DevStructure Redux

Five years ago Matt Tanase and I built a tool called Blueprint under the banner of a company we called DevStructure.  We believed configuring development environments was too difficult and that existing configuration management tools were too hard to use.  This, we concluded, resulted in development environments that were too far removed from their production environments and thus offered little proof that software will work well once deployed to production.

We started with a paid semi-hosted development environment and eventually backed off to a set of open-source tools we hoped to support with production hosting offerings.  To cut a long story short, we never made it that far.  As a business, DevStructure was a failure.

Early this morning while standing in the dark gently bouncing my two-month-old daughter back to sleep I was thinking about what we built and why we failed.  If I was ever bitter about our failure as a business I’m long since over it but now I see the world through DevStructure-colored glasses; I see shades of what we tried to do in all sorts of tools being released all across our industry.  Perhaps that’s confirmation bias but it keeps me thinking about how I’d approach the problem today.  And let’s be very clear:  The management of development environments that are reliable predictors of how software will behave in production environments is still a major threat to software quality five years later.

However, since I adore my job and since chasing this idea would be a costly distraction for us right now, DevStructure Redux isn’t going to be coming from me anytime soon.  Consider this a free idea or idle ramblings, whichever you choose.

Blueprint was trying to put a better user interface on configuration management and even the idea of infrastructure as code but the tools we built were quite hostile towards code review.  Nowadays, I believe code review is essential so I’ve resumed seeking code-centric user interface improvements.

Today’s configuration management suffers as a user interface in part because it is built on command-line tools readily available on the servers being managed.  Most operators can resist the temptation to apt-get install ... in production but basically no one can resist in a development environment.  And why would they?  The edit-save-run cycle is a drag and over time most Chef or Puppet runs tend towards oh-my-god-this-is-slow.

So this is where my mind was at 5 o’clock this morning.  I thought that if configuration management was a concern of the operating system then the interface it ultimately exposed could be made more amenable to code review.  Here we go with the hand-waving.

Code review-conducive package installation is the easiest to tackle.  Whatever the interface, there just needs to be a file or directory tree naming the packages to be installed and specifying their versions.  When changes are made the operating system needs to add and remove packages to suit. Changes may be allowed via editor, command-line tool, or pulling changes from an upstream source code repository.  Hypothetically this could be as simple as files like /etc/packages/mysql-server that contain the desired version number.  Version control /etc and send all the changes through code review.

Immediately after installing a package folks tend to configure the newly-installed software.  Detecting meaningful changes to these files was always a major problem for Blueprint.  Well-behaved packages mark their configuration files and include hashes of the packaged content to enable detection of changes.  In my opinion we’d be better-served by packages devoid of configuration files.  Programs would thus be required to function sanely by default or fail immediately at the author’s discretion.  Imagine how wonderful it would be if this requirement were imposed on all packages:  /etc would be empty on a fresh operating system installation!  If the defaults don't suit you, create /etc/my.cnf containing your desired configuration and commit it, too.

While we’re moving configuration files out of packages, let’s get rid of maintainer scripts, too.  I want to control when services are started and restarted.  I want to control user and group IDs.  And I reserve the right to manage these actions and resources differently in my infrastructure than you do in yours.  Store service dependencies in systemd units like /etc/systemd/system/mysql.unit.  Store users and groups in /etc/passwd and /etc/group but without all the cruft or better yet in /etc/passwd.d and /etc/group.d directories.  And version them just like packages and configuration files.

Oddly, despite my desire to move maintainer scripts from packages out into configuration management, I also want to get rid of exec or execute resources that allow today’s configuration management tools to run arbitrary commands.  Puppet’s documentation has long said that if you find yourself using a lot of exec resources you’re wrong and should be writing your own types and providers.  Everyone ignored that because exec resources are easy.  The real problem underlying all of this, though, is that it is easier to manage many aspects of a server today via commands than via files (iptables(8), mount(8), sysctl(8), to name but a few) and tough to manage small changes to large sets of defaults (basically every default configuration file).  This needs to change if we're to make anything but superficial lipstick-on-a-pig progress.  It's no accident this is starting to sound a little bit like Plan 9.

If we were starting DevStructure again today our company would look a whole lot more like CoreOS than like Chef or Puppet Labs.  My vision for a minimal operating system, however, is quite different than that of CoreOS.  I envision a base installation chock full of all the tools I could ever need to introspect the system.  I envision a package manager that enables and encourages fat packages, explicit dependency versioning and vendoring, and then gets out of the business of configuration and orchestration.  I envision configuration management being the only interface for adding and removing software or changing its configuration.  I envision that all aspects of the operating system will be designed to be version-controlled.  I envision packaging to be the obvious best way to deploy your own software into production.

DevStructure Redux would sell an appliance or service (customer’s choice) that turn this minimal operating system distribution into a streamlined development-to-production pipeline.  At a minimum, it would need to accept and serve package artifacts, integrate with test runners and code review tools, store package artifacts installed from the Internet so precise versions are available to all servers, manage server inventory, and orchestrate reconfiguration and deployments.  The toughest product challenge is, I think, how to infiltrate brownfield environments.  Failure is imminent if this product gives anyone the impression it’s an all-or-nothing proposition; customers need to derive value immediately and in proportion to their investment.

Alas, I will not have the opportunity to take up this cause again for a long time because I love my job.  Our development environments at Slack are not terribly portable to other applications or workflows but they are amazingly production-like and highly optimized for our 100+ engineers getting work done.  Seeing this scale does nothing but reinforce my belief that development environments should be as close of a match to production as possible.  I’m not satisfied with the latest batch of abstractions over the ever more complex software systems being built today but I’ll keep searching.  I sincerely hope someone builds what I’ve described above because I’d love to buy it.