2024.02 Release Notes Notes

Substrate 2024.02 is out, most notably featuring the end of Terraform as a load-bearing component of Substrate. Instead of setting folks up to succeed with Terraform and then partially sabotaging that success by generating Terraform code into those very same directories, we just…set folks up to succeed with Terraform. Bold strategy, Cotton; let’s see if it works out for ‘im.

Technological death knells

In much the same way your phone goes on the fritz the day you order a new one, technologies whose days are numbered in Substrate staged a bit of a rebellion today on the way to the 2024.02 release.

Terraform, probably as a result of the massive recent growth in the number of AWS organizations Substrate tests against in parallel, began to corrupt its .terraform.lock.hcl files, claim hashes of long-stable provider binaries had changed, and even crash. I was able to trace this behavior to races when the plugin_cache_dir directive is set in ~/.terraformrc. Without that setting, Terraform uses a lot more disk space by storing duplicate copies of provider binaries but at least it doesn’t crash.

More mysteriously, AWS CodeBuild lost its ability to clone private repositories from GitHub. I tried all the usual variations of “turn it off and turn it back on again” and got absolutely nowhere. I think it heard me conspiring to replace it with GitHub Actions, where I can now get both Intel and ARM Mac builds for less than $50 each.

The perils of mixing local and remote state

This is also about Terraform, yes, but it’s about “state” as in data that’s persisted because I want it back later, not “state” as in Terraform state files.

It’s awkward that Terraform persists some state locally and some remotely. Current configuration of some esoteric AWS resource? Remote. Version and hash of the provider binary that manages it? Local. Committed to source control, possibly, which I guess makes that state eventually, potentially consistent remote state.

Substrate is awkward in this way, too. For historical, accidental reasons (the worst kind of reason, at least in software) it stores metadata on networks locally and metadata on accounts and roles remotely. This is confusing! And it’s also the most significant blocker to offering more features via web UIs as well as / instead of command-line tools. The future is bright.

I think the design lesson in this is to pick one — either local state with the option to upload/synchronize (in the local-first spirit) or remote state with local caching and really buttoned-up cache invalidation. Mixing the two only leads to awkwardness, either when others see one’s changes sooner than expected or others miss one’s changes because they were never uploaded/synchronized.

A look ahead

I’m starting to build a few Substrate-managed AWS organizations as demos. Imagining myself, fresh on my first week at any job I’ve ever had, putting together this caliber of infrastructure, this quickly — it fills me with the high of doing a damn good job and knowing it.

Code and commentary coming soon.