2024.01 Release Notes Notes

New year, new schtick. Substrate 2024.01 was released yesterday. In honor of the self-inflicted magnitude of that effort, I’m resolving to start writing commentary on the releases and the tangents I took along the way. Monthly releases, monthly release notes, and monthly release notes notes.

I’ve long admired Simon Willison’s TIL and weeknotes writing. And, since I’ve been able to keep up Substrate’s monthly release cadence since June 2020, I figure I’m up to this challenge.

TIL, in homage

TIL about aws ec2 wait instance-terminated --instance-ids ... which I’ve for years implemented myself, poorly, and at great risk of smashing into rate limits.

Mac the unshavable yak

The plan for the Substrate 2024.01 release was to manage VPC peering directly (in Substrate’s Go codebase) instead of by generating and applying Terraform code. This is the not-first-maybe-second-but-definitely-not-last step in removing generated Terraform code from Substrate, which I’m doing to make Substrate more robust, decouple Substrate being able to manage the basics from whatever state folks leave their Terraform code in, and free folks to upgrade (or not) Terraform and its providers whenever and however they see fit.

The plan was simple. Too simple. We couldn’t help ourselves. So we creeped that scope up to whatever level working-late-on-New-Year’s-Eve is. For one big change, we adopted Cobra to dispatch subcommands, parse options, and handle autocomplete. More on that in the next section. For the other, we added an integration with the macOS keychain.

Substrate mints temporary AWS credentials, a massive security improvement already, and then goes to some trouble to discourage folks writing even those temporary credentials to disk. It’s harder to steal credentials from a running process’ environment than from a file in the victim’s home directory, the thinking goes. But also it’s really annoying to have to mint new credentials in every shell. We wanted to share credentials between Substrate processes without disclosing them to other processes with filesystem access.

Enter the macOS keychain, which allows a particular binary (apparently identified by both name and code) to get one-time or persistent access to certain data. We taught every Substrate process to look first in the environment and second in the macOS keychain for AWS credentials. It works brilliantly. Hats off to Keybase for publishing Go bindings to the C library that provides access to keychains.

I finished the keychain integration on December 11 and all but forgot about it in the run-up to release.

So I tag the release and set the computers to work building, packaging, and announcing it. It explodes because it can’t cross-compile macOS binaries on Linux anymore because I linked to a C library. CGo strikes again.

No matter, I think. I bought a Mac for just such a purpose. Wrong again! I can build for ARM but not for Intel. I ask Travis, “Do you have an Intel Mac?” He does not.

EC2 to the rescue? EC2 to the rescue! It took some research to confirm but, yes, mac1.metal means Intel and mac2.metal means ARM. I remember from the launch that the EC2 Mac instances were quite a chore to use and quite expensive. I priced it out and thought we could do a build for about $5. Not great but a price I’m quite willing to pay.

Joke’s on me! Hidden in the fine print, AWS says that you have to keep EC2 Mac instances for at least 24 hours. A build will actually cost about $50. Travis is on eBay shopping for Intel Mac Minis. Here’s how it should actually be presented in the AWS documentation:

EC2 Mac instances are billed for a minimum of 24 hours

Big. Also, wow do they take a long time to launch! It’s things like this that drive people to run homelabs.

Thankfully, the EC2 Macs could actually build Substrate, CGo keychain integration and all, and Substrate 2024.01 is out, with an absolutely cracking new experience for macOS users.

Cobra

Cobra is a Go library for constructing command-line tools that use the subcommand pattern that version control tools have used since CVS. It’s good kit and I’m happy we adopted it, even if it made for a busy December.

I really liked my old subcommand dispatch code, though. It worked in much the same way as Go’s testing package. Functions with the proper signature were collected into a map[string]func(...) for main to use when dispatching subcommands.

Here’s the thing, though: Of the three jobs that libraries for constructing command-line tools do — autocomplete, help/usage documentation, and actually executing code — actually executing code is trivial to the point of almost not mattering. Substrate’s first 3½ years featured afterthought autocomplete and usage messages that were hand-typeset with fmt.Printf. And nesting subcommands completely broke it.

Cobra is exceptionally good at autocomplete and documentation. That it’s annoying when actually executing code is a price worth paying.

Boxed pointer types in Go

In the course of transitioning Substrate from my homegrown subcommand dispatching and the standard library flag package to Cobra and the “drop-in replacement” pflag package, I found myself writing a lot of types that were all essentially boxes around a primitive type (mostly bool and string) with a Set method and a few others that control completion and documentation.

My micro-optimizer personality had a little fight with my readable-code personality that got me wondering whether there was actually any performance difference between a boxing type that uses a pointer and a boxing type that uses a struct with a single field.

Even with all my years as a C programmer, I still look at this twice and wonder whether I’ve done it right:

type String string

func (s *String) Set(v string) { *s = String(v) }

func (s *String) String() string { return string(*s) }

To my eye, this is much more obviously correct:

type Struct struct{ s string }

func (s *Struct) Set(v string) { s.s = v }

func (s *Struct) String() string { return s.s }

Benchmark results validate the desire for clarity, in this case, at least:

=== RUN   TestInt
--- PASS: TestInt (0.00s)
=== RUN   TestString
--- PASS: TestString (0.00s)
=== RUN   TestStruct
--- PASS: TestStruct (0.00s)
goos: linux
goarch: amd64
pkg: github.com/rcrowley/src/play/boxed-pointer-types
cpu: 12th Gen Intel(R) Core(TM) i7-1260P
BenchmarkInt
BenchmarkInt-16       	1000000000	         0.2416 ns/op
BenchmarkString
BenchmarkString-16    	1000000000	         0.3482 ns/op
BenchmarkStruct
BenchmarkStruct-16    	1000000000	         0.3595 ns/op
PASS
ok  	github.com/rcrowley/src/play/boxed-pointer-types	1.059s

For bonus curiosity-satiation, I made a version of the benchmark with a boxed int, too, which shows that the size and complexity of the typed being boxed has a lot more to do with the box’s performance than whether the box uses a pointer or a struct field.

type Int int

func (i *Int) Set(v int) { *i = Int(v) }

func (i *Int) Int() int { return int(*i) }

Could you possibly come up with a more cliché New Year’s Resolution?

No. But I’m really going to stick with it. I swear.