WhoTF am I?
- Led Slack ops from 2014 to 2020
- It was my fault when Slack was down
- (It may sometimes still be)
- Made EKM, started Disasterpiece Theater, got all of engineering on call
- Now I help startups use AWS the right way via Substrate
- Mostly, I tell them to have more AWS accounts
Substrate
https://substrate.tools/
The right way to use AWS, designed for startups with immediate security and reliability needs, and informed by everything I did at Slack
- Configures AWS Organizations, IAM, and VPC
- Manages all your AWS accounts
- Sets up Terraform
- Best cross-account IAM role management tools in all the land
AWS at Tiny Speck in 2009
- EC2 and S3
- PHP everywhere they could, Java everywhere they had to
- Glitch, the game
- AWS accounts: 1
AWS at Tiny Speck Slack in 2014
- Still EC2 and S3
- Still PHP everywhere we could, Java everywhere we had to
- But now it was all Slack, all the time
- And it was now my fault if it was down
- AWS accounts: 1
“We only use EC2 and S3, so we can leave whenever we want”
(No one in particular said that but the sentiment was in the air.)
EC2 is lock-in
- What are EC2 and GCE?
- Are they more than Linux boxen as a service?
- One time, Slack tried to use GCE, too…
EC2 is lock-in
The peculiarities of the networks make EC2 and GCE fundamentally different
Rate limits and service quotas
- Near constant friction in our tools
- More alarmingly, when you’re rate-limited, your load balancers don’t react as quickly
- Sometimes, we had all the i2 or i3 instances
Rate limits and service quotas
The best way to avoid running into rate limits and service quotas is to have lots of AWS accounts
Reserved Instances bin packing
- Try your best to standardize on very few instance families
- Scaling reservations up and down in size is theoretically possible but practically difficult
- In 2024, thankfully, you can opt out of all of it by buying Savings Plans
Reserved Instances bin packing
Save yourself less money but a ton of time by buying Savings Plans instead of Reserved Instances
Backups and existential dread
- It would be bad if an attacker deleted Slack’s infrastructure
- It would be existentially bad if an attacker also deleted Slack’s backups
Backups and existential dread
Store your backups in a separate AWS account and don’t give anyone enough privileges to delete them
SecOps
- A second clear case of absolute isolation being an absolute requirement
- This was the sink for kernel audit logs from every EC2 instance we ran
- And I still don’t know what all they were monitoring
SecOps
Use an AWS account boundary to ensure that your all-seeing security eyes can’t themselves be seen
Futile separation of dev and prod
- Limped along with less awesome isolation for quite a while
- AWS Organizations didn’t launch until 2017
Futile separation of dev and prod
Accounts are the one true unit of isolation in AWS
2015 security incident and the move to VPC
- EC2 Classic to VPC
- Production first
- Three days
2015 security incident and the move to VPC
Security monitoring, security audits, and disaster recovery exercises are very important
Kubernetes vs Mesos vs me sitting on everyone’s hands
- The cost of being wrong is going it nearly alone
- And we already had that going with Hack/HHVM
- Waiting until a clear winner emerged was prudent
Kubernetes vs Mesos vs me sitting on everyone’s hands
Not only did waiting ensure we chose the winning path but we reaped the benefits of tooling advances like EKS that the earliest adopters didn’t
Project White Castle
- Lots of AWS accounts
- Okta integration to get into each one
- Terraform to configure each one
- VPC Sharing to bring them all back together
Project White Castle
Accounts are the one true unit of isolation in AWS and VPC Sharing is the smartest way to network them back together
AWS at Slack in 2020
- Using a lot more AWS-managed services
- Hosting a smattering of services in Kubernetes
- PHP became Hack, Go joined Java (sorry / you’re welcome), Elixir got called up
- AWS accounts: O(10)
Takeaways
- Everything about every cloud provider is lock-in
- So use everything that’s useful in your cloud provider
- Accounts are the one true unit of isolation in AWS