Things I will not tell you are AWS best practices

Immutable infrastructure
Playing CNCF bingo
Trying to avoid vendor lock-in by only using EC2 and S3
Micromanaging Reserved Instances
Active-active multi-cloud
Ruby on Rails
Not Ruby on Rails
Serverless
Kubernetes

Have lots of AWS accounts

Accounts are the one true unit of isolation in AWS
Cost allocation by account is often all you need, too

By the way: I work on Substrate <https://substrate.tools/>, the whole point of which is to make having lots of AWS accounts so easy that everyone does it

Isolate environments

Reduce the blast-radius of changes
Protect production data from development code
Ensure you can test all kinds of changes in development before making them in production

Isolate backup data

Prevent even a malicious attacker from deleting your backups
Ensure a security breach can’t become a company-ending event

Story time: Ma.gnolia’s influence on Slack

Isolate services

Further reduce the blast-radius of changes
Prevent one service from impacting another
Prevent one team from stepping on another’s toes
Give each service their own AWS rate limits and service quotas

Use Shared VPCs

Don’t waste time figuring our VPC Endpoints and PrivateLink
Don’t pay per byte to cross VPC boundaries
Use security groups just like you expect and get on with things
And use standard VPC Peering if you run in multiple regions

Use three availability zones

Four AZs use IPv4 addresses more efficiently but
Regional databases use three AZs
Almost every region has three AZs but only one region has four

Configure your one free CloudTrail

Make it a global, multi-region trail in your management account
Store the logs in S3 for a rainy day
Definitely don’t create a second trail because it will be very expensive

Story time: redacted

Configure GuardDuty

Low-effort, potentially high-reward security monitoring
Checks a compliance box
Might save your bacon!
Probably won’t cost much at all

Particularly pay attention to findings like: InitialAccess:IAMUser/AnomalousBehavior, Persistence:IAMUser/AnomalousBehavior, Policy:IAMUser/RootCredentialUsage, and UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration

Avoid IAM users for humans

Require manual onboarding and offboarding
Have long-lived access keys that will end up sitting plaintext in home directories waiting to be stolen
Instead, watch for GuardDuty’s Persistence:IAMUser/AnomalousBehavior finding

Story time: Bob

Use IAM roles and an IdP instead

Humans assume them via your OAuth OIDC or SAML identity provider
Services assume them via EC2 instance profiles, ECS container instance role, EKS OAuth OIDC integration, etc.
IAM roles are your one mechanism for expressing and propagating identity
Their credentials automatically expire

Simplify IAM policies

Grant broad permissions within an account and few, if any, permissions between accounts

{
  "Version": "2012-10-17",
  "Statement": {
    "Action": "*",
    "Effect": "Allow",
    "Resource": "*"
  }
}

{
  "Version": "2012-10-17",
  "Statement": {
    "Action": "sts:AssumeRole",
    "Effect": "Allow",
    "Principal": {
      "AWS: [
        "arn:aws:iam::123456789012:role/Example"
      ]
    }
  }
}

Skip AWS Code* and use GitHub Actions

AWS Code* services are good at assuming IAM roles and kind of bad at everything else
GitHub Actions has a joyous and magical OAuth OIDC integration for getting AWS credentials
And most of your other CI/CD tasks are if not elegant than at least well understood

Even if you ignore this advice, don’t go running a Jenkins server

Use any high-level AWS service that saves you time

Avoid “undifferentiated heavy lifting”
Hosted databases, especially Aurora, will get you a long way
Serverless (i.e. Lambda) for long-tail and, especially, internal services
Even oldies like Elastic Beanstalk can be real time-savers
Don’t pretend your infrastructure is portable because you “only” use EC2 and S3

Story time: redacted

Savings Plans

Not the absolute best discounts available
But can be had for next to zero effort
Wait until you’re spending more than $5M per year to negotiate bigger “EDP” or “cross-service” discounts

tl;dr

Have lots of AWS accounts to isolate environments, services, and sensitive data
Have a development/testing version of everything
Share VPCs between accounts to network services together
Configure CloudTrail and GuardDuty
Avoid IAM users in favor of IAM roles and an IdP
Use any high-level AWS service that saves you time
Buy Savings Plans to save some easy money

AWS for startups

The big little things everyone has to get right

How would I know?

The big picture