Developing Operability
Richard Crowley
Betable operations
Richard Crowley
Betable operations
(probably not OCD)
Linux development environments
No automation
One development environment
Lots of automation
No tests
Staging used production data
Reduce the distance between development and production
Gambling as a Service
Heavily regulated
Green field
(or however you capitalize it)
Dev : Ops :: BD : COO, some PMs
Not good at multitasking
“Ops guy” obscures the meaning
Gift cards for Heroku Dynos
at Wal-Mart
Rube Goldberg machines
“Orthogonality makes it easier to understand what happens when things combine.” - golang.org/doc/faq
No moving targets
The degree to which software behaves unsurprisingly in surprising situations
Hint: it starts in development
Thousands
Unit and integration
At least services and not “services”
Six services
API-driven
node-betable-admin
node-betable-accounting
node-betable-cassandra
Except when testing their client libraries
Especially service client libraries
Especially dependent service failures
Predictable and standard command-line interface
Yep
Build numbers
Git commit SHAs
Semantic versions
fpm and freight make things easy
Fat packages keep things orthogonal
How do I know the package with a Git commit in its version actually contains that Git commit?
Give up and deploy Git repositories
The analogue to fat packages
Compilers everywhere!
Compilers everywhere!
mvn package
restart betable-accounting
master to CI to staging to production
Running process to source code to version
Composability
Orthogonality
package { "betable-accounting":
ensure => latest,
}
puppet apply manifests/site.pp
git push git@prod01.betable.com:betable-accounting.git
Can’t be bothered to git pull
But not too up-to-date
The important part of every engineer’s laptop should be identical
git pull origin master puppet apply manifests/site.pp
puppetme
Reduce the distance between development and production
Mutiny brewin’
Don’t care to manage Spotify
Reduce the distance between development and production
puppetme anytime, anywhere
package.json
pom.xml
bootstrap.sh if all else fails
integrate anytime, anywhere
You’re likely only working on one thing at a time
For what’s not under development
For everything else
If you do this in production, people will laugh at you
All your logs in one terminal
This is both good and bad
git push origin master
Initiates Jenkins build, deploy to staging, and integrate
Coworkers run puppetme at their leisure
You’re not going to watch them, are you?
SIGTERM
Canonical’s init(8) replacement
Direct parent supervision
File descriptor inheritance
SCM_RIGHTS and unix(7)
Not typically compatible with direct parent supervision
WANT
Easier to achieve
No one here has only one server, right?
Manage upstream services
Let upstream services manage themselves
We get more out of logs than graphs
We log full requests and responses
Reduce the distance between development and production
Be mindful of how you deploy