Take a BlaBlaCar to the stars with rkt !

Simon Lallemand
System Engineer
@slallema
Agenda

BlaBlaCar Facts, Figures & History

Key principles Directions for driving change

New infrastructure
ecosystem 100% rkt powered carpooling
Facts and figures

40 million
members 21 million mobile
app downloaded
12 million travelers
per quarter
(iPhone + Android)

CO2
Currently in
22 countries
France, Spain, UK, Italy, Poland,
Hungary, Croatia, Serbia, Romania,
Founded 1 million Germany, Belgium, India, Mexico,
tonnes less CO2 in the The Netherlands, Luxembourg,
in 2006 past year Portugal, Ukraine, Czech Republic,
Slovakia, Russia, Brazil and Turkey.
Our server growth

The evolution of the infrastructure from the begining to now:

Baremetal Baremetal
Web Dedicated Baremetal Baremetal Baremetal Baremetal 14 racks 17 racks
hosting servers 1 rack 3 racks 5 racks 8 racks 150 servers 300 servers
2 DC 3 DC

2006 2008 2010 2012 2013 2014 2015 2016

~300 bare-metal servers
~400 container images
4000+ running containers
Tech evolution

The major tech changes in our infrastructure :

Hardware
Virtu Chef Foreman Containers
uniformization

2015
2012 2013 2014 2015
2016
Key principles
Leading the industrialization at BlaBlaCar
Metal is invisible
Decouple hardware management and
what runs on it
Route everything
Keep the network simple and scalable
Remove snowflakes
Be service oriented
New Infrastructure Ecosystem
100% rkt powered carpooling
Containers
For everything !
CoreOS Container Linux
On 100% of our new servers
rkt
as container runtime
New Infrastructure Ecosystem
The tools
 Harmonize the way we build

Quick build
One way of doing things
Easy to understand for newcomers
As little code replication as possible

dgr Templating at container start

A good integration with rkt
Container build
and runtime tool
github.com/blablacar/dgr
dgr : build directory of an ACI

aci-manifest.yml
attributes
redis.yml
A standardized structure
runlevels Ease maintenance and teamwork
build
install.sh Inspired by config management
templates Separation of templates, attributes and scripts.
etc
redis
redis.conf.tmpl
dgr : ACI manifest

name: redis:0.1 Simpler manifest format

Fill only what is important
aci:
app: YAML <3
exec:
- /usr/bin/redis-server One process per ACI
Composition is done only with POD
- /etc/redis/redis.conf
dgr : ACI manifest

name: redis:0.1
aci:
app:
exec:
- /usr/bin/redis-server
- /etc/redis/redis.conf Use of dependencies
Composition
dependencies: Lighter images
- debian:8.6
dgr : runlevel build

aci-manifest.yml
attributes
redis.yml
runlevels Scripts executed in the container
build Build from inside the container with all the
install.sh dependencies
templates
etc
redis
redis.conf.tmpl
tests
dgr : runlevel build

#!/bin/bash Install packages and stuff

For debian based containers you can install
apt-get install -y redis-server packages using apt-get for instance.
dgr : runlevel builder

name: example.com/aci-redis-dicator:1
builder:
dependencies:
- example.org/aci-go Build from outside of the container
- example.org/aci-git Dedicated builder image with go & git
aci:
app:
exec:
- /bin/dictator
dgr : runlevel builder

#!/dgr/bin/busybox sh
Builder script
git clone \ Clone, build and copy binary to
https://github.com/blablacar/redis-dictator target ACI
cd redis-dictator Only /bin/dictator in final ACI
go build
cp dist/dictator ${ROOTFS}/bin
dgr : runlevel builder

name: example.org/gentoo-redis:1
builder:
dependencies:
- example.org/gentoo-stage4 Other example
Builder with gentoos stage4
aci:
dependencies:
- example.org/base
app:
exec: [ /usr/bin/redis-server ]
dgr : runlevel builder

Install packages from outside

Use emerge to install your app and its
#!/dgr/bin/busybox sh dependencies.
emerge -v redis-server Redis with dependencies only in the final
ACI
dgr : templates & attributes

aci-manifest.yml
attributes
redis.yml
runlevels Render configuration files
Templates stored in the aci
build Default attributes stored in the aci
install.sh Overridable when used as dependencies
templates Overridable by environment var

etc
redis
redis.conf.tmpl
dgr : templates & attributes

# templates/etc/redis.conf.tmpl
daemonize no
port {{ .redis.port }}

# attributes/redis.yml
default:
redis:
port: 6379
dgr : runlevel prestart

aci-manifest.yml
attributes
redis.yml
runlevels Initialize container
prestart-late prestart-early and prestart-late scripts
Before and after templating
10-init-db.sh Initialize environment before exec
templates
etc
redis
redis.conf.tmpl
dgr : testing

aci-manifest.yml
attributes Testing
runlevels Bats as default tester
templates
tests wait.sh
wait.sh Wait for service to be ready
my_cool_tests.bats
dgr : testing

#!/dgr/bin/bats -x

@test "Redis should be running" {

run bash -c "ps -aux | grep redis-server"
[ "$status" -eq 0 ]
echo ${lines[0]}
[[ "${lines[0]}" =~ "/usr/bin/redis-server" ]]
}

@test "Redis should listen on port: 6379" {

run bash -c "netstat -peanut | grep redis-server"
[ "$status" -eq 0 ]
[[ "${lines[0]}" =~ ":6379" ]]
[[ "${lines[0]}" =~ "redis-server" ]]
}
dgr : subcommands

~ # dgr init
~ # dgr try
~ # dgr build
~ # dgr test
~ # dgr install
~ # dgr push

github.com/blablacar/dgr
 INSERT
Pods Services and Environments
LOGO
HERE Different services that use the same pods
Environments (1 per DC for prod + dev)
Abstraction of fleet commands
ggn
Manage services
in fleet clusters
github.com/blablacar/ggn
ggn

prod-dc1 Single directory tree

attributes Describes the services
services The environments
Versioned in git
redis-quota
attributes
service-manifest.yml
unit.tmpl
redis-rating
preprod

...
ggn : service manifest

containers: Define the service

ACI images and versions combined
- blablacar.com/aci-redis:3 to make a pod
- blablacar.com/aci-redis-exporter:1 Nodes composing the service
nodes:
- hostname: redis-quota1
- hostname: redis-quota2
- hostname: redis-quota3
ggn : attributes

# attributes/redis.yml
override:
redis:
maxmemory: 4GB Override attributes
Inject run context attributes as environment var
TEMPLATER_OVERRIDE={redis:{max
memory: 4GB}}
ggn : unit.tmpl

[Unit]
Description=Redis POD {{.

[Service]
KillMode=mixed
Restart=always

ExecStart=/usr/bin/rkt run \

--set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}'
{{.acis}}
ggn : run a POD

$ ggn prod-dc1 redis-quota update

$ ggn prod-dc1 redis-quota1 journal -f

github.com/blablacar/ggn
 Dynamic topology of services

Scalability requires service discovery

Service We started with smartstack of AirBnB

Since then we improved it and rewrote
it in go
discovery
with nerve & synapse
github.com/blablacar/go-nerve
AirBnBs smartstack github.com/blablacar/go-synapse
Service Discovery
Zookeeper

go-synapse go-nerve

/database/node1

/database
go-synapse watches zookeeper
service keys and reloads haproxy
if changes are detected go-nerve does health checks and
reports to zookeeper in service
keys

HAProxy

node1
Applications hit their local
haproxy to access backends
Our infrastructure ecosystem
front_1

nerve nerve

php mysql-main_1
Service Discovery
create
nginx nerve zookeeper
Service Codebase synapse

dgr build monitoring mysqld

synapse monitoring
Container Registry store

ggn run rkt PODs

Distributed init system fleet etcd fleet machines

1 type of hardware
Hardware CoreOS
host bare-metal servers
3 disk profiles
Work In Progress

( Were hiring )
 @slallema
@BlaBlaCarTech
Thanks!
@BlaBlaCarTech
BlaBlaTech.com