Redis is easy. Redis is trivial. Redis is HARD!

We all love Redis. It handles all of our key/value and queuing needs.

Single server Redis is easy

// redis1.ourdomain.io //
sudo apt-get install redis-server

Primary/slave replication is trivial

// redis2.ourdomain.io //
sudo apt-get install redis-server

# edit /etc/redis/redis.conf
slaveof redis1.ourdomain.io 6379

sudo service redis-server restart

Automatic service discovery and failover is HARD

The minimum number of servers for automatic service discovery and high availability using Sentinel is 3. So we need another server.

// redis3.ourdomain.io //
sudo apt-get install redis-server

# edit /etc/redis/redis.conf
slaveof redis1.ourdomain.io 6379

The default redis-server package on Ubuntu does not include an init script for redis-sentinel, so on each server we have to set that up.

cd /etc/init.d
sudo touch redis-sentinel
sudo chmod +x redis-sentinel

Put the following into /etc/init.d/redis-sentinel on each server:

#!/bin/bash
### BEGIN INIT INFO
# Provides:          redis sentinel
# Required-Start:    $all
# Required-Stop:    $all
# Default-Start:    2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts redis sentinel
# Description:      Starts redis sentinel using start-stop-daemon
### END INIT INFO

NAME=redis-sentinel
BIN=/usr/bin/redis-server
SENTINEL_PID=/var/run/redis/sentinel.pid
CMD=$1

start() {
        echo "Starting $NAME ..."
        exec 2>&1 $BIN /etc/redis/sentinel.conf --sentinel | logger -t sentinel &
        echo $! > "${SENTINEL_PID}";
}

stop() {
        PID=`cat $SENTINEL_PID`
        echo "Stopping $NAME ($PID) ..."
        kill $PID
}

restart() {
        echo "Restarting $NAME ..."
        stop
        start
}

case "$CMD" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                restart
                ;;
        *)
                echo "Usage $0 {start|stop|restart}"
esac

Now we need to setup the redis-sentinel service to start at boot on each server.

sudo apt-get install sysv-rc-conf
sysv-rc-conf redis-sentinel on

Now in /etc/redis/sentinel.conf on each server add the following lines:

 daemonize yes
 pidfile "/var/run/redis/sentinel.pid"
 loglevel verbose
 logfile "/var/log/redis/sentinel.log"
 sentinel monitor mymaster redis1.mydomain.io 6379 2
 sentinel down-after-milliseconds mymaster 10000
 sentinel failover-timeout mymaster 60000
 sentinel parallel-syncs mymaster 1

Now in /etc/redis/redis.conf on each server add the following lines:

  repl-ping-slave-period 5
  slave-serve-stale-data no
  repl-backlog-size 8mb
  min-slaves-to-write 1
  min-slaves-max-lag 10

Assuming that all worked, let's try and start all redis-server and redis-sentinel daemons on each server:

 sudo service redis-server start
 sudo service redis-sentinel start

Testing it all out

Let's take down the primary Redis server for 2 minutes, and see if Sentinel kicks in.

 // redis1.mydomain.io //
 redis-cli -p 6379 DEBUG sleep 120

And finally check which server is the Redis primary on redis2.mydomain.io:

 // redis2.mydomain.io //
 redis-cli -p 26379
     127.0.0.1:26379> SENTINEL get-master-addr-by-name mymaster