Things break, servers crash, software has bugs, water is wet, so to keep the sky blue and clear we need backups. While backing up a disk index in Sphinx is basically a matter of copying files around, properly backing up RT is a little bit more complicated. (But only a little bit.) Read on to find out how binlogs, periodic flushes, and two newly added 2.0.2-beta features (FLUSH RTINDEX and ATTACH INDEX) can all work together for backup/recovery.
Let’s begin with a quick reminder as to how a RT index works. It keeps a bunch of disk chunks, which are just plain old files. But it also keeps an in-memory RAM chunk that is responsible for all the Real Time magic. The RAM chunk data occasionally gets written to disk, but only occasionally. So to avoid losing that in-memory data in the event of a crash, Sphinx appends all your changes to a binary log, and replays that log after a crash to recover the data.
Hooray, you now know how to implement some RT backup scheme. Just backup all the RT data files and the binlog are you’re all set. Only, make sure that you backup binlog files after you backup the data files. The other way around, you risk ending up in a situation where that rt.ram file was written after you backed up the binlogs, and the two are inconsistent. Also, binlogs are appended to all the time, so it generally makes sense to backup them last to have the latest and greatest data backed up.
Speaking of backing up the files in proper order, something resembling the following example should work. It backs up (immutable) disk chunk data first, then the more dynamic RAM chunk, K-list, and header files (.ram, .kill, and .meta files respectively), and then the binlog files:
tar zcvf /backups/myrtbackup.tar.gz /sphinxdata/myrtindex.{*.sp*,ram,kill,meta} /sphinxdata/binlog.*
This scheme just above is easy, operational, and gets the data backed up all right. However, there might be an issue when recovering from that backup. Remember that Sphinx needs to replay the binary log? And what if the binary log has many GBs of updates which were not yet written to .ram file on disk? Replaying that is going to take time, which is not great when you want to get back up and running ASAP. Fortunately, we can further improve the backup scheme and fix that.
First, you can configure searchd to periodically flush .ram chunks to disk, there is a rt_flush_period config directive for that. That will implicitly limit the (total) binlogs size, because every time .ram is written to disk, old and now-redundant binlogs get killed. The recovery time will be implicitly limited accordingly. Worst case scenario, Sphinx would need to replay rt_flush_period minus 1 seconds worth of changes when doing recovery from backup.
Second, starting with 2.0.2-beta you can issue a FLUSH RTINDEX statement just before you begin that backup. FLUSH forces an immediate save of .ram chunk for a given index, and the subsequent binlog cleanup. It also blocks until all the work is done. Bottom line? If you backup just after FLUSH RTINDEX, starting up from that backup is going to be quickest, recovery is going to take a few seconds tops. You can easily fire it from the command line, too:
echo FLUSH RTINDEX myrtindex | mysql -h0 -P9306
tar zcvf /backups/myrtbackup.tar.gz /sphinxdata/myrtindex.{*.sp*,ram,kill,meta} /sphinxdata/binlog.*
Last but not least, there’s now an ATTACH INDEX statement, also added in 2.0.2-beta. It’s not really about backups, and was intended for huge initial data imports, but it’s also useful for disaster recovery. I mean, if all hell just broke loose, and aliens attacked and destroyed not only your Sphinx production cluster but also all the datacenters you kept backups at, ATTACH INDEX lets you recreate your RT index much quicker than before. The trick is, you just build a regular disk index, and then convert it to a RT index in a matter of seconds.
Bottom line, crashes happen, they tend to occur at the worst times, so make sure your backups are in place and set up in such a way that your recovery is as painless as possible. And, of course, I wish you never have to use those backups.
| « November 16, 2011. Sphinx 2.0.2-beta is out | December 29, 2011. Sphinx 2.0.3-Release » |
Can’t believe. Is it really works?
Well, try and tell us if it doesn’t..