Read more

Memcache: Your cache node may degenerate over time, check your settings

Tobias Kraze
January 28, 2013Software engineer at makandra GmbH

We recently had a problem on a Memcache cluster, where one of the servers showed a significantly worse cache hit rate and a lot more evictions.

Illustration online protection

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

  • Prevents you from data breaches and liability risks
  • Upgrade at your own pace
  • Works with modern Rubies
Read more Show archive.org snapshot

It turned out that the only reason was that the server was running for a few months longer than the others. Some investigation showed this to be a known problem with Memcache: Once your cache gets full, it might be "hardwired" for your specific usage patterns. If those change (and you for example start to store larger values), memory is no longer allocated optimally, in extreme cases Memcache might not be storing anything at all.

Here is an excellent article Show archive.org snapshot on the gruesome details.

Memcache actually has a feature to alleviate this. Make sure you run at least version 14.11 and enable the slab_automove and slab_reassign features. Compare the release notes Show archive.org snapshot .

This might still not fix your issues. There are memcache forks Show archive.org snapshot with a whole list of eviction strategies Show archive.org snapshot to choose from. I suggest you start by monitoring your own hit and miss rates, and see if you actually require more complicated approaches.

Posted by Tobias Kraze to makandra dev (2013-01-28 12:20)