Compression in the Node.js Memcached Library

Here at Tagged we use memcached extensively with our PHP layer, primarily to store compressed, serialized PHP Objects. I’ve been working on a project that utilizes Node.js to interact with these objects in memcached and I’ve run into a few issues since the PHP memcached extension compresses objects above a certain size and the Node.js memcached library doesn’t support compression. Fixing this issue was not as simple as just compressing the data before putting it in memcached or decompressing the data that comes back from memcached.

The issue is how Node.js handled strings with bytes above the regular ascii range. The Node.js memcached library used strings to read data coming back from memcached, however when a byte was encountered that was not in the regular ascii range the byte would be converted into an UTF8 error byte sequence. In addition, the memcached flags indicating the data was compressed had to be properly set.

Tip: To avoid problems when dealing with buffers with non-ascii bytes, keep it as a buffer as long as possible. If necessary, the buffer can be converted to a string using the ‘binary’ protocol. However, if something is appended to this string, the non-ascii bytes will get converted to UTF8 error byte sequences. Hence, if something is added to this string, it should be converted back to a buffer and the necessary bytes should be appended.

With this in mind, I helped create this Node.js memcached library to solve these issues. It is useful to anyone who wants to use compression with memcached in node, as there are currently no other memcached libraries for Node.js that support compression. This is also useful for people who want to be able to have Node.js applications that are interoperable with the same memcached stores as their PHP applications.

Let me know what you think and please post any questions you may have! This is just one of the many issues with memcached and Node.js we’ve encountered and solved along the way. As we continue to use these technologies more extensively, we will be able to share more project stories, tips and tricks on how to use them efficiently. Stay tuned.

Nalin DeZoysa is a Software Engineer at Tagged and you can follow this project on Github.