Written by Jon Stern on 36:04 reading time

… found in translation.

A while back, Kris wrote about some of the interesting ways redis configuration choices can harm the performance of your applications. The investigation he described was the most illuminating so far in our efforts to understand and tame the microservice beasts we have bred, but I’d like to talk a bit about my own pet preoccupations and how I came by them: specifically, how in translating the library we use to build Python workers into Clojure, I learned a few new things about the code that had been under my nose all along.

A Pattern for Distributing Synchronous Tasks Using Redis

So Kris mentioned PSH, which stands for Psomething Service Hub. (Well, almost—the Psomething part is Top Psecret!) To build it, we wrote a couple of “tasklib” libraries, one in Ruby and one in Python, to enqueue synchrononous1 “tasks"—remote function calls that return all the disparate pieces of context we need to build our pages—in redis. On the other end, worker processes use one...

Read more

Written by Kris Schultz on 04:25 reading time

Redis has become an increasingly important part of our production stack at Leapfrog. We use it in two critical functions: as a session store and as the transport medium for our microservice implementation (PSH). The throughput of Redis was a big draw when looking at technologies to use for PSH and the list data types are also very useful for feeding tasks to workers. However, Over time we started to identify some performance issues with task throughput. Overall, the performance was great but we would occasionally see spikes of 20+ seconds when pushing a new task (a small JSON blob with parameters) onto the end of a list. Thus began the great redis investigation of 2015.

Under The Microscope

The entire ordeal began as we prepared to migrate datacenters. Every component of the infrastructure was under the microscope to make sure that the new data center was running properly. As our primary platform (FOX) was being tested, we started taking note of these latency spikes. We noticed the...

Read more