Oops! We screwed up!

Published on February 21st, 2015 01:42 by lightstalker

So we inadvertently had a bit of downtime over the past few days.

What went wrong?

Our backend task scheduler, Sidekiq, stopped running. As a result, messages were not being sent out.

Why did this happen?

I (Brian) was a moron, and the application server ran out of memory. Normally there is swap space allocated for this situation, but unfortunately it was not running at the time.

So sorry about the delay in messages sending. To prevent a huge flux of messages to be sent out when they're horribly out of date, the notification queue was emptied.