Sidekiq queues haven't broken past 500 all day, amazing.

"Fix your scaling issues with this one weird trick! systemd *hates* it!"

it's actually mentioned in the official docs: I should send a pull request to emphasize how impactful splitting up the Sidekiq daemon can be.

Today went really well. The new Sidekiq setup (multiple systemd services, to scale up specific in-demand queues) has worked out perfectly! CPUs aren’t idling as much and load average is still super-reasonable. I never saw the queue size exceed 500 all day, while yesterday we reached 45k :ohnocat:

The instance scaling continues to be a success. I haven't touched it since Tuesday night and memory/CPU/load have all levelled off 🎉

This graph shows the typical metrics before Monday, then during the initial influx, our first attempt to scale, and our second (successful) attempt to scale.

No more idling CPUs! This isn't the whole picture, as our Sidekiq queues were overwhelmed Monday&Tuesday. No more of that either :cat_uwu:

I learned a lot this week. I learned about systemd, Postgres, and Sidekiq, but I also learned more about the boundary between what I do/don't know.

A Mastodon server is a complex thing and it needs to be understood in order to scale it up. You can't just throw money at this kind of problem because that won't work. I enjoy this learning process, but it can be stressful when the server is under heavy load 😅


@ashfurrow if you've got any questions about Postgres feel free to give a shout, it tends to be pretty robust but you've had the instance running long enough that memory tuning might be necessary.

· · Web · 1 · 0 · 3

@RandomDamage yeah, I gave it a few extra GBs of RAM and it seems happy :)

Sign in to participate in the conversation
Mastodon for Tech Folks

This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!