Sidekiq queues haven't broken past 500 all day, amazing.
"Fix your scaling issues with this one weird trick! systemd *hates* it!"
it's actually mentioned in the official docs: https://docs.joinmastodon.org/admin/scaling/ I should send a pull request to emphasize how impactful splitting up the Sidekiq daemon can be.
Today went really well. The new Sidekiq setup (multiple systemd services, to scale up specific in-demand queues) has worked out perfectly! CPUs aren’t idling as much and load average is still super-reasonable. I never saw the queue size exceed 500 all day, while yesterday we reached 45k 
The instance scaling continues to be a success. I haven't touched it since Tuesday night and memory/CPU/load have all levelled off 🎉
This graph shows the typical metrics before Monday, then during the initial influx, our first attempt to scale, and our second (successful) attempt to scale.
No more idling CPUs! This isn't the whole picture, as our Sidekiq queues were overwhelmed Monday&Tuesday. No more of that either 
@ashfurrow if you've got any questions about Postgres feel free to give a shout, it tends to be pretty robust but you've had the instance running long enough that memory tuning might be necessary.
@RandomDamage yeah, I gave it a few extra GBs of RAM and it seems happy :)
@ashfurrow very cool dude
@orta my niece is at a great age: “are those the moms?” “those are the aunts” “mmmm no those are the moms actually”
I learned a lot this week. I learned about systemd, Postgres, and Sidekiq, but I also learned more about the boundary between what I do/don't know.
A Mastodon server is a complex thing and it needs to be understood in order to scale it up. You can't just throw money at this kind of problem because that won't work. I enjoy this learning process, but it can be stressful when the server is under heavy load 😅