Sidekiq queues haven't broken past 500 all day, amazing.
"Fix your scaling issues with this one weird trick! systemd *hates* it!"
it's actually mentioned in the official docs: https://docs.joinmastodon.org/admin/scaling/ I should send a pull request to emphasize how impactful splitting up the Sidekiq daemon can be.
The instance scaling continues to be a success. I haven't touched it since Tuesday night and memory/CPU/load have all levelled off 🎉
This graph shows the typical metrics before Monday, then during the initial influx, our first attempt to scale, and our second (successful) attempt to scale.
No more idling CPUs! This isn't the whole picture, as our Sidekiq queues were overwhelmed Monday&Tuesday. No more of that either
I learned a lot this week. I learned about systemd, Postgres, and Sidekiq, but I also learned more about the boundary between what I do/don't know.
A Mastodon server is a complex thing and it needs to be understood in order to scale it up. You can't just throw money at this kind of problem because that won't work. I enjoy this learning process, but it can be stressful when the server is under heavy load 😅
@ashfurrow if you've got any questions about Postgres feel free to give a shout, it tends to be pretty robust but you've had the instance running long enough that memory tuning might be necessary.
@orta my niece is at a great age: “are those the moms?” “those are the aunts” “mmmm no those are the moms actually”
This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!