Show newer
Ash Furrow boosted

SOME 

pasta once told me the world is rigatoni

Yesterday we processed a record-setting 1M Sidekiq jobs. Today is at 1.4M already! Sidekiq seems to be the bottleneck, but I can’t scale it up without running out of Postgres connections. That makes me think PgBouncer might be the necessary next step. It gives me more connections, so Sidekiq can do more at once, so my idling CPUs can get to work.

Does that make sense? (Bonus: the Mastodon docs have a guide on how to do this.)

Show thread

43k jobs queued up. default has a latency of 22 minutes, pull is over two hours lol okay

I see a few jobs taking a long time (>10 minutes) but most finish instantly. I’m trying to figure out where the next bottleneck to scale up is. How do I get the CPU cores doing more? PgBouncer? Multiple Sidekiq services? It seems like I have CPU headroom, so should I get more aggressive with my Postgres config? Is there a quick win without prolonged downtime? These are the questions on my mind.

Show thread

Sidekiq queues have continued to get worse. (But at a slower rate! Yes!!!) I’ve played with the config of everything as much as I’m comfortable – a sluggish instance is better than an unresponsive instance.

Show thread

Another cool thing users can do to alleviate strain on the fediverse is to pause your scripts that delete your old posts.

You might not think it, but those deletion requests take a lot of work to churn through. I have several dozen processing in Sidekiq right now, from a single remote user!

Show thread

Probably not a huge deal, but I've been keeping an eye on our queue sizes and when they shoot up by 8k suddenly, I dunno. Doesn't feel great 😅

Show thread

Something I've noticed clogging up our Sidekiq queues have been user account *deletion* jobs. When you delete a Mastodon account, it appears that a lot of jobs get queued to notify other instances of that deletion.

Servers are already busy due to the influx of new users. If you're rediscovering the fediverse, and you want to delete some previously-created account, maybe wait a week or two? Much appreciated 🙏

One year ago, this instance was running on Docker. This configuration had been officially supported, but discouraged, by Mastodon maintainers – and for good reason. It introduced so many problems and so much overhead that I'm really grateful that everyone peer-pressured me into migrating off it. Now it runs as standard linux services. I'm positive this instance would be in tears right now, otherwise 😆

Show thread

This instance is funded by a Patreon – if you'd like to contribute, head over to patreon.com/ashfurrow

Show thread

Along with myself, this instance is moderated by @bclindner and @fuzzface – I'm fortunate to have such a great team 🙌

Show thread

Here are the DigitalOcean graphs for mastodon.technology over the past 7 days, and the past 24 hours. Yesterday required us to rebalance our resource utilization (before this week, we were over-provisioned and were preparing to decrease server resources to save costs 😅).

I think we're in an okay spot. We don't have a lot of headroom left. Fingers crossed today. We may need to close new registrations, but we'll be upfront if we make that decision.

Ash Furrow boosted

Tips for new folks 

Content warnings and image captions are an important part of fedi culture. Both features promote inclusivity in their own way, and should be used liberally! You may find people are reluctant to boost otherwise good posts that lack one or both of these features.

"Politics", mentions of violence or other triggering subjects, spoilers, long posts, and plenty of other things greatly benefit from a content warning and keep fedi safer for more people.

Ash Furrow boosted

Welcome to Mastodon. We got weird emotes, like this one:

:cate:

Babylon 5 Watch Through 

I feel like the writing in season one could have really benefited from a second writer. Someone to push back a bit against JMS, who appears to want total, absolute creative control.

Actors are creative people, too – the few times actors on this show are permitted to bring their own ideas to their characters, it really shines. JMS reminds of Star Wars prequel-era George Lucas, treating actors like puppets.

Show thread
Ash Furrow boosted

“Michael Tsai - Blog - App Store Removing Old Apps That Still Work”

An overview of Apple’s latest instance of self-sabotage. mjtsai.com/blog/2022/04/25/app

Ash Furrow boosted
Ash Furrow boosted
Show older
Mastodon for Tech Folks

This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!