Pinned post

Hello everyone 👋 We have a lot of new users, so, welcome! :mastodon:

Just a head up that this instance is moderated by a team who are fairly progressive, and who enforce our rules. If you see something breaking the Code of Conduct, please use the report feature. You can find the rules here: mastodon.technology/about/more

Thank you and take care!

Pinned post

For mastodon.technology status updates from admin, especially during downtime, please use status.mastodon.technology

Thank you to @SuperFloppies for building this and to @sporiff for helping test it ❤️

In five minutes, the instance server will be rebooted. Expect a minute or so of downtime.

We have completed the maintenance! Please let @ashfurrow know if you see anything not working correctly. We might have intermittent reboots throughout today as we adjust things.

Please note that we’ve had to temporarily disable our ElasticSearch capabilities, so search on m.t will be reduced to basic functions for a few days.

Thanks for your patience and take care!

Show thread

Reminder: Friday morning EST will have extended downtime while server upgrades are done.

Show thread

On Friday July 30 at 7am EDT (11:00 UTC) the mastodon.technology instance will be undergoing extensive server maintenance. Expect several hours of downtime. Thank you for your patience.

Okay, I tested on staging first this time. Need to remember to do that *before* announcing reboots…

We’ll be going down for a quick reboot in five minutes. Expect <1 minute of downtime.

Show thread

The instance will be rebooting in five minutes, expect <1 minute of downtime.

We've completed the server upgrades and will monitor things. Please let @ashfurrow know if you see any problems with the instance.

We have upgraded from Mastodon 3.3.0 to 3.4.1. The release notes, including new features and bug fixes, are available at github.com/tootsuite/mastodon/

Show thread

Tomorrow morning at 8am EDT (12:00pm UTC) the mastodon.technology instance will be undergoing server upgrades. Expect intermittent downtime. We will let you know when everything is done. Thanks!

In case anyone was curious why the m.t instance went down for an hour last night, I've opened an issue on the Mastodon project with details of our unexpected downtime. I suspect it was a fluke and not a problem with the software, but I've included remediation steps in case they are helpful to another admin someday.

github.com/tootsuite/mastodon/

So far the new Sidekiq instance is stable. The ETA for rebuilding user timelines is about an hour.

Thanks for your patience – I know it sucks to lose any data at all. @ashfurrow will check in on this in the morning but currently needs sleep.

Show thread

Hi everyone. We experienced a catastrophic failure in our Sidekiq setup, which handles background job processing. We had to clear its queue because it had become backed up by millions of duplicate jobs. The cause is still unknown.

Feeds for local users are rebuilding now. We probably lost some incoming federated posts, and some outgoing posts for the past hour won’t be delivered.

We just recovered from unexpected downtime caused by runaway ram use. We’ll investigate the root cause to tomorrow, hoping things stay calm overnight 🤞

The server is going down for a quick reboot in about five minutes.

Show older
Mastodon for Tech Folks

This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!