Pinned toot

For status updates from admin, especially during downtime, please use

Thank you to @SuperFloppies for building this and to @sporiff for helping test it ❤️

We have added language to the Code of Conduct to prohibit fascism and fascist-sympathizing from our instance. We will also not federate with instances who choose to federate with fascist instances. Let @ashfurrow know if you have any questions.

The CoC is here:

Okay, sorry for the extended downtime! Those database migrations, eh? We've upgraded from 2.8.0 -> 2.9.1 – all the release notes are here:

Notable is a new moderation API and a new single-column layout. Check your user settings for more info. Let @ashfurrow know if you see something wrong with the instance.

Tomorrow morning at 10:00 EDT (14:00 UTC) will enter a 2-hour maintenance window to upgrade our servers. Expect intermittent periods of downtime while severs are rebooted, etc.

We’re aware of spammers signing up for m.t and have taken steps to stop them. Thank you for our local and remote users who’ve reported them.

We’ve identified a problem where some requests were failing with 400 “Request header or cookie too large” and a fix has been deployed. Thank you to users who helped troubleshoot this!

We just had a reboot related to the issue a few weeks ago; I'll be tuning some more nginx config tomorrow to hopefully address in the longterm. Thanks everyone.

We've added language to the Code of Conduct to forbid scraping content from the instance's web interface or API. You're free to archive your own content, of course (it's also exportable from your user settings).

Sorry for the intermittent downtime; I've got the server into a stable-ish state and will take another stab at reconfiguring nginx tomorrow. Thanks for your patience, everyone.

I’ve reconfigured nginx to support a higher number of simultaneous connections, which should alleviate the immediate problems while I do more investigation into nginx tuning. Thanks for your patience everyone.

We had a bit of an outage earlier, with intermittent accessibility of the m.t instance. I’ve done another reboot and am monitoring.

Site monitoring for indicates intermittent refused connections. DigitalOcean, our cloud hosting provider, is experiencing network issues that may be related. I will continue to monitor.

If anyone has problems accessing the instance, let @ashfurrow know 👍

Good morning. I'm seeing some timeouts on the server so I'm going to do a quick reboot to hopefully resolve. Please standby.

Following up on the degraded service from yesterday, here is the postmortem with details and learnings for next time:

Reach out to @ashfurrow with any questions. Thanks again, everyone.

Good morning. Overnight, a disk on our server ran out of space, causing a few API endpoints and media uploads to fail. The issue's been fixed for now and I'll be doing a proper postmortem once I'm done the workday. Thanks for reporting the problems, everyone!

Almost forgot! Mastodon 2.8 release notes are here:

Keybase integration requires some more setup that we're working on now.

On Saturday, April 13, at 10:00 EDT (14:00 UTC), will enter a maintenance window to upgrade to Mastodon 2.8.0. Expect periods of downtime while we reboot servers, apply database migrations, etc.

Okay, got the ElasticSearch problem sorted out – turns out ES goes into a read-only mode once the disk gets full, and needs to be manually unstuck. This blog post was helpful:

Sidekiq jobs have stopped failing; I'll keep the re-index job going (to fill in the gap while ES index was read-only). Thanks again!

Alright, not sure what's up with the search indexes. The volume it was on was pretty full, too, so I've also moved it over to a larger hard disk and have begun re-indexing. Hopefully Sidekiq indexing jobs stop complaining after that. Expect some degraded performance as those search indexes are sync'd.

Thanks everyone for your patience!

Show more
Mastodon for Tech Folks

This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either! We adhere to an adapted version of the TootCat Code of Conduct and follow the Toot Café list of blocked instances. Ash is the admin and is supported by Fuzzface, Brian!, and Daniel Glus as moderators. Hosting costs are largely covered by our generous supporters on Patreon – thanks for all the help!