Pinned toot

For mastodon.technology status updates from admin, especially during downtime, please use status.mastodon.technology

Thank you to @SuperFloppies for building this and to @sporiff for helping test it ❤️

Small update: we have added language to our Code of Conduct to prohibit casteism. Have a good evening ❤️

Just had to reboot the docker containers after the instance started timing out. Will investigate later.

We have updated our instance configuration to default to excluding user profiles from search engine indexes. Previously, accounts were included by default. Now, they are excluded by default. This change doesn’t impact users who have modified this setting.

You can remain in search indexes in your user settings.

Existing user content may remain cached in search indexes for some time, but should be removed over time to match our new config.

Reach out with any questions.

Downtime was roughly 28 minutes, according to StatusCake. I'll be purchasing SMS credits to get notified on my phone, since the email notification went to spam :(

Just saw that the instance was down. Not sure why, still looking into it. Neither an nginx restart or a restart of the docker containers was successful, only a full reboot of the host VM had any effect. Load+CPU+memory were all steady.

Okay, Content-Security-Policy headers are now updated to work with with the new client-side OCR in the image upload flow, but Safari specifically still isn't working. I'll try to take a look later, we'll see!

Okay, I'm going to try some more CSP configuration changes. There will be Docker restarts and nginx reloads, so expect intermittent downtime this morning. Promise to keep it brief 😚✌️

Tracked down the problem a bit, and learned more about nginx configs. The CSP headers are still an issue but for now I'm stopping faffing with the configs.

We're diagnosing some Content-Security Policy HTTP header configurations this morning, which involves restarting the nginx service on the host VM. Expect intermittent unavailability.

Okay! ElasticSearch indexes are still being regenerated, but otherwise things seems stable.

We've upgraded from 2.9.2 -> 3.0.1. The release notes are here, you should look at the 3.0.0 notes for the most exciting features: github.com/tootsuite/mastodon/

We've got trending hashtags, we've got new client-side OCR for image descriptions, we've got better search results UI, we've got better 2FA settings, we've got new spam controls... check it out, folks!

Database migrations are complete and the servers are running Mastodon 3.0.1 code, but our ElasticSearch needs to be re-indexed. I'll follow-up once that's done with the release notes and everything.

Thanks for the patience, everyone!

Mastodon upgrade went well on staging, looking to get it done on the main domain next. Expect some intermittent downtime for a few hours.

Sorry for the short notice, I'll try to plan these better next time.

Apologies for the extended downtime there. I didn't realize one of the packages that apt was upgrading was, Docker itself. Sorry!

We're going to apply some minor software updates to the host VM this morning. Expect a few short periods (<60 seconds) of intermittent downtime during server reboots :ReBoot:

We have completed the migration from Cloudflare for mastodon.technology. It was quite a process, so @ashfurrow has a retrospective written up here: ashfurrow.com/blog/migrating-f

Special thanks to @IzzyOnDroid for their help researching alternatives 🎉

Hmm, due to a quirk with Namecheap’s user interface, it appears that some mail servers have cached incomplete DNS records from the new nameserver. Email delivery (for notifications, password resets, etc) will have intermittent delivery failures while DNS completes propagating. Will monitor and keep you updated, retrospective blog post is coming 👍

We have updated our DNS settings, pointing to a new nameserver. You shouldn't see any downtime, just letting you know ❤️

Cloudflare was an appealing option because it’s free, and the new CDN is not. Luckily, donations from our instance’s Patrons *should* already be enough to cover the additional cost. Thank you all so much ❤️

Okay, reboots complete! We’ve moved our traffic over to use our new CDN provider. You may need to refresh browsers to receive a new Content-Security Policy response header.

The migration away from Cloudflare is still an ongoing project, since they are still DNS nameservers for m.t and we want this CDN change to stabilize first. But this is the biggest step. A blog post outlining the steps we took and lessons learned is forthcoming.

Reminder: we are about to enter a maintenance window, please excuse the intermittent downtime.

Show more
Mastodon for Tech Folks

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!