I'm noticing a huge surge in 404 errors on my website because a bunch of fediverse crawlers seem to think that beesbuzz.biz is a Mastodon instance. I wonder what gave them that idea.

It's not a huge deal or anything, but it's also a bit baffling why these crawlers have each been retrying every 10 minutes for the past few days. fediverse.space and fediverse.network are the two big ones but there's a handful of others as well.

@fluffy fediverse.network tries all the possible URLs

if it's happening every 10 minutes you've enabled monitoring

(log in and disable monitoring to make it slow down)

@ben How would I have enabled monitoring and how do I log in? I don't have an account there or anything, I've never even heard of this before just now.

@fluffy actually, it looks like fediverse.network doesn't have monitoring enabled for that domain

fediverse.network/beesbuzz.biz

so the "every 10 minutes" thing is probably fediverse.space (which I've never heard of)

@ben okay, you're right, fediverse.space is doing it every 10 minutes. fediverse.network is doing it every 30 minutes. Which is still a lot more often than it should for something that is not and never will be a Mastodon instance.

@fluffy yeah, @href is aware the crawler needs better detection of non-instances

@ben @href I'd think returning a 404 on /api/v1/instance would be a pretty good indication :)

@ben @href Any idea why it would have started doing this just today?

@fluffy hmm, maybe some instance claims it's a peer? are there any activitystreams documents on there?

@ben I've been using Bridgy Fed to republish my stuff as @beesbuzz.biz for a while but that hasn't changed at all since last October. Unless something changed on Bridgy Fed itself today. @snarfed.org ?

Follow

@fluffy @ben @beesbuzz.biz nope, no recent bridgy fed changes. it definitely does make the fediverse think your site is an activitypub/mastodon instance, which will trigger requests like these, but that’s always been true. more details: github.com/snarfed/bridgy-fed/

· · Web · 1 · 0 · 0

@snarfed @ben @beesbuzz.biz I wonder what caused things to change such that I suddenly was getting the influx of traffic then.

@snarfed actually it looks like this has been going on for a while, it just only happens every few days and lasts for the whole day and then they stop for a while. Whatever.

@fluffy @snarfed probably what happened is some instance claimed your domain as a peer and the two spiders picked that up at around the same time

@ben @snarfed here's my stats graph from the last couple of weeks. each of those 404 spikes is one of the instance lists going weird, and then it stops for a while.

I suspect that what's happening is some instance where someone's following me via bridgy fed gets set off whenever I post a new blog entry or something.

@fluffy @snarfed fediverse.space just launched today so it might have been in development during that time

@ben @snarfed I’m tempted to just add a Mastodon instance info route to my site just to make it a means of advertising publ :P

Sign in to participate in the conversation
Mastodon for Tech Folks

This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!