@neil > There might be an approach based on allow-listing known IPs of spiders, but it's hardly optimal.
It will also block any chance for new independent small search engines to pop-up, effectively locking the search mainstream engine space down for the oligopolists.
@neil good piece though! Thank you for writing it and sharing it.
This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!