Video of my #MCH2022 talk about #LibResilient and an idea for decentralized, community CDNs is online:
> I want a Web where CDNs are unnecessary.
> Where different organizations, different website operators, can help each other out by hosting assets for each others' websites, thus spreading the load across many orgs in solidarity, instead of centralizing it in gatekeepers.
An older fedi thread about just that:
@dredmorbius Squid is a caching reverse proxy. It can be used for caching and microcaching, which I mention in this talk as generally a good idea.
But Squid will not help you if your website is down because all your edge nodes are overwhelmed or because you are censored on DNS level.
LibResilient works in visitors' browsers, and *will* help in such circumstances (depending on the config).
@rysiek I may be misreading / misremembering the origins of Squid, but my understanding was that Harvest (the project from which it originated) was meant to provide the sort of generalised caching system you seem to be getting at. (I need to view your preso, about to and will hopefully reduce my ignorance...).
In this paper we introduced the Harvest system, which addresses these problems through a
combination of topic-specific content indexing made possible by a very efficient distributed information gathering architecture; topology-adaptive index replication and hierarchical object caching; and structure-preserving indexes, flexible search engines, and data type-specific manipulation and integration mechanisms.
AFAIR Akamai ended up productionalising much of this as a commercial service.
(Hazy distant understanding / memories.)
@rysiek And, having listened to the preso: Resilient sounds a lot like what you were calling Samizdat at one point, or is building off of it. Browser + web-workers with a server-side config for alternate origins.
The proxy cache concept (Squid) satisfies content requests by sourcing them through proxies. The original request goes through the proxy, either client-side (e.g., a local Squid caching proxy) or server-side.
What Squid configuration allows for is specification of peers for content.
A chief problem with Squid, as with other Web proxy tools (e.g., Junkbuster Proxy, if you remember that, or Dansguardian), is that they play poorly with TLS/SSL traffic. Certificate pinning and other mechanisms make this hard to accomodate with the modern Web ... unless if like Cloudflare the TLS request terminates with the caching provider. (Problematic.)
I don't know that Squid has been used to provide robust support for content access even where the origin is dead ... though that does seem to be its basic function in fact.
@dredmorbius Squid doesn't run in the browser, and unless a user runs it themselves, it does not offer what LibResilient offers. So Squid is not a comparable solution.
I think we've had versions of this conversation before. You're looking at a sites-and-users in-the-browser solution.
Squid might operate independently of sites, and on a server basis. Mind, "server" might be a network of small home/office routers running OpenWRT or similar.
One thought which occurs to me is that Tor may have an interest, in that Tor is often blocked by websites, but could potentially offer greater utility if it would sideline request and cache sites (via distributed proxy) in cases where the site itself fails to respond at all or as normally. Yes, that last means some case-by-case and whack-a-mole elements to this.
Samizdat / libResilient might be part of that sideline access.
@dredmorbius not sure, really. Libresilient is client-side but website-deployed, as in: it is deployed on a website but operates on visitors' browsers.
Squid et al can be part of the picture as alternative endpoints, for example (I mention the reverse caching proxy option in my talk). But not sure how LibResilient could help Tor out, as LR is not a thing that is deployed with/in a browser.
@rysiek Tor would need access to sites blocking Tor exit points, to feed to its own (proposed by me) caching network.
A model for that might be:
Tor client makes site request.
Tor exit point (TEP) notes that request is denied.
TEP checks for Tor Cache Instance of requested content.
TEP communicates request for alternative access (RFAA).
Request is farmed out to subscribing libResilient host(s).
LR client makes request, feeds to LR network.
Tor network is included in this distribution, and caches and distributes the content (w/ expiry and other aspects).
TEP alternative result requesting client in its result set. (This offer could be simultaneous w/ RFAA, or on Tor Cache check.)
Client has option of requesting cached version instead of origin.
Or something like that.
@dredmorbius yeah, but you don't need LibResilient for that. This can be implemented directly in Tor or Tor Browser. LR is specifically meant for regular websites.
@rysiek Tor traffic is (mostly) regular websites.
The problem Tor faces is that Tor excit nodes are both determinable and explicitly blocked or filtered.
LR clients effectively offer an alternative request network in normal-user IP space.
LR is not a general proxy tool. It's a very specific tool that is very specifically useful for *websites that deploy it* not for random tools that happen to interact with websites.
@rysiek LR would have to be server-independent for my proposal to make sense.
To that degree ... it's probably not a good match, though some alternate capability dropped in place of LR might work. Again, as a browser function, though in this case likely an extension rather than service-workers based on a website's source code.
Though that's now making me wonder how a server might use RS for evil....
@rysiek Anywho: my initial question was about the comparison / relationship between LR and Squid.
I think I've established a sufficient understanding for now. Thanks.
This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!