Building a Decentralized CDN for the Web

We’ve been building decentralized technology for the past 7 years now. When we got going, our focus was on making standalone applications because integrating with the traditional web without sacrificing decentralization seemed like a tall challenge.

We’ve come a long way since then. We’re still figuring out the best way to decentralize the Internet, but browsers have come a long way in the past few years and we’ve discovered several surprising techniques that allow us to bring decentralization to users without requiring them to download new software. The latest of course being a decentralized CDN that works on top of centralized websites.

Before we get any further, it’s important to establish what we mean by decentralization. Decentralization is a gradient, and in its purest form it means that there is no failure mode for the user. The poster child for decentralization is Bitcoin, and at least if you take its security claims at face value, it achieves that well. There is no single party or collective of parties that are reasonably capable of censoring transactions or modifying the transaction history. Whether those claims hold up to scrutiny is important, but not for the purposes of this post.

In the context of a decentralized CDN, the focus is around the application user. When a user hears that an application is decentralized, that should be indicating to the user that the app will have reliability independent of any single entity or group of entities. This means that the data itself needs to be stored in a reliable way, and then there need to be reliable avenues for the end user to retrieve the data.

And then if you want it to be web friendly — meaning we don’t require the user to download any software or install any browser extensions — you have to accomplish all of that from inside of the web browser sandbox.

Sia: The Storage Backbone

The backbone for our CDN is going to be Sia, a decentralized storage network. When data is uploaded to the Sia network, several (typically 30) hosts are paid to keep the data. The entire Sia network is powered by money. Hosts are paid when data is uploaded, hosts are paid over time as they store the data, and hosts are paid for their bandwidth when the data is downloaded again.

People often get upset at this part, but Sia is powered by a blockchain. The payments are made using cryptocurrency, and the hosts are bound to storage contracts that check cryptographic proofs that the host is still storing the data in question before making a payout.

The typical size of a payment on the Sia network is $0.000004, which is much too small for a traditional payment system. Traditional payment systems also don’t support conditional payouts, and even if they had all of these features they are still centralized. Blockchains aren’t good at many things, but when you need decentralized payments, they are often the only viable choice. If you eschew the idea of payments altogether, you find that it’s almost impossible to build a reliable storage network. There have been dozens, and perhaps hundreds of attempts at this over the past 2 decades, and none of them have been successful. For reliable, decentralized storage, there’s only one option and it’s Sia.

Even Sia has weaknesses. Data is ultimately stored with a finite number of people, and while those people are kept online by financial incentives, there is ultimately nothing we can do if all of those people decide together to unplug their machines. If you are constrained to having other people store data, this is the absolute best you can do.

Skynet: The Web-Friendly Interface to Sia

About a year ago, we came up with a method to allow normal web users to access and share data over the Sia network. We called this layer Skynet.

The major structure that enables Skynet is called a “Skynet Portal”. A Skynet portal is a centralized web service that acts as a pass-through to the Sia network. Importantly, the Skynet portal is [almost] entirely stateless. Switching from one portal to another has no impact on the user experience — content uploaded to one portal is available using any other portal, and the portals do not need to coordinate with each other or run any sort of consensus protocol (other than the Sia network daemon itself).

This creates a pseudo-decentralization where any user can access any content or application by choosing a portal of their choice. If a particular portal is down or is blocking access, you can switch to another portal.

I call this pseudo-decentralization because if a particular centralized service goes down, the user has to manually intervene to restore their access to Skynet. This both requires the user to know how Skynet works (to a minor degree at least) and also requires that the user be aware of alternative portals which are still operational. A complete solution can invisibly repair itself as things come online and go offline.

MySky: The Self-Repairing Layer

We can fix the dependence on centralized portals by giving users a way to load their favorite portal (with any list of fallback portals) directly from their web browser, independent of the URL provided by the application.

The key enabling technology is iframes. Instead of having the web application make the content request directly, it opens an iframe to a separate application called MySky, and MySky will manage the request and return the response using the postmessage protocol supported by all major web browsers.

On the backend, MySky uses local storage to load a list of portals that the user prefers, and then try each portal until one of them returns a useful response. If the user hasn’t used MySky from this web browser before, it can use a set of default portals to fetch the user’s list of preferred portals, and if the user hasn’t ever configured MySky at all (on any browser), it’ll just use the set of default portals.

And that’s the crux of the idea. We can turn any content fetch into a more robust, decentralized retrieval process by putting in some middleware that tracks user preferences and can failover to alternative services if a preferred service is down.

The One Remaining Shim

The one gap here is that MySky is a bundle of javascript, and that javascript needs to be downloaded from some centralized service. That’s a fundamental limitation of the web browser. When the web browser connects to an online application, it must connect to a centralized service, that’s just how modern web browsers work.

One way to get around this is to install a browser extension which can replace requests for MySky with locally served files, but the long term approach would be to convince web browsers to support MySky natively. Native MySky support is nothing more than a code injection that replaces a call to https://mysky.hns.siasky.net/ with a locally served piece of code. Politics aside, this is less than a day of work for the right browser engineer.

In the meantime, web applications still need to fetch the MySky code from a centralized service, and users may experience a disruption if that code is unavailable. We can even use tricks like caching to minimize the impact of downtime from the main service. Dependence on a single piece of code is a huge step forward from the status quo — where all of the content is being channeled through a single centralized service — and we have a highly viable path to full decentralization and mass adoption.

User Configuration

As much as possible, we try to minimize the total amount of user configuration required. Skynet operates on a freemium model where anyone can make use of the infrastructure for free, but will be subject to ratelimits and lower quality of service until they upgrade to the paid service.

For most users, no configuration is necessary at all until they want to upgrade to the paid service. You don’t even need to make an account, everything will just work. When you do decide that you want premium speeds however, you pick a portal and then the MySky app can walk you through the portal signup process. Portals can accept both fiat and crypto for payment, and then in return for a monthly subscription the portal can give the user full speed access to the decentralized web.

When signup is done, MySky will configure the paid portal as the primary portal for the user, and if the paid portal has any partnerships or payment setups with other portals it’ll configure the partnered portals as the first attempt for fail-overs. Portals outside the partnership programs will also be used as fall backs, but only if all of the portals inside the partnership are unavailable.

The Roadmap for Decentralization

We’re seven years into our journey at Skynet Labs, and perhaps the most surprising part of this idea for decentralized CDN was how simple it was overall. We’re still very early when it comes to decentralization. Unlike fields like physics, mathematics, or even machine learning, there are lots of little, easy ideas that people just haven’t had yet.

Especially the past year, the Skynet team has been blessed by a series of simple yet potent ideas, and there’s no sign yet that this is going to slow down. Looking into the past at areas of growth like physics, aviation, and semiconductors, it feels reasonable to project that we’re going to continue to see enormous growth and creativity in the field of the decentralized web for another 30 to 50 years, and that the really interesting stuff is just getting going.

The most exciting part though is that decentralized technology is just… better. We started down the path of decentralization because we believed that it was a healthier way to build a society. But we also spent a big part of the past seven years stressing about the path to widespread adoption. People aren’t willing to adopt a technology simply because it’s healthier for them. And thankfully, it appears that we don’t need to change that. We appear live in a reality where the healthiest technology is also the best. And that makes it inevitable.

Building a Decentralized CDN for the Web was originally published in Sia and Skynet Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.