Community Spotlight: S5

A Content-Addressed Data Routing Network that is decentralized, scalable, efficient, secure, and FOSS.

Hey, I’m redsolver, and I love building innovative open-source apps powered by decentralized networks. I joined the Sia Community when the Sia Skynet project launched in early 2020 because the technology made it possible to build fast and user-friendly applications on top of Sia, which was really exciting! Unfortunately, this excitement was short-lived when, in 2022, Skynet was unable to continue development. Undeterred by this setback, I took my favorite parts from Skynet, IPFS, and Hypercore and began developing the S5 protocol that aims to make decentralized content delivery as efficient and scalable as possible. I believe that S5 has the ability to enable the development of decentralized applications in a user-friendly way that wasn’t possible before. The Sia Foundation has been funding my development work on S5 via their Grants Program since early 2023, and in this spotlight, I’ll talk a bit about S5 and the grants program.

Why do we need S5?

So, let’s start with why we even want decentralized storage and data solutions. One good example is the number of issues traditional cloud storage providers face, like a lack of end-to-end encryption that allows company personnel unrestricted access to user data. Many traditional storage providers also face risks created by the centralization of their networks. For example, if a provider experiences downtime, you, as the customer, will be unable to access your files. Or, in a more extreme case, if the storage provider ceases to exist, you could lose everything.

Another benefit of S5 is its ability to deliver public content, like videos, music, images, or live streams. Existing platforms have a strong lock-in effect that prevents you from easily switching platforms. While many of the challenges in building decentralized social applications are being solved by the AT Protocol. S5 can solve the media delivery requirement by serving as a cryptographically secure decentralized CDN routing layer.

Finally, there are the problems that current messenger and chat platforms face. Although there are some that offer end-to-end encryption, none of the popular ones are decentralized. This makes these platforms susceptible to risks like the loss of your identity and contacts or experiencing service downtime. It’s also hard to switch platforms because you will lose your ability to interact with all of your friends. While some decentralized chat solutions are available, they either don’t scale well or are too unreliable for day-to-day use. A problem S5 solves through its native routing layer and the messaging layer security protocol (MLS).

Ok, but why not just use the Sia network directly?

The main reason for this is that while Sia can reliably store private files in a secure and truly decentralized way that eliminates trust, it does not have a content-addressed file retrieval/streaming or real-time routing layer that natively works with web apps. Sia also has some usability challenges for new users, like needing to purchase Siacoin, syncing the blockchain, or higher bandwidth consumption due to the required file repairs needed to maintain redundancy. S5 aims to abstract these pain points away while still keeping most of the guarantees that the Sia network provides, such as decentralized storage and confidentiality.

What is S5?

At its core, S5 consists of some basic concepts and data structures that have been designed to be efficient and easy to use in a decentralized network.

The first concept is content-addressed data. Any blob of bytes, such as an image, can be globally identified by its unique fixed-length 32-byte cryptographic hash. This means that as long as you get the hash of a file you’re interested in from a reliable source, you can download it from any available node and verify that it is an exact copy of the file you requested without needing to trust the node. S5 uses the BLAKE3 hash function by default because it’s fast, secure, parallelizable, and capable of verified streaming. Thanks to BLAKE3 being structured like a Merkle tree, it’s possible to stream only small parts of a giant file and verify that they are part of the root hash using minimal additional metadata. S5 then encodes the BLAKE3 hash and file size as a short human-readable content identifier (CID) that can be shared easily. While S5 CIDs are not directly compatible with IPFS due to the included file size, they use different magic bytes to prevent collisions and can easily be converted to IPFS CIDs if needed. BLAKE3 is also used by other modern protocols like Iroh, which initially started as an IPFS implementation.

The key difference to most other content-addressed networks is that S5 doesn’t really care where you store your files and can work with almost any storage backend directly, as long as it supports the HTTPS protocol and HTTP range requests. This makes it possible to combine the strengths of different storage networks. Using the S5 Pinning API makes it quick and easy to securely mirror the same data on as many different storage backends as you want. Primarily, using HTTPS for file transfers has multiple advantages, like being able to use already existing highly-optimized implementations, infrastructure like CDNs, and keeping full compatibility with web apps. The S5 web proxy is a small piece of software that can be easily added to a web app as a service worker and stream content directly from the S5 network through the user’s web browser while verifying the integrity of every single byte.

While S5 does have a p2p network, its primary purpose is to find storage locations for a specific hash/file (CID) and not to transfer the file data over p2p. So it’s pretty much just a content-addressed routing layer and could be combined or extended with most other distributed hash tables (DHT) that already exist. The network supports multiple different transports, and you can mix them however you want as long as your platform supports it. The default one is WSS (WebSocket), but the Iroh networking layer for truly p2p connections will also be supported soon.

There are additional concepts and data structures like the registry for storing mutable persistent key-value pairs with values smaller than 48 bytes and message streams for routing real-time messages (which are more temporary) and also support larger payload sizes. The registry is very simple; keys are public ed25519 keys by default and updates to an entry need to be signed by that key pair. This also makes it possible to resolve registry entries on other DHTs in addition to just the S5 p2p network, for example, Pkarr, which uses the battle-tested Mainline DHT. The message streams are still a work in progress, but they share some similarities with other projects in the space, like Hypercore and Namakemono.

These two mutable layers, combined with the basic BLAKE3 blobs, verified streaming, and hybrid P2P network, make building most types of applications possible. The data structures are also designed to be as lightweight as possible, so combining and integrating S5 with other cool decentralized technologies is very much something I want to support! All of these concepts are explained in more detail in the S5 documentation.

Development Progress and Future Plans

The work on the core S5 tech is coming along nicely. It’s still a bit experimental, but all the basic features, like streaming giant files securely, already work reliably. Most of the code is written in Dart+Rust and can be viewed on the S5 GitHub page.

The biggest app I’m working on is Vup, which is an open-source, cross-platform, end-to-end-encrypted cloud storage app built on top of S5. It uses most of the features and is a great demo of what’s possible with a fast content-addressed storage network like S5. I encourage everyone to give it a try and let me know what you think!

Some more experimental applications that I have built are the Tube5 video-sharing platform that streams videos directly from the S5 Network in any modern web browser, Vup Chat which will be a secure alternative to messenger applications like Discord, and maybe even a new decentralized App Store that securely downloads applications and games from the S5 network. I am also building a simple collaborative document editor in Flutter to showcase all the different S5 features and make it easier for developers to get started building something nice! Another cool application that will soon use S5 for secure real-time collaboration features is Butterfly, a powerful visual note-taking app written in Flutter.

Also, as I mentioned earlier, the AT Protocol is, in my opinion, the best solution for decentralized and scalable public platforms. So, I will likely build something like a Bandcamp alternative on top of the AT Protocol that uses S5 and Sia to deliver media files securely at scale.

As for improving the Sia integration, it would be awesome if Sia hosts could support the HTTP protocol natively in the future and make it possible for S5 users to stream data directly from the Sia network without needing native software or a proxy. That wouldn't make things more secure, but it would be quite a bit more efficient bandwidth-wise and reduce some of the centralization risks.

The Sia Grants Program

Thanks to funding from the Sia Foundation, I’ve been able to work full-time on S5 and related applications for almost one year now. I think the Grants Program has been an awesome way to support developers with innovative ideas and enable them to work on free and open-source software that benefits society without needing to rely on ads or selling user data to fund development.

I’m biased because I’m part of the grant committee, but I feel like the Grants Program is really starting to pick up steam, and a lot of cool projects are coming to life thanks to it. So, I recommend that anyone who has a cool idea for a project built on top of Sia submit a grant proposal. The Sia Foundation’s new hostd and renterd software makes developing on Sia easier than ever before!

If you have any ideas, feedback, questions, or want to talk about decentralized networks, send me a message on Matrix (@red:y4y.me), E-Mail (s5@redsolver.net), or Discord (@redsolver). I would love to answer any questions you might have about the tech!

Community Spotlight: S5 was originally published in The Sia Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.