Lessons Learned (the hard way) from Building a Global, Decentralized Storage Network

Durability and performance in an S3 alternative storage platform are complex problems, but not owning the hard drives brings another level of difficulty. At Storj, we have 13,500+ independent node operators putting their unutilized hard drive space to work by joining our decentralized network. Learn how Storj developed an architecture that could deliver the demands of an S3 workload and also ensure durability. Assuming any of the node operators could be malicious, required a focus on encryption. We’ll go over how encrypting files the typical way (the whole thing) makes it difficult to support video streaming and what we did instead. What if there’s a blackout bringing thousands of nodes offline or a country deciding to block internet access? We’ll share the details about how these scenarios played out and what happened with our network. Spoiler, we didn’t lose a file. How do we maintain a system that is fault tolerant despite being susceptible to a large amount of threats? Short answer: erasure coding, encryption, and repair! We’ll show you how it works. Dan will discuss how Storj addresses these complexities and more. Join us for this casual, yet technical discussion: - Erasure codes for fun and profit - Streaming 4k video from encrypted content across thousands of nodes - Ensuring performance and eliminating long-tail effects - Managing data audits, repair, and reputation at scale