Scaling Magic Pocket - Exabyte Scale Object Storage System: Architectural Insights and SMR Adoption Journey

Magic Pocket represents Dropbox's proprietary Exabyte-scale object storage infrastructure, tailored specifically for the demands of Dropbox's file synchronization and sharing functionalities. As a highly available and durable system, it caters to the needs of hundreds of millions of users worldwide. Since its inception, Magic Pocket has expanded to a multi-Exabyte scale, showcasing Dropbox's pioneering adoption of Shingled Magnetic Recording (SMR) technology on a large scale.

In this presentation, I delve into the architecture of Magic Pocket, the journey of integrating SMR technology, and the valuable insights gained from operating such a colossal system with a small team of engineers. Additionally, I address the ongoing challenges posed by the growth of disk capacities juxtaposed with stagnant disk IOPS and throughput. Join me as we explore the intricacies of Magic Pocket's evolution and the strategies employed to tackle the evolving landscape of storage technology.

Abstract

Magic Pocket represents Dropbox's proprietary Exabyte-scale object storage infrastructure, tailored specifically for the demands of Dropbox's file synchronization and sharing functionalities. As a highly available and durable system, it caters to the needs of hundreds of millions of users worldwide. Since its inception, Magic Pocket has expanded to a multi-Exabyte scale, showcasing Dropbox's pioneering adoption of Shingled Magnetic Recording (SMR) technology on a large scale.

In this presentation, I delve into the architecture of Magic Pocket, the journey of integrating SMR technology, and the valuable insights gained from operating such a colossal system with a small team of engineers. Additionally, I address the ongoing challenges posed by the growth of disk capacities juxtaposed with stagnant disk IOPS and throughput. Join me as we explore the intricacies of Magic Pocket's evolution and the strategies employed to tackle the evolving landscape of storage technology.

Learning Objectives

Understand the architecture of Magic Pocket and its role in supporting Dropbox's file synchronization and sharing services.
Learn about the operational lessons and best practices derived from managing a massive storage system with a small team of engineers.
Gain insights into the adoption of Shingled Magnetic Recording (SMR) technology at scale and its implications for storage infrastructure.

Related Sessions