A prominent trend in disaggregated storage is the use of Non-Volatile Memory Express over Fabric (NVME-oF) to connect to storage clusters. DPUs provide local NVMe controller to remote NVMe over Fabric storage offload, where the NVMe devices can appear as local PCIe controllers to the VMs and containers but can reach to namespaces provided over NVMe over Fabric target storage with the help of the DPU without the involvement of the host. This paradigm is getting standardized in AI/ML storage workloads as we see the newer transport protocols like Falcon/UEC adopting this method of storage disaggregation. As on-prem, hybrid and bare metal tenancy become more prevalent, most hyperscalers and datacenter operators are looking at DPUs to provide streamlined and fine-grained control to their infrastructure. DPUs are designed to be a host-independent infrastructure endpoint that not only exposes the device’s storage services to the host but also at the same time securely exposes the device’s capabilities and provisioning control to the broader infrastructure and orchestration mechanisms.
At SDC 2023 we delved into the details and capabilities of DPUs as a solution for disaggregated storage over fabric, specifically NVMe over TCP (https://www.youtube.com/watch?v=d5uqAR0Ny04&list=PLH_ag5Km-YUae2Ba7JNPqI...).
In this presentation we will focus on a specific challenge associated with using DPUs for disaggregated storage when using NVMe over TCP which uses DPU memory as bounce buffers for transmission and reception of NVMe over TCP traffic. Since the NVMe over TCP traffic runs on the DPU, there is an intermediate step of bouncing the data through the DPU DDR memory. This increases latency, causes DDR bandwidth issues on DPU and reduces the achievable performance overall. With recent linux networking stack advancements like dmabuf, TCPdevicemem techniques , these shortcomings can now be overcome.
This talk will explore the details of these techniques and optimizations.
Optimizing NVMe Over TCP Disaggregated Storage architecture on DPUs
Mon Sep 16 | 4:35pm
Location:
Winchester
Abstract
Learning Objectives
Understand how DPUs provide disaggregated storage for VMs and containers in cloud and datacenter environment, the existing trends and future directions.
Understand the performance and latency problems in current DPU architectures for disaggregated storage and the solutions for overcoming these problems.
Understand how DPU based disaggregated storage is being standardized through newer transport protocols being developed in the industry.
---
Satananda Burla
Marvell
Related Sessions
SNIA Developer Conference
September 16-18, 2024
SDC 2024 is brought to you by SNIA. SNIA is an industry association committed to its mission of worldwide leadership developing and promoting architectures, standards, education and vendor-neutral collaboration.
