LVC21-208: Embracing High Performance Flash Storage with Arm

Session Abstract

Solid-state storage media is in the process of taking over the data center, whose growing advantages in performance bring a major challenge that current storage software stack overhead becomes a major bottleneck for developing high performance storage applications. Traditional kernel I/O stacks bring obvious overhead due to context switch, data copy, interrupt, resource synchronization and etc. SPDK minimizes the impact on CPU and Memory bus cycles during IO processing mainly by three ways. One is using the user-mode for storage applications rather than kernel mode. The devices that SPDK takes control are unbound from kernel space driver. And instead, UIO or VFIO driver is used for SPDK to operate devices directly from user space, which thereby eliminates costly kernel context switch. The second is that SPDK runs in a polled mode instead of interrupt mode. It polls devices for completions instead of waiting for interrupts, which reduces the related interrupt handling overhead. And the third one, the shared-nothing thread model. Separate queues exposed by a device like NVMe can be accessed without coordination. SPDK can send requests to the device from multiple threads of execution in parallel without locks for SPDK requires a hardware queue is only ever accessed from one thread at a time. Building upon such basis, SPDK further provides a full block stack as a user space library and provides NVMe-oF and vhost servers that are capable of serving disks over the network or to other processes. We can learn from these targets about how to implement a high performance storage target, or use them as the basis for production deployments. In this session, we will walk through our practice about running SPDK NVMe-over-TCP on Arm servers. We will show the IO performance and multi-dimentional comparison with the linux kernel driver. And we will show our tuning and optimization details to get performance improved. The session will have this outline: 1.software framework and the key techniques of SPDK; 2.pratice on SPDK NVMe-over-TCP to accelerate accessing remote NVMe devices on Arm servers; 3.SPDK NVMe-over-TCP tuning and optimization on Arm servers.

Session Speakers

richael zhuang

Arm (Arm China, software engineer)

Richael is software engineer from Arm. She has strong expertise in cloud storage technology, covering NVMe-oF, kernel, distribute storage system and solutions. Richael has rich experience in workloads profiling, tuning and optimization on Arm. As an active contributor to open source community, Richael is now focusing on SPDK and Ceph projects.

Solid-state storage media is in the process of taking over the data center, whose growing advantages in performance bring a major challenge that current storage software stack overhead becomes a major bottleneck for developing high performance storage applications.

Traditional kernel I/O stacks bring obvious overhead due to context switch, data copy, interrupt, resource synchronization and etc. SPDK minimizes the impact on CPU and Memory bus cycles during IO processing mainly by three ways. One is using the user-mode for storage applications rather than kernel mode. The devices that SPDK takes control are unbound from kernel space driver. And instead, UIO or VFIO driver is used for SPDK to operate devices directly from user space, which thereby eliminates costly kernel context switch. The second is that SPDK runs in a polled mode instead of interrupt mode. It polls devices for completions instead of waiting for interrupts, which reduces the related interrupt handling overhead. And the third one, the shared-nothing thread model. Separate queues exposed by a device like NVMe can be accessed without coordination. SPDK can send requests to the device from multiple threads of execution in parallel without locks for SPDK requires a hardware queue is only ever accessed from one thread at a time.

Building upon such basis, SPDK further provides a full block stack as a user space library and provides NVMe-oF and vhost servers that are capable of serving disks over the network or to other processes. We can learn from these targets about how to implement a high performance storage target, or use them as the basis for production deployments.

In this session, we will walk through our practice about running SPDK NVMe-over-TCP on Arm servers. We will show the IO performance and multi-dimentional comparison with the linux kernel driver. And we will show our tuning and optimization details to get performance improved.

The session will have this outline: 1.software framework and the key techniques of SPDK; 2.pratice on SPDK NVMe-over-TCP to accelerate accessing remote NVMe devices on Arm servers; 3.SPDK NVMe-over-TCP tuning and optimization on Arm servers.

comments powered by Disqus

Other Posts

Sign up. Receive Updates. Stay informed.

Sign up to our mailing list to receive updates on the latest Linaro Connect news!