NVMe has already established as storage choice for modern workloads requiring low latency and high performance. On cloud it is available as NVMe ephemeral resources that can be easily attached to computing instances. Though this option is within reach it comes with other challenges
MayaScale provides Elastic NVMe using NVMe Fabric over TCP on AWS Cloud for high-performance shared storage and its performance is compared against running locally using the ephemeral local NVMe SSDs.
MayaScale works better when configured with multiple cores to provide same parallelism seen with NVMe hardware queues and also the compute instance network and storage performance is also tied to the number of provisioned CPU cores. The software has very low memory requirement only. The following i3 instances can be used for MayaScale server.
Instance Type | vCPU | Memory (GiB) | Network | Capacity | Total HW Queues |
---|---|---|---|---|---|
i3.large | 2 | 15.25 | ~10 Gbps | ||
i3.xlarge | 4 | 30.5 | ~10 Gbps | 0.8T | 4 |
i3.2xlarge | 8 | 61 | ~10 Gbps | 1.7T | 8 |
i3.4xl | 16 | 122 | ~10 Gbps | 3.5T | 32 |
i3.8xl | 32 | 244 | 10 Gbps | 6.9T | 64 |
i3.16xl | 64 | 488 | 25 Gbps | 13.8T | 248 |
The FIO benchmark ran with options numthreads=4, iodepth=32, bs=4k
Instance Type | IO pattern | Block Size | Local IOPS | Client IOPS | Local Latency (us) | Client Latency (us) |
---|---|---|---|---|---|---|
i3.large | random write | 4K | ||||
i3.large | random read | 4K | ||||
i3.xlarge | random write | 4K | 71.8k | 71.3K | 1781.89 | 1793.06 |
i3.xlarge | random read | 4K | 210k | 175K | 609.61 | 727.94 |
i3.2xlarge | random write | 4K | 181K | 180k | 706.52 | 706.55 |
i3.2xlarge | random read | 4K | 413K | 189k | 308.56 | 674.64 |
i3.4xlarge | random write | 4K | ||||
i3.4xlarge | random write | 4K | ||||
i3.8xlarge | random write | 4K |
The output shows that NVME over Fabrics is a power technology that can match the local NVMe performance with little added latency for read operations only limited by the network bandwidth of 10Gbps that was used in the testing.