MayaNAS: ZFS on S3 Cloud Object Storage
Posted on Jun 23, 2023 at 7:09 PM
Introduction
MayaNAS Cloud from ZettaLane Systems is a full featured enterprise-grade software defined storage solution that provides high-performance, high-throughput file storage on standard object storage (S3) without any additional disk resources. In this article, I will detail why MayaNAS with ZFS is a better way to store files on the public cloud for data analytics, AI/ML training workloads demanding high-throughput.MayaNAS provides the scalability and reliability of object storage with the flexiblity of POSIX file storage.
ZFS is the Best File System
ZFS is a robust, scalable file system with some of the key features listed below.Pooled Storage - ZFS with its integrated volume management easily pools available storage drives into zpool for large-scale data storage and I/O performance using striping. If needed, more drives can be added to the zpool without downtime.
Data Integrity - ZFS is the best file system with numerous techniques to provide data integrity. The file system uses 256-bit checksum as part of its metadata to detect hardware, software related data corruption and accidental overwrites.
Copy on Write - With Copy-on-write feature ZFS ensures that data is always consistent on the disk as new data is always written to a new block with calculated checksum before block pointers are changed. The old data may be retained for snapshots of the file system. Further, ZFS writes are transactional as in database — either everything or nothing is committed to disk
Data scrubbing - ZFS can be configured to scrub all the data periodically to verify its integrity and detect any silent data corruption. If possible it can also correct such data errors if the storage pool is configured with data redundancy in a mirrored or RAIDZ type configuration.
Snapshots and Checkpoint - ZFS supports unlimited low-overhead efficient snapshots that can be configured to snapshot on a regular basis. These snapshots provide application level consistency and are rolled back to the previous state if needed. In addition ZFS has a pool checkpoint which can be viewed as a pool-wide snapshot and allow user to revert the entire pool to that state or discard it.
TRIM support - When data blocks are deleted ZFS can notify the SSD with TRIM command about these unused blocks. This helps SSD GC (Garbage Collection) management to function better thereby improving performance and lifespan of storage device.
Highly Scalable object storage
Amazon S3 pioneered object storage with high availability, security, performance and unlimited scalability with bottomless storage. It provides such mammoth storage capabilities with a simple, standardized data interface over HTTPS famously known as S3. Any cloud-native solution on the cloud ought to use S3 storage for its reliability.MayaNAS S3-backed
Now MayaNAS without creating yet another FUSE based file system combines the best of the two worlds and delivers a robust cloud file storage solution that can be accessed from NFS/SMB clients with high-throughput. The architecture includes a new VDEV objbacker to ZFS, which appears as standard block disk to ZFS forwarding IO requests to user-mode program objbacker.io Objbacker.io translates read/writes requests as S3 GET/PUT and completes the IO returning back the results.
MayaNAS architecture uses all the salient features of ZFS:
Pooled storage — Any number of cloud storage buckets can be added to storage pool for increased capacity and performance increases linearly with data blocks striped on all the buckets.
ZFS large recordsize — Large blocksize of 1MB helps in achieving great throughput and reduced S3 API costs.
TRIM feature of ZFS allows deleted data blocks to be deleted on the object storage bucket saving storage costs. Data management features including compression, snapshots, cloning are readily available.
FIO Performance
We conducted a performance analysis using the fio tool giving stunningly high performance in excess of 3 GB/s even on a single VM instance. As a bonus we configured Wasabi hot storage also and compared its performance with AWS S3 side by side. The FIO arguments to evaluate basic sequential read and write performance is as follows. Here block size of 1MB with sync option is given to get the true performance without any caching skewing our results.fio --ioengine=sync --direct=0 --fsync_on_close=1 \ --randrepeat=0 --nrfiles=1 --name=seqwrite --rw=write --bs=1m --size=10G \ --end_fsync=1 --fallocate=none --overwrite=0 --numjobs=10 \ --directory=/testwasabi/testfs/test --loops=1 fio --ioengine=sync --direct=0 \ --fsync_on_close=1 --randrepeat=0 --nrfiles=1 --name=seqread --rw=read \ --bs=1m --size=10G --end_fsync=1 --fallocate=none --overwrite=0 --numjobs=10 \ --directory=/testwasabi/testfs --invalidate=1 --loops=1
| Wasabi | S3 | |
|---|---|---|
| 1 bucket | ||
| Write | 354 MB/s | 537 MB/s |
| Read | 975 MB/s | 945 MB/s |
| 3 buckets | ||
| Write | 849 MB/s | 1367 MB/s |
| Read | 2024 MB/s | 2674 MB/s |
| 5 buckets | ||
| Write | 1321 MB/s | 2073 MB/s |
| Read | 2957 MB/s | 3242 MB/s |
Summary
With MayaNAS experience great throughput using cost-effective cloud storage even on a single VM instance for your demanding file based workloads.
MayaNAS use of open source ZFS means no vendor lock in, and your data is protected with end-to-end data integrity. Further, use of highly-scalable, unlimited S3 storage results in tremendous cost savings without traditional block disks or SSDs.
If you would like to try MayaNAS, it is available on AWS, GCP and Azure as marketplace solutions.