MayaNAS: ZFS on S3 Cloud Object Storage

Introduction to MayaNAS Cloud

Posted on Jun 23, 2023 at 7:09 PM


MayaNAS Cloud from ZettaLane Systems is a full featured enterprise-grade software defined storage solution that provides high-performance, high-throughput file storage on standard object storage (S3) without any additional disk resources. In this article, I will detail why MayaNAS with ZFS is a better way to store files on the public cloud for data analytics, AI/ML training workloads demanding high-throughput.
MayaNAS provides the scalability and reliability of object storage with the flexiblity of POSIX file storage.

ZFS is the Best File System

ZFS is a robust, scalable file system with some of the key features listed below.

Pooled Storage - ZFS with its integrated volume management easily pools available storage drives into zpool for large-scale data storage and I/O performance using striping. If needed, more drives can be added to the zpool without downtime.

Data Integrity - ZFS is the best file system with numerous techniques to provide data integrity. The file system uses 256-bit checksum as part of its metadata to detect hardware, software related data corruption and accidental overwrites.

Copy on Write - With Copy-on-write feature ZFS ensures that data is always consistent on the disk as new data is always written to a new block with calculated checksum before block pointers are changed. The old data may be retained for snapshots of the file system. Further, ZFS writes are transactional as in database — either everything or nothing is committed to disk

Data scrubbing - ZFS can be configured to scrub all the data periodically to verify its integrity and detect any silent data corruption. If possible it can also correct such data errors if the storage pool is configured with data redundancy in a mirrored or RAIDZ type configuration.

Snapshots and Checkpoint - ZFS supports unlimited low-overhead efficient snapshots that can be configured to snapshot on a regular basis. These snapshots provide application level consistency and are rolled back to the previous state if needed. In addition ZFS has a pool checkpoint which can be viewed as a pool-wide snapshot and allow user to revert the entire pool to that state or discard it.

TRIM support - When data blocks are deleted ZFS can notify the SSD with TRIM command about these unused blocks. This helps SSD GC (Garbage Collection) management to function better thereby improving performance and lifespan of storage device.

Highly Scalable object storage

Amazon S3 pioneered object storage with high availability, security, performance and unlimited scalability with bottomless storage. It provides such mammoth storage capabilities with a simple, standardized data interface over HTTPS famously known as S3. Any cloud-native solution on the cloud ought to use S3 storage for its reliability.

MayaNAS S3-backed

Now MayaNAS without creating yet another FUSE based file system combines the best of the two worlds and delivers a robust cloud file storage solution that can be accessed from NFS/SMB clients with high-throughput. The architecture includes a new VDEV objbacker to ZFS, which appears as standard block disk to ZFS forwarding IO requests to user-mode program translates read/writes requests as S3 GET/PUT and completes the IO returning back the results.
ZFS On S3  Object storage architecture

MayaNAS architecture uses all the salient features of ZFS:

Pooled storage — Any number of cloud storage buckets can be added to storage pool for increased capacity and performance increases linearly with data blocks striped on all the buckets.

ZFS large recordsize — Large blocksize of 1MB helps in achieving great throughput and reduced S3 API costs.

TRIM feature of ZFS allows deleted data blocks to be deleted on the object storage bucket saving storage costs. Data management features including compression, snapshots, cloning are readily available.

FIO Performance

We conducted a performance analysis using the fio tool giving stunningly high performance in excess of 3 GB/s even on a single VM instance. As a bonus we configured Wasabi hot storage also and compared its performance with AWS S3 side by side. The FIO arguments to evaluate basic sequential read and write performance is as follows. Here block size of 1MB with sync option is given to get the true performance without any caching skewing our results.
fio --ioengine=sync --direct=0 --fsync_on_close=1 \
--randrepeat=0 --nrfiles=1 --name=seqwrite --rw=write --bs=1m --size=10G \
--end_fsync=1 --fallocate=none --overwrite=0 --numjobs=10 \
--directory=/testwasabi/testfs/test --loops=1

fio --ioengine=sync --direct=0 \
--fsync_on_close=1 --randrepeat=0 --nrfiles=1  --name=seqread --rw=read \
--bs=1m --size=10G --end_fsync=1 --fallocate=none  --overwrite=0 --numjobs=10 \
--directory=/testwasabi/testfs --invalidate=1 --loops=1

From AWS c5n.9xlarge (36 vcpus, 96G) with buckets 1, 3, and 5
1 bucket
Write 354 MB/s537 MB/s
Read975 MB/s945 MB/s
3 buckets
Write849 MB/s1367 MB/s
Read2024 MB/s2674 MB/s
5 buckets
Write1321 MB/s2073 MB/s
Read2957 MB/s3242 MB/s
Wasabi Object storage performance on MayaNAS
When compared with one open-source FUSE based project s3backer, the performance is quite a bit slower around 50–70 MB/s unable to saturate the network bandwidth (50 Gbps) available on c5n.9xlarge. While MayaNAS with direct interface to ZFS is able to fully utilize the network bandwidth to achieve throughput in excess of 3 GB/s.


With MayaNAS experience great throughput using cost-effective cloud storage even on a single VM instance for your demanding file based workloads.

MayaNAS use of open source ZFS means no vendor lock in, and your data is protected with end-to-end data integrity. Further, use of highly-scalable, unlimited S3 storage results in tremendous cost savings without traditional block disks or SSDs.

If you would like to try MayaNAS, it is available on AWS, GCP and Azure as marketplace solutions.