Beyond the FUSE Debate: Why Native ZFS on Object Storage Changes Everything

The Debate That Misses the Point

In early 2024, MinIO published a provocative article titled "Filesystem on Object Store is a Bad Idea". JuiceFS responded with their rebuttal, defending POSIX filesystems on object storage.

Both make valid points. Both miss the bigger picture. There's a third approach that sidesteps the entire FUSE debate: native ZFS integration with object storage via objbacker.io.

MinIO's Argument: FUSE Is Fundamentally Broken

To prove their point, MinIO benchmarked s3fs-fuse copying a 10GB file. The result? Over 5 minutes with I/O errors, versus their native S3 API.

Here's the problem: they generalized an entire architectural approach based on ONE implementation—and they picked the worst one. s3fs-fuse is widely known to be rudimentary. It's a rookie mistake to dismiss an entire category based on its weakest example.

They also never showed their own mc cp timing for comparison. Classic strawman argument.

The Fundamental Flaw in MinIO's Argument

MinIO claims: "There is simply no need for a filesystem utility in the middle of MinIO and your application!"

This ignores a fundamental reality: Linux syscalls are POSIX.

Not every application can be rewritten to use S3 SDKs. Consider:

• HPC workloads — Supercomputers and scientific computing run on Lustre, GPFS, BeeGFS—all POSIX
• Legacy applications — Millions of lines of code expecting file I/O
• Databases — PostgreSQL, MySQL, Oracle all use file I/O
• ML frameworks — PyTorch, TensorFlow checkpoint to files
• Analytics tools — Spark, Hadoop expect HDFS/POSIX interfaces
• Media workflows — Video editing, rendering pipelines use files

The world's fastest supercomputers—Frontier, Aurora, LUMI—all use POSIX filesystems. National labs, research institutions, and enterprises have decades of code built on POSIX I/O. Telling them to "just use S3" is disconnected from reality.

And what about decades of data management practices?

• Snapshots — Instant, space-efficient point-in-time copies
• Clones — Writable snapshots for dev/test
• Replication — Efficient block-level sync
• Compression — Transparent, algorithm-selectable
• Checksums — End-to-end data integrity
• Scrubbing — Proactive corruption detection

"But S3 has versioning!" Yes—at the object level, not the bucket level.

With S3 versioning, you can't say "restore my bucket to how it was at 3pm yesterday." You'd have to script through every object, find the right version, restore each one—and hope nothing was added or deleted in between. That's not data management. That's data archaeology.

Object storage has none of this natively. MinIO's answer? "Rewrite your applications." That's not a solution—it's an abdication.

JuiceFS's Rebuttal: Implementation Matters

JuiceFS ran the fair comparison MinIO avoided:

JuiceFS proved that a well-implemented POSIX layer can match native S3 performance. But they're still using FUSE. They still need external metadata (Redis). They're still working around the fundamental architecture rather than solving it.

The Third Way: Native ZFS Integration

MayaNAS doesn't put a filesystem on top of object storage. It extends ZFS to use object storage as a block device tier. The distinction is fundamental.

Why objbacker.io Is Different

┌─────────────────────────────────────────────────────────────────┐
│                     APPLICATION LAYER                           │
│                    (NFS/SMB Clients)                           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                         ZFS (Kernel)                            │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐  │
│  │   ARC Cache  │  │  L2ARC SSD   │  │   ZIL (SLOG on SSD)  │  │
│  └──────────────┘  └──────────────┘  └──────────────────────┘  │
│                              │                                  │
│  ┌───────────────────────────┴───────────────────────────────┐ │
│  │                      VDEV Layer                            │ │
│  │  ┌─────────────────┐         ┌─────────────────────────┐  │ │
│  │  │  Special VDEV   │         │    objbacker.io VDEV    │  │ │
│  │  │  (NVMe SSD)     │         │    (Object Storage)     │  │ │
│  │  │                 │         │                         │  │ │
│  │  │  • Metadata     │         │  • 1MB blocks           │  │ │
│  │  │  • Small blocks │         │  • Large files          │  │ │
│  │  │  • Dedup tables │         │  • Cold data            │  │ │
│  │  └────────┬────────┘         └────────────┬────────────┘  │ │
│  └───────────┼───────────────────────────────┼───────────────┘ │
└──────────────┼───────────────────────────────┼─────────────────┘
               │                               │
               ▼                               ▼
        ┌──────────────┐              ┌──────────────────┐
        │  Local NVMe  │              │   S3/GCS/Azure   │
        │    SSD       │              │  Object Storage  │
        └──────────────┘              └──────────────────┘

Key Architectural Differences

The Metadata Problem—Solved by ZFS Architecture

MinIO's core criticism is valid: metadata operations on object storage are slow. Every ls, every stat, every directory traversal becomes an HTTP request.

JuiceFS solved this by adding Redis. Now you need a Redis cluster. More infrastructure. More failure modes.

The ZFS special VDEV stores:

• All metadata — Directory entries, file attributes, extended attributes
• Small blocks — Files smaller than a configurable threshold (e.g., 64KB)
• Deduplication tables — If dedup is enabled

Only large data blocks (1MB recordsize) go to object storage via objbacker.io. This means a 10GB file becomes ~10,000 objects—but metadata operations hit local NVMe at microsecond latency.

Performance: Real Numbers

Understanding the Benchmark Methodology

MinIO's 10GB file copy test is equivalent to running:

fio --name=test --size=10G --numjobs=1 --rw=write --bs=1M --direct=1

That's a single-threaded sequential write. It's the simplest possible storage benchmark. And here's the critical limitation they don't mention:

Real workloads don't operate this way. They have concurrent I/O, multiple files, parallel threads. This is where architecture matters.

MayaNAS Validated Throughput

Test configurations:
AWS: c5n.9xlarge, 6 S3 buckets, 1MB recordsize, special VDEV on NVMe
GCP: n2-standard-48, 20 GCS buckets, 75 Gbps TIER_1 networking

Test 1: 10GB File Copy (mc cp equivalent)

MinIO's benchmark used mc cp with a 10GB taxi dataset CSV. We replicated this with a simple Linux cp command—the most straightforward comparison possible:

# Create 10GB test file (random data, incompressible)
dd if=/dev/urandom of=/tmp/10gb-testfile bs=1M count=10240

# Copy to MayaNAS (ZFS + objbacker.io)
time cp /tmp/10gb-testfile /minio-pool/testfs/

Same test with fio for precise measurement:

fio --name=test --size=10G --numjobs=1 --rw=write --bs=1M \
    --ioengine=psync --directory=/pool/test --end_fsync=1

# Result: 10GB in 5.13s = 2.09 GB/s (with explicit fsync)

The difference? objbacker.io writes to multiple buckets in parallel. Each bucket handles ~400 MB/s, and they aggregate. No FUSE context switches. No userspace overhead. No Redis round-trips.

The Scaling Advantage mc cp Can Never Match

Here's what MinIO can't do: mc cp writes a single object to a single bucket. Even with multipart upload, it's still one destination. You can't stripe across buckets. You can't use multiple prefixes in parallel. The architecture has a hard ceiling.

With objbacker.io, adding more buckets directly increases throughput. The ZFS VDEV layer stripes I/O across all configured buckets automatically. This is the architectural advantage of treating object storage as block devices rather than as a file destination.

Read Performance: Where ZFS Really Shines

What about reading? S3 supports byte-range GET requests, and mc cp can use ~4 parallel connections for downloads. But again—it's reading one object from one bucket.

ZFS has aggressive read-ahead prefetching for sequential workloads:

• Pattern detection — ZFS detects sequential access and prefetches ahead
• Parallel bucket reads — With objbacker.io, prefetch triggers concurrent GETs across all buckets
• ARC cache — Hot data stays in memory, eliminating repeated object fetches
• L2ARC on SSD — Warm data cached on local NVMe for sub-millisecond access

10GB Cold Read Benchmark (cache dropped)

# Cache dropped before test to ensure cold read from object storage:
echo 3 | sudo tee /proc/sys/vm/drop_caches

fio --name=test --size=10G --numjobs=1 --rw=read --bs=1M \
    --ioengine=psync --directory=/pool/test

Peak throughput reached 1.98 GB/s before the 10GB file ended—ZFS prefetch was still ramping up. With larger files, throughput stabilizes even higher as prefetch fully engages across all buckets.

For sustained sequential reads with larger datasets, ZFS prefetch combined with multi-bucket striping delivers 8.14 GB/s on GCP—far beyond what any single-object download can achieve.

Why ZFS 1MB Recordsize Matters

ZFS with objbacker.io uses 1MB recordsize. A 10GB file becomes ~10,000 S3 PUT operations. This is optimal for object storage because:

• Reduced API costs — Fewer PUT requests than smaller blocks
• Better throughput — Each PUT transfers meaningful data
• TRIM support — Deleted blocks are removed from object storage (cost savings)
• Compression efficiency — 1MB gives LZ4/ZSTD good compression ratios

Test 2: Pandas Small-File Iterations (JuiceFS benchmark)

JuiceFS also tested a pandas workload: 100 iterations of read-modify-write on a small CSV file. This tests metadata operations and small-file handling:

# 100 iterations: read CSV → append row → write CSV
for i in range(100):
    df = pd.read_csv(test_file)
    df = pd.concat([df, new_row])
    df.to_csv(test_file)

Why is MayaNAS so fast? ZFS ARC (Adaptive Replacement Cache):

100 pandas iterations (0.12s total):
     ↓
All reads/writes hit ZFS ARC (memory)
     ↓
TXG commit at end → single write to storage
     ↓
File size: 1,508 bytes → tiny, unnoticeable commit

ZFS keeps hot data in ARC during the 100 iterations. The final commit is fast because of layered intelligence:

• ZFS ARC — Hot data stays in memory during iterations
• ZFS Special VDEV — If configured, small files commit directly to local NVMe SSD
• objbacker.io — Even without special VDEV, handles small/unaligned writes efficiently

No 100 round-trips to object storage. No API cost explosion.

This is intelligent tiering. MinIO's approach (direct S3 writes) means every iteration hits object storage—100 PUT operations, 100 latency penalties, 100x the API cost. ZFS + objbacker.io batches intelligently at multiple layers. You get POSIX semantics without the overhead.

When Each Approach Makes Sense

Use MinIO Directly When:

• Your application is S3-native (data lakes, analytics)
• You don't need POSIX semantics
• Simple archival storage

Use JuiceFS When:

• You need POSIX but can manage Redis infrastructure
• Kubernetes-native deployments
• You're already invested in their ecosystem

Use MayaNAS + objbacker.io When:

• You need maximum performance with POSIX
• You want no external dependencies (no Redis, no FUSE)
• You need enterprise features: snapshots, clones, replication, compression
• You have mixed workloads: hot data on SSD, cold data on object storage
• You want data integrity: ZFS checksums, scrubbing, self-healing
• You're running NFS/SMB for traditional file sharing

Conclusion: Architecture Matters

MinIO's argument boiled down to: "Don't put a filesystem between your application and object storage." But here's what our benchmarks prove—having a filesystem in between doesn't break anything. It makes everything better.

Pandas doesn't slow down with MayaNAS between it and object storage. It speeds up. cp doesn't become a bottleneck. It becomes 7x faster than mc cp. Why? Because an intelligent filesystem layer does what applications cannot:

• Intelligent caching: Hot data stays in ARC, cold data lives in object storage
• Transaction batching: 100 writes become a handful of efficient TXG commits
• Parallel streaming: Multi-bucket striping saturates network bandwidth
• Aggressive read-ahead: ZFS prefetches what your application will need next
• Write coalescing: Small random writes become large sequential objects

MinIO generalized from one bad implementation (s3fs-fuse) to condemn all filesystems. That's like condemning all databases because one SQL implementation was slow.

The question was never "filesystem or object storage?" The question is "Why would you talk directly to object storage when such an intelligent filesystem layer is available?"

objbacker.io answers that question.