Major Updates

Happy Thanksgiving! This release has a major architectural change.

Before:

S3 Client → S3 API → Filer → Volume Server

After (Direct Access):

S3 Client → S3 API → Volume Server
                ↓
              Filer (metadata only)

This change is benchmarked in test cases from user @yahooguntu

• reduces write latency by 30%
• reduce read latency by 5%

What's Changed

• S3

  • S3: prevent deleting buckets with object locking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7434
  • do delete expired entries on s3 list request by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7426
  • S3: Perf related by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7463
  • S3: Enforce bucket policy by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7471
  • S3: add context aware action resolution by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7479
  • Refactor data structure by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7472
  • S3: Lazy Versioning Check, Conditional SSE Entry Fetch, HEAD Request Optimization by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7480
  • S3: Directly read write volume servers by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7481
  • S3: Add tests for PyArrow with native S3 filesystem by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7508
  • Account Info by @digitalinfobr in https://github.com/seaweedfs/seaweedfs/pull/7507
  • S3: JWT generation for volume server authentication by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7514
  • S3: S3 Object Retention API to include XML namespace support by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7517
  • S3: adds FilerClient to use cached volume id by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7518
  • test read write by s3fs and PyArrow native file system for s3 by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7520
  • S3: list owned buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7519
  • S3: set identity to request context, and remove obsolete code by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7523
  • S3: Add Vary header for non-wildcard AllowOrigin by @llc1123 in https://github.com/seaweedfs/seaweedfs/pull/7547
  • S3: Fix encrypted file copy with multiple chunks (#7530) by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7546
  • fix copying for paused versioning buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7548
  • S3: Auto create bucket by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7549
  • Support multiple filers for S3 and IAM servers with automatic failover by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7550
  • Filer, S3: Feature/add concurrent file upload limit by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7554
  • fix(s3api): fix AWS Signature V2 format and validation by @qzhello in https://github.com/seaweedfs/seaweedfs/pull/7488
  • S3: pass HTTP 429 from volume servers to S3 clients by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7556
  • Metrics: Add Prometheus metrics for concurrent upload tracking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7555
  • s3api: Fix response-content-disposition query parameter not being honored by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7559

• Misc

  • Nit: use time.Durations instead of constants in seconds. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7438
  • Fix date string parsing bug for the SQL Engine. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7446
  • chore: fix the diagram in RDMA sidecar readme by @dimaqq in https://github.com/seaweedfs/seaweedfs/pull/7503
  • add build info metrics by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7525
  • hide millseconds in up time by @trim21 in https://github.com/seaweedfs/seaweedfs/pull/7553

• Docker, Helm Charts

  • Don't make nginx the default ingress controller by @facorazza in https://github.com/seaweedfs/seaweedfs/pull/7436
  • docker: fix /data ownership and permission by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7451
  • Add explicit IP and binding parameters in Docker Compose by @gophlet in https://github.com/seaweedfs/seaweedfs/pull/7533

• Master

  • master: fix negative active volumes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7440

• Shell

  • Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7450
  • Rework parameters passing for functions within volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7448
  • Rework parameters passing for functions within ec.rebuild by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7445
  • backup: handle volume not found when backing up by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7465
  • Split logic for volume.check.disk into writable and read-only volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7476
  • Parallelize ec.rebuild operations per affected volume. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7466
  • volume.check.disk: add support for uni- or bi-directional sync between volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7484
  • feat(volume.fix): show all replica locations for misplaced volumes by @CodeRusher in https://github.com/seaweedfs/seaweedfs/pull/7560

• Volume Server

  • Volume Server: avoid aggressive volume assignment by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7501
  • fix: pass proxied query param by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7477
  • Bootstrap logic to fix read-only volumes with volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7531

• Filer

  • de/compress the fs meta file if filename ends with gz/gzip by @shaofeng66 in https://github.com/seaweedfs/seaweedfs/pull/7500
  • filer store: add foundationdb by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7178
  • Add error list each entry func by @tam-i13 in https://github.com/seaweedfs/seaweedfs/pull/7485
  • fix(filer-ui): support folder creation with JWT token in URL by @littlemilkwu in https://github.com/seaweedfs/seaweedfs/pull/7271
  • fix(tikv): replace DeleteRange with transaction-based batch deletes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7557
  • fix(tikv): improve context propagation and refactor batch delete logic by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7558

• Java, Hadoop Compatible File System

  • HDFS: Java client replication configuration by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7526

New Contributors

• @dimaqq made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7503
• @shaofeng66 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7500
• @digitalinfobr made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7507
• @gophlet made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7533
• @llc1123 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7547
• @tam-i13 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7485
• @littlemilkwu made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7271

Full Changelog: https://github.com/seaweedfs/seaweedfs/compare/4.00...4.01