Pain points: small files, metadata, and false parallelism
Use this guide when two to ten dedicated M4 nodes compile iOS, Flutter, or AI-sidecar projects across regions. The goal is not a distributed filesystem fashion contest. The goal is predictable build lanes, observable storage pressure, and a clean promotion path to new Mac mini M4 nodes.
- Small-file pressure: package managers, DerivedData, coverage shards, and symbol maps create thousands of low-value writes.
- Filer metadata heat: one overloaded metadata process can make every volume server look slow.
- Artifact races: rsync and CI jobs overwrite the same release folder unless promotion is locked.
Topology matrix: where SeaweedFS belongs
| Pattern | Use when | Guardrail |
|---|---|---|
| Local volume + shared filer | Most builds read many small cached files. | Best default for 2-6 nodes. |
| Regional filer pair | Asia and US workers need independent reads. | Mirror metadata backups hourly. |
| Rsync release export | Only signed artifacts move across regions. | Simple and auditable. |
| Full mesh writes | Rare; only for active-active labs. | Avoid for release CI. |
Start with one filer metadata plane per geography and volume servers on each Mac node. Keep SeaweedFS for active namespace reads. Keep rsync for promoted artifacts. This matches the parallel cluster artifact series without turning every cache into a consensus problem.
SeaweedFS volume and filer parameters
| Knob | Starting value | Why it matters |
|---|---|---|
| volume.max | 64-128 volumes per node | Limits compaction noise on APFS. |
| volume.index | leveldb or memory-mapped index | Choose after measuring small-file churn. |
| filer.collection | ci-cache, artifacts, logs | Separates retention and delete policy. |
| filer store | SQLite for tiny labs, external DB for scale | Protects metadata from node rebuilds. |
| replication | 001 locally, 010 cross-rack when needed | Do not replicate disposable caches twice. |
Pin the filer process with a health probe and log its p95 metadata latency. If filer latency rises before CPU, move metadata storage first; do not blindly add volume servers.
Synchronization plan: rsync bandwidth, concurrency, and locks
For release artifacts, use rsync with explicit ceilings. A useful baseline is two streams per region, --bwlimit=45000 to 70000 KiB/s per stream, --partial, --delete-delay, and a wall-clock timeout. Raise limits only after SSH, VNC, and Git fetch remain responsive.
flock -n /var/tmp/release-promote.lock bash -c ' rsync -az --partial --delete-delay --bwlimit=60000 ./release/ mac-sg-02:/Volumes/Artifacts/release/ ' || exit 17
- Concurrency: cap artifact exports at two streams per WAN path; local SeaweedFS reads can remain parallel.
- Build lock: lock only the promote step, not compilation, tests, or cache warm-up.
- Audit: keep dry-run logs for deletes and checksum mismatches before every signed build.
Expansion checklist: 1TB and 2TB disk watermarks
On 1TB nodes, treat SeaweedFS cache plus DerivedData as a shared budget. On 2TB nodes, reserve the extra space for retention and hot cache, not silent growth. Before expanding, pass this acceptance list: p95 filer latency stable, rsync dry-run clean, APFS free space above twenty percent, and build lock failures below one percent of jobs.
Keep one weekly capacity review separate from incident response. Record the largest archive, the biggest simulator runtime set, the longest rsync wall time, and the filer metadata backup size. Those four numbers make node expansion a measurable decision instead of a late-night argument.
Disaster recovery FAQ: restore order and OpenClaw links
What is restored first? Restore filer metadata, then required SeaweedFS volumes, then rsync release exports. Disposable caches can rebuild after CI is green.
How do OpenClaw clusters fit? Use the same gateway discipline as the OpenClaw cluster logs guide: bounded retries, append-only incident logs, and one digest per failed promotion.
When should we buy more capacity? Add multi-node Mac mini M4 capacity when two waterline checks fail in one sprint, or when release exports need more than two rsync streams to meet the window while developers still need responsive SSH sessions and stable VNC screens.
Build a cleaner SeaweedFS and rsync lane on dedicated M4 nodes
Choose matched Mac mini M4 nodes for filer metadata, volume servers, and release exporters. Scale from a two-node pilot to a cross-region cluster before disk waterlines interrupt delivery.