-wal and -shm growth beside DerivedData.
Why WAL checkpoints feel like “random” CI slowdowns
Write-Ahead Logging is the right default for many developer tools: readers can overlap writers, commits append to the WAL file, and the database stays consistent if a process crashes mid-transaction. The pain arrives when the same Mac Mini M4 hosts are also running parallel compiles, large test shards, and container layer unpacks. A checkpoint merges WAL pages back into the main file and issues fsync-heavy IO; on a busy APFS volume—or worse, a network share with lazy metadata—that burst lands on top of build graphs and shows up as tail latency in CI dashboards, not as a SQLite error string.
- Shared volume semantics: WAL needs coherent locking across the DB,
-wal, and-shmfiles. SMB and some NAS stacks implement byte-range and advisory locks unevenly; “it worked on my laptop” is not a proof for multi-host pytest. - IO jitter coupling: Concurrent builds already randomize disk queues. Checkpoint work is periodic but large; together they produce jitter that the Watchman versus Git disk matrix treats as a first-class signal.
- Hidden disk pressure: WAL files can grow while utilization meters still look “green” if your alert ignores sidecar files next to hot databases.
If artifacts leave the node after builds, keep rsync policy aligned with the cross-region Mac Mini M4 artifact matrix so database paths and sync excludes stay consistent across regions.
WAL on shared storage: what must be true before you scale writers
Treat “shared volume” as a spectrum. Local APFS on each Mac Mini M4 with NFS used only for read-mostly inputs is very different from a single SMB-backed folder where every worker opens the same build_state.db. SQLite’s documentation is explicit that network filesystems are a policy choice, not a guarantee: you need locking behavior you have measured under parallel opens from every region that mounts the share.
- Validated stack: Run a torture script that opens the DB from two hosts, runs mixed read/write transactions, and forces checkpoints while a fake compile job hammers temp directories on the same volume.
- Prefer single-writer lanes: Even when WAL is supported, scaling writers linearly with CPU count rarely helps; you gain contention on the WAL and on the filesystem journal.
- Escape hatch: Replicate to local SSD per worker (copy-in, work, copy-out) or move hot metadata to Postgres—keep SQLite for edge caches and local developer sandboxes when cross-host coherence is the bottleneck.
For interactive shells while you run those drills on jittery links, pair with the Mosh versus SSH collaboration matrix so operators are not debugging transport and storage in the same incident.
Executable PRAGMA and journal_mode notes (staging-first)
Run these on a disposable clone of production data. Order matters: set journal_mode before heavy writes. synchronous=FULL is safer on dubious storage but amplifies checkpoint cost; NORMAL is the usual CI compromise once you trust the stack.
-- Open the DB, then: PRAGMA journal_mode=WAL; -- requires write access; may rebuild journal PRAGMA synchronous=NORMAL; -- use FULL if storage fsync semantics are unknown PRAGMA temp_store=MEMORY; -- keep temp tables off shared disk when RAM allows PRAGMA busy_timeout=8000; -- ms; backoff busy writers instead of failing fast PRAGMA wal_autocheckpoint=1000; -- pages; lower = more frequent, smaller checkpoints -- Optional: bound cache (negative = KiB); tune per host RAM, not copy-paste blindly PRAGMA cache_size=-64000; -- Inspect before/after a load test: PRAGMA journal_mode; PRAGMA page_count; PRAGMA wal_checkpoint(PASSIVE); -- TRUNCATE/RESTART only when maintenance window allows
If wal_autocheckpoint is too aggressive relative to build bursts, operators see frequent small stalls; if it is too lazy, the WAL file balloons and a single later checkpoint becomes a multi-second freeze. Adjust against your measured page size and write rate, not defaults from a template repo.
Skip mmap_size tuning on network-backed files until you profile page-in behavior; large mmap can amplify read storms during checkpoints.
Queue concurrency caps (same file, many workers)
These are conservative starting caps for one hot SQLite file on shared storage backing CI metadata (build IDs, artifact indices, flaky-test history). Re-benchmark after every major Xcode or Swift upgrade because linker IO shifts the ceiling.
| Pattern | Max concurrent writers | Max concurrent readers | Operator note |
|---|---|---|---|
| Validated SMB/NFS + WAL | 1 | 2–4 | Serialize writer tasks in the queue; readers only if torture tests pass. |
| Local APFS replica per host | 1 per replica file | Unbounded reads locally | Merge results through a single promotion step under lock. |
| Unknown / legacy NAS | 0 shared (use DELETE mode or remote DB) | Do not scale parallel opens | Risk of silent corruption outweighs convenience. |
Your orchestrator’s global job concurrency can stay high—only the tasks touching the same DB path need the narrow cap. That is the same philosophy as Nomad affinity plus build locks on Mac Mini M4, but applied at the datastore boundary.
Build locks and checkpoints: keep the critical section tiny
A build lock around “publish canonical tree” should not span long-running tests—only the atomic rename, signing, or manifest swap. Long locks stretch WAL growth because writers pile up behind the promotion gate, then checkpoints catch up all at once when the lock drops. Mirror the two-layer pattern from the Nomad guide: scheduler serialization for who may run the promote task, plus flock on a lockfile for scripts that bypass the scheduler.
Run heavy PRAGMA wal_checkpoint(RESTART) only in quiet windows—never alongside a full compile fleet unless you enjoy coordinated fsync latency incidents.
Decision matrix: where should the SQLite file live?
Use this table when a platform team proposes “one SQLite on the share for every region.” It complements the disk utilization bands you already use for 1TB versus 2TB planning.
| Topology | WAL recommendation | Checkpoint / IO risk | When to choose it |
|---|---|---|---|
| Per-node local DB + rsync/merge | WAL on APFS | Low; checkpoints local. | High parallel compile count; eventual consistency acceptable with explicit merge. |
| Single SMB/NFS gold copy | WAL only after lock torture tests | Medium–high; coupled to network RTT and lock latency. | Small team, strict single-writer queue, strong NAS ops. |
| Multi-writer “fan out” to same path | Avoid | Severe jitter and integrity risk. | Redesign: shard DBs, queue writers, or central server. |
1TB / 2TB disk waterline acceptance checklist (WAL + CI)
Before you sign off a new parallel pool, walk this list with storage and CI owners. Numbers are starting bands—tighten if snapshots or Time Machine targets share the same APFS container.
- 1TB hosts: Alert when projected
.wal+.shm+ DB footprint exceeds ~8–12 GB sustained for the largest catalog, or when total volume use crosses ~78% for more than one sprint—whichever comes first triggers a capacity review. - 2TB hosts: Allow larger WAL headroom but still cap sidecar growth: checkpoint latency scales with WAL bytes, not only free space. Treat ~85% volume use as freeze-new-experiments territory.
- Co-tenancy: Measure checkpoint duration while a representative compile job runs; p95 wall time should not exceed your job SLA budget.
- Backups: Verify backup agents copy
-wal/-shmatomically or use SQLite’s online backup API—partial copies corrupt restores. - Cross-region: If the DB is truly shared across regions, add RTT and lock wait metrics to the same dashboard as split DNS and registry latency so operators see one story.
FAQ
Should I switch to DELETE journal mode on the share? DELETE mode reduces sidecar files but increases write amplification on the main DB during transactions. It can be a reasonable interim mitigation on flaky NAS, but the durable fix is better storage or local replicas—not endless mode churn.
Does Apple Silicon change SQLite tuning? Unified memory helps page cache, but NVMe and network contention still dominate. Profile on real Mac Mini M4 hosts, not Intel VMs.
Who owns the checklist sign-off? Platform SRE plus the CI maintainer who controls queue depth; finance uses the same 1TB/2TB signals to time disk upgrades before incidents become executive threads.