1. Tooling drift. Mixing AnalysisRun semantics with Flux event payloads breaks the canary verdict because the controllers measure different things and write different fields into their status objects.
2. Token churn. Single-secret bearer tokens stall ImageUpdateAutomation reconciles the moment a rotation lands mid-window, and Flux quietly retries until somebody notices the lag.
3. Silent failures. Flux retries a non-2xx receiver politely, and operators only see the trail when failure digests are broadcast on purpose with stable correlation IDs and digest links.
Webhook contract
Pin the contract before you wire anything. The Flux Receiver POSTs a JSON body to the OpenClaw ingress, and the gateway decides whether the canary slice is allowed to absorb the new image, hold at the current weight, or roll back to baseline.
Treat this contract as the source of truth: store it in the same repo as the Flux ImageRepository, ImagePolicy, and ImageUpdateAutomation manifests so reviewers see contract drift in the same pull request as the controller changes.
| Field | Source | Required |
|---|---|---|
x-flux-event |
Flux Alert provider | yes |
authorization |
Bearer token (rotated) | yes |
imageRef |
ImageUpdateAutomation status | yes |
canaryHeader |
Gateway slice routing | optional |
Multi-node config slices & canary split
Keep one OpenClaw fragment per Mac so the merge step stays diff-friendly. Then attach a five to fifteen percent canary split on the slice tagged canary=true—stable nodes keep serving baseline traffic without operator handholding.
Do this once and repeat it for every clustervps Mac that joins the fleet. The combination of declarative fragments and explicit canary headers keeps GitOps reviews short, even when ImageUpdateAutomation rewrites the same image tag twice in one afternoon.
- Mint dual bearer tokens. Mount
flux-token-currentandflux-token-next; accept either for one rotation window so ImageUpdateAutomation never trips authentication mid-cycle. - Shard config per Mac. One YAML fragment per
openclaw/nodes/<hostname>.yaml, merged through deterministic fragment merge. - Tag a canary slice. Promote one node to
canary=true; the gateway routes the configured percent of webhook replay traffic there. - Split traffic at the receiver. Flux Receiver forwards
canaryHeaderverbatim; the OpenClaw ingress applies weights aligned with multi-AZ gateway routing. - Cap retries. Bounded exponential backoff with jitter; never let a wedged gateway storm clustervps.
Merged doctor probes
Run openclaw doctor --json after each Flux event and merge the result into the response envelope. Reviewers get one artifact per webhook, not three browser tabs, and dashboards stop arguing with the gateway.
Probe verdicts feed straight into the canary decision: green allows promotion, yellow holds at the current percent, and red triggers an automatic rollback to the previous image reference recorded by ImageUpdateAutomation status.
| Probe key | Source command | Verdict |
|---|---|---|
gateway.health |
openclaw doctor --gateway |
green / yellow |
canary.latency |
p95 over fifteen minutes | versus baseline |
image.match |
ImagePolicy resolved tag | strict |
Failure summary broadcast
Mirror non-2xx replies and verdict red events to one notifier channel. Reuse the digest pipeline documented in the cluster logs webhook guide so on-call sees Flux and gateway evidence side by side.
Pair every digest with the originating Flux event UID and the OpenClaw doctor probe key so reviewers correlate ImageUpdateAutomation reconciles with gateway responses without scrolling through unrelated controller logs.
- Citable rule: keep dual bearer tokens valid for at least one full ImageUpdateAutomation interval before retiring the previous secret.
- Citable rule: cap webhook retries at four attempts with jitter; longer storms hide the root cause from the digest.
- Citable rule: ship every failure digest with the Flux event UID so GitOps reviewers cross-link automation logs without grep gymnastics.
FAQ
Is this Argo Rollouts? No. Rollouts AnalysisRun handles progressive delivery measurements with its own analysis templates; Flux ImageUpdateAutomation drives Git commits, image bumps, and reconcile loops. The webhook contract above belongs strictly to the Flux side and never reads Rollouts status.
Why per-node fragments? Deterministic merges remove rebase noise when many Macs join clustervps; reviewers diff one file per host instead of one giant manifest, and the canary tag flips with a one-line commit.
What if doctor returns yellow? Hold the canary at the current percent and broadcast a digest; promote only after two consecutive green windows so a single noisy probe does not push a regressed image to every gateway slice on clustervps.