# Crash lessons from the fleet ## Purpose This is the first evidence-based failure ledger for Raindesk and the systems around it. It records what the archives actually show, what we infer, and what we should change in response. The aim is not to make the work look more reliable than it was. A crash, a quota wall, a blank surface, a corrupt file, a missing credential, and a misleading health signal are all useful evidence when they remain attached to their context. ## Reading the evidence - **Observed** means an archive or log records the event directly. - **Documented risk** means a stress test or design document predicts a failure mode that may not yet have happened in production. - **Inference** means a likely explanation that still needs a confirming probe. - **Control** means a later design already introduced to prevent recurrence. ## Incident register ### 1. Abyss HELM returned HTTP 503 **Observed:** `2026-07-28T15:13:02Z` and `2026-07-28T15:17:26Z` in `/Users/uprootiny/abyss/ops/logs/abyss-helm.log`. ```text ERROR tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms ``` **What we know:** the service emitted two immediate 503 responses. The log does not identify the request path, caller, upstream dependency, or whether the service recovered afterward. **What we must not claim:** this is not enough to say that the whole service crashed. It is a confirmed failed response, not a confirmed process exit. **Lesson:** every liveness record must preserve the request, status, latency, process state, and recovery state. “503” without a path and a follow-up check is an incomplete incident. **Next control:** add structured health events with `service`, `route`, `status_code`, `observed_at`, `process_id`, and `recovered_at` where known. ### 2. Abyss services were relaunched repeatedly during development **Observed:** `abyss-vessels.log` records repeated successful launches of `abyss-web` on ports `3033` and `3034`, with several rebuild cycles and package or artifact lock waits. `abyss-helm.log` records repeated rebuilds before the 503s. **What we know:** the development processes were repeatedly stopped and started. The archive does not distinguish intentional reloads from crashes. **Inference:** development churn, file-lock contention, or an unstable process supervision loop may have been contributing to the restart pattern. **Lesson:** a restart is an event worth recording even when it ends in a clean launch. The archive must distinguish `manual_restart`, `build_restart`, `crash_restart`, and `supervisor_restart`. **Next control:** assign each process a run id and record start, stop, exit code, signal, reason, and duration. ### 3. One Gemini key stalled the system at the quota wall **Observed:** the Raindesk engineering journal records a hard 20-request/day Gemini free-tier ceiling and a single-key implementation that stalled for hours after the ceiling was reached. Source: `/Users/uprootiny/common-lisp/raindesk/LEARNING_JOURNAL.md` and `JOURNAL.md`, entry dated `2026-07-08`. **Control introduced:** key rotation, model rotation, parsed retry delays, and adaptive backoff. **Lesson:** credentials are runtime capacity, not merely configuration. A provider key needs a budget, remaining capacity, cooldown horizon, and failure history. When a key is exhausted, the interface should say so plainly rather than appearing frozen. **Raindesk mark:** `quota wall` with a link to the provider history and the recommended fallback. ### 4. Incorrect model names produced 404s **Observed:** the same engineering journal records 404 errors when the system steered to model names that were not valid for the `v1beta` endpoint. **Lesson:** a model identifier is an endpoint contract, not a display label. Provider, endpoint version, model name, and last successful probe must travel together. **Next control:** store the exact echoed model and endpoint version in every usage event; probe model compatibility before allowing it into a fallback rotation. ### 5. The Lisp process was vulnerable to `Killed: 9` **Observed:** the engineering journal identifies aggressive garbage collection and insufficient SBCL dynamic-space sizing as necessary on macOS to avoid `Killed: 9`. **What we know:** the archive documents the failure mode and its operational control, but does not contain a core dump or a per-process memory trace. **Lesson:** an abrupt process death needs memory context. Record peak RSS, dynamic-space size, queue length, and the last completed mission so a future death can be distinguished from an OS kill, an application panic, or a host restart. ### 6. Intake source was corrupted by a blind write **Observed:** `intake.lisp` was corrupted with line-number prefixes during a blind write, then manually repaired and load-tested. Source: `/Users/uprootiny/common-lisp/raindesk/JOURNAL.md`, entry dated `2026-07-08`. **Lesson:** generated or transformed source must be checked before it replaces the working file. The safe pattern is inspect → write a temporary artifact → parse/load/test → atomic rename. **Raindesk mark:** `artifact integrity` rather than `crash`; the damage was to the source artifact, not necessarily to a running process. ### 7. Ledger startup scales linearly with history **Documented risk:** `STRESS_TEST.md` marks `rebuild-stats-from-ledger` as failed for scalability. At one million missions, a full `ledger.jsonl` scan is expected to exceed 30 seconds at startup. **Lesson:** durable history and fast startup are separate requirements. A ledger must have checkpoints, offsets, compaction, and a way to resume after a partial tail. **Next control:** checkpoint the stats hash and last read offset; expose scan duration and replayed event count as liveness indicators. ### 8. Fixed cooldowns mishandled provider retry timing **Documented risk:** the stress test says `drain-pit` used a fixed three-second sleep and a blocking 60-second cooldown. A five-second provider retry delay caused unnecessary waiting; a two-minute delay caused premature retries. **Lesson:** retry timing is evidence from the provider, not an aesthetic delay. Honor `retry-after` or provider-specific retry metadata, cap it, jitter it, and show the next attempt time. ### 9. Append-only JSON could be left partially written **Documented risk:** `ledger-write!` appends directly. A full disk or SIGKILL during a write could leave a partial JSON line and break the next boot. **Lesson:** “append-only” does not mean “atomic.” The reader needs a recovery policy for a truncated tail, and the writer needs a durable flush or a safer segment/swap strategy. **Next control:** record a checksum or length-prefixed envelope, fsync the completed segment, and make tail repair visible rather than silently dropping data. ### 10. Single-threaded mission draining is a throughput bottleneck **Documented risk:** the stress test marks `drain-pit` as limited because one worker processes the mission queue. **Lesson:** concurrency should be bounded and observable. Adding workers without a thread-safe queue, idempotency key, and provider budget would trade a slow system for a noisy one. **Next control:** expose queue depth, active workers, oldest mission age, and per-provider concurrency. ### 11. Provider availability was mostly blocked by credentials or account state **Observed:** Tokenmill’s `2026-06-21T01:31:04Z` probe status recorded Pollinations and OpenRouter as `live_ok` with HTTP 200. Most other providers were `blocked_missing_credential`; Cloudflare required both a token and an account id. Earlier probe history recorded invalid GitHub CLI authentication. Source: `/Users/uprootiny/landings/tokenmill/probe_status.json`, `probe_status.md`, and `probe_history.json`. **Lesson:** “offline” is too coarse. A provider can be live, missing a key, invalidly authenticated, account-gated, endpoint-uncertain, rate-limited, or actually down. These states deserve different marks and different suggested resolutions. **Raindesk mark set:** ```text live_ok · missing_credential · invalid_credential account_gate · endpoint_uncertain · rate_limited · failed_probe ``` ### 12. Public Abyss exposure was deliberately constrained **Control:** the Abyss deployment contract keeps the service on loopback behind Caddy, disables inference publicly, stores credentials in memory only, pauses on authentication failure, and cancels sibling calls on pause or forget. Source: `/Users/uprootiny/pretty/abyss-next/README.md` and `/Users/uprootiny/abyss/DEPLOYMENT_STRATEGY.md`. **Lesson:** this is a good example of learning from anticipated failure. The interface should communicate that a public heartbeat is not an API-health assertion, and a configured key is not permission to start recurring work. ### 13. Deployments have had 502/offline and partial-copy risks **Observed/documented:** `landings.caddyfile` describes previously-502 hosts, including `report.solvulator.com` forwarding while port `8091` was offline. The landings roadmap identifies partial remote copies, unclear live-versus- snapshot state, and DNS/cache confusion as deployment risks. **Lesson:** a deployment mark needs both routing evidence and content evidence. “Caddy has a block” is not the same as “the page is reachable,” and “the page is reachable” is not the same as “the intended commit is serving.” **Next control:** post-deploy checks should record DNS, TLS, root HTML, assets, provenance, manifest, snapshot URL, and served commit. ## Recurring patterns ### We confuse configured with healthy Several archives contain a configured route, a running process, or a positive heartbeat without a complete end-to-end observation. Raindesk should show the evidence chain: ```text source exists → route configured → process running → endpoint reachable → expected content served → dependency probe passed → recovered after last failure ``` ### We hide capacity failures inside waiting Quota ceilings, fixed cooldowns, missing credentials, and account gates can all look like a quiet interface. Every waiting state should have a reason, a next attempt or resolution, and the evidence that produced it. ### We protect the happy path better than the history Partial writes, corrupted intake, missing snapshots, and incomplete failure records all damage the future reader. The archive must be designed for recovery, not only successful execution. ### We have useful controls but inconsistent vocabulary The newer Abyss and Tokenmill designs already distinguish paused, faulted, missing credential, live probe, and public read-only states. Those terms should become shared Raindesk vocabulary rather than remaining project-local. ## What the calm desk should show The main page should keep its calm list-first composition. Each project row can carry a small evidence strip: ```text WEFT SCHEMA · TEXT · ARCHIVE source ↗ deployed ↗ last checked 2d ago ▁▃▅▇▇▂▁ history: 1 warning · no confirmed crash record ``` For a project with real incidents: ```text ABYSS HELM INSTRUMENT · SERVICE deployed ↗ last checked 4h ago 2 failed responses in archive ↓ read incident history ↓ suggested controls ``` Use quiet states, not traffic-light theater: - `verified` — recent end-to-end observation succeeded - `deployed` — route or artifact is configured, health not established - `unverified` — source or route exists, but no recent check - `faulted` — recent failure observed - `recovering` — failure observed and a later successful observation exists - `dormant` — no recent activity; not necessarily broken - `blocked` — missing credential, account, or required dependency ## Future incident record ```json { "id": "incident-abyss-helm-20260728-151302", "project": "abyss-helm", "kind": "failed_response", "severity": "warning", "observed_at": "2026-07-28T15:13:02Z", "source": "/Users/uprootiny/abyss/ops/logs/abyss-helm.log", "evidence": { "status_code": 503, "latency_ms": 0 }, "cause": {"status": "unknown"}, "recovery": {"status": "unknown"}, "lesson": "Record route, process state, and follow-up recovery.", "resolution": "Add structured health events and post-failure probes." } ``` ## Priorities 1. Add the incident vocabulary and evidence strip to the Raindesk project records. 2. Build a small collector for local logs, probe JSON, deployment manifests, and known service records. 3. Preserve raw evidence paths and timestamps beside every normalized record. 4. Add recovery probes so an incident can close with evidence rather than inference. 5. Only then add server-side fleet observations or recurring health checks. The guiding rule is simple: > A failure is not an embarrassing exception to the archive. It is one of the > archive’s most valuable primary sources.