robfox.dev

Rob Fox

Software engineer, Dublin. Open-source reliability tooling.

Your dashboards say it’s up. I build the tooling that tells you whether it was right.

Agents, rules and people make decisions no test can check. I created NthLayer and the OpenSRM spec to turn whether those decisions hold up into SLOs and alerts in the Prometheus and Grafana you already run.

Email Rob GitHub LinkedIn
service.yaml
apiVersion: opensrm.nthlayer.io/v2kind: ServiceManifestspec:  service:    name: refund-approver    type: ai-gate  # Operational health: is the gate up and fast?  slo:    - apiVersion: openslo/v1      objectives:        - displayName: 99.9% of decision requests answered          target: 0.999  # Decision quality: is the gate RIGHT?  judgment_slo:    - metadata: { name: refund-approver-reversal-rate }      spec:        judgment_type: reversal_rate        measurement: { window: 7d, source: lineage }        target: { maximum_reversal_rate: 0.05 }        breach_actions:          - create_case: { priority: P2 }# … trimmed from the OpenSRM v2 ai-gate example
Hover an underlined value to trace it. Run nthlayer apply to compile.

Up is not the same as right.

A code review bot with 99.9% availability can approve terrible code dozens of times a day. Nobody’s tracking that.

Classical SRE gave us SLOs, error budgets and deployment gates for systems we can test. But a growing share of production runs on judgment: an AI SRE suppressing an alert, a model approving a refund, a rule engine passing a deploy, a person signing off a migration. Those calls stay available, fast and cheap while they quietly go wrong.

The evidence already exists. The on-call overrides the suppression; a reviewer reverses the refund. NthLayer records each decision, waits for its outcome, and turns how often decisions hold up into judgment SLOs: reversal rate, high-confidence failure, calibration. They get the same targets, burn alerts and gates as latency.

NthLayer: declare what “right” means, compile it, then measure against it.

An open-source reliability platform built on the OpenSRM specification. A deterministic compiler produces the monitoring at build time; a three-tier runtime records decisions and settles their outcomes.

nthlayer.io GitHub
Build timeDefine → Generate
Define
opensrm

The spec. One manifest declares classical SLOs for whether a service is up and judgment SLOs for whether its decisions are right.

compiled by
Generate
nthlayer-generate

A pure compiler: deterministic, stateless, no LLM. Runs locally or as a GitHub Action gate in CI.

emits
Output
  • Prometheus rules
  • Sloth SLOs
  • Grafana dashboards
  • Backstage entities
  • PagerDuty config
  • Loki alerts
  • Service docs
RuntimeObserve → Measure → Correlate → Respond → Learn
Tier 1 · state
nthlayer-core

Verdict store, cases and the manifest catalogue. Stateful, no LLM. Core availability is product availability.

Decisions arrive as POST /verdicts

HTTP only
Tier 2 · computation
nthlayer-workers
  • observeSLOs, drift, deploy gate
  • measurejudgment SLOs
  • correlatesession windows
  • respondproposes, never acts
  • learnoutcomes, retros
Tier 3 · operator
nthlayer-bench

A terminal UI for SREs: situation board, case bench, approval flows, on-call status.

VerdictsWritten once, settled later

A verdict is one decision about one subject at one point in time. It is written once and never edited; its outcome settles later as a new verdict linked to the original.

When a judgment SLO breaches, the manifest’s declared action runs (notify, open a case, reduce autonomy), and a human acts on your systems.

RepositoryRoleWhat it does
opensrmSpecificationThe manifest format for declaring reliability requirements as code, validated by a JSON Schema.
nthlayer-generateCompilerSpecs to SLOs, alerts, dashboards, recording rules and Backstage entities; CI/CD gate via GitHub Action.
nthlayer-coreTier 1API server, verdict store, case management, manifest catalogue.
nthlayer-workersTier 2Five worker modules (observe, measure, correlate, respond, learn), all talking to core over HTTP.
nthlayer-benchTier 3Textual TUI: situation board, case bench, approval flows.
nthlayer-commonLibraryVerdict model, manifest parser, LLM wrapper and API client shared by every repo.

What’s under the hood

  • 7,000+ testsacross the compiler, the three runtime tiers and the shared library.
  • 10,000 trialsnthlayer simulate runs Monte Carlo SLO simulation over a 90-day horizon, with what-if redundancy. Pure standard library, no numpy.
  • 90+ invalid manifestsThe OpenSRM schema is tested both ways: every valid fixture must pass and every deliberately broken one must be rejected.
  • exit 2nthlayer-workers gate blocks a deploy when the service’s reliability says no.
  • 3 tiers, end to endA Docker integration test drives a reversal-rate breach through core, workers and bench and asserts the verdict chain.
  • SARIF + PR commentsA GitHub Action for CI, plus a separate OpenSRM validator Action.

Principles I build by

These aren’t aspirations. They are written into the specs and enforced in the code.

Zero Framework Cognition
Transport is code; judgment is the model’s. Shared libraries move data and never own prompts, decision trees or agent orchestration. The library does not own reasoning. It provides transport.NthLayer Common spec, §2
If it doesn’t reason, it isn’t an agent
Anything that can be deterministic is. The compiler has no LLM in it, so the same manifest produces the same monitoring every time. If a component doesn’t need to reason, it isn’t an agent.OpenSRM README, principle 5
Propose, never execute
Respond records what it would do and opens a case; a person acts on production. Autonomy only ratchets down automatically. Raising it takes a human. NthLayer executes nothing.nthlayer.io
Not evaluable is never a pass
Missing data is a state of its own. Every generated threshold alert has a twin that fires when the signal can’t be evaluated, so silence never reads as health. No data is reported as not evaluable — never as a pass.OpenSRM authoring guide
Spec first, then shift left
Every component is defined by a specification before it’s built, and reliability is decided in CI rather than discovered in production. Every component is defined by a specification first.OpenSRM README
Promote rules into code
When a convention keeps getting broken, documentation has failed, so the rule becomes a check. When documentation proves insufficient to prevent a recurring violation, the rule MUST be promoted into code.NthLayer golden principles

Sloppylint

A linter for the mistakes AI-generated Python makes and traditional linters miss: hallucinated imports, other languages leaking in, placeholder code, and confident wrongness.

  • Scores findings on four axes (noise, lies, soul, structure) and rolls them into a verdict: clean, acceptable, sloppy or disaster.
  • CI mode with a score budget (--ci --max-score 50) and JSON reports.
  • Python 3.9+, MIT licensed, on PyPI. My most-starred project, and I run it on NthLayer itself.
Sloppylint on GitHubPyPI
terminal
$ pip install sloppylint$ sloppylint . CRITICAL (2 issues)============================================================  src/api.py:23  mutable_default_arg    Mutable default argument - use None instead    > def process(items=[]):   src/db.py:15  bare_except    Bare except catches everything including SystemExit    > except: SLOPPY INDEX══════════════════════════════════════════════════Information Utility (Noise)    : 24 ptsInformation Quality (Lies)     : 105 ptsStyle / Taste (Soul)           : 31 ptsStructural Issues              : 45 pts──────────────────────────────────────────────────TOTAL SLOP SCORE               : 205 pts Verdict: SLOPPY

Landed upstream in Gas Town

Gas Town is Steve Yegge’s multi-agent workspace manager. NthLayer’s quality-measurement ideas were first proven inside its merge pipeline.

Fix-merged by Steve Yeggebd47754#2263

Guardian: quality review in the merge pipeline

Adds a judgment step to the Refinery’s merge patrol. After tests pass, the agent reviews the merge diff for correctness, security, clarity and style, scores it from 0 to 1, and records the result so quality can be tracked per worker over time. It ships as a Deacon plugin with trend analysis and alerts on degradation.

Phase 1 is deliberately measurement-only and fail-open: a review never blocks a merge. Measure first; gate later.

mol-refinery-patrol.formula.toml
@@ -346,10 +358,82 @@[[steps]]id = "quality-review"title = "Quality review merge diff"needs = ["run-tests"]Review the merge diff for quality issues. This step is measurement-only (Phase 1):reviews are recorded but do NOT gate merges.…Assign a score from 0.0 to 1.0:- 0.9-1.0: Excellent — no issues or only trivial nits- 0.7-0.89: Good — minor issues only- 0.45-0.69: Needs work — significant issues found- 0.0-0.44: Breach — critical issues (security, correctness)…Record the quality review result. This is fail-open: if the command fails,log the error and continue — never block the merge.… [[steps]] id = "handle-failures"needs = ["run-tests"]needs = ["quality-review"]

Writing

The arguments behind the code: why decisions need SLOs, why reliability needs a spec, and why it belongs in CI.

Classical SLOs measure the system, not the judgment. Judgment SLOs (reversal rate, high-confidence failure) bring targets, error budgets, burn alerts and deploy gates to decision quality, using human overrides as the signal instead of ground-truth labels.

Decision quality is a reliability problem. Let’s treat it like one.

SLOs set one service at a time hide impossible targets: three dependencies at 0.999, 0.9995 and 0.999 cap you at 99.75%, whatever your 99.99% goal says. OpenSRM declares SLOs, contracts and dependencies in one format that tooling can validate and enforce.

The syntax is deliberately boring. The value isn’t in clever YAML; it’s in having a standard format that tooling can validate, generate from, and enforce.

Shift-Left Reliability

dev.to · 12 Jan 2026 · 4 min

Teams are good at incident response but decide reliability only after changes are live. Reliability needs the same spec, validate and enforce loop in CI/CD that security already shifted left.

The only question is whether you address it before deployment — or learn about it in the postmortem.

Let’s talk about reliability.

Whether it’s NthLayer, judgment SLOs, a role, or something you’re building, I’m happy to hear from you.

GitHub LinkedIn dev.to nthlayer.io