New Break the graph: seven experiments on a GRAPH.COPY

What broke.
Why.
Will the fix work?

Hypocenter follows an outage down the call graph to the change that started it, argues every suspect with evidence it can cite, and proves the fix on a copy of the FalkorDB graph before anyone approves it.

Run the hero incident Break the graph

Click the cube

localhost:8090/app · INC-302open it ↗

Real outages · higher is better

0.81

of 360 real RCAEval failures named first, in three systems (Online Boutique, Sock Shop, Train Ticket).

RCAEval's own code on the same cases: NSigma 0.78 · BARO 0.28

Fix, proved first · same means right

4/4

planted fixes cleared exactly the alerts the graph copy predicted. The decoy rollback was shown to clear nothing before anyone approved it.

GRAPH.COPY · CAUSED_BY · the judge asked again

At size · lower is better

73ms

per verdict on a 3,000-service graph (211k nodes, 252k edges), naming 197 of 198 planted causes first.

python -m faultline.scale · docs/scale.md

Measured, not claimed · make eval reproduces every number · on held-out Train Ticket NSigma ranks better (0.76 vs 0.71), and we say so

An on-call agent that proves before it blames. Every suspect is a hypothesis in the graph, every piece of evidence cites a node, and every fix is tried on a copy of FalkorDB first.

Fig 0.1

Follow it down

A change is a suspect only if every service between it and the symptom started failing after it, in order. Off the path, or too late: out, whatever the model thinks.

Fig 0.2

Argue every suspect

Each suspect is a Hypothesis node with evidence for and against, every piece citing the alert, change or past incident it rests on. Confidence is the margin, not the score.

Fig 0.3

Prove it on a copy

The fix is applied to a GRAPH.COPY: what clears, what stays, and what explains the rest. After a person approves, the real alerts are checked against that prediction.

Where it sits. It reads what your stack already emits, and decides only through the graph.

Hover a part
the routes and tools are real
Watch it run live

How it decides. INC-302, step by step: every verdict, and the query that made it.

INC-302 · Kestrel Payfrom the seed data
Fig 2.1

Product tour

See it work. The same graph behind every screen.

Published evaluation

Measured on 360 real outages. Against the published methods, on the same cases.

0.81right first · all 360
0.71held-out Train Ticket · NSigma 0.76
94.5–96.8%planted faults · three held-out seeds
Every result, the losses too

Run it

Watch an investigation. Pick an open incident; it runs live against the graph, step by step.

Run a scenario

offline investigator · no API key

Loading incidents…

Solved incidents

memory

Loading…

Getting started

Before you run it.

Does it need an LLM or an API key?

No. The offline investigator runs the same tools in the same order through the graph and writes the same case file. With ANTHROPIC_API_KEY set, Claude drives the tools instead; every claim it makes is still checked against the graph before it is shown.

What does it need to run?

Docker. docker compose up -d --build starts FalkorDB 4.14.10 and the app, seeds every tenant, and serves this site on :8090.

What happens if FalkorDB is down?

Every answer is a graph query, so nothing is shown rather than something guessed. The status pill turns red, the pages say so, and they reconnect by themselves.

Can it change production on its own?

No. It proposes one action with its proof attached; a person approves it, and then the alerts decide whether it worked.

Is the data real?

Kestrel Pay and Harbor are synthetic companies. Online Boutique, Sock Shop and Train Ticket are all 360 real failures from the public RCAEval benchmark, with real telemetry, real call graphs and real root causes.

Where do the numbers on this page come from?

make eval downloads the public data, runs Hypocenter and RCAEval's own baselines, and prints the tables in docs/rcaeval-results.md. A test fails if this page and that file disagree.

Don't trust this page

Run it yourself.

The app

docker compose up -d --build
open http://localhost:8090

Every RCAEval number

make eval        # data, signals, Hypocenter, baselines
make eval-quick  # re-score in under a minute

From any MCP client

claude mcp add hypocenter -- docker compose \
  exec -T app python -m faultline.mcp_server