Results

Measured, not claimed. Every number here is read from the file that publishes it.

Real outages from the public RCAEval benchmark against the published methods run on the same cases, a held-out set kept out of development, planted faults with ablations, learning, scale, and the fix proofs run live. Where Hypocenter loses, this page says so.

01 · Real outages

All 360 RCAEval failures, three systems.

RE2 injects CPU, memory, disk, delay, loss and socket faults; RE3 plants bugs in the code. Every method is RCAEval's own implementation, run here on the same cases, scored with RCAEval's service mapping. Our BARO run reproduces the paper's Table 6 exactly.

brighter = better · RE2 Train Ticket was held out of development

All 360, ranked

What this says

    02 · Held out

    90 Train Ticket failures nobody looked at while building it.

    Settings were committed before any result outside Online Boutique was seen. RE2 Train Ticket was opened once, after v2 was frozen, and v3 was built on the other 270 cases only.

    RE2 Train Ticket · AC@1 and Avg@5

    AC@1Avg@5

    How the judge got here

    1. v1
      Pre-registered 623d340

      Settings fixed before any Sock Shop, Train Ticket or RE3 result. 155 of 270 dev cases first.

    2. v2
      Three changes from the dev cases 8de51f4

      The symptom can be the cause; Online Boutique's graph from traces plus its manifests; anomaly strength at weight 0.5, chosen on dev.

    3. v3
      Ranking past the suspects, span profile 0b4ef76

      Every alerting service ranked, then the quiet ones; where there are traces, whether a service's own work slowed (weight 0.15, chosen on dev).

    03 · Planted faults

    Hundreds of faults planted on Kestrel's topology, each with decoys.

    Five kinds: a deep change with alerts climbing to it, a silent one whose cause never alerts, an infrastructure failure with nothing changed, one change breaking two services, and a failing host. Each scenario carries a closer change, a newer one and an old one on the path. The generator is ours, so this measures the patterns the judge was built for; the held-out seeds are the fairer number.

    Switch one factor off · top-1, seed 7, 150 scenarios

    When it says…

    Confidence comes from the margin over the runner-up, not the raw score. It is conservative and ordered: high is right more often than medium, medium more than low.

    04 · Learning

    Each company's judge, refitted on its own solved incidents.

    A logistic model over the judge's factors plus each service's track record, used only if, cross-validated on solved incidents alone, it names the cause at least as often as the hand-set weights. The open incidents are the test.

    05 · Scale

    Same queries, a company of 3,000 services.

    A synthetic company built in FalkorDB with every layer the demo has: operations, pods, changes, alerts and incidents. p50 over repeated runs on a laptop.

    Query time, p50 · 1,000 vs 3,000 services log scale

    1,0003,000

    06 · Fix proofs, live

    Every planted fix, proved on a copy of the graph right now.

    Each fix is applied to its own GRAPH.COPY, the alerts whose CAUSED_BY chain ends at what it removes are cleared, and the judge is asked again. Then the prediction is checked against what the seed planted. The decoy should clear nothing.

    five copies, made and dropped
    Runs when you reach this section.

    Break it

    Seven one-line changes, each judged on a GRAPH.COPY.
    Open →

    07 · Depth, and what isn't there

    The same question at one hop, then at five. And a question about nothing.

    A one-hop look is what a join could do: the symptom's direct dependencies. The cause is often further down. Each row below is the judge run live with the search capped at that depth, checked against the cause the seed planted.

    INC-301 · checkout slow, judged at each depth

    Runs when you reach this section.

    Every open Kestrel incident, right at each depth

    On the RCAEval systems the front end calls nearly every service directly, so one hop already reaches the cause and depth doesn't change their accuracy. It matters in a layered company like Kestrel, where it is the difference between blaming a victim and finding the change.

    Ask about something that isn't in the graph

    what breaks if billing-svc goes down?

    
    
    

    
    
      
    08 · Limits
    What these numbers don't say.
    
      
    
    

    
    
      
    09 · Reproduce
    Don't trust this page. Run it.
    
      
    
        
    Every RCAEval number
    make eval        # data, signals, Hypocenter, baselines
make eval-quick  # re-score in under a minute
    
        
    Planted faults
    docker compose run --rm app \
  python -m faultline.eval --n 400 --seed 3
    
        
    Scale
    docker compose run --rm app \
  python -m faultline.scale --services 3000