Evals Console

Every number shows its denominator, interval and rows. Hover a metric for its definition.

Real only Real + synthetic

Prompt for new drafts

Trust level 0: Review everything

Real decisions only

Every draft needs a rep's approval. Every action still needs a rep's approval in this prototype.

To reach level 1

Real decisions ≥ 10: 1 (needs ≥ 10)
Clean approval rate: not enough data (0/1) (needs ≥ 50%)
Critical errors in last 20 = 0: 1 (needs = 0)shield block, wf_b730a5fb

To reach level 2

Real decisions ≥ 20: 1 (needs ≥ 20)
Clean approval rate: not enough data (0/1) (needs ≥ 70%)
Fact correction rate: not enough data (0/0) (needs ≤ 20%)
Rubber-stamp rate: not enough data (0/0) (needs ≤ 30%)

To reach level 3

Real decisions ≥ 40: 1 (needs ≥ 40)
Clean approval, lower 95% bound: 0% (needs ≥ 75%)
Completion rate: not enough data (0/0) (needs = 100%)
Write fidelity: not enough data (0/0) (needs = 100%)

Rep decisions

Clean approval rate

Show rows

Not enough data (n=1)

95% CI 0%–79%

1 real · 0 synthetic

Fact correction rate

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Rubber-stamp rate

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Reject reason mix

Show rows

Not enough data (n=1)

95% CI 21%–100%

1 real · 0 synthetic

too risky1/1 · 100%

Completion rate

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Critical errors (last 20)

Show rows

1of 2 recent events

2 real · 0 synthetic

Execution and grounding

Write fidelity

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Grounding pass rate

Show rows

0% (0/10)

95% CI 0%–28%

10 real · 0 synthetic

Replay pass rate

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Experiments

Experiment cards →

Unsupported claim rate

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Routing accuracy

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Graph corrections

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Classifier accuracy

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Human burden

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Critical actions auto-run

Show rows

0of 0 recent events

0 real · 0 synthetic

Judge agreement (verify.ts)

Show rows

Not enough data (n=0)

0 real · 0 synthetic

Edit classifier

No classifier results yet. Run the feedback experiment on /evals/experiments.

Replay (2 active tests)

Test	generate@v1	generate@baseline
Follow-up states renewal date as December 31, 2026	fail · diff Expected a sentence containing: December 31, 2026 The draft said: ~~Great to hear all 140 seats are live.~~ ~~As Luis asked, I'll send over the SOC 2 Type II report and API rate-limit doc today.~~ ~~Next step on our side: Security review call, week of October 12, 2026.~~ ~~I have your renewal down for December 31, 2026.~~ ~~I understand the budget owner for the expansion is not decided yet.~~ live generation failed; fell back to fixture	fail · diff Expected a sentence containing: December 31, 2026 The draft said: ~~Great to hear all 140 seats are live.~~ ~~As Luis asked, I'll send over the SOC 2 Type II report and API rate-limit doc today.~~ ~~Next step on our side: Security review call, week of October 12, 2026.~~ ~~I have your renewal down for December 31, 2026.~~ ~~I understand the budget owner for the expansion is not decided yet.~~ live generation failed; fell back to fixture
Keeps the rep's addition: "week, including Jana Svoboda's maintenance savings"	fail · diff Expected a sentence containing: week, including Jana Svoboda's maintenance savings The draft said: ~~Congratulations on the pilot: 18% faster line changeover.~~ ~~Thanks for confirming you're about 450 people after the spin-off; I'll size the proposal on that.~~ ~~I'll put together the one-page ROI summary this week.~~ ~~I'll address it to Tomas Hruby, who you mentioned has final sign-off.~~ live generation failed; fell back to fixture	fail · diff Expected a sentence containing: week, including Jana Svoboda's maintenance savings The draft said: ~~Congratulations on the pilot: 18% faster line changeover.~~ ~~Thanks for confirming you're about 450 people after the spin-off; I'll size the proposal on that.~~ ~~I'll put together the one-page ROI summary this week.~~ ~~I'll address it to Tomas Hruby, who you mentioned has final sign-off.~~ live generation failed; fell back to fixture

Decision log (8)

Workflow	Account	Outcome	Review time	Edits
wf_03c6c571	Brightline Freight	rejected: too risky	15.9s	0
wf_b730a5fb	Brightline Freight	held by the shield	–	0
wf_3fcf69a2	Walkthrough · Ostrava Robotics SYNTHETIC	approved edited	31.0s	1
wf_23ca674d	Walkthrough · Ostrava Robotics SYNTHETIC	rejected: wrong fact (Final approver)	9.0s	0
wf_6f300246	Walkthrough · Halcyon Clinics SYNTHETIC	approved clean	18.0s	0
wf_beca6a8d	Walkthrough · Halcyon Clinics SYNTHETIC	held by the shield	–	0
wf_2d4f8e98	Walkthrough · Brightline Freight SYNTHETIC	approved clean	12.0s	0
wf_6383231b	Walkthrough · Brightline Freight SYNTHETIC	approved edited	24.0s	1

Edits (2)

Before	After	Category	Severity	Decided by	Fix goes to	Workflow
week.	week, including Jana Svoboda's maintenance savings.	missing context	major	rule	graph	wf_3fcf69a2
March 31, 2027.	December 31, 2026.	fact correction	critical	rule	graph	wf_6383231b

Regression tests (2)

Test	Latest result	Why	Created from
Keeps the rep's addition: "week, including Jana Svoboda's maintenance savings"	fail	live generation failed; fell back to fixture	Maya Reyes's edit on wf_3fcf69a2
Follow-up states renewal date as December 31, 2026	fail	live generation failed; fell back to fixture	Maya Reyes's edit on wf_6383231b