Evals Console
Every number shows its denominator, interval and rows. Hover a metric for its definition.
Metrics include flagged synthetic demo history (a four-week sample for illustration, not a measurement). Every card shows its real/synthetic split; the trust level below uses real decisions only. Show real only
Trust level 0: Review everything
Real decisions onlyEvery draft needs a rep's approval. Every action still needs a rep's approval in this prototype.
To reach level 1
- Real decisions ≥ 10: 1 (needs ≥ 10)
- Clean approval rate: not enough data (0/1) (needs ≥ 50%)
- Critical errors in last 20 = 0: 1 (needs = 0)shield block, wf_b730a5fb
To reach level 2
- Real decisions ≥ 20: 1 (needs ≥ 20)
- Clean approval rate: not enough data (0/1) (needs ≥ 70%)
- Fact correction rate: not enough data (0/0) (needs ≤ 20%)
- Rubber-stamp rate: not enough data (0/0) (needs ≤ 30%)
To reach level 3
- Real decisions ≥ 40: 1 (needs ≥ 40)
- Clean approval, lower 95% bound: 0% (needs ≥ 75%)
- Completion rate: not enough data (0/0) (needs = 100%)
- Write fidelity: not enough data (0/0) (needs = 100%)
Rep decisions
Reject reason mix
Show rows33% (4/12)
95% CI 14%–61%
1 real · 11 synthetic
- wrong fact4/12 · 33%
- bad timing4/12 · 33%
- too risky3/12 · 25%
- wrong action1/12 · 8%
Execution and grounding
Experiments
Experiment cards →Edit classifier
No classifier results yet. Run the feedback experiment on /evals/experiments.
Replay (2 active tests)
| Test | generate@v1 | generate@baseline |
|---|---|---|
| Follow-up states renewal date as December 31, 2026 | fail · diffExpected a sentence containing: December 31, 2026 The draft said:
live generation failed; fell back to fixture | fail · diffExpected a sentence containing: December 31, 2026 The draft said:
live generation failed; fell back to fixture |
| Keeps the rep's addition: "week, including Jana Svoboda's maintenance savings" | fail · diffExpected a sentence containing: week, including Jana Svoboda's maintenance savings The draft said:
live generation failed; fell back to fixture | fail · diffExpected a sentence containing: week, including Jana Svoboda's maintenance savings The draft said:
live generation failed; fell back to fixture |
Decision log (8)
| Workflow | Account | Outcome | Review time | Edits |
|---|---|---|---|---|
| wf_03c6c571 | Brightline Freight | rejected: too risky | 15.9s | 0 |
| wf_b730a5fb | Brightline Freight | held by the shield | – | 0 |
| wf_3fcf69a2 | Walkthrough · Ostrava Robotics SYNTHETIC | approved edited | 31.0s | 1 |
| wf_23ca674d | Walkthrough · Ostrava Robotics SYNTHETIC | rejected: wrong fact (Final approver) | 9.0s | 0 |
| wf_6f300246 | Walkthrough · Halcyon Clinics SYNTHETIC | approved clean | 18.0s | 0 |
| wf_beca6a8d | Walkthrough · Halcyon Clinics SYNTHETIC | held by the shield | – | 0 |
| wf_2d4f8e98 | Walkthrough · Brightline Freight SYNTHETIC | approved clean | 12.0s | 0 |
| wf_6383231b | Walkthrough · Brightline Freight SYNTHETIC | approved edited | 24.0s | 1 |
Edits (2)
| Before | After | Category | Severity | Decided by | Fix goes to | Workflow | |
|---|---|---|---|---|---|---|---|
| week. | week, including Jana Svoboda's maintenance savings. | missing context | major | rule | graph | wf_3fcf69a2 | |
| March 31, 2027. | December 31, 2026. | fact correction | critical | rule | graph | wf_6383231b |
Regression tests (2)
| Test | Latest result | Why | Created from |
|---|---|---|---|
| Keeps the rep's addition: "week, including Jana Svoboda's maintenance savings" | fail | live generation failed; fell back to fixture | Maya Reyes's edit on wf_3fcf69a2 |
| Follow-up states renewal date as December 31, 2026 | fail | live generation failed; fell back to fixture | Maya Reyes's edit on wf_6383231b |