Ship the model update without the 2am rollback.

Orrery runs your evals on every prompt, model, and tool change, then shows you which customers would have seen the regression while there is still time to fix it.

Passing test caseRegressionNew coverage

Swap the model. See what breaks.

This is a real Orrery run on a support assistant. Switch between the current model and the candidate, and watch which cases regress and who they belong to.

Pass rate96.8% Regressions0 Cost / 1k runs$4.10
Test caseCustomer segmentGradeVerdict

Fast enough to run on every pull request.

1.2B

tokens evaluated every day across customer workloads

340ms

median time to the first verdict on a pull request

97.4%

agreement between Orrery graders and expert human reviewers

61%

fewer production incidents in the first two quarters

Three lines in CI. Then it watches every change.

01

Point it at your traces

Orrery samples real production conversations into a test set, grouped by customer segment, and keeps it fresh as usage changes.

02

Write graders in plain language

Describe what good looks like. Orrery turns it into a grader, checks it against your own labels, and tells you where it disagrees.

03

Block the merge on a regression

Every pull request that touches a prompt, model, or tool gets a verdict, a diff of the worst cases, and the cost change.

# .github/workflows/evals.yml
- name: Run Orrery
  uses: orrery/eval-action@v2
  with:
    suite: support-assistant
    compare: main
    fail-on: regression > 0.5%

# verdict, 340ms
# 412 cases, 3 regressions (refunds, enterprise)
# cost per 1k runs: $4.10 -> $2.35

Your traces stay yours.

No training on your data

Traces and grades are used to run your evals and nothing else, by contract.

EU or US residency

Pick the region at sign-up. Data never leaves it, including backups.

PII redacted on ingest

Names, emails, and card numbers are masked before a trace is stored.

SOC 2 Type II

Audited every year. The report is available under NDA.

Priced on eval runs. Never on seats.

Free

$0
  • 10,000 eval runs a month
  • One project
  • Pull request checks
Get an API key

Team

$400 / month
  • 500,000 eval runs a month
  • Unlimited projects
  • Trace sampling by segment
  • Slack verdicts
Start a trial

Enterprise

Custom
  • Self-hosted graders
  • Single sign-on and audit log
  • Dedicated region
Talk to us

Know before your customers do.