01 / DEFINE
Make success explicit.
Give your agent a task, a dataset,
and a clear definition of done.
case: refund-eligible-order
expected: refund.processed
expected: refund.processed
Open source. Ready for your next agent.
Ship agents.
Ship agents.
With evidence.
Know what works. See what breaks.
One evaluation engine for LLMs, agents,
and everything between them.
return-policyInteractive example
3/ 3cases passed
Run completeReturn on day 7Passed
Return on day 30Passed
Return on day 31Passed
EXPLORE WHAT YOU CAN EVALUATE
THE EVALUATION LOOPA good answer is
A good answer is
only part of the story.
Measure what your agent actually did.
From the first input to the final outcome,
Evals keeps the evidence in view.
02 / OBSERVE
Keep the whole picture.
Collect responses, tool calls, retrieval
context, conversations, and state.
agentverify_identityprocess_refund
03 / VERIFY
Turn runs into evidence.
Apply your graders, inspect failures,
and compare against a baseline.
Outcome verified1.0
“Refund processed” is a response. A refund in the database is evidence.
Fits the way
you already build.
Start with your coding agent. Let it handle the integration.
Or take the controls yourself.
your-terminal
›
Use $evals to add meaningful
evaluations to this application.
Let’s make the behavior measurable.
- Inspect your application
- Create a suite and adapter
- Run, inspect, and verify
A few commands.
A lot less guesswork.
Read the documentationTHE CODING-AGENT ROUTE
CODEXMeet your new evaluation skill.
1
Install from the marketplace
Run these in your terminal with Codex CLI installed.
codex plugin marketplace add Vatsa10/evals
codex plugin add evals@evals2
Start a fresh session in your app
Restart Codex so it can discover the installed Evals skill.
codex3
Give it a real task
Your agent writes the adapter, creates the suite, and runs it.
Use $evals to inspect this application, add meaningful
evaluations, run them, and explain the failures.Verified in live Codex sessions. See the evidence ↗
KEEP THE FEEDBACK LOOP OPENYour agent evolves.
Your agent evolves.
Your checks should too.
Run suites continuously. Capture real requests.
Bring production failures back into your dataset.
Nested tracesConversation IDsLocal alerts
Explore monitoringsupport-conversation-1SAMPLE TRACE
support-requestCompleted
get_ordertool
format_responsellm
valid-response-jsonpassed
Your history stays with you.SQLite storage + a local JSON API.
LOCALClear answers.
Before you start.
Explore the repositoryIs Evals built on DeepEval?
Evals has its own runner, graders, tracing, and storage. We studied DeepEval, Langfuse, LangSmith, and Anthropic’s evaluation guidance for ideas. There is no DeepEval runtime dependency.
Do I need a hosted account?
No. The engine runs locally with Python 3.11+ and no third-party runtime dependencies. Your application or custom LLM judges use their own model credentials. The coding-agent route uses your existing Claude Code or Codex setup.
Can I evaluate my existing application?
Yes. Connect it with a Python function or a command that reads and writes JSON. Your adapter returns the available evidence: responses, tool calls, retrieved documents, messages, or environment state. The plugin can write this integration for you.
Can I bring my own graders and models?
Yes. Built-in checks cover exact values, patterns, JSON, numeric limits, tools, and labeled retrieval. Custom adapters support domain checks and LLM judges with your provider, rubric, scores, and evidence.
What does monitoring include today?
Continuous suite runs, request tracing, conversation IDs, online grading policies, SQLite history, local alerts, and a loopback JSON API. Hosted dashboards, external alert delivery, and provider auto-instrumentation are future extensions.
Build the next version.
Know it’s better.
Your first evaluation is a good place to start.
Get startedOpen source · Apache 2.0 · Yours to build on