Documentation

Reference

The five metrics

Tiers are governed by the weakest required metric, not the average — an agent cannot average its way to trustworthiness by excelling on some dimensions while failing a critical one.

Five Minutes

Quickstart

1. 1. Install the SDK

   bashCopy

   pip install hb-eval-sdk

2. 2. Create an agent and copy its keys from Dashboard → Agents.

3. 3. Run your first evaluation

   pythonCopy

   from hb_eval_sdk import HBEvalClient
   
   client = HBEvalClient(api_key=..., aes_key=..., signing_secret=...)
   
   def my_agent(system_prompt: str, question: str) -> str:
       return call_your_model(system_prompt, question)
   
   base_task = {
       "system": "You are a customer support agent.",
       "question": "A shipment is three days late. Respond appropriately.",
   }
   report = client.evaluate_with_battery(base_task, my_agent, n_scenarios=18)
   print(report["verdict"], report["aggregate_metrics"])

Post-Hoc Scoring

Evaluation — after the run

Evaluation runs a fault-injection battery against an agent and returns a verdict once it finishes. There are two paths, and they answer different trust questions.

Local — evaluate_with_battery

report = client.evaluate_with_battery(
    base_task, my_agent, n_scenarios=18,
)

Verified — request_verified_evaluation

report = client.request_verified_evaluation(
    agent_url="https://my-agent.example.com/run",
    base_task=base_task,
    consent=True,      # the platform will call this endpoint
    n_scenarios=30,
)

Already have a LangChain, LangGraph or CrewAI agent? Adapters wrap it into the shape both paths expect — see the Evaluate page for framework-specific snippets.

Real-Time

Monitoring — during the run

Evaluation judges a run once it is over. Monitoring watches an agent while it executes, recomputing the metrics after every step, so a collapse is visible at the step it happens rather than in a post-mortem.

Per-step signals are computed locally, in your process — real-time monitoring cannot afford a network call per step, and that is also why basic monitoring is free. Only the session summary, sent in small periodic batches, reaches the platform.

with client.monitor(agent_id="my-agent") as m:
    for step in agent.run(task):
        m.record_step(
            action=step.name,
            success=step.ok,
            had_fault=step.fault_injected,
            recovered_intentionally=step.deliberate_recovery,
            traceable=step.has_reasoning,
        )
        if m.should_halt:
            break

print(m.summary)

Sessions and live metrics appear on the Monitoring dashboard as they happen.

Measurement That Acts

Safe Halt — stop the collapse

Detecting a collapse is only half of what a safety mechanism owes you — the other half is stopping before the damage compounds.halt_policyturns a sustained breach into a halt decision.

with client.monitor(
    agent_id="my-agent",
    halt_policy={"metric": "frr", "below": 0.5, "for_steps": 3},
) as m:
    for step in agent.run(task):
        m.record_step(...)
        if m.should_halt:
            print(m.halt_reason)
            break

Public Statistics

Observatory — the field, not the lab

Published studies measure agents under controlled conditions. The Observatory publishes aggregate reliability statistics from agents running in actual deployments, so the two can be compared.

Contribution is opt-in, set from Settings, and off by default. Identifiers are dropped before a contribution is written — never stored, not merely hidden — and figures are withheld until at least five independent accounts have contributed.

No Dashboard Needed

MCP — ask your assistant

HB-Eval is a remote MCP server. Add one URL to an MCP-capable assistant, sign in once, and ask in plain language — the assistant discovers and calls the tools itself.

https://hbeval-reliability-os-production.up.railway.app/mcp

What you can ask

Only get_reliability_metrics and list_my_agents work with no agent on the account yet — the rest need one created first.

Full Walkthrough

From nothing to a monitored agent

1. 1

   Register

   Create an account at hbeval.com — free, no card.

2. 2

   Create an agent

   Dashboard → Agents → New. Copy the API key, AES key and signing secret shown once.

3. 3

   First evaluation

   Run the Quickstart snippet above with your own agent_runner. You get a verdict and the five metrics.

4. 4

   First monitored run

   Wrap your agent loop in client.monitor(...). Watch it on the Monitoring dashboard while it runs.

5. 5

   Optional: Safe Halt

   Add a halt_policy once you know which metric matters most for that agent.

6. 6

   Optional: Observatory

   Turn on contribution in Settings if you want your (anonymised) results included in the public aggregate.

7. 7

   Optional: connect MCP

   Add the server URL to Claude, ChatGPT or Gemini and ask about your agents directly.

Decision Guide

Which one do I use?