OpenAgent Eval

Key features

• Retrieval metrics — context precision and recall, Precision@K, Recall@K, hit rate, MRR, NDCG
• Generation metrics — NLI-based faithfulness and answer relevancy, hallucination detection, BLEU, ROUGE, BERTScore
• LLM-as-judge with custom criteria
• Component diagnosis — blame retrieval vs generation vs chunking
• Corpus health auditor for contradictions, staleness and duplicates
• Latency, token and cost tracking per provider

1. Dataset + config

   config.yaml

2. Retriever

   context under test

3. Metrics

   retrieval · generation · cost

4. Reports

   terminal · MD · HTML · JSON

1. 1oaeval init --interactive walks through dataset, LLM provider, retriever, metric preset and report format, then writes config.yaml.
2. 2oaeval validate checks the config before anything runs.
3. 3oaeval run executes the evaluation across retrieval, generation and performance metrics.
4. 4oaeval report renders the latest results as terminal, Markdown, HTML or JSON output.

• Python
• CLI + SDK
• LLM-as-judge
• NLI models