harveyai/harvey-labsPublic

A benchmark built to evaluate and improve agent capabilities for supporting legal work.

AI summary: An open-source benchmark for evaluating AI agent capabilities on complex, real-world legal tasks.

Stars: 1.3K; +12 today
Forks: 229
Watchers: 15
Open issues: 21
Open PRs: 36
Contributors: ~15
Commits: 37
Branches: 14

PythonMITCreated Mar 30, 2026Last push 1d ago+42 stars this week+367 this month

Star history

since May 3, 2026

1.3K stars as of Sep 9, 2026, tracked back to May 3, 2026. Historical curve reconstructed from public GitHub event archives, calibrated to the current total.

Contribution activity

commits per day, last 52 weeks

37 commits in the last yearLessMore

Signals and awards

derived from tracked data

Actively maintained
Pushed within 48 hours
Permissive license
MIT
Continuous integration
Automated checks passing
Repeat trending
4 trending appearances

What harvey-labs does

The Legal Agent Benchmark (LAB) is an open-source evaluation framework designed specifically to assess how well AI agents perform complex tasks related to the legal profession. It provides a standardized set of challenges, datasets, and metrics based on actual legal scenarios, such as contract analysis, case law summarization, and regulatory compliance checks. The benchmark aims to move beyond generic LLM evaluations by testing agents on their ability to handle domain-specific reasoning, parse dense legal terminology, and execute multi-step workflows typical of legal practice. It includes an execution harness for running agents and sophisticated all-pass rubric scoring to ensure rigorous evaluation.

AI researchers, legal technology developers, and data scientists working on applying large language models to the legal industry.

Domain-Specific Evaluation: Tests AI capabilities using realistic, complex scenarios drawn directly from legal practice and M&A data rooms.
Standardized Benchmarking: Provides a uniform execution harness for comparing the performance of different AI models and agent architectures.
Multi-Step Task Assessment: Evaluates agents on their ability to execute complex workflows requiring sustained reasoning over long, dense documents.
Open-Source Dataset: Offers a curated collection of thousands of legal tasks, documents, and instructions available for public research.
Rigorous Scoring Methodology: Utilizes all-pass rubric scoring and LLM judge behaviors to precisely measure accuracy, compliance, and legal reasoning.

Where teams use it

Model Evaluation

AI researchers use the benchmark to rigorously test how well a newly trained LLM performs on dense legal terminology.

Agent Framework Testing

Developers of AI agents assess whether their multi-step reasoning frameworks can handle the complexities of contract analysis.

Legal Tech Procurement

Law firms utilize the standardized metrics to compare the capabilities of different commercial AI legal assistants before purchasing.

Dataset Curation

Data scientists leverage the open-source legal datasets to fine-tune their own models for better performance in the legal domain.

Getting started: git clone https://github.com/harveyai/harvey-labs.git

README

main branch

Legal Agent Benchmark (LAB): An open-source benchmark for evaluating agents on real legal work.

Harvey LAB is an open-source project aimed at benchmarking LLM agents' abilities to perform legal work in realistic environments.

LAB consists of two parts: a dataset of tasks containing agent instructions, documents, and rubrics as well as an execution harness for running and evaluating agents against those tasks.

LAB is an ongoing project and we expect to consistently add to and refine the task set and execution harness.

Read the announcement post: Introducing Harvey's Legal Agent Benchmark

Getting Started

Start with the full walkthrough in docs/tutorial.md — it takes one realistic M&A data-room assignment end to end: setup, task inspection, agent run, scoring, report review, and comparison dashboards.

Additional Documentation

Guide	Description
Architecture	Task model, harness, tools, adapters, reports, and sweeps
Evaluation Methodology	All-pass rubric scoring and LLM judge behavior
Contributing	Add tasks, model adapters, evaluation improvements, and docs

Citation

If you use Harvey LAB in your research, please cite it as:

@misc{harveylab2026,
  title   = {Harvey LAB: The Legal Agent Benchmark},
  author  = {{Harvey AI}},
  year    = {2026},
  version = {v1.0},
  url     = {https://github.com/harveyai/harvey-labs/tree/v1.0},
  note    = {Announcement: \url{https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark}}
}

View on GitHub

Recent activity

commits and pull requests

CI to validate task.json schemasMerged
#17 by spencerp · updated 9d ago
Add banking practice area: two operations-boundary tasksOpen
#153 by thefazzer · updated 12d ago
[judge] Make standard dual evaluation the defaultMerged
#150 by calvinqi · updated 15d ago
[judge] Make standard dual evaluation the default (#150)
calvinqi committed a2b429e · 15d ago
[models] Pass optional request controls through literallyDraft
#151 by calvinqi · updated 16d ago
Order judge schema reasoning-first so the verdict follows the analysisMerged
#105 by stevenobiajulu · updated 17d ago
Order judge schema reasoning-first so the verdict follows the analysis (#105)
stevenobiajulu committed 9380740 · 17d ago
[judge] Request reasoning before verdictClosed
#131 by calvinqi · updated 17d ago
test(anthropic): assert temperature omission at the SDK boundaryOpen
#103 by stevenobiajulu · updated 27d ago
firm-knowledge: add response.md output instruction + deliverable hooksMerged
#142 by JulioPereyra93 · updated 29d ago
firm-knowledge: add response.md output instruction + deliverable hooks (#142)
JulioPereyra93 committed 7be41d5 · 29d ago
firm-knowledge: update tasks to v3 rubric (#138)
JulioPereyra93 committed 60071cc · 1mo ago

Recent open issues

view all

assess-litigation-and-regulatory-risk: 5 criteria grade a $31M escrow that appears in no document or instruction
#152 · 16d ago · by samik-bind
C-045 appears unsatisfiable: requires an IC meeting date that exists only in task.json, not in any data room document
#149 · 17d ago · by hshadab
Proposal: EvalPort adapter for LAB's task/rubric/scores format (all-pass grading, criteria → graders)
#148 · 17d ago · by adhabnr-ux
firm-knowledge task 142 can score 1.0 without answering the escrow-percentage question
#147 · 23d ago · by narcolepticchicken
firm-knowledge v3 precision allowlists disagree with fixed counts/rates in 7 tasks (task 005 says 4 of 7 but lists 8)
#146 · 23d ago · by narcolepticchicken
Evaluator silently converts unreadable deliverables into judge input, producing confident wrong scores (missing pandoc -> 7/50 instead of an error)
#145 · 23d ago · by hshadab

Code frequency

additions and deletions

+4.4M lines added, -1.6M removed over the last year.

Commits per week

last 52 weeks

37 commits in the last 52 weeks.

When work happens

weekday and hour

Commit volume by weekday and hour (UTC). Larger dots mean more commits.

Date	List	Rank	Stars gained
Aug 13, 2026	daily	#11	+28
Aug 12, 2026	daily	#11	+28
Aug 11, 2026	daily	#7	+47
Aug 10, 2026	daily	#7	+47

public-apis/public-apis
A collective list of free APIs
478.6K stars · Python
donnemartin/system-design-primer
Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
369.3K stars · Python
practical-tutorials/project-based-learning
Curated list of project-based tutorials
282.9K stars · Python
NousResearch/hermes-agent
The agent that grows with you
244.2K stars · Python
yt-dlp/yt-dlp
A feature-rich command-line audio/video downloader
190.2K stars · Python
Significant-Gravitas/AutoGPT
AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
187.2K stars · Python