Evaluation

This folder contains code and resources to run experiments and evaluations.

For Benchmark Users

Setup

Before starting evaluation, follow the instructions here to setup your local development environment and LLM.

Once you are done with setup, you can follow the benchmark-specific instructions in each subdirectory of the evaluation directory. Generally these will involve running run_infer.py to perform inference with the agents.

Implementing and Evaluating an Agent

To add an agent to OpenHands, you will need to implement it in the agenthub directory. There is a README there with more information.

To evaluate an agent, you can provide the agent's name to the run_infer.py program.

Evaluating Different LLMs

OpenHands in development mode uses config.toml to keep track of most configuration. Here's an example configuration file you can use to define and use multiple LLMs:

[llm]
# IMPORTANT: add your API key here, and set the model to the one you want to evaluate
model = "gpt-4o-2024-05-13"
api_key = "sk-XXX"

[llm.eval_gpt4_1106_preview_llm]
model = "gpt-4-1106-preview"
api_key = "XXX"
temperature = 0.0

[llm.eval_some_openai_compatible_model_llm]
model = "openai/MODEL_NAME"
base_url = "https://OPENAI_COMPATIBLE_URL/v1"
api_key = "XXX"
temperature = 0.0

Supported Benchmarks

The OpenHands evaluation harness supports a wide variety of benchmarks across software engineering, web browsing, and miscellaneous assistance tasks.

Software Engineering

• SWE-Bench: evaluation/benchmarks/swe_bench
• HumanEvalFix: evaluation/benchmarks/humanevalfix
• BIRD: evaluation/benchmarks/bird
• BioCoder: evaluation/benchmarks/ml_bench
• ML-Bench: evaluation/benchmarks/ml_bench
• APIBench: evaluation/benchmarks/gorilla
• ToolQA: evaluation/benchmarks/toolqa
• AiderBench: evaluation/benchmarks/aider_bench
• Commit0: evaluation/benchmarks/commit0_bench
• DiscoveryBench: evaluation/benchmarks/discoverybench

Web Browsing

• WebArena: evaluation/benchmarks/webarena
• MiniWob++: evaluation/benchmarks/miniwob
• Browsing Delegation: evaluation/benchmarks/browsing_delegation

Misc. Assistance

• GAIA: evaluation/benchmarks/gaia
• GPQA: evaluation/benchmarks/gpqa
• AgentBench: evaluation/benchmarks/agent_bench
• MINT: evaluation/benchmarks/mint
• Entity deduction Arena (EDA): evaluation/benchmarks/EDA
• ProofWriter: evaluation/benchmarks/logic_reasoning
• ScienceAgentBench: evaluation/benchmarks/scienceagentbench

Result Visualization

Check this huggingface space for visualization of existing experimental results.

You can start your own fork of our huggingface evaluation outputs and submit a PR of your evaluation results to our hosted huggingface repo via PR following the guide here.

For Benchmark Developers

To learn more about how to integrate your benchmark into OpenHands, check out tutorial here. Briefly,

• Each subfolder contains a specific benchmark or experiment. For example, evaluation/benchmarks/swe_bench should contain all the preprocessing/evaluation/analysis scripts.
• Raw data and experimental records should not be stored within this repo.
• For model outputs, they should be stored at this huggingface space for visualization.
• Important data files of manageable size and analysis scripts (e.g., jupyter notebooks) can be directly uploaded to this repo.