Automatic benchmarking of large multimodal models via iterative experiment programming

Setup

Install dependencies

# clone project
git clone https://github.com/altndrr/apex
cd apex

# install requirements
# it will create a .venv folder in the project root
# and install all the dependencies using flit
make install

# activate virtual environment
source .venv/bin/activate

Setup environment variables

# copy .env.example to .env
cp .env.example .env

# edit .env file
vim .env

Usage

The only entry points is main.py. It must be called with the query argument, which is the question to ask the model.

#  ask the model a question
python main.py query="Can models ... ?"

Note: the first run will take a while, as it will download the necessary models and datasets.

Configuration

The full list of parameters can be found under configs, but the most important one is:

• main.yaml: main configuration file for the entry point.

Parameters can be overwritten by passing them as command line arguments. You can additionally override any parameter from the config file by using the ++ prefix.

# limit the number of experiments to 3
python main.py query="Can models ... ?" ++max_experiments=3

Development

Install pre-commit hooks

# install pre-commit hooks
pre-commit install

Run tests

# run fast tests
make test

# run all tests
make test-full

Format code

# run linters
make format

Clean repository

# remove autogenerated files
make clean

# remove logs
make clean-logs