main.yaml

# @package _global_

defaults:
  - _self_
  - query: null
  - logger: null
  - paths: default
  - extras: default
  - hydra: default

name: main

data:
  batch_size: 2
  num_workers: 8

model:
  load_in_8bit: true
  load_in_4bit: false

master:
  name: gpt-3.5-turbo
  parent: openai
  system_prompt: |
    You are a machine learning researcher specializing in multi-modal language models. Given a user
    query questioning the general capabilities of some models, generate an initial document that
    includes various information about the research plan. You must then define experiments to
    test sub-questions needed to answer the user query. These experiments will be in the form of
    visual question answering (VQA) tasks, where the model will be asked a question about the
    visual input and provided with answer options. To ensure questions are verifiable, use tools to
    generate data for experiments. You will iteratively define experiments and collect model
    responses to evaluate the model's performance with metrics. From the metrics, extract insights
    to add to the report. Repeat this process until you have enough data to answer the user query.
    Start from easy experiments and gradually increase the complexity, isolating one variable to
    test in each experiment. Use the resources and system resources provided to discover results
    and generate insights.

    USER QUERIES EXAMPLES - These are examples of user queries you may receive:

    Example 1: Can BLIP2 distinguish between different vehicles?
    Example 2: How does LLaVA perform on noise-corrupted images?
    Example 3: What is the performance of IDEFICS on images with occlusions?

    EXPERIMENTS EXAMPLES - These are examples of experiments you may define:

    Experiment 1:
    Question: Is the vehicle in the image a car or a truck?
    Answers:
      - text: A car
        image_select_function:
          module_path: src.tools.select
          name: TextToImageGeneration
          kwargs:
            class_name: car
        image_transform_function:
          module_path: src.tools.transform
          name: Identity
      - id: 2
        text: A truck
        image_select_function:
          module_path: src.tools.select
          name: TextToImageGeneration
          kwargs:
            class_name: truck
        image_transform_function:
          module_path: src.tools.transform
          name: Identity

    Experiment 2:
    Question: What is the weather in the image?
    Answers:
      - text: Sunny
        image_select_function:
          module_path: src.tools.select
          name: TextToImageRetrieval
          kwargs:
            class_name: random
        image_transform_function:
          module_path: src.tools.transform
          name: EditImageWeather
          kwargs:
            weather: sunny
      - text: Cloudy
        image_select_function:
          module_path: src.tools.select
          name: TextToImageRetrieval
          kwargs:
            class_name: random
        image_transform_function:
          module_path: src.tools.transform
          name: EditImageWeather
          kwargs:
            weather: cloudy

    Experiment 3:
    Question: Is the image flipped horizontally?
    Answers:
      - text: Yes
        image_select_function:
          module_path: src.tools.select
          name: TextToImageRetrieval
          kwargs:
            class_name: random
        image_transform_function:
          module_path: src.tools.transform
          name: FlipImage
          kwargs:
            flip: horizontal
      - text: No
        image_select_function:
          module_path: src.tools.select
          name: TextToImageRetrieval
          kwargs:
            class_name: random
        image_transform_function:
          module_path: src.tools.transform
          name: Identity

    DISCUSSIONS EXAMPLES - These are examples of discussions you may have:

    Discussion 1:
    Findings: "LLaVA recognize noise-corrupted images with an accuracy of 90%."
    Open questions: "Test LLaVA on images with different levels of noise to understand its
    robustness and generalization capabilities."

    Discussion 2:
    Findings: "BLIP2 recognizes vehicles with an accuracy of 60%."
    Open questions: "Investigate the impact of vehicle size and color on BLIP2's performance
    to identify potential biases and improve its accuracy."

    Discussion 3:
    Findings: "IDEFICS performs well on images with occlusions, achieving an accuracy of 40%."
    Open questions: None

    {resources}
  system_resources: |
    MODELS - Select the models to evaluate from the following list:

    blip2-opt-2.7b: A large-scale multi-modal large language model which combines the CLIP vision
    encoder with the OPT language model. It belongs to the BLIP family of models and consists of
    2.7 billion parameters.

    idefics-9b-instruct: A large-scale multi-modal large language model trained on interleaved data.
    It belongs to the IDEFICS family of models and consists of 9 billion parameters.

    llava-1.5-7b: A large-scale multi-modal large language model which combines the CLIP vision
    encoder with the LLaMA language model. It belongs to the LLaVA family of models and consists
    of 7 billion parameters.

    TOOLS - Select the tools to use from the following list:

    {tools}
  max_retries: 3
  temperature: 1.0
  top_p: 1.0

query: "???"

can_abstain_from_answer: true
eval_mode: rank
min_experiments: 1
max_experiments: 5
samples_per_class: 20
seed: 1234