Efficient Language Model Personal Viability Benchmarking

Simple, Opinionated benchmark for testing the viability of Efficient Language Models (ELMs) for personal use cases.

Uses bun, promptfoo, and ollama for a minimalist, cross-platform, local LLM prompt testing & benchmarking experience.

Setup

• Install Bun
• Install Ollama
  • Install llama3 ollama run llama3
  • Install phi3 ollama run phi3
  • Install gemma ollama run gemma
• Setup .env variables
  • cp .env.sample .env
  • Add your OpenAI API key to the .env file
• Install dependencies: bun i
• Run the minimal tests: bun minimal
• Open test viewer: bun view
• Run the ELM-ITV tests: bun elm

Guide

• First, watch the video where we walk through ELMs and this codebase.
• To get started take a look at BENCH__minimal_test_suite/ to get an idea of how to structure a basic test suite.
• Next take a look at the BENCH__efficient_language_models/ test suite to get an idea of how you can setup tests for your own viability tests for ELMs.
• Explore other ollama based models you can test
  • Or OpenAI models
  • Or Anthropic models
  • Or Groq models
• Modify the BENCH__minimal_test_suite/ or BENCH__efficient_language_models/ to suit your needs
• Create a new test with the Create a new test suite script

Folder Structure

• /BENCH__<name of test suite>
  • /prompt.txt - the prompt(s) to test
  • /test.yaml - variables and assertions
  • /promptfooconfig.yaml - llm model config

Scripts

• Create a new test suite: bun run ./scripts/new_prompt_test
• Run a test prompt against a running ollama server bun run ./scripts/ollama_local_model_call

Resources

• Ollama model library
  • https://ollama.com/library
• LMSYS Chatbot Arena Leaderboard
  • https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
• Ollama api.md docs
  • https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-chat-completion
• Promptfoo Ollama Provider
  • https://promptfoo.dev/docs/providers/ollama
• Promptfoo LLM Providers
  • https://www.promptfoo.dev/docs/providers
• Promptfoo Assertions
  • https://www.promptfoo.dev/docs/configuration/expected-outputs/