Multi-Programming Language Evaluation of Large Language Models of Code (MultiPL-E)

MultiPL-E is a system for translating unit test-driven neural code generation benchmarks to new languages. We have used MultiPL-E to translate two popular Python benchmarks (HumanEval and MBPP) to 18 other programming languages.

For more information:

• MultiPL-E is part of the BigCode Code Generation LM Harness. This is the easiest way to use MultiPL-E.
• The Multilingual Code Models Evaluation by BigCode evaluates Code LLMs using several benchmarks, including MultiPL-E.
• We have a tutorial on how to use MultiPL-E directly.
• Read our paper MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation.
• The MultiPL-E dataset of translated prompts is available on the Hugging Face Hub.

Versions

• Version 0.4.0: Work in progress.

  • New languages: OCaml, MATLAB
  • Using .jsonl instead of .json for prompts
  • Several bugfixes to prompts

• Version 0.3.0: used to evaluate StarCoder

  • This version corrects several bugs in prompts and test cases that resulted in lower pass@k rates for some of the statically typed languages. The most significant difference is that the pass@k for Java increases by about 2% on HumanEval.

• Version 0.2.0: used to evaluate SantaCoder