Skip to content

Transforming Neural Architecture Search (NAS) into multi-objective optimization problems. A benchmark suite for testing evolutionary algorithms in deep learning.

License

Notifications You must be signed in to change notification settings

ranchengcn/evoxbench

This branch is 1 commit behind EMI-Group/evoxbench:main.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

6bbe554 Β· Apr 15, 2024
Jan 23, 2024
Mar 12, 2024
Feb 29, 2024
Apr 15, 2024
Aug 10, 2022
Aug 10, 2022
Aug 10, 2022
Mar 12, 2024
Mar 12, 2024

Repository files navigation

EvoXBench Logo
Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment arXiv


EvoXBench is a platfrom offering instant benchmarking of evolutionary multi-objective optimization (EMO) algorithms in neural architecture search (NAS), with ready to use test suites. It facilitates efficient performance assessments with NO requirement of GPUs or PyTorch/TensorFlow, enhancing accessibility for a broader range of research applications. It encompasses extensive test suites that cover a variety of datasets (CIFAR10, ImageNet, Cityscapes, etc.), search spaces (NASBench101, NASBench201, NATS, DARTS, ResNet50, Transformer, MNV3, MoSegNAS, etc.), and hardware devices (Eyeriss, GPUs, Samsung Note10, etc.). It provides a versatile interface compatible with multiple programming languages (Java, Matlab, Python, etc.).


πŸ“’ Latest News & Updates

  • EvoXBench has been updated to version 1.0.5! This latest release addresses bugs in CitySeg/MOP10 and HV calculation in the CitySeg/MOP test suite.

    If you're already onboard with EvoXBench, give this command a spin: pip install evoxbench==1.0.5.

⭐️ Key Features

πŸ“ General NAS Problem Formulation

  • Formulating NAS tasks into general multi-objective optimization problems.
  • Exploring NAS's nuanced traits through the prism of evolutionary optimization.

πŸ› οΈ Efficient Benchmarking Pipeline

  • Presenting an end-to-end worflow for instant benchmark assessments of EMO algorithms.
  • Providing instant fitness evaluations as numerical optimization.

πŸ“Š Comprehensive Test Suites

  • Encompassing a wide spectrum of datasets, search spaces, and hardware devices.
  • Ready-to-use test multi-objective optimization suites with up to eight objectives.

Get Started

Dive into the tutorial
Tap the image to embark on the introductory video voyage.

Setup & Installation

  1. Download requisite files:

  2. Run pip install evoxbench to get the benchmark.

  3. Configure the benchmark:

    from evoxbench.database.init import config

    config("Path to database", "Path to data")
    # For instance:
    # With this structure:
    # /home/Downloads/
    # └─ database/
    # |  |  __init__.py
    # |  |  db.sqlite3
    # |  |  ...
    # |
    # └─ data/
    #    └─ darts/
    #    └─ mnv3/
    #    └─ ...
    # Then, execute:
    # config("/home/Downloads/database", "/home/Downloads/data")

About the Database

Explore our comprehensive database and understand its structure and content. Check it out here.

Community & Support

  • Use the issue tracker for bugs or questions.
  • Submit your enhancements through a pull request (PR).
  • We have an active QQ group (ID: 297969717).
  • Official Website: https://evox.group/

Sister Projects

  • EvoX: A computing framework for distributed GPU-aceleration of evolutionary computation, supporting a wide spectrum of evolutionary algorithms and test problems. Check out here.

Citing EvoXBench

If you use EvoXBench in your research and want to cite it in your work, please use:

@article{EvoXBench,
  title={Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment},
  author={Lu, Zhichao and Cheng, Ran and Jin, Yaochu and Tan, Kay Chen and Deb, Kalyanmoy},
  journal={IEEE Transactions on Evolutionary Computation},
  year={2023},
  publisher={IEEE}
}

Acknowledgements

A big shoutout to the following projects that have made EvoXBench possible:

NAS-Bench-101, NAS-Bench-201, NAS-Bench-301, NATS-Bench, Once for All, AutoFormer, Django, pymoo.

About

Transforming Neural Architecture Search (NAS) into multi-objective optimization problems. A benchmark suite for testing evolutionary algorithms in deep learning.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 94.6%
  • Java 2.8%
  • Jupyter Notebook 1.7%
  • MATLAB 0.9%