Skip to content

kazemnejad/len_gen_lm

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Oct 21, 2023
674a841 · Oct 21, 2023

History

71 Commits
May 15, 2023
Oct 17, 2023
Oct 7, 2023
Oct 7, 2023
Oct 21, 2023
Oct 7, 2023
May 16, 2023
Aug 5, 2023
Aug 3, 2023
Aug 3, 2023
Oct 7, 2023
Oct 7, 2023
Oct 7, 2023
Aug 7, 2023
Aug 5, 2023
Oct 18, 2023
May 15, 2023
Oct 17, 2023
May 13, 2023
Aug 3, 2023
Aug 3, 2023
Oct 17, 2023
Oct 20, 2023
Oct 7, 2023
Oct 7, 2023
Oct 7, 2023
Oct 7, 2023
Oct 7, 2023
Aug 8, 2023
Oct 7, 2023
Oct 7, 2023

Repository files navigation

len_gen_lm

Prepare Experiment Environment (Only for the first time)

  1. Clone this repository
git clone [email protected]:kazemnejad/len_gen_lm.git
  1. Create a conda environment
conda create -n len_gen_lm python=3.9
conda activate len_gen_lm
  1. Install requirements
# Install pytorch
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
# Install other requirements
pip install -r requirements.txt
  1. Fill the environment variables in env.sh with
# We save checkpoints and logs here. It should be shared network storage accessible from all nodes.
export PROJECT_DIR=/path/to/network/storage/projects/len_gen_lm

# Go to comet.ml and get your API token
export COMET_API_KEY="..."

Train

./run_training.sh <pe> <size>

<pe> can be chosen from:

  • alibi: Alibi
  • none: NoPE

<size> can be chosen from:

  • 100m
  • 300m
  • 1b

What compute resources should be used?

at least:

  • CPU: 6 cores
  • Memory: 32GB

it will use all gpus available on the node. So, the more gpus we have, the faster it will be.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published