Skip to content

Latest commit

 

History

History
249 lines (177 loc) · 20.8 KB

README.md

File metadata and controls

249 lines (177 loc) · 20.8 KB
Raw

GenéLive! Generating Rhythm Actions in Love Live!

This repository provides the source code and trained model to reproduce the results in the following paper:

Atsushi Takada, Daichi Yamazaki, Likun Liu, Yudai Yoshida, Nyamkhuu Ganbat, Takayuki Shimotomai, Taiga Yamamoto, Daisuke Sakurai, Naoki Hamada, "GenéLive! Generating Rhythm Actions in Love Live!", DOI: 10.48550/arXiv.2202.12823, to appear at AAAI-23.

Implementations of the Proposed Method

The proposed method contains two novel techniques:

  • Beat guide is implemented in notes_generator/models/beats.py file.
  • Multi-scale conv-stack is implemented as ConvStackV7 class in notes_generator/layers/base_layers.py file.

Getting Started

Using this repository, you can reproduce the results based on the Stepmania dataset in the paper. (The results on the other datasets are excluded for copyrights reasons.) The proposed GenéLive! model trained on the Stepmain dataset is also provided to generate a chart for your own audio file. The chart is a MIDI file that you can open in DAW software and play it.

Prerequisites

Packages

  • libsndfile
    • Linux (Debian) 
      apt-get install libsndfile1
    • MacOS 
      brew install libsndfile

Python environment

To run the code, a conda environment is required. We recommend Miniconda, see Conda official site for how to install it.

Installation

  1. Create and activate a new conda environment 
    conda create -n my-mlflow
conda activate my-mlflow
  2. Install Mlflow to the created environment 
    conda install mlflow
  3. Clone the repo 
    git clone https://github.com/KLab/AAAI-23.6040.git

Usage

All the functionalities in the code can be invoked through mlflow command.

Fetch dataset

Run the following command.

mlflow run -e fetch --experiment-name fetch .

After running the command, data/ directory is created and data is downloaded under the directory.
The directory structure is like below:

data
└── raw
    ├── fraxtil
    │   ├── Fraxtil's Arrow Arrangements
    │   ├── Fraxtil's Beast Beats
    │   └── Tsunamix III
    └── itg
        ├── In The Groove
        └── In The Groove 2

Parameters

Name Default Description
data_dir data Path to a directory in which raw data is downloaded.

Generate notes

Run the following command.

mlflow run -e generate --experiment-name generate . \
  -P audio_path=path/to/audio \  # specify a path to the audio file you want to generate notes for
  -P midi_save_path=save/path \
  -P bpm_info=""  # example: [(180.0,600,4)]

Parameters

Name Default Description
model_path pretrained_model/model.pth Path to a model used for prediction. Default is a path to a model which achieved the highest f1_micro score with STEPMANIA_F dataset.
audio_path - Path to an audio file to input to the model.
midi_save_path data/midi_out Directory to which output MIDI file is saved.
bpm_info - A string that contains BPM information of the song. The content of a string is python literal of list which contains tuples of (bpm, millisecond from head of song, beat).

Check results

The results are stored in mlruns/ directory. You can check them on the browser by the following command:

mlflow ui

Preprocessing

Before moving forward to Evaluation and Training section, data preprocessing is required in advance.
Run the following command.

mlflow run -e preprocess --experiment-name preprocess .

Parameters

Name Default Description
data_dir data Path to data directory which contains raw data.

Evaluation

  1. Create a directory named model_test_result under data/.

  2. Run the following command.

    mlflow run -e test --experiment-name test .

Parameters

Name Default Description
model_dir data/onsets_model The directory where models are saved.
app_name STEPMANIA The dataset name. Choices: STEPMANIA, STEPMANIA_F, STEPMANIA_I. If STEPMANIA_F is chosen, only use the songs belonging to “Fraxtil” dataset, and if STEPMANIA_I is chosen, only use “In The Groove” dataset.
score_dir data/train_data/score_onsets_1 The directory containing training labels.
mel_dir data/train_data/mel_log The directory containing Mel-spectrograms.
seq_length 20480 The desired length of the sequence of the input to the model.
batch 1 The mini batch size.
num_layers 2 The number of LSTM layers.
onset_weight 64 The weight multiplied to positive labels when calculating binary cross entropy loss.
with_beats 1 If set to 1, model accept beat information.
conv_stack_type v7 The type of convolution stack. Choices: v1 (DDC), v7 (GenéLive!).
csv_save_dir data/model_test_result The directory the evaluation result will be saved.

Note

Since it requires 179 GB of models to fully reproduce the results in the paper, we couldn't include all models in this repository. Instead, we provide commands to reproduce equivalent experiments in next section.

Training

  1. Create a directory named onsets_model under data/.

  2. Run the following command.

    mlflow run -e train --experiment-name train .

Parameters

Name Default Description
app_name STEPMANIA The dataset name. Choices: STEPMANIA, STEPMANIA_F, STEPMANIA_I. If STEPMANIA_F is chosen, only use the songs belonging to “Fraxtil” dataset, and if STEPMANIA_I is chosen, only use “In The Groove” dataset.
model_dir data/onset_model The directory to save models.
score_dir data/train_data/score_onsets_1 The directory containing training labels.
mel_dir data/train_data/mel_log The directory containing Mel-spectrograms.
resume 0 If nonzero value is specified, resume training from specified step.
epochs 35 The epoch of training.
batch 32 The mini batch size.
lr_start 9e-4 The start value of learning rate scheduling.
lr_end 9e-4 The end rate of learning rate scheduling
lr_scheduler CosineAnnealingLR The type of learning rate scheduler. Choices: CosineAnnealingLR, CyclicLR,
seq_length 20480 The desired length of the sequence of the input to the model.
aug_count 0 The augmentation count of mel-spectrogram.
num_layers 2 The number of LSTM layers.
onset_weight 64 The weight multiplied to positive labels when calculating binary cross entropy loss.
dropout 0.3 The dropout rate.
fuzzy_width 5 The width of fuzzy label.
fuzzy_scale 0.2 The scale of fuzzy label.
with_beats 1 Either 0 or 1. If set to 1, train a model with beat information.
difficulties - If a tuple of difficulty ids is set, train a model with data of solely specified difficulties.
send_model 0 If set to 1, save the best model in the local millruns directory.
n_saved_model 20 The maximum number of recent models saved.
log_artifacts "" The comma separated string which contains path to files wanted to be saved in the local mlruns directory.
augmentation_setting loader_aug_config.yaml The yaml file containing audio augmentation settings.
warmup_steps 400 The warming up steps for learning rate scheduler.
weight_decay 0 The L2 regularization coefficient.
is_parallel 0 If set to 1, use multiple GPU if available.
eta_min 1e-6 The hyper-parameter for CosineAnnealingLR scheduler.
conv_stack_type v7 The type of convolution stack. Choices: v1 (DDC), v7 (GenéLive!).
pretrained_model_path "" If specified, load weights before start training.
rnn_dropout 0.1 The dropout rate in RNN layers.

Commands for Reproducing Experiments

Experiment Dataset Command
Proposed method Fraxtil mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_F
Beat guide ablation Fraxtil mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_F -P with_beats=0
Difficulty ablation (Beginner only) Fraxtil mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_F -P difficulties='(10,)'
Difficulty ablation (Intermediate only) Fraxtil mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_F -P difficulties='(20,)'
Difficulty ablation (Advanced only) Fraxtil mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_F -P difficulties='(30,)'
Difficulty ablation (Expert only) Fraxtil mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_F -P difficulties='(40,)'
Conv-stack ablation Fraxtil mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_F -P conv_stack_type=v1
Proposed method In the Groove mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_I
Beat guide ablation In the Groove mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_I -P with_beats=0
Difficulty ablation (Beginner only) In the Groove mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_I -P difficulties='(10,)'
Difficulty ablation (Intermediate only) In the Groove mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_I -P difficulties='(20,)'
Difficulty ablation (Advanced only) In the Groove mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_I -P difficulties='(30,)'
Difficulty ablation (Expert only) In the Groove mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_I -P difficulties='(40,)'
Conv-stack ablation In the Groove mlflow run -e train --experiment-name train . -P app_name=STEPMANIA_I -P conv_stack_type=v1

Reproducing figures in the paper

See figure.ipynb file.

License

  • The code in notes_generator/ddc/ directory is re-distributed under the MIT License by Chris Donahue. See notes_generator/ddc/LICENSE file and the original repository for more information.
  • The other code is distributed under the MIT License by KLab Inc. See LICENSE file for more information.