Workout Detector

Installation

git clone https://github.com/iucario/workoutdetector.git
cd WorkoutDetector

conda env create -f conda_env.yml
pip install openmim
mim install mmcv
pip install -r requirements.txt
pip install -e .

Docker

Build image for dev environment example:

docker built -t workout/dev docker
docker run -it \
    --gpus=all \
    --shm-size=32gb \
    --volume="$PWD:/work" \
    --volume="/home/$USER/data:/home/user/data:ro" \
    -w /work \
    --entrypoint zsh \
    --name devbox \
    workout/dev

sudo apt install vim tree -y
pip install wandb pytest
sudo pip install -e .
export $PROJ_ROOT=$PWD

Run docker example

docker run --rm -it \
  --gpus=all \
  --shm-size=32gb \
  --volume="$PWD:/work" \
  --volume="/home/$USER/data:/home/user/data:ro" \
  workout/dev:latest python3 workoutdetector/trainer.py

React demo

Download onnx model. OneDrive
cd app && uvicorn server:app --port 8000
open http://localhost:8000/

Run Gradio demo

Download onnx model. Same as the React demo. OneDrive
Copy to checkpoints
python WorkoutDetector/demo.py
open http://localhost:7860/

Repetition counting

Two model types, image and video, can be used.

Method is naive. The transition of states is counted as one repetition. It's online counting. Only previous frames are used.

Evaluation

Inference videos and save results to a directory. Results of each video will be saved in a JSON file. workoutdetector/utils/inference_count.py

ckpt = 'checkpoints/model.onnx'
model = onnxruntime.InferenceSession(ckpt, providers=['CUDAExecutionProvider'])
inference_dataset(model, ['train', 'val', 'test'], out_dir='out/tsm_rep_scores', checkpoint=ckpt)

Scores of each video are saved in out/tsm_rep_scores/video.mp4.score.json.

Evaluating mean absolute error and off-by-one accuracy workoutdetector/utils/eval_count.py

json_dir = 'out/tsm_rep_scores'
anno_path = 'data/RepCount/annotation.csv'
out_csv = 'out/tsm_rep_scores.csv'
main(json_dir, anno_path, out_csv, softmax=True)
analyze_count(out_csv, out_csv.replace('.csv', '_meta.csv'))

Results of every video are saved in out/tsm_rep_scores.csv. Metrics are saved in out/tsm_rep_scores_meta.csv.

Visualization notebooks/rep_analysis.ipynb

Train an action recognition model

Local

Be sure to modify config WorkoutDetector/settings/global_settings.py to your project root. E.g. PROJ_ROOT = '/home/your_name/WorkoutDetector/' Or set environment variable PROJ_ROOT to your project root. E.g. export PROJ_ROOT=$PWD

Build dataset

Workouts are subsets taking from Countix and RepCount. For now, I am using 11 classes. Dataset.md

Use soft links to avoid copying files.

See WorkoutDetector/scripts/build_datasets.py for details.

data/Workouts/
├── rawframes
│   ├── Countix
│   │   ├── train
│   │   └── val
│   ├── RepCount
│   │   ├── test
│   │   ├── train
│   │   └── val
│   ├── test.txt
│   ├── test_repcount.txt
│   ├── train.txt
│   ├── train_countix.txt
│   ├── train_repcount.txt
│   ├── val.txt
│   ├── val_countix.txt
│   └── val_repcount.txt
├── test.txt
├── train.txt
└── val.txt

Train video with mmaction2

python workoutdetector/train.py config: configs/tsm_action_recogition_sthv2.py

Train an exercise state recognition model

Train video with tsm

Build label files scripts/build_label_files.py

 anno_file = 'datasets/RepCount/annotation.csv'
 data_root = os.path.join(PROJ_ROOT, 'data')
 dst_dir = os.path.join(data_root, 'Binary')
 build_with_start(data_root, anno_file, dst_dir)

Download weights pretrained on SSV2: https://hanlab.mit.edu/projects/tsm/models/TSM_somethingv2_RGB_resnet50_shift8_blockres_avg_segment8_e45.pth
Modify config file configs/repcount_12_tsm.yaml
Train python workoutdetector/trainer.py --cfg workoutdetector/configs/tsm.yaml

Configs Best weigths are saved in directory {cfg.trainer.default_root_dir}/checkpoints, in format best-val-acc={val/acc:.2f}-epoch={epoch:02d}" + f"-{timenow}.ckpt. Modify callbacks.modelcheckpoint.dirpath to save weights in a different directory.

Tensorboard logs are in {cfg.log.output_dir}/{cfg.log.name}/version_{num}.

Wandb logs are in {cfg.log.output_dir}/wandb.

Train rep with mmaction2

python workoutdetector/train_rep.py \
 --action=pull_up \
 --data-prefix=data/RepCount/rawframes/ \
 --ann-dir=data/relabeled/pull_up \
 --ckpt=work_dirs/tsm_MultiActionRepCount_sthv2_20220625-224626/best_top1_acc_epoch_5.pth

Classification of action states, e.g. for action push_up, up and down are two states. Also supports multiple actions and multiple states.

Training code is in workoutdetector/train_rep.py. Uses mmaction2 to train the time shift module. Configs are in workoutdetector/configs.

Prepare label files. The row in the label file is of format: path/to/rawframe_dir start_frame num_frames label. Frames of indices start_frame to start_frame + num_frames will be used. Don't need to move and rename frames in this way. Just need to modify the label file.

Train an image recognition model

python workoutdetector/train_img.py --cfg workoutdetector/configs/pull_up.yaml

Uses PyTorch Lightning to train a model.