Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarification on Reproducing PFC Score from EDGE Paper #24

Open
AkideLiu opened this issue Jul 24, 2023 · 2 comments
Open

Clarification on Reproducing PFC Score from EDGE Paper #24

AkideLiu opened this issue Jul 24, 2023 · 2 comments

Comments

@AkideLiu
Copy link

Dear EDGE Authors,

I am fascinated by your paper and wish to gain a better understanding of your methodology, particularly concerning the process to replicate the evaluation metrics detailed in your publication.

In your work, it's mentioned that for automatic evaluations such as PFC, beat alignment, Distk, and Distg, 5-second clips were obtained from each model using slices from the test music set with a 2.5-second stride. However, the process of deriving these 5-second clips from the AIST++ dataset remains unclear to me. The dataset test set, as far as I understand, comprises 20 musical pieces ranging from 8-15 seconds each.

I have attempted to replicate the PFC metrics using the following approaches:

  1. I used the raw input music from the AIST++ test set, selecting the initial 5 seconds, resulting in a PFC of 1.6836428132824115.
  2. I made use of pre-split slices in the test set, incorporating 186 samples, each 5 seconds long, which led to a PFC of 1.2385500567535723.
  3. By employing the original implementation in the test.py file with an output length (interpreted as a random slice selection for motion generation), two generation runs yielded PFCs of 1.5676957425076252 and 1.7647031391283114, respectively.

I am trying to replicate the PFC score (1.5363) reported in your paper, and I would greatly appreciate your guidance in this matter. Please let me know if there are any misconceptions in my understanding.

Looking forward to your kind assistance.

@applesleam
Copy link

@AkideLiu I tried approach 1, same to you, I got a result at 1.628. However, approach 2 seems more reasonable.

Have you ever tried to compute the PFC metrics of ground truth motion? I use smpl.forward(q, pos) to compute the full_pose for evaluation but result in a very low score ~0.3.

@Andy010902
Copy link

@AkideLiu I tried approach 1, same to you, I got a result at 1.628. However, approach 2 seems more reasonable.

Have you ever tried to compute the PFC metrics of ground truth motion? I use smpl.forward(q, pos) to compute the full_pose for evaluation but result in a very low score ~0.3.

Just do downsample like dance.dataset.py (60fps -> 30fps)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants