Inquiry about training setting

[Nyquist0]

Hi Sir or Madam,

Thanks for sharing your great work.
I would like to consult 2 questions regarding your training setting.

1. I found the identity consistency between your generated frames and reference image is higher than hallo. For results of Hallo, the chin from generated frames would become stronger than reference. May I ask why?
2. Referring your paper, I found the dataset jdh-hallo is all of front-view. But for testing data, the side-view is also good. Why is this happened?

Looking forward to your reply.
Thanks

[DBDXSS]

Thanks for your question, it's valuable.

1. As mentioned in the paper, we propose a semi-decoupled structure. It can capture inter-feature relationships among lip, expression, and pose features, which makes the result more vivid.

2. Actually, it is not all of the front-view in jdh-Hallo. We just show one frame of the video.

[Nyquist0]

Thanks for your quick reply.

1. But referring to your paper, I think the semi-decoupled structure you proposed is the same as Hierarchical Audio-Visual Cross Attention in Hallo. That should not lead to different results for testing.. Is there any misunderstanding here?
2. Got it. Thanks. So the dataset is similar as MEAD, but without emotions.

Best~

[DBDXSS]

But in Hallo's repository, if you read the codes, you will find they are different. There is a misleading in Hallo's paper. Hallo's Hierarchical Audio-Visual Cross Attention is actually a full-decoupled structure.

[Nyquist0]

Git it. Thanks for explanation~