question about self.pos_embed in models_dww.py

[stillwaterman]

Hi, nice work!
I found this code ‘self.pos_embed = nn.Parameter(torch.zeros(1, self.num_patches + 1, embed_dim), requires_grad=False) # fixed sin-cos embedding’ in models_dww.py. I think self.pos_embed is not a fixed sin-cos embedding, it's a fixed zeros embedding, and adding it to patch embedding has no effect. I want to know what the purpose of this line of code is. @ShadowXZT

[xiong-zhitong]

Hi, thanks for your interests!
During training, pos_embed is initialized here.
During inference, we will load the parameters (fixed sin-cos embedding) from the pre-trained model.

[stillwaterman]

thanks for reply! I understand now, close issue.