Initial commit

CODE_OF_CONDUCT.md

@@ -0,0 +1,80 @@
+# Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to make participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all project spaces, and it also applies when
+an individual is representing the project or its community in public spaces.
+Examples of representing a project or community include using an official
+project e-mail address, posting via an official social media account, or acting
+as an appointed representative at an online or offline event. Representation of
+a project may be further defined and clarified by project maintainers.
+
+This Code of Conduct also applies outside the project spaces when there is a
+reasonable belief that an individual's behavior may have a negative impact on
+the project or its community.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at <[email protected]>. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONFIG.md

@@ -0,0 +1,210 @@
+## MoCo v3 Reference Setups and Models
+
+Here we document the reference commands for pre-training and evaluating various MoCo v3 models.
+
+### ResNet-50 models
+
+With batch 4096, the training of all ResNet-50 models can fit into 2 nodes with a total of 16 Volta 32G GPUs. 
+
+<details>
+<summary>ResNet-50, 100-epoch pre-training.</summary>
+
+On the first node, run:
+```
+python main_moco.py \
+  --moco-m-cos --crop-min=.2 \
+  --dist-url 'tcp://[your first node address]:[specified port]' \
+  --multiprocessing-distributed --world-size 2 --rank 0 \
+  [your imagenet-folder with train and val folders]
+```
+On the second node, run the same command with `--rank 1`.
+</details>
+
+<details>
+<summary>ResNet-50, 300-epoch pre-training.</summary>
+
+On the first node, run:
+```
+python main_moco.py \
+  --lr=.3 --epochs=300 \
+  --moco-m-cos --crop-min=.2 \
+  --dist-url 'tcp://[your first node address]:[specified port]' \
+  --multiprocessing-distributed --world-size 2 --rank 0 \
+  [your imagenet-folder with train and val folders]
+```
+On the second node, run the same command with `--rank 1`.
+</details>
+
+<details>
+<summary>ResNet-50, 1000-epoch pre-training.</summary>
+
+On the first node, run:
+```
+python main_moco.py \
+  --lr=.3 --wd=1.5e-6 --epochs=1000 \
+  --moco-m=0.996 --moco-m-cos --crop-min=.2 \
+  --dist-url 'tcp://[your first node address]:[specified port]' \
+  --multiprocessing-distributed --world-size 2 --rank 0 \
+  [your imagenet-folder with train and val folders]
+```
+On the second node, run the same command with `--rank 1`.
+</details>
+
+<details>
+<summary>ResNet-50, linear classification.</summary>
+
+Run on single node:
+```
+python main_lincls.py \
+  --dist-url 'tcp://localhost:10001' \
+  --multiprocessing-distributed --world-size 1 --rank 0 \
+  --pretrained [your checkpoint path]/[your checkpoint file].pth.tar \
+  [your imagenet-folder with train and val folders]
+```
+</details>
+
+Below are our pre-trained ResNet-50 models and logs.
+
+<table><tbody>
+<!-- START TABLE -->
+<!-- TABLE HEADER -->
+<th valign="center">pretrain<br/>epochs</th>
+<th valign="center">linear<br/>acc</th>
+<th valign="center">pretrain<br/>files</th>
+<th valign="center">linear<br/>files</th>
+<!-- TABLE BODY -->
+<tr>
+<td align="right">100</td>
+<td align="center">68.9</td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-100ep/r-50-100ep.pth.tar">chpt</a></td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-100ep/linear-100ep.pth.tar">chpt</a> /
+                  <a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-100ep/linear-100ep.std">log</a></td>
+</tr>
+<tr>
+<td align="right">300</td>
+<td align="center">72.8</td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-300ep/r-50-300ep.pth.tar">chpt</a></td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-300ep/linear-300ep.pth.tar">chpt</a> /
+                  <a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-300ep/linear-300ep.std">log</a></td>
+</tr>
+<tr>
+<td align="right">1000</td>
+<td align="center">74.6</td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-1000ep/r-50-1000ep.pth.tar">chpt</a></td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-1000ep/linear-1000ep.pth.tar">chpt</a> /
+                  <a href="https://dl.fbaipublicfiles.com/moco-v3/r-50-1000ep/linear-1000ep.std">log</a></td>
+</tr>
+</tbody></table>
+
+
+### ViT Models
+
+All ViT models are pre-trained for 300 epochs with AdamW.
+
+<details>
+<summary>ViT-Small, 1-node (8-GPU), 1024-batch pre-training.</summary>
+
+This setup fits into a single node of 8 Volta 32G GPUs, for ease of debugging.
+```
+python main_moco.py \
+  -a vit_small -b 1024 \
+  --optimizer=adamw --lr=1.5e-4 --weight-decay=.1 \
+  --epochs=300 --warmup-epochs=40 \
+  --stop-grad-conv1 --moco-m-cos --moco-t=.2 \
+  --dist-url 'tcp://localhost:10001' \
+  --multiprocessing-distributed --world-size 1 --rank 0 \
+  [your imagenet-folder with train and val folders]
+```
+
+</details>
+
+<details>
+<summary>ViT-Small, 4-node (32-GPU) pre-training.</summary>
+
+On the first node, run:
+```
+python main_moco.py \
+  -a vit_small \
+  --optimizer=adamw --lr=1.5e-4 --weight-decay=.1 \
+  --epochs=300 --warmup-epochs=40 \
+  --stop-grad-conv1 --moco-m-cos --moco-t=.2 \
+  --dist-url 'tcp://[your first node address]:[specified port]' \
+  --multiprocessing-distributed --world-size 8 --rank 0 \
+  [your imagenet-folder with train and val folders]
+```
+On other nodes, run the same command with `--rank 1`, ..., `--rank 3` respectively.
+</details>
+
+<details>
+<summary>ViT-Small, linear classification.</summary>
+
+Run on single node:
+```
+python main_lincls.py \
+  -a vit_small --lr=3 \
+  --dist-url 'tcp://localhost:10001' \
+  --multiprocessing-distributed --world-size 1 --rank 0 \
+  --pretrained [your checkpoint path]/[your checkpoint file].pth.tar \
+  [your imagenet-folder with train and val folders]
+```
+</details>
+
+<details>
+<summary>ViT-Base, 8-node (64-GPU) pre-training.</summary>
+
+```
+python main_moco.py \
+  -a vit_base \
+  --optimizer=adamw --lr=1.5e-4 --weight-decay=.1 \
+  --epochs=300 --warmup-epochs=40 \
+  --stop-grad-conv1 --moco-m-cos --moco-t=.2 \
+  --dist-url 'tcp://[your first node address]:[specified port]' \
+  --multiprocessing-distributed --world-size 8 --rank 0 \
+  [your imagenet-folder with train and val folders]
+```
+On other nodes, run the same command with `--rank 1`, ..., `--rank 7` respectively.
+</details>
+
+<details>
+<summary>ViT-Base, linear classification.</summary>
+
+Run on single node:
+```
+python main_lincls.py \
+  -a vit_base --lr=3 \
+  --dist-url 'tcp://localhost:10001' \
+  --multiprocessing-distributed --world-size 1 --rank 0 \
+  --pretrained [your checkpoint path]/[your checkpoint file].pth.tar \
+  [your imagenet-folder with train and val folders]
+```
+</details>
+
+
+Below are our pre-trained ViT models and logs (batch 4096).
+
+<table><tbody>
+<!-- START TABLE -->
+<!-- TABLE HEADER -->
+<th valign="center">model</th>
+<th valign="center">pretrain<br/>epochs</th>
+<th valign="center">linear<br/>acc</th>
+<th valign="center">pretrain<br/>files</th>
+<th valign="center">linear<br/>files</th>
+<!-- TABLE BODY -->
+<tr>
+<td align="left">ViT-Small</td>
+<td align="center">300</td>
+<td align="center">73.2</td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/vit-s-300ep/vit-s-300ep.pth.tar">chpt</a></td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/vit-s-300ep/linear-vit-s-300ep.pth.tar">chpt</a> /
+                  <a href="https://dl.fbaipublicfiles.com/moco-v3/vit-s-300ep/linear-vit-s-300ep.std">log</a></td>
+</tr>
+<tr>
+<td align="left">ViT-Base</td>
+<td align="center">300</td>
+<td align="center">76.7</td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/vit-b-300ep/vit-b-300ep.pth.tar">chpt</a></td>
+<td align="center"><a href="https://dl.fbaipublicfiles.com/moco-v3/vit-b-300ep/linear-vit-b-300ep.pth.tar">chpt</a> /
+                  <a href="https://dl.fbaipublicfiles.com/moco-v3/vit-b-300ep/linear-vit-b-300ep.std">log</a></td>
+</tr>
+</tbody></table>

CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing to moco-v3
+We want to make contributing to this project as easy and transparent as
+possible.
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `master`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+disclosure of security bugs. In those cases, please go through the process
+outlined on that page and do not file a public issue.
+
+## License
+By contributing to moco-v3, you agree that your contributions will be licensed
+under the LICENSE file in the root directory of this source tree.