|
| 1 | +<!--Copyright 2022 The HuggingFace Team. All rights reserved. |
| 2 | + |
| 3 | +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with |
| 4 | +the License. You may obtain a copy of the License at |
| 5 | + |
| 6 | +http://www.apache.org/licenses/LICENSE-2.0 |
| 7 | + |
| 8 | +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on |
| 9 | +an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the |
| 10 | +specific language governing permissions and limitations under the License. |
| 11 | +--> |
| 12 | + |
| 13 | +# Reproducibility |
| 14 | + |
| 15 | +Before reading about reproducibility for Diffusers, it is strongly recommended to take a look at |
| 16 | +[PyTorch's statement about reproducibility](https://pytorch.org/docs/stable/notes/randomness.html). |
| 17 | + |
| 18 | +PyTorch states that |
| 19 | +> *completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms.* |
| 20 | +While one can never expect the same results across platforms, one can expect results to be reproducible |
| 21 | +across releases, platforms, etc... within a certain tolerance. However, this tolerance strongly varies |
| 22 | +depending on the diffusion pipeline and checkpoint. |
| 23 | + |
| 24 | +In the following, we show how to best control sources of randomness for diffusion models. |
| 25 | + |
| 26 | +## Inference |
| 27 | + |
| 28 | +During inference, diffusion pipelines heavily rely on random sampling operations, such as the creating the |
| 29 | +gaussian noise tensors to be denoised and adding noise to the scheduling step. |
| 30 | + |
| 31 | +Let's have a look at an example. We run the [DDIM pipeline](./api/pipelines/ddim.mdx) |
| 32 | +for just two inference steps and return a numpy tensor to look into the numerical values of the output. |
| 33 | + |
| 34 | +```python |
| 35 | +from diffusers import DDIMPipeline |
| 36 | +import numpy as np |
| 37 | + |
| 38 | +model_id = "google/ddpm-cifar10-32" |
| 39 | + |
| 40 | +# load model and scheduler |
| 41 | +ddim = DDIMPipeline.from_pretrained(model_id) |
| 42 | + |
| 43 | +# run pipeline for just two steps and return numpy tensor |
| 44 | +image = ddim(num_inference_steps=2, output_type="np").images |
| 45 | +print(np.abs(image).sum()) |
| 46 | +``` |
| 47 | + |
| 48 | +Running the above prints a value of 1464.2076, but running it again prints a different |
| 49 | +value of 1495.1768. What is going on here? Every time the pipeline is run, gaussian noise |
| 50 | +is created and step-wise denoised. To create the gaussian noise with [`torch.randn`](https://pytorch.org/docs/stable/generated/torch.randn.html), a different random seed is taken every time, thus leading to a different result. |
| 51 | +This is a desired property of diffusion pipelines, as it means that the pipeline can create a different random image every time it is run. In many cases, one would like to generate the exact same image of a certain |
| 52 | +run, for which case an instance of a [PyTorch generator](https://pytorch.org/docs/stable/generated/torch.randn.html) has to be passed: |
| 53 | + |
| 54 | +```python |
| 55 | +import torch |
| 56 | +from diffusers import DDIMPipeline |
| 57 | +import numpy as np |
| 58 | + |
| 59 | +model_id = "google/ddpm-cifar10-32" |
| 60 | + |
| 61 | +# load model and scheduler |
| 62 | +ddim = DDIMPipeline.from_pretrained(model_id) |
| 63 | + |
| 64 | +# create a generator for reproducibility |
| 65 | +generator = torch.Generator(device="cpu").manual_seed(0) |
| 66 | + |
| 67 | +# run pipeline for just two steps and return numpy tensor |
| 68 | +image = ddim(num_inference_steps=2, output_type="np", generator=generator).images |
| 69 | +print(np.abs(image).sum()) |
| 70 | +``` |
| 71 | + |
| 72 | +Running the above always prints a value of 1491.1711 - also upon running it again because we |
| 73 | +define the generator object to be passed to all random functions of the pipeline. |
| 74 | + |
| 75 | +If you run this code snippet on your specific hardware and version, you should get a similar, if not the same, result. |
| 76 | + |
| 77 | +<Tip> |
| 78 | + |
| 79 | +It might be a bit unintuitive at first to pass `generator` objects to the pipelines instead of |
| 80 | +just integer values representing the seed, but this is the recommended design when dealing with |
| 81 | +probabilistic models in PyTorch as generators are *random states* that are advanced and can thus be |
| 82 | +passed to multiple pipelines in a sequence. |
| 83 | + |
| 84 | +</Tip> |
| 85 | + |
| 86 | +Great! Now, we know how to write reproducible pipelines, but it gets a bit trickier since the above example only runs on the CPU. How do we also achieve reproducibility on GPU? |
| 87 | +In short, one should not expect full reproducibility across different hardware when running pipelines on GPU |
| 88 | +as matrix multiplications are less deterministic on GPU than on CPU and diffusion pipelines tend to require |
| 89 | +a lot of matrix multiplications. Let's see what we can do to keep the randomness within limits across |
| 90 | +different GPU hardware. |
| 91 | + |
| 92 | +To achieve maximum speed performance, it is recommended to create the generator directly on GPU when running |
| 93 | +the pipeline on GPU: |
| 94 | + |
| 95 | +```python |
| 96 | +import torch |
| 97 | +from diffusers import DDIMPipeline |
| 98 | +import numpy as np |
| 99 | + |
| 100 | +model_id = "google/ddpm-cifar10-32" |
| 101 | + |
| 102 | +# load model and scheduler |
| 103 | +ddim = DDIMPipeline.from_pretrained(model_id) |
| 104 | +ddim.to("cuda") |
| 105 | + |
| 106 | +# create a generator for reproducibility |
| 107 | +generator = torch.Generator(device="cuda").manual_seed(0) |
| 108 | + |
| 109 | +# run pipeline for just two steps and return numpy tensor |
| 110 | +image = ddim(num_inference_steps=2, output_type="np", generator=generator).images |
| 111 | +print(np.abs(image).sum()) |
| 112 | +``` |
| 113 | + |
| 114 | +Running the above now prints a value of 1389.8634 - even though we're using the exact same seed! |
| 115 | +This is unfortunate as it means we cannot reproduce the results we achieved on GPU, also on CPU. |
| 116 | +Nevertheless, it should be expected since the GPU uses a different random number generator than the CPU. |
| 117 | + |
| 118 | +To circumvent this problem, we created a [`randn_tensor`](#diffusers.utils.randn_tensor) function, which can create random noise |
| 119 | +on the CPU and then move the tensor to GPU if necessary. The function is used everywhere inside the pipelines allowing the user to **always** pass a CPU generator even if the pipeline is run on GPU: |
| 120 | + |
| 121 | +```python |
| 122 | +import torch |
| 123 | +from diffusers import DDIMPipeline |
| 124 | +import numpy as np |
| 125 | + |
| 126 | +model_id = "google/ddpm-cifar10-32" |
| 127 | + |
| 128 | +# load model and scheduler |
| 129 | +ddim = DDIMPipeline.from_pretrained(model_id) |
| 130 | +ddim.to("cuda") |
| 131 | + |
| 132 | +# create a generator for reproducibility |
| 133 | +generator = torch.manual_seed(0) |
| 134 | + |
| 135 | +# run pipeline for just two steps and return numpy tensor |
| 136 | +image = ddim(num_inference_steps=2, output_type="np", generator=generator).images |
| 137 | +print(np.abs(image).sum()) |
| 138 | +``` |
| 139 | + |
| 140 | +Running the above now prints a value of 1491.1713, much closer to the value of 1491.1711 when |
| 141 | +the pipeline is fully run on the CPU. |
| 142 | + |
| 143 | +<Tip> |
| 144 | + |
| 145 | +As a consequence, we recommend always passing a CPU generator if Reproducibility is important. |
| 146 | +The loss of performance is often neglectable, but one can be sure to generate much more similar |
| 147 | +values than if the pipeline would have been run on CPU. |
| 148 | + |
| 149 | +</Tip> |
| 150 | + |
| 151 | +Finally, we noticed that more complex pipelines, such as [`UnCLIPPipeline`] are often extremely |
| 152 | +susceptible to precision error propagation and thus one cannot expect even similar results across |
| 153 | +different GPU hardware or PyTorch versions. In such cases, one has to make sure to run |
| 154 | +exactly the same hardware and PyTorch version for full Reproducibility. |
| 155 | + |
| 156 | +## Randomness utilities |
| 157 | + |
| 158 | +### randn_tensor |
| 159 | +[[autodoc]] diffusers.utils.randn_tensor |
0 commit comments