Initial commit

Co-authored-by: Zach Nagengast <[email protected]>
Co-authored-by: Atila Orhon <[email protected]>

.gitignore

@@ -0,0 +1,234 @@
+### Common ###
+
+.vscode
+.DS_Store
+models
+*tar
+
+### Python ###
+
+# Models
+*.safetensors
+*.ipynb
+*.png
+*.npy
+*.npz
+mlx_model/
+wandb/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+poetry.lock
+
+# pdm
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Output files
+output.png
+*/output.png
+
+### Swift ###
+
+# Core ML Model Files
+Models
+**/*.mlpackage
+**/*.mlmodel
+**/*.mlmodelc
+**/*.zip
+**/*.tar.gz
+.swiftpm/configuration/registries.json
+.swiftpm/xcode/package.xcworkspace/contents.xcworkspacedata
+.swiftpm/xcode/xcshareddata/
+**/*.xcscheme
+
+## Xcode
+# Build generated
+build/
+DerivedData/
+
+# Various settings
+*.pbxuser
+!default.pbxuser
+*.mode1v3
+!default.mode1v3
+*.mode2v3
+!default.mode2v3
+*.perspectivev3
+!default.perspectivev3
+xcuserdata/
+
+# Other
+*.moved-aside
+*.xccheckout
+*.xcscmblueprint
+
+# Obj-C/Swift specific
+*.hmap
+*.ipa
+*.dSYM.zip
+*.dSYM
+
+# fastlane
+fastlane/report.xml
+fastlane/Preview.html
+fastlane/screenshots
+fastlane/test_output
+
+### Xcode Patch ###
+*.xcodeproj/*
+!*.xcodeproj/project.pbxproj
+!*.xcodeproj/xcshareddata/
+!*.xcworkspace/contents.xcworkspacedata
+/*.gcno

README.md

@@ -0,0 +1,91 @@
+# DiffusionKit
+
+Run Diffusion Models on Apple Silicon with Core ML and MLX
+
+This repository comprises
+- `diffusionkit`, a Python package for converting PyTorch models to Core ML format and performing image generation with [MLX](https://github.com/ml-explore/mlx) in Python
+- `DiffusionKit`, a Swift package for on-device inference of diffusion models using Core ML and MLX
+
+<div align="center">
+<img src="assets/diffusionkit.png" width=256>
+</div>
+
+
+## Installation
+
+<details>
+
+The following installation steps are required for:
+- MLX inference
+- PyTorch to Core ML model conversion
+
+### Python Environment Setup
+
+```bash
+conda create -n diffusionkit python=3.11 -y
+conda activate diffusionkit
+cd /path/to/diffusionkit/repo
+pip install -e .
+```
+
+### Hugging Face Hub Credentials
+
+[Stable Diffusion 3](https://huggingface.co/stabilityai/stable-diffusion-3-medium) requires users to accept the terms before downloading the checkpoint. Once you accept the terms, sign in with your Hugging Face hub READ token as below:
+
+```bash
+huggingface-cli login --token YOUR_HF_HUB_TOKEN
+```
+
+</details>
+
+## <a name="converting-models-to-coreml"></a> Converting Models from PyTorch to Core ML
+
+<details>
+  <summary> Click to expand </summary>
+
+**Step 1:** Follow the installation steps from the previous section
+**Step 2:** Prepare the denoise model (MMDiT) Core ML model files (`.mlpackage`)
+
+```shell
+python -m tests.torch2coreml.test_mmdit --sd3-ckpt-path <path-to-sd3-mmdit.safetensors> --model-version {2b} -o <output-mlpackages-directory> --latent-size {64, 128}
+```
+
+**Step 3:** Prepare the VAE Decoder Core ML model files (`.mlpackage`)
+
+```shell
+python -m tests.torch2coreml.test_vae --sd3-ckpt-path <path-to-sd3-mmdit.safetensors> -o <output-mlpackages-directory> --latent-size {64, 128}
+```
+</details>
+
+## <a name="image-generation-with-python-mlx"></a> Image Generation with Python MLX
+
+<details>
+  <summary> Click to expand </summary>
+
+For simple text-to-image in float16 precision:
+```shell
+diffusionkit-cli --prompt "a photo of a cat" --output-path </path/to/output/image.png> --seed 0 --w16 --a16
+```
+
+Some notable optional arguments:
+- For image-to-image, use `--image-path` (path to input image) and `--denoise` (value between 0. and 1.)
+- T5 text embeddings, use `--t5`
+- For different resolutions, use `--height` and `--width`
+
+Please refer to the help menu for all available arguments: `diffusionkit-cli -h`.
+
+</details>
+
+## Image Generation with Swift
+
+<details>
+  <summary> Click to expand </summary>
+
+### Core ML Swift
+
+[Apple Core ML Stable Diffusion](https://github.com/apple/ml-stable-diffusion) is the initial Core ML backend for DiffusionKit. Stable Diffusion 3 support is upstreamed to that repository while we build the holistic Swift inference package.
+
+### MLX Swift
+🚧
+
+</details>