TouchDiffusion

TouchDesigner implementation for real-time Stable Diffusion interactive generation with StreamDiffusion.

Benchmarks with stabilityai/sd-turbo, 512x512 and 1 batch size.

GPU	FPS
4090	55-60 FPS
4080	47 FPS
3090ti	37 FPS
3090	30-32 FPS
4070 Laptop	24 FPS
3060 12GB	16 FPS

Disclaimer

Notice: This repository is in an early testing phase and may undergo significant changes. Use it at your own risk.

Usage

Required TouchDesigner 2023 & Python 3.11

You can follow YouTube tutorial

Install:

1. Install Python 3.11
2. Install Git
3. Install CUDA Toolkit 11.8 (required PC restart)
4. Download TouchDiffusion.
5. Open webui.bat with text editor and set path to Python 3.11 in set PYTHON_PATH=. (ex. set PYTHON_PATH="C:\Program Files\Python311\python.exe")
6. Run webui.bat. After installation it will provide url to web interface (ex. http://127.0.0.1:7860)
7. Open install & update tab and run Update dependencies. (could take ~10 minutes, depending on your internet connection)
8. If you get pop up window with error related to .dll, run Fix pop up
9. Restart webui.bat

Accelerate model:

Models in .safetensors format must be in models\checkpoints folder. (as for sd_turbo, it will be auto-downloaded).

Internet connection required, while making engines.

1. Run webui.bat
2. Select model type.
3. Select model.
4. Set width, height and amount of sampling steps (Batch size)
5. Select acceleration lora if available.
6. Run Make engine and wait for acceleration to finish. (could take ~10 minutes, depending on your hardware)

TouchDesigner inference:

1. Add TouchDiffusion.tox to project
2. On Settings page change path to TouchDiffusion folder (same as where webui.bat).
3. Save and restart TouchDesigner project.
4. On Settings page select Engine and click Load Engine.
5. Connect animated TOP to input. Component cook only if input updates.

Known issues / Roadmap:

• Fix Re-init. Sometimes required to restart TouchDesigner for initializing site-packages.
• Code clean-up and rework.
• Custom resolution (for now fixed 512x512)
• CFG not affecting image
• Add Lora
• Add Hyper Lora support
• Add ControlNet support
• Add SDXL support

Acknowledgement

Based on the following projects:

• StreamDiffusion - Pipeline-Level Solution for Real-Time Interactive Generation
• TopArray - Interaction between Python/PyTorch tensor operations and TouchDesigner TOPs.