Let's finetune video generation models!

Agent Laboratory is an end-to-end autonomous research workflow meant to assist you as the human researcher toward implementing your research ideas

VideoVAE+: Large Motion Video Autoencoding with Cross-modal Video VAE

Next-Token Prediction is All You Need

Depth Pro: Sharp Monocular Metric Depth in Less Than a Second.

High-resolution models for human tasks.

Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

Official code for "RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control"

xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism

[Arxiv 2024] Official code for MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions

Use PEFT or Full-parameter to finetune 450+ LLMs (Qwen2.5, InternLM3, GLM4, Llama3.3, Mistral, Yi1.5, Baichuan2, DeepSeek-R1, ...) and 150+ MLLMs (Qwen2.5-VL, Qwen2-Audio, Llama3.2-Vision, Llava, I…

CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets

A high-throughput and memory-efficient inference and serving engine for LLMs

Creative Commons Licenses for Github

Pythonic bindings for FFmpeg's libraries.

Text-to-3D Generation within 5 Minutes

DeepSeek-VL: Towards Real-World Vision-Language Understanding

[WIP] Layer Diffusion for WebUI (via Forge)

[CVPR 2024 Highlight] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

Transparent Image Layer Diffusion using Latent Transparency

PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

Official Code for MotionCtrl [SIGGRAPH 2024]

🔥🔥🔥 A curated list of papers on LLMs-based multimodal generation (image, video, 3D and audio).

Code for the paper "Pix2Video: Video Editing using Image Diffusion"

Focus on prompting and generating

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

A feature-rich command-line audio/video downloader