mirror of
https://github.com/modelscope/DiffSynth-Studio.git
synced 2026-03-18 22:08:13 +00:00
91 lines
5.1 KiB
Markdown
91 lines
5.1 KiB
Markdown
# DiffSynth-Studio Documentation
|
|
|
|
Welcome to the magical world of Diffusion models! `DiffSynth-Studio` is an open-source Diffusion model engine developed and maintained by the [ModelScope Community](https://www.modelscope.cn/). We aim to build a universal Diffusion model framework that fosters technological innovation through framework construction, aggregates the power of the open-source community, and explores the boundaries of generative model technology!
|
|
|
|
<details>
|
|
|
|
<summary>Documentation Reading Guide</summary>
|
|
|
|
```mermaid
|
|
graph LR;
|
|
I_want_to_use_models_for_inference_and_training-->sec1[Section 1: Getting Started];
|
|
I_want_to_use_models_for_inference_and_training-->sec2[Section 2: Model Details];
|
|
I_want_to_use_models_for_inference_and_training-->sec3[Section 3: Training Framework];
|
|
I_want_to_develop_based_on_this_framework-->sec3[Section 3: Training Framework];
|
|
I_want_to_develop_based_on_this_framework-->sec4[Section 4: Model Integration];
|
|
I_want_to_develop_based_on_this_framework-->sec5[Section 5: API Reference];
|
|
I_want_to_explore_new_technologies_based_on_this_project-->sec4[Section 4: Model Integration];
|
|
I_want_to_explore_new_technologies_based_on_this_project-->sec5[Section 5: API Reference];
|
|
I_want_to_explore_new_technologies_based_on_this_project-->sec6[Section 6: Academic Guide];
|
|
I_encountered_a_problem-->sec7[Section 7: Frequently Asked Questions];
|
|
```
|
|
|
|
</details>
|
|
|
|
## Section 1: Getting Started
|
|
|
|
This section introduces the basic usage of `DiffSynth-Studio`, including how to enable VRAM management for inference on GPUs with extremely low VRAM, and how to train various base models, LoRAs, ControlNets, and other models.
|
|
|
|
* [Installation Dependencies](./Pipeline_Usage/Setup.md)
|
|
* [Model Inference](./Pipeline_Usage/Model_Inference.md)
|
|
* [VRAM Management](./Pipeline_Usage/VRAM_management.md)
|
|
* [Model Training](./Pipeline_Usage/Model_Training.md)
|
|
* [Environment Variables](./Pipeline_Usage/Environment_Variables.md)
|
|
* [GPU/NPU Support](./Pipeline_Usage/GPU_support.md)
|
|
|
|
## Section 2: Model Details
|
|
|
|
This section introduces the Diffusion models supported by `DiffSynth-Studio`. Some model pipelines feature special functionalities such as controllable generation and parallel acceleration.
|
|
|
|
* [FLUX.1](./Model_Details/FLUX.md)
|
|
* [Wan](./Model_Details/Wan.md)
|
|
* [Qwen-Image](./Model_Details/Qwen-Image.md)
|
|
* [FLUX.2](./Model_Details/FLUX2.md)
|
|
* [Z-Image](./Model_Details/Z-Image.md)
|
|
* [Anima](./Model_Details/Anima.md)
|
|
* [LTX-2](./Model_Details/LTX-2.md)
|
|
|
|
## Section 3: Training Framework
|
|
|
|
This section introduces the design philosophy of the training framework in `DiffSynth-Studio`, helping developers understand the principles of Diffusion model training algorithms.
|
|
|
|
* [Basic Principles of Diffusion Models](./Training/Understanding_Diffusion_models.md)
|
|
* [Standard Supervised Training](./Training/Supervised_Fine_Tuning.md)
|
|
* [Enabling FP8 Precision in Training](./Training/FP8_Precision.md)
|
|
* [End-to-End Distillation Accelerated Training](./Training/Direct_Distill.md)
|
|
* [Two-Stage Split Training](./Training/Split_Training.md)
|
|
* [Differential LoRA Training](./Training/Differential_LoRA.md)
|
|
|
|
## Section 4: Model Integration
|
|
|
|
This section introduces how to integrate models into `DiffSynth-Studio` to utilize the framework's basic functions, helping developers provide support for new models in this project or perform inference and training of private models.
|
|
|
|
* [Integrating Model Architecture](./Developer_Guide/Integrating_Your_Model.md)
|
|
* [Building a Pipeline](./Developer_Guide/Building_a_Pipeline.md)
|
|
* [Enabling Fine-Grained VRAM Management](./Developer_Guide/Enabling_VRAM_management.md)
|
|
* [Model Training Integration](./Developer_Guide/Training_Diffusion_Models.md)
|
|
|
|
## Section 5: API Reference
|
|
|
|
This section introduces the independent core module `diffsynth.core` in `DiffSynth-Studio`, explaining how internal functions are designed and operate. Developers can use these functional modules in other codebase developments if needed.
|
|
|
|
* [`diffsynth.core.attention`](./API_Reference/core/attention.md): Attention mechanism implementation
|
|
* [`diffsynth.core.data`](./API_Reference/core/data.md): Data processing operators and general datasets
|
|
* [`diffsynth.core.gradient`](./API_Reference/core/gradient.md): Gradient checkpointing
|
|
* [`diffsynth.core.loader`](./API_Reference/core/loader.md): Model download and loading
|
|
* [`diffsynth.core.vram`](./API_Reference/core/vram.md): VRAM management
|
|
|
|
## Section 6: Academic Guide
|
|
|
|
This section introduces how to use `DiffSynth-Studio` to train new models, helping researchers explore new model technologies.
|
|
|
|
* [Training models from scratch](./Research_Tutorial/train_from_scratch.md)
|
|
* [Inference improvement techniques](./Research_Tutorial/inference_time_scaling.md)
|
|
* Designing controllable generation models 【coming soon】
|
|
* Creating new training paradigms 【coming soon】
|
|
|
|
## Section 7: Frequently Asked Questions
|
|
|
|
This section summarizes common developer questions. If you encounter issues during usage or development, please refer to this section. If you still cannot resolve the problem, please submit an issue on GitHub.
|
|
|
|
* [Frequently Asked Questions](./QA.md) |