mirror of
https://github.com/modelscope/DiffSynth-Studio.git
synced 2026-03-19 06:23:43 +00:00
64 lines
5.7 KiB
Markdown
64 lines
5.7 KiB
Markdown
# EliGen: Entity-Level Controlled Image Generation
|
|
|
|
## Introduction
|
|
|
|
We propose EliGen, a novel approach that leverages fine-grained entity-level information to enable precise and controllable text-to-image generation. EliGen excels in tasks such as entity-level controlled image generation and image inpainting, while its applicability is not limited to these areas. Additionally, it can be seamlessly integrated with existing community models, such as the IP-Adpater.
|
|
|
|
* Paper: Comming soon
|
|
* Github: https://github.com/modelscope/DiffSynth-Studio
|
|
* Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen)
|
|
* Training dataset: Coming soon
|
|
|
|
## Methodology
|
|
|
|
|
|
|
|
We introduce a regional attention mechanism within the DiT framework to effectively process the conditions of each entity. This mechanism enables the local prompt associated with each entity to semantically influence specific regions through regional attention. To further enhance the layout control capabilities of EliGen, we meticulously curate an entity-annotated dataset and fine-tune the model using the LoRA framework.
|
|
|
|
1. **Regional Attention**: Regional attention is shown in above figure, which can be easily applied to other text-to-image models. Its core principle involves transforming the positional information of each entity into an attention mask, ensuring that the mechanism only affects the designated regions.
|
|
|
|
2. **Dataset with Entity Annotation**: To curate a dedicated entity control dataset, we start by randomly selecting captions from DiffusionDB and generating the corresponding source image using Flux. Next, we employ Qwen2-VL 72B, recognized for its advanced grounding capabilities among MLLMs, to randomly identify entities within the image. These entities are annotated with local prompts and bounding boxes for precise localization, forming the foundation of our dataset for further training.
|
|
|
|
3. **Training**: We apply LoRA and deepspeed to finetune regional attention with curated dataset, enabling our EliGen performing effective entity-level control.
|
|
|
|
## Usage
|
|
1. **Entity-Level Controlled Image Generation**
|
|
See [./entity_control.py](./entity_control.py) for usage.
|
|
2. **Image Inpainting**
|
|
To apply EliGen to image inpainting task, we propose a inpainting fusion pipeline to preserve the non-painting areas while enabling precise, entity-level modifications over inpaining regions.
|
|
See [./entity_inpaint.py](./entity_inpaint.py) for usage.
|
|
3. **Styled Entity Control**
|
|
EliGen can be seamlessly integrated with existing community models. We have provided an example of how to integrate it with the IP-Adpater. See [./entity_control_ipadapter.py](./entity_control_ipadapter.py) for usage.
|
|
4. **Play with EliGen using UI**
|
|
Download the checkpoint of EliGen from [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen) to `models/lora/entity_control` and run the following command to try interactive UI:
|
|
```bash
|
|
python apps/gradio/entity_level_control.py
|
|
```
|
|
## Examples
|
|
### Entity-Level Controlled Image Generation
|
|
|
|
1. The effect of generating images with continuously changing entity positions.
|
|
<div style="text-align:center;">
|
|
<video width="768" height="768" controls>
|
|
<source src="https://github.com/user-attachments/assets/4fc76df1-b26a-46e8-a950-865cdf02a38d" type="video/mp4">
|
|
your browser does not support the video tag
|
|
</video>
|
|
</div>
|
|
|
|
2. The image generation effect of complex Entity combinations, demonstrating the strong generalization of EliGen.
|
|
|
|
|||
|
|
|-|-|
|
|
|||
|
|
|||
|
|
|||
|
|
### Image Inpainting
|
|
|Inpainting Input|Inpainting Output|
|
|
|-|-|
|
|
|||
|
|
|||
|
|
### Styled Entity Control
|
|
|Style Reference|Entity Control Variance 1|Entity Control Variance 2|Entity Control Variance 3|
|
|
|-|-|-|-|
|
|
|||||
|
|
||||| |