From 03cba5e59ebf7751f34af4f460a7475ba67e69f3 Mon Sep 17 00:00:00 2001 From: wenmeng zhou Date: Thu, 27 Jun 2024 15:56:51 +0800 Subject: [PATCH] Update README.md --- README.md | 94 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 52 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index d74ea0a..7a3b908 100644 --- a/README.md +++ b/README.md @@ -4,49 +4,59 @@ DiffSynth Studio is a Diffusion engine. We have restructured architectures including Text Encoder, UNet, VAE, among others, maintaining compatibility with models from the open-source community while enhancing computational performance. We provide many interesting features. Enjoy the magic of Diffusion models! -## Roadmap +Until now, DiffSynth Studio has supported the following models: + +* [ExVideo](https://huggingface.co/ECNU-CILab/ExVideo-SVD-128f-v1) +* [Stable Video Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt) +* [Hunyuan-DiT](https://github.com/Tencent/HunyuanDiT) +* [RIFE](https://github.com/hzwer/ECCV2022-RIFE) +* [ESRGAN](https://github.com/xinntao/ESRGAN) +* [Ip-Adapter](https://github.com/tencent-ailab/IP-Adapter) +* [AnimateDiff](https://github.com/guoyww/animatediff/) +* [ControlNet](https://github.com/lllyasviel/ControlNet) +* [Stable Diffusion XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) +* [Stable Diffusion](https://huggingface.co/runwayml/stable-diffusion-v1-5) + +## News + + +- **June 21, 2024.** We propose ExVideo, a post-tuning technique aimed at enhancing the capability of video generation models. We have extended Stable Video Diffusion to achieve the generation of long videos up to 128 frames. + - [Project Page](https://ecnu-cilab.github.io/ExVideoProjectPage/) + - Source code is released in this repo. See [`examples/ExVideo`](./examples/ExVideo/). + - Models are released on [HuggingFace](https://huggingface.co/ECNU-CILab/ExVideo-SVD-128f-v1) and [ModelScope](https://modelscope.cn/models/ECNU-CILab/ExVideo-SVD-128f-v1). + - Technical report is released on [arXiv](https://arxiv.org/abs/2406.14130). + +- **June 13, 2024.** DiffSynth Studio is transferred to ModelScope. The developers have transitioned from "I" to "we". Of course, I will still participate in development and maintenance. + +- **Jan 29, 2024.** We propose Diffutoon, a fantastic solution for toon shading. + - [Project Page](https://ecnu-cilab.github.io/DiffutoonProjectPage/) + - The source codes are released in this project. + - The technical report (IJCAI 2024) is released on [arXiv](https://arxiv.org/abs/2401.16224). + +- **Dec 8, 2023.** We decide to develop a new Project, aiming to release the potential of diffusion models, especially in video synthesis. The development of this project is started. + +- **Nov 15, 2023.** We propose FastBlend, a powerful video deflickering algorithm. + - The sd-webui extension is released on [GitHub](https://github.com/Artiprocher/sd-webui-fastblend). + - Demo videos are shown on Bilibili, including three tasks. + - [Video deflickering](https://www.bilibili.com/video/BV1d94y1W7PE) + - [Video interpolation](https://www.bilibili.com/video/BV1Lw411m71p) + - [Image-driven video rendering](https://www.bilibili.com/video/BV1RB4y1Z7LF) + - The technical report is released on [arXiv](https://arxiv.org/abs/2311.09265). + - An unofficial ComfyUI extension developed by other users is released on [GitHub](https://github.com/AInseven/ComfyUI-fastblend). + +- **Oct 1, 2023.** We release an early version of this project, namely FastSDXL. A try for building a diffusion engine. + - The source codes are released on [GitHub](https://github.com/Artiprocher/FastSDXL). + - FastSDXL includes a trainable OLSS scheduler for efficiency improvement. + - The original repo of OLSS is [here](https://github.com/alibaba/EasyNLP/tree/master/diffusion/olss_scheduler). + - The technical report (CIKM 2023) is released on [arXiv](https://arxiv.org/abs/2305.14677). + - A demo video is shown on [Bilibili](https://www.bilibili.com/video/BV1w8411y7uj). + - Since OLSS requires additional training, we don't implement it in this project. + +- **Aug 29, 2023.** We propose DiffSynth, a video synthesis framework. + - [Project Page](https://ecnu-cilab.github.io/DiffSynth.github.io/). + - The source codes are released in [EasyNLP](https://github.com/alibaba/EasyNLP/tree/master/diffusion/DiffSynth). + - The technical report (ECML PKDD 2024) is released on [arXiv](https://arxiv.org/abs/2308.03463). -* Aug 29, 2023. We propose DiffSynth, a video synthesis framework. - * [Project Page](https://ecnu-cilab.github.io/DiffSynth.github.io/). - * The source codes are released in [EasyNLP](https://github.com/alibaba/EasyNLP/tree/master/diffusion/DiffSynth). - * The technical report (ECML PKDD 2024) is released on [arXiv](https://arxiv.org/abs/2308.03463). -* Oct 1, 2023. We release an early version of this project, namely FastSDXL. A try for building a diffusion engine. - * The source codes are released on [GitHub](https://github.com/Artiprocher/FastSDXL). - * FastSDXL includes a trainable OLSS scheduler for efficiency improvement. - * The original repo of OLSS is [here](https://github.com/alibaba/EasyNLP/tree/master/diffusion/olss_scheduler). - * The technical report (CIKM 2023) is released on [arXiv](https://arxiv.org/abs/2305.14677). - * A demo video is shown on [Bilibili](https://www.bilibili.com/video/BV1w8411y7uj). - * Since OLSS requires additional training, we don't implement it in this project. -* Nov 15, 2023. We propose FastBlend, a powerful video deflickering algorithm. - * The sd-webui extension is released on [GitHub](https://github.com/Artiprocher/sd-webui-fastblend). - * Demo videos are shown on Bilibili, including three tasks. - * [Video deflickering](https://www.bilibili.com/video/BV1d94y1W7PE) - * [Video interpolation](https://www.bilibili.com/video/BV1Lw411m71p) - * [Image-driven video rendering](https://www.bilibili.com/video/BV1RB4y1Z7LF) - * The technical report is released on [arXiv](https://arxiv.org/abs/2311.09265). - * An unofficial ComfyUI extension developed by other users is released on [GitHub](https://github.com/AInseven/ComfyUI-fastblend). -* Dec 8, 2023. We decide to develop a new Project, aiming to release the potential of diffusion models, especially in video synthesis. The development of this project is started. -* Jan 29, 2024. We propose Diffutoon, a fantastic solution for toon shading. - * [Project Page](https://ecnu-cilab.github.io/DiffutoonProjectPage/). - * The source codes are released in this project. - * The technical report (IJCAI 2024) is released on [arXiv](https://arxiv.org/abs/2401.16224). -* June 13, 2024. DiffSynth Studio is transferred to ModelScope. The developers have transitioned from "I" to "we". Of course, I will still participate in development and maintenance. -* June 21, 2024. We propose ExVideo, a post-tuning technique aimed at enhancing the capability of video generation models. We have extended Stable Video Diffusion to achieve the generation of long videos up to 128 frames. - * [Project Page](https://ecnu-cilab.github.io/ExVideoProjectPage/). - * Source code is released in this repo. See [`examples/ExVideo`](./examples/ExVideo/). - * Models are released on [HuggingFace](https://huggingface.co/ECNU-CILab/ExVideo-SVD-128f-v1) and [ModelScope](https://modelscope.cn/models/ECNU-CILab/ExVideo-SVD-128f-v1). - * Technical report is released on [arXiv](https://arxiv.org/abs/2406.14130). -* Until now, DiffSynth Studio has supported the following models: - * [Stable Diffusion](https://huggingface.co/runwayml/stable-diffusion-v1-5) - * [Stable Diffusion XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) - * [ControlNet](https://github.com/lllyasviel/ControlNet) - * [AnimateDiff](https://github.com/guoyww/animatediff/) - * [Ip-Adapter](https://github.com/tencent-ailab/IP-Adapter) - * [ESRGAN](https://github.com/xinntao/ESRGAN) - * [RIFE](https://github.com/hzwer/ECCV2022-RIFE) - * [Hunyuan-DiT](https://github.com/Tencent/HunyuanDiT) - * [Stable Video Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt) - * [ExVideo](https://huggingface.co/ECNU-CILab/ExVideo-SVD-128f-v1) ## Installation