Skip to playerSkip to main content
Unlock the power of Video-to-Video AI generation! In this tutorial, I'll show you how to take a real-world source video and use it to drive stylized animations in Stable Diffusion by combining the forces of AnimatedDiff and ControlNet.

I outline best practices for prepping your video (keeping it short and optimized to 8 FPS to save rendering time) before diving into the setup process. You'll learn how to feed the clip into AnimatedDiff's Video Source menu and leverage a dual-ControlNet stack (DW OpenPose and Canny) to capture motion accurately. I also show you how disabling the Canny node allows the prompt to take creative control over the clothing and background while maintaining the original dance choreography.

Original YouTube Tutorial: https://youtu.be/R2eysgnhmWE

Connected Videos in this Series:
* How to Install Stable Diffusion Automatic1111: https://youtu.be/xrVb6GIKhJQ
* How to Install ControlNet: https://youtu.be/pR0t_f3OGgY
* How to Install AnimatedDiff: https://youtu.be/2vMr8pqPXtc


Source/Model Resources:
* Animatediff Motion Modules: https://huggingface.co/guoyww/animatediff/tree/main
* ControlNet Models (OpenPose & Canny): https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main

Video Details:
* Original Publish Date: May 17, 2024
* Focus: Video-to-Video / AnimatedDiff / ControlNet / Stable Diffusion
* Hardware Environment: RTX 4060 Ti (16GB VRAM) — Rendered a 11-second clip at 8 FPS in roughly 24 minutes.

Follow lordcaocao2025 on Dailymotion for more advanced AI animation techniques, ComfyUI workflows, and technical design research!

---
Connect with me:
📺 YouTube: https://www.youtube.com/@CaoCao2025
📱 TikTok: https://www.tiktok.com/@caocao20250
💎 Patreon: https://www.patreon.com/cw/Caocao2025

#AnimatedDiff #ControlNet #StableDiffusion #VideoToVideo #AIAnimation #Automatic1111 #lordcaocao2025
Transcript
00:00Hello, guys. Welcome back with me, CiaoCiao2025.
00:04Today, we're going to explore more of Stable Diffusion Automatis 11.11 and its extension,
00:12AnimatedIF, and ControlNet. By combining these two extensions, we could make a video that are
00:18inspired by other video. Before we continue, make sure you already have Stable Diffusion,
00:24ControlNet, and AnimatedIF. I already make video about how to install them, so just check the link
00:30on the description. Now that we have all the extension, the first thing you do is to have the
00:39video source. I suggest have a very short video, like a few seconds, just for testing. If you can,
00:46convert the video into very low FPS, I suggest max 8 FPS, so the process won't take a lot of
00:54time.
00:54There are a lot of tools and software for converting and reducing FPS, so I will not make it part
01:01of the
01:01video. In this example, I will have a 10-second video with 8 FPS of a woman dancing. I got
01:09this
01:09video from YouTube, and we will create a video that are inspired by this.
01:23After you have the source video, open your Stable Diffusion. Use your favorite model.
01:29Now activate AnimatedIF by checking Enable AnimatedIF. Put your video on Video Source.
01:40Your FPS and number of frame will be set up depending on your source video.
01:46Next part is to activate ControlNet.
01:50In this video, I use two ControlNet unit, OpenPose with DW OpenPose model and Canny.
01:58Both I use, ControlNet is more important.
02:24Now I put a positive prompt of Pretty Woman, Asia, Dancing, Masterpiece, High Quality.
02:31I also put negative prompt as Low Quality and Worst Quality.
02:36Now we're going to process. It's going to take some time.
02:40And here are the results.
03:02Remember, doing this will take a lot of time, and the more ControlNet unit you use, the more FPS that
03:08you have, or the more frame you generate, will cost you time.
03:40Hmm.
03:41ControlNet, and I add RedShirt in the positive prompt.
03:45And here are the results.
03:53As you can see, the dancing is the same, but the character and background now more following the prompt.
04:00So now you know how it's time for you to generate more long and high FPS video.
04:08Anyway, guys, that's it for me.
04:13Don't forget to like, share and comment.
04:15See you again on the next episode.
04:17Have a nice day.
04:19Please subscribe.
Comments

Recommended