Stable Diffusion ControlNet Explained Control Net Examples
Hey friends, welcome to Image Pipeline, You have been hearing about stable diffusion with Controlnet almost everywhere, everywhere you go. If you’re a stable diffusion fan, everybody has been using a controlnet. So in this post, I’m going to try my level best to explain to you about controlnet and show you things that people have been doing with Controlnet.
It is quite amazing what people have been doing with controlnet and we are going to see every one of them. And to start with first, let’s try to understand what is even controlnet. So if you see controlnet, it’s a very simple thing. I mean, it’s not simple. So controlnet is a neural net architecture.
Controlnet 1.1 — Canny
Controlnet 1.1 — Depth
Controlnet 1.1 — HED
Controlnet 1.1 — Lineart
Controlnet 1.1 — Seg
Controlnet 1.1 — Scribble
So it’s a new neural net structure that helps you control diffusion models like stable diffusion models by adding extra conditions. So to show you what controlnet can do, I have come up with a very, weird example, but, if you have not watched the movie, Logan, it’s a spoiler for you. So if you have watched the movie, Logan, what happens that they have taken Wolverine’s DNA and then made some modifications and came up with this girl.
So that’s what the entire Logan movie is about. So you have got Wolverine who is actually a natural mutant and they have taken the natural mutant and then they have, they have made this girl from Wolverine’s DNA and this is exactly what controlnet can do. So controlnet can take an existing stable diffusion or diffusion model architecture, and then just make slight changes to the architecture and then add whatever you want to just like what they’ve done with this girl here.
So if you now look at controlnet examples. So, you can upload an image and then ask controlnet to hold some properties of the image and then change other properties. I’ll give you the easiest example that everybody has been looking at. So for example, if you look at this, this is controlnet, stable diffusion controlnet with the pose.
So you can see here that you can upload an image and it is going to preserve the post. It’s not going to do anything else. It’s not going to preserve the scenery. It’s going to preserve the post and it is going to come up with new poses or new images for the same post based on the prompt that you give.
And it is not only for posts like you would have seen a lot of examples of posts. A lot of people have been showing examples of posts, but it works for a lot of other things. And that’s like, you can see, for example, here you have got a. It’s a shoe and it can hold the scribble map, the fake scribble map, and from that it can design new things.
And you can see a lot of examples here. How it can hold edges, how it can, you know, take a simple scribble and then build images on top of it. So you can see a lot of examples here, but I’m not going to show you the examples here. But first I want to show you the growth of controlnet. Like if you have been wondering why everybody’s talking about controlnet, you know, right now that controlnet is quite amazing in holding certain properties of a neural network.
And even to say how it does, it actually makes a copy of a neural network, just like in Logan. It makes a copy of a neural network and it holds one neural network and it makes changes in another neural network And when you get the final output, it has both the properties That’s why when you upload a picture of a man standing, you can hold the post and then instead of a man, you can put a woman, you can put a kite, you can put a kid, you can put a robot with the same post, you can do anything.
So that is happening because now there is a duplicate copy and then it combines further. So now this is about controlling. Now if you simply look at controlnets growth, this has been tweeted by hugging face CEO. You can see that controlnet, like what Clem has said is that there are already 50. Public and open controlnet models on the hugging face model up.
It has got more than a hundred and 1200 likes, and you can see the growth of controlnet is almost as exponential as stable division. So people have been really grow going crazy with what controlnet can do. And this is a great example of what controlnet can do. And you can see how the trajectories growth is.
It’s quite amazing what it can do. It has a lot of things people are still figuring out every day what they’re doing. So now that we have learned about controlnet, what is controlnet or controlnet explanation. Now that we have also learned about how controlnet is growing. Now let’s go and then look at some examples of what people are doing with controlnet.
So first you can see, just search for controlnet. You can see these kinds of animations where people have taken an image with different poses. And then they are trying to use controlnet. Also sometimes with the blender, EpiSynth, lot of different applications try to combine those images. Like for example, you can see.
How you can take an image, take the pose, and then create certain aspects around it. Then now you can create another image. So this is one thing that people are doing. The other thing that people are doing is. Controlnet with NERF. N E R F. Controlnet with NERF. So this is by Bilawal Sidhu. So you can see this tweet, how you.
If you have to emulate a drone shot, people would use typically drones or, you know, robotic arm cameras for like this, but here this is all simply screenshots or photos, snapshots and controlnet, and a bunch of other things. this is another new trend that a lot of people have been doing.
Use controlnet image. And ask it to, you know, capture either, the, the edges or so what you can do is you can upload a logo, a brand logo. And as control needs to capture the edges, as we have seen, like certain control unit models can keep the edges intact and use that to come up with new landscapes, and new images.
This is, this is quite amazing from a brand perspective. Let’s say you want to make an advertisement copy and you want to show your brand logo in the middle of a desert, a tennis court, or a football court somewhere. And this is quite amazing. As you can see how. They’ve naturally embedded Nike logos in literally any place that you say, like a hill station, desert, volcano, landscape, sea, wherever you want.
And this is all not very difficult to do. This is quite easy to do and all it requires is the right Controlnet model that you want to use with stable deficient and the right prompt that you want to give. And, again, this is another popular use case. A lot of people have been playing with this. I’ve made a short about it.
It’s called a scribble diffusion. com. You can go to scribble diffusion. com. You can scribble something and give a nice prompt, and then it is going to create that prompt as a result. For example, now, if you think this looks like. I mean, you’re, you think that you’re actually making a cat, so make a cat like this and then, you know, mention whatever you want, like cat wearing a cowboy hat.
And it is going to help you create a cat wearing a cowboy hat because we have actually seen that controlnet can create that or hold that scribble map. And then from that it can, it can generate new images. So, so like I said, the applications are amazing. So the latest one is how you can use Controlnet to create a movie or a scene.
Like for example, the problem that people have been having with stable diffusion is Control, stable division is really good, but creating a consistent scene, like having a control over what you want has not been very easy with stable division always. So that’s where controlnet comes and plays a very vital role.
Imagine you want a character to be on like this and now how do you do it? It’s quite simple with controlnet. You need to just create the post. Put the post and then that will do it. And you can see certain examples like how you can feel like you’re a movie director or you can make an animation Using controlnet and put the characters in the right place And finally, I would like to show you another important.
Discovery by Dushyant It’s not just you can give an input image and extract the pose and then create an output image You can also create your own pose and then based on that pose you can generate image Like for example, this is a pose that was not extracted from an image but rather it is a pose that has been created with the right colors for the open post model and using that you can actually create any image that you want and I mean, like I said, I can just keep on going on.
So people have used control with Dream Booth. For example, if you have got your model or if you have got a model for a, let’s say celebrity or, and you are an advertisement company, you’ve got a dream booth model for somebody. Now you can use controlnet and place them. In such a way that they have been posing for that thing.
So overall controlnet is quite amazing. It is taking stable diffusion completely forward into a totally new place So if you want to use controlnet the easiest place to start with is the hugging face models and hugging face demos I’ll link it in the end of this post.
That is also something that you can explore. I’m yet to make hands-on tutorials with controlling, but I thought I needed to actually explain what is controlnet doing here and how, and what kind of potential Controlnet has. If you have not started using Controlnet, I think today is the best day to start with.
It is quite amazing. I hope this post was helpful to you in learning about controlnet and what controlnet is actually doing here. So there are a lot of examples about what kind of properties controlnet can hold.
Please check it out. Try out the Controlnet. Thanks, Links Below
Download All Models from Hugging Face
