AI Art Generation Handbook/ControlNet/Canny

Canny in this context meant Canny Edge Detection algorithm which is kind of popular edge detection used in computer vision.

To use Canny mode , you need to ensure you have downloaded Canny models from here https://huggingface.co/lllyasviel/sd_control_collection/tree/main and search for the following Canny pre-trained ControlNet models .

ControlNet Canny Pre-processed Model	GPU VRAM Recommendation
diffusers_xl_canny_full.safetensors	X > 12GB VRAM
diffusers_xl_canny_mid.safetensors	8GB VRAM >X > 12 GB VRAM
diffusers_xl_canny_small.safetensors	X < 8GB VRAM

Workflow:

First of all, we need to have a base image to work with. Such as picture of this geisha playing shamisen.

Then , you can think of a similar picture such as a rockstar playing guitar.

We shall use prompt : "Photo of a rockstar playing heavy metal music with electric guitar,  kneeling on the stage"

Each of the effect are displayed below

Note: All of the settings are the same as per default unless mentioned otherwise .

Control Weight

Control Weight controls the amount of influence that the reference image has on the generated image. A higher Control Weight will result in a more similar image, while a lower Control Weight will result in a more original image.

Control Weight	1.0	0.8	0.6	0.4	0.2
Generated ControlNet Images

Preprocessing Resolution

Pre-Processing Resolution setting in Stable Diffusion ControlNet controls the resolution of the image that is used to train the model. A higher Pre-Processing Resolution will make the model more accurate, but it will also require more computation. Note: If you are using a reference image with a lot of detail, you may need to use a higher Pre-Processing Resolution to capture all of the detail in the generated image. If you are limited by computation, you should use a lower Pre-Processing Resolution.

Preprocessing Resolution	300	600	900	1200
Generated ControlNet Images
	1200	1500	1800	2048
Generated ControlNet Images