IMG2IMG+Partial Repainting

Learn about ComfyUI's image-to-image workflow and four powerful partial repainting methods: VAE Encode, Set Latent Noise Mask, ControlNet Inpaint, and CLIPSeg.

Basic Image to Image

Img to Img can be adjusted based on Text to Image, adding the "Load Image" and "VAE Encode" nodes.

Since the input image is just a pixel image, it can't be directly placed into the latent space. Therefore, it requires a VAE encoder to encode the image so that the latent space can recognize it. Here, the final generated image size is consistent with the original image.

Empty Latent Image: The previous Text to Image must denoise through Empty Latent Image before generating a new image. Now that an image has been added, the Empty Latent Image is no longer needed.

Workflow: Upload an Image → Select Models → Enter Prompts → Adjust Parameters → Generate.

Parameters:

denoise: Equivalent to Denoising Strength, can be adjusted between 0-1.

Using Img2Img allows for changing styles, repairing images, extending images, high-definition restoration, etc.

Pre-processing Image

You can scale or crop the image by adding different nodes, or choose not to add any.

Upscale Image/Upscale Image By: Upscale the image.

ImageCrop: Crop the image.

Partial Repainting

Four Methods: VAE Encode (for Inpainting), Set Latent Noise Mask, ControlNet Inpaint, and CLIPSeg.

1. VAE Encode (for Inpainting)

Add VAE Encode (for Inpainting), connect Mask, right-click on the image, and select Open in MaskEditor to draw a mask. If there are issues with a section of the drawn mask, you can erase it by holding down the right mouse button.

Workflow: Choose a model similar to the original image, and enter prompts for the masked part.

VAE Encode (for Inpainting): Equivalent to repainting, with higher randomness, and the original masked area will not be preserved.

1. Set Latent Noise Mask

First, encode the image through VAE to turn it into content recognizable by the latent space, then regenerate the masked area as noise content.

Set Latent Noise Mask: It will refer to the original image for repainting, ensuring a better understanding of the generated content, with a lower probability of generating incorrect images, thus suitable for fine-tuning while maintaining similarity to the original image.

1. ControlNet Inpaint

Add the ControlNet, select the Inpaint model, and preprocess the image accordingly (Inpaint Preprocessor).

Note: Don't forget to add the VAE encoder so the image can enter the latent space.

1. CLIPSeg

Enter prompts to automatically divide the mask areas, eliminating the need for manual paint-over. It can be used together with Set Latent Noise Mask.

Parameters:

text: Input the area you want to repaint.

threshold: The precision level of content recognition.

dilation_factor: The diffusion degree of content recognition.

Output:

Heatmap Mask: Heatmap image.

BW Mask: Black and white image.

You can preview the recognized mask areas separately.

Differences between the four repainting methods:

1. VAE Encode (for Inpainting): Equivalent to erasing and repainting with higher randomness, suitable for generation from scratch.

2. Set Latent Noise Mask: It will refer to the original image for repainting, ensuring a certain similarity to the original image, suitable for fine-tuning.

3. ControlNet Inpaint: Relatively stable and refined.

4. CLIPSeg: Automatically recognizes the mask areas, so there's no need for manual paint-over, making it more convenient.