++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling
++: Instruction-Based Image Creation and Editing
Chaojie Mao
·
Jingfeng Zhang
·
Yulin Pan
·
Zeyinzi Jiang
·
Zhen Han
·
Yu Liu
·
Jingren Zhou
Tongyi Lab, Alibaba Group
|
| Workflow | Description | Setting | ||||
| ACE_Plus_FFT_workflow_no_preprocess.json | Use the preprocessed images, such as depth and contour, as input, or the super-resolution. | Task_type: no_preprocess (you don't need to install dependencies like scepter) | ||||
| ACE_Plus_FFT_workflow_controlpreprocess.json | Controllable image-to-image translation capability. To preprocess depth and contour information from images, we use externally-provided models that are typically downloaded from the ModelScope Hub. Because download success can vary depending on the user's environment, we offer alternatives: users can either leverage existing community nodes (depth extration node or contour extraction node) for this task (then choosing the 'no_preprocess' option), or users can pre-download the required models contour and depth and adjust the configuration file 'workflow/ComfyUI-ACE_Plus/config/ace_plus_fft_processor.yaml' to specify the models' local paths. | Task_type: contour_repainting/depth_repainting/recolorizing (you need to install dependencies like scepter) | ||||
| ACE_Plus_FFT_workflow_reference_generation.json | Reference image generation capability for portrait or subject. | Task_type: repainting (you don't need to install dependencies like scepter) | ||||
| ACE_Plus_FFT_workflow_referenceediting_generation.json | Reference image editing capability | Task_type: repainting (you don't need to install dependencies like scepter) | ||||
| Input Reference Image | Input Edit Image | Input Edit Mask | Output | Instruction | Function |
 |
 |
"Maintain the facial features, A girl is wearing a neat police uniform and sporting a badge. She is smiling with a friendly and confident demeanor. The background is blurred, featuring a cartoon logo." | "Character ID Consistency Generation" | ||
 |
 |
"Display the logo in a minimalist style printed in white on a matte black ceramic coffee mug, alongside a steaming cup of coffee on a cozy cafe table." | "Subject Consistency Generation" | ||
 |
 |
 |
 |
"The item is put on the table." | "Subject Consistency Editing" |
 |
 |
 |
 |
"The logo is printed on the headphones." | "Subject Consistency Editing" |
 |
 |
 |
 |
"The woman dresses this skirt." | "Try On" |
 |
|
 |
 |
"{image}, the man faces the camera." | "Face swap" |
 |
 |
 |
"{image} features a close-up of a young, furry tiger cub on a rock. The tiger, which appears to be quite young, has distinctive orange, black, and white striped fur, typical of tigers. The cub's eyes have a bright and curious expression, and its ears are perked up, indicating alertness. The cub seems to be in the act of climbing or resting on the rock. The background is a blurred grassland with trees, but the focus is on the cub, which is vividly colored while the rest of the image is in grayscale, drawing attention to the tiger's details. The photo captures a moment in the wild, depicting the charming and tenacious nature of this young tiger, as well as its typical interaction with the environment." | "Super-resolution" | |
|
 |
 |
"a blue hand" | "Regional Editing" | |
|
 |
 |
"Mechanical hands like a robot" | "Regional Editing" | |
 |
 |
 |
"{image} Beautiful female portrait, Robot with smooth White transparent carbon shell, rococo detailing, Natural lighting, Highly detailed, Cinematic, 4K." | "Recolorizing" | |
 |
|
 |
"{image} Beautiful female portrait, Robot with smooth White transparent carbon shell, rococo detailing, Natural lighting, Highly detailed, Cinematic, 4K." | "Depth Guided Generation" | |
 |
|
 |
"{image} Beautiful female portrait, Robot with smooth White transparent carbon shell, rococo detailing, Natural lighting, Highly detailed, Cinematic, 4K." | "Contour Guided Generation" |
| Tuning Method | Input | Output | Instruction | Models |
| LoRA + ACE Data |
|
 |
"Maintain the facial features. A girl is wearing a neat police uniform and sporting a badge. She is smiling with a friendly and confident demeanor. The background is blurred, featuring a cartoon logo." |   |
| Tuning Method | Input | Output | Instruction | Models |
| LoRA + ACE Data |
|
 |
"Display the logo in a minimalist style printed in white on a matte black ceramic coffee mug, alongside a steaming cup of coffee on a cozy cafe table." | |
| Tuning Method | Input | Output | Instruction | Models |
| LoRA + ACE Data |
  |
 |
"By referencing the mask, restore a partial image from the doodle {image} that aligns with the textual explanation: "1 white old owl"." | |
| Application | ACE++ Model | Examples | ||||
|---|---|---|---|---|---|---|
| Try On | ACE++ Subject | |
|
|
 |
"The woman dresses this skirt." |
| Logo Paste | ACE++ Subject | |
|
|
 |
"The logo is printed on the headphones." |
| Photo Editing | ACE++ Subject | |
 |
 |
 |
"The item is put on the ground." |
| Movie Poster Editor | ACE++ Portrait | |
 |
 |
 |
"The man is facing the camera and is smiling." |
export FLUX_FILL_PATH="path/to/FLUX.1-Fill-dev"
export PORTRAIT_MODEL_PATH="path/to/ACE++ PORTRAIT PATH"
export SUBJECT_MODEL_PATH="path/to/ACE++ SUBJECT PATH"
export LOCAL_MODEL_PATH="path/to/ACE++ LOCAL EDITING PATH"export FLUX_FILL_PATH="hf://black-forest-labs/FLUX.1-Fill-dev"
export PORTRAIT_MODEL_PATH="${scepter_path}"
export SUBJECT_MODEL_PATH="${scepter_path}"
export LOCAL_MODEL_PATH="${scepter_path}"