MagicQuill V2

Precise and Interactive Image Editing with Layered Visual Cues

Zichen Liu*,1,2 Yue Yu*,1,2 Hao Ouyang2 Qiuyu Wang2 Shuailei Ma2,3 Ka Leong Cheng2 Wen Wang2,4 Qingyan Bai1,2 Yuxuan Zhang5 Yanhong Zeng2 Yixuan Li2,5 Xing Zhu2 Yujun Shen2 Qifeng Chen1
1HKUST, 2Ant Group, 3NEU, 4ZJU, 5CUHK
📄 Paper 💻 GitHub 🤗 HF Demo

Abstract

We propose MagicQuill V2, a novel system that introduces a layered composition paradigm to generative image editing, bridging the gap between the semantic power of diffusion models and the granular control of traditional graphics software. While diffusion transformers excel at holistic generation, their use of singular, monolithic prompts fails to disentangle distinct user intentions for content, position, and appearance. To overcome this, our method deconstructs creative intent into a stack of controllable visual cues: a content layer for what to create, a spatial layer for where to place it, a structural layer for how it is shaped, and a color layer for its palette. Our technical contributions include a specialized data generation pipeline for context-aware content integration, a unified control module to process all visual cues, and a fine-tuned spatial branch for precise local editing, including object removal. Extensive experiments validate that this layered approach effectively resolves the user intention gap, granting creators direct, intuitive control over the generative process.

Layered Editing Examples

(Hover over Layered Composition to decompose layers)

Content Layer

Layered Composition
hover me
Background Layer
Fill Mask Layer
Prop 3 Layer
Prop 2 Layer
Prop 1 Layer
Editing Result
Result
Editing Prompt: A girl sits and embraces her dog, with her vintage car parked near by.
Layered Composition
hover me
Background Layer
Fill Mask Layer
Prop 2 Layer
Prop 1 Layer
Editing Result
Result
Editing Prompt: A man in cloak rides a white horse through a serene lake at dawn.
Layered Composition
Background Layer
Fill Mask Layer
Prop 1 Layer
Prop 2 Layer
Editing Result
Result
Editing Prompt: A group photo of 4 people in a sunny day.
Layered Composition
Background Layer
Prop 4 Layer
Prop 3 Layer
Prop 2 Layer
Prop 1 Layer
Editing Result
Result
Editing Prompt: A shot of a modern industrial living room with furnitures.

Control Layer

Structural Layer: Edge

Layered Composition
Background Layer
Add Brush Layer
Editing Result
Result
Editing Prompt: Add a leather collar with bell to the dog.
Layered Composition
Background Layer
Subtract Brush Layer
Add Brush Layer
Editing Result
Result
Editing Prompt: Make this cup extra large.
Layered Composition
Background Layer
Add Brush Layer
Editing Result
Result
Editing Prompt: Add a flower in this hand.
Layered Composition
Background Layer
Add Brush Layer
Editing Result
Result
Editing Prompt: Let him wear a golden dollar-shape glasses.

Color Layer: Color

Layered Composition
Background Layer
Color Layer
Editing Result
Result
Editing Prompt: Turn her T-shirt's color to deep green.
Layered Composition
Background Layer
Color Layer
Editing Result
Result
Editing Prompt: Turn his T-shirt into vibrant, seamless multi-color horizontal gradient
Layered Composition
Background Layer
Color Layer
Editing Result
Result
Editing Prompt: Turn the flowers into blue and red.
Layered Composition
Background Layer
Color Layer
Editing Result
Result
Editing Prompt: Turn her crown into blue diamond.

Spatial Layer: Local Edit

Layered Composition
Background Layer
Fill Mask Layer
Editing Result
Result
Editing Prompt: Turn the dog's head to face the camera.
Layered Composition
Background Layer
Fill Mask Layer
Editing Result
Result
Editing Prompt: Convert the man into black and white lineart style.
Layered Composition
Background Layer
Fill Mask Layer
Editing Result
Result
Editing Prompt: Add candles on the cake and a fork by the cake.
Layered Composition
Background Layer
Fill Mask Layer
Editing Result
Result
Editing Prompt: Change the text from "NanoBanana" to "MagicQuill", and the banana neon icon to a feather quill icon.

Spatial Layer: Removal

Layered Composition
Background Layer
Subtract Brush Layer
Editing Result
Result
Editing Prompt: Remove the instance.
Layered Composition
Background Layer
Subtract Brush Layer
Editing Result
Result
Editing Prompt: Remove the instance.
Layered Composition
Background Layer
Subtract Brush Layer
Editing Result
Result
Editing Prompt: Remove the instance.
Layered Composition
Background Layer
Subtract Brush Layer
Editing Result
Result
Editing Prompt: Remove the instance.

Tutorial

System Overview

The Toolbar (A) features a new Local Edit Brush for defining the target editing area, along with tools from MagicQuill V1.

The Visual Cue Manager (B) holds all content layer visual cues (foreground props) that users can drag onto the canvas to define what to generate.

Users can refine these cues using the Image Segmentation Panel (C) by clicking the segment icon . This panel allows precise object extraction using dots or bounding boxes, powered by SAM.

MagicQuill V2 UI

Image Segmentation

Click the segment icon to enter the segmentation UI. Users can perform four operations:

  1. Add positive dots to indicate areas to include.
  2. Add negative dots to indicate areas to exclude.
  3. Add bounding box to bound the region of interest.
  4. Use eraser to refine and erase unwanted areas.

After segmentation, click the Save/Save as new prop button to add the foreground prop to the Visual Cue Manager, or fill with any brush.

Layer Operations

1. Content Layer

Users can click a foreground prop in the Visual Cue Manager to add it to the canvas for puzzle-like editing. Use the local edit brush to specify the edit location. The result will respect the user-provided foreground props.

Layered Composition
Background Layer
Fill Mask Layer
Prop 1 Layer
Prop 2 Layer
Prop 3 Layer
Prop 4 Layer
Editing Result
Result
Editing Prompt: A man in denim jacket holds a coke and shows a finger heart.
Layered Composition
Background Layer
Fill Mask Layer
Prop 1 Layer
Prop 2 Layer
Editing Result
Result
Editing Prompt: A man hugs the rabbit and the fox.

2. Structural Layer

Users can use the add brush to sketch edges and the subtract brush to mask existing edges. The result will follow the user's edge map.

Layered Composition
Background Layer
Subtract Brush Layer
Add Brush Layer
Editing Result
Result
Editing Prompt: Let this cute ghost have arms and legs.
Layered Composition
Background Layer
Subtract Brush Layer
Add Brush Layer
Editing Result
Result
Editing Prompt: Make the rabbit's ears bigger.

3. Color Layer

Use the color brush to overlay semi-transparent color to change the color of specific regions. The result will follow the user's color map.

Layered Composition
Background Layer
Color Layer
Editing Result
Result
Editing Prompt: Turn the dolphin into pink.
Layered Composition
Background Layer
Color Layer
Editing Result
Result
Editing Prompt: Turn the woman's dress into yellow.

4. Spatial Layer (Local Edit)

Use the local edit brush to paint an area for local editing. The edit will occur within the specified region.

Layered Composition
Background Layer
Fill Mask Layer
Editing Result
Result
Editing Prompt: Add a white horse in the forest.
Layered Composition
Background Layer
Fill Mask Layer
Editing Result
Result
Editing Prompt: Let the man wear white formal suit.

5. Spatial Layer (Removal)

Use the subtract brush to paint over an object for precise removal.

Layered Composition
Background Layer
Subtract Brush Layer
Editing Result
Result
Editing Prompt: Remove the instance.
Layered Composition
Background Layer
Subtract Brush Layer
Editing Result
Result
Editing Prompt: Remove the instance.