Manga Color Generator
An AI manga colorization pipeline combining Stable Diffusion, ControlNet lineart conditioning and YOLOv8 segmentation to color manga panels without destroying their line work or text.
The problem
Coloring manga by hand is slow, skilled work. Naive AI manga colorization has two failure modes: it distorts the line work — faces, anatomy and clothing edges drift from the original drawing — and it paints over speech bubbles, destroying the text.
The solution
A four-stage pipeline. A YOLOv8 segmentation model detects and masks the speech bubbles; ControlNet's LineartDetector extracts a clean lineart conditioning image; a Stable Diffusion 1.5 + ControlNet model colorizes from that lineart so the original structure is preserved; then the original bubble pixels are composited back over the colored panel with OpenCV. Coloring runs at 512px with aspect ratio preserved, while bubble compositing happens at the panel's native resolution.
The result
Iterating on the baseline made the output cleaner and faster: switching to a flat 2D anime checkpoint (AnyLoRA) for cel-style color, the UniPC scheduler to cut steps 20 → 15 at equal quality, aspect-preserving resize to stop squashing panels, and structured color-forcing prompts to kill the muddy/3D look. Models were also chosen for free-tier availability so the demo runs on a free Colab T4.
Skills used
- PyTorch
- Stable Diffusion
- ControlNet
- YOLOv8
- OpenCV
- Gradio
- Hugging Face
