# ControlNet Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/controlnet
> Markdown URL: https://aitinkerers.org/technologies/controlnet.md
> Technology record last updated: 2026-02-26T21:35:41Z
> Generated: 2026-08-24T08:41:13Z

ControlNet is a neural network architecture that adds precise spatial conditioning (e.g., Canny edges, OpenPose keypoints) to large, pretrained text-to-image diffusion models like Stable Diffusion.

ControlNet is a game-changer for diffusion models, providing fine-grained control over image composition. It works by creating two copies of the model’s weights: a 'locked' copy to preserve the original Stable Diffusion capabilities and a 'trainable' copy for new conditions. This connection uses 'zero convolutions' to ensure no harmful noise affects the finetuning. This architecture allows training on small datasets (under 50k images) for tasks like pose-to-image or depth-to-image, efficiently steering the output with specific inputs like OpenPose keypoints or Canny edge maps.

- Official technology site: https://github.com/lllyasviel/ControlNet
- Public AI Tinkerers demos and talks: 6
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Hybrid AI/VFX workflow](https://montreal.aitinkerers.org/talks/rsvp_dME8XrE7GlI)

Hyprid VFX / AI pipeline. Generative AI used in a controlled way to enchance film footage. The "Hockeyville" project is a hybrid VFX production for a KRAFT commercial that utilizes local Generative AI to transform live-action urban footage into a frozen landscape. https://vimeo.com/1173898332

- Event context: AI Tinkerers Montreal - March Demo Night — 2026-03-24 — Montreal
- Public talk page: https://montreal.aitinkerers.org/talks/rsvp_dME8XrE7GlI

### [StableGen - diffusion powered texturing within Blender](https://prague.aitinkerers.org/talks/rsvp_5HqwtbOGNaU)

This will be a 5-minute live demonstration of StableGen, a Blender plugin designed for AI-driven 3D texturing. I will begin with a standard untextured 3D model in Blender and walk through a typical texturing workflow. First, I'll set up the camera viewpoints and assign a text prompt for the desired texture. I will then initiate the generation process, which runs asynchronously in the background. While the plugin is working, I will show the key artifacts and workflows it manages, including the exported ControlNet images (like depth maps) used for geometric guidance and the dynamically generated ComfyUI workflow graph sent to the backend. I will then show the final output images generated for each viewpoint. To conclude, I will show the final, fully-textured 3D model in the Blender viewport and point to the complex material node tree that StableGen automatically built to seamlessly blend all the individual textures together. I could also make it interactive by having the audience choose what to generate. Note: I could also go through the code, but as the project is pretty large, I don't think 5 minutes is nearly enough to even start diving into that.

- Event context: 🍂 AI Tinkerers Prague – October 2025 Meetup — 2025-10-23 — Prague
- Public talk page: https://prague.aitinkerers.org/talks/rsvp_5HqwtbOGNaU

### [An image is worth a thousand words - How a simple sketch can help your image generation process](https://lausanne.aitinkerers.org/talks/rsvp_KzlATt5sa20)

When using AI to generate images, it is sometimes difficult to clearly explain in a prompt what we have in mind, especially in terms of spatial placement of the different elements in the image. Leveraging a webcam, a pen and a piece of paper, this demo shows how a simple sketch can drastically help the image generation process and end up with the dream image with fewer efforts.

- Event context: AI Tinkerers Lausanne April 2025 Meetup — 2025-04-01 — Lausanne
- Public talk page: https://lausanne.aitinkerers.org/talks/rsvp_KzlATt5sa20

### [Training custom controlnets for virtual staging](https://paris.aitinkerers.org/talks/rsvp_0OsZ87CsZcM)

I have been training several custom models for AI fully automated virtual staging of rooms. These allow the input of empty room images, and output a room with furniture. I would like to share several experiments (different architectures), and show the results of each.

- Event context: AI Tinkerers - Paris Meetup on January 30th — 2025-01-30 — Paris
- Public talk page: https://paris.aitinkerers.org/talks/rsvp_0OsZ87CsZcM

### [Stable Diffusion ControlNets for controlled GenAI Data Viz](https://dc.aitinkerers.org/talks/rsvp_xEa4tHjvi6I)

Will Angel will demonstrate using Generative AI for data visualization, including the use of Control Nets and Stable Diffusion to accurately encode information in generated data visualizations to get the creativity of AI image generation for accurate data visualization. The demos will include multiple image generation models such as Flux and Stable Diffusion, some examples of prompt engineering, different methods for controlling image models and using ControlNet to structure the output of image generation into usable and correct charts. Link: https://github.com/DataDrivenAngel/gen_ai_viz_demo

- Event context: AI Tinkerers - DC Metro Meetup (January 15th 2025) — 2025-01-15 — DC
- Public talk page: https://dc.aitinkerers.org/talks/rsvp_xEa4tHjvi6I

### [Diffusion based Interior Customisation for VR Home Tours](https://mumbai.aitinkerers.org/talks/rsvp_mcNaq7cpLUQ)

This project uses a Flux1.dev diffusion model to customise the look of an apartment in a VR Home Tour. The user can pick a style of their choice (Modern, Minimalist, Boho, etc) and the apartment will be styled to match the user's preferences while maintaining the same structure and dimensions. The approach uses Controlnets and LoRAs to create conditioning based on the original apartment layout and then styles the image based on a prompt.

- Event context: AI Tinkerers Mumbai Chapter - 2nd Event Incoming! — 2024-11-23 — Mumbai
- Public talk page: https://mumbai.aitinkerers.org/talks/rsvp_mcNaq7cpLUQ

## Related Technologies

- [ComfyUI](https://aitinkerers.org/technologies/comfyui) ([Markdown](https://aitinkerers.org/technologies/comfyui.md)) — 13 public demos
- [Stable Diffusion](https://aitinkerers.org/technologies/stable-diffusion) ([Markdown](https://aitinkerers.org/technologies/stable-diffusion.md)) — 32 public demos
- [Blender](https://aitinkerers.org/technologies/blender) ([Markdown](https://aitinkerers.org/technologies/blender.md)) — 4 public demos
- [DaVinci Resolve](https://aitinkerers.org/technologies/davinci-resolve) ([Markdown](https://aitinkerers.org/technologies/davinci-resolve.md)) — 1 public demo
- [Diffusion model](https://aitinkerers.org/technologies/diffusion-model) ([Markdown](https://aitinkerers.org/technologies/diffusion-model.md)) — 3 public demos
- [Diffusion models](https://aitinkerers.org/technologies/diffusion-models) ([Markdown](https://aitinkerers.org/technologies/diffusion-models.md)) — 5 public demos
- [Flux](https://aitinkerers.org/technologies/flux) ([Markdown](https://aitinkerers.org/technologies/flux.md)) — 8 public demos
- [Flux1](https://aitinkerers.org/technologies/flux1) ([Markdown](https://aitinkerers.org/technologies/flux1.md)) — 2 public demos
- [GitHub](https://aitinkerers.org/technologies/github) ([Markdown](https://aitinkerers.org/technologies/github.md)) — 73 public demos
- [Houdini](https://aitinkerers.org/technologies/houdini) ([Markdown](https://aitinkerers.org/technologies/houdini.md)) — 1 public demo
- [In-painting](https://aitinkerers.org/technologies/in-painting) ([Markdown](https://aitinkerers.org/technologies/in-painting.md)) — 1 public demo
- [IPAdapter](https://aitinkerers.org/technologies/ipadapter) ([Markdown](https://aitinkerers.org/technologies/ipadapter.md)) — 1 public demo
- [Laptop](https://aitinkerers.org/technologies/laptop) ([Markdown](https://aitinkerers.org/technologies/laptop.md)) — 4 public demos
- [LoRA](https://aitinkerers.org/technologies/lora) ([Markdown](https://aitinkerers.org/technologies/lora.md)) — 16 public demos
- [Prompt Engineering](https://aitinkerers.org/technologies/prompt-engineering) ([Markdown](https://aitinkerers.org/technologies/prompt-engineering.md)) — 28 public demos
- [Virtual Reality](https://aitinkerers.org/technologies/virtual-reality) ([Markdown](https://aitinkerers.org/technologies/virtual-reality.md)) — 1 public demo
- [Webcam](https://aitinkerers.org/technologies/webcam) ([Markdown](https://aitinkerers.org/technologies/webcam.md)) — 1 public demo