ControlNets are a technique for constraining the output of image diffusion so that they don't generate images that are too different from a reference image. I recently created some QR codes for Asheville Digital Nomads, where I am a event host.

Instructions

based on https://www.youtube.com/watch?v=nFfHEmNu0Zk

1. create a QR code using a generator, e.g. https://qrcode.antfu.me/#generator
2. download stable-diffusion-webui
   1. download StableDiffusion model to the webui/models/stablediffusion folder. I used "I cant believe it's not a photograph" and "Cheese Daddy's Landscapes mix". Protip: do not under any circumstances disable the NSFW filter. There are things on the internet that you cannot unsee.
3. text2img Settings:
   • width, height to 768
   • increase sampling steps to 60
   • Sampling: DPM++ 2M Karras
   • Hires fix: on
4. ControlNet settings (requires ControlNet plugin):
   1. for first ControlNet (unit 0), click enable
      1. input QR code image
      2. Control type: all
      3. preprocessor: Invert (from white bg)
      4. model: control_v1p_sd15_qrcode_monster.safetensors from https://huggingface.co/monster-labs/control_v1p_sd15_qrcode_monster/tree/main
      5. Increase control weight to 1-2 (used 1.25)
      6. Starting control step: 0 - 0.2 (i.e. wait until 20% of image generation to start applying control)
         1. I needed to delay to 0.2 so that a person would show up. Then, I compensated by setting end step to 1.0
      7. End Step: 0.7-1.0.
      8. Resize mode: resize and fill
5. Lora (optional): you can tweak a base model using LoRA.

Example prompt:
> score_9, score_8_up, score_7_up, score_6_up, source anime, scenery, {{natural language description}}, {{tags}}](<score_9, score_8_up, score_7_up, score_6_up, Miyazaki, scenery, (Appalachian mountains:1.2), sunset, fall foliage colors %3Clora:Detailed_Anime_Style_Backgrounds_for_PonyXL:0.6%3E <lora:naturepron_nolabels:1>

Negative prompt:
> score_4, score_5, score_6, monochrome, (censor), render, simple background, white background, text, watermark

Settings used for QR Codes

• SD1.5 EMA, prompt: Miyazaki, scenery, (Appalachian mountains:1.2), the sun, sunset rays, (fall foliage colors:1.2), cliff rocks, clouds over horizon, photograph <lora:Detailed_Anime_Style_Backgrounds_for_PonyXL:0.6> <lora:naturepron_nolabels:0.5>. Negative prompt = monochrome, (censor), render, simple background, white background, text, watermark, buildings, DPM++ 2M Karras, 60 steps. ControlNet settings: control_v1p_sd15_qrcode_monster.safetensors, no ControlNet preprocessor, LoRAs: NaturePron, control weight = 1.25, start step = 20%, end step= 100%
• I ended up having better luck with simpler prompts, e.g. "nebula, stars". No LoRA. higher control weight (1.5). end step ending at 90%. the QR codes are less artistic but they all scan.