Cosmos Data Factory (CDF) AnomalyGen

CDF AnomalyGen is a diffusion-based pipeline for Synthetic Data Generation (SDG) of anomaly images in a few-shot scenario. It adapts the base Cosmos generator into an anomaly-generation model for your defect types: given a clean product image and a mask marking where a defect should go, the fine-tuned model paints a realistic defect of the requested type into that region. Because only a small parameter subset is trained on top of a frozen base network, a handful of real examples per defect is enough to fine-tune, runs are cheap, and the resulting checkpoints are tiny — a scalable source of synthetic anomaly data for training downstream defect detectors and segmenters.

High-level flow: the texture workflow runs end to end in seven stages, each feeding the next.

1. Dataset preparation — turn an upstream source into the anomaly_image/ + mask/ + clean_image/ layout the trainer expects.
2. Mask generation — build the testcase.jsonl of placed defect masks that fine-tuning and generation consume. The default is Agentic Mask Generation (AMG): a self-hostable propose→judge→refine VLM loop that authors a judged mask per clean image. The classic Auto Mask Placement (AMP) route places training submasks by declared routing and remains available.
3. Fine-tuning — train the model on the anomaly_image/ + mask/ pairs.
4. Generation (SDG) — run the fine-tuned checkpoint over a batch of testcases to synthesize defect images.
5. Evaluation — score the batch against real references with the KPI.
6. Quality refinement — search per-sample generation parameters over repeated draw → generate → evaluate rounds, then keep the best variant of each sample. NN is the default objective; optional aq_rank cumulatively re-ranks NN, completeness, precision, and boundary IoU over the observed candidate pool. See the refinement tutorial. Optionally filter the refined batch into a clean keep/ set.
7. Pseudo-labeling — turn the refined batch into a labeled dataset: a COCO instance dataset, a per-class classification layout, and optional captions.

Zooming into Generation (SDG): once the per-defect-type LoRA is trained, generation is conditioned on two things at once. The text condition turns a prompt into tokens for the VLM. The vision condition takes a clean image plus a target mask, replaces the masked region with noise, and encodes it into a source latent. The Diffusion model — frozen, with a small per-defect-type LoRA on top — denoises the target latent from both, and the VAE decoder turns it back into an image with the defect painted into the masked region.

Requirements:

• NVIDIA driver supporting CUDA 13.2, and a visible GPU at build time — some of the compiled CUDA extensions autodetect the target arch, and silently build without GPU kernels if they cannot see one.
• A host C/C++ toolchain (gcc/g++) and git — several dependencies are compiled from source or installed straight from a git URL.

git clone https://github.com/NVIDIA/cdf-anomalygen.git
cd cdf-anomalygen

Install uv as the virtual environment manager.

curl -LsSf https://astral.sh/uv/install.sh | sh

Build the venv:

# MAX_JOBS: parallel compile jobs (default 16)
# NVCC_THREADS: threads per nvcc invocation (default 1)
bash scripts/env_setup.sh [MAX_JOBS] [NVCC_THREADS]

Then authenticate to Hugging Face, fetch the model checkpoints, and confirm the environment is sound. The checkpoints are gated, so accept both the nvidia/Cosmos3-Nano and nvidia/Cosmos3-Edge licenses on Hugging Face with the same account before downloading:

source .venv/bin/activate
hf auth login                   # or: export HF_TOKEN=<your-hf-token>
bash scripts/download_checkpoints.sh
scripts/preflight_env_ckpt.sh   # GPU, Python, CUDA/torch, deps, HF auth, checkpoints

That fetches every checkpoint (≈ 140 GB under checkpoints/, ≈ 60 GB of it the Gemma VLM for self-hosted mask generation), which is what a full checkout needs. A run that only ever uses some of them can name the groups instead — scripts/download_checkpoints.sh --list prints them, and scripts/download_checkpoints.sh base-edge wan-vae edge-processor guardrail is what an edge generation run opens:

bash scripts/download_checkpoints.sh --list

A prebuilt image is published publicly on NGC — no registry login required:

docker pull nvcr.io/nvidia/cdf-anomalygen:1.2.0

The tag tracks this repo's VERSION. If the pull fails (manifest unknown, not found or pull access denied) or the tag is not yet on the tag list, that version has not been released yet — build the image from this checkout rather than pulling an older tag.

Building — including the develop / product / air-gapped targets — is covered in docker/README.md.

An end-to-end walkthrough of the pipeline above — from a clean environment to a filtered, pseudo-labeled set of synthetic anomalies — lives under docs/tutorial/texture/. Start at 1 · Overview & Environment Setup, which explains what you will build and sets up the environment (venv or Docker, checkpoints, smoke test). The pages:

1. Overview & Environment Setup — install and smoke test.
2. Dataset Preparation — build the dataset layout.
3. Agentic Mask Generation — author judged defect masks to build the testcases (or classic Auto Mask Placement).
4. Fine-tuning — train the per-defect-type LoRAs.
5. Generation — generate synthetic anomalies from the fine-tuned checkpoint.
6. Evaluation, Refinement & Pseudo-labeling — evaluate, refine, filter & pseudo-label.
7. Agentic End-to-End Flow — run all seven stages from one prompt with the agent skill.

The pipeline ships as agent skills so an AI coding agent can drive it end to end, with the per-stage gates and failure modes encoded. Ask for it in plain language and pass the run's parameters as key=value lines. The agent asks for anything required that you leave out, rather than guessing:

Use AnomalyGen skill with
name=phone_screen
mode=full
dataset_dir=datasets/phone_screen
num_sdg=100
max_iter=15000
validation_iter=1000
save_iter=1000

That runs all seven stages above; 7-agentic-end-to-end-flow.md walks through it. For a single stage, switch mode — e.g. mode=generation_only with a checkpoint= and recipe= from an earlier run generates more images without retraining.

Contributions are welcome. All commits must be signed off under the Developer Certificate of Origin — see CONTRIBUTING.md for sign-off instructions and the full DCO text.

NOTICE AND DISCLAIMER: This software automatically retrieves, accesses or interacts with external materials. Those retrieved materials are not distributed with this software and are governed solely by separate terms, conditions and licenses. You are solely responsible for finding, reviewing and complying with all applicable terms, conditions, and licenses, and for verifying the security, integrity and suitability of any retrieved materials for your specific use case. This software is provided "AS IS", without warranty of any kind. The author makes no representations or warranties regarding any retrieved materials, and assumes no liability for any losses, damages, liabilities or legal consequences from your use or inability to use this software or any retrieved materials. Use this software and the retrieved materials at your own risk.