stable-diffusion.cpp/docs/minimax_h3.md

MiniMax-H3

MiniMax-H3 jointly generates video and stereo audio with a packed diffusion transformer. The implementation supports text-to-audio-video (T2VA), optional first-frame conditioning (I2VA), first/last-frame conditioning (FL2VA), and image/video/audio reference conditioning (Ref2VA).

Model files

Pass the four MiniMax-H3 components separately:

• --diffusion-model: MiniMax-H3 diffusion transformer
• --vae: MiniMax-H3 video VAE
• --audio-vae: MiniMax-H3 audio VAE
• --llm: the MiniMax-H3 Qwen3-VL-32B text encoder checkpoint

The text encoder must be the MiniMax-H3 variant: Qwen3-VL-32B truncated to 50 language layers and exported without the final language-model normalization. Its Qwen3-VL vision tower, including the three DeepStack mergers, must also be present. If the vision tower is stored separately, pass it with --llm_vision.

Both the original time-embedder DiT and the smaller AdaLN curve-table variant are detected from their weights.

Download weights

• Download minimax_h3_fl2va/minimax_h3_ref2va
  • safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models
  • gguf: https://huggingface.co/leejet/MiniMax-H3-GGUF/tree/main
• Download qwen3vl_32b_minimax_h3
  • safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders
  • gguf: https://huggingface.co/leejet/MiniMax-H3-GGUF/tree/main
• Download vae
  • safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/vae
• Download audio vae
  • safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/vae

Text-to-audio-video

.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model  ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "A cute American Shorthair silver tabby kitten surfs on a tropical ocean wave, riding a white surfboard with the clear text 'sd.cpp' on it. Cinematic tracking shot, realistic water, bright sunlight, smooth motion, and consistent character appearance. Add upbeat tropical surf-rock background music with cheerful drums and guitar, synchronized with the kitten’s energetic surfing." --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56

Omitting --audio-vae still runs the joint diffusion model but produces video without a decoded audio track.

First/last-frame conditioning

Add --init-img for I2VA, or both --init-img and --end-img for FL2VA:

.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model  ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "a lovely cat" -i ..\assets\ernie_image\turbo_example.png --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56

Reference-to-audio-video conditioning

Ref2VA accepts any combination of reference images, reference videos, paired video soundtracks, and standalone audio references:

.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model  ..\models\diffusion_models\minimax_h3_ref2va_pruned-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "Use the cat from <Picture 1> as the main character. Keep the cat’s appearance, fur color, facial features, and identity consistent with the reference image. Create a 2-second cinematic video: start with an extreme close-up shot of the cat’s face, focusing on its cute expression and detailed fur texture. The camera slowly rotates around the cat’s head, creating a dynamic reveal. Then smoothly pull back and zoom out to reveal the full scene: the cat is standing confidently on a surfboard, riding ocean waves. Water splashes around the board, sea breeze gently moves the cat’s fur, and the cat maintains a cute and fearless expression while surfing. Smooth camera movement, cinematic orbit shot, seamless zoom-out transition, low-angle wide shot, realistic ocean environment, golden sunlight, dynamic waves, high-quality realistic style, natural motion, no distortion, keep the cat’s identity unchanged." -r ..\assets\ernie_image\turbo_example.png --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56

--ref-image, --ref-video, and --ref-audio can each be repeated. A reference video is a directory of image frames sorted lexicographically and is treated as 24 fps. Repeated --ref-video-audio WAV files are paired by index with repeated --ref-video inputs. WAV PCM (8/16/24/32-bit) and 32/64-bit floating-point samples are accepted; audio is converted to stereo 32 kHz by the pipeline.

Reference inputs are presented to Qwen3-VL in image, video, then audio order. Videos are sampled at 2 fps for the Qwen presentation while their full 24 fps latents condition the diffusion transformer. Paired video and audio references share the same timeline. Ref2VA cannot be combined with --init-img or --end-img in one request.

Reference images keep their aspect ratio and are only downscaled when they exceed the configured limit; neither mode scales a reference up. The limit is selected by ref_image_size. The default, match, uses the pixel area of the requested generation canvas. Pass --ref-image-args "ref_image_size=max" (or ref_image_size in sd_vid_gen_params_t.ref_image_args) to use a 2048 px short edge instead, which preserves more identity detail at a higher sampling cost: reference latents are attended at every sampling step.

Whichever limit applies, each axis is then snapped to the nearest 32-pixel multiple. An axis of at least 16 pixels can therefore grow by up to 16 pixels past the scaled size (a 48-pixel axis becomes 64), and references are never narrower than 32 pixels. resize_before_vae=false is not supported here; MiniMax-H3 keeps references on the 32-pixel grid and reports a warning.

The C API exposes the same inputs through ref_images, ref_videos, and ref_audios in sd_vid_gen_params_t. Each sd_ref_video_t supplies its own frame rate and optional soundtrack; non-24-fps inputs are resampled internally.

Latent hires second pass (two-stage sampling)

MiniMax-H3 supports --hires with two interchangeable second-stage upscalers:

• an interpolating latent upscaler (Latent, Latent (nearest), Latent (bicubic), ...): after the first pass finishes, the packed AV latent is resampled in latent space to --hires-width x --hires-height and refined by a second sampling pass;
• a model-backed upscaler (--hires-upscaler <name> + --hires-upscalers-dir): a H3-shaped learned latent upscaler checkpoint runs between the passes, like the LTX spatial upsampler. This is the sd.cpp equivalent of the two-sampler workflows used for H3 in node-based frontends (e.g. the community 3D-conv latent upscaler from LBH-123-AI/Minimax_h3_latent_Upscaler).

Draft at a low megapixel count (e.g. 0.4 MP) and refine at a higher one (e.g. 0.8 MP) instead of paying for the full resolution in every step:

.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model  ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "a lovely cat" --cfg-scale 1.0 -v -W 864 -H 480 --diffusion-fa --offload-to-cpu --rng cpu --video-frames 56 --hires --hires-upscaler "Latent (bicubic)" --hires-width 1216 --hires-height 672 --hires-steps 4 --hires-denoising-strength 0.4

or with the learned upscaler:

.\bin\Release\sd-cli.exe -M vid_gen ... --hires --hires-upscalers-dir ..\models\latent_upscale_models --hires-upscaler h3-latent-upscaler-3d --hires-width 1216 --hires-height 672 --hires-steps 4 --hires-denoising-strength 0.4

Notes for this mode:

• The learned upscaler expects a second normalization of the diffusion latent, matching its training convention. The runner applies and reverses this extra transform internally; callers should pass the usual H3 diffusion latents.
• The frame count is never changed by the second stage, so the packed audio latent length stays valid and the generated soundtrack is preserved.
• Width and height still have to be multiples of 32; the hires target is aligned like any other resolution. The learned upscaler additionally wants an even latent size on both axes, which 32-pixel targets provide.
• --hires-sigmas can replace --hires-denoising-strength when the second stage should resume from an explicit sigma schedule, which is how most two-stage H3 workflows split the sampler.
• Keyframe conditioning (--init-img / --end-img) is not available together with hires yet: those latents are encoded at the first stage resolution and would have to be re-encoded for the hires canvas. Use T2VA or Ref2VA.
• The second stage runs at the larger latent, so it is the memory-hungry half; combine it with quantization, --vae-tiling and --offload-to-cpu on small cards.
• The learned upscaler is scale-conditional: the target-to-source ratio is fed into the network as an embedding, 1.2-2.0x is the trained domain, and downscaling is rejected.

Shape and runtime notes

• Width and height are aligned upward to a multiple of 32.
• Frame count is aligned upward to the 17k + 5 grid, with a minimum of 5.
• MiniMax-H3 runs at 24 fps; another requested value is overridden.
• Attention head grouping is controlled by --model-args minimax_h3_attn_chunk_budget=2, in GiB (default: 2). The budget estimates the size of a single FP32 Q/K tensor per group: head_dim * tokens * batch * 4 bytes * heads_per_group. The group size adapts to the sequence length, with a minimum of one head; inputs that fit use all heads together. Set 0 to disable grouping, or a smaller budget such as 0.25 to reduce peak activation memory for long sequences at the cost of extra launches and concatenation. Each head still attends to the full sequence. This is a grouping target, not a cap on total attention memory or VRAM usage; other intermediates, backend workspace, and weights require additional memory. --max-vram and --offload-to-cpu control weight residency, but the active block's working memory must also fit within the VRAM budget.
• The default video flow shift is 12. The audio stream is mapped internally to its shift of 3, so the regular samplers can operate on the packed AV latent.