stable-diffusion.cpp/docs/minimax_h3.md

MiniMax-H3

MiniMax-H3 jointly generates video and stereo audio with a packed diffusion transformer. The implementation supports text-to-audio-video (T2VA), optional first-frame conditioning (I2VA), first/last-frame conditioning (FL2VA), and image/video/audio reference conditioning (Ref2VA).

Model files

Pass the four MiniMax-H3 components separately:

• --diffusion-model: MiniMax-H3 diffusion transformer
• --vae: MiniMax-H3 video VAE
• --audio-vae: MiniMax-H3 audio VAE
• --llm: the MiniMax-H3 Qwen3-VL-32B text encoder checkpoint

The text encoder must be the MiniMax-H3 variant: Qwen3-VL-32B truncated to 50 language layers and exported without the final language-model normalization. Its Qwen3-VL vision tower, including the three DeepStack mergers, must also be present. If the vision tower is stored separately, pass it with --llm_vision.

Both the original time-embedder DiT and the smaller AdaLN curve-table variant are detected from their weights.

Download weights

• Download minimax_h3_fl2va/minimax_h3_ref2va
  • safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models
  • gguf: https://huggingface.co/leejet/MiniMax-H3-GGUF/tree/main
• Download qwen3vl_32b_minimax_h3
  • safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders
  • gguf: https://huggingface.co/leejet/MiniMax-H3-GGUF/tree/main
• Download vae
  • safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/vae
• Download audio vae
  • safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/vae

Text-to-audio-video

.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model  ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "A cute American Shorthair silver tabby kitten surfs on a tropical ocean wave, riding a white surfboard with the clear text 'sd.cpp' on it. Cinematic tracking shot, realistic water, bright sunlight, smooth motion, and consistent character appearance. Add upbeat tropical surf-rock background music with cheerful drums and guitar, synchronized with the kitten’s energetic surfing." --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56

Omitting --audio-vae still runs the joint diffusion model but produces video without a decoded audio track.

First/last-frame conditioning

Add --init-img for I2VA, or both --init-img and --end-img for FL2VA:

.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model  ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "a lovely cat" -i ..\assets\ernie_image\turbo_example.png --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56

Reference-to-audio-video conditioning

Ref2VA accepts any combination of reference images, reference videos, paired video soundtracks, and standalone audio references:

.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model  ..\models\diffusion_models\minimax_h3_ref2va_pruned-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "Use the cat from <Picture 1> as the main character. Keep the cat’s appearance, fur color, facial features, and identity consistent with the reference image. Create a 2-second cinematic video: start with an extreme close-up shot of the cat’s face, focusing on its cute expression and detailed fur texture. The camera slowly rotates around the cat’s head, creating a dynamic reveal. Then smoothly pull back and zoom out to reveal the full scene: the cat is standing confidently on a surfboard, riding ocean waves. Water splashes around the board, sea breeze gently moves the cat’s fur, and the cat maintains a cute and fearless expression while surfing. Smooth camera movement, cinematic orbit shot, seamless zoom-out transition, low-angle wide shot, realistic ocean environment, golden sunlight, dynamic waves, high-quality realistic style, natural motion, no distortion, keep the cat’s identity unchanged." -r ..\assets\ernie_image\turbo_example.png --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56

--ref-image, --ref-video, and --ref-audio can each be repeated. A reference video is a directory of image frames sorted lexicographically and is treated as 24 fps. Repeated --ref-video-audio WAV files are paired by index with repeated --ref-video inputs. WAV PCM (8/16/24/32-bit) and 32/64-bit floating-point samples are accepted; audio is converted to stereo 32 kHz by the pipeline.

Reference inputs are presented to Qwen3-VL in image, video, then audio order. Videos are sampled at 2 fps for the Qwen presentation while their full 24 fps latents condition the diffusion transformer. Paired video and audio references share the same timeline. Ref2VA cannot be combined with --init-img or --end-img in one request.

Reference images keep their aspect ratio and are only downscaled when they exceed the configured limit; neither mode scales a reference up. The limit is selected by ref_image_size. The default, match, uses the pixel area of the requested generation canvas. Pass --ref-image-args "ref_image_size=max" (or ref_image_size in sd_vid_gen_params_t.ref_image_args) to use a 2048 px short edge instead, which preserves more identity detail at a higher sampling cost: reference latents are attended at every sampling step.

Whichever limit applies, each axis is then snapped to the nearest 32-pixel multiple. An axis of at least 16 pixels can therefore grow by up to 16 pixels past the scaled size (a 48-pixel axis becomes 64), and references are never narrower than 32 pixels. resize_before_vae=false is not supported here; MiniMax-H3 keeps references on the 32-pixel grid and reports a warning.

The C API exposes the same inputs through ref_images, ref_videos, and ref_audios in sd_vid_gen_params_t. Each sd_ref_video_t supplies its own frame rate and optional soundtrack; non-24-fps inputs are resampled internally.

Latent hires second pass (two-stage sampling)

MiniMax-H3 supports --hires with two interchangeable second-stage upscalers:

• an interpolating latent upscaler (Latent, Latent (nearest), Latent (bicubic), ...): after the first pass finishes, the packed AV latent is resampled in latent space to --hires-width x --hires-height and refined by a second sampling pass;
• a model-backed upscaler (--hires-upscaler <name> + --hires-upscalers-dir): a H3-shaped learned latent upscaler checkpoint runs between the passes, like the LTX spatial upsampler. This is the sd.cpp equivalent of the two-sampler workflows used for H3 in node-based frontends (e.g. the community 3D-conv latent upscaler from LBH-123-AI/Minimax_h3_latent_Upscaler).

Draft at a low megapixel count (e.g. 0.4 MP) and refine at a higher one (e.g. 0.8 MP) instead of paying for the full resolution in every step:

.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model  ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "a lovely cat" --cfg-scale 1.0 -v -W 864 -H 480 --diffusion-fa --offload-to-cpu --rng cpu --video-frames 56 --hires --hires-upscaler "Latent (bicubic)" --hires-width 1216 --hires-height 672 --hires-steps 4 --hires-denoising-strength 0.4

or with the learned upscaler:

.\bin\Release\sd-cli.exe -M vid_gen ... --hires --hires-upscalers-dir ..\models\latent_upscale_models --hires-upscaler h3-latent-upscaler-3d --hires-width 1216 --hires-height 672 --hires-steps 4 --hires-denoising-strength 0.4

Notes for this mode:

• The learned upscaler expects a second normalization of the diffusion latent, matching its training convention. The runner applies and reverses this extra transform internally; callers should pass the usual H3 diffusion latents.
• The frame count is never changed by the second stage, so the packed audio latent length stays valid and the generated soundtrack is preserved.
• Width and height still have to be multiples of 32; the hires target is aligned like any other resolution. The learned upscaler additionally wants an even latent size on both axes, which 32-pixel targets provide.
• --hires-sigmas can replace --hires-denoising-strength when the second stage should resume from an explicit sigma schedule, which is how most two-stage H3 workflows split the sampler.
• Keyframe conditioning (--init-img / --end-img) is not available together with hires yet: those latents are encoded at the first stage resolution and would have to be re-encoded for the hires canvas. Use T2VA or Ref2VA.
• The second stage runs at the larger latent, so it is the memory-hungry half; combine it with quantization, --vae-tiling and --offload-to-cpu on small cards.
• The learned upscaler is scale-conditional: the target-to-source ratio is fed into the network as an embedding, 1.2-2.0x is the trained domain, and downscaling is rejected.

Shape and runtime notes

• Width and height are aligned upward to a multiple of 32.
• Frame count is aligned upward to the 17k + 5 grid, with a minimum of 5.
• MiniMax-H3 runs at 24 fps; another requested value is overridden.
• The default video flow shift is 12. The audio stream is mapped internally to its shift of 3, so the regular samplers can operate on the packed AV latent.