mirror of
https://github.com/leejet/stable-diffusion.cpp.git
synced 2026-11-11 03:24:30 +00:00
172 lines
11 KiB
Markdown
172 lines
11 KiB
Markdown
# MiniMax-H3
|
||
|
||
MiniMax-H3 jointly generates video and stereo audio with a packed diffusion
|
||
transformer. The implementation supports text-to-audio-video (T2VA), optional
|
||
first-frame conditioning (I2VA), first/last-frame conditioning (FL2VA), and
|
||
image/video/audio reference conditioning (Ref2VA).
|
||
|
||
## Model files
|
||
|
||
Pass the four MiniMax-H3 components separately:
|
||
|
||
- `--diffusion-model`: MiniMax-H3 diffusion transformer
|
||
- `--vae`: MiniMax-H3 video VAE
|
||
- `--audio-vae`: MiniMax-H3 audio VAE
|
||
- `--llm`: the MiniMax-H3 Qwen3-VL-32B text encoder checkpoint
|
||
|
||
The text encoder must be the MiniMax-H3 variant: Qwen3-VL-32B truncated to 50
|
||
language layers and exported without the final language-model normalization.
|
||
Its Qwen3-VL vision tower, including the three DeepStack mergers, must also be
|
||
present. If the vision tower is stored separately, pass it with `--llm_vision`.
|
||
|
||
Both the original time-embedder DiT and the smaller AdaLN curve-table variant
|
||
are detected from their weights.
|
||
|
||
### Download weights
|
||
|
||
- Download minimax_h3_fl2va/minimax_h3_ref2va
|
||
- safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models
|
||
- gguf: https://huggingface.co/leejet/MiniMax-H3-GGUF/tree/main
|
||
- Download qwen3vl_32b_minimax_h3
|
||
- safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders
|
||
- gguf: https://huggingface.co/leejet/MiniMax-H3-GGUF/tree/main
|
||
- Download vae
|
||
- safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/vae
|
||
- Download audio vae
|
||
- safetensors: https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/vae
|
||
|
||
## Text-to-audio-video
|
||
|
||
```sh
|
||
.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "A cute American Shorthair silver tabby kitten surfs on a tropical ocean wave, riding a white surfboard with the clear text 'sd.cpp' on it. Cinematic tracking shot, realistic water, bright sunlight, smooth motion, and consistent character appearance. Add upbeat tropical surf-rock background music with cheerful drums and guitar, synchronized with the kitten’s energetic surfing." --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56
|
||
```
|
||
|
||
<video src=../assets/minimax-h3/t2av.mp4 controls="controls" muted="muted" type="video/mp4"></video>
|
||
|
||
Omitting `--audio-vae` still runs the joint diffusion model but produces video without a
|
||
decoded audio track.
|
||
|
||
## First/last-frame conditioning
|
||
|
||
Add `--init-img` for I2VA, or both `--init-img` and `--end-img` for FL2VA:
|
||
|
||
```sh
|
||
.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "a lovely cat" -i ..\assets\ernie_image\turbo_example.png --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56
|
||
```
|
||
|
||
<video src=../assets/minimax-h3/i2av.mp4 controls="controls" muted="muted" type="video/mp4"></video>
|
||
|
||
## Reference-to-audio-video conditioning
|
||
|
||
Ref2VA accepts any combination of reference images, reference videos, paired
|
||
video soundtracks, and standalone audio references:
|
||
|
||
```sh
|
||
.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model ..\models\diffusion_models\minimax_h3_ref2va_pruned-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "Use the cat from <Picture 1> as the main character. Keep the cat’s appearance, fur color, facial features, and identity consistent with the reference image. Create a 2-second cinematic video: start with an extreme close-up shot of the cat’s face, focusing on its cute expression and detailed fur texture. The camera slowly rotates around the cat’s head, creating a dynamic reveal. Then smoothly pull back and zoom out to reveal the full scene: the cat is standing confidently on a surfboard, riding ocean waves. Water splashes around the board, sea breeze gently moves the cat’s fur, and the cat maintains a cute and fearless expression while surfing. Smooth camera movement, cinematic orbit shot, seamless zoom-out transition, low-angle wide shot, realistic ocean environment, golden sunlight, dynamic waves, high-quality realistic style, natural motion, no distortion, keep the cat’s identity unchanged." -r ..\assets\ernie_image\turbo_example.png --cfg-scale 1.0 -v -W 864 -H 480 --offload-to-cpu --rng cpu --fps 24 --video-frames 56
|
||
```
|
||
|
||
<video src=../assets/minimax-h3/r2av.mp4 controls="controls" muted="muted" type="video/mp4"></video>
|
||
|
||
`--ref-image`, `--ref-video`, and `--ref-audio` can each be repeated. A
|
||
reference video is a directory of image frames sorted lexicographically and is
|
||
treated as 24 fps. Repeated `--ref-video-audio` WAV files are paired by index
|
||
with repeated `--ref-video` inputs. WAV PCM (8/16/24/32-bit) and 32/64-bit
|
||
floating-point samples are accepted; audio is converted to stereo 32 kHz by the
|
||
pipeline.
|
||
|
||
Reference inputs are presented to Qwen3-VL in image, video, then audio order.
|
||
Videos are sampled at 2 fps for the Qwen presentation while their full 24 fps
|
||
latents condition the diffusion transformer. Paired video and audio references
|
||
share the same timeline. Ref2VA cannot be combined with `--init-img` or
|
||
`--end-img` in one request.
|
||
|
||
Reference images keep their aspect ratio and are only downscaled when they
|
||
exceed the configured limit; neither mode scales a reference up. The limit is
|
||
selected by `ref_image_size`. The default, `match`, uses the pixel area of the
|
||
requested generation canvas. Pass `--ref-image-args "ref_image_size=max"` (or
|
||
`ref_image_size` in `sd_vid_gen_params_t.ref_image_args`) to use a 2048 px short
|
||
edge instead, which preserves more identity detail at a higher sampling cost:
|
||
reference latents are attended at every sampling step.
|
||
|
||
Whichever limit applies, each axis is then snapped to the nearest 32-pixel
|
||
multiple. An axis of at least 16 pixels can therefore grow by up to 16 pixels
|
||
past the scaled size (a 48-pixel axis becomes 64), and references are never
|
||
narrower than 32 pixels.
|
||
`resize_before_vae=false` is not supported here; MiniMax-H3 keeps references on
|
||
the 32-pixel grid and reports a warning.
|
||
|
||
The C API exposes the same inputs through `ref_images`, `ref_videos`, and
|
||
`ref_audios` in `sd_vid_gen_params_t`. Each `sd_ref_video_t` supplies its own
|
||
frame rate and optional soundtrack; non-24-fps inputs are resampled internally.
|
||
|
||
## Latent hires second pass (two-stage sampling)
|
||
|
||
MiniMax-H3 supports `--hires` with two interchangeable second-stage upscalers:
|
||
|
||
- an interpolating latent upscaler (`Latent`, `Latent (nearest)`,
|
||
`Latent (bicubic)`, ...): after the first pass finishes, the packed AV latent
|
||
is resampled in latent space to `--hires-width` x `--hires-height` and
|
||
refined by a second sampling pass;
|
||
- a model-backed upscaler (`--hires-upscaler <name>` + `--hires-upscalers-dir`):
|
||
a H3-shaped learned latent upscaler checkpoint runs between the passes,
|
||
like the LTX spatial upsampler. This is the sd.cpp equivalent of the
|
||
two-sampler workflows used for H3 in node-based frontends (e.g. the
|
||
community 3D-conv latent upscaler from
|
||
[LBH-123-AI/Minimax_h3_latent_Upscaler](https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler)).
|
||
|
||
Draft at a low megapixel count (e.g. 0.4 MP) and refine at a higher one
|
||
(e.g. 0.8 MP) instead of paying for the full resolution in every step:
|
||
|
||
```sh
|
||
.\bin\Release\sd-cli.exe -M vid_gen --diffusion-model ..\models\diffusion_models\minimax_h3_fl2va-Q4_K_M.gguf --vae ..\models\vae\minimax_h3_video_vae_fp16.safetensors --audio-vae ..\models\vae\minimax_h3_audio_vae_fp32.safetensors --llm ..\models\text_encoders\qwen3vl_32b_minimax_h3-Q4_K_M.gguf -p "a lovely cat" --cfg-scale 1.0 -v -W 864 -H 480 --diffusion-fa --offload-to-cpu --rng cpu --video-frames 56 --hires --hires-upscaler "Latent (bicubic)" --hires-width 1216 --hires-height 672 --hires-steps 4 --hires-denoising-strength 0.4
|
||
```
|
||
|
||
or with the learned upscaler:
|
||
|
||
```sh
|
||
.\bin\Release\sd-cli.exe -M vid_gen ... --hires --hires-upscalers-dir ..\models\latent_upscale_models --hires-upscaler h3-latent-upscaler-3d --hires-width 1216 --hires-height 672 --hires-steps 4 --hires-denoising-strength 0.4
|
||
```
|
||
|
||
Notes for this mode:
|
||
|
||
- The learned upscaler expects a second normalization of the diffusion latent,
|
||
matching its training convention. The runner applies and reverses this extra
|
||
transform internally; callers should pass the usual H3 diffusion latents.
|
||
- The frame count is never changed by the second stage, so the packed audio
|
||
latent length stays valid and the generated soundtrack is preserved.
|
||
- Width and height still have to be multiples of 32; the hires target is
|
||
aligned like any other resolution. The learned upscaler additionally wants
|
||
an even latent size on both axes, which 32-pixel targets provide.
|
||
- `--hires-sigmas` can replace `--hires-denoising-strength` when the second
|
||
stage should resume from an explicit sigma schedule, which is how most
|
||
two-stage H3 workflows split the sampler.
|
||
- Keyframe conditioning (`--init-img` / `--end-img`) is not available together
|
||
with hires yet: those latents are encoded at the first stage resolution and
|
||
would have to be re-encoded for the hires canvas. Use T2VA or Ref2VA.
|
||
- The second stage runs at the larger latent, so it is the memory-hungry half;
|
||
combine it with quantization, `--vae-tiling` and `--offload-to-cpu` on
|
||
small cards.
|
||
- The learned upscaler is scale-conditional: the target-to-source ratio is
|
||
fed into the network as an embedding, 1.2-2.0x is the trained domain, and
|
||
downscaling is rejected.
|
||
|
||
## Shape and runtime notes
|
||
|
||
- Width and height are aligned upward to a multiple of 32.
|
||
- Frame count is aligned upward to the `17k + 5` grid, with a minimum of 5.
|
||
- MiniMax-H3 runs at 24 fps; another requested value is overridden.
|
||
- Attention head grouping is controlled by
|
||
`--model-args minimax_h3_attn_chunk_budget=2`, in GiB (default: `2`).
|
||
The budget estimates the size of a single FP32 Q/K tensor per group:
|
||
`head_dim * tokens * batch * 4 bytes * heads_per_group`. The group size adapts
|
||
to the sequence length, with a minimum of one head; inputs that fit use all
|
||
heads together. Set `0` to disable grouping, or a smaller budget such as
|
||
`0.25` to reduce peak activation memory for long sequences at the cost of
|
||
extra launches and concatenation. Each head still attends to the full sequence.
|
||
This is a grouping target, not a cap on total attention memory or VRAM usage;
|
||
other intermediates, backend workspace, and weights require additional memory.
|
||
`--max-vram` and `--offload-to-cpu` control weight residency, but the active
|
||
block's working memory must also fit within the VRAM budget.
|
||
- The default video flow shift is 12. The audio stream is mapped internally to
|
||
its shift of 3, so the regular samplers can operate on the packed AV latent.
|