DarthAffe/stable-diffusion.cpp
1
0 Fork 0
mirror of https://github.com/leejet/stable-diffusion.cpp.git synced 2026-11-11 03:24:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add SeedVR2 image and video upscaling (#2300)

Browse Source
This commit is contained in:
leejet authored and GitHub committed 2026-11-09 01:13:57 +08:00
1 parent abc6718455
commit cd5594e9f9
14 files changed
+1035 -89
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/relay_vsr.md
+31 -51
View File
@@ -1,24 +1,24 @@
# RelayVSR video super-resolution
# RelayVSR image and video super-resolution
RelayVSR uses a Wan model to sample sparse reference latents, followed by the
Flash conditional decoder to reconstruct every input frame. The model code
follows [RelayVSR](https://github.com/kopperx/RelayVSR); the loader maps
[ComfyUI repackaged weights](https://huggingface.co/Comfy-Org/RelayVSR) to the
original module names.
[RelayVSR](https://github.com/kopperx/RelayVSR) upscales images and videos by
2x or 4x with `-M vid_upscale`.
Use `-M vid_upscale` with an ordered directory of RGB images. Filenames are
sorted lexicographically. Supply the source frame rate with `--fps`. Output
keeps the input frame count. Use a video output supported by the CLI, such as
`.avi` or `.webm`, or a numbered PNG output such as `output/%04d.png`.
## Weights
The scale is an explicit, required argument. It is not read from checkpoint
metadata. Use matching Wan and Flash checkpoints:
[ComfyUI safetensors weights](https://huggingface.co/Comfy-Org/RelayVSR) load
directly. Select the matching pair for the requested scale:
| Scale | `--diffusion-model` | `--vae` |
| `--upscale-factor` | `--diffusion-model` | `--vae` |
| --- | --- | --- |
| 2 | `relayvsr_x2_lightx2v_int8_convrot.safetensors` | `relayvsr_flash_vae_x2_fp32.safetensors` |
| 4 | `relayvsr_x4_int8_convrot.safetensors` | `relayvsr_flash_vae_x4_fp32.safetensors` |
No separate text encoder or ordinary Wan VAE is needed.
## Usage
For a video, extract numbered frames into a directory, then run:
```sh
sd-cli -M vid_upscale \
--diffusion-model models/diffusion_models/relayvsr_x2_lightx2v_int8_convrot.safetensors \
@@ -27,44 +27,24 @@ sd-cli -M vid_upscale \
--vsr-window-size 16 -s 42 -o output.avi
```
`--vsr-window-size` includes both reference endpoints. The default 16 samples
references at frame indices 0, 15, 30, and so on, always including the last
frame. Shared endpoints update Flash history in the following window but are
emitted only once. Single-frame input is also supported.
Frames are read in filename order. Set `--fps` to the source video's frame rate.
Video outputs support AVI, WebM, and animated WebP. To save individual frames,
use an output pattern such as `-o output/%04d.png`.
Wan performs one flow-matching Euler step at timestep 1000 with guidance 1,
using the prompt embedding stored in the checkpoint. No text encoder or
ordinary Wan VAE is needed. Generic generation sampling settings do not
configure this fixed sampling path.
For a single image, replace `--input-frames input_frames` with `-i input.png`
and use `-o output.png` (or JPEG/WebP). No FPS setting is needed for an image.
`-i` / `--init-img` and `--input-frames` are mutually exclusive.
The Flash scale cannot be verified from its tensor shapes: the x2 and x4 files
have identical shapes. The caller must select the correct pair. Wan's input
projection is checked against the requested scale. Inputs are replicate-padded
internally and outputs are cropped to exactly the input size times the scale.
## Settings and limits
INT8 convrot uses the existing [quantized-weight execution path](int8_convrot.md).
Backend and parameter placement use `--backend` and `--params-backend`, with
Wan assigned to `diffusion` and Flash to `vae`. CUDA builds with CUTLASS use
fused window attention on compute capability 8.0 or newer. It loads spatial
neighborhoods directly, with FP16 Q/K/V and FP32 accumulation, and preserves
Wan's boundary-shifted windows and Flash's boundary-truncated windows.
Other backends use bounded spatial tiles with ordinary attention operators.
Set `GGML_CUDA_DISABLE_WINDOW_ATTENTION=1` to use that fallback on CUDA for
comparison. Floating-point results can differ between these implementations.
Library callers initialize `sd_video_upscaler_ctx_params_t` with
`sd_video_upscaler_ctx_params_init()`, set the model paths, explicit scale and
backend options, and create a `video_upscaler_ctx_t` with
`new_video_upscaler_ctx()`. Model architecture is detected from
weight names and shapes; the scale remains caller-supplied.
Initialize `sd_video_upscale_params_t` with `sd_video_upscale_params_init()`,
then set the RGB frames, frame count, FPS, seed and RelayVSR reference window.
Call `upscale_video()` and free the result with
`free_sd_video_generation_result()`. Defaults are 24 FPS, seed 42 and a
16-frame reference window. No sampler or scheduler selection is required.
Each call starts fresh
temporal state. Model caches are bounded by the temporal windows; the public
API currently holds the complete input and output frame arrays in host memory.
Audio is not copied by this path.
- `--upscale-factor` is required. Use 2 or 4 with the corresponding weights.
Output width and height are exactly the input dimensions times the scale.
- `--vsr-window-size` sets the reference-frame window, including both endpoints.
The default 16 selects reference frames 0, 15, 30, and so on, and always
includes the final frame. Every input frame appears once in the output.
- `--flash-attn auto` and `--conv-direct auto` are enabled by default. See
[backend selection](backend.md) to choose a device or weight placement.
- Sampler, scheduler, and step settings do not configure RelayVSR.
`--vsr-temporal-chunk-size` is a SeedVR2 option.
- Long videos require enough system memory to hold the input and output frames.
Audio is not copied to the output.
docs/seedvr2.md
+69
View File
@@ -0,0 +1,69 @@
# SeedVR2 image and video super-resolution
SeedVR2 upscales images and videos with `-M vid_upscale`. Support is experimental.
## Weights
These [ComfyUI safetensors weights](https://huggingface.co/Comfy-Org/SeedVR2)
load directly, without conversion or a separate text encoder:
| Model | `--diffusion-model` | `--vae` |
| --- | --- | --- |
| 3B | `seedvr2_3b_int8_convrot.safetensors` | `seedvr2_ema_vae_fp16.safetensors` |
| 7B Sharp | `seedvr2_7b_sharp_int8_convrot.safetensors` | `seedvr2_ema_vae_fp16.safetensors` |
Use the matching SeedVR2 VAE for either model. The 7B model requires more memory.
## Upscale an image
```sh
sd-cli -M vid_upscale \
--diffusion-model models/diffusion_models/seedvr2_3b_int8_convrot.safetensors \
--vae models/vae/seedvr2_ema_vae_fp16.safetensors \
-i input.png --upscale-factor 2 -s 42 -o restored.png
```
`-i` / `--init-img` accepts a single image. Outputs can be PNG, JPEG, or WebP.
No FPS or temporal chunk setting is needed for an image.
## Upscale a video
Extract the video into numbered frames, then pass the frame directory:
```sh
mkdir frames
ffmpeg -i input.mp4 -fps_mode passthrough frames/%06d.png
sd-cli -M vid_upscale \
--diffusion-model models/diffusion_models/seedvr2_3b_int8_convrot.safetensors \
--vae models/vae/seedvr2_ema_vae_fp16.safetensors \
--input-frames frames --upscale-factor 2 --fps 24 \
--vsr-temporal-chunk-size 1 -s 42 -o restored.avi
```
Set `--fps` to the source video's frame rate; 24 is only the example above.
Frames are read in filename order. Use either `-i` or `--input-frames`, not both.
Video outputs support AVI, WebM, and animated WebP. To save individual frames,
use an output pattern such as `-o restored/%06d.png`.
## Settings and limits
- `--upscale-factor` is required and must be a positive integer. Output width
and height are exactly the input dimensions multiplied by this value.
- `--vsr-temporal-chunk-size` controls how many video frames are processed
together. Valid values are 1, 5, 9, 13, and so on; the default is 5. A value of
1 processes each frame independently and uses less memory. Larger values use
neighboring frames but require more memory. Chunk boundaries may be visible.
- Frame count and order are preserved, including a shorter final chunk. Audio
is not copied to the output.
- If you run out of GPU memory, try `--vsr-temporal-chunk-size 1`. A five-frame
chunk at 1728x960 output resolution exceeded 24 GB of GPU memory in testing.
- `--flash-attn auto` and `--conv-direct auto` are enabled by default. See
[backend selection](backend.md) to choose a device or weight placement.
- SeedVR2 does not use sampler, scheduler, or step settings.
`--vsr-window-size` is a RelayVSR option and does not affect SeedVR2.
3B has been tested on CUDA and CPU; 7B has been tested on CUDA with small inputs.
Other backends have not been tested with this integration.
Model reference: [ByteDance-Seed/SeedVR](https://github.com/ByteDance-Seed/SeedVR).