feat: add SeedVR2 image and video upscaling (#2300)

docs/relay_vsr.md

@@ -1,24 +1,24 @@
-# RelayVSR video super-resolution
+# RelayVSR image and video super-resolution
 
-RelayVSR uses a Wan model to sample sparse reference latents, followed by the
-Flash conditional decoder to reconstruct every input frame. The model code
-follows [RelayVSR](https://github.com/kopperx/RelayVSR); the loader maps
-[ComfyUI repackaged weights](https://huggingface.co/Comfy-Org/RelayVSR) to the
-original module names.
+[RelayVSR](https://github.com/kopperx/RelayVSR) upscales images and videos by
+2x or 4x with `-M vid_upscale`.
 
-Use `-M vid_upscale` with an ordered directory of RGB images. Filenames are
-sorted lexicographically. Supply the source frame rate with `--fps`. Output
-keeps the input frame count. Use a video output supported by the CLI, such as
-`.avi` or `.webm`, or a numbered PNG output such as `output/%04d.png`.
+## Weights
 
-The scale is an explicit, required argument. It is not read from checkpoint
-metadata. Use matching Wan and Flash checkpoints:
+[ComfyUI safetensors weights](https://huggingface.co/Comfy-Org/RelayVSR) load
+directly. Select the matching pair for the requested scale:
 
-| Scale | `--diffusion-model` | `--vae` |
+| `--upscale-factor` | `--diffusion-model` | `--vae` |
 | --- | --- | --- |
 | 2 | `relayvsr_x2_lightx2v_int8_convrot.safetensors` | `relayvsr_flash_vae_x2_fp32.safetensors` |
 | 4 | `relayvsr_x4_int8_convrot.safetensors` | `relayvsr_flash_vae_x4_fp32.safetensors` |
 
+No separate text encoder or ordinary Wan VAE is needed.
+
+## Usage
+
+For a video, extract numbered frames into a directory, then run:
+
 ```sh
 sd-cli -M vid_upscale \
   --diffusion-model models/diffusion_models/relayvsr_x2_lightx2v_int8_convrot.safetensors \
@@ -27,44 +27,24 @@ sd-cli -M vid_upscale \
   --vsr-window-size 16 -s 42 -o output.avi
 ```
 
-`--vsr-window-size` includes both reference endpoints. The default 16 samples
-references at frame indices 0, 15, 30, and so on, always including the last
-frame. Shared endpoints update Flash history in the following window but are
-emitted only once. Single-frame input is also supported.
+Frames are read in filename order. Set `--fps` to the source video's frame rate.
+Video outputs support AVI, WebM, and animated WebP. To save individual frames,
+use an output pattern such as `-o output/%04d.png`.
 
-Wan performs one flow-matching Euler step at timestep 1000 with guidance 1,
-using the prompt embedding stored in the checkpoint. No text encoder or
-ordinary Wan VAE is needed. Generic generation sampling settings do not
-configure this fixed sampling path.
+For a single image, replace `--input-frames input_frames` with `-i input.png`
+and use `-o output.png` (or JPEG/WebP). No FPS setting is needed for an image.
+`-i` / `--init-img` and `--input-frames` are mutually exclusive.
 
-The Flash scale cannot be verified from its tensor shapes: the x2 and x4 files
-have identical shapes. The caller must select the correct pair. Wan's input
-projection is checked against the requested scale. Inputs are replicate-padded
-internally and outputs are cropped to exactly the input size times the scale.
+## Settings and limits
 
-INT8 convrot uses the existing [quantized-weight execution path](int8_convrot.md).
-Backend and parameter placement use `--backend` and `--params-backend`, with
-Wan assigned to `diffusion` and Flash to `vae`. CUDA builds with CUTLASS use
-fused window attention on compute capability 8.0 or newer. It loads spatial
-neighborhoods directly, with FP16 Q/K/V and FP32 accumulation, and preserves
-Wan's boundary-shifted windows and Flash's boundary-truncated windows.
-Other backends use bounded spatial tiles with ordinary attention operators.
-Set `GGML_CUDA_DISABLE_WINDOW_ATTENTION=1` to use that fallback on CUDA for
-comparison. Floating-point results can differ between these implementations.
-
-Library callers initialize `sd_video_upscaler_ctx_params_t` with
-`sd_video_upscaler_ctx_params_init()`, set the model paths, explicit scale and
-backend options, and create a `video_upscaler_ctx_t` with
-`new_video_upscaler_ctx()`. Model architecture is detected from
-weight names and shapes; the scale remains caller-supplied.
-
-Initialize `sd_video_upscale_params_t` with `sd_video_upscale_params_init()`,
-then set the RGB frames, frame count, FPS, seed and RelayVSR reference window.
-Call `upscale_video()` and free the result with
-`free_sd_video_generation_result()`. Defaults are 24 FPS, seed 42 and a
-16-frame reference window. No sampler or scheduler selection is required.
-
-Each call starts fresh
-temporal state. Model caches are bounded by the temporal windows; the public
-API currently holds the complete input and output frame arrays in host memory.
-Audio is not copied by this path.
+- `--upscale-factor` is required. Use 2 or 4 with the corresponding weights.
+  Output width and height are exactly the input dimensions times the scale.
+- `--vsr-window-size` sets the reference-frame window, including both endpoints.
+  The default 16 selects reference frames 0, 15, 30, and so on, and always
+  includes the final frame. Every input frame appears once in the output.
+- `--flash-attn auto` and `--conv-direct auto` are enabled by default. See
+  [backend selection](backend.md) to choose a device or weight placement.
+- Sampler, scheduler, and step settings do not configure RelayVSR.
+  `--vsr-temporal-chunk-size` is a SeedVR2 option.
+- Long videos require enough system memory to hold the input and output frames.
+  Audio is not copied to the output.