mirror of
https://github.com/leejet/stable-diffusion.cpp.git
synced 2026-09-24 20:20:37 +00:00
107 lines
4.8 KiB
Markdown
107 lines
4.8 KiB
Markdown
# Sol-Attn
|
|
|
|
`--sol-attn` enables native CUDA Sol-Attn in the diffusion model, including the
|
|
high-noise diffusion model when present. It uses the shared attention dispatcher
|
|
without classifying tokens as text, images, or video. Python, PyTorch, Triton,
|
|
and CuTe DSL are not needed to build or run it.
|
|
|
|
This implementation follows the diagonal-threshold algorithm in
|
|
[NVlabs/Sana's Sol-Attn](https://github.com/NVlabs/Sana/tree/sol-engine/techniques/sparse_backends/sol_attn).
|
|
It summarizes 64-token KV blocks, selects exact blocks using proxy scores and
|
|
an online threshold, and approximates the remaining blocks using their K means
|
|
and V sums. Adjacent blocks remain exact. Both contributions share an online
|
|
softmax normalizer. Q/K/V and probability tiles use BF16 Tensor Cores with FP32
|
|
accumulation; the BF16 result is returned through the existing FP32 interface.
|
|
|
|
## Build
|
|
|
|
Use patched GGML, CUDA Toolkit 12.0 or newer, and an NVIDIA GPU with compute
|
|
capability 8.0 or newer. Compile kernels for the target GPU:
|
|
|
|
```sh
|
|
cmake -S . -B build -DSD_CUDA=ON -DSD_USE_UPSTREAM_GGML=OFF
|
|
cmake --build build --config Release
|
|
```
|
|
|
|
The feature is compiled with the CUDA backend; no separate build option is
|
|
required. Upstream GGML and non-CUDA backends do not support it. A system GGML
|
|
must provide the matching patched API and CUDA implementation. Tensor-parallel
|
|
row splitting is not supported; layer splitting requires supported devices.
|
|
|
|
## Use
|
|
|
|
Add `--sol-attn` to an existing generation command:
|
|
|
|
```sh
|
|
sd-cli ... --sol-attn
|
|
sd-cli ... --sol-attn --sol-attn-tau 1.0
|
|
```
|
|
|
|
The default threshold coefficient is `1.0`. Larger coefficients select fewer
|
|
blocks for exact attention. The coefficient must be finite; zero does not mean
|
|
dense attention. Omit `--sol-attn` to disable the feature.
|
|
|
|
The native kernel supports unmasked, noncausal attention with head dimension
|
|
128, equal Q/K/V sequence lengths and head counts, and multiple batches. Other
|
|
attention operations fall back to FlashAttention when available, then ordinary
|
|
attention. Existing attention scaling overrides remain effective. `--fa` and
|
|
`--diffusion-fa` may be used together with Sol-Attn; `--sage-attn` is mutually
|
|
exclusive. Text encoders and VAEs retain their existing attention selection.
|
|
|
|
Initialization reports an error if the requested diffusion backend cannot run
|
|
Sol-Attn. Graph logs report the number of Sol-Attn and FlashAttention nodes and
|
|
warn when no Sol-Attn nodes are selected. CUDA execution errors are not silently
|
|
converted into dense attention.
|
|
|
|
This is approximate attention. Validate quality and end-to-end speed with the
|
|
same prompt, seed, dimensions, frame count, and sampling settings. Include
|
|
packing, preprocessing, offload, and decode time in comparisons. Short sequences
|
|
may not benefit. Upstream combined pipeline speedups are not measurements of
|
|
this native kernel. Exact-covariance thresholds, text sinks, Morton ordering,
|
|
and step/layer schedules are not implemented.
|
|
|
|
## Validation
|
|
|
|
On an RTX 4090 with CUDA 12.4, Wan 2.1 T2V 1.3B was tested at 832x480,
|
|
33 frames, 20 Euler steps, seed 42, CFG 6, and flow shift 3, using the prompt
|
|
`a lovely cat` and the same negative prompt for every run:
|
|
|
|
| Attention | Sampling time | Total process time |
|
|
| --- | ---: | ---: |
|
|
| FlashAttention | 45.73 s | 74.63 s |
|
|
| Sol-Attn, tau 1 | 37.17 s | 66.20 s |
|
|
| Sol-Attn, tau 0 | 40.66 s | 68.50 s |
|
|
|
|
These are single-run measurements. The graph selected 30 Sol-Attn nodes and
|
|
30 FlashAttention nodes. At tau 1, sampled video frames showed washed-out
|
|
colors and reduced detail. Tau 0 improved clarity in this example, but still
|
|
changed the composition. Neither setting guarantees the baseline's quality.
|
|
For this Wan command, `--sol-attn --sol-attn-tau 0` is a more conservative
|
|
starting point. In the one-frame case, tau 1 increased warm sampling time from
|
|
0.140 to 0.148 seconds per step.
|
|
|
|
Validation also covered 15 numerical reference cases, 11 layout/scaling/fallback
|
|
cases, CUDA memory checking, and 36 existing SageAttention regression cases.
|
|
CLI and server CUDA builds and the upstream GGML CPU library build passed.
|
|
Other GPU architectures, multi-GPU execution, and other models have not been
|
|
tested.
|
|
|
|
## Library API
|
|
|
|
Configure Sol-Attn in `sd_ctx_params_t` before creating the context:
|
|
|
|
```cpp
|
|
sd_ctx_params_t params;
|
|
sd_ctx_params_init(¶ms);
|
|
// Set model paths and other context options here.
|
|
params.sol_attn = true;
|
|
params.sol_attn_tau = 1.0f;
|
|
sd_ctx_t* ctx = new_sd_ctx(¶ms);
|
|
```
|
|
|
|
`sd_ctx_params_init` defaults `sol_attn` to false and `sol_attn_tau` to 1.0.
|
|
`new_sd_ctx` returns null for a nonfinite threshold, unavailable requested
|
|
backends, or a conflict with SageAttention. The context owns a copy of these
|
|
settings; changing the input structure after creation does not reconfigure it.
|
|
Applications must be rebuilt against the updated `sd_ctx_params_t` definition.
|