fix: benchmark Vulkan direct convolution in auto mode (#2141)

This commit is contained in:
leejet authored and GitHub committed 2026-10-15 23:08:30 +08:00
1 parent 2891378bb1
commit e47426c044
4 files changed
+84 -17

No files matched your search

+20 -2
View File
@@ -5,8 +5,13 @@ eg.:
- flux 768x768 ~600mb
- SD2 768x768 ~1400mb
For most backends, it slows things down, but for cuda it generally speeds it up too.
At the moment, it is only supported for some models and some backends (like cpu, cuda/rocm, metal).
`--flash-attn on|off|auto` controls flash attention. Its effect on speed depends
on GPU architecture, backend, driver, model, and tensor shapes. Automatic
selection uses backend rules and, for some paths, small probes; it does not
guarantee the fastest end-to-end generation. Compare `on` and `off` with the
same model and generation settings over repeated runs, separating startup/probe
time from steady-state generation time. See [backend selection](backend.md)
for automatic selection rules and per-module overrides.
Watch for:
```
@@ -21,6 +26,19 @@ CPU fallback. It excludes weights and cache buffers. Within a runner lifecycle,
the summary is printed only on the first graph or when backend capacities or the
segment count change.
## Tune direct convolution.
`--conv-direct on|off|auto` selects direct convolution or the im2col-based
path. Performance depends on GPU architecture, backend, driver, model, and
tensor shapes, so enabling direct convolution can either improve or reduce speed.
Automatic selection uses small benchmarks on Vulkan and implicit GEMM checks
on other backends. It does not guarantee the fastest end-to-end generation.
Compare `on` and `off` with the same model and generation settings over
repeated runs, keeping flash attention unchanged and separating startup/probe
time from steady-state generation time. See [backend selection](backend.md)
for automatic selection rules and per-module overrides.
## Run conditional and unconditional CFG in one batched UNet forward.
For UNet models, the conditional and unconditional guidance branches are