feat: support disk params backend (#1651)

This commit is contained in:
leejet authored and GitHub committed 2026-06-14 14:48:50 +08:00
1 parent 276025e054
commit bdb431ad95
27 files changed
+134 -65

No files matched your search

+33 -1
View File
@@ -21,6 +21,38 @@ and the compute buffer shrink in the debug log:
Using `--offload-to-cpu` allows you to offload weights to the CPU, saving VRAM without reducing generation speed.
## Use params backend to reduce VRAM or RAM usage.
`--params-backend` controls where model parameters are kept. If it is not set, parameters use the same backend as `--backend`, so a GPU runtime backend also keeps parameters in VRAM.
Use CPU params to reduce VRAM usage:
```shell
--backend cuda0 --params-backend cpu
```
This keeps model weights in system RAM and moves them to the runtime backend when needed. `--offload-to-cpu` is a compatibility shortcut that prepends `*=cpu` to `--params-backend`, so explicit module assignments can still override it:
```shell
--offload-to-cpu --params-backend te=disk
```
Use disk params to reduce both VRAM and RAM usage:
```shell
--backend cuda0 --params-backend disk
```
This reloads parameters from the model file on demand and releases them after use. It has the lowest memory residency, but can be slower because weights must be read again. `disk` is never selected implicitly; set it explicitly when RAM usage matters more than reload cost.
Per-module assignments can target only the largest modules:
```shell
--backend cuda0 --params-backend diffusion=disk,te=cpu,vae=cpu
```
See [backend selection](./backend.md) for full syntax.
## Use quantization to reduce memory usage.
[quantization](./quantization_and_gguf.md)
[quantization](./quantization_and_gguf.md)