feat: implement the complete bpe function (#119)

* implement the complete bpe function
---------

Co-authored-by: leejet <leejet714@gmail.com>
This commit is contained in:
旺旺碎冰冰andleejet authored and GitHub committed 2023-12-23 12:11:07 +08:00
1 parent 8f6b4a39d6
commit 0e64238e4c
4 files changed
+95 -11

No files matched your search

-1
View File
@@ -51,7 +51,6 @@ Inference of [Stable Diffusion](https://github.com/CompVis/stable-diffusion) in
- The current implementation of ggml_conv_2d is slow and has high memory usage
- Implement Winograd Convolution 2D for 3x3 kernel filtering
- [ ] Continuing to reduce memory usage (quantizing the weights of ggml_conv_2d)
- [ ] Implement BPE Tokenizer
- [ ] Implement [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN/tree/master) upscaler
- [ ] k-quants support