feat: support external Hugging Face tokenizer JSON files (#1973)

This commit is contained in:
leejet authored and GitHub committed 2026-09-15 00:27:39 +08:00
1 parent 42d6c0ab92
commit 4964abdfc5
58 files changed
+97390 -180

No files matched your search

+7 -1
View File
@@ -7,4 +7,10 @@
- httplib.h from: https://github.com/yhirose/cpp-httplib/blob/master/httplib.h
- LICENSE: https://github.com/yhirose/cpp-httplib/blob/master/LICENSE
- stb_image.h/stb_image_resize.h/stb_image_write.h from: https://github.com/nothings/stb
- LICENSE: https://github.com/nothings/stb/blob/master/LICENSE
- LICENSE: https://github.com/nothings/stb/blob/master/LICENSE
- Oniguruma from: https://github.com/kkos/oniguruma
- Version and source details: [README](oniguruma/README.md)
- LICENSE: [BSD-2-Clause](oniguruma/COPYING)
- utf8proc from: https://github.com/JuliaStrings/utf8proc
- Version and source details: [README](utf8proc/README.md)
- LICENSE: [MIT and Unicode data licenses](utf8proc/LICENSE.md)