refactor: require external Gemma 2 and GPT-OSS tokenizers (#1974)

This commit is contained in:
leejet authored and GitHub committed 2026-09-15 01:25:13 +08:00
1 parent 4964abdfc5
commit f9ddc0f388
19 files changed
+73 -337

No files matched your search

+2 -1
View File
@@ -118,7 +118,8 @@ documentation.
6. Run the narrowest useful build, test, or inspection command available.
Follow `CONTRIBUTING.md` for formatting, naming, PR expectations, dependency
update policy, and security rules.
update policy, and security rules. For tokenizer additions, follow its embedded-data
allowlist and default to an external `tokenizer.json`.
---