mirror of
https://github.com/leejet/stable-diffusion.cpp.git
synced 2025-12-12 13:28:37 +00:00
beb99a2de2
* add qwen tokenizer * add qwen2.5 vl support * mv qwen.hpp -> qwenvl.hpp * add qwen image model * add qwen image t2i pipeline * fix qwen image flash attn * add qwen image i2i pipline * change encoding of vocab_qwen.hpp to utf8 * fix get_first_stage_encoding * apply jeffbolz f32 patch https://github.com/leejet/stable-diffusion.cpp/pull/851#issuecomment-3335515302 * fix the issue that occurs when using CUDA with k-quants weights * optimize the handling of the FeedForward precision fix * to_add_out precision fix * update docs
10 lines
307 B
C++
10 lines
307 B
C++
#ifndef __TOKENIZE_UTIL__
|
|
#define __TOKENIZE_UTIL__
|
|
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
std::vector<std::string> token_split(const std::string& text);
|
|
std::vector<std::string> split_with_special_tokens(const std::string& text, const std::vector<std::string>& special_tokens);
|
|
|
|
#endif // __TOKENIZE_UTIL__
|