Release pretrained_bpe
Unclaimed project
Are you a maintainer of flash-tokenizer? Claim this project to take control of your public changelog and roadmap.
Changelog
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
Release pretrained_bpe
⚠️⚠️⚠️⚠️⚠️
The configuration file is downloaded from here by the from_pretrained method.
VC_redist.x64.exe : Latest Microsoft Visual C++ Redistributable Version.⚠️⚠️⚠️⚠️⚠️
texts_en.all.parquettexts_ko.all.parquettexts_cn.all.parquettexts_multilingual.all.parquet