Commit graph

4 commits

Author SHA1 Message Date
Michael Yang
855de683ca get eos_token_id from generation_config.json 2025-05-05 23:51:35 -07:00
Michael Yang
4456012956 fix unmarshaling merges 2024-12-04 09:21:56 -08:00
Michael Yang
60e47573a6 more tokenizer tests 2024-08-27 14:51:10 -07:00
Michael Yang
3eb08377f8 detect chat template from configs that contain lists 2024-08-27 10:49:33 -07:00