Where is the tokenizer?

by bobby-nakamoto - opened Aug 23, 2024

I can't find the tokenizer for this model. Can someone direct me to it?

This comment has been hidden

I haven't found the tokenizer yet.

I haven't found the tokenizer yet.

give me solution

I have the same problem...

I ended up using a tokenizer from a model that was trained off of this one.

I forget which one, but search for models that were trained off of this one and go in and take that model's tokenizer from their files.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment