metadata
library_name: transformers
tags: []
SuperBPE tokenizer
SuperBPE tokenizer trained on mix of code + edu data
"Big cat sat on a mat" -> ['Big', 'Ġcat', 'Ġsat', 'ĠonĠa', 'Ġmat']
"this function does summation on a last axis and then returns a scalar." -> ['this', 'Ġfunction', 'Ġdoes', 'Ġsummation', 'ĠonĠa', 'Ġlast', 'Ġaxis', 'ĠandĠthen', 'ĠreturnsĠa', 'Ġscalar', '.']