privacy-filter-multilingual / tokenizer_config.json

Initial publish of OpenMed/privacy-filter-multilingual. Renamed from privacy-filter-openmed-pii54-multilingual; clean README matching OpenMed/privacy-filter-nemotron format. Training artifact JSONs intentionally not redistributed here.

7656d47 verified 18 days ago

raw

history blame contribute delete

234 Bytes

	{
	"backend": "tokenizers",
	"eos_token": "<\|endoftext\|>",
	"model_input_names": [
	"input_ids",
	"attention_mask"
	],
	"model_max_length": 128000,
	"pad_token": "<\|endoftext\|>",
	"tokenizer_class": "TokenizersBackend"
	}