Instructions to use BramVanroy/fietje-2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BramVanroy/fietje-2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BramVanroy/fietje-2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BramVanroy/fietje-2")
model = AutoModelForCausalLM.from_pretrained("BramVanroy/fietje-2")

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use BramVanroy/fietje-2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BramVanroy/fietje-2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BramVanroy/fietje-2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/BramVanroy/fietje-2

SGLang

How to use BramVanroy/fietje-2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BramVanroy/fietje-2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BramVanroy/fietje-2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BramVanroy/fietje-2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BramVanroy/fietje-2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use BramVanroy/fietje-2 with Docker Model Runner:
```
docker model run hf.co/BramVanroy/fietje-2
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Fietje 2

An open and efficient LLM for Dutch

👱‍♀️ Base version (this one) - 🤖 Instruct version - 💬 Chat version - 🚀 GGUF of base

Chat with Fietje here!

Fietje is an adapated version of microsoft/phi-2, tailored to Dutch text generation by training on 28B tokens. It is small and efficient with a size of 2.7 billion parameters while performing almost on par with more powerful Dutch LLMs of twice its size like GEITje 7B Ultra.

A thorough description of the creation and evaluation of Fietje as well as usage examples are available in this Github repository.

Citation

If you use Fietje or the CulturaX + Wikipedia filtered subset in your work, please cite to the following paper:

@misc{vanroy2024fietjeopenefficientllm,
      title={Fietje: An open, efficient LLM for Dutch}, 
      author={Bram Vanroy},
      year={2024},
      eprint={2412.15450},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.15450}, 
}

Intended uses & limitations

The same limitations as phi-2, and LLMs in general, apply here. LLMs hallucinate, make mistakes, and should not be trusted. Use at your own risk!

Training data

Fietje was continue-pretrained on 28B Dutch tokens, which includes the full Dutch component of Wikipedia (accounting for around 15%), supplemented with Dutch tokens from CulturaX. A newer version of this dataset can be found here, which also describes the filtering that took place to ensure high data quality.

Training procedure

I am thankful to the Flemish Supercomputer Center (VSC) for providing the computational power to accomplish this project. Accounting for waiting for jobs, training took around two weeks on four nodes of 4x A100 80GB each (16 total).

Training was done with the wonderful alignment-handbook, using DeepSpeed as a back-end. Exact training recipes and SLURM script are given in the Github repository.