Instructions to use bineric/lynx-instruct-30b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bineric/lynx-instruct-30b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bineric/lynx-instruct-30b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("bineric/lynx-instruct-30b")
model = AutoModelForCausalLM.from_pretrained("bineric/lynx-instruct-30b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bineric/lynx-instruct-30b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bineric/lynx-instruct-30b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bineric/lynx-instruct-30b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bineric/lynx-instruct-30b

SGLang

How to use bineric/lynx-instruct-30b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bineric/lynx-instruct-30b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bineric/lynx-instruct-30b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bineric/lynx-instruct-30b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bineric/lynx-instruct-30b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bineric/lynx-instruct-30b with Docker Model Runner:
```
docker model run hf.co/bineric/lynx-instruct-30b
```

lynx-instruct-30b / assets /quantization_comparison.svg

Umair Imam

Add comprehensive model card with Nordic evaluation results and graphs

50b9f98 about 2 months ago

preview code

raw

history blame contribute delete

3.29 kB

	<svg width="700" height="400" xmlns="http://www.w3.org/2000/svg">
	<style>
	.title { font: bold 18px sans-serif; fill: #333; }
	.label { font: 12px sans-serif; fill: #333; }
	.value { font: 11px sans-serif; fill: #333; }
	.axis { font: 11px sans-serif; fill: #666; }
	.legend { font: 13px sans-serif; fill: #333; }
	</style>

	<!-- Background -->
	<rect width="700" height="400" fill="#fafafa"/>

	<!-- Title -->
	<text x="350" y="30" text-anchor="middle" class="title">8-bit vs 4-bit Quantization (Norwegian)</text>

	<!-- Legend -->
	<rect x="480" y="50" width="16" height="16" fill="#3b82f6"/>
	<text x="502" y="63" class="legend">8-bit</text>
	<rect x="560" y="50" width="16" height="16" fill="#94a3b8"/>
	<text x="582" y="63" class="legend">4-bit</text>

	<!-- Y-axis labels -->
	<text x="70" y="100" text-anchor="end" class="label">Sentiment</text>
	<text x="70" y="145" text-anchor="end" class="label">NER (nb)</text>
	<text x="70" y="190" text-anchor="end" class="label">NER (nn)</text>
	<text x="70" y="235" text-anchor="end" class="label">Reading</text>
	<text x="70" y="280" text-anchor="end" class="label">Summary</text>
	<text x="70" y="325" text-anchor="end" class="label">Common Sense</text>
	<text x="70" y="370" text-anchor="end" class="label">Ling. Accept.</text>

	<!-- Bars - Sentiment -->
	<rect x="80" y="85" width="306" height="18" fill="#3b82f6" rx="2"/>
	<rect x="80" y="105" width="298.2" height="18" fill="#94a3b8" rx="2"/>
	<text x="392" y="98" class="value">51.0%</text>
	<text x="384" y="118" class="value">49.7%</text>

	<!-- Bars - NER nb -->
	<rect x="80" y="130" width="394.2" height="18" fill="#3b82f6" rx="2"/>
	<rect x="80" y="150" width="390.6" height="18" fill="#94a3b8" rx="2"/>
	<text x="480" y="143" class="value">65.7%</text>
	<text x="476" y="163" class="value">65.1%</text>

	<!-- Bars - NER nn -->
	<rect x="80" y="175" width="426" height="18" fill="#3b82f6" rx="2"/>
	<rect x="80" y="195" width="419.4" height="18" fill="#94a3b8" rx="2"/>
	<text x="512" y="188" class="value">71.0%</text>
	<text x="505" y="208" class="value">69.9%</text>

	<!-- Bars - Reading -->
	<rect x="80" y="220" width="367.2" height="18" fill="#3b82f6" rx="2"/>
	<rect x="80" y="240" width="353.4" height="18" fill="#94a3b8" rx="2"/>
	<text x="453" y="233" class="value">61.2%</text>
	<text x="439" y="253" class="value">58.9%</text>

	<!-- Bars - Summary -->
	<rect x="80" y="265" width="380.4" height="18" fill="#3b82f6" rx="2"/>
	<rect x="80" y="285" width="378.6" height="18" fill="#94a3b8" rx="2"/>
	<text x="466" y="278" class="value">63.4%</text>
	<text x="464" y="298" class="value">63.1%</text>

	<!-- Bars - Common Sense -->
	<rect x="80" y="310" width="415.8" height="18" fill="#3b82f6" rx="2"/>
	<rect x="80" y="330" width="411" height="18" fill="#94a3b8" rx="2"/>
	<text x="501" y="323" class="value">69.3%</text>
	<text x="497" y="343" class="value">68.5%</text>

	<!-- Bars - Linguistic Acceptability (biggest difference) -->
	<rect x="80" y="355" width="218.4" height="18" fill="#3b82f6" rx="2"/>
	<rect x="80" y="375" width="178.8" height="18" fill="#94a3b8" rx="2"/>
	<text x="304" y="368" class="value">36.4% (+6.6%)</text>
	<text x="264" y="388" class="value">29.8%</text>
	</svg>