Instructions to use Data-Selection/PDS-1.7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Data-Selection/PDS-1.7B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Data-Selection/PDS-1.7B")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Data-Selection/PDS-1.7B") model = AutoModelForCausalLM.from_pretrained("Data-Selection/PDS-1.7B") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use Data-Selection/PDS-1.7B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Data-Selection/PDS-1.7B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Data-Selection/PDS-1.7B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Data-Selection/PDS-1.7B
- SGLang
How to use Data-Selection/PDS-1.7B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Data-Selection/PDS-1.7B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Data-Selection/PDS-1.7B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Data-Selection/PDS-1.7B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Data-Selection/PDS-1.7B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Data-Selection/PDS-1.7B with Docker Model Runner:
docker model run hf.co/Data-Selection/PDS-1.7B
Add link to Github and improve description
Browse filesThis PR adds the link to the Github repository and improves the model card description by including information from the abstract.
README.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
datasets:
|
| 4 |
- togethercomputer/RedPajama-Data-1T
|
| 5 |
language:
|
| 6 |
- en
|
| 7 |
-
pipeline_tag: text-generation
|
| 8 |
library_name: transformers
|
|
|
|
|
|
|
| 9 |
---
|
| 10 |
|
| 11 |
## PDS-1.7B
|
|
@@ -14,7 +14,8 @@ library_name: transformers
|
|
| 14 |
|
| 15 |
**PDS-1.7B** is a 1.7B model with [Mistral](https://arxiv.org/abs/2310.06825) achitecture pre-trained from scratch on the data selected from the CC split of [Redpajama](https://github.com/togethercomputer/RedPajama-Data), using the PDS framework.
|
| 16 |
|
| 17 |
-
|
|
|
|
| 18 |
|
| 19 |
Please refer to our [paper](https://arxiv.org/abs/2410.07064) for more details.
|
| 20 |
|
|
@@ -42,6 +43,21 @@ PDS-selected data improves the performance of language models pre-trained from s
|
|
| 42 |
|
| 43 |
[Conventional Pre-training](https://huggingface.co/Data-Selection/BSL-1.7B)
|
| 44 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
### Citation
|
| 46 |
|
| 47 |
```bibtex
|
|
@@ -51,4 +67,4 @@ PDS-selected data improves the performance of language models pre-trained from s
|
|
| 51 |
journal={arXiv preprint arXiv:2410.07064},
|
| 52 |
year={2024}
|
| 53 |
}
|
| 54 |
-
```
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
datasets:
|
| 3 |
- togethercomputer/RedPajama-Data-1T
|
| 4 |
language:
|
| 5 |
- en
|
|
|
|
| 6 |
library_name: transformers
|
| 7 |
+
license: apache-2.0
|
| 8 |
+
pipeline_tag: text-generation
|
| 9 |
---
|
| 10 |
|
| 11 |
## PDS-1.7B
|
|
|
|
| 14 |
|
| 15 |
**PDS-1.7B** is a 1.7B model with [Mistral](https://arxiv.org/abs/2310.06825) achitecture pre-trained from scratch on the data selected from the CC split of [Redpajama](https://github.com/togethercomputer/RedPajama-Data), using the PDS framework.
|
| 16 |
|
| 17 |
+
This work investigates the selection of high-quality pre-training data from massive corpora to enhance LMs' capabilities for downstream usage.
|
| 18 |
+
We formulate data selection as a generalized Optimal Control problem, which can be solved theoretically by Pontryagin's Maximum Principle (PMP), yielding a set of necessary conditions that characterize the relationship between optimal data selection and LM training dynamics. Based on these theoretical results, we introduce PMP-based Data Selection (PDS), a framework that approximates optimal data selection by solving the PMP conditions.
|
| 19 |
|
| 20 |
Please refer to our [paper](https://arxiv.org/abs/2410.07064) for more details.
|
| 21 |
|
|
|
|
| 43 |
|
| 44 |
[Conventional Pre-training](https://huggingface.co/Data-Selection/BSL-1.7B)
|
| 45 |
|
| 46 |
+
### Sample Usage
|
| 47 |
+
|
| 48 |
+
```python
|
| 49 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 50 |
+
|
| 51 |
+
model_id = "Data-Selection/PDS-1.7B"
|
| 52 |
+
|
| 53 |
+
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
| 54 |
+
model = AutoModelForCausalLM.from_pretrained(model_id)
|
| 55 |
+
|
| 56 |
+
inputs = tokenizer("Hello, my name is", return_tensors="pt")
|
| 57 |
+
outputs = model.generate(**inputs)
|
| 58 |
+
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
### Citation
|
| 62 |
|
| 63 |
```bibtex
|
|
|
|
| 67 |
journal={arXiv preprint arXiv:2410.07064},
|
| 68 |
year={2024}
|
| 69 |
}
|
| 70 |
+
```
|