| --- |
| tags: |
| - language-model |
| - next-token-prediction |
| - neural-network |
| - English |
| - Amharic |
| license: apache-2.0 |
| datasets: |
| - my-dataset |
| metrics: |
| - accuracy |
| - perplexity |
| model_type: nn |
| language: |
| - en |
| - am |
| --- |
| |
| # Neural Network-Based Language Model for Next Token Prediction |
|
|
| ## Overview |
| This project is a midterm assignment focused on developing a neural network-based language model for next token prediction. The model was trained using a custom dataset with two languages, English and Amharic. The project incorporates techniques in neural networks to predict the next token in a sequence, demonstrating a non-transformer approach to language modeling. |
|
|
| ## Project Objectives |
| The main objective of this project was to: |
| - Develop a neural network-based model for next token prediction without using transformers or encoder-decoder architectures. |
| - Experiment with multiple languages to observe model performance. |
| - Implement checkpointing to save model progress and generate text during different training stages. |
| - Present a video demo showcasing the model's performance in generating text in both English and Amharic. |
|
|
| ## Project Details |
|
|
| ### 1. Training Languages |
| The model was trained using datasets in English and Amharic. The datasets were cleaned and prepared, including tokenization and embedding for improved model training. |
|
|
| ### 2. Tokenizer |
| A custom tokenizer was created using Byte Pair Encoding (BPE). This tokenizer was trained on five languages: English, Amharic, Sanskrit, Nepali, and Hindi, but the model specifically utilized English and Amharic for this task. |
|
|
| ### 3. Embedding Model |
| A custom embedding model was employed to convert tokens into vector representations, allowing the neural network to better understand the structure and meaning of the input data. |
|
|
| ### 4. Model Architecture |
| The project uses an LSTM (Long Short-Term Memory) neural network to predict the next token in a sequence. LSTMs are well-suited for sequential data and are a popular choice for language modeling due to their ability to capture long-term dependencies. |
|
|
| ## Results and Evaluation |
|
|
| ### Training Curve and Loss |
| The model’s training and validation loss over time are documented and included in the repository (`loss_values.csv`). The training curve demonstrates the model's learning progress, with explanations provided for key observations in the loss trends. |
|
|
| ### Checkpoint Implementation |
| Checkpointing was implemented to save model states at different training stages, allowing for partial model evaluations and text generation demos. Checkpoints are included in the repository for reference. |
|
|
| ### Perplexity Score |
| The model's perplexity score, calculated during training, is available in the `perplexity.csv` file. This score provides an indication of the model's predictive accuracy over time. |
|
|
| ## Demonstration |
| A video demo, linked below, demonstrates: |
| - Random initialization text generation in English. |
| - Text generation using the trained model in both English and Amharic, with English translations provided using Google Translate. |
|
|
| **Video Demo Link:** [YouTube Demo](https://youtu.be/1m21NYmLSC4) |
|
|
| ## Instructions for Reproducing the Results |
| 1. Install dependencies (Python, PyTorch, and other required libraries). |
| 2. Load the .ipynb notebook and run cells sequentially to replicate training and evaluation. |
| 3. Refer to HuggingFace documentation for downloading the model and tokenizer files. |
|
|
| Note: The data for the project has been taken from [saillab/taco-datasets](https://huggingface.co/datasets/saillab/taco-datasets) |
