| 2025-11-06 10:18:05 - experiment_save_merged_model - INFO - Starting merged model save process | |
| 2025-11-06 10:18:05 - experiment_save_merged_model - INFO - Arguments: {'lambdas_path': '/work/gj26/b20042/LLM-AdaMerge/outputs/deepseek-7b/task-wise/cross_entropy-ep2-10%dataset-lambda09/llm_adamerge_lambdas.json', 'model_config': '/work/gj26/b20042/LLM-AdaMerge/outputs/deepseek-7b/task-wise/cross_entropy-ep2-10%dataset-lambda09/model_config.yaml', 'output_dir': '/work/gj26/b20042/LLM-AdaMerge/mergekit/outputs/deepseek-7b/llmadamerge/task-wise/cross_entropy-ep2-10%dataset/lambda09', 'model_name': 'merged-model', 'push_to_hub': False, 'hub_repo_id': 'lejelly/ds7b-ep2-data10-id4-taskwise-lambda09', 'private': False, 'device': 'cuda', 'debug': False} | |
| 2025-11-06 10:18:05 - experiment_save_merged_model - INFO - Loading lambdas from /work/gj26/b20042/LLM-AdaMerge/outputs/deepseek-7b/task-wise/cross_entropy-ep2-10%dataset-lambda09/llm_adamerge_lambdas.json | |
| 2025-11-06 10:18:05 - experiment_save_merged_model - INFO - Auto-detected parameter-wise merge from JSON structure | |
| 2025-11-06 10:18:05 - experiment_save_merged_model - INFO - Merge type: parameter_wise | |
| 2025-11-06 10:18:06 - experiment_save_merged_model - INFO - [Initial] Memory Usage: | |
| 2025-11-06 10:18:06 - experiment_save_merged_model - INFO - Process: 0.42 GB (0.2%) | |
| 2025-11-06 10:18:06 - experiment_save_merged_model - INFO - System: 7.72 GB / 212.49 GB (8.2%) | |
| 2025-11-06 10:18:06 - experiment_save_merged_model - INFO - Available: 194.96 GB | |
| 2025-11-06 10:18:06 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-11-06 10:18:06 - experiment_save_merged_model - INFO - Loading models | |
| 2025-11-06 10:18:15 - experiment_save_merged_model - INFO - [After loading models] Memory Usage: | |
| 2025-11-06 10:18:15 - experiment_save_merged_model - INFO - Process: 0.64 GB (0.3%) | |
| 2025-11-06 10:18:15 - experiment_save_merged_model - INFO - System: 49.40 GB / 212.49 GB (31.2%) | |
| 2025-11-06 10:18:15 - experiment_save_merged_model - INFO - Available: 146.24 GB | |
| 2025-11-06 10:18:15 - experiment_save_merged_model - INFO - GPU 0: Allocated: 38.61 GB, Reserved: 40.64 GB, Total: 94.50 GB | |
| 2025-11-06 10:18:15 - experiment_save_merged_model - INFO - Initializing parameter_wise AdaMerge | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Loading learned lambdas | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Deleting original models to free memory (task vectors already computed) | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - [Before deleting models] Memory Usage: | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Process: 39.08 GB (18.4%) | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - System: 109.81 GB / 212.49 GB (59.6%) | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Available: 85.81 GB | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - GPU 0: Allocated: 38.61 GB, Reserved: 40.64 GB, Total: 94.50 GB | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Clearing model_loader references | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Deleting model variables | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Running garbage collection | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - [After deleting models and GC] Memory Usage: | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Process: 39.08 GB (18.4%) | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - System: 68.64 GB / 212.49 GB (40.2%) | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Available: 126.98 GB | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - [After loading lambdas] Memory Usage: | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Process: 39.08 GB (18.4%) | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - System: 68.64 GB / 212.49 GB (40.2%) | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Available: 126.98 GB | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - GPU 0: Allocated: 0.00 GB, Reserved: 0.00 GB, Total: 94.50 GB | |
| 2025-11-06 10:22:41 - experiment_save_merged_model - INFO - Creating merged model with learned lambdas | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - [After merging models] Memory Usage: | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Process: 39.06 GB (18.4%) | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - System: 91.10 GB / 212.49 GB (50.8%) | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Available: 104.52 GB | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - GPU 0: Allocated: 38.61 GB, Reserved: 43.18 GB, Total: 94.50 GB | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Freeing memory from AdaMerge object (task vectors and base params no longer needed) | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Deleting task vectors | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Deleting base params | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Deleting functional model | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - [After freeing AdaMerge memory] Memory Usage: | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Process: 0.41 GB (0.2%) | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - System: 49.35 GB / 212.49 GB (31.2%) | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Available: 146.28 GB | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - GPU 0: Allocated: 38.61 GB, Reserved: 40.49 GB, Total: 94.50 GB | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Saving merged model to /work/gj26/b20042/LLM-AdaMerge/mergekit/outputs/deepseek-7b/llmadamerge/task-wise/cross_entropy-ep2-10%dataset/lambda09 | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Moving merged model to CPU for saving | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - ERROR - No safetensors files were saved! | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Attempting alternative save method... | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Saved as pytorch_model.bin | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - [After saving model] Memory Usage: | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Process: 0.41 GB (0.2%) | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - System: 49.35 GB / 212.49 GB (31.2%) | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Available: 146.28 GB | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - GPU 0: Allocated: 38.61 GB, Reserved: 40.49 GB, Total: 94.50 GB | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Saving tokenizer | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Copied lambdas file to /work/gj26/b20042/LLM-AdaMerge/mergekit/outputs/deepseek-7b/llmadamerge/task-wise/cross_entropy-ep2-10%dataset/lambda09/learned_lambdas.json | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Creating model card | |
| 2025-11-06 10:22:43 - experiment_save_merged_model - INFO - Cleaning up models | |
| 2025-11-06 10:22:44 - experiment_save_merged_model - INFO - [After cleanup] Memory Usage: | |
| 2025-11-06 10:22:44 - experiment_save_merged_model - INFO - Process: 0.42 GB (0.2%) | |
| 2025-11-06 10:22:44 - experiment_save_merged_model - INFO - System: 49.35 GB / 212.49 GB (31.2%) | |
| 2025-11-06 10:22:44 - experiment_save_merged_model - INFO - Available: 146.28 GB | |
| 2025-11-06 10:22:44 - experiment_save_merged_model - INFO - GPU 0: Allocated: 38.61 GB, Reserved: 40.49 GB, Total: 94.50 GB | |
| 2025-11-06 10:22:44 - experiment_save_merged_model - INFO - Model saved successfully to /work/gj26/b20042/LLM-AdaMerge/mergekit/outputs/deepseek-7b/llmadamerge/task-wise/cross_entropy-ep2-10%dataset/lambda09 | |