| | --- |
| | language: en |
| | license: apache-2.0 |
| | tags: |
| | - vision-language-model |
| | - visual-storytelling |
| | - chain-of-thought |
| | - grounded-text-generation |
| | - cross-frame-consistency |
| | - storytelling |
| | - image-to-text |
| | datasets: |
| | - daniel3303/StoryReasoning |
| | metrics: |
| | - precision |
| | - recall |
| | - bleu |
| | - meteor |
| | - rouge |
| | base_model: |
| | - Qwen/Qwen2.5-VL-7B-Instruct |
| | pipeline_tag: image-to-text |
| | model-index: |
| | - name: QwenStoryteller |
| | results: |
| | - task: |
| | type: visual-storytelling |
| | name: Visual Storytelling |
| | dataset: |
| | name: StoryReasoning |
| | type: daniel3303/StoryReasoning |
| | split: test |
| | metrics: |
| | - name: Character Precision |
| | type: precision |
| | value: 0.83 |
| | - name: Object Precision |
| | type: precision |
| | value: 0.46 |
| | - name: Total Precision |
| | type: precision |
| | value: 0.57 |
| | - name: mAP |
| | type: mean_average_precision |
| | value: 0.27 |
| | - name: Character Recall |
| | type: recall |
| | value: 0.62 |
| | - name: Object Recall |
| | type: recall |
| | value: 0.25 |
| | - name: Total Recall |
| | type: recall |
| | value: 0.40 |
| | - name: METEOR |
| | type: meteor |
| | value: 0.14 |
| | - name: ROUGE-L |
| | type: rouge-l |
| | value: 0.16 |
| | - name: BLEU-4 |
| | type: bleu-4 |
| | value: 0.054 |
| | - name: Description Accuracy |
| | type: accuracy |
| | value: 2.76 |
| | description: "Rating on a scale of 1-5" |
| | - name: Average Hallucinations |
| | type: error_rate |
| | value: 3.56 |
| | description: "Average number of hallucinations per story" |
| | library_name: transformers |
| | --- |
| | |
| | # QwenStoryteller |
| |
|
| | QwenStoryteller is a fine-tuned version of Qwen2.5-VL 7B specialized for grounded visual storytelling with cross-frame consistency, capable of generating coherent narratives from multiple images while maintaining character and object identity throughout the story. |
| |
|
| | ## Model Description |
| |
|
| | **Base Model:** Qwen2.5-VL 7B |
| | **Training Method:** LoRA fine-tuning (rank 2048, alpha 4096) |
| | **Training Dataset:** [StoryReasoning](https://huggingface.co/datasets/daniel3303/StoryReasoning) |
| |
|
| | QwenStoryteller processes sequences of images to perform: |
| | - End-to-end object detection |
| | - Cross-frame object re-identification |
| | - Landmark detection |
| | - Chain-of-thought reasoning for scene understanding |
| | - Grounded story generation with explicit visual references |
| |
|
| | The model was fine-tuned on the StoryReasoning dataset using LoRA with a rank of 2048 and alpha scaling factor of 4096, targeting self-attention layers of the language components. Training used a peak learning rate of 1×10⁻⁴ with batch size 32, warmup for the first 3% of steps for 4 epochs, AdamW optimizer with weight decay 0.01, and bfloat16 precision. |
| |
|
| | ## System Prompt |
| | The model was trained with the following system prompt, and we recommend using it as it is for inference. |
| |
|
| | ``` |
| | You are an AI storyteller that can analyze sequences of images and create creative narratives. |
| | First think step-by-step to analyze characters, objects, settings, and narrative structure. |
| | Then create a grounded story that maintains consistent character identity and object references across frames. |
| | Use <think></think> tags to show your reasoning process before writing the final story. |
| | ``` |
| |
|
| | ## Key Features |
| |
|
| | - **Cross-Frame Consistency:** Maintains consistent character and object identity across multiple frames through visual similarity and face recognition techniques |
| | - **Structured Reasoning:** Employs chain-of-thought reasoning to analyze scenes with explicit modeling of characters, objects, settings, and narrative structure |
| | - **Grounded Storytelling:** Uses specialized XML tags to link narrative elements directly to visual entities |
| | - **Reduced Hallucinations:** Achieves 12.3% fewer hallucinations compared to the non-fine-tuned base model |
| |
|
| | ## Usage |
| |
|
| | ```python |
| | from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor |
| | from qwen_vl_utils import process_vision_info |
| | import torch |
| | from PIL import Image |
| | |
| | # Load the model |
| | model = Qwen2_5_VLForConditionalGeneration.from_pretrained( |
| | "daniel3303/QwenStoryteller", torch_dtype="auto", device_map="auto" |
| | ) |
| | |
| | # Load processor |
| | processor = AutoProcessor.from_pretrained("daniel3303/QwenStoryteller") |
| | |
| | # Load images |
| | images = [ |
| | Image.open("image1.jpg"), |
| | Image.open("image2.jpg"), |
| | Image.open("image3.jpg"), |
| | Image.open("image4.jpg"), |
| | Image.open("image5.jpg") |
| | ] |
| | |
| | # Create image content list |
| | image_content = [] |
| | for img in images: |
| | image_content.append({ |
| | "type": "image", |
| | "image": img, |
| | }) |
| | |
| | # Add text prompt at the end |
| | image_content.append({"type": "text", "text": "Generate a story based on these images."}) |
| | |
| | # Create messages with system prompt |
| | messages = [ |
| | { |
| | "role": "system", |
| | "content": "You are an AI storyteller that can analyze sequences of images and create creative narratives. First think step-by-step to analyze characters, objects, settings, and narrative structure. Then create a grounded story that maintains consistent character identity and object references across frames. Use <think></think> tags to show your reasoning process before writing the final story." |
| | }, |
| | { |
| | "role": "user", |
| | "content": image_content, |
| | } |
| | ] |
| | |
| | # Preparation for inference |
| | text = processor.apply_chat_template( |
| | messages, tokenize=False, add_generation_prompt=True |
| | ) |
| | image_inputs, video_inputs = process_vision_info(messages) |
| | inputs = processor( |
| | text=[text], |
| | images=image_inputs, |
| | videos=video_inputs, |
| | padding=True, |
| | return_tensors="pt", |
| | ) |
| | inputs = inputs.to(model.device) |
| | |
| | # Inference: Generation of the output |
| | generated_ids = model.generate( |
| | **inputs, |
| | max_new_tokens=4096, |
| | do_sample=True, |
| | temperature=0.7, |
| | top_p=0.9 |
| | ) |
| | generated_ids_trimmed = [ |
| | out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) |
| | ] |
| | story = processor.batch_decode( |
| | generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False |
| | )[0] |
| | |
| | print(story) |
| | ``` |
| |
|
| | ### Using vLLM for faster inference |
| |
|
| | For significantly faster inference, you can use vLLM to serve the model. Simply install vLLM and run: |
| |
|
| | ```bash |
| | # Install vLLM |
| | pip install vllm |
| | |
| | # Serve the model with vLLM |
| | vllm serve daniel3303/QwenStoryteller |
| | ``` |
| |
|
| | ## Output Format |
| |
|
| | QwenStoryteller produces two main outputs: |
| |
|
| | 1. **Chain-of-Thought Analysis (`<think></think>`):** A structured analysis containing: |
| | - Character tables with consistent identity references, emotions, actions, and spatial locations |
| | - Object tables with functions, interactions, and spatial coordinates |
| | - Setting tables categorizing environmental elements |
| | - Narrative structure tables modeling story progression |
| |
|
| | 2. **Grounded Story:** A narrative with specialized XML tags linking text to visual elements: |
| | - `<gdi>`: Image tags for specific frames |
| | - `<gdo>`: Entity reference tags for character and object mentions |
| | - `<gda>`: Action tags for character actions |
| | - `<gdl>`: Location/landmark tags for background elements |
| |
|
| | ## Limitations |
| |
|
| | - Re-identification relies primarily on object appearance rather than overall context, which can lead to confusion with similar-looking objects/persons |
| | - Movie-derived training data introduces biases from cinematic composition that may not generalize to candid visual sequences |
| | - Low grounding rates for first-person pronouns as they primarily appear in character dialogues |
| | - May still produce hallucinations, albeit at a reduced rate compared to the base model |
| |
|
| | ## Citation |
| |
|
| | ``` |
| | @misc{oliveira2025storyreasoningdatasetusingchainofthought, |
| | title={StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation}, |
| | author={Daniel A. P. Oliveira and David Martins de Matos}, |
| | year={2025}, |
| | eprint={2505.10292}, |
| | archivePrefix={arXiv}, |
| | primaryClass={cs.CV}, |
| | url={https://arxiv.org/abs/2505.10292}, |
| | } |
| | ``` |
| |
|
| | ## Contact |
| |
|
| | For questions or feedback regarding this model, please contact: |
| | - Daniel A. P. Oliveira (daniel.oliveira@inesc-id.pt) |
