Generates gibberish
I have tried the Q8 version which is on ollama and that works quite well. I'd say about 95% of the quality of the one hosted here: https://huggingface.co/spaces/Souvik3333/Nanonets-ocr-s
When I run the Q8 from Unsloth though, it's just gibberish; same for Q6. I've set the parameters correctly. Any idea why that would be? Do these quants work for anyone else?
Is this through Ollama?
Is this through Ollama?
Yes through Ollama with Msty . I tried the same models in LM Studio and they don't work properly for some reason. Not sure what Ollama does differently.
@danihend can you try redownloading and see if its fixed?
I downloaded in LM Studio as Ollama is not updated with the newest model yet. It works the same as others in LM Studio - not very well. I then imported it to Ollama and it just outputs nonsense like this in response to a image:
pdf
{
"bbox_2d": [1, 368, 497, 504],
"label": "εΎ"
}
Not sure if it's due to an error on my part or what. Does it work for you with Ollama?
Hi, I was wondering if I could get some help. I tried the BF16 model with llama-cpp-python but I don't get any output. (My llama_cpp version is 0.3.14)
Can the resolution of the images be an issue? Do I need to change the parameters? Any advice is welcome!
my code:
import os
from llama_cpp import Llama
model_path = "Nanonets-OCR-s-BF16.gguf"
mmproj_path = "mmproj-BF16.gguf"
image_path = "Swiss_QR-Bill_example.jpg" # wget https://upload.wikimedia.org/wikipedia/commons/9/9f/Swiss_QR-Bill_example.jpg
llm = Llama(
model_path=model_path,
mmproj_model_path=mmproj_path,
n_ctx=128000,
n_gpu_layers=0,
n_threads=4,
mlock=True,
verbose=False
)
prompt = """Extract the text from the above document as if you were reading it naturally. Return the tables in html format. Return the equations in LaTeX representation. If there is an image in the document and image caption is not present, add a small description of the image inside the <img></img> tag; otherwise, add the image caption inside <img></img>. Watermarks should be wrapped in brackets. Ex: <watermark>OFFICIAL COPY</watermark>. Page numbers should be wrapped in brackets. Ex: <page_number>14</page_number> or <page_number>9/22</page_number>. Prefer using β and β for check boxes."""
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": [
{"type": "image", "image": f"file://{os.path.abspath(image_path)}"},
{"type": "text", "text": prompt},
]},
]
response = llm.create_chat_completion(
messages=messages,
temperature=0.7,
min_p=0.01,
top_p=1.0,
top_k=-1,
repeat_penalty=1.0,
max_tokens=-1
)
print(response)
I get this:
{'id': 'chatcmpl-00eea211-5e4e-4675-b1a2-bdde4748271e',
'object': 'chat.completion',
'created': 1754567520,
'model': 'Nanonets-OCR-s-BF16.gguf',
'choices': [{'index': 0,
'message': {'role': 'assistant',
'content': '<watermark>Watermark Removed</watermark>'},
'logprobs': None,
'finish_reason': 'stop'}],
'usage': {'prompt_tokens': 296, 'completion_tokens': 11, 'total_tokens': 307}}