Inference failed: Torch not compiled with CUDA enabled
Using moondream-station from pip. I get this error after downloading Moondream 3 Preview and attempting inference with the interactive CLI.
My GPUs are a pair of RTX8000 (Turing generation)
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:28:36_Pacific_Standard_Time_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0
Is there a way to fix this? I assume the software didn't specify to pull the cuda wheels?
I tried to go back to torch 2.6 which was the last that was precompiled for cuda v12.4. Now I don't get an error, but it doesn't use the GPU at all. I can see it running the cpu at 60% for several minutes now.
It would be great to have any info to let me debug this. It's weird that it required torch to be compiled with CUDA enabled if it works on CPU anyway? I'd rather get it onto one of the GPUs if possible.
Well I dont think it is doing inference on the CPU. Memory utilisation is < 1.5gb but processor utilisation still above 60% for nearly 30 mins. I have no idea what it would be doing. No GPU memory or processor utilisation.
The RTX8000 might run into some issues as this model uses BFloat16 and FlexAttention. I recently updated how Moondream Station is resolving your CUDA version, it might be worth updating and trying again.
pip install -U moondream-station
Same with a 3080TI.
The RTX8000 might run into some issues as this model uses BFloat16 and FlexAttention. I recently updated how Moondream Station is resolving your CUDA version, it might be worth updating and trying again.
pip install -U moondream-station
I'm getting the same error when running on the CPU.
Why do we need cuda to run on CPU?