I recently acquired a very powerful GPU but my training times only went down by half compared to training locally on my laptop. im using pytorch lightning and i see that GPU usage is very low (< 5%) and 100% cpu usage. training till early stopping still takes about 1.5 hours per set of hyperparams im testing which is a little too slow for me. can i have a bit of intuition/suggestions as to what might be the potential issues? thanks!
It might help to put the code in the pastebin and indicate which parts you changed after migrating to the better GPU