apologies in advance if this isn't the correct channel to ask: I recently acquired a very powerful GPU but my training times only went down by half compared to training locally on my laptop. im using pytorch lightning and i see that GPU usage is very low (< 5%) and 100% cpu usage. training till early stopping still takes about 1.5 hours per set of hyperparams im testing which is a little too slow for me. can i have a bit of intuition/suggestions as to what might be the potential issues? thanks! edit: training a bi-directional LSTM model on a small dataset (less than 100K rows)
Just to get the obvious out: did you check if you were looking at Compute, and not just overall GPU usage (including 3d, video, etc...)?