GPU utilization is not 100%.
29 views (last 30 days)
Joss Knight on 31 May 2019
Your question is very hard to answer in it's current form. You want to know why GPU utilisation is not 100%? The answer is, because the GPU isn't running kernels 100% of the time. Why? I don't know, because you haven't provided any information about what you're doing. Maybe, as Walter says, a lot of time is being spent doing file I/O, perhaps because you have a very slow disk or slow network file access. Maybe you have a transformed datastore, or an imageDatastore with a custom ReadFcn, and the data processing is very complex and takes place on the CPU, blocking GPU execution while it is carried out. Maybe you have a very small network, or a low resolution network, or you don't have a high enough mini-batch size, and so you are not successfully occupying all the cores on the GPU. Maybe your network is so small that the amount of time spent running the MATLAB interpreter in order to generate the GPU kernels to do the computation outweighs the amount of time it takes to run those kernels.
If you want to know more, run the MATLAB profiler and find out where time is being spent during training.
Abolfazl Nejatian on 15 Dec 2019
Thank you for the information you provided.
the strange thing is when i was testing my code on Linux and building my network with Python the GPU utilization grew up to around 100 percent but in windows with Matlab, it stays around 45 percent.
Abolfazl Nejatian on 16 Dec 2019
well, i know these are the different OS, but the vague point is, with the same resource(both of them use Tesla V100, actually i install both of OS on one machin), why they can't use GPU in a similar percent!
yes, absolutely i used MEX code on my Matlab.
then i try to train a Resnet with Python. ( and all of the initial value were same, input size, network layers and etc).
there is no code conversion between Matlab and Python i used Matlab function and Pretrained Net for this work and for Python use Keras and Pycharm.
but in a windows environment with Matlab, my GPU utilization goes around 45% and in Python, at Linux OS it was around 90%!
now the question is, do you recommend me reInstall my Matlab on Linux OS and then i can use more from my hardware resource?