CUDA_ERROR​_ILLEGAL_A​DDRESS

I am trying to follow the instruction about using neural networks on GPU:

The graphic card is GIGABYTE GTX 1070 mini ITX with the latest driver. The Matlab is 2016a. The CUDA ToolkitVersion reported by Matlab is 7.5.

1. gpuDevice(1) call takes 4 minutes to complete. Each call. Later I realized that actually any GPU call after Matlab start causes NVIDIA to rebuild its cache in ...\AppData\Roaming\NVIDIA\ComputeCache\. This is what takes so long. A remark: only the first call of train(net1,x,t,'useGPU','yes') has this overhead, the subsequent calls are fast. But gpuDevice(1) forces the NVIDIA cache rebuild always.
2. A call to train(net2,xg,tg) stops with error: "Error using gpuArray/gather.An unexpected error occurred during CUDA execution. The CUDA error was:CUDA_ERROR_ILLEGAL_ADDRESS". And this is end of the game. After a train call with arguments prepared by gpu2nndata no other calls to GPU can be made. Not even gpuArray(). All calls end with some error. Matlab must be restarted (and NVIDIA cache rebuild) to be able to use train with useGPU flag at least.

Anybody has similar experience? Or does anybody has a clue what's going on and if this is solvable? Thank you.

[Dalibor Knis]

AN Update: I can confirm that the first part of the issue, cache rebuild, was solved by defining Windows system variables:

CUDA_CACHE_MAXSIZE=2147483648
CUDA_CACHE_DISABLE=0

The cache folders size then grew from 256MB to 440MB. But the initial gpuDevice call takes now 4s, not 4 minutes. I found the solution in some other MATLAB Central question: parallel.gpu.CUDAKernel slow on GTX 1080

[mizuki]

GTX 1070 and 1080 have some problems with MATLAB R2016a and R2016b, which is probably caused by cuDNN library. I recommend you to use another GPU if you have.

[Dalibor Knis]

Right, it seems as the only viable option. Thank you.

Answer 1

[Joss Knight]

Both these issues stem from the fact that Pascal architecture cards are not fully supported by the current CUDA toolkit. This means that all kernels have to be compiled on the fly. This includes the 400+MB of NVIDIA and MathWorks kernels in MATLAB.

The second issue is the first time I've heard of an actual runtime error on a GTX 10xx card when running cuDNN, but no doubt it's a manifestation of the same issue, see: https://www.mathworks.com/support/bugreports/details/1439741

The first issue can't be resolved until a future version of MATLAB, but the second issue will hopefully be resolved at some point with a patch.

[Joss Knight]

The fixes have now been published and can be downloaded from the bug report link. If you are using your own version of cuDNN you can download a patched version of cuDNN v3 and v4 from the NVIDIA download site.

[peng sun]

Hi Joss,

I have a titan x pascal running with matlabR16a. I tried to test the performance with:

b=rand([10000 10000]); bg=gpuArray(b); a=rand([1000 1000]); ag=gpuArray(a); for ii=1:1000 bg=sum(imfilter(bg, ag)); display(ii); end

then I will get the error like:

Error using gpuArray/rot90 An unexpected error occurred during CUDA execution. The CUDA error was: an illegal memory access was encountered

Is matlab 16 generally not supporting nvidia pascal card?

Thanks, hope you hear from you.

[Joss Knight]

There are significant issues, yes:

https://www.mathworks.com/matlabcentral/answers/309235-can-i-use-my-nvidia-pascal-architecture-gpu-with-matlab-for-gpu-computing

Answer 2

[David Jaquier]

Hi everybody, I had the same problem as Dalibor. I have a gtx 1070 (version 399.07) and Matlab R2018a. Last week the example from matlab "DeepLearningFasterRCNNObjectDetectionExample.mlx" run properly and today the program make an CUDA_ERROR_ILLEGAL_ADDRESS. I had to clean reinstall the new driver and the problem was solved.

I hope that's will can help somebody else.

Answer 3

[Nicao]

Hi everybody, I have the same problem as you. I got a K80 (driver version 384.130) with Matlab R2017a. I am trying to use a VGG19 network with large images (800*800 pixels) and I got this following message : "An unexpected error occurred during CUDA execution. The CUDA error was: an illegal memory access was encountered ", "CUDA_ERROR_ILLEGAL_ADDRESS " whereas there is still 5Go of free memory on the GPU.

The error occurs during the backpropagation.

According to vl_simplenn_display the data will just take 2Go from the memory.

I also try on a P100 with the same driver version (384.130) and I still have the problem.

I don't have any problem if I use smaller image (250*250 pixels).

Do you have any idea of what can be the problem ?