Error using gpuDevice call

I am trying to run vl_testnn for gpu in matconvnet ...I get CUDA error if I try to access the device using gpuDevice I get the same error below. There is nothing wrong with my installation as I run other project using tensorflow, cafe and others that use the gpu and cudnn just fine. This seem to be only happening for matlab. Any ideas on how to resolve? I am on a nivida gpu under ubuntu16.

gpuDevice(1)
Error using gpuDevice (line 26)
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_UNKNOWN

[Walter Roberson]

Which MATLAB version are you using? Which Cuda drivers and toolkit do you have installed? I suspect that you have Cuda 9 installed, which is not supported in any released MATLAB version.

[Richard Nunziata]

Cuda 8.0 with cudnn 6.0 and cudnn 5.1 I point at cudnn 5.1 when I compile matconvnet. The gpu is a nivida gtx 1060. I tried running the following form nivida for mathlab and get the same error so I will take this to them.

https://devblogs.nvidia.com/parallelforall/deep-learning-for-computer-vision-with-matlab-and-cudnn/

[Walter Roberson]

Which MATLAB version are you using?

[Richard Nunziata]

I am using R2017b..trail download. Nivia does not seem to have a support site for matlab all there support links take me back here. I am kind of lost as to how to analysis this issue.

nvidia-smi
Tue Jan  2 14:58:53 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.66                 Driver Version: 384.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1060    Off  | 00000000:01:00.0  On |                  N/A |
| N/A   46C    P8     8W /  N/A |    304MiB /  6064MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1009    G   /usr/lib/xorg/Xorg                             177MiB |
|    0      1722    G   compiz                                          44MiB |
|    0      2192    G   ...el-token=8D68399EE953F378B0E8CE015F0B51AA    79MiB |
+-----------------------------------------------------------------------------+

[Walter Roberson]

R2017b should accept CUDA 8.0 and the *1060 series, so I do not have suggestions at the moment (other than contacting Mathworks for installation support)

[Richard Nunziata]

test still fails with cuda device not found

./deviceQuery.NotSet 
./deviceQuery.NotSet Starting...

   CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 30
-> unknown error
Result = FAIL

[Richard Nunziata]

problem resolved after reinstall cuda drivers and related packages

[Joss Knight]

Just to clarify Walter, MATLAB R2017a and R2017b support CUDA 8, CUDA 9, and any future versions of CUDA. MATLAB is always forward-compatible to new CUDA versions.

[Walter Roberson]

Joss, that contradicts quite a number of postings. For example discussions of the 1060 series it was made clear that CUDA 8 was needed to run those devices properly but that on the other hand at the time no MATLAB supported CUDA 8 and the people needed to have CUDA 7 installed to talk to the device at all.

[Petter Stefansson]

Yes I can confirm that Matlab R2017a does not support CUDA 9. As soon as you install it and attempt to compile something you get the following warning:

Warning: Version 8.0 of the CUDA toolkit could not be found. 

No supported compiler or SDK was found

So Matlab 2017a explicitly says in its warning messages that it needs toolkit 8.0

[Joss Knight]

There is a lot of confusion here, understandably, so it's worth being clear.

Applications like MATLAB are built with a version of the CUDA compiler and libraries that comes with the CUDA toolkit. These applications (if built correctly) are forward compatible to any future CUDA architecture using the JIT compilation pipeline through the driver. This pipeline has its limitations - the one-off compilation takes time and the resulting binaries may not be well optimised. MATLAB ships all those libraries in its installation so if you install a new CUDA toolkit it makes no difference to MATLAB, only to the things you compile with it.

New device architectures require new drivers that are themselves labelled with a new version of the CUDA toolkit, because they are built using it. This is what people mean when they say a card 'requires' a particular CUDA version, they mean the driver, not the toolkit. These drivers work with older cards and with applications built with older CUDA toolkits. Occasionally, there are problems with the forward/backward compatibility. For instance, the cuDNN library has twice failed to support new device architectures. These problems are bugs - the expected behaviour is that a version of MATLAB will work with any new card as long as you have up-to-date drivers.

Where future versions of the CUDA toolkit are not supported is for compiling MEX functions, that is all. And even that is just a precaution to avoid binary incompatibilities. In fact I have several times posted here the workaround that allows you to compile MEX functions using a different toolkit from the one MATLAB was built with. Sometimes it's useful if a new toolkit brings support for your latest host compiler.

Anywhere where someone has explicitly posted in this forum that MATLAB doesn't yet support some future CUDA version, then either they are mistaken, or they are referring to a bug. It's not even particularly meaningful because it's new cards that sometimes have limitations due to not being properly supported by the JIT pipeline, not new 'CUDA versions' (which come with drivers, and with applications like MATLAB, not with cards). The most honest thing to say would be to say a new GPU architecture (not CUDA version) is not fully supported.

Case in point, here is me using a Pascal GPU in R2016a:

                                 < M A T L A B (R) >
                       Copyright 1984-2016 The MathWorks, Inc.
                       R2016a (9.0.0.341360) 64-bit (glnxa64)
                                  February 11, 2016

>> gpuDevice

ans =

CUDADevice with properties:

                      Name: 'TITAN Xp'
                     Index: 1
         ComputeCapability: '6.1'
            SupportsDouble: 1
             DriverVersion: 9
            ToolkitVersion: 7.5000
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 1.2782e+10
           AvailableMemory: 1.2546e+10
       MultiprocessorCount: 30
              ClockRateKHz: 1582000
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 0
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

>> gpuArray.rand(5) * gpuArray.rand(5,1)

ans =

    1.5732
    2.0223
    3.2198
    2.0951
    1.4465