Two tesla c2050 + quadro 600 - GPU error. Ideas ?

Hello, I've got a remote cluster with 2 teslas c2050 inside (compute capability 2.0) + quadro 600 for OS purposes. All drivers are the newest from NVIDIA. CUDA toolkit - the latest one 3.2. MATLAB R2010b x64 + PCT + MDCS. Parallel computing test works fine. win7 x64 + 16Gb of RAM. This machine is public one, so some other software (CUDA compatible) is being tested on it. MATLAB is being run under system administrator. Some time ago it worked, now I have an error:

>> gpuDevice ??? Error using ==> feval An error occurred during: setting up the CUDA device. The CUDA error was: (Unknown code: 10100).

Error in ==> currentDeviceIndex at 7 idx = feval( '_gpu_selectedDeviceIdx' );

Error in ==> deviceProperties at 8 idx = parallel.internal.gpu.currentDeviceIndex;

Error in ==> GPUDevice.GPUDevice>GPUDevice.current at 34 [props, E] = parallel.internal.gpu.deviceProperties();

Error in ==> gpuDevice at 18 dev = parallel.gpu.GPUDevice.current();

Actually, I don't have an idea what to test or to check. Please help.

Answer 1

[Edric Ellis]

That error code usually indicates that CUDA is having a problem connecting to the device. Are you connecting to the remote machines using remote desktop? (If so, you'll need the latest Tesla-specific "TCC" drivers).

[Denis Zhegalin]

Thanks for your answer.

Is it correct that I'm not able to install this drivers on win7 (only win2008 R2 - in my case) and this solution wil not work with c2050 (supports only S and M series) ?

[Edric Ellis]

Do things work if you access the machines not through remote desktop?

[Denis Zhegalin]

It worked when machine was few meters far from me...Now it is located on another part of the city. And Yes, I connect via rdp.

[Edric Ellis]

Hm, I'm still not clear quite how things worked. What I do know is that to have CUDA work over RDP, you must use a Tesla driver and enable TCC. According to the driver release notes (http://us.download.nvidia.com/Windows/Quadro_Certified/260.61/260.61-Win7-WinVista-Tesla-Release-Notes.pdf) TCC is not the default for C2050 - but the release notes do tell you how to enable it.

[Denis Zhegalin]

Ok, I will try and give feedback here. thanks for help.

[Denis Zhegalin]

Current status is that drivers are correct, the latest ones with TCC. But I'm not able to configure it via RDP. Need to visit machine personally.

[Jason Ross]

If possible, you can save a trip to the machine by using VNC, which will not interfere with the CUDA driver in the same way RDP does.

[Najeeb Ahmad]

Hi, I have exact same problem. I am using Tesla C1060 as non-display compute only GPU. Operating system is Windows 7 and driver version is 263.06 WHQL. It is supposed to support TCC mode as per release notes but it needs nvidia-smi.exe utility to switch to TCC mode. Unfortunately can't find it anywhere on my computer. Is there some other way OR can you guide me where to find the utility? Thanks

[Jason Ross]

I find nvidia-smi.exe in "C:\Program Files\NVIDIA Corporation\NVSMI"

[Najeeb Ahmad]

I don't see NVSMI folder in NVIDA Corporation. The folders there are

3D Vision

Control Panel Client

Display Drs

Installer2

and a file named license.txt.

Can you help? Thanks

[Jason Ross]

You might want to try downloading the developer driver from the CUDA downloads page. That's what we have and we definitely have the utility.

http://developer.nvidia.com/object/cuda_3_2_downloads.html

[Najeeb Ahmad]

Thanks a lot for your reply.

As per your advice, I downloaded and installed developer driver for Vista and Windows 7 from the link you provided. The file name was devdriver_3.2_winvista-win7_32_263.06_general.exe. However, I still have the folders I mentioned in my previous reply and no NVSMI folder. Tried running 'gpuDevice' command and still get the same error:

??? Error using ==> feval

An error occurred during: setting up the CUDA

device. The CUDA error was: (Unknown code: 10100).

Error in ==> currentDeviceIndex at 7

idx = feval( '_gpu_selectedDeviceIdx' );

Error in ==> deviceProperties at 8

idx = parallel.internal.gpu.currentDeviceIndex;

Error in ==> GPUDevice.GPUDevice>GPUDevice.current

at 34

[props, E] =

parallel.internal.gpu.deviceProperties();

Error in ==> gpuDevice at 18

dev = parallel.gpu.GPUDevice.current();

Am I doing something wrong? Thank you for your co-operation.

Answer 2

[Najeeb Ahmad]

The problem was with Windows 7 32-bit as it doesn't support TCC driver. I installed Windows 7 64-bit and associated TCC drivers for Tesla C1060 and don't get the error anymore. Moroever, now I do have NVSMI folder in its rightful place but it never required me to run nvidia-smi.exe utility to switch driver into TCC mode which indicates that driver is in TCC mode by default.