Problem to access OPeNDAP data behind https:// urls

Hi!

I'm trying to access an OPeNDAP server with the https:// handle. I have the R2019b version of matlab. As far as I understand, matlab can't access the file directly because of security reasons(?). But it seems as though sometimes my code works and extracts the variable, sometimes not. See here

>> m=1
chl_A(m)=ncread(url, 'chlor_a', [lonnumber(m)-1,latnumber(m)-1,timeM(m)-1],[1 1 1]); %this code extracts data
>> m=2
>> chl_A(m)=ncread(url, 'chlor_a', [lonnumber(m)-1,latnumber(m)-1,timeM(m)-1],[1 1 1]); %this does not
Error using netcdflib
The NetCDF library encountered an error during execution of 'open' function - 'I/O failure (NC_EIO)'. If the data source was an  OPeNDAP URL,
see the OPeNDAP Troubleshooting section in the Users Guide.

Error in netcdf.open (line 67)
        [varargout{:}] = netcdflib ( 'open', filename, varargin{1} );

Error in internal.matlab.imagesci.nc/openToRead (line 1278)
            this.ncRootid = netcdf.open(this.Filename,'NOWRITE');

Error in internal.matlab.imagesci.nc (line 121)
                    this.openToRead();

Error in ncread (line 61)
ncObj   = internal.matlab.imagesci.nc(ncFile);
 

and basically matlab does this to the whole data. I have 158 points to extract and it only extracts about 30 of them, for the others the error comes up. Why?

This code used to work when the server was a http:// one. Is there any way to fix it or bypass it?

Thanks in advance!

[Rashed Mohammed]

Hi Sofya

Could you please provide the URL and Operating System you are using so that we can replicate the issue

[Sofya Kuzmina]

Yes, sorry:

url = 'https://rsg.pml.ac.uk/thredds/dodsC/CCI_ALL-v4.2-DAILY';

and i'm using Windows 10.

Hopefully that'll be enough now.

Answer 1

[Rashed Mohammed]

Hi Sofya

Thanks for the information. I was able to reproduce the issue and I have brought this issue to the notice of our developers. They will investigate the matter further.

In the meantime, can you try upgrading to the latest release of MATLAB and see if the issue persists as I was not able to reproduce the issue in R2020b.

[Sofya Kuzmina]

Thank you so much for your help, I will try upgrading later!

[Subhrajit Rath]

In 2020b i am also facing same issue.

url =http://nomads.ncep.noaa.gov:80/dods/gfs_0p25_1hr/gfs20210406/gfs_0p25_1hr_06z

temp=ncread('http://nomads.ncep.noaa.gov:80/dods/gfs_0p25_1hr/gfs20210406/gfs_0p25_1hr_06z','tmax2m');

Error using netcdflib

The NetCDF library encountered an error during execution of 'open' function - 'Malformed or inaccessible DAP

DATADDS (NC_EDATADDS)'. If the data source was an OPeNDAP URL, see the OPeNDAP Troubleshooting section in

the Users Guide.

Error in netcdf.open (line 67)

[varargout{:}] = netcdflib ( 'open', filename, varargin{1} );

Error in internal.matlab.imagesci.nc/openToRead (line 1278)

this.ncRootid = netcdf.open(this.Filename,'NOWRITE');

Error in internal.matlab.imagesci.nc (line 121)

this.openToRead();

Error in ncread (line 61)

ncObj = internal.matlab.imagesci.nc(ncFile);

[Barbara Bentz]

I am having the same issue with R2020a. It is not a problem with the site, because I can access the site with R code. Similar to Sofya, it was working, then all of sudden quit with the same error as above. Thanks.

[Barbara Bentz]

Here's the code and error (Matlab 2020a)

myVar = 'ppt';

%enter your terraclimate OPeNDAP URL

%find this here: http://thredds.northwestknowledge.net:8080/thredds/terraclimate_aggregated.html

myURL = ['http://thredds.northwestknowledge.net:8080/thredds/dodsC/agg_terraclimate_',myVar,'_1980_CurrentYear_GLOBE.nc'];

% enter your point locations as lat/lon pairs in the following arrays

latList=[41 44 45 41]; %list of north latitudes for each point location

lonList=[-110 -100 -101 -95]; %list of East longitudes for each point location

NUM_POINTS = length(latList);

%enter your time start and time end

year_start = 1958;

month_start=1;

year_end = 2017;

month_end=12;

%===================================================

timebounds_start =datenum(year_start,month_start,1)-datenum(1900,1,1);

timebounds_end = datenum(year_end,month_end,1)-datenum(1900,1,1);

timebounds = [timebounds_start timebounds_end];

lat=ncread(myURL,'lat');

lon=ncread(myURL,'lon');

time=ncread(myURL,'time');

[Y,M,D]=datevec(time+datenum(1900,1,1));

ftimei=find(time>=timebounds(1) & time<=timebounds(2));

time = time(ftimei);

NUM_TIME= length(time);

myData = NaN(NUM_POINTS,NUM_TIME);

myLat = NaN(NUM_POINTS,1);

myLon = NaN(NUM_POINTS,1);

for i= 1:NUM_POINTS;

[dist,flati]=min(abs(lat-latList(i)));

[dist,floni]=min(abs(lon-lonList(i)));

myLat(i)=lat(flati);

myLon(i)=lon(floni);

% to change variables, see list here: http://thredds.northwestknowledge.net:8080/thredds/terraclimate_aggregated.html

%dimensions of the data should be:

myData(i,:)=ncread(myURL,myVar,[floni(1) flati(1) ftimei(1)],[length(floni) length(flati) length(ftimei)],[1 1 1]);

end;

Error using netcdflib

The NetCDF library encountered an error during execution of 'open' function - 'Malformed or inaccessible DAP DDS

(NC_EDDS)'. If the data source was an OPeNDAP URL, see the OPeNDAP Troubleshooting section in the Users Guide.

Error in netcdf.open (line 67)

[varargout{:}] = netcdflib ( 'open', filename, varargin{1} );

Error in internal.matlab.imagesci.nc/openToRead (line 1278)

this.ncRootid = netcdf.open(this.Filename,'NOWRITE');

Error in internal.matlab.imagesci.nc (line 121)

this.openToRead();

Error in ncread (line 61)

ncObj = internal.matlab.imagesci.nc(ncFile);

[BN]

I have same problem in R2021a. Did you find any solution?

[Sofya Kuzmina]

Hello everyone. Based on my experience with different servers, this problem seems to just pop up every once in a while. I have since installed R2021b and it still occurs sometimes, albeit less often.

My only solution (@BN, thanks for the nudge, your reply was the only one that managed to get to my inbox!) would be:

a) give it some time, try again later. I know, not the most elegant solution, but if your desired data is not large, perhaps a try-catch loop can work.

b) find other ways to access your data, lol. For me the only problem has been with a specific dataset (now I think about it, it was quite a big one, while other .nc files I've accessed were smaller, so maybe that's part of the issue? idk), whereas other https:// servers worked fine. Never encountered this problem with an http:// server though, so I have no idea how one could fix that...

c) make sure your url is working! access it through your brщwser to check that it does exist on the server! @Subhrajit Rath and @Barbara Bentz's errors appear to be different to mine, so maybe that's a whole different issue.

Sorry I can't be of more help.