netcdf reading parameters "Error using double Out of memory. Type HELP MEMORY for your options"

Hi everyone ! I have an "out of memory" error while reading chlorophyl data from an netcdf file. The file contains :

• latitude
• longitude
• time
• CHL ( chl_a)

when I use ncread for latitude longitude and time , it works without any issue , but when I use the ame function for chlorophyl read, it shows the out of memory error.

I used this code :

ncdisp('dataset-oc-med-chl-multi-l4-interp_1km_daily-rt-v02_1541025401266.nc'); % to display the nc file
lon2 = ncread('dataset-oc-med-chl-multi-l4-interp_1km_daily-rt-v02_1541025401266.nc','lon') ; %to read longitude
lat2 = ncread('dataset-oc-med-chl-multi-l4-interp_1km_daily-rt-v02_1541025401266.nc','lat') ; %to read latitude
time2 = ncread('dataset-oc-med-chl-multi-l4-interp_1km_daily-rt-v02_1541025401266.nc','time') ; %to read time
chl = ncread('dataset-oc-med-chl-multi-l4-interp_1km_daily-rt-v02_1541025401266.nc','CHL') ; %to read chlorophyl , the error shows in this line

the file size is 955 Mo.

any ideas ? Thks in advance!

Answer 1

[Kelly Kearney]

What does the ncdisp command tell you about the variable sizes? Just paste the output of the following:

ncdisp('dataset-oc-med-chl-multi-l4-interp_1km_daily-rt-v02_1541025401266.nc')

This is probably a pretty straightforward error... you're trying to read a very big dataset on a computer without enough memory to hold it all. So you either need more memory, or you need to read the data in smaller chunks.

[Silver]

Thkans for your answer ! I tried to devide my file into a smaller size , and it works for me ! thanks for the answer , but now am having another issue.

Actually I have two kind of data :

• satellite data : netcdf files
• Ferrybox data : xlsx files

the satellite as you know are having data with this format 818x839x120 double : 3D

the ferrybox data form a 2D line like the photo below.

Actually I want to extract chlorophyle data from the netcdf file while the latitude and longitude of the atellite data correspond of those from the ferrybox file (xlsx file) . So the idea is to indentify the range of the coincidance points between long and lat from both files.

I used this code that does not work as I wish so :

 B=[]; %vector of range of similitude points
 % latitudebis :  latitude from ferrybox data file 
 % longitudebis : longitude from ferrybox data file 
 % m : latitude from satellite data file
 % n :  longitude from satellite data file 
 
 for b=1:818   % 818 is the size of the latitude and longitude
     if ((m(1,b)== latitudebis(1,b)) && (n(1,b)== longitudebis(1,b))) == 0 
         B (1,b) = b 
     end 
 end 

I will appreciate your help again !

[Kelly Kearney]

Well, first off, you say that your data is a 818 x 839( x 120) array, but then are assuming lat/lon dimensions of 818 x 818 in your loop... which is it?

But the main error in your logic here is that you are assuming there will be an exact match in your satellite grid coordinates to each of the trackline coordinates. That is very, very unlikely to be the case, unless the ferrybox data was subsampled from the satellite data.

What you really want to do is find the closest grid cell in the satellite product to each coordinate in the spreadsheet. (Or better yet, the closest 5-10 points, if you want a more robust average).

There are a few approaches you could use:

1. If your satellite data is higher-resolution than the observation dataset, and well away from the poles, such that the nearest-neighbor calculation will be approximately the same in either geographic or cartesian lon/lat space, you could try using interp2. Just be careful if you're close to the dateline or prime meridan that your longitudes are properly wrapped to either [0 360] or [-180 180] as appropriate.
2. If you're dealing with data with lots of pole crossings, or where the gridded dataset has a coarser resolution that the trackline data such that the nonlinearity of latitude and longitude becomes important, then you may want to look at pdist2 with a custom geographic-distance-based metric to build your own custom interpolant.

[Silver]

1. So you mean that I am going to find verry few points from the satellite data that I am going to interpolate so I could fine a line of satellite data that could be superposed to the FerryBox line ? is that what you mean ? If it is the case , I have to first compare both data and extract the coincidance and that's what I didnt find how to do it. could you be more exlicit in a code please ?
2. PS : while you are weeling with coordinate system , I want to ask you another question if you dont mind :) , in the photo I attached normally the line of the Ferrybox must be in the red line (as bellow) and not where it was located while the plot. The coordinate system of the satellite data is WGS 84 / Plate Carree (EPSG 32662). What can I do for the ferrybox coordinate to make it in the right location?

Thanks again!

[Kelly Kearney]

Regarding reading, yes, you'll need to read a certain subset of the data. I'd start by seeing if you can read in just the grid that falls inside the bounding box of your line data. Take a look at the START and COUNT inputs for ncread.

As for the offset coordinates, that really depends on how the data is actually stored in each file. Porjection and reference ellipsoid shouldn't matter if you're dealing with lat/lon coordinates. Is one of your datasets actually stored using projected coordinates (i.e. X/Y)? If so, you'll need to find out all the details of that projection to back-calculate the appropriate lat/lon coordinates and compare the two datasets.