writing binary file segment by segment

hi,

i am wondering if i can carry out the following task in matlab:

i have a large multiband image that is >10 GB. since i cannot open the entire image due to limited memory, i want to open the image row by row, ie if my image is R rows by C columns by B bands, i want to open 1 row x C columns x B bands at any one time. i then want to carry out some mathematical analysis, which returns 2 values per pixel. i want to save the returned values to separate binary files, by writing them to the files row by row (again due to limited memory). this entire process is then looped over all the rows of the image, which means i will need some function/script to allow me to append data to existing files. at the end of the day, i just want to open the output files in another software.

can this be done? if so, pls let me know which functions i should work with. or even better, are there scripts on the File Exchange that allow me to do this?

thanks for any help.

Answer 1

[Ashish Uthama]

Some hints:

1. Use multibandread with the 'Offset' parameter inside a loop. (Also look at multibandwrite).

2. Use raw file access functions like fopen, fread and fwrite in a loop.

Air code: (assumes data is in ordered in 'bil' (look at multibandwrite doc)).

      fid_input  =  fopen('input.bin','rb');
      fid_output =  fopen('output.bin','wb');

      for rowInd=1:1000
          rowData = fread(fid_input, rowLength*NumBands,'precision');
          processedData = processData(rowData);
          fwrite(fid_output, processedData,'precision');
      end

      fclose the handles.

Answer 2

[bob bob]

hi Ashish,

thanks for your answer. i managed to come up with this:

--------------------------

x = xlsread('x-axis.xlsx');

outslope = fopen('slope', 'a');

outrsq = fopen('rsq', 'a');

for r = 1:100

    im = multibandread('image', [100,100,54],...
    'float', 0, 'bsq', 'ieee-le', {'Row','Range',[r,1,r]});

    for c = 1:100
        imrc = im(1,c,:);
        imrc = reshape(imrc,54,1);          
        imrc = strrep(imrc,0,NaN);
        imrc = imrc';
        gdrows = ~isnan(imrc);
        imrc = imrc(gdrows);
        xrc = x(gdrows);
        stats = regstats(imrc, xrc);
        slope(c) = stats.beta(2);
        rsq(c) = stats.rsquare;

    end

    fwrite(outslope, slope);
    fwrite(outrsq, rsq);
end

fclose('all');

---------------------------

however, the final output files (outslope and outrsq) only contain one value each (ie scalars) and happen to be 5 and 6 respectively, when in fact they should each be a 100x100 doubles array. any idea where i have gone wrong?

Answer 3

[Ashish Uthama]

Its hard for me to debug your code without the data (or dummy data) and without an idea on what the code is supposed to do.

Notes on the code:

1. Always check the return status of FOPEN to ensure success.

2. Look at squeeze to remove singleton dimensions.

3. strrep is for strings. You could use the following instead to replace 0's with NaN in a numeric array. (Not sure what you do next, i.e index using non-NaN's).

 imrc(iszero(imrc))=NaN;

You could try debugging the code, i.e set a breakpoint at the fwrite and inspect the variables to see if they are what you expected them to be.

Answer 4

[bob bob]

sorry for the lack of comments. anyhow, i finally got it to work, by changing the fwrite statement to:

fwrite(outslope, slope, 'float');

thanks for your help.