How would I manipulate the following functions to use a tall array?

Hello,

I'm having difficulty figuring out how to turn these functions to utilize tall arrays. The only examples online about using tall arrays I've found so far have been using NYC taxi data as examples, which doesn't help me at all. I can't find examples that go beyond multiplying two columns togethor, or getting the mean of a column. I'm working with a single column, no header array of data. I essentially need to apply of series of transformations on the data.

Any help is appreciated, thanks.

 % Say both files are 6,000,000,000 (6e9) data points long, representing a wave. 
 % I need to flatten with a linear fit every 400,000, then gather variance every 100,000
 Dir = 'C:/';
 Data{1} = 'File1';
 Data{2} = 'File2';

 for filestoread = 1:length(Data);
      PathFile = fullfile(Dir, Data{filestoread};
      fidi = fopen(PathFile);
      while ~feof(fidi)
           yraw = fscanf(fidi, '%f\n', 1200000);
           yflat = detrend(reshape(yraw, 400000, []));
           yflat = yflat(:);
           variance = var(reshape(yflat, 100000, []));
           dlmwrite(varASCII, transpose(variance), 'delimiter', '\t', '-append', 'precision', 16);
      end
 end

Answer 1

[Josh Meyer]

First you need to make a datastore that references the underlying files for each array (one datastore per array). Then you can convert the datastores into tall arrays with T = tall(ds).

While the movvar function (moving variance) supports tall arrays, the detrend function does not (the bottom of each doc page has an Extended Capabilities section that says whether it supports tall arrays). So to detrend the data, you would probably have to use the package function matlab.tall.movingWindow to operate on the tall array with a window function you write that does the detrending.