How to efficiently parse flat file into 3-D array?

3 Ansichten (letzte 30 Tage)
Andria Bilich
Andria Bilich am 30 Aug. 2018
Kommentiert: Rishi Binda am 4 Sep. 2018
I am processing large flat files of data; I have attached a small sample. The files contain no headers, so are straightforward to read with a 'load' command. In this example, the last 3 columns are my data, and the data have two independent variables: time (columns 1 & 2 are modified Julian date) and satellite number (column 3). Note that the data columns may contain NaN when data are missing. I actually need to work with 4 of these files at a time, and each of them have data with time and satellite number as independent variables. For a typical dataset, each file would have 150K+ lines spread out over 36K+ timetags.
To facilitate matrix math and also time- and satellite-matching operations at the processing phase, I want to load these data into a 3D array, with dimensions:
  1. time
  2. satellite
  3. data
I'm currently looping through the timetags, but I'm confident this isn't the most efficient way to accomplish my dataload. On my PC, looping through 25K timetags takes well over 3 minutes.
Here is a simplified version of the current code, where I'm working with only one of the 4 files. I preallocate storage with NaNs, because at a later stage I difference the 4 arrays and want operations such as 'number minus NaN equals NaN' to work out.
hk = load('antr_24118_1kLines.txt');
tk = datetime(hk(:,1) + hk(:,2)/86400, ...
'ConvertFrom','modifiedjuliandate');
ttag = unique(tk);
sats = unique(hk(:,3));
sz = [length(ttag) length([1:max(sats)])];
PhaseK = ones([sz 3])*NaN;
for i = 1:length(ttag)
% find all obs at ttag
q.k = tk==ttag(i);
% store station data into Phase arrays
PhaseK(i,hk(q.k,3),:) = hk(q.k,4:end);
end
So, how can I parse the data and store them into my 3D arrays more quickly? Can ya'll spot other places where I could be more efficient?
As you could guess, the data can potentially be sparse, with satellites coming and going, and the occasionally missing data at epochs, or entirely missing data types. So I started looking into sparse arrays, but the documentation looked like 'sparse' cannot do greater than 2D arrays and I wasn't sure how my desire for NaN differencing operations could work with sparse matricies.
  1 Kommentar
Rishi Binda
Rishi Binda am 4 Sep. 2018
You can have look at this link. It talks about generating 3-D sparse matrix from a large 2-D sparse matrix but the answer suggests using cell arrays.

Melden Sie sich an, um zu kommentieren.

Antworten (0)

Produkte


Version

R2017b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by