MATLAB Answers

What's the upper limit for how much data can be stored in a structure?

19 views (last 30 days)
Ibro Tutic
Ibro Tutic on 15 May 2017
Commented: Ibro Tutic on 15 May 2017
I am working with about 20 datasets, each with a size of about 2500x300. Is it possible to open these datasets, store them, and save off the resulting structure as a mat file without running into memory issues? This is my current code, and I run into an "Out of memory" error when I run it.
for i=1:length(fieldnames(dat))
a=fieldnames(dat); %files to work with
a=a{i}; %selects upper structure
a2=fieldnames(foldername); %foldername where .dat files are held
a2=a2{i}; %select which folder
for j=1:length(fieldnames(dat.(strcat(a))))
b=fieldnames(dat.(strcat(a))); %list of filenames in upper structure
b=b{j}; %select file to open
for k=1:length(dat.(strcat(a)).(strcat(b)))
filepath = strcat(foldername.(strcat(a2)),'\', dat.(strcat(a)).(strcat(b))); %sets file path
fid = fopen(filepath{k}); %opens file
fmt = repmat('%s', 1, 360); %reads file, text string delimter
C= textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
fclose(fid); %closes file
DLG.(strcat(a)).(strcat(b)){k}=C{1}; %writes data to structure
end
end
end

  4 Comments

Show 1 older comment
Ibro Tutic
Ibro Tutic on 15 May 2017
I can't call Dat.a. I need to turn the character array into a string before I can have the program be pointed to the correct file within the structure.
Guillaume
Guillaume on 15 May 2017
"I can't call Dat.a"
Indeed, the proper syntax is simply:
dat.(a)
No need for the strcat that does absolutely nothing.

Sign in to comment.

Accepted Answer

Guillaume
Guillaume on 15 May 2017
Your code is a bit confusing with all these apparently useless strcat (see comment to the question).
Anyway, to answer your question, you at least need 2500*300*20*8 bytes ~= 120 MB to store your datasets. Add to that the overhead of the structure fields (around 20*(numberofcharactersinfieldname + constant) and of the structure itself (not much).
Unless you've got very little memory on your computer or the data size is vastly different from what you've stated, it should all fit in memory.

  1 Comment

Ibro Tutic
Ibro Tutic on 15 May 2017
Just found a way to shrink the data sets even further, thanks for the reply, it was helpful.

Sign in to comment.

More Answers (1)

Jan
Jan on 15 May 2017
A cleaner version of the code:
fmt = repmat('%s', 1, 360); %reads file, text string delimter
datFields = fieldnames(dat);
folderFields = fieldnames(foldername); % "foldername" is a bad name for a struct
for i = 1:length(datFields)
a = datFields{i}; %selects upper structure
a2 = folderFields{i}; %select which folder
aFields = fieldnames(dat.(a));
for j = 1:length(aFields)
b = aFields{j}; %select file to open
filepath = fullfile(foldername.(a2), dat.(a).(b)); %sets file path
DLG.(a).(b) = cell(1, length(dat.(a).(b))); % Pre-allocate
for k = 1:length(dat.(a).(b))
fid = fopen(filepath{k}); %opens file
C = textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
fclose(fid); %closes file
DLG.(a).(b){k} = C{1}; %writes data to structure
end
end
end
I tried to move all repeated work before the loops, removed the useless strcat and there is still much potential to improve the readability. Fieldnames like "a" and "b" are not clear.

  0 Comments

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by