Find more time-efficient way to extract data from a .txt file

I have a code that takes over 4minutes to converge to a solution, and I need to get convergence about 30 times for synthesis purposes, resulting in over 2hrs of run time. I found that what is slowing my code down so much is the fopen function.

The files with the function name "intermediate_GW" and "intermediate_GW2" read a single data point with the format #####.# from a text file outputted from my gross weight solver (not listed above), and then, they output this value to the functions FF, DISK, and SW_TOTAL. "intermediate_GW2" stays the same per iteration of this WEIGHT_FUEL program while "intermediate_GW" changes per call. Is there a way for me to have the values from these two programs in a better format for this purpose? Below is a rough flowchart of how my programs are going to run. I can't have the TOTAL_WEIGHT program directly output to my other programs because this causes an infinite recursion, and I had to bypass that by using txt files.

If it could be done such that the below loop of intermediate_GW2, DISK, and SW_TOTAL can be called only once instead of 40000 times, this alone would drastically improve the runtime of my code.

[Mario Malic]

Is it possible to put this single data point with the format #####.# in to a variable and pass it to functions?

[Albert Garcia]

Doing this will cause infinite recursions because the WEIGHT_SIZING program requires the size of the wings and the rotors in order to calculate the weight of the aircraft, so it makes an infinite loop between WEIGHT_SIZING and SW_TOTAL/DISK

edit: WEIGHT_SIZING is a function of SW_TOTAL and SW_TOTAL is a function of WING_SIZING

[Mario Malic]

"intermediate_GW2" - if it doesn't change per iteration, then initialise it only during the first step of convergence.

I don't understand completely the program, but maybe you put this what you've drawn in this recursive piece of code and make some conditions so that it can break out when needed

[dpb]

If computing the value and writing it to a file can be done w/o infinite recursion, then the program can be constructed to output that to a variable without causing such.

We simply don't have enough code to be able to know how, specifically, to tell you to do so, but it most certainly CAN be done.

Having a flow chart will undoubtedly help you in determing the flow pattern to get where need to go, but without the actual code and data structures it's not possible to say just what to modify where.

[Albert Garcia]

You're right. I've been thinking about this, and I think all my problems arise because I just don't know how to properly structure my code. So this is what I want to do...

I have this main code, and I can have this program run quickly if I just had all my calculations within one .m file. However, I've been told that this is poor coding and that I should write my functions and such into different files so that it's easier to digest; this makes sense, and I wouldn't want to have thousands of lines of code in 1 .m file. This, however, leads to a very slow program. I guess what I really want is for my seperate functions to run as if they were embeded in this main code without having all this code in one program.

Below, I threw in my convergence code and one of the functions that this code needs.

This here is my major code that runs my other functions.

function [GW,W] = WEIGHT_SIZING(H,IT,x,y)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%   Function:   [W_total,W] = WEIGHT_ALLOCATION()
%   Program:    Weight Allocation
%   Input:      
%               H:          Altitude [ft]               -FOR USAGE IN STD_ATM
%               IT:         Sea Level Temperature [°F]  -FOR USAGE IN STD_ATM
%               x:          Gear Ratio Combination      -FOR USAGE IN TRANSMISSION
%               y:          Mission                     -FOR USAGE IN SW_TOTAL, FF, WEIGHT_FUEL
%   Output:     
%               W_total:    Sum of Component Weights [lbf]
%               W:          Component Weights [lbf]
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

W_ammo  = .32*2.205;            %Ammo Weight [lbf]
W_hefi  = 100;                  %Hellfire Weight [lbf]
W_PL_AR = W_ammo*500+4*W_hefi;  %Armed Recon Payload weight [lbf]
W_PL_AT = W_ammo*500+8*W_hefi;  %Attack Payload weight [lbf]
W_PL    = [W_PL_AR W_PL_AT];    %Payload Weight Conditional [lbf]

epsilon = 2;    %Used for convergence
i       = 1;    %Used for iterations
while epsilon > 1
    
    if i == 1   %Sets gross weight to guess gross weight
        GW  = 12180;     %Guess gross weight [lbf]
    else
	%Opens gross weight text file and pulls data from it
        file	= fopen('intermediate_GW.txt','r');
        GW      = fscanf(file,'%f');        fclose(file);
        GW2     = GW;
    %Gross weight calculations
        W_F             = WEIGHT_FUEL(H,IT,x,y);    %Fuel weight from WEIGHT_FUEL
        [SW]            = SW_TOTAL(y);              %Wing area from SW_TOTAL
        [HPmast,~,Qm]   = TRANSMISSION(x);          %Power and torque transmitted to mast from TRANSMISSION
        EW_GW           = .6776;                    %Empty weight to gross weight ratio from HS data
        W_E             = GW / EW_GW;               %Empty weight [lbf]
        Nr              = 2;                        %Number of rotors
        Lf              = 38.38;                    %Fuselage length [ft]
        HPeng           = 3000;                     %Engine power [HP]
        v_ktas          = [220 200];                %Forward flight dash speed [ktas]
        [~,Tmr]         = FF(H,IT,v_ktas(y),x,y,0); %Thrust of main rotor from FF

        %        1   2   3    4   5   6   7  8     9 
        W_SYS = [175 100 1050 700 525 100 20 143.6 15];

        W(1)  = 20.906 * SW^.601;                       %Wing
        W(2)  = 30.537 * (Tmr/1000)^1.042 * Nr^1.136;   %Main Rotor
        W(3)  = GW*.0114;                               %Tail
        W(4)  = 7.5986*Lf^.946*(Tmr/1000)^.636;         %Body
        W(5)  = .0127*GW^1.095;                         %Landing Gear
        W(6)  = .05418*HPmast^1.22;                     %Engine Section
        W(7)  = .01703*HPeng^1.053;                     %Air Induction
        W(8)  = 456*.8;%5.1403*HPeng^0.702;                     %Propulsion
        W(9)  = 46.1558*(Qm/1000)^0.921*Nr;             %Drive
        W(10) = GW * 0.0427;                            %Flight Controls
        W(11) = W_SYS(1);                               %Auxiliary Power Unit
        W(12) = W_SYS(2);                               %Instruments
        W(13) = GW * .0104;                             %Hydraulics
        W(14) = GW * .0333;                             %Electrical
        W(15) = W_SYS(3);                               %Avionics
        W(16) = W_SYS(4);                               %Armament
        W(17) = W_SYS(5);                               %Furniture and Equipment
        W(18) = W_SYS(6);                               %Air Conditioning
        W(19) = W_SYS(7);                               %Anti-Ice
        W(20) = W_SYS(8);                               %Fluids
        W(21) = W_SYS(9);                               %Load and Hand
        W(22) = W_E * .03;                              %Contingency

        W(23) = 400;%480;                               %Crew
        W(24) = W_PL(y);                                %Payload
        W(25) = W_F;                                    %Fuel

        GW  = sum(W);               %Calculating new gross weight
        epsilon = abs(GW-GW2);      %Difference in new and old gross weights to determine if reiteration is necessary
    end
    i   = i + 1;                    %Just to make i not equal to 1. 
%Opens gross weight text file and edits data for this iteration
    fileID = fopen('intermediate_GW.txt','w');
    fprintf(fileID,'%.1f\n',GW);
    fclose(fileID);
    fileID = fopen('intermediate_GW2.txt','w');
    fprintf(fileID,'%.1f\n',GW);
    fclose(fileID);
end

This here is a child function

function [Sw_w,b,CL,CD] = SW_TOTAL(y)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%   Function:   [Sw_t] = SW_TOTAL(y)
%   Program:    Wing Sizing
%   Input:      
%               y:          Mission
%   Output:     
%               Sw_w:       Total Usable Wing Area [ft^2]
%               b:          Total Span of Wing [ft]
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

[~,~,~,Den,~] = STD_ATM(4000,95);

GW  = intermediate_GW2;

v_ktas  = [220 200];        %Objective dash for AR and AT [ktas]

v       = v_ktas(y)*1.68781;%Design point velocity [f/s]
Lw      = GW * .2;          %Lift produced by wing [lbs]
q       = .5 * Den * v^2;   %Design point dynamic pressure [lbs/ft^2]
ao      = 2*pi;             %2-D Lift-Curve Slope 

AR      = 5;
a       = ao / (1+pi*ao/(pi*AR));

aoa     = 5*pi/180;         %Design angle of attack of wing [rad]
c_r     = 6;                %Root chord [ft]
w_f     = 6.25;             %Fuselage width [ft]
tr      = .5;               %Taper Ratio 

CL      = a*aoa;            %Lift coefficient
CD      = .00847;           %Drag coefficient
Sw_w    = Lw / (q*CL) ;     %Wetted area of wing [ft^2]
b_w     = Sw_w/(c_r*(1+tr));%Span of each wetted wing [ft]

b   = w_f + 2 * b_w;        %Total Wing Span
h   = c_r*(1-tr)/b_w*w_f;   %Height of Triangle

Sw_t    = Sw_w + c_r*w_f + 2*h*(w_f/2)/2;

[dpb]

What actually changes in the loop to allow convergence inside function WEIGHT_SIZING?

I'd guess if you recast the functional correctly, you could simply solve for W-W_tgt=0 w/ fzero or at worst, one of the nonlinear solvers.

At worst, you just save the last computed value and iterate...it's setting both W and W2 to the same value every pass that looks like is where the problem lies, but I'd have to be able to spend quite a lot more time and have the rest of the code this needs to really be able to say that for certain.

[Albert Garcia]

The values of GW, SW, W_F, and Tmr all change (calculated in the functions SW_TOTAL, WEIGHT_FUEL, and FF), but there are interrelations between these functions, too, as seen in the profiler screenshot above.

Well, I am saving the last computed value and iterating. I do set both W and W2 to the same value; the SW_TOTAL function has the wings sized to match the initial weight of the aircraft since the wings won't change size during flight, and this program along with others so use this W2 since it's meant to be a constant value. The W value is used for the FF program since it calculates how much power the aircraft needs to fly, and this W value decreases as fuel is burnt off, making the aircraft lighter, allowing it to fly at the same velocities using less power/faster with the same power.

As for the converging part, me setting it twice doesn't really do any harm. I have the W2 term, the weight calculated in the previous iteration, compared to the new GW term. Then, I go convert the data in both text files to the newest GW term so that the aircraft's geometries are sized appropriately for the weight, with one text file changing and the other remaining constant.

Yeah, I would have to restructure the whole code if I wanted to try fzero. I guess that's what I would have to do regardless.

[dpb]

"...with one text file changing and the other remaining constant."

That doesn't seem to be so...

  ...
  i=i+1;
  fileID = fopen('intermediate_GW.txt','w');
  fprintf(fileID,'%.1f\n',GW);
  fclose(fileID);
  fileID = fopen('intermediate_GW2.txt','w');
  fprintf(fileID,'%.1f\n',GW);
  fclose(fileID);
end

writes the same value to both files every iteration.

Presuming the

GW  = intermediate_GW2;

function is reading the file of that name, the new GW value in SW_TOTAL(y) is the same one just written.

In which case, just pass it (new GW) in the argument list and bypass the file.

When you get to the bottom and GW=sum(W), THEN you do have a difference again that you can test for convergence.

I can't see any reason for the file at all if you just pass the information as an additional argument to the functions that need it.

[Albert Garcia]

In the programs above, it doesn't show the GW file changing values within the first text file, but it does change values when it is used by other functions.

That's how I first tried it, having the main file pass the information to the beneath arguments, but I couldn't figure out how to do that without infinite recursions, so I ended up with this inconvenient solution.

Answer 1

[dpb]

While I'm convinced you really don't need the files at all and probably with only a minor amount of rearranging if it works as is, one really expedient workaround would be to not open/close the files every time...

function [GW,W] = WEIGHT_SIZING(H,IT,x,y)
  ...
  i       = 1;    %Used for iterations
  fID1 = fopen('intermediate_GW.txt','w');
  fID2 = fopen('intermediate_GW2.txt','w');
  ...
  while epsilon > 1
    ...
    i=i+1;
    frewind(fID1)
    fprintf(fID1,'%.1f\n',GW);
    frewind(fID1)
    frewind(fID2)
    fprintf(fID2,'%.1f\n',GW);
    frewind(fID2)
  end
  fID1=fclose(fID1);       % release file handles only when completed iteration loop
  fID2=fclose(fID2);
end

You then either add fID1, fID2 to the argument list to the functions that need them, or, (ugly, but expedient) make them GLOBAL variables.

[Albert Garcia]

I'll try these methods out. Thanks mate

[Albert Garcia]

Went with the global variable approach, and my synthesis code went from almost 300s to 10s. I believe I have found love, no matter how ugly global variables may be xD

[dpb]

If this is going to be an ongoing simulation code for the longer term rather than just a thowaway "get 'er done!" exercise, I'd suggest would be well worth looking at the overall structure again to see where the issue is that isn't being solved by keeping track of the two iteration weights.

While not opening and closing the file every pass is significant, I'd guess removing the write entirely would probably be close to another order of magnitude reduction in compute time.

[Albert Garcia]

Well, I will be using this program quite a bit for the next month, but I don't think I should invest more time in making this code better since it's good enough now. I do need to learn how to properly structure my code when I write it though; my terrible structure skills are definitely what have made me waste so much time fixing this code when I should've written it write the first time around.

[dpb]

Your call, of course. If it's fast enough this way, then perhaps there's no real justification for spending the time now to refactor it.