Code Not working...sort of

I have a very long script that works, but not all the way through. When I run the non working part on its own, it works just fine. I have no idea what is going on and why it doesn't work when I press run, but works when I copy and paste the script into the command window. I don't get any errors. I just don't get the plot that I want, which is a change of atmospheric pressure over time. I tried setting the xlimits, but it doesn't let me in a scatter plot. It would also be a high inconvenience as I need to run the script for multiple months over multiple years.

I have attached my script as it is rather long and an example of the type of files I am working with. There are several hundred thousand that will need to be processed. In the script they are not text files, but I needed to make it one to attach it here.

I have also attached two image files, one showing the incorrect graph in the last subplot and the other showing the correct graph.

If anyone can help me to figure out what is going on, I would appreciate it. I am still new to matlab and I am not completely versed in why the graph is created incorrectly when I press Run, but not when I copy and paste it into the command window.

[C G]

In the script, I create two different structures. I did this because the first two scripts like the z1Data structure and fail when I add an extra column. The second script likes the p1Data structure and also fails when I add an extra column.

[dpb]

The problem with the third plot is the x-axis range...all the data are there the xlim values are such it's all visually compressed as the range is 1800-2000 instead of 1990.x

Use debugger and see where you didn't set xlim correctly.

As far as the issue between the two conditions; again use the debugger from the script and work your way through to see what goes wrong where.

"I tried setting the xlimits, but it doesn't let me in a scatter plot."

That's simply not true; there's nothing unique about scatter plot axes but you don't show where you made any call to xlim to see what the issue might be in your syntax.

Looking at the data file it appears the first portion is a date vector but there are only nine times given but there are many more points than that on the plots -- where's the rest of the data come from or did you just truncate a file to illustrate format?

[jonas]

When I run the non working part on its own, it works just fine. I have no idea what is going on and why it doesn't work when I press run, but works when I copy and paste the script into the command window.

Try adding drawnow to update the figures after each input.

[Image Analyst]

I can't even run your code because you didn't give code to define pathdist, decyear, kmmetersTraveled1, and maybe some other variables. Please include all code needed to run it.

It made me chuckle when you said your 72 line long code was "very long". You should see my programs. A typical program is several thousand lines of code.

[dpb]

Use the debugger...stepping through will likely uncover the cause of the difficulties.

[Adam Danz]

Learning how to debug in matlab will solve these types of problems. Here's a tutorial and there are several videos out there you can search for, too.

https://www.mathworks.com/help/matlab/matlab_prog/debugging-process-and-features.html

When in debug mode, it's helpful to use these shortcuts:

• F9: run the section that is highlighted
• F10: run the next line of code
• F11: run the next line of code and step into the function it calls
• F5: Run the code until the next break point or until the end; also, start running the code.
• F12: toggle (on/off) a break point at current line of code.

[C G]

Thank you for all of the help! I will try the debugger program to see what is going on.

I didn't truncate the attached text file, just the code. I figured if I can get it to work once on one month, then it will work for the others. As for the file, this is how each shows up. I only attached one as there are actually 8760(minimum) of these files. They are the output from another program, so cannot be changed. So, think of it like this. There are 365(6) days in a year and 24 files per day. That adds up to a lot of files.

I will try drawnow. Would that need to go in the loop or just before?

[dpb]

for i = 3625:4344 %Set this to the number of files you want to import.
  C = importdata(tdumpnamesOneHr(i),delimiterIn,headerlinesIn);
  z1Data = [z1Data C.data(:,3:12)];
end

This looks pretty risky to me...first of all having to fixup the loop limits and then where's the tdumpnamesOneHr array come from?

Using dynamic reallocation to "grow" the array is exceedingly overhead-intensive; at least allocate it to the size based on the loop limits.

Also, the sample data file has a count record at the beginning but the 'headerlines' is fixed and ignores that inside the file; is it absolutely inviolate that there are always 9 of those initial records?

Even if you're going to do this, you've made things much more difficult indexing-wise by concatenating horizontally on one long record instead of vertically to make each

Also, what's the 3:12 subset; what variable is what column and what do the two records represent?

My initial best guess would be that somewhere in here is where things go wrong in that all the assumptions made don't hold up in court for every case.

[C G]

I agree, having to import the files like that is annoying at best. But, it was the best way I could come up with to call up just one month's of files.

Anything with 'tdumpnames' in the variable refers to a list of the file names being used for that particular run. I have attached a small sample of it. Each file name has the same general format, tdumpYRMODYHR. I know tdump90...isn't shown. The last year I was working with was 1988, so that is where the sample came from. (FYI: I have to do this from 1950-2017.)

In the text files, the first 9 lines of header are a part of each and every file and are dictated by an outside program (not MATLAB). They can have less, but that will change the output of the data. They are telling you which data files you used to generate the data in the text file. Short answer, they must stay.

In each and every text file, there are 22 columns of different data. For example, column 9 is hours back and columns 10 and 11 are latitude and longitude, respectively. '3:12' is calling up columns 3-12. The first 6 (3:8)are the time/date of each line of data. Then the hour back and latitude and longitude.

It gets more fun because after that you have elevation, pressure, potential temperature, actual temperature, humidity,...

If there is a way to import the data once and then use it monthly, that would be great. The only problem is the length of each run would have to be manually set as not every run has the same number of days. Example, January has 744 files, where February only has 696 in a leap year.

[dpb]

You're not really answering the fundamental question -- what's the file naming convention--you should be able to get the desired files for whatever dates wanted simply by giving that date range and then retrieving the correct files by building the file name range.

I'm fully aware the header lines will be there; the issue is as you say "They can have less, but that will change the output of the data" which is exactly the issue I was asking about; you need to read that initial record to find out how many header lines are actually in the file.

Again, I understand what 3:12 indexing is; the question was what are the data in the file because the header doesn't seem to make any sense as compared to what the values are on a 1:1 basis. I guessed the date vector off first six; the question was what are the others and then what is the actual end result desired to do with them. Altho it's peculiar if the date vector is actually yr,mo,da,hr,mn,se that they're in reverse order in the sample file as the first would be 10:00:00 and the second 9:00:00 unless there's an issue that they wrote the file as AM/PM and didn't encode that minor piece of data as to which it actually is and so that is 10AM and 9PM. That's a real piece o' work if that's the case...some like this you could infer if T2<T1 but others might be totally indeterminate, depending.

Since that is the way the data are organized as I suspected, concatenating horizontally instead of vertically makes no sense at all.

As far as the last comment, I'm positive there's a much easier and simpler way to read the files reliably; it revolves around how they are named and if that naming was reasonable such that they can be identified from the date wanted; there's no issue in determining how many days are in any month of any year as far as the number; there's no reason to have to manually hardcode that in and change the function itself.

What is needed is the actual base problem definition from which to develop the problem specification and then the solution can be coded.

[C G]

I have added a few descriptions to the text file. I hope this clears things up a bit.

Basically, the underlying problem is to plot a map of each day's trajectories. For example, in the file attached, the start point is the latitude and longitude of line 14 and the end point is the latitude and longitude of line 15. In addition, along the way, over time not space, I would like to track the changes in distance traveled, speed, pressure, and temperature (for now). The mapping I have down. The plotting no so much.

The layout of the text (not text files in real life) cannot be changed. All of the data will be from lines 14 and 15, columns 3-22. The actual name of each data represented is irrelevant since they are all numeric values. I included them in the text file in shorthand. Line 13 also contains the names as they are produced by the outside program.

The dates go YR,MO,DY,HR,MN,SC, are in UTC, and do not contain AM/PM. They are in a 24 hour clock. MN and SC are always 0. Line 14 always has a larger date than line 15. Line 14 reads 7 June 1990, 1000. Line 15 reads 7 June 1990, 0900. These are back trajectories so they need to go backwards in time. This cannot be changed as it is impossible to know where the trajectory is coming from before reaching my start point. The start point never changes.

Vertically or horizontally... makes no difference to me as long as I can plot the data.

I think though we are straying from the original question, which was why doesn't it work when I press run and not when I run just the problem section on its own.

[dpb]

" why doesn't it work when I press run and not when I run just the problem section on its own."

Don't be so defensive on the file structure; I'm not trying to change anything but have to understand it well enough to be able to write code to parse it reliably and part of that is knowing just what it is in the file/data that needs be treated, and how.

Undoubtedly related to the data and the code and the context but can't solve one without knowing how to do the other correctly. When that's done, it'll all work all the time automagically provided all the variables have been correctly accounted for.

You also have to recognize that we only know what we're told here, we haven't worked on the problem a priori so don't know all the details and can't tell what is and what is (or may) not be important. Only ascertaining such leads to nirvana in the end...for instance, how were we to have any idea about reversed time w/o asking Q???

[C G]

I wasn't trying to be offensive, just trying to explain it in detail. It was not meant to be anything other than detailed. I cannot get pathdist.m to attach and the file is quite long. It can be found in the Antarctic Mapping Toolbox by Chad Greene. Antarctic Mapping Tools

I still haven't figured out why it wasn't working, but did find a work around. If I just use the original t1data instead of importing new files, it works just fine. Maybe the glitch had something to do with creating two structures in the same script?

[dpb]

Glad to hear..."real world" interfered before I got back to try to actually write code, sorry.

I can't actually run the full code even if had; the TB requires the Mapping TB I don't have.

I'm still pretty sure there was/is a clear reason, just didn't have the time to get far enough along to actually uncover it...if it pops up again, repost and somebody if not I is bound to come along again.

[dpb]

Understand on the intent...just weren't answering the questions trying to get to the bottom of the "how and why"... :)

One thing from the attaching the file-naming description and example of the files list; it's trivial to retrieve the files for any year or year/month or year/month/day simply by giving the date information

yr=90;
mo=2;
fnWild=num2str([yr mo],'tdump%02d%02d*%10');  % filename wildcard string
d=dir(fnWild);                           % retrieve those matching
for i=1:length(d)                        % iterate over the list
  data=importdata(d(i).name, ...         % retrieve each and process in turn

I've got meetings in town again at the moment but that logic would help at least some in the repetitive fixups. I'll have to think a little more about how to build a range; of course to read everything you have on hand is trivial, just use

d=dir('tdump*');

[C G]

I think the last bit of your for loop was cut off.

[dpb]

That was just showing a structure by which you can read the data without the counting by using the file naming convention to select files...as noted, meetings interrupted precluding more details at the time...

[C G]

I tried your code and got this error message.

 Warning: Invalid format. 
> In num2str>cellPrintf (line 174)
  In num2str (line 121) 
Index exceeds matrix dimensions.

What comes after

 data=importdata(d(i).name, ...

[dpb]

Well, would need to see the actual code and error in context to decipher the above; that's the lower-level error in the ML routine, not the top-level user code.

I had meetings out of town that took all of today; mayhaps tomorrow I can try to flesh that out a little...basically it carries on from there with your code as starters, all I did was to replace having to manually count and build file names manually by using a wild card string to return the directory structure contaning the list...I ran out of time at that point...

[C G]

 fnWild = num2str([yr mo],'tdump%02d%02d*%10');
Warning: Invalid format. 
> In num2str>cellPrintf (line 174)
  In num2str (line 121)

[dpb]

Ah, a typo...I shoulda' pasted instead of re-typing. The image of the file names list has the trailing "%10" string; is that real? The above assumed it was and might be significant in selecting a specific subset of files. But, to incorporate a "%" in a format string as a literal it has to be "doubled up" and I missed doing it there...

>> yr=90;mo=2;  % random yr, month for demo
>> fnWild = num2str([yr mo],'tdump%02d%02d*%%10');
>> fnWild
fnWild =
  'tdump9002*%10'
>>

If that isn't a significant piece or isn't really part of the file names (your attached sample file didn't have such) then you can just stop with the wildcard "*"

fnWild = num2str([yr mo],'tdump%02d%02d*');

It also assumes there's not a file extension but perhaps from the example attached there should be a '.txt' added on the end?

fnWild = num2str([yr mo],'tdump%02d%02d*.txt');

[C G]

The % sign is both a typo and a way of designating the files by group. In my original windows command line script, there is a variable defined %HT%. When the variable is created by the computer it skips the first % but when it generated the name of the file, it kept it in. It was easier to leave it then figure out the problem.

The files are indeed without extension. What comes after this?

 for i=1:length(d)                        % iterate over the list
  data=importdata(d(i).name, ...         % retrieve each and process in turn

[dpb]

A. OK that 'splains the file naming. The % while unorthodox may not be a bad thing to have in there at that point to make visualization a little easier--having two more digits tacked onto the base name would really make it hard to scan visually. You could, of course, just use REN to do a substitution to make it go away...

B. I'm working on it... :) I have yet another meeting this evening here in a few minutes and had to spend the day catching up on farm work after rain and being gone yesterday so "all's been quiet on this front" for most part...

I did download the toolset from FEX so at least should be able to run the functions for the file or two that do have when get that far along...

[C G]

No worries on time. I am slowly working through my code and getting stuff that looks fine.

The naming convention has been a blessing. I have 5 %HT that I am working with. It makes my life easier if I accidentally mix up the variables, I can easily sort through them. Yay! Happy accident!

Answer 1

[C G]

delimiterIn = ' ';
headerlinesIn = 13;
% Create an empty Table that will hold ALL imported data
z1Data = [];
% Use 'for loop' to import all the data into one place
for i = 3625:4344 %Set this to the number of files you want to import.
    C = importdata(tdumpnamesOneHr(i),delimiterIn,headerlinesIn);
    z1Data = [z1Data C.data(:,3:15)];
end

   %Now let's loop through all of the kmmetersTraveled.
  kmmetersTraveled1 = pathdist(z1Data(:,8),z1Data(:,9),'km');
  DateNumber = 1900 + (decyear([z1Data(:,1),z1Data(:,2),z1Data(:,3),z1Data(:,4),z1Data(:,5),z1Data(:,6)]));
  figure
  subplot(2,2,1)
  sz = 15;
  scatter(DateNumber(2,:),kmmetersTraveled1(2,:),sz,'b.','filled');
  box off;
  xlabel('time')
  ylabel('kilometers traveled')
  hold on;
  for j = 14:13:size(z1Data,2)
      DateNumber = 1900 + (decyear([z1Data(:,j),z1Data(:,j+1),z1Data(:,j+2),z1Data(:,j+3),z1Data(:,j+4),z1Data(:,j+5)]));
      kmmetersTraveled = pathdist(z1Data(:,j+7),z1Data(:,j+8),'km');
      scatter(DateNumber(2,:),kmmetersTraveled(2,:),sz,'b.');
   end

   %Now calculate and plot speed in m/s.
  subplot(2,2,2)
  speed_1 = (((kmmetersTraveled)*1000)/3600);
  DateNumber = 1900 + (decyear([z1Data(:,1),z1Data(:,2),z1Data(:,3),z1Data(:,4),z1Data(:,5),z1Data(:,6)]));
  scatter(DateNumber(2,:),speed_1(2,:),sz,'b.','filled');
  box off; 
  xlabel('time')
  ylabel('speed (m/s)')
  hold on;
  for j = 14:13:size(z1Data,2)
      DateNumber = 1900 + (decyear([z1Data(:,j),z1Data(:,j+1),z1Data(:,j+2),z1Data(:,j+3),z1Data(:,j+4),z1Data(:,j+5)]));
      kmmetersTraveled = pathdist(z1Data(:,j+7),z1Data(:,j+8),'km');
      speed_1 = (((kmmetersTraveled)*1000)/3600);
      scatter(DateNumber(2,:),speed_1(2,:),sz,'b.');
  end

   %Plot Atmospheric Pressure
  subplot(2,2,3)
  DateNumber2 = 1900 + (decyear([z1Data(:,j),z1Data(:,j+1),z1Data(:,j+2),z1Data(:,j+3),z1Data(:,j+4),z1Data(:,j+5)]));
  %scatter(DateNumber2(1,:),z1Data(1,11),sz,'r.'); hold on;
  scatter(DateNumber2(2,:),z1Data(2,11),sz,'b.');
  box off; 
  xlabel('time')
  ylabel('hPa')
  hold on;
  for j = 14:13:size(z1Data,2)
      DateNumber2 = 1900 + (decyear([z1Data(:,j),z1Data(:,j+1),z1Data(:,j+2),z1Data(:,j+3),z1Data(:,j+4),z1Data(:,j+5)]));
      %scatter(DateNumber2(1,:),z1Data(1,j+10),sz,'r.'); hold on;
      scatter(DateNumber2(2,:),z1Data(2,j+10),sz,'b.');
  end

   %Plot Temperature.
  subplot(2,2,4)
  DateNumber2 = 1900 + (decyear([z1Data(:,j),z1Data(:,j+1),z1Data(:,j+2),z1Data(:,j+3),z1Data(:,j+4),z1Data(:,j+5)]));
  scatter(DateNumber2(2,:),z1Data(2,13),sz,'b.');
  box off; 
  xlabel('time')
  ylabel('Degrees (°C)')
  hold on;
  for j = 14:13:size(z1Data,2)
      DateNumber2 = 1900 + (decyear([z1Data(:,j),z1Data(:,j+1),z1Data(:,j+2),z1Data(:,j+3),z1Data(:,j+4),z1Data(:,j+5)]));
      scatter(DateNumber2(2,:),z1Data(2,j+12),sz,'b.');
  end