Code runs very slow after making small changes

I have a script containing ~1500 lines that I run in sections as each section produces a different plot. If I run from scratch after clearing everything, each section runs as expected, taking 20-30 seconds each. However, if I make any change (e.g. something as small as changing the colours in a plot), it then gets stuck. Rather than taking 20-30 seconds, a section could take over 2 minutes. I realised this was occuring specifically after making changes, so I started running single lines of code to see what was happening. I would enter something simple like x=1; into the command window and this would take a minute or two to complete (which obviously is ridiculous). However, once I had done this, it was as if it had reset, and everything then started running normally again - until I made another change.

I've tried shutting Matlab down and restarting the computer, but the problem persists. It has been like this for about a month now, I've got into a habbit of typing x=1; into the command window after every change I make so I know whether Matlab is ready for me to run the actual script again. Sometimes it isn't too bad, and will only take about 10 seconds, but other times I can be waiting for 5 minutes.

For extra context, this only happens when I am working on scripts that use a fair bit of memory. The one I mention in this example reads in ~500mb of data and also does MCMC analysis with ~200,000 samples.

[Rik]

Perhaps this might be caused by paging to the harddrive? I would generally recommend strongly against using scripts for actual analysis. You might want to consider using functions. That way you also make it easier for Matlab to do garbage collection and avoid memory issues.

Without the script itself I can't go beyond this pure speculation, and even with the script I doubt I could help you.

[John D'Errico]

@Rik - you missed your chance. You should have predicted an error on some random line. Yes, well over 99% of the time you would be wrong, but, just imagine you guessed it right? This will immediately elevate you to the status of god of MATLAB. :)

Sadly, with a 1500 line script, as @Rik suggests, nothing can easily be done from our side. This script will likely be massively complex, and literally impossible to debug.

In turn, this strongly suggests doing as he has suggested. Learn to use functions. SMALL, modular functions. Now you can test each piece. Essentially, you need to learn to write better code. A very nice thing about functions is they work better with memory, releasing memory fragments you no longer need and returning only what is important. With a script, all of that useless baggage stays around forever.

What can you do without a massive rewrite, which is something I would STRONGLY recommend regardless? You can learn to use the profile tool. See what part of your code is taking the time. Which lines are causing the problems?

I might make a guess the issue is a memory one. Somewhere in your code, you have done something that creates some absolutely massive object. In order to procede, MATLAB is forced to deal with that massive boulder in its way. But that is only a guess. At least I'll predict it happens at or around line 537. If I'm right...

[Rik]

In that case I'm going to guess that on line 537 you made a small mistake: you created an array with A=ones(length(B));, which generates a square array, instead of the vector you might have meant (bonus points of using length where you should be using numel). Paging this array in and out is the source of this problem.

See John? THAT is how you do pure speculation ;)

[hg24]

Thank you @Rik and @John D'Errico for your responses. I appreciate that I could absolutely write the code better, but it does everything I want it to in a reasonable amount of time - it works! The problem isn't when running the code. It is not the code I have written that is taking too long, it is Matlab itself. After everything has run perfectly, one small change can make it take up to 5 minutes just to clear the command window. Therefore, I think the actual code itself is irrelevant (other than the memory that is used by reading in files etc.). Is this just a memory issue?

@John D'Errico, you mentioned using the profile tool to see which lines are taking too long but it is not my code that is taking too long, it takes a few minutes before it even starts running any of my code - it is like it is doing something in the background first.

To make this clearer, I will lay out step by step what happens with and without making any changes:

Without changes:

Section 1 - 5 seconds

Section 2 - 40 seconds

Secion 3 - 30 seconds

Section 4 - 1 minute

Section 5 - 10 seconds

Section 6 - 1 minute

With changes:

Section 1 - 5 seconds

Section 2 - 40 seconds

Section 3 - 30 seconds

Small change in section 3 (e.g. change marker in plot from circle to square)

Section 3 rerun - 8 minutes

Section 4, 5, 6 - same as before

[Rik]

Neither I nor John has experienced this ourselves in our years of writing Matlab code. While you may think it unlikely, both John and I expect it to be the source of the issue. It may or may not be. Without the code itself we can't tell for sure.

The only thing we can therefore tell you is how we try to avoid this issue: by using real functions for real work. Scripts are nice for debugging, but not work. I use scripts to figure out what works, but I use functions to make sure I will have the same result tomorrow.

Having a function doesn't mean you can't change anything anymore. You can pass in all the options you like. But clearly something about that 1500 lines of code causes Matlab problems. I have a 10k lines files that runs without problems, even with GNU Octave (a slow, semi-alternative to Matlab) on a Raspberry Pi. But those 10k lines are divided into 179 function, some off which have several dozen lines of documentation.

[DGM]

There's plenty of ways you can burn a ton of time with an unexpectedly expensive plot creation or manipulation within a loop. We still can't do anything but guess. I don't even know if a loop is involved.

Answer 1

[Constantino Carlos Reyes-Aldasoro]

Are you plotting your results? If you are, and you are doing many of these, it may be the case that that is what is slowing down your code if every time you are using plot(...). I once had problem like that and solved it by plotting once, grabbing the handle of the plot and then changing the XData and YData of the handles. Hope this helps.