2024a vs 2024b detectImportOptions error

I just updated my matlab to start using a new toolbox added to my license. I'm now getting an error that I was previously not getting. I can run both versions of matlab beside each other and am still successfully running it on 2024a but getting an error on 2024b. Because it's not got an error in 2024a I know that the file DOES exist and the path IS valid. Help?

"Error using detectImportOptions (line 428)

Unable to find file. Ensure file exists and path is valid.

Error in Scriptname (line 79)

opts = detectImportOptions(url,'FileType','text');"

[dpb]

What does

dir(url)

return if a breakpoint is set at that line in the offending case?

What does

pwd

return for each?

[Drew Jordison]

The line is in an if/for loop so when it errors out,

dir(url) = https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"

pwd on both version is my working folder, the same for both.

[Walter Roberson]

Works for me on my Intel MacOS system, in R2024b and R2025a-pre-release.

[Drew Jordison]

Weird. I'm on Windows but really can't imagine what the issue is. :(

[dpb]

Show us the command output, don't tell us... :)

url='https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"'

url = 'https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"'

opt=detectImportOptions(url,'FileType','text')

opt = 
  DelimitedTextImportOptions with properties:

   Format Properties:
                    Delimiter: {','}
                   Whitespace: '\b\t '
                   LineEnding: {'\n'  '\r'  '\r\n'}
                 CommentStyle: {}
    ConsecutiveDelimitersRule: 'split'
        LeadingDelimitersRule: 'keep'
       TrailingDelimitersRule: 'ignore'
                EmptyLineRule: 'skip'
                     Encoding: 'UTF-8'

   Replacement Properties:
                  MissingRule: 'fill'
              ImportErrorRule: 'fill'
             ExtraColumnsRule: 'addvars'

   Variable Import Properties: Set types by name using setvartype
                VariableNames: {'work_area', 'cruise', 'hakai_id' ... and 63 more}
                VariableTypes: {'char', 'char', 'char' ... and 63 more}
        SelectedVariableNames: {'work_area', 'cruise', 'hakai_id' ... and 63 more}
              VariableOptions: [1-by-66 matlab.io.VariableImportOptions] 
	Access VariableOptions sub-properties using setvaropts/getvaropts
           VariableNamingRule: 'modify'

   Location Properties:
                    DataLines: [2 Inf]
            VariableNamesLine: 1
               RowNamesColumn: 0
            VariableUnitsLine: 0
     VariableDescriptionsLine: 0 
	To display a preview of the table, use preview

dir(url)

https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"  

version

ans = '24.2.0.2871072 (R2024b) Update 5'

Works here w/ R2024b, too...something about the local connection, for that one somehow if others succeed and it doesn't, maybe?

I don't have installed locally to test...

[Drew Jordison]

>> url

url =

"https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z""

>> opts = detectImportOptions(url,'FileType','text')

Error using detectImportOptions (line 428)

Unable to find file. Ensure file exists and path is valid.

>> version

ans =

'24.2.0.2863752 (R2024b) Update 5'

>> dir(url)

https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"

vs 2024a below:

>> url='https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"'

url =

'https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"'

>> opt=detectImportOptions(url,'FileType','text')

opt =

DelimitedTextImportOptions with properties:

Format Properties:

Delimiter: {','}

Whitespace: '\b\t '

LineEnding: {'\n' '\r' '\r\n'}

CommentStyle: {}

ConsecutiveDelimitersRule: 'split'

LeadingDelimitersRule: 'keep'

TrailingDelimitersRule: 'ignore'

EmptyLineRule: 'skip'

Encoding: 'UTF-8'

Replacement Properties:

MissingRule: 'fill'

ImportErrorRule: 'fill'

ExtraColumnsRule: 'addvars'

Variable Import Properties: Set types by name using setvartype

VariableNames: {'work_area', 'cruise', 'hakai_id' ... and 63 more}

VariableTypes: {'char', 'char', 'char' ... and 63 more}

SelectedVariableNames: {'work_area', 'cruise', 'hakai_id' ... and 63 more}

VariableOptions: Show all 66 VariableOptions

Access VariableOptions sub-properties using setvaropts/getvaropts

VariableNamingRule: 'modify'

Location Properties:

DataLines: [2 Inf]

VariableNamesLine: 1

RowNamesColumn: 0

VariableUnitsLine: 0

VariableDescriptionsLine: 0

To display a preview of the table, use preview

>> version

ans =

'24.1.0.2653294 (R2024a) Update 5'

>> dir(url)

'https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"' contains an unsupported provider.

>>

I'd assume it was some obvious error on my part but the script hasn't changed between the two versions running side by side.

[Walter Roberson]

The most likely cause of that kind of problem, is if you have some third-party .m file with the same name as a Mathworks function.

You might notice it in R2024b but not R2023a because possibly the function was not called in R2023a.

One of the more common names that leads to this kind of problem is contains.m

[dpb]

Does the error occur for all files or only the specific one? If Walter's hypothesis were correct it would seem to be all unless there's some branching of a function call that is dependent upon the content of the filename which wouldn't seem likely?

It would seem would be down to tracking the differences in the two supplied m-files to see where they differ...

I think I would submit that the two have different results to Mathworks as a bug report; should be able to give you a place to look for the different behavior anyway...

[Steven Lord]

Since this is related to file I/O, if Walter is correct (which I suspect as well) let's see if the fopen function is being shadowed by a different fopen.m that behaves differently than the built-in function. What do you see when you run the following command in both release R2024a and R2024b?

which -all fopen

built-in (/MATLAB/toolbox/matlab/iofun/fopen)
/MATLAB/toolbox/matlab/serial/@serial/fopen.m           % serial method
/MATLAB/toolbox/instrument/instrument/@i2c/fopen.m      % i2c method
/MATLAB/toolbox/shared/instrument/@icinterface/fopen.m  % icinterface method

[dpb]

This implies Drew would have loaded some additional/other add-on in his b release as opposed to a...which is possible, of course, but would seem to be unknownst to him thinking he's running the same environment, just a newer MATLAB release. Unless the update triggers a refresh of installed add-ons as well, maybe?

That would seem to indicate should look at versions for any/all addins does have installed on the two releases...

[Walter Roberson]

Suppose Drew had his own contains.m file. Suppose the 2024a detectImportOptions did not call contains() but the 2024b version does call contains(). Then the code would work in 2024a but would not work in 2024b.

The interfering file is not necessarily contains.m — that is just a common culprit. The real culprit could be something like width.m with the older version using size(obj,2) but the new version calling width() instead.

[dpb]

Anything is possible, yes, Walter, but as Steve points out, this failure to be able to open the file appears to point to something in the iofun category. I'm having difficulty wrapping my head around the fact that apparently it works for some, but not this particular file.

But, it's all conjecture until Drew comes back with results of further diagnostics...I think I'd set

dbstop if error

so can poke around inside the MATLAB function when it dies and see just what was the actual calling code that fails...although this is part of what Mathworks has so obfuscated by packaging internals in objects it's essentially impossible for the user to figure out any more...

Line 428 in R2021b of detectImportOptions appears to be no different than a current version -- it is

func = matlab.io.internal.functions.FunctionStore.getFunctionByName('detectImportOptions');

and from there it gets ever more convoluted to trace down to the final executed code to determine what might be the actual aliased function name, presuming that really is what is happening.

[Drew Jordison]

So the only change in addition to the new version is that I've installed the Curve Fitting Toolbox

From above,

which -all fopen

built-in (C:\Program Files\MATLAB\R2024a\toolbox\matlab\iofun\fopen)

C:\Program Files\MATLAB\R2024a\toolbox\matlab\serial\@serial\fopen.m % serial method

C:\Program Files\MATLAB\R2024a\toolbox\shared\instrument\@icinterface\fopen.m % icinterface method

which -all fopen

built-in (C:\Program Files\MATLAB\R2024a\toolbox\matlab\iofun\fopen)

C:\Program Files\MATLAB\R2024a\toolbox\matlab\serial\@serial\fopen.m % serial method

C:\Program Files\MATLAB\R2024a\toolbox\shared\instrument\@icinterface\fopen.m % icinterface method

[dpb]

That is as I expected you would find, Drew. My recommendation for next step is as I mentioned in comment back to Walter is to set

dbstop if error

and see if it will open up in the bowels to let you see where it actually dies instead of just echoing the high-level calling line that is uninformative as to root cause.

I'm still curious if R2024b works for other files and only fails on this particular one or is it any file?

Oh! And also, do you get the same behavior from straight at the command line as in your app/function/code?

[Drew Jordison]

In addition, it looks like this error occurs in the for loop in the first instance of a new instrument's data,

url =

"https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="18032_2022-11-25T22:13:22.500Z""

encounters no errors.

[Drew Jordison]

If I input straight into the command line the result is the same.

dbstop if error

doesn't seem to change anything? The error thrown is the same, pin pointing the error to the last line of detectImportOptions

[dpb]

I don't know what "a new instrument" means here, but perhaps there's something unique/different there? Although again, if the other release handles it, then it does still imply something has been changed internally by Mathworks. Does this use an instrumentation toolboxes or the like or is it all base MATLAB functionality?

Given that this a url and not a local or even network file, my hunch is it has to do with timing/communication with the server somehow instead of actual function aliasing or such...

What if you put the call in a try...catch block and then first, just retry?

Secondly, if that still fails, see if you can just copy the url content to local file first...at this point just to see if behavior changes.

Again, the dbtop if error to see where it will point in the morass of internal MATLAB code...

[Drew Jordison]

I don't have experience with try.

Would it look like this:

try 
    section
catch
    warning('Error using detectImportOptions (line 428) Unable to find file. Ensure file exists and path is valid.')
    i=i+1;
end

where section is the name of a script of just the section throwing the error?

[dpb]

Yeah, figured as much since in

throw(ME)

throw inside the try/catch block creates the stack trace from the location--a rethrow there instead might be of some help.

Would still be curious if either of the other above ideas about a retry (not likely, I'm guessing) or trying to copy to a local file would make any difference, but...

Unless @Steven Lord has some other ideas, I'm of the opinion since it is reproducible on your system you should submit to official support; otherwise delving into the bowels of these obfuscated internal functions is beyond much hope for the end user.

[Drew Jordison]

I have submitted a bug report. Thanks for all the help!

[Walter Roberson]

Tracing this would require

dbstop if caught error

rather than

dbstop if error

[dpb]

I missed your earlier comment about the try/catch block, Drew, sorry...

Instead of

try 
    section
catch
    warning('Error using detectImportOptions (line 428) Unable to find file. Ensure file exists and path is valid.')
    i=i+1;
end

What you would do is something like

.... existing code here
try    % the problematic line
  opts = detectImportOptions(url,'FileType','text');"
catch ME
  % try it again just in case -- this will almost certainly not work, but "ya' never know!"
  disp(ME.message) % show what happened; this will be the failed to open but then you know it was caught
  opts = detectImportOptions(url,'FileType','text');"
end
...

The second perhaps more interesting case would be something like

....
try    % the problematic line
  opts = detectImportOptions(url,'FileType','text');"
catch ME
  % try to copy to local temp location
  disp(ME.message) % show what happened; this will be the failed to open but then you know it was caught
  [f,n,e]=fileparts(url);                   % parse the url parts
  tempfile=fullfile(tempdir,strcat(n,e));   % create a temporary file name in system temp folder
  [status,msg]=copyfile(url,tempfile);      % see if can copy it local file
  assert(status==1,strcat('Copy failed: ',msg)) % if unsuccessful, abort but let us know
  opts = detectImportOptions(tempfile,'FileType','text'); % then try it
  urlold=url;                               % hang onto the original url in case need it later...
  url=tempfile;                             % if got here, it worked so redirect to the local copy...
end
...

This would be inelegant solution, but would be interesting as to see if the regular OS can get to the file and then whether detectImportOptions is barfing on the name or the fact it isn't local...

[Drew Jordison]

First case:

disp(ME.message) 

Unable to find file. Ensure file exists and path is valid.

Second case:

Error using assert

Copy failed:'C:\Users\DrewJordison\AppData\Local\Temp\HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"' is not a valid location.

[dpb]

Yeah, you'll probably have to do something more like

...
[f,n,e]=fileparts(url);                             % parse the url parts
tempfile=char(fullfile(tempdir,strcat(n,'.txt')));  % create a temporary .txt file name
[status,msg]=copyfile(url,tempfile);                % see if can copy it local file
...

to create a normal filesystem filename. This one would be the base name

>> [f,n,e]=fileparts(url)
f =
    'https://catalogue.hakai.org/erddap/tabledap'
n =
    'HakaiWaterPropertiesInstrumentProfileProvisional'
e =
    '.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"'
>> 

HakaiWaterPropertiesInstrumentProfileProvisional.txt in your temp dir

Or you could try and get more creative and try some variation such as

...
[f,n,e]=fileparts(url);                                 % parse the url parts
tstr=extractBetween(url,'"','"');                       % pull the time string out of the url
tempfile=char(fullfile(tempdir,strcat(n,tstr,'.txt'))); % create a temporary .txt file name
[status,msg]=copyfile(url,tempfile);                    % see if can copy it local file
...

The ? and " characters at least are invalid in Windows filenames, I guess the other alternative would be to replace them...

url=replace(url,{'&?"><'},{'_'});

first.

[dpb]

url='https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"'

url = 'https://catalogue.hakai.org/erddap/tabledap/HakaiWaterPropertiesInstrumentProfileProvisional.csvp?&hakai_id="1907674_2021-09-23T17:01:55Z"'

[f,n,e]=fileparts(url);
tstr=extractBetween(url,'"','"');
tempfile=char(fullfile(strcat(n,tstr,'.txt')))

tempfile = 'HakaiWaterPropertiesInstrumentProfileProvisional1907674_2021-09-23T17:01:55Z.txt'

[status,msg]=copyfile(url,tempfile)

status = logical
   1

msg =

  0x0 empty char array

dir *.txt

HakaiWaterPropertiesInstrumentProfileProvisional1907674_2021-09-23T17:01:55Z.txt  

opt=detectImportOptions(tempfile)  % will be 'Filetype','Text' based on file extension

opt = 
  DelimitedTextImportOptions with properties:

   Format Properties:
                    Delimiter: {','}
                   Whitespace: '\b\t '
                   LineEnding: {'\n'  '\r'  '\r\n'}
                 CommentStyle: {}
    ConsecutiveDelimitersRule: 'split'
        LeadingDelimitersRule: 'keep'
       TrailingDelimitersRule: 'ignore'
                EmptyLineRule: 'skip'
                     Encoding: 'UTF-8'

   Replacement Properties:
                  MissingRule: 'fill'
              ImportErrorRule: 'fill'
             ExtraColumnsRule: 'addvars'

   Variable Import Properties: Set types by name using setvartype
                VariableNames: {'work_area', 'cruise', 'hakai_id' ... and 63 more}
                VariableTypes: {'char', 'char', 'char' ... and 63 more}
        SelectedVariableNames: {'work_area', 'cruise', 'hakai_id' ... and 63 more}
              VariableOptions: [1-by-66 matlab.io.VariableImportOptions] 
	Access VariableOptions sub-properties using setvaropts/getvaropts
           VariableNamingRule: 'modify'

   Location Properties:
                    DataLines: [2 Inf]
            VariableNamesLine: 1
               RowNamesColumn: 0
            VariableUnitsLine: 0
     VariableDescriptionsLine: 0 
	To display a preview of the table, use preview

So, the copy does seem to work; the Q? then would be whether your system will then be able to open/process the local file that fails from the direct url request....

Answer 1

[Kautuk Raj]

I was facing a similar error during one of my workflows wherein the "detectImportOptions" threw a "Unable to find file" error when filename was a URL in MATLAB R2024b.

As a workaround, I switched to using MATLAB R2024a.

Alternatively, you use a local copy of the CSV file to mitigate the issue in MATLAB R2024b.

Answer 2

[Drew Jordison]

I reported this and it was a replicable bug. I've reverted to 2024a for the time being.