How can I rename variables containing forbidden characters once they are already imported to workspace?

Hello together,

I have imported a *.mat-file containing measurement values. One of the values' name looks like follows

Variable_[2]

It seems to be within the workspace but cannot be looked into due to forbidden characters. Is there any chance of renaming and work with it again. I tried to just assign it to another variable but it didn't work. Following error message was displayed.

??? openvar('Variable_[2]', Variable_[2]); Error: Unbalanced or unexpected parenthesis or bracket.

I appreciate any help , Thank you

[Azzi Abdelmalek]

This is not clear, variable_[2] is just a string or what? how your mat file was created before?

[Christoph]

Variable_[2] is something like a 2x23423 double depending on the length of the measurement.

The mat-file was created by an import tool to convert MDF-files to mat-files which I just use without further knowledge of how it really works. Am kind of new to matlab.

Answer 1

[James Tursa]

You can do this with a mex routine. I have already posted code for fixing workspace names for this exact problem. I don't have time to look for this code right now but will do so this evening. It is probably on the Newsgroup somewhere, not on Answers.

EDIT ----------------------------------------------------

OK, here's the file. I will confess it was only finished this evening so has not been through a lot of testing, but it seems to work on the simple test files I threw at it (bad variable names with spaces, characters like @#$^%^(, beginning with numbers, etc.). It reads the mat file directly, fixes up the name, and then puts it into the workspace of the caller. Give it a shot. I will try to formally complete this project and post it to the FEX soon ...

I will point out that this preliminary version of the routine is not particularly efficient. It loads each variable into the mex routine, fixes, the name, copies it to the workspace using mexPutVariable (which I think does a deep data copy), then deletes the mex copy of the variable. So it does an unnecessary deep variable copy to get the job done. I didn't have time this evening, but my final production version that I post to the FEX will likely hack into the mxArray when copying stuff to the workspace to avoid the data copy. It will probably take another week or two to get those changes into the code. But as long as your variables are not too large this should not make much of a difference to you (I hope!).

If you have not worked with mex files before you can do this:

1) Copy the program into a file called loadfixnames.c, which should be in a directory that is on the MATLAB path.

2) Make that directory your current default directory.

3) Type the following at the MATLAB prompt:

> mex loadfixnames.c

That should compile the file into a mex routine. Instructions for using it are in the comments in the file. If you get a request to "look for compilers", answer yes and then pick a C compiler from the list (such as lcc).

---------------------------------------------------------------------

/*************************************************************************************
 *
 * MATLAB (R) is a trademark of The Mathworks (R) Corporation
 *
 * Function:    loadfixnames
 * Filename:    locafixnames.c
 * Programmer:  James Tursa
 * Version:     1.00
 * Date:        June 06, 2013
 * Copyright:   (c) 2013 by James Tursa, All Rights Reserved
 *
 *  This code uses the BSD License:
 *
 *  Redistribution and use in source and binary forms, with or without 
 *  modification, are permitted provided that the following conditions are 
 *  met:
 *
 *     * Redistributions of source code must retain the above copyright 
 *       notice, this list of conditions and the following disclaimer.
 *     * Redistributions in binary form must reproduce the above copyright 
 *       notice, this list of conditions and the following disclaimer in 
 *       the documentation and/or other materials provided with the distribution
 *      
 *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 
 *  AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
 *  IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 
 *  ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 
 *  LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
 *  CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 
 *  SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
 *  INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
 *  CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
 *  ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 
 *  POSSIBILITY OF SUCH DAMAGE.
 *
 * Building:
 *
 * LOADFIXNAMES is typically self building. That is, the first time you call it,
 * the loadfixnames.m file recognizes that the mex routine needs to be compiled and
 * then the compilation will happen automatically.
 *
 * The usage is as follows (arguments in brackets [ ] are optional):
 *
 * Syntax
 *
 * loadfixnames(FILENAME [,names] [,verbose])
 *
 *     FILENAME = The mat file to be loaded
 *     names    = string or cell array of strings (variable name(s) to load)
 *     verbose  = 1 (optional, causes name change log to be displayed)
 *
 * Description
 *
 * LOADFILENAMES loads a mat file into the workspace, fixing invalid names.
 * All invalid characters are replaced with an underscore. Also, if the
 * first character is not a letter, an 'A' is added to the front. If the
 * variable is a structure, fixes the field names also. In the case of
 * field names, the name length is kept constant. So if a field name
 * begins with a digit it will be replaced with 'A' - 'J' instead. If the
 * field name begins with an invalid non-digit it will be replaced with
 * 'A' - 'Z' or 'a' - 'z' (letters are cycled in an attempt to avoid
 * name clashes).
 *
 * Limitations: The current renaming scheme makes a mild attempt to avoid
 *              name clashes, but does not guarantee this.
 *
 * Change Log:
 * 2013/Jun/06 --> 1.00, Initial Release
 *
 ****************************************************************************/

// Includes -----------------------------------------------------------

#include "mex.h"
#include "mat.h"
#include <ctype.h>
#include <string.h>

// Macros -------------------------------------------------------------

#ifndef MWSIZE_MAX
#define  mwIndex        int
#define  mwSignedIndex  int
#define  mwSize         int
#endif

#define  MAXNAME  63

// Prototypes ---------------------------------------------------------

void fixvarname(char *, char *, int);
void fixfieldnames(mxArray *, int);
char *appendmat(char *, mwSize);
int isamatch(const char *, mxArray *);
int wildcmp(const char *wild, const char *str);

// Global variables ---------------------------------------------------

char alphabet[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
char alphanum[] = "AaABbBCcCDdDEeEFfFGgGHhHIiIJjJKkKLlLMmMNnNOoOPpPQqQRrRSsSTtTUuUVvVWwWXxXYyYZzZ_0a1b2c3d4e5f6g7h8i9j";

// Gateway Function ---------------------------------------------------

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
  mxArray *mx, *compare = NULL;
  char *filename, *matfilename;
  char mode[] = "r";
  const char *name;
  char *cname;
  char newname[MAXNAME+2];
  char matlabmat[] = "matlab.mat";
  MATFile *mfp;
  int verbose = 0;
  mwSize n;

/* Check arguments */

  if( nlhs > 0 ) {
    mexErrMsgTxt("Too many outputs.");
  }
  if( nrhs > 3 ) {
    mexErrMsgTxt("Too many inputs.");
  }
  if( !mxIsChar(prhs[0]) ) {
    mexErrMsgTxt("1st argument must be filename char string.");
  }
  if( mxGetNumberOfDimensions(prhs[0]) != 2 || (mxGetM(prhs[0]) != 1 && mxGetN(prhs[0]) != 1) ) {
    mexErrMsgTxt("1st argument must be filename char string.");
  }
  if( nrhs >= 2 ) {
    if( mxIsNumeric(prhs[1]) ) {
      verbose = (mxGetScalar(prhs[1]) != 0.0);
    } else if( mxIsChar(prhs[1]) || mxIsCell(prhs[1]) ) {
      compare = prhs[1];
    } else {
      mexErrMsgTxt("Invalid 2nd argument.");
    }
  }
  if( nrhs >= 3 ) {
    if( mxIsNumeric(prhs[2]) ) {
      verbose = (mxGetScalar(prhs[2]) != 0.0);
    } else if( mxIsChar(prhs[2]) || mxIsCell(prhs[2]) ) {
      compare = prhs[2];
    } else {
      mexErrMsgTxt("Invalid 3rd argument.");
    }
  }
  if( nrhs == 0 ) {
    matfilename = matlabmat; /* If no filename given, use "matlab.mat" */
  } else {
    filename = mxArrayToString(prhs[0]); /* Get the filename as a C-string */
    n = mxGetNumberOfElements(prhs[0]);
    matfilename = appendmat(filename, n); /* Append ".mat" if necessary */
  }
  mfp = matOpen(matfilename, mode); /* Open the mat file */
  if( mfp == NULL ) {
    mexPrintf("Tried to open file %s\n",matfilename);
    if( matfilename != matlabmat ) mxFree(matfilename);
    mexErrMsgTxt("Unable to open file.");
  }
  if( matfilename != matlabmat ) mxFree(matfilename); /* Done with temp filename C-string */
  while( 1 ) {
    mx = matGetNextVariable( mfp, &name );
    if( mx == NULL ) break;
    if( !isamatch(name,compare) ) continue;
    fixvarname( name, newname, verbose );
    fixfieldnames( mx, verbose );
    if( mexPutVariable("caller", newname, mx) ) {
      mexPrintf("Tried to put variable into caller workspace: %s\n",newname);
      mexWarnMsgTxt("Unable to put variable into caller workspace.");
    }
    mxDestroyArray(mx);
  }
  matClose(mfp);
}

/**************************************************************************************************/

void fixvarname( char *name, char *newname, int verbose )
{
  char *oldn = name;
  char *newn = newname;
  int badname = 0;

  if( !isalpha(*name) ) { /* Check that first char is a letter */
    badname = 1;
    *newname++ = 'A';
  }
  while( *name ) {
    if( isalnum(*name) || *name == '_' ) { /* Check for alphanumeric or underscore */
      *newname++ = *name++;
    } else {
      badname = 1;
      *newname++ = '_'; /* Replace bad characters with underscores */
      name++;
    }
  }
  *newname = '\0'; /* Attach the final null termination character */
  if( verbose && badname ) {
    mexPrintf("Invalid Name:  <%s>\n",oldn);
    mexPrintf("Replaced With: <%s>\n",newn);
  }
}

/**************************************************************************************************/

void fixfieldnames(mxArray *mx, int verbose)
{
  int badname;
  mwSize i, j, n, f;
  char *fieldname, *fname, *fi, *fj, *newchar;
  int z, nameclash;

  if( mx == NULL ) {
    ; /* Nothing to fix */
  } else if( mxIsStruct(mx) ) {
    n = mxGetNumberOfElements(mx);
    f = mxGetNumberOfFields(mx);
    z = 0;
    for( j=0; j<f; j++ ) {
      badname = 0;
      fname = fieldname = (char *) mxGetFieldNameByNumber(mx,j);
      if( *fieldname == '\0' ) {
        mexWarnMsgTxt("Unable to fix empty fieldname.");
        continue;
      }
      if( !isalpha(*fieldname) ) { /* Check that first char is a letter */
        if( verbose ) {
          mexPrintf("Invalid Field Name:  <%s>\n",fname);
        }
        badname = 1;
        if( isdigit(*fieldname) ) {
          *fieldname = alphabet[*fieldname - '0'];
        } else {
          *fieldname = alphabet[z];
          z = (z+1) % 52;
        }
      }
      while( *fieldname ) {
        if( !isalnum(*fieldname) && *fieldname != '_' ) { /* Check for alphanumeric or underscore */
          if( !badname && verbose ) {
            mexPrintf("Invalid Field Name:  <%s>\n",fname);
          }
          badname = 1;
          *fieldname++ = '_'; /* Replace bad characters with underscores */
        } else {
          fieldname++;
        }
      }
      if( verbose && badname ) {
        mexPrintf("Replaced With:       <%s>\n",fname);
      }
      nameclash = 1; /* Check for field name clashes */
      while( nameclash ) {
        nameclash = 0;
        for( i=0; i<f; i++ ) {
          fi = (char *) mxGetFieldNameByNumber(mx,i);
          for( j=i+1; j<f; j++ ) {
            fj = (char *) mxGetFieldNameByNumber(mx,j);
            if( fi && fj && strcmp(fi,fj) == 0 ) { /* Name clash */
              nameclash = 1;
              while( *fj ) { /* Fix the name clash */
                  newchar = alphanum;
                while( *newchar ) {
                  if( *fj == *newchar ) {
                    *fj = *(newchar+1);
                    break;
                  }
                  newchar++;
                }
                fj++;
              }
            }
          }
        }
      }
    }
    for( i=0; i<n; i++ ) {
      for( j=0; j<f; j++ ) {
                fixfieldnames(mxGetFieldByNumber(mx,i,j), verbose);
      }
    }
  } else if( mxIsCell(mx) ) {
    n = mxGetNumberOfElements(mx);
    for( i=0; i<n; i++ ) {
            fixfieldnames(mxGetCell(mx,i), verbose);
    }
  }
}

/**************************************************************************************************/

char *appendmat(char *filename, mwSize n)
{
  char *fname = filename;
  char *mname, *matfilename;

  if( n < 5 || (strcmp(filename+n-4,".mat") != 0 && strcmp(filename+n-4,".MAT") != 0 && strcmp(filename+n-4,".Mat") != 0) ) {
    mname = matfilename = mxMalloc(n+5);
    while( *filename ) {
      *matfilename++ = *filename++;
    }
    *matfilename++ = '.';
    *matfilename++ = 'm';
    *matfilename++ = 'a';
    *matfilename++ = 't';
    *matfilename = '\0';
    mxFree(fname);
  } else {
    mname = filename;
  }
  return mname;
}

/**************************************************************************************************/

int isamatch(const char *name, mxArray *compare)
{
  mwSize i, n;
  char *cname;

  if( compare == NULL ) return 1;
  if( mxIsChar(compare) ) {
    cname = mxArrayToString(compare);
//    i = strcmp(name,cname);
    i = !wildcmp(cname,name);
    mxFree(cname);
    return (i == 0);
  } else if( mxIsCell(compare) ) {
    n = mxGetNumberOfElements(compare);
    for( i=0; i<n; i++ ) {
      if( isamatch(name,mxGetCell(compare,i)) ) return 1;
    }
    return 0;
  } else {
    mexErrMsgTxt("Invalid cell. Must be char string with variable name.");
  }
}

/* wildcmp obtained from Jack Handy. Seems to work OK. */

int wildcmp(const char *wild, const char *str)
{
  const char *cp = NULL, *mp = NULL;

    while ((*str) && (*wild != '*')) {
      if ((*wild != *str) && (*wild != '?')) {
        return 0;
      }
      wild++;
      str++;
    }

    while (*str) {
      if (*wild == '*') {
        if (!*++wild) {
          return 1;
        }
        mp = wild;
        cp = str+1;
      } else if ((*wild == *str) || (*wild == '?')) {
        wild++;
        str++;
      } else {
        wild = mp;
        str = cp++;
      }
    }

    while (*wild == '*') {
      wild++;
    }
    return !*wild;
  }

[Christoph]

Hi James, I checked on the Newsgroup for mex and your name but found too many entires which are not too self explaining for a beginner : ) Would be great if you can provide the code or link later. I'll go to bed in the meantime, good night and Greetings from Germany and thanks a lot.

[James Tursa]

Found it. Looks like the original problem I worked on was variable names with spaces (which I think was posted to the Newsgroup). Then I expanded that to a routine that fixed all invalid characters for both variable names and field names of structures. I never finished that expanded project so it was never posted. I will work on completing that code tonight ...

[Christoph]

Hi James, sorry for being so late with saying thank you, but it took me awhile to get it to run since I am quite inexperienced. Thanks a lot for your help. Christoph

[Walter Roberson]

Thanks, James!

[James Tursa]

UPDATE: I have uploaded the shared-data-copy version of this routine to the FEX, which might make a difference to someone working with large variables. It can be found here:

http://www.mathworks.com/matlabcentral/fileexchange/42274-loadfixnames-loads-a-mat-file-fixing-invalid-names

Answer 2

[Walter Roberson]

Use the function form of load of the .mat file:

filevarstruct = load('TheFile.mat');

Then

fn = fieldnames(filevarstruct);
for K = 1 : length(fn)
  newfieldname = genvarname(fn{K});
  memvarstruct.(newfieldname) = filevarstruct.(fn{K});
end
clear filevarstruct

Now memvarstruct is a structure whose fieldnames are versions of the original names but adjusted to be valid variable names. In the case of 'Variable_[2]' the result would be 'Variable_0x5B20x5D' . You could use whatever string processing you though was appropriate to construct the new fieldnames, such as

newfieldname = regexprep(fn{K}, '\[(\d+)\]', '$1');

which would change the '[2]' to '2'.

I seem to recall there is a FEX contribution to rename invalid fields in .mat files. Jan knows this stuff.

Once you have the structure with valid fieldnames, you can use it in further processing. Maybe even assign portions to non-structure variables in the workspace. But preferably not with unrestricted dynamically-created variable names: beyond there be dragons.

http://matlab.wikia.com/wiki/FAQ#How_can_I_create_variables_A1.2C_A2.2C....2CA10_in_a_loop.3F

[Christoph]

Hi Walter, thanks for your quick reply. I tried as you suggested but I already fail to import the mat-file via

filevarstruct = load...

There seems to be a difference between just "drag and drop" the mat-file into workspace which leads to at least showing the variable_[2] and the load function where matlab already complains about the "Invalid field name: 'Variable_[2]' " Variable_[2] is supposed to be a 2x5600 double or so

Answer 3

[Justin Mai]

There is no current way to rename fieldnames in a MAT file, because loading it to a variable will fail.

That is, doing load('myfile.mat') will load into the workspace and not fail, doing x = load('myfile.mat') will fail.

[Walter Roberson]

Did you try Jame's code?

[James Tursa]

Unfortunately, TMW changed the API library functions (I'm not sure when) to check for invalid names and will not load the variables even in a mex routine. So the only way my code will work is to use an older version of MATLAB. It would have been nice if they would have maintained the old functionality in some way so that a mex routine could fix things up, but that is no longer the case.

[Walter Roberson]

I wonder if something like https://www.mathworks.com/matlabcentral/fileexchange/6893-matcat-mat-file-corruption-analysis-tool could be modified to scribble in new variable names?

[James Tursa]

So, if one is willing to write a mat file reader from scratch that only operates on uncompressed mat files, then yes you could open as a regular binary file and read in the data and fix up invalid names on the fly. Could be done either in an m-file or mex routine. Looks like the tool in your link might be a good start on this, but it would take a fair amount of research on the mat file structure to add the code necessary to actually load the variables instead of just looking for errors. I don't have any plans to do this in the near future.