Why do I get a matrix dimensions must agree error in PCT on external server but not on local cluster?

Hello,

When I run some code in matlab using parallel computing on my local cluster (2 workers) it works perfectly. However, when I run the same code on an external cluster (20 workers) I get a "matrix dimensions must agree" error for the last line of this chunk of code. I can't figure out why! Any suggestions appreciated.

        parfor it = 1:num_noise_iter

            y_i = normrnd(y_no_noise,outcome_stdev);
            z=[group_id, y_i, y_no_noise, preferences, effort_vector, u1];
            z_sorted=sortrows(z, [1,-2]); 
            y_i_gp = accumarray(z_sorted(:,1),z_sorted(:,2),[],@(n){n});
            y_i_gp_mat=padcat(cell2mat(y_i_gp(1,:)), cell2mat(y_i_gp(2,:)))';
            Ninsp = max(cellfun(@numel,y_i_gp));

            y_i_g = y_i_gp_mat(z_sorted(:,1),:);

            y_out=y_i_g';
% error in following line: "Matrix dimensions must agree"
            y_out=reshape(y_out(y_out ~= z_sorted(:,2)'),[],num_inspectors);

[Edric Ellis]

Without a minimal, complete, verifiable example, it's hard to tell what's going wrong. You could try and work things out by splitting the combined expression over several lines, and using disp to print the sizes of the various expressions.

Answer 1

[Walter Roberson]

Are you running R2016b or newer on your local system, and is the external cluster running R2016a or earlier? If so then you might be benefiting from the new implicit expansion feature of R2016b that operates like bsxfun()

[Garima Sharma]

Thank you--yes, I am! (and yes, it is) How would I fix this?

[Walter Roberson]

You should re-check the dimensions of all of your variables to ensure that they are what you want, and that you have the orientation that you expect. If, after the check, you find that you still need the expansion that is new in R2016b, then you will need to recode that part of the calculation using bsxfun()

It could turn out to be as simple as accidentally comparing a row vector to a column vector of the same length when you expected both to be rows or both to be columns.

[Walter Roberson]

Ah, you are comparing an 11 x 21 matrix to a 1 x 21 matrix. With R2016b or later, you would get an 11 x 21 logical result.

At the moment I do not follow the math of how z_sorted and y_out are connected, so at the moment I do not see any reason why the number of matches per row should be consistent. You can theoretically get duplicate random numbers, or random numbers that just happen to match integers. It is after 3 AM here so I although I have traced through the construction logic some, I am not currently convinced that you will get a fixed number of rows or columns selected out of the comparison.

If you do not get a fixed number of matches per row then the reshape() is not going to work because the number of total elements would not match the number of inspectors. If we suppose that the fixed number of per row might not be exactly 1, then we run into the logic difficulty that you would be reshaping 11 x something into [] by 11 without having done a transpose, and that is unlikely to get you the order of elements that you want. If we somehow determine that there will be exactly one match per row, so the number of elements selected matches the number of inspectors, then the reshape() in that form would have the effect of making the result into a row vector -- but there are clearer ways of making that into a row vector. For example, reshape(Expression, 1, []) makes it clear that your desired result is a row vector. Or, considering that the result of the logical masking is going to be a column vector, you could just use .' instead of reshape() in the way you do.

Anyhow, the direct repair to your code to do what you are doing now is

reshape(y_out( bsxfun(@ne, y_out, z_sorted(:,2)')),[], num_inspectors)

Answer 2

[Mostafa Nakhaei]

I have the same issue and I only access to Matlab 2016a

What should I do?

[Walter Roberson]

If your local system and your cluster system are both R2016a or before, then you have some other issue, and you should start a new Question showing your code and posting the error message.

[Mostafa Nakhaei]

Thanks Walter, I fixed the problem with a bit of mathematical trick to change the dimension and preserve the original results.