Which dimension(s) does trainingOptions shuffle apply to?

Hi all,

How do I know which dimension(s) the trainingOptions shuffle shuffles or how could I figure it out? The wiki does not state it sadly.

My assumption would be for e.g. TxCxB data that is shuffles the batches.

Side question, I would assume that it would keep input-output pairs intact, including for multi-task learning models?

Thanks in advance!

Edit:

I made the following code to try to understand what is going on. If you set a breakpoint at the end of the loss function you can inspect variable b. Assuming that reshape does not alter the data, the first and second dimension of b are not altered compared to A. Therefor, only the batches have been shuffled. However, I do not know and haven't been able to figure out if it always shuffles the batches or that some other thing is playing a part.

close all
clear all
clc

% Generate simple input and output data
A = reshape(1:210, [7,5,6]); % Example input data

% % Convert to table for trainnet compatibility
% trainData = array2table(A, 'VariableNames', {'Input'});
% trainLabels = array2table(A, 'VariableNames', {'Output'});

% Define a simple feedforward network
layers = [
    sequenceInputLayer(5)
    fullyConnectedLayer(5)
];

% Set training options with shuffle 'every-epoch'
MLoptions = trainingOptions('adam', ...
    'MaxEpochs', 5, ...
    'Shuffle', 'every-epoch', ...
    'Verbose', true,...
    'InputDataFormats','TCB');

% Train the network with a custom loss function
net = trainnet(A, A, layers, @customLossFunction, MLoptions);

% ----------------------------
% Inline Custom Loss Function
% ----------------------------
function loss = customLossFunction(Y,T)
    % Example custom loss: Mean Squared Error (MSE)
    loss = mean((Y - T).^2, 'all');

    b=permute(extractdata(T), [3,1,2]);
end

[Stephen23]

According to an AI tool that I asked, the shuffle parameter in trainingOptions for MATLAB's Deep Learning Toolbox shuffles the observations (samples) in your training data before each training epoch. It affects how data is presented to the network during training but doesn't change the structure of individual samples. But AI is not very reliable.

In any case, the documentation should be clearer, so give a rating at the bottom of the documentation page. Make sure to leave a comment which clearly explains which information is lacking and how it could be improved.

[Matt J]

For TxCxB data: It shuffles the observations within each mini-batch, not the batches themselves. Your batches are still processed in sequence.

That sounds doubtful to me.

[Corné Dijkstra]

I think so far my findings (see edit) are similar to the shuffleing of batches within the mini-batch.

And I asked in the forum here, since an AI would not know the answer if it has not been stated before on the web which it hasn't according to my quick search.

[Stephen23]

"... if it has not been stated before on the web which it hasn't according to my quick search."

Then please do make a documentation enhancement suggestion here:

https://www.mathworks.com/support/contact_us.html

Answer 1

[Matt J]

You can use analyzeNetwork to see which dimension is the batch dimension.However, I believe it will always be the last dimension.

[Corné Dijkstra]

As far as I understand I cannot see in analyzeNetwork how it shuffles the data. It does provide the structure on which dimension is which data.

[Matt J]

It is always the (B) dimension which is shuffled. If your data is TxCxB, the shuffling process is equivalent to doing,

shuffledData = Data(:,:,randperm(end));

[Matt J]

I think so far my findings (see edit) are similar to the shuffleing of batches within the mini-batch.

I don't understand how your are using the terminology "batch" as distinct from "minibatch". The process of shuffling training samples and the process of dividing training samples into minibatches are entirely separate and independent (assuming you are not using a custom datastore). Both processes, however, operate along the (B) dimension of your data.