You can create custom layers and define custom loss functions for output layers.
The output layer uses two functions to compute the loss and the derivatives: forwardLoss and backwardLoss. The forwardLoss function computes the loss L. The backwardLoss function computes the derivatives of the loss with respect to the predictions.
For eg., to write a weighted cross entropy classification loss, try running this in the MATLAB command window
Hope this helps