The operating system is designed such that only one audio source is allocated all the child processes of a process.
MATLAB is the parent process and every code run from MATLAB is a child of MATLAB.
The system command is initiated from MATLAB, hence VLC which is initiated from MATLAB is also a child of MATLAB.
Only one(VLC/MATLAB code) can use the audio resource.
Even though there is no audio output from VLC, VLC uses the audio resource.
Hence MATLAB can not access the Audio resource.
However, external applications can play audio at the same time because these applications are not a a child process of MATLAB.
To achieve your task, you can use MATLAB’s in-buit video player instead of VLC.
To play a video stored in your computer using MATLAB’s movie player