Just if someone faces the same problem as me.
The problem was that my Simulink model was inside a Matlab Project. A part of the startup function of this project was to open the simulink model.
open('Test.slx')
When starting the optimization the startup function is called, which opens the model without fast restart, which the optimization function does not override..
I got arround adding those lines to the startup function:
open('Test.slx')
sim_obj = sdo.SimulationTest('Test');
sim_obj = fastRestart(sim_obj,'On');