Seems like you were close! To get this to work,
- You need to connect 1D-3D interfaces between the Simscape Driveline model and the Simscape Multibody model.
- An inertia needs to be placed on the Simscape Driveline side of the interface (can be small)
- A stiff spring damper must appear between the two interface blocks so that the system is not overconstrained
I adapted shipping example smdoc_worm_gear, commenting out the Worm and Gear Constraint and putting in the Simscape Driveline block. The model is attached for your reference. Some tuning of the inertia, spring, and damper may be necessary to get fast simulation.