A couple comments / suggestions:
syms f(t) cos sin M m1 m2 xddot xddotsq l1 l2 L1 l2 alpha alphadot alphaddot alphadot beta betaddot betadot g gamma sq theta xddot
The identifiers cos, sin, alpha, beta, and gamma each already have meanings in MATLAB. By defining them as symbolic variables you will not be able to call the cosine, sine, alpha (graphics transparency), beta (special function), or gamma (special function) functions while those symbolic variables exist.
eqn1=-f(t)*(M+m1+m2)*xddot + (m1*l1+m2*L1)*alphaddot*cos*alpha + m2*l2*betaddot*cos*beta - (m1*l1+m2*L1)*alpha*sq*sin*alpha - m2*l2*betadot*sq*sin*beta;
eqn2=(m1*l1*sq+m2*L1*sq+l1)*alphaddot + (m1*l1*m2*L1)*xddot*cos*alpha + (m2*l2*L1*betaddot*cos*gamma) + (m2*l2*L1*beta*sq*sin*gamma) - g(m1*l1+m2*L1)*(sin*alpha);
You can't ask for element m1*l1+m2*L1 of g like you did in the last term on the line above. If I'd written:
what would you expect result to be? I'm guessing you meant to multiply g by that expression but forgot the *.
eqn3=(m2*L1*sq+l2)*betaddot + (m2*l2)*xddot*(cos*beta) + (m2*l2*L1)*alphaddot*(cos*gamma) - (m2*l2*L1)*alphadot*sq*(sin*gamma) - (m2*g*l2*sin*beta);
sol = solve([eqn1, eqn2, eqn3], [f(t),cos,sin,M,m1,m2,xddot,xddotsq,l1,l2,L1,l2,alpha,alphadot,alphaddot,alphadot,beta,betaddot,betadot,g,gamma,sq,theta,xddot]);
You've got three equations in many more than 3 unknowns. That could be a problem.
Based on your naming scheme, I'm guessing you're trying to solve a system of differential equations. If I'm correct you might want to try the approach described on this documentation page instead.