First, ‘t’ needs to be defined as a sym object;
Second, use the convenient heaviside function to define the unit step function:
syms x t
t = sym(5);
y = int(-(t-x)*exp(-(t-x))*heaviside(t-x)*x^2*exp(-x)*sin(x)*heaviside(x));
This produces an expression for ‘y’ incorporating heaviside terms, since ‘x’ is not defined numerically.
I am not certain what result you want. At the very least, this solves the NaN problem.