Hello
hgtransform is primarily used for Graphic Object transformations and doesn't support overlaps functionality.
If your objects are represented in 2D consider making use of polyshape as shown in the following code
pts = [4 4 -1 -1 2 2 1 1 0 0 3 3; ...
1 3 3 -1 -1 1 1 0 0 2 2 1; ...
0 0 0 0 0 0 0 0 0 0 0 0; ...
1 1 1 1 1 1 1 1 1 1 1 1];
h1 = patch('FaceColor',[0.3010 0.7450 0.9330]);
h1.XData = pts(1,:) ./ pts(4,:);
h1.YData = pts(2,:) ./ pts(4,:);
h1.ZData = pts(3,:) ./ pts(4,:);
mrot = makehgtform('zrotate',pi/5);
h2 = patch('FaceColor',[0.9290 0.6940 0.1250]);
h2.XData = p2(1,:) ./ p2(4,:);
h2.YData = p2(2,:) ./ p2(4,:);
h2.ZData = p2(3,:) ./ p2(4,:);
p2 = mrot*pts;
poly1 = polyshape(pts(1,:),pts(2,:));
poly2 = polyshape(p2(1,:),p2(2,:));
figure
polyvec = [poly1 poly2];
plot(polyvec)
TF = overlaps(polyvec)
I would suggest making use of Driving Scenario Designer in Automated Driving Toolbox, designed for designing, simulating, and testing ADAS and autonomous driving systems.