I assume that you are following Euler Angle convention of roll-pitch-yaw in the order of X-Y-Z. You have three coplanar points P1, P2 and P3 on the body in clockwise order (looking from the top) and that the X-axis of the body-fixed frame can be taken along the vector starting from P3 passing through the midpoint of the segment joining P2 and P3 i.e., x = (P1+P2)/2 -P3.
First we need to obtain the unit vectors along the XYZ-axes of the body-fixed frame. To do this:
- Construct vectors v1=P2-P1 and v2=P3-P1.
- Now Z is along the vector cross(v1,v2). Normalize it to get the unit vector Z.
- The unit vector (X), along X-axis is obtained by normalizing the vector x obtained above.
- Unit vector (Y) along Y-axis can be obtained as cross(Z,X). This will be a unit vector as Z and X are unit vectors.
Now, the Rotation Matrix representing the orientation of the body-fixed frame can be written as R=[X1,Y1,Z1; X2,Y2,Z2; X3,Y3,Z3], where X=[X1; X2; X3;] and so on for Y and Z. Assuming roll, pitch and yaw angles are called alpha, beta and gamma respectively. These can be obtained as:
- alpha= atan2(-Z2, Z3)
- beta= asin(Z1)
- gamma = atan2(-Y1,X1)
I hope this is what you were looking for. More about Euler angles and finding them from rotation matrix, can be found here. If you have access to MATLAB's Robotic System Toolbox, you can use rotm2eul function to get Euler Angles from rotation matrix.