Disclaimer: I have never read any literature on this subject, so there may be a better way to do this. This solution is what I was thinking about after reading your question.
A rectangle has two important dimensions: height and width
now if we start with a polygon and a rectangle:

1: go around the perimeter of the polygon and pay attention to all the places where the height of the rectangle will fit in the polygon (you can save this as a polygon *):

2: go around the perimeter of the new polygon that you just made and pay attention to all the places where the width of the rectangle will fit in the polygon (again, you can save this as a polygon):

3: the rectangle should fit into this new polygon (just be careful to correctly place the rectangle inside the polygon, as it is a polygon, not a rectangle. If you align the upper left node of the rectangle with the upper left node of this new polygon, you should be fine )
4: if no area is found that the rectangle will fit into, rotate the polygon a couple of degrees and try again.
* Note: in some polygons you will get more than one place where a rectangle can be set:

stormCloud
source share