Your devices are disconnected. Consider this expression from your code, for example:
RichText.Width * Screen.Pixelsperinch
The left term is in pixels and the right term is in pixels / inches, so the units of the result are ² / inch pixels. The expected unit for the rectangles used in em_FormatRange is twips. If you want to convert pixels to twips, you will need the following:
const TwipsPerInch = 1440; RichText.Width / Screen.PixelsPerInch * TwipsPerInch
You do not need an off-screen advanced editing control. You just need a contactless editor with advanced editing , which you can instruct to draw directly on the tooltip. I posted some Delphi code that simplifies the basics. Beware that it does not support Unicode, and I have no plans to do so (although it may not be too difficult to do).
The main function from my DrawRTF code, shown below, is in RTFPaint.pas. However, this does not quite meet your needs; you want to know the size before drawing it, while my code assumes that you already know the size of the drawing target. To measure RTF text size, call ITextServices.TxGetNaturalSize .
Word wrap is important. Without it, the control will assume that it has infinite width to work, and it will only start a new line when the RTF text requests it.
procedure DrawRTF(Canvas: TCanvas; const RTF: string; const Rect: TRect; const Transparent, WordWrap: Boolean); var Host: ITextHost; Unknown: IUnknown; Services: ITextServices; HostImpl: TTextHostImpl; Stream: TEditStream; Cookie: TCookie; res: Integer; begin HostImpl := TDrawRTFTextHost.Create(Rect, Transparent, WordWrap); Host := CreateTextHost(HostImpl); OleCheck(CreateTextServices(nil, Host, Unknown)); Services := Unknown as ITextServices; Unknown := nil; PatchTextServices(Services); Cookie.dwCount := 0; Cookie.dwSize := Length(RTF); Cookie.Text := PChar(RTF); Stream.dwCookie := Integer(@Cookie); Stream.dwError := 0; Stream.pfnCallback := EditStreamInCallback; OleCheck(Services.TxSendMessage(em_StreamIn, sf_RTF or sff_PlainRTF, lParam(@Stream), res)); OleCheck(Services.TxDraw(dvAspect_Content, 0, nil, nil, Canvas.Handle, 0, Rect, PRect(nil)^, PRect(nil)^, nil, 0, txtView_Inactive)); Services := nil; Host := nil; end;