You definitely don't need to invoke Application.Run(a)F # Interactive because it manages it with its own message loop (you really can't do this).
Creating the form and setting Visibleup trueshould work (and it works on my machine!) Unfortunately, I'm not sure what might cause the problem you reported, but this is definitely not expected behavior (this seems to be some error).
ShowDialog - , , F # Interactive . - F # Interactive - , . :
> let a = new Form();;
val a : Form = System.Windows.Forms.Form, Text:
> a.Visible <- true;;
val it : unit = ()
> a.Text <- "Hello";;
val it : unit = ()
( , .)