If you want your menu to be inside the firefox window, then ... I do not know the details, but the usual way in X is to create an X window (for example, GtkWindow) with your GtkMenu. Then you should make firefox "swallow" this window, something like all systray utilities, that (keywords: reparenting X window - should be enough). I suppose there must be a special XUL element for this, but I do not see it.
If you do not want this, you can simply create your own GtkWindow with GtkMenu and show it. I think you should do it from another thread. As for your other question here, it is entirely possible to write an extension in C ++ if you use the Gecko SDK and Gecko API. You can then link any library you want, including GTK.
I think you should ask this question on one of the Mozilla mailing lists, for example mozilla.dev.extensions (they are listed at http://www.mozilla.org/community/developer-forums.html ).
liori
source share