Try connecting via ssh using the -X switch. This will allow you to remotely use X applications. It will be a little slow, but it should be used for many applications.
ssh -X you@yourserver $ firefox &
Edit: I just checked this on one of my ubuntu servers. It took ~ 140 MB of additional depots to install firefox (without ubuntu-desktop or xserver).
dubvfan87
source share