Do not disable /Library/LaunchDaemons/org.macosforge.xquartz.privileged_startx.plist. This will prevent XQuartz.app from starting.
Disabling / Library / LaunchAgents / org.macosforge.xquartz.startx.plist (and switching) will cause the DISPLAY environment variable to not be set and thus automatically launch XQuartz.app when you start X11 applications, but you all you can run XQuartz.app directly.
There is no default X server. X11 has been removed from OS X in Mountain Lion. X11.app in earlier versions of OS X was simply XQuartz.app with a different name.
source share