Construction is a CPU intensive process. The idea of the build server is that it sits there and builds after each check to make sure that nothing is broken. If you put this on a computer with other resources, it will slow down the rest. Even if you virtualized it, that would probably be a problem.
Another thing is that you want to install a “clean” OS so that other programs do not pollute the computer, adding dependencies that may not be present on client computers.
If you used it only for nightly builds, I would have thought that you would normally work with a virtual machine to do this.