In general, the impact on performance is not affected; depending on what you do, you can result in exactly the same runtime configuration (and therefore performance).
The most noticeable difference will be caused by the server.xml setting: instead of allowing buildpack to choose the runtime configuration (and the "functions" that will be executed), you can trim the functions to a minimum (by deleting anything you cannot use, for example, the log ) or add additional features (https, jax-rs, ...).
Clicking the entire server is usually necessary if your settings simply do not correspond only to server.xml changes. You might be adding libraries (jar files, such as database drivers) that are not part of the runtime provided by buildpack. Again, this may or may not be related to performance; it depends on what changes you make.
For more information, see the server.xml file by default and some possible settings here . You can also read an example of adding drivers to the buildpack server, in this case MySQL - this does not apply to Bluemix, but rather to Liberty which you can also apply to Bluemix by clicking on the server directory.