There are pros and cons to each ...
jQueryUI gives you a whole bunch of tools with a whole bunch of configuration options. jQueryUI also provides you with themes that you can apply to all tools. You do not need to install the entire jQueryUI library to use a single component, so bulk work should not be a problem. However, I think that with all the parameters a steeper learning curve appears.
Individual plugins, as a rule, are a more complete solution out of the box, IMHO. Some plugins are well written and some are messy. You will have to research each and thoroughly test the demo version. You may need to tweak them and customize the CSS to your liking, but I think that when you choose a quality plugin like SimpleModal, you are much closer to the final work product than if you started with jQueryUI.