I want to avoid the fact that user settings for fonts affect the font size for various TextViews, etc. in my application.
Users will only change the font size if they believe that larger (or possibly smaller) fonts improve the use of their device. Your application should be customized to match using sp units for text sizes. Users may not like your application if your application refuses to fulfill their request.
What is the best way?
Do not use sp units for text sizes and be prepared for negative reviews in the Play Store.