If you are wondering about a “general solution”, and you can track a lot in comparing asymptotic functions. Here is what I recommend:
Use the BigO notation definition constraint as soon as you know:
f(n) = O(g(n)) iff limit (n approaches +inf) f(n)/g(n) exists and is not +inf
You can use a Computer Algebra system , for example open source Maxima , here is the Maxima Documentation on Limits .
So checking lg(n)*lg(n) = O(sqrt(n)) can be a date - this is checking the limit (lg(n)lg(n))/sqrt(n) :
(%i1) limit( (log(n)^2) / (sqrt(n)), n, inf); (%o1) 0
If you like, a longer descriptive notation:
(%i1) f(n) := log(n)^2 ; 2 (%o1) f(n) := log (n) (%i2) g(n) := sqrt(n) ; (%o2) g(n) := sqrt(n) (%i3) limit(f(n)/g(n), n, inf); (%o3) 0