(Nothing in your question uses existential types. You have an Approximate constructor that has a polymorphic argument, resulting in Approximate having a rank-2 type and leading to problems with higher-ranking types and output type.)
Short answer: style without dots and types of a higher rank do not combine with each other. Avoid using composition of functions if polymorphic arguments are involved and stick to a simple application or $ , and everything will be fine. The direct way to write approx in a way that is accepted is as follows:
approx :: (forall a . Floating a => a -> a) -> AugmentedRational -> AugmentedRational approx f ar = Approximate (f (approximateValue ar))
The problem is that the GHC does not support the proper support for "unclean" types. This means: if a function is polymorphic, its type variables can be created with monomorphic types, but not with those types that are again polymorphic. Why does it matter here?
See what you wrote:
approx :: (forall a.Floating a => a -> a) -> AugmentedRational -> AugmentedRational approx f = Approximate . f . approximateValue
Here you use the composition function ( . ) Twice. The type of function composition is as follows:
(.) :: (b -> c) -> (a -> b) -> a -> c infixr 9 .
So, the above definition is analyzed as
Approximate . (f . approximateValue)
But
Approximate :: (forall a. Floating a => a) -> AugmentedRational
has a rank-2 type. Thus, matching an Approximate type with the first argument (.) Means that:
b = forall a. Floating a => a c = AugmentedRational
.
This is a description of instance b for a polymorphic type that the GHC does not allow. It offers ImpredicativeTypes as a language extension that can make it work, but unfortunately it is a very fragile language extension, and using this is usually not recommended. And, as you saw, even with ImpredicativeTypes enabled, GHC, as a rule, still requires quite a few additional type annotations, so your program will not work without additional changes.
An application with a normal function is built into the GHC and is handled differently during type checking. This is why a more accurate definition of approx works. Using $ also good, but only because the GHC has a special hack telling about type checking that $ really no different from an application.