as far as I can judge that the problem seems to be possible overflows when the numbers get too big for Int - in this case, most likely in the x * x - 1 func ( Int is maxBound from 9223372036854775807 in my system)
So, the easiest option is to simply switch to Integer everywhere, as they are not limited:
func :: Integer -> Integer -> Integer ... pollardStep :: Integer -> Integer -> Integer -> Integer -> Integer -> Integer ... pollard_rho :: Integer -> Integer ...
this of course will make things a little slower though
source share