This is pretty well studied, as pointed out by mcdowella's comment. This is how the random Cantor-Sassenhaus algorithm works for the case when you want to find the roots of a polynomial, rather than a more general factorization.
Note that in the ring of polynomials with coefficients mod p the product x (x-1) (x-2) ... (xp + 1) has all possible roots and is equal to x ^ px via Fermat’s Little Theorem and the only factorization in this ring.
Set g = GCD (f, x ^ px). Using the Euclidean algorithm to compute the GCD of two polynomials, is fast overall, performing a series of steps that are logarithmic to the maximum extent. This does not require you to define polynomials. g has the same roots as f in the field, and non-repeating factors.
Due to the special form of x ^ px with two nonzero terms, the first step of the Euclidean algorithm can be performed by re-squaring approximately 2 log_2 (p) with the participation of only polynomials of degree no more than two times the degree of f with coefficients mod p. We can compute x mod f, x ^ 2 mod f, x ^ 4 mod f, etc., and then multiply the numbers corresponding to nonzero places in the binary expansion of p to calculate x ^ p mod f and finally subtract x .
Repeat the following: select random d in Z / p. Calculate the GCD of g with r_d = (x + d) ^ ((p-1) / 2) -1, which we can quickly calculate using the Euclidean algorithm using repeated squaring in the first step. If the degree of this GCD is strictly between 0 and the degree of g, we find a non-trivial factor g, and we can recurse until we find linear factors, therefore, the roots of g and, therefore, f.
How often does it work? r_d has as its roots numbers that are less than d than the nonzero square mod p. Consider two different roots of g, a and b, therefore (xa) and (xb) are factors of g. If a + d is a nonzero square, but b + d is not, then (xa) is a common factor of g and r_d, while (xb) is not, which means that GCD (g, r_d) is a nontrivial factor of g, Similarly, if b + d is a nonzero square, but a + d is not, then (xb) is a common factor of g and r_d, and (xa) is not. According to number theory, one case or another case is close to half of the possible options for d, which means that on average it takes a constant number of options d before it finds a non-trivial factor g, which actually divides (xa) from (xb).