It seems to me that you are facing the NP-Hard problem.
This is equivalent to a graphic coloring with colors k , where the edges are prohibition lists.
Graphic arts:
Given a graph G=(V,E), and an integer `k`, create coloring function f:V->{1,2,..,k} such that: f(v) = f(u) -> (v,u) is NOT in E
Reduction from coloring the graph to your problem:
Given the problem of coloring the graph (G,k) , where G=(V,E) , create an instance of your problem with:
students = V for each student: student.denial = { student' | for each edge (student,student')}
Intuitively, each vertex is represented by the student, and the student denies all students that there is an edge between the vertices representing them. Number of groups - a given number of colors.
Now, given the solution to your problem, we get groups k that, if student u rejects student v - they are not in the same group, but this is the same as coloring u and v in different colors, so for each edge (u , v) in the original graph, u and v are in different colors.
Another way is similar
So, we got here a polynomial reduction from the problem of graph coloring, and thus find the optimal solution to your problem - NP-Hard, and there is no effective solution to this problem , and most believe that this does not exist.
Some alternatives use heuristics, such as Genetic algorithms , which do not provide optimal solutions or use brute force for a lot of time (which is not possible for a large number of students).
Brute force will generate all possible partitions into groups k and check if this is an acceptable solution, in the end - the best solution will be found.