This has little or nothing to do with the nil values in the table. An error message is generated if the comparison function itself is invalid. From the documentation for table.sort :
If comp is specified, then it must be a function that receives two tables of elements and returns true when first less than the second (so not comp(a[i+1],a[i]) will be true after sorting).
In other words, comp(a,b) should mean not comp(b,a) . If this relation is not satisfied, then the error “incorrect order function for sorting” is likely to be raised. (Note that in all cases this may not be raised.)
To be more useful, we really need to see that the entire function is passed to table.sort .
source share