Is it possible to display the sorting progress in linux?

Question

My work involves a large number of sorting fields from very large files. I usually do this with the sort command in bash. Unfortunately, when I start a variety, I’m never sure how long it will take. Do I have to wait a second for the results to appear, or should I start working on something else during its launch?

Is there any possible way to get an idea of ​​how a career progresses or how fast it works?

 $ cut -d , -f 3 VERY_BIG_FILE | sort -du > output 

Answer 1

No, GNU sort does not execute progress reports.

However, if you use sort only to remove duplicates, and you really don't care about ordering, then a more scalable way of doing this:

 awk '! a[$0]++' 

This records the first occurrence of the string as soon as it is visible, which may give you an idea of ​​progress.

Answer 2

You might want to give pv a try, it should give you a pretty good idea of ​​what is going on in your pipe in terms of bandwidth.

An example (untested) introducing pv before and after the sort command to get an idea of ​​throughput:

 $ cut -d , -f 3 VERY_BIG_FILE | pv -cN cut | sort -du | pv -cN sort > output 

EDIT : I skipped -u in your sort command, so calculating strings first to get percentage output is not valid. Removed this part from my answer.

Answer 3

You can do your sorting in the background, you will get a hint, and you will be able to complete other tasks

$ sort ...... and # (&. means running in the background)