How to track the progress of a linux command?

Question

Currently, I am working with a large data set, where even converting the file format takes at least an hour for one subject, and as a result I often do not know if my command was executed or the program froze. I was wondering if anyone has a hint on how to monitor the progress of the commands / scripts that I'm trying to run on Linux?

Your help would be greatly appreciated.

Answer 1

If the long-lasting command produces some output in the foo.out file, you can do watch ls -l foo.out or tail -f foo.out

You can also list /proc/$(pidof prog)/fd to find out the open files of some prog

Answer 2

In addition to @ basile-starynkevitch's answer,
I have a bash script that can measure how many files you processed as a percentage.

He looks in procfs to get the current position from the fd information ( /proc/pid/fdinfo ) and count it as a percentage of the total file size.

See https://gist.github.com/azat/2830255

 curl -s https://gist.github.com/azat/2830255/raw >| progress_fds.sh \ && chmod +x progress_fds.sh Usage: ./progress_fds.sh /path/to/file [ PID] 

May be useful to someone

Answer 3

You can monitor program system calls with strace , which allows you to monitor open calls.

Answer 4

You can use verbose output, but it will slow down work even more.

Answer 5

I think there can be no general answer to this question, it just depends on the type of program (which you don’t even need to do with Linux, see the “stopping problem”).

If you use the handset during the conversion, I find the pv(1) tool quite useful. Even if pv cannot know the total size of the data, it helps to see if there is real progress and how good the datarate is. It is not part of most standard installations and should probably be installed explicitly.