Because it seems to you that you are in a terrible state, I offer some terrible options (and one option that sounds awful, but not so bad).
First option: use a different tool. strace(1) and ltrace(1) can both provide an amazing array of information - although nothing knows anything about shell variables, they will show you how scripts interact with the rest of the system and some internal states of the program. This can be quite good. (If you are new to strace(1) , try strace -f -o /tmp/foo ./program - it will follow the calls to fork(2) , vfork(2) and clone(2) , so the child processes will execute and reset too. The output goes to /tmp/foo .)
Second choice: replace all #!/bin/bash with #!/bin/bash -x in all your scripts:
find . -type f -print0 | xargs -0 sed -i -e 's/^#!\/bin\/bash$/#!\/bin\/bash -x/'
There may be a better mechanism for replacing the shebang string in all of your scripts, but that's all right. It will skip all uses of system(3) , popen(3) , etc. In C programs that invoke through your scripts, but this can be very good. If any scripts rely on the /bin/bash arguments, you may need to work harder to add -x correctly.
Last choice: run the sash(1) shell as root. Copy /bin/bash to /bin/real.bash . Write a quick, dirty C program that adds -x to the command line arguments in /bin/real.bash and /bin/real.bash them in /bin/bash . This will get everything: every system(3) , every popen(3) , every init script, all cron jobs, all user logins, everything.
You can slightly change this last choice; The Linux kernel provides private namespaces for each process, which can be used to make the new /bin/bash visible only to children of the process. This is a little trickier ...
- Make a copy of
bash : cp /bin/bash /bin/real.bash - Write your C shell to add
-x to the command line arguments and call /bin/real.bash . (Lets call him /bin/wrapper.bash .) - Install
/ mount shared : mount --make-shared / - Create a new file system namespace:
unshare --mount bash - In the new
bash make / subordinate: mount --make-slave / - In the new
bash bind your bash replacement: mount -B /bin/wrapper.bash /bin/bash - In the new
bash run shell scripts. All of them will be redirected to your wrapper. Processes not related to the new bash will continue to use the "real" /bin/bash , which has remained unchanged.
When the last (large) * child process dies, so does the personal namespace and the fun binding.
The result of all this ridiculous business is that all your shell scripts remain unchanged, and you can use the debugging tool, which, in your opinion, will be most useful. If you squint your eyes right to the right, it’s not very intrusive.
I followed many of these steps on my workstation (except that I used --make-rshared and --make-rslave and tested using /bin/dir and /bin/ls instead of /bin/bash )) and checked the inode numbers with ls -li /bin/dir /bin/ls from shells in the new namespace and outside the new namespace, and processes outside the namespace continued to see index numbers that remained unchanged, but processes in the namespace saw dir and ls common index numbers.
For detailed information on namespaces, private mounts, and binding bindings, I recommend reading the Documentation/filesystems/sharedsubtree.txt from the kernel source tree, the clone(2) unshare(1) page, unshare(1) page and mount(8) unshare(1) page .
Forced mounts disappear when the system reboots, so you can't mess up anything. :)