Subprocess command cannot find files with ls?

Question

I am creating a program that pulls out a list of account numbers and then runs the ls -lh to find a file for each of them. When I run my command on our Linux server without Python, it does not cause problems with the files, but then when I do it through Python, it says that it cannot find them.

 import subprocess as sp sp.call(['cd', input_dir]) for i, e in enumerate(piv_id_list): proc_out = sp.Popen(['ls', '-lh', '*CSV*APP*{0}.zip'.format(e)]) proc_out_list.append(proc_out) print(proc_out) 

Here is a sample output when I run commands through the Python interpreter:

→> ls: cannot access * CSV1000 * APP *: no such file or directory

But through Linux, the same command:

 ls -lh *CSV*APP* 

It returns the result as it should.

Answer 1

ls , as it works through Python, is probably right: I think that in the current directory there is no file named *CSV*APP* . There is probably a file with a name that matches this glob pattern. But ls doesn't care about balls. What happens when you run the command in the shell is that the shell extends glob to match the names of the files that it can see in the current directory, and these extended names are what the shell passes to ls .

To get the same result in the shell as in Python (for demonstration, not because you need it), protect the arguments from the glob extension with single quotes:

 ls -lh '*CVS*APP*'${e}'.zip' 

But how can you get shell behavior in Python? You can use shell=True , as some other answers suggest, but this is a slippery slope, since accessing the real shell of your dynamically generated strings (it may depend on user input in a more complex application) can make you vulnerable to command injections and other nasty things .

Here you need only one specific shell behavior, the globbing file name. And Python can do this on its own :

 import subprocess as sp from glob import glob sp.call(['cd', input_dir]) for i, e in enumerate(piv_id_list): proc_out = sp.Popen(['ls', '-lh', glob('*CSV*APP*{0}.zip'.format(e))]) proc_out_list.append(proc_out) print(proc_out) 

As JunCompressor pointed out , it will still look in the wrong directory, because cd will only affect the subdial of the cd call, so let's fix this too:

 import subprocess as sp from glob import glob os.chdir(input_dir) for i, e in enumerate(piv_id_list): proc_out = sp.Popen(['ls', '-lh', glob('*CSV*APP*{0}.zip'.format(e))]) proc_out_list.append(proc_out) print(proc_out) 

The note

Perhaps you can use a slightly higher level of sp.check_output instead of lower sp.Popen .

Answer 2

This is because the shell replaces wildcards with existing files that match the pattern. For example, if you have a.txt and b.txt , then ls *.txt will be expanded from the shell to ls a.txt b.txt . With your team, you really ask ls to return information about the file containing an asterisk in the file name. Follow these steps to confirm:

 sp.Popen(['bash', '-c', 'ls', '-lh', '*CSV*APP*{0}.zip'.format(e)]) 

You should also use os.chdir to change the directory, since sp.call(['cd', input_dir]) changes the current directory for the new process that you created, not the parent.

Answer 3

You must use the cwd argument of Popen and shell=True , then communicate to get the result.

Your code will look like this:

 import subprocess as sp for i, e in enumerate(piv_id_list): proc = sp.Popen(['ls', '-lh', '*CSV*APP*{0}.zip'.format(e)], cwd=input_dir, stdout=sp.PIPE, shell=True) proc_out_list.append(proc.communicate()[0]) print(proc_out_list[-1]) 

But why are you doing a subprocess instead of using the standard library?

• https://docs.python.org/2/library/os.html#os.listdir

Edit

Like @tripleee , it replaces only a few features. I think it is better to use builtins / stdlib whenever possible; in your case, you “only” want to list the files for this template ( glob ) and show sorted information about their size ( stat ).

Using stdlib makes your code more portable; even if you don't care about portability of Microsoft Windows, you might want to avoid the unexpectedness of running your code on a computer without GNU binutils (that is: Mac OS, BSD, ...).

You want to use the subprocess module for things that cannot (easily) be implemented in pure Python (i.e. .: encode a video with ffmpeg , change userpassowrd with passwd , elevate privileges with sudo , ...).

Answer 4

I believe that you need to add shell=True as the Popen parameter and replace the list with one line:

 proc_out = sp.Popen('ls -lh *CSV*APP*{0}.zip'.format(e), shell=True) 

See here for more info and possible use of glob : Python subprocess subprocess