Paramiko hangs while executing a large wget command

Question

Hi, I am having problems executing a command that executes wget from a 100mb file through an Ubuntu 10 server. Shorter commands work fine except for this. The class below contains information on how I use paramiko and other attempts to overcome this problem (see Various run or exec methods). In the case of exec_cmd, execution hangs on this line:

out = self.in_buffer.read(nbytes, self.timeout) 

from the recv method of the channel.py module from paramiko.

The same wget command works fine in the shell using the regular Mac ssh utility.

 """ Management of SSH connections """ import logging import os import paramiko import socket import time import StringIO class SSHClient(): def __init__(self): self._ssh_client = paramiko.SSHClient() self._ssh_client.load_system_host_keys() self._ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) self.time_out = 300 self.wait = 5 def connect(self, hostname, user, pkey): retry = self.time_out self.hostname = hostname logging.info("connecting to:%s user:%s key:%s" % (hostname, user, pkey)) while retry > 0: try: self._ssh_client.connect(hostname, username=user, key_filename=os.path.expanduser(pkey), timeout=self.time_out) return except socket.error, (value,message): if value == 61 or value == 111: logging.warning('SSH Connection refused, will retry in 5 seconds') time.sleep(self.wait) retry -= self.wait else: raise except paramiko.BadHostKeyException: logging.warning("%s has an entry in ~/.ssh/known_hosts and it doesn't match" % self.server.hostname) logging.warning('Edit that file to remove the entry and then try again') retry = 0 except EOFError: logging.warning('Unexpected Error from SSH Connection, retry in 5 seconds') time.sleep(self.wait) retry -= self.wait logging.error('Could not establish SSH connection') def exists(self, path): status = self.run('[ -a %s ] || echo "FALSE"' % path) if status[1].startswith('FALSE'): return 0 return 1 def shell(self): """ Start an interactive shell session on the remote host. """ channel = self._ssh_client.invoke_shell() interactive_shell(channel) def run(self, command): """ Execute a command on the remote host. Return a tuple containing an integer status and a string containing all output from the command. """ logging.info('running:%s on %s' % (command, self.hostname)) log_fp = StringIO.StringIO() status = 0 try: t = self._ssh_client.exec_command(command) except paramiko.SSHException: logging.error("Error executing command: " + command) status = 1 log_fp.write(t[1].read()) log_fp.write(t[2].read()) t[0].close() t[1].close() t[2].close() logging.info('output: %s' % log_fp.getvalue()) return (status, log_fp.getvalue()) def run_pty(self, command): """ Execute a command on the remote host with a pseudo-terminal. Returns a string containing the output of the command. """ logging.info('running:%s on %s' % (command, self.hostname)) channel = self._ssh_client.get_transport().open_session() channel.get_pty() status = 0 try: channel.exec_command(command) except: logging.error("Error executing command: " + command) status = 1 return status, channel.recv(1024) def close(self): transport = self._ssh_client.get_transport() transport.close() def run_remote(self, cmd, check_exit_status=True, verbose=True, use_sudo=False): logging.info('running:%s on %s' % (cmd, self.hostname)) ssh = self._ssh_client chan = ssh.get_transport().open_session() stdin = chan.makefile('wb') stdout = chan.makefile('rb') stderr = chan.makefile_stderr('rb') processed_cmd = cmd if use_sudo: processed_cmd = 'sudo -S bash -c "%s"' % cmd.replace('"', '\\"') chan.exec_command(processed_cmd) result = { 'stdout': [], 'stderr': [], } exit_status = chan.recv_exit_status() result['exit_status'] = exit_status def print_output(): for line in stdout: result['stdout'].append(line) logging.info(line) for line in stderr: result['stderr'].append(line) logging.info(line) if verbose: print processed_cmd print_output() return exit_status,result def exec_cmd(self, cmd): import select ssh = self._ssh_client channel = ssh.get_transport().open_session() END = "CMD_EPILOGqwkjidksjk58754dskhjdksjKDSL" cmd += ";echo " + END logging.info('running:%s on %s' % (cmd, self.hostname)) channel.exec_command(cmd) out = "" buf = "" while END not in buf: rl, wl, xl = select.select([channel],[],[],0.0) if len(rl) > 0: # Must be stdout buf = channel.recv(1024) logging.info(buf) out += buf return 0, out 

Answer 1

• In this case, I would go with adding a list and then concatenating. What for? Well, strings are immutable in Python. This means that every time you use += , you basically create two new lines and read the third. On the other hand, if you create a list and add it, you will halve the number of lines created.
• Do you really need to call select several times? My understanding is that you don't care if the process is thread blocking. Since select is more or less a wrapper around a C method with the same name:

  select () and pselect () allow the program to track multiple file descriptors, expecting one or more file descriptors to become "ready" for some class of I / O operations (for example, input is possible). The file descriptor, if possible, to perform the corresponding input-output operation (for example, read (2)) without blocking.

• You are not listening on socket.timeout exception in your code.
• Writing to stdout / file system can be expensive, but you register every line that recv returns. Can you move the log line?
• Have you considered handling channel reading manually? The only code you need:

 try: out = self.in_buffer.read(nbytes, self.timeout) except PipeTimeout, e: # do something with error 

This is not guaranteed, but it cuts out additional processing.

Answer 2

I had the same problem, my python script hung when the shell script that I ran on the remote ssh client ran the wget command in a 400 MB file.

I found that adding a timeout to the wget command fixes the problem. I initially had:

wget http: // blah: 8888 / file.zip

now with this:

wget -q -T90 http: // blah: 8888 / file.zip

It works like a charm!

Hope this helps.