Geek Answers Handbook

In Python, how to write a string to a file on a remote machine?

On Machine1, I have a Python2.7 script that calculates a large (up to 10 MB) binary string in RAM that I would like to write to a file on Machine2, which is the remote machine. What is the best way to do this?

Limitations:

  • Both machines are Ubuntu 13.04. Communication between them is fast - they are on the same network.

  • The destination directory does not yet exist on Machine2, so it must be created.

  • If this is easy, I would like to avoid writing a line from RAM to a temporary disk file on Machine1. Does this fix the solutions that can use the system call for rsync?

  • Since the string is binary, it can contain bytes that can be interpreted as a new string. This seems to rule out solutions that might use the system invocation of the echo command on Machine2.

  • I would like it to be as small as possible on Machine2. Thus, I would like to avoid starting services such as ftp on Machine2, or participate in other configuration activities there. In addition, I do not understand that this is good, and therefore I would like to avoid opening additional ports if it is really necessary.

  • I have ssh keys installed on Machine1 and Machine2 and would like to use them for authentication.

  • EDIT: Machine1 works with multiple threads, and therefore it is possible that more than one thread might try to write to the same file on Machine2 at the time of overlapping. I do not mind the inefficiency caused by the fact that the file is written twice (or more) in this case, but the resulting data file on Machine2 should not be corrupted by simultaneous recording. Perhaps OS lock on Machine2 is required?

I use the rsync solution because it is a standalone object that I understand quite well and does not require configuration on Machine2.

+8
python file ssh network-programming rsync
source share
4 answers

You open a new SSH process for Machine2 using subprocess.Popen , and then write your data to STDIN.

 import subprocess cmd = ['ssh', 'user@machine2', 'mkdir -p output/dir; cat - > output/dir/file.dat'] p = subprocess.Popen(cmd, stdin=subprocess.PIPE) your_inmem_data = 'foobarbaz\0' * 1024 * 1024 for chunk_ix in range(0, len(your_inmem_data), 1024): chunk = your_inmem_data[chunk_ix:chunk_ix + 1024] p.stdin.write(chunk)

I just checked that it works as advertised, and copies all empty bytes 10485760.

PS . A potentially cleaner or more elegant solution would be for the Python program to write its output to sys.stdout instead and make the ssh pipe from the outside:

 $ python process.py | ssh <the same ssh command>
+5
source share

Paramiko supports opening files on remote machines:

 import paramiko def put_file(machinename, username, dirname, filename, data): ssh = paramiko.SSHClient() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh.connect(machinename, username=username) sftp = ssh.open_sftp() try: sftp.mkdir(dirname) except IOError: pass f = sftp.open(dirname + '/' + filename, 'w') f.write(data) f.close() ssh.close() data = 'This is arbitrary data\n'.encode('ascii') put_file('v13', 'rob', '/tmp/dir', 'file.bin', data)
+13
source share

Just calling the subprocess is all you need, maybe sh.py might be right.

 from sh import ssh remote_host = ssh.bake(<remote host>) remote_host.dd(_in = <your binary string>, of=<output filename on remote host>)
0
source share

A solution in which you do not explicitly send data over some connection will use sshfs . You can use it to mount a directory from Machine2 somewhere on Machine1, and writing to a file in that directory will automatically result in writing data to Machine2.

0
source share

More articles:


All Articles