Printing in stdout in parallel IPython processes

Question

I am new to IPython and would like to print intermediate results for stdout when running IPython parallel cluster functions. (I know that with several processes this can lead to output failure, but it is fine - it is just for testing / debugging, and the processes that I started are long enough so that such a collision is unlikely.) I checked the documentation for IPython, but not can find an example when the parallelism function is printed. Basically, I'm looking for a way to redirect subprocess output to the main stdout, the equivalent of IPython

subprocess.Popen( ... , stdout=...) 

Printing inside the process does not work:

 rc = Client() dview = rc() def ff(x): print(x) return x**2 sync = dview.map_sync(ff,[1,2,3,4]) print('sync res=%s'%repr(sync)) async = dview.map_async(ff,[1,2,3,4]) print('async res=%s'%repr(async)) print(async.display_outputs()) 

returns

 sync res=[1, 4, 9, 16] async res=[1, 4, 9, 16] 

So, the calculation is done correctly, but the print statement in the ff function is never printed, even when all processes are returned. What am I doing wrong? How to get a "print" for work?

Answer 1

Actually this is more like subprocess.Popen( ... , stdout=PIPE) than you expect. Just as a Popen object has a stdout attribute that you can read to see the stdout subprocess, AsyncResult has a stdout attribute that contains stdout taken from the engines. It differs in that AsyncResult.stdout is a list of strings , where each item in the list is the standard tone of one engine as a string.

So, for starters:

 rc = parallel.Client() dview = rc[:] def ff(x): print(x) return x**2 sync = dview.map_sync(ff,[1,2,3,4]) print('sync res=%r' % sync) async = dview.map_async(ff,[1,2,3,4]) print('async res=%r' % async) async.get() 

gives

 sync res=[1, 4, 9, 16] async res=<AsyncMapResult: ff> 

We can see the list of AsyncResult.stdout lines:

 print(async.stdout) ['1\n2\n', '3\n4\n'] 

We can see the source code of the asynchronous result:

 print('async output:') async.display_outputs() 

which prints:

 async output: [stdout:0] 1 2 [stdout:1] 3 4 

And here is a laptop with everything demonstrated.

Some notes based on your question:

• you need to wait for AsyncResult to complete before the outputs are ready ( async.get() )
• display_outputs() does not return anything - it actually does print / display, so print(async.display_outputs()) does not make sense.