I would like to open several kernels from the same ipython session, run the code on these kernels, and then collect the results. But I cannot figure out how to collect the results, or even see stdout / stderr. How can i do this?
What i still have
I managed to complete the first two steps (open the kernel and run the code on them) with code like the following:
from jupyter_client import MultiKernelManager kernelmanager = MultiKernelManager() remote_id = kernelmanager.start_kernel('python3') remote_kernel = kernelmanager.get_kernel(remote_id) remote = remote_kernel.client() sent_msg_id = remote.execute('2+2')
[I welcome any suggestions on how to improve this, or how to close these kernels and clients.]
Here, python3 can be the name of any of the kernels I installed (which can be specified on the command line using jupyter-kernelspec list ). And I seem to be able to use any reasonable code instead of '2+2' . For example, I can write to a file, and this file is actually created.
Now the problem is how to get the result. I can get a message that seems to be related to
reply = remote.get_shell_msg(sent_msg_id)
This answer is a dictionary:
{'buffers': [], 'content': {'execution_count': 2, 'payload': [], 'status': 'ok', 'user_expressions': {}}, 'header': {'date': datetime.datetime(2015, 10, 19, 14, 34, 34, 378577), 'msg_id': '98e216b4-3251-4085-8eb1-bfceedbae3b0', 'msg_type': 'execute_reply', 'session': 'ca4d615d-82b7-487f-88ff-7076c2bdd109', 'username': 'me', 'version': '5.0'}, 'metadata': {'dependencies_met': True, 'engine': '868de9dd-054b-4630-99b7-0face61915a6', 'started': '2015-10-19T14:34:34.265718', 'status': 'ok'}, 'msg_id': '98e216b4-3251-4085-8eb1-bfceedbae3b0', 'msg_type': 'execute_reply', 'parent_header': {'date': datetime.datetime(2015, 10, 19, 14, 34, 34, 264508), 'msg_id': '2674c61a-c79a-48a6-b88a-1f2e8da68a80', 'msg_type': 'execute_request', 'session': '767ae562-38d6-41a3-a9dc-6faf37d83222', 'username': 'me', 'version': '5.0'}}
This is described in Messages in Jupyter . What is not documented is how to actually use this - that is, what functions I use, when and where to find messages, etc. I saw this question and its answer, which has useful related information, but does not give me an answer. And this answer does not get any useful conclusion.
So, for example, I also tried to get msg with msg_id specified in the result above, but it just freezes. I tried everything I could think of, but I can’t figure out how to get something back from the kernel. How can I do it? Can I transfer data from the kernel to some string? Can I see its stdout and stderr?
Background
I am writing ipython magic to run a piece of code on remote kernels. [Edit: This now exists and is available here .] The idea is that I will have a laptop on my laptop and collect data from several remote servers, just having a small magic cell:
%%remote_exec -kernels server1,server2 2+2 ! hostname
I use remote_ikernel to easily and automatically connect to these remote kernels. This seems to work fine; I have my magic team with all its bells and whistles working perfectly, opening these remote kernels and running the code. Now I want to get some data from a remote send back to my laptop - apparently serializing it somehow. At the moment, I think pickle.dumps and pickle.loads would be ideal for this part; I just need to get these bytes created and used by these functions from one core to another. I would prefer not to use actual files for etching, although that would be potentially acceptable.
Edit:
It seems like it's possible with some kind of monster:
remote.get_shell_msg(remote.execute('import pickle')) sent_msg_id = remote.execute('a=2+2', user_expressions={'output':'pickle.dumps({"a":a})'}) reply = remote.get_shell_msg(sent_msg_id) output_bytes = reply['content']['user_expressions']['output']['data']['text/plain'] variable_dict = pickle.loads(eval(output_bytes))
And now variable_dict['a'] just 4 . Note, however, that output_bytes is a string representing these bytes, so it should be eval ed. This seems ridiculous (and still doesn't show how I get stdout). Is there a better way? And how do I get stdout?
Edit 2:
Although I am not happy with my hack above, I have successfully used it to write a small module called remote_exec hosted on github as described above. The module gives me some ipython magic that I can use to remotely run code on one or more other kernels. This is a more or less automated process that I am definitely pleased with - except that they know what is going on below.