Mixing Python and Go

Question

I am working on a library in Python, and I would like to make some performance improvements.

Is it possible to write code in Python and some code in Go and pass data between them? And if possible, are there any examples of how to do this?

Like this:

# Python def python_foo(): data = {'foo': 'val', 'bar': [1, 2, 3]} go_process(json.dumps(data)) def python_got_data_from_go(data): # deal with data from Go # Go func go_process(json string) { // do some processing python_got_data_from_go(someData) } 

Answer 1

You need glue between them, for example C programming language or network connection. The most painful solution is if you mix https://docs.python.org/2/extending/extending.html with http://golang.org/cmd/cgo/ and good C programming skills.

You can create a server in python http://pymotw.com/2/socket/tcp.html and in go https://coderwall.com/p/wohavg and exchange data between them.

Edit: see Writing a Python extension in Go (Golang) and Calling a Python function from Go and getting the return value of the function

Answer 2

The easiest way to connect languages ​​is to communicate through a socket, such as a TCP / IP or Unix socket, or through a high-level protocol such as HTTP or XML-RPC. This is associated with very high overhead, although due to query processing and serialization / deserialization to / from JSON / XML, which can be significant if you need to make a lot of calls. Socket communication is usually best if the workload of the request is high.

If you don't want to pay for socket overhead (say if you make thousands of requests back and forth between python and go in seconds), there are other solutions that may have less overhead. You might be able to use shared memory in OSs that have them. Shared memory usually incurs a much cheaper cost of data access, but it can incur the cost of boxing / unpacking from the general memory structure to Python data types. Also, note that you may need to operate the locks yourself.

If you make only a very small number of calls, and they do not need to share between states between calls, I would not recommend communicating using the standard stdin / stdout.

The final alternative is to write a Python Extension; I would not recommend this for the faint of heart.

Answer 3

The easiest solution is to start the Go process from Python and link them using the standard Go process threads ( os.Stdin and os.Stdout ). You should come up with a protocol on which both parties agree (it seems you already installed JSON) and don't forget to clear the stream after streaming a logical atomic request from both ends.

Thus, your solution is (almost) cross-platform and very easy to configure.