Pass a lot of data from a Python program to C

Question

I have a Python script and a C program, and I need to transfer large amounts of data from a Python script that repeatedly call a C program. Right now I let the user choose between transferring them using an ASCII file or a binary file, but both are pretty slow and useless (I mean files that are useful if you want to save data, but I delete these files at the end of the script).

os.system does not work, there are too many arguments, since the C program also uses files to return data in Python, but this is much less data.

I wonder what I can use for a quick exchange? Writing files to a RAM disk? If so, how can I do this?

I heard that you can call functions from a DLL using ctypes, but I don’t know how to compile my program as a DLL (I use wxdevc + on Windows 7 64). Or wrap it, but still don't know if it can work and whether it is effective.

Data are the vertices of a three-dimensional grid.

I run a Python script inside another program ( blender (open source)), and it is called many times (usually more than 500 times) because it is inside a loop. The script sends vertex information (1 int index and 3 floatordinates) to the program and program should return many vertices (only int index, because I can find the corresponding vertices using Python).

So this is not interactive, it is more like a function (but it is written in C). The script + C program (which is in addition to blender) that I am writing must be cross-platform because it will be distributed.

The program is actually written in C, and from Python I can find out the address in the memory of the structure containing the vertex data. If only I know how to do this, it would be better to transfer only the address to the C-program, and from there find all the other vertices (stored in the list).

But, as far as I know, I can’t access the memory space of another program, and I don’t know if it calls a program with pipelines or something else, it initializes a new thread or runs inside a script (which actually runs under a blender thread )

Here is the source, and blender/source/blender/makesdna/DNA_meshdata_types.h should be a structure definition

Answer 1

Pipes are the obvious way; if your c program accepts input from stdin, you can use Popen . This does not create a “stream”, as you say in your editing; it creates a completely new process with separate memory:

 from subprocess import Popen, PIPE input = "some input" cproc = Popen("c_prog", stdin=PIPE, stdout=PIPE) out, err = cproc.communicate(input) 

Here is a more detailed example. Firstly, a simple c-program that has something in common with stdin:

 #include<stdio.h> #include<stdlib.h> #define BUFMAX 100 int main() { char buffer[BUFMAX + 1]; char *bp = buffer; int c; FILE *in; while (EOF != (c = fgetc(stdin)) && (bp - buffer) < BUFMAX) { *bp++ = c; } *bp = 0; // Null-terminate the string printf("%s", buffer); } 

Then a python program that inputs the input (from argv in this case) to the following:

 from subprocess import Popen, PIPE from sys import argv input = ' '.join(argv[1:]) if not input: input = "no arguments given" cproc = Popen("./c_prog", stdin=PIPE, stdout=PIPE) out, err = cproc.communicate(input) print "output:", out print "errors:", err 

If you don't plan on using the c program without the python interface, however, you might be better off nesting a c function, possibly using instant .

 from instant import inline c_code = """ [ ... some c code ... ] //see the below page for a more complete example. """ c_func = inline(c_code) 

As Joe points out, you can also write a python module in c: Extending Python with C or C ++

This answer discusses other ways to combine c and python: How do I plug in a Python and C program?

EDIT: Based on your editing, it sounds like you really have to create a cpython extension. If you need some sample code, let me know; but a full explanation would make for an unreasonably long answer. See the link above (Python Extension ...) for everything you need to know.

Answer 2

If your operating system supports it, named pipes is a file replacement.

Answer 3

The idea here is slightly different from the others: write your C program as a Python module. Here is all the information necessary for this. You can then pass large buffers between your Python code and C code.

Answer 4

I was never happy with the answers to linking python and C, so I wrote the answer after a lot of research and thinking.

test.c

 #include <stdio.h> #include <stdlib.h> int main() { FILE *fp; char path[1035]; fp = popen("python3 output2.py", "r"); // Open the command for reading. if (fp == NULL) { printf("Failed to run command\n" ); exit(1); } while (fgets(path, sizeof(path), fp) != NULL) printf("C received %s", path); // Read the output. pclose(fp); // close return 0; } 

output2.py

 import time import os, sys i = 0 while True : print("%d" %(i), flush=True) time.sleep(1) i = i + 1