Subprocess.check_output without using large memory

Question

In my current project, I have a web server that calls Linux commands to retrieve information that is then displayed on the website. The problem with this is that the web server is running on a tiny embedded device (it is basically a configuration tool for the device), which only has 256 MB of RAM. The web server itself takes up more than half of the free RAM that I have on this device.

Now, when I try to use subprocess.check_output () to call the command, fork soon doubles the RAM usage (because it clones the parent process or something, as I understand it) and thus breaks everything into "Out of Memory", though the process being called is pretty tiny.

Since the device uses fairly cheap flash chips, which turned out to be unsuccessful if abused, I do not want to use any swap or other solutions based on increasing virtual memory.

What I tried to do so far is to pop up the sh session at the beginning of the program, when it still remains low when using memory, and then write commands to this sh session and read the output. This view works, but it is rather unstable, because the wrong "exit" or something like that can destroy it all.

Is there any solution like subprocess.check_output () that doesn't double my memory usage?

Answer 1

So, with the help of J. F. Sebastian, I realized this.

This is the code I used at the end:

from multiprocessing import Process, Queue from subprocess import check_output, CalledProcessError def cmdloop(inQueue,outQueue): while True: command = inQueue.get() try: result = check_output(command,shell=True) except CalledProcessError as e: result = e outQueue.put(result) inQueue = Queue() outQueue = Queue() cmdHostProcess = Process(target=cmdloop, args=(inQueue,outQueue,)) cmdHostProcess.start() def callCommand(command): inQueue.put(command) return outQueue.get() def killCmdHostProcess(): cmdHostProcess.terminate() 

In Python 3.4+, I could use multiprocessing.set_start_method ('forkserver'), but since this works on Python 2.7, this is unfortunately not available.

However, this reduces my memory usage in a long shot and fixes the problem in a clean way. Thank you for help!