The piserial buffer fills faster than I can read

Question

I am reading data from the microcontroller through the serial port, at a baud rate of 921600. I am reading a lot of ASCII csv data, and since it arrives so quickly, the buffer is full and all other data is lost before I can read it. I know that I can manually edit the pyserial source code for serialwin32 to increase the buffer size, but I was wondering if it has another way?

I can estimate the amount of data that I will receive, but somewhere around 200 KB of data.

Answer 1

There is a “Receive Buffer” slider, available on the COM port properties page in the device manager. You can find it by clicking the "Advanced" button on the "Port Settings" tab.

Additional Information:

http://support.microsoft.com/kb/131016 under the heading Receive Buffer

http://tldp.org/HOWTO/Serial-HOWTO-4.html under the Interrupt heading

Try to knock it with an incision or two.

Answer 2

Have you considered reading from the serial interface in a separate thread that works before to send a command to uC to send data?

This will remove some delay after the write command and start reading. There are other SO users who have succeeded in this method, provided that they did not have a buffer overflow.

If this is unclear, let me know and I can put something together to show it.

EDIT

Thinking about it a bit more, if you try to read from the buffer and write it to the file system, even a stand-alone stream may not save you. To minimize processing time, you can consider reading 100 bytes at a time serial.Read(size=100) and pushing this data into the queue to process all of this after the transfer is complete

Pseudocode Example

 def thread_main_loop(myserialobj, data_queue): data_queue.put_no_wait(myserialobj.Read(size=100)) def process_queue_when_done(data_queue): while(1): if len(data_queue) > 0: poped_data = data_queue.get_no_wait() # Process the data as needed else: break; 

Answer 3

You do not need to manually change the alias code.

If you run your code on a Windows platform, you just need to add a line to your code

 ser.set_buffer_size(rx_size = 12800, tx_size = 12800) 

Where 12800 is an arbitrary number that I chose. You can make a receive (rx) and transmit (tx) buffer up to 2147483647a in size.

---

See also:

https://docs.python.org/3/library/ctypes.html

https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.readbuffersize(v=vs.110).aspx

Perhaps you can configure the serial port from the DLL // Setup serial

  mySerialPort.BaudRate = 9600; mySerialPort.PortName = comPort; mySerialPort.Parity = Parity.None; mySerialPort.StopBits = StopBits.One; mySerialPort.DataBits = 8; mySerialPort.Handshake = Handshake.None; mySerialPort.RtsEnable = true; mySerialPort.ReadBufferSize = 32768; 

Property Value Type: System.Int32 Buffer size in bytes. The default value is 4096; maximum value is a positive integer or 2147483647

And then open and use it in Python

Answer 4

I am somewhat surprised that no one has already mentioned the correct solution to such problems (when it is available), namely effective flow control through software (XON / XOFF) or hardware flow control between the microcontroller and its receiver. This problem is well described in this article .

It may happen that the source device does not comply with such protocols, and in this case you are stuck with a number of solutions that delegate the problem up to where more resources are available (move it from the UART buffer to the driver and up towards your application code). If you are losing data, it would be wise to try to implement a lower data rate, if possible.