Queue in Python and Threading Confusion

Question

I made a sniffer in Python that calculates the size of IP packets. I tried to get the size in two ways:

1) just calculating len (pkt)

2) by extracting the packet length from the IP header and adding 14 bytes for the ethernet header

When comparing the result from len (pkt) with the extracted value from the IP header, they were almost always the same (ok, for very few packets the difference was 4-6 bytes, but this is another question).

But as soon as I implemented the queues and threads into my code, the sizes from len (pkt) and the extracted value from the IP header are completely different in most cases. Sometimes there is a difference of several bytes, and sometimes several hundred bytes. But very rarely they are the same.

Below is the code in which I implemented the threads. Does anyone have an idea if I make a mistake in how I implemented threads / queues or what am I doing wrong?

import pcap
import struct
import dpkt
from Queue import Queue
from threading import Thread

def packet_handler():
    ts,pkt=q.get()
    eth=dpkt.ethernet.Ethernet(pkt)
    if eth.type != dpkt.ethernet.ETH_TYPE_IP:
        return
    a=struct.unpack('!BBHHHBBH4s4s', pkt[14:34])
    print a[2]+14,len(pkt)

def start():  
    pc.loop(0,lambda ts,pkt: q.put((ts,pkt)))

q=Queue()
pc=pcap.pcap(name="eth0")
start_sniffer=Thread(target=start)
start_sniffer.start()

while True:
    packet_handler()

Part of the output is as follows:

419 1454
419 419
54 60
389 60
389 389
405 60
405 405
405 60
405 405
54 60
54 60
493 491
491 492
491 493
491 491
502 502
54 60

Answer 1

I think I decided. I changed this line:

pc.loop(0,lambda ts,pkt: q.put((str(ts),str(pkt))))

This means that I converted ts and pkt to strings before adding them to the queue.

The sizes for multiple packets still vary for 4-6 bytes, but this also applies to the non-streaming version. I will probably ask about this in a new post.

Thank you all for your help !!!