Python: how to calculate multi-threaded download speed

I wrote a multi-threaded HTTP downloader, now it can download a file faster than a single-threaded down-loader, and the MD5 sum is correct. However, I found that the speed he showed is so fast that I do not believe that this is the true value. enter image description here The unit has not yet been printed, but I'm sure it is KB / s, please take a look at my measure code.

 # Setup the slaver
    def _download(self):
        # Start download partital content when queue not empty
        while not self.configer.down_queue.empty():
            data_range = self.configer.down_queue.get()
            headers = {
                'Range': 'bytes={}-{}'.format(*data_range)
            }
            response = requests.get(
                self.configer.url, stream = True,
                headers = headers
            )
            start_point = data_range[0]
            for bunch in response.iter_content(self.block_size):
                _time = time.time()
                with self.file_lock:
                    with open(
                        self.configer.path, 'r+b',
                        buffering = 1
                    ) as f:
                        f.seek(start_point)
                        f.write(bunch)
                        f.flush()
                start_point += self.block_size
                self.worker_com.put((
                    threading.current_thread().name,
                    int(self.block_size / (time.time() - _time))
                ))
            self.configer.down_queue.task_done()

    # speed monitor
    def speed_monitor(self):
        while len(self.thread_list)>0:
            try:
                info = self.worker_com.get_nowait()
                self.speed[info[0]] = info[1]
            except queue.Empty:
                time.sleep(0.1)
                continue
            sys.stdout.write('\b'*64 + '{:10}'.format(self.total_speed)
                + '  thread num ' + '{:2}'.format(self.worker_count))
            sys.stdout.flush()

If you need more info, visit my github repository . I will appreciate if you can point out my mistake. thank.

+4
source share

All Articles