Redirect print command in python script via tqdm.write ()

Question

I use tqdm in Python to display console-progressbars in our scripts. However, I have to call functions that print messages to the console as well and which I cannot change. In general, writing to the console while displaying progress indicators in the console causes a mess on the display as follows:

 from time import sleep from tqdm import tqdm def blabla(): print "Foo blabla" for k in tqdm(range(3)): blabla() sleep(.5) 

This produces a conclusion:

 0%| | 0/3 [00:00<?, ?it/s]Foo blabla 33%|###########6 | 1/3 [00:00<00:01, 2.00it/s]Foo blabla 67%|#######################3 | 2/3 [00:01<00:00, 2.00it/s]Foo blabla 100%|###################################| 3/3 [00:01<00:00, 2.00it/s] 

According to the tqdm documentation tqdm the tqdm.write() method provides a means for writing messages to the console without violating the displayed control bars. Thus, the correct output is provided by this fragment:

 from time import sleep from tqdm import tqdm def blabla(): tqdm.write("Foo blabla") for k in tqdm(range(3)): blabla() sleep(.5) 

And it looks like this:

 Foo blabla Foo blabla Foo blabla 100%|###################################| 3/3 [00:01<00:00, 1.99it/s] 

On the other hand, there is a solution that allows you to disable these functions , pretty elegantly redirecting sys.stdout to void. This works great for jamming features.

Since I want to display messages from these functions without violating the progress indicators, I tried to combine both solutions into one by redirecting sys.stdout to tqdm.write() and, in turn, allowing tqdm.write() to be written to old sys.stdout The result is a fragment:

 from time import sleep import contextlib import sys from tqdm import tqdm class DummyFile(object): file = None def __init__(self, file): self.file = file def write(self, x): tqdm.write(x, file=self.file) @contextlib.contextmanager def nostdout(): save_stdout = sys.stdout sys.stdout = DummyFile(save_stdout) yield sys.stdout = save_stdout def blabla(): print "Foo blabla" for k in tqdm(range(3)): with nostdout(): blabla() sleep(.5) 

However, this actually creates an even more confusing conclusion, as before:

 0%| | 0/3 [00:00<?, ?it/s]Foo blabla 33%|###########6 | 1/3 [00:00<00:01, 2.00it/s]Foo blabla 67%|#######################3 | 2/3 [00:01<00:00, 2.00it/s]Foo blabla 100%|###################################| 3/3 [00:01<00:00, 2.00it/s] 

FYI: calling tqdm.write(..., end="") inside DummyFile.write() produces the same result as the first output, which is still confused.

I don’t understand why this will not work, since tqdm.write() must control the clearing of the progress bar before recording the message, and then rewrite the progress bar.

What am I missing?

Answer 1

Forwarding sys.stdout always complicated, and it becomes a nightmare when two applications are running at the same time.

Here's the trick that tqdm prints to sys.stderr by default, not sys.stdout . Typically, tqdm has an anti-mix strategy for these two special channels, but since you redirect sys.stdout , tqdm gets confused because the file descriptor changes.

So you just need to explicitly point file=sys.stdout to tqdm and it will work:

 from time import sleep import contextlib import sys from tqdm import tqdm class DummyFile(object): file = None def __init__(self, file): self.file = file def write(self, x): # Avoid print() second call (useless \n) if len(x.rstrip()) > 0: tqdm.write(x, file=self.file) @contextlib.contextmanager def nostdout(): save_stdout = sys.stdout sys.stdout = DummyFile(sys.stdout) yield sys.stdout = save_stdout def blabla(): print("Foo blabla") # tqdm call to sys.stdout must be done BEFORE stdout redirection # and you need to specify sys.stdout, not sys.stderr (default) for _ in tqdm(range(3), file=sys.stdout): with nostdout(): blabla() sleep(.5) print('Done!') 

I added a few more tricks to make the output more enjoyable (for example, without using \n when using print() without end='' ).

/ EDIT: in fact, you can do a stdout redirection after running tqdm , you just need to set dynamic_ncols=True in tqdm .

Answer 2

This may not be a good way, but I am changing the built-in print function. A.

 import inspect import tqdm # store builtin print old_print = print def new_print(*args, **kwargs): # if tqdm.tqdm.write raises error, use builtin print try: tqdm.tqdm.write(*args, **kwargs) except: old_print(*args, ** kwargs) # globaly replace print with new_print inspect.builtins.print = new_print 

Answer 3

By mixing user493630 and gaborous responses, I created this context manager, which should not use the file=sys.stdout tqdm .

 import inspect import contextlib import tqdm @contextlib.contextmanager def redirect_to_tqdm(): # Store builtin print old_print = print def new_print(*args, **kwargs): # If tqdm.tqdm.write raises error, use builtin print try: tqdm.tqdm.write(*args, **kwargs) except: old_print(*args, ** kwargs) try: # Globaly replace print with new_print inspect.builtins.print = new_print yield finally: inspect.builtins.print = old_print 

To use it, simply:

 for i in tqdm.tqdm(range(100)): with redirect_to_tqdm(): time.sleep(.1) print(i) 

To simplify even more, you can put the code in a new function:

 def tqdm_redirect(*args, **kwargs): with redirect_to_tqdm(): for x in tqdm.tqdm(*args, **kwargs): yield x for i in tqdm_redirect(range(20)): time.sleep(.1) print(i) 

Answer 4

The OP solution is almost correct. Here is a test in the tqdm library that messed up your output ( https://github.com/tqdm/tqdm/blob/master/tqdm/_tqdm.py#L546-L549 ):

 if hasattr(inst, "start_t") and (inst.fp == fp or all( f in (sys.stdout, sys.stderr) for f in (fp, inst. inst.clear(nolock=True) inst_cleared.append(inst) 

tqdm.write checks the file you provided to see if there is a risk of a collision between the print text and potential tqdm bars. In your case, stdout and stderr are mixed in the terminal, so there is a collision. To counter this, when the test passes, tqdm clears the columns, prints the text, and displays the columns back after.

Here the fp == sys.stdout test fails because sys.stdout became a DummyFile and fp is a real sys.stdout , so the cleanup behavior is not enabled. A simple equality operator in DummyFile fixes everything.

 class DummyFile(object): def __init__(self, file): self.file = file def write(self, x): tqdm.write(x, end="", file=self.file) def __eq__(self, other): return other is self.file 

In addition, since print passes the new sys.stdout to sys.stdout (or does not depend on the user's choice), you do not want tqdm to add another, so it is better to set the end='' option than doing strip on the content.

Benefits of this solution

With a loud answer, tqdm(..., file=sys.stdout) pollutes your output stream with pieces of the bar. By keeping file=sys.stdout (the default), you are sharing your streams.
With Conchylicultor and user493630 answers, you only print the patch. However, other systems, such as logging, are passed directly to sys.stdout, so they do not go through tqdm.write .