stdout으로 인쇄하는 것이 왜 그렇게 느립니까?

stdout으로 인쇄하는 것이 왜 그렇게 느립니까? 속도를 올릴 수 있습니까?

---

나는 항상 print 문으로 터미널에 출력하는 데 걸리는 시간에 놀랐습니다. 최근의 고통스럽게 느린 로깅 후 나는 그것을 조사하기로 결정했고 거의 모든 시간이 터미널이 결과를 처리하기를 기다리고 있음을 알게되어 매우 놀랐 습니다.

어떻게 든 stdout에 글을 쓸 수 있습니까?

print_timer.py100k 줄을 stdout, 파일 및 stdout으로 리디렉션 할 때의 타이밍을 비교 하는 스크립트 ( 이 질문의 맨 아래에 ' ')를 작성했습니다 /dev/null. 타이밍 결과는 다음과 같습니다.

$ python print_timer.py
this is a test
this is a test
<snipped 99997 lines>
this is a test
-----
timing summary (100k lines each)
-----
print                         :11.950 s
write to file (+ fsync)       : 0.122 s
print with stdout = /dev/null : 0.050 s

와. 파이썬이 stdout을 / dev / null 또는 다른 것으로 재 지정했음을 인식하는 것과 같이 무대 뒤에서 무언가를하지 않도록하기 위해 스크립트 외부에서 리디렉션을 수행했습니다 ...

$ python print_timer.py > /dev/null
-----
timing summary (100k lines each)
-----
print                         : 0.053 s
write to file (+fsync)        : 0.108 s
print with stdout = /dev/null : 0.045 s

그래서 그것은 파이썬 트릭이 아니며 단지 터미널 일뿐입니다. 나는 항상 출력을 / dev / null 속도로 덤프하는 것을 알고 있었지만 그것이 그렇게 중요하다고 생각하지는 않았다!

tty가 얼마나 느린 지 놀랍습니다. 물리 디스크에 쓰는 것이 "스크린"(아마도 모든 RAM op)에 쓰는 것보다 더 빠르며 / dev / null을 사용하여 가비지에 덤프하는 것만 큼 효과적입니까?

이 링크 는 터미널이 I / O를 차단하는 방법에 대해 설명하여 "[입력] 구문 분석, 프레임 버퍼 업데이트, X 서버와 통신하여 창을 스크롤하는 등"을 수행 할 수 있습니다. 완전히 가져옵니다. 무엇이 오래 걸리나요?

나는 더 빠른 tty 구현이 부족한 방법이 없을 것으로 예상하지만 어쨌든 묻습니다.

---

업데이트 : 일부 의견을 읽은 후 화면 크기가 실제로 인쇄 시간에 얼마나 큰 영향을 미치는지 궁금해했으며 약간의 중요성이 있습니다. 위의 정말 느린 숫자는 내 Gnome 터미널이 1920x1200으로 날아간 것입니다. 아주 작게 줄이면 ...

-----
timing summary (100k lines each)
-----
print                         : 2.920 s
write to file (+fsync)        : 0.121 s
print with stdout = /dev/null : 0.048 s

확실히 나아지지만 (~ 4 배) 내 질문은 바뀌지 않습니다. 그것은 단지 추가 터미널 화면 렌더링이 표준 출력에 쓰는 응용 프로그램을 느리게해야하는 이유를 이해하지 않는 내 질문에. 프로그램에서 화면 렌더링이 계속되기를 기다려야하는 이유는 무엇입니까?

모든 터미널 / tty 앱이 동일하게 생성되지 않습니까? 아직 실험하지 않았습니다. 터미널이 들어오는 모든 데이터를 버퍼링하고, 보이지 않게 파싱 / 렌더링하고, 현재 화면 구성에서 볼 수있는 가장 최근의 청크 만 적절한 프레임 속도로 렌더링 할 수 있어야합니다. 따라서 ~ 0.1 초 안에 디스크에 쓰거나 fsync 할 수 있다면, 터미널은 그 순서에 따라 동일한 작업을 완료 할 수 있어야합니다 (화면이 업데이트 된 동안 화면이 약간 업데이트 될 수 있음).

나는 프로그래머 에게이 행동을 더 잘하기 위해 응용 프로그램 측에서 변경할 수있는 tty 설정이 있기를 바라고 있습니다. 이것이 엄격하게 터미널 응용 프로그램 문제인 경우 StackOverflow에 속하지 않을 수도 있습니다.

내가 무엇을 놓치고 있습니까?

---

타이밍을 생성하는 데 사용되는 파이썬 프로그램은 다음과 같습니다.

import time, sys, tty
import os

lineCount = 100000
line = "this is a test"
summary = ""

cmd = "print"
startTime_s = time.time()
for x in range(lineCount):
    print line
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)

#Add a newline to match line outputs above...
line += "\n"

cmd = "write to file (+fsync)"
fp = file("out.txt", "w")
startTime_s = time.time()
for x in range(lineCount):
    fp.write(line)
os.fsync(fp.fileno())
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)

cmd = "print with stdout = /dev/null"
sys.stdout = file(os.devnull, "w")
startTime_s = time.time()
for x in range(lineCount):
    fp.write(line)
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)

print >> sys.stderr, "-----"
print >> sys.stderr, "timing summary (100k lines each)"
print >> sys.stderr, "-----"
print >> sys.stderr, summary

---

How can it be that writing to physical disk is WAY faster than writing to the "screen" (presumably an all-RAM op), and is effectively as fast as simply dumping to the garbage with /dev/null?

Congratulations, you have just discovered the importance of I/O buffering. :-)

The disk appears to be faster, because it is highly buffered: all Python's write() calls are returning before anything is actually written to physical disk. (The OS does this later, combining many thousands of individual writes into a big, efficient chunks.)

The terminal, on the other hand, does little or no buffering: each individual print / write(line) waits for the full write (i.e. display to output device) to complete.

To make the comparison fair, you must make the file test use the same output buffering as the terminal, which you can do by modifying your example to:

fp = file("out.txt", "w", 1)   # line-buffered, like stdout
[...]
for x in range(lineCount):
    fp.write(line)
    os.fsync(fp.fileno())      # wait for the write to actually complete

I ran your file writing test on my machine, and with buffering, it also 0.05s here for 100,000 lines.

However, with the above modifications to write unbuffered, it takes 40 seconds to write only 1,000 lines to disk. I gave up waiting for 100,000 lines to write, but extrapolating from the previous, it would take over an hour.

That puts the terminal's 11 seconds into perspective, doesn't it?

So to answer your original question, writing to a terminal is actually blazingly fast, all things considered, and there's not a lot of room to make it much faster (but individual terminals do vary in how much work they do; see Russ's comment to this answer).

(You could add more write buffering, like with disk I/O, but then you wouldn't see what was written to your terminal until after the buffer gets flushed. It's a trade-off: interactivity versus bulk efficiency.)

---

Thanks for all the comments! I've ended up answering it myself with your help. It feels dirty answering your own question, though.

Question 1: Why is printing to stdout slow?

Answer: Printing to stdout is not inherently slow. It is the terminal you work with that is slow. And it has pretty much zero to do with I/O buffering on the application side (eg: python file buffering). See below.

Question 2: Can it be sped up?

Answer: Yes it can, but seemingly not from the program side (the side doing the 'printing' to stdout). To speed it up, use a faster different terminal emulator.

Explanation...

I tried a self-described 'lightweight' terminal program called wterm and got significantly better results. Below is the output of my test script (at the bottom of the question) when running in wterm at 1920x1200 in on the same system where the basic print option took 12s using gnome-terminal:

-----
timing summary (100k lines each)
-----
print                         : 0.261 s
write to file (+fsync)        : 0.110 s
print with stdout = /dev/null : 0.050 s

0.26s is MUCH better than 12s! I don't know whether wterm is more intelligent about how it renders to screen along the lines of how I was suggesting (render the 'visible' tail at a reasonable frame rate), or whether it just "does less" than gnome-terminal. For the purposes of my question I've got the answer, though. gnome-terminal is slow.

So - If you have a long running script that you feel is slow and it spews massive amounts of text to stdout... try a different terminal and see if it is any better!

Note that I pretty much randomly pulled wterm from the ubuntu/debian repositories. This link might be the same terminal, but I'm not sure. I did not test any other terminal emulators.

---

Update: Because I had to scratch the itch, I tested a whole pile of other terminal emulators with the same script and full screen (1920x1200). My manually collected stats are here:

wterm           0.3s
aterm           0.3s
rxvt            0.3s
mrxvt           0.4s
konsole         0.6s
yakuake         0.7s
lxterminal        7s
xterm             9s
gnome-terminal   12s
xfce4-terminal   12s
vala-terminal    18s
xvt              48s

The recorded times are manually collected, but they were pretty consistent. I recorded the best(ish) value. YMMV, obviously.

As a bonus, it was an interesting tour of some of the various terminal emulators available out there! I'm amazed my first 'alternate' test turned out to be the best of the bunch.

---

Your redirection probably does nothing as programs can determine whether their output FD points to a tty.

It's likely that stdout is line buffered when pointing to a terminal (the same as C's stdout stream behaviour).

As an amusing experiment, try piping the output to cat.

---

I've tried my own amusing experiment, and here are the results.

$ python test.py 2>foo
...
$ cat foo
-----
timing summary (100k lines each)
-----
print                         : 6.040 s
write to file                 : 0.122 s
print with stdout = /dev/null : 0.121 s

$ python test.py 2>foo |cat
...
$ cat foo
-----
timing summary (100k lines each)
-----
print                         : 1.024 s
write to file                 : 0.131 s
print with stdout = /dev/null : 0.122 s

---

I can't talk about the technical details because I don't know them, but this doesn't surprise me: the terminal was not designed for printing lots of data like this. Indeed, you even provide a link to a load of GUI stuff that it has to do every time you want to print something! Notice that if you call the script with pythonw instead, it does not take 15 seconds; this is entirely a GUI issue. Redirect stdout to a file to avoid this:

import contextlib, io
@contextlib.contextmanager
def redirect_stdout(stream):
    import sys
    sys.stdout = stream
    yield
    sys.stdout = sys.__stdout__

output = io.StringIO
with redirect_stdout(output):
    ...

---

Printing to the terminal is going to be slow. Unfortunately short of writing a new terminal implementation I can't really see how you'd speed this up significantly.

---

In addition to the output probably defaulting to a line-buffered mode, output to a terminal is also causing your data to flow into a terminal and serial line with a maximum throughput, or a pseudo-terminal and a separate process that is handling a display event loop, rendering characters from some font, moving display bits to implement a scrolling display. The latter scenario is probably spread over multiple processes (e.g. telnet server/client, terminal app, X11 display server) so there are context switching and latency issues too.

참고URL : https://stackoverflow.com/questions/3857052/why-is-printing-to-stdout-so-slow-can-it-be-sped-up