Watching something be written to a file live with tail
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I have a python program which is, slowly, generating some output.
I want to capture that in a file, but I also thought I could watch it live with tail.
So in one terminal I'm doing :
python myprog.py > output.txt
and in another terminal :
tail -f output.txt
But it seems like the tail isn't showing me anything while the python program is running.
If I hit ctrl-c to kill the python script, suddenly the tail of output.txt
starts filling up. But not while python is running.
What am I doing wrong?
linux command-line redirection stdout
add a comment |
I have a python program which is, slowly, generating some output.
I want to capture that in a file, but I also thought I could watch it live with tail.
So in one terminal I'm doing :
python myprog.py > output.txt
and in another terminal :
tail -f output.txt
But it seems like the tail isn't showing me anything while the python program is running.
If I hit ctrl-c to kill the python script, suddenly the tail of output.txt
starts filling up. But not while python is running.
What am I doing wrong?
linux command-line redirection stdout
10
How aboutpython myprog.py | tee output.txt
instead?
– n8te
Apr 3 at 23:35
7
@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.
– JPhi1618
Apr 4 at 2:42
1
stdbuf
can be used to alter the buffering status of file descriptors.
– studog
2 days ago
1
Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.
– Peter Cordes
2 days ago
2
Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…
– Mark Wagner
2 days ago
add a comment |
I have a python program which is, slowly, generating some output.
I want to capture that in a file, but I also thought I could watch it live with tail.
So in one terminal I'm doing :
python myprog.py > output.txt
and in another terminal :
tail -f output.txt
But it seems like the tail isn't showing me anything while the python program is running.
If I hit ctrl-c to kill the python script, suddenly the tail of output.txt
starts filling up. But not while python is running.
What am I doing wrong?
linux command-line redirection stdout
I have a python program which is, slowly, generating some output.
I want to capture that in a file, but I also thought I could watch it live with tail.
So in one terminal I'm doing :
python myprog.py > output.txt
and in another terminal :
tail -f output.txt
But it seems like the tail isn't showing me anything while the python program is running.
If I hit ctrl-c to kill the python script, suddenly the tail of output.txt
starts filling up. But not while python is running.
What am I doing wrong?
linux command-line redirection stdout
linux command-line redirection stdout
edited 2 days ago
Peter Cordes
2,5451622
2,5451622
asked Apr 3 at 23:31
interstarinterstar
433513
433513
10
How aboutpython myprog.py | tee output.txt
instead?
– n8te
Apr 3 at 23:35
7
@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.
– JPhi1618
Apr 4 at 2:42
1
stdbuf
can be used to alter the buffering status of file descriptors.
– studog
2 days ago
1
Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.
– Peter Cordes
2 days ago
2
Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…
– Mark Wagner
2 days ago
add a comment |
10
How aboutpython myprog.py | tee output.txt
instead?
– n8te
Apr 3 at 23:35
7
@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.
– JPhi1618
Apr 4 at 2:42
1
stdbuf
can be used to alter the buffering status of file descriptors.
– studog
2 days ago
1
Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.
– Peter Cordes
2 days ago
2
Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…
– Mark Wagner
2 days ago
10
10
How about
python myprog.py | tee output.txt
instead?– n8te
Apr 3 at 23:35
How about
python myprog.py | tee output.txt
instead?– n8te
Apr 3 at 23:35
7
7
@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.
– JPhi1618
Apr 4 at 2:42
@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.
– JPhi1618
Apr 4 at 2:42
1
1
stdbuf
can be used to alter the buffering status of file descriptors.– studog
2 days ago
stdbuf
can be used to alter the buffering status of file descriptors.– studog
2 days ago
1
1
Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.
– Peter Cordes
2 days ago
Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.
– Peter Cordes
2 days ago
2
2
Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…
– Mark Wagner
2 days ago
Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…
– Mark Wagner
2 days ago
add a comment |
4 Answers
4
active
oldest
votes
You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.
import sys
...
print('Some message')
sys.stdout.flush()
New contributor
7
If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).
– mckenzm
2 days ago
10
You can also useprint
'sflush
parameter to do just as well. For example,print('some message', flush=True)
.
– Dan
2 days ago
12
It has nothing to do with the pipe's buffer, but with thestdout
mechanism which doesn't flush after newline if it doesn't write to a tty.
– glglgl
2 days ago
1
Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.
– Roel Schroeven
yesterday
1
@glglgl Please explain. I don't actually know what you mean by that; how is "thestdout
mechanism" different to that of any other file?
– wizzwizz4
yesterday
|
show 1 more comment
Run python with the unbuffered flag:
python -u myprog.py > output.txt
Output will then print in real time.
New contributor
4
This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.
– Roel Schroeven
yesterday
Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."
– ivanivan
yesterday
add a comment |
Instead of trying to tail a live file, use tee
instead. It was made to do exactly what you're trying to do.
From man tee:
tee(1) - Linux man page
Name tee - read from standard input and write to standard output and files
Synopsis
tee [OPTION]... [FILE]...
Description
Copy standard input to each FILE, and also to standard output.
-a, --append
append to the given FILEs, do not overwrite
-i, --ignore-interrupts
ignore interrupt signals
--help
display this help and exit
--version
output version information and exit
If a FILE is -, copy again to standard output.
So in your case you'd run:
python myprog.py | tee output.txt
EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush()
is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.
tee
can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.
1
tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.
– Baldrickk
2 days ago
11
That requires a permanent console session, this is why it’s often much easier to usetail -F
or even better the follow function ofless
. But in all cases theflush
should be used.
– eckes
2 days ago
8
This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.
– Barmar
2 days ago
add a comment |
Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).
This is a redirect to a regular file.
C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write()
system call) when it's full.
You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.
On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.
Options include running python -u
(Or I guess putting #!/usr/bin/python -u
at the top of your script), or using the PYTHONUNBUFFERED
environment variable for that program. Or explicit flushing after some/all print
functions, like @Davey's answer suggests.
Some other programs have similar options, e.g. GNU grep has --line-buffered
, and GNU sed
has -u
/ --unbuffered
, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'
.
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1421123%2fwatching-something-be-written-to-a-file-live-with-tail%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
4 Answers
4
active
oldest
votes
4 Answers
4
active
oldest
votes
active
oldest
votes
active
oldest
votes
You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.
import sys
...
print('Some message')
sys.stdout.flush()
New contributor
7
If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).
– mckenzm
2 days ago
10
You can also useprint
'sflush
parameter to do just as well. For example,print('some message', flush=True)
.
– Dan
2 days ago
12
It has nothing to do with the pipe's buffer, but with thestdout
mechanism which doesn't flush after newline if it doesn't write to a tty.
– glglgl
2 days ago
1
Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.
– Roel Schroeven
yesterday
1
@glglgl Please explain. I don't actually know what you mean by that; how is "thestdout
mechanism" different to that of any other file?
– wizzwizz4
yesterday
|
show 1 more comment
You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.
import sys
...
print('Some message')
sys.stdout.flush()
New contributor
7
If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).
– mckenzm
2 days ago
10
You can also useprint
'sflush
parameter to do just as well. For example,print('some message', flush=True)
.
– Dan
2 days ago
12
It has nothing to do with the pipe's buffer, but with thestdout
mechanism which doesn't flush after newline if it doesn't write to a tty.
– glglgl
2 days ago
1
Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.
– Roel Schroeven
yesterday
1
@glglgl Please explain. I don't actually know what you mean by that; how is "thestdout
mechanism" different to that of any other file?
– wizzwizz4
yesterday
|
show 1 more comment
You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.
import sys
...
print('Some message')
sys.stdout.flush()
New contributor
You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.
import sys
...
print('Some message')
sys.stdout.flush()
New contributor
edited 2 days ago
user2313067
2,1001911
2,1001911
New contributor
answered Apr 4 at 0:05
DaveyDavey
46225
46225
New contributor
New contributor
7
If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).
– mckenzm
2 days ago
10
You can also useprint
'sflush
parameter to do just as well. For example,print('some message', flush=True)
.
– Dan
2 days ago
12
It has nothing to do with the pipe's buffer, but with thestdout
mechanism which doesn't flush after newline if it doesn't write to a tty.
– glglgl
2 days ago
1
Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.
– Roel Schroeven
yesterday
1
@glglgl Please explain. I don't actually know what you mean by that; how is "thestdout
mechanism" different to that of any other file?
– wizzwizz4
yesterday
|
show 1 more comment
7
If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).
– mckenzm
2 days ago
10
You can also useprint
'sflush
parameter to do just as well. For example,print('some message', flush=True)
.
– Dan
2 days ago
12
It has nothing to do with the pipe's buffer, but with thestdout
mechanism which doesn't flush after newline if it doesn't write to a tty.
– glglgl
2 days ago
1
Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.
– Roel Schroeven
yesterday
1
@glglgl Please explain. I don't actually know what you mean by that; how is "thestdout
mechanism" different to that of any other file?
– wizzwizz4
yesterday
7
7
If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).
– mckenzm
2 days ago
If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).
– mckenzm
2 days ago
10
10
You can also use
print
's flush
parameter to do just as well. For example, print('some message', flush=True)
.– Dan
2 days ago
You can also use
print
's flush
parameter to do just as well. For example, print('some message', flush=True)
.– Dan
2 days ago
12
12
It has nothing to do with the pipe's buffer, but with the
stdout
mechanism which doesn't flush after newline if it doesn't write to a tty.– glglgl
2 days ago
It has nothing to do with the pipe's buffer, but with the
stdout
mechanism which doesn't flush after newline if it doesn't write to a tty.– glglgl
2 days ago
1
1
Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.
– Roel Schroeven
yesterday
Instead of add flush() calls to the program, you can also use Python's -u command line switch; see BHC's answer.
– Roel Schroeven
yesterday
1
1
@glglgl Please explain. I don't actually know what you mean by that; how is "the
stdout
mechanism" different to that of any other file?– wizzwizz4
yesterday
@glglgl Please explain. I don't actually know what you mean by that; how is "the
stdout
mechanism" different to that of any other file?– wizzwizz4
yesterday
|
show 1 more comment
Run python with the unbuffered flag:
python -u myprog.py > output.txt
Output will then print in real time.
New contributor
4
This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.
– Roel Schroeven
yesterday
Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."
– ivanivan
yesterday
add a comment |
Run python with the unbuffered flag:
python -u myprog.py > output.txt
Output will then print in real time.
New contributor
4
This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.
– Roel Schroeven
yesterday
Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."
– ivanivan
yesterday
add a comment |
Run python with the unbuffered flag:
python -u myprog.py > output.txt
Output will then print in real time.
New contributor
Run python with the unbuffered flag:
python -u myprog.py > output.txt
Output will then print in real time.
New contributor
New contributor
answered 2 days ago
BHCBHC
39913
39913
New contributor
New contributor
4
This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.
– Roel Schroeven
yesterday
Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."
– ivanivan
yesterday
add a comment |
4
This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.
– Roel Schroeven
yesterday
Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."
– ivanivan
yesterday
4
4
This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.
– Roel Schroeven
yesterday
This is the correct answer. Python by default writes unbuffered (or actually line-buffered for text I/O) when writing to the console, but buffered when stdout is redirected to a file. -u forces Python to be unbuffered (or line-buffered for text) for writes.
– Roel Schroeven
yesterday
Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."
– ivanivan
yesterday
Know nothing of python but when I read what was happening I thought "gee, sounds like it is buffering output...."
– ivanivan
yesterday
add a comment |
Instead of trying to tail a live file, use tee
instead. It was made to do exactly what you're trying to do.
From man tee:
tee(1) - Linux man page
Name tee - read from standard input and write to standard output and files
Synopsis
tee [OPTION]... [FILE]...
Description
Copy standard input to each FILE, and also to standard output.
-a, --append
append to the given FILEs, do not overwrite
-i, --ignore-interrupts
ignore interrupt signals
--help
display this help and exit
--version
output version information and exit
If a FILE is -, copy again to standard output.
So in your case you'd run:
python myprog.py | tee output.txt
EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush()
is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.
tee
can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.
1
tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.
– Baldrickk
2 days ago
11
That requires a permanent console session, this is why it’s often much easier to usetail -F
or even better the follow function ofless
. But in all cases theflush
should be used.
– eckes
2 days ago
8
This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.
– Barmar
2 days ago
add a comment |
Instead of trying to tail a live file, use tee
instead. It was made to do exactly what you're trying to do.
From man tee:
tee(1) - Linux man page
Name tee - read from standard input and write to standard output and files
Synopsis
tee [OPTION]... [FILE]...
Description
Copy standard input to each FILE, and also to standard output.
-a, --append
append to the given FILEs, do not overwrite
-i, --ignore-interrupts
ignore interrupt signals
--help
display this help and exit
--version
output version information and exit
If a FILE is -, copy again to standard output.
So in your case you'd run:
python myprog.py | tee output.txt
EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush()
is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.
tee
can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.
1
tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.
– Baldrickk
2 days ago
11
That requires a permanent console session, this is why it’s often much easier to usetail -F
or even better the follow function ofless
. But in all cases theflush
should be used.
– eckes
2 days ago
8
This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.
– Barmar
2 days ago
add a comment |
Instead of trying to tail a live file, use tee
instead. It was made to do exactly what you're trying to do.
From man tee:
tee(1) - Linux man page
Name tee - read from standard input and write to standard output and files
Synopsis
tee [OPTION]... [FILE]...
Description
Copy standard input to each FILE, and also to standard output.
-a, --append
append to the given FILEs, do not overwrite
-i, --ignore-interrupts
ignore interrupt signals
--help
display this help and exit
--version
output version information and exit
If a FILE is -, copy again to standard output.
So in your case you'd run:
python myprog.py | tee output.txt
EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush()
is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.
tee
can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.
Instead of trying to tail a live file, use tee
instead. It was made to do exactly what you're trying to do.
From man tee:
tee(1) - Linux man page
Name tee - read from standard input and write to standard output and files
Synopsis
tee [OPTION]... [FILE]...
Description
Copy standard input to each FILE, and also to standard output.
-a, --append
append to the given FILEs, do not overwrite
-i, --ignore-interrupts
ignore interrupt signals
--help
display this help and exit
--version
output version information and exit
If a FILE is -, copy again to standard output.
So in your case you'd run:
python myprog.py | tee output.txt
EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush()
is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.
tee
can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.
edited 2 days ago
answered Apr 4 at 0:11
n8ten8te
5,38072235
5,38072235
1
tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.
– Baldrickk
2 days ago
11
That requires a permanent console session, this is why it’s often much easier to usetail -F
or even better the follow function ofless
. But in all cases theflush
should be used.
– eckes
2 days ago
8
This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.
– Barmar
2 days ago
add a comment |
1
tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.
– Baldrickk
2 days ago
11
That requires a permanent console session, this is why it’s often much easier to usetail -F
or even better the follow function ofless
. But in all cases theflush
should be used.
– eckes
2 days ago
8
This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.
– Barmar
2 days ago
1
1
tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.
– Baldrickk
2 days ago
tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.
– Baldrickk
2 days ago
11
11
That requires a permanent console session, this is why it’s often much easier to use
tail -F
or even better the follow function of less
. But in all cases the flush
should be used.– eckes
2 days ago
That requires a permanent console session, this is why it’s often much easier to use
tail -F
or even better the follow function of less
. But in all cases the flush
should be used.– eckes
2 days ago
8
8
This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.
– Barmar
2 days ago
This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.
– Barmar
2 days ago
add a comment |
Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).
This is a redirect to a regular file.
C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write()
system call) when it's full.
You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.
On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.
Options include running python -u
(Or I guess putting #!/usr/bin/python -u
at the top of your script), or using the PYTHONUNBUFFERED
environment variable for that program. Or explicit flushing after some/all print
functions, like @Davey's answer suggests.
Some other programs have similar options, e.g. GNU grep has --line-buffered
, and GNU sed
has -u
/ --unbuffered
, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'
.
add a comment |
Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).
This is a redirect to a regular file.
C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write()
system call) when it's full.
You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.
On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.
Options include running python -u
(Or I guess putting #!/usr/bin/python -u
at the top of your script), or using the PYTHONUNBUFFERED
environment variable for that program. Or explicit flushing after some/all print
functions, like @Davey's answer suggests.
Some other programs have similar options, e.g. GNU grep has --line-buffered
, and GNU sed
has -u
/ --unbuffered
, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'
.
add a comment |
Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).
This is a redirect to a regular file.
C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write()
system call) when it's full.
You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.
On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.
Options include running python -u
(Or I guess putting #!/usr/bin/python -u
at the top of your script), or using the PYTHONUNBUFFERED
environment variable for that program. Or explicit flushing after some/all print
functions, like @Davey's answer suggests.
Some other programs have similar options, e.g. GNU grep has --line-buffered
, and GNU sed
has -u
/ --unbuffered
, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'
.
Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).
This is a redirect to a regular file.
C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write()
system call) when it's full.
You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.
On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.
Options include running python -u
(Or I guess putting #!/usr/bin/python -u
at the top of your script), or using the PYTHONUNBUFFERED
environment variable for that program. Or explicit flushing after some/all print
functions, like @Davey's answer suggests.
Some other programs have similar options, e.g. GNU grep has --line-buffered
, and GNU sed
has -u
/ --unbuffered
, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'
.
answered 2 days ago
Peter CordesPeter Cordes
2,5451622
2,5451622
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1421123%2fwatching-something-be-written-to-a-file-live-with-tail%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
10
How about
python myprog.py | tee output.txt
instead?– n8te
Apr 3 at 23:35
7
@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.
– JPhi1618
Apr 4 at 2:42
1
stdbuf
can be used to alter the buffering status of file descriptors.– studog
2 days ago
1
Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.
– Peter Cordes
2 days ago
2
Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…
– Mark Wagner
2 days ago