Hi, just wondering if using input() or sys.stdin is the preferred method for reading from stdin in python. What are the differences and what is more readable/pythonesque?
# Using input()
while True:
try:
line = input()
...
except EOFError:
break
# Using sys.stdin
for line in sys.stdin:
line = line.strip()
...io - Python reading from stdin while doing other tasks - Software Engineering Stack Exchange
Python wait until data is in sys.stdin - Stack Overflow
python - How do I read from stdin? - Stack Overflow
Python sys.stdin.read(1) in a while(True) loop consistently executes 1 time getting input and multiple times not getting input - Stack Overflow
This is a perfect example to use multithreading.
Create a 'worker' thread and another thread, which waits for user input. When the 'input' thread receives input, it can add an item to a 'todo' list (with some sort of locking to exclude race conditions).
Then, the worker thread can fetch the topmost element and execute it. In your case it should also add another item for 'unbanning' the IP later, which means each task should have a 'time' tag of some sort.
A reasonable structure to implement this is a red-black tree or a binary heap with sorting by the 'time' value.
Then, your worker thread checks if the time has come to execute the top object in the 'todo' list, executes it and sleeps some reasonable amount of time (say 100 ms).
On the other hand, the input thread waits for input, and when presented with it, adds a task with the current time (which guarantees it would be almost immediately picked up by the worker thread).
Threads are fine, and there's really nothing wrong with using them here. For completeness sake, there's also select(), which allows you to do a blocking check for pending input on a file descriptor, with a timeout. If you use this function in a loop, interspersed with the timed activity, you'll get roughly the same concurrent behavior, but implemented on a single thread. This is the python binding for select() and its friends, btw.
The following should just work.
import sys
for line in sys.stdin:
# whatever
Rationale:
The code will iterate over lines in stdin as they come in. If the stream is still open, but there isn't a complete line then the loop will hang until either a newline character is encountered (and the whole line returned) or the stream is closed (and the whatever is left in the buffer is returned).
Once the stream has been closed, no more data can be written to or read from stdin. Period.
The reason that your code was overloading your cpu is that once the stdin has been closed any subsequent attempts to iterate over stdin will return immediately without doing anything. In essence your code was equivalent to the following.
for line in sys.stdin:
# do something
while 1:
pass # infinite loop, very CPU intensive
Maybe it would be useful if you posted how you were writing data to stdin.
EDIT:
Python will (for the purposes of for loops, iterators and readlines() consider a stream closed when it encounters an EOF character. You can ask python to read more data after this, but you cannot use any of the previous methods. The python man page recommends using
import sys
while True:
line = sys.stdin.readline()
# do something with line
When an EOF character is encountered readline will return an empty string. The next call to readline will function as normal if the stream is still open. You can test this out yourself by running the command in a terminal. Pressing ctrl+D will cause a terminal to write the EOF character to stdin. This will cause the first program in this post to terminate, but the last program will continue to read data until the stream is actually closed. The last program should not 100% your CPU as readline will wait until there is data to return rather than returning an empty string.
I only have the problem of a busy loop when I try readline from an actual file. But when reading from stdin, readline happily blocks.
This actually works flawlessly (i.e. no runnaway CPU) - when you call the script from the shell, like so:
tail -f input-file | yourscript.py
Obviously, that is not ideal - since you then have to write all relevant stdout to that file -
but it works without a lot of overhead!
Namely because of using readline() - I think:
while 1:
line = sys.stdin.readline()
It will actually stop and wait at that line until it gets more input.
Hope this helps someone!
Use the fileinput module:
import fileinput
for line in fileinput.input():
pass
fileinput will loop through all the lines in the input specified as file names given in command-line arguments, or the standard input if no arguments are provided.
Note: line will contain a trailing newline; to remove it use line.rstrip().
There's a few ways to do it.
sys.stdinis a file-like object on which you can call functionsreadorreadlinesif you want to read everything or you want to read everything and split it by newline automatically. (You need toimport sysfor this to work.)If you want to prompt the user for input, you can use
raw_inputin Python 2.X, and justinputin Python 3.If you actually just want to read command-line options, you can access them via the sys.argv list.
You will probably find this Wikibook article on I/O in Python to be a useful reference as well.
Problem is probably due to flushing of stdin since the \n lingers on.
as an alternative, use raw_input
while True:
c = raw_input('please enter a character: ')
print 'you entered', c
For the flushing part, see this
sys.stdin is line-buffered by default i.e., your sys.stdin.read(1) won't return until there is full line in stdin buffer.
It means if you enter a character and hit Enter then after you get the first character with sys.stdin.read(1), there is a newline in the buffer (one or two characters: os.linesep) that are read immediately on the next loop iterations.
You could avoid hitting Enter by reading exactly one character at a time (msvcrt.getch()).
The fileinput module in the standard library is just what you want:
import fileinput
for line in fileinput.input(): ...
import fileinput
for line in fileinput.input():
process(line)
This iterates over the lines of all files listed in sys.argv[1:], defaulting to sys.stdin if the list is empty.
It's simpler:
for line in sys.stdin:
chrCounter += len(line)
The file-like object sys.stdin is automatically iterated over line by line; if you call .readline() on it, you only read the first line (and iterate over that character-by-character); if you call read(), then you'll read the entire input into a single string and iterate over that character-by.character.
The answer from Tim Pietzcker is IMHO the correct one. There are 2 similar ways of doing this. Using:
for line in sys.stdin:
and
for line in sys.stdin.readlines():
The second option is closer to your original code. The difference between these two options is made clear by using e.g. the following modification of the for-loop body and using keyboard for input:
for line in sys.stdin.readlines():
line_len = len(line)
print('Last line was', line_len, 'chars long.')
chrCounter += len(line)
If you use the first option (for line in sys.stdin:), then the lines are processed right after you hit enter.
If you use the second option (for line in sys.stdin.readlines():), then the whole file is first read, split into lines and only then they are processed.
You can wrap stdin to strip the newlines; if you can strip all trailing whitespace (usually okay), then it's just:
for name in map(str.rstrip, sys.stdin):
...
You're on Py3, so that works as is; if you're on Py2, you'll need to add an import, from future_builtins import map, so you get a lazy, generator based map (that yields the lines as they're requested, rather than slurping stdin until it ends, then returning a list of all the lines).
If you need to limit to newlines, a generator expression can do it:
for name in (line.rstrip("\r\n") for line in sys.stdin):
...
or with an import to allow map to push work to C layer for (slightly) faster code (a matter of 30-some nanoseconds per line faster than the genexpr, but still 40 ns per line slower than the argumentless option at the top of this answer):
from operator import methodcaller
for name in map(methodcaller('rstrip', '\r\n'), sys.stdin):
...
Like the first solution, on Py2, make sure to get the map from future_builtins.
I wouldn't recommend you this, but you can create a generator to be used in a for loop to iterate through input line by line:
def getlines():
while True:
yield input()
for name in getlines():
print(name)
## Remember to break out of the loop at some point
Hey everyone, I just want to talk about reading in data from standard input and the 4 main ways it can be done.
I'm not going to talk about the input() or raw_input() functions today, instead ill be talking about how to read from standard input using the sys module.
To get access to the sys module we first need to import it
import sys
Ok now we have access to this module, there are 3 ways to read from standard input:
-
sys.stdin.read([size])
-
sys.stdin.readline()
-
sys.stdin.readlines()
Lets look at how all of these work first and the ways to use them.
First off we can read lines directly from the console, this will look something like this
lines = sys.stdin.read() print(lines) $ python3 stdin.py Line1 Line 2 **END** Line 1 Line 2
Our lines variable looks like this: "Line1\nLine2"
Here when we run our program, it waits until it we pass some data through the console window. We specify end of input using ctrl+z on windows and I believe ctrl+d on linux.
The sys.stdin.read() function also has an optional parameter for the size of the data we want to read. For example if we pass 10 then it reads 10 characters including any newline characters.
The read() function will read everything, or the size of data specified, and return it as one string. This is useful for small amounts of data but if we read large files this way, it can use up a lot of memory.
The second way is sys.stdin.readline() which is self explanatory and reads a single line from standard input with a newline character at the end.
line = sys.stdin.readline() print(line) $ python3 stdin.py hello hello
The next way is sys.stdin.readlines(). I find myself using this way most often. With this way, we read lines from the console and are returned a list containing all the lines we entered.
lines = sys.stdin.readlines() print(lines) $ python3 stdin.py line1 line2 line3 ['line1\n', 'line2\n', 'line3\n']
This is very useful if we wish to process a file line by line although, we do have a large list sitting in memory which we may not want with large files. I will show you how to read from files in a moment.
Reading from files:
To read from a file we can do this a couple of ways, we can open and read the file within our program.
with open('FILENAME', [rw]) as our_file:
for line in our_file:
print(line)The optional [rw] specifies whether we wish to open the file for reading, r or writing, w. This will work depending on the access permission on the file. You can check this on linux from the command line by navigating to your directory where the file is and typing:
$ ls -l
This will display the access permissions of the file in that directory.
An error will be thrown if you try to read or write without having permission to do so.
If the file name you entered doesn't exist, an empty file will be created for you.
The use of with open() here is very useful as it closes our file for us when we are finished.
Another way to read a file is passing it at the command line
$ python3 stdin.py < FILENAME.txt
Presuming FILENAME.txt looks like this:
Line 1 Line 2 Line 3
Running the following program, we get the following output:
import sys lines = sys.stdin.readlines() print(lines) $ python3 stdin.py < FILENAME.txt ['Line 1\n', 'Line 2\n', 'Line 3']
I dont want to talk to much about the different ways of reading and writing files as I only wanted to talk about the different methods we have available to use for reading so I wont discuss any further ways of reading.
If we wish to strip the newline characters from our lines we can use the strip() method, I'm going to use a list comprehension here as it is a good example of their usage:
lines = [line.strip() for line in sys.stdin.readlines()] print(lines) $ python3 stdin.py < FILENAME.txt ['Line 1', Line 2', 'Line 3']
Whats the list comprehension doing? It uses a for loop to loop through each line in standard input, takes each line and strips it then appends it to our list, lines.
Now our newline characters are gone.
We covered a fair bit of stuff here and got the chance to see some extra things in use such as list comprehensions. If you found anything here confusing, play around with it yourself, after all its one of the best ways to learn.