If you have 100 million numbers like in the question, then this is actually manageable in-memory (it takes about 0.5 GB).
As DSM pointed out, this can be done with the standard modules in an efficient way:
>>> import array
>>> a = array.array('I', xrange(10**8)) # a.itemsize indicates 4 bytes per element => about 0.5 GB
>>> import random
>>> random.shuffle(a)
It is also possible to use the third-party NumPy package, which is the standard Python tool for managing arrays in an efficient way:
>>> import numpy
>>> ids = numpy.arange(100000000, dtype='uint32') # 32 bits is enough for numbers up to about 4 billion
>>> numpy.random.shuffle(ids)
(this is only useful if your program already uses NumPy, as the standard module approach is about as efficient).
Both method take about the same amount of time on my machine (maybe 1 minute for the shuffling), but the 0.5 GB they use is not too big for current computers.
PS: There are too many elements for the shuffling to be really random because there are way too many permutations possible, compared to the period of the random generators used. In other words, there are fewer Python shuffles than the number of possible shuffles!
Answer from Eric O. Lebigot on Stack OverflowIf you have 100 million numbers like in the question, then this is actually manageable in-memory (it takes about 0.5 GB).
As DSM pointed out, this can be done with the standard modules in an efficient way:
>>> import array
>>> a = array.array('I', xrange(10**8)) # a.itemsize indicates 4 bytes per element => about 0.5 GB
>>> import random
>>> random.shuffle(a)
It is also possible to use the third-party NumPy package, which is the standard Python tool for managing arrays in an efficient way:
>>> import numpy
>>> ids = numpy.arange(100000000, dtype='uint32') # 32 bits is enough for numbers up to about 4 billion
>>> numpy.random.shuffle(ids)
(this is only useful if your program already uses NumPy, as the standard module approach is about as efficient).
Both method take about the same amount of time on my machine (maybe 1 minute for the shuffling), but the 0.5 GB they use is not too big for current computers.
PS: There are too many elements for the shuffling to be really random because there are way too many permutations possible, compared to the period of the random generators used. In other words, there are fewer Python shuffles than the number of possible shuffles!
Maybe something like (won't be consecutive, but will be unique):
from uuid import uuid4
def unique_nums(): # Not strictly unique, but *practically* unique
while True:
yield int(uuid4().hex, 16)
# alternative yield uuid4().int
unique_num = unique_nums()
next(unique_num)
next(unique_num) # etc...
Which method is best for generating N unique numbers from a small range
Does random.shuffle produce unique items if the input list already has duplicates
How to handle an OverflowError when using range(very_large_number)
Instead of picking random integers, shuffle a list and pick the first two items:
import random
choices = ['Candy', 'Steak', 'Vegetables']
random.shuffle(choices)
item1, item2 = choices[:2]
Because we shuffled a list of possible choices first, then picked the first two, you can guarantee that item1 and item2 are never equal to one another.
Using random.shuffle() leaves the option open to do something with the remaining choices; you only have 1 here, but in a larger set you can continue to pick items that have so far not been picked:
choices = list(range(100))
random.shuffle(choices)
while choices:
if input('Do you want another random number? (Y/N)' ).lower() == 'n':
break
print(choices.pop())
would give you 100 random numbers without repeating.
If all you need is a random sample of 2, use random.sample() instead:
import random
choices = ['Candy', 'Steak', 'Vegetables']
item1, item2 = random.sample(choices, 2)
You can use the random module in Python to do the heavy lifting for you, specifically random.sample():
>>> import random
>>> random.sample(['candy', 'steak', 'vegetable'], 2)
['vegetable', 'steak']
Every number from 1,2,5,6,9,10... is divisible by 4 with remainder 1 or 2.
>>> ','.join(str(i) for i in xrange(100) if i % 4 in (1,2))
'1,2,5,6,9,10,13,14,...'
>>> ','.join('{},{}'.format(i, i + 1) for i in range(1, 100, 4))
'1,2,5,6,9,10,13,14,17,18,21,22,25,26,29,30,33,34,37,38,41,42,45,46,49,50,53,54,57,58,61,62,65,66,69,70,73,74,77,78,81,82,85,86,89,90,93,94,97,98'
That was a quick and quite dirty solution.
Now, for a solution that is suitable for different kinds of progression problems:
def deltas():
while True:
yield 1
yield 3
def numbers(start, deltas, max):
i = start
while i <= max:
yield i
i += next(deltas)
print(','.join(str(i) for i in numbers(1, deltas(), 100)))
And here are similar ideas implemented using itertools:
from itertools import cycle, takewhile, accumulate, chain
def numbers(start, deltas, max):
deltas = cycle(deltas)
numbers = accumulate(chain([start], deltas))
return takewhile(lambda x: x <= max, numbers)
print(','.join(str(x) for x in numbers(1, [1, 3], 100)))