Best / most pythonic way to get an ordered list of unique items

Question

I have one or more unordered sequences of (immutable, hashed) objects with possible duplicates, and I want to get a sorted sequence of all these objects without duplicates.

Now I use the kit to quickly collect all the elements that discard duplicates, convert them to a list, and then sort:

result = set() for s in sequences: result = result.union(s) result = list(result) result.sort() return result 

It works, but I would not call it "pretty." Is there a better way?

Answer 1

This should work:

 sorted(set(itertools.chain.from_iterable(sequences))) 

Answer 2

I really like your code. It is simple and straightforward.

We can shorten it a bit by concatenating list() :

 result = set() for s in sequences: result = result.union(s) return sorted(result) 

I really have no desire to try to boil it further, but you can do it with reduce() :

 result = reduce(lambda s, x: s.union(x), sequences, set()) return sorted(result) 

Personally, I find it harder to understand than the higher, but people immersed in functional programming may prefer it.

EDIT: @agf is much better at this reduce() than me. From the comments below:

 return sorted(reduce(set().union, sequences)) 

I had no idea this would work. If I understand how this works correctly, we provide a reduce() call that is really a function of the method on a single instance of a set() (call it x for discussion, but note that I'm not saying that Python will bind the name x to this object). Then reduce() will feed this function the first two iterators from sequences , returning an x instance, the method of which we are using. Then reduce() will repeatedly call the .union() method and ask it to accept the union of x and the next iterable of sequences . Since the .union() method is most likely smart enough to notice that he is being asked to take an alliance with his own instance and not worry about any work, you need to call x.union(x, some_iterable) just as quickly to call x.union(some_iterable) >. Finally, reduce() will return x , and we will have the desired set.

It is a little complicated for my personal taste. I had to think about this in order to understand this, while the itertools.chain() solution made sense to me right away.

EDIT: @agf made this less complicated:

 return sorted(reduce(set.union, sequences, set())) 

What this does is much easier to understand! If we again call the instance returned by set() name x (and the same as above, on the understanding that I am not saying that Python will associate the name x with this instance); and if we use the name n to denote each "next" value from sequences ; then reduce() will repeatedly call set.union(x, n) . And of course, this is exactly the same as x.union(n) . IMHO, if you want a reduce() solution, this is the best option.

-

If you want it to be fast, ask yourself: is there a way we can apply itertools to this? There is a pretty good way:

 from itertools import chain return sorted(set(chain(*sequences))) 

itertools.chain() , called with *sequences , serves to "smooth" the list of lists into one iterable. This is a little complicated, but only a little, and this is a common idiom.

EDIT: As @Jbernardo wrote in the most popular answer, and as @agf notes in the comments, itertools.chain() returns an object with the .from_iterable() method, and the documentation says that it lazily evaluates the iterable. The designation * forces you to create a list that can consume significant memory if iterability is a long sequence. In fact, you can have an infinite generator, and with itertools.chain().from_iterable() you can extract values ​​from it as much as you want to run your program, while the notation * just ends without memory.

As @Jbernardo wrote:

 sorted(set(itertools.chain.from_iterable(sequences))) 

This is the best answer, and I have already supported it.