So make a temporary dict with the key being the id. This filters out the duplicates. The values() of the dict will be the list

In Python2.7

>>> L=[
... {'id':1,'name':'john', 'age':34},
... {'id':1,'name':'john', 'age':34},
... {'id':2,'name':'hanna', 'age':30},
... ]
>>> {v['id']:v for v in L}.values()
[{'age': 34, 'id': 1, 'name': 'john'}, {'age': 30, 'id': 2, 'name': 'hanna'}]

In Python3

>>> L=[
... {'id':1,'name':'john', 'age':34},
... {'id':1,'name':'john', 'age':34},
... {'id':2,'name':'hanna', 'age':30},
... ] 
>>> list({v['id']:v for v in L}.values())
[{'age': 34, 'id': 1, 'name': 'john'}, {'age': 30, 'id': 2, 'name': 'hanna'}]

In Python2.5/2.6

>>> L=[
... {'id':1,'name':'john', 'age':34},
... {'id':1,'name':'john', 'age':34},
... {'id':2,'name':'hanna', 'age':30},
... ] 
>>> dict((v['id'],v) for v in L).values()
[{'age': 34, 'id': 1, 'name': 'john'}, {'age': 30, 'id': 2, 'name': 'hanna'}]

data = [{"name": "John", "age": "20", "occupation": "Python pro"}, \
        {"name": "Jack", "age": "80", "occupation": "Olympic Athlete"}, \
        {"name": "John", "age": "20", "occupation": "Student"}]

new_data = []
names_ages = set([])
for d in data:
    name_age = (d["name"], d["age"])
    if name_age not in names_ages:
        new_data.append(d)
    names_ages.add(name_age)

print new_data
# [{'age': '20', 'name': 'John', 'occupation': 'Python pro'},
#  {'age': '80', 'name': 'Jack', 'occupation': 'Olympic Athlete'}]

If your value is hashable this will work:

>>> [dict(y) for y in set(tuple(x.items()) for x in d)]
[{'y': 4, 'x': 3}, {'y': 2, 'x': 1}]

EDIT:

I tried it with no duplicates and it seemed to work fine

>>> d = [{'x':1, 'y':2}, {'x':3, 'y':4}]
>>> [dict(y) for y in set(tuple(x.items()) for x in d)]
[{'y': 4, 'x': 3}, {'y': 2, 'x': 1}]

and

>>> d = [{'x':1,'y':2}]
>>> [dict(y) for y in set(tuple(x.items()) for x in d)]
[{'y': 2, 'x': 1}]

d = {'res': [1.1, 2.2, 1.2, 4.5, 1.5, 3.4], 'sp': [1, 1, 2, 3, 4, 4], 'obs': [1, 2, 3, 4, 5, 6]}

Given that input:

r = {}
for i, v in enumerate(d['sp']):
    r.setdefault(v, {'res':[],'obs':[]})
    r[v]['res'].append(d['res'][i])
    r[v]['obs'].append(d['obs'][i])

This results in:

>>> r
{1: {'res': [1.1, 2.2], 'obs': [1, 2]}, 2: {'res': [1.2], 'obs': [3]}, 3: {'res': [4.5], 'obs': [4]}, 4: {'res': [1.5, 3.4], 'obs': [5, 6]}}

To extend the first solution for any number of arguments is simple:

r = {}
keys = set(d) - set(['sp'])
for i, v in enumerate(d['sp']):
    if v not in r:
        r[v] = dict((k, []) for k in keys)

    for k in keys:
        r[v][k].append(d[k][i])

This results in the same result.

Using if v not in r: over setdefault saves creating a lot of dictionary and list objects but when the structure is simple that is not much of a cost.

I think your first approach is already pretty close to being optimal. Dictionary lookup is fast (just as fast as in a set) and the loop is easy to understand, even though a bit lengthy (by Python standards), but you should not sacrifice readability for brevity.

You can, however, shave off one line using setdefault, and you might want to use collections.OrderedDict() so that the elements in the resulting list are in their orginal order. Also, note that in Python 3, unique.values() is not a list but a view on the dict.

unique = collections.OrderedDict()
for elem in elems:
    unique.setdefault(elem["country_code"], elem)

If you really, really want to use reduce, you can use the empty dict as an initializer and then use d.setdefault(k,v) and d to set the value (if not present) and return the modified dict.

unique = reduce(lambda unique, elem: unique.setdefault(elem["country_code"], elem) and unique,
                elems, collections.OrderedDict())

I would just use the loop, though.

With all of the pairs appearing in the list, the implementation should be simple enough.

Sort the list. The "lower" labels will come to the front. In particular, the first entry will be a viable main element. Put every such link into the results list, keeping track of the second elements. Then pass over the links starting with "other" names. Repeat this until you've covered the entire original list. Here's a painfully detailed version:

start = [
    { 'name00': 'test1', 'name01': 'test2' },
    { 'name00': 'test1', 'name01': 'test3' },
    { 'name00': 'test2', 'name01': 'test1' },
    { 'name00': 'test2', 'name01': 'test3' },
    { 'name00': 'test3', 'name01': 'test1' },
    { 'name00': 'test3', 'name01': 'test2' },
    { 'name00': 'test4', 'name01': 'test5' },
    { 'name00': 'test5', 'name01': 'test4' }
]

result = []

start.sort(key=lambda link: link["name00"]+link["name01"]) 

for link in start:
    print(link)

link_key = None
pos = 0
while pos < len(start):

    other_name = []

    link = start[pos]
    if not link_key:
        link_key = link['name00']

    # Gather all of the links that start with the lowest name.
    # Keep track of the other names for later use.
    while start[pos]['name00'] == link_key:
        link = start[pos]
        result.append(link)
        other_name.append(link["name01"])
        pos += 1

    # Now is "later" ... ignore all links that start with other names.
    while pos < len(start) and              \
          start[pos]['name00'] in other_name:
        link = start[pos]
        pos += 1

    link_key = None


# Print the resulting pairs
for link in result:
    print(link["name00"], link["name01"])

Output:

test1 test2
test1 test3
test4 test5