Sort both lists with an efficient sorting algorithm (or ensure that the lists are "pre-sorted" by whoever/whatever created them).

Then, if the first name in both lists is the same you've found a match, otherwise discard whichever name is "earlier"; and do that until one of the lists are empty.

Some crude pseudo-code:

    do {
        status = compare(shortList[i], longList[j]);
        if(status == EQUAL) {
            // Found match!
            i++;
            j++;
        } else if(status == EARLIER) {
            // No match, discard first entry in short list
            i++;
        } else {
            // No match, discard first entry in long list
            j++;
        }
    } while( (i < shortListEntries) && (j < longListEntries) );

If you need just a general algorithm to reduce number iterations, you can limit the range of the inner loop

for i, A in enumerate(items):
   for B in items[:i]:
      sim[A][B] = calc_sim(A, B)

But if you are looking for Python-specific optimization, it would be much better to use numpy vectorization. For example, if calc_sim(a, b) computes squared difference between a and b, then it can be vectorized the following way:

import numpy as np

list = [1, 2, 3]
array = np.array(list)
sim = np.square(array[:,np.newaxis] - array)

[[0 1 4]
 [1 0 1]
 [4 1 0]]

You can do it with one loop, by continuously maintaining the 3-way result list. If you can override equals() to depend on ids, that's fine. If not, check the code under update.

Here's the code, see the explanation below:

List<T> origList = ...;
List<T> newList = ...;

List<T> addedList = new ArrayList<T>();
List<T> deletedList = new ArrayList<T>();
List<T> changedList = new ArrayList<T>();

deletedList.addAll(origList);

for(T t : newList) {
    int origIndex = deletedList.indexOf(t);
    if (origIndex < 0) {
        addedList.add(t);
    } else {
        T origT = deletedList.remove(origIndex);
        if(t.compareTo(origT) != 0) {
            changedList.add(t);
        }
    }
}

Note that I presumed that equals() will check the id, and compareTo() will check all other fields.

Explanation:

You remove all elements from deletedList that were also present in newList, so the result is the deleted items.

You add all new elements to the addedList that were not present in the original list.

If both are present and the objects differ, they'll go to the changedList.

If both are present and the objects are the same then we don't add it anywhere.

Notes:

If new objects have an ID of 0, they will not be present in origList (since we suppose they have been already created).

When I last time implemented this, I created a separate method to compare the objects field-by-field, so I could separate the comparison logic from standard Java methods (actually it was also declared on an interface)

I created this with Lists, but actually you can use it with any type of Collection. Using with List will preserve the original order.

Update:

Try overriding your equals this way (I skipped the typecheck and the casting):

public boolean equals(T other) {
    if (this.id == 0) {
        return this == other;
    }
    return this.id == other.id;
}

The not-yet-created instances are equal to themselves only. The already-created ones are checked by id.

Using hash table you can solve this in linear time.

You could store one list in a hash table using python dictionary where the key would be the element (in this case an integer) and the value would be the number of occurrences of the element. Running time: O(n)

Then iterate through the other list and do a hash table lookup for each element. Keep a variable for counting the common values. Running time: O(n).

To avoid counting duplicates, as you iterate check if the previous element is the same, in which case move to the next element. You will need an extra variable to keep track of the previous element.

If you're using C++, there is support in the standard library for this (std::set_difference)

http://www.cplusplus.com/reference/algorithm/set_difference/

That documentation even includes equivalent "pseudocode" (really, just more C++) that you can use to port the idea to other languages. The algorithm is close to the "merging" part of mergesort. Note that std::set_difference operates upon sorted ranges, not std::sets (this is a good thing - means sorted std::vector's are adequate).