Both .lower() and .casefold() act on the full range of Unicode codepoints

There's some confusion in the existing answers, even the accepted one (EDIT: I was referring to this currently outdated version; the current one is fine). The distinction between .lower() and .casefold() has nothing to do with ASCII vs. Unicode, both act on the whole Unicode range of codepoints, just in slightly different ways. But both perform relatively complex mappings which they need to look up in the Unicode database, for instance:

>>> "Ť".lower()
'ť'

Both can involve single-to-multiple codepoint mappings, like we saw with "ß".casefold(). Look what happens to ß when you apply .lower()'s counterpart .upper():

>>> "ß".upper()
'SS'

And the one example I found where .lower() also does this:

>>> list("İ".lower())
['i', '̇']

So the performance claims, like "lower() will require less memory or less time because there are no lookups, and it's only dealing with 26 characters it has to transform", are simply not true.

The vast majority of the time, both operations yield the same thing, but there are a few cases (297 as of Unicode 13.0.0) where they don't. You can identify them like this:

import sys
import unicodedata as ud

print("Unicode version:", ud.unidata_version, "\n")
total = 0
for codepoint in map(chr, range(sys.maxunicode)):
    lower, casefold = codepoint.lower(), codepoint.casefold()
    if lower != casefold:
        total += 1
        for conversion, converted in zip(
            ("orig", "lower", "casefold"),
            (codepoint, lower, casefold)
        ):
            print(conversion, [ud.name(cp) for cp in converted], converted)
        print()
print("Total differences:", total)

When to use which

The Unicode standard covers lowercasing as part of Default Case Conversion in Section 3.13, and Default Case Folding is described right below that. The first paragraph says:

Case folding is related to case conversion. However, the main purpose of case folding is to contribute to caseless matching of strings, whereas the main purpose of case conversion is to put strings into a particular cased form.

My rule of thumb based on this:

• Want to display a lowercased version of a string to users? Use .lower().
• Want to do case-insensitive string comparison? Use .casefold().

(As a sidenote, I routinely break this rule of thumb and use .lower() across the board, just because it's shorter to type, the output is overwhelmingly the same, and what differences there are don't affect the languages I typically come across and work with. Don't be like me though ;) )

Just to hammer home that in terms of complexity, both operations are basically the same, they just use slightly different mappings -- this is Unicode's abstract definition of lowercasing:

R2 toLowercase(X): Map each character C in X to Lowercase_Mapping(C).

And this is its abstract definition of case folding:

R4 toCasefold(X): Map each character C in X to Case_Folding(C).

In Python's official documentation

The Python docs are quite clear that this is what the respective methods do, they even point the user to the aforementioned Section 3.13.

They describe .lower() as converting cased characters to lowercase, where cased characters are "those with general category property being one of “Lu” (Letter, uppercase), “Ll” (Letter, lowercase), or “Lt” (Letter, titlecase)". Same with .upper() and uppercase.

With .casefold(), the docs explicitly state that it's meant for "caseless matching", and that it's "similar to lowercasing but more aggressive because it is intended to remove all case distinctions in a string".

I'll flesh out the discussion of case folding and insensitive matching in Python and Unicode.

Unicode defines two sets of operations, the first is case-mapping. For case mapping, Unicode defines three cases: lowercase, uppercase, and titlecase. These are string transformations changing text from one case to another.

Likewise, Unicode defines case folding operations. Case folding is designed to remove case distinctions before comparing strings or matching strings. This is different from the comparison operations between two strings defined in collation.

If you need to match or compare strings case insensitively, use case folding.

If you want to transform text or to do case sensitive string comparison use case mapping operations.

The key source of data on case folding is the CaseFolding.txt file in the UCD.

Three types methodologies are defined for case folding:

1. Simple casefolding. This is used when you want to minimise the size of the data you need to work with. It can be found in embedded systems, and is used in some regex engines. It involves folding single characters to single characters. Simple casefolding uses the mappings with status C and S.
2. Full casefolding. This is what str.casefold uses. Individual characters could be mapped to a sequence of characters. Full casefolding uses the mappings with status C and F.
3. Turkic tailoring for Turkish, Azerbaijani, Uzbek, Tatar and Kazakh. This is an optional folding that by default isn't used, but is an available option to casefolding in Unicode.

Casefolding is not a string operation that is sensitive to locales or languages, except for the option to use Turkic exceptions. It is also important to note that case insensitivity using str.casefold differs from case insensitivity in the re module.

Casefolding is a building block to other matching algorithms, including canonical caseless matching, compatibility caseless matching, and identifier matching.

As has been noted in other answers , Python doesn't provide access to the Turkish tailorings when casefolding.

There are two approaches:

1. Build a custom function to class to handle casefolding while using str.casefold, or
2. Make use of PyICU, a wrapper around icu4c

I'll use PyICU using icu.Char and icu.CaseMap classes:

def toCasefold(text:str, full:bool = True, turkic:bool = False) -> str:
    # Enumerated consonants to use with icu.CaseMap:
    # icu.U_FOLD_CASE_DEFAULT : 0
    # icu.U_FOLD_CASE_EXCLUDE_SPECIAL_I : 1
    # 
    # Enumerated consonants in icu.Char:
    # icu.Char.FOLD_CASE_DEFAULT : 0
    # icu.Char.FOLD_CASE_EXCLUDE_SPECIAL_I : 1

    option:int = 1 if turkic else 0
    if not full:
        return "".join([icu.Char.foldCase(char, option) for char in text])
    return icu.CaseMap.fold(option, text)

city = 'DİYARBAKIR'

# Default casefold of string in Python,
print(city.casefold())
# di̇yarbakir

# Full case folding in PyICU, using a wrapper:
# Could also use icu.UnicodeString.foldCase
print(toCasefold(city))
# di̇yarbakir

# Full case folding in PyICU, using a wrapper, with Turkic rules in Casefolding.txt
# Could also use icu.UnicodeString.foldCase
print(toCasefold(city, turkic=True))
# diyarbakır

# Simple case folding in PyICU, using a wrapper
print(toCasefold(city, full=False))
# dİyarbakir

# Simple case folding in PyICU, using a wrapper, with Turkic rules in Casefolding.txt
# Could also use icu.UnicodeString.foldCase
print(toCasefold(city, full=False, turkic=True))
# diyarbakır

So DİYARBAKIR can be casefolded according to Unicode rules to di̇yarbakir, diyarbakır, or dİyarbakir, depending on the type of case folding and the options applied.