» pip install pycountry-convert
The DRC in Pycountry
python - Basic function to convert Country name to ISO code using Pycountry - Code Review Stack Exchange
python - I cannot get pycountry_convert to work in my code - Stack Overflow
python - How to use pycountry_convert to create a new column in a DataFrame that lists the continent that the country is in? - Stack Overflow
» pip install country-converter
Hey, I am trying to change country names to ones that are recognised by pycountry so that I can get their ISO-3 codes, does anyone know what it is for the DRC (Democratic Republic of the Congo)?
Wikipedia says: 'Congo, Democratic Republic of the' but it is not working for me
Edit: its 'Congo, The Democratic Republic of the' if anyone cares
I always get my code to be as clean as possible before starting work on performance. Here you have a bare except, which can hide errors. Change it to something better, for now I'll change it to Exception. You are also hiding a potential IndexError and AttributeError if the data ever changes shape as they are in the try.
def do_fuzzy_search(country):
try:
result = pycountry.countries.search_fuzzy(country)
except Exception:
return np.nan
else:
return result[0].alpha_3
Since there are roughly 200 sovereign states and you're working with 17003 rows you likely have a lot of the same values hitting a costly function. To resolve this issue you can use functools.lru_cache. Running in amortized \$O(n)\$ time and \$O(n)\$ space.
@functools.lru_cache(None)
def do_fuzzy_search(country):
...
Alternately you can sort the data by the provided countries name and then get each country once. Running in \$O(n\log n)\$ time and \$O(1)\$ space.
Apart from speeding up your code, there are some other improvements possible that I don't see mentioned yet.
The speedup
As other users have already picked up, you only need to transform around 200 countries instead of 17k rows. You can do this efficiently by storing the mapping in a dictionary:
iso_map = {country: do_fuzzy_search(country) for country in df["Country"].unique()}
df["country_code"] = df["Country"].map(iso_map)
This is in essence very similar to using the lru_cache proposed by Peilonrayz, but it's a bit more explicit.
Consistent naming
As a side note, I would advise you to use a consistent naming scheme in your DataFrame. This doesn't have to follow the python convention, but it should be easy to remember.
For example, turn all columns to lowercase by using something like:
df = df.rename(columns = {name: name.lower() for name in df.columns}
Using categoricals
Since the number of countries (and isocodes) is limited, you might want to consider using categorical columns. These have the advantage of requiring less memory and being faster on some operations.
It might not be worth the extra lines of code in your situation, but it's something useful to keep in the back of your mind.