The answer to your misunderstanding is a Unix concept, softlinks which we could say that in Windows are similar to shortcuts. Let's explain this.

When you spacy download en, spaCy tries to find the best small model that matches your spaCy distribution. The small model that I am talking about defaults to en_core_web_sm which can be found in different variations which correspond to the different spaCy versions (for example spacy, spacy-nightly have en_core_web_sm of different sizes).

When spaCy finds the best model for you, it downloads it and then links the name en to the package it downloaded, e.g. en_core_web_sm. That basically means that whenever you refer to en you will be referring to en_core_web_sm. In other words, en after linking is not a "real" package, is just a name for en_core_web_sm.

However, it doesn't work the other way. You can't refer directly to en_core_web_sm because your system doesn't know you have it installed. When you did spacy download en you basically did a pip install. So pip knows that you have a package named en installed for your python distribution, but knows nothing about the package en_core_web_sm. This package is just replacing package en when you import it, which means that package en is just a softlink to en_core_web_sm.

Of course, you can directly download en_core_web_sm, using the command: python -m spacy download en_core_web_sm, or you can even link the name en to other models as well. For example, you could do python -m spacy download en_core_web_lg and then python -m spacy link en_core_web_lg en. That would make en a name for en_core_web_lg, which is a large spaCy model for the English language.

I had the same problem. After executing the python -m spacy download en_core_web_sm, it was giving an error. Then I executed below lines of code and it successfully installed and runs. I hope it helps you too.

try:
    nlp = spacy.load("en_core_web_sm")
except OSError:
    from spacy.cli import download
    download("en_core_web_sm")
    nlp = spacy.load("en_core_web_sm")

You might need to install the specific module too after installing spacy. Try:

python -m spacy.en.download all

Here is the reference link: https://pypi.python.org/pypi/spacy

Hi @PRADEEPCHEEKATLA
Here are the details

Cluster - 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)
Python - 3.8.10
spaCy version 3.3

Code snippet

import spacy  
!python -m spacy download en_core_web_sm   
from spacy import displacy  
  
nlp = spacy.load("en_core_web_sm")  
# Process whole documents  
text = ("When Sebastian Thrun started working on self-driving cars at "  
        "Google in 2007, few people outside of the company took him "  
        "seriously. “I can tell you very senior CEOs of major American "  
        "car companies would shake my hand and turn away because I wasn’t "  
        "worth talking to,” said Thrun, in an interview with Recode earlier "  
        "this week.")  
doc = nlp(text)  

The error I get is "OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a Python package or a valid path to a data directory."

I also see these messages. I don't know if it is relevant

/databricks/python3/lib/python3.8/site-packages/spacy/util.py:845: UserWarning: [W094] Model 'en_core_web_sm' (2.2.5) specifies an under-constrained spaCy version requirement: >=2.2.2. This can lead to compatibility problems with older versions, or as new spaCy versions are released, because the model may say it's compatible when it's not. Consider changing the "spacy_version" in your meta.json to a version range, with a lower and upper pin. For example: >=3.3.0,<3.4.0 warnings.warn(warn_msg)

Also the message when installing 'en_core_web_sm"

"Defaulting to user installation because normal site-packages is not writeable"

As I had mentioned, this worked briefly and then stopped.

find a location of your 'en_core_web_sm' in spacy use in load directory. e.g model = spacy.load('path/....')

this is happening because spacy vocab for English is missing, try to activate the python environment and run : python -m spacy download en this will download English vocabulary.

It worked for me, hope it works for you. :)

Thanks Regards