If spaCy’s NER isn’t picking up what you need, you’ll probably need to look into creating your own annotations and fine tuning a model or training a custom model. It isn’t too hard using BIO/BILOU tags. Things like “raw materials” and particularly niche models and brands are unlikely to be picked up by off the shelf solutions. More on reddit.com

Hi, NER is basically a token level text classification problem, which can be considered to be similar to semantic segmentation in vision tasks, which is pixel level classification. To prepare the dataset, first you need to have a fixed number of labels, like any other classification problem, and each word should be labelled (an label for all words doesn't have an entity). Please ensure no words are left unlabelled. Once you have this dataset, you can try these, based on your dataset aswell: as mentioned in other comments, few-shot learning with LLMs using spacy custom NER model ( Ref: https://medium.com/@mjghadge9007/building-your-own-custom-named-entity-recognition-ner-model-with-spacy-v3-a-step-by-step-guide-15c7dcb1c416 ) BERT token level classifier (Ref: https://huggingface.co/docs/transformers/en/tasks/token_classification ) An RNN or LSTM classifier with some dense embedded features (glove, word2vec etc), and a prediction layer at each time step after the stack of (if multi-layer) RNNs I would suggest you try the 4th one only if you have enough time, otherwise invest more on preparing a good enough custom dataset and work on any of the first 3. More on reddit.com

Is there a reason you're using the OpenAI API for this specific function? This is an open source model that can be run locally for such tasks, in conjunction with OpenAI API calls. I use it to minimise my usage costs on tasks such as this and tokenising messages etc. More on reddit.com

You can try our zero-shot and few-shot NER library which can use GPT to perform predictions. https://github.com/plncmm/llmner More on reddit.com