Named-entity recognition

Check this out: https://hitz-zentroa.github.io/GoLLIE/ ICLR 2024 paper, current SOTA on IE including NER, you write your expected classes and describe them as python dataclasses specified by guidelines and get all the entities, sub-attributes included. Works amazingly! More on reddit.com

In industry we're mostly pragmatic engineers who aren't aiming for SOTA but "whatever works and is cheapest". NER - perhaps in combination with some form of co-reference resolution can be useful in and of itself for some use cases: for example clients might want to group/filter documents by which people and organisations are mentioned most within them. Likewise POS tagging for identifying verb chunks and noun chunks for the purpose of metadata enrichment or to improve document retrieval is quite common. Both NER and POS are useful upstream tasks that help with co-reference resolution and entity linking. Regarding transformers and older methods "no longer" being useful: whilst some companies in industry (typically the well funded incumbents like FAANG and unicorns) are obsessed with transformers, the rest of the industry is decidedly /NOT/ blinded by the transformers trend. At my company the philosophy is to start with simple models and move towards more complex modelling approaches only if you have to. If I can get ~0.93 micro F1 on a text classification problem using bag-of-words features and a logistic regression model that will happily chug through 100k inferences/min on a $25/month virtual server, it is unlikely my customer will want to pay $500/month for the same throughput and 0.96 micro F1 using a fine-tuned huggingface BERTForClassification model. Whether you're planning on getting into industry or whether you're planning on staying in academia I would strongly recommend reading around and familiarising yourself with what is now considered "old school". In industry you might find you're using "old school" methods a lot more than you are new shiny models and in academia you might find that deeply understanding old models and new models helps you to unlock new ways to think about problems and model them like this paper by someone in my PhD cohort who found that combining "old school" LDA topic modelling with BERT contextual embeddings improved their model performance at semantic similarity detection. More on reddit.com

If spaCy’s NER isn’t picking up what you need, you’ll probably need to look into creating your own annotations and fine tuning a model or training a custom model. It isn’t too hard using BIO/BILOU tags. Things like “raw materials” and particularly niche models and brands are unlikely to be picked up by off the shelf solutions. More on reddit.com

You can try our zero-shot and few-shot NER library which can use GPT to perform predictions. https://github.com/plncmm/llmner More on reddit.com