Are you committed to using NLTK/Python? I ran into the same problems as you, and had much better results using Stanford's named-entity recognizer: http://nlp.stanford.edu/software/CRF-NER.shtml. The process for training the classifier using your own data is very well-documented in the FAQ.
If you really need to use NLTK, I'd hit up the mailing list for some advice from other users: http://groups.google.com/group/nltk-users.
Hope this helps!
Answer from jjdubs on Stack OverflowAre you committed to using NLTK/Python? I ran into the same problems as you, and had much better results using Stanford's named-entity recognizer: http://nlp.stanford.edu/software/CRF-NER.shtml. The process for training the classifier using your own data is very well-documented in the FAQ.
If you really need to use NLTK, I'd hit up the mailing list for some advice from other users: http://groups.google.com/group/nltk-users.
Hope this helps!
You can easily use the Stanford NER alongwith nltk. The python script is like
from nltk.tag.stanford import NERTagger
import os
java_path = "/Java/jdk1.8.0_45/bin/java.exe"
os.environ['JAVAHOME'] = java_path
st = NERTagger('../ner-model.ser.gz','../stanford-ner.jar')
tagging = st.tag(text.split())
To train your own data and to create a model you can refer to the first question on Stanford NER FAQ.
The link is http://nlp.stanford.edu/software/crf-faq.shtml
Videos
The default NE chunker in nltk is a maximum entropy chunker trained on the ACE corpus (http://catalog.ldc.upenn.edu/LDC2005T09). It has not been trained to recognise dates and times, so you need to train your own classifier if you want to do that.
Have a look at http://mattshomepage.com/articles/2016/May/23/nltk_nec/, the whole process is explained very well.
Also, there is a module called timex in nltk_contrib which might help you with your needs. https://github.com/nltk/nltk_contrib/blob/master/nltk_contrib/timex.py
Named entity recognition is not an easy problem, do not expect any library to be 100% accurate. You shouldn't make any conclusions about NLTK's performance based on one sentence. Here's another example:
sentence = "I went to New York to meet John Smith";
I get
(S
I/PRP
went/VBD
to/TO
(NE New/NNP York/NNP)
to/TO
meet/VB
(NE John/NNP Smith/NNP))
As you can see, NLTK does very well here. However, I couldn't get NLTK to recognise today or tomorrow as temporal expressions. You can try Stanford SUTime, it is a part of Stanford CoreNLP- I have used it before I it works quite well (it is in Java though).
Hey!
I am looking for a way to train the NLTK chunker using my own text, For e.g., If I want to get all the car manufacturers name out of a text, It won't be possible from the existing corpus. can someone point me to the right direction?
nltk.ne_chunk returns a nested nltk.tree.Tree object so you would have to traverse the Tree object to get to the NEs.
Take a look at Named Entity Recognition with Regular Expression: NLTK
>>> from nltk import ne_chunk, pos_tag, word_tokenize
>>> from nltk.tree import Tree
>>>
>>> def get_continuous_chunks(text):
... chunked = ne_chunk(pos_tag(word_tokenize(text)))
... continuous_chunk = []
... current_chunk = []
... for i in chunked:
... if type(i) == Tree:
... current_chunk.append(" ".join([token for token, pos in i.leaves()]))
... if current_chunk:
... named_entity = " ".join(current_chunk)
... if named_entity not in continuous_chunk:
... continuous_chunk.append(named_entity)
... current_chunk = []
... else:
... continue
... return continuous_chunk
...
>>> my_sent = "WASHINGTON -- In the wake of a string of abuses by New York police officers in the 1990s, Loretta E. Lynch, the top federal prosecutor in Brooklyn, spoke forcefully about the pain of a broken trust that African-Americans felt and said the responsibility for repairing generations of miscommunication and mistrust fell to law enforcement."
>>> get_continuous_chunks(my_sent)
['WASHINGTON', 'New York', 'Loretta E. Lynch', 'Brooklyn']
>>> my_sent = "How's the weather in New York and Brooklyn"
>>> get_continuous_chunks(my_sent)
['New York', 'Brooklyn']
You can also extract the label of each Name Entity in the text using this code:
import nltk
for sent in nltk.sent_tokenize(sentence):
for chunk in nltk.ne_chunk(nltk.pos_tag(nltk.word_tokenize(sent))):
if hasattr(chunk, 'label'):
print(chunk.label(), ' '.join(c[0] for c in chunk))
Output:
GPE WASHINGTON
GPE New York
PERSON Loretta E. Lynch
GPE Brooklyn
You can see Washington, New York and Brooklyn are GPE means geo-political entities
and Loretta E. Lynch is a PERSON
Once you have your own lists of named entities, and you're only interested in extracting the relations, I believe there are simpler solutions (although I never tried relation extraction in NLTK, so I might be wrong):
ReVerb - a tool written in Java. Once it produces the results, you can simply keep the rows, where your labels are present as objects of the relation.
OpenIE - the successor of ReVerb (also written in Java). The authors claim better perfomance, and the output might be more informative.
IEPY - a relation extraction tool in Python. You should be able to provide your own labels/named entities using gazetees.
MITIE - this library has bindings in Python, and it offers relation extraction functionality.
Lots of keyword extraction techniques are there depends on factors like:
Grammatical quality of text. Length of text Are you looking for a single keyword or phrasal keyword etc. But in general If you have long text and you want to extract keywords automatically from that, I would recommend you to go through follow articles:
TextRank
Rake [Rapid Automatic Keyword Extraction]
Topica
Also to extract custom(special) keywords which is not coming through above techniques, have a look at this post.