How can I create a new entity and use it to find that entity in my test data? How do I get my tokenization to work?


Joe

I want to create a new entity: let's call it "medicine", and train it using my corpus. From there, identify all entities of "medicine". My code doesn't work, can someone help me?

import nltk


test= input("Please enter your file name")
test1= input("Please enter your second file name")

with open(test, "r") as file:  
    new = file.read().splitlines()


with open(test1, "r") as file2:
    new1= file2.read().splitlines()


for s in new:
    for x in new1:
        sample = s.replace('value', x)

        sample1 = ''.join(str(v) for v in sample)

        print(sample1)


        sentences = nltk.sent_tokenize(sample1)
        tokenized_sentences = [nltk.word_tokenize(sentence) for sentence in sentences]
        tagged_sentences = [nltk.pos_tag(sentence) for sentence in tokenized_sentences]
        chunked_sentences = nltk.ne_chunk_sents(tagged_sentences, binary=True)


        print(sentences)

def extract_entity_names(t):
    entity_names = []

    if hasattr(t, 'label') and t.label:
        if t.label() == 'NE':
            entity_names.append(' '.join([child[0] for child in t]))
        else:
            for child in t:
                entity_names.extend(extract_entity_names(child))

    return entity_names
0x5050

How can I create a new entity and use it to find that entity in my test data?

Named entity recognizers are probabilistic models, neural models or linear models. in your code

chunked_sentences = nltk.ne_chunk_sents(tagged_sentences, binary=True)

make this prediction. Therefore, if you want it to recognize new entity types, you should first train the classifier on annotated data containing the new entity types.

Somehow my code doesn't work,

As mentioned, you didn't train the NLTK model with your own data, so it won't work.

How do I get my tokenization to work?

The tokenizer extracts only word tokens, this line does it in code

tokenized_sentences = [nltk.word_tokenize(sentence) for sentence in sentences]

However, the tokenizer does not directly predict named entities.

If you want to use NLTK to train a model to predict custom named entities like medicine, try this tutorial.

From my personal experience, NLTK may not be suitable for this, check out Spacy.

Related


How can I get the current user into my entity class?

Mike I'm developing my first Symfony 4 application and migrating from Symfony 2+ and symfony 3+. Right now I'm developing a backend where all my entity classes have addedBy()and updatedBy()methods where I need to log the currently logged in admin. I want somet

How can I get the current user into my entity class?

Mike I'm developing my first Symfony 4 application and migrating from Symfony 2+ and symfony 3+. Right now I'm developing a backend where all my entity classes have addedBy()and updatedBy()methods where I need to log the currently logged in admin. I want somet

How can I get the current user into my entity class?

Mike I'm developing my first Symfony 4 application and migrating from Symfony 2+ and symfony 3+. Right now I'm developing a backend where all my entity classes have addedBy()and updatedBy()methods where I need to log the currently logged in admin. I want somet

How can I get the current user into my entity class?

Mike I'm developing my first Symfony 4 application and migrating from Symfony 2+ and symfony 3+. Right now I'm developing a backend where all my entity classes have addedBy()and updatedBy()methods where I need to log the currently logged in admin. I want somet

How can I connect my entity fields to the same entity?

say I added a field "Parent Category" to my entity "Category" to be able to connect one category to another: class Category { /** * @ORM\Id() * @ORM\GeneratedValue() * @ORM\Column(type="integer") */ private $id; /** * @ORM\ManyToOne(targetEn

How do I get my torrents to work?

Aelo-74 I'm still new to Ruby/Rails and need your help to make sure the seed is good before running the command.rails db:seed Here is my seed: require 'faker' puts "Creating 10 fake user heros..." 10.times do user = User.create!( username: Faker::DcComi

How do I get my torrents to work?

Aelo-74 I'm still new to Ruby/Rails and need your help to make sure the seed is good before running the command.rails db:seed Here is my seed: require 'faker' puts "Creating 10 fake user heros..." 10.times do user = User.create!( username: Faker::DcComi

How do I get my torrents to work?

Aelo-74 I'm still new to Ruby/Rails and need your help to make sure the seed is good before running the command.rails db:seed Here is my seed: require 'faker' puts "Creating 10 fake user heros..." 10.times do user = User.create!( username: Faker::DcComi

How do I get my $_Post to work

turquoise I'm trying to get my site to submit an email every time someone sends a message (now I'm struggling to get variables to work...). It's a learning process, so I'm giving it my all here, just trying to teach myself. But don't like global php commands $

How do I get my ListBox to work

Niermag I'm trying to get a listbox to work that will function very much like this. These Availableexercises are derived from Migration/Configuration. I need to be able to add multiples Availableto each exercise Selected Regime. I'm using viewmodels to access

How do I get my torrents to work?

Aelo-74 I'm still new to Ruby/Rails and need your help to make sure the seed is good before running the command.rails db:seed Here is my seed: require 'faker' puts "Creating 10 fake user heros..." 10.times do user = User.create!( username: Faker::DcComi

How do I get my torrents to work?

Aelo-74 I'm still new to Ruby/Rails and need your help to make sure the seed is good before running the command.rails db:seed Here is my seed: require 'faker' puts "Creating 10 fake user heros..." 10.times do user = User.create!( username: Faker::DcComi

How do I get my torrents to work?

Aelo-74 I'm still new to Ruby/Rails and need your help to make sure the seed is good before running the command.rails db:seed Here is my seed: require 'faker' puts "Creating 10 fake user heros..." 10.times do user = User.create!( username: Faker::DcComi

How do I get my torrents to work?

Aelo-74 I'm still new to Ruby/Rails and need your help to make sure the seed is good before running the command.rails db:seed Here is my seed: require 'faker' puts "Creating 10 fake user heros..." 10.times do user = User.create!( username: Faker::DcComi

How do I get my ListBox to work

Niermag I'm trying to get a listbox to work that will function very much like this. These Availableexercises are derived from Migration/Configuration. I need to be able to add multiples Availableto each exercise Selected Regime. I'm using viewmodels to access

How can I create or generate new entity on RunTIme in Swift

Hussein Ahmed I'm working on a data based core project. I want to create a dynamic core data entity at runtime with the name of the current date. thanks Jerry Crannock You cannot dynamically create entities, but you can dynamically create managed objects in an

How can I create or generate new entity on RunTIme in Swift

Hussein Ahmed I'm working on a data based core project. I want to create a dynamic core data entity at runtime with the name of the current date. thanks Jerry Crannock You cannot dynamically create entities, but you can dynamically create managed objects in an

How can I set my entity fields to "nullable=false"?

User 10800263 I am connecting my entity "data" with my entity "document" /** * @ORM\ManyToOne(targetEntity="Documents") * @ORM\JoinColumn(name="document", referencedColumnName="id") */ private $document; My database shows NULL if the data is not

How can I add another variable to my entity?

Chris Gugel I'm really new to hibernate/spring data, my entity is set up just fine, but now I try to add another variable (orchestral) to the entity, but I keep getting the following error: Failed to call init method; nested Exception is javax.persistence.Pers