Обработка натурального языка: как обрабатывать данные NLP?

text = '''The Time Traveller (for so it will be convenient to speak of him) was
expounding a recondite matter to us. His pale grey eyes shone and
twinkled, and his usually pale face was flushed and animated. The fire
burnt brightly, and the soft radiance of the incandescent lights in the
lilies of silver caught the bubbles that flashed and passed in our
glasses. Our chairs, being his patents, embraced and caressed us rather
than submitted to be sat upon, and there was that luxurious
after-dinner atmosphere, when thought runs gracefully free of the
trammels of precision. And he put it to us in this way—marking the
points with a lean forefinger—as we sat and lazily admired his
earnestness over this new paradox (as we thought it) and his fecundity.

"You must follow me carefully. I shall have to controvert one or two
ideas that are almost universally accepted. The geometry, for instance,
they taught you at school is founded on a misconception."

"Is not that rather a large thing to expect us to begin upon?" said
Filby, an argumentative person with red hair.

"I do not mean to ask you to accept anything without reasonable ground
for it. You will soon admit as much as I need from you. You know of
course that a mathematical line, a line of thickness _nil_, has no real
existence. They taught you that? Neither has a mathematical plane.
These things are mere abstractions."

"That is all right," said the Psychologist.

"Nor, having only length, breadth, and thickness, can a cube have a
real existence."

"There I object," said Filby. "Of course a solid body may exist. All
real things—"

"So most people think. But wait a moment. Can an _instantaneous_ cube
exist?"

"Don't follow you," said Filby.

"Can a cube that does not last for any time at all, have a real
existence?"

Filby became pensive. "Clearly," the Time Traveller proceeded, "any
real body must have extension in _four_ directions: it must have
Length, Breadth, Thickness, and—Duration. But through a natural
infirmity of the flesh, which I will explain to you in a moment, we
incline to overlook this fact. There are really four dimensions, three
which we call the three planes of Space, and a fourth, Time. There is,
however, a tendency to draw an unreal distinction between the former
three dimensions and the latter, because it happens that our
consciousness moves intermittently in one direction along the latter
from the beginning to the end of our lives."'''

import re
text = re.sub("\n"," ",text)

import nltk
from nltk.stem import PorterStemmer

word_stemmer = PorterStemmer()
for word in text.split(" "):
  if len(word)>10:
    print((word,word_stemmer.stem(word)))

('incandescent', 'incandesc') ('after-dinner', 'after-dinn') ('atmosphere,', 'atmosphere,') ('way—marking', 'way—mark') ('forefinger—as', 'forefinger—a') ('earnestness', 'earnest') ('universally', 'univers') ('misconception."', 'misconception."') ('argumentative', 'argument') ('mathematical', 'mathemat') ('mathematical', 'mathemat') ('abstractions."', 'abstractions."') ('Psychologist.', 'psychologist.') ('existence."', 'existence."') ('_instantaneous_', '_instantaneous_') ('existence?"', 'existence?"') ('directions:', 'directions:') ('and—Duration.', 'and—duration.') ('dimensions,', 'dimensions,') ('distinction', 'distinct') ('consciousness', 'conscious') ('intermittently', 'intermitt')

from nltk.stem import LancasterStemmer
Lanc_stemmer = LancasterStemmer()

for word in text.split(" "):
  if len(word)>10:
    print((word,Lanc_stemmer.stem(word)))

('incandescent', 'incandesc') ('after-dinner', 'after-dinn') ('atmosphere,', 'atmosphere,') ('way—marking', 'way—marking') ('forefinger—as', 'forefinger—as') ('earnestness', 'earnest') ('universally', 'univers') ('misconception."', 'misconception."') ('argumentative', 'argu') ('mathematical', 'mathem') ('mathematical', 'mathem') ('abstractions."', 'abstractions."') ('Psychologist.', 'psychologist.') ('existence."', 'existence."') ('_instantaneous_', '_instantaneous_') ('existence?"', 'existence?"') ('directions:', 'directions:') ('and—Duration.', 'and—duration.') ('dimensions,', 'dimensions,') ('distinction', 'distinct') ('consciousness', 'conscy') ('intermittently', 'intermit')

nltk.download('wordnet')
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()

for word in text.split():
  if len(word)>5 and word!=lemmatizer.lemmatize(word):
    print((word,lemmatizer.lemmatize(word)))
  elif len(word)>10:
    print((word,lemmatizer.lemmatize(word)))

('incandescent', 'incandescent') ('lights', 'light') ('lilies', 'lily') ('bubbles', 'bubble') ('after-dinner', 'after-dinner') ('atmosphere,', 'atmosphere,') ('trammels', 'trammel') ('way—marking', 'way—marking') ('points', 'point') ('forefinger—as', 'forefinger—as') ('earnestness', 'earnestness') ('universally', 'universally') ('misconception."', 'misconception."') ('argumentative', 'argumentative') ('mathematical', 'mathematical') ('mathematical', 'mathematical') ('things', 'thing') ('abstractions."', 'abstractions."') ('Psychologist.', 'Psychologist.') ('existence."', 'existence."') ('_instantaneous_', '_instantaneous_') ('existence?"', 'existence?"') ('directions:', 'directions:') ('and—Duration.', 'and—Duration.') ('dimensions,', 'dimensions,') ('planes', 'plane') ('distinction', 'distinction') ('dimensions', 'dimension') ('consciousness', 'consciousness') ('intermittently', 'intermittently')

nltk.download('averaged_perceptron_tagger')
from nltk import pos_tag

for sentence in text.split(".")[0]:
  token = sentence.split(" ")[1:]
  token = [i for i in token if i] 
  tokens_tag = pos_tag(token)
  print(tokens_tag)

[('Time', 'NNP'), ('Traveller', 'NNP'), ('(for', 'NNP'), ('so', 'IN'), ('it', 'PRP'), ('will', 'MD'), ('be', 'VB'), ('convenient', 'JJ'), ('to', 'TO'), ('speak', 'VB'), ('of', 'IN'), ('him)', 'NN'), ('was', 'VBD'), ('expounding', 'VBG'), ('a', 'DT'), ('recondite', 'JJ'), ('matter', 'NN'), ('to', 'TO'), ('us', 'PRP')]

import re
text = re.sub("\n"," ",text)

import nltk
nltk.download('wordnet')
from nltk.corpus import wordnet

for word in text.split(" "):
  if len(word)>5:
    list_syn = []
    for syn in wordnet.synsets(word): 
      for lemm in syn.lemmas():
        if lemm.name() not in list_syn:
          list_syn.append(lemm.name())
    if list_syn:
      print(word + ":-")
      print(" "+str(list_syn))

Traveller:-
 ['traveler', 'traveller']
convenient:-
 ['convenient', 'commodious']
expounding:-
 ['exposition', 'expounding', 'elaborate', 'lucubrate', 'expatiate', 'exposit', 'enlarge', 'flesh_out', 'expand', 'expound', 'dilate', 'set_forth']
recondite:-
 ['abstruse', 'deep', 'recondite']
matter:-
 ['matter', 'affair', 'thing', 'topic', 'subject', 'issue', 'count', 'weigh']
usually:-
 ['normally', 'usually', 'unremarkably', 'commonly', 'ordinarily']
flushed:-
 ['blush', 'crimson', 'flush', 'redden', 'level', 'even_out', 'even', 'scour', 'purge', 'sluice', 'flushed', 'rose-cheeked', 'rosy', 'rosy-cheeked', 'red', 'reddened', 'red-faced']
radiance:-
 ['radiance', 'glow', 'glowing', 'radiancy', 'shine', 'effulgence', 'refulgence', 'refulgency']
incandescent:-
 ['incandescent', 'candent']
lights:-
 ['light', 'visible_light', 'visible_radiation', 'light_source', 'luminosity', 'brightness', 'brightness_level', 'luminance', 'luminousness', 'illumination', 'lightness', 'lighting', 'sparkle', 'twinkle', 'spark', 'Inner_Light', 'Light', 'Light_Within', 'Christ_Within', 'lighter', 'igniter', 'ignitor', 'illume', 'illumine', 'light_up', 'illuminate', 'fire_up', 'alight', 'perch', 'ignite', 'fall', 'unhorse', 'dismount', 'get_off', 'get_down']

for word in text.split(" "):
  if len(word)>5:
    list_ant = []
    for syn in wordnet.synsets(word): 
      for lemm in syn.lemmas():
        if lemm.antonyms(): 
            list_ant.append(lemm.antonyms()[0].name())
    if list_ant:
      print(word + ":-")
      print(" "+str(list_ant))

convenient:- ['inconvenient', 'incommodious'] 
expounding:- ['contract'] 
usually:- ['remarkably'] 
lights:- ['dark', 'extinguish'] 
caught:- ['unhitch'] 
passed:- ['fail', 'fail', 'be_born'] 
thought:- ['forget'] 
gracefully:- ['gracelessly', 'ungraciously', 'ungracefully'] 
points:- ['unpointedness'] 
admired:- ['look_down_on'] 
earnestness:- ['frivolity'] 
thought:- ['forget'] 
follow:- ['precede', 'predate', 'precede'] 
founded:- ['abolish'] 
argumentative:- ['unargumentative'] 
accept:- ['reject', 'refuse', 'refuse'] 
reasonable:- ['unreasonable'] 
ground:- ['figure'] 
course:- ['unnaturally'] 
mathematical:- ['verbal'] 
thickness:- ['thinness', 'thinness'] 
mathematical:- ['verbal'] 
having:- ['lack', 'abstain', 'refuse'] 
course:- ['unnaturally'] 
follow:- ['precede', 'predate', 'precede'] 
extension:- ['flexion'] 
natural:- ['unnatural', 'artificial', 'supernatural', 'flat'] 
incline:- ['indispose'] 
overlook:- ['attend_to'] 
unreal:- ['real', 'real', 'natural', 'substantial'] 
former:- ['latter', 'latter'] 
happens:- ['dematerialize', 'dematerialise'] 
consciousness:- ['unconsciousness', 'incognizance'] 
latter:- ['former', 'former'] 
beginning:- ['ending', 'end','finish']

import spacy
spacy_obj = spacy.load('en_core_web_sm')

spacy_text = spacy_obj(text)
for phrase in spacy_text.noun_chunks:
  print(phrase)

The Time Traveller
a recondite matter
His pale grey eyes
his usually pale face
the soft radiance
the incandescent lights
a lean forefinger
this new paradox
one or two
ideas
an argumentative person
reasonable ground
a mathematical line
no real
existence
a mathematical plane
mere abstractions
the Psychologist
a
real existence
an _instantaneous_ cube
a real
existence
the Time Traveller
_four_ directions
a natural
infirmity
the three planes
an unreal distinction
the former
three dimensions
our
consciousness