102 lines
2.4 KiB
Markdown
102 lines
2.4 KiB
Markdown
# Processing text data
|
|
|
|
Polysemy - One words maps to many concepts
|
|
Synonymy - One concept maps to many words
|
|
|
|
Word order is important
|
|
> Venetian blind vs blind Venetian
|
|
|
|
> man bites dog vs dog bites man
|
|
|
|
Language is generative, there are many ways of expressing the same proposition or assertion.
|
|
> Starbucks coffe is my favorite
|
|
|
|
> The place I like most to feed my caffeine addiction is the company from Seattle with branches everywhere
|
|
|
|
Language is changing
|
|
> I want to buy a mobile
|
|
|
|
Ill-formed input
|
|
> accomodation office
|
|
|
|
Co-ordination, negation, etc.
|
|
> This is not a talk about neuro-linguistic programming
|
|
|
|
Multi-linguality
|
|
> Claudia Schiffer is on the cover of Elle
|
|
|
|
Sarcasm, irony, slang, jargon, etc.
|
|
> That was a wicked lecture
|
|
|
|
> Yep - the coffee break was the best part
|
|
|
|
# Text processing fundamentals 1
|
|
## Processing text data (normalizing text)
|
|
|
|
As humans, we process text data effortlessly, don't we?
|
|
|
|
> DRUNK GETS NINE YEARS IN VIOLIN CASE
|
|
> STOLEN PAITING FOUND BY TREE
|
|
> RED TAPE HOLDS UP NEW BRIDGE
|
|
|
|
Language is **ambiguous**.
|
|
|
|
To determine structure, we must resolve ambiguity. Ambiguity exists at many levels.
|
|
|
|
**Lexical analysis (tokenization)**
|
|
> The cat sat on the map
|
|
>
|
|
> I can't tokenize this sentence
|
|
|
|
**Stop word removal**
|
|
> The Who, The The, Take That...
|
|
>
|
|
> To be or not to be (all are stop words)
|
|
|
|
**Stemming** - We remove endings to get stems.
|
|
> fishing, fished, fish, fisher -> fish
|
|
>
|
|
> argue, argued, argues, arguing -> argu
|
|
|
|
**Lemmatization** - Linguistically principled analysis
|
|
> Passing -> pass + ING
|
|
>
|
|
> Were -> be + PAST
|
|
>
|
|
> Delegate = de-leg-ate (?)
|
|
>
|
|
> Ratify = rat-ify (?)
|
|
|
|
**Morphology** (prefixes, suffixes, etc.)
|
|
> gebäudereinigungsfirmenangestellter -> Gebaude + Reiningung + Firma + Angestellter (bulding + cleaning + company + employee)
|
|
|
|
**Syntax** - part of speech tagging
|
|
* noun, pronoun, adjective, determiner, verb, adverb, preposition, conjunction, and interjection
|
|
> book -> NOUN, VERB
|
|
> that -> DETERMINER
|
|
> flight -> NOUN
|
|
> Book that flight -> VERB DET NOUN
|
|
|
|
Ambiguity problem
|
|
> Time flies like an arrow -> NOUN VERB PREP DET NOUN
|
|
|
|
> Fruit flies like a banana -> ?
|
|
|
|
> Eat shoots and leaves -> ?
|
|
|
|
**Parsing** (grammar)
|
|
> I saw a venetian blind
|
|
>
|
|
> I saw a blind venetian
|
|
>
|
|
> I saw the man on the hill with a telescope
|
|
>
|
|
> Ruby is a game played by men with odd-shaped balls
|
|
|
|
|
|
|
|
**Sentence boundary detection**
|
|
> Punctuation denotes the end of a sentence!
|
|
>
|
|
> "But not always!", said Fred...
|