You can access the course materials quickly from

https://ayoubbagheri.nl/r_tm/

• Text data is everywhere!
• A lot of world’s data is in unstructured text format
• The course teaches
  • text mining techniques
  • using R
  • on a variety of applications
  • in many domains.

• Different things can mean more or less the same (“data science” vs. “statistics”)
• Context dependency (“You have very nice shoes”);
• Same words with different meanings (“to sanction”, “bank”);
• Lexical ambiguity (“we saw her duck”)
• Irony, sarcasm (“That’s just what I needed today!”, “Great!”, “Well, what a surprise.”)
• Figurative language (“He has a heart of stone”)
• Negation (“not good” vs. “good”), spelling variations, jargon, abbreviations
• All the above are different over languages, 99% of work is on English!

• “the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources” Hearst (1999)

• Text mining is about looking for patterns in text, in a similar way that data mining can be loosely described as looking for patterns in data.

• Text mining describes a set of linguistic, statistical, and machine learning techniques that model and structure the information content of textual sources. (Wikipedia)

• We won’t solve linguistics …
• In spite of the problems, text mining can be quite effective!

https://dh2017.adho.org/abstracts/079/079.pdf

https://asreview.nl/

• Automatically classify political news from sports news

• Authorship identification

• Age/gender identification

• Language Identification

• …

Regular Expression one of the first and important tm techniques we must learn about!

Really clever “wild card” expressions for matching and parsing strings.

http://en.wikipedia.org/wiki/Regular_expression

• A formal language for specifying text strings

• How can we search for any of these?

  • netherland

  • netherlands

  • Netherland

  • Netherlands

There are some rules on how to write REs…

• Define the text string
  • text <- “How much wood would a woodchuck chuck if a woodchucks could chuck wood?”
• Check if “woodchuck” is in the text
  • found <- grepl(“woodchuck”, text)
• Print the result
  • print(found)

The use of the brackets [] to specify a disjunction of characters:

The pipe sympol | is also for disjunction:

The use of the brackets [] plus the dash - to specify a range:

The caret ^ for negation or just to mean ^:

The question mark ? marks optionality of the previous expression:

The use of the period . to specify any character:

Aliases for common sets of characters:

The backslash for escaping!

• Patterns are greedy: In these cases regular expressions always match the largest string they can, expanding to cover as much of a string as they can.

• Enforce non-greedy matching, using another meaning of the ? qualifier.

  • The operator *? is a Kleene star that matches as little text as possible.
  • The operator +? is a Kleene plus that matches as little text as possible.

Some characters that need to be backslashed:

• Very powerful and quite cryptic

• Fun once you understand them

• Regular expressions are a programming language with characters

• It is kind of an “old school” language

The primary R functions for dealing with regular expressions are:

• grep(), grepl(): Search for matches of a regular expression/pattern in a character vector

• regexpr(), gregexpr(): Search a character vector for regular expression matches and return the indices where the match begins; useful in conjunction with regmatches()

• sub(), gsub(): Search a character vector for regular expression matches and replace that match with another string

• The stringr package provides a series of functions implementing much of the regular expression functionality in R but with a more consistent and rationalized interface.

• Find all instances of the word “the” in a text.

Misses capitalized examples

Incorrectly returns words such as other or Netherlands

[^a-zA-Z] [tT]he [^a-zA-Z]

Still not compeletly correct! What is missing?

txt <- "The other the Netherlands will then be without the"
r   <- gregexpr("[^a-zA-Z][tT]he[^a-zA-Z]", txt)
print(regmatches(txt, r))

## [[1]]
## [1] " the "

r   <- gregexpr("(^|[^a-zA-Z])[tT]he($|[^a-zA-Z])", txt)
print(regmatches(txt, r))

## [[1]]
## [1] "The "  " the " " the"

• The process we just went through was based on

  • Matching strings that we should not have matched (there, Netherlands)

  
  

  • Not matching things that we should have matched (The)

  
  

• The process we just went through was based on fixing two kinds of errors

  • Matching strings that we should not have matched (there, Netherlands)

    • False positives (Type I)

  • Not matching things that we should have matched (The)

    • False negatives (Type II)

• In NLP we are always dealing with these kinds of errors.

• Reducing the error rate for an application often involves:

  • Increasing precision (minimizing false positives)

  • Increasing recall (minimizing false negatives).

Suppose we want to build an application to help a user buy a car from textual catalogues. The user looks for any car cheaper than $10,000.00.

Assume we are using the following data: txt <- c(“Price of Tesla S is $8599.99.”, “Audi Q4 is $7000.”, “BMW X5 costs $900”)

Which RE will help us to do this?

• (ˆ|\W)\$[0-9]{0,4}(\.[0-9][0-9])*
• (ˆ|\W)\$[0-9]{0,3}(\.[0-9][0-9])+
• (ˆ|\W)\$[0-9]{0,4}(\.[0-9][0-9])?
• (ˆ|\W)\$[0-9][0-9][0-9][0-9](\.[0-9][0-9])*

txt <- c("Price of Tesla S is $8599.99.", 
         "Audi Q4 is $7000.", 
         "BMW X5 costs $900") 
r <- gregexpr("(ˆ|\\W)\\$[0-9]{0,4}(\\.[0-9][0-9])?", txt)
print(regmatches(txt, r))

## [[1]]
## [1] " $8599.99"
## 
## [[2]]
## [1] " $7000"
## 
## [[3]]
## [1] " $900"

• Text data is everywhere!
• Language is hard!
• Sophisticated sequences of regular expressions are often the first model for any text processing tool
• Regular expressions are a cryptic but powerful language for matching strings and extracting elements from those strings
• The basic problem of text mining is that text is not a neat data set
• One solution: text pre-processing

• is an approach for cleaning and noise removal of text data.
• brings your text into a form that is analyzable for your task.
• transforms text into a more digestible form so that machine learning algorithms can perform better.