In this practical, we will apply both dictionary- and deep learning-based sentiment analysis approaches on the IMDB sentiment classification task.
We are going to use the following libraries. Take care to have them installed!
import numpy as np
import pandas as pd
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_datasets as tfds
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
from sklearn import metrics
Let's get started!¶
Here we are going to classify movie reviews as positive or negative using the text of the review. We will use the IMDB dataset that contains the text of 50,000 movie reviews from the Internet Movie Database (IMDb). These are split into 25,000 reviews for training and 25,000 reviews for testing. The training and test sets are balanced, meaning they contain an equal number of positive and negative reviews.
1. The IMDB dataset is available on TensorFlow datasets. Use the following code to download the IMDB dataset.
# Split the training set into 60% and 40% to end up with 15,000 examples
# for training, 10,000 examples for validation and 25,000 examples for testing.
train_data, validation_data, test_data = tfds.load(
name="imdb_reviews",
split=('train[:60%]', 'train[60%:]', 'test'),
as_supervised=True)
2. Use the following code to explore the data and print the first 4 examples.
train_examples_batch, train_labels_batch = next(iter(train_data.batch(4)))
train_examples_batch
train_labels_batch
The label is an integer value of either 0 or 1, where 0 is a negative review, and 1 is a positive review.
Lexicon-based sentiment analysis¶
Vader (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.
The VADER lexicon is an empirically validated by multiple independent human judges, VADER incorporates a "gold-standard" sentiment lexicon that is especially attuned to microblog-like contexts.
It has some advantages:
- Unsupervised
- Fast and deployable
- Reasonable performance even without preprocessing
However, there are some disadvantages:
- It is a rule-based approach, meaning it utilizes a list of predefined polarity scores for each word
- It cannot exceed beyond a certain performance compared to state-of-the-art NLP approaches
3. Create a Vader analyzer using the SentimentIntensityAnalyzer function, and look at the polarity scores of some example sentences.
analyzer = SentimentIntensityAnalyzer()
print(analyzer.polarity_scores("you cannot be negative"))
The output is 50% positive ad 50% neutral. The compound score is 0.4585.
4. Calculate the compound sentiment scores of the first 1,000 training data. Convert the final scores to 0 (negative) and 1 (positive).
train_examples_batch, train_labels_batch = next(iter(train_data.batch(1000)))
score = [0 for x in range(1000)]
for i in range(1000):
text = train_examples_batch.numpy()[i].decode("utf-8")
sent = analyzer.polarity_scores(text)['compound']
if(sent > 0):
score[i] = 1
5. Evaluate the performance of the predicted sentiment socres using the classification_report function. How do you analyze your results?
print(metrics.classification_report(train_labels_batch, score, target_names=['negative', 'positive']))
Deep learning-based sentiment analysis¶
In this part of the practical, we are going to use pre-trained word embedding models from TensorFlow Hub (https://tfhub.dev/) to do sentiment classification on movie reviews. TensorFlow Hub is a repository of trained machine learning models.
6. Use a pre-trained model from TensorFlow Hub called "google/nnlm-en-dim50/2", and create a Keras embedding layer that uses this model to embed the sentences, and try it out on a couple of input examples.
# Token based text embedding trained on English Google News 7B corpus.
embedding = "https://tfhub.dev/google/nnlm-en-dim50/2"
hub_layer = hub.KerasLayer(embedding, input_shape=[],
dtype=tf.string, trainable=True)
hub_layer(train_examples_batch[:3])
Here you see that no matter the length of the input text, the output shape of the embeddings is: (num_examples, embedding_dimension).
7. Build a deep learning model using the embedding layer and one hidden layer.
model = tf.keras.Sequential()
model.add(hub_layer)
model.add(tf.keras.layers.Dense(16, activation='relu'))
model.add(tf.keras.layers.Dense(1))
model.summary()
8. Compile and train the model for 10 epochs in batches of 512 samples.
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
history = model.fit(train_data.shuffle(10000).batch(512),
epochs=10,
validation_data=validation_data.batch(512),
verbose=1)
9. Evaluate the model on the test set.
results = model.evaluate(test_data.batch(512), verbose=2)
for name, value in zip(model.metrics_names, results):
print("%s: %.3f" % (name, value))
This fairly simple approach achieves an accuracy of about 85%.
10. For your next experiment load a more complex pretrained word embedding for the embedding layer. Train and evaluate your model.
embedding = "https://tfhub.dev/google/nnlm-en-dim128-with-normalization/2"
hub_layer = hub.KerasLayer(embedding, input_shape=[],
dtype=tf.string, trainable=True)
# hub_layer(train_examples_batch[:3])
Here we tried google/nnlm-en-dim128-with-normalization/2 - trained with the same NNLM (Neural Network Language Model) architecture on the same data as google/nnlm-en-dim50/2, but with a larger embedding dimension. Larger dimensional embeddings can improve on your task but it may take longer to train your model. This new model has additional text normalization such as removing punctuation. This can help if the text in your task contains additional characters or punctuation. You can try more pretrained embeddings from TensorFlow Hub, for example BERT, but rememeber that these are huge models and need a lot of training time.
In Practical 10, you will fine-tune and fit BERT!
model = tf.keras.Sequential()
model.add(hub_layer)
model.add(tf.keras.layers.Dense(16, activation='relu'))
model.add(tf.keras.layers.Dense(1))
model.summary()
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
history = model.fit(train_data.shuffle(10000).batch(512),
epochs=10,
validation_data=validation_data.batch(512),
verbose=1)
results = model.evaluate(test_data.batch(512), verbose=2)
for name, value in zip(model.metrics_names, results):
print("%s: %.3f" % (name, value))