Natural Language Processing with TensorFlow
- Sumit Dey
- Mar 29, 2022
- 25 min read
Language processing(Natural) is used for computers to understand the human natural language. The main goal of natural language processing (NLP) is to derive information from natural language. Natural language is a broad term but you can consider it to cover any of the following:
Text (such as that contained in an email, blog post, book, Tweet)
Speech (a conversation you have with a lawyer, voice commands you give to a smart speaker)
Under the umbrellas of text and speech, there are many different things you might want to do. If you're building an email application, you might want to scan incoming emails to see if they're spam or not spam (classification). If you're trying to analyze customer feedback complaints, you might want to discover which section of your business they're for.
To get hands-on with NLP in TensorFlow, we're going to practice the steps we've used previously but this time with text data:
Text -> turn into numbers -> build a model -> train the model to find patterns -> use patterns (make predictions)Helper Functions
We can able to create a bunch of helper functions to do small tasks, rather than rewrite all these, however, here are the helper functions below
# Create function to unzip a zipfile into current working directory
# (since we're going to be downloading and unzipping a few files)
import zipfile
import datetime
import matplotlib.pyplot as plt
def unzip_data(filename):
"""
Unzips filename into the current working directory.
Args:
filename (str): a filepath to a target zip folder to be unzipped.
"""
zip_ref = zipfile.ZipFile(filename, "r")
zip_ref.extractall()
zip_ref.close()
def create_tensorboard_callback(dir_name, experiment_name):
"""
Creates a TensorBoard callback instand to store log files.
Stores log files with the filepath:
"dir_name/experiment_name/current_datetime/"
Args:
dir_name: target directory to store TensorBoard log files
experiment_name: name of experiment directory (e.g. efficientnet_model_1)
"""
log_dir = dir_name + "/" + experiment_name + "/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(
log_dir=log_dir
)
print(f"Saving TensorBoard log files to: {log_dir}")
return tensorboard_callback
# Plot the validation and training data separately
def plot_loss_curves(history):
"""
Returns separate loss curves for training and validation metrics.
Args:
history: TensorFlow model History object (see: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/History)
"""
loss = history.history['loss']
val_loss = history.history['val_loss']
accuracy = history.history['accuracy']
val_accuracy = history.history['val_accuracy']
epochs = range(len(history.history['loss']))
# Plot loss
plt.plot(epochs, loss, label='training_loss')
plt.plot(epochs, val_loss, label='val_loss')
plt.title('Loss')
plt.xlabel('Epochs')
plt.legend()
# Plot accuracy
plt.figure()
plt.plot(epochs, accuracy, label='training_accuracy')
plt.plot(epochs, val_accuracy, label='val_accuracy')
plt.title('Accuracy')
plt.xlabel('Epochs')
plt.legend();
def compare_historys(original_history, new_history, initial_epochs=5):
"""
Compares two TensorFlow model History objects.
Args:
original_history: History object from original model (before new_history)
new_history: History object from continued model training (after original_history)
initial_epochs: Number of epochs in original_history (new_history plot starts from here)
"""
# Get original history measurements
acc = original_history.history["accuracy"]
loss = original_history.history["loss"]
val_acc = original_history.history["val_accuracy"]
val_loss = original_history.history["val_loss"]
# Combine original history with new history
total_acc = acc + new_history.history["accuracy"]
total_loss = loss + new_history.history["loss"]
total_val_acc = val_acc + new_history.history["val_accuracy"]
total_val_loss = val_loss + new_history.history["val_loss"]
# Make plots
plt.figure(figsize=(8, 8))
plt.subplot(2, 1, 1)
plt.plot(total_acc, label='Training Accuracy')
plt.plot(total_val_acc, label='Validation Accuracy')
plt.plot([initial_epochs-1, initial_epochs-1],
plt.ylim(), label='Start Fine Tuning') # reshift plot around epochs
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')
plt.subplot(2, 1, 2)
plt.plot(total_loss, label='Training Loss')
plt.plot(total_val_loss, label='Validation Loss')
plt.plot([initial_epochs-1, initial_epochs-1],
plt.ylim(), label='Start Fine Tuning') # reshift plot around epochs
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()Download a test dataset
Now we need a text dataset to build a model, Let's start by downloading a test dataset. We'll be using the Real or Not? dataset from Kaggle which contains text-based Tweets about natural disasters.
# Download data (same as from Kaggle)
!wget "https://github.com/sumitdeyonline/machinelearning/raw/main/nlp_getting_started.zip"
# Unzip data
unzip_data("nlp_getting_started.zip")Unzipping nlp_getting_started.zip gives the following 3 .csv files:
sample_submission.csv - an example of the file you'd submit to the Kaggle competition of your model's predictions.
train.csv - training samples of real and not real disaster Tweets.
test.csv - testing samples of real and not real disaster Tweets.
Visualize the Data
Let's visualize the data, our text data samples are in the form of .csv files. For an easy way to make them visual, let's turn them into pandas DataFrame's.
# Turn .csv files into pandas DataFrame's
import pandas as pd
train_df = pd.read_csv("train.csv")
test_df = pd.read_csv("test.csv")
train_df.head()
The training data we downloaded is probably shuffled already. But just to be sure, let's shuffle it again.
# Shuffle training dataframe
train_df_shuffled = train_df.sample(frac=1, random_state=42) # shuffle with random_state=42 for reproducibility
train_df_shuffled.head()
Notice how the training data has a "target" column.
We're going to be writing code to find patterns (e.g. different combinations of words) in the "text" column of the training dataset to predict the value of the "target" column. The test dataset doesn't have a "target" column.
Inputs (text column) -> Machine Learning Algorithm -> Outputs (target column)
Let's check how many examples of each target we have.
# How many examples of each class?
train_df.target.value_counts()
Since we have two target values, we're dealing with a binary classification problem.
It's fairly balanced too, about 60% negative class (target = 0) and 40% positive class (target = 1).Where,
1 = a real disaster Tweet
0 = not a real disaster Tweet
And what about the total number of samples we have?
# How many samples total?
print(f"Total training samples: {len(train_df)}")
print(f"Total test samples: {len(test_df)}")
print(f"Total samples: {len(train_df) + len(test_df)}")
Alright, seems like we've got a decent amount of training and test data. If anything, we've got an abundance of testing examples, usually a split of 90/10 (90% training, 10% testing) or 80/20 is sufficient. Time to visualize, let's write some code to visualize random text samples.
# Let's visualize some random training examples
import random
random_index = random.randint(0, len(train_df)-5) # create random indexes not higher than the total number of samples
for row in train_df_shuffled[["text", "target"]][random_index:random_index+5].itertuples():
_, text, target = row
print(f"Target: {target}", "(real disaster)" if target > 0 else "(not real disaster)")
print(f"Text:\n{text}\n")
print("---\n")
Preparation of Data
Split data into training and validation sets, since the test set has no labels and we need a way to evaluate our trained models, we'll split off some of the training data and create a validation set. When our model trains (tries patterns in the Tweet samples), it'll only see data from the training set and we can see how it performs on unseen data using the validation set. We'll convert our splits from pandas Series datatypes to lists of strings (for the text) and lists of ints (for the labels) for ease of use later. To split our training dataset and create a validation dataset, we'll use Scikit-Learn's train_test_split() method and dedicate 10% of the training samples to the validation set.
from sklearn.model_selection import train_test_split
# Use train_test_split to split training data into training and validation sets
train_sentences, val_sentences, train_labels, val_labels = train_test_split(train_df_shuffled["text"].to_numpy(),
train_df_shuffled["target"].to_numpy(),
test_size=0.1, # dedicate 10% of samples to validation set
random_state=42) # random state for reproducibility
# Check the lengths
len(train_sentences), len(train_labels), len(val_sentences), len(val_labels)
# View the first 10 training sentences and their labels
train_sentences[:10], train_labels[:10]
Converting text into numbers
We've got a training set and a validation set containing Tweets and labels.
Our labels are in numerical form (0 and 1) but our Tweets are in string form.
A machine learning algorithm requires its inputs to be in numerical form.
In NLP, there are two main concepts for turning text into numbers:
Tokenization - A straight mapping from a word or character or sub-word to a numerical value. There are three main levels of tokenization:
Using word-level tokenization with the sentence "I love TensorFlow" might result in "I" being 0, "love" being 1 and "TensorFlow" being 2. In this case, every word in a sequence is considered a single token.
Character-level tokenization, such as converting the letters A-Z to values 1-26. In this case, every character in a sequence is considered a single token.
Sub-word tokenization is in between word-level and character-level tokenization. It involves breaking individual words into smaller parts and then converting those smaller parts into numbers. For example, "my favorite food is pineapple pizza" might become "my, fav, vo, rite, fo, oo, od, is, pin, ine, app, le, piz, za". After doing this, these sub-words would then be mapped to a numerical value. In this case, every word could be considered multiple tokens
Embeddings - An embedding is a representation of natural language which can be learned. Representation comes in the form of a feature vector. For example, the word "dance" could be represented by the 5-dimensional vector [-0.8547, 0.4559, -0.3332, 0.9877, 0.1112]. It's important to note here, the size of the feature vector is tuneable. There are two ways to use embeddings:
Create your own embedding - Once your text has been turned into numbers (required for an embedding), you can put them through an embedding layer (such as tf.keras.layers.Embedding) and an embedding representation will be learned during model training.
Reuse a pre-learned embedding - Many pre-trained embeddings exist online. These pre-trained embeddings have often been learned on large corpora of text (such as all of Wikipedia) and thus have a good underlying representation of natural language. You can use a pre-trained embedding to initialize your model and fine-tune it to your own specific task.
It depends on your problem. You could try character-level tokenization/embeddings and word-level tokenization/embeddings and see which perform best. You might even want to try stacking them (e.g. combining the outputs of your embedding layers using tf.keras.layers.concatenate). If you're looking for pre-trained word embeddings, Word2vec embeddings, GloVe embeddings, and many of the options available on TensorFlow Hub are great places to start.
Text vectorization (tokenization)
To tokenize our words, we'll use the helpful preprocessing layer tf.keras.layers.experimental.preprocessing.TextVectorization.
The TextVectorization layer takes the following parameters:
max_tokens - The maximum number of words in your vocabulary (e.g. 20000 or the number of unique words in your text), includes a value for OOV (out of vocabulary) tokens.
standardize - Method for standardizing text. Default is "lower_and_strip_punctuation" which lowers text and removes all punctuation marks.
split - How to split text, default is "whitespace" which splits into spaces.
ngrams - How many words to contain per token split, for example, ngrams=2 splits tokens into continuous sequences of 2.
output_mode - How to output tokens, can be "int" (integer mapping), "binary" (one-hot encoding), "count" or "tf-idf". See documentation for more.
output_sequence_length - Length of tokenized sequence to output. For example, if output_sequence_length=150, all tokenized sequences will be 150 tokens long.
pad_to_max_tokens - Defaults to False, if True, the output feature axis will be padded to max_tokens even if the number of unique tokens in the vocabulary is less than max_tokens. Only valid in certain modes, see docs for more.
import tensorflow as tf
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
# Note: in TensorFlow 2.6+, you no longer need "layers.experimental.preprocessing"
# you can use: "tf.keras.layers.TextVectorization", see https://github.com/tensorflow/tensorflow/releases/tag/v2.6.0 for more
# Use the default TextVectorization variables
text_vectorizer = TextVectorization(max_tokens=None, # how many words in the vocabulary (all of the different words in your text)
standardize="lower_and_strip_punctuation", # how to process text
split="whitespace", # how to split tokens
ngrams=None, # create groups of n-words?
output_mode="int", # how to map tokens to numbers
output_sequence_length=None) # how long should the output sequence of tokens be?
# pad_to_max_tokens=True) # Not valid if using max_tokens=NoneWe've initialized a TextVectorization object with the default settings but let's customize it a little bit for our own use case. In particular, let's set values for max_tokens and output_sequence_length.
For max_tokens (the number of words in the vocabulary), multiples of 10,000 (10,000, 20,000, 30,000) or the exact number of unique words in your text (e.g. 32,179) are common values. For our use case, we'll use 10,000. And for the output_sequence_length we'll use the average number of tokens per Tweet in the training set. But first, we'll need to find it.
And for the output_sequence_length we'll use the average number of tokens per Tweet in the training set. But first, we'll need to find it.
# Find average number of tokens (words) in training Tweets
round(sum([len(i.split()) for i in train_sentences])/len(train_sentences))15
Now let's create another TextVectorization object using our custom parameters.
# Setup text vectorization with custom variables
max_vocab_length = 10000 # max number of words to have in our vocabulary
max_length = 15 # max length our sequences will be (e.g. how many words from a Tweet does our model see?)
text_vectorizer = TextVectorization(max_tokens=max_vocab_length,
output_mode="int",
output_sequence_length=max_length)To map our TextVectorization instance text_vectorizer to our data, we can call the adapt() method on it whilst passing it our training text.
# Fit the text vectorizer to the training text
text_vectorizer.adapt(train_sentences)it seems we've got a way to turn our text into numbers (in this case, word-level tokenization). Notice the 0's at the end of the returned tensor, this is because we set output_sequence_length=15, meaning no matter the size of the sequence we pass to text_vectorizer, it always returns a sequence with a length of 15.
How about we try our text_vectorizer on a few random sentences?
# Choose a random sentence from the training dataset and tokenize it
random_sentence = random.choice(train_sentences)
print(f"Original text:\n{random_sentence}\
\n\nVectorized version:")
text_vectorizer([random_sentence])
Finally, we can check the unique tokens in our vocabulary using the get_vocabulary() method.
# Get the unique words in the vocabulary
words_in_vocab = text_vectorizer.get_vocabulary()
top_5_words = words_in_vocab[:5] # most common tokens (notice the [UNK] token for "unknown" words)
bottom_5_words = words_in_vocab[-5:] # least common tokens
print(f"Number of words in vocab: {len(words_in_vocab)}")
print(f"Top 5 most common words: {top_5_words}")
print(f"Bottom 5 least common words: {bottom_5_words}")
Creating an Embedding using an Embedding Layer
We've got a way to map our text to numbers. How about we go a step further and turn those numbers into an embedding?
The powerful thing about embedding is it can be learned during training. This means rather than just being static (e.g. 1 = I, 2 = love, 3 = TensorFlow), a word's numeric representation can be improved as a model goes through data samples.
We can see what an embedding of a word looks like by using the tf.keras.layers.Embedding layer.
The main parameters we're concerned about here are:
input_dim - The size of the vocabulary (e.g. len(text_vectorizer.get_vocabulary()).
output_dim - The size of the output embedding vector, for example, a value of 100 outputs a feature vector of size 100 for each word.
embeddings_initializer - How to initialize the embeddings matrix, default is "uniform" which randomly initialized embedding matrix with uniform distribution. This can be changed by using pre-learned embeddings.
input_length - Length of sequences being passed to embedding layer.
tf.random.set_seed(42)
from tensorflow.keras import layers
embedding = layers.Embedding(input_dim=max_vocab_length, # set input shape
output_dim=128, # set size of embedding vector
embeddings_initializer="uniform", # default, intialize randomly
input_length=max_length, # how long is each input
name="embedding_1")
embedding
Excellent, notice how embedding is a TensoFlow layer? This is important because we can use it as part of a model, meaning its parameters (word representations) can be updated and improved as the model learns.
How about we try it out on a sample sentence?
# Get a random sentence from training set
random_sentence = random.choice(train_sentences)
print(f"Original text:\n{random_sentence}\
\n\nEmbedded version:")
# Embed the random sentence (turn it into numerical representation)
sample_embed = embedding(text_vectorizer([random_sentence]))
sample_embed
Modeling a text dataset
Now that we've got a way to turn our text data into numbers, we can start to build machine learning models to model it. To get plenty of practice, we're going to build a series of different models, each as its own experiment. We'll then compare the results of each model and see which one performed best.
More specifically, we'll be building the following:
Model 0: Naive Bayes (baseline)
Model 1: Feed-forward neural network (dense model)
Model 2: LSTM model
Model 3: GRU model
Model 4: Bidirectional-LSTM model
Model 5: 1D Convolutional Neural Network
Model 6: TensorFlow Hub Pretrained Feature Extractor
Model 7: Same as model 6 with 10% of training data
Model 0 is the simplest to acquire a baseline which we'll expect each other of the other deeper models to beat.
Each experiment will go through the following steps:
Construct the model
Train the model
Make predictions with the model
Track prediction evaluation metrics for later comparison
Getting a baseline - Model 0
As with all machine learning modeling experiments, it's important to create a baseline model so you've got a benchmark for future experiments to build upon.
To create our baseline, we'll create a Scikit-Learn Pipeline using the TF-IDF (term frequency-inverse document frequency) formula to convert our words to numbers and then model them with the Multinomial Naive Bayes algorithm. This was chosen via referring to the Scikit-Learn machine learning map.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
# Create tokenization and modelling pipeline
model_0 = Pipeline([
("tfidf", TfidfVectorizer()), # convert words to numbers using tfidf
("clf", MultinomialNB()) # model the text
])
# Fit the pipeline to the training data
model_0.fit(train_sentences, train_labels)
The benefit of using a shallow model like Multinomial Naive Bayes is that training is very fast.Let's evaluate our model and find our baseline metric.
baseline_score = model_0.score(val_sentences, val_labels)
print(f"Our baseline model achieves an accuracy of: {baseline_score*100:.2f}%")
How about we make some predictions with our baseline model?
# Make predictions
baseline_preds = model_0.predict(val_sentences)
baseline_preds[:20]
Creating an evaluation function for our model experiments
We could evaluate these as they are but since we're going to be evaluating several models, in the same way, going forward, let's create a helper function that takes an array of predictions and ground truth labels and computes the following:
Accuracy
Precision
Recall
F1-score
# Function to evaluate: accuracy, precision, recall, f1-score
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
def calculate_results(y_true, y_pred):
"""
Calculates model accuracy, precision, recall and f1 score of a binary classification model.
Args:
-----
y_true = true labels in the form of a 1D array
y_pred = predicted labels in the form of a 1D array
Returns a dictionary of accuracy, precision, recall, f1-score.
"""
# Calculate model accuracy
model_accuracy = accuracy_score(y_true, y_pred) * 100
# Calculate model precision, recall and f1 score using "weighted" average
model_precision, model_recall, model_f1, _ = precision_recall_fscore_support(y_true, y_pred, average="weighted")
model_results = {"accuracy": model_accuracy,
"precision": model_precision,
"recall": model_recall,
"f1": model_f1}
return model_results# Get baseline results
baseline_results = calculate_results(y_true=val_labels,
y_pred=baseline_preds)
baseline_results
A simple dense model - Model 1
It'll take our text and labels as input, tokenize the text, create an embedding, find the average of the embedding (using Global Average Pooling) and then pass the average through a fully connected layer with one output unit and a sigmoid activation function.
We have already created create_tensorboard_callback() function in our previous blogs, we would reuse those functions to keep track of the results of each. Here is the function
import tensorflow as tf
import datetime
import matplotlib.pyplot as plt
# Create directory to save TensorBoard logs
SAVE_DIR = "model_logs"
def create_tensorboard_callback(dir_name, experiment_name):
"""
Creates a TensorBoard callback instand to store log files.
Stores log files with the filepath:
"dir_name/experiment_name/current_datetime/"
Args:
dir_name: target directory to store TensorBoard log files
experiment_name: name of experiment directory (e.g. efficientnet_model_1)
"""
log_dir = dir_name + "/" + experiment_name + "/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(
log_dir=log_dir
)
print(f"Saving TensorBoard log files to: {log_dir}")
return tensorboard_callbackNow we've got a TensorBoard callback function ready to go, let's build our first deep model.
# Build model with the Functional API
from tensorflow.keras import layers
inputs = layers.Input(shape=(1,), dtype="string") # inputs are 1-dimensional strings
x = text_vectorizer(inputs) # turn the input text into numbers
x = embedding(x) # create an embedding of the numerized numbers
x = layers.GlobalAveragePooling1D()(x) # lower the dimensionality of the embedding (try running the model without this layer and see what happens)
outputs = layers.Dense(1, activation="sigmoid")(x) # create the output layer, want binary outputs so use sigmoid activation
model_1 = tf.keras.Model(inputs, outputs, name="model_1_dense") # construct the model
# Compile model
model_1.compile(loss="binary_crossentropy",
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])Looking good. Our model takes a 1-dimensional string as input (in our case, a Tweet), it then tokenizes the string using text_vectorizer and creates an embedding using embedding.
Finally, we pass the output of the pooling layer to a dense layer with sigmoid activation (we use sigmoid since our problem is binary classification).
our model is compiled, let's fit it to our training data for 5 epochs. We'll also pass our TensorBoard callback function to make sure our model's training metrics are logged.
# Fit the model
model_1_history = model_1.fit(train_sentences, # input sentences can be a list of strings due to text preprocessing layer built-in model
train_labels,
epochs=5,
validation_data=(val_sentences, val_labels),
callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR,
experiment_name="simple_dense_model")])
Let's check our model's performance on the validation set.
# Check the results
model_1.evaluate(val_sentences, val_labels)
And since we tracked our model's training logs with TensorBoard, how about we visualize them? We can do so by uploading our TensorBoard log files (contained in the model_logs directory) to TensorBoard.dev.
We've built and trained our first deep model, the next step is to make some predictions with it.
# Make predictions (these come back in the form of probabilities)
model_1_pred_probs = model_1.predict(val_sentences)
# Turn prediction probabilities into single-dimension tensor of floats
model_1_preds = tf.squeeze(tf.round(model_1_pred_probs)) # squeeze removes single dimensions
model_1_preds[:20]
Now we've got our model's predictions in the form of classes, we can use our calculate_results() function to compare them to the ground truth validation labels.
# Calculate model_1 metrics
model_1_results = calculate_results(y_true=val_labels,
y_pred=model_1_preds)
model_1_results
Since we'll be doing this kind of comparison (baseline compared to the new model) quite a few times, let's create a function to help us out.
# Create a helper function to compare our baseline results to new model results
def compare_baseline_to_new_results(baseline_results, new_model_results):
for key, value in baseline_results.items():
print(f"Baseline {key}: {value:.2f}, New {key}: {new_model_results[key]:.2f}, Difference: {new_model_results[key]-value:.2f}")
compare_baseline_to_new_results(baseline_results=baseline_results,
new_model_results=model_1_results)
Recurrent Neural Networks (RNN's)
For our next series of modeling experiments, we're going to be using a special kind of neural network called a Recurrent Neural Network (RNN).
The premise of an RNN is simple: use information from the past to help you with the future (this is where the term recurrent comes from). In other words, take an input (X) and compute an output (y) based on all previous inputs.
This concept is especially helpful when dealing with sequences such as passages of natural language text (such as our Tweets).
For example, when you read this sentence, you take into context the previous words when deciphering the meaning of the current word dog.
See what happened there?
I put the word "dog" at the end which is a valid word but it doesn't make sense in the context of the rest of the sentence.
When an RNN looks at a sequence of text (already in numerical form), the patterns it learns are continually updated based on the order of the sequence.
For a simple example, take two sentences:
The massive earthquake last week, no?
No massive earthquake last week.
Both contain exactly the same words but have different meanings. The order of the words determines the meaning (one could argue punctuation marks also dictate the meaning but for simplicity's sake, let's stay focused on the words).
Recurrent neural networks can be used for a number of sequence-based problems:
One to one: one input, one output, such as image classification.
One to many: one input, many outputs, such as image captioning (image input, a sequence of text as caption output).
Many to one: many inputs, one output, such as text classification (classifying a Tweet as real diaster or not real disaster).
Many to many: many inputs, many outputs, such as machine translation (translating English to Spanish) or speech to text (audio wave as input, text as output).
When you come across RNN's in the wild, you'll most likely come across variants of the following:
Long short-term memory cells (LSTMs).
Gated recurrent units (GRUs).
Bidirectional RNN's (passes forward and backward along a sequence, left to right and right to left).
LSTM - Model 2
With all this talk about what RNN's are and what they're good for, I'm sure you're eager to build one. We're going to start with an LSTM-powered RNN.
We're not getting reusing trained embeddings (this would involve data leakage between models, leading to an uneven comparison later on), we'll create another embedding layer (model_2_embedding) for our model. The text_vectorizer layer can be reused since it doesn't get updated during training.
# Set random seed and create embedding layer (new embedding layer for each model)
tf.random.set_seed(42)
from tensorflow.keras import layers
model_2_embedding = layers.Embedding(input_dim=max_vocab_length,
output_dim=128,
embeddings_initializer="uniform",
input_length=max_length,
name="embedding_2")
# Create LSTM model
inputs = layers.Input(shape=(1,), dtype="string")
x = text_vectorizer(inputs)
x = model_2_embedding(x)
print(x.shape)
# x = layers.LSTM(64, return_sequences=True)(x) # return vector for each word in the Tweet (you can stack RNN cells as long as return_sequences=True)
x = layers.LSTM(64)(x) # return vector for whole sequence
print(x.shape)
# x = layers.Dense(64, activation="relu")(x) # optional dense layer on top of output of LSTM cell
outputs = layers.Dense(1, activation="sigmoid")(x)
model_2 = tf.keras.Model(inputs, outputs, name="model_2_LSTM")
# Compile model
model_2.compile(loss="binary_crossentropy",
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])
Now our first RNN model's compiled let's fit it to our training data, validating it on the validation data and tracking its training parameters using our TensorBoard callback. Let's fit the model.
# Fit model
model_2_history = model_2.fit(train_sentences,
train_labels,
epochs=5,
validation_data=(val_sentences, val_labels),
callbacks=[create_tensorboard_callback(SAVE_DIR,
"LSTM")])
The same thing will happen as before, due to the sigmoid activation function in the final layer, when we call the predict() method on our model, it'll return prediction probabilities rather than classes.
# Make predictions on the validation dataset
model_2_pred_probs = model_2.predict(val_sentences)
# Round out predictions and reduce to 1-dimensional array
model_2_preds = tf.squeeze(tf.round(model_2_pred_probs))
model_2_preds[:10]
now let's use our caculate_results() function to evaluate our LSTM model and our compare_baseline_to_new_results() function to compare it to our baseline model.
# Calculate LSTM model results
model_2_results = calculate_results(y_true=val_labels,
y_pred=model_2_preds)
model_2_results
# Compare model 2 to baseline
compare_baseline_to_new_results(baseline_results, model_2_results)
GRU - Model 3
Another popular and effective RNN component is the GRU or gated recurrent unit.
The GRU cell has similar features to an LSTM cell but has fewer parameters.
To use the GRU cell in TensorFlow, we can call the tensorflow.keras.layers.GRU() class.
# Set random seed and create embedding layer (new embedding layer for each model)
tf.random.set_seed(42)
from tensorflow.keras import layers
model_3_embedding = layers.Embedding(input_dim=max_vocab_length,
output_dim=128,
embeddings_initializer="uniform",
input_length=max_length,
name="embedding_3")
# Build an RNN using the GRU cell
inputs = layers.Input(shape=(1,), dtype="string")
x = text_vectorizer(inputs)
x = model_3_embedding(x)
# x = layers.GRU(64, return_sequences=True) # stacking recurrent cells requires return_sequences=True
x = layers.GRU(64)(x)
# x = layers.Dense(64, activation="relu")(x) # optional dense layer after GRU cell
outputs = layers.Dense(1, activation="sigmoid")(x)
model_3 = tf.keras.Model(inputs, outputs, name="model_3_GRU")
# Compile GRU model
model_3.compile(loss="binary_crossentropy",
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])We'll fit our model just as we've been doing previously. We'll also track our models results using our create_tensorboard_callback() function.
# Fit model
model_3_history = model_3.fit(train_sentences,
train_labels,
epochs=5,
validation_data=(val_sentences, val_labels),
callbacks=[create_tensorboard_callback(SAVE_DIR, "GRU")])
Time to make some predictions on the validation samples.
# Make predictions on the validation data
model_3_pred_probs = model_3.predict(val_sentences)
# Convert prediction probabilities to prediction classes
model_3_preds = tf.squeeze(tf.round(model_3_pred_probs))
model_3_preds[:10]
Now we've got predicted classes, let's evaluate them against the ground truth labels.
# Calcuate model_3 results
model_3_results = calculate_results(y_true=val_labels,
y_pred=model_3_preds)
model_3_results
Finally, we can compare our GRU model's results to our baseline.
# Compare to baseline
compare_baseline_to_new_results(baseline_results, model_3_results)
Bidirectonal RNN model - Model 4
We've already built two RNN's with GRU and LSTM cells. Now we're going to look into another kind of RNN, the bidirectional RNN.
A standard RNN will process a sequence from left to right, whereas a bidirectional RNN will process the sequence from left to right and then again from right to left. Intuitively, this can be thought of as if you were reading a sentence for the first time in the normal fashion (left to right) but for some reason, it didn't make sense so you traverse back through the words and go back over them again (right to left). In practice, many sequence models often see an improvement in performance when using bidirectional RNN's
Let's build a bidirectional RNN. TensorFlow helps us out by providing the tensorflow.keras.layers.Bidirectional class. We can use the Bidirectional class to wrap our existing RNNs, instantly making them bidirectional.
# Set random seed and create embedding layer (new embedding layer for each model)
tf.random.set_seed(42)
from tensorflow.keras import layers
model_4_embedding = layers.Embedding(input_dim=max_vocab_length,
output_dim=128,
embeddings_initializer="uniform",
input_length=max_length,
name="embedding_4")
# Build a Bidirectional RNN in TensorFlow
inputs = layers.Input(shape=(1,), dtype="string")
x = text_vectorizer(inputs)
x = model_4_embedding(x)
# x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(x) # stacking RNN layers requires return_sequences=True
x = layers.Bidirectional(layers.LSTM(64))(x) # bidirectional goes both ways so has double the parameters of a regular LSTM layer
outputs = layers.Dense(1, activation="sigmoid")(x)
model_4 = tf.keras.Model(inputs, outputs, name="model_4_Bidirectional")
# Compile
model_4.compile(loss="binary_crossentropy",
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])Notice the increased number of trainable parameters in model_4 (bidirectional LSTM) compared to model_2 (regular LSTM). This is due to the bidirectionality we added to our RNN. Time to fit our bidirectional model and track its performance.
# Fit the model (takes longer because of the bidirectional layers)
model_4_history = model_4.fit(train_sentences,
train_labels,
epochs=5,
validation_data=(val_sentences, val_labels),
callbacks=[create_tensorboard_callback(SAVE_DIR, "bidirectional_RNN")])
Let's make some predictions about it.
# Make predictions with bidirectional RNN on the validation data
model_4_pred_probs = model_4.predict(val_sentences)
# Convert prediction probabilities to labels
model_4_preds = tf.squeeze(tf.round(model_4_pred_probs))
model_4_preds[:10]
Calculate bidirectional RNN model results
model_4_results = calculate_results(val_labels, model_4_preds)
model_4_results
# Check to see how the bidirectional model performs against the baseline
compare_baseline_to_new_results(baseline_results, model_4_results)
Convolutional Neural Networks for Text
You might've used convolutional neural networks (CNNs) for images before but they can also be used for sequences. The main difference between using CNNs for images and sequences in the shape of the data. Images come in 2-dimensions (height x width) whereas sequences are often 1-dimensional (a string of text). So to use CNNs with sequences, we use a 1-dimensional convolution instead of a 2-dimensional convolution. A typical CNN architecture for sequences will look like the following:
Inputs (text) -> Tokenization -> Embedding -> Layers -> Outputs (class probabilities)The difference again is in the layers component. Instead of using an LSTM or GRU cell, we're going to use a tensorflow.keras.layers.Conv1D() layer followed by a tensorflow.keras.layers.GlobablMaxPool1D() layer.
Conv1D - Model 5
Before we build a full 1-dimensional CNN model, let's see a 1-dimensional convolutional layer (also called a temporal convolution) in action. We'll first create an embedding of a sample of text and experiment by passing it through a Conv1D() layer and GlobalMaxPool1D() layer.
# Test out the embedding, 1D convolutional and max pooling
embedding_test = embedding(text_vectorizer(["this is a test sentence"])) # turn target sentence into embedding
conv_1d = layers.Conv1D(filters=32, kernel_size=5, activation="relu") # convolve over target sequence 5 words at a time
conv_1d_output = conv_1d(embedding_test) # pass embedding through 1D convolutional layer
max_pool = layers.GlobalMaxPool1D()
max_pool_output = max_pool(conv_1d_output) # get the most important features
embedding_test.shape, conv_1d_output.shape, max_pool_output.shape
Notice the output shapes of each layer. The embedding has an output shape dimension of the parameters we set it to (input_length=15 and output_dim=128). The 1-dimensional convolutional layer has an output that has been compressed in line with its parameters. And the same goes for the max-pooling layer output.
Our text starts out as a string but gets converted to a feature vector of length 64 through various transformation steps (from tokenization to embedding to 1-dimensional convolution to the max pool).
We've seen the outputs of several components of a CNN for sequences, let's put them together and construct a full model, compile it (just as we've done with our other models)
# Set random seed and create embedding layer (new embedding layer for each model)
tf.random.set_seed(42)
from tensorflow.keras import layers
model_5_embedding = layers.Embedding(input_dim=max_vocab_length,
output_dim=128,
embeddings_initializer="uniform",
input_length=max_length,
name="embedding_5")
# Create 1-dimensional convolutional layer to model sequences
from tensorflow.keras import layers
inputs = layers.Input(shape=(1,), dtype="string")
x = text_vectorizer(inputs)
x = model_5_embedding(x)
x = layers.Conv1D(filters=32, kernel_size=5, activation="relu")(x)
x = layers.GlobalMaxPool1D()(x)
# x = layers.Dense(64, activation="relu")(x) # optional dense layer
outputs = layers.Dense(1, activation="sigmoid")(x)
model_5 = tf.keras.Model(inputs, outputs, name="model_5_Conv1D")
# Compile Conv1D model
model_5.compile(loss="binary_crossentropy",
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])Let's fit our 1D CNN model to our text data. In line with previous experiments, we'll save its results using our create_tensorboard_callback() function.
# Fit the model
model_5_history = model_5.fit(train_sentences,
train_labels,
epochs=5,
validation_data=(val_sentences, val_labels),
callbacks=[create_tensorboard_callback(SAVE_DIR,
"Conv1D")])
Let's make some predictions with it and evaluate them just as before.
# Make predictions with model_5
model_5_pred_probs = model_5.predict(val_sentences)
# Convert model_5 prediction probabilities to labels
model_5_preds = tf.squeeze(tf.round(model_5_pred_probs))
model_5_preds[:10]
# Calculate model_5 evaluation metrics
model_5_results = calculate_results(y_true=val_labels,
y_pred=model_5_preds)
model_5_results
# Compare model_5 results to baseline
compare_baseline_to_new_results(baseline_results, model_5_results)
Using Pretrained Embeddings (transfer learning for NLP)
For all of the previous deep learning models we've built and trained, we've created and used our own embeddings from scratch each time.
However, a common practice is to leverage pre-trained embeddings through transfer learning. This is one of the main benefits of using deep models: being able to take what one (often larger) model has learned (often on a large amount of data) and adjust it for our own use case. For our next model, instead of using our own embedding layer, we're going to replace it with a pre-trained embedding layer. More specifically, we're going to be using the Universal Sentence Encoder from TensorFlow Hub (a great resource containing a plethora of pre-trained model resources for a variety of tasks).
TensorFlow Hub Pretrained Sentence Encoder - Model 6
The main difference between the embedding layer we created and the Universal Sentence Encoder is that rather than create a word-level embedding, the Universal Sentence Encoder, as you might've guessed, creates a whole sentence-level embedding.
Our embedding layer also outputs a 128-dimensional vector for each word, whereas, the Universal Sentence Encoder outputs a 512-dimensional vector for each sentence.
As usual, this is best demonstrated with an example. We can load in a TensorFlow Hub module using the hub.load() method and pass it the target URL of the module we'd like to use, in our case, it's "https://tfhub.dev/google/universal-sentence-encoder/4".
Let's load the Universal Sentence Encoder model and test it on a couple of sentences.
# Example of pretrained embedding with universal sentence encoder - https://tfhub.dev/google/universal-sentence-encoder/4
import tensorflow_hub as hub
embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder/4") # load Universal Sentence EncoderPassing our sentences to the Universal Sentence Encoder (USE) encodes them from strings to 512-dimensional vectors, which make no sense to us but hopefully make sense to our machine learning models. Speaking of models, let's build one with the USE as our embedding layer. We can convert the TensorFlow Hub USE module into a Keras layer using the hub.KerasLayer class.
# We can use this encoding layer in place of our text_vectorizer and embedding layer
sentence_encoder_layer = hub.KerasLayer("https://tfhub.dev/google/universal-sentence-encoder/4",
input_shape=[], # shape of inputs coming to our model
dtype=tf.string, # data type of inputs coming to the USE layer
trainable=False, # keep the pretrained weights (we'll create a feature extractor)
name="USE") Now we've got the USE as a Keras layer, we can use it in a Keras Sequential model.
# Create model using the Sequential API
model_6 = tf.keras.Sequential([
sentence_encoder_layer, # take in sentences and then encode them into an embedding
layers.Dense(64, activation="relu"),
layers.Dense(1, activation="sigmoid")
], name="model_6_USE")
# Compile model
model_6.compile(loss="binary_crossentropy",
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])Now we've got a feature extractor model ready, let's train it and track its results to TensorBoard using our create_tensorboard_callback() function.
# Train a classifier on top of pretrained embeddings
model_6_history = model_6.fit(train_sentences,
train_labels,
epochs=5,
validation_data=(val_sentences, val_labels),
callbacks=[create_tensorboard_callback(SAVE_DIR,
"tf_hub_sentence_encoder")])
Let's make some predictions with it and evaluate them as we've done with our other models.
# Make predictions with USE TF Hub model
model_6_pred_probs = model_6.predict(val_sentences)
# Convert prediction probabilities to labels
model_6_preds = tf.squeeze(tf.round(model_6_pred_probs))
model_6_preds[:10]
# Calculate model 6 performance metrics
model_6_results = calculate_results(val_labels, model_6_preds)
model_6_results
# Compare TF Hub model to baseline
compare_baseline_to_new_results(baseline_results, model_6_results)
TensorFlow Hub Pretrained Sentence Encoder 10% of the training data - Model 7
One of the benefits of using transfer learning methods, such as the pre-trained embeddings within the USE is the ability to get great results on a small amount of data (the USE paper even mentions this in the abstract).To put this to the test, we're going to make a small subset of the training data (10%), train a model, and evaluate it.
from sklearn.model_selection import train_test_split
import numpy as np
# Create subsets of 10% of the training data
train_10_percent = train_df_shuffled[["text", "target"]].sample(frac=0.1, random_state=42)
train_sentences_10_percent = train_10_percent["text"].to_list()
train_labels_10_percent = train_10_percent["target"].to_list()
len(train_sentences_10_percent), len(train_labels_10_percent)
# One kind of correct way (there are more) to make data subset
# (split the already split train_sentences/train_labels)
train_sentences_90_percent, train_sentences_10_percent, train_labels_90_percent, train_labels_10_percent = train_test_split(np.array(train_sentences),
train_labels,
test_size=0.1,
random_state=42)# Check length of 10 percent datasets
print(f"Total training examples: {len(train_sentences)}")
print(f"Length of 10% training examples: {len(train_sentences_10_percent)}")
To make sure we're making an appropriate comparison between our model's ability to learn from the full training set and the 10% subset, we'll clone our USE model (model_6) using the tf.keras.models.clone_model() method. Doing this will create the same architecture but reset the learned weights of the clone target (pre-trained weights from the USE will remain but all others will be reset).
# Clone model_6 but reset weights
model_7 = tf.keras.models.clone_model(model_6)
# Compile model
model_7.compile(loss="binary_crossentropy",
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])Now let's train the newly created model on our 10% training data subset.
# Fit the model to 10% of the training data
model_7_history = model_7.fit(x=train_sentences_10_percent,
y=train_labels_10_percent,
epochs=5,
validation_data=(val_sentences, val_labels),
callbacks=[create_tensorboard_callback(SAVE_DIR, "10_percent_tf_hub_sentence_encoder")])
Due to the smaller amount of training data, training happens even quicker than before.
Let's evaluate our model's performance after learning on 10% of the training data.
# Make predictions with the model trained on 10% of the data
model_7_pred_probs = model_7.predict(val_sentences)
# Convert prediction probabilities to labels
model_7_preds = tf.squeeze(tf.round(model_7_pred_probs))
model_7_preds[:10]
# Calculate model results
model_7_results = calculate_results(val_labels, model_7_preds)
model_7_results
# Compare to baseline
compare_baseline_to_new_results(baseline_results, model_7_results)
Comparing the performance of each of our models
We've come a long way! From training a baseline to several deep models.
Now it's time to compare our model's results. But just before we do, it's worthwhile mentioning, this type of practice is a standard deep learning workflow. Training various different models, then comparing them to see which one performed best and continuing to train it if necessary. The important thing to note is that for all of our modeling experiments we used the same training data (except for model_7 where we used 10% of the training data). To visualize our model's performances, let's create a pandas DataFrame results dictionary and then plot it.
# Combine model results into a DataFrame
all_model_results = pd.DataFrame({"baseline": baseline_results,
"simple_dense": model_1_results,
"lstm": model_2_results,
"gru": model_3_results,
"bidirectional": model_4_results,
"conv1d": model_5_results,
"tf_hub_sentence_encoder": model_6_results,
"tf_hub_10_percent_data": model_7_results})
all_model_results = all_model_results.transpose()
all_model_results
# Plot and compare all of the model results
all_model_results.plot(kind="bar", figsize=(10, 7)).legend(bbox_to_anchor=(1.0, 1.0));
Looks like our pre-trained USE TensorFlow Hub models have the best performance, even the one with only 10% of the training data seems to outperform the other models. This goes to show the power of transfer learning. How about we drill down and get the F1 scores of each model?
# Sort model results by f1-score
all_model_results.sort_values("f1", ascending=False)["f1"].plot(kind="bar", figsize=(10, 7));
Combining our models (model ensembling/stacking)
Many production systems use an ensemble (multiple different models combined) of models to make a prediction.
The idea behind model stacking is that if several uncorrelated models agree on a prediction, then the prediction must be more robust than a prediction made by a singular model.
The keyword in the sentence above is uncorrelated, which is another way of saying, different types of models. For example, in our case, we might combine our baseline, our bidirectional model, and our TensorFlow Hub USE model.
Although these models are all trained on the same data, they all have a different way of finding patterns.
If we were to use three similarly trained models, such as three LSTM models, the predictions they output will likely be very similar.
Think of it as trying to decide where to eat with your friends. If you all have similar tastes, you'll probably all pick the same restaurant. But if you've all got different tastes and still end up picking the same restaurant, the restaurant must be good.
Since we're working with a classification problem, there are a few of ways we can combine our models:
Averaging - Take the output prediction probabilities of each model for each sample, combine them and then average them.
Majority vote (mode) - Make class predictions with each of your models on all samples, the predicted class is the one in the majority. For example, if three different models predict [1, 0, 1] respectively, the majority class is 1, therefore, that would be the predicted label.
Model stacking - Take the outputs of each of your chosen models and use them as inputs to another model.
We're going to combine our baseline model (model_0), LSTM model (model_2) and our USE model trained on the full training data (model_6) by averaging the combined prediction probabilities of each.
# Get mean pred probs for 3 models
baseline_pred_probs = np.max(model_0.predict_proba(val_sentences), axis=1) # get the prediction probabilities from baseline model
combined_pred_probs = baseline_pred_probs + tf.squeeze(model_2_pred_probs, axis=1) + tf.squeeze(model_6_pred_probs)
combined_preds = tf.round(combined_pred_probs/3) # average and round the prediction probabilities to get prediction classes
combined_preds[:20]
We've got a combined predictions array of different classes, let's evaluate them against the true labels and add our stacked model's results to our all_model_results DataFrame.
# Calculate results from averaging the prediction probabilities
ensemble_results = calculate_results(val_labels, combined_preds)
ensemble_results
# Add our combined model's results to the results DataFrame
all_model_results.loc["ensemble_results"] = ensemble_results
# Convert the accuracy to the same scale as the rest of the results
all_model_results.loc["ensemble_results"]["accuracy"] = all_model_results.loc["ensemble_results"]["accuracy"]/100
all_model_results
It seems many of our model's results are similar. This may mean there are some limitations to what can be learned from our data. When many of your modeling experiments return similar results, it's a good idea to revisit your data.
Making predictions on the test dataset
We've seen how our models perform on the validation set.
But how about the test dataset? We don't have labels for the test dataset so we're going to have to make some predictions and inspect them for ourselves. Let's write some code to make predictions on random samples from the test dataset and visualize them.
# Making predictions on the test dataset
test_sentences = test_df["text"].to_list()
test_samples = random.sample(test_sentences, 10)
for test_sample in test_samples:
pred_prob = tf.squeeze(model_6.predict([test_sample])) # has to be list
pred = tf.round(pred_prob)
print(f"Pred: {int(pred)}, Prob: {pred_prob}")
print(f"Text:\n{test_sample}\n")
print("----\n")
------
Predicting on Tweets from the wild
How about we find some Tweets and use our model to predict whether or not they're about a disaster or not?
def predict_on_sentence(model, sentence):
"""
Uses model to make a prediction on sentence.
Returns the sentence, the predicted label and the prediction probability.
"""
pred_prob = model.predict([sentence])
pred_label = tf.squeeze(tf.round(pred_prob)).numpy()
print(f"Pred: {pred_label}", "(real disaster)" if pred_label > 0 else "(not real disaster)", f"Prob: {pred_prob[0][0]}")
print(f"Text:\n{sentence}")How about we find a few Tweets about actual disasters? Such as the following two Tweets about the 2020 Beirut explosions.
# Source - https://twitter.com/BeirutCityGuide/status/1290696551376007168
beirut_tweet_1 = "Reports that the smoke in Beirut sky contains nitric acid, which is toxic. Please share and refrain from stepping outside unless urgent. #Lebanon"
# Source - https://twitter.com/BeirutCityGuide/status/1290773498743476224
beirut_tweet_2 = "#Beirut declared a “devastated city”, two-week state of emergency officially declared. #Lebanon"
# Predict on diaster Tweet 1
predict_on_sentence(model=model_6,
sentence=beirut_tweet_1)
# Predict on diaster Tweet 2
predict_on_sentence(model=model_6,
sentence=beirut_tweet_2)
Nice !!!! Looks like our model is performing as expected, predicting both of the disaster Tweets as actual diasters.
Comments