Classification Intro
Types of Classification
Binary Classification
is this A or B?, SPAM or NOT SPAM?
Multi-Class Classification
Is this pizza, steak, or sushi? Is this a 1 or a 9?
Multi-Label Classification
What tags/labels should this article get?
Topics To Consider With Classification Models
- the architecture of a neural network
- Input & Output shapes
- photos of food to identified foods
- Steps in modeling (creating, compiling, fitting, and evaluationg)
- Evaluation methods
- saving + loading a model
Classification Example: Photo-to-food
The Overview
- photos are (square) pixels
- photos have dimensions, say 256x256
- photos have pixels with colors, say rgb strings
- the pixels can be broken down into numbers based on img pixel coordinates
- the pixel numbers can be converted to tensors
- the tensors can get passed to models....etc
The Inputs & Outputs
Pixels get converted to tensors & passed to machine learning models.
Inputs could look like...
[batch_size, img_width, img_height, colour_channels]Outputs could look like...[%likeSushi, %likeSteak, %likePotatos, ...etc]
Architecture of A Classification model
A Classification model could...
- be a sequential model
- contain several layers
- an Input layer with a defined shape
- an activiation-specified set of "inner" layers
- a "softmax" final layer
Classification Model Details & Hyperparameters
- Input Layer Shape: same as the number of features
- Hidden Layers: Problem-specific (minimum 1 & maximum unlimited...)
- Neuros-Per-Hidden-Layer: Problem-specific (generally 10-100)
- Output layer shape: 1-per-classification
- Hidden Activation: usually ReLu (rectified linear unit)
- Output Activation: usually softmax for multi-class or sigmoid for binary class
- Loss Function: Cross Entropy (categorical or binary)
- Optimizer: SGD or Adam
Classification Evaluation Methods
(tp: true-positive, tn: true-negative, fp: false-positive, fn: false-negative)
Accuracy
- (tp + tn) / (tp + tn + fp + fn)
tf.keras.metrics.Accuracy()sklearn.metrics.acuracy_score()- this is a default for classification
- NOT the best for imbalanced data (10_000 of A and 1_000 of B)
Precision
- tp / (tp + fp)
tf.keras.metrics.Precision()sklearn.metrics.precision_score()- HIGHER precision means LESS fp
Recall
- tp / (tp + fn)
tf.keras.metrics.Recall()sklearn.metrics.precision_score()- HIGHER recall means LESS fn
Precision-to-Recall Tradeoff
Can't have both be high.
Usually improving one decreases the other.
F1-Score
- (precision * recall) / (recision + recall)
sklearn.metrics.f1_score()- a good "overall metric" for a classification model
A Mock Classification Example
Make Some Data
In [34]:
import tensorflow as tf
from sklearn.datasets import make_circles
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
# make_circles:
# A simple toy dataset to visualize clustering and classification algorithms.
# returns X as array of numbers
# returns y as binary classification options
# how much data
sampleCount = 1000
X, y = make_circles(sampleCount, noise=0.03, random_state=42)
# convert to pd datafram
circles = pd.DataFrame({"X0":X[:, 0], "X1":X[:, 1], "label":y})
circles.head()Out [34]:
Visualize the data
In [2]:
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu);
Inspect Data Shapes
In [3]:
print(f'data SHAPES --> X: {X.shape}, Y: {y.shape}')
print(f'data LENGTHS --> X: {len(X)}, Y: {len(y)}')In [4]:
X[0]Out [4]:
In [5]:
y[0]Out [5]:
Model: Creation, Compilation, & Fit
In [6]:
tf.random.set_seed(42)
l1 = tf.keras.layers.Dense(1)
m1EpochCount = 5
# Create
m1 = tf.keras.Sequential()
m1.add(l1)
# Compile
m1.compile(loss=tf.keras.losses.BinaryCrossentropy(), # binary since we are working with 2 clases (0 & 1)
optimizer=tf.keras.optimizers.SGD(),
metrics=['accuracy'])
# Fit
m1.fit(X, y, epochs=m1EpochCount)Out [6]:
Model: More Epochs
The previous run looks like the accuracy is ~50%. That's pretty bad.
In [7]:
moreEpochs = 200
m1.fit(X, y, epochs=moreEpochs, verbose=0)Out [7]:
In [8]:
m1.evaluate(X, y)Out [8]:
Model: Add Another Layer
Looks like more epochs did NOT make a meaningful impact.
In [9]:
midEpochs = 100
m2 = tf.keras.Sequential()
m2.add(l1)
m2.add(l1)
m2.compile(loss=tf.keras.losses.BinaryCrossentropy(),
optimizer=tf.keras.optimizers.SGD(),
metrics=['accuracy'])
m2.fit(X, y, epochs=midEpochs, verbose=0)Out [9]:
In [10]:
m2.evaluate(X, y)Out [10]:
Looks like MORE LAYERS didn't really help either: the accuracy is still ~50%.
Models: Ways To Update
- add layers
- increase the number of hidden units
- change the activation function
- change the optimization function
- change the learning rate
- fit more data
- fit for longer
Models: Add Layer Hidden Units
In [11]:
m3 = tf.keras.Sequential()
dense100 = tf.keras.layers.Dense(100)
dense10 = tf.keras.layers.Dense(10)
dense1 = tf.keras.layers.Dense(1)
m3.add(dense100)
m3.add(dense10)
m3.add(dense1)
m3.compile(loss=tf.keras.losses.BinaryCrossentropy(),
optimizer=tf.keras.optimizers.SGD(),
metrics=['accuracy'])
m3.fit(X,y,epochs=100, verbose=0)Out [11]:
In [12]:
m3.evaluate(X, y)Out [12]:
Still no meaningful updates :(
Model: Visualize Predictions
Create a reusable function that takes the model, the x & the y input and uses matplotlib to draw a chart
In [13]:
def plot_decision_boundary(model, X, y):
"""
This function has been adapted from two phenomenal resources:
1. CS231n - https://cs231n.github.io/neural-networks-case-study/
2. Made with ML basics - https://github.com/GokuMohandas/MadeWithML/blob/main/notebooks/08_Neural_Networks.ipynb
"""
#
# Define the axis boundaries of the plot and create a meshgrid
#
x_axis_min, x_axis_max = X[:, 0].min() - 0.1, X[:, 0].max() + 0.1
y_axis_min, y_axis_max = X[:, 1].min() - 0.1, X[:, 1].max() + 0.1
axis_points = np.linspace(x_axis_min, x_axis_max, 100)
xx, yy = np.meshgrid(axis_points,axis_points)
#
# Create X values (we're going to predict on all of these)
#
x_in = np.c_[xx.ravel(), yy.ravel()] # stack 2D arrays together: https://numpy.org/devdocs/reference/generated/numpy.c_.html
#
# Make predictions using the trained model
#
y_pred = model.predict(x_in)
#
# Check for multi-class
#
if model.output_shape[-1] > 1: # checks the final dimension of the model's output shape, if this is > (greater than) 1, it's multi-class
print("doing multiclass classification...")
# We have to reshape our predictions to get them ready for plotting
y_pred = np.argmax(y_pred, axis=1).reshape(xx.shape)
else:
print("doing binary classifcation...")
y_pred = np.round(np.max(y_pred, axis=1)).reshape(xx.shape)
#
# Plot decision boundary
#
plt.contourf(xx, yy, y_pred, cmap=plt.cm.RdYlBu, alpha=0.7)
plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.RdYlBu)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())In [14]:
plot_decision_boundary(m3, X, y)
Model: Several updates
- 2x dense layer with 4 neurons with relu activation
- output layer with 1 neuron, sigmoid activation
- learning-rate of .001
In [15]:
denseRelu4 = tf.keras.layers.Dense(4, activation="relu")
denseSigmoid1 = tf.keras.layers.Dense(1, activation="sigmoid")
tf.keras.layers.Dense(1, activation=tf.keras.activations.sigmoid)
# Create a model
m5 = tf.keras.Sequential()
m5.add(denseRelu4)
m5.add(denseRelu4)
m5.add(denseSigmoid1)
# Compile the model
m5.compile(loss=tf.keras.losses.binary_crossentropy,
optimizer=tf.keras.optimizers.Adam(),
metrics=['accuracy'])
# Fit the model
m5History = m5.fit(X, y, epochs=100, verbose=0)In [16]:
m5.evaluate(X,y)Out [16]:
In [17]:
plot_decision_boundary(m5, X, y)
Model: Change Optimizer Learning Rate
In [18]:
# testDataPercentage = .15 # how much of our data should we use for "testing"
# randomVal = 42
# feature_training_data, feature_testing_data, label_training_data, label_testing_data = train_test_split(X,
# y,
# test_size=testDataPercentage,
# random_state=randomVal) # set random state for reproducible splits
# Split data into train and test sets
feature_training_data, label_training_data = X[:800], y[:800] # 80% of the data for the training set
feature_testing_data, label_testing_data = X[800:], y[800:] # 20% of the data for the test setIn [19]:
m6 = tf.keras.Sequential()
m6.add(denseRelu4)
m6.add(denseRelu4)
m6.add(denseSigmoid1)
# Compile the model
m6.compile(loss=tf.keras.losses.binary_crossentropy,
optimizer=tf.keras.optimizers.Adam(learning_rate=0.01), # increase learning rate from 0.001 to 0.01 for faster learning
metrics=['accuracy'])
# Fit the model
m6History = m6.fit(feature_training_data, label_training_data, epochs=30, verbose=0)In [20]:
loss, accuracy = m6.evaluate(feature_testing_data, label_testing_data)
print(f"Model loss on the test set: {loss}")
print(f"Model accuracy on the test set: {100*accuracy:.2f}%")In [21]:
plt.figure(figsize=(12, 6))
plt.subplot(1, 2, 1)
plt.title("Train")
plot_decision_boundary(m6, X=feature_training_data, y=label_training_data)
plt.subplot(1, 2, 2)
plt.title("Test")
plot_decision_boundary(m6, X=feature_testing_data, y=label_testing_data)
plt.show()
More Model Analysis
Plot The Loss Curve
In [22]:
pd.DataFrame(m6History.history)Out [22]:
In [23]:
pd.DataFrame(m6History.history).plot()
plt.title("m6 training curves")Out [23]:
Find The Best Learning Rate
Playing around with the learning rate can be part of experimenting with machine learning.
A learning-rate callback function can be used to edit the model learning-rate based on the epoch number. Here, a lambda function will be used to increase the learning rate every epoch.
In [24]:
tf.random.set_seed(42)
# Create a model (same as model_8)
m7 = tf.keras.Sequential([
tf.keras.layers.Dense(4, activation="relu"),
tf.keras.layers.Dense(4, activation="relu"),
tf.keras.layers.Dense(1, activation="sigmoid")
])
# Compile the model
m7.compile(loss="binary_crossentropy",
# tf.keras.optimizers.Adam()
optimizer="Adam",
metrics=["accuracy"])
# Create a learning rate scheduler callback
lr_scheduler = tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-4 * 10**(epoch/20)) # traverse a set of learning rate values starting from 1e-4, increasing by 10**(epoch/20) every epoch
# Fit the model (passing the lr_scheduler callback)
m7History = m7.fit(feature_training_data,
label_training_data,
epochs=100,
callbacks=[lr_scheduler])Visualize The Model Performance
Visualizing by epoch...
- loss: how much error is happening between "guessed" output and "real" output, per-epoch. An ideal would be to see a downward slop from beginning to end
- accuracy: how accurate the model is per-epoch. An ideal would be to see an increase over time
- learning-rate (lr): how "fast" the model is learning, per-epoch. An ideal here could be to see a "hockey stick", increasing learning-rate over time
In [25]:
pd.DataFrame(m7History.history).plot(figsize=(10,7), xlabel="epochs");
Visualize Loss Vs Learning
Loss vs. learning can be super critical for "finding" the "ideal" learning-rate.
The LEAST loss means the best guess.
The learning-rate that "intersects" with the least loss is, potentially, the "best" learning-rate.
In [26]:
lrs = 1e-4 * (10 ** (np.arange(100)/20))
plt.figure(figsize=(10, 7))
plt.semilogx(lrs, m7History.history["loss"]) # we want the x-axis (learning rate) to be log scale
plt.xlabel("Learning Rate")
plt.ylabel("Loss")
plt.title("Learning rate vs. loss");
In [27]:
# using 10 exp -2 (.01), lets try using .02
# Create the model
m8 = tf.keras.Sequential([
tf.keras.layers.Dense(4, activation="relu"),
tf.keras.layers.Dense(4, activation="relu"),
tf.keras.layers.Dense(1, activation="sigmoid")
])
# Compile the model with the ideal learning rate
m8.compile(loss="binary_crossentropy",
optimizer=tf.keras.optimizers.Adam(learning_rate=0.02), # to adjust the learning rate, you need to use tf.keras.optimizers.Adam (not "adam")
metrics=["accuracy"])
# Fit the model for 20 epochs (5 less than before)
m8History = m8.fit(feature_training_data, label_training_data, epochs=30)In [31]:
m8Loss, m8Accuracy = m8.evaluate(feature_training_data, label_training_data)In [29]:
plt.figure(figsize=(12, 6))
plt.subplot(1, 2, 1)
plt.title("Train")
plot_decision_boundary(m8, X=feature_training_data, y=label_training_data)
plt.subplot(1, 2, 2)
plt.title("Test")
plot_decision_boundary(m8, X=feature_testing_data, y=label_testing_data)
plt.show()
Check Accuracy of Model
In [33]:
print(f'Loss on test Data: {m8Loss}')
print(f'Accuracy on test Data: {m8Accuracy*100:.2f}%')Make a Confusion Matrix
a visual square of 4 squares, comparing tp, tn, fp,fn
In [46]:
m8LabelPredictions = m8.predict(feature_testing_data)In [47]:
# AN ERROR!
confusion_matrix(label_testing_data, m8LabelPredictions)In [48]:
label_testing_data[0]Out [48]:
In [49]:
m8LabelPredictions[0]Out [49]:
The datatypes between the testing and prediction data are not the same :/
Predictions data contains prediction probabilities.
For the sake of visualizing a confusion matrix, these can be converted to 0-and-1.
In [50]:
reshapedPredictionData = tf.round(m8LabelPredictions)In [51]:
confusion_matrix(label_testing_data, reshapedPredictionData)Out [51]:
In [58]:
# Note: The following confusion matrix code is a remix of Scikit-Learn's
# plot_confusion_matrix function - https://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_confusion_matrix.html
# and Made with ML's introductory notebook - https://github.com/GokuMohandas/MadeWithML/blob/main/notebooks/08_Neural_Networks.ipynb
import itertools
figsize = (10, 10)
titleText = "Confusion Matrix"
xLabelText= "Predicted label"
yLabelText = "True label"
axisSize = 20
# Create the confusion matrix
cm = confusion_matrix(label_testing_data, reshapedPredictionData)
cm_norm = cm.astype("float") / cm.sum(axis=1)[:, np.newaxis] # normalize it
n_classes = cm.shape[0]
fig, ax = plt.subplots(figsize=figsize)
# Create a matrix plot
cax = ax.matshow(cm, cmap=plt.cm.Blues) # https://matplotlib.org/3.2.0/api/_as_gen/matplotlib.axes.Axes.matshow.html
fig.colorbar(cax)
# Create classes
classes = False
if classes:
labels = classes
else:
labels = np.arange(cm.shape[0])
# Label the axes
ax.set(title=titleText,
xlabel=xLabelText,
ylabel=yLabelText,
xticks=np.arange(n_classes),
yticks=np.arange(n_classes),
xticklabels=labels,
yticklabels=labels)
# Set x-axis labels to bottom
ax.xaxis.set_label_position("bottom")
ax.xaxis.tick_bottom()
# Adjust label size
ax.xaxis.label.set_size(axisSize)
ax.yaxis.label.set_size(axisSize)
ax.title.set_size(axisSize)
# Set threshold for different colors
threshold = (cm.max() + cm.min()) / 2.
# Plot the text on each cell
for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
plt.text(j, i, f"{cm[i, j]} ({cm_norm[i, j]*100:.1f}%)",
horizontalalignment="center",
color="white" if cm[i, j] > threshold else "black",
size=15)