Skip to content
Search
Generic filters
Exact matches only

How to Use Metrics for Deep Learning with Keras in Python

Last Updated on January 8, 2020

The Keras library provides a way to calculate and report on a suite of standard metrics when training deep learning models.

In addition to offering standard metrics for classification and regression problems, Keras also allows you to define and report on your own custom metrics when training deep learning models. This is particularly useful if you want to keep track of a performance measure that better captures the skill of your model during training.

In this tutorial, you will discover how to use the built-in metrics and how to define and use your own metrics when training deep learning models in Keras.

After completing this tutorial, you will know:

  • How Keras metrics work and how you can use them when training your models.
  • How to use regression and classification metrics in Keras with worked examples.
  • How to define and use your own custom metric in Keras with a worked example.

Discover how to develop deep learning models for a range of predictive modeling problems with just a few lines of code in my new book, with 18 step-by-step tutorials and 9 projects.

Let’s get started.

  • Update Jan/2020: Updated API for Keras 2.3 and TensorFlow 2.0.

Metrics and How to Use Custom Metrics for Deep Learning with Keras in Python

Metrics and How to Use Custom Metrics for Deep Learning with Keras in Python
Photo by Indi Samarajiva, some rights reserved.

Tutorial Overview

This tutorial is divided into 4 parts; they are:

  1. Keras Metrics
  2. Keras Regression Metrics
  3. Keras Classification Metrics
  4. Custom Metrics in Keras

Keras Metrics

Keras allows you to list the metrics to monitor during the training of your model.

You can do this by specifying the “metrics” argument and providing a list of function names (or function name aliases) to the compile() function on your model.

For example:

model.compile(…, metrics=[‘mse’])

model.compile(…, metrics=[‘mse’])

The specific metrics that you list can be the names of Keras functions (like mean_squared_error) or string aliases for those functions (like ‘mse‘).

Metric values are recorded at the end of each epoch on the training dataset. If a validation dataset is also provided, then the metric recorded is also calculated for the validation dataset.

All metrics are reported in verbose output and in the history object returned from calling the fit() function. In both cases, the name of the metric function is used as the key for the metric values. In the case of metrics for the validation dataset, the “val_” prefix is added to the key.

Both loss functions and explicitly defined Keras metrics can be used as training metrics.

Keras Regression Metrics

Below is a list of the metrics that you can use in Keras on regression problems.

  • Mean Squared Error: mean_squared_error, MSE or mse
  • Mean Absolute Error: mean_absolute_error, MAE, mae
  • Mean Absolute Percentage Error: mean_absolute_percentage_error, MAPE, mape
  • Cosine Proximity: cosine_proximity, cosine

The example below demonstrates these 4 built-in regression metrics on a simple contrived regression problem.

from numpy import array
from keras.models import Sequential
from keras.layers import Dense
from matplotlib import pyplot
# prepare sequence
X = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
# create model
model = Sequential()
model.add(Dense(2, input_dim=1))
model.add(Dense(1))
model.compile(loss=’mse’, optimizer=’adam’, metrics=[‘mse’, ‘mae’, ‘mape’, ‘cosine’])
# train model
history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)
# plot metrics
pyplot.plot(history.history[‘mean_squared_error’])
pyplot.plot(history.history[‘mean_absolute_error’])
pyplot.plot(history.history[‘mean_absolute_percentage_error’])
pyplot.plot(history.history[‘cosine_proximity’])
pyplot.show()

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

from numpy import array

from keras.models import Sequential

from keras.layers import Dense

from matplotlib import pyplot

# prepare sequence

X = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])

# create model

model = Sequential()

model.add(Dense(2, input_dim=1))

model.add(Dense(1))

model.compile(loss=’mse’, optimizer=’adam’, metrics=[‘mse’, ‘mae’, ‘mape’, ‘cosine’])

# train model

history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)

# plot metrics

pyplot.plot(history.history[‘mean_squared_error’])

pyplot.plot(history.history[‘mean_absolute_error’])

pyplot.plot(history.history[‘mean_absolute_percentage_error’])

pyplot.plot(history.history[‘cosine_proximity’])

pyplot.show()

Running the example prints the metric values at the end of each epoch.


Epoch 96/100
0s – loss: 1.0596e-04 – mean_squared_error: 1.0596e-04 – mean_absolute_error: 0.0088 – mean_absolute_percentage_error: 3.5611 – cosine_proximity: -1.0000e+00
Epoch 97/100
0s – loss: 1.0354e-04 – mean_squared_error: 1.0354e-04 – mean_absolute_error: 0.0087 – mean_absolute_percentage_error: 3.5178 – cosine_proximity: -1.0000e+00
Epoch 98/100
0s – loss: 1.0116e-04 – mean_squared_error: 1.0116e-04 – mean_absolute_error: 0.0086 – mean_absolute_percentage_error: 3.4738 – cosine_proximity: -1.0000e+00
Epoch 99/100
0s – loss: 9.8820e-05 – mean_squared_error: 9.8820e-05 – mean_absolute_error: 0.0085 – mean_absolute_percentage_error: 3.4294 – cosine_proximity: -1.0000e+00
Epoch 100/100
0s – loss: 9.6515e-05 – mean_squared_error: 9.6515e-05 – mean_absolute_error: 0.0084 – mean_absolute_percentage_error: 3.3847 – cosine_proximity: -1.0000e+00

Epoch 96/100

0s – loss: 1.0596e-04 – mean_squared_error: 1.0596e-04 – mean_absolute_error: 0.0088 – mean_absolute_percentage_error: 3.5611 – cosine_proximity: -1.0000e+00

Epoch 97/100

0s – loss: 1.0354e-04 – mean_squared_error: 1.0354e-04 – mean_absolute_error: 0.0087 – mean_absolute_percentage_error: 3.5178 – cosine_proximity: -1.0000e+00

Epoch 98/100

0s – loss: 1.0116e-04 – mean_squared_error: 1.0116e-04 – mean_absolute_error: 0.0086 – mean_absolute_percentage_error: 3.4738 – cosine_proximity: -1.0000e+00

Epoch 99/100

0s – loss: 9.8820e-05 – mean_squared_error: 9.8820e-05 – mean_absolute_error: 0.0085 – mean_absolute_percentage_error: 3.4294 – cosine_proximity: -1.0000e+00

Epoch 100/100

0s – loss: 9.6515e-05 – mean_squared_error: 9.6515e-05 – mean_absolute_error: 0.0084 – mean_absolute_percentage_error: 3.3847 – cosine_proximity: -1.0000e+00

A line plot of the 4 metrics over the training epochs is then created.

Line Plot of Built-in Keras Metrics for Regression

Line Plot of Built-in Keras Metrics for Regression

Note that the metrics were specified using string alias values [‘mse‘, ‘mae‘, ‘mape‘, ‘cosine‘] and were referenced as key values on the history object using their expanded function name.

We could also specify the metrics using their expanded name, as follows:

model.compile(loss=’mse’, optimizer=’adam’, metrics=[‘mean_squared_error’, ‘mean_absolute_error’, ‘mean_absolute_percentage_error’, ‘cosine_proximity’])

model.compile(loss=’mse’, optimizer=’adam’, metrics=[‘mean_squared_error’, ‘mean_absolute_error’, ‘mean_absolute_percentage_error’, ‘cosine_proximity’])

We can also specify the function names directly if they are imported into the script.

from keras import metrics
model.compile(loss=’mse’, optimizer=’adam’, metrics=[metrics.mean_squared_error, metrics.mean_absolute_error, metrics.mean_absolute_percentage_error, metrics.cosine_proximity])

from keras import metrics

model.compile(loss=’mse’, optimizer=’adam’, metrics=[metrics.mean_squared_error, metrics.mean_absolute_error, metrics.mean_absolute_percentage_error, metrics.cosine_proximity])

You can also use the loss functions as metrics.

For example, you could use the Mean squared Logarithmic Error (mean_squared_logarithmic_error, MSLE or msle) loss function as a metric as follows:

model.compile(loss=’mse’, optimizer=’adam’, metrics=[‘msle’])

model.compile(loss=’mse’, optimizer=’adam’, metrics=[‘msle’])

Keras Classification Metrics

Below is a list of the metrics that you can use in Keras on classification problems.

  • Binary Accuracy: binary_accuracy, acc
  • Categorical Accuracy: categorical_accuracy, acc
  • Sparse Categorical Accuracy: sparse_categorical_accuracy
  • Top k Categorical Accuracy: top_k_categorical_accuracy (requires you specify a k parameter)
  • Sparse Top k Categorical Accuracy: sparse_top_k_categorical_accuracy (requires you specify a k parameter)

Accuracy is special.

Regardless of whether your problem is a binary or multi-class classification problem, you can specify the ‘accuracy‘ metric to report on accuracy.

Below is an example of a binary classification problem with the built-in accuracy metric demonstrated.

from numpy import array
from keras.models import Sequential
from keras.layers import Dense
from matplotlib import pyplot
# prepare sequence
X = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
y = array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
# create model
model = Sequential()
model.add(Dense(2, input_dim=1))
model.add(Dense(1, activation=’sigmoid’))
model.compile(loss=’binary_crossentropy’, optimizer=’adam’, metrics=[‘accuracy’])
# train model
history = model.fit(X, y, epochs=400, batch_size=len(X), verbose=2)
# plot metrics
pyplot.plot(history.history[‘accuracy’])
pyplot.show()

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

from numpy import array

from keras.models import Sequential

from keras.layers import Dense

from matplotlib import pyplot

# prepare sequence

X = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])

y = array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])

# create model

model = Sequential()

model.add(Dense(2, input_dim=1))

model.add(Dense(1, activation=’sigmoid’))

model.compile(loss=’binary_crossentropy’, optimizer=’adam’, metrics=[‘accuracy’])

# train model

history = model.fit(X, y, epochs=400, batch_size=len(X), verbose=2)

# plot metrics

pyplot.plot(history.history[‘accuracy’])

pyplot.show()

Running the example reports the accuracy at the end of each training epoch.


Epoch 396/400
0s – loss: 0.5934 – acc: 0.9000
Epoch 397/400
0s – loss: 0.5932 – acc: 0.9000
Epoch 398/400
0s – loss: 0.5930 – acc: 0.9000
Epoch 399/400
0s – loss: 0.5927 – acc: 0.9000
Epoch 400/400
0s – loss: 0.5925 – acc: 0.9000

Epoch 396/400

0s – loss: 0.5934 – acc: 0.9000

Epoch 397/400

0s – loss: 0.5932 – acc: 0.9000

Epoch 398/400

0s – loss: 0.5930 – acc: 0.9000

Epoch 399/400

0s – loss: 0.5927 – acc: 0.9000

Epoch 400/400

0s – loss: 0.5925 – acc: 0.9000

A line plot of accuracy over epoch is created.

Line Plot of Built-in Keras Metrics for Classification

Line Plot of Built-in Keras Metrics for Classification

Custom Metrics in Keras

You can also define your own metrics and specify the function name in the list of functions for the “metrics” argument when calling the compile() function.

A metric I often like to keep track of is Root Mean Square Error, or RMSE.

You can get an idea of how to write a custom metric by examining the code for an existing metric.

For example, below is the code for the mean_squared_error loss function and metric in Keras.

def mean_squared_error(y_true, y_pred):
return K.mean(K.square(y_pred – y_true), axis=-1)

def mean_squared_error(y_true, y_pred):

    return K.mean(K.square(y_pred – y_true), axis=-1)

K is the backend used by Keras.

From this example and other examples of loss functions and metrics, the approach is to use standard math functions on the backend to calculate the metric of interest.

For example, we can write a custom metric to calculate RMSE as follows:

from keras import backend

def rmse(y_true, y_pred):
return backend.sqrt(backend.mean(backend.square(y_pred – y_true), axis=-1))

from keras import backend

def rmse(y_true, y_pred):

return backend.sqrt(backend.mean(backend.square(y_pred – y_true), axis=-1))

You can see the function is the same code as MSE with the addition of the sqrt() wrapping the result.

We can test this in our regression example as follows. Note that we simply list the function name directly rather than providing it as a string or alias for Keras to resolve.

from numpy import array
from keras.models import Sequential
from keras.layers import Dense
from matplotlib import pyplot
from keras import backend

def rmse(y_true, y_pred):
return backend.sqrt(backend.mean(backend.square(y_pred – y_true), axis=-1))

# prepare sequence
X = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])
# create model
model = Sequential()
model.add(Dense(2, input_dim=1, activation=’relu’))
model.add(Dense(1))
model.compile(loss=’mse’, optimizer=’adam’, metrics=[rmse])
# train model
history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)
# plot metrics
pyplot.plot(history.history[‘rmse’])
pyplot.show()

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

from numpy import array

from keras.models import Sequential

from keras.layers import Dense

from matplotlib import pyplot

from keras import backend

def rmse(y_true, y_pred):

return backend.sqrt(backend.mean(backend.square(y_pred – y_true), axis=-1))

# prepare sequence

X = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0])

# create model

model = Sequential()

model.add(Dense(2, input_dim=1, activation=’relu’))

model.add(Dense(1))

model.compile(loss=’mse’, optimizer=’adam’, metrics=[rmse])

# train model

history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)

# plot metrics

pyplot.plot(history.history[‘rmse’])

pyplot.show()

Running the example reports the custom RMSE metric at the end of each training epoch.


Epoch 496/500
0s – loss: 1.2992e-06 – rmse: 9.7909e-04
Epoch 497/500
0s – loss: 1.2681e-06 – rmse: 9.6731e-04
Epoch 498/500
0s – loss: 1.2377e-06 – rmse: 9.5562e-04
Epoch 499/500
0s – loss: 1.2079e-06 – rmse: 9.4403e-04
Epoch 500/500
0s – loss: 1.1788e-06 – rmse: 9.3261e-04

Epoch 496/500

0s – loss: 1.2992e-06 – rmse: 9.7909e-04

Epoch 497/500

0s – loss: 1.2681e-06 – rmse: 9.6731e-04

Epoch 498/500

0s – loss: 1.2377e-06 – rmse: 9.5562e-04

Epoch 499/500

0s – loss: 1.2079e-06 – rmse: 9.4403e-04

Epoch 500/500

0s – loss: 1.1788e-06 – rmse: 9.3261e-04

At the end of the run, a line plot of the custom RMSE metric is created.

Line Plot of Custom RMSE Keras Metric for Regression

Line Plot of Custom RMSE Keras Metric for Regression

Your custom metric function must operate on Keras internal data structures that may be different depending on the backend used (e.g. tensorflow.python.framework.ops.Tensor when using tensorflow) rather than the raw yhat and y values directly.

For this reason, I would recommend using the backend math functions wherever possible for consistency and execution speed.

Further Reading

This section provides more resources on the topic if you are looking go deeper.

Summary

In this tutorial, you discovered how to use Keras metrics when training your deep learning models.

Specifically, you learned:

  • How Keras metrics works and how you configure your models to report on metrics during training.
  • How to use classification and regression metrics built into Keras.
  • How to define and report on your own custom metrics efficiently while training your deep learning models.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Develop Deep Learning Projects with Python!

Deep Learning with Python

 What If You Could Develop A Network in Minutes

…with just a few lines of Python

Discover how in my new Ebook:
Deep Learning With Python

It covers end-to-end projects on topics like:
Multilayer Perceptrons, Convolutional Nets and Recurrent Neural Nets, and more…

Finally Bring Deep Learning To
Your Own Projects

Skip the Academics. Just Results.

See What’s Inside

error: Content is protected !!