Note
Click here to download the full example code
Transfer learning with MobileNet for cats vs. dogs
This tutorial presents a demonstration of how transfer learning is applied with our quantized models to get an Akida model.
The transfer learning example is derived from the Tensorflow tutorial:
Our base model is an Akida-compatible version of MobileNet v1, trained on ImageNet.
The new dataset for transfer learning is cats vs. dogs (link).
We use transfer learning to customize the model to the new task of classifying cats and dogs.
Note
This tutorial only shows the inference of the trained Keras model and its conversion to an Akida network. A textual explanation of the training is given below.
Transfer learning process
Transfer learning allows to classify on a specific task by using a pre-trained base model. For an introduction to transfer learning, please refer to the Tensorflow transfer learning tutorial before exploring this tutorial. Here, we focus on how to quantize the Keras model in order to convert it to an Akida one.
The model is composed of:
a base quantized MobileNet model used to extract image features
a top layer to classify cats and dogs
a sigmoid activation function to interpret model outputs as a probability
Base model
The base model is a quantized version of MobileNet v1. This model was trained and quantized using the ImageNet dataset. Please refer to the corresponding example for more information. The layers have 4-bit weights (except for the first layer having 8-bit weights) and the activations are quantized to 4 bits. This base model ends with a classification layer for 1000 classes. To classify cats and dogs, the feature extractor is preserved but the classification layer must be removed to be replaced by a new top layer focusing on the new task.
In our transfer learning process, the base model is frozen, i.e., the weights are not updated during training. Pre-trained weights for the frozen quantized model are provided on our data server.
Top layer
While a fully-connected top layer is added in the Tensorflow tutorial, we decided to use a separable convolutional layer with one output neuron for the top layer of our model. The reason is that the separable convolutional layer is the only Akida layer supporting 4-bit weights (see hardware compatibility).
Training process
The transfer learning process for quantized models can be handled in different ways:
From a quantized base model, the new transferred model is composed of a frozen base model and a float top layer. The top layer is trained. Then, the top layer is quantized and fine-tuned. If necessary, the base model can be unfrozen to be slightly trained to improve accuracy.
From a float base model, the new transferred model is also composed of a frozen base model (with float weights/activations) and a float top layer. The top layer is trained. Then the full model is quantized, unfrozen and fine-tuned. This option requires longer training operations since we don’t take advantage of an already quantized base model. Option 2 can be used alternatively if option 1 doesn’t give suitable performance.
In this example, option 1 is chosen. The training steps are described below.
1. Load and preprocess data
In this section, we will load and preprocess the ‘cats_vs_dogs’ dataset to match the required model’s inputs.
1.A - Load and split data
The cats_vs_dogs
dataset
is loaded and split into train, validation and test sets. The train and
validation sets were used for the transfer learning process. Here only
the test set is used. We use here tf.Dataset objects to load and
preprocess batches of data (one can look at the TensorFlow guide
here for more information).
Note
The cats_vs_dogs dataset version used here is 4.0.0.
import tensorflow_datasets as tfds
splits = ['train[:80%]', 'train[80%:90%]', 'train[90%:]']
tfds.disable_progress_bar()
(raw_train, raw_validation,
raw_test), metadata = tfds.load('cats_vs_dogs:4.0.0',
split=splits,
with_info=True,
as_supervised=True)
Out:
Downloading and preparing dataset 786.68 MiB (download: 786.68 MiB, generated: Unknown size, total: 786.68 MiB) to /root/tensorflow_datasets/cats_vs_dogs/4.0.0...
WARNING:absl:1738 images were corrupted and were skipped
Dataset cats_vs_dogs downloaded and prepared to /root/tensorflow_datasets/cats_vs_dogs/4.0.0. Subsequent calls will reuse this data.
1.B - Preprocess the test set
We must apply the same preprocessing as for training: rescaling and resizing. Since Akida models directly accept integer-valued images, we also define a preprocessing function for Akida:
for Keras: images are rescaled between 0 and 1, and resized to 160x160
for Akida: images are only resized to 160x160 (uint8 values).
Keras and Akida models require 4-dimensional (N,H,W,C) arrays as inputs. We must then create batches of images to feed the model. For inference, the batch size is not relevant; you can set it such that the batch of images can be loaded in memory depending on your CPU/GPU.
import tensorflow as tf
IMG_SIZE = 160
def format_example(image, label):
image = tf.cast(image, tf.float32)
image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))
return image, label
BATCH_SIZE = 32
test_batches = raw_test.map(format_example).batch(BATCH_SIZE)
1.C - Get labels
Labels are contained in the test set as ‘0’ for cats and ‘1’ for dogs. We read through the batches to extract the labels.
import numpy as np
labels = np.array([])
for _, label_batch in test_batches:
labels = np.concatenate((labels, label_batch))
num_images = labels.shape[0]
print(f"Test set composed of {num_images} images: "
f"{np.count_nonzero(labels==0)} cats and "
f"{np.count_nonzero(labels==1)} dogs.")
Out:
Test set composed of 2326 images: 1160 cats and 1166 dogs.
2. Modify a pre-trained base Keras model
In this section, we will describe how to modify a base model to specify
the classification for cats_vs_dogs.
2.A - Instantiate a Keras base model
Here, we instantiate a quantized Keras model based on a MobileNet model. This base model was previously trained using the 1000 classes of the ImageNet dataset. For more information, please see the ImageNet tutorial.
The quantized MobileNet model satisfies the Akida NSoC requirements:
The model relies on a convolutional layer (first layer) and separable convolutional layers, all being Akida-compatible.
All the separable convolutional layers have 4-bit weights, the first convolutional layer has 8-bit weights.
The activations are quantized with 4 bits.
from akida_models import mobilenet_imagenet
# Instantiate a quantized MobileNet model
base_model_keras = mobilenet_imagenet(input_shape=(IMG_SIZE, IMG_SIZE, 3),
weight_quantization=4,
activ_quantization=4,
input_weight_quantization=8)
# Load pre-trained weights for the base model
pretrained_weights = tf.keras.utils.get_file(
"mobilenet_imagenet_iq8_wq4_aq4.h5",
"http://data.brainchip.com/models/mobilenet/mobilenet_imagenet_224_iq8_wq4_aq4.h5",
file_hash="d9eabb514a7db6d823ab108b0fbc64fe2872ad1113bd6c04c9a3329b6a41e135",
cache_subdir='models/mobilenet')
base_model_keras.load_weights(pretrained_weights)
base_model_keras.summary()
Out:
Downloading data from http://data.brainchip.com/models/mobilenet/mobilenet_imagenet_224_iq8_wq4_aq4.h5
16384/17001760 [..............................] - ETA: 14s
139264/17001760 [..............................] - ETA: 8s
401408/17001760 [..............................] - ETA: 5s
663552/17001760 [>.............................] - ETA: 4s
925696/17001760 [>.............................] - ETA: 4s
1187840/17001760 [=>............................] - ETA: 4s
1449984/17001760 [=>............................] - ETA: 3s
1712128/17001760 [==>...........................] - ETA: 3s
1974272/17001760 [==>...........................] - ETA: 3s
2236416/17001760 [==>...........................] - ETA: 3s
2498560/17001760 [===>..........................] - ETA: 3s
2760704/17001760 [===>..........................] - ETA: 3s
3022848/17001760 [====>.........................] - ETA: 3s
3284992/17001760 [====>.........................] - ETA: 3s
3547136/17001760 [=====>........................] - ETA: 3s
3809280/17001760 [=====>........................] - ETA: 3s
4071424/17001760 [======>.......................] - ETA: 2s
4333568/17001760 [======>.......................] - ETA: 2s
4595712/17001760 [=======>......................] - ETA: 2s
4857856/17001760 [=======>......................] - ETA: 2s
5120000/17001760 [========>.....................] - ETA: 2s
5382144/17001760 [========>.....................] - ETA: 2s
5644288/17001760 [========>.....................] - ETA: 2s
5906432/17001760 [=========>....................] - ETA: 2s
6168576/17001760 [=========>....................] - ETA: 2s
6430720/17001760 [==========>...................] - ETA: 2s
6692864/17001760 [==========>...................] - ETA: 2s
6955008/17001760 [===========>..................] - ETA: 2s
7217152/17001760 [===========>..................] - ETA: 2s
7479296/17001760 [============>.................] - ETA: 2s
7741440/17001760 [============>.................] - ETA: 2s
8003584/17001760 [=============>................] - ETA: 2s
8265728/17001760 [=============>................] - ETA: 1s
8527872/17001760 [==============>...............] - ETA: 1s
8790016/17001760 [==============>...............] - ETA: 1s
9052160/17001760 [==============>...............] - ETA: 1s
9314304/17001760 [===============>..............] - ETA: 1s
9576448/17001760 [===============>..............] - ETA: 1s
9838592/17001760 [================>.............] - ETA: 1s
10100736/17001760 [================>.............] - ETA: 1s
10362880/17001760 [=================>............] - ETA: 1s
10625024/17001760 [=================>............] - ETA: 1s
10887168/17001760 [==================>...........] - ETA: 1s
11149312/17001760 [==================>...........] - ETA: 1s
11411456/17001760 [===================>..........] - ETA: 1s
11673600/17001760 [===================>..........] - ETA: 1s
11935744/17001760 [====================>.........] - ETA: 1s
12197888/17001760 [====================>.........] - ETA: 1s
12460032/17001760 [====================>.........] - ETA: 1s
12722176/17001760 [=====================>........] - ETA: 0s
12984320/17001760 [=====================>........] - ETA: 0s
13246464/17001760 [======================>.......] - ETA: 0s
13508608/17001760 [======================>.......] - ETA: 0s
13770752/17001760 [=======================>......] - ETA: 0s
14032896/17001760 [=======================>......] - ETA: 0s
14295040/17001760 [========================>.....] - ETA: 0s
14557184/17001760 [========================>.....] - ETA: 0s
14819328/17001760 [=========================>....] - ETA: 0s
15081472/17001760 [=========================>....] - ETA: 0s
15343616/17001760 [==========================>...] - ETA: 0s
15605760/17001760 [==========================>...] - ETA: 0s
15867904/17001760 [==========================>...] - ETA: 0s
16130048/17001760 [===========================>..] - ETA: 0s
16392192/17001760 [===========================>..] - ETA: 0s
16654336/17001760 [============================>.] - ETA: 0s
16916480/17001760 [============================>.] - ETA: 0s
17006592/17001760 [==============================] - 4s 0us/step
17014784/17001760 [==============================] - 4s 0us/step
Model: "mobilenet_1.00_160_1000"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_26 (InputLayer) [(None, 160, 160, 3)] 0
_________________________________________________________________
rescaling (Rescaling) (None, 160, 160, 3) 0
_________________________________________________________________
conv_0 (QuantizedConv2D) (None, 80, 80, 32) 896
_________________________________________________________________
conv_0_relu (ActivationDiscr (None, 80, 80, 32) 0
_________________________________________________________________
separable_1 (QuantizedSepara (None, 80, 80, 64) 2400
_________________________________________________________________
separable_1_relu (Activation (None, 80, 80, 64) 0
_________________________________________________________________
separable_2 (QuantizedSepara (None, 80, 80, 128) 8896
_________________________________________________________________
separable_2_maxpool (MaxPool (None, 40, 40, 128) 0
_________________________________________________________________
separable_2_relu (Activation (None, 40, 40, 128) 0
_________________________________________________________________
separable_3 (QuantizedSepara (None, 40, 40, 128) 17664
_________________________________________________________________
separable_3_relu (Activation (None, 40, 40, 128) 0
_________________________________________________________________
separable_4 (QuantizedSepara (None, 40, 40, 256) 34176
_________________________________________________________________
separable_4_maxpool (MaxPool (None, 20, 20, 256) 0
_________________________________________________________________
separable_4_relu (Activation (None, 20, 20, 256) 0
_________________________________________________________________
separable_5 (QuantizedSepara (None, 20, 20, 256) 68096
_________________________________________________________________
separable_5_relu (Activation (None, 20, 20, 256) 0
_________________________________________________________________
separable_6 (QuantizedSepara (None, 20, 20, 512) 133888
_________________________________________________________________
separable_6_maxpool (MaxPool (None, 10, 10, 512) 0
_________________________________________________________________
separable_6_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_7 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_7_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_8 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_8_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_9 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_9_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_10 (QuantizedSepar (None, 10, 10, 512) 267264
_________________________________________________________________
separable_10_relu (Activatio (None, 10, 10, 512) 0
_________________________________________________________________
separable_11 (QuantizedSepar (None, 10, 10, 512) 267264
_________________________________________________________________
separable_11_relu (Activatio (None, 10, 10, 512) 0
_________________________________________________________________
separable_12 (QuantizedSepar (None, 10, 10, 1024) 529920
_________________________________________________________________
separable_12_maxpool (MaxPoo (None, 5, 5, 1024) 0
_________________________________________________________________
separable_12_relu (Activatio (None, 5, 5, 1024) 0
_________________________________________________________________
separable_13 (QuantizedSepar (None, 5, 5, 1024) 1058816
_________________________________________________________________
separable_13_global_avg (Glo (None, 1024) 0
_________________________________________________________________
separable_13_relu (Activatio (None, 1024) 0
_________________________________________________________________
reshape_1 (Reshape) (None, 1, 1, 1024) 0
_________________________________________________________________
dropout (Dropout) (None, 1, 1, 1024) 0
_________________________________________________________________
separable_14 (QuantizedSepar (None, 1, 1, 1000) 1033216
_________________________________________________________________
act_softmax (Activation) (None, 1, 1, 1000) 0
_________________________________________________________________
reshape_2 (Reshape) (None, 1000) 0
=================================================================
Total params: 4,224,288
Trainable params: 4,224,288
Non-trainable params: 0
_________________________________________________________________
2.B - Modify the network for the new task
As explained in section 1,
we replace the 1000-class top layer with a separable convolutional layer with
one output neuron.
The new model is now appropriate for the cats_vs_dogs dataset and is
Akida-compatible. Note that a sigmoid activation is added at the end of
the model: the output neuron returns a probability between 0 and 1 that
the input image is a dog.
from akida_models.layer_blocks import separable_conv_block
# Add a top layer for "cats_vs_dogs" classification
x = base_model_keras.get_layer('reshape_1').output
x = separable_conv_block(x,
filters=1,
kernel_size=(3, 3),
padding='same',
use_bias=False,
add_activation=False,
name='top_layer_separable')
x = tf.keras.layers.Activation('sigmoid')(x)
preds = tf.keras.layers.Reshape((1,), name='reshape_2')(x)
model_keras = tf.keras.Model(inputs=base_model_keras.input,
outputs=preds,
name="model_cats_vs_dogs")
model_keras.summary()
Out:
Model: "model_cats_vs_dogs"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_26 (InputLayer) [(None, 160, 160, 3)] 0
_________________________________________________________________
rescaling (Rescaling) (None, 160, 160, 3) 0
_________________________________________________________________
conv_0 (QuantizedConv2D) (None, 80, 80, 32) 896
_________________________________________________________________
conv_0_relu (ActivationDiscr (None, 80, 80, 32) 0
_________________________________________________________________
separable_1 (QuantizedSepara (None, 80, 80, 64) 2400
_________________________________________________________________
separable_1_relu (Activation (None, 80, 80, 64) 0
_________________________________________________________________
separable_2 (QuantizedSepara (None, 80, 80, 128) 8896
_________________________________________________________________
separable_2_maxpool (MaxPool (None, 40, 40, 128) 0
_________________________________________________________________
separable_2_relu (Activation (None, 40, 40, 128) 0
_________________________________________________________________
separable_3 (QuantizedSepara (None, 40, 40, 128) 17664
_________________________________________________________________
separable_3_relu (Activation (None, 40, 40, 128) 0
_________________________________________________________________
separable_4 (QuantizedSepara (None, 40, 40, 256) 34176
_________________________________________________________________
separable_4_maxpool (MaxPool (None, 20, 20, 256) 0
_________________________________________________________________
separable_4_relu (Activation (None, 20, 20, 256) 0
_________________________________________________________________
separable_5 (QuantizedSepara (None, 20, 20, 256) 68096
_________________________________________________________________
separable_5_relu (Activation (None, 20, 20, 256) 0
_________________________________________________________________
separable_6 (QuantizedSepara (None, 20, 20, 512) 133888
_________________________________________________________________
separable_6_maxpool (MaxPool (None, 10, 10, 512) 0
_________________________________________________________________
separable_6_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_7 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_7_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_8 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_8_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_9 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_9_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_10 (QuantizedSepar (None, 10, 10, 512) 267264
_________________________________________________________________
separable_10_relu (Activatio (None, 10, 10, 512) 0
_________________________________________________________________
separable_11 (QuantizedSepar (None, 10, 10, 512) 267264
_________________________________________________________________
separable_11_relu (Activatio (None, 10, 10, 512) 0
_________________________________________________________________
separable_12 (QuantizedSepar (None, 10, 10, 1024) 529920
_________________________________________________________________
separable_12_maxpool (MaxPoo (None, 5, 5, 1024) 0
_________________________________________________________________
separable_12_relu (Activatio (None, 5, 5, 1024) 0
_________________________________________________________________
separable_13 (QuantizedSepar (None, 5, 5, 1024) 1058816
_________________________________________________________________
separable_13_global_avg (Glo (None, 1024) 0
_________________________________________________________________
separable_13_relu (Activatio (None, 1024) 0
_________________________________________________________________
reshape_1 (Reshape) (None, 1, 1, 1024) 0
_________________________________________________________________
top_layer_separable (Separab (None, 1, 1, 1) 10240
_________________________________________________________________
activation (Activation) (None, 1, 1, 1) 0
_________________________________________________________________
reshape_2 (Reshape) (None, 1) 0
=================================================================
Total params: 3,201,312
Trainable params: 3,201,312
Non-trainable params: 0
_________________________________________________________________
3. Train the transferred model for the new task
The transferred model must be trained to learn how to classify cats and dogs. The quantized base model is frozen: only the float top layer will effectively be trained. One can take a look at the training section of the corresponding TensorFlow tutorial to reproduce the training stage.
The float top layer is trained for 20 epochs. We don’t illustrate the training phase in this tutorial; instead we directly load the pre-trained weights obtained after the 20 epochs.
# Freeze the base model part of the new model
base_model_keras.trainable = False
# Load pre-trained weights
pretrained_weights = tf.keras.utils.get_file(
"mobilenet_cats_vs_dogs_iq8_wq4_aq4.h5",
"http://data.brainchip.com/models/mobilenet/mobilenet_cats_vs_dogs_iq8_wq4_aq4.h5",
file_hash="b021fccaba676de9549430336ff27875a85b2aea7ca767a5e70a76185362fa4b",
cache_subdir='models')
model_keras.load_weights(pretrained_weights)
Out:
Downloading data from http://data.brainchip.com/models/mobilenet/mobilenet_cats_vs_dogs_iq8_wq4_aq4.h5
16384/12909984 [..............................] - ETA: 10s
139264/12909984 [..............................] - ETA: 6s
401408/12909984 [..............................] - ETA: 3s
663552/12909984 [>.............................] - ETA: 3s
925696/12909984 [=>............................] - ETA: 3s
1187840/12909984 [=>............................] - ETA: 2s
1449984/12909984 [==>...........................] - ETA: 2s
1712128/12909984 [==>...........................] - ETA: 2s
1974272/12909984 [===>..........................] - ETA: 2s
2236416/12909984 [====>.........................] - ETA: 2s
2498560/12909984 [====>.........................] - ETA: 2s
2760704/12909984 [=====>........................] - ETA: 2s
3022848/12909984 [======>.......................] - ETA: 2s
3284992/12909984 [======>.......................] - ETA: 2s
3547136/12909984 [=======>......................] - ETA: 2s
3809280/12909984 [=======>......................] - ETA: 2s
4071424/12909984 [========>.....................] - ETA: 2s
4333568/12909984 [=========>....................] - ETA: 1s
4595712/12909984 [=========>....................] - ETA: 1s
4857856/12909984 [==========>...................] - ETA: 1s
5120000/12909984 [==========>...................] - ETA: 1s
5382144/12909984 [===========>..................] - ETA: 1s
5644288/12909984 [============>.................] - ETA: 1s
5906432/12909984 [============>.................] - ETA: 1s
6168576/12909984 [=============>................] - ETA: 1s
6430720/12909984 [=============>................] - ETA: 1s
6692864/12909984 [==============>...............] - ETA: 1s
6955008/12909984 [===============>..............] - ETA: 1s
7217152/12909984 [===============>..............] - ETA: 1s
7479296/12909984 [================>.............] - ETA: 1s
7741440/12909984 [================>.............] - ETA: 1s
8003584/12909984 [=================>............] - ETA: 1s
8265728/12909984 [==================>...........] - ETA: 1s
8527872/12909984 [==================>...........] - ETA: 0s
8790016/12909984 [===================>..........] - ETA: 0s
9052160/12909984 [====================>.........] - ETA: 0s
9314304/12909984 [====================>.........] - ETA: 0s
9576448/12909984 [=====================>........] - ETA: 0s
9838592/12909984 [=====================>........] - ETA: 0s
10100736/12909984 [======================>.......] - ETA: 0s
10362880/12909984 [=======================>......] - ETA: 0s
10625024/12909984 [=======================>......] - ETA: 0s
10887168/12909984 [========================>.....] - ETA: 0s
11149312/12909984 [========================>.....] - ETA: 0s
11411456/12909984 [=========================>....] - ETA: 0s
11673600/12909984 [==========================>...] - ETA: 0s
11935744/12909984 [==========================>...] - ETA: 0s
12197888/12909984 [===========================>..] - ETA: 0s
12460032/12909984 [===========================>..] - ETA: 0s
12722176/12909984 [============================>.] - ETA: 0s
12910592/12909984 [==============================] - 3s 0us/step
12918784/12909984 [==============================] - 3s 0us/step
# Check performance on the test set
model_keras.compile(metrics=['accuracy'])
_, keras_accuracy = model_keras.evaluate(test_batches)
print(f"Keras accuracy (float top layer): {keras_accuracy*100:.2f} %")
Out:
1/73 [..............................] - ETA: 1:34 - loss: 2.0341 - accuracy: 1.0000
4/73 [>.............................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9922
7/73 [=>............................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9955
10/73 [===>..........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9937
13/73 [====>.........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9928
16/73 [=====>........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9941
18/73 [======>.......................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9931
20/73 [=======>......................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9859
22/73 [========>.....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9858
24/73 [========>.....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9857
26/73 [=========>....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9868
28/73 [==========>...................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9866
30/73 [===========>..................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9865
32/73 [============>.................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9873
34/73 [============>.................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9871
36/73 [=============>................] - ETA: 0s - loss: 2.0341 - accuracy: 0.9861
38/73 [==============>...............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9860
40/73 [===============>..............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9859
42/73 [================>.............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9866
44/73 [=================>............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9858
46/73 [=================>............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9844
48/73 [==================>...........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9850
50/73 [===================>..........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9850
52/73 [====================>.........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9850
54/73 [=====================>........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9850
56/73 [======================>.......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9844
58/73 [======================>.......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9838
60/73 [=======================>......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9839
62/73 [========================>.....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9819
64/73 [=========================>....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9824
66/73 [==========================>...] - ETA: 0s - loss: 2.0341 - accuracy: 0.9830
68/73 [==========================>...] - ETA: 0s - loss: 2.0341 - accuracy: 0.9830
70/73 [===========================>..] - ETA: 0s - loss: 2.0341 - accuracy: 0.9826
72/73 [============================>.] - ETA: 0s - loss: 2.0341 - accuracy: 0.9831
73/73 [==============================] - ETA: 0s - loss: 2.0341 - accuracy: 0.9828
73/73 [==============================] - 4s 31ms/step - loss: 2.0341 - accuracy: 0.9828
Keras accuracy (float top layer): 98.28 %
4 Quantize the top layer
To get an Akida-compatible model, the float top layer must be quantized. We decide to quantize its weights to 4 bits. The performance of the new quantized model is then assessed.
Here, the quantized model gives suitable performance compared to the model with the float top layer. If that had not been the case, a fine-tuning step would have been necessary to recover the drop in accuracy.
from cnn2snn import quantize_layer
# Quantize the top layer to 4 bits
model_keras = quantize_layer(model_keras, 'top_layer_separable', bitwidth=4)
# Check performance for the quantized Keras model
model_keras.compile(metrics=['accuracy'])
_, keras_accuracy = model_keras.evaluate(test_batches)
print(f"Quantized Keras accuracy: {keras_accuracy*100:.2f} %")
Out:
1/73 [..............................] - ETA: 1:14 - loss: 2.0341 - accuracy: 1.0000
4/73 [>.............................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9922
7/73 [=>............................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9955
9/73 [==>...........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9931
11/73 [===>..........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9943
13/73 [====>.........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9928
15/73 [=====>........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9937
17/73 [=====>........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9926
19/73 [======>.......................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9885
21/73 [=======>......................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9866
23/73 [========>.....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9851
25/73 [=========>....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9862
27/73 [==========>...................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9861
29/73 [==========>...................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9860
31/73 [===========>..................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9869
33/73 [============>.................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9877
35/73 [=============>................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9857
37/73 [==============>...............] - ETA: 1s - loss: 2.0341 - accuracy: 0.9856
39/73 [===============>..............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9864
41/73 [===============>..............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9863
43/73 [================>.............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9862
45/73 [=================>............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9847
47/73 [==================>...........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9847
49/73 [===================>..........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9847
51/73 [===================>..........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9853
53/73 [====================>.........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9853
55/73 [=====================>........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9847
57/73 [======================>.......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9830
59/73 [=======================>......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9831
61/73 [========================>.....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9826
63/73 [========================>.....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9816
65/73 [=========================>....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9822
67/73 [==========================>...] - ETA: 0s - loss: 2.0341 - accuracy: 0.9827
69/73 [===========================>..] - ETA: 0s - loss: 2.0341 - accuracy: 0.9819
71/73 [============================>.] - ETA: 0s - loss: 2.0341 - accuracy: 0.9824
73/73 [==============================] - ETA: 0s - loss: 2.0341 - accuracy: 0.9824
73/73 [==============================] - 3s 29ms/step - loss: 2.0341 - accuracy: 0.9824
Quantized Keras accuracy: 98.24 %
5. Convert to Akida
The new quantized Keras model is now converted to an Akida model. The ‘sigmoid’ final activation has no SNN equivalent and will be simply ignored during the conversion.
Performance of the Akida model is then computed. Compared to Keras inference, remember that:
Input images in Akida are uint8 and not scaled like Keras inputs. But remember that the conversion process needs to know what scaling was applied during Keras training, in order to compensate (see CNN2SNN guide)
The Akida evaluate function takes a NumPy array containing the images and returns potentials before the sigmoid activation. We must therefore explicitly apply the ‘sigmoid’ activation on the model outputs to obtain the Akida probabilities.
Since activations sparsity has a great impact on Akida inference time, we also have a look at the average input and output sparsity of each layer on one batch of the test set.
from cnn2snn import convert
# Convert the model
model_akida = convert(model_keras)
model_akida.summary()
Out:
Model Summary
________________________________________________
Input shape Output shape Sequences Layers
================================================
[160, 160, 3] [1, 1, 1] 1 15
________________________________________________
SW/conv_0-top_layer_separable (Software)
____________________________________________________________________
Layer (type) Output shape Kernel shape
====================================================================
conv_0 (InputConv.) [80, 80, 32] (3, 3, 3, 32)
____________________________________________________________________
separable_1 (Sep.Conv.) [80, 80, 64] (3, 3, 32, 1)
____________________________________________________________________
(1, 1, 32, 64)
____________________________________________________________________
separable_2 (Sep.Conv.) [40, 40, 128] (3, 3, 64, 1)
____________________________________________________________________
(1, 1, 64, 128)
____________________________________________________________________
separable_3 (Sep.Conv.) [40, 40, 128] (3, 3, 128, 1)
____________________________________________________________________
(1, 1, 128, 128)
____________________________________________________________________
separable_4 (Sep.Conv.) [20, 20, 256] (3, 3, 128, 1)
____________________________________________________________________
(1, 1, 128, 256)
____________________________________________________________________
separable_5 (Sep.Conv.) [20, 20, 256] (3, 3, 256, 1)
____________________________________________________________________
(1, 1, 256, 256)
____________________________________________________________________
separable_6 (Sep.Conv.) [10, 10, 512] (3, 3, 256, 1)
____________________________________________________________________
(1, 1, 256, 512)
____________________________________________________________________
separable_7 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_8 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_9 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_10 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_11 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_12 (Sep.Conv.) [5, 5, 1024] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 1024)
____________________________________________________________________
separable_13 (Sep.Conv.) [1, 1, 1024] (3, 3, 1024, 1)
____________________________________________________________________
(1, 1, 1024, 1024)
____________________________________________________________________
top_layer_separable (Sep.Conv.) [1, 1, 1] (3, 3, 1024, 1)
____________________________________________________________________
(1, 1, 1024, 1)
____________________________________________________________________
from timeit import default_timer as timer
from progressbar import ProgressBar
# Run inference with Akida model
n_batches = num_images // BATCH_SIZE + 1
pots_akida = np.array([], dtype=np.float32)
pbar = ProgressBar(maxval=n_batches)
pbar.start()
start = timer()
i = 1
for batch, _ in test_batches:
pots_batch_akida = model_akida.evaluate(batch.numpy().astype('uint8'))
pots_akida = np.concatenate((pots_akida, pots_batch_akida.squeeze()))
pbar.update(i)
i = i + 1
pbar.finish()
end = timer()
print(f"Akida inference on {num_images} images took {end-start:.2f} s.\n")
# Compute predictions and accuracy
preds_akida = tf.keras.layers.Activation('sigmoid')(pots_akida) > 0.5
akida_accuracy = np.mean(np.equal(preds_akida, labels))
print(f"Akida accuracy: {akida_accuracy*100:.2f} %")
# For non-regression purpose
assert akida_accuracy > 0.97
Out:
0% | |
1% | |
2% |# |
4% |## |
5% |### |
6% |#### |
8% |##### |
9% |###### |
10% |####### |
12% |######## |
13% |######### |
15% |########## |
16% |########### |
17% |############ |
19% |############# |
20% |############## |
21% |############### |
23% |################ |
24% |################# |
26% |################## |
27% |################### |
28% |#################### |
30% |##################### |
31% |###################### |
32% |####################### |
34% |######################## |
35% |######################### |
36% |########################## |
38% |########################### |
39% |############################ |
41% |############################# |
42% |############################## |
43% |############################### |
45% |################################ |
46% |################################# |
47% |################################## |
49% |################################### |
50% |#################################### |
52% |##################################### |
53% |###################################### |
54% |####################################### |
56% |######################################## |
57% |######################################### |
58% |########################################## |
60% |########################################### |
61% |############################################ |
63% |############################################# |
64% |############################################## |
65% |############################################### |
67% |################################################ |
68% |################################################# |
69% |################################################## |
71% |################################################### |
72% |#################################################### |
73% |##################################################### |
75% |###################################################### |
76% |####################################################### |
78% |######################################################## |
79% |######################################################### |
80% |########################################################## |
82% |########################################################### |
83% |############################################################ |
84% |############################################################# |
86% |############################################################## |
87% |############################################################### |
89% |################################################################ |
90% |################################################################# |
91% |################################################################## |
93% |################################################################### |
94% |#################################################################### |
95% |##################################################################### |
97% |###################################################################### |
98% |####################################################################### |
100% |########################################################################|
100% |########################################################################|
Akida inference on 2326 images took 84.50 s.
Akida accuracy: 98.11 %
Let’s summarize the accuracy for the quantized Keras and the Akida model.
Model |
Accuracy |
|---|---|
quantized Keras |
98.28 % |
Akida |
98.32 % |
6. Plot confusion matrix
import matplotlib.pyplot as plt
def confusion_matrix_2classes(labels, predictions):
tp = np.count_nonzero(labels + predictions == 2)
tn = np.count_nonzero(labels + predictions == 0)
fp = np.count_nonzero(predictions - labels == 1)
fn = np.count_nonzero(labels - predictions == 1)
return np.array([[tp, fn], [fp, tn]])
def plot_confusion_matrix_2classes(cm, classes):
cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
plt.xticks([0, 1], classes)
plt.yticks([0, 1], classes)
for i, j in zip([0, 0, 1, 1], [0, 1, 0, 1]):
plt.text(j,
i,
f"{cm[i, j]:.2f}",
horizontalalignment="center",
color="white" if cm[i, j] > cm.max() / 2. else "black")
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.autoscale()
# Plot confusion matrix for Akida
cm_akida = confusion_matrix_2classes(labels, preds_akida.numpy())
plot_confusion_matrix_2classes(cm_akida, ['dog', 'cat'])
plt.show()
Total running time of the script: ( 2 minutes 41.833 seconds)