Note
Click here to download the full example code
Transfer learning with MobileNet for cats vs. dogs¶
This tutorial presents a demonstration of how transfer learning is applied with our quantized models to get an Akida model.
The transfer learning example is derived from the Tensorflow tutorial:
Our base model is an Akida-compatible version of MobileNet v1, trained on ImageNet.
The new dataset for transfer learning is cats vs. dogs (link).
We use transfer learning to customize the model to the new task of classifying cats and dogs.
Note
This tutorial only shows the inference of the trained Keras model and its conversion to an Akida network. A textual explanation of the training is given below.
Transfer learning process¶
Transfer learning allows to classify on a specific task by using a pre-trained base model. For an introduction to transfer learning, please refer to the Tensorflow transfer learning tutorial before exploring this tutorial. Here, we focus on how to quantize the Keras model in order to convert it to an Akida one.
The model is composed of:
a base quantized MobileNet model used to extract image features
a top layer to classify cats and dogs
a sigmoid activation function to interpret model outputs as a probability
Base model
The base model is a quantized version of MobileNet v1. This model was trained and quantized using the ImageNet dataset. Please refer to the corresponding example for more information. The layers have 4-bit weights (except for the first layer having 8-bit weights) and the activations are quantized to 4 bits. This base model ends with a classification layer for 1000 classes. To classify cats and dogs, the feature extractor is preserved but the classification layer must be removed to be replaced by a new top layer focusing on the new task.
In our transfer learning process, the base model is frozen, i.e., the weights are not updated during training. Pre-trained weights for the frozen quantized model are provided on our data server.
Top layer
While a fully-connected top layer is added in the Tensorflow tutorial, we decided to use a separable convolutional layer with one output neuron for the top layer of our model. The reason is that the separable convolutional layer is the only Akida layer supporting 4-bit weights (see hardware compatibility).
Training process
The transfer learning process for quantized models can be handled in different ways:
From a quantized base model, the new transferred model is composed of a frozen base model and a float top layer. The top layer is trained. Then, the top layer is quantized and fine-tuned. If necessary, the base model can be unfrozen to be slightly trained to improve accuracy.
From a float base model, the new transferred model is also composed of a frozen base model (with float weights/activations) and a float top layer. The top layer is trained. Then the full model is quantized, unfrozen and fine-tuned. This option requires longer training operations since we don’t take advantage of an already quantized base model. Option 2 can be used alternatively if option 1 doesn’t give suitable performance.
In this example, option 1 is chosen. The training steps are described below.
1. Load and preprocess data¶
In this section, we will load and preprocess the ‘cats_vs_dogs’ dataset to match the required model’s inputs.
1.A - Load and split data¶
The cats_vs_dogs
dataset
is loaded and split into train, validation and test sets. The train and
validation sets were used for the transfer learning process. Here only
the test set is used. We use here tf.Dataset objects to load and
preprocess batches of data (one can look at the TensorFlow guide
here for more information).
Note
The cats_vs_dogs dataset version used here is 4.0.0.
import tensorflow_datasets as tfds
splits = ['train[:80%]', 'train[80%:90%]', 'train[90%:]']
tfds.disable_progress_bar()
(raw_train, raw_validation,
raw_test), metadata = tfds.load('cats_vs_dogs:4.0.0',
split=splits,
with_info=True,
as_supervised=True)
Out:
[1mDownloading and preparing dataset 786.68 MiB (download: 786.68 MiB, generated: Unknown size, total: 786.68 MiB) to /root/tensorflow_datasets/cats_vs_dogs/4.0.0...[0m
WARNING:absl:1738 images were corrupted and were skipped
[1mDataset cats_vs_dogs downloaded and prepared to /root/tensorflow_datasets/cats_vs_dogs/4.0.0. Subsequent calls will reuse this data.[0m
1.B - Preprocess the test set¶
We must apply the same preprocessing as for training: rescaling and resizing. Since Akida models directly accept integer-valued images, we also define a preprocessing function for Akida:
for Keras: images are rescaled between 0 and 1, and resized to 160x160
for Akida: images are only resized to 160x160 (uint8 values).
Keras and Akida models require 4-dimensional (N,H,W,C) arrays as inputs. We must then create batches of images to feed the model. For inference, the batch size is not relevant; you can set it such that the batch of images can be loaded in memory depending on your CPU/GPU.
import tensorflow as tf
IMG_SIZE = 160
input_scaling = (127.5, 127.5)
def format_example_keras(image, label):
image = tf.cast(image, tf.float32)
image = (image - input_scaling[1]) / input_scaling[0]
image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))
return image, label
def format_example_akida(image, label):
image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))
image = tf.cast(image, tf.uint8)
return image, label
BATCH_SIZE = 32
test_batches_keras = raw_test.map(format_example_keras).batch(BATCH_SIZE)
test_batches_akida = raw_test.map(format_example_akida).batch(BATCH_SIZE)
1.C - Get labels¶
Labels are contained in the test set as ‘0’ for cats and ‘1’ for dogs. We read through the batches to extract the labels.
import numpy as np
labels = np.array([])
for _, label_batch in test_batches_keras:
labels = np.concatenate((labels, label_batch))
num_images = labels.shape[0]
print(f"Test set composed of {num_images} images: "
f"{np.count_nonzero(labels==0)} cats and "
f"{np.count_nonzero(labels==1)} dogs.")
Out:
Test set composed of 2326 images: 1160 cats and 1166 dogs.
2. Modify a pre-trained base Keras model¶
In this section, we will describe how to modify a base model to specify
the classification for cats_vs_dogs.
2.A - Instantiate a Keras base model¶
Here, we instantiate a quantized Keras model based on a MobileNet model. This base model was previously trained using the 1000 classes of the ImageNet dataset. For more information, please see the ImageNet tutorial.
The quantized MobileNet model satisfies the Akida NSoC requirements:
The model relies on a convolutional layer (first layer) and separable convolutional layers, all being Akida-compatible.
All the separable convolutional layers have 4-bit weights, the first convolutional layer has 8-bit weights.
The activations are quantized with 4 bits.
from akida_models import mobilenet_imagenet
# Instantiate a quantized MobileNet model
base_model_keras = mobilenet_imagenet(input_shape=(IMG_SIZE, IMG_SIZE, 3),
weight_quantization=4,
activ_quantization=4,
input_weight_quantization=8)
# Load pre-trained weights for the base model
pretrained_weights = tf.keras.utils.get_file(
"mobilenet_imagenet_iq8_wq4_aq4.h5",
"http://data.brainchip.com/models/mobilenet/mobilenet_imagenet_224_iq8_wq4_aq4.h5",
file_hash="d9eabb514a7db6d823ab108b0fbc64fe2872ad1113bd6c04c9a3329b6a41e135",
cache_subdir='models/mobilenet')
base_model_keras.load_weights(pretrained_weights)
base_model_keras.summary()
Out:
Downloading data from http://data.brainchip.com/models/mobilenet/mobilenet_imagenet_224_iq8_wq4_aq4.h5
8192/16998088 [..............................] - ETA: 33s
73728/16998088 [..............................] - ETA: 18s
204800/16998088 [..............................] - ETA: 10s
401408/16998088 [..............................] - ETA: 7s
598016/16998088 [>.............................] - ETA: 6s
794624/16998088 [>.............................] - ETA: 5s
991232/16998088 [>.............................] - ETA: 5s
1187840/16998088 [=>............................] - ETA: 5s
1384448/16998088 [=>............................] - ETA: 5s
1581056/16998088 [=>............................] - ETA: 4s
1777664/16998088 [==>...........................] - ETA: 4s
1974272/16998088 [==>...........................] - ETA: 4s
2170880/16998088 [==>...........................] - ETA: 4s
2367488/16998088 [===>..........................] - ETA: 4s
2424832/16998088 [===>..........................] - ETA: 6s
2441216/16998088 [===>..........................] - ETA: 6s
2498560/16998088 [===>..........................] - ETA: 6s
2629632/16998088 [===>..........................] - ETA: 6s
2826240/16998088 [===>..........................] - ETA: 6s
3022848/16998088 [====>.........................] - ETA: 5s
3219456/16998088 [====>.........................] - ETA: 5s
3416064/16998088 [=====>........................] - ETA: 5s
3612672/16998088 [=====>........................] - ETA: 5s
3809280/16998088 [=====>........................] - ETA: 5s
4005888/16998088 [======>.......................] - ETA: 5s
4202496/16998088 [======>.......................] - ETA: 4s
4399104/16998088 [======>.......................] - ETA: 4s
4595712/16998088 [=======>......................] - ETA: 4s
4792320/16998088 [=======>......................] - ETA: 4s
4988928/16998088 [=======>......................] - ETA: 4s
5185536/16998088 [========>.....................] - ETA: 4s
5382144/16998088 [========>.....................] - ETA: 4s
5578752/16998088 [========>.....................] - ETA: 4s
5775360/16998088 [=========>....................] - ETA: 3s
5971968/16998088 [=========>....................] - ETA: 3s
6168576/16998088 [=========>....................] - ETA: 3s
6365184/16998088 [==========>...................] - ETA: 3s
6561792/16998088 [==========>...................] - ETA: 3s
6758400/16998088 [==========>...................] - ETA: 3s
6955008/16998088 [===========>..................] - ETA: 3s
7151616/16998088 [===========>..................] - ETA: 3s
7348224/16998088 [===========>..................] - ETA: 3s
7544832/16998088 [============>.................] - ETA: 3s
7741440/16998088 [============>.................] - ETA: 3s
7938048/16998088 [=============>................] - ETA: 2s
8134656/16998088 [=============>................] - ETA: 2s
8331264/16998088 [=============>................] - ETA: 2s
8527872/16998088 [==============>...............] - ETA: 2s
8724480/16998088 [==============>...............] - ETA: 2s
8921088/16998088 [==============>...............] - ETA: 2s
9117696/16998088 [===============>..............] - ETA: 2s
9314304/16998088 [===============>..............] - ETA: 2s
9510912/16998088 [===============>..............] - ETA: 2s
9707520/16998088 [================>.............] - ETA: 2s
9904128/16998088 [================>.............] - ETA: 2s
10100736/16998088 [================>.............] - ETA: 2s
10297344/16998088 [=================>............] - ETA: 2s
10493952/16998088 [=================>............] - ETA: 2s
10690560/16998088 [=================>............] - ETA: 1s
10887168/16998088 [==================>...........] - ETA: 1s
11083776/16998088 [==================>...........] - ETA: 1s
11280384/16998088 [==================>...........] - ETA: 1s
11476992/16998088 [===================>..........] - ETA: 1s
11673600/16998088 [===================>..........] - ETA: 1s
11870208/16998088 [===================>..........] - ETA: 1s
12066816/16998088 [====================>.........] - ETA: 1s
12263424/16998088 [====================>.........] - ETA: 1s
12460032/16998088 [====================>.........] - ETA: 1s
12656640/16998088 [=====================>........] - ETA: 1s
12853248/16998088 [=====================>........] - ETA: 1s
13049856/16998088 [======================>.......] - ETA: 1s
13246464/16998088 [======================>.......] - ETA: 1s
13443072/16998088 [======================>.......] - ETA: 1s
13639680/16998088 [=======================>......] - ETA: 1s
13836288/16998088 [=======================>......] - ETA: 0s
14032896/16998088 [=======================>......] - ETA: 0s
14229504/16998088 [========================>.....] - ETA: 0s
14426112/16998088 [========================>.....] - ETA: 0s
14622720/16998088 [========================>.....] - ETA: 0s
14819328/16998088 [=========================>....] - ETA: 0s
15015936/16998088 [=========================>....] - ETA: 0s
15212544/16998088 [=========================>....] - ETA: 0s
15409152/16998088 [==========================>...] - ETA: 0s
15605760/16998088 [==========================>...] - ETA: 0s
15802368/16998088 [==========================>...] - ETA: 0s
15998976/16998088 [===========================>..] - ETA: 0s
16195584/16998088 [===========================>..] - ETA: 0s
16392192/16998088 [===========================>..] - ETA: 0s
16588800/16998088 [============================>.] - ETA: 0s
16785408/16998088 [============================>.] - ETA: 0s
16982016/16998088 [============================>.] - ETA: 0s
16998400/16998088 [==============================] - 5s 0us/step
Model: "mobilenet_1.00_160_1000"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_26 (InputLayer) [(None, 160, 160, 3)] 0
_________________________________________________________________
conv_0 (QuantizedConv2D) (None, 80, 80, 32) 896
_________________________________________________________________
conv_0_relu (ActivationDiscr (None, 80, 80, 32) 0
_________________________________________________________________
separable_1 (QuantizedSepara (None, 80, 80, 64) 2400
_________________________________________________________________
separable_1_relu (Activation (None, 80, 80, 64) 0
_________________________________________________________________
separable_2 (QuantizedSepara (None, 80, 80, 128) 8896
_________________________________________________________________
separable_2_maxpool (MaxPool (None, 40, 40, 128) 0
_________________________________________________________________
separable_2_relu (Activation (None, 40, 40, 128) 0
_________________________________________________________________
separable_3 (QuantizedSepara (None, 40, 40, 128) 17664
_________________________________________________________________
separable_3_relu (Activation (None, 40, 40, 128) 0
_________________________________________________________________
separable_4 (QuantizedSepara (None, 40, 40, 256) 34176
_________________________________________________________________
separable_4_maxpool (MaxPool (None, 20, 20, 256) 0
_________________________________________________________________
separable_4_relu (Activation (None, 20, 20, 256) 0
_________________________________________________________________
separable_5 (QuantizedSepara (None, 20, 20, 256) 68096
_________________________________________________________________
separable_5_relu (Activation (None, 20, 20, 256) 0
_________________________________________________________________
separable_6 (QuantizedSepara (None, 20, 20, 512) 133888
_________________________________________________________________
separable_6_maxpool (MaxPool (None, 10, 10, 512) 0
_________________________________________________________________
separable_6_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_7 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_7_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_8 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_8_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_9 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_9_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_10 (QuantizedSepar (None, 10, 10, 512) 267264
_________________________________________________________________
separable_10_relu (Activatio (None, 10, 10, 512) 0
_________________________________________________________________
separable_11 (QuantizedSepar (None, 10, 10, 512) 267264
_________________________________________________________________
separable_11_relu (Activatio (None, 10, 10, 512) 0
_________________________________________________________________
separable_12 (QuantizedSepar (None, 10, 10, 1024) 529920
_________________________________________________________________
separable_12_maxpool (MaxPoo (None, 5, 5, 1024) 0
_________________________________________________________________
separable_12_relu (Activatio (None, 5, 5, 1024) 0
_________________________________________________________________
separable_13 (QuantizedSepar (None, 5, 5, 1024) 1058816
_________________________________________________________________
separable_13_global_avg (Glo (None, 1024) 0
_________________________________________________________________
separable_13_relu (Activatio (None, 1024) 0
_________________________________________________________________
reshape_1 (Reshape) (None, 1, 1, 1024) 0
_________________________________________________________________
dropout (Dropout) (None, 1, 1, 1024) 0
_________________________________________________________________
separable_14 (QuantizedSepar (None, 1, 1, 1000) 1033216
_________________________________________________________________
act_softmax (Activation) (None, 1, 1, 1000) 0
_________________________________________________________________
reshape_2 (Reshape) (None, 1000) 0
=================================================================
Total params: 4,224,288
Trainable params: 4,224,288
Non-trainable params: 0
_________________________________________________________________
2.B - Modify the network for the new task¶
As explained in section 1,
we replace the 1000-class top layer with a separable convolutional layer with
one output neuron.
The new model is now appropriate for the cats_vs_dogs dataset and is
Akida-compatible. Note that a sigmoid activation is added at the end of
the model: the output neuron returns a probability between 0 and 1 that
the input image is a dog.
from akida_models.layer_blocks import separable_conv_block
# Add a top layer for "cats_vs_dogs" classification
x = base_model_keras.get_layer('reshape_1').output
x = separable_conv_block(x,
filters=1,
kernel_size=(3, 3),
padding='same',
use_bias=False,
add_activation=False,
name='top_layer_separable')
x = tf.keras.layers.Activation('sigmoid')(x)
preds = tf.keras.layers.Reshape((1,), name='reshape_2')(x)
model_keras = tf.keras.Model(inputs=base_model_keras.input,
outputs=preds,
name="model_cats_vs_dogs")
model_keras.summary()
Out:
Model: "model_cats_vs_dogs"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_26 (InputLayer) [(None, 160, 160, 3)] 0
_________________________________________________________________
conv_0 (QuantizedConv2D) (None, 80, 80, 32) 896
_________________________________________________________________
conv_0_relu (ActivationDiscr (None, 80, 80, 32) 0
_________________________________________________________________
separable_1 (QuantizedSepara (None, 80, 80, 64) 2400
_________________________________________________________________
separable_1_relu (Activation (None, 80, 80, 64) 0
_________________________________________________________________
separable_2 (QuantizedSepara (None, 80, 80, 128) 8896
_________________________________________________________________
separable_2_maxpool (MaxPool (None, 40, 40, 128) 0
_________________________________________________________________
separable_2_relu (Activation (None, 40, 40, 128) 0
_________________________________________________________________
separable_3 (QuantizedSepara (None, 40, 40, 128) 17664
_________________________________________________________________
separable_3_relu (Activation (None, 40, 40, 128) 0
_________________________________________________________________
separable_4 (QuantizedSepara (None, 40, 40, 256) 34176
_________________________________________________________________
separable_4_maxpool (MaxPool (None, 20, 20, 256) 0
_________________________________________________________________
separable_4_relu (Activation (None, 20, 20, 256) 0
_________________________________________________________________
separable_5 (QuantizedSepara (None, 20, 20, 256) 68096
_________________________________________________________________
separable_5_relu (Activation (None, 20, 20, 256) 0
_________________________________________________________________
separable_6 (QuantizedSepara (None, 20, 20, 512) 133888
_________________________________________________________________
separable_6_maxpool (MaxPool (None, 10, 10, 512) 0
_________________________________________________________________
separable_6_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_7 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_7_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_8 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_8_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_9 (QuantizedSepara (None, 10, 10, 512) 267264
_________________________________________________________________
separable_9_relu (Activation (None, 10, 10, 512) 0
_________________________________________________________________
separable_10 (QuantizedSepar (None, 10, 10, 512) 267264
_________________________________________________________________
separable_10_relu (Activatio (None, 10, 10, 512) 0
_________________________________________________________________
separable_11 (QuantizedSepar (None, 10, 10, 512) 267264
_________________________________________________________________
separable_11_relu (Activatio (None, 10, 10, 512) 0
_________________________________________________________________
separable_12 (QuantizedSepar (None, 10, 10, 1024) 529920
_________________________________________________________________
separable_12_maxpool (MaxPoo (None, 5, 5, 1024) 0
_________________________________________________________________
separable_12_relu (Activatio (None, 5, 5, 1024) 0
_________________________________________________________________
separable_13 (QuantizedSepar (None, 5, 5, 1024) 1058816
_________________________________________________________________
separable_13_global_avg (Glo (None, 1024) 0
_________________________________________________________________
separable_13_relu (Activatio (None, 1024) 0
_________________________________________________________________
reshape_1 (Reshape) (None, 1, 1, 1024) 0
_________________________________________________________________
top_layer_separable (Separab (None, 1, 1, 1) 10240
_________________________________________________________________
activation (Activation) (None, 1, 1, 1) 0
_________________________________________________________________
reshape_2 (Reshape) (None, 1) 0
=================================================================
Total params: 3,201,312
Trainable params: 3,201,312
Non-trainable params: 0
_________________________________________________________________
3. Train the transferred model for the new task¶
The transferred model must be trained to learn how to classify cats and dogs. The quantized base model is frozen: only the float top layer will effectively be trained. One can take a look at the training section of the corresponding TensorFlow tutorial to reproduce the training stage.
The float top layer is trained for 20 epochs. We don’t illustrate the training phase in this tutorial; instead we directly load the pre-trained weights obtained after the 20 epochs.
# Freeze the base model part of the new model
base_model_keras.trainable = False
# Load pre-trained weights
pretrained_weights = tf.keras.utils.get_file(
"mobilenet_cats_vs_dogs_iq8_wq4_aq4.h5",
"http://data.brainchip.com/models/mobilenet/mobilenet_cats_vs_dogs_iq8_wq4_aq4.h5",
file_hash="b021fccaba676de9549430336ff27875a85b2aea7ca767a5e70a76185362fa4b",
cache_subdir='models')
model_keras.load_weights(pretrained_weights)
Out:
Downloading data from http://data.brainchip.com/models/mobilenet/mobilenet_cats_vs_dogs_iq8_wq4_aq4.h5
8192/12906128 [..............................] - ETA: 25s
73728/12906128 [..............................] - ETA: 14s
270336/12906128 [..............................] - ETA: 6s
466944/12906128 [>.............................] - ETA: 4s
663552/12906128 [>.............................] - ETA: 4s
860160/12906128 [>.............................] - ETA: 4s
1056768/12906128 [=>............................] - ETA: 3s
1253376/12906128 [=>............................] - ETA: 3s
1449984/12906128 [==>...........................] - ETA: 3s
1646592/12906128 [==>...........................] - ETA: 3s
1843200/12906128 [===>..........................] - ETA: 3s
2039808/12906128 [===>..........................] - ETA: 3s
2236416/12906128 [====>.........................] - ETA: 3s
2433024/12906128 [====>.........................] - ETA: 3s
2629632/12906128 [=====>........................] - ETA: 2s
2826240/12906128 [=====>........................] - ETA: 2s
3022848/12906128 [======>.......................] - ETA: 2s
3219456/12906128 [======>.......................] - ETA: 2s
3416064/12906128 [======>.......................] - ETA: 2s
3612672/12906128 [=======>......................] - ETA: 2s
3809280/12906128 [=======>......................] - ETA: 2s
4005888/12906128 [========>.....................] - ETA: 2s
4202496/12906128 [========>.....................] - ETA: 2s
4399104/12906128 [=========>....................] - ETA: 2s
4595712/12906128 [=========>....................] - ETA: 2s
4792320/12906128 [==========>...................] - ETA: 2s
4988928/12906128 [==========>...................] - ETA: 2s
5185536/12906128 [===========>..................] - ETA: 2s
5382144/12906128 [===========>..................] - ETA: 2s
5578752/12906128 [===========>..................] - ETA: 2s
5775360/12906128 [============>.................] - ETA: 1s
5971968/12906128 [============>.................] - ETA: 1s
6168576/12906128 [=============>................] - ETA: 1s
6365184/12906128 [=============>................] - ETA: 1s
6561792/12906128 [==============>...............] - ETA: 1s
6758400/12906128 [==============>...............] - ETA: 1s
6955008/12906128 [===============>..............] - ETA: 1s
7151616/12906128 [===============>..............] - ETA: 1s
7348224/12906128 [================>.............] - ETA: 1s
7544832/12906128 [================>.............] - ETA: 1s
7741440/12906128 [================>.............] - ETA: 1s
7938048/12906128 [=================>............] - ETA: 1s
8134656/12906128 [=================>............] - ETA: 1s
8331264/12906128 [==================>...........] - ETA: 1s
8527872/12906128 [==================>...........] - ETA: 1s
8724480/12906128 [===================>..........] - ETA: 1s
8921088/12906128 [===================>..........] - ETA: 1s
9117696/12906128 [====================>.........] - ETA: 1s
9314304/12906128 [====================>.........] - ETA: 0s
9510912/12906128 [=====================>........] - ETA: 0s
9707520/12906128 [=====================>........] - ETA: 0s
9904128/12906128 [======================>.......] - ETA: 0s
10100736/12906128 [======================>.......] - ETA: 0s
10297344/12906128 [======================>.......] - ETA: 0s
10493952/12906128 [=======================>......] - ETA: 0s
10690560/12906128 [=======================>......] - ETA: 0s
10887168/12906128 [========================>.....] - ETA: 0s
11083776/12906128 [========================>.....] - ETA: 0s
11280384/12906128 [=========================>....] - ETA: 0s
11476992/12906128 [=========================>....] - ETA: 0s
11673600/12906128 [==========================>...] - ETA: 0s
11870208/12906128 [==========================>...] - ETA: 0s
12066816/12906128 [===========================>..] - ETA: 0s
12263424/12906128 [===========================>..] - ETA: 0s
12460032/12906128 [===========================>..] - ETA: 0s
12656640/12906128 [============================>.] - ETA: 0s
12853248/12906128 [============================>.] - ETA: 0s
12910592/12906128 [==============================] - 3s 0us/step
# Check performance on the test set
model_keras.compile(metrics=['accuracy'])
_, keras_accuracy = model_keras.evaluate(test_batches_keras)
print(f"Keras accuracy (float top layer): {keras_accuracy*100:.2f} %")
Out:
1/73 [..............................] - ETA: 1:46 - loss: 2.0341 - accuracy: 1.0000
3/73 [>.............................] - ETA: 1s - loss: 2.0341 - accuracy: 1.0000
5/73 [=>............................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9959
7/73 [=>............................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9943
9/73 [==>...........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9923
11/73 [===>..........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9915
13/73 [====>.........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9907
15/73 [=====>........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9902
17/73 [=====>........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9899
19/73 [======>.......................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9895
21/73 [=======>......................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9886
23/73 [========>.....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9877
25/73 [=========>....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9871
27/73 [==========>...................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9866
29/73 [==========>...................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9862
31/73 [===========>..................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9859
33/73 [============>.................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9857
35/73 [=============>................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9854
37/73 [==============>...............] - ETA: 1s - loss: 2.0341 - accuracy: 0.9852
39/73 [===============>..............] - ETA: 1s - loss: 2.0341 - accuracy: 0.9850
41/73 [===============>..............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9848
43/73 [================>.............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9847
45/73 [=================>............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9845
47/73 [==================>...........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9843
49/73 [===================>..........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9841
51/73 [===================>..........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9840
53/73 [====================>.........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9838
55/73 [=====================>........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9837
57/73 [======================>.......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9836
59/73 [=======================>......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9835
61/73 [========================>.....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9834
63/73 [========================>.....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9832
65/73 [=========================>....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9831
67/73 [==========================>...] - ETA: 0s - loss: 2.0341 - accuracy: 0.9830
69/73 [===========================>..] - ETA: 0s - loss: 2.0341 - accuracy: 0.9829
71/73 [============================>.] - ETA: 0s - loss: 2.0341 - accuracy: 0.9827
73/73 [==============================] - ETA: 0s - loss: 2.0341 - accuracy: 0.9826
73/73 [==============================] - 4s 34ms/step - loss: 2.0341 - accuracy: 0.9826
Keras accuracy (float top layer): 97.89 %
4 Quantize the top layer¶
To get an Akida-compatible model, the float top layer must be quantized. We decide to quantize its weights to 4 bits. The performance of the new quantized model is then assessed.
Here, the quantized model gives suitable performance compared to the model with the float top layer. If that had not been the case, a fine-tuning step would have been necessary to recover the drop in accuracy.
from cnn2snn import quantize_layer
# Quantize the top layer to 4 bits
model_keras = quantize_layer(model_keras, 'top_layer_separable', bitwidth=4)
# Check performance for the quantized Keras model
model_keras.compile(metrics=['accuracy'])
_, keras_accuracy = model_keras.evaluate(test_batches_keras)
print(f"Quantized Keras accuracy: {keras_accuracy*100:.2f} %")
Out:
1/73 [..............................] - ETA: 1:28 - loss: 2.0341 - accuracy: 1.0000
3/73 [>.............................] - ETA: 2s - loss: 2.0341 - accuracy: 1.0000
5/73 [=>............................] - ETA: 2s - loss: 2.0341 - accuracy: 0.9959
7/73 [=>............................] - ETA: 2s - loss: 2.0341 - accuracy: 0.9943
9/73 [==>...........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9923
11/73 [===>..........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9915
13/73 [====>.........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9907
15/73 [=====>........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9902
17/73 [=====>........................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9899
19/73 [======>.......................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9895
21/73 [=======>......................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9886
23/73 [========>.....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9877
25/73 [=========>....................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9871
27/73 [==========>...................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9866
29/73 [==========>...................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9862
31/73 [===========>..................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9859
33/73 [============>.................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9857
35/73 [=============>................] - ETA: 1s - loss: 2.0341 - accuracy: 0.9855
37/73 [==============>...............] - ETA: 1s - loss: 2.0341 - accuracy: 0.9853
39/73 [===============>..............] - ETA: 1s - loss: 2.0341 - accuracy: 0.9851
41/73 [===============>..............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9850
43/73 [================>.............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9849
45/73 [=================>............] - ETA: 0s - loss: 2.0341 - accuracy: 0.9847
47/73 [==================>...........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9845
49/73 [===================>..........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9844
51/73 [===================>..........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9843
53/73 [====================>.........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9842
55/73 [=====================>........] - ETA: 0s - loss: 2.0341 - accuracy: 0.9842
57/73 [======================>.......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9841
59/73 [=======================>......] - ETA: 0s - loss: 2.0341 - accuracy: 0.9840
61/73 [========================>.....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9839
63/73 [========================>.....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9837
65/73 [=========================>....] - ETA: 0s - loss: 2.0341 - accuracy: 0.9836
67/73 [==========================>...] - ETA: 0s - loss: 2.0341 - accuracy: 0.9835
69/73 [===========================>..] - ETA: 0s - loss: 2.0341 - accuracy: 0.9834
71/73 [============================>.] - ETA: 0s - loss: 2.0341 - accuracy: 0.9833
73/73 [==============================] - 3s 31ms/step - loss: 2.0341 - accuracy: 0.9832
Quantized Keras accuracy: 97.98 %
5. Convert to Akida¶
The new quantized Keras model is now converted to an Akida model. The ‘sigmoid’ final activation has no SNN equivalent and will be simply ignored during the conversion.
Performance of the Akida model is then computed. Compared to Keras inference, remember that:
Input images in Akida are uint8 and not scaled like Keras inputs. But remember that the conversion process needs to know what scaling was applied during Keras training, in order to compensate (see CNN2SNN guide)
The Akida evaluate function takes a NumPy array containing the images and returns potentials before the sigmoid activation. We must therefore explicitly apply the ‘sigmoid’ activation on the model outputs to obtain the Akida probabilities.
Since activations sparsity has a great impact on Akida inference time, we also have a look at the average input and output sparsity of each layer on one batch of the test set.
from cnn2snn import convert
# Convert the model
model_akida = convert(model_keras, input_scaling=input_scaling)
model_akida.summary()
Out:
Model Summary
________________________________________________
Input shape Output shape Sequences Layers
================================================
[160, 160, 3] [1, 1, 1] 1 15
________________________________________________
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(a, dtype, copy=False, order=order)
SW/conv_0-top_layer_separable (Software)
____________________________________________________________________
Layer (type) Output shape Kernel shape
====================================================================
conv_0 (InputConv.) [80, 80, 32] (3, 3, 3, 32)
____________________________________________________________________
separable_1 (Sep.Conv.) [80, 80, 64] (3, 3, 32, 1)
____________________________________________________________________
(1, 1, 32, 64)
____________________________________________________________________
separable_2 (Sep.Conv.) [40, 40, 128] (3, 3, 64, 1)
____________________________________________________________________
(1, 1, 64, 128)
____________________________________________________________________
separable_3 (Sep.Conv.) [40, 40, 128] (3, 3, 128, 1)
____________________________________________________________________
(1, 1, 128, 128)
____________________________________________________________________
separable_4 (Sep.Conv.) [20, 20, 256] (3, 3, 128, 1)
____________________________________________________________________
(1, 1, 128, 256)
____________________________________________________________________
separable_5 (Sep.Conv.) [20, 20, 256] (3, 3, 256, 1)
____________________________________________________________________
(1, 1, 256, 256)
____________________________________________________________________
separable_6 (Sep.Conv.) [10, 10, 512] (3, 3, 256, 1)
____________________________________________________________________
(1, 1, 256, 512)
____________________________________________________________________
separable_7 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_8 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_9 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_10 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_11 (Sep.Conv.) [10, 10, 512] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 512)
____________________________________________________________________
separable_12 (Sep.Conv.) [5, 5, 1024] (3, 3, 512, 1)
____________________________________________________________________
(1, 1, 512, 1024)
____________________________________________________________________
separable_13 (Sep.Conv.) [1, 1, 1024] (3, 3, 1024, 1)
____________________________________________________________________
(1, 1, 1024, 1024)
____________________________________________________________________
top_layer_separable (Sep.Conv.) [1, 1, 1] (3, 3, 1024, 1)
____________________________________________________________________
(1, 1, 1024, 1)
____________________________________________________________________
from timeit import default_timer as timer
from progressbar import ProgressBar
# Run inference with Akida model
n_batches = num_images // BATCH_SIZE + 1
pots_akida = np.array([], dtype=np.float32)
pbar = ProgressBar(maxval=n_batches)
pbar.start()
start = timer()
i = 1
for batch, _ in test_batches_akida:
pots_batch_akida = model_akida.evaluate(batch.numpy())
pots_akida = np.concatenate((pots_akida, pots_batch_akida.squeeze()))
pbar.update(i)
i = i + 1
pbar.finish()
end = timer()
print(f"Akida inference on {num_images} images took {end-start:.2f} s.\n")
# Compute predictions and accuracy
preds_akida = tf.keras.layers.Activation('sigmoid')(pots_akida) > 0.5
akida_accuracy = np.mean(np.equal(preds_akida, labels))
print(f"Akida accuracy: {akida_accuracy*100:.2f} %")
# For non-regression purpose
assert akida_accuracy > 0.97
Out:
0% | |
1% | |
2% |# |
4% |## |
5% |### |
6% |#### |
8% |##### |
9% |###### |
10% |####### |
12% |######## |
13% |######### |
15% |########## |
16% |########### |
17% |############ |
19% |############# |
20% |############## |
21% |############### |
23% |################ |
24% |################# |
26% |################## |
27% |################### |
28% |#################### |
30% |##################### |
31% |###################### |
32% |####################### |
34% |######################## |
35% |######################### |
36% |########################## |
38% |########################### |
39% |############################ |
41% |############################# |
42% |############################## |
43% |############################### |
45% |################################ |
46% |################################# |
47% |################################## |
49% |################################### |
50% |#################################### |
52% |##################################### |
53% |###################################### |
54% |####################################### |
56% |######################################## |
57% |######################################### |
58% |########################################## |
60% |########################################### |
61% |############################################ |
63% |############################################# |
64% |############################################## |
65% |############################################### |
67% |################################################ |
68% |################################################# |
69% |################################################## |
71% |################################################### |
72% |#################################################### |
73% |##################################################### |
75% |###################################################### |
76% |####################################################### |
78% |######################################################## |
79% |######################################################### |
80% |########################################################## |
82% |########################################################### |
83% |############################################################ |
84% |############################################################# |
86% |############################################################## |
87% |############################################################### |
89% |################################################################ |
90% |################################################################# |
91% |################################################################## |
93% |################################################################### |
94% |#################################################################### |
95% |##################################################################### |
97% |###################################################################### |
98% |####################################################################### |
100% |########################################################################|
100% |########################################################################|
Akida inference on 2326 images took 112.41 s.
Akida accuracy: 98.11 %
# Print model statistics
print("Model statistics")
print(model_akida.statistics)
Out:
Model statistics
Sequence SW/conv_0-top_layer_separable
Average framerate = 20.75 fps
Layer (type) output sparsity
conv_0 (InputConv.) 0.38
Layer (type) output sparsity
separable_1 (Sep.Conv.) 0.40
Layer (type) output sparsity
separable_2 (Sep.Conv.) 0.39
Layer (type) output sparsity
separable_3 (Sep.Conv.) 0.40
Layer (type) output sparsity
separable_4 (Sep.Conv.) 0.55
Layer (type) output sparsity
separable_5 (Sep.Conv.) 0.43
Layer (type) output sparsity
separable_6 (Sep.Conv.) 0.59
Layer (type) output sparsity
separable_7 (Sep.Conv.) 0.59
Layer (type) output sparsity
separable_8 (Sep.Conv.) 0.66
Layer (type) output sparsity
separable_9 (Sep.Conv.) 0.73
Layer (type) output sparsity
separable_10 (Sep.Conv.) 0.70
Layer (type) output sparsity
separable_11 (Sep.Conv.) 0.63
Layer (type) output sparsity
separable_12 (Sep.Conv.) 0.90
Layer (type) output sparsity
separable_13 (Sep.Conv.) 0.69
Layer (type) output sparsity
top_layer_separable (Sep.Conv N/A
Let’s summarize the accuracy for the quantized Keras and the Akida model.
Model |
Accuracy |
|---|---|
quantized Keras |
98.28 % |
Akida |
98.32 % |
6. Plot confusion matrix¶
import matplotlib.pyplot as plt
def confusion_matrix_2classes(labels, predictions):
tp = np.count_nonzero(labels + predictions == 2)
tn = np.count_nonzero(labels + predictions == 0)
fp = np.count_nonzero(predictions - labels == 1)
fn = np.count_nonzero(labels - predictions == 1)
return np.array([[tp, fn], [fp, tn]])
def plot_confusion_matrix_2classes(cm, classes):
cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
plt.xticks([0, 1], classes)
plt.yticks([0, 1], classes)
for i, j in zip([0, 0, 1, 1], [0, 1, 0, 1]):
plt.text(j,
i,
f"{cm[i, j]:.2f}",
horizontalalignment="center",
color="white" if cm[i, j] > cm.max() / 2. else "black")
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.autoscale()
# Plot confusion matrix for Akida
cm_akida = confusion_matrix_2classes(labels, preds_akida.numpy())
plot_confusion_matrix_2classes(cm_akida, ['dog', 'cat'])
plt.show()
Total running time of the script: ( 3 minutes 7.974 seconds)