AkidaNet/ImageNet inference

This CNN2SNN tutorial presents how to convert an AkidaNet pre-trained model into Akida.

As ImageNet images are not publicly available, performances are assessed using a set of 10 copyright free images that were found on Google using ImageNet class names.

1. Dataset preparation

Test images all have at least 256 pixels in the smallest dimension. They must be preprocessed to fit in the model. The imagenet.preprocessing.preprocess_image function decodes, crops and extracts a square 224x224x3 patch from an input image.

Note

Input size is here set to 224x224x3 as this is what is used by the model presented in the next section.

import akida
import os
import numpy as np

from tensorflow.io import read_file
from tensorflow.image import decode_jpeg
from tensorflow.keras.utils import get_file

from akida_models.imagenet import preprocessing

# Model specification and hyperparameters
NUM_CHANNELS = 3
IMAGE_SIZE = 224
NUM_CLASSES = 1000

num_images = 10

# Retrieve dataset file from Brainchip data server
file_path = get_file(
    "imagenet_like.zip",
    "http://data.brainchip.com/dataset-mirror/imagenet_like/imagenet_like.zip",
    cache_subdir='datasets/imagenet_like',
    extract=True)
data_folder = os.path.dirname(file_path)

# Load images for test set
x_test_files = []
x_test = np.zeros((num_images, 224, 224, 3)).astype('uint8')
for id in range(num_images):
    test_file = 'image_' + str(id + 1).zfill(2) + '.jpg'
    x_test_files.append(test_file)
    img_path = os.path.join(data_folder, test_file)
    base_image = read_file(img_path)
    image = decode_jpeg(base_image, channels=NUM_CHANNELS)
    image = preprocessing.preprocess_image(image, IMAGE_SIZE)
    x_test[id, :, :, :] = np.expand_dims(image, axis=0)

print(f'{num_images} images loaded and preprocessed.')

Out:

Downloading data from http://data.brainchip.com/dataset-mirror/imagenet_like/imagenet_like.zip

   16384/20418307 [..............................] - ETA: 0s
  139264/20418307 [..............................] - ETA: 9s
  335872/20418307 [..............................] - ETA: 7s
  532480/20418307 [..............................] - ETA: 6s
  729088/20418307 [>.............................] - ETA: 6s
  925696/20418307 [>.............................] - ETA: 6s
 1122304/20418307 [>.............................] - ETA: 6s
 1318912/20418307 [>.............................] - ETA: 6s
 1515520/20418307 [=>............................] - ETA: 5s
 1712128/20418307 [=>............................] - ETA: 5s
 1908736/20418307 [=>............................] - ETA: 5s
 2105344/20418307 [==>...........................] - ETA: 5s
 2301952/20418307 [==>...........................] - ETA: 5s
 2498560/20418307 [==>...........................] - ETA: 5s
 2695168/20418307 [==>...........................] - ETA: 5s
 2883584/20418307 [===>..........................] - ETA: 5s
 3022848/20418307 [===>..........................] - ETA: 5s
 3219456/20418307 [===>..........................] - ETA: 5s
 3416064/20418307 [====>.........................] - ETA: 5s
 3612672/20418307 [====>.........................] - ETA: 5s
 3809280/20418307 [====>.........................] - ETA: 5s
 4005888/20418307 [====>.........................] - ETA: 5s
 4202496/20418307 [=====>........................] - ETA: 4s
 4399104/20418307 [=====>........................] - ETA: 4s
 4595712/20418307 [=====>........................] - ETA: 4s
 4792320/20418307 [======>.......................] - ETA: 4s
 4988928/20418307 [======>.......................] - ETA: 4s
 5185536/20418307 [======>.......................] - ETA: 4s
 5382144/20418307 [======>.......................] - ETA: 4s
 5578752/20418307 [=======>......................] - ETA: 4s
 5775360/20418307 [=======>......................] - ETA: 4s
 5971968/20418307 [=======>......................] - ETA: 4s
 6168576/20418307 [========>.....................] - ETA: 4s
 6365184/20418307 [========>.....................] - ETA: 4s
 6561792/20418307 [========>.....................] - ETA: 4s
 6758400/20418307 [========>.....................] - ETA: 4s
 6955008/20418307 [=========>....................] - ETA: 4s
 7151616/20418307 [=========>....................] - ETA: 4s
 7348224/20418307 [=========>....................] - ETA: 3s
 7544832/20418307 [==========>...................] - ETA: 3s
 7741440/20418307 [==========>...................] - ETA: 3s
 7938048/20418307 [==========>...................] - ETA: 3s
 8134656/20418307 [==========>...................] - ETA: 3s
 8331264/20418307 [===========>..................] - ETA: 3s
 8527872/20418307 [===========>..................] - ETA: 3s
 8724480/20418307 [===========>..................] - ETA: 3s
 8921088/20418307 [============>.................] - ETA: 3s
 9117696/20418307 [============>.................] - ETA: 3s
 9314304/20418307 [============>.................] - ETA: 3s
 9510912/20418307 [============>.................] - ETA: 3s
 9707520/20418307 [=============>................] - ETA: 3s
 9904128/20418307 [=============>................] - ETA: 3s
10100736/20418307 [=============>................] - ETA: 3s
10297344/20418307 [==============>...............] - ETA: 3s
10493952/20418307 [==============>...............] - ETA: 3s
10690560/20418307 [==============>...............] - ETA: 2s
10887168/20418307 [==============>...............] - ETA: 2s
11083776/20418307 [===============>..............] - ETA: 2s
11280384/20418307 [===============>..............] - ETA: 2s
11476992/20418307 [===============>..............] - ETA: 2s
11673600/20418307 [================>.............] - ETA: 2s
11870208/20418307 [================>.............] - ETA: 2s
12066816/20418307 [================>.............] - ETA: 2s
12263424/20418307 [=================>............] - ETA: 2s
12460032/20418307 [=================>............] - ETA: 2s
12656640/20418307 [=================>............] - ETA: 2s
12853248/20418307 [=================>............] - ETA: 2s
13049856/20418307 [==================>...........] - ETA: 2s
13246464/20418307 [==================>...........] - ETA: 2s
13443072/20418307 [==================>...........] - ETA: 2s
13639680/20418307 [===================>..........] - ETA: 2s
13836288/20418307 [===================>..........] - ETA: 1s
14032896/20418307 [===================>..........] - ETA: 1s
14229504/20418307 [===================>..........] - ETA: 1s
14426112/20418307 [====================>.........] - ETA: 1s
14622720/20418307 [====================>.........] - ETA: 1s
14819328/20418307 [====================>.........] - ETA: 1s
15015936/20418307 [=====================>........] - ETA: 1s
15212544/20418307 [=====================>........] - ETA: 1s
15409152/20418307 [=====================>........] - ETA: 1s
15605760/20418307 [=====================>........] - ETA: 1s
15802368/20418307 [======================>.......] - ETA: 1s
15998976/20418307 [======================>.......] - ETA: 1s
16195584/20418307 [======================>.......] - ETA: 1s
16392192/20418307 [=======================>......] - ETA: 1s
16588800/20418307 [=======================>......] - ETA: 1s
16785408/20418307 [=======================>......] - ETA: 1s
16982016/20418307 [=======================>......] - ETA: 1s
17178624/20418307 [========================>.....] - ETA: 0s
17375232/20418307 [========================>.....] - ETA: 0s
17571840/20418307 [========================>.....] - ETA: 0s
17768448/20418307 [=========================>....] - ETA: 0s
17965056/20418307 [=========================>....] - ETA: 0s
18161664/20418307 [=========================>....] - ETA: 0s
18358272/20418307 [=========================>....] - ETA: 0s
18554880/20418307 [==========================>...] - ETA: 0s
18751488/20418307 [==========================>...] - ETA: 0s
18948096/20418307 [==========================>...] - ETA: 0s
19144704/20418307 [===========================>..] - ETA: 0s
19341312/20418307 [===========================>..] - ETA: 0s
19537920/20418307 [===========================>..] - ETA: 0s
19734528/20418307 [===========================>..] - ETA: 0s
19931136/20418307 [============================>.] - ETA: 0s
20127744/20418307 [============================>.] - ETA: 0s
20324352/20418307 [============================>.] - ETA: 0s
20422656/20418307 [==============================] - 6s 0us/step

20430848/20418307 [==============================] - 6s 0us/step
10 images loaded and preprocessed.

Labels for test images are stored in the akida_models package. The matching between names (string) and labels (integer) is given through the imagenet.preprocessing.index_to_label method.

import csv

# Parse labels file
fname = os.path.join(data_folder, 'labels_validation.txt')
validation_labels = dict()
with open(fname, newline='') as csvfile:
    reader = csv.reader(csvfile, delimiter=' ')
    for row in reader:
        validation_labels[row[0]] = row[1]

# Get labels for the test set by index
labels_test = np.zeros(num_images)
for i in range(num_images):
    labels_test[i] = int(validation_labels[x_test_files[i]])

2. Create a Keras AkidaNet model

The AkidaNet architecture is available in the Akida model zoo as akidanet_imagenet. In this tutorial, the alpha = 0.5 version of AkidaNet will be used, where alpha is the parameter controlling the width of the model.

from tensorflow.keras.models import load_model

# Retrieve the float model with pretrained weights and load it
model_file = get_file(
    "akidanet_imagenet_224_alpha_50.h5",
    "http://data.brainchip.com/models/akidanet/akidanet_imagenet_224_alpha_50.h5",
    cache_subdir='models/akidanet_imagenet')
model_keras = load_model(model_file)
model_keras.summary()

Out:

Downloading data from http://data.brainchip.com/models/akidanet/akidanet_imagenet_224_alpha_50.h5

  16384/5704248 [..............................] - ETA: 0s
 139264/5704248 [..............................] - ETA: 2s
 335872/5704248 [>.............................] - ETA: 2s
 532480/5704248 [=>............................] - ETA: 1s
 729088/5704248 [==>...........................] - ETA: 1s
 925696/5704248 [===>..........................] - ETA: 1s
1122304/5704248 [====>.........................] - ETA: 1s
1318912/5704248 [=====>........................] - ETA: 1s
1515520/5704248 [======>.......................] - ETA: 1s
1712128/5704248 [========>.....................] - ETA: 1s
1908736/5704248 [=========>....................] - ETA: 1s
2105344/5704248 [==========>...................] - ETA: 1s
2301952/5704248 [===========>..................] - ETA: 1s
2498560/5704248 [============>.................] - ETA: 1s
2695168/5704248 [=============>................] - ETA: 0s
2891776/5704248 [==============>...............] - ETA: 0s
3088384/5704248 [===============>..............] - ETA: 0s
3284992/5704248 [================>.............] - ETA: 0s
3481600/5704248 [=================>............] - ETA: 0s
3678208/5704248 [==================>...........] - ETA: 0s
3874816/5704248 [===================>..........] - ETA: 0s
4071424/5704248 [====================>.........] - ETA: 0s
4268032/5704248 [=====================>........] - ETA: 0s
4464640/5704248 [======================>.......] - ETA: 0s
4661248/5704248 [=======================>......] - ETA: 0s
4857856/5704248 [========================>.....] - ETA: 0s
5054464/5704248 [=========================>....] - ETA: 0s
5251072/5704248 [==========================>...] - ETA: 0s
5447680/5704248 [===========================>..] - ETA: 0s
5644288/5704248 [============================>.] - ETA: 0s
5709824/5704248 [==============================] - 2s 0us/step

5718016/5704248 [==============================] - 2s 0us/step
Model: "akidanet_0.50_224_1000"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 input_1 (InputLayer)        [(None, 224, 224, 3)]     0

 rescaling (Rescaling)       (None, 224, 224, 3)       0

 conv_0 (Conv2D)             (None, 112, 112, 16)      432

 conv_0_BN (BatchNormalizati  (None, 112, 112, 16)     64
 on)

 conv_0_relu (ReLU)          (None, 112, 112, 16)      0

 conv_1 (Conv2D)             (None, 112, 112, 32)      4608

 conv_1_BN (BatchNormalizati  (None, 112, 112, 32)     128
 on)

 conv_1_relu (ReLU)          (None, 112, 112, 32)      0

 conv_2 (Conv2D)             (None, 56, 56, 64)        18432

 conv_2_BN (BatchNormalizati  (None, 56, 56, 64)       256
 on)

 conv_2_relu (ReLU)          (None, 56, 56, 64)        0

 conv_3 (Conv2D)             (None, 56, 56, 64)        36864

 conv_3_BN (BatchNormalizati  (None, 56, 56, 64)       256
 on)

 conv_3_relu (ReLU)          (None, 56, 56, 64)        0

 separable_4 (SeparableConv2  (None, 28, 28, 128)      8768
 D)

 separable_4_BN (BatchNormal  (None, 28, 28, 128)      512
 ization)

 separable_4_relu (ReLU)     (None, 28, 28, 128)       0

 separable_5 (SeparableConv2  (None, 28, 28, 128)      17536
 D)

 separable_5_BN (BatchNormal  (None, 28, 28, 128)      512
 ization)

 separable_5_relu (ReLU)     (None, 28, 28, 128)       0

 separable_6 (SeparableConv2  (None, 14, 14, 256)      33920
 D)

 separable_6_BN (BatchNormal  (None, 14, 14, 256)      1024
 ization)

 separable_6_relu (ReLU)     (None, 14, 14, 256)       0

 separable_7 (SeparableConv2  (None, 14, 14, 256)      67840
 D)

 separable_7_BN (BatchNormal  (None, 14, 14, 256)      1024
 ization)

 separable_7_relu (ReLU)     (None, 14, 14, 256)       0

 separable_8 (SeparableConv2  (None, 14, 14, 256)      67840
 D)

 separable_8_BN (BatchNormal  (None, 14, 14, 256)      1024
 ization)

 separable_8_relu (ReLU)     (None, 14, 14, 256)       0

 separable_9 (SeparableConv2  (None, 14, 14, 256)      67840
 D)

 separable_9_BN (BatchNormal  (None, 14, 14, 256)      1024
 ization)

 separable_9_relu (ReLU)     (None, 14, 14, 256)       0

 separable_10 (SeparableConv  (None, 14, 14, 256)      67840
 2D)

 separable_10_BN (BatchNorma  (None, 14, 14, 256)      1024
 lization)

 separable_10_relu (ReLU)    (None, 14, 14, 256)       0

 separable_11 (SeparableConv  (None, 14, 14, 256)      67840
 2D)

 separable_11_BN (BatchNorma  (None, 14, 14, 256)      1024
 lization)

 separable_11_relu (ReLU)    (None, 14, 14, 256)       0

 separable_12 (SeparableConv  (None, 7, 7, 512)        133376
 2D)

 separable_12_BN (BatchNorma  (None, 7, 7, 512)        2048
 lization)

 separable_12_relu (ReLU)    (None, 7, 7, 512)         0

 separable_13 (SeparableConv  (None, 7, 7, 512)        266752
 2D)

 separable_13_global_avg (Gl  (None, 512)              0
 obalAveragePooling2D)

 separable_13_BN (BatchNorma  (None, 512)              2048
 lization)

 separable_13_relu (ReLU)    (None, 512)               0

 reshape_1 (Reshape)         (None, 1, 1, 512)         0

 dropout (Dropout)           (None, 1, 1, 512)         0

 separable_14 (SeparableConv  (None, 1, 1, 1000)       516608
 2D)

 act_softmax (Activation)    (None, 1, 1, 1000)        0

 reshape_2 (Reshape)         (None, 1000)              0

=================================================================
Total params: 1,388,464
Trainable params: 1,382,480
Non-trainable params: 5,984
_________________________________________________________________

Top-1 accuracy on the actual ImageNet is 64.58%, the perfomance given below uses the 10 images subset.

from timeit import default_timer as timer


# Check model performance
def check_model_performances(model, x_test=x_test, labels_test=labels_test):
    num_images = len(x_test)

    start = timer()
    potentials_keras = model.predict(x_test, batch_size=100)
    end = timer()
    print(f'Keras inference on {num_images} images took {end-start:.2f} s.\n')

    preds_keras = np.squeeze(np.argmax(potentials_keras, 1))
    accuracy_keras = np.sum(np.equal(preds_keras, labels_test)) / num_images

    print(f"Keras accuracy: {accuracy_keras*100:.2f} %")


check_model_performances(model_keras)

Out:

Keras inference on 10 images took 0.38 s.

Keras accuracy: 100.00 %

3. Quantized model

Quantizing a model is done using cnn2snn.quantize.

The quantized model satisfies the Akida NSoC requirements:

  • the first layer has 8-bit weights,

  • all other convolutional layers have 4-bit weights,

  • all convolutional layers have 4-bit activations.

However, this model will suffer from a drop in accuracy due to quantization as shown in the table below for ImageNet and in the next cell for the 10 images set.

Float accuracy

Quantized accuracy

64.58 %

1.00 %

from cnn2snn import quantize

# Quantize the model to 4-bit weights and activations, 8-bit weights for the
# first convolutional layer
model_keras_quantized = quantize(model_keras, 4, 4, 8)

# Check Model performance
check_model_performances(model_keras_quantized)

Out:

Keras inference on 10 images took 0.53 s.

Keras accuracy: 0.00 %

4. Pretrained quantized model

The Akida models zoo also contains a pretrained quantized helper that was obtained after fine tuning the model for 10 epochs.

Tuning the model, that is training with a lowered learning rate, allows to recover performances up to the initial floating point accuracy.

Performances on the full ImageNet dataset are:

Float accuracy

Quantized accuracy

After tuning

64.58 %

1.00 %

61.30 %

from akida_models import akidanet_imagenet_pretrained

# Use a quantized model with pretrained quantized weights
model_keras_quantized_pretrained = akidanet_imagenet_pretrained(0.5)
model_keras_quantized_pretrained.summary()

Out:

Downloading data from http://data.brainchip.com/models/akidanet/akidanet_imagenet_224_alpha_50_iq8_wq4_aq4.h5

  16384/5613320 [..............................] - ETA: 0s
 139264/5613320 [..............................] - ETA: 2s
 335872/5613320 [>.............................] - ETA: 1s
 532480/5613320 [=>............................] - ETA: 1s
 729088/5613320 [==>...........................] - ETA: 1s
 925696/5613320 [===>..........................] - ETA: 1s
1122304/5613320 [====>.........................] - ETA: 1s
1318912/5613320 [======>.......................] - ETA: 1s
1515520/5613320 [=======>......................] - ETA: 1s
1712128/5613320 [========>.....................] - ETA: 1s
1908736/5613320 [=========>....................] - ETA: 1s
2105344/5613320 [==========>...................] - ETA: 1s
2301952/5613320 [===========>..................] - ETA: 1s
2498560/5613320 [============>.................] - ETA: 0s
2695168/5613320 [=============>................] - ETA: 0s
2891776/5613320 [==============>...............] - ETA: 0s
3088384/5613320 [===============>..............] - ETA: 0s
3284992/5613320 [================>.............] - ETA: 0s
3481600/5613320 [=================>............] - ETA: 0s
3678208/5613320 [==================>...........] - ETA: 0s
3874816/5613320 [===================>..........] - ETA: 0s
4071424/5613320 [====================>.........] - ETA: 0s
4268032/5613320 [=====================>........] - ETA: 0s
4464640/5613320 [======================>.......] - ETA: 0s
4661248/5613320 [=======================>......] - ETA: 0s
4800512/5613320 [========================>.....] - ETA: 0s
4898816/5613320 [=========================>....] - ETA: 0s
5054464/5613320 [==========================>...] - ETA: 0s
5251072/5613320 [===========================>..] - ETA: 0s
5390336/5613320 [===========================>..] - ETA: 0s
5398528/5613320 [===========================>..] - ETA: 0s
5488640/5613320 [============================>.] - ETA: 0s
5619712/5613320 [==============================] - 3s 0us/step

5627904/5613320 [==============================] - 3s 0us/step
Model: "akidanet_0.50_224_1000"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 input_1 (InputLayer)        [(None, 224, 224, 3)]     0

 rescaling (Rescaling)       (None, 224, 224, 3)       0

 conv_0 (QuantizedConv2D)    (None, 112, 112, 16)      448

 conv_0_relu (ActivationDisc  (None, 112, 112, 16)     0
 reteRelu)

 conv_1 (QuantizedConv2D)    (None, 112, 112, 32)      4640

 conv_1_relu (ActivationDisc  (None, 112, 112, 32)     0
 reteRelu)

 conv_2 (QuantizedConv2D)    (None, 56, 56, 64)        18496

 conv_2_relu (ActivationDisc  (None, 56, 56, 64)       0
 reteRelu)

 conv_3 (QuantizedConv2D)    (None, 56, 56, 64)        36928

 conv_3_relu (ActivationDisc  (None, 56, 56, 64)       0
 reteRelu)

 separable_4 (QuantizedSepar  (None, 28, 28, 128)      8896
 ableConv2D)

 separable_4_relu (Activatio  (None, 28, 28, 128)      0
 nDiscreteRelu)

 separable_5 (QuantizedSepar  (None, 28, 28, 128)      17664
 ableConv2D)

 separable_5_relu (Activatio  (None, 28, 28, 128)      0
 nDiscreteRelu)

 separable_6 (QuantizedSepar  (None, 14, 14, 256)      34176
 ableConv2D)

 separable_6_relu (Activatio  (None, 14, 14, 256)      0
 nDiscreteRelu)

 separable_7 (QuantizedSepar  (None, 14, 14, 256)      68096
 ableConv2D)

 separable_7_relu (Activatio  (None, 14, 14, 256)      0
 nDiscreteRelu)

 separable_8 (QuantizedSepar  (None, 14, 14, 256)      68096
 ableConv2D)

 separable_8_relu (Activatio  (None, 14, 14, 256)      0
 nDiscreteRelu)

 separable_9 (QuantizedSepar  (None, 14, 14, 256)      68096
 ableConv2D)

 separable_9_relu (Activatio  (None, 14, 14, 256)      0
 nDiscreteRelu)

 separable_10 (QuantizedSepa  (None, 14, 14, 256)      68096
 rableConv2D)

 separable_10_relu (Activati  (None, 14, 14, 256)      0
 onDiscreteRelu)

 separable_11 (QuantizedSepa  (None, 14, 14, 256)      68096
 rableConv2D)

 separable_11_relu (Activati  (None, 14, 14, 256)      0
 onDiscreteRelu)

 separable_12 (QuantizedSepa  (None, 7, 7, 512)        133888
 rableConv2D)

 separable_12_relu (Activati  (None, 7, 7, 512)        0
 onDiscreteRelu)

 separable_13 (QuantizedSepa  (None, 7, 7, 512)        267264
 rableConv2D)

 separable_13_global_avg (Gl  (None, 512)              0
 obalAveragePooling2D)

 separable_13_relu (Activati  (None, 512)              0
 onDiscreteRelu)

 reshape_1 (Reshape)         (None, 1, 1, 512)         0

 dropout (Dropout)           (None, 1, 1, 512)         0

 separable_14 (QuantizedSepa  (None, 1, 1, 1000)       516608
 rableConv2D)

 act_softmax (Activation)    (None, 1, 1, 1000)        0

 reshape_2 (Reshape)         (None, 1000)              0

=================================================================
Total params: 1,379,488
Trainable params: 1,379,488
Non-trainable params: 0
_________________________________________________________________
# Check model performance
check_model_performances(model_keras_quantized_pretrained)

Out:

Keras inference on 10 images took 0.54 s.

Keras accuracy: 90.00 %

5. Conversion to Akida

5.1 Convert to Akida model

Here, the Keras quantized model is converted into a suitable version for the Akida NSoC. The cnn2snn.convert function needs as arguments the Keras model and the input scaling parameters.

from cnn2snn import convert

model_akida = convert(model_keras_quantized_pretrained)

The Model.summary method provides a detailed description of the Model layers.

model_akida.summary()

Out:

                 Model Summary
________________________________________________
Input shape    Output shape  Sequences  Layers
================================================
[224, 224, 3]  [1, 1, 1000]  1          15
________________________________________________

              SW/conv_0-separable_14 (Software)
_____________________________________________________________
Layer (type)              Output shape    Kernel shape
=============================================================
conv_0 (InputConv.)       [112, 112, 16]  (3, 3, 3, 16)
_____________________________________________________________
conv_1 (Conv.)            [112, 112, 32]  (3, 3, 16, 32)
_____________________________________________________________
conv_2 (Conv.)            [56, 56, 64]    (3, 3, 32, 64)
_____________________________________________________________
conv_3 (Conv.)            [56, 56, 64]    (3, 3, 64, 64)
_____________________________________________________________
separable_4 (Sep.Conv.)   [28, 28, 128]   (3, 3, 64, 1)
_____________________________________________________________
                                          (1, 1, 64, 128)
_____________________________________________________________
separable_5 (Sep.Conv.)   [28, 28, 128]   (3, 3, 128, 1)
_____________________________________________________________
                                          (1, 1, 128, 128)
_____________________________________________________________
separable_6 (Sep.Conv.)   [14, 14, 256]   (3, 3, 128, 1)
_____________________________________________________________
                                          (1, 1, 128, 256)
_____________________________________________________________
separable_7 (Sep.Conv.)   [14, 14, 256]   (3, 3, 256, 1)
_____________________________________________________________
                                          (1, 1, 256, 256)
_____________________________________________________________
separable_8 (Sep.Conv.)   [14, 14, 256]   (3, 3, 256, 1)
_____________________________________________________________
                                          (1, 1, 256, 256)
_____________________________________________________________
separable_9 (Sep.Conv.)   [14, 14, 256]   (3, 3, 256, 1)
_____________________________________________________________
                                          (1, 1, 256, 256)
_____________________________________________________________
separable_10 (Sep.Conv.)  [14, 14, 256]   (3, 3, 256, 1)
_____________________________________________________________
                                          (1, 1, 256, 256)
_____________________________________________________________
separable_11 (Sep.Conv.)  [14, 14, 256]   (3, 3, 256, 1)
_____________________________________________________________
                                          (1, 1, 256, 256)
_____________________________________________________________
separable_12 (Sep.Conv.)  [7, 7, 512]     (3, 3, 256, 1)
_____________________________________________________________
                                          (1, 1, 256, 512)
_____________________________________________________________
separable_13 (Sep.Conv.)  [1, 1, 512]     (3, 3, 512, 1)
_____________________________________________________________
                                          (1, 1, 512, 512)
_____________________________________________________________
separable_14 (Sep.Conv.)  [1, 1, 1000]    (3, 3, 512, 1)
_____________________________________________________________
                                          (1, 1, 512, 1000)
_____________________________________________________________

5.2 Check performance

While we compute accuracy for the 10 images set in the next cell, the following table summarizes results obtained on ImageNet.

Keras accuracy

Akida accuracy

61.30 %

61.37 %

# Check Model performance
start = timer()
accuracy_akida = model_akida.evaluate(x_test, labels_test)
end = timer()
print(f'Inference on {num_images} images took {end-start:.2f} s.\n')
print(f"Accuracy: {accuracy_akida*100:.2f} %")

# For non-regression purpose
assert accuracy_akida >= 0.89

Out:

Inference on 10 images took 0.30 s.

Accuracy: 90.00 %

5.3 Show predictions for a random image

import matplotlib.pyplot as plt
import matplotlib.lines as lines


# Functions used to display the top5 results
def get_top5(potentials, true_label):
    """
    Returns the top 5 classes from the output potentials
    """
    tmp_pots = potentials.copy()
    top5 = []
    min_val = np.min(tmp_pots)
    for ii in range(5):
        best = np.argmax(tmp_pots)
        top5.append(best)
        tmp_pots[best] = min_val

    vals = np.zeros((6,))
    vals[:5] = potentials[top5]
    if true_label not in top5:
        vals[5] = potentials[true_label]
    else:
        vals[5] = 0
    vals /= np.max(vals)

    class_name = []
    for ii in range(5):
        class_name.append(preprocessing.index_to_label(top5[ii]).split(',')[0])
    if true_label in top5:
        class_name.append('')
    else:
        class_name.append(
            preprocessing.index_to_label(true_label).split(',')[0])

    return top5, vals, class_name


def adjust_spines(ax, spines):
    for loc, spine in ax.spines.items():
        if loc in spines:
            spine.set_position(('outward', 10))  # outward by 10 points
        else:
            spine.set_color('none')  # don't draw spine
    # turn off ticks where there is no spine
    if 'left' in spines:
        ax.yaxis.set_ticks_position('left')
    else:
        # no yaxis ticks
        ax.yaxis.set_ticks([])
    if 'bottom' in spines:
        ax.xaxis.set_ticks_position('bottom')
    else:
        # no xaxis ticks
        ax.xaxis.set_ticks([])


def prepare_plots():
    fig = plt.figure(figsize=(8, 4))
    # Image subplot
    ax0 = plt.subplot(1, 3, 1)
    imgobj = ax0.imshow(np.zeros((224, 224, 3), dtype=np.uint8))
    ax0.set_axis_off()
    # Top 5 results subplot
    ax1 = plt.subplot(1, 2, 2)
    bar_positions = (0, 1, 2, 3, 4, 6)
    rects = ax1.barh(bar_positions, np.zeros((6,)), align='center', height=0.5)
    plt.xlim(-0.2, 1.01)
    ax1.set(xlim=(-0.2, 1.15), ylim=(-1.5, 12))
    ax1.set_yticks(bar_positions)
    ax1.invert_yaxis()
    ax1.yaxis.set_ticks_position('left')
    ax1.xaxis.set_ticks([])
    adjust_spines(ax1, 'left')
    ax1.add_line(lines.Line2D((0, 0), (-0.5, 6.5), color=(0.0, 0.0, 0.0)))
    # Adjust Plot Positions
    ax0.set_position([0.05, 0.055, 0.3, 0.9])
    l1, b1, w1, h1 = ax1.get_position().bounds
    ax1.set_position([l1 * 1.05, b1 + 0.09 * h1, w1, 0.8 * h1])
    # Add title box
    plt.figtext(0.5,
                0.9,
                "Imagenet Classification by Akida",
                size=20,
                ha="center",
                va="center",
                bbox=dict(boxstyle="round",
                          ec=(0.5, 0.5, 0.5),
                          fc=(0.9, 0.9, 1.0)))

    return fig, imgobj, ax1, rects


def update_bars_chart(rects, vals, true_label):
    counter = 0
    for rect, h in zip(rects, yvals):
        rect.set_width(h)
        if counter < 5:
            if top5[counter] == true_label:
                if counter == 0:
                    rect.set_facecolor((0.0, 1.0, 0.0))
                else:
                    rect.set_facecolor((0.0, 0.5, 0.0))
            else:
                rect.set_facecolor('gray')
        elif counter == 5:
            rect.set_facecolor('red')
        counter += 1


# Prepare plots
fig, imgobj, ax1, rects = prepare_plots()

# Get a random image
img = np.random.randint(num_images)

# Predict image class
potentials_akida = model_akida.predict(np.expand_dims(x_test[img],
                                                      axis=0)).squeeze()

# Get top 5 prediction labels and associated names
true_label = int(validation_labels[x_test_files[img]])
top5, yvals, class_name = get_top5(potentials_akida, true_label)

# Draw Plots
imgobj.set_data(x_test[img])
ax1.set_yticklabels(class_name, rotation='horizontal', size=9)
update_bars_chart(rects, yvals, true_label)
fig.canvas.draw()
plt.show()
plot 1 akidanet imagenet

6. Hardware mapping and performance

6.1. Map on hardware

List Akida available devices and check that an NSoC V2 (production chip) is available

devices = akida.devices()
print(f'Available devices: {[dev.desc for dev in devices]}')
device = devices[0]
assert device.version == akida.NSoC_v2

Out:

Available devices: ['PCIe/NSoC_v2/0']

Map the model on the device

model_akida.map(device)

# Check model mapping: NP allocation and binary size
model_akida.summary()

Out:

                 Model Summary
________________________________________________
Input shape    Output shape  Sequences  Layers
================================================
[224, 224, 3]  [1, 1, 1000]  1          15
________________________________________________

     HW/conv_0-separable_14 (Hardware) - size: 1241424 bytes
__________________________________________________________________
Layer (type)              Output shape    Kernel shape       NPs
==================================================================
conv_0 (InputConv.)       [112, 112, 16]  (3, 3, 3, 16)      N/A
__________________________________________________________________
conv_1 (Conv.)            [112, 112, 32]  (3, 3, 16, 32)     4
__________________________________________________________________
conv_2 (Conv.)            [56, 56, 64]    (3, 3, 32, 64)     6
__________________________________________________________________
conv_3 (Conv.)            [56, 56, 64]    (3, 3, 64, 64)     3
__________________________________________________________________
separable_4 (Sep.Conv.)   [28, 28, 128]   (3, 3, 64, 1)      3
__________________________________________________________________
                                          (1, 1, 64, 128)
__________________________________________________________________
separable_5 (Sep.Conv.)   [28, 28, 128]   (3, 3, 128, 1)     2
__________________________________________________________________
                                          (1, 1, 128, 128)
__________________________________________________________________
separable_6 (Sep.Conv.)   [14, 14, 256]   (3, 3, 128, 1)     2
__________________________________________________________________
                                          (1, 1, 128, 256)
__________________________________________________________________
separable_7 (Sep.Conv.)   [14, 14, 256]   (3, 3, 256, 1)     1
__________________________________________________________________
                                          (1, 1, 256, 256)
__________________________________________________________________
separable_8 (Sep.Conv.)   [14, 14, 256]   (3, 3, 256, 1)     1
__________________________________________________________________
                                          (1, 1, 256, 256)
__________________________________________________________________
separable_9 (Sep.Conv.)   [14, 14, 256]   (3, 3, 256, 1)     1
__________________________________________________________________
                                          (1, 1, 256, 256)
__________________________________________________________________
separable_10 (Sep.Conv.)  [14, 14, 256]   (3, 3, 256, 1)     1
__________________________________________________________________
                                          (1, 1, 256, 256)
__________________________________________________________________
separable_11 (Sep.Conv.)  [14, 14, 256]   (3, 3, 256, 1)     1
__________________________________________________________________
                                          (1, 1, 256, 256)
__________________________________________________________________
separable_12 (Sep.Conv.)  [7, 7, 512]     (3, 3, 256, 1)     2
__________________________________________________________________
                                          (1, 1, 256, 512)
__________________________________________________________________
separable_13 (Sep.Conv.)  [1, 1, 512]     (3, 3, 512, 1)     4
__________________________________________________________________
                                          (1, 1, 512, 512)
__________________________________________________________________
separable_14 (Sep.Conv.)  [1, 1, 1000]    (3, 3, 512, 1)     7
__________________________________________________________________
                                          (1, 1, 512, 1000)
__________________________________________________________________

6.2. Performances measurement

Power measurement must be enabled on the device’ soc (disabled by default). After sending data for inference, performances measurements are available in the model statistics.

# Enable power measurement
device.soc.power_measurement_enabled = True

# Send data for inference
_ = model_akida.forward(x_test)

# Display floor current
floor_power = device.soc.power_meter.floor
print(f'Floor power: {floor_power:.2f} mW')

# Retrieve statistics
print(model_akida.statistics)

Out:

Floor power: 889.42 mW
Average framerate = 21.32 fps
Last inference power range (mW):  Avg 1017.67 / Min 889.00 / Max 1116.00 / Std 92.18
Last inference energy consumed (mJ/frame): 47.73

Total running time of the script: ( 0 minutes 19.616 seconds)

Gallery generated by Sphinx-Gallery