DS-CNN/KWS inference

This tutorial illustrates how to build a basic speech recognition Akida network that recognizes thirty-two different words.

The model will be first defined as a CNN and trained in Keras, then converted using the CNN2SNN toolkit.

This example uses a Keyword Spotting Dataset prepared using TensorFlow audio recognition example utils.

The words to recognize are first converted to spectrogram images that allows us to use a model architecture that is typically used for image recognition tasks.

1. Load the preprocessed dataset

The TensorFlow speech_commands dataset is used for training and validation. All keywords except “backward”, “follow” and “forward”, are retrieved. These three words are kept to illustrate the edge learning in this edge example. The data are not directly used for training. They are preprocessed, transforming the audio files into MFCC features, well-suited for CNN networks. A pickle file containing the preprocessed data is available on our data server.

import pickle

from tensorflow.keras.utils import get_file

# Fetch pre-processed data for 32 keywords
fname = get_file(
    fname='kws_preprocessed_all_words_except_backward_follow_forward.pkl',
    origin="http://data.brainchip.com/dataset-mirror/kws/kws_preprocessed_all_words_except_backward_follow_forward.pkl",
    cache_subdir='datasets/kws')
with open(fname, 'rb') as f:
    [_, _, x_valid, y_valid, _, _, word_to_index, _] = pickle.load(f)

# Preprocessed dataset parameters
num_classes = len(word_to_index)

print("Wanted words and labels:\n", word_to_index)

Out:

Downloading data from http://data.brainchip.com/dataset-mirror/kws/kws_preprocessed_all_words_except_backward_follow_forward.pkl

   16384/62628765 [..............................] - ETA: 3s
  139264/62628765 [..............................] - ETA: 28s
  335872/62628765 [..............................] - ETA: 23s
  532480/62628765 [..............................] - ETA: 21s
  729088/62628765 [..............................] - ETA: 20s
  925696/62628765 [..............................] - ETA: 20s
 1122304/62628765 [..............................] - ETA: 19s
 1318912/62628765 [..............................] - ETA: 19s
 1515520/62628765 [..............................] - ETA: 19s
 1712128/62628765 [..............................] - ETA: 19s
 1908736/62628765 [..............................] - ETA: 19s
 2105344/62628765 [>.............................] - ETA: 18s
 2301952/62628765 [>.............................] - ETA: 18s
 2498560/62628765 [>.............................] - ETA: 18s
 2695168/62628765 [>.............................] - ETA: 18s
 2891776/62628765 [>.............................] - ETA: 18s
 3088384/62628765 [>.............................] - ETA: 18s
 3284992/62628765 [>.............................] - ETA: 18s
 3481600/62628765 [>.............................] - ETA: 18s
 3678208/62628765 [>.............................] - ETA: 18s
 3874816/62628765 [>.............................] - ETA: 18s
 4071424/62628765 [>.............................] - ETA: 18s
 4268032/62628765 [=>............................] - ETA: 17s
 4464640/62628765 [=>............................] - ETA: 17s
 4661248/62628765 [=>............................] - ETA: 17s
 4857856/62628765 [=>............................] - ETA: 17s
 5054464/62628765 [=>............................] - ETA: 17s
 5251072/62628765 [=>............................] - ETA: 17s
 5447680/62628765 [=>............................] - ETA: 17s
 5644288/62628765 [=>............................] - ETA: 17s
 5840896/62628765 [=>............................] - ETA: 17s
 6037504/62628765 [=>............................] - ETA: 17s
 6234112/62628765 [=>............................] - ETA: 17s
 6430720/62628765 [==>...........................] - ETA: 17s
 6627328/62628765 [==>...........................] - ETA: 17s
 6823936/62628765 [==>...........................] - ETA: 17s
 7020544/62628765 [==>...........................] - ETA: 16s
 7217152/62628765 [==>...........................] - ETA: 16s
 7413760/62628765 [==>...........................] - ETA: 16s
 7610368/62628765 [==>...........................] - ETA: 16s
 7806976/62628765 [==>...........................] - ETA: 16s
 8003584/62628765 [==>...........................] - ETA: 16s
 8200192/62628765 [==>...........................] - ETA: 16s
 8396800/62628765 [===>..........................] - ETA: 16s
 8593408/62628765 [===>..........................] - ETA: 16s
 8790016/62628765 [===>..........................] - ETA: 16s
 8986624/62628765 [===>..........................] - ETA: 16s
 9183232/62628765 [===>..........................] - ETA: 16s
 9379840/62628765 [===>..........................] - ETA: 16s
 9576448/62628765 [===>..........................] - ETA: 16s
 9773056/62628765 [===>..........................] - ETA: 16s
 9969664/62628765 [===>..........................] - ETA: 16s
10166272/62628765 [===>..........................] - ETA: 15s
10362880/62628765 [===>..........................] - ETA: 15s
10559488/62628765 [====>.........................] - ETA: 15s
10756096/62628765 [====>.........................] - ETA: 15s
10952704/62628765 [====>.........................] - ETA: 15s
11149312/62628765 [====>.........................] - ETA: 15s
11345920/62628765 [====>.........................] - ETA: 15s
11542528/62628765 [====>.........................] - ETA: 15s
11739136/62628765 [====>.........................] - ETA: 15s
11935744/62628765 [====>.........................] - ETA: 15s
12132352/62628765 [====>.........................] - ETA: 15s
12328960/62628765 [====>.........................] - ETA: 15s
12525568/62628765 [====>.........................] - ETA: 15s
12722176/62628765 [=====>........................] - ETA: 15s
12918784/62628765 [=====>........................] - ETA: 15s
13115392/62628765 [=====>........................] - ETA: 15s
13312000/62628765 [=====>........................] - ETA: 15s
13508608/62628765 [=====>........................] - ETA: 14s
13705216/62628765 [=====>........................] - ETA: 14s
13901824/62628765 [=====>........................] - ETA: 14s
14098432/62628765 [=====>........................] - ETA: 14s
14295040/62628765 [=====>........................] - ETA: 14s
14491648/62628765 [=====>........................] - ETA: 14s
14688256/62628765 [======>.......................] - ETA: 14s
14884864/62628765 [======>.......................] - ETA: 14s
15081472/62628765 [======>.......................] - ETA: 14s
15278080/62628765 [======>.......................] - ETA: 14s
15474688/62628765 [======>.......................] - ETA: 14s
15671296/62628765 [======>.......................] - ETA: 14s
15867904/62628765 [======>.......................] - ETA: 14s
16064512/62628765 [======>.......................] - ETA: 14s
16261120/62628765 [======>.......................] - ETA: 14s
16457728/62628765 [======>.......................] - ETA: 14s
16654336/62628765 [======>.......................] - ETA: 13s
16850944/62628765 [=======>......................] - ETA: 13s
17047552/62628765 [=======>......................] - ETA: 13s
17244160/62628765 [=======>......................] - ETA: 13s
17440768/62628765 [=======>......................] - ETA: 13s
17637376/62628765 [=======>......................] - ETA: 13s
17833984/62628765 [=======>......................] - ETA: 13s
18030592/62628765 [=======>......................] - ETA: 13s
18227200/62628765 [=======>......................] - ETA: 13s
18423808/62628765 [=======>......................] - ETA: 13s
18620416/62628765 [=======>......................] - ETA: 13s
18817024/62628765 [========>.....................] - ETA: 13s
19013632/62628765 [========>.....................] - ETA: 13s
19210240/62628765 [========>.....................] - ETA: 13s
19406848/62628765 [========>.....................] - ETA: 13s
19603456/62628765 [========>.....................] - ETA: 13s
19800064/62628765 [========>.....................] - ETA: 13s
19996672/62628765 [========>.....................] - ETA: 12s
20193280/62628765 [========>.....................] - ETA: 12s
20389888/62628765 [========>.....................] - ETA: 12s
20586496/62628765 [========>.....................] - ETA: 12s
20783104/62628765 [========>.....................] - ETA: 12s
20979712/62628765 [=========>....................] - ETA: 12s
21176320/62628765 [=========>....................] - ETA: 12s
21372928/62628765 [=========>....................] - ETA: 12s
21569536/62628765 [=========>....................] - ETA: 12s
21766144/62628765 [=========>....................] - ETA: 12s
21962752/62628765 [=========>....................] - ETA: 12s
22159360/62628765 [=========>....................] - ETA: 12s
22355968/62628765 [=========>....................] - ETA: 12s
22552576/62628765 [=========>....................] - ETA: 12s
22749184/62628765 [=========>....................] - ETA: 12s
22945792/62628765 [=========>....................] - ETA: 12s
23142400/62628765 [==========>...................] - ETA: 11s
23339008/62628765 [==========>...................] - ETA: 11s
23535616/62628765 [==========>...................] - ETA: 11s
23732224/62628765 [==========>...................] - ETA: 11s
23928832/62628765 [==========>...................] - ETA: 11s
24125440/62628765 [==========>...................] - ETA: 11s
24322048/62628765 [==========>...................] - ETA: 11s
24518656/62628765 [==========>...................] - ETA: 11s
24715264/62628765 [==========>...................] - ETA: 11s
24911872/62628765 [==========>...................] - ETA: 11s
25108480/62628765 [===========>..................] - ETA: 11s
25305088/62628765 [===========>..................] - ETA: 11s
25501696/62628765 [===========>..................] - ETA: 11s
25698304/62628765 [===========>..................] - ETA: 11s
25894912/62628765 [===========>..................] - ETA: 11s
26091520/62628765 [===========>..................] - ETA: 11s
26288128/62628765 [===========>..................] - ETA: 11s
26484736/62628765 [===========>..................] - ETA: 10s
26681344/62628765 [===========>..................] - ETA: 10s
26877952/62628765 [===========>..................] - ETA: 10s
27074560/62628765 [===========>..................] - ETA: 10s
27271168/62628765 [============>.................] - ETA: 10s
27467776/62628765 [============>.................] - ETA: 10s
27664384/62628765 [============>.................] - ETA: 10s
27860992/62628765 [============>.................] - ETA: 10s
28057600/62628765 [============>.................] - ETA: 10s
28254208/62628765 [============>.................] - ETA: 10s
28450816/62628765 [============>.................] - ETA: 10s
28647424/62628765 [============>.................] - ETA: 10s
28844032/62628765 [============>.................] - ETA: 10s
29040640/62628765 [============>.................] - ETA: 10s
29237248/62628765 [=============>................] - ETA: 10s
29433856/62628765 [=============>................] - ETA: 10s
29630464/62628765 [=============>................] - ETA: 10s
29827072/62628765 [=============>................] - ETA: 9s 
30023680/62628765 [=============>................] - ETA: 9s
30220288/62628765 [=============>................] - ETA: 9s
30416896/62628765 [=============>................] - ETA: 9s
30613504/62628765 [=============>................] - ETA: 9s
30810112/62628765 [=============>................] - ETA: 9s
31006720/62628765 [=============>................] - ETA: 9s
31203328/62628765 [=============>................] - ETA: 9s
31399936/62628765 [==============>...............] - ETA: 9s
31596544/62628765 [==============>...............] - ETA: 9s
31793152/62628765 [==============>...............] - ETA: 9s
31989760/62628765 [==============>...............] - ETA: 9s
32186368/62628765 [==============>...............] - ETA: 9s
32382976/62628765 [==============>...............] - ETA: 9s
32579584/62628765 [==============>...............] - ETA: 9s
32776192/62628765 [==============>...............] - ETA: 9s
32972800/62628765 [==============>...............] - ETA: 8s
33169408/62628765 [==============>...............] - ETA: 8s
33366016/62628765 [==============>...............] - ETA: 8s
33562624/62628765 [===============>..............] - ETA: 8s
33759232/62628765 [===============>..............] - ETA: 8s
33955840/62628765 [===============>..............] - ETA: 8s
34152448/62628765 [===============>..............] - ETA: 8s
34349056/62628765 [===============>..............] - ETA: 8s
34471936/62628765 [===============>..............] - ETA: 8s
34611200/62628765 [===============>..............] - ETA: 8s
34807808/62628765 [===============>..............] - ETA: 8s
35004416/62628765 [===============>..............] - ETA: 8s
35201024/62628765 [===============>..............] - ETA: 8s
35397632/62628765 [===============>..............] - ETA: 8s
35594240/62628765 [================>.............] - ETA: 8s
35790848/62628765 [================>.............] - ETA: 8s
35987456/62628765 [================>.............] - ETA: 8s
36184064/62628765 [================>.............] - ETA: 8s
36380672/62628765 [================>.............] - ETA: 7s
36577280/62628765 [================>.............] - ETA: 7s
36773888/62628765 [================>.............] - ETA: 7s
36970496/62628765 [================>.............] - ETA: 7s
37167104/62628765 [================>.............] - ETA: 7s
37363712/62628765 [================>.............] - ETA: 7s
37560320/62628765 [================>.............] - ETA: 7s
37756928/62628765 [=================>............] - ETA: 7s
37953536/62628765 [=================>............] - ETA: 7s
38150144/62628765 [=================>............] - ETA: 7s
38346752/62628765 [=================>............] - ETA: 7s
38543360/62628765 [=================>............] - ETA: 7s
38739968/62628765 [=================>............] - ETA: 7s
38936576/62628765 [=================>............] - ETA: 7s
39133184/62628765 [=================>............] - ETA: 7s
39329792/62628765 [=================>............] - ETA: 7s
39526400/62628765 [=================>............] - ETA: 7s
39723008/62628765 [==================>...........] - ETA: 6s
39919616/62628765 [==================>...........] - ETA: 6s
40116224/62628765 [==================>...........] - ETA: 6s
40312832/62628765 [==================>...........] - ETA: 6s
40509440/62628765 [==================>...........] - ETA: 6s
40706048/62628765 [==================>...........] - ETA: 6s
40902656/62628765 [==================>...........] - ETA: 6s
41099264/62628765 [==================>...........] - ETA: 6s
41295872/62628765 [==================>...........] - ETA: 6s
41492480/62628765 [==================>...........] - ETA: 6s
41689088/62628765 [==================>...........] - ETA: 6s
41885696/62628765 [===================>..........] - ETA: 6s
42082304/62628765 [===================>..........] - ETA: 6s
42278912/62628765 [===================>..........] - ETA: 6s
42475520/62628765 [===================>..........] - ETA: 6s
42672128/62628765 [===================>..........] - ETA: 6s
42868736/62628765 [===================>..........] - ETA: 6s
43065344/62628765 [===================>..........] - ETA: 5s
43261952/62628765 [===================>..........] - ETA: 5s
43458560/62628765 [===================>..........] - ETA: 5s
43655168/62628765 [===================>..........] - ETA: 5s
43851776/62628765 [====================>.........] - ETA: 5s
44048384/62628765 [====================>.........] - ETA: 5s
44244992/62628765 [====================>.........] - ETA: 5s
44441600/62628765 [====================>.........] - ETA: 5s
44638208/62628765 [====================>.........] - ETA: 5s
44834816/62628765 [====================>.........] - ETA: 5s
45031424/62628765 [====================>.........] - ETA: 5s
45228032/62628765 [====================>.........] - ETA: 5s
45424640/62628765 [====================>.........] - ETA: 5s
45621248/62628765 [====================>.........] - ETA: 5s
45817856/62628765 [====================>.........] - ETA: 5s
46014464/62628765 [=====================>........] - ETA: 5s
46211072/62628765 [=====================>........] - ETA: 4s
46407680/62628765 [=====================>........] - ETA: 4s
46604288/62628765 [=====================>........] - ETA: 4s
46800896/62628765 [=====================>........] - ETA: 4s
46997504/62628765 [=====================>........] - ETA: 4s
47169536/62628765 [=====================>........] - ETA: 4s
47325184/62628765 [=====================>........] - ETA: 4s
47521792/62628765 [=====================>........] - ETA: 4s
47718400/62628765 [=====================>........] - ETA: 4s
47915008/62628765 [=====================>........] - ETA: 4s
48111616/62628765 [======================>.......] - ETA: 4s
48308224/62628765 [======================>.......] - ETA: 4s
48504832/62628765 [======================>.......] - ETA: 4s
48701440/62628765 [======================>.......] - ETA: 4s
48898048/62628765 [======================>.......] - ETA: 4s
49094656/62628765 [======================>.......] - ETA: 4s
49291264/62628765 [======================>.......] - ETA: 4s
49487872/62628765 [======================>.......] - ETA: 3s
49684480/62628765 [======================>.......] - ETA: 3s
49881088/62628765 [======================>.......] - ETA: 3s
50077696/62628765 [======================>.......] - ETA: 3s
50274304/62628765 [=======================>......] - ETA: 3s
50470912/62628765 [=======================>......] - ETA: 3s
50667520/62628765 [=======================>......] - ETA: 3s
50864128/62628765 [=======================>......] - ETA: 3s
51060736/62628765 [=======================>......] - ETA: 3s
51257344/62628765 [=======================>......] - ETA: 3s
51453952/62628765 [=======================>......] - ETA: 3s
51650560/62628765 [=======================>......] - ETA: 3s
51847168/62628765 [=======================>......] - ETA: 3s
52043776/62628765 [=======================>......] - ETA: 3s
52240384/62628765 [========================>.....] - ETA: 3s
52436992/62628765 [========================>.....] - ETA: 3s
52633600/62628765 [========================>.....] - ETA: 3s
52830208/62628765 [========================>.....] - ETA: 2s
53026816/62628765 [========================>.....] - ETA: 2s
53223424/62628765 [========================>.....] - ETA: 2s
53420032/62628765 [========================>.....] - ETA: 2s
53616640/62628765 [========================>.....] - ETA: 2s
53813248/62628765 [========================>.....] - ETA: 2s
54009856/62628765 [========================>.....] - ETA: 2s
54206464/62628765 [========================>.....] - ETA: 2s
54403072/62628765 [=========================>....] - ETA: 2s
54599680/62628765 [=========================>....] - ETA: 2s
54796288/62628765 [=========================>....] - ETA: 2s
54992896/62628765 [=========================>....] - ETA: 2s
55189504/62628765 [=========================>....] - ETA: 2s
55386112/62628765 [=========================>....] - ETA: 2s
55582720/62628765 [=========================>....] - ETA: 2s
55779328/62628765 [=========================>....] - ETA: 2s
55975936/62628765 [=========================>....] - ETA: 2s
56172544/62628765 [=========================>....] - ETA: 1s
56369152/62628765 [==========================>...] - ETA: 1s
56565760/62628765 [==========================>...] - ETA: 1s
56762368/62628765 [==========================>...] - ETA: 1s
56958976/62628765 [==========================>...] - ETA: 1s
57155584/62628765 [==========================>...] - ETA: 1s
57352192/62628765 [==========================>...] - ETA: 1s
57548800/62628765 [==========================>...] - ETA: 1s
57745408/62628765 [==========================>...] - ETA: 1s
57942016/62628765 [==========================>...] - ETA: 1s
58138624/62628765 [==========================>...] - ETA: 1s
58335232/62628765 [==========================>...] - ETA: 1s
58531840/62628765 [===========================>..] - ETA: 1s
58728448/62628765 [===========================>..] - ETA: 1s
58925056/62628765 [===========================>..] - ETA: 1s
59121664/62628765 [===========================>..] - ETA: 1s
59318272/62628765 [===========================>..] - ETA: 1s
59514880/62628765 [===========================>..] - ETA: 0s
59711488/62628765 [===========================>..] - ETA: 0s
59908096/62628765 [===========================>..] - ETA: 0s
60104704/62628765 [===========================>..] - ETA: 0s
60301312/62628765 [===========================>..] - ETA: 0s
60497920/62628765 [===========================>..] - ETA: 0s
60694528/62628765 [============================>.] - ETA: 0s
60891136/62628765 [============================>.] - ETA: 0s
61087744/62628765 [============================>.] - ETA: 0s
61284352/62628765 [============================>.] - ETA: 0s
61480960/62628765 [============================>.] - ETA: 0s
61677568/62628765 [============================>.] - ETA: 0s
61874176/62628765 [============================>.] - ETA: 0s
62070784/62628765 [============================>.] - ETA: 0s
62267392/62628765 [============================>.] - ETA: 0s
62464000/62628765 [============================>.] - ETA: 0s
62636032/62628765 [==============================] - 19s 0us/step

62644224/62628765 [==============================] - 19s 0us/step
Wanted words and labels:
 {'six': 23, 'three': 25, 'seven': 21, 'bed': 1, 'eight': 6, 'yes': 31, 'cat': 3, 'on': 18, 'one': 19, 'stop': 24, 'two': 27, 'house': 11, 'five': 7, 'down': 5, 'four': 8, 'go': 9, 'up': 28, 'learn': 12, 'no': 16, 'bird': 2, 'zero': 32, 'nine': 15, 'visual': 29, 'wow': 30, 'sheila': 22, 'marvin': 14, 'off': 17, 'right': 20, 'left': 13, 'happy': 10, 'dog': 4, 'tree': 26, '_silence_': 0}

2. Load a pre-trained native Keras model

The model consists of:

  • a first convolutional layer accepting dense inputs (images),

  • several separable convolutional layers preserving spatial dimensions,

  • a global pooling reducing the spatial dimensions to a single pixel,

  • a last separable convolutional to reduce the number of outputs

  • a final fully connected layer to classify words

All layers are followed by a batch normalization and a ReLU activation.

This model was obtained with unconstrained float weights and activations after 16 epochs of training.

from tensorflow.keras.models import load_model

# Retrieve the model file from the BrainChip data server
model_file = get_file("ds_cnn_kws.h5",
                      "http://data.brainchip.com/models/ds_cnn/ds_cnn_kws.h5",
                      cache_subdir='models')

# Load the native Keras pre-trained model
model_keras = load_model(model_file)
model_keras.summary()

Out:

Downloading data from http://data.brainchip.com/models/ds_cnn/ds_cnn_kws.h5

 16384/158432 [==>...........................] - ETA: 0s
139264/158432 [=========================>....] - ETA: 0s
163840/158432 [===============================] - 0s 0us/step

172032/158432 [================================] - 0s 0us/step
Model: "ds_cnn_kws"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 input_1 (InputLayer)        [(None, 49, 10, 1)]       0

 rescaling (Rescaling)       (None, 49, 10, 1)         0

 conv_0 (Conv2D)             (None, 25, 5, 64)         1600

 conv_0_BN (BatchNormalizati  (None, 25, 5, 64)        256
 on)

 conv_0_relu (ReLU)          (None, 25, 5, 64)         0

 separable_1 (SeparableConv2  (None, 25, 5, 64)        4672
 D)

 separable_1_BN (BatchNormal  (None, 25, 5, 64)        256
 ization)

 separable_1_relu (ReLU)     (None, 25, 5, 64)         0

 separable_2 (SeparableConv2  (None, 25, 5, 64)        4672
 D)

 separable_2_BN (BatchNormal  (None, 25, 5, 64)        256
 ization)

 separable_2_relu (ReLU)     (None, 25, 5, 64)         0

 separable_3 (SeparableConv2  (None, 25, 5, 64)        4672
 D)

 separable_3_BN (BatchNormal  (None, 25, 5, 64)        256
 ization)

 separable_3_relu (ReLU)     (None, 25, 5, 64)         0

 separable_4 (SeparableConv2  (None, 25, 5, 64)        4672
 D)

 separable_4_global_avg (Glo  (None, 64)               0
 balAveragePooling2D)

 separable_4_BN (BatchNormal  (None, 64)               256
 ization)

 separable_4_relu (ReLU)     (None, 64)                0

 reshape_1 (Reshape)         (None, 1, 1, 64)          0

 flatten (Flatten)           (None, 64)                0

 dense_5 (Dense)             (None, 33)                2145

 act_softmax (Activation)    (None, 33)                0

=================================================================
Total params: 23,713
Trainable params: 23,073
Non-trainable params: 640
_________________________________________________________________
import numpy as np

from sklearn.metrics import accuracy_score

# Check Keras Model performance
potentials_keras = model_keras.predict(x_valid)
preds_keras = np.squeeze(np.argmax(potentials_keras, 1))

accuracy = accuracy_score(y_valid, preds_keras)
print("Accuracy: " + "{0:.2f}".format(100 * accuracy) + "%")

Out:

Accuracy: 92.85%

3. Load a pre-trained quantized Keras model satisfying Akida NSoC requirements

The above native Keras model is quantized and fine-tuned to get a quantized Keras model satisfying the Akida NSoC requirements. The first convolutional layer uses 8 bits weights, but other layers use 4 bits weights.

All activations are 4 bits except for the final Separable Convolutional that uses binary activations.

Pre-trained weights were obtained after a few training episodes:

  • we train the model with quantized activations only, with weights initialized from those trained in the previous episode (native Keras model),

  • then, we train the model with quantized weights, with both weights and activations initialized from those trained in the previous episode,

  • finally, we train the model with quantized weights and activations and by gradually increasing quantization in the last layer.

The table below summarizes the results obtained when preparing the weights stored under http://data.brainchip.com/models/ds_cnn/ :

Episode

Weights Quant.

Activ. Quant. / last layer

Accuracy

Epochs

1

N/A

N/A

93.06 %

16

2

N/A

4 bits / 4 bits

92.30 %

16

3

8/4 bits

4 bits / 4 bits

92.11 %

16

4

8/4 bits

4 bits / 3 bits

92.38 %

16

5

8/4 bits

4 bits / 2 bits

92.23 %

16

6

8/4 bits

4 bits / 1 bit

92.22 %

16

from akida_models import ds_cnn_kws_pretrained

# Load the pre-trained quantized model
model_keras_quantized = ds_cnn_kws_pretrained()
model_keras_quantized.summary()

# Check Model performance
potentials_keras_q = model_keras_quantized.predict(x_valid)
preds_keras_q = np.squeeze(np.argmax(potentials_keras_q, 1))

accuracy_q = accuracy_score(y_valid, preds_keras_q)
print("Accuracy: " + "{0:.2f}".format(100 * accuracy_q) + "%")

Out:

Downloading data from http://data.brainchip.com/models/ds_cnn/ds_cnn_kws_iq8_wq4_aq4_laq1.h5

 16384/135704 [==>...........................] - ETA: 0s
139264/135704 [==============================] - 0s 0us/step

147456/135704 [================================] - 0s 0us/step
Model: "ds_cnn_kws"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 input_1 (InputLayer)        [(None, 49, 10, 1)]       0

 rescaling (Rescaling)       (None, 49, 10, 1)         0

 conv_0 (QuantizedConv2D)    (None, 25, 5, 64)         1664

 conv_0_relu (ActivationDisc  (None, 25, 5, 64)        0
 reteRelu)

 separable_1 (QuantizedSepar  (None, 25, 5, 64)        4736
 ableConv2D)

 separable_1_relu (Activatio  (None, 25, 5, 64)        0
 nDiscreteRelu)

 separable_2 (QuantizedSepar  (None, 25, 5, 64)        4736
 ableConv2D)

 separable_2_relu (Activatio  (None, 25, 5, 64)        0
 nDiscreteRelu)

 separable_3 (QuantizedSepar  (None, 25, 5, 64)        4736
 ableConv2D)

 separable_3_relu (Activatio  (None, 25, 5, 64)        0
 nDiscreteRelu)

 separable_4 (QuantizedSepar  (None, 25, 5, 64)        4736
 ableConv2D)

 separable_4_global_avg (Glo  (None, 64)               0
 balAveragePooling2D)

 separable_4_relu (Activatio  (None, 64)               0
 nDiscreteRelu)

 reshape_1 (Reshape)         (None, 1, 1, 64)          0

 flatten (Flatten)           (None, 64)                0

 dense_5 (QuantizedDense)    (None, 33)                2145

 act_softmax (Activation)    (None, 33)                0

=================================================================
Total params: 22,753
Trainable params: 22,753
Non-trainable params: 0
_________________________________________________________________
Accuracy: 91.43%

4. Conversion to Akida

We convert the model to Akida and then evaluate the performances on the dataset.

from cnn2snn import convert

# Convert the model
model_akida = convert(model_keras_quantized)
model_akida.summary()

Out:

                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[49, 10, 1]  [1, 1, 33]    1          6
______________________________________________

             SW/conv_0-dense_5 (Software)
_______________________________________________________
Layer (type)             Output shape  Kernel shape
=======================================================
conv_0 (InputConv.)      [25, 5, 64]   (5, 5, 1, 64)
_______________________________________________________
separable_1 (Sep.Conv.)  [25, 5, 64]   (3, 3, 64, 1)
_______________________________________________________
                                       (1, 1, 64, 64)
_______________________________________________________
separable_2 (Sep.Conv.)  [25, 5, 64]   (3, 3, 64, 1)
_______________________________________________________
                                       (1, 1, 64, 64)
_______________________________________________________
separable_3 (Sep.Conv.)  [25, 5, 64]   (3, 3, 64, 1)
_______________________________________________________
                                       (1, 1, 64, 64)
_______________________________________________________
separable_4 (Sep.Conv.)  [1, 1, 64]    (3, 3, 64, 1)
_______________________________________________________
                                       (1, 1, 64, 64)
_______________________________________________________
dense_5 (Fully.)         [1, 1, 33]    (1, 1, 64, 33)
_______________________________________________________
# Check Akida model performance
preds_akida = model_akida.predict_classes(x_valid, num_classes=num_classes)

accuracy = accuracy_score(y_valid, preds_akida)
print("Accuracy: " + "{0:.2f}".format(100 * accuracy) + "%")

# For non-regression purpose
assert accuracy > 0.9

Out:

Accuracy: 91.34%

5. Confusion matrix

The confusion matrix provides a good summary of what mistakes the network is making.

Per scikit-learn convention it displays the true class in each row (ie on each row you can see what the network predicted for the corresponding word).

Please refer to the Tensorflow audio recognition example for a detailed explanation of the confusion matrix.

import itertools
import matplotlib.pyplot as plt

from sklearn.metrics import confusion_matrix

# Create confusion matrix
cm = confusion_matrix(y_valid, preds_akida,
                      labels=list(word_to_index.values()))

# Normalize
cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

# Display confusion matrix
plt.rcParams["figure.figsize"] = (16, 16)
plt.figure()

title = 'Confusion matrix'
cmap = plt.cm.Blues

plt.imshow(cm, interpolation='nearest', cmap=cmap)
plt.title(title)
plt.colorbar()
tick_marks = np.arange(len(word_to_index))
plt.xticks(tick_marks, word_to_index, rotation=45)
plt.yticks(tick_marks, word_to_index)

thresh = cm.max() / 2.
for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
    plt.text(j,
             i,
             format(cm[i, j], '.2f'),
             horizontalalignment="center",
             color="white" if cm[i, j] > thresh else "black")

plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.autoscale()
plt.show()
Confusion matrix

Total running time of the script: ( 0 minutes 26.522 seconds)

Gallery generated by Sphinx-Gallery