CNN conversion flow tutorial

This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.

The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.

Note

Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.

MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.

../../_images/cnn2snn_flow_small.jpg

1. Load and reshape MNIST dataset

After loading, we make 2 transformations on the dataset:

  1. Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.

Note

At this point, we’ll set aside the raw data for testing our converted model in the Akida Execution Engine later.

  1. Rescale the 8-bit loaded data to the range 0-to-1 for training.

Note

Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.

This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.

Also note that we store the scaling values input_scaling for use when preparing the model for the Akida Execution Engine. The implementation of the Akida neural network allows us to completely skip the rescaling step (i.e. the Akida model should be fed with the raw 8-bit values) but that does require information about what scaling was applied prior to training - see below for more details.

import tensorflow as tf
from tensorflow import keras

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)

# Set aside raw test data for use with Akida Execution Engine later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test

# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a

2. Model definition

Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.

An appropriate model for MNIST (inspired by this example) might look something like the following:

model_keras = keras.models.Sequential([
    keras.layers.Conv2D(filters=32, kernel_size=3, input_shape=(28, 28, 1)),
    keras.layers.MaxPool2D(),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.SeparableConv2D(filters=64, kernel_size=3, padding='same'),
    keras.layers.MaxPool2D(padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 26, 26, 32)        320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0
_________________________________________________________________
batch_normalization (BatchNo (None, 13, 13, 32)        128
_________________________________________________________________
re_lu (ReLU)                 (None, 13, 13, 32)        0
_________________________________________________________________
separable_conv2d (SeparableC (None, 13, 13, 64)        2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64)          0
_________________________________________________________________
batch_normalization_1 (Batch (None, 7, 7, 64)          256
_________________________________________________________________
re_lu_1 (ReLU)               (None, 7, 7, 64)          0
_________________________________________________________________
flatten (Flatten)            (None, 3136)              0
_________________________________________________________________
dense (Dense)                (None, 10)                31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________

The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):

  • Standard Conv2D and Dense layers are supported

  • Hidden layers must be followed by a ReLU layer.

  • BatchNormalization must always happen before activations.

  • Convolutional blocks can optionally be followed by a MaxPooling.

The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).

from cnn2snn import check_model_compatibility

print("Model compatible for Akida conversion:",
      check_model_compatibility(model_keras))

Out:

Model compatible for Akida conversion: True

3. Model training

Before going any further, train the model and get its performance. The created model should have achieved a test accuracy a little over 99% after 10 epochs.

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)

score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/10

   1/1688 [..............................] - ETA: 10:35 - loss: 2.8026 - accuracy: 0.0625
  30/1688 [..............................] - ETA: 2s - loss: 1.8196 - accuracy: 0.4397   
  59/1688 [>.............................] - ETA: 2s - loss: 1.3904 - accuracy: 0.5719
  89/1688 [>.............................] - ETA: 2s - loss: 1.1593 - accuracy: 0.6441
 119/1688 [=>............................] - ETA: 2s - loss: 1.0128 - accuracy: 0.6893
 148/1688 [=>............................] - ETA: 2s - loss: 0.9124 - accuracy: 0.7204
 177/1688 [==>...........................] - ETA: 2s - loss: 0.8361 - accuracy: 0.7442
 206/1688 [==>...........................] - ETA: 2s - loss: 0.7748 - accuracy: 0.7634
 235/1688 [===>..........................] - ETA: 2s - loss: 0.7243 - accuracy: 0.7791
 265/1688 [===>..........................] - ETA: 2s - loss: 0.6807 - accuracy: 0.7926
 294/1688 [====>.........................] - ETA: 2s - loss: 0.6446 - accuracy: 0.8038
 324/1688 [====>.........................] - ETA: 2s - loss: 0.6125 - accuracy: 0.8137
 353/1688 [=====>........................] - ETA: 2s - loss: 0.5853 - accuracy: 0.8222
 382/1688 [=====>........................] - ETA: 2s - loss: 0.5612 - accuracy: 0.8296
 411/1688 [======>.......................] - ETA: 2s - loss: 0.5399 - accuracy: 0.8362
 440/1688 [======>.......................] - ETA: 2s - loss: 0.5207 - accuracy: 0.8421
 469/1688 [=======>......................] - ETA: 2s - loss: 0.5033 - accuracy: 0.8474
 498/1688 [=======>......................] - ETA: 2s - loss: 0.4874 - accuracy: 0.8523
 528/1688 [========>.....................] - ETA: 2s - loss: 0.4724 - accuracy: 0.8569
 557/1688 [========>.....................] - ETA: 1s - loss: 0.4591 - accuracy: 0.8609
 587/1688 [=========>....................] - ETA: 1s - loss: 0.4463 - accuracy: 0.8647
 616/1688 [=========>....................] - ETA: 1s - loss: 0.4349 - accuracy: 0.8682
 645/1688 [==========>...................] - ETA: 1s - loss: 0.4243 - accuracy: 0.8714
 675/1688 [==========>...................] - ETA: 1s - loss: 0.4141 - accuracy: 0.8745
 704/1688 [===========>..................] - ETA: 1s - loss: 0.4048 - accuracy: 0.8773
 734/1688 [============>.................] - ETA: 1s - loss: 0.3958 - accuracy: 0.8800
 764/1688 [============>.................] - ETA: 1s - loss: 0.3874 - accuracy: 0.8826
 793/1688 [=============>................] - ETA: 1s - loss: 0.3798 - accuracy: 0.8849
 822/1688 [=============>................] - ETA: 1s - loss: 0.3725 - accuracy: 0.8871
 851/1688 [==============>...............] - ETA: 1s - loss: 0.3657 - accuracy: 0.8892
 881/1688 [==============>...............] - ETA: 1s - loss: 0.3589 - accuracy: 0.8912
 910/1688 [===============>..............] - ETA: 1s - loss: 0.3527 - accuracy: 0.8931
 939/1688 [===============>..............] - ETA: 1s - loss: 0.3468 - accuracy: 0.8949
 968/1688 [================>.............] - ETA: 1s - loss: 0.3412 - accuracy: 0.8966
 997/1688 [================>.............] - ETA: 1s - loss: 0.3358 - accuracy: 0.8982
1026/1688 [=================>............] - ETA: 1s - loss: 0.3306 - accuracy: 0.8998
1055/1688 [=================>............] - ETA: 1s - loss: 0.3257 - accuracy: 0.9013
1084/1688 [==================>...........] - ETA: 1s - loss: 0.3210 - accuracy: 0.9027
1114/1688 [==================>...........] - ETA: 0s - loss: 0.3163 - accuracy: 0.9042
1144/1688 [===================>..........] - ETA: 0s - loss: 0.3118 - accuracy: 0.9055
1173/1688 [===================>..........] - ETA: 0s - loss: 0.3076 - accuracy: 0.9068
1202/1688 [====================>.........] - ETA: 0s - loss: 0.3035 - accuracy: 0.9080
1232/1688 [====================>.........] - ETA: 0s - loss: 0.2995 - accuracy: 0.9093
1262/1688 [=====================>........] - ETA: 0s - loss: 0.2957 - accuracy: 0.9104
1291/1688 [=====================>........] - ETA: 0s - loss: 0.2921 - accuracy: 0.9115
1320/1688 [======================>.......] - ETA: 0s - loss: 0.2886 - accuracy: 0.9126
1350/1688 [======================>.......] - ETA: 0s - loss: 0.2852 - accuracy: 0.9136
1380/1688 [=======================>......] - ETA: 0s - loss: 0.2819 - accuracy: 0.9146
1409/1688 [========================>.....] - ETA: 0s - loss: 0.2788 - accuracy: 0.9155
1438/1688 [========================>.....] - ETA: 0s - loss: 0.2758 - accuracy: 0.9165
1467/1688 [=========================>....] - ETA: 0s - loss: 0.2729 - accuracy: 0.9173
1496/1688 [=========================>....] - ETA: 0s - loss: 0.2701 - accuracy: 0.9182
1525/1688 [==========================>...] - ETA: 0s - loss: 0.2673 - accuracy: 0.9190
1554/1688 [==========================>...] - ETA: 0s - loss: 0.2647 - accuracy: 0.9198
1583/1688 [===========================>..] - ETA: 0s - loss: 0.2621 - accuracy: 0.9206
1613/1688 [===========================>..] - ETA: 0s - loss: 0.2595 - accuracy: 0.9214
1642/1688 [============================>.] - ETA: 0s - loss: 0.2571 - accuracy: 0.9221
1672/1688 [============================>.] - ETA: 0s - loss: 0.2547 - accuracy: 0.9228
1688/1688 [==============================] - ETA: 0s - loss: 0.2534 - accuracy: 0.9232
1688/1688 [==============================] - 4s 2ms/step - loss: 0.2533 - accuracy: 0.9232 - val_loss: 0.0706 - val_accuracy: 0.9793
Epoch 2/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0573 - accuracy: 0.9688
  32/1688 [..............................] - ETA: 2s - loss: 0.0696 - accuracy: 0.9707
  61/1688 [>.............................] - ETA: 2s - loss: 0.0634 - accuracy: 0.9739
  90/1688 [>.............................] - ETA: 2s - loss: 0.0594 - accuracy: 0.9763
 119/1688 [=>............................] - ETA: 2s - loss: 0.0577 - accuracy: 0.9776
 148/1688 [=>............................] - ETA: 2s - loss: 0.0564 - accuracy: 0.9785
 178/1688 [==>...........................] - ETA: 2s - loss: 0.0551 - accuracy: 0.9794
 207/1688 [==>...........................] - ETA: 2s - loss: 0.0539 - accuracy: 0.9801
 236/1688 [===>..........................] - ETA: 2s - loss: 0.0527 - accuracy: 0.9808
 265/1688 [===>..........................] - ETA: 2s - loss: 0.0518 - accuracy: 0.9812
 294/1688 [====>.........................] - ETA: 2s - loss: 0.0512 - accuracy: 0.9815
 323/1688 [====>.........................] - ETA: 2s - loss: 0.0509 - accuracy: 0.9817
 353/1688 [=====>........................] - ETA: 2s - loss: 0.0508 - accuracy: 0.9819
 383/1688 [=====>........................] - ETA: 2s - loss: 0.0507 - accuracy: 0.9820
 412/1688 [======>.......................] - ETA: 2s - loss: 0.0506 - accuracy: 0.9821
 442/1688 [======>.......................] - ETA: 2s - loss: 0.0504 - accuracy: 0.9822
 471/1688 [=======>......................] - ETA: 2s - loss: 0.0502 - accuracy: 0.9823
 500/1688 [=======>......................] - ETA: 2s - loss: 0.0500 - accuracy: 0.9825
 529/1688 [========>.....................] - ETA: 2s - loss: 0.0498 - accuracy: 0.9826
 558/1688 [========>.....................] - ETA: 1s - loss: 0.0497 - accuracy: 0.9827
 587/1688 [=========>....................] - ETA: 1s - loss: 0.0495 - accuracy: 0.9828
 616/1688 [=========>....................] - ETA: 1s - loss: 0.0493 - accuracy: 0.9829
 645/1688 [==========>...................] - ETA: 1s - loss: 0.0491 - accuracy: 0.9831
 674/1688 [==========>...................] - ETA: 1s - loss: 0.0490 - accuracy: 0.9832
 703/1688 [===========>..................] - ETA: 1s - loss: 0.0488 - accuracy: 0.9833
 733/1688 [============>.................] - ETA: 1s - loss: 0.0487 - accuracy: 0.9834
 763/1688 [============>.................] - ETA: 1s - loss: 0.0486 - accuracy: 0.9834
 793/1688 [=============>................] - ETA: 1s - loss: 0.0485 - accuracy: 0.9835
 823/1688 [=============>................] - ETA: 1s - loss: 0.0484 - accuracy: 0.9836
 852/1688 [==============>...............] - ETA: 1s - loss: 0.0483 - accuracy: 0.9837
 882/1688 [==============>...............] - ETA: 1s - loss: 0.0482 - accuracy: 0.9837
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0482 - accuracy: 0.9838
 940/1688 [===============>..............] - ETA: 1s - loss: 0.0482 - accuracy: 0.9838
 969/1688 [================>.............] - ETA: 1s - loss: 0.0481 - accuracy: 0.9839
 998/1688 [================>.............] - ETA: 1s - loss: 0.0481 - accuracy: 0.9839
1027/1688 [=================>............] - ETA: 1s - loss: 0.0480 - accuracy: 0.9839
1056/1688 [=================>............] - ETA: 1s - loss: 0.0480 - accuracy: 0.9840
1086/1688 [==================>...........] - ETA: 1s - loss: 0.0480 - accuracy: 0.9840
1116/1688 [==================>...........] - ETA: 0s - loss: 0.0480 - accuracy: 0.9840
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0480 - accuracy: 0.9841
1175/1688 [===================>..........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9841
1204/1688 [====================>.........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9841
1233/1688 [====================>.........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9841
1262/1688 [=====================>........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9841
1292/1688 [=====================>........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9842
1322/1688 [======================>.......] - ETA: 0s - loss: 0.0479 - accuracy: 0.9842
1351/1688 [=======================>......] - ETA: 0s - loss: 0.0478 - accuracy: 0.9842
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0478 - accuracy: 0.9842
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0478 - accuracy: 0.9842
1439/1688 [========================>.....] - ETA: 0s - loss: 0.0478 - accuracy: 0.9842
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0478 - accuracy: 0.9843
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1644/1688 [============================>.] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1673/1688 [============================>.] - ETA: 0s - loss: 0.0476 - accuracy: 0.9844
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0476 - accuracy: 0.9844 - val_loss: 0.0597 - val_accuracy: 0.9857
Epoch 3/10

   1/1688 [..............................] - ETA: 3s - loss: 9.3639e-04 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9940    
  60/1688 [>.............................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9930
  89/1688 [>.............................] - ETA: 2s - loss: 0.0259 - accuracy: 0.9925
 118/1688 [=>............................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9922
 148/1688 [=>............................] - ETA: 2s - loss: 0.0290 - accuracy: 0.9922
 178/1688 [==>...........................] - ETA: 2s - loss: 0.0294 - accuracy: 0.9920
 207/1688 [==>...........................] - ETA: 2s - loss: 0.0298 - accuracy: 0.9918
 236/1688 [===>..........................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9916
 265/1688 [===>..........................] - ETA: 2s - loss: 0.0304 - accuracy: 0.9915
 294/1688 [====>.........................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9914
 323/1688 [====>.........................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9913
 352/1688 [=====>........................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9913
 382/1688 [=====>........................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9913
 411/1688 [======>.......................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9913
 440/1688 [======>.......................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9912
 470/1688 [=======>......................] - ETA: 2s - loss: 0.0307 - accuracy: 0.9912
 499/1688 [=======>......................] - ETA: 2s - loss: 0.0307 - accuracy: 0.9911
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0308 - accuracy: 0.9911
 557/1688 [========>.....................] - ETA: 1s - loss: 0.0308 - accuracy: 0.9910
 586/1688 [=========>....................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9909
 616/1688 [=========>....................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9909
 645/1688 [==========>...................] - ETA: 1s - loss: 0.0310 - accuracy: 0.9908
 674/1688 [==========>...................] - ETA: 1s - loss: 0.0311 - accuracy: 0.9908
 703/1688 [===========>..................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9907
 732/1688 [============>.................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9907
 762/1688 [============>.................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9907
 791/1688 [=============>................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9907
 820/1688 [=============>................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9906
 849/1688 [==============>...............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9906
 879/1688 [==============>...............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9906
 909/1688 [===============>..............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9906
 939/1688 [===============>..............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9906
 968/1688 [================>.............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9906
 997/1688 [================>.............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9906
1026/1688 [=================>............] - ETA: 1s - loss: 0.0310 - accuracy: 0.9906
1055/1688 [=================>............] - ETA: 1s - loss: 0.0310 - accuracy: 0.9906
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0310 - accuracy: 0.9906
1114/1688 [==================>...........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9906
1143/1688 [===================>..........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9906
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9906
1202/1688 [====================>.........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9906
1231/1688 [====================>.........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9905
1260/1688 [=====================>........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9905
1289/1688 [=====================>........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9905
1318/1688 [======================>.......] - ETA: 0s - loss: 0.0311 - accuracy: 0.9905
1347/1688 [======================>.......] - ETA: 0s - loss: 0.0311 - accuracy: 0.9905
1376/1688 [=======================>......] - ETA: 0s - loss: 0.0312 - accuracy: 0.9905
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0312 - accuracy: 0.9905
1435/1688 [========================>.....] - ETA: 0s - loss: 0.0312 - accuracy: 0.9904
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0312 - accuracy: 0.9904
1493/1688 [=========================>....] - ETA: 0s - loss: 0.0313 - accuracy: 0.9904
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0313 - accuracy: 0.9904
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0313 - accuracy: 0.9904
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0314 - accuracy: 0.9904
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0314 - accuracy: 0.9904
1639/1688 [============================>.] - ETA: 0s - loss: 0.0314 - accuracy: 0.9904
1668/1688 [============================>.] - ETA: 0s - loss: 0.0314 - accuracy: 0.9904
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0315 - accuracy: 0.9903 - val_loss: 0.0483 - val_accuracy: 0.9870
Epoch 4/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0030 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9922
  60/1688 [>.............................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9921
  88/1688 [>.............................] - ETA: 2s - loss: 0.0251 - accuracy: 0.9916
 117/1688 [=>............................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9914
 146/1688 [=>............................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9913
 175/1688 [==>...........................] - ETA: 2s - loss: 0.0255 - accuracy: 0.9914
 204/1688 [==>...........................] - ETA: 2s - loss: 0.0250 - accuracy: 0.9915
 234/1688 [===>..........................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9917
 263/1688 [===>..........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9918
 292/1688 [====>.........................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9920
 321/1688 [====>.........................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9921
 350/1688 [=====>........................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9923
 379/1688 [=====>........................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9924
 408/1688 [======>.......................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9925
 437/1688 [======>.......................] - ETA: 2s - loss: 0.0224 - accuracy: 0.9926
 466/1688 [=======>......................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9926
 495/1688 [=======>......................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9927
 524/1688 [========>.....................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9927
 553/1688 [========>.....................] - ETA: 1s - loss: 0.0219 - accuracy: 0.9928
 582/1688 [=========>....................] - ETA: 1s - loss: 0.0218 - accuracy: 0.9928
 611/1688 [=========>....................] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
 639/1688 [==========>...................] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
 668/1688 [==========>...................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 697/1688 [===========>..................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 727/1688 [===========>..................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 756/1688 [============>.................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 785/1688 [============>.................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 814/1688 [=============>................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 844/1688 [==============>...............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
 873/1688 [==============>...............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
 902/1688 [===============>..............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
 931/1688 [===============>..............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
 960/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
 989/1688 [================>.............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1018/1688 [=================>............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1048/1688 [=================>............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
1106/1688 [==================>...........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
1135/1688 [===================>..........] - ETA: 0s - loss: 0.0218 - accuracy: 0.9928
1165/1688 [===================>..........] - ETA: 0s - loss: 0.0218 - accuracy: 0.9928
1195/1688 [====================>.........] - ETA: 0s - loss: 0.0219 - accuracy: 0.9927
1225/1688 [====================>.........] - ETA: 0s - loss: 0.0219 - accuracy: 0.9927
1254/1688 [=====================>........] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1283/1688 [=====================>........] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9927
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9927
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0222 - accuracy: 0.9927
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0222 - accuracy: 0.9926
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0222 - accuracy: 0.9926
1486/1688 [=========================>....] - ETA: 0s - loss: 0.0223 - accuracy: 0.9926
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0223 - accuracy: 0.9926
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0224 - accuracy: 0.9926
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0224 - accuracy: 0.9926
1603/1688 [===========================>..] - ETA: 0s - loss: 0.0225 - accuracy: 0.9926
1632/1688 [============================>.] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1661/1688 [============================>.] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0226 - accuracy: 0.9925 - val_loss: 0.0524 - val_accuracy: 0.9870
Epoch 5/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0098 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9972
  59/1688 [>.............................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9961
  88/1688 [>.............................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9958
 118/1688 [=>............................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9956
 147/1688 [=>............................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9954
 176/1688 [==>...........................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9953
 205/1688 [==>...........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9953
 235/1688 [===>..........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9952
 264/1688 [===>..........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9952
 293/1688 [====>.........................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9951
 323/1688 [====>.........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9951
 352/1688 [=====>........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9951
 381/1688 [=====>........................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9951
 410/1688 [======>.......................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9951
 439/1688 [======>.......................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9951
 468/1688 [=======>......................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9951
 497/1688 [=======>......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
 526/1688 [========>.....................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
 555/1688 [========>.....................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9951
 584/1688 [=========>....................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
 614/1688 [=========>....................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
 643/1688 [==========>...................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
 672/1688 [==========>...................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
 702/1688 [===========>..................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
 731/1688 [===========>..................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
 760/1688 [============>.................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9950
 789/1688 [=============>................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9950
 818/1688 [=============>................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9950
 847/1688 [==============>...............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9950
 876/1688 [==============>...............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9950
 905/1688 [===============>..............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9950
 934/1688 [===============>..............] - ETA: 1s - loss: 0.0145 - accuracy: 0.9949
 963/1688 [================>.............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9949
 992/1688 [================>.............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9949
1022/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9949
1051/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9949
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0148 - accuracy: 0.9948
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0149 - accuracy: 0.9948
1138/1688 [===================>..........] - ETA: 0s - loss: 0.0150 - accuracy: 0.9948
1166/1688 [===================>..........] - ETA: 0s - loss: 0.0151 - accuracy: 0.9948
1195/1688 [====================>.........] - ETA: 0s - loss: 0.0152 - accuracy: 0.9948
1224/1688 [====================>.........] - ETA: 0s - loss: 0.0153 - accuracy: 0.9947
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0154 - accuracy: 0.9947
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0155 - accuracy: 0.9947
1311/1688 [======================>.......] - ETA: 0s - loss: 0.0156 - accuracy: 0.9947
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0157 - accuracy: 0.9946
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0158 - accuracy: 0.9946
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9946
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0160 - accuracy: 0.9945
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0160 - accuracy: 0.9945
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0161 - accuracy: 0.9945
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0162 - accuracy: 0.9945
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9944
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0164 - accuracy: 0.9944
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0164 - accuracy: 0.9944
1634/1688 [============================>.] - ETA: 0s - loss: 0.0165 - accuracy: 0.9944
1663/1688 [============================>.] - ETA: 0s - loss: 0.0166 - accuracy: 0.9943
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0167 - accuracy: 0.9943 - val_loss: 0.0754 - val_accuracy: 0.9817
Epoch 6/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0020 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9942
  62/1688 [>.............................] - ETA: 2s - loss: 0.0273 - accuracy: 0.9937
  91/1688 [>.............................] - ETA: 2s - loss: 0.0265 - accuracy: 0.9934
 120/1688 [=>............................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9932
 149/1688 [=>............................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9932
 178/1688 [==>...........................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9933
 207/1688 [==>...........................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9934
 236/1688 [===>..........................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9935
 266/1688 [===>..........................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9936
 296/1688 [====>.........................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9937
 325/1688 [====>.........................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9938
 354/1688 [=====>........................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9939
 383/1688 [=====>........................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9940
 412/1688 [======>.......................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9941
 441/1688 [======>.......................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9941
 470/1688 [=======>......................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9942
 499/1688 [=======>......................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9943
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9943
 558/1688 [========>.....................] - ETA: 1s - loss: 0.0186 - accuracy: 0.9944
 587/1688 [=========>....................] - ETA: 1s - loss: 0.0185 - accuracy: 0.9944
 616/1688 [=========>....................] - ETA: 1s - loss: 0.0183 - accuracy: 0.9944
 645/1688 [==========>...................] - ETA: 1s - loss: 0.0182 - accuracy: 0.9944
 674/1688 [==========>...................] - ETA: 1s - loss: 0.0181 - accuracy: 0.9945
 703/1688 [===========>..................] - ETA: 1s - loss: 0.0180 - accuracy: 0.9945
 732/1688 [============>.................] - ETA: 1s - loss: 0.0179 - accuracy: 0.9945
 762/1688 [============>.................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9945
 791/1688 [=============>................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9946
 821/1688 [=============>................] - ETA: 1s - loss: 0.0176 - accuracy: 0.9946
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0175 - accuracy: 0.9946
 879/1688 [==============>...............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9946
 908/1688 [===============>..............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9947
 936/1688 [===============>..............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9947
 965/1688 [================>.............] - ETA: 1s - loss: 0.0171 - accuracy: 0.9947
 995/1688 [================>.............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9947
1024/1688 [=================>............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9947
1053/1688 [=================>............] - ETA: 1s - loss: 0.0169 - accuracy: 0.9947
1082/1688 [==================>...........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
1111/1688 [==================>...........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
1140/1688 [===================>..........] - ETA: 0s - loss: 0.0167 - accuracy: 0.9948
1169/1688 [===================>..........] - ETA: 0s - loss: 0.0167 - accuracy: 0.9948
1198/1688 [====================>.........] - ETA: 0s - loss: 0.0167 - accuracy: 0.9948
1228/1688 [====================>.........] - ETA: 0s - loss: 0.0166 - accuracy: 0.9948
1258/1688 [=====================>........] - ETA: 0s - loss: 0.0166 - accuracy: 0.9948
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1317/1688 [======================>.......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1346/1688 [======================>.......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1375/1688 [=======================>......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1404/1688 [=======================>......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1433/1688 [========================>.....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1638/1688 [============================>.] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1668/1688 [============================>.] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0164 - accuracy: 0.9948 - val_loss: 0.0481 - val_accuracy: 0.9887
Epoch 7/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0089 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9959
  61/1688 [>.............................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9958
  91/1688 [>.............................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9961
 121/1688 [=>............................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9963
 151/1688 [=>............................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9964
 180/1688 [==>...........................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9965
 210/1688 [==>...........................] - ETA: 2s - loss: 0.0103 - accuracy: 0.9965
 240/1688 [===>..........................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9967
 269/1688 [===>..........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9967
 298/1688 [====>.........................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9968
 327/1688 [====>.........................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9968
 356/1688 [=====>........................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9968
 386/1688 [=====>........................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9968
 416/1688 [======>.......................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9968
 446/1688 [======>.......................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9968
 476/1688 [=======>......................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9968
 505/1688 [=======>......................] - ETA: 2s - loss: 0.0100 - accuracy: 0.9967
 534/1688 [========>.....................] - ETA: 1s - loss: 0.0100 - accuracy: 0.9967
 564/1688 [=========>....................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9967
 593/1688 [=========>....................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9966
 622/1688 [==========>...................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9966
 651/1688 [==========>...................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9966
 680/1688 [===========>..................] - ETA: 1s - loss: 0.0103 - accuracy: 0.9965
 709/1688 [===========>..................] - ETA: 1s - loss: 0.0104 - accuracy: 0.9965
 738/1688 [============>.................] - ETA: 1s - loss: 0.0105 - accuracy: 0.9965
 767/1688 [============>.................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9964
 797/1688 [=============>................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9964
 827/1688 [=============>................] - ETA: 1s - loss: 0.0107 - accuracy: 0.9964
 856/1688 [==============>...............] - ETA: 1s - loss: 0.0108 - accuracy: 0.9964
 885/1688 [==============>...............] - ETA: 1s - loss: 0.0108 - accuracy: 0.9963
 914/1688 [===============>..............] - ETA: 1s - loss: 0.0109 - accuracy: 0.9963
 944/1688 [===============>..............] - ETA: 1s - loss: 0.0110 - accuracy: 0.9963
 973/1688 [================>.............] - ETA: 1s - loss: 0.0111 - accuracy: 0.9963
1002/1688 [================>.............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9962
1031/1688 [=================>............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9962
1060/1688 [=================>............] - ETA: 1s - loss: 0.0113 - accuracy: 0.9962
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0114 - accuracy: 0.9961
1118/1688 [==================>...........] - ETA: 0s - loss: 0.0115 - accuracy: 0.9961
1147/1688 [===================>..........] - ETA: 0s - loss: 0.0116 - accuracy: 0.9961
1177/1688 [===================>..........] - ETA: 0s - loss: 0.0117 - accuracy: 0.9961
1206/1688 [====================>.........] - ETA: 0s - loss: 0.0118 - accuracy: 0.9960
1235/1688 [====================>.........] - ETA: 0s - loss: 0.0119 - accuracy: 0.9960
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0119 - accuracy: 0.9960
1293/1688 [=====================>........] - ETA: 0s - loss: 0.0120 - accuracy: 0.9960
1322/1688 [======================>.......] - ETA: 0s - loss: 0.0120 - accuracy: 0.9960
1351/1688 [=======================>......] - ETA: 0s - loss: 0.0121 - accuracy: 0.9959
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0122 - accuracy: 0.9959
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0122 - accuracy: 0.9959
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0123 - accuracy: 0.9959
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0123 - accuracy: 0.9959
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0124 - accuracy: 0.9958
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0124 - accuracy: 0.9958
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0125 - accuracy: 0.9958
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0125 - accuracy: 0.9958
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0125 - accuracy: 0.9958
1643/1688 [============================>.] - ETA: 0s - loss: 0.0126 - accuracy: 0.9958
1672/1688 [============================>.] - ETA: 0s - loss: 0.0126 - accuracy: 0.9957
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0127 - accuracy: 0.9957 - val_loss: 0.0547 - val_accuracy: 0.9880
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0013 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0291 - accuracy: 0.9966
  59/1688 [>.............................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9966
  88/1688 [>.............................] - ETA: 2s - loss: 0.0185 - accuracy: 0.9966
 117/1688 [=>............................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9967
 146/1688 [=>............................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9968
 175/1688 [==>...........................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9969
 204/1688 [==>...........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9969
 233/1688 [===>..........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9969
 262/1688 [===>..........................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9970
 292/1688 [====>.........................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9970
 322/1688 [====>.........................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9970
 351/1688 [=====>........................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9970
 380/1688 [=====>........................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9971
 410/1688 [======>.......................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9971
 439/1688 [======>.......................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9971
 468/1688 [=======>......................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9971
 497/1688 [=======>......................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9971
 526/1688 [========>.....................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9971
 555/1688 [========>.....................] - ETA: 1s - loss: 0.0108 - accuracy: 0.9972
 584/1688 [=========>....................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9972
 613/1688 [=========>....................] - ETA: 1s - loss: 0.0105 - accuracy: 0.9972
 642/1688 [==========>...................] - ETA: 1s - loss: 0.0104 - accuracy: 0.9972
 672/1688 [==========>...................] - ETA: 1s - loss: 0.0103 - accuracy: 0.9972
 701/1688 [===========>..................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9972
 730/1688 [===========>..................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9972
 758/1688 [============>.................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9972
 787/1688 [============>.................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9972
 816/1688 [=============>................] - ETA: 1s - loss: 0.0100 - accuracy: 0.9972
 845/1688 [==============>...............] - ETA: 1s - loss: 0.0100 - accuracy: 0.9972
 874/1688 [==============>...............] - ETA: 1s - loss: 0.0100 - accuracy: 0.9972
 903/1688 [===============>..............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9972
 933/1688 [===============>..............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9972
 963/1688 [================>.............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9971
 992/1688 [================>.............] - ETA: 1s - loss: 0.0098 - accuracy: 0.9971
1021/1688 [=================>............] - ETA: 1s - loss: 0.0098 - accuracy: 0.9971
1049/1688 [=================>............] - ETA: 1s - loss: 0.0098 - accuracy: 0.9971
1078/1688 [==================>...........] - ETA: 1s - loss: 0.0098 - accuracy: 0.9971
1107/1688 [==================>...........] - ETA: 1s - loss: 0.0097 - accuracy: 0.9971
1136/1688 [===================>..........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9971
1165/1688 [===================>..........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9971
1194/1688 [====================>.........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9971
1223/1688 [====================>.........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9971
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1283/1688 [=====================>........] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9970
1636/1688 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9970
1665/1688 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9970
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0096 - accuracy: 0.9970 - val_loss: 0.0478 - val_accuracy: 0.9893
Epoch 9/10

   1/1688 [..............................] - ETA: 2s - loss: 4.8947e-04 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0052 - accuracy: 0.9970    
  61/1688 [>.............................] - ETA: 2s - loss: 0.0051 - accuracy: 0.9977
  90/1688 [>.............................] - ETA: 2s - loss: 0.0054 - accuracy: 0.9978
 119/1688 [=>............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9978
 148/1688 [=>............................] - ETA: 2s - loss: 0.0057 - accuracy: 0.9978
 177/1688 [==>...........................] - ETA: 2s - loss: 0.0058 - accuracy: 0.9979
 207/1688 [==>...........................] - ETA: 2s - loss: 0.0059 - accuracy: 0.9978
 236/1688 [===>..........................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9978
 266/1688 [===>..........................] - ETA: 2s - loss: 0.0063 - accuracy: 0.9977
 295/1688 [====>.........................] - ETA: 2s - loss: 0.0064 - accuracy: 0.9977
 324/1688 [====>.........................] - ETA: 2s - loss: 0.0065 - accuracy: 0.9976
 354/1688 [=====>........................] - ETA: 2s - loss: 0.0066 - accuracy: 0.9976
 383/1688 [=====>........................] - ETA: 2s - loss: 0.0067 - accuracy: 0.9976
 412/1688 [======>.......................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9976
 441/1688 [======>.......................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9976
 470/1688 [=======>......................] - ETA: 2s - loss: 0.0070 - accuracy: 0.9975
 499/1688 [=======>......................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9975
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9975
 557/1688 [========>.....................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9975
 587/1688 [=========>....................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9975
 616/1688 [=========>....................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
 645/1688 [==========>...................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
 674/1688 [==========>...................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
 703/1688 [===========>..................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
 732/1688 [============>.................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
 761/1688 [============>.................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
 790/1688 [=============>................] - ETA: 1s - loss: 0.0074 - accuracy: 0.9975
 819/1688 [=============>................] - ETA: 1s - loss: 0.0074 - accuracy: 0.9975
 848/1688 [==============>...............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9975
 877/1688 [==============>...............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9975
 906/1688 [===============>..............] - ETA: 1s - loss: 0.0075 - accuracy: 0.9975
 935/1688 [===============>..............] - ETA: 1s - loss: 0.0075 - accuracy: 0.9975
 964/1688 [================>.............] - ETA: 1s - loss: 0.0075 - accuracy: 0.9975
 993/1688 [================>.............] - ETA: 1s - loss: 0.0076 - accuracy: 0.9975
1022/1688 [=================>............] - ETA: 1s - loss: 0.0076 - accuracy: 0.9975
1051/1688 [=================>............] - ETA: 1s - loss: 0.0076 - accuracy: 0.9975
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0077 - accuracy: 0.9975
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0077 - accuracy: 0.9975
1138/1688 [===================>..........] - ETA: 0s - loss: 0.0077 - accuracy: 0.9975
1167/1688 [===================>..........] - ETA: 0s - loss: 0.0077 - accuracy: 0.9975
1196/1688 [====================>.........] - ETA: 0s - loss: 0.0078 - accuracy: 0.9975
1226/1688 [====================>.........] - ETA: 0s - loss: 0.0078 - accuracy: 0.9975
1256/1688 [=====================>........] - ETA: 0s - loss: 0.0078 - accuracy: 0.9974
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0079 - accuracy: 0.9974
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0079 - accuracy: 0.9974
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0080 - accuracy: 0.9974
1374/1688 [=======================>......] - ETA: 0s - loss: 0.0080 - accuracy: 0.9974
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0080 - accuracy: 0.9974
1433/1688 [========================>.....] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1462/1688 [========================>.....] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0083 - accuracy: 0.9974
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1640/1688 [============================>.] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1669/1688 [============================>.] - ETA: 0s - loss: 0.0084 - accuracy: 0.9973
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0084 - accuracy: 0.9973 - val_loss: 0.0639 - val_accuracy: 0.9845
Epoch 10/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0489 - accuracy: 0.9688
  32/1688 [..............................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9895
  61/1688 [>.............................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9921
  91/1688 [>.............................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9932
 121/1688 [=>............................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9941
 150/1688 [=>............................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9947
 179/1688 [==>...........................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9951
 208/1688 [==>...........................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9955
 237/1688 [===>..........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9958
 266/1688 [===>..........................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9960
 296/1688 [====>.........................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9961
 325/1688 [====>.........................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9962
 354/1688 [=====>........................] - ETA: 2s - loss: 0.0104 - accuracy: 0.9963
 383/1688 [=====>........................] - ETA: 2s - loss: 0.0103 - accuracy: 0.9964
 412/1688 [======>.......................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9965
 441/1688 [======>.......................] - ETA: 2s - loss: 0.0100 - accuracy: 0.9966
 470/1688 [=======>......................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9966
 499/1688 [=======>......................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9967
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9967
 557/1688 [========>.....................] - ETA: 1s - loss: 0.0094 - accuracy: 0.9968
 585/1688 [=========>....................] - ETA: 1s - loss: 0.0093 - accuracy: 0.9968
 614/1688 [=========>....................] - ETA: 1s - loss: 0.0092 - accuracy: 0.9968
 643/1688 [==========>...................] - ETA: 1s - loss: 0.0092 - accuracy: 0.9969
 672/1688 [==========>...................] - ETA: 1s - loss: 0.0091 - accuracy: 0.9969
 702/1688 [===========>..................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9969
 731/1688 [===========>..................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9969
 760/1688 [============>.................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9969
 788/1688 [=============>................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9969
 817/1688 [=============>................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9969
 846/1688 [==============>...............] - ETA: 1s - loss: 0.0089 - accuracy: 0.9970
 875/1688 [==============>...............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
 904/1688 [===============>..............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
 933/1688 [===============>..............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
 962/1688 [================>.............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
 991/1688 [================>.............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1021/1688 [=================>............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1050/1688 [=================>............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1139/1688 [===================>..........] - ETA: 0s - loss: 0.0088 - accuracy: 0.9970
1168/1688 [===================>..........] - ETA: 0s - loss: 0.0088 - accuracy: 0.9970
1198/1688 [====================>.........] - ETA: 0s - loss: 0.0088 - accuracy: 0.9970
1227/1688 [====================>.........] - ETA: 0s - loss: 0.0088 - accuracy: 0.9970
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1316/1688 [======================>.......] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1346/1688 [======================>.......] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1375/1688 [=======================>......] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1434/1688 [========================>.....] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0090 - accuracy: 0.9970
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0090 - accuracy: 0.9970
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0090 - accuracy: 0.9970
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0090 - accuracy: 0.9970
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0091 - accuracy: 0.9970
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0091 - accuracy: 0.9970
1638/1688 [============================>.] - ETA: 0s - loss: 0.0091 - accuracy: 0.9970
1667/1688 [============================>.] - ETA: 0s - loss: 0.0091 - accuracy: 0.9970
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0091 - accuracy: 0.9969 - val_loss: 0.0432 - val_accuracy: 0.9907
Test score: 0.04563219100236893
Test accuracy: 0.9890000224113464

4. Model quantization

We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.

For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).

The quantized model is a Keras model where the neural layers (Conv2D, Dense) and the ReLU layers are replaced with custom CNN2SNN quantized layers (QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function folds the batch normalization layers into the corresponding neural layer. The new weights are computed according to this folding operation.

from cnn2snn import quantize

model_quantized = quantize(model_keras,
                           input_weight_quantization=8,
                           weight_quantization=4,
                           activ_quantization=4)
model_quantized.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (QuantizedConv2D)     (None, 26, 26, 32)        320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0
_________________________________________________________________
re_lu (ActivationDiscreteRel (None, 13, 13, 32)        0
_________________________________________________________________
separable_conv2d (QuantizedS (None, 13, 13, 64)        2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64)          0
_________________________________________________________________
re_lu_1 (ActivationDiscreteR (None, 7, 7, 64)          0
_________________________________________________________________
flatten (Flatten)            (None, 3136)              0
_________________________________________________________________
dense (QuantizedDense)       (None, 10)                31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________

Check the quantized model accuracy.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])

Out:

Test accuracy after 8-4-4 quantization: 0.9866999983787537

Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.

Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.

Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.

from cnn2snn import quantize_layer

model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])

# To recover the original model accuracy, a quantization-aware training phase
# is required.

Out:

Test accuracy after low bitwidth quantization: 0.7461000084877014

5. Model fine tuning (quantization-aware training)

This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.

Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])

Out:

Epoch 1/5

   1/1688 [..............................] - ETA: 18:55 - loss: 0.5271 - accuracy: 0.7812
  21/1688 [..............................] - ETA: 4s - loss: 0.4219 - accuracy: 0.8586   
  41/1688 [..............................] - ETA: 4s - loss: 0.3135 - accuracy: 0.9017
  61/1688 [>.............................] - ETA: 4s - loss: 0.2557 - accuracy: 0.9232
  81/1688 [>.............................] - ETA: 4s - loss: 0.2172 - accuracy: 0.9363
 101/1688 [>.............................] - ETA: 4s - loss: 0.1944 - accuracy: 0.9418
 121/1688 [=>............................] - ETA: 4s - loss: 0.1734 - accuracy: 0.9486
 141/1688 [=>............................] - ETA: 3s - loss: 0.1555 - accuracy: 0.9546
 161/1688 [=>............................] - ETA: 3s - loss: 0.1457 - accuracy: 0.9571
 181/1688 [==>...........................] - ETA: 3s - loss: 0.1387 - accuracy: 0.9584
 201/1688 [==>...........................] - ETA: 3s - loss: 0.1301 - accuracy: 0.9611
 221/1688 [==>...........................] - ETA: 3s - loss: 0.1228 - accuracy: 0.9632
 241/1688 [===>..........................] - ETA: 3s - loss: 0.1167 - accuracy: 0.9651
 261/1688 [===>..........................] - ETA: 3s - loss: 0.1141 - accuracy: 0.9658
 281/1688 [===>..........................] - ETA: 3s - loss: 0.1104 - accuracy: 0.9667
 301/1688 [====>.........................] - ETA: 3s - loss: 0.1079 - accuracy: 0.9674
 321/1688 [====>.........................] - ETA: 3s - loss: 0.1055 - accuracy: 0.9679
 341/1688 [=====>........................] - ETA: 3s - loss: 0.1021 - accuracy: 0.9690
 361/1688 [=====>........................] - ETA: 3s - loss: 0.0991 - accuracy: 0.9697
 381/1688 [=====>........................] - ETA: 3s - loss: 0.0964 - accuracy: 0.9704
 401/1688 [======>.......................] - ETA: 3s - loss: 0.0945 - accuracy: 0.9709
 421/1688 [======>.......................] - ETA: 3s - loss: 0.0919 - accuracy: 0.9715
 441/1688 [======>.......................] - ETA: 3s - loss: 0.0895 - accuracy: 0.9722
 461/1688 [=======>......................] - ETA: 3s - loss: 0.0873 - accuracy: 0.9726
 481/1688 [=======>......................] - ETA: 3s - loss: 0.0846 - accuracy: 0.9736
 500/1688 [=======>......................] - ETA: 3s - loss: 0.0822 - accuracy: 0.9743
 520/1688 [========>.....................] - ETA: 2s - loss: 0.0802 - accuracy: 0.9749
 540/1688 [========>.....................] - ETA: 2s - loss: 0.0793 - accuracy: 0.9752
 560/1688 [========>.....................] - ETA: 2s - loss: 0.0779 - accuracy: 0.9757
 580/1688 [=========>....................] - ETA: 2s - loss: 0.0770 - accuracy: 0.9759
 600/1688 [=========>....................] - ETA: 2s - loss: 0.0757 - accuracy: 0.9764
 620/1688 [==========>...................] - ETA: 2s - loss: 0.0742 - accuracy: 0.9769
 640/1688 [==========>...................] - ETA: 2s - loss: 0.0736 - accuracy: 0.9770
 660/1688 [==========>...................] - ETA: 2s - loss: 0.0730 - accuracy: 0.9772
 680/1688 [===========>..................] - ETA: 2s - loss: 0.0718 - accuracy: 0.9776
 700/1688 [===========>..................] - ETA: 2s - loss: 0.0709 - accuracy: 0.9779
 720/1688 [===========>..................] - ETA: 2s - loss: 0.0695 - accuracy: 0.9783
 740/1688 [============>.................] - ETA: 2s - loss: 0.0682 - accuracy: 0.9785
 760/1688 [============>.................] - ETA: 2s - loss: 0.0673 - accuracy: 0.9789
 780/1688 [============>.................] - ETA: 2s - loss: 0.0666 - accuracy: 0.9790
 800/1688 [=============>................] - ETA: 2s - loss: 0.0656 - accuracy: 0.9793
 820/1688 [=============>................] - ETA: 2s - loss: 0.0649 - accuracy: 0.9795
 840/1688 [=============>................] - ETA: 2s - loss: 0.0645 - accuracy: 0.9795
 860/1688 [==============>...............] - ETA: 2s - loss: 0.0637 - accuracy: 0.9797
 880/1688 [==============>...............] - ETA: 2s - loss: 0.0633 - accuracy: 0.9798
 900/1688 [==============>...............] - ETA: 2s - loss: 0.0627 - accuracy: 0.9800
 920/1688 [===============>..............] - ETA: 1s - loss: 0.0620 - accuracy: 0.9803
 940/1688 [===============>..............] - ETA: 1s - loss: 0.0611 - accuracy: 0.9805
 960/1688 [================>.............] - ETA: 1s - loss: 0.0602 - accuracy: 0.9807
 980/1688 [================>.............] - ETA: 1s - loss: 0.0597 - accuracy: 0.9808
1000/1688 [================>.............] - ETA: 1s - loss: 0.0592 - accuracy: 0.9809
1020/1688 [=================>............] - ETA: 1s - loss: 0.0588 - accuracy: 0.9811
1040/1688 [=================>............] - ETA: 1s - loss: 0.0586 - accuracy: 0.9812
1060/1688 [=================>............] - ETA: 1s - loss: 0.0581 - accuracy: 0.9814
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0575 - accuracy: 0.9816
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0574 - accuracy: 0.9816
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0571 - accuracy: 0.9817
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0567 - accuracy: 0.9818
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0566 - accuracy: 0.9817
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0560 - accuracy: 0.9819
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0560 - accuracy: 0.9821
1220/1688 [====================>.........] - ETA: 1s - loss: 0.0555 - accuracy: 0.9822
1240/1688 [=====================>........] - ETA: 1s - loss: 0.0552 - accuracy: 0.9823
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0547 - accuracy: 0.9824
1280/1688 [=====================>........] - ETA: 1s - loss: 0.0542 - accuracy: 0.9826
1300/1688 [======================>.......] - ETA: 0s - loss: 0.0539 - accuracy: 0.9826
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0534 - accuracy: 0.9828
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0531 - accuracy: 0.9829
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0529 - accuracy: 0.9829
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0527 - accuracy: 0.9830
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0525 - accuracy: 0.9830
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0520 - accuracy: 0.9831
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0515 - accuracy: 0.9833
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0511 - accuracy: 0.9834
1480/1688 [=========================>....] - ETA: 0s - loss: 0.0507 - accuracy: 0.9835
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0504 - accuracy: 0.9836
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0501 - accuracy: 0.9837
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0497 - accuracy: 0.9838
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0496 - accuracy: 0.9839
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0492 - accuracy: 0.9840
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0492 - accuracy: 0.9839
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0489 - accuracy: 0.9840
1640/1688 [============================>.] - ETA: 0s - loss: 0.0486 - accuracy: 0.9841
1660/1688 [============================>.] - ETA: 0s - loss: 0.0485 - accuracy: 0.9841
1680/1688 [============================>.] - ETA: 0s - loss: 0.0481 - accuracy: 0.9843
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0480 - accuracy: 0.9844 - val_loss: 0.0479 - val_accuracy: 0.9877
Epoch 2/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0017 - accuracy: 1.0000
  21/1688 [..............................] - ETA: 4s - loss: 0.0281 - accuracy: 0.9940
  41/1688 [..............................] - ETA: 4s - loss: 0.0288 - accuracy: 0.9916
  61/1688 [>.............................] - ETA: 4s - loss: 0.0259 - accuracy: 0.9913
  81/1688 [>.............................] - ETA: 4s - loss: 0.0228 - accuracy: 0.9923
 101/1688 [>.............................] - ETA: 4s - loss: 0.0235 - accuracy: 0.9920
 121/1688 [=>............................] - ETA: 4s - loss: 0.0236 - accuracy: 0.9917
 141/1688 [=>............................] - ETA: 3s - loss: 0.0252 - accuracy: 0.9916
 161/1688 [=>............................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9920
 181/1688 [==>...........................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9922
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9919
 221/1688 [==>...........................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9924
 241/1688 [===>..........................] - ETA: 3s - loss: 0.0237 - accuracy: 0.9922
 261/1688 [===>..........................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9925
 281/1688 [===>..........................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9922
 301/1688 [====>.........................] - ETA: 3s - loss: 0.0244 - accuracy: 0.9919
 321/1688 [====>.........................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9920
 341/1688 [=====>........................] - ETA: 3s - loss: 0.0244 - accuracy: 0.9919
 361/1688 [=====>........................] - ETA: 3s - loss: 0.0248 - accuracy: 0.9917
 381/1688 [=====>........................] - ETA: 3s - loss: 0.0252 - accuracy: 0.9916
 401/1688 [======>.......................] - ETA: 3s - loss: 0.0250 - accuracy: 0.9918
 421/1688 [======>.......................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9920
 441/1688 [======>.......................] - ETA: 3s - loss: 0.0242 - accuracy: 0.9920
 461/1688 [=======>......................] - ETA: 3s - loss: 0.0244 - accuracy: 0.9919
 481/1688 [=======>......................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9920
 501/1688 [=======>......................] - ETA: 3s - loss: 0.0241 - accuracy: 0.9921
 521/1688 [========>.....................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9921
 541/1688 [========>.....................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9921
 561/1688 [========>.....................] - ETA: 2s - loss: 0.0248 - accuracy: 0.9918
 581/1688 [=========>....................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9918
 601/1688 [=========>....................] - ETA: 2s - loss: 0.0246 - accuracy: 0.9918
 621/1688 [==========>...................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9919
 641/1688 [==========>...................] - ETA: 2s - loss: 0.0239 - accuracy: 0.9922
 661/1688 [==========>...................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9922
 681/1688 [===========>..................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9922
 701/1688 [===========>..................] - ETA: 2s - loss: 0.0236 - accuracy: 0.9923
 721/1688 [===========>..................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9924
 741/1688 [============>.................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9924
 761/1688 [============>.................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9923
 781/1688 [============>.................] - ETA: 2s - loss: 0.0239 - accuracy: 0.9922
 801/1688 [=============>................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9922
 821/1688 [=============>................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9922
 841/1688 [=============>................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9922
 860/1688 [==============>...............] - ETA: 2s - loss: 0.0239 - accuracy: 0.9922
 880/1688 [==============>...............] - ETA: 2s - loss: 0.0242 - accuracy: 0.9922
 900/1688 [==============>...............] - ETA: 2s - loss: 0.0243 - accuracy: 0.9921
 920/1688 [===============>..............] - ETA: 1s - loss: 0.0245 - accuracy: 0.9921
 940/1688 [===============>..............] - ETA: 1s - loss: 0.0246 - accuracy: 0.9921
 960/1688 [================>.............] - ETA: 1s - loss: 0.0244 - accuracy: 0.9922
 980/1688 [================>.............] - ETA: 1s - loss: 0.0242 - accuracy: 0.9923
1000/1688 [================>.............] - ETA: 1s - loss: 0.0243 - accuracy: 0.9922
1020/1688 [=================>............] - ETA: 1s - loss: 0.0242 - accuracy: 0.9923
1040/1688 [=================>............] - ETA: 1s - loss: 0.0242 - accuracy: 0.9922
1060/1688 [=================>............] - ETA: 1s - loss: 0.0243 - accuracy: 0.9922
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0242 - accuracy: 0.9921
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0243 - accuracy: 0.9920
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0243 - accuracy: 0.9920
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0245 - accuracy: 0.9919
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0244 - accuracy: 0.9919
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0244 - accuracy: 0.9920
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0244 - accuracy: 0.9920
1220/1688 [====================>.........] - ETA: 1s - loss: 0.0245 - accuracy: 0.9920
1240/1688 [=====================>........] - ETA: 1s - loss: 0.0245 - accuracy: 0.9921
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0246 - accuracy: 0.9920
1280/1688 [=====================>........] - ETA: 1s - loss: 0.0246 - accuracy: 0.9919
1300/1688 [======================>.......] - ETA: 0s - loss: 0.0244 - accuracy: 0.9920
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0243 - accuracy: 0.9921
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0242 - accuracy: 0.9921
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0246 - accuracy: 0.9920
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0245 - accuracy: 0.9919
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0246 - accuracy: 0.9919
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0248 - accuracy: 0.9918
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0250 - accuracy: 0.9918
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0252 - accuracy: 0.9918
1480/1688 [=========================>....] - ETA: 0s - loss: 0.0252 - accuracy: 0.9918
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0251 - accuracy: 0.9918
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0250 - accuracy: 0.9919
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0249 - accuracy: 0.9919
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0249 - accuracy: 0.9919
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0248 - accuracy: 0.9919
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0247 - accuracy: 0.9920
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0246 - accuracy: 0.9920
1640/1688 [============================>.] - ETA: 0s - loss: 0.0246 - accuracy: 0.9919
1660/1688 [============================>.] - ETA: 0s - loss: 0.0246 - accuracy: 0.9919
1680/1688 [============================>.] - ETA: 0s - loss: 0.0247 - accuracy: 0.9919
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0247 - accuracy: 0.9919 - val_loss: 0.0577 - val_accuracy: 0.9845
Epoch 3/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0016 - accuracy: 1.0000
  21/1688 [..............................] - ETA: 4s - loss: 0.0271 - accuracy: 0.9926
  41/1688 [..............................] - ETA: 4s - loss: 0.0209 - accuracy: 0.9931
  61/1688 [>.............................] - ETA: 4s - loss: 0.0220 - accuracy: 0.9913
  81/1688 [>.............................] - ETA: 4s - loss: 0.0190 - accuracy: 0.9927
 101/1688 [>.............................] - ETA: 4s - loss: 0.0181 - accuracy: 0.9929
 121/1688 [=>............................] - ETA: 4s - loss: 0.0183 - accuracy: 0.9933
 141/1688 [=>............................] - ETA: 3s - loss: 0.0177 - accuracy: 0.9936
 161/1688 [=>............................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9932
 181/1688 [==>...........................] - ETA: 3s - loss: 0.0194 - accuracy: 0.9933
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9928
 221/1688 [==>...........................] - ETA: 3s - loss: 0.0222 - accuracy: 0.9924
 241/1688 [===>..........................] - ETA: 3s - loss: 0.0215 - accuracy: 0.9926
 261/1688 [===>..........................] - ETA: 3s - loss: 0.0225 - accuracy: 0.9921
 281/1688 [===>..........................] - ETA: 3s - loss: 0.0231 - accuracy: 0.9920
 301/1688 [====>.........................] - ETA: 3s - loss: 0.0229 - accuracy: 0.9919
 321/1688 [====>.........................] - ETA: 3s - loss: 0.0227 - accuracy: 0.9919
 341/1688 [=====>........................] - ETA: 3s - loss: 0.0226 - accuracy: 0.9918
 361/1688 [=====>........................] - ETA: 3s - loss: 0.0222 - accuracy: 0.9920
 381/1688 [=====>........................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9922
 401/1688 [======>.......................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9921
 421/1688 [======>.......................] - ETA: 3s - loss: 0.0229 - accuracy: 0.9920
 441/1688 [======>.......................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9922
 461/1688 [=======>......................] - ETA: 3s - loss: 0.0221 - accuracy: 0.9923
 481/1688 [=======>......................] - ETA: 3s - loss: 0.0221 - accuracy: 0.9923
 501/1688 [=======>......................] - ETA: 3s - loss: 0.0220 - accuracy: 0.9923
 521/1688 [========>.....................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9923
 541/1688 [========>.....................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9923
 561/1688 [========>.....................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9923
 581/1688 [=========>....................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9924
 601/1688 [=========>....................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9924
 621/1688 [==========>...................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9924
 641/1688 [==========>...................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9926
 661/1688 [==========>...................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9926
 681/1688 [===========>..................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9926
 701/1688 [===========>..................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9925
 721/1688 [===========>..................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9924
 741/1688 [============>.................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9924
 761/1688 [============>.................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9925
 781/1688 [============>.................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9925
 801/1688 [=============>................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9925
 821/1688 [=============>................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9925
 841/1688 [=============>................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9926
 861/1688 [==============>...............] - ETA: 2s - loss: 0.0210 - accuracy: 0.9927
 881/1688 [==============>...............] - ETA: 2s - loss: 0.0210 - accuracy: 0.9928
 901/1688 [===============>..............] - ETA: 2s - loss: 0.0212 - accuracy: 0.9927
 921/1688 [===============>..............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9927
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9927
 961/1688 [================>.............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9926
 981/1688 [================>.............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9926
1001/1688 [================>.............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9927
1021/1688 [=================>............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9927
1041/1688 [=================>............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9926
1061/1688 [=================>............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9926
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0210 - accuracy: 0.9926
1101/1688 [==================>...........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9925
1121/1688 [==================>...........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9925
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9926
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9926
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0214 - accuracy: 0.9925
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9924
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9924
1241/1688 [=====================>........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9924
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9924
1281/1688 [=====================>........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9924
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1321/1688 [======================>.......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0211 - accuracy: 0.9925
1361/1688 [=======================>......] - ETA: 0s - loss: 0.0214 - accuracy: 0.9924
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0214 - accuracy: 0.9924
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9924
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0212 - accuracy: 0.9924
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0212 - accuracy: 0.9925
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0212 - accuracy: 0.9925
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0214 - accuracy: 0.9924
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0214 - accuracy: 0.9925
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0216 - accuracy: 0.9925
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9925
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9925
1640/1688 [============================>.] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1660/1688 [============================>.] - ETA: 0s - loss: 0.0218 - accuracy: 0.9924
1680/1688 [============================>.] - ETA: 0s - loss: 0.0218 - accuracy: 0.9924
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0220 - accuracy: 0.9923 - val_loss: 0.0596 - val_accuracy: 0.9840
Epoch 4/5

   1/1688 [..............................] - ETA: 5s - loss: 0.0060 - accuracy: 1.0000
  21/1688 [..............................] - ETA: 4s - loss: 0.0253 - accuracy: 0.9911
  41/1688 [..............................] - ETA: 4s - loss: 0.0208 - accuracy: 0.9924
  61/1688 [>.............................] - ETA: 4s - loss: 0.0194 - accuracy: 0.9933
  81/1688 [>.............................] - ETA: 4s - loss: 0.0216 - accuracy: 0.9931
 101/1688 [>.............................] - ETA: 4s - loss: 0.0190 - accuracy: 0.9941
 121/1688 [=>............................] - ETA: 4s - loss: 0.0196 - accuracy: 0.9941
 141/1688 [=>............................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9936
 161/1688 [=>............................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9930
 181/1688 [==>...........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9926
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9927
 221/1688 [==>...........................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9929
 241/1688 [===>..........................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9931
 261/1688 [===>..........................] - ETA: 3s - loss: 0.0213 - accuracy: 0.9934
 281/1688 [===>..........................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9932
 301/1688 [====>.........................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9931
 321/1688 [====>.........................] - ETA: 3s - loss: 0.0221 - accuracy: 0.9930
 341/1688 [=====>........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9929
 361/1688 [=====>........................] - ETA: 3s - loss: 0.0227 - accuracy: 0.9926
 381/1688 [=====>........................] - ETA: 3s - loss: 0.0228 - accuracy: 0.9926
 401/1688 [======>.......................] - ETA: 3s - loss: 0.0233 - accuracy: 0.9923
 421/1688 [======>.......................] - ETA: 3s - loss: 0.0226 - accuracy: 0.9925
 441/1688 [======>.......................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9925
 461/1688 [=======>......................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9927
 481/1688 [=======>......................] - ETA: 3s - loss: 0.0220 - accuracy: 0.9926
 501/1688 [=======>......................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9926
 521/1688 [========>.....................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9925
 541/1688 [========>.....................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9925
 561/1688 [========>.....................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9926
 581/1688 [=========>....................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9928
 601/1688 [=========>....................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9929
 621/1688 [==========>...................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9928
 641/1688 [==========>...................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9926
 661/1688 [==========>...................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9927
 681/1688 [===========>..................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9927
 701/1688 [===========>..................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9926
 721/1688 [===========>..................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9925
 741/1688 [============>.................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9923
 761/1688 [============>.................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9924
 781/1688 [============>.................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9923
 801/1688 [=============>................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9922
 821/1688 [=============>................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9922
 841/1688 [=============>................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9922
 861/1688 [==============>...............] - ETA: 2s - loss: 0.0213 - accuracy: 0.9923
 881/1688 [==============>...............] - ETA: 2s - loss: 0.0212 - accuracy: 0.9923
 901/1688 [===============>..............] - ETA: 2s - loss: 0.0211 - accuracy: 0.9924
 921/1688 [===============>..............] - ETA: 1s - loss: 0.0208 - accuracy: 0.9925
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9926
 961/1688 [================>.............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9927
 981/1688 [================>.............] - ETA: 1s - loss: 0.0205 - accuracy: 0.9928
1001/1688 [================>.............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9929
1021/1688 [=================>............] - ETA: 1s - loss: 0.0205 - accuracy: 0.9929
1041/1688 [=================>............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9929
1061/1688 [=================>............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9930
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0203 - accuracy: 0.9929
1101/1688 [==================>...........] - ETA: 1s - loss: 0.0202 - accuracy: 0.9929
1121/1688 [==================>...........] - ETA: 1s - loss: 0.0205 - accuracy: 0.9929
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0204 - accuracy: 0.9929
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0204 - accuracy: 0.9928
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0205 - accuracy: 0.9928
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0205 - accuracy: 0.9928
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0206 - accuracy: 0.9928
1241/1688 [=====================>........] - ETA: 1s - loss: 0.0208 - accuracy: 0.9926
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0206 - accuracy: 0.9927
1281/1688 [=====================>........] - ETA: 1s - loss: 0.0206 - accuracy: 0.9927
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0206 - accuracy: 0.9927
1321/1688 [======================>.......] - ETA: 0s - loss: 0.0206 - accuracy: 0.9926
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0207 - accuracy: 0.9926
1361/1688 [=======================>......] - ETA: 0s - loss: 0.0205 - accuracy: 0.9927
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0204 - accuracy: 0.9928
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0207 - accuracy: 0.9928
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9927
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9928
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9927
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0210 - accuracy: 0.9927
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0212 - accuracy: 0.9926
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0212 - accuracy: 0.9925
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0214 - accuracy: 0.9925
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0215 - accuracy: 0.9924
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0215 - accuracy: 0.9924
1601/1688 [===========================>..] - ETA: 0s - loss: 0.0215 - accuracy: 0.9924
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1641/1688 [============================>.] - ETA: 0s - loss: 0.0215 - accuracy: 0.9925
1661/1688 [============================>.] - ETA: 0s - loss: 0.0214 - accuracy: 0.9925
1681/1688 [============================>.] - ETA: 0s - loss: 0.0212 - accuracy: 0.9925
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0212 - accuracy: 0.9926 - val_loss: 0.0532 - val_accuracy: 0.9873
Epoch 5/5

   1/1688 [..............................] - ETA: 5s - loss: 2.5233e-04 - accuracy: 1.0000
  21/1688 [..............................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9955    
  41/1688 [..............................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9962
  61/1688 [>.............................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9964
  81/1688 [>.............................] - ETA: 4s - loss: 0.0156 - accuracy: 0.9954
 101/1688 [>.............................] - ETA: 4s - loss: 0.0147 - accuracy: 0.9954
 121/1688 [=>............................] - ETA: 4s - loss: 0.0163 - accuracy: 0.9954
 141/1688 [=>............................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9953
 161/1688 [=>............................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9953
 181/1688 [==>...........................] - ETA: 3s - loss: 0.0159 - accuracy: 0.9953
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9952
 221/1688 [==>...........................] - ETA: 3s - loss: 0.0171 - accuracy: 0.9949
 241/1688 [===>..........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9949
 261/1688 [===>..........................] - ETA: 3s - loss: 0.0167 - accuracy: 0.9951
 281/1688 [===>..........................] - ETA: 3s - loss: 0.0175 - accuracy: 0.9948
 301/1688 [====>.........................] - ETA: 3s - loss: 0.0172 - accuracy: 0.9949
 321/1688 [====>.........................] - ETA: 3s - loss: 0.0175 - accuracy: 0.9946
 341/1688 [=====>........................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9947
 361/1688 [=====>........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9947
 381/1688 [=====>........................] - ETA: 3s - loss: 0.0166 - accuracy: 0.9950
 401/1688 [======>.......................] - ETA: 3s - loss: 0.0162 - accuracy: 0.9952
 421/1688 [======>.......................] - ETA: 3s - loss: 0.0159 - accuracy: 0.9952
 441/1688 [======>.......................] - ETA: 3s - loss: 0.0156 - accuracy: 0.9953
 461/1688 [=======>......................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9954
 481/1688 [=======>......................] - ETA: 3s - loss: 0.0151 - accuracy: 0.9954
 501/1688 [=======>......................] - ETA: 3s - loss: 0.0151 - accuracy: 0.9953
 521/1688 [========>.....................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9952
 541/1688 [========>.....................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9951
 561/1688 [========>.....................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9952
 581/1688 [=========>....................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9952
 601/1688 [=========>....................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9952
 621/1688 [==========>...................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9954
 641/1688 [==========>...................] - ETA: 2s - loss: 0.0158 - accuracy: 0.9950
 661/1688 [==========>...................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9949
 681/1688 [===========>..................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9949
 701/1688 [===========>..................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9948
 721/1688 [===========>..................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9947
 740/1688 [============>.................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9947
 760/1688 [============>.................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9947
 780/1688 [============>.................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9948
 800/1688 [=============>................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9949
 820/1688 [=============>................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9950
 840/1688 [=============>................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9949
 860/1688 [==============>...............] - ETA: 2s - loss: 0.0156 - accuracy: 0.9948
 880/1688 [==============>...............] - ETA: 2s - loss: 0.0154 - accuracy: 0.9949
 900/1688 [==============>...............] - ETA: 2s - loss: 0.0161 - accuracy: 0.9947
 920/1688 [===============>..............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9947
 940/1688 [===============>..............] - ETA: 1s - loss: 0.0160 - accuracy: 0.9947
 960/1688 [================>.............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9947
 980/1688 [================>.............] - ETA: 1s - loss: 0.0163 - accuracy: 0.9946
1000/1688 [================>.............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9947
1020/1688 [=================>............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9946
1040/1688 [=================>............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9947
1060/1688 [=================>............] - ETA: 1s - loss: 0.0162 - accuracy: 0.9947
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0163 - accuracy: 0.9947
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0163 - accuracy: 0.9947
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0165 - accuracy: 0.9946
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9946
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0167 - accuracy: 0.9946
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9945
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9945
1220/1688 [====================>.........] - ETA: 1s - loss: 0.0170 - accuracy: 0.9944
1240/1688 [=====================>........] - ETA: 1s - loss: 0.0171 - accuracy: 0.9945
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0174 - accuracy: 0.9945
1280/1688 [=====================>........] - ETA: 1s - loss: 0.0178 - accuracy: 0.9943
1300/1688 [======================>.......] - ETA: 0s - loss: 0.0181 - accuracy: 0.9942
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0180 - accuracy: 0.9942
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0182 - accuracy: 0.9941
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0183 - accuracy: 0.9941
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0182 - accuracy: 0.9942
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0184 - accuracy: 0.9942
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0182 - accuracy: 0.9942
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0181 - accuracy: 0.9943
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0180 - accuracy: 0.9942
1480/1688 [=========================>....] - ETA: 0s - loss: 0.0180 - accuracy: 0.9942
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0179 - accuracy: 0.9943
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0180 - accuracy: 0.9942
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0181 - accuracy: 0.9942
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0182 - accuracy: 0.9942
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0182 - accuracy: 0.9941
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0181 - accuracy: 0.9941
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0184 - accuracy: 0.9940
1640/1688 [============================>.] - ETA: 0s - loss: 0.0184 - accuracy: 0.9940
1660/1688 [============================>.] - ETA: 0s - loss: 0.0184 - accuracy: 0.9940
1680/1688 [============================>.] - ETA: 0s - loss: 0.0185 - accuracy: 0.9940
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0186 - accuracy: 0.9940 - val_loss: 0.0461 - val_accuracy: 0.9883
Test accuracy after fine tuning: 0.987500011920929

6. Model conversion

After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida Execution Engine.

Note

One needs to supply the coefficients used to rescale the input dataset before the training - here input_scaling.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()

results = model_akida.predict(raw_x_test)
accuracy = (raw_y_test == results).mean()

print('Test accuracy after conversion:', accuracy)

# For non-regression purpose
assert accuracy > 0.97

Out:

                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          3
______________________________________________

/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)
                  SW/conv2d-dense (Software)
______________________________________________________________
Layer (type)                  Output shape  Kernel shape
==============================================================
conv2d (InputConv.)           [13, 13, 32]  (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.)  [7, 7, 64]    (3, 3, 32, 1)
______________________________________________________________
                                            (1, 1, 32, 64)
______________________________________________________________
dense (Fully.)                [1, 1, 10]    (1, 1, 3136, 10)
______________________________________________________________


Test accuracy after conversion: 0.9876

Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).

Total running time of the script: ( 1 minutes 2.414 seconds)

Gallery generated by Sphinx-Gallery