CNN conversion flow tutorial

This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.

The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.

Note

Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.

MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.

../../_images/cnn2snn_flow_small.jpg

1. Load and reshape MNIST dataset

After loading, we make 2 transformations on the dataset:

  1. Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.

Note

At this point, we’ll set aside the raw data for testing our converted model in the Akida Execution Engine later.

  1. Rescale the 8-bit loaded data to the range 0-to-1 for training.

Note

Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.

This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.

Also note that we store the scaling values input_scaling for use when preparing the model for the Akida Execution Engine. The implementation of the Akida neural network allows us to completely skip the rescaling step (i.e. the Akida model should be fed with the raw 8-bit values) but that does require information about what scaling was applied prior to training - see below for more details.

import tensorflow as tf
from tensorflow import keras

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)

# Set aside raw test data for use with Akida Execution Engine later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test

# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a

2. Model definition

Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.

An appropriate model for MNIST (inspired by this example) might look something like the following:

model_keras = keras.models.Sequential([
    keras.layers.Conv2D(filters=32, kernel_size=3, input_shape=(28, 28, 1)),
    keras.layers.MaxPool2D(),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.SeparableConv2D(filters=64, kernel_size=3, padding='same'),
    keras.layers.MaxPool2D(padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 26, 26, 32)        320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0
_________________________________________________________________
batch_normalization (BatchNo (None, 13, 13, 32)        128
_________________________________________________________________
re_lu (ReLU)                 (None, 13, 13, 32)        0
_________________________________________________________________
separable_conv2d (SeparableC (None, 13, 13, 64)        2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64)          0
_________________________________________________________________
batch_normalization_1 (Batch (None, 7, 7, 64)          256
_________________________________________________________________
re_lu_1 (ReLU)               (None, 7, 7, 64)          0
_________________________________________________________________
flatten (Flatten)            (None, 3136)              0
_________________________________________________________________
dense (Dense)                (None, 10)                31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________

The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):

  • Standard Conv2D and Dense layers are supported

  • Hidden layers must be followed by a ReLU layer.

  • BatchNormalization must always happen before activations.

  • Convolutional blocks can optionally be followed by a MaxPooling.

The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).

from cnn2snn import check_model_compatibility

print("Model compatible for Akida conversion:",
      check_model_compatibility(model_keras))

Out:

Model compatible for Akida conversion: True

3. Model training

Before going any further, train the model and get its performance. The created model should have achieved a test accuracy a little over 99% after 10 epochs.

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)

score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/10

   1/1688 [..............................] - ETA: 11:45 - loss: 2.9876 - accuracy: 0.0625
  31/1688 [..............................] - ETA: 2s - loss: 1.0383 - accuracy: 0.6845   
  61/1688 [>.............................] - ETA: 2s - loss: 0.6852 - accuracy: 0.7961
  91/1688 [>.............................] - ETA: 2s - loss: 0.5450 - accuracy: 0.8365
 122/1688 [=>............................] - ETA: 2s - loss: 0.4669 - accuracy: 0.8604
 152/1688 [=>............................] - ETA: 2s - loss: 0.4098 - accuracy: 0.8793
 183/1688 [==>...........................] - ETA: 2s - loss: 0.3704 - accuracy: 0.8911
 213/1688 [==>...........................] - ETA: 2s - loss: 0.3434 - accuracy: 0.8982
 243/1688 [===>..........................] - ETA: 2s - loss: 0.3158 - accuracy: 0.9068
 273/1688 [===>..........................] - ETA: 2s - loss: 0.2963 - accuracy: 0.9125
 303/1688 [====>.........................] - ETA: 2s - loss: 0.2785 - accuracy: 0.9178
 333/1688 [====>.........................] - ETA: 2s - loss: 0.2678 - accuracy: 0.9209
 364/1688 [=====>........................] - ETA: 2s - loss: 0.2554 - accuracy: 0.9244
 394/1688 [======>.......................] - ETA: 2s - loss: 0.2431 - accuracy: 0.9274
 423/1688 [======>.......................] - ETA: 2s - loss: 0.2334 - accuracy: 0.9302
 454/1688 [=======>......................] - ETA: 2s - loss: 0.2238 - accuracy: 0.9330
 484/1688 [=======>......................] - ETA: 2s - loss: 0.2169 - accuracy: 0.9349
 515/1688 [========>.....................] - ETA: 1s - loss: 0.2097 - accuracy: 0.9367
 545/1688 [========>.....................] - ETA: 1s - loss: 0.2026 - accuracy: 0.9387
 575/1688 [=========>....................] - ETA: 1s - loss: 0.1963 - accuracy: 0.9405
 605/1688 [=========>....................] - ETA: 1s - loss: 0.1912 - accuracy: 0.9421
 635/1688 [==========>...................] - ETA: 1s - loss: 0.1881 - accuracy: 0.9437
 665/1688 [==========>...................] - ETA: 1s - loss: 0.1844 - accuracy: 0.9448
 695/1688 [===========>..................] - ETA: 1s - loss: 0.1796 - accuracy: 0.9460
 725/1688 [===========>..................] - ETA: 1s - loss: 0.1758 - accuracy: 0.9472
 756/1688 [============>.................] - ETA: 1s - loss: 0.1722 - accuracy: 0.9484
 787/1688 [============>.................] - ETA: 1s - loss: 0.1691 - accuracy: 0.9492
 816/1688 [=============>................] - ETA: 1s - loss: 0.1663 - accuracy: 0.9499
 846/1688 [==============>...............] - ETA: 1s - loss: 0.1647 - accuracy: 0.9504
 876/1688 [==============>...............] - ETA: 1s - loss: 0.1618 - accuracy: 0.9511
 907/1688 [===============>..............] - ETA: 1s - loss: 0.1589 - accuracy: 0.9521
 937/1688 [===============>..............] - ETA: 1s - loss: 0.1565 - accuracy: 0.9528
 967/1688 [================>.............] - ETA: 1s - loss: 0.1538 - accuracy: 0.9535
 997/1688 [================>.............] - ETA: 1s - loss: 0.1515 - accuracy: 0.9541
1027/1688 [=================>............] - ETA: 1s - loss: 0.1491 - accuracy: 0.9548
1057/1688 [=================>............] - ETA: 1s - loss: 0.1471 - accuracy: 0.9553
1087/1688 [==================>...........] - ETA: 1s - loss: 0.1457 - accuracy: 0.9559
1117/1688 [==================>...........] - ETA: 0s - loss: 0.1446 - accuracy: 0.9561
1147/1688 [===================>..........] - ETA: 0s - loss: 0.1421 - accuracy: 0.9568
1177/1688 [===================>..........] - ETA: 0s - loss: 0.1404 - accuracy: 0.9573
1208/1688 [====================>.........] - ETA: 0s - loss: 0.1394 - accuracy: 0.9576
1238/1688 [=====================>........] - ETA: 0s - loss: 0.1377 - accuracy: 0.9581
1269/1688 [=====================>........] - ETA: 0s - loss: 0.1362 - accuracy: 0.9585
1300/1688 [======================>.......] - ETA: 0s - loss: 0.1346 - accuracy: 0.9590
1331/1688 [======================>.......] - ETA: 0s - loss: 0.1323 - accuracy: 0.9597
1361/1688 [=======================>......] - ETA: 0s - loss: 0.1305 - accuracy: 0.9602
1392/1688 [=======================>......] - ETA: 0s - loss: 0.1289 - accuracy: 0.9607
1422/1688 [========================>.....] - ETA: 0s - loss: 0.1276 - accuracy: 0.9612
1452/1688 [========================>.....] - ETA: 0s - loss: 0.1265 - accuracy: 0.9614
1482/1688 [=========================>....] - ETA: 0s - loss: 0.1249 - accuracy: 0.9620
1512/1688 [=========================>....] - ETA: 0s - loss: 0.1239 - accuracy: 0.9623
1541/1688 [==========================>...] - ETA: 0s - loss: 0.1225 - accuracy: 0.9628
1571/1688 [==========================>...] - ETA: 0s - loss: 0.1212 - accuracy: 0.9632
1602/1688 [===========================>..] - ETA: 0s - loss: 0.1207 - accuracy: 0.9634
1631/1688 [===========================>..] - ETA: 0s - loss: 0.1196 - accuracy: 0.9637
1661/1688 [============================>.] - ETA: 0s - loss: 0.1186 - accuracy: 0.9640
1688/1688 [==============================] - ETA: 0s - loss: 0.1177 - accuracy: 0.9643
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1177 - accuracy: 0.9643 - val_loss: 0.0792 - val_accuracy: 0.9803
Epoch 2/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9688
  33/1688 [..............................] - ETA: 2s - loss: 0.0471 - accuracy: 0.9839
  63/1688 [>.............................] - ETA: 2s - loss: 0.0415 - accuracy: 0.9851
  94/1688 [>.............................] - ETA: 2s - loss: 0.0405 - accuracy: 0.9864
 124/1688 [=>............................] - ETA: 2s - loss: 0.0384 - accuracy: 0.9869
 154/1688 [=>............................] - ETA: 2s - loss: 0.0402 - accuracy: 0.9868
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0419 - accuracy: 0.9861
 215/1688 [==>...........................] - ETA: 2s - loss: 0.0389 - accuracy: 0.9872
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0421 - accuracy: 0.9869
 276/1688 [===>..........................] - ETA: 2s - loss: 0.0428 - accuracy: 0.9865
 299/1688 [====>.........................] - ETA: 2s - loss: 0.0415 - accuracy: 0.9868
 328/1688 [====>.........................] - ETA: 2s - loss: 0.0445 - accuracy: 0.9856
 356/1688 [=====>........................] - ETA: 2s - loss: 0.0441 - accuracy: 0.9860
 385/1688 [=====>........................] - ETA: 2s - loss: 0.0441 - accuracy: 0.9860
 415/1688 [======>.......................] - ETA: 2s - loss: 0.0431 - accuracy: 0.9861
 443/1688 [======>.......................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9859
 473/1688 [=======>......................] - ETA: 2s - loss: 0.0461 - accuracy: 0.9857
 503/1688 [=======>......................] - ETA: 2s - loss: 0.0461 - accuracy: 0.9855
 533/1688 [========>.....................] - ETA: 1s - loss: 0.0462 - accuracy: 0.9853
 563/1688 [=========>....................] - ETA: 1s - loss: 0.0463 - accuracy: 0.9853
 593/1688 [=========>....................] - ETA: 1s - loss: 0.0463 - accuracy: 0.9852
 623/1688 [==========>...................] - ETA: 1s - loss: 0.0467 - accuracy: 0.9850
 653/1688 [==========>...................] - ETA: 1s - loss: 0.0471 - accuracy: 0.9849
 683/1688 [===========>..................] - ETA: 1s - loss: 0.0466 - accuracy: 0.9850
 713/1688 [===========>..................] - ETA: 1s - loss: 0.0478 - accuracy: 0.9849
 743/1688 [============>.................] - ETA: 1s - loss: 0.0474 - accuracy: 0.9851
 774/1688 [============>.................] - ETA: 1s - loss: 0.0482 - accuracy: 0.9847
 805/1688 [=============>................] - ETA: 1s - loss: 0.0487 - accuracy: 0.9847
 835/1688 [=============>................] - ETA: 1s - loss: 0.0493 - accuracy: 0.9845
 865/1688 [==============>...............] - ETA: 1s - loss: 0.0487 - accuracy: 0.9847
 896/1688 [==============>...............] - ETA: 1s - loss: 0.0484 - accuracy: 0.9847
 926/1688 [===============>..............] - ETA: 1s - loss: 0.0485 - accuracy: 0.9848
 956/1688 [===============>..............] - ETA: 1s - loss: 0.0487 - accuracy: 0.9848
 986/1688 [================>.............] - ETA: 1s - loss: 0.0484 - accuracy: 0.9849
1017/1688 [=================>............] - ETA: 1s - loss: 0.0492 - accuracy: 0.9846
1047/1688 [=================>............] - ETA: 1s - loss: 0.0490 - accuracy: 0.9848
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0490 - accuracy: 0.9848
1107/1688 [==================>...........] - ETA: 0s - loss: 0.0489 - accuracy: 0.9848
1137/1688 [===================>..........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9847
1167/1688 [===================>..........] - ETA: 0s - loss: 0.0496 - accuracy: 0.9845
1197/1688 [====================>.........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9845
1227/1688 [====================>.........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9845
1258/1688 [=====================>........] - ETA: 0s - loss: 0.0494 - accuracy: 0.9845
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9846
1318/1688 [======================>.......] - ETA: 0s - loss: 0.0492 - accuracy: 0.9846
1348/1688 [======================>.......] - ETA: 0s - loss: 0.0491 - accuracy: 0.9847
1377/1688 [=======================>......] - ETA: 0s - loss: 0.0489 - accuracy: 0.9847
1407/1688 [========================>.....] - ETA: 0s - loss: 0.0491 - accuracy: 0.9846
1436/1688 [========================>.....] - ETA: 0s - loss: 0.0488 - accuracy: 0.9846
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0486 - accuracy: 0.9847
1496/1688 [=========================>....] - ETA: 0s - loss: 0.0487 - accuracy: 0.9847
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0488 - accuracy: 0.9846
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0487 - accuracy: 0.9847
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0487 - accuracy: 0.9847
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0487 - accuracy: 0.9847
1645/1688 [============================>.] - ETA: 0s - loss: 0.0486 - accuracy: 0.9847
1676/1688 [============================>.] - ETA: 0s - loss: 0.0482 - accuracy: 0.9848
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0482 - accuracy: 0.9849 - val_loss: 0.0450 - val_accuracy: 0.9877
Epoch 3/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0015 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9915
  63/1688 [>.............................] - ETA: 2s - loss: 0.0316 - accuracy: 0.9906
  93/1688 [>.............................] - ETA: 2s - loss: 0.0283 - accuracy: 0.9909
 122/1688 [=>............................] - ETA: 2s - loss: 0.0250 - accuracy: 0.9921
 152/1688 [=>............................] - ETA: 2s - loss: 0.0259 - accuracy: 0.9922
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0266 - accuracy: 0.9914
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9912
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0290 - accuracy: 0.9910
 272/1688 [===>..........................] - ETA: 2s - loss: 0.0285 - accuracy: 0.9909
 302/1688 [====>.........................] - ETA: 2s - loss: 0.0287 - accuracy: 0.9908
 332/1688 [====>.........................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9909
 362/1688 [=====>........................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9909
 392/1688 [=====>........................] - ETA: 2s - loss: 0.0308 - accuracy: 0.9904
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0313 - accuracy: 0.9899
 452/1688 [=======>......................] - ETA: 2s - loss: 0.0316 - accuracy: 0.9897
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0318 - accuracy: 0.9896
 512/1688 [========>.....................] - ETA: 1s - loss: 0.0328 - accuracy: 0.9895
 542/1688 [========>.....................] - ETA: 1s - loss: 0.0326 - accuracy: 0.9893
 572/1688 [=========>....................] - ETA: 1s - loss: 0.0337 - accuracy: 0.9893
 603/1688 [=========>....................] - ETA: 1s - loss: 0.0337 - accuracy: 0.9893
 633/1688 [==========>...................] - ETA: 1s - loss: 0.0343 - accuracy: 0.9892
 663/1688 [==========>...................] - ETA: 1s - loss: 0.0350 - accuracy: 0.9891
 693/1688 [===========>..................] - ETA: 1s - loss: 0.0353 - accuracy: 0.9888
 723/1688 [===========>..................] - ETA: 1s - loss: 0.0353 - accuracy: 0.9888
 753/1688 [============>.................] - ETA: 1s - loss: 0.0351 - accuracy: 0.9888
 783/1688 [============>.................] - ETA: 1s - loss: 0.0349 - accuracy: 0.9889
 812/1688 [=============>................] - ETA: 1s - loss: 0.0347 - accuracy: 0.9890
 843/1688 [=============>................] - ETA: 1s - loss: 0.0343 - accuracy: 0.9891
 873/1688 [==============>...............] - ETA: 1s - loss: 0.0342 - accuracy: 0.9891
 903/1688 [===============>..............] - ETA: 1s - loss: 0.0341 - accuracy: 0.9891
 933/1688 [===============>..............] - ETA: 1s - loss: 0.0341 - accuracy: 0.9892
 963/1688 [================>.............] - ETA: 1s - loss: 0.0342 - accuracy: 0.9891
 994/1688 [================>.............] - ETA: 1s - loss: 0.0349 - accuracy: 0.9890
1025/1688 [=================>............] - ETA: 1s - loss: 0.0351 - accuracy: 0.9888
1055/1688 [=================>............] - ETA: 1s - loss: 0.0353 - accuracy: 0.9886
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0353 - accuracy: 0.9886
1115/1688 [==================>...........] - ETA: 0s - loss: 0.0350 - accuracy: 0.9887
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1175/1688 [===================>..........] - ETA: 0s - loss: 0.0349 - accuracy: 0.9889
1205/1688 [====================>.........] - ETA: 0s - loss: 0.0345 - accuracy: 0.9890
1235/1688 [====================>.........] - ETA: 0s - loss: 0.0345 - accuracy: 0.9889
1265/1688 [=====================>........] - ETA: 0s - loss: 0.0344 - accuracy: 0.9889
1295/1688 [======================>.......] - ETA: 0s - loss: 0.0344 - accuracy: 0.9889
1325/1688 [======================>.......] - ETA: 0s - loss: 0.0345 - accuracy: 0.9888
1355/1688 [=======================>......] - ETA: 0s - loss: 0.0344 - accuracy: 0.9888
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0347 - accuracy: 0.9889
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0345 - accuracy: 0.9889
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0352 - accuracy: 0.9888
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0352 - accuracy: 0.9888
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0351 - accuracy: 0.9888
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0348 - accuracy: 0.9888
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0347 - accuracy: 0.9889
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1654/1688 [============================>.] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1684/1688 [============================>.] - ETA: 0s - loss: 0.0348 - accuracy: 0.9888
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0349 - accuracy: 0.9888 - val_loss: 0.0496 - val_accuracy: 0.9873
Epoch 4/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0038 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0297 - accuracy: 0.9896
  63/1688 [>.............................] - ETA: 2s - loss: 0.0319 - accuracy: 0.9886
  93/1688 [>.............................] - ETA: 2s - loss: 0.0352 - accuracy: 0.9869
 123/1688 [=>............................] - ETA: 2s - loss: 0.0299 - accuracy: 0.9886
 154/1688 [=>............................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9890
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9898
 214/1688 [==>...........................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9902
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9903
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9899
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9895
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0262 - accuracy: 0.9901
 365/1688 [=====>........................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9906
 396/1688 [======>.......................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9911
 427/1688 [======>.......................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9913
 457/1688 [=======>......................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9912
 488/1688 [=======>......................] - ETA: 2s - loss: 0.0243 - accuracy: 0.9914
 519/1688 [========>.....................] - ETA: 1s - loss: 0.0238 - accuracy: 0.9918
 549/1688 [========>.....................] - ETA: 1s - loss: 0.0244 - accuracy: 0.9917
 579/1688 [=========>....................] - ETA: 1s - loss: 0.0252 - accuracy: 0.9916
 609/1688 [=========>....................] - ETA: 1s - loss: 0.0256 - accuracy: 0.9916
 639/1688 [==========>...................] - ETA: 1s - loss: 0.0259 - accuracy: 0.9914
 668/1688 [==========>...................] - ETA: 1s - loss: 0.0267 - accuracy: 0.9912
 698/1688 [===========>..................] - ETA: 1s - loss: 0.0262 - accuracy: 0.9914
 728/1688 [===========>..................] - ETA: 1s - loss: 0.0257 - accuracy: 0.9916
 758/1688 [============>.................] - ETA: 1s - loss: 0.0258 - accuracy: 0.9915
 789/1688 [=============>................] - ETA: 1s - loss: 0.0253 - accuracy: 0.9918
 819/1688 [=============>................] - ETA: 1s - loss: 0.0256 - accuracy: 0.9917
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
 909/1688 [===============>..............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9915
 939/1688 [===============>..............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9915
 969/1688 [================>.............] - ETA: 1s - loss: 0.0253 - accuracy: 0.9916
 999/1688 [================>.............] - ETA: 1s - loss: 0.0250 - accuracy: 0.9916
1030/1688 [=================>............] - ETA: 1s - loss: 0.0251 - accuracy: 0.9916
1060/1688 [=================>............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9916
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
1121/1688 [==================>...........] - ETA: 0s - loss: 0.0257 - accuracy: 0.9915
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0259 - accuracy: 0.9914
1181/1688 [===================>..........] - ETA: 0s - loss: 0.0265 - accuracy: 0.9912
1211/1688 [====================>.........] - ETA: 0s - loss: 0.0266 - accuracy: 0.9913
1242/1688 [=====================>........] - ETA: 0s - loss: 0.0267 - accuracy: 0.9912
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0265 - accuracy: 0.9912
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0261 - accuracy: 0.9913
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0258 - accuracy: 0.9914
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0254 - accuracy: 0.9915
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1422/1688 [========================>.....] - ETA: 0s - loss: 0.0256 - accuracy: 0.9914
1452/1688 [========================>.....] - ETA: 0s - loss: 0.0256 - accuracy: 0.9913
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0256 - accuracy: 0.9913
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0254 - accuracy: 0.9914
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1635/1688 [============================>.] - ETA: 0s - loss: 0.0255 - accuracy: 0.9913
1664/1688 [============================>.] - ETA: 0s - loss: 0.0254 - accuracy: 0.9913
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0255 - accuracy: 0.9914 - val_loss: 0.0499 - val_accuracy: 0.9875
Epoch 5/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0020 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0233 - accuracy: 0.9912
  63/1688 [>.............................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9916
  93/1688 [>.............................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9929
 123/1688 [=>............................] - ETA: 2s - loss: 0.0198 - accuracy: 0.9921
 153/1688 [=>............................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9922
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9925
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0187 - accuracy: 0.9931
 243/1688 [===>..........................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9931
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9928
 303/1688 [====>.........................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9931
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9934
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9936
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9938
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9939
 455/1688 [=======>......................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9940
 485/1688 [=======>......................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9937
 515/1688 [========>.....................] - ETA: 1s - loss: 0.0194 - accuracy: 0.9936
 545/1688 [========>.....................] - ETA: 1s - loss: 0.0193 - accuracy: 0.9936
 575/1688 [=========>....................] - ETA: 1s - loss: 0.0191 - accuracy: 0.9937
 605/1688 [=========>....................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9935
 634/1688 [==========>...................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9935
 665/1688 [==========>...................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9935
 695/1688 [===========>..................] - ETA: 1s - loss: 0.0199 - accuracy: 0.9933
 725/1688 [===========>..................] - ETA: 1s - loss: 0.0201 - accuracy: 0.9933
 755/1688 [============>.................] - ETA: 1s - loss: 0.0202 - accuracy: 0.9932
 785/1688 [============>.................] - ETA: 1s - loss: 0.0202 - accuracy: 0.9932
 815/1688 [=============>................] - ETA: 1s - loss: 0.0208 - accuracy: 0.9929
 845/1688 [==============>...............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9930
 873/1688 [==============>...............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9930
 904/1688 [===============>..............] - ETA: 1s - loss: 0.0208 - accuracy: 0.9929
 934/1688 [===============>..............] - ETA: 1s - loss: 0.0205 - accuracy: 0.9930
 964/1688 [================>.............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9931
 994/1688 [================>.............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9929
1024/1688 [=================>............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9927
1055/1688 [=================>............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9927
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9927
1115/1688 [==================>...........] - ETA: 0s - loss: 0.0208 - accuracy: 0.9929
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0207 - accuracy: 0.9929
1176/1688 [===================>..........] - ETA: 0s - loss: 0.0209 - accuracy: 0.9929
1206/1688 [====================>.........] - ETA: 0s - loss: 0.0208 - accuracy: 0.9928
1236/1688 [====================>.........] - ETA: 0s - loss: 0.0206 - accuracy: 0.9929
1266/1688 [=====================>........] - ETA: 0s - loss: 0.0209 - accuracy: 0.9928
1296/1688 [======================>.......] - ETA: 0s - loss: 0.0210 - accuracy: 0.9927
1325/1688 [======================>.......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9927
1355/1688 [=======================>......] - ETA: 0s - loss: 0.0211 - accuracy: 0.9927
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0212 - accuracy: 0.9927
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9928
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0210 - accuracy: 0.9928
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0214 - accuracy: 0.9927
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0219 - accuracy: 0.9925
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0216 - accuracy: 0.9927
1627/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1657/1688 [============================>.] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1687/1688 [============================>.] - ETA: 0s - loss: 0.0224 - accuracy: 0.9924
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0224 - accuracy: 0.9924 - val_loss: 0.0493 - val_accuracy: 0.9877
Epoch 6/10

   1/1688 [..............................] - ETA: 2s - loss: 2.0065e-04 - accuracy: 1.0000
  34/1688 [..............................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9945    
  64/1688 [>.............................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9937
  94/1688 [>.............................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9947
 125/1688 [=>............................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9942
 155/1688 [=>............................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9948
 185/1688 [==>...........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9949
 215/1688 [==>...........................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9951
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9950
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9949
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9951
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9954
 363/1688 [=====>........................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9954
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9955
 424/1688 [======>.......................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9955
 454/1688 [=======>......................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9955
 484/1688 [=======>......................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9955
 513/1688 [========>.....................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
 543/1688 [========>.....................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9955
 573/1688 [=========>....................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
 603/1688 [=========>....................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9952
 633/1688 [==========>...................] - ETA: 1s - loss: 0.0136 - accuracy: 0.9952
 662/1688 [==========>...................] - ETA: 1s - loss: 0.0134 - accuracy: 0.9953
 692/1688 [===========>..................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9954
 722/1688 [===========>..................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9955
 752/1688 [============>.................] - ETA: 1s - loss: 0.0129 - accuracy: 0.9955
 782/1688 [============>.................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9954
 812/1688 [=============>................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
 843/1688 [=============>................] - ETA: 1s - loss: 0.0130 - accuracy: 0.9955
 873/1688 [==============>...............] - ETA: 1s - loss: 0.0133 - accuracy: 0.9955
 903/1688 [===============>..............] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
 932/1688 [===============>..............] - ETA: 1s - loss: 0.0134 - accuracy: 0.9955
 962/1688 [================>.............] - ETA: 1s - loss: 0.0137 - accuracy: 0.9955
 993/1688 [================>.............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9955
1023/1688 [=================>............] - ETA: 1s - loss: 0.0139 - accuracy: 0.9954
1053/1688 [=================>............] - ETA: 1s - loss: 0.0140 - accuracy: 0.9953
1083/1688 [==================>...........] - ETA: 1s - loss: 0.0143 - accuracy: 0.9952
1113/1688 [==================>...........] - ETA: 0s - loss: 0.0143 - accuracy: 0.9953
1142/1688 [===================>..........] - ETA: 0s - loss: 0.0144 - accuracy: 0.9953
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0150 - accuracy: 0.9951
1203/1688 [====================>.........] - ETA: 0s - loss: 0.0155 - accuracy: 0.9950
1234/1688 [====================>.........] - ETA: 0s - loss: 0.0157 - accuracy: 0.9949
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1294/1688 [=====================>........] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1324/1688 [======================>.......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1355/1688 [=======================>......] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1385/1688 [=======================>......] - ETA: 0s - loss: 0.0162 - accuracy: 0.9947
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0162 - accuracy: 0.9947
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0165 - accuracy: 0.9947
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0170 - accuracy: 0.9946
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0170 - accuracy: 0.9946
1564/1688 [==========================>...] - ETA: 0s - loss: 0.0171 - accuracy: 0.9946
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0170 - accuracy: 0.9947
1625/1688 [===========================>..] - ETA: 0s - loss: 0.0170 - accuracy: 0.9947
1655/1688 [============================>.] - ETA: 0s - loss: 0.0170 - accuracy: 0.9947
1685/1688 [============================>.] - ETA: 0s - loss: 0.0170 - accuracy: 0.9946
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0170 - accuracy: 0.9946 - val_loss: 0.0490 - val_accuracy: 0.9890
Epoch 7/10

   1/1688 [..............................] - ETA: 3s - loss: 6.6795e-04 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0048 - accuracy: 1.0000    
  61/1688 [>.............................] - ETA: 2s - loss: 0.0047 - accuracy: 0.9995
  91/1688 [>.............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9986
 121/1688 [=>............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9985
 152/1688 [=>............................] - ETA: 2s - loss: 0.0058 - accuracy: 0.9984
 181/1688 [==>...........................] - ETA: 2s - loss: 0.0058 - accuracy: 0.9981
 211/1688 [==>...........................] - ETA: 2s - loss: 0.0072 - accuracy: 0.9979
 241/1688 [===>..........................] - ETA: 2s - loss: 0.0074 - accuracy: 0.9979
 271/1688 [===>..........................] - ETA: 2s - loss: 0.0074 - accuracy: 0.9979
 301/1688 [====>.........................] - ETA: 2s - loss: 0.0084 - accuracy: 0.9975
 331/1688 [====>.........................] - ETA: 2s - loss: 0.0085 - accuracy: 0.9973
 361/1688 [=====>........................] - ETA: 2s - loss: 0.0082 - accuracy: 0.9973
 391/1688 [=====>........................] - ETA: 2s - loss: 0.0088 - accuracy: 0.9972
 421/1688 [======>.......................] - ETA: 2s - loss: 0.0091 - accuracy: 0.9970
 452/1688 [=======>......................] - ETA: 2s - loss: 0.0089 - accuracy: 0.9970
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9968
 512/1688 [========>.....................] - ETA: 1s - loss: 0.0100 - accuracy: 0.9966
 542/1688 [========>.....................] - ETA: 1s - loss: 0.0105 - accuracy: 0.9964
 572/1688 [=========>....................] - ETA: 1s - loss: 0.0107 - accuracy: 0.9963
 603/1688 [=========>....................] - ETA: 1s - loss: 0.0113 - accuracy: 0.9961
 633/1688 [==========>...................] - ETA: 1s - loss: 0.0118 - accuracy: 0.9959
 663/1688 [==========>...................] - ETA: 1s - loss: 0.0117 - accuracy: 0.9958
 694/1688 [===========>..................] - ETA: 1s - loss: 0.0117 - accuracy: 0.9958
 724/1688 [===========>..................] - ETA: 1s - loss: 0.0121 - accuracy: 0.9957
 755/1688 [============>.................] - ETA: 1s - loss: 0.0122 - accuracy: 0.9956
 785/1688 [============>.................] - ETA: 1s - loss: 0.0120 - accuracy: 0.9956
 816/1688 [=============>................] - ETA: 1s - loss: 0.0118 - accuracy: 0.9957
 846/1688 [==============>...............] - ETA: 1s - loss: 0.0116 - accuracy: 0.9958
 877/1688 [==============>...............] - ETA: 1s - loss: 0.0114 - accuracy: 0.9959
 908/1688 [===============>..............] - ETA: 1s - loss: 0.0114 - accuracy: 0.9959
 937/1688 [===============>..............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9959
 967/1688 [================>.............] - ETA: 1s - loss: 0.0111 - accuracy: 0.9960
 998/1688 [================>.............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9960
1028/1688 [=================>............] - ETA: 1s - loss: 0.0116 - accuracy: 0.9959
1059/1688 [=================>............] - ETA: 1s - loss: 0.0119 - accuracy: 0.9958
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0119 - accuracy: 0.9958
1119/1688 [==================>...........] - ETA: 0s - loss: 0.0120 - accuracy: 0.9957
1150/1688 [===================>..........] - ETA: 0s - loss: 0.0121 - accuracy: 0.9956
1180/1688 [===================>..........] - ETA: 0s - loss: 0.0120 - accuracy: 0.9956
1210/1688 [====================>.........] - ETA: 0s - loss: 0.0119 - accuracy: 0.9956
1240/1688 [=====================>........] - ETA: 0s - loss: 0.0117 - accuracy: 0.9957
1270/1688 [=====================>........] - ETA: 0s - loss: 0.0116 - accuracy: 0.9958
1299/1688 [======================>.......] - ETA: 0s - loss: 0.0116 - accuracy: 0.9958
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0116 - accuracy: 0.9958
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0119 - accuracy: 0.9956
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0120 - accuracy: 0.9956
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0120 - accuracy: 0.9956
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9956
1479/1688 [=========================>....] - ETA: 0s - loss: 0.0120 - accuracy: 0.9955
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0121 - accuracy: 0.9955
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0122 - accuracy: 0.9954
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0125 - accuracy: 0.9954
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0125 - accuracy: 0.9954
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0128 - accuracy: 0.9952
1661/1688 [============================>.] - ETA: 0s - loss: 0.0127 - accuracy: 0.9953
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0128 - accuracy: 0.9951 - val_loss: 0.0794 - val_accuracy: 0.9837
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0106 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9951
  62/1688 [>.............................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9965
  92/1688 [>.............................] - ETA: 2s - loss: 0.0084 - accuracy: 0.9973
 121/1688 [=>............................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9979
 152/1688 [=>............................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9984
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9986
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9984
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0070 - accuracy: 0.9974
 302/1688 [====>.........................] - ETA: 2s - loss: 0.0087 - accuracy: 0.9966
 332/1688 [====>.........................] - ETA: 2s - loss: 0.0090 - accuracy: 0.9965
 362/1688 [=====>........................] - ETA: 2s - loss: 0.0091 - accuracy: 0.9965
 392/1688 [=====>........................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9963
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9961
 452/1688 [=======>......................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9961
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9963
 512/1688 [========>.....................] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
 542/1688 [========>.....................] - ETA: 1s - loss: 0.0093 - accuracy: 0.9965
 572/1688 [=========>....................] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
 603/1688 [=========>....................] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
 633/1688 [==========>...................] - ETA: 1s - loss: 0.0095 - accuracy: 0.9964
 664/1688 [==========>...................] - ETA: 1s - loss: 0.0097 - accuracy: 0.9963
 694/1688 [===========>..................] - ETA: 1s - loss: 0.0097 - accuracy: 0.9964
 724/1688 [===========>..................] - ETA: 1s - loss: 0.0099 - accuracy: 0.9965
 754/1688 [============>.................] - ETA: 1s - loss: 0.0098 - accuracy: 0.9965
 785/1688 [============>.................] - ETA: 1s - loss: 0.0099 - accuracy: 0.9965
 815/1688 [=============>................] - ETA: 1s - loss: 0.0097 - accuracy: 0.9966
 844/1688 [==============>...............] - ETA: 1s - loss: 0.0103 - accuracy: 0.9963
 875/1688 [==============>...............] - ETA: 1s - loss: 0.0105 - accuracy: 0.9962
 905/1688 [===============>..............] - ETA: 1s - loss: 0.0104 - accuracy: 0.9963
 935/1688 [===============>..............] - ETA: 1s - loss: 0.0104 - accuracy: 0.9963
 965/1688 [================>.............] - ETA: 1s - loss: 0.0105 - accuracy: 0.9963
 995/1688 [================>.............] - ETA: 1s - loss: 0.0110 - accuracy: 0.9961
1025/1688 [=================>............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9961
1055/1688 [=================>............] - ETA: 1s - loss: 0.0111 - accuracy: 0.9961
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0111 - accuracy: 0.9961
1115/1688 [==================>...........] - ETA: 0s - loss: 0.0110 - accuracy: 0.9961
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0109 - accuracy: 0.9962
1175/1688 [===================>..........] - ETA: 0s - loss: 0.0109 - accuracy: 0.9962
1205/1688 [====================>.........] - ETA: 0s - loss: 0.0110 - accuracy: 0.9962
1234/1688 [====================>.........] - ETA: 0s - loss: 0.0109 - accuracy: 0.9962
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0109 - accuracy: 0.9962
1293/1688 [=====================>........] - ETA: 0s - loss: 0.0110 - accuracy: 0.9962
1324/1688 [======================>.......] - ETA: 0s - loss: 0.0112 - accuracy: 0.9961
1354/1688 [=======================>......] - ETA: 0s - loss: 0.0115 - accuracy: 0.9961
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0117 - accuracy: 0.9959
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0121 - accuracy: 0.9958
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0118 - accuracy: 0.9958
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1564/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1653/1688 [============================>.] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1683/1688 [============================>.] - ETA: 0s - loss: 0.0118 - accuracy: 0.9958
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0118 - accuracy: 0.9958 - val_loss: 0.0493 - val_accuracy: 0.9898
Epoch 9/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0040 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0031 - accuracy: 0.9991
  64/1688 [>.............................] - ETA: 2s - loss: 0.0027 - accuracy: 0.9995
  94/1688 [>.............................] - ETA: 2s - loss: 0.0037 - accuracy: 0.9990
 124/1688 [=>............................] - ETA: 2s - loss: 0.0040 - accuracy: 0.9990
 154/1688 [=>............................] - ETA: 2s - loss: 0.0051 - accuracy: 0.9986
 185/1688 [==>...........................] - ETA: 2s - loss: 0.0076 - accuracy: 0.9978
 215/1688 [==>...........................] - ETA: 2s - loss: 0.0080 - accuracy: 0.9975
 245/1688 [===>..........................] - ETA: 2s - loss: 0.0080 - accuracy: 0.9976
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0080 - accuracy: 0.9974
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0085 - accuracy: 0.9973
 336/1688 [====>.........................] - ETA: 2s - loss: 0.0087 - accuracy: 0.9973
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0082 - accuracy: 0.9975
 397/1688 [======>.......................] - ETA: 2s - loss: 0.0083 - accuracy: 0.9974
 427/1688 [======>.......................] - ETA: 2s - loss: 0.0088 - accuracy: 0.9971
 457/1688 [=======>......................] - ETA: 2s - loss: 0.0087 - accuracy: 0.9971
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0085 - accuracy: 0.9971
 516/1688 [========>.....................] - ETA: 1s - loss: 0.0086 - accuracy: 0.9972
 547/1688 [========>.....................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9971
 577/1688 [=========>....................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9971
 607/1688 [=========>....................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9971
 638/1688 [==========>...................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9971
 669/1688 [==========>...................] - ETA: 1s - loss: 0.0091 - accuracy: 0.9971
 699/1688 [===========>..................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9971
 729/1688 [===========>..................] - ETA: 1s - loss: 0.0091 - accuracy: 0.9970
 758/1688 [============>.................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9971
 789/1688 [=============>................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9971
 819/1688 [=============>................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9971
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9969
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9969
 909/1688 [===============>..............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9969
 939/1688 [===============>..............] - ETA: 1s - loss: 0.0094 - accuracy: 0.9969
 969/1688 [================>.............] - ETA: 1s - loss: 0.0096 - accuracy: 0.9969
1000/1688 [================>.............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9968
1030/1688 [=================>............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9968
1060/1688 [=================>............] - ETA: 1s - loss: 0.0097 - accuracy: 0.9968
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0096 - accuracy: 0.9968
1120/1688 [==================>...........] - ETA: 0s - loss: 0.0098 - accuracy: 0.9967
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0100 - accuracy: 0.9967
1182/1688 [====================>.........] - ETA: 0s - loss: 0.0099 - accuracy: 0.9967
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0099 - accuracy: 0.9968
1242/1688 [=====================>........] - ETA: 0s - loss: 0.0098 - accuracy: 0.9968
1273/1688 [=====================>........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9968
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0098 - accuracy: 0.9968
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0097 - accuracy: 0.9967
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0097 - accuracy: 0.9967
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0098 - accuracy: 0.9966
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0097 - accuracy: 0.9966
1455/1688 [========================>.....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9966
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9966
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9966
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0099 - accuracy: 0.9965
1635/1688 [============================>.] - ETA: 0s - loss: 0.0100 - accuracy: 0.9965
1665/1688 [============================>.] - ETA: 0s - loss: 0.0100 - accuracy: 0.9965
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0100 - accuracy: 0.9965 - val_loss: 0.0507 - val_accuracy: 0.9893
Epoch 10/10

   1/1688 [..............................] - ETA: 2s - loss: 5.2356e-04 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0046 - accuracy: 0.9991    
  64/1688 [>.............................] - ETA: 2s - loss: 0.0046 - accuracy: 0.9990
  94/1688 [>.............................] - ETA: 2s - loss: 0.0050 - accuracy: 0.9990
 124/1688 [=>............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9987
 154/1688 [=>............................] - ETA: 2s - loss: 0.0051 - accuracy: 0.9986
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0055 - accuracy: 0.9981
 214/1688 [==>...........................] - ETA: 2s - loss: 0.0064 - accuracy: 0.9978
 244/1688 [===>..........................] - ETA: 2s - loss: 0.0065 - accuracy: 0.9978
 275/1688 [===>..........................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9977
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0067 - accuracy: 0.9978
 336/1688 [====>.........................] - ETA: 2s - loss: 0.0064 - accuracy: 0.9980
 367/1688 [=====>........................] - ETA: 2s - loss: 0.0064 - accuracy: 0.9980
 397/1688 [======>.......................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9982
 427/1688 [======>.......................] - ETA: 2s - loss: 0.0060 - accuracy: 0.9982
 457/1688 [=======>......................] - ETA: 2s - loss: 0.0057 - accuracy: 0.9983
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9983
 517/1688 [========>.....................] - ETA: 1s - loss: 0.0055 - accuracy: 0.9984
 548/1688 [========>.....................] - ETA: 1s - loss: 0.0055 - accuracy: 0.9984
 578/1688 [=========>....................] - ETA: 1s - loss: 0.0056 - accuracy: 0.9984
 609/1688 [=========>....................] - ETA: 1s - loss: 0.0054 - accuracy: 0.9985
 640/1688 [==========>...................] - ETA: 1s - loss: 0.0060 - accuracy: 0.9983
 670/1688 [==========>...................] - ETA: 1s - loss: 0.0067 - accuracy: 0.9981
 701/1688 [===========>..................] - ETA: 1s - loss: 0.0071 - accuracy: 0.9980
 731/1688 [===========>..................] - ETA: 1s - loss: 0.0075 - accuracy: 0.9979
 761/1688 [============>.................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9980
 791/1688 [=============>................] - ETA: 1s - loss: 0.0074 - accuracy: 0.9979
 821/1688 [=============>................] - ETA: 1s - loss: 0.0077 - accuracy: 0.9978
 851/1688 [==============>...............] - ETA: 1s - loss: 0.0080 - accuracy: 0.9976
 881/1688 [==============>...............] - ETA: 1s - loss: 0.0079 - accuracy: 0.9976
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0081 - accuracy: 0.9975
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9975
 972/1688 [================>.............] - ETA: 1s - loss: 0.0081 - accuracy: 0.9976
1002/1688 [================>.............] - ETA: 1s - loss: 0.0081 - accuracy: 0.9975
1032/1688 [=================>............] - ETA: 1s - loss: 0.0080 - accuracy: 0.9975
1062/1688 [=================>............] - ETA: 1s - loss: 0.0080 - accuracy: 0.9975
1092/1688 [==================>...........] - ETA: 1s - loss: 0.0081 - accuracy: 0.9974
1123/1688 [==================>...........] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1152/1688 [===================>..........] - ETA: 0s - loss: 0.0080 - accuracy: 0.9974
1182/1688 [====================>.........] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0083 - accuracy: 0.9974
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1273/1688 [=====================>........] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1333/1688 [======================>.......] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0082 - accuracy: 0.9973
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0084 - accuracy: 0.9973
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0084 - accuracy: 0.9972
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0085 - accuracy: 0.9972
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0085 - accuracy: 0.9973
1636/1688 [============================>.] - ETA: 0s - loss: 0.0085 - accuracy: 0.9973
1666/1688 [============================>.] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0088 - accuracy: 0.9972 - val_loss: 0.0524 - val_accuracy: 0.9895
Test score: 0.04670824855566025
Test accuracy: 0.9882000088691711

4. Model quantization

We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.

For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).

The quantized model is a Keras model where the neural layers (Conv2D, Dense) and the ReLU layers are replaced with custom CNN2SNN quantized layers (QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function folds the batch normalization layers into the corresponding neural layer. The new weights are computed according to this folding operation.

from cnn2snn import quantize

model_quantized = quantize(model_keras,
                           input_weight_quantization=8,
                           weight_quantization=4,
                           activ_quantization=4)
model_quantized.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (QuantizedConv2D)     (None, 26, 26, 32)        320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0
_________________________________________________________________
re_lu (ActivationDiscreteRel (None, 13, 13, 32)        0
_________________________________________________________________
separable_conv2d (QuantizedS (None, 13, 13, 64)        2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64)          0
_________________________________________________________________
re_lu_1 (ActivationDiscreteR (None, 7, 7, 64)          0
_________________________________________________________________
flatten (Flatten)            (None, 3136)              0
_________________________________________________________________
dense (QuantizedDense)       (None, 10)                31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________

Check the quantized model accuracy.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])

Out:

Test accuracy after 8-4-4 quantization: 0.9843999743461609

Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.

Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.

Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.

from cnn2snn import quantize_layer

model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])

# To recover the original model accuracy, a quantization-aware training phase
# is required.

Out:

Test accuracy after low bitwidth quantization: 0.6850000023841858

5. Model fine tuning (quantization-aware training)

This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.

Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])

Out:

Epoch 1/5

   1/1688 [..............................] - ETA: 16:13 - loss: 0.7107 - accuracy: 0.7188
  22/1688 [..............................] - ETA: 4s - loss: 0.5075 - accuracy: 0.8239   
  43/1688 [..............................] - ETA: 4s - loss: 0.3858 - accuracy: 0.8743
  64/1688 [>.............................] - ETA: 3s - loss: 0.3116 - accuracy: 0.9023
  85/1688 [>.............................] - ETA: 3s - loss: 0.2666 - accuracy: 0.9173
 106/1688 [>.............................] - ETA: 3s - loss: 0.2376 - accuracy: 0.9263
 127/1688 [=>............................] - ETA: 3s - loss: 0.2171 - accuracy: 0.9323
 148/1688 [=>............................] - ETA: 3s - loss: 0.1984 - accuracy: 0.9383
 169/1688 [==>...........................] - ETA: 3s - loss: 0.1871 - accuracy: 0.9406
 190/1688 [==>...........................] - ETA: 3s - loss: 0.1748 - accuracy: 0.9441
 212/1688 [==>...........................] - ETA: 3s - loss: 0.1656 - accuracy: 0.9472
 233/1688 [===>..........................] - ETA: 3s - loss: 0.1567 - accuracy: 0.9505
 254/1688 [===>..........................] - ETA: 3s - loss: 0.1482 - accuracy: 0.9531
 275/1688 [===>..........................] - ETA: 3s - loss: 0.1402 - accuracy: 0.9555
 296/1688 [====>.........................] - ETA: 3s - loss: 0.1339 - accuracy: 0.9577
 317/1688 [====>.........................] - ETA: 3s - loss: 0.1289 - accuracy: 0.9590
 338/1688 [=====>........................] - ETA: 3s - loss: 0.1247 - accuracy: 0.9598
 359/1688 [=====>........................] - ETA: 3s - loss: 0.1194 - accuracy: 0.9616
 380/1688 [=====>........................] - ETA: 3s - loss: 0.1163 - accuracy: 0.9623
 402/1688 [======>.......................] - ETA: 3s - loss: 0.1131 - accuracy: 0.9632
 423/1688 [======>.......................] - ETA: 3s - loss: 0.1105 - accuracy: 0.9637
 444/1688 [======>.......................] - ETA: 3s - loss: 0.1073 - accuracy: 0.9648
 465/1688 [=======>......................] - ETA: 2s - loss: 0.1045 - accuracy: 0.9659
 486/1688 [=======>......................] - ETA: 2s - loss: 0.1021 - accuracy: 0.9668
 507/1688 [========>.....................] - ETA: 2s - loss: 0.1002 - accuracy: 0.9675
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0976 - accuracy: 0.9682
 549/1688 [========>.....................] - ETA: 2s - loss: 0.0953 - accuracy: 0.9689
 570/1688 [=========>....................] - ETA: 2s - loss: 0.0934 - accuracy: 0.9693
 591/1688 [=========>....................] - ETA: 2s - loss: 0.0916 - accuracy: 0.9699
 612/1688 [=========>....................] - ETA: 2s - loss: 0.0904 - accuracy: 0.9701
 633/1688 [==========>...................] - ETA: 2s - loss: 0.0891 - accuracy: 0.9705
 654/1688 [==========>...................] - ETA: 2s - loss: 0.0875 - accuracy: 0.9709
 675/1688 [==========>...................] - ETA: 2s - loss: 0.0860 - accuracy: 0.9713
 696/1688 [===========>..................] - ETA: 2s - loss: 0.0843 - accuracy: 0.9719
 717/1688 [===========>..................] - ETA: 2s - loss: 0.0830 - accuracy: 0.9724
 738/1688 [============>.................] - ETA: 2s - loss: 0.0816 - accuracy: 0.9729
 759/1688 [============>.................] - ETA: 2s - loss: 0.0808 - accuracy: 0.9732
 780/1688 [============>.................] - ETA: 2s - loss: 0.0797 - accuracy: 0.9736
 802/1688 [=============>................] - ETA: 2s - loss: 0.0788 - accuracy: 0.9740
 824/1688 [=============>................] - ETA: 2s - loss: 0.0774 - accuracy: 0.9744
 845/1688 [==============>...............] - ETA: 2s - loss: 0.0767 - accuracy: 0.9749
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0755 - accuracy: 0.9753
 887/1688 [==============>...............] - ETA: 1s - loss: 0.0744 - accuracy: 0.9756
 909/1688 [===============>..............] - ETA: 1s - loss: 0.0740 - accuracy: 0.9757
 930/1688 [===============>..............] - ETA: 1s - loss: 0.0730 - accuracy: 0.9760
 951/1688 [===============>..............] - ETA: 1s - loss: 0.0721 - accuracy: 0.9764
 972/1688 [================>.............] - ETA: 1s - loss: 0.0715 - accuracy: 0.9766
 993/1688 [================>.............] - ETA: 1s - loss: 0.0707 - accuracy: 0.9769
1014/1688 [=================>............] - ETA: 1s - loss: 0.0702 - accuracy: 0.9770
1036/1688 [=================>............] - ETA: 1s - loss: 0.0693 - accuracy: 0.9772
1057/1688 [=================>............] - ETA: 1s - loss: 0.0685 - accuracy: 0.9775
1078/1688 [==================>...........] - ETA: 1s - loss: 0.0677 - accuracy: 0.9777
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0672 - accuracy: 0.9779
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0665 - accuracy: 0.9782
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0661 - accuracy: 0.9784
1162/1688 [===================>..........] - ETA: 1s - loss: 0.0655 - accuracy: 0.9785
1183/1688 [====================>.........] - ETA: 1s - loss: 0.0654 - accuracy: 0.9785
1204/1688 [====================>.........] - ETA: 1s - loss: 0.0653 - accuracy: 0.9786
1225/1688 [====================>.........] - ETA: 1s - loss: 0.0648 - accuracy: 0.9787
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0645 - accuracy: 0.9788
1268/1688 [=====================>........] - ETA: 1s - loss: 0.0641 - accuracy: 0.9790
1290/1688 [=====================>........] - ETA: 0s - loss: 0.0638 - accuracy: 0.9792
1311/1688 [======================>.......] - ETA: 0s - loss: 0.0636 - accuracy: 0.9792
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0630 - accuracy: 0.9794
1353/1688 [=======================>......] - ETA: 0s - loss: 0.0624 - accuracy: 0.9796
1374/1688 [=======================>......] - ETA: 0s - loss: 0.0619 - accuracy: 0.9797
1396/1688 [=======================>......] - ETA: 0s - loss: 0.0615 - accuracy: 0.9799
1418/1688 [========================>.....] - ETA: 0s - loss: 0.0608 - accuracy: 0.9801
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0603 - accuracy: 0.9803
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0597 - accuracy: 0.9805
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0592 - accuracy: 0.9806
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0587 - accuracy: 0.9808
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0584 - accuracy: 0.9808
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0583 - accuracy: 0.9809
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0578 - accuracy: 0.9811
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0575 - accuracy: 0.9812
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0571 - accuracy: 0.9813
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0572 - accuracy: 0.9813
1651/1688 [============================>.] - ETA: 0s - loss: 0.0569 - accuracy: 0.9814
1673/1688 [============================>.] - ETA: 0s - loss: 0.0569 - accuracy: 0.9814
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0567 - accuracy: 0.9814 - val_loss: 0.0608 - val_accuracy: 0.9838
Epoch 2/5

   1/1688 [..............................] - ETA: 4s - loss: 0.1173 - accuracy: 0.9688
  23/1688 [..............................] - ETA: 3s - loss: 0.0318 - accuracy: 0.9878
  44/1688 [..............................] - ETA: 3s - loss: 0.0346 - accuracy: 0.9886
  65/1688 [>.............................] - ETA: 3s - loss: 0.0333 - accuracy: 0.9894
  86/1688 [>.............................] - ETA: 3s - loss: 0.0324 - accuracy: 0.9895
 107/1688 [>.............................] - ETA: 3s - loss: 0.0292 - accuracy: 0.9907
 129/1688 [=>............................] - ETA: 3s - loss: 0.0315 - accuracy: 0.9898
 151/1688 [=>............................] - ETA: 3s - loss: 0.0308 - accuracy: 0.9897
 172/1688 [==>...........................] - ETA: 3s - loss: 0.0297 - accuracy: 0.9900
 193/1688 [==>...........................] - ETA: 3s - loss: 0.0278 - accuracy: 0.9901
 214/1688 [==>...........................] - ETA: 3s - loss: 0.0277 - accuracy: 0.9902
 235/1688 [===>..........................] - ETA: 3s - loss: 0.0275 - accuracy: 0.9904
 256/1688 [===>..........................] - ETA: 3s - loss: 0.0278 - accuracy: 0.9904
 277/1688 [===>..........................] - ETA: 3s - loss: 0.0267 - accuracy: 0.9907
 298/1688 [====>.........................] - ETA: 3s - loss: 0.0268 - accuracy: 0.9908
 319/1688 [====>.........................] - ETA: 3s - loss: 0.0281 - accuracy: 0.9901
 340/1688 [=====>........................] - ETA: 3s - loss: 0.0288 - accuracy: 0.9899
 361/1688 [=====>........................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9901
 382/1688 [=====>........................] - ETA: 3s - loss: 0.0276 - accuracy: 0.9903
 403/1688 [======>.......................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9901
 424/1688 [======>.......................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9901
 446/1688 [======>.......................] - ETA: 3s - loss: 0.0276 - accuracy: 0.9903
 467/1688 [=======>......................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9904
 488/1688 [=======>......................] - ETA: 2s - loss: 0.0280 - accuracy: 0.9903
 510/1688 [========>.....................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9902
 531/1688 [========>.....................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9900
 552/1688 [========>.....................] - ETA: 2s - loss: 0.0280 - accuracy: 0.9900
 574/1688 [=========>....................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9901
 595/1688 [=========>....................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9901
 616/1688 [=========>....................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9901
 637/1688 [==========>...................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9899
 658/1688 [==========>...................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9901
 679/1688 [===========>..................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9899
 700/1688 [===========>..................] - ETA: 2s - loss: 0.0286 - accuracy: 0.9899
 721/1688 [===========>..................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9900
 743/1688 [============>.................] - ETA: 2s - loss: 0.0283 - accuracy: 0.9902
 764/1688 [============>.................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9902
 785/1688 [============>.................] - ETA: 2s - loss: 0.0283 - accuracy: 0.9902
 806/1688 [=============>................] - ETA: 2s - loss: 0.0286 - accuracy: 0.9902
 827/1688 [=============>................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9903
 848/1688 [==============>...............] - ETA: 2s - loss: 0.0281 - accuracy: 0.9903
 869/1688 [==============>...............] - ETA: 1s - loss: 0.0281 - accuracy: 0.9903
 890/1688 [==============>...............] - ETA: 1s - loss: 0.0282 - accuracy: 0.9902
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0280 - accuracy: 0.9903
 932/1688 [===============>..............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9903
 953/1688 [===============>..............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9902
 974/1688 [================>.............] - ETA: 1s - loss: 0.0284 - accuracy: 0.9902
 996/1688 [================>.............] - ETA: 1s - loss: 0.0280 - accuracy: 0.9903
1017/1688 [=================>............] - ETA: 1s - loss: 0.0278 - accuracy: 0.9903
1039/1688 [=================>............] - ETA: 1s - loss: 0.0275 - accuracy: 0.9904
1060/1688 [=================>............] - ETA: 1s - loss: 0.0275 - accuracy: 0.9904
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0279 - accuracy: 0.9903
1102/1688 [==================>...........] - ETA: 1s - loss: 0.0279 - accuracy: 0.9903
1123/1688 [==================>...........] - ETA: 1s - loss: 0.0285 - accuracy: 0.9902
1144/1688 [===================>..........] - ETA: 1s - loss: 0.0292 - accuracy: 0.9900
1166/1688 [===================>..........] - ETA: 1s - loss: 0.0292 - accuracy: 0.9899
1187/1688 [====================>.........] - ETA: 1s - loss: 0.0291 - accuracy: 0.9899
1208/1688 [====================>.........] - ETA: 1s - loss: 0.0288 - accuracy: 0.9900
1229/1688 [====================>.........] - ETA: 1s - loss: 0.0288 - accuracy: 0.9900
1250/1688 [=====================>........] - ETA: 1s - loss: 0.0291 - accuracy: 0.9900
1271/1688 [=====================>........] - ETA: 1s - loss: 0.0290 - accuracy: 0.9900
1292/1688 [=====================>........] - ETA: 0s - loss: 0.0289 - accuracy: 0.9900
1313/1688 [======================>.......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9900
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9901
1356/1688 [=======================>......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9902
1377/1688 [=======================>......] - ETA: 0s - loss: 0.0286 - accuracy: 0.9903
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0285 - accuracy: 0.9903
1419/1688 [========================>.....] - ETA: 0s - loss: 0.0285 - accuracy: 0.9902
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0286 - accuracy: 0.9902
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9903
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9903
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9902
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0283 - accuracy: 0.9902
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0282 - accuracy: 0.9903
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0282 - accuracy: 0.9902
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0283 - accuracy: 0.9902
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0282 - accuracy: 0.9903
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0281 - accuracy: 0.9903
1650/1688 [============================>.] - ETA: 0s - loss: 0.0280 - accuracy: 0.9903
1671/1688 [============================>.] - ETA: 0s - loss: 0.0280 - accuracy: 0.9903
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0280 - accuracy: 0.9903 - val_loss: 0.0534 - val_accuracy: 0.9863
Epoch 3/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0284 - accuracy: 0.9688
  23/1688 [..............................] - ETA: 3s - loss: 0.0303 - accuracy: 0.9918
  44/1688 [..............................] - ETA: 3s - loss: 0.0258 - accuracy: 0.9929
  66/1688 [>.............................] - ETA: 3s - loss: 0.0262 - accuracy: 0.9920
  87/1688 [>.............................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9928
 108/1688 [>.............................] - ETA: 3s - loss: 0.0224 - accuracy: 0.9931
 129/1688 [=>............................] - ETA: 3s - loss: 0.0218 - accuracy: 0.9932
 151/1688 [=>............................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9934
 172/1688 [==>...........................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9940
 193/1688 [==>...........................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9945
 214/1688 [==>...........................] - ETA: 3s - loss: 0.0175 - accuracy: 0.9950
 235/1688 [===>..........................] - ETA: 3s - loss: 0.0176 - accuracy: 0.9949
 256/1688 [===>..........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9949
 277/1688 [===>..........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9945
 298/1688 [====>.........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9947
 319/1688 [====>.........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9944
 338/1688 [=====>........................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9944
 359/1688 [=====>........................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9936
 381/1688 [=====>........................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9935
 402/1688 [======>.......................] - ETA: 3s - loss: 0.0206 - accuracy: 0.9934
 423/1688 [======>.......................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9934
 444/1688 [======>.......................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9933
 465/1688 [=======>......................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9931
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9931
 508/1688 [========>.....................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9932
 529/1688 [========>.....................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9932
 550/1688 [========>.....................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9930
 571/1688 [=========>....................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9930
 592/1688 [=========>....................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9930
 613/1688 [=========>....................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9930
 634/1688 [==========>...................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9931
 655/1688 [==========>...................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9930
 676/1688 [===========>..................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9929
 698/1688 [===========>..................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9930
 719/1688 [===========>..................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9930
 740/1688 [============>.................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9931
 761/1688 [============>.................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9931
 782/1688 [============>.................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9930
 803/1688 [=============>................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9930
 824/1688 [=============>................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9929
 845/1688 [==============>...............] - ETA: 2s - loss: 0.0212 - accuracy: 0.9930
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0213 - accuracy: 0.9930
 887/1688 [==============>...............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
 908/1688 [===============>..............] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
 929/1688 [===============>..............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
 950/1688 [===============>..............] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
 971/1688 [================>.............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
 993/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
1015/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
1036/1688 [=================>............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1057/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0223 - accuracy: 0.9927
1098/1688 [==================>...........] - ETA: 1s - loss: 0.0226 - accuracy: 0.9926
1119/1688 [==================>...........] - ETA: 1s - loss: 0.0227 - accuracy: 0.9926
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0229 - accuracy: 0.9926
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0229 - accuracy: 0.9925
1182/1688 [====================>.........] - ETA: 1s - loss: 0.0231 - accuracy: 0.9925
1203/1688 [====================>.........] - ETA: 1s - loss: 0.0230 - accuracy: 0.9925
1224/1688 [====================>.........] - ETA: 1s - loss: 0.0230 - accuracy: 0.9925
1245/1688 [=====================>........] - ETA: 1s - loss: 0.0233 - accuracy: 0.9924
1266/1688 [=====================>........] - ETA: 1s - loss: 0.0233 - accuracy: 0.9924
1287/1688 [=====================>........] - ETA: 0s - loss: 0.0233 - accuracy: 0.9924
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0235 - accuracy: 0.9923
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0234 - accuracy: 0.9923
1351/1688 [=======================>......] - ETA: 0s - loss: 0.0233 - accuracy: 0.9923
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0234 - accuracy: 0.9922
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0233 - accuracy: 0.9923
1436/1688 [========================>.....] - ETA: 0s - loss: 0.0235 - accuracy: 0.9922
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0235 - accuracy: 0.9921
1478/1688 [=========================>....] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0232 - accuracy: 0.9923
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0234 - accuracy: 0.9922
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0234 - accuracy: 0.9922
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0235 - accuracy: 0.9922
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0233 - accuracy: 0.9923
1647/1688 [============================>.] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1668/1688 [============================>.] - ETA: 0s - loss: 0.0230 - accuracy: 0.9923
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0229 - accuracy: 0.9924 - val_loss: 0.0564 - val_accuracy: 0.9883
Epoch 4/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0055 - accuracy: 1.0000
  23/1688 [..............................] - ETA: 3s - loss: 0.0107 - accuracy: 0.9973
  45/1688 [..............................] - ETA: 3s - loss: 0.0218 - accuracy: 0.9931
  66/1688 [>.............................] - ETA: 3s - loss: 0.0212 - accuracy: 0.9938
  87/1688 [>.............................] - ETA: 3s - loss: 0.0186 - accuracy: 0.9946
 109/1688 [>.............................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9948
 130/1688 [=>............................] - ETA: 3s - loss: 0.0161 - accuracy: 0.9952
 151/1688 [=>............................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9950
 172/1688 [==>...........................] - ETA: 3s - loss: 0.0155 - accuracy: 0.9947
 193/1688 [==>...........................] - ETA: 3s - loss: 0.0167 - accuracy: 0.9943
 214/1688 [==>...........................] - ETA: 3s - loss: 0.0171 - accuracy: 0.9942
 235/1688 [===>..........................] - ETA: 3s - loss: 0.0172 - accuracy: 0.9940
 256/1688 [===>..........................] - ETA: 3s - loss: 0.0179 - accuracy: 0.9939
 277/1688 [===>..........................] - ETA: 3s - loss: 0.0189 - accuracy: 0.9931
 298/1688 [====>.........................] - ETA: 3s - loss: 0.0196 - accuracy: 0.9927
 319/1688 [====>.........................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9929
 340/1688 [=====>........................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9928
 361/1688 [=====>........................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9926
 382/1688 [=====>........................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9926
 403/1688 [======>.......................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9922
 424/1688 [======>.......................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9921
 445/1688 [======>.......................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9923
 466/1688 [=======>......................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9921
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9922
 508/1688 [========>.....................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9922
 529/1688 [========>.....................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9920
 550/1688 [========>.....................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9920
 571/1688 [=========>....................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9923
 592/1688 [=========>....................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9924
 613/1688 [=========>....................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9926
 634/1688 [==========>...................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9926
 655/1688 [==========>...................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9926
 676/1688 [===========>..................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9926
 697/1688 [===========>..................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9926
 718/1688 [===========>..................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9926
 739/1688 [============>.................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9926
 760/1688 [============>.................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9926
 781/1688 [============>.................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9926
 802/1688 [=============>................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9926
 823/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9927
 844/1688 [==============>...............] - ETA: 2s - loss: 0.0204 - accuracy: 0.9926
 865/1688 [==============>...............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9927
 886/1688 [==============>...............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9928
 907/1688 [===============>..............] - ETA: 1s - loss: 0.0199 - accuracy: 0.9929
 928/1688 [===============>..............] - ETA: 1s - loss: 0.0198 - accuracy: 0.9928
 948/1688 [===============>..............] - ETA: 1s - loss: 0.0197 - accuracy: 0.9929
 969/1688 [================>.............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9929
 990/1688 [================>.............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9929
1011/1688 [================>.............] - ETA: 1s - loss: 0.0199 - accuracy: 0.9929
1032/1688 [=================>............] - ETA: 1s - loss: 0.0199 - accuracy: 0.9928
1053/1688 [=================>............] - ETA: 1s - loss: 0.0197 - accuracy: 0.9929
1074/1688 [==================>...........] - ETA: 1s - loss: 0.0199 - accuracy: 0.9927
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0198 - accuracy: 0.9928
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0199 - accuracy: 0.9929
1137/1688 [===================>..........] - ETA: 1s - loss: 0.0198 - accuracy: 0.9929
1158/1688 [===================>..........] - ETA: 1s - loss: 0.0197 - accuracy: 0.9929
1179/1688 [===================>..........] - ETA: 1s - loss: 0.0196 - accuracy: 0.9929
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0196 - accuracy: 0.9929
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0195 - accuracy: 0.9929
1243/1688 [=====================>........] - ETA: 1s - loss: 0.0195 - accuracy: 0.9929
1264/1688 [=====================>........] - ETA: 1s - loss: 0.0195 - accuracy: 0.9930
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0194 - accuracy: 0.9930
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0193 - accuracy: 0.9930
1327/1688 [======================>.......] - ETA: 0s - loss: 0.0195 - accuracy: 0.9930
1348/1688 [======================>.......] - ETA: 0s - loss: 0.0197 - accuracy: 0.9929
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0198 - accuracy: 0.9929
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0197 - accuracy: 0.9929
1411/1688 [========================>.....] - ETA: 0s - loss: 0.0199 - accuracy: 0.9928
1432/1688 [========================>.....] - ETA: 0s - loss: 0.0199 - accuracy: 0.9928
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0198 - accuracy: 0.9928
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0202 - accuracy: 0.9927
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0202 - accuracy: 0.9927
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0201 - accuracy: 0.9928
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0204 - accuracy: 0.9927
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0205 - accuracy: 0.9927
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0206 - accuracy: 0.9926
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0209 - accuracy: 0.9925
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0211 - accuracy: 0.9923
1642/1688 [============================>.] - ETA: 0s - loss: 0.0211 - accuracy: 0.9924
1663/1688 [============================>.] - ETA: 0s - loss: 0.0209 - accuracy: 0.9924
1684/1688 [============================>.] - ETA: 0s - loss: 0.0210 - accuracy: 0.9924
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0210 - accuracy: 0.9924 - val_loss: 0.0558 - val_accuracy: 0.9892
Epoch 5/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0463 - accuracy: 0.9688
  23/1688 [..............................] - ETA: 3s - loss: 0.0238 - accuracy: 0.9932
  44/1688 [..............................] - ETA: 3s - loss: 0.0230 - accuracy: 0.9943
  65/1688 [>.............................] - ETA: 3s - loss: 0.0229 - accuracy: 0.9933
  87/1688 [>.............................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9932
 108/1688 [>.............................] - ETA: 3s - loss: 0.0237 - accuracy: 0.9928
 129/1688 [=>............................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9930
 150/1688 [=>............................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9925
 171/1688 [==>...........................] - ETA: 3s - loss: 0.0231 - accuracy: 0.9929
 192/1688 [==>...........................] - ETA: 3s - loss: 0.0220 - accuracy: 0.9932
 213/1688 [==>...........................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9931
 234/1688 [===>..........................] - ETA: 3s - loss: 0.0201 - accuracy: 0.9936
 255/1688 [===>..........................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9940
 276/1688 [===>..........................] - ETA: 3s - loss: 0.0197 - accuracy: 0.9935
 297/1688 [====>.........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9939
 318/1688 [====>.........................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9939
 339/1688 [=====>........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9937
 360/1688 [=====>........................] - ETA: 3s - loss: 0.0183 - accuracy: 0.9938
 381/1688 [=====>........................] - ETA: 3s - loss: 0.0181 - accuracy: 0.9938
 402/1688 [======>.......................] - ETA: 3s - loss: 0.0180 - accuracy: 0.9939
 424/1688 [======>.......................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9941
 445/1688 [======>.......................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9940
 466/1688 [=======>......................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9938
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9938
 508/1688 [========>.....................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9938
 529/1688 [========>.....................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9940
 550/1688 [========>.....................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9939
 571/1688 [=========>....................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9939
 591/1688 [=========>....................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9940
 612/1688 [=========>....................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9939
 633/1688 [==========>...................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9939
 654/1688 [==========>...................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9937
 675/1688 [==========>...................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9936
 696/1688 [===========>..................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9936
 717/1688 [===========>..................] - ETA: 2s - loss: 0.0194 - accuracy: 0.9936
 738/1688 [============>.................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9936
 759/1688 [============>.................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9936
 780/1688 [============>.................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9937
 801/1688 [=============>................] - ETA: 2s - loss: 0.0198 - accuracy: 0.9933
 822/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9932
 843/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9931
 864/1688 [==============>...............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9931
 885/1688 [==============>...............] - ETA: 1s - loss: 0.0207 - accuracy: 0.9930
 906/1688 [===============>..............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
 927/1688 [===============>..............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
 948/1688 [===============>..............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9929
 969/1688 [================>.............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9930
 990/1688 [================>.............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
1011/1688 [================>.............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9931
1032/1688 [=================>............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
1054/1688 [=================>............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9930
1075/1688 [==================>...........] - ETA: 1s - loss: 0.0209 - accuracy: 0.9931
1096/1688 [==================>...........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9930
1118/1688 [==================>...........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9930
1139/1688 [===================>..........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9930
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9930
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9929
1202/1688 [====================>.........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9930
1223/1688 [====================>.........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
1244/1688 [=====================>........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9927
1265/1688 [=====================>........] - ETA: 1s - loss: 0.0219 - accuracy: 0.9927
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0219 - accuracy: 0.9927
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0219 - accuracy: 0.9926
1329/1688 [======================>.......] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1350/1688 [======================>.......] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0215 - accuracy: 0.9927
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1435/1688 [========================>.....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1498/1688 [=========================>....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0215 - accuracy: 0.9927
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0219 - accuracy: 0.9926
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1603/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1645/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1666/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1687/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0216 - accuracy: 0.9926 - val_loss: 0.0595 - val_accuracy: 0.9877
Test accuracy after fine tuning: 0.98580002784729

6. Model conversion

After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida Execution Engine.

Note

One needs to supply the coefficients used to rescale the input dataset before the training - here input_scaling.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()

results = model_akida.predict(raw_x_test)
accuracy = (raw_y_test == results).mean()

print('Test accuracy after conversion:', accuracy)

# For non-regression purpose
assert accuracy > 0.97

Out:

                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          3
______________________________________________

                  SW/conv2d-dense (Software)
______________________________________________________________
Layer (type)                  Output shape  Kernel shape
==============================================================
conv2d (InputConv.)           [13, 13, 32]  (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.)  [7, 7, 64]    (3, 3, 32, 1)
______________________________________________________________
                                            (1, 1, 32, 64)
______________________________________________________________
dense (Fully.)                [1, 1, 10]    (1, 1, 3136, 10)
______________________________________________________________

Test accuracy after conversion: 0.9857

Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).

Total running time of the script: ( 0 minutes 57.164 seconds)

Gallery generated by Sphinx-Gallery