CNN conversion flow tutorial

This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.

The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.

Note

Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.

MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.

../../_images/cnn2snn_flow_small.jpg

1. Load and reshape MNIST dataset

After loading, we make 2 transformations on the dataset:

  1. Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.

Note

At this point, we’ll set aside the raw data for testing our converted model in the Akida Execution Engine later.

  1. Rescale the 8-bit loaded data to the range 0-to-1 for training.

Note

Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.

This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.

Also note that we store the scaling values input_scaling for use when preparing the model for the Akida Execution Engine. The implementation of the Akida neural network allows us to completely skip the rescaling step (i.e. the Akida model should be fed with the raw 8-bit values) but that does require information about what scaling was applied prior to training - see below for more details.

import tensorflow as tf
from tensorflow import keras

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)

# Set aside raw test data for use with Akida Execution Engine later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test

# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a

2. Model definition

Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.

An appropriate model for MNIST (inspired by this example) might look something like the following:

model_keras = keras.models.Sequential([
    keras.layers.Conv2D(filters=32, kernel_size=3, input_shape=(28, 28, 1)),
    keras.layers.MaxPool2D(),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.SeparableConv2D(filters=64, kernel_size=3, padding='same'),
    keras.layers.MaxPool2D(padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 26, 26, 32)        320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0
_________________________________________________________________
batch_normalization (BatchNo (None, 13, 13, 32)        128
_________________________________________________________________
re_lu (ReLU)                 (None, 13, 13, 32)        0
_________________________________________________________________
separable_conv2d (SeparableC (None, 13, 13, 64)        2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64)          0
_________________________________________________________________
batch_normalization_1 (Batch (None, 7, 7, 64)          256
_________________________________________________________________
re_lu_1 (ReLU)               (None, 7, 7, 64)          0
_________________________________________________________________
flatten (Flatten)            (None, 3136)              0
_________________________________________________________________
dense (Dense)                (None, 10)                31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________

The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):

  • Standard Conv2D and Dense layers are supported

  • Hidden layers must be followed by a ReLU layer.

  • BatchNormalization must always happen before activations.

  • Convolutional blocks can optionally be followed by a MaxPooling.

The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).

from cnn2snn import check_model_compatibility

print("Model compatible for Akida conversion:",
      check_model_compatibility(model_keras))

Out:

Model compatible for Akida conversion: True

3. Model training

Before going any further, train the model and get its performance. The created model should have achieved a test accuracy a little over 99% after 10 epochs.

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)

score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/10

   1/1688 [..............................] - ETA: 10:10 - loss: 3.3138 - accuracy: 0.0625
  30/1688 [..............................] - ETA: 2s - loss: 1.7556 - accuracy: 0.4486   
  59/1688 [>.............................] - ETA: 2s - loss: 1.3453 - accuracy: 0.5822
  89/1688 [>.............................] - ETA: 2s - loss: 1.1229 - accuracy: 0.6538
 119/1688 [=>............................] - ETA: 2s - loss: 0.9790 - accuracy: 0.6995
 149/1688 [=>............................] - ETA: 2s - loss: 0.8779 - accuracy: 0.7315
 179/1688 [==>...........................] - ETA: 2s - loss: 0.8024 - accuracy: 0.7555
 209/1688 [==>...........................] - ETA: 2s - loss: 0.7433 - accuracy: 0.7742
 238/1688 [===>..........................] - ETA: 2s - loss: 0.6966 - accuracy: 0.7888
 268/1688 [===>..........................] - ETA: 2s - loss: 0.6562 - accuracy: 0.8014
 297/1688 [====>.........................] - ETA: 2s - loss: 0.6226 - accuracy: 0.8118
 327/1688 [====>.........................] - ETA: 2s - loss: 0.5923 - accuracy: 0.8212
 356/1688 [=====>........................] - ETA: 2s - loss: 0.5668 - accuracy: 0.8291
 386/1688 [=====>........................] - ETA: 2s - loss: 0.5435 - accuracy: 0.8363
 416/1688 [======>.......................] - ETA: 2s - loss: 0.5226 - accuracy: 0.8427
 446/1688 [======>.......................] - ETA: 2s - loss: 0.5039 - accuracy: 0.8484
 475/1688 [=======>......................] - ETA: 2s - loss: 0.4876 - accuracy: 0.8535
 505/1688 [=======>......................] - ETA: 2s - loss: 0.4721 - accuracy: 0.8582
 534/1688 [========>.....................] - ETA: 1s - loss: 0.4584 - accuracy: 0.8623
 564/1688 [=========>....................] - ETA: 1s - loss: 0.4453 - accuracy: 0.8663
 593/1688 [=========>....................] - ETA: 1s - loss: 0.4336 - accuracy: 0.8698
 623/1688 [==========>...................] - ETA: 1s - loss: 0.4224 - accuracy: 0.8732
 653/1688 [==========>...................] - ETA: 1s - loss: 0.4119 - accuracy: 0.8763
 683/1688 [===========>..................] - ETA: 1s - loss: 0.4022 - accuracy: 0.8793
 713/1688 [===========>..................] - ETA: 1s - loss: 0.3930 - accuracy: 0.8820
 744/1688 [============>.................] - ETA: 1s - loss: 0.3842 - accuracy: 0.8847
 773/1688 [============>.................] - ETA: 1s - loss: 0.3764 - accuracy: 0.8871
 803/1688 [=============>................] - ETA: 1s - loss: 0.3688 - accuracy: 0.8893
 833/1688 [=============>................] - ETA: 1s - loss: 0.3616 - accuracy: 0.8915
 862/1688 [==============>...............] - ETA: 1s - loss: 0.3550 - accuracy: 0.8935
 891/1688 [==============>...............] - ETA: 1s - loss: 0.3488 - accuracy: 0.8954
 921/1688 [===============>..............] - ETA: 1s - loss: 0.3426 - accuracy: 0.8972
 950/1688 [===============>..............] - ETA: 1s - loss: 0.3370 - accuracy: 0.8989
 980/1688 [================>.............] - ETA: 1s - loss: 0.3315 - accuracy: 0.9005
1010/1688 [================>.............] - ETA: 1s - loss: 0.3262 - accuracy: 0.9021
1040/1688 [=================>............] - ETA: 1s - loss: 0.3211 - accuracy: 0.9036
1070/1688 [==================>...........] - ETA: 1s - loss: 0.3163 - accuracy: 0.9051
1099/1688 [==================>...........] - ETA: 1s - loss: 0.3118 - accuracy: 0.9064
1129/1688 [===================>..........] - ETA: 0s - loss: 0.3073 - accuracy: 0.9078
1159/1688 [===================>..........] - ETA: 0s - loss: 0.3030 - accuracy: 0.9090
1189/1688 [====================>.........] - ETA: 0s - loss: 0.2989 - accuracy: 0.9103
1219/1688 [====================>.........] - ETA: 0s - loss: 0.2950 - accuracy: 0.9114
1249/1688 [=====================>........] - ETA: 0s - loss: 0.2912 - accuracy: 0.9126
1278/1688 [=====================>........] - ETA: 0s - loss: 0.2877 - accuracy: 0.9136
1308/1688 [======================>.......] - ETA: 0s - loss: 0.2842 - accuracy: 0.9146
1338/1688 [======================>.......] - ETA: 0s - loss: 0.2808 - accuracy: 0.9157
1367/1688 [=======================>......] - ETA: 0s - loss: 0.2777 - accuracy: 0.9166
1397/1688 [=======================>......] - ETA: 0s - loss: 0.2745 - accuracy: 0.9176
1427/1688 [========================>.....] - ETA: 0s - loss: 0.2715 - accuracy: 0.9185
1456/1688 [========================>.....] - ETA: 0s - loss: 0.2686 - accuracy: 0.9193
1485/1688 [=========================>....] - ETA: 0s - loss: 0.2659 - accuracy: 0.9202
1514/1688 [=========================>....] - ETA: 0s - loss: 0.2632 - accuracy: 0.9210
1544/1688 [==========================>...] - ETA: 0s - loss: 0.2605 - accuracy: 0.9218
1574/1688 [==========================>...] - ETA: 0s - loss: 0.2579 - accuracy: 0.9226
1604/1688 [===========================>..] - ETA: 0s - loss: 0.2553 - accuracy: 0.9233
1634/1688 [============================>.] - ETA: 0s - loss: 0.2529 - accuracy: 0.9241
1664/1688 [============================>.] - ETA: 0s - loss: 0.2505 - accuracy: 0.9248
1688/1688 [==============================] - ETA: 0s - loss: 0.2487 - accuracy: 0.9253
1688/1688 [==============================] - 4s 2ms/step - loss: 0.2486 - accuracy: 0.9253 - val_loss: 0.0547 - val_accuracy: 0.9853
Epoch 2/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0060 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0425 - accuracy: 0.9876
  62/1688 [>.............................] - ETA: 2s - loss: 0.0430 - accuracy: 0.9870
  92/1688 [>.............................] - ETA: 2s - loss: 0.0425 - accuracy: 0.9869
 122/1688 [=>............................] - ETA: 2s - loss: 0.0423 - accuracy: 0.9869
 152/1688 [=>............................] - ETA: 2s - loss: 0.0436 - accuracy: 0.9866
 181/1688 [==>...........................] - ETA: 2s - loss: 0.0444 - accuracy: 0.9866
 210/1688 [==>...........................] - ETA: 2s - loss: 0.0448 - accuracy: 0.9867
 240/1688 [===>..........................] - ETA: 2s - loss: 0.0449 - accuracy: 0.9868
 270/1688 [===>..........................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9868
 299/1688 [====>.........................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9868
 328/1688 [====>.........................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9868
 358/1688 [=====>........................] - ETA: 2s - loss: 0.0451 - accuracy: 0.9868
 387/1688 [=====>........................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9867
 416/1688 [======>.......................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9867
 445/1688 [======>.......................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9866
 475/1688 [=======>......................] - ETA: 2s - loss: 0.0454 - accuracy: 0.9866
 504/1688 [=======>......................] - ETA: 2s - loss: 0.0455 - accuracy: 0.9866
 533/1688 [========>.....................] - ETA: 1s - loss: 0.0455 - accuracy: 0.9865
 563/1688 [=========>....................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9865
 593/1688 [=========>....................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9865
 623/1688 [==========>...................] - ETA: 1s - loss: 0.0458 - accuracy: 0.9865
 652/1688 [==========>...................] - ETA: 1s - loss: 0.0459 - accuracy: 0.9864
 682/1688 [===========>..................] - ETA: 1s - loss: 0.0460 - accuracy: 0.9864
 711/1688 [===========>..................] - ETA: 1s - loss: 0.0462 - accuracy: 0.9864
 741/1688 [============>.................] - ETA: 1s - loss: 0.0463 - accuracy: 0.9863
 771/1688 [============>.................] - ETA: 1s - loss: 0.0464 - accuracy: 0.9863
 801/1688 [=============>................] - ETA: 1s - loss: 0.0465 - accuracy: 0.9863
 831/1688 [=============>................] - ETA: 1s - loss: 0.0465 - accuracy: 0.9862
 860/1688 [==============>...............] - ETA: 1s - loss: 0.0466 - accuracy: 0.9862
 890/1688 [==============>...............] - ETA: 1s - loss: 0.0467 - accuracy: 0.9862
 921/1688 [===============>..............] - ETA: 1s - loss: 0.0468 - accuracy: 0.9861
 951/1688 [===============>..............] - ETA: 1s - loss: 0.0468 - accuracy: 0.9861
 981/1688 [================>.............] - ETA: 1s - loss: 0.0469 - accuracy: 0.9861
1011/1688 [================>.............] - ETA: 1s - loss: 0.0469 - accuracy: 0.9861
1040/1688 [=================>............] - ETA: 1s - loss: 0.0470 - accuracy: 0.9860
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0470 - accuracy: 0.9860
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0470 - accuracy: 0.9860
1129/1688 [===================>..........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1159/1688 [===================>..........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1189/1688 [====================>.........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1219/1688 [====================>.........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1248/1688 [=====================>........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0472 - accuracy: 0.9858
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0472 - accuracy: 0.9858
1338/1688 [======================>.......] - ETA: 0s - loss: 0.0473 - accuracy: 0.9858
1368/1688 [=======================>......] - ETA: 0s - loss: 0.0473 - accuracy: 0.9858
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0473 - accuracy: 0.9858
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0473 - accuracy: 0.9857
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0473 - accuracy: 0.9857
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1637/1688 [============================>.] - ETA: 0s - loss: 0.0474 - accuracy: 0.9856
1667/1688 [============================>.] - ETA: 0s - loss: 0.0474 - accuracy: 0.9856
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0474 - accuracy: 0.9856 - val_loss: 0.0561 - val_accuracy: 0.9835
Epoch 3/10

   1/1688 [..............................] - ETA: 3s - loss: 3.8257e-04 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9933    
  61/1688 [>.............................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9923
  91/1688 [>.............................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9923
 120/1688 [=>............................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9925
 150/1688 [=>............................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9927
 180/1688 [==>...........................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9928
 210/1688 [==>...........................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9927
 239/1688 [===>..........................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9926
 269/1688 [===>..........................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9925
 299/1688 [====>.........................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9923
 328/1688 [====>.........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9922
 358/1688 [=====>........................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9921
 388/1688 [=====>........................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9920
 417/1688 [======>.......................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9919
 447/1688 [======>.......................] - ETA: 2s - loss: 0.0251 - accuracy: 0.9918
 477/1688 [=======>......................] - ETA: 2s - loss: 0.0253 - accuracy: 0.9918
 507/1688 [========>.....................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9917
 537/1688 [========>.....................] - ETA: 1s - loss: 0.0256 - accuracy: 0.9917
 567/1688 [=========>....................] - ETA: 1s - loss: 0.0257 - accuracy: 0.9916
 597/1688 [=========>....................] - ETA: 1s - loss: 0.0258 - accuracy: 0.9916
 626/1688 [==========>...................] - ETA: 1s - loss: 0.0260 - accuracy: 0.9915
 656/1688 [==========>...................] - ETA: 1s - loss: 0.0261 - accuracy: 0.9915
 686/1688 [===========>..................] - ETA: 1s - loss: 0.0263 - accuracy: 0.9915
 716/1688 [===========>..................] - ETA: 1s - loss: 0.0264 - accuracy: 0.9914
 745/1688 [============>.................] - ETA: 1s - loss: 0.0266 - accuracy: 0.9914
 774/1688 [============>.................] - ETA: 1s - loss: 0.0267 - accuracy: 0.9914
 804/1688 [=============>................] - ETA: 1s - loss: 0.0269 - accuracy: 0.9913
 833/1688 [=============>................] - ETA: 1s - loss: 0.0270 - accuracy: 0.9913
 863/1688 [==============>...............] - ETA: 1s - loss: 0.0272 - accuracy: 0.9912
 893/1688 [==============>...............] - ETA: 1s - loss: 0.0273 - accuracy: 0.9912
 923/1688 [===============>..............] - ETA: 1s - loss: 0.0275 - accuracy: 0.9911
 953/1688 [===============>..............] - ETA: 1s - loss: 0.0276 - accuracy: 0.9911
 982/1688 [================>.............] - ETA: 1s - loss: 0.0277 - accuracy: 0.9911
1011/1688 [================>.............] - ETA: 1s - loss: 0.0278 - accuracy: 0.9911
1041/1688 [=================>............] - ETA: 1s - loss: 0.0279 - accuracy: 0.9910
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0280 - accuracy: 0.9910
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0281 - accuracy: 0.9910
1130/1688 [===================>..........] - ETA: 0s - loss: 0.0282 - accuracy: 0.9909
1160/1688 [===================>..........] - ETA: 0s - loss: 0.0283 - accuracy: 0.9909
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0284 - accuracy: 0.9909
1220/1688 [====================>.........] - ETA: 0s - loss: 0.0285 - accuracy: 0.9908
1250/1688 [=====================>........] - ETA: 0s - loss: 0.0286 - accuracy: 0.9908
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0287 - accuracy: 0.9908
1310/1688 [======================>.......] - ETA: 0s - loss: 0.0287 - accuracy: 0.9908
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0288 - accuracy: 0.9907
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9907
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0290 - accuracy: 0.9907
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0290 - accuracy: 0.9907
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0291 - accuracy: 0.9907
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0292 - accuracy: 0.9906
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0292 - accuracy: 0.9906
1546/1688 [==========================>...] - ETA: 0s - loss: 0.0293 - accuracy: 0.9906
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0293 - accuracy: 0.9906
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0294 - accuracy: 0.9905
1635/1688 [============================>.] - ETA: 0s - loss: 0.0295 - accuracy: 0.9905
1665/1688 [============================>.] - ETA: 0s - loss: 0.0295 - accuracy: 0.9905
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0296 - accuracy: 0.9905 - val_loss: 0.0468 - val_accuracy: 0.9863
Epoch 4/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0101 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9908
  61/1688 [>.............................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9896
  91/1688 [>.............................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9900
 121/1688 [=>............................] - ETA: 2s - loss: 0.0262 - accuracy: 0.9905
 151/1688 [=>............................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9908
 180/1688 [==>...........................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9910
 209/1688 [==>...........................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9912
 239/1688 [===>..........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9913
 269/1688 [===>..........................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9914
 298/1688 [====>.........................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9915
 328/1688 [====>.........................] - ETA: 2s - loss: 0.0236 - accuracy: 0.9916
 358/1688 [=====>........................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9917
 388/1688 [=====>........................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9918
 418/1688 [======>.......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9918
 447/1688 [======>.......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9919
 476/1688 [=======>......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9919
 505/1688 [=======>......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9920
 535/1688 [========>.....................] - ETA: 1s - loss: 0.0233 - accuracy: 0.9920
 564/1688 [=========>....................] - ETA: 1s - loss: 0.0233 - accuracy: 0.9921
 593/1688 [=========>....................] - ETA: 1s - loss: 0.0232 - accuracy: 0.9921
 622/1688 [==========>...................] - ETA: 1s - loss: 0.0231 - accuracy: 0.9922
 651/1688 [==========>...................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9922
 680/1688 [===========>..................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9923
 709/1688 [===========>..................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9923
 739/1688 [============>.................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9923
 768/1688 [============>.................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9924
 798/1688 [=============>................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9924
 827/1688 [=============>................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9924
 857/1688 [==============>...............] - ETA: 1s - loss: 0.0230 - accuracy: 0.9924
 888/1688 [==============>...............] - ETA: 1s - loss: 0.0230 - accuracy: 0.9924
 917/1688 [===============>..............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9924
 947/1688 [===============>..............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9924
 977/1688 [================>.............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9924
1007/1688 [================>.............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9924
1037/1688 [=================>............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9923
1067/1688 [=================>............] - ETA: 1s - loss: 0.0234 - accuracy: 0.9923
1097/1688 [==================>...........] - ETA: 1s - loss: 0.0234 - accuracy: 0.9923
1127/1688 [===================>..........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9923
1156/1688 [===================>..........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9923
1186/1688 [====================>.........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9923
1216/1688 [====================>.........] - ETA: 0s - loss: 0.0236 - accuracy: 0.9923
1245/1688 [=====================>........] - ETA: 0s - loss: 0.0236 - accuracy: 0.9923
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0236 - accuracy: 0.9922
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9922
1452/1688 [========================>.....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9922
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9922
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9922
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0239 - accuracy: 0.9922
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0239 - accuracy: 0.9922
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0239 - accuracy: 0.9922
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0240 - accuracy: 0.9922
1659/1688 [============================>.] - ETA: 0s - loss: 0.0240 - accuracy: 0.9921
1688/1688 [==============================] - ETA: 0s - loss: 0.0241 - accuracy: 0.9921
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0241 - accuracy: 0.9921 - val_loss: 0.0664 - val_accuracy: 0.9842
Epoch 5/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0053 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9965
  60/1688 [>.............................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9964
  90/1688 [>.............................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9963
 119/1688 [=>............................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9963
 149/1688 [=>............................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9964
 179/1688 [==>...........................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9964
 209/1688 [==>...........................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9962
 239/1688 [===>..........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9961
 269/1688 [===>..........................] - ETA: 2s - loss: 0.0121 - accuracy: 0.9959
 299/1688 [====>.........................] - ETA: 2s - loss: 0.0126 - accuracy: 0.9957
 329/1688 [====>.........................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9955
 358/1688 [=====>........................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9954
 388/1688 [=====>........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9952
 417/1688 [======>.......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
 447/1688 [======>.......................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9950
 477/1688 [=======>......................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9949
 507/1688 [========>.....................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9947
 537/1688 [========>.....................] - ETA: 1s - loss: 0.0154 - accuracy: 0.9946
 567/1688 [=========>....................] - ETA: 1s - loss: 0.0157 - accuracy: 0.9946
 597/1688 [=========>....................] - ETA: 1s - loss: 0.0159 - accuracy: 0.9945
 627/1688 [==========>...................] - ETA: 1s - loss: 0.0161 - accuracy: 0.9944
 656/1688 [==========>...................] - ETA: 1s - loss: 0.0163 - accuracy: 0.9944
 685/1688 [===========>..................] - ETA: 1s - loss: 0.0164 - accuracy: 0.9943
 715/1688 [===========>..................] - ETA: 1s - loss: 0.0165 - accuracy: 0.9943
 744/1688 [============>.................] - ETA: 1s - loss: 0.0166 - accuracy: 0.9943
 774/1688 [============>.................] - ETA: 1s - loss: 0.0167 - accuracy: 0.9943
 804/1688 [=============>................] - ETA: 1s - loss: 0.0168 - accuracy: 0.9942
 834/1688 [=============>................] - ETA: 1s - loss: 0.0169 - accuracy: 0.9942
 865/1688 [==============>...............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9942
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0171 - accuracy: 0.9941
 923/1688 [===============>..............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9941
 952/1688 [===============>..............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9941
 981/1688 [================>.............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9941
1010/1688 [================>.............] - ETA: 1s - loss: 0.0175 - accuracy: 0.9940
1040/1688 [=================>............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9940
1069/1688 [=================>............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9940
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0177 - accuracy: 0.9940
1129/1688 [===================>..........] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1159/1688 [===================>..........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1189/1688 [====================>.........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1219/1688 [====================>.........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1248/1688 [=====================>........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1338/1688 [======================>.......] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1397/1688 [=======================>......] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0181 - accuracy: 0.9938
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0181 - accuracy: 0.9938
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0181 - accuracy: 0.9938
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1633/1688 [============================>.] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1663/1688 [============================>.] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0183 - accuracy: 0.9938 - val_loss: 0.0430 - val_accuracy: 0.9890
Epoch 6/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9688
  32/1688 [..............................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9953
  62/1688 [>.............................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9956
  92/1688 [>.............................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9956
 121/1688 [=>............................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9958
 151/1688 [=>............................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9960
 180/1688 [==>...........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9961
 209/1688 [==>...........................] - ETA: 2s - loss: 0.0114 - accuracy: 0.9962
 239/1688 [===>..........................] - ETA: 2s - loss: 0.0114 - accuracy: 0.9963
 269/1688 [===>..........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9963
 298/1688 [====>.........................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9963
 328/1688 [====>.........................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9963
 358/1688 [=====>........................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9963
 388/1688 [=====>........................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9963
 417/1688 [======>.......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9963
 447/1688 [======>.......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9963
 469/1688 [=======>......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9963
 499/1688 [=======>......................] - ETA: 2s - loss: 0.0120 - accuracy: 0.9963
 529/1688 [========>.....................] - ETA: 2s - loss: 0.0121 - accuracy: 0.9963
 558/1688 [========>.....................] - ETA: 1s - loss: 0.0122 - accuracy: 0.9962
 587/1688 [=========>....................] - ETA: 1s - loss: 0.0122 - accuracy: 0.9962
 617/1688 [=========>....................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
 647/1688 [==========>...................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
 677/1688 [===========>..................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
 707/1688 [===========>..................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
 736/1688 [============>.................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
 766/1688 [============>.................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
 796/1688 [=============>................] - ETA: 1s - loss: 0.0124 - accuracy: 0.9962
 826/1688 [=============>................] - ETA: 1s - loss: 0.0124 - accuracy: 0.9962
 856/1688 [==============>...............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
 886/1688 [==============>...............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
 915/1688 [===============>..............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
 945/1688 [===============>..............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
 974/1688 [================>.............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
1004/1688 [================>.............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
1034/1688 [=================>............] - ETA: 1s - loss: 0.0125 - accuracy: 0.9961
1064/1688 [=================>............] - ETA: 1s - loss: 0.0125 - accuracy: 0.9961
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0125 - accuracy: 0.9960
1125/1688 [==================>...........] - ETA: 0s - loss: 0.0125 - accuracy: 0.9960
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0125 - accuracy: 0.9960
1185/1688 [====================>.........] - ETA: 0s - loss: 0.0126 - accuracy: 0.9960
1214/1688 [====================>.........] - ETA: 0s - loss: 0.0126 - accuracy: 0.9960
1244/1688 [=====================>........] - ETA: 0s - loss: 0.0126 - accuracy: 0.9960
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0127 - accuracy: 0.9959
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0127 - accuracy: 0.9959
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0127 - accuracy: 0.9959
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0128 - accuracy: 0.9959
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0128 - accuracy: 0.9959
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0129 - accuracy: 0.9959
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0129 - accuracy: 0.9958
1483/1688 [=========================>....] - ETA: 0s - loss: 0.0129 - accuracy: 0.9958
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0130 - accuracy: 0.9958
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0130 - accuracy: 0.9958
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0131 - accuracy: 0.9958
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0131 - accuracy: 0.9958
1632/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9957
1661/1688 [============================>.] - ETA: 0s - loss: 0.0132 - accuracy: 0.9957
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0132 - accuracy: 0.9957 - val_loss: 0.0523 - val_accuracy: 0.9873
Epoch 7/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0027 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0026 - accuracy: 0.9995
  64/1688 [>.............................] - ETA: 2s - loss: 0.0033 - accuracy: 0.9991
  94/1688 [>.............................] - ETA: 2s - loss: 0.0039 - accuracy: 0.9987
 124/1688 [=>............................] - ETA: 2s - loss: 0.0047 - accuracy: 0.9983
 154/1688 [=>............................] - ETA: 2s - loss: 0.0052 - accuracy: 0.9981
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9979
 214/1688 [==>...........................] - ETA: 2s - loss: 0.0058 - accuracy: 0.9977
 244/1688 [===>..........................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9976
 274/1688 [===>..........................] - ETA: 2s - loss: 0.0066 - accuracy: 0.9974
 304/1688 [====>.........................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9971
 333/1688 [====>.........................] - ETA: 2s - loss: 0.0075 - accuracy: 0.9970
 362/1688 [=====>........................] - ETA: 2s - loss: 0.0078 - accuracy: 0.9969
 392/1688 [=====>........................] - ETA: 2s - loss: 0.0080 - accuracy: 0.9968
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0082 - accuracy: 0.9967
 451/1688 [=======>......................] - ETA: 2s - loss: 0.0083 - accuracy: 0.9967
 480/1688 [=======>......................] - ETA: 2s - loss: 0.0084 - accuracy: 0.9966
 509/1688 [========>.....................] - ETA: 2s - loss: 0.0085 - accuracy: 0.9966
 539/1688 [========>.....................] - ETA: 1s - loss: 0.0086 - accuracy: 0.9966
 569/1688 [=========>....................] - ETA: 1s - loss: 0.0087 - accuracy: 0.9965
 599/1688 [=========>....................] - ETA: 1s - loss: 0.0087 - accuracy: 0.9965
 628/1688 [==========>...................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9965
 658/1688 [==========>...................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9965
 687/1688 [===========>..................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9965
 717/1688 [===========>..................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9965
 747/1688 [============>.................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9965
 776/1688 [============>.................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9965
 806/1688 [=============>................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9965
 836/1688 [=============>................] - ETA: 1s - loss: 0.0091 - accuracy: 0.9965
 865/1688 [==============>...............] - ETA: 1s - loss: 0.0091 - accuracy: 0.9965
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9965
 924/1688 [===============>..............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9965
 953/1688 [===============>..............] - ETA: 1s - loss: 0.0093 - accuracy: 0.9964
 983/1688 [================>.............] - ETA: 1s - loss: 0.0093 - accuracy: 0.9964
1013/1688 [=================>............] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
1043/1688 [=================>............] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
1073/1688 [==================>...........] - ETA: 1s - loss: 0.0095 - accuracy: 0.9964
1103/1688 [==================>...........] - ETA: 1s - loss: 0.0095 - accuracy: 0.9964
1133/1688 [===================>..........] - ETA: 0s - loss: 0.0096 - accuracy: 0.9964
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0096 - accuracy: 0.9963
1193/1688 [====================>.........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9963
1222/1688 [====================>.........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9963
1252/1688 [=====================>........] - ETA: 0s - loss: 0.0098 - accuracy: 0.9963
1281/1688 [=====================>........] - ETA: 0s - loss: 0.0098 - accuracy: 0.9963
1310/1688 [======================>.......] - ETA: 0s - loss: 0.0099 - accuracy: 0.9963
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0099 - accuracy: 0.9963
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0100 - accuracy: 0.9963
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0100 - accuracy: 0.9963
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0101 - accuracy: 0.9962
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0101 - accuracy: 0.9962
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0102 - accuracy: 0.9962
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0102 - accuracy: 0.9962
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0103 - accuracy: 0.9962
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9962
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0104 - accuracy: 0.9962
1637/1688 [============================>.] - ETA: 0s - loss: 0.0104 - accuracy: 0.9962
1667/1688 [============================>.] - ETA: 0s - loss: 0.0105 - accuracy: 0.9961
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0105 - accuracy: 0.9961 - val_loss: 0.0502 - val_accuracy: 0.9883
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 2.6362e-04 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9969    
  61/1688 [>.............................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9968
  91/1688 [>.............................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9964
 120/1688 [=>............................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9961
 150/1688 [=>............................] - ETA: 2s - loss: 0.0120 - accuracy: 0.9961
 180/1688 [==>...........................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9962
 210/1688 [==>...........................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9963
 239/1688 [===>..........................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9964
 268/1688 [===>..........................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9965
 297/1688 [====>.........................] - ETA: 2s - loss: 0.0104 - accuracy: 0.9966
 326/1688 [====>.........................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9967
 355/1688 [=====>........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9967
 385/1688 [=====>........................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9968
 415/1688 [======>.......................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9969
 446/1688 [======>.......................] - ETA: 2s - loss: 0.0093 - accuracy: 0.9970
 475/1688 [=======>......................] - ETA: 2s - loss: 0.0091 - accuracy: 0.9970
 504/1688 [=======>......................] - ETA: 2s - loss: 0.0090 - accuracy: 0.9971
 534/1688 [========>.....................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9971
 564/1688 [=========>....................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9972
 594/1688 [=========>....................] - ETA: 1s - loss: 0.0087 - accuracy: 0.9972
 624/1688 [==========>...................] - ETA: 1s - loss: 0.0086 - accuracy: 0.9972
 653/1688 [==========>...................] - ETA: 1s - loss: 0.0085 - accuracy: 0.9972
 683/1688 [===========>..................] - ETA: 1s - loss: 0.0084 - accuracy: 0.9973
 713/1688 [===========>..................] - ETA: 1s - loss: 0.0084 - accuracy: 0.9973
 743/1688 [============>.................] - ETA: 1s - loss: 0.0083 - accuracy: 0.9973
 772/1688 [============>.................] - ETA: 1s - loss: 0.0083 - accuracy: 0.9973
 801/1688 [=============>................] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
 830/1688 [=============>................] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
 860/1688 [==============>...............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
 890/1688 [==============>...............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
 920/1688 [===============>..............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
 950/1688 [===============>..............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
 980/1688 [================>.............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1009/1688 [================>.............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1039/1688 [=================>............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1069/1688 [=================>............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1098/1688 [==================>...........] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1128/1688 [===================>..........] - ETA: 0s - loss: 0.0082 - accuracy: 0.9973
1158/1688 [===================>..........] - ETA: 0s - loss: 0.0082 - accuracy: 0.9973
1188/1688 [====================>.........] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1217/1688 [====================>.........] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1276/1688 [=====================>........] - ETA: 0s - loss: 0.0084 - accuracy: 0.9973
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0084 - accuracy: 0.9972
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0084 - accuracy: 0.9972
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0085 - accuracy: 0.9972
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0085 - accuracy: 0.9972
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0087 - accuracy: 0.9972
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0087 - accuracy: 0.9971
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0087 - accuracy: 0.9971
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0088 - accuracy: 0.9971
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0088 - accuracy: 0.9971
1632/1688 [============================>.] - ETA: 0s - loss: 0.0089 - accuracy: 0.9971
1662/1688 [============================>.] - ETA: 0s - loss: 0.0089 - accuracy: 0.9971
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0089 - accuracy: 0.9970 - val_loss: 0.0454 - val_accuracy: 0.9905
Epoch 9/10

   1/1688 [..............................] - ETA: 2s - loss: 3.8184e-06 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0052 - accuracy: 0.9985    
  61/1688 [>.............................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9976
  91/1688 [>.............................] - ETA: 2s - loss: 0.0073 - accuracy: 0.9972
 120/1688 [=>............................] - ETA: 2s - loss: 0.0073 - accuracy: 0.9971
 150/1688 [=>............................] - ETA: 2s - loss: 0.0072 - accuracy: 0.9971
 179/1688 [==>...........................] - ETA: 2s - loss: 0.0073 - accuracy: 0.9971
 209/1688 [==>...........................] - ETA: 2s - loss: 0.0072 - accuracy: 0.9972
 238/1688 [===>..........................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9973
 268/1688 [===>..........................] - ETA: 2s - loss: 0.0070 - accuracy: 0.9973
 298/1688 [====>.........................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9974
 327/1688 [====>.........................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9974
 357/1688 [=====>........................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9975
 386/1688 [=====>........................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9975
 416/1688 [======>.......................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9975
 446/1688 [======>.......................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9975
 476/1688 [=======>......................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9974
 505/1688 [=======>......................] - ETA: 2s - loss: 0.0070 - accuracy: 0.9974
 534/1688 [========>.....................] - ETA: 1s - loss: 0.0070 - accuracy: 0.9974
 563/1688 [=========>....................] - ETA: 1s - loss: 0.0071 - accuracy: 0.9974
 592/1688 [=========>....................] - ETA: 1s - loss: 0.0071 - accuracy: 0.9973
 622/1688 [==========>...................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
 652/1688 [==========>...................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
 682/1688 [===========>..................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
 712/1688 [===========>..................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
 741/1688 [============>.................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
 770/1688 [============>.................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
 799/1688 [=============>................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9973
 829/1688 [=============>................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9973
 858/1688 [==============>...............] - ETA: 1s - loss: 0.0073 - accuracy: 0.9973
 888/1688 [==============>...............] - ETA: 1s - loss: 0.0073 - accuracy: 0.9972
 918/1688 [===============>..............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
 947/1688 [===============>..............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
 977/1688 [================>.............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
1006/1688 [================>.............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
1035/1688 [=================>............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
1065/1688 [=================>............] - ETA: 1s - loss: 0.0075 - accuracy: 0.9972
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0075 - accuracy: 0.9972
1124/1688 [==================>...........] - ETA: 0s - loss: 0.0075 - accuracy: 0.9972
1154/1688 [===================>..........] - ETA: 0s - loss: 0.0076 - accuracy: 0.9972
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0076 - accuracy: 0.9971
1214/1688 [====================>.........] - ETA: 0s - loss: 0.0076 - accuracy: 0.9971
1244/1688 [=====================>........] - ETA: 0s - loss: 0.0077 - accuracy: 0.9971
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0077 - accuracy: 0.9971
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0077 - accuracy: 0.9971
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0078 - accuracy: 0.9971
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0078 - accuracy: 0.9971
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0078 - accuracy: 0.9971
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0079 - accuracy: 0.9971
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0079 - accuracy: 0.9971
1483/1688 [=========================>....] - ETA: 0s - loss: 0.0080 - accuracy: 0.9971
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0080 - accuracy: 0.9970
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0081 - accuracy: 0.9970
1573/1688 [==========================>...] - ETA: 0s - loss: 0.0081 - accuracy: 0.9970
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0081 - accuracy: 0.9970
1632/1688 [============================>.] - ETA: 0s - loss: 0.0082 - accuracy: 0.9970
1662/1688 [============================>.] - ETA: 0s - loss: 0.0082 - accuracy: 0.9970
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0082 - accuracy: 0.9970 - val_loss: 0.0587 - val_accuracy: 0.9882
Epoch 10/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0014 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0049 - accuracy: 0.9980
  61/1688 [>.............................] - ETA: 2s - loss: 0.0052 - accuracy: 0.9975
  91/1688 [>.............................] - ETA: 2s - loss: 0.0049 - accuracy: 0.9976
 121/1688 [=>............................] - ETA: 2s - loss: 0.0046 - accuracy: 0.9977
 151/1688 [=>............................] - ETA: 2s - loss: 0.0046 - accuracy: 0.9978
 181/1688 [==>...........................] - ETA: 2s - loss: 0.0045 - accuracy: 0.9979
 211/1688 [==>...........................] - ETA: 2s - loss: 0.0044 - accuracy: 0.9979
 240/1688 [===>..........................] - ETA: 2s - loss: 0.0043 - accuracy: 0.9980
 270/1688 [===>..........................] - ETA: 2s - loss: 0.0043 - accuracy: 0.9981
 300/1688 [====>.........................] - ETA: 2s - loss: 0.0042 - accuracy: 0.9981
 329/1688 [====>.........................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9982
 359/1688 [=====>........................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9982
 389/1688 [=====>........................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
 418/1688 [======>.......................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
 448/1688 [======>.......................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
 477/1688 [=======>......................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
 507/1688 [========>.....................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
 537/1688 [========>.....................] - ETA: 1s - loss: 0.0041 - accuracy: 0.9983
 567/1688 [=========>....................] - ETA: 1s - loss: 0.0041 - accuracy: 0.9983
 597/1688 [=========>....................] - ETA: 1s - loss: 0.0041 - accuracy: 0.9983
 626/1688 [==========>...................] - ETA: 1s - loss: 0.0041 - accuracy: 0.9983
 656/1688 [==========>...................] - ETA: 1s - loss: 0.0042 - accuracy: 0.9983
 686/1688 [===========>..................] - ETA: 1s - loss: 0.0042 - accuracy: 0.9983
 716/1688 [===========>..................] - ETA: 1s - loss: 0.0043 - accuracy: 0.9983
 746/1688 [============>.................] - ETA: 1s - loss: 0.0043 - accuracy: 0.9983
 776/1688 [============>.................] - ETA: 1s - loss: 0.0044 - accuracy: 0.9983
 806/1688 [=============>................] - ETA: 1s - loss: 0.0044 - accuracy: 0.9983
 836/1688 [=============>................] - ETA: 1s - loss: 0.0045 - accuracy: 0.9982
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0045 - accuracy: 0.9982
 895/1688 [==============>...............] - ETA: 1s - loss: 0.0046 - accuracy: 0.9982
 924/1688 [===============>..............] - ETA: 1s - loss: 0.0046 - accuracy: 0.9982
 953/1688 [===============>..............] - ETA: 1s - loss: 0.0047 - accuracy: 0.9982
 982/1688 [================>.............] - ETA: 1s - loss: 0.0047 - accuracy: 0.9982
1012/1688 [================>.............] - ETA: 1s - loss: 0.0048 - accuracy: 0.9982
1042/1688 [=================>............] - ETA: 1s - loss: 0.0048 - accuracy: 0.9982
1072/1688 [==================>...........] - ETA: 1s - loss: 0.0049 - accuracy: 0.9982
1102/1688 [==================>...........] - ETA: 1s - loss: 0.0049 - accuracy: 0.9981
1132/1688 [===================>..........] - ETA: 0s - loss: 0.0050 - accuracy: 0.9981
1161/1688 [===================>..........] - ETA: 0s - loss: 0.0050 - accuracy: 0.9981
1191/1688 [====================>.........] - ETA: 0s - loss: 0.0051 - accuracy: 0.9981
1221/1688 [====================>.........] - ETA: 0s - loss: 0.0052 - accuracy: 0.9981
1250/1688 [=====================>........] - ETA: 0s - loss: 0.0052 - accuracy: 0.9980
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0053 - accuracy: 0.9980
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0054 - accuracy: 0.9980
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0054 - accuracy: 0.9980
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0055 - accuracy: 0.9980
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0055 - accuracy: 0.9979
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0056 - accuracy: 0.9979
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9979
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9979
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0058 - accuracy: 0.9979
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9979
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9979
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9979
1635/1688 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9978
1665/1688 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9978
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0060 - accuracy: 0.9978 - val_loss: 0.0483 - val_accuracy: 0.9902
Test score: 0.052414294332265854
Test accuracy: 0.9876999855041504

4. Model quantization

We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.

For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).

The quantized model is a Keras model where the neural layers (Conv2D, Dense) and the ReLU layers are replaced with custom CNN2SNN quantized layers (QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function folds the batch normalization layers into the corresponding neural layer. The new weights are computed according to this folding operation.

from cnn2snn import quantize

model_quantized = quantize(model_keras,
                           input_weight_quantization=8,
                           weight_quantization=4,
                           activ_quantization=4)
model_quantized.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (QuantizedConv2D)     (None, 26, 26, 32)        320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0
_________________________________________________________________
re_lu (ActivationDiscreteRel (None, 13, 13, 32)        0
_________________________________________________________________
separable_conv2d (QuantizedS (None, 13, 13, 64)        2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64)          0
_________________________________________________________________
re_lu_1 (ActivationDiscreteR (None, 7, 7, 64)          0
_________________________________________________________________
flatten (Flatten)            (None, 3136)              0
_________________________________________________________________
dense (QuantizedDense)       (None, 10)                31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________

Check the quantized model accuracy.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])

Out:

Test accuracy after 8-4-4 quantization: 0.9846000075340271

Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.

Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.

Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.

from cnn2snn import quantize_layer

model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])

# To recover the original model accuracy, a quantization-aware training phase
# is required.

Out:

Test accuracy after low bitwidth quantization: 0.7494999766349792

5. Model fine tuning (quantization-aware training)

This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.

Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])

Out:

Epoch 1/5

   1/1688 [..............................] - ETA: 18:51 - loss: 0.8136 - accuracy: 0.7188
  22/1688 [..............................] - ETA: 4s - loss: 0.5149 - accuracy: 0.8438   
  43/1688 [..............................] - ETA: 4s - loss: 0.3889 - accuracy: 0.8844
  63/1688 [>.............................] - ETA: 4s - loss: 0.3238 - accuracy: 0.9038
  84/1688 [>.............................] - ETA: 3s - loss: 0.2795 - accuracy: 0.9200
 105/1688 [>.............................] - ETA: 3s - loss: 0.2472 - accuracy: 0.9301
 126/1688 [=>............................] - ETA: 3s - loss: 0.2279 - accuracy: 0.9355
 146/1688 [=>............................] - ETA: 3s - loss: 0.2143 - accuracy: 0.9390
 167/1688 [=>............................] - ETA: 3s - loss: 0.1976 - accuracy: 0.9435
 188/1688 [==>...........................] - ETA: 3s - loss: 0.1830 - accuracy: 0.9475
 209/1688 [==>...........................] - ETA: 3s - loss: 0.1706 - accuracy: 0.9508
 230/1688 [===>..........................] - ETA: 3s - loss: 0.1609 - accuracy: 0.9538
 250/1688 [===>..........................] - ETA: 3s - loss: 0.1524 - accuracy: 0.9559
 271/1688 [===>..........................] - ETA: 3s - loss: 0.1450 - accuracy: 0.9580
 291/1688 [====>.........................] - ETA: 3s - loss: 0.1386 - accuracy: 0.9597
 311/1688 [====>.........................] - ETA: 3s - loss: 0.1334 - accuracy: 0.9608
 331/1688 [====>.........................] - ETA: 3s - loss: 0.1296 - accuracy: 0.9619
 352/1688 [=====>........................] - ETA: 3s - loss: 0.1253 - accuracy: 0.9632
 372/1688 [=====>........................] - ETA: 3s - loss: 0.1213 - accuracy: 0.9644
 393/1688 [=====>........................] - ETA: 3s - loss: 0.1178 - accuracy: 0.9652
 413/1688 [======>.......................] - ETA: 3s - loss: 0.1145 - accuracy: 0.9662
 433/1688 [======>.......................] - ETA: 3s - loss: 0.1119 - accuracy: 0.9666
 453/1688 [=======>......................] - ETA: 3s - loss: 0.1086 - accuracy: 0.9676
 473/1688 [=======>......................] - ETA: 3s - loss: 0.1068 - accuracy: 0.9680
 493/1688 [=======>......................] - ETA: 2s - loss: 0.1044 - accuracy: 0.9688
 513/1688 [========>.....................] - ETA: 2s - loss: 0.1017 - accuracy: 0.9696
 533/1688 [========>.....................] - ETA: 2s - loss: 0.0998 - accuracy: 0.9702
 553/1688 [========>.....................] - ETA: 2s - loss: 0.0979 - accuracy: 0.9706
 573/1688 [=========>....................] - ETA: 2s - loss: 0.0960 - accuracy: 0.9711
 594/1688 [=========>....................] - ETA: 2s - loss: 0.0939 - accuracy: 0.9716
 614/1688 [=========>....................] - ETA: 2s - loss: 0.0926 - accuracy: 0.9719
 634/1688 [==========>...................] - ETA: 2s - loss: 0.0924 - accuracy: 0.9717
 654/1688 [==========>...................] - ETA: 2s - loss: 0.0912 - accuracy: 0.9721
 675/1688 [==========>...................] - ETA: 2s - loss: 0.0896 - accuracy: 0.9726
 695/1688 [===========>..................] - ETA: 2s - loss: 0.0882 - accuracy: 0.9732
 715/1688 [===========>..................] - ETA: 2s - loss: 0.0866 - accuracy: 0.9736
 736/1688 [============>.................] - ETA: 2s - loss: 0.0856 - accuracy: 0.9738
 757/1688 [============>.................] - ETA: 2s - loss: 0.0842 - accuracy: 0.9741
 778/1688 [============>.................] - ETA: 2s - loss: 0.0834 - accuracy: 0.9742
 798/1688 [=============>................] - ETA: 2s - loss: 0.0821 - accuracy: 0.9746
 818/1688 [=============>................] - ETA: 2s - loss: 0.0808 - accuracy: 0.9750
 839/1688 [=============>................] - ETA: 2s - loss: 0.0797 - accuracy: 0.9753
 859/1688 [==============>...............] - ETA: 2s - loss: 0.0784 - accuracy: 0.9757
 879/1688 [==============>...............] - ETA: 2s - loss: 0.0774 - accuracy: 0.9759
 899/1688 [==============>...............] - ETA: 1s - loss: 0.0764 - accuracy: 0.9762
 920/1688 [===============>..............] - ETA: 1s - loss: 0.0758 - accuracy: 0.9762
 941/1688 [===============>..............] - ETA: 1s - loss: 0.0751 - accuracy: 0.9764
 961/1688 [================>.............] - ETA: 1s - loss: 0.0742 - accuracy: 0.9767
 981/1688 [================>.............] - ETA: 1s - loss: 0.0734 - accuracy: 0.9769
1002/1688 [================>.............] - ETA: 1s - loss: 0.0728 - accuracy: 0.9770
1022/1688 [=================>............] - ETA: 1s - loss: 0.0721 - accuracy: 0.9771
1042/1688 [=================>............] - ETA: 1s - loss: 0.0716 - accuracy: 0.9772
1063/1688 [=================>............] - ETA: 1s - loss: 0.0711 - accuracy: 0.9775
1083/1688 [==================>...........] - ETA: 1s - loss: 0.0705 - accuracy: 0.9776
1103/1688 [==================>...........] - ETA: 1s - loss: 0.0699 - accuracy: 0.9778
1123/1688 [==================>...........] - ETA: 1s - loss: 0.0694 - accuracy: 0.9779
1144/1688 [===================>..........] - ETA: 1s - loss: 0.0691 - accuracy: 0.9780
1164/1688 [===================>..........] - ETA: 1s - loss: 0.0686 - accuracy: 0.9781
1185/1688 [====================>.........] - ETA: 1s - loss: 0.0681 - accuracy: 0.9783
1206/1688 [====================>.........] - ETA: 1s - loss: 0.0677 - accuracy: 0.9784
1226/1688 [====================>.........] - ETA: 1s - loss: 0.0670 - accuracy: 0.9787
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0664 - accuracy: 0.9789
1268/1688 [=====================>........] - ETA: 1s - loss: 0.0660 - accuracy: 0.9790
1288/1688 [=====================>........] - ETA: 1s - loss: 0.0654 - accuracy: 0.9793
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0648 - accuracy: 0.9795
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0642 - accuracy: 0.9797
1350/1688 [======================>.......] - ETA: 0s - loss: 0.0637 - accuracy: 0.9798
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0632 - accuracy: 0.9799
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0626 - accuracy: 0.9801
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0620 - accuracy: 0.9803
1430/1688 [========================>.....] - ETA: 0s - loss: 0.0616 - accuracy: 0.9804
1450/1688 [========================>.....] - ETA: 0s - loss: 0.0610 - accuracy: 0.9805
1470/1688 [=========================>....] - ETA: 0s - loss: 0.0605 - accuracy: 0.9807
1490/1688 [=========================>....] - ETA: 0s - loss: 0.0604 - accuracy: 0.9808
1510/1688 [=========================>....] - ETA: 0s - loss: 0.0598 - accuracy: 0.9809
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0592 - accuracy: 0.9812
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0589 - accuracy: 0.9812
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0586 - accuracy: 0.9813
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0585 - accuracy: 0.9812
1611/1688 [===========================>..] - ETA: 0s - loss: 0.0582 - accuracy: 0.9812
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0578 - accuracy: 0.9814
1651/1688 [============================>.] - ETA: 0s - loss: 0.0576 - accuracy: 0.9814
1671/1688 [============================>.] - ETA: 0s - loss: 0.0578 - accuracy: 0.9814
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0575 - accuracy: 0.9814 - val_loss: 0.0611 - val_accuracy: 0.9843
Epoch 2/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0625 - accuracy: 0.9688
  22/1688 [..............................] - ETA: 4s - loss: 0.0196 - accuracy: 0.9915
  42/1688 [..............................] - ETA: 4s - loss: 0.0253 - accuracy: 0.9896
  63/1688 [>.............................] - ETA: 4s - loss: 0.0245 - accuracy: 0.9896
  83/1688 [>.............................] - ETA: 4s - loss: 0.0274 - accuracy: 0.9876
 103/1688 [>.............................] - ETA: 3s - loss: 0.0278 - accuracy: 0.9882
 123/1688 [=>............................] - ETA: 3s - loss: 0.0305 - accuracy: 0.9876
 144/1688 [=>............................] - ETA: 3s - loss: 0.0287 - accuracy: 0.9887
 164/1688 [=>............................] - ETA: 3s - loss: 0.0275 - accuracy: 0.9897
 184/1688 [==>...........................] - ETA: 3s - loss: 0.0280 - accuracy: 0.9895
 204/1688 [==>...........................] - ETA: 3s - loss: 0.0272 - accuracy: 0.9899
 225/1688 [==>...........................] - ETA: 3s - loss: 0.0299 - accuracy: 0.9893
 246/1688 [===>..........................] - ETA: 3s - loss: 0.0291 - accuracy: 0.9896
 267/1688 [===>..........................] - ETA: 3s - loss: 0.0297 - accuracy: 0.9892
 287/1688 [====>.........................] - ETA: 3s - loss: 0.0297 - accuracy: 0.9892
 308/1688 [====>.........................] - ETA: 3s - loss: 0.0290 - accuracy: 0.9893
 328/1688 [====>.........................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9896
 348/1688 [=====>........................] - ETA: 3s - loss: 0.0293 - accuracy: 0.9893
 369/1688 [=====>........................] - ETA: 3s - loss: 0.0289 - accuracy: 0.9896
 389/1688 [=====>........................] - ETA: 3s - loss: 0.0284 - accuracy: 0.9899
 409/1688 [======>.......................] - ETA: 3s - loss: 0.0289 - accuracy: 0.9895
 429/1688 [======>.......................] - ETA: 3s - loss: 0.0287 - accuracy: 0.9895
 449/1688 [======>.......................] - ETA: 3s - loss: 0.0289 - accuracy: 0.9894
 470/1688 [=======>......................] - ETA: 3s - loss: 0.0291 - accuracy: 0.9894
 490/1688 [=======>......................] - ETA: 3s - loss: 0.0298 - accuracy: 0.9893
 511/1688 [========>.....................] - ETA: 2s - loss: 0.0297 - accuracy: 0.9894
 531/1688 [========>.....................] - ETA: 2s - loss: 0.0295 - accuracy: 0.9895
 551/1688 [========>.....................] - ETA: 2s - loss: 0.0291 - accuracy: 0.9897
 571/1688 [=========>....................] - ETA: 2s - loss: 0.0298 - accuracy: 0.9895
 591/1688 [=========>....................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9894
 611/1688 [=========>....................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9893
 631/1688 [==========>...................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9893
 652/1688 [==========>...................] - ETA: 2s - loss: 0.0309 - accuracy: 0.9894
 672/1688 [==========>...................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9894
 692/1688 [===========>..................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9893
 713/1688 [===========>..................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9893
 733/1688 [============>.................] - ETA: 2s - loss: 0.0310 - accuracy: 0.9895
 754/1688 [============>.................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9894
 774/1688 [============>.................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9892
 794/1688 [=============>................] - ETA: 2s - loss: 0.0314 - accuracy: 0.9893
 814/1688 [=============>................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9894
 834/1688 [=============>................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9896
 855/1688 [==============>...............] - ETA: 2s - loss: 0.0314 - accuracy: 0.9895
 875/1688 [==============>...............] - ETA: 2s - loss: 0.0313 - accuracy: 0.9895
 895/1688 [==============>...............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9895
 915/1688 [===============>..............] - ETA: 1s - loss: 0.0310 - accuracy: 0.9896
 936/1688 [===============>..............] - ETA: 1s - loss: 0.0306 - accuracy: 0.9897
 956/1688 [===============>..............] - ETA: 1s - loss: 0.0304 - accuracy: 0.9897
 977/1688 [================>.............] - ETA: 1s - loss: 0.0302 - accuracy: 0.9898
 997/1688 [================>.............] - ETA: 1s - loss: 0.0304 - accuracy: 0.9898
1018/1688 [=================>............] - ETA: 1s - loss: 0.0301 - accuracy: 0.9899
1038/1688 [=================>............] - ETA: 1s - loss: 0.0299 - accuracy: 0.9900
1058/1688 [=================>............] - ETA: 1s - loss: 0.0298 - accuracy: 0.9900
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0299 - accuracy: 0.9900
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0300 - accuracy: 0.9899
1119/1688 [==================>...........] - ETA: 1s - loss: 0.0300 - accuracy: 0.9899
1139/1688 [===================>..........] - ETA: 1s - loss: 0.0299 - accuracy: 0.9900
1159/1688 [===================>..........] - ETA: 1s - loss: 0.0296 - accuracy: 0.9901
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0294 - accuracy: 0.9901
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0294 - accuracy: 0.9901
1220/1688 [====================>.........] - ETA: 1s - loss: 0.0291 - accuracy: 0.9901
1241/1688 [=====================>........] - ETA: 1s - loss: 0.0292 - accuracy: 0.9902
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0292 - accuracy: 0.9902
1281/1688 [=====================>........] - ETA: 1s - loss: 0.0289 - accuracy: 0.9903
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0290 - accuracy: 0.9903
1322/1688 [======================>.......] - ETA: 0s - loss: 0.0293 - accuracy: 0.9903
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0293 - accuracy: 0.9903
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0293 - accuracy: 0.9903
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0291 - accuracy: 0.9904
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9904
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0289 - accuracy: 0.9903
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0288 - accuracy: 0.9904
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0291 - accuracy: 0.9903
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0292 - accuracy: 0.9902
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0294 - accuracy: 0.9901
1529/1688 [==========================>...] - ETA: 0s - loss: 0.0294 - accuracy: 0.9901
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0295 - accuracy: 0.9901
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0294 - accuracy: 0.9901
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0292 - accuracy: 0.9902
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0293 - accuracy: 0.9902
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0292 - accuracy: 0.9902
1651/1688 [============================>.] - ETA: 0s - loss: 0.0293 - accuracy: 0.9902
1671/1688 [============================>.] - ETA: 0s - loss: 0.0294 - accuracy: 0.9902
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0293 - accuracy: 0.9902 - val_loss: 0.0620 - val_accuracy: 0.9835
Epoch 3/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0024 - accuracy: 1.0000
  22/1688 [..............................] - ETA: 4s - loss: 0.0204 - accuracy: 0.9943
  42/1688 [..............................] - ETA: 4s - loss: 0.0227 - accuracy: 0.9918
  63/1688 [>.............................] - ETA: 4s - loss: 0.0233 - accuracy: 0.9921
  83/1688 [>.............................] - ETA: 4s - loss: 0.0206 - accuracy: 0.9921
 103/1688 [>.............................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9924
 124/1688 [=>............................] - ETA: 3s - loss: 0.0199 - accuracy: 0.9932
 145/1688 [=>............................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9922
 166/1688 [=>............................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9921
 186/1688 [==>...........................] - ETA: 3s - loss: 0.0222 - accuracy: 0.9916
 206/1688 [==>...........................] - ETA: 3s - loss: 0.0227 - accuracy: 0.9914
 226/1688 [===>..........................] - ETA: 3s - loss: 0.0231 - accuracy: 0.9913
 247/1688 [===>..........................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9905
 267/1688 [===>..........................] - ETA: 3s - loss: 0.0247 - accuracy: 0.9905
 287/1688 [====>.........................] - ETA: 3s - loss: 0.0242 - accuracy: 0.9907
 307/1688 [====>.........................] - ETA: 3s - loss: 0.0234 - accuracy: 0.9911
 327/1688 [====>.........................] - ETA: 3s - loss: 0.0225 - accuracy: 0.9916
 348/1688 [=====>........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9920
 368/1688 [=====>........................] - ETA: 3s - loss: 0.0215 - accuracy: 0.9922
 388/1688 [=====>........................] - ETA: 3s - loss: 0.0210 - accuracy: 0.9923
 408/1688 [======>.......................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9922
 428/1688 [======>.......................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9920
 448/1688 [======>.......................] - ETA: 3s - loss: 0.0212 - accuracy: 0.9921
 469/1688 [=======>......................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9921
 489/1688 [=======>......................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9924
 510/1688 [========>.....................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9924
 530/1688 [========>.....................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9923
 550/1688 [========>.....................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9923
 571/1688 [=========>....................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9922
 591/1688 [=========>....................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9921
 611/1688 [=========>....................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9920
 631/1688 [==========>...................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9922
 652/1688 [==========>...................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9920
 672/1688 [==========>...................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9921
 692/1688 [===========>..................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9920
 713/1688 [===========>..................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9921
 733/1688 [============>.................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9920
 754/1688 [============>.................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9921
 774/1688 [============>.................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9921
 794/1688 [=============>................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9921
 814/1688 [=============>................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9921
 834/1688 [=============>................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9923
 855/1688 [==============>...............] - ETA: 2s - loss: 0.0210 - accuracy: 0.9923
 875/1688 [==============>...............] - ETA: 2s - loss: 0.0210 - accuracy: 0.9923
 896/1688 [==============>...............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9924
 917/1688 [===============>..............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9923
 937/1688 [===============>..............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9924
 957/1688 [================>.............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9924
 977/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9924
 998/1688 [================>.............] - ETA: 1s - loss: 0.0217 - accuracy: 0.9923
1018/1688 [=================>............] - ETA: 1s - loss: 0.0218 - accuracy: 0.9923
1038/1688 [=================>............] - ETA: 1s - loss: 0.0219 - accuracy: 0.9922
1058/1688 [=================>............] - ETA: 1s - loss: 0.0222 - accuracy: 0.9922
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0220 - accuracy: 0.9923
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0219 - accuracy: 0.9923
1119/1688 [==================>...........] - ETA: 1s - loss: 0.0219 - accuracy: 0.9922
1139/1688 [===================>..........] - ETA: 1s - loss: 0.0220 - accuracy: 0.9922
1159/1688 [===================>..........] - ETA: 1s - loss: 0.0222 - accuracy: 0.9921
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0223 - accuracy: 0.9921
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0223 - accuracy: 0.9921
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0225 - accuracy: 0.9921
1242/1688 [=====================>........] - ETA: 1s - loss: 0.0229 - accuracy: 0.9920
1263/1688 [=====================>........] - ETA: 1s - loss: 0.0230 - accuracy: 0.9920
1283/1688 [=====================>........] - ETA: 1s - loss: 0.0232 - accuracy: 0.9919
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0234 - accuracy: 0.9918
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0236 - accuracy: 0.9917
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0239 - accuracy: 0.9916
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9917
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9916
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0239 - accuracy: 0.9916
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0241 - accuracy: 0.9916
1546/1688 [==========================>...] - ETA: 0s - loss: 0.0243 - accuracy: 0.9915
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0246 - accuracy: 0.9914
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0250 - accuracy: 0.9912
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0253 - accuracy: 0.9912
1628/1688 [===========================>..] - ETA: 0s - loss: 0.0254 - accuracy: 0.9912
1649/1688 [============================>.] - ETA: 0s - loss: 0.0253 - accuracy: 0.9912
1670/1688 [============================>.] - ETA: 0s - loss: 0.0256 - accuracy: 0.9912
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0258 - accuracy: 0.9911 - val_loss: 0.0540 - val_accuracy: 0.9872
Epoch 4/5

   1/1688 [..............................] - ETA: 4s - loss: 9.5040e-04 - accuracy: 1.0000
  22/1688 [..............................] - ETA: 4s - loss: 0.0211 - accuracy: 0.9929    
  42/1688 [..............................] - ETA: 4s - loss: 0.0268 - accuracy: 0.9903
  63/1688 [>.............................] - ETA: 4s - loss: 0.0305 - accuracy: 0.9901
  84/1688 [>.............................] - ETA: 4s - loss: 0.0246 - accuracy: 0.9922
 104/1688 [>.............................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9928
 125/1688 [=>............................] - ETA: 3s - loss: 0.0215 - accuracy: 0.9935
 146/1688 [=>............................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9940
 167/1688 [=>............................] - ETA: 3s - loss: 0.0186 - accuracy: 0.9944
 187/1688 [==>...........................] - ETA: 3s - loss: 0.0184 - accuracy: 0.9943
 207/1688 [==>...........................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9940
 227/1688 [===>..........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9942
 248/1688 [===>..........................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9937
 268/1688 [===>..........................] - ETA: 3s - loss: 0.0194 - accuracy: 0.9939
 289/1688 [====>.........................] - ETA: 3s - loss: 0.0194 - accuracy: 0.9941
 309/1688 [====>.........................] - ETA: 3s - loss: 0.0188 - accuracy: 0.9943
 330/1688 [====>.........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9942
 350/1688 [=====>........................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9941
 370/1688 [=====>........................] - ETA: 3s - loss: 0.0188 - accuracy: 0.9943
 390/1688 [=====>........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9945
 410/1688 [======>.......................] - ETA: 3s - loss: 0.0178 - accuracy: 0.9947
 431/1688 [======>.......................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9943
 451/1688 [=======>......................] - ETA: 3s - loss: 0.0184 - accuracy: 0.9942
 472/1688 [=======>......................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9944
 492/1688 [=======>......................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9940
 512/1688 [========>.....................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9940
 532/1688 [========>.....................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9939
 553/1688 [========>.....................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9937
 573/1688 [=========>....................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9935
 594/1688 [=========>....................] - ETA: 2s - loss: 0.0194 - accuracy: 0.9935
 614/1688 [=========>....................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9936
 634/1688 [==========>...................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9935
 655/1688 [==========>...................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9934
 675/1688 [==========>...................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9933
 696/1688 [===========>..................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9934
 716/1688 [===========>..................] - ETA: 2s - loss: 0.0198 - accuracy: 0.9935
 737/1688 [============>.................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9936
 757/1688 [============>.................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9936
 777/1688 [============>.................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9935
 797/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9934
 817/1688 [=============>................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9932
 838/1688 [=============>................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9932
 858/1688 [==============>...............] - ETA: 2s - loss: 0.0214 - accuracy: 0.9930
 878/1688 [==============>...............] - ETA: 2s - loss: 0.0212 - accuracy: 0.9931
 898/1688 [==============>...............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 918/1688 [===============>..............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9930
 938/1688 [===============>..............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 958/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 978/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
 998/1688 [================>.............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9930
1018/1688 [=================>............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9929
1038/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
1059/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9929
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9929
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0214 - accuracy: 0.9928
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9927
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9927
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9927
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0214 - accuracy: 0.9927
1242/1688 [=====================>........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9928
1262/1688 [=====================>........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9929
1282/1688 [=====================>........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9928
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0215 - accuracy: 0.9927
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1385/1688 [=======================>......] - ETA: 0s - loss: 0.0222 - accuracy: 0.9924
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0227 - accuracy: 0.9923
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0225 - accuracy: 0.9923
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0225 - accuracy: 0.9923
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0225 - accuracy: 0.9923
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0225 - accuracy: 0.9923
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0228 - accuracy: 0.9922
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0227 - accuracy: 0.9922
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0229 - accuracy: 0.9922
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0228 - accuracy: 0.9921
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0227 - accuracy: 0.9922
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0228 - accuracy: 0.9921
1650/1688 [============================>.] - ETA: 0s - loss: 0.0230 - accuracy: 0.9921
1671/1688 [============================>.] - ETA: 0s - loss: 0.0229 - accuracy: 0.9921
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0231 - accuracy: 0.9921 - val_loss: 0.0561 - val_accuracy: 0.9873
Epoch 5/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
  21/1688 [..............................] - ETA: 4s - loss: 0.0230 - accuracy: 0.9911
  41/1688 [..............................] - ETA: 4s - loss: 0.0160 - accuracy: 0.9947
  62/1688 [>.............................] - ETA: 4s - loss: 0.0149 - accuracy: 0.9950
  82/1688 [>.............................] - ETA: 4s - loss: 0.0175 - accuracy: 0.9935
 102/1688 [>.............................] - ETA: 4s - loss: 0.0178 - accuracy: 0.9936
 123/1688 [=>............................] - ETA: 3s - loss: 0.0162 - accuracy: 0.9942
 143/1688 [=>............................] - ETA: 3s - loss: 0.0165 - accuracy: 0.9934
 163/1688 [=>............................] - ETA: 3s - loss: 0.0164 - accuracy: 0.9933
 183/1688 [==>...........................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9925
 203/1688 [==>...........................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9929
 223/1688 [==>...........................] - ETA: 3s - loss: 0.0186 - accuracy: 0.9927
 243/1688 [===>..........................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9927
 263/1688 [===>..........................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9928
 283/1688 [====>.........................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9927
 303/1688 [====>.........................] - ETA: 3s - loss: 0.0210 - accuracy: 0.9927
 324/1688 [====>.........................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9929
 344/1688 [=====>........................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9932
 364/1688 [=====>........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9934
 384/1688 [=====>........................] - ETA: 3s - loss: 0.0184 - accuracy: 0.9933
 404/1688 [======>.......................] - ETA: 3s - loss: 0.0189 - accuracy: 0.9933
 425/1688 [======>.......................] - ETA: 3s - loss: 0.0195 - accuracy: 0.9929
 445/1688 [======>.......................] - ETA: 3s - loss: 0.0189 - accuracy: 0.9931
 466/1688 [=======>......................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9932
 487/1688 [=======>......................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9931
 507/1688 [========>.....................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9931
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9933
 548/1688 [========>.....................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9932
 568/1688 [=========>....................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9930
 589/1688 [=========>....................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9929
 610/1688 [=========>....................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9928
 630/1688 [==========>...................] - ETA: 2s - loss: 0.0197 - accuracy: 0.9930
 651/1688 [==========>...................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9929
 671/1688 [==========>...................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9929
 691/1688 [===========>..................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9928
 711/1688 [===========>..................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9927
 731/1688 [===========>..................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9926
 751/1688 [============>.................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9926
 771/1688 [============>.................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9925
 791/1688 [=============>................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9924
 811/1688 [=============>................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9923
 831/1688 [=============>................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9923
 851/1688 [==============>...............] - ETA: 2s - loss: 0.0214 - accuracy: 0.9924
 871/1688 [==============>...............] - ETA: 2s - loss: 0.0211 - accuracy: 0.9924
 892/1688 [==============>...............] - ETA: 2s - loss: 0.0208 - accuracy: 0.9924
 912/1688 [===============>..............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9925
 933/1688 [===============>..............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9925
 953/1688 [===============>..............] - ETA: 1s - loss: 0.0207 - accuracy: 0.9925
 973/1688 [================>.............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9924
 993/1688 [================>.............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9923
1013/1688 [=================>............] - ETA: 1s - loss: 0.0217 - accuracy: 0.9922
1033/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9923
1053/1688 [=================>............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9923
1073/1688 [==================>...........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9923
1093/1688 [==================>...........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9923
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9923
1134/1688 [===================>..........] - ETA: 1s - loss: 0.0210 - accuracy: 0.9924
1154/1688 [===================>..........] - ETA: 1s - loss: 0.0210 - accuracy: 0.9925
1175/1688 [===================>..........] - ETA: 1s - loss: 0.0208 - accuracy: 0.9925
1196/1688 [====================>.........] - ETA: 1s - loss: 0.0208 - accuracy: 0.9925
1216/1688 [====================>.........] - ETA: 1s - loss: 0.0208 - accuracy: 0.9925
1237/1688 [====================>.........] - ETA: 1s - loss: 0.0209 - accuracy: 0.9925
1257/1688 [=====================>........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9924
1277/1688 [=====================>........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9924
1297/1688 [======================>.......] - ETA: 0s - loss: 0.0214 - accuracy: 0.9923
1317/1688 [======================>.......] - ETA: 0s - loss: 0.0212 - accuracy: 0.9923
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0212 - accuracy: 0.9924
1358/1688 [=======================>......] - ETA: 0s - loss: 0.0214 - accuracy: 0.9923
1378/1688 [=======================>......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9924
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0216 - accuracy: 0.9922
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9921
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9921
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9922
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0218 - accuracy: 0.9922
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0220 - accuracy: 0.9921
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0223 - accuracy: 0.9920
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0222 - accuracy: 0.9920
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0220 - accuracy: 0.9921
1642/1688 [============================>.] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1662/1688 [============================>.] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1683/1688 [============================>.] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0219 - accuracy: 0.9921 - val_loss: 0.0633 - val_accuracy: 0.9855
Test accuracy after fine tuning: 0.986299991607666

6. Model conversion

After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida Execution Engine.

Note

One needs to supply the coefficients used to rescale the input dataset before the training - here input_scaling.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()

results = model_akida.predict(raw_x_test)
accuracy = (raw_y_test == results).mean()

print('Test accuracy after conversion:', accuracy)

# For non-regression purpose
assert accuracy > 0.97

Out:

                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          3
______________________________________________

                               SW/conv2d-dense (Software)
________________________________________________________________________________________
Layer (type)                               Output shape  Kernel shape
========================================================================================
conv2d (InputConvolutional)                [13, 13, 32]  (3, 3, 1, 32)
________________________________________________________________________________________
separable_conv2d (SeparableConvolutional)  [7, 7, 64]    (3, 3, 32, 1), (1, 1, 32, 64)
________________________________________________________________________________________
dense (FullyConnected)                     [1, 1, 10]    (1, 1, 3136, 10)
________________________________________________________________________________________


Test accuracy after conversion: 0.986

Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).

Total running time of the script: ( 1 minutes 1.440 seconds)

Gallery generated by Sphinx-Gallery