CNN conversion flow tutorial

This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.

The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.

Note

Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.

MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.

../../_images/cnn2snn_flow_small.jpg

1. Load and reshape MNIST dataset

After loading, we make 2 transformations on the dataset:

  1. Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.

Note

At this point, we’ll set aside the raw data for testing our converted model in the Akida runtime later.

  1. Rescale the 8-bit loaded data to the range 0-to-1 for training.

Note

Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.

This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.

Also note that we store the scaling values input_scaling for use when preparing the model for the Akida neuromorphic IP. The implementation of the Akida neural network allows us to completely skip the rescaling step (i.e. the Akida model should be fed with the raw 8-bit values) but that does require information about what scaling was applied prior to training - see below for more details.

from tensorflow import keras

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)

# Set aside raw test data for use with Akida runtime later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test

# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a

2. Model definition

Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.

An appropriate model for MNIST (inspired by this example) might look something like the following:

model_keras = keras.models.Sequential([
    keras.layers.Conv2D(
        filters=32, kernel_size=3, input_shape=(28, 28, 1), strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.SeparableConv2D(
        filters=64, kernel_size=3, padding='same', strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 conv2d (Conv2D)             (None, 13, 13, 32)        320

 batch_normalization (BatchN  (None, 13, 13, 32)       128
 ormalization)

 re_lu (ReLU)                (None, 13, 13, 32)        0

 separable_conv2d (Separable  (None, 7, 7, 64)         2400
 Conv2D)

 batch_normalization_1 (Batc  (None, 7, 7, 64)         256
 hNormalization)

 re_lu_1 (ReLU)              (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (Dense)               (None, 10)                31370

=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________

The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):

  • Standard Conv2D and Dense layers are supported

  • Hidden layers must be followed by a ReLU layer.

  • BatchNormalization must always happen before activations.

  • Convolutional blocks can optionally be followed by a MaxPooling.

The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).

from cnn2snn import check_model_compatibility

print("Model compatible for Akida conversion:",
      check_model_compatibility(model_keras))

Out:

Model compatible for Akida conversion: True

3. Model training

Before going any further, train the model and get its performance. The created model should have achieved a test accuracy over 98% after 10 epochs.

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)

score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])

Out:

Epoch 1/10

   1/1688 [..............................] - ETA: 11:53 - loss: 2.8070 - accuracy: 0.1250
  29/1688 [..............................] - ETA: 2s - loss: 1.2656 - accuracy: 0.6088   
  57/1688 [>.............................] - ETA: 2s - loss: 0.9022 - accuracy: 0.7314
  85/1688 [>.............................] - ETA: 2s - loss: 0.7527 - accuracy: 0.7776
 114/1688 [=>............................] - ETA: 2s - loss: 0.6401 - accuracy: 0.8133
 143/1688 [=>............................] - ETA: 2s - loss: 0.5821 - accuracy: 0.8285
 171/1688 [==>...........................] - ETA: 2s - loss: 0.5348 - accuracy: 0.8423
 199/1688 [==>...........................] - ETA: 2s - loss: 0.5011 - accuracy: 0.8525
 227/1688 [===>..........................] - ETA: 2s - loss: 0.4727 - accuracy: 0.8597
 256/1688 [===>..........................] - ETA: 2s - loss: 0.4460 - accuracy: 0.8684
 284/1688 [====>.........................] - ETA: 2s - loss: 0.4237 - accuracy: 0.8747
 312/1688 [====>.........................] - ETA: 2s - loss: 0.4071 - accuracy: 0.8795
 341/1688 [=====>........................] - ETA: 2s - loss: 0.3899 - accuracy: 0.8848
 370/1688 [=====>........................] - ETA: 2s - loss: 0.3756 - accuracy: 0.8894
 399/1688 [======>.......................] - ETA: 2s - loss: 0.3602 - accuracy: 0.8947
 427/1688 [======>.......................] - ETA: 2s - loss: 0.3475 - accuracy: 0.8985
 456/1688 [=======>......................] - ETA: 2s - loss: 0.3363 - accuracy: 0.9017
 485/1688 [=======>......................] - ETA: 2s - loss: 0.3266 - accuracy: 0.9048
 514/1688 [========>.....................] - ETA: 2s - loss: 0.3157 - accuracy: 0.9082
 543/1688 [========>.....................] - ETA: 2s - loss: 0.3062 - accuracy: 0.9106
 571/1688 [=========>....................] - ETA: 1s - loss: 0.2976 - accuracy: 0.9132
 600/1688 [=========>....................] - ETA: 1s - loss: 0.2924 - accuracy: 0.9148
 629/1688 [==========>...................] - ETA: 1s - loss: 0.2869 - accuracy: 0.9159
 657/1688 [==========>...................] - ETA: 1s - loss: 0.2804 - accuracy: 0.9178
 686/1688 [===========>..................] - ETA: 1s - loss: 0.2745 - accuracy: 0.9194
 714/1688 [===========>..................] - ETA: 1s - loss: 0.2684 - accuracy: 0.9212
 743/1688 [============>.................] - ETA: 1s - loss: 0.2635 - accuracy: 0.9228
 771/1688 [============>.................] - ETA: 1s - loss: 0.2600 - accuracy: 0.9240
 800/1688 [=============>................] - ETA: 1s - loss: 0.2557 - accuracy: 0.9255
 829/1688 [=============>................] - ETA: 1s - loss: 0.2510 - accuracy: 0.9271
 857/1688 [==============>...............] - ETA: 1s - loss: 0.2472 - accuracy: 0.9283
 885/1688 [==============>...............] - ETA: 1s - loss: 0.2430 - accuracy: 0.9294
 914/1688 [===============>..............] - ETA: 1s - loss: 0.2396 - accuracy: 0.9304
 943/1688 [===============>..............] - ETA: 1s - loss: 0.2353 - accuracy: 0.9316
 972/1688 [================>.............] - ETA: 1s - loss: 0.2317 - accuracy: 0.9325
1001/1688 [================>.............] - ETA: 1s - loss: 0.2277 - accuracy: 0.9334
1030/1688 [=================>............] - ETA: 1s - loss: 0.2247 - accuracy: 0.9344
1059/1688 [=================>............] - ETA: 1s - loss: 0.2216 - accuracy: 0.9351
1088/1688 [==================>...........] - ETA: 1s - loss: 0.2191 - accuracy: 0.9360
1118/1688 [==================>...........] - ETA: 1s - loss: 0.2163 - accuracy: 0.9368
1147/1688 [===================>..........] - ETA: 0s - loss: 0.2138 - accuracy: 0.9375
1176/1688 [===================>..........] - ETA: 0s - loss: 0.2113 - accuracy: 0.9383
1205/1688 [====================>.........] - ETA: 0s - loss: 0.2100 - accuracy: 0.9387
1234/1688 [====================>.........] - ETA: 0s - loss: 0.2074 - accuracy: 0.9393
1262/1688 [=====================>........] - ETA: 0s - loss: 0.2054 - accuracy: 0.9399
1291/1688 [=====================>........] - ETA: 0s - loss: 0.2033 - accuracy: 0.9403
1319/1688 [======================>.......] - ETA: 0s - loss: 0.2012 - accuracy: 0.9409
1347/1688 [======================>.......] - ETA: 0s - loss: 0.1987 - accuracy: 0.9416
1376/1688 [=======================>......] - ETA: 0s - loss: 0.1972 - accuracy: 0.9421
1404/1688 [=======================>......] - ETA: 0s - loss: 0.1953 - accuracy: 0.9427
1433/1688 [========================>.....] - ETA: 0s - loss: 0.1931 - accuracy: 0.9432
1461/1688 [========================>.....] - ETA: 0s - loss: 0.1916 - accuracy: 0.9437
1490/1688 [=========================>....] - ETA: 0s - loss: 0.1903 - accuracy: 0.9442
1519/1688 [=========================>....] - ETA: 0s - loss: 0.1884 - accuracy: 0.9448
1548/1688 [==========================>...] - ETA: 0s - loss: 0.1862 - accuracy: 0.9454
1578/1688 [===========================>..] - ETA: 0s - loss: 0.1846 - accuracy: 0.9458
1607/1688 [===========================>..] - ETA: 0s - loss: 0.1827 - accuracy: 0.9464
1637/1688 [============================>.] - ETA: 0s - loss: 0.1812 - accuracy: 0.9468
1665/1688 [============================>.] - ETA: 0s - loss: 0.1798 - accuracy: 0.9471
1688/1688 [==============================] - ETA: 0s - loss: 0.1785 - accuracy: 0.9475
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1785 - accuracy: 0.9475 - val_loss: 0.0860 - val_accuracy: 0.9765
Epoch 2/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0795 - accuracy: 0.9688
  30/1688 [..............................] - ETA: 2s - loss: 0.0810 - accuracy: 0.9719
  59/1688 [>.............................] - ETA: 2s - loss: 0.0760 - accuracy: 0.9746
  87/1688 [>.............................] - ETA: 2s - loss: 0.0702 - accuracy: 0.9759
 116/1688 [=>............................] - ETA: 2s - loss: 0.0703 - accuracy: 0.9771
 145/1688 [=>............................] - ETA: 2s - loss: 0.0682 - accuracy: 0.9780
 174/1688 [==>...........................] - ETA: 2s - loss: 0.0645 - accuracy: 0.9799
 203/1688 [==>...........................] - ETA: 2s - loss: 0.0664 - accuracy: 0.9791
 233/1688 [===>..........................] - ETA: 2s - loss: 0.0688 - accuracy: 0.9784
 261/1688 [===>..........................] - ETA: 2s - loss: 0.0674 - accuracy: 0.9790
 290/1688 [====>.........................] - ETA: 2s - loss: 0.0668 - accuracy: 0.9793
 319/1688 [====>.........................] - ETA: 2s - loss: 0.0653 - accuracy: 0.9794
 348/1688 [=====>........................] - ETA: 2s - loss: 0.0678 - accuracy: 0.9783
 378/1688 [=====>........................] - ETA: 2s - loss: 0.0670 - accuracy: 0.9782
 406/1688 [======>.......................] - ETA: 2s - loss: 0.0665 - accuracy: 0.9784
 434/1688 [======>.......................] - ETA: 2s - loss: 0.0665 - accuracy: 0.9784
 463/1688 [=======>......................] - ETA: 2s - loss: 0.0665 - accuracy: 0.9786
 492/1688 [=======>......................] - ETA: 2s - loss: 0.0677 - accuracy: 0.9780
 521/1688 [========>.....................] - ETA: 2s - loss: 0.0687 - accuracy: 0.9780
 550/1688 [========>.....................] - ETA: 2s - loss: 0.0682 - accuracy: 0.9782
 579/1688 [=========>....................] - ETA: 1s - loss: 0.0673 - accuracy: 0.9786
 607/1688 [=========>....................] - ETA: 1s - loss: 0.0666 - accuracy: 0.9788
 635/1688 [==========>...................] - ETA: 1s - loss: 0.0665 - accuracy: 0.9790
 663/1688 [==========>...................] - ETA: 1s - loss: 0.0659 - accuracy: 0.9793
 691/1688 [===========>..................] - ETA: 1s - loss: 0.0658 - accuracy: 0.9793
 719/1688 [===========>..................] - ETA: 1s - loss: 0.0665 - accuracy: 0.9792
 748/1688 [============>.................] - ETA: 1s - loss: 0.0657 - accuracy: 0.9794
 778/1688 [============>.................] - ETA: 1s - loss: 0.0657 - accuracy: 0.9793
 807/1688 [=============>................] - ETA: 1s - loss: 0.0660 - accuracy: 0.9794
 836/1688 [=============>................] - ETA: 1s - loss: 0.0668 - accuracy: 0.9788
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0679 - accuracy: 0.9785
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0683 - accuracy: 0.9785
 922/1688 [===============>..............] - ETA: 1s - loss: 0.0678 - accuracy: 0.9787
 951/1688 [===============>..............] - ETA: 1s - loss: 0.0682 - accuracy: 0.9785
 980/1688 [================>.............] - ETA: 1s - loss: 0.0683 - accuracy: 0.9784
1008/1688 [================>.............] - ETA: 1s - loss: 0.0681 - accuracy: 0.9784
1037/1688 [=================>............] - ETA: 1s - loss: 0.0679 - accuracy: 0.9785
1066/1688 [=================>............] - ETA: 1s - loss: 0.0673 - accuracy: 0.9786
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0673 - accuracy: 0.9787
1124/1688 [==================>...........] - ETA: 0s - loss: 0.0677 - accuracy: 0.9786
1153/1688 [===================>..........] - ETA: 0s - loss: 0.0676 - accuracy: 0.9785
1181/1688 [===================>..........] - ETA: 0s - loss: 0.0680 - accuracy: 0.9783
1210/1688 [====================>.........] - ETA: 0s - loss: 0.0683 - accuracy: 0.9784
1238/1688 [=====================>........] - ETA: 0s - loss: 0.0688 - accuracy: 0.9782
1267/1688 [=====================>........] - ETA: 0s - loss: 0.0687 - accuracy: 0.9781
1296/1688 [======================>.......] - ETA: 0s - loss: 0.0685 - accuracy: 0.9782
1325/1688 [======================>.......] - ETA: 0s - loss: 0.0685 - accuracy: 0.9781
1354/1688 [=======================>......] - ETA: 0s - loss: 0.0690 - accuracy: 0.9780
1382/1688 [=======================>......] - ETA: 0s - loss: 0.0692 - accuracy: 0.9780
1411/1688 [========================>.....] - ETA: 0s - loss: 0.0695 - accuracy: 0.9779
1439/1688 [========================>.....] - ETA: 0s - loss: 0.0696 - accuracy: 0.9779
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0696 - accuracy: 0.9779
1496/1688 [=========================>....] - ETA: 0s - loss: 0.0697 - accuracy: 0.9779
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0699 - accuracy: 0.9779
1552/1688 [==========================>...] - ETA: 0s - loss: 0.0700 - accuracy: 0.9781
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0697 - accuracy: 0.9781
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0701 - accuracy: 0.9781
1638/1688 [============================>.] - ETA: 0s - loss: 0.0702 - accuracy: 0.9780
1666/1688 [============================>.] - ETA: 0s - loss: 0.0705 - accuracy: 0.9779
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0703 - accuracy: 0.9780 - val_loss: 0.0572 - val_accuracy: 0.9845
Epoch 3/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0247 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0550 - accuracy: 0.9802
  60/1688 [>.............................] - ETA: 2s - loss: 0.0481 - accuracy: 0.9844
  90/1688 [>.............................] - ETA: 2s - loss: 0.0449 - accuracy: 0.9851
 119/1688 [=>............................] - ETA: 2s - loss: 0.0508 - accuracy: 0.9837
 148/1688 [=>............................] - ETA: 2s - loss: 0.0525 - accuracy: 0.9823
 177/1688 [==>...........................] - ETA: 2s - loss: 0.0512 - accuracy: 0.9827
 205/1688 [==>...........................] - ETA: 2s - loss: 0.0528 - accuracy: 0.9822
 234/1688 [===>..........................] - ETA: 2s - loss: 0.0536 - accuracy: 0.9820
 263/1688 [===>..........................] - ETA: 2s - loss: 0.0518 - accuracy: 0.9830
 291/1688 [====>.........................] - ETA: 2s - loss: 0.0490 - accuracy: 0.9841
 320/1688 [====>.........................] - ETA: 2s - loss: 0.0486 - accuracy: 0.9848
 349/1688 [=====>........................] - ETA: 2s - loss: 0.0479 - accuracy: 0.9847
 377/1688 [=====>........................] - ETA: 2s - loss: 0.0470 - accuracy: 0.9849
 406/1688 [======>.......................] - ETA: 2s - loss: 0.0483 - accuracy: 0.9844
 435/1688 [======>.......................] - ETA: 2s - loss: 0.0474 - accuracy: 0.9848
 463/1688 [=======>......................] - ETA: 2s - loss: 0.0473 - accuracy: 0.9845
 493/1688 [=======>......................] - ETA: 2s - loss: 0.0472 - accuracy: 0.9847
 521/1688 [========>.....................] - ETA: 2s - loss: 0.0481 - accuracy: 0.9846
 550/1688 [========>.....................] - ETA: 2s - loss: 0.0487 - accuracy: 0.9847
 579/1688 [=========>....................] - ETA: 1s - loss: 0.0495 - accuracy: 0.9844
 608/1688 [=========>....................] - ETA: 1s - loss: 0.0494 - accuracy: 0.9844
 636/1688 [==========>...................] - ETA: 1s - loss: 0.0490 - accuracy: 0.9844
 666/1688 [==========>...................] - ETA: 1s - loss: 0.0488 - accuracy: 0.9845
 695/1688 [===========>..................] - ETA: 1s - loss: 0.0479 - accuracy: 0.9848
 724/1688 [===========>..................] - ETA: 1s - loss: 0.0476 - accuracy: 0.9848
 753/1688 [============>.................] - ETA: 1s - loss: 0.0472 - accuracy: 0.9850
 782/1688 [============>.................] - ETA: 1s - loss: 0.0477 - accuracy: 0.9847
 811/1688 [=============>................] - ETA: 1s - loss: 0.0486 - accuracy: 0.9844
 840/1688 [=============>................] - ETA: 1s - loss: 0.0500 - accuracy: 0.9839
 869/1688 [==============>...............] - ETA: 1s - loss: 0.0506 - accuracy: 0.9835
 897/1688 [==============>...............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9835
 926/1688 [===============>..............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9835
 956/1688 [===============>..............] - ETA: 1s - loss: 0.0501 - accuracy: 0.9836
 985/1688 [================>.............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9833
1013/1688 [=================>............] - ETA: 1s - loss: 0.0512 - accuracy: 0.9831
1042/1688 [=================>............] - ETA: 1s - loss: 0.0516 - accuracy: 0.9829
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0516 - accuracy: 0.9829
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0518 - accuracy: 0.9827
1128/1688 [===================>..........] - ETA: 0s - loss: 0.0515 - accuracy: 0.9828
1157/1688 [===================>..........] - ETA: 0s - loss: 0.0513 - accuracy: 0.9828
1185/1688 [====================>.........] - ETA: 0s - loss: 0.0519 - accuracy: 0.9827
1214/1688 [====================>.........] - ETA: 0s - loss: 0.0525 - accuracy: 0.9826
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0524 - accuracy: 0.9826
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0523 - accuracy: 0.9826
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0521 - accuracy: 0.9827
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0519 - accuracy: 0.9828
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0524 - accuracy: 0.9825
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0528 - accuracy: 0.9825
1417/1688 [========================>.....] - ETA: 0s - loss: 0.0525 - accuracy: 0.9826
1446/1688 [========================>.....] - ETA: 0s - loss: 0.0523 - accuracy: 0.9826
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0526 - accuracy: 0.9826
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0523 - accuracy: 0.9827
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0521 - accuracy: 0.9828
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0526 - accuracy: 0.9827
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0523 - accuracy: 0.9828
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0524 - accuracy: 0.9828
1650/1688 [============================>.] - ETA: 0s - loss: 0.0524 - accuracy: 0.9829
1679/1688 [============================>.] - ETA: 0s - loss: 0.0524 - accuracy: 0.9829
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0523 - accuracy: 0.9829 - val_loss: 0.0773 - val_accuracy: 0.9763
Epoch 4/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0232 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9902
  60/1688 [>.............................] - ETA: 2s - loss: 0.0346 - accuracy: 0.9885
  89/1688 [>.............................] - ETA: 2s - loss: 0.0298 - accuracy: 0.9909
 118/1688 [=>............................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9915
 147/1688 [=>............................] - ETA: 2s - loss: 0.0268 - accuracy: 0.9915
 175/1688 [==>...........................] - ETA: 2s - loss: 0.0288 - accuracy: 0.9907
 203/1688 [==>...........................] - ETA: 2s - loss: 0.0297 - accuracy: 0.9900
 231/1688 [===>..........................] - ETA: 2s - loss: 0.0303 - accuracy: 0.9897
 260/1688 [===>..........................] - ETA: 2s - loss: 0.0326 - accuracy: 0.9893
 289/1688 [====>.........................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9896
 318/1688 [====>.........................] - ETA: 2s - loss: 0.0329 - accuracy: 0.9893
 346/1688 [=====>........................] - ETA: 2s - loss: 0.0339 - accuracy: 0.9892
 375/1688 [=====>........................] - ETA: 2s - loss: 0.0332 - accuracy: 0.9893
 404/1688 [======>.......................] - ETA: 2s - loss: 0.0338 - accuracy: 0.9892
 432/1688 [======>.......................] - ETA: 2s - loss: 0.0347 - accuracy: 0.9889
 460/1688 [=======>......................] - ETA: 2s - loss: 0.0362 - accuracy: 0.9885
 488/1688 [=======>......................] - ETA: 2s - loss: 0.0361 - accuracy: 0.9882
 517/1688 [========>.....................] - ETA: 2s - loss: 0.0365 - accuracy: 0.9880
 545/1688 [========>.....................] - ETA: 2s - loss: 0.0363 - accuracy: 0.9881
 574/1688 [=========>....................] - ETA: 1s - loss: 0.0362 - accuracy: 0.9881
 603/1688 [=========>....................] - ETA: 1s - loss: 0.0361 - accuracy: 0.9884
 631/1688 [==========>...................] - ETA: 1s - loss: 0.0359 - accuracy: 0.9885
 660/1688 [==========>...................] - ETA: 1s - loss: 0.0364 - accuracy: 0.9882
 689/1688 [===========>..................] - ETA: 1s - loss: 0.0368 - accuracy: 0.9881
 718/1688 [===========>..................] - ETA: 1s - loss: 0.0369 - accuracy: 0.9879
 747/1688 [============>.................] - ETA: 1s - loss: 0.0373 - accuracy: 0.9876
 776/1688 [============>.................] - ETA: 1s - loss: 0.0370 - accuracy: 0.9878
 806/1688 [=============>................] - ETA: 1s - loss: 0.0373 - accuracy: 0.9876
 834/1688 [=============>................] - ETA: 1s - loss: 0.0381 - accuracy: 0.9873
 862/1688 [==============>...............] - ETA: 1s - loss: 0.0388 - accuracy: 0.9871
 891/1688 [==============>...............] - ETA: 1s - loss: 0.0392 - accuracy: 0.9870
 919/1688 [===============>..............] - ETA: 1s - loss: 0.0390 - accuracy: 0.9869
 948/1688 [===============>..............] - ETA: 1s - loss: 0.0388 - accuracy: 0.9870
 977/1688 [================>.............] - ETA: 1s - loss: 0.0393 - accuracy: 0.9869
1006/1688 [================>.............] - ETA: 1s - loss: 0.0396 - accuracy: 0.9869
1034/1688 [=================>............] - ETA: 1s - loss: 0.0399 - accuracy: 0.9868
1063/1688 [=================>............] - ETA: 1s - loss: 0.0396 - accuracy: 0.9868
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0396 - accuracy: 0.9867
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0397 - accuracy: 0.9867
1149/1688 [===================>..........] - ETA: 0s - loss: 0.0405 - accuracy: 0.9864
1178/1688 [===================>..........] - ETA: 0s - loss: 0.0406 - accuracy: 0.9865
1207/1688 [====================>.........] - ETA: 0s - loss: 0.0407 - accuracy: 0.9864
1236/1688 [====================>.........] - ETA: 0s - loss: 0.0408 - accuracy: 0.9864
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0414 - accuracy: 0.9862
1293/1688 [=====================>........] - ETA: 0s - loss: 0.0417 - accuracy: 0.9861
1321/1688 [======================>.......] - ETA: 0s - loss: 0.0416 - accuracy: 0.9861
1349/1688 [======================>.......] - ETA: 0s - loss: 0.0418 - accuracy: 0.9860
1377/1688 [=======================>......] - ETA: 0s - loss: 0.0422 - accuracy: 0.9859
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0423 - accuracy: 0.9859
1434/1688 [========================>.....] - ETA: 0s - loss: 0.0425 - accuracy: 0.9859
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0423 - accuracy: 0.9860
1491/1688 [=========================>....] - ETA: 0s - loss: 0.0421 - accuracy: 0.9860
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0420 - accuracy: 0.9860
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0417 - accuracy: 0.9861
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0418 - accuracy: 0.9861
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0417 - accuracy: 0.9862
1634/1688 [============================>.] - ETA: 0s - loss: 0.0419 - accuracy: 0.9862
1662/1688 [============================>.] - ETA: 0s - loss: 0.0416 - accuracy: 0.9863
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0416 - accuracy: 0.9863 - val_loss: 0.0912 - val_accuracy: 0.9758
Epoch 5/10

   1/1688 [..............................] - ETA: 3s - loss: 0.1936 - accuracy: 0.9688
  31/1688 [..............................] - ETA: 2s - loss: 0.0397 - accuracy: 0.9889
  60/1688 [>.............................] - ETA: 2s - loss: 0.0323 - accuracy: 0.9891
  89/1688 [>.............................] - ETA: 2s - loss: 0.0345 - accuracy: 0.9884
 118/1688 [=>............................] - ETA: 2s - loss: 0.0325 - accuracy: 0.9897
 147/1688 [=>............................] - ETA: 2s - loss: 0.0296 - accuracy: 0.9904
 176/1688 [==>...........................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9902
 206/1688 [==>...........................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9907
 235/1688 [===>..........................] - ETA: 2s - loss: 0.0290 - accuracy: 0.9903
 264/1688 [===>..........................] - ETA: 2s - loss: 0.0300 - accuracy: 0.9901
 293/1688 [====>.........................] - ETA: 2s - loss: 0.0294 - accuracy: 0.9902
 322/1688 [====>.........................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9909
 351/1688 [=====>........................] - ETA: 2s - loss: 0.0280 - accuracy: 0.9907
 380/1688 [=====>........................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9903
 409/1688 [======>.......................] - ETA: 2s - loss: 0.0304 - accuracy: 0.9904
 437/1688 [======>.......................] - ETA: 2s - loss: 0.0303 - accuracy: 0.9906
 464/1688 [=======>......................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9907
 493/1688 [=======>......................] - ETA: 2s - loss: 0.0304 - accuracy: 0.9904
 522/1688 [========>.....................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9903
 551/1688 [========>.....................] - ETA: 1s - loss: 0.0316 - accuracy: 0.9900
 580/1688 [=========>....................] - ETA: 1s - loss: 0.0316 - accuracy: 0.9899
 609/1688 [=========>....................] - ETA: 1s - loss: 0.0319 - accuracy: 0.9898
 638/1688 [==========>...................] - ETA: 1s - loss: 0.0324 - accuracy: 0.9896
 668/1688 [==========>...................] - ETA: 1s - loss: 0.0318 - accuracy: 0.9898
 696/1688 [===========>..................] - ETA: 1s - loss: 0.0313 - accuracy: 0.9901
 726/1688 [===========>..................] - ETA: 1s - loss: 0.0308 - accuracy: 0.9902
 756/1688 [============>.................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9901
 784/1688 [============>.................] - ETA: 1s - loss: 0.0311 - accuracy: 0.9901
 813/1688 [=============>................] - ETA: 1s - loss: 0.0304 - accuracy: 0.9904
 843/1688 [=============>................] - ETA: 1s - loss: 0.0301 - accuracy: 0.9905
 873/1688 [==============>...............] - ETA: 1s - loss: 0.0304 - accuracy: 0.9906
 901/1688 [===============>..............] - ETA: 1s - loss: 0.0306 - accuracy: 0.9906
 930/1688 [===============>..............] - ETA: 1s - loss: 0.0313 - accuracy: 0.9902
 958/1688 [================>.............] - ETA: 1s - loss: 0.0314 - accuracy: 0.9902
 986/1688 [================>.............] - ETA: 1s - loss: 0.0320 - accuracy: 0.9900
1014/1688 [=================>............] - ETA: 1s - loss: 0.0323 - accuracy: 0.9900
1044/1688 [=================>............] - ETA: 1s - loss: 0.0325 - accuracy: 0.9898
1074/1688 [==================>...........] - ETA: 1s - loss: 0.0328 - accuracy: 0.9898
1102/1688 [==================>...........] - ETA: 1s - loss: 0.0330 - accuracy: 0.9898
1130/1688 [===================>..........] - ETA: 0s - loss: 0.0330 - accuracy: 0.9897
1159/1688 [===================>..........] - ETA: 0s - loss: 0.0333 - accuracy: 0.9896
1187/1688 [====================>.........] - ETA: 0s - loss: 0.0334 - accuracy: 0.9896
1216/1688 [====================>.........] - ETA: 0s - loss: 0.0333 - accuracy: 0.9895
1245/1688 [=====================>........] - ETA: 0s - loss: 0.0341 - accuracy: 0.9893
1273/1688 [=====================>........] - ETA: 0s - loss: 0.0343 - accuracy: 0.9892
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0343 - accuracy: 0.9892
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0344 - accuracy: 0.9892
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0344 - accuracy: 0.9892
1389/1688 [=======================>......] - ETA: 0s - loss: 0.0344 - accuracy: 0.9892
1418/1688 [========================>.....] - ETA: 0s - loss: 0.0345 - accuracy: 0.9891
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0346 - accuracy: 0.9891
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0346 - accuracy: 0.9891
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0345 - accuracy: 0.9891
1532/1688 [==========================>...] - ETA: 0s - loss: 0.0343 - accuracy: 0.9891
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0343 - accuracy: 0.9891
1588/1688 [===========================>..] - ETA: 0s - loss: 0.0344 - accuracy: 0.9891
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0344 - accuracy: 0.9891
1646/1688 [============================>.] - ETA: 0s - loss: 0.0345 - accuracy: 0.9891
1675/1688 [============================>.] - ETA: 0s - loss: 0.0345 - accuracy: 0.9891
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0345 - accuracy: 0.9891 - val_loss: 0.0748 - val_accuracy: 0.9778
Epoch 6/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0548 - accuracy: 0.9688
  32/1688 [..............................] - ETA: 2s - loss: 0.0521 - accuracy: 0.9854
  61/1688 [>.............................] - ETA: 2s - loss: 0.0321 - accuracy: 0.9903
  90/1688 [>.............................] - ETA: 2s - loss: 0.0296 - accuracy: 0.9906
 119/1688 [=>............................] - ETA: 2s - loss: 0.0303 - accuracy: 0.9908
 148/1688 [=>............................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9916
 177/1688 [==>...........................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9919
 206/1688 [==>...........................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9921
 234/1688 [===>..........................] - ETA: 2s - loss: 0.0262 - accuracy: 0.9919
 262/1688 [===>..........................] - ETA: 2s - loss: 0.0259 - accuracy: 0.9919
 291/1688 [====>.........................] - ETA: 2s - loss: 0.0250 - accuracy: 0.9918
 320/1688 [====>.........................] - ETA: 2s - loss: 0.0253 - accuracy: 0.9916
 349/1688 [=====>........................] - ETA: 2s - loss: 0.0255 - accuracy: 0.9916
 378/1688 [=====>........................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9912
 408/1688 [======>.......................] - ETA: 2s - loss: 0.0259 - accuracy: 0.9912
 436/1688 [======>.......................] - ETA: 2s - loss: 0.0248 - accuracy: 0.9917
 465/1688 [=======>......................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9917
 494/1688 [=======>......................] - ETA: 2s - loss: 0.0251 - accuracy: 0.9918
 522/1688 [========>.....................] - ETA: 2s - loss: 0.0253 - accuracy: 0.9916
 551/1688 [========>.....................] - ETA: 2s - loss: 0.0253 - accuracy: 0.9915
 580/1688 [=========>....................] - ETA: 1s - loss: 0.0255 - accuracy: 0.9916
 608/1688 [=========>....................] - ETA: 1s - loss: 0.0261 - accuracy: 0.9914
 637/1688 [==========>...................] - ETA: 1s - loss: 0.0258 - accuracy: 0.9915
 666/1688 [==========>...................] - ETA: 1s - loss: 0.0259 - accuracy: 0.9915
 695/1688 [===========>..................] - ETA: 1s - loss: 0.0264 - accuracy: 0.9913
 723/1688 [===========>..................] - ETA: 1s - loss: 0.0264 - accuracy: 0.9912
 752/1688 [============>.................] - ETA: 1s - loss: 0.0264 - accuracy: 0.9912
 781/1688 [============>.................] - ETA: 1s - loss: 0.0263 - accuracy: 0.9912
 809/1688 [=============>................] - ETA: 1s - loss: 0.0270 - accuracy: 0.9909
 837/1688 [=============>................] - ETA: 1s - loss: 0.0270 - accuracy: 0.9909
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0267 - accuracy: 0.9911
 895/1688 [==============>...............] - ETA: 1s - loss: 0.0270 - accuracy: 0.9910
 924/1688 [===============>..............] - ETA: 1s - loss: 0.0270 - accuracy: 0.9910
 953/1688 [===============>..............] - ETA: 1s - loss: 0.0268 - accuracy: 0.9911
 982/1688 [================>.............] - ETA: 1s - loss: 0.0272 - accuracy: 0.9910
1011/1688 [================>.............] - ETA: 1s - loss: 0.0278 - accuracy: 0.9908
1040/1688 [=================>............] - ETA: 1s - loss: 0.0279 - accuracy: 0.9909
1069/1688 [=================>............] - ETA: 1s - loss: 0.0277 - accuracy: 0.9909
1098/1688 [==================>...........] - ETA: 1s - loss: 0.0280 - accuracy: 0.9907
1127/1688 [===================>..........] - ETA: 0s - loss: 0.0282 - accuracy: 0.9906
1156/1688 [===================>..........] - ETA: 0s - loss: 0.0281 - accuracy: 0.9906
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0280 - accuracy: 0.9906
1213/1688 [====================>.........] - ETA: 0s - loss: 0.0281 - accuracy: 0.9906
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0281 - accuracy: 0.9906
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0284 - accuracy: 0.9906
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0284 - accuracy: 0.9906
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0285 - accuracy: 0.9907
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0283 - accuracy: 0.9908
1387/1688 [=======================>......] - ETA: 0s - loss: 0.0284 - accuracy: 0.9908
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0280 - accuracy: 0.9909
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0281 - accuracy: 0.9908
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0282 - accuracy: 0.9907
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9907
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0288 - accuracy: 0.9906
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0290 - accuracy: 0.9905
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0293 - accuracy: 0.9904
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0293 - accuracy: 0.9904
1647/1688 [============================>.] - ETA: 0s - loss: 0.0292 - accuracy: 0.9904
1676/1688 [============================>.] - ETA: 0s - loss: 0.0293 - accuracy: 0.9903
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0293 - accuracy: 0.9903 - val_loss: 0.0559 - val_accuracy: 0.9863
Epoch 7/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0219 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0229 - accuracy: 0.9919
  59/1688 [>.............................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9926
  88/1688 [>.............................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9933
 116/1688 [=>............................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9925
 145/1688 [=>............................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9922
 173/1688 [==>...........................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9928
 202/1688 [==>...........................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9932
 231/1688 [===>..........................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9934
 260/1688 [===>..........................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9936
 288/1688 [====>.........................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9936
 316/1688 [====>.........................] - ETA: 2s - loss: 0.0177 - accuracy: 0.9934
 345/1688 [=====>........................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9931
 374/1688 [=====>........................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9932
 402/1688 [======>.......................] - ETA: 2s - loss: 0.0185 - accuracy: 0.9933
 431/1688 [======>.......................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9930
 460/1688 [=======>......................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9927
 488/1688 [=======>......................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9930
 517/1688 [========>.....................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9931
 547/1688 [========>.....................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9930
 576/1688 [=========>....................] - ETA: 1s - loss: 0.0200 - accuracy: 0.9929
 605/1688 [=========>....................] - ETA: 1s - loss: 0.0199 - accuracy: 0.9930
 633/1688 [==========>...................] - ETA: 1s - loss: 0.0200 - accuracy: 0.9930
 662/1688 [==========>...................] - ETA: 1s - loss: 0.0203 - accuracy: 0.9929
 690/1688 [===========>..................] - ETA: 1s - loss: 0.0204 - accuracy: 0.9928
 719/1688 [===========>..................] - ETA: 1s - loss: 0.0203 - accuracy: 0.9929
 748/1688 [============>.................] - ETA: 1s - loss: 0.0205 - accuracy: 0.9926
 777/1688 [============>.................] - ETA: 1s - loss: 0.0209 - accuracy: 0.9923
 805/1688 [=============>................] - ETA: 1s - loss: 0.0211 - accuracy: 0.9923
 834/1688 [=============>................] - ETA: 1s - loss: 0.0217 - accuracy: 0.9921
 863/1688 [==============>...............] - ETA: 1s - loss: 0.0219 - accuracy: 0.9921
 892/1688 [==============>...............] - ETA: 1s - loss: 0.0223 - accuracy: 0.9921
 921/1688 [===============>..............] - ETA: 1s - loss: 0.0227 - accuracy: 0.9921
 951/1688 [===============>..............] - ETA: 1s - loss: 0.0235 - accuracy: 0.9919
 980/1688 [================>.............] - ETA: 1s - loss: 0.0237 - accuracy: 0.9918
1009/1688 [================>.............] - ETA: 1s - loss: 0.0240 - accuracy: 0.9917
1038/1688 [=================>............] - ETA: 1s - loss: 0.0241 - accuracy: 0.9918
1066/1688 [=================>............] - ETA: 1s - loss: 0.0237 - accuracy: 0.9919
1094/1688 [==================>...........] - ETA: 1s - loss: 0.0238 - accuracy: 0.9918
1122/1688 [==================>...........] - ETA: 1s - loss: 0.0236 - accuracy: 0.9919
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0236 - accuracy: 0.9919
1179/1688 [===================>..........] - ETA: 0s - loss: 0.0237 - accuracy: 0.9919
1207/1688 [====================>.........] - ETA: 0s - loss: 0.0237 - accuracy: 0.9919
1236/1688 [====================>.........] - ETA: 0s - loss: 0.0237 - accuracy: 0.9919
1265/1688 [=====================>........] - ETA: 0s - loss: 0.0240 - accuracy: 0.9919
1294/1688 [=====================>........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9919
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9919
1352/1688 [=======================>......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9919
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9919
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0235 - accuracy: 0.9921
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9919
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0239 - accuracy: 0.9919
1496/1688 [=========================>....] - ETA: 0s - loss: 0.0240 - accuracy: 0.9918
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0242 - accuracy: 0.9918
1554/1688 [==========================>...] - ETA: 0s - loss: 0.0243 - accuracy: 0.9918
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0242 - accuracy: 0.9919
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0243 - accuracy: 0.9919
1639/1688 [============================>.] - ETA: 0s - loss: 0.0244 - accuracy: 0.9919
1668/1688 [============================>.] - ETA: 0s - loss: 0.0244 - accuracy: 0.9919
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0242 - accuracy: 0.9919 - val_loss: 0.0569 - val_accuracy: 0.9863
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0072 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0091 - accuracy: 0.9970
  60/1688 [>.............................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9958
  89/1688 [>.............................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9958
 117/1688 [=>............................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9960
 146/1688 [=>............................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9961
 175/1688 [==>...........................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9946
 203/1688 [==>...........................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9942
 232/1688 [===>..........................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9946
 261/1688 [===>..........................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9945
 289/1688 [====>.........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9950
 317/1688 [====>.........................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9951
 346/1688 [=====>........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9948
 374/1688 [=====>........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9949
 402/1688 [======>.......................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9949
 430/1688 [======>.......................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9948
 458/1688 [=======>......................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9948
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9948
 517/1688 [========>.....................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9948
 545/1688 [========>.....................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9947
 574/1688 [=========>....................] - ETA: 1s - loss: 0.0156 - accuracy: 0.9948
 602/1688 [=========>....................] - ETA: 1s - loss: 0.0157 - accuracy: 0.9948
 631/1688 [==========>...................] - ETA: 1s - loss: 0.0158 - accuracy: 0.9948
 660/1688 [==========>...................] - ETA: 1s - loss: 0.0164 - accuracy: 0.9946
 689/1688 [===========>..................] - ETA: 1s - loss: 0.0167 - accuracy: 0.9943
 718/1688 [===========>..................] - ETA: 1s - loss: 0.0169 - accuracy: 0.9943
 747/1688 [============>.................] - ETA: 1s - loss: 0.0171 - accuracy: 0.9942
 776/1688 [============>.................] - ETA: 1s - loss: 0.0176 - accuracy: 0.9941
 805/1688 [=============>................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9941
 834/1688 [=============>................] - ETA: 1s - loss: 0.0187 - accuracy: 0.9938
 862/1688 [==============>...............] - ETA: 1s - loss: 0.0192 - accuracy: 0.9936
 891/1688 [==============>...............] - ETA: 1s - loss: 0.0192 - accuracy: 0.9936
 920/1688 [===============>..............] - ETA: 1s - loss: 0.0193 - accuracy: 0.9935
 949/1688 [===============>..............] - ETA: 1s - loss: 0.0191 - accuracy: 0.9936
 978/1688 [================>.............] - ETA: 1s - loss: 0.0194 - accuracy: 0.9935
1007/1688 [================>.............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9934
1036/1688 [=================>............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9935
1065/1688 [=================>............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9935
1094/1688 [==================>...........] - ETA: 1s - loss: 0.0195 - accuracy: 0.9935
1122/1688 [==================>...........] - ETA: 1s - loss: 0.0194 - accuracy: 0.9935
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0193 - accuracy: 0.9935
1179/1688 [===================>..........] - ETA: 0s - loss: 0.0190 - accuracy: 0.9936
1208/1688 [====================>.........] - ETA: 0s - loss: 0.0191 - accuracy: 0.9937
1237/1688 [====================>.........] - ETA: 0s - loss: 0.0190 - accuracy: 0.9937
1266/1688 [=====================>........] - ETA: 0s - loss: 0.0193 - accuracy: 0.9937
1295/1688 [======================>.......] - ETA: 0s - loss: 0.0192 - accuracy: 0.9936
1324/1688 [======================>.......] - ETA: 0s - loss: 0.0193 - accuracy: 0.9935
1352/1688 [=======================>......] - ETA: 0s - loss: 0.0194 - accuracy: 0.9935
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0194 - accuracy: 0.9935
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0194 - accuracy: 0.9935
1439/1688 [========================>.....] - ETA: 0s - loss: 0.0196 - accuracy: 0.9935
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0197 - accuracy: 0.9934
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0197 - accuracy: 0.9934
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0200 - accuracy: 0.9933
1554/1688 [==========================>...] - ETA: 0s - loss: 0.0201 - accuracy: 0.9932
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0202 - accuracy: 0.9932
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0201 - accuracy: 0.9932
1641/1688 [============================>.] - ETA: 0s - loss: 0.0204 - accuracy: 0.9931
1670/1688 [============================>.] - ETA: 0s - loss: 0.0205 - accuracy: 0.9930
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0204 - accuracy: 0.9931 - val_loss: 0.0718 - val_accuracy: 0.9823
Epoch 9/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0023 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9960
  61/1688 [>.............................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9964
  90/1688 [>.............................] - ETA: 2s - loss: 0.0102 - accuracy: 0.9969
 119/1688 [=>............................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9958
 148/1688 [=>............................] - ETA: 2s - loss: 0.0197 - accuracy: 0.9943
 177/1688 [==>...........................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9944
 206/1688 [==>...........................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9948
 235/1688 [===>..........................] - ETA: 2s - loss: 0.0177 - accuracy: 0.9947
 263/1688 [===>..........................] - ETA: 2s - loss: 0.0169 - accuracy: 0.9949
 292/1688 [====>.........................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9949
 321/1688 [====>.........................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9948
 350/1688 [=====>........................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9949
 379/1688 [=====>........................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9950
 407/1688 [======>.......................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9951
 437/1688 [======>.......................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9954
 465/1688 [=======>......................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9952
 494/1688 [=======>......................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9955
 524/1688 [========>.....................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9955
 552/1688 [========>.....................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9956
 582/1688 [=========>....................] - ETA: 1s - loss: 0.0149 - accuracy: 0.9954
 611/1688 [=========>....................] - ETA: 1s - loss: 0.0151 - accuracy: 0.9953
 640/1688 [==========>...................] - ETA: 1s - loss: 0.0154 - accuracy: 0.9953
 669/1688 [==========>...................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9953
 698/1688 [===========>..................] - ETA: 1s - loss: 0.0154 - accuracy: 0.9952
 726/1688 [===========>..................] - ETA: 1s - loss: 0.0154 - accuracy: 0.9952
 755/1688 [============>.................] - ETA: 1s - loss: 0.0155 - accuracy: 0.9952
 784/1688 [============>.................] - ETA: 1s - loss: 0.0156 - accuracy: 0.9951
 813/1688 [=============>................] - ETA: 1s - loss: 0.0158 - accuracy: 0.9950
 841/1688 [=============>................] - ETA: 1s - loss: 0.0155 - accuracy: 0.9951
 869/1688 [==============>...............] - ETA: 1s - loss: 0.0155 - accuracy: 0.9950
 898/1688 [==============>...............] - ETA: 1s - loss: 0.0160 - accuracy: 0.9948
 927/1688 [===============>..............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9949
 956/1688 [===============>..............] - ETA: 1s - loss: 0.0160 - accuracy: 0.9948
 985/1688 [================>.............] - ETA: 1s - loss: 0.0163 - accuracy: 0.9947
1014/1688 [=================>............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9946
1043/1688 [=================>............] - ETA: 1s - loss: 0.0168 - accuracy: 0.9945
1073/1688 [==================>...........] - ETA: 1s - loss: 0.0170 - accuracy: 0.9944
1102/1688 [==================>...........] - ETA: 1s - loss: 0.0170 - accuracy: 0.9944
1132/1688 [===================>..........] - ETA: 0s - loss: 0.0171 - accuracy: 0.9943
1161/1688 [===================>..........] - ETA: 0s - loss: 0.0170 - accuracy: 0.9943
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0171 - accuracy: 0.9943
1218/1688 [====================>.........] - ETA: 0s - loss: 0.0170 - accuracy: 0.9943
1248/1688 [=====================>........] - ETA: 0s - loss: 0.0174 - accuracy: 0.9942
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0173 - accuracy: 0.9942
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0175 - accuracy: 0.9942
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0175 - accuracy: 0.9941
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0176 - accuracy: 0.9941
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0177 - accuracy: 0.9940
1419/1688 [========================>.....] - ETA: 0s - loss: 0.0179 - accuracy: 0.9941
1448/1688 [========================>.....] - ETA: 0s - loss: 0.0179 - accuracy: 0.9940
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0177 - accuracy: 0.9941
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0177 - accuracy: 0.9941
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0178 - accuracy: 0.9941
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0181 - accuracy: 0.9940
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0181 - accuracy: 0.9940
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0181 - accuracy: 0.9940
1648/1688 [============================>.] - ETA: 0s - loss: 0.0183 - accuracy: 0.9939
1677/1688 [============================>.] - ETA: 0s - loss: 0.0183 - accuracy: 0.9939
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0182 - accuracy: 0.9940 - val_loss: 0.0579 - val_accuracy: 0.9847
Epoch 10/10

   1/1688 [..............................] - ETA: 3s - loss: 5.7492e-04 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0086 - accuracy: 0.9990    
  60/1688 [>.............................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9969
  89/1688 [>.............................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9965
 117/1688 [=>............................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9963
 146/1688 [=>............................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9966
 175/1688 [==>...........................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9970
 204/1688 [==>...........................] - ETA: 2s - loss: 0.0102 - accuracy: 0.9969
 233/1688 [===>..........................] - ETA: 2s - loss: 0.0102 - accuracy: 0.9972
 261/1688 [===>..........................] - ETA: 2s - loss: 0.0104 - accuracy: 0.9969
 290/1688 [====>.........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9969
 319/1688 [====>.........................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9968
 348/1688 [=====>........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9968
 377/1688 [=====>........................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9969
 406/1688 [======>.......................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9968
 434/1688 [======>.......................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9968
 463/1688 [=======>......................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9967
 491/1688 [=======>......................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9965
 520/1688 [========>.....................] - ETA: 2s - loss: 0.0103 - accuracy: 0.9963
 549/1688 [========>.....................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9958
 579/1688 [=========>....................] - ETA: 1s - loss: 0.0120 - accuracy: 0.9957
 607/1688 [=========>....................] - ETA: 1s - loss: 0.0120 - accuracy: 0.9957
 637/1688 [==========>...................] - ETA: 1s - loss: 0.0120 - accuracy: 0.9957
 666/1688 [==========>...................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9955
 695/1688 [===========>..................] - ETA: 1s - loss: 0.0129 - accuracy: 0.9953
 724/1688 [===========>..................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9953
 752/1688 [============>.................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9952
 781/1688 [============>.................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9951
 810/1688 [=============>................] - ETA: 1s - loss: 0.0138 - accuracy: 0.9951
 839/1688 [=============>................] - ETA: 1s - loss: 0.0139 - accuracy: 0.9951
 868/1688 [==============>...............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9950
 896/1688 [==============>...............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9950
 925/1688 [===============>..............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9950
 954/1688 [===============>..............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9950
 982/1688 [================>.............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9950
1010/1688 [================>.............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9949
1039/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9949
1068/1688 [=================>............] - ETA: 1s - loss: 0.0154 - accuracy: 0.9948
1097/1688 [==================>...........] - ETA: 1s - loss: 0.0157 - accuracy: 0.9947
1126/1688 [===================>..........] - ETA: 0s - loss: 0.0158 - accuracy: 0.9947
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0158 - accuracy: 0.9948
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0158 - accuracy: 0.9948
1213/1688 [====================>.........] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1241/1688 [=====================>........] - ETA: 0s - loss: 0.0161 - accuracy: 0.9947
1269/1688 [=====================>........] - ETA: 0s - loss: 0.0164 - accuracy: 0.9946
1298/1688 [======================>.......] - ETA: 0s - loss: 0.0164 - accuracy: 0.9945
1327/1688 [======================>.......] - ETA: 0s - loss: 0.0163 - accuracy: 0.9946
1356/1688 [=======================>......] - ETA: 0s - loss: 0.0162 - accuracy: 0.9946
1385/1688 [=======================>......] - ETA: 0s - loss: 0.0162 - accuracy: 0.9946
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0161 - accuracy: 0.9946
1442/1688 [========================>.....] - ETA: 0s - loss: 0.0162 - accuracy: 0.9946
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0161 - accuracy: 0.9946
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9945
1529/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9945
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0162 - accuracy: 0.9945
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0161 - accuracy: 0.9946
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0160 - accuracy: 0.9946
1644/1688 [============================>.] - ETA: 0s - loss: 0.0160 - accuracy: 0.9946
1672/1688 [============================>.] - ETA: 0s - loss: 0.0160 - accuracy: 0.9946
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0160 - accuracy: 0.9946 - val_loss: 0.0543 - val_accuracy: 0.9868
Test accuracy: 0.9866999983787537

4. Model quantization

We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.

For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).

The quantized model is a Keras model where the neural layers (Conv2D, Dense) and the ReLU layers are replaced with custom CNN2SNN quantized layers (QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function folds the batch normalization layers into the corresponding neural layer. The new weights are computed according to this folding operation.

from cnn2snn import quantize

model_quantized = quantize(model_keras,
                           input_weight_quantization=8,
                           weight_quantization=4,
                           activ_quantization=4)
model_quantized.summary()

Out:

Model: "sequential_48"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 conv2d (QuantizedConv2D)    (None, 13, 13, 32)        320

 re_lu (QuantizedReLU)       (None, 13, 13, 32)        0

 separable_conv2d (Quantized  (None, 7, 7, 64)         2400
 SeparableConv2D)

 re_lu_1 (QuantizedReLU)     (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (QuantizedDense)      (None, 10)                31370

=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________

Check the quantized model accuracy.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])

Out:

Test accuracy after 8-4-4 quantization: 0.8393999934196472

Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.

Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.

Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.

from cnn2snn import quantize_layer

model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])

# To recover the original model accuracy, a quantization-aware training phase
# is required.

Out:

Test accuracy after low bitwidth quantization: 0.11349999904632568

5. Model fine tuning (quantization-aware training)

This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.

Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])

Out:

Epoch 1/5

   1/1688 [..............................] - ETA: 17:09 - loss: 2.3055 - accuracy: 0.0312
  21/1688 [..............................] - ETA: 4s - loss: 2.2993 - accuracy: 0.1250   
  41/1688 [..............................] - ETA: 4s - loss: 2.2990 - accuracy: 0.1197
  61/1688 [>.............................] - ETA: 4s - loss: 2.2981 - accuracy: 0.1142
  82/1688 [>.............................] - ETA: 4s - loss: 2.3140 - accuracy: 0.1204
 102/1688 [>.............................] - ETA: 4s - loss: 2.3273 - accuracy: 0.1275
 122/1688 [=>............................] - ETA: 3s - loss: 2.3340 - accuracy: 0.1465
 142/1688 [=>............................] - ETA: 3s - loss: 2.2837 - accuracy: 0.1875
 162/1688 [=>............................] - ETA: 3s - loss: 2.2281 - accuracy: 0.2388
 183/1688 [==>...........................] - ETA: 3s - loss: 2.1510 - accuracy: 0.2968
 203/1688 [==>...........................] - ETA: 3s - loss: 2.0784 - accuracy: 0.3478
 224/1688 [==>...........................] - ETA: 3s - loss: 1.9933 - accuracy: 0.3927
 244/1688 [===>..........................] - ETA: 3s - loss: 1.9069 - accuracy: 0.4314
 264/1688 [===>..........................] - ETA: 3s - loss: 1.8362 - accuracy: 0.4657
 284/1688 [====>.........................] - ETA: 3s - loss: 1.7809 - accuracy: 0.4946
 305/1688 [====>.........................] - ETA: 3s - loss: 1.7250 - accuracy: 0.5214
 325/1688 [====>.........................] - ETA: 3s - loss: 1.6896 - accuracy: 0.5440
 345/1688 [=====>........................] - ETA: 3s - loss: 1.6553 - accuracy: 0.5639
 365/1688 [=====>........................] - ETA: 3s - loss: 1.6073 - accuracy: 0.5830
 385/1688 [=====>........................] - ETA: 3s - loss: 1.5767 - accuracy: 0.5989
 405/1688 [======>.......................] - ETA: 3s - loss: 1.5383 - accuracy: 0.6140
 425/1688 [======>.......................] - ETA: 3s - loss: 1.5078 - accuracy: 0.6277
 445/1688 [======>.......................] - ETA: 3s - loss: 1.4978 - accuracy: 0.6403
 465/1688 [=======>......................] - ETA: 3s - loss: 1.4841 - accuracy: 0.6524
 485/1688 [=======>......................] - ETA: 3s - loss: 1.4628 - accuracy: 0.6633
 505/1688 [=======>......................] - ETA: 2s - loss: 1.4499 - accuracy: 0.6733
 526/1688 [========>.....................] - ETA: 2s - loss: 1.4384 - accuracy: 0.6834
 546/1688 [========>.....................] - ETA: 2s - loss: 1.4270 - accuracy: 0.6924
 566/1688 [=========>....................] - ETA: 2s - loss: 1.4297 - accuracy: 0.7001
 586/1688 [=========>....................] - ETA: 2s - loss: 1.4138 - accuracy: 0.7078
 606/1688 [=========>....................] - ETA: 2s - loss: 1.4042 - accuracy: 0.7150
 626/1688 [==========>...................] - ETA: 2s - loss: 1.4091 - accuracy: 0.7216
 646/1688 [==========>...................] - ETA: 2s - loss: 1.3991 - accuracy: 0.7282
 666/1688 [==========>...................] - ETA: 2s - loss: 1.3914 - accuracy: 0.7343
 686/1688 [===========>..................] - ETA: 2s - loss: 1.3760 - accuracy: 0.7403
 706/1688 [===========>..................] - ETA: 2s - loss: 1.3560 - accuracy: 0.7462
 726/1688 [===========>..................] - ETA: 2s - loss: 1.3341 - accuracy: 0.7520
 746/1688 [============>.................] - ETA: 2s - loss: 1.3249 - accuracy: 0.7570
 766/1688 [============>.................] - ETA: 2s - loss: 1.3172 - accuracy: 0.7617
 786/1688 [============>.................] - ETA: 2s - loss: 1.3035 - accuracy: 0.7664
 806/1688 [=============>................] - ETA: 2s - loss: 1.2964 - accuracy: 0.7705
 826/1688 [=============>................] - ETA: 2s - loss: 1.2858 - accuracy: 0.7749
 847/1688 [==============>...............] - ETA: 2s - loss: 1.2814 - accuracy: 0.7786
 867/1688 [==============>...............] - ETA: 2s - loss: 1.2792 - accuracy: 0.7822
 887/1688 [==============>...............] - ETA: 2s - loss: 1.2739 - accuracy: 0.7857
 907/1688 [===============>..............] - ETA: 1s - loss: 1.2691 - accuracy: 0.7892
 927/1688 [===============>..............] - ETA: 1s - loss: 1.2589 - accuracy: 0.7925
 947/1688 [===============>..............] - ETA: 1s - loss: 1.2520 - accuracy: 0.7955
 967/1688 [================>.............] - ETA: 1s - loss: 1.2428 - accuracy: 0.7987
 987/1688 [================>.............] - ETA: 1s - loss: 1.2376 - accuracy: 0.8015
1008/1688 [================>.............] - ETA: 1s - loss: 1.2297 - accuracy: 0.8046
1029/1688 [=================>............] - ETA: 1s - loss: 1.2234 - accuracy: 0.8075
1049/1688 [=================>............] - ETA: 1s - loss: 1.2149 - accuracy: 0.8102
1069/1688 [=================>............] - ETA: 1s - loss: 1.2079 - accuracy: 0.8129
1089/1688 [==================>...........] - ETA: 1s - loss: 1.1992 - accuracy: 0.8156
1109/1688 [==================>...........] - ETA: 1s - loss: 1.1936 - accuracy: 0.8177
1129/1688 [===================>..........] - ETA: 1s - loss: 1.1941 - accuracy: 0.8199
1149/1688 [===================>..........] - ETA: 1s - loss: 1.1914 - accuracy: 0.8221
1169/1688 [===================>..........] - ETA: 1s - loss: 1.1926 - accuracy: 0.8241
1189/1688 [====================>.........] - ETA: 1s - loss: 1.1822 - accuracy: 0.8265
1210/1688 [====================>.........] - ETA: 1s - loss: 1.1720 - accuracy: 0.8288
1230/1688 [====================>.........] - ETA: 1s - loss: 1.1691 - accuracy: 0.8306
1250/1688 [=====================>........] - ETA: 1s - loss: 1.1585 - accuracy: 0.8327
1270/1688 [=====================>........] - ETA: 1s - loss: 1.1506 - accuracy: 0.8347
1289/1688 [=====================>........] - ETA: 1s - loss: 1.1440 - accuracy: 0.8365
1309/1688 [======================>.......] - ETA: 0s - loss: 1.1395 - accuracy: 0.8382
1330/1688 [======================>.......] - ETA: 0s - loss: 1.1339 - accuracy: 0.8399
1350/1688 [======================>.......] - ETA: 0s - loss: 1.1348 - accuracy: 0.8413
1370/1688 [=======================>......] - ETA: 0s - loss: 1.1296 - accuracy: 0.8428
1391/1688 [=======================>......] - ETA: 0s - loss: 1.1279 - accuracy: 0.8444
1411/1688 [========================>.....] - ETA: 0s - loss: 1.1273 - accuracy: 0.8458
1432/1688 [========================>.....] - ETA: 0s - loss: 1.1176 - accuracy: 0.8475
1452/1688 [========================>.....] - ETA: 0s - loss: 1.1175 - accuracy: 0.8488
1473/1688 [=========================>....] - ETA: 0s - loss: 1.1137 - accuracy: 0.8503
1493/1688 [=========================>....] - ETA: 0s - loss: 1.1087 - accuracy: 0.8517
1513/1688 [=========================>....] - ETA: 0s - loss: 1.1015 - accuracy: 0.8531
1534/1688 [==========================>...] - ETA: 0s - loss: 1.0938 - accuracy: 0.8546
1555/1688 [==========================>...] - ETA: 0s - loss: 1.0868 - accuracy: 0.8561
1575/1688 [==========================>...] - ETA: 0s - loss: 1.0820 - accuracy: 0.8575
1595/1688 [===========================>..] - ETA: 0s - loss: 1.0772 - accuracy: 0.8588
1616/1688 [===========================>..] - ETA: 0s - loss: 1.0734 - accuracy: 0.8600
1636/1688 [============================>.] - ETA: 0s - loss: 1.0704 - accuracy: 0.8610
1656/1688 [============================>.] - ETA: 0s - loss: 1.0642 - accuracy: 0.8622
1676/1688 [============================>.] - ETA: 0s - loss: 1.0629 - accuracy: 0.8633
1688/1688 [==============================] - 5s 3ms/step - loss: 1.0626 - accuracy: 0.8639 - val_loss: 0.7314 - val_accuracy: 0.9647
Epoch 2/5

   1/1688 [..............................] - ETA: 4s - loss: 0.2309 - accuracy: 0.9688
  22/1688 [..............................] - ETA: 4s - loss: 0.5171 - accuracy: 0.9616
  43/1688 [..............................] - ETA: 4s - loss: 0.4916 - accuracy: 0.9637
  64/1688 [>.............................] - ETA: 4s - loss: 0.5410 - accuracy: 0.9624
  84/1688 [>.............................] - ETA: 3s - loss: 0.5352 - accuracy: 0.9609
 104/1688 [>.............................] - ETA: 3s - loss: 0.6131 - accuracy: 0.9594
 124/1688 [=>............................] - ETA: 3s - loss: 0.6482 - accuracy: 0.9594
 144/1688 [=>............................] - ETA: 3s - loss: 0.6494 - accuracy: 0.9592
 164/1688 [=>............................] - ETA: 3s - loss: 0.7114 - accuracy: 0.9558
 184/1688 [==>...........................] - ETA: 3s - loss: 0.7062 - accuracy: 0.9560
 204/1688 [==>...........................] - ETA: 3s - loss: 0.6935 - accuracy: 0.9563
 224/1688 [==>...........................] - ETA: 3s - loss: 0.6985 - accuracy: 0.9562
 244/1688 [===>..........................] - ETA: 3s - loss: 0.7312 - accuracy: 0.9545
 264/1688 [===>..........................] - ETA: 3s - loss: 0.7030 - accuracy: 0.9557
 284/1688 [====>.........................] - ETA: 3s - loss: 0.7097 - accuracy: 0.9559
 304/1688 [====>.........................] - ETA: 3s - loss: 0.7030 - accuracy: 0.9561
 324/1688 [====>.........................] - ETA: 3s - loss: 0.6777 - accuracy: 0.9572
 344/1688 [=====>........................] - ETA: 3s - loss: 0.6724 - accuracy: 0.9579
 364/1688 [=====>........................] - ETA: 3s - loss: 0.6692 - accuracy: 0.9583
 384/1688 [=====>........................] - ETA: 3s - loss: 0.6946 - accuracy: 0.9578
 405/1688 [======>.......................] - ETA: 3s - loss: 0.6794 - accuracy: 0.9583
 425/1688 [======>.......................] - ETA: 3s - loss: 0.6810 - accuracy: 0.9584
 446/1688 [======>.......................] - ETA: 3s - loss: 0.7071 - accuracy: 0.9577
 466/1688 [=======>......................] - ETA: 3s - loss: 0.7179 - accuracy: 0.9579
 486/1688 [=======>......................] - ETA: 3s - loss: 0.7140 - accuracy: 0.9583
 506/1688 [=======>......................] - ETA: 2s - loss: 0.7066 - accuracy: 0.9581
 527/1688 [========>.....................] - ETA: 2s - loss: 0.7042 - accuracy: 0.9584
 547/1688 [========>.....................] - ETA: 2s - loss: 0.7179 - accuracy: 0.9584
 567/1688 [=========>....................] - ETA: 2s - loss: 0.6997 - accuracy: 0.9590
 587/1688 [=========>....................] - ETA: 2s - loss: 0.7051 - accuracy: 0.9590
 607/1688 [=========>....................] - ETA: 2s - loss: 0.7032 - accuracy: 0.9592
 627/1688 [==========>...................] - ETA: 2s - loss: 0.7000 - accuracy: 0.9596
 647/1688 [==========>...................] - ETA: 2s - loss: 0.7056 - accuracy: 0.9593
 667/1688 [==========>...................] - ETA: 2s - loss: 0.6983 - accuracy: 0.9597
 687/1688 [===========>..................] - ETA: 2s - loss: 0.7035 - accuracy: 0.9598
 707/1688 [===========>..................] - ETA: 2s - loss: 0.7089 - accuracy: 0.9596
 727/1688 [===========>..................] - ETA: 2s - loss: 0.7173 - accuracy: 0.9596
 747/1688 [============>.................] - ETA: 2s - loss: 0.7265 - accuracy: 0.9595
 767/1688 [============>.................] - ETA: 2s - loss: 0.7279 - accuracy: 0.9596
 787/1688 [============>.................] - ETA: 2s - loss: 0.7223 - accuracy: 0.9599
 807/1688 [=============>................] - ETA: 2s - loss: 0.7215 - accuracy: 0.9600
 827/1688 [=============>................] - ETA: 2s - loss: 0.7202 - accuracy: 0.9601
 847/1688 [==============>...............] - ETA: 2s - loss: 0.7192 - accuracy: 0.9601
 867/1688 [==============>...............] - ETA: 2s - loss: 0.7136 - accuracy: 0.9603
 887/1688 [==============>...............] - ETA: 2s - loss: 0.7145 - accuracy: 0.9604
 908/1688 [===============>..............] - ETA: 1s - loss: 0.7170 - accuracy: 0.9602
 928/1688 [===============>..............] - ETA: 1s - loss: 0.7131 - accuracy: 0.9603
 947/1688 [===============>..............] - ETA: 1s - loss: 0.7244 - accuracy: 0.9603
 967/1688 [================>.............] - ETA: 1s - loss: 0.7169 - accuracy: 0.9606
 987/1688 [================>.............] - ETA: 1s - loss: 0.7157 - accuracy: 0.9607
1007/1688 [================>.............] - ETA: 1s - loss: 0.7077 - accuracy: 0.9609
1027/1688 [=================>............] - ETA: 1s - loss: 0.7039 - accuracy: 0.9610
1048/1688 [=================>............] - ETA: 1s - loss: 0.7054 - accuracy: 0.9609
1068/1688 [=================>............] - ETA: 1s - loss: 0.7057 - accuracy: 0.9611
1088/1688 [==================>...........] - ETA: 1s - loss: 0.7080 - accuracy: 0.9611
1108/1688 [==================>...........] - ETA: 1s - loss: 0.7013 - accuracy: 0.9615
1128/1688 [===================>..........] - ETA: 1s - loss: 0.7018 - accuracy: 0.9616
1149/1688 [===================>..........] - ETA: 1s - loss: 0.7036 - accuracy: 0.9617
1169/1688 [===================>..........] - ETA: 1s - loss: 0.7110 - accuracy: 0.9615
1189/1688 [====================>.........] - ETA: 1s - loss: 0.7187 - accuracy: 0.9614
1209/1688 [====================>.........] - ETA: 1s - loss: 0.7175 - accuracy: 0.9615
1230/1688 [====================>.........] - ETA: 1s - loss: 0.7189 - accuracy: 0.9616
1250/1688 [=====================>........] - ETA: 1s - loss: 0.7182 - accuracy: 0.9615
1270/1688 [=====================>........] - ETA: 1s - loss: 0.7217 - accuracy: 0.9615
1290/1688 [=====================>........] - ETA: 1s - loss: 0.7227 - accuracy: 0.9616
1310/1688 [======================>.......] - ETA: 0s - loss: 0.7272 - accuracy: 0.9615
1330/1688 [======================>.......] - ETA: 0s - loss: 0.7261 - accuracy: 0.9615
1350/1688 [======================>.......] - ETA: 0s - loss: 0.7255 - accuracy: 0.9614
1370/1688 [=======================>......] - ETA: 0s - loss: 0.7238 - accuracy: 0.9616
1390/1688 [=======================>......] - ETA: 0s - loss: 0.7209 - accuracy: 0.9617
1410/1688 [========================>.....] - ETA: 0s - loss: 0.7215 - accuracy: 0.9616
1430/1688 [========================>.....] - ETA: 0s - loss: 0.7231 - accuracy: 0.9617
1450/1688 [========================>.....] - ETA: 0s - loss: 0.7274 - accuracy: 0.9616
1470/1688 [=========================>....] - ETA: 0s - loss: 0.7288 - accuracy: 0.9614
1490/1688 [=========================>....] - ETA: 0s - loss: 0.7289 - accuracy: 0.9615
1510/1688 [=========================>....] - ETA: 0s - loss: 0.7280 - accuracy: 0.9616
1530/1688 [==========================>...] - ETA: 0s - loss: 0.7243 - accuracy: 0.9618
1550/1688 [==========================>...] - ETA: 0s - loss: 0.7241 - accuracy: 0.9619
1571/1688 [==========================>...] - ETA: 0s - loss: 0.7223 - accuracy: 0.9620
1591/1688 [===========================>..] - ETA: 0s - loss: 0.7196 - accuracy: 0.9622
1611/1688 [===========================>..] - ETA: 0s - loss: 0.7158 - accuracy: 0.9623
1631/1688 [===========================>..] - ETA: 0s - loss: 0.7149 - accuracy: 0.9624
1651/1688 [============================>.] - ETA: 0s - loss: 0.7135 - accuracy: 0.9624
1671/1688 [============================>.] - ETA: 0s - loss: 0.7150 - accuracy: 0.9624
1688/1688 [==============================] - 5s 3ms/step - loss: 0.7147 - accuracy: 0.9624 - val_loss: 0.8547 - val_accuracy: 0.9588
Epoch 3/5

   1/1688 [..............................] - ETA: 4s - loss: 1.9000 - accuracy: 0.9375
  21/1688 [..............................] - ETA: 4s - loss: 0.8871 - accuracy: 0.9613
  41/1688 [..............................] - ETA: 4s - loss: 0.7236 - accuracy: 0.9657
  61/1688 [>.............................] - ETA: 4s - loss: 0.6480 - accuracy: 0.9703
  81/1688 [>.............................] - ETA: 4s - loss: 0.5800 - accuracy: 0.9715
 101/1688 [>.............................] - ETA: 4s - loss: 0.6169 - accuracy: 0.9709
 121/1688 [=>............................] - ETA: 3s - loss: 0.6744 - accuracy: 0.9672
 141/1688 [=>............................] - ETA: 3s - loss: 0.6676 - accuracy: 0.9672
 161/1688 [=>............................] - ETA: 3s - loss: 0.7085 - accuracy: 0.9653
 181/1688 [==>...........................] - ETA: 3s - loss: 0.7326 - accuracy: 0.9650
 201/1688 [==>...........................] - ETA: 3s - loss: 0.7382 - accuracy: 0.9647
 221/1688 [==>...........................] - ETA: 3s - loss: 0.7072 - accuracy: 0.9661
 241/1688 [===>..........................] - ETA: 3s - loss: 0.6861 - accuracy: 0.9664
 261/1688 [===>..........................] - ETA: 3s - loss: 0.6845 - accuracy: 0.9664
 281/1688 [===>..........................] - ETA: 3s - loss: 0.6964 - accuracy: 0.9662
 301/1688 [====>.........................] - ETA: 3s - loss: 0.6816 - accuracy: 0.9666
 321/1688 [====>.........................] - ETA: 3s - loss: 0.6887 - accuracy: 0.9663
 341/1688 [=====>........................] - ETA: 3s - loss: 0.6706 - accuracy: 0.9664
 361/1688 [=====>........................] - ETA: 3s - loss: 0.6571 - accuracy: 0.9668
 381/1688 [=====>........................] - ETA: 3s - loss: 0.6471 - accuracy: 0.9671
 401/1688 [======>.......................] - ETA: 3s - loss: 0.6295 - accuracy: 0.9677
 421/1688 [======>.......................] - ETA: 3s - loss: 0.6317 - accuracy: 0.9675
 441/1688 [======>.......................] - ETA: 3s - loss: 0.6208 - accuracy: 0.9680
 461/1688 [=======>......................] - ETA: 3s - loss: 0.6081 - accuracy: 0.9684
 481/1688 [=======>......................] - ETA: 3s - loss: 0.5993 - accuracy: 0.9684
 501/1688 [=======>......................] - ETA: 3s - loss: 0.5935 - accuracy: 0.9689
 521/1688 [========>.....................] - ETA: 2s - loss: 0.5855 - accuracy: 0.9690
 541/1688 [========>.....................] - ETA: 2s - loss: 0.5885 - accuracy: 0.9688
 561/1688 [========>.....................] - ETA: 2s - loss: 0.5816 - accuracy: 0.9691
 581/1688 [=========>....................] - ETA: 2s - loss: 0.5779 - accuracy: 0.9692
 601/1688 [=========>....................] - ETA: 2s - loss: 0.5724 - accuracy: 0.9694
 621/1688 [==========>...................] - ETA: 2s - loss: 0.5759 - accuracy: 0.9693
 641/1688 [==========>...................] - ETA: 2s - loss: 0.5821 - accuracy: 0.9692
 661/1688 [==========>...................] - ETA: 2s - loss: 0.5765 - accuracy: 0.9695
 681/1688 [===========>..................] - ETA: 2s - loss: 0.5821 - accuracy: 0.9693
 702/1688 [===========>..................] - ETA: 2s - loss: 0.5904 - accuracy: 0.9692
 722/1688 [===========>..................] - ETA: 2s - loss: 0.5871 - accuracy: 0.9693
 742/1688 [============>.................] - ETA: 2s - loss: 0.5920 - accuracy: 0.9692
 762/1688 [============>.................] - ETA: 2s - loss: 0.6028 - accuracy: 0.9685
 782/1688 [============>.................] - ETA: 2s - loss: 0.5937 - accuracy: 0.9689
 802/1688 [=============>................] - ETA: 2s - loss: 0.5875 - accuracy: 0.9690
 823/1688 [=============>................] - ETA: 2s - loss: 0.5975 - accuracy: 0.9685
 843/1688 [=============>................] - ETA: 2s - loss: 0.5955 - accuracy: 0.9686
 863/1688 [==============>...............] - ETA: 2s - loss: 0.5952 - accuracy: 0.9688
 883/1688 [==============>...............] - ETA: 2s - loss: 0.5991 - accuracy: 0.9687
 903/1688 [===============>..............] - ETA: 2s - loss: 0.5923 - accuracy: 0.9689
 923/1688 [===============>..............] - ETA: 1s - loss: 0.5938 - accuracy: 0.9688
 943/1688 [===============>..............] - ETA: 1s - loss: 0.5944 - accuracy: 0.9688
 963/1688 [================>.............] - ETA: 1s - loss: 0.5920 - accuracy: 0.9690
 983/1688 [================>.............] - ETA: 1s - loss: 0.5915 - accuracy: 0.9690
1003/1688 [================>.............] - ETA: 1s - loss: 0.5900 - accuracy: 0.9690
1023/1688 [=================>............] - ETA: 1s - loss: 0.5893 - accuracy: 0.9691
1043/1688 [=================>............] - ETA: 1s - loss: 0.5872 - accuracy: 0.9692
1063/1688 [=================>............] - ETA: 1s - loss: 0.5933 - accuracy: 0.9690
1083/1688 [==================>...........] - ETA: 1s - loss: 0.5978 - accuracy: 0.9688
1103/1688 [==================>...........] - ETA: 1s - loss: 0.6013 - accuracy: 0.9687
1123/1688 [==================>...........] - ETA: 1s - loss: 0.5994 - accuracy: 0.9687
1143/1688 [===================>..........] - ETA: 1s - loss: 0.6019 - accuracy: 0.9687
1163/1688 [===================>..........] - ETA: 1s - loss: 0.5999 - accuracy: 0.9688
1184/1688 [====================>.........] - ETA: 1s - loss: 0.6042 - accuracy: 0.9688
1204/1688 [====================>.........] - ETA: 1s - loss: 0.6062 - accuracy: 0.9688
1224/1688 [====================>.........] - ETA: 1s - loss: 0.6014 - accuracy: 0.9688
1244/1688 [=====================>........] - ETA: 1s - loss: 0.6062 - accuracy: 0.9688
1264/1688 [=====================>........] - ETA: 1s - loss: 0.6103 - accuracy: 0.9687
1284/1688 [=====================>........] - ETA: 1s - loss: 0.6095 - accuracy: 0.9687
1304/1688 [======================>.......] - ETA: 0s - loss: 0.6050 - accuracy: 0.9688
1324/1688 [======================>.......] - ETA: 0s - loss: 0.6068 - accuracy: 0.9687
1344/1688 [======================>.......] - ETA: 0s - loss: 0.6065 - accuracy: 0.9686
1364/1688 [=======================>......] - ETA: 0s - loss: 0.6084 - accuracy: 0.9685
1384/1688 [=======================>......] - ETA: 0s - loss: 0.6070 - accuracy: 0.9685
1404/1688 [=======================>......] - ETA: 0s - loss: 0.6050 - accuracy: 0.9686
1424/1688 [========================>.....] - ETA: 0s - loss: 0.6048 - accuracy: 0.9685
1444/1688 [========================>.....] - ETA: 0s - loss: 0.6154 - accuracy: 0.9683
1464/1688 [=========================>....] - ETA: 0s - loss: 0.6139 - accuracy: 0.9683
1484/1688 [=========================>....] - ETA: 0s - loss: 0.6165 - accuracy: 0.9683
1504/1688 [=========================>....] - ETA: 0s - loss: 0.6173 - accuracy: 0.9684
1525/1688 [==========================>...] - ETA: 0s - loss: 0.6167 - accuracy: 0.9685
1545/1688 [==========================>...] - ETA: 0s - loss: 0.6181 - accuracy: 0.9684
1565/1688 [==========================>...] - ETA: 0s - loss: 0.6199 - accuracy: 0.9685
1586/1688 [===========================>..] - ETA: 0s - loss: 0.6175 - accuracy: 0.9685
1607/1688 [===========================>..] - ETA: 0s - loss: 0.6192 - accuracy: 0.9685
1627/1688 [===========================>..] - ETA: 0s - loss: 0.6188 - accuracy: 0.9687
1647/1688 [============================>.] - ETA: 0s - loss: 0.6179 - accuracy: 0.9688
1667/1688 [============================>.] - ETA: 0s - loss: 0.6176 - accuracy: 0.9688
1687/1688 [============================>.] - ETA: 0s - loss: 0.6191 - accuracy: 0.9688
1688/1688 [==============================] - 5s 3ms/step - loss: 0.6189 - accuracy: 0.9688 - val_loss: 0.7367 - val_accuracy: 0.9698
Epoch 4/5

   1/1688 [..............................] - ETA: 4s - loss: 0.2057 - accuracy: 0.9375
  21/1688 [..............................] - ETA: 4s - loss: 0.3711 - accuracy: 0.9717
  41/1688 [..............................] - ETA: 4s - loss: 0.4600 - accuracy: 0.9756
  61/1688 [>.............................] - ETA: 4s - loss: 0.4669 - accuracy: 0.9754
  82/1688 [>.............................] - ETA: 4s - loss: 0.5125 - accuracy: 0.9752
 102/1688 [>.............................] - ETA: 4s - loss: 0.4721 - accuracy: 0.9758
 122/1688 [=>............................] - ETA: 3s - loss: 0.4819 - accuracy: 0.9744
 142/1688 [=>............................] - ETA: 3s - loss: 0.5163 - accuracy: 0.9734
 162/1688 [=>............................] - ETA: 3s - loss: 0.4983 - accuracy: 0.9743
 182/1688 [==>...........................] - ETA: 3s - loss: 0.4929 - accuracy: 0.9749
 202/1688 [==>...........................] - ETA: 3s - loss: 0.4716 - accuracy: 0.9760
 222/1688 [==>...........................] - ETA: 3s - loss: 0.4810 - accuracy: 0.9755
 242/1688 [===>..........................] - ETA: 3s - loss: 0.5107 - accuracy: 0.9746
 262/1688 [===>..........................] - ETA: 3s - loss: 0.5211 - accuracy: 0.9741
 282/1688 [====>.........................] - ETA: 3s - loss: 0.5190 - accuracy: 0.9743
 302/1688 [====>.........................] - ETA: 3s - loss: 0.5258 - accuracy: 0.9741
 322/1688 [====>.........................] - ETA: 3s - loss: 0.5478 - accuracy: 0.9734
 342/1688 [=====>........................] - ETA: 3s - loss: 0.5536 - accuracy: 0.9732
 362/1688 [=====>........................] - ETA: 3s - loss: 0.5560 - accuracy: 0.9731
 382/1688 [=====>........................] - ETA: 3s - loss: 0.5392 - accuracy: 0.9732
 402/1688 [======>.......................] - ETA: 3s - loss: 0.5473 - accuracy: 0.9726
 422/1688 [======>.......................] - ETA: 3s - loss: 0.5431 - accuracy: 0.9722
 442/1688 [======>.......................] - ETA: 3s - loss: 0.5612 - accuracy: 0.9718
 462/1688 [=======>......................] - ETA: 3s - loss: 0.5611 - accuracy: 0.9719
 482/1688 [=======>......................] - ETA: 3s - loss: 0.5672 - accuracy: 0.9714
 502/1688 [=======>......................] - ETA: 3s - loss: 0.5724 - accuracy: 0.9710
 522/1688 [========>.....................] - ETA: 2s - loss: 0.5753 - accuracy: 0.9709
 543/1688 [========>.....................] - ETA: 2s - loss: 0.5657 - accuracy: 0.9715
 563/1688 [=========>....................] - ETA: 2s - loss: 0.5595 - accuracy: 0.9717
 583/1688 [=========>....................] - ETA: 2s - loss: 0.5501 - accuracy: 0.9721
 603/1688 [=========>....................] - ETA: 2s - loss: 0.5576 - accuracy: 0.9719
 623/1688 [==========>...................] - ETA: 2s - loss: 0.5595 - accuracy: 0.9719
 643/1688 [==========>...................] - ETA: 2s - loss: 0.5539 - accuracy: 0.9720
 663/1688 [==========>...................] - ETA: 2s - loss: 0.5536 - accuracy: 0.9719
 684/1688 [===========>..................] - ETA: 2s - loss: 0.5492 - accuracy: 0.9720
 704/1688 [===========>..................] - ETA: 2s - loss: 0.5475 - accuracy: 0.9722
 724/1688 [===========>..................] - ETA: 2s - loss: 0.5547 - accuracy: 0.9718
 744/1688 [============>.................] - ETA: 2s - loss: 0.5596 - accuracy: 0.9714
 764/1688 [============>.................] - ETA: 2s - loss: 0.5628 - accuracy: 0.9714
 784/1688 [============>.................] - ETA: 2s - loss: 0.5705 - accuracy: 0.9712
 804/1688 [=============>................] - ETA: 2s - loss: 0.5710 - accuracy: 0.9712
 824/1688 [=============>................] - ETA: 2s - loss: 0.5696 - accuracy: 0.9713
 844/1688 [==============>...............] - ETA: 2s - loss: 0.5751 - accuracy: 0.9713
 864/1688 [==============>...............] - ETA: 2s - loss: 0.5744 - accuracy: 0.9712
 884/1688 [==============>...............] - ETA: 2s - loss: 0.5767 - accuracy: 0.9712
 904/1688 [===============>..............] - ETA: 1s - loss: 0.5792 - accuracy: 0.9712
 925/1688 [===============>..............] - ETA: 1s - loss: 0.5731 - accuracy: 0.9715
 945/1688 [===============>..............] - ETA: 1s - loss: 0.5693 - accuracy: 0.9716
 965/1688 [================>.............] - ETA: 1s - loss: 0.5688 - accuracy: 0.9717
 985/1688 [================>.............] - ETA: 1s - loss: 0.5698 - accuracy: 0.9718
1005/1688 [================>.............] - ETA: 1s - loss: 0.5664 - accuracy: 0.9720
1025/1688 [=================>............] - ETA: 1s - loss: 0.5694 - accuracy: 0.9719
1045/1688 [=================>............] - ETA: 1s - loss: 0.5719 - accuracy: 0.9718
1065/1688 [=================>............] - ETA: 1s - loss: 0.5768 - accuracy: 0.9719
1085/1688 [==================>...........] - ETA: 1s - loss: 0.5800 - accuracy: 0.9717
1105/1688 [==================>...........] - ETA: 1s - loss: 0.5845 - accuracy: 0.9717
1125/1688 [==================>...........] - ETA: 1s - loss: 0.5820 - accuracy: 0.9717
1145/1688 [===================>..........] - ETA: 1s - loss: 0.5814 - accuracy: 0.9716
1166/1688 [===================>..........] - ETA: 1s - loss: 0.5782 - accuracy: 0.9717
1186/1688 [====================>.........] - ETA: 1s - loss: 0.5831 - accuracy: 0.9717
1206/1688 [====================>.........] - ETA: 1s - loss: 0.5828 - accuracy: 0.9715
1226/1688 [====================>.........] - ETA: 1s - loss: 0.5792 - accuracy: 0.9717
1246/1688 [=====================>........] - ETA: 1s - loss: 0.5777 - accuracy: 0.9718
1266/1688 [=====================>........] - ETA: 1s - loss: 0.5790 - accuracy: 0.9719
1286/1688 [=====================>........] - ETA: 1s - loss: 0.5776 - accuracy: 0.9718
1306/1688 [======================>.......] - ETA: 0s - loss: 0.5774 - accuracy: 0.9719
1326/1688 [======================>.......] - ETA: 0s - loss: 0.5744 - accuracy: 0.9720
1347/1688 [======================>.......] - ETA: 0s - loss: 0.5751 - accuracy: 0.9718
1367/1688 [=======================>......] - ETA: 0s - loss: 0.5751 - accuracy: 0.9718
1388/1688 [=======================>......] - ETA: 0s - loss: 0.5728 - accuracy: 0.9720
1408/1688 [========================>.....] - ETA: 0s - loss: 0.5779 - accuracy: 0.9720
1428/1688 [========================>.....] - ETA: 0s - loss: 0.5743 - accuracy: 0.9720
1448/1688 [========================>.....] - ETA: 0s - loss: 0.5747 - accuracy: 0.9720
1468/1688 [=========================>....] - ETA: 0s - loss: 0.5733 - accuracy: 0.9721
1488/1688 [=========================>....] - ETA: 0s - loss: 0.5717 - accuracy: 0.9721
1508/1688 [=========================>....] - ETA: 0s - loss: 0.5702 - accuracy: 0.9722
1528/1688 [==========================>...] - ETA: 0s - loss: 0.5703 - accuracy: 0.9721
1548/1688 [==========================>...] - ETA: 0s - loss: 0.5715 - accuracy: 0.9720
1568/1688 [==========================>...] - ETA: 0s - loss: 0.5720 - accuracy: 0.9719
1588/1688 [===========================>..] - ETA: 0s - loss: 0.5726 - accuracy: 0.9719
1608/1688 [===========================>..] - ETA: 0s - loss: 0.5744 - accuracy: 0.9718
1628/1688 [===========================>..] - ETA: 0s - loss: 0.5774 - accuracy: 0.9716
1649/1688 [============================>.] - ETA: 0s - loss: 0.5784 - accuracy: 0.9716
1669/1688 [============================>.] - ETA: 0s - loss: 0.5762 - accuracy: 0.9717
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5802 - accuracy: 0.9716 - val_loss: 0.7767 - val_accuracy: 0.9698
Epoch 5/5

   1/1688 [..............................] - ETA: 4s - loss: 1.8920 - accuracy: 0.9688
  21/1688 [..............................] - ETA: 4s - loss: 0.9109 - accuracy: 0.9673
  41/1688 [..............................] - ETA: 4s - loss: 0.7397 - accuracy: 0.9688
  61/1688 [>.............................] - ETA: 4s - loss: 0.6536 - accuracy: 0.9693
  81/1688 [>.............................] - ETA: 4s - loss: 0.6163 - accuracy: 0.9707
 101/1688 [>.............................] - ETA: 4s - loss: 0.5715 - accuracy: 0.9706
 122/1688 [=>............................] - ETA: 3s - loss: 0.5337 - accuracy: 0.9711
 142/1688 [=>............................] - ETA: 3s - loss: 0.5377 - accuracy: 0.9710
 162/1688 [=>............................] - ETA: 3s - loss: 0.5098 - accuracy: 0.9718
 182/1688 [==>...........................] - ETA: 3s - loss: 0.4926 - accuracy: 0.9727
 202/1688 [==>...........................] - ETA: 3s - loss: 0.4899 - accuracy: 0.9725
 222/1688 [==>...........................] - ETA: 3s - loss: 0.4787 - accuracy: 0.9728
 242/1688 [===>..........................] - ETA: 3s - loss: 0.4910 - accuracy: 0.9730
 262/1688 [===>..........................] - ETA: 3s - loss: 0.5024 - accuracy: 0.9722
 282/1688 [====>.........................] - ETA: 3s - loss: 0.4946 - accuracy: 0.9723
 302/1688 [====>.........................] - ETA: 3s - loss: 0.4934 - accuracy: 0.9728
 322/1688 [====>.........................] - ETA: 3s - loss: 0.5007 - accuracy: 0.9726
 342/1688 [=====>........................] - ETA: 3s - loss: 0.4960 - accuracy: 0.9729
 362/1688 [=====>........................] - ETA: 3s - loss: 0.5016 - accuracy: 0.9726
 383/1688 [=====>........................] - ETA: 3s - loss: 0.4998 - accuracy: 0.9731
 403/1688 [======>.......................] - ETA: 3s - loss: 0.4950 - accuracy: 0.9729
 423/1688 [======>.......................] - ETA: 3s - loss: 0.4881 - accuracy: 0.9730
 443/1688 [======>.......................] - ETA: 3s - loss: 0.4809 - accuracy: 0.9734
 463/1688 [=======>......................] - ETA: 3s - loss: 0.4723 - accuracy: 0.9740
 483/1688 [=======>......................] - ETA: 3s - loss: 0.4613 - accuracy: 0.9744
 504/1688 [=======>......................] - ETA: 2s - loss: 0.4586 - accuracy: 0.9748
 524/1688 [========>.....................] - ETA: 2s - loss: 0.4633 - accuracy: 0.9749
 544/1688 [========>.....................] - ETA: 2s - loss: 0.4621 - accuracy: 0.9748
 564/1688 [=========>....................] - ETA: 2s - loss: 0.4573 - accuracy: 0.9750
 584/1688 [=========>....................] - ETA: 2s - loss: 0.4603 - accuracy: 0.9749
 604/1688 [=========>....................] - ETA: 2s - loss: 0.4590 - accuracy: 0.9749
 623/1688 [==========>...................] - ETA: 2s - loss: 0.4672 - accuracy: 0.9746
 643/1688 [==========>...................] - ETA: 2s - loss: 0.4762 - accuracy: 0.9743
 663/1688 [==========>...................] - ETA: 2s - loss: 0.4770 - accuracy: 0.9743
 683/1688 [===========>..................] - ETA: 2s - loss: 0.4838 - accuracy: 0.9741
 703/1688 [===========>..................] - ETA: 2s - loss: 0.4859 - accuracy: 0.9739
 723/1688 [===========>..................] - ETA: 2s - loss: 0.4872 - accuracy: 0.9740
 743/1688 [============>.................] - ETA: 2s - loss: 0.4872 - accuracy: 0.9741
 763/1688 [============>.................] - ETA: 2s - loss: 0.4861 - accuracy: 0.9742
 783/1688 [============>.................] - ETA: 2s - loss: 0.4914 - accuracy: 0.9739
 803/1688 [=============>................] - ETA: 2s - loss: 0.4867 - accuracy: 0.9740
 823/1688 [=============>................] - ETA: 2s - loss: 0.4923 - accuracy: 0.9739
 843/1688 [=============>................] - ETA: 2s - loss: 0.4971 - accuracy: 0.9738
 863/1688 [==============>...............] - ETA: 2s - loss: 0.5015 - accuracy: 0.9737
 883/1688 [==============>...............] - ETA: 2s - loss: 0.5103 - accuracy: 0.9737
 903/1688 [===============>..............] - ETA: 1s - loss: 0.5155 - accuracy: 0.9736
 923/1688 [===============>..............] - ETA: 1s - loss: 0.5165 - accuracy: 0.9736
 944/1688 [===============>..............] - ETA: 1s - loss: 0.5225 - accuracy: 0.9733
 964/1688 [================>.............] - ETA: 1s - loss: 0.5264 - accuracy: 0.9731
 984/1688 [================>.............] - ETA: 1s - loss: 0.5393 - accuracy: 0.9728
1004/1688 [================>.............] - ETA: 1s - loss: 0.5481 - accuracy: 0.9726
1024/1688 [=================>............] - ETA: 1s - loss: 0.5415 - accuracy: 0.9728
1043/1688 [=================>............] - ETA: 1s - loss: 0.5398 - accuracy: 0.9729
1063/1688 [=================>............] - ETA: 1s - loss: 0.5419 - accuracy: 0.9729
1083/1688 [==================>...........] - ETA: 1s - loss: 0.5454 - accuracy: 0.9728
1103/1688 [==================>...........] - ETA: 1s - loss: 0.5437 - accuracy: 0.9729
1123/1688 [==================>...........] - ETA: 1s - loss: 0.5390 - accuracy: 0.9729
1143/1688 [===================>..........] - ETA: 1s - loss: 0.5422 - accuracy: 0.9727
1163/1688 [===================>..........] - ETA: 1s - loss: 0.5406 - accuracy: 0.9728
1183/1688 [====================>.........] - ETA: 1s - loss: 0.5403 - accuracy: 0.9728
1203/1688 [====================>.........] - ETA: 1s - loss: 0.5370 - accuracy: 0.9728
1224/1688 [====================>.........] - ETA: 1s - loss: 0.5412 - accuracy: 0.9726
1244/1688 [=====================>........] - ETA: 1s - loss: 0.5430 - accuracy: 0.9727
1264/1688 [=====================>........] - ETA: 1s - loss: 0.5399 - accuracy: 0.9729
1284/1688 [=====================>........] - ETA: 1s - loss: 0.5412 - accuracy: 0.9728
1304/1688 [======================>.......] - ETA: 0s - loss: 0.5466 - accuracy: 0.9727
1325/1688 [======================>.......] - ETA: 0s - loss: 0.5482 - accuracy: 0.9726
1345/1688 [======================>.......] - ETA: 0s - loss: 0.5457 - accuracy: 0.9727
1365/1688 [=======================>......] - ETA: 0s - loss: 0.5466 - accuracy: 0.9726
1385/1688 [=======================>......] - ETA: 0s - loss: 0.5469 - accuracy: 0.9726
1405/1688 [=======================>......] - ETA: 0s - loss: 0.5458 - accuracy: 0.9726
1425/1688 [========================>.....] - ETA: 0s - loss: 0.5510 - accuracy: 0.9725
1445/1688 [========================>.....] - ETA: 0s - loss: 0.5566 - accuracy: 0.9723
1465/1688 [=========================>....] - ETA: 0s - loss: 0.5574 - accuracy: 0.9724
1486/1688 [=========================>....] - ETA: 0s - loss: 0.5586 - accuracy: 0.9724
1506/1688 [=========================>....] - ETA: 0s - loss: 0.5572 - accuracy: 0.9724
1526/1688 [==========================>...] - ETA: 0s - loss: 0.5563 - accuracy: 0.9725
1546/1688 [==========================>...] - ETA: 0s - loss: 0.5604 - accuracy: 0.9724
1566/1688 [==========================>...] - ETA: 0s - loss: 0.5652 - accuracy: 0.9723
1586/1688 [===========================>..] - ETA: 0s - loss: 0.5651 - accuracy: 0.9723
1606/1688 [===========================>..] - ETA: 0s - loss: 0.5641 - accuracy: 0.9723
1627/1688 [===========================>..] - ETA: 0s - loss: 0.5631 - accuracy: 0.9724
1648/1688 [============================>.] - ETA: 0s - loss: 0.5669 - accuracy: 0.9723
1668/1688 [============================>.] - ETA: 0s - loss: 0.5672 - accuracy: 0.9723
1687/1688 [============================>.] - ETA: 0s - loss: 0.5654 - accuracy: 0.9724
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5660 - accuracy: 0.9723 - val_loss: 0.6566 - val_accuracy: 0.9720
Test accuracy after fine tuning: 0.9693999886512756

6. Model conversion

After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida software simulator.

Note

One needs to supply the coefficients used to rescale the input dataset before the training - here input_scaling.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()

accuracy = model_akida.evaluate(raw_x_test, raw_y_test)
print('Test accuracy after conversion:', accuracy)

# For non-regression purpose
assert accuracy > 0.96

Out:

                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          3
______________________________________________

                  SW/conv2d-dense (Software)
______________________________________________________________
Layer (type)                  Output shape  Kernel shape
==============================================================
conv2d (InputConv.)           [13, 13, 32]  (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.)  [7, 7, 64]    (3, 3, 32, 1)
______________________________________________________________
                                            (1, 1, 32, 64)
______________________________________________________________
dense (Fully.)                [1, 1, 10]    (1, 1, 3136, 10)
______________________________________________________________

Test accuracy after conversion: 0.9692999720573425

Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).

Total running time of the script: ( 0 minutes 59.283 seconds)

Gallery generated by Sphinx-Gallery