CNN conversion flow tutorial

This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.

The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.

Note

Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.

MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.

../../_images/cnn2snn_flow_small.jpg

1. Load and reshape MNIST dataset

After loading, we make 2 transformations on the dataset:

  1. Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.

Note

At this point, we’ll set aside the raw data for testing our converted model in the Akida runtime later.

  1. Rescale the 8-bit loaded data to the range 0-to-1 for training.

Note

Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.

This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.

Also note that we store the scaling values input_scaling for use when preparing the model for the Akida neuromorphic IP. The implementation of the Akida neural network allows us to completely skip the rescaling step (i.e. the Akida model should be fed with the raw 8-bit values) but that does require information about what scaling was applied prior to training - see below for more details.

from tensorflow import keras

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)

# Set aside raw test data for use with Akida runtime later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test

# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a

2. Model definition

Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.

An appropriate model for MNIST (inspired by this example) might look something like the following:

model_keras = keras.models.Sequential([
    keras.layers.Conv2D(
        filters=32, kernel_size=3, input_shape=(28, 28, 1), strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.SeparableConv2D(
        filters=64, kernel_size=3, padding='same', strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 conv2d (Conv2D)             (None, 13, 13, 32)        320

 batch_normalization (BatchN  (None, 13, 13, 32)       128
 ormalization)

 re_lu (ReLU)                (None, 13, 13, 32)        0

 separable_conv2d (Separable  (None, 7, 7, 64)         2400
 Conv2D)

 batch_normalization_1 (Batc  (None, 7, 7, 64)         256
 hNormalization)

 re_lu_1 (ReLU)              (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (Dense)               (None, 10)                31370

=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________

The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):

  • Standard Conv2D and Dense layers are supported

  • Hidden layers must be followed by a ReLU layer.

  • BatchNormalization must always happen before activations.

  • Convolutional blocks can optionally be followed by a MaxPooling.

The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).

from cnn2snn import check_model_compatibility

print("Model compatible for Akida conversion:",
      check_model_compatibility(model_keras))

Out:

Model compatible for Akida conversion: True

3. Model training

Before going any further, train the model and get its performance. The created model should have achieved a test accuracy over 98% after 10 epochs.

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)

score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])

Out:

Epoch 1/10

   1/1688 [..............................] - ETA: 11:51 - loss: 2.7239 - accuracy: 0.0938
  28/1688 [..............................] - ETA: 3s - loss: 1.3301 - accuracy: 0.5938   
  56/1688 [..............................] - ETA: 2s - loss: 0.9390 - accuracy: 0.7109
  84/1688 [>.............................] - ETA: 2s - loss: 0.7748 - accuracy: 0.7638
 113/1688 [=>............................] - ETA: 2s - loss: 0.6793 - accuracy: 0.7929
 142/1688 [=>............................] - ETA: 2s - loss: 0.6182 - accuracy: 0.8134
 171/1688 [==>...........................] - ETA: 2s - loss: 0.5609 - accuracy: 0.8300
 199/1688 [==>...........................] - ETA: 2s - loss: 0.5208 - accuracy: 0.8430
 228/1688 [===>..........................] - ETA: 2s - loss: 0.4973 - accuracy: 0.8509
 256/1688 [===>..........................] - ETA: 2s - loss: 0.4710 - accuracy: 0.8586
 284/1688 [====>.........................] - ETA: 2s - loss: 0.4531 - accuracy: 0.8637
 312/1688 [====>.........................] - ETA: 2s - loss: 0.4338 - accuracy: 0.8694
 340/1688 [=====>........................] - ETA: 2s - loss: 0.4129 - accuracy: 0.8753
 368/1688 [=====>........................] - ETA: 2s - loss: 0.3958 - accuracy: 0.8809
 397/1688 [======>.......................] - ETA: 2s - loss: 0.3807 - accuracy: 0.8852
 426/1688 [======>.......................] - ETA: 2s - loss: 0.3723 - accuracy: 0.8878
 454/1688 [=======>......................] - ETA: 2s - loss: 0.3604 - accuracy: 0.8918
 482/1688 [=======>......................] - ETA: 2s - loss: 0.3483 - accuracy: 0.8959
 511/1688 [========>.....................] - ETA: 2s - loss: 0.3396 - accuracy: 0.8987
 540/1688 [========>.....................] - ETA: 2s - loss: 0.3321 - accuracy: 0.9008
 568/1688 [=========>....................] - ETA: 2s - loss: 0.3227 - accuracy: 0.9031
 597/1688 [=========>....................] - ETA: 1s - loss: 0.3135 - accuracy: 0.9062
 626/1688 [==========>...................] - ETA: 1s - loss: 0.3059 - accuracy: 0.9087
 654/1688 [==========>...................] - ETA: 1s - loss: 0.2988 - accuracy: 0.9105
 683/1688 [===========>..................] - ETA: 1s - loss: 0.2919 - accuracy: 0.9126
 712/1688 [===========>..................] - ETA: 1s - loss: 0.2848 - accuracy: 0.9147
 740/1688 [============>.................] - ETA: 1s - loss: 0.2795 - accuracy: 0.9165
 769/1688 [============>.................] - ETA: 1s - loss: 0.2738 - accuracy: 0.9178
 798/1688 [=============>................] - ETA: 1s - loss: 0.2690 - accuracy: 0.9191
 826/1688 [=============>................] - ETA: 1s - loss: 0.2638 - accuracy: 0.9209
 853/1688 [==============>...............] - ETA: 1s - loss: 0.2584 - accuracy: 0.9224
 881/1688 [==============>...............] - ETA: 1s - loss: 0.2540 - accuracy: 0.9235
 910/1688 [===============>..............] - ETA: 1s - loss: 0.2493 - accuracy: 0.9248
 938/1688 [===============>..............] - ETA: 1s - loss: 0.2458 - accuracy: 0.9258
 966/1688 [================>.............] - ETA: 1s - loss: 0.2422 - accuracy: 0.9268
 994/1688 [================>.............] - ETA: 1s - loss: 0.2380 - accuracy: 0.9281
1022/1688 [=================>............] - ETA: 1s - loss: 0.2344 - accuracy: 0.9291
1050/1688 [=================>............] - ETA: 1s - loss: 0.2310 - accuracy: 0.9299
1078/1688 [==================>...........] - ETA: 1s - loss: 0.2273 - accuracy: 0.9311
1107/1688 [==================>...........] - ETA: 1s - loss: 0.2237 - accuracy: 0.9321
1135/1688 [===================>..........] - ETA: 0s - loss: 0.2214 - accuracy: 0.9327
1163/1688 [===================>..........] - ETA: 0s - loss: 0.2195 - accuracy: 0.9333
1192/1688 [====================>.........] - ETA: 0s - loss: 0.2163 - accuracy: 0.9342
1221/1688 [====================>.........] - ETA: 0s - loss: 0.2135 - accuracy: 0.9351
1250/1688 [=====================>........] - ETA: 0s - loss: 0.2103 - accuracy: 0.9361
1279/1688 [=====================>........] - ETA: 0s - loss: 0.2079 - accuracy: 0.9367
1307/1688 [======================>.......] - ETA: 0s - loss: 0.2051 - accuracy: 0.9377
1335/1688 [======================>.......] - ETA: 0s - loss: 0.2022 - accuracy: 0.9386
1363/1688 [=======================>......] - ETA: 0s - loss: 0.2000 - accuracy: 0.9391
1391/1688 [=======================>......] - ETA: 0s - loss: 0.1986 - accuracy: 0.9396
1420/1688 [========================>.....] - ETA: 0s - loss: 0.1959 - accuracy: 0.9404
1448/1688 [========================>.....] - ETA: 0s - loss: 0.1933 - accuracy: 0.9411
1476/1688 [=========================>....] - ETA: 0s - loss: 0.1917 - accuracy: 0.9416
1505/1688 [=========================>....] - ETA: 0s - loss: 0.1901 - accuracy: 0.9421
1533/1688 [==========================>...] - ETA: 0s - loss: 0.1886 - accuracy: 0.9426
1561/1688 [==========================>...] - ETA: 0s - loss: 0.1866 - accuracy: 0.9431
1590/1688 [===========================>..] - ETA: 0s - loss: 0.1846 - accuracy: 0.9438
1619/1688 [===========================>..] - ETA: 0s - loss: 0.1839 - accuracy: 0.9439
1647/1688 [============================>.] - ETA: 0s - loss: 0.1823 - accuracy: 0.9443
1676/1688 [============================>.] - ETA: 0s - loss: 0.1812 - accuracy: 0.9446
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1805 - accuracy: 0.9448 - val_loss: 0.1017 - val_accuracy: 0.9705
Epoch 2/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0141 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0691 - accuracy: 0.9758
  59/1688 [>.............................] - ETA: 2s - loss: 0.0758 - accuracy: 0.9783
  89/1688 [>.............................] - ETA: 2s - loss: 0.0867 - accuracy: 0.9754
 119/1688 [=>............................] - ETA: 2s - loss: 0.0814 - accuracy: 0.9761
 148/1688 [=>............................] - ETA: 2s - loss: 0.0814 - accuracy: 0.9753
 177/1688 [==>...........................] - ETA: 2s - loss: 0.0853 - accuracy: 0.9748
 205/1688 [==>...........................] - ETA: 2s - loss: 0.0814 - accuracy: 0.9753
 233/1688 [===>..........................] - ETA: 2s - loss: 0.0826 - accuracy: 0.9753
 262/1688 [===>..........................] - ETA: 2s - loss: 0.0801 - accuracy: 0.9759
 290/1688 [====>.........................] - ETA: 2s - loss: 0.0773 - accuracy: 0.9767
 319/1688 [====>.........................] - ETA: 2s - loss: 0.0763 - accuracy: 0.9769
 347/1688 [=====>........................] - ETA: 2s - loss: 0.0753 - accuracy: 0.9771
 375/1688 [=====>........................] - ETA: 2s - loss: 0.0746 - accuracy: 0.9771
 402/1688 [======>.......................] - ETA: 2s - loss: 0.0741 - accuracy: 0.9771
 431/1688 [======>.......................] - ETA: 2s - loss: 0.0756 - accuracy: 0.9765
 459/1688 [=======>......................] - ETA: 2s - loss: 0.0789 - accuracy: 0.9756
 489/1688 [=======>......................] - ETA: 2s - loss: 0.0799 - accuracy: 0.9750
 517/1688 [========>.....................] - ETA: 2s - loss: 0.0797 - accuracy: 0.9750
 545/1688 [========>.....................] - ETA: 2s - loss: 0.0793 - accuracy: 0.9752
 574/1688 [=========>....................] - ETA: 1s - loss: 0.0788 - accuracy: 0.9754
 602/1688 [=========>....................] - ETA: 1s - loss: 0.0778 - accuracy: 0.9758
 631/1688 [==========>...................] - ETA: 1s - loss: 0.0767 - accuracy: 0.9762
 659/1688 [==========>...................] - ETA: 1s - loss: 0.0774 - accuracy: 0.9762
 688/1688 [===========>..................] - ETA: 1s - loss: 0.0769 - accuracy: 0.9763
 716/1688 [===========>..................] - ETA: 1s - loss: 0.0762 - accuracy: 0.9766
 745/1688 [============>.................] - ETA: 1s - loss: 0.0755 - accuracy: 0.9768
 774/1688 [============>.................] - ETA: 1s - loss: 0.0755 - accuracy: 0.9769
 801/1688 [=============>................] - ETA: 1s - loss: 0.0748 - accuracy: 0.9770
 829/1688 [=============>................] - ETA: 1s - loss: 0.0753 - accuracy: 0.9769
 857/1688 [==============>...............] - ETA: 1s - loss: 0.0755 - accuracy: 0.9768
 885/1688 [==============>...............] - ETA: 1s - loss: 0.0762 - accuracy: 0.9764
 913/1688 [===============>..............] - ETA: 1s - loss: 0.0762 - accuracy: 0.9763
 942/1688 [===============>..............] - ETA: 1s - loss: 0.0764 - accuracy: 0.9761
 970/1688 [================>.............] - ETA: 1s - loss: 0.0761 - accuracy: 0.9763
 999/1688 [================>.............] - ETA: 1s - loss: 0.0753 - accuracy: 0.9765
1027/1688 [=================>............] - ETA: 1s - loss: 0.0753 - accuracy: 0.9766
1056/1688 [=================>............] - ETA: 1s - loss: 0.0753 - accuracy: 0.9765
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0751 - accuracy: 0.9766
1113/1688 [==================>...........] - ETA: 1s - loss: 0.0750 - accuracy: 0.9765
1141/1688 [===================>..........] - ETA: 0s - loss: 0.0748 - accuracy: 0.9767
1169/1688 [===================>..........] - ETA: 0s - loss: 0.0744 - accuracy: 0.9769
1197/1688 [====================>.........] - ETA: 0s - loss: 0.0746 - accuracy: 0.9768
1225/1688 [====================>.........] - ETA: 0s - loss: 0.0741 - accuracy: 0.9769
1254/1688 [=====================>........] - ETA: 0s - loss: 0.0741 - accuracy: 0.9770
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0742 - accuracy: 0.9770
1310/1688 [======================>.......] - ETA: 0s - loss: 0.0741 - accuracy: 0.9769
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0749 - accuracy: 0.9767
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0746 - accuracy: 0.9768
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0741 - accuracy: 0.9770
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0738 - accuracy: 0.9771
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0729 - accuracy: 0.9774
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0730 - accuracy: 0.9774
1510/1688 [=========================>....] - ETA: 0s - loss: 0.0726 - accuracy: 0.9775
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0726 - accuracy: 0.9776
1567/1688 [==========================>...] - ETA: 0s - loss: 0.0727 - accuracy: 0.9775
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0724 - accuracy: 0.9775
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0723 - accuracy: 0.9775
1653/1688 [============================>.] - ETA: 0s - loss: 0.0726 - accuracy: 0.9775
1682/1688 [============================>.] - ETA: 0s - loss: 0.0723 - accuracy: 0.9776
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0725 - accuracy: 0.9775 - val_loss: 0.0751 - val_accuracy: 0.9782
Epoch 3/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0191 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0580 - accuracy: 0.9771
  57/1688 [>.............................] - ETA: 2s - loss: 0.0508 - accuracy: 0.9836
  86/1688 [>.............................] - ETA: 2s - loss: 0.0470 - accuracy: 0.9851
 114/1688 [=>............................] - ETA: 2s - loss: 0.0498 - accuracy: 0.9855
 142/1688 [=>............................] - ETA: 2s - loss: 0.0505 - accuracy: 0.9855
 171/1688 [==>...........................] - ETA: 2s - loss: 0.0507 - accuracy: 0.9854
 199/1688 [==>...........................] - ETA: 2s - loss: 0.0518 - accuracy: 0.9846
 227/1688 [===>..........................] - ETA: 2s - loss: 0.0493 - accuracy: 0.9851
 255/1688 [===>..........................] - ETA: 2s - loss: 0.0477 - accuracy: 0.9857
 283/1688 [====>.........................] - ETA: 2s - loss: 0.0481 - accuracy: 0.9856
 311/1688 [====>.........................] - ETA: 2s - loss: 0.0511 - accuracy: 0.9845
 339/1688 [=====>........................] - ETA: 2s - loss: 0.0489 - accuracy: 0.9851
 367/1688 [=====>........................] - ETA: 2s - loss: 0.0480 - accuracy: 0.9854
 396/1688 [======>.......................] - ETA: 2s - loss: 0.0467 - accuracy: 0.9858
 424/1688 [======>.......................] - ETA: 2s - loss: 0.0463 - accuracy: 0.9860
 452/1688 [=======>......................] - ETA: 2s - loss: 0.0459 - accuracy: 0.9858
 480/1688 [=======>......................] - ETA: 2s - loss: 0.0459 - accuracy: 0.9859
 507/1688 [========>.....................] - ETA: 2s - loss: 0.0455 - accuracy: 0.9861
 535/1688 [========>.....................] - ETA: 2s - loss: 0.0456 - accuracy: 0.9860
 563/1688 [=========>....................] - ETA: 2s - loss: 0.0465 - accuracy: 0.9860
 591/1688 [=========>....................] - ETA: 1s - loss: 0.0466 - accuracy: 0.9859
 619/1688 [==========>...................] - ETA: 1s - loss: 0.0467 - accuracy: 0.9858
 646/1688 [==========>...................] - ETA: 1s - loss: 0.0470 - accuracy: 0.9857
 674/1688 [==========>...................] - ETA: 1s - loss: 0.0479 - accuracy: 0.9855
 703/1688 [===========>..................] - ETA: 1s - loss: 0.0475 - accuracy: 0.9853
 732/1688 [============>.................] - ETA: 1s - loss: 0.0481 - accuracy: 0.9850
 760/1688 [============>.................] - ETA: 1s - loss: 0.0481 - accuracy: 0.9850
 789/1688 [=============>................] - ETA: 1s - loss: 0.0493 - accuracy: 0.9848
 818/1688 [=============>................] - ETA: 1s - loss: 0.0495 - accuracy: 0.9846
 847/1688 [==============>...............] - ETA: 1s - loss: 0.0499 - accuracy: 0.9842
 875/1688 [==============>...............] - ETA: 1s - loss: 0.0505 - accuracy: 0.9839
 903/1688 [===============>..............] - ETA: 1s - loss: 0.0501 - accuracy: 0.9840
 930/1688 [===============>..............] - ETA: 1s - loss: 0.0497 - accuracy: 0.9841
 958/1688 [================>.............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9841
 987/1688 [================>.............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9841
1015/1688 [=================>............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9841
1043/1688 [=================>............] - ETA: 1s - loss: 0.0507 - accuracy: 0.9840
1071/1688 [==================>...........] - ETA: 1s - loss: 0.0508 - accuracy: 0.9838
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0505 - accuracy: 0.9839
1127/1688 [===================>..........] - ETA: 1s - loss: 0.0511 - accuracy: 0.9838
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0509 - accuracy: 0.9838
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0508 - accuracy: 0.9839
1213/1688 [====================>.........] - ETA: 0s - loss: 0.0508 - accuracy: 0.9839
1241/1688 [=====================>........] - ETA: 0s - loss: 0.0509 - accuracy: 0.9839
1269/1688 [=====================>........] - ETA: 0s - loss: 0.0504 - accuracy: 0.9842
1297/1688 [======================>.......] - ETA: 0s - loss: 0.0507 - accuracy: 0.9841
1325/1688 [======================>.......] - ETA: 0s - loss: 0.0507 - accuracy: 0.9841
1353/1688 [=======================>......] - ETA: 0s - loss: 0.0507 - accuracy: 0.9839
1382/1688 [=======================>......] - ETA: 0s - loss: 0.0511 - accuracy: 0.9839
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0517 - accuracy: 0.9837
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0521 - accuracy: 0.9836
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0524 - accuracy: 0.9836
1494/1688 [=========================>....] - ETA: 0s - loss: 0.0519 - accuracy: 0.9838
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0518 - accuracy: 0.9838
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0516 - accuracy: 0.9838
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0515 - accuracy: 0.9838
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0520 - accuracy: 0.9836
1635/1688 [============================>.] - ETA: 0s - loss: 0.0520 - accuracy: 0.9837
1663/1688 [============================>.] - ETA: 0s - loss: 0.0523 - accuracy: 0.9836
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0520 - accuracy: 0.9836 - val_loss: 0.0637 - val_accuracy: 0.9833
Epoch 4/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0853 - accuracy: 0.9688
  30/1688 [..............................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9865
  59/1688 [>.............................] - ETA: 2s - loss: 0.0394 - accuracy: 0.9894
  88/1688 [>.............................] - ETA: 2s - loss: 0.0391 - accuracy: 0.9879
 117/1688 [=>............................] - ETA: 2s - loss: 0.0362 - accuracy: 0.9885
 146/1688 [=>............................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9876
 175/1688 [==>...........................] - ETA: 2s - loss: 0.0367 - accuracy: 0.9887
 203/1688 [==>...........................] - ETA: 2s - loss: 0.0356 - accuracy: 0.9886
 232/1688 [===>..........................] - ETA: 2s - loss: 0.0372 - accuracy: 0.9881
 261/1688 [===>..........................] - ETA: 2s - loss: 0.0384 - accuracy: 0.9880
 289/1688 [====>.........................] - ETA: 2s - loss: 0.0392 - accuracy: 0.9880
 317/1688 [====>.........................] - ETA: 2s - loss: 0.0385 - accuracy: 0.9883
 346/1688 [=====>........................] - ETA: 2s - loss: 0.0379 - accuracy: 0.9886
 374/1688 [=====>........................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9886
 403/1688 [======>.......................] - ETA: 2s - loss: 0.0389 - accuracy: 0.9881
 432/1688 [======>.......................] - ETA: 2s - loss: 0.0394 - accuracy: 0.9878
 461/1688 [=======>......................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9876
 489/1688 [=======>......................] - ETA: 2s - loss: 0.0397 - accuracy: 0.9877
 517/1688 [========>.....................] - ETA: 2s - loss: 0.0393 - accuracy: 0.9878
 546/1688 [========>.....................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9876
 575/1688 [=========>....................] - ETA: 1s - loss: 0.0399 - accuracy: 0.9874
 603/1688 [=========>....................] - ETA: 1s - loss: 0.0404 - accuracy: 0.9873
 632/1688 [==========>...................] - ETA: 1s - loss: 0.0406 - accuracy: 0.9869
 661/1688 [==========>...................] - ETA: 1s - loss: 0.0414 - accuracy: 0.9866
 689/1688 [===========>..................] - ETA: 1s - loss: 0.0418 - accuracy: 0.9864
 718/1688 [===========>..................] - ETA: 1s - loss: 0.0417 - accuracy: 0.9865
 747/1688 [============>.................] - ETA: 1s - loss: 0.0410 - accuracy: 0.9868
 776/1688 [============>.................] - ETA: 1s - loss: 0.0413 - accuracy: 0.9868
 804/1688 [=============>................] - ETA: 1s - loss: 0.0416 - accuracy: 0.9867
 832/1688 [=============>................] - ETA: 1s - loss: 0.0418 - accuracy: 0.9867
 861/1688 [==============>...............] - ETA: 1s - loss: 0.0426 - accuracy: 0.9866
 890/1688 [==============>...............] - ETA: 1s - loss: 0.0421 - accuracy: 0.9868
 918/1688 [===============>..............] - ETA: 1s - loss: 0.0427 - accuracy: 0.9867
 946/1688 [===============>..............] - ETA: 1s - loss: 0.0424 - accuracy: 0.9867
 974/1688 [================>.............] - ETA: 1s - loss: 0.0421 - accuracy: 0.9868
1002/1688 [================>.............] - ETA: 1s - loss: 0.0417 - accuracy: 0.9869
1030/1688 [=================>............] - ETA: 1s - loss: 0.0414 - accuracy: 0.9870
1058/1688 [=================>............] - ETA: 1s - loss: 0.0411 - accuracy: 0.9871
1087/1688 [==================>...........] - ETA: 1s - loss: 0.0410 - accuracy: 0.9871
1115/1688 [==================>...........] - ETA: 1s - loss: 0.0407 - accuracy: 0.9872
1143/1688 [===================>..........] - ETA: 0s - loss: 0.0405 - accuracy: 0.9872
1172/1688 [===================>..........] - ETA: 0s - loss: 0.0405 - accuracy: 0.9872
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0409 - accuracy: 0.9871
1229/1688 [====================>.........] - ETA: 0s - loss: 0.0412 - accuracy: 0.9869
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0410 - accuracy: 0.9869
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0414 - accuracy: 0.9869
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0416 - accuracy: 0.9868
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0414 - accuracy: 0.9869
1371/1688 [=======================>......] - ETA: 0s - loss: 0.0415 - accuracy: 0.9869
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0417 - accuracy: 0.9867
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0417 - accuracy: 0.9867
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0417 - accuracy: 0.9867
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0416 - accuracy: 0.9867
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0421 - accuracy: 0.9866
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0422 - accuracy: 0.9865
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0420 - accuracy: 0.9866
1598/1688 [===========================>..] - ETA: 0s - loss: 0.0429 - accuracy: 0.9864
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0430 - accuracy: 0.9864
1655/1688 [============================>.] - ETA: 0s - loss: 0.0430 - accuracy: 0.9863
1684/1688 [============================>.] - ETA: 0s - loss: 0.0429 - accuracy: 0.9863
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0428 - accuracy: 0.9863 - val_loss: 0.0564 - val_accuracy: 0.9855
Epoch 5/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0470 - accuracy: 0.9688
  31/1688 [..............................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9940
  60/1688 [>.............................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9906
  88/1688 [>.............................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9908
 116/1688 [=>............................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9908
 143/1688 [=>............................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9910
 172/1688 [==>...........................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9907
 201/1688 [==>...........................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9905
 229/1688 [===>..........................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9907
 258/1688 [===>..........................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9908
 286/1688 [====>.........................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9908
 314/1688 [====>.........................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9908
 342/1688 [=====>........................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9906
 371/1688 [=====>........................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9906
 399/1688 [======>.......................] - ETA: 2s - loss: 0.0273 - accuracy: 0.9908
 427/1688 [======>.......................] - ETA: 2s - loss: 0.0268 - accuracy: 0.9912
 455/1688 [=======>......................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9913
 484/1688 [=======>......................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9915
 513/1688 [========>.....................] - ETA: 2s - loss: 0.0269 - accuracy: 0.9913
 541/1688 [========>.....................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9909
 569/1688 [=========>....................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9910
 598/1688 [=========>....................] - ETA: 1s - loss: 0.0276 - accuracy: 0.9910
 626/1688 [==========>...................] - ETA: 1s - loss: 0.0284 - accuracy: 0.9907
 654/1688 [==========>...................] - ETA: 1s - loss: 0.0287 - accuracy: 0.9907
 682/1688 [===========>..................] - ETA: 1s - loss: 0.0283 - accuracy: 0.9907
 710/1688 [===========>..................] - ETA: 1s - loss: 0.0283 - accuracy: 0.9905
 738/1688 [============>.................] - ETA: 1s - loss: 0.0287 - accuracy: 0.9904
 766/1688 [============>.................] - ETA: 1s - loss: 0.0292 - accuracy: 0.9900
 794/1688 [=============>................] - ETA: 1s - loss: 0.0303 - accuracy: 0.9897
 823/1688 [=============>................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9896
 851/1688 [==============>...............] - ETA: 1s - loss: 0.0313 - accuracy: 0.9895
 879/1688 [==============>...............] - ETA: 1s - loss: 0.0314 - accuracy: 0.9894
 907/1688 [===============>..............] - ETA: 1s - loss: 0.0317 - accuracy: 0.9894
 936/1688 [===============>..............] - ETA: 1s - loss: 0.0313 - accuracy: 0.9894
 964/1688 [================>.............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9895
 993/1688 [================>.............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9896
1021/1688 [=================>............] - ETA: 1s - loss: 0.0309 - accuracy: 0.9897
1048/1688 [=================>............] - ETA: 1s - loss: 0.0306 - accuracy: 0.9899
1076/1688 [==================>...........] - ETA: 1s - loss: 0.0308 - accuracy: 0.9897
1104/1688 [==================>...........] - ETA: 1s - loss: 0.0310 - accuracy: 0.9896
1133/1688 [===================>..........] - ETA: 0s - loss: 0.0311 - accuracy: 0.9896
1162/1688 [===================>..........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9896
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0308 - accuracy: 0.9896
1218/1688 [====================>.........] - ETA: 0s - loss: 0.0306 - accuracy: 0.9896
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0309 - accuracy: 0.9896
1275/1688 [=====================>........] - ETA: 0s - loss: 0.0306 - accuracy: 0.9897
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0312 - accuracy: 0.9895
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0309 - accuracy: 0.9896
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0312 - accuracy: 0.9895
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0315 - accuracy: 0.9895
1416/1688 [========================>.....] - ETA: 0s - loss: 0.0316 - accuracy: 0.9895
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0319 - accuracy: 0.9895
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0322 - accuracy: 0.9894
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0323 - accuracy: 0.9894
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0327 - accuracy: 0.9893
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0329 - accuracy: 0.9892
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0332 - accuracy: 0.9891
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0332 - accuracy: 0.9892
1644/1688 [============================>.] - ETA: 0s - loss: 0.0332 - accuracy: 0.9891
1673/1688 [============================>.] - ETA: 0s - loss: 0.0331 - accuracy: 0.9891
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0332 - accuracy: 0.9891 - val_loss: 0.0600 - val_accuracy: 0.9848
Epoch 6/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0193 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9950
  60/1688 [>.............................] - ETA: 2s - loss: 0.0186 - accuracy: 0.9943
  89/1688 [>.............................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9944
 117/1688 [=>............................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9931
 145/1688 [=>............................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9925
 173/1688 [==>...........................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9924
 202/1688 [==>...........................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9918
 230/1688 [===>..........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9917
 259/1688 [===>..........................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9919
 287/1688 [====>.........................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9921
 315/1688 [====>.........................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9918
 344/1688 [=====>........................] - ETA: 2s - loss: 0.0243 - accuracy: 0.9917
 373/1688 [=====>........................] - ETA: 2s - loss: 0.0243 - accuracy: 0.9916
 402/1688 [======>.......................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9917
 430/1688 [======>.......................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9915
 459/1688 [=======>......................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9918
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9919
 515/1688 [========>.....................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9915
 543/1688 [========>.....................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9918
 570/1688 [=========>....................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9921
 597/1688 [=========>....................] - ETA: 1s - loss: 0.0231 - accuracy: 0.9921
 625/1688 [==========>...................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9922
 654/1688 [==========>...................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9923
 682/1688 [===========>..................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9920
 710/1688 [===========>..................] - ETA: 1s - loss: 0.0245 - accuracy: 0.9918
 738/1688 [============>.................] - ETA: 1s - loss: 0.0246 - accuracy: 0.9918
 767/1688 [============>.................] - ETA: 1s - loss: 0.0245 - accuracy: 0.9917
 795/1688 [=============>................] - ETA: 1s - loss: 0.0246 - accuracy: 0.9917
 823/1688 [=============>................] - ETA: 1s - loss: 0.0249 - accuracy: 0.9917
 851/1688 [==============>...............] - ETA: 1s - loss: 0.0252 - accuracy: 0.9917
 879/1688 [==============>...............] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
 908/1688 [===============>..............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9915
 936/1688 [===============>..............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9915
 964/1688 [================>.............] - ETA: 1s - loss: 0.0257 - accuracy: 0.9914
 992/1688 [================>.............] - ETA: 1s - loss: 0.0257 - accuracy: 0.9914
1021/1688 [=================>............] - ETA: 1s - loss: 0.0258 - accuracy: 0.9914
1050/1688 [=================>............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9915
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0252 - accuracy: 0.9916
1107/1688 [==================>...........] - ETA: 1s - loss: 0.0252 - accuracy: 0.9916
1135/1688 [===================>..........] - ETA: 0s - loss: 0.0254 - accuracy: 0.9915
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0257 - accuracy: 0.9915
1191/1688 [====================>.........] - ETA: 0s - loss: 0.0260 - accuracy: 0.9914
1219/1688 [====================>.........] - ETA: 0s - loss: 0.0261 - accuracy: 0.9913
1247/1688 [=====================>........] - ETA: 0s - loss: 0.0263 - accuracy: 0.9913
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0262 - accuracy: 0.9913
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0266 - accuracy: 0.9911
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0269 - accuracy: 0.9909
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9909
1387/1688 [=======================>......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9909
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0274 - accuracy: 0.9907
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0279 - accuracy: 0.9907
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0277 - accuracy: 0.9908
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0281 - accuracy: 0.9906
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0283 - accuracy: 0.9905
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0293 - accuracy: 0.9902
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0295 - accuracy: 0.9901
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0298 - accuracy: 0.9900
1641/1688 [============================>.] - ETA: 0s - loss: 0.0300 - accuracy: 0.9900
1669/1688 [============================>.] - ETA: 0s - loss: 0.0298 - accuracy: 0.9901
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0298 - accuracy: 0.9901 - val_loss: 0.0498 - val_accuracy: 0.9863
Epoch 7/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0012 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0229 - accuracy: 0.9922
  60/1688 [>.............................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9932
  89/1688 [>.............................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9947
 118/1688 [=>............................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9950
 147/1688 [=>............................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9945
 175/1688 [==>...........................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9943
 204/1688 [==>...........................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9946
 232/1688 [===>..........................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9950
 261/1688 [===>..........................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9952
 289/1688 [====>.........................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9948
 318/1688 [====>.........................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9947
 346/1688 [=====>........................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9947
 374/1688 [=====>........................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9944
 401/1688 [======>.......................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9938
 430/1688 [======>.......................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9936
 459/1688 [=======>......................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9936
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0194 - accuracy: 0.9937
 516/1688 [========>.....................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9934
 543/1688 [========>.....................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9932
 572/1688 [=========>....................] - ETA: 1s - loss: 0.0222 - accuracy: 0.9927
 600/1688 [=========>....................] - ETA: 1s - loss: 0.0232 - accuracy: 0.9923
 629/1688 [==========>...................] - ETA: 1s - loss: 0.0231 - accuracy: 0.9923
 657/1688 [==========>...................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9924
 685/1688 [===========>..................] - ETA: 1s - loss: 0.0234 - accuracy: 0.9923
 714/1688 [===========>..................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9922
 743/1688 [============>.................] - ETA: 1s - loss: 0.0238 - accuracy: 0.9922
 771/1688 [============>.................] - ETA: 1s - loss: 0.0234 - accuracy: 0.9923
 799/1688 [=============>................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9921
 827/1688 [=============>................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9921
 855/1688 [==============>...............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9923
 883/1688 [==============>...............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9924
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0230 - accuracy: 0.9924
 940/1688 [===============>..............] - ETA: 1s - loss: 0.0229 - accuracy: 0.9925
 968/1688 [================>.............] - ETA: 1s - loss: 0.0227 - accuracy: 0.9926
 996/1688 [================>.............] - ETA: 1s - loss: 0.0224 - accuracy: 0.9927
1025/1688 [=================>............] - ETA: 1s - loss: 0.0222 - accuracy: 0.9927
1053/1688 [=================>............] - ETA: 1s - loss: 0.0223 - accuracy: 0.9927
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0224 - accuracy: 0.9926
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0224 - accuracy: 0.9926
1137/1688 [===================>..........] - ETA: 0s - loss: 0.0222 - accuracy: 0.9926
1165/1688 [===================>..........] - ETA: 0s - loss: 0.0224 - accuracy: 0.9925
1193/1688 [====================>.........] - ETA: 0s - loss: 0.0226 - accuracy: 0.9925
1221/1688 [====================>.........] - ETA: 0s - loss: 0.0226 - accuracy: 0.9924
1249/1688 [=====================>........] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1391/1688 [=======================>......] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1419/1688 [========================>.....] - ETA: 0s - loss: 0.0235 - accuracy: 0.9921
1448/1688 [========================>.....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9920
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9920
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9920
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0238 - accuracy: 0.9920
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0240 - accuracy: 0.9919
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0240 - accuracy: 0.9919
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0241 - accuracy: 0.9918
1646/1688 [============================>.] - ETA: 0s - loss: 0.0241 - accuracy: 0.9918
1674/1688 [============================>.] - ETA: 0s - loss: 0.0242 - accuracy: 0.9918
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0243 - accuracy: 0.9918 - val_loss: 0.0545 - val_accuracy: 0.9878
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0141 - accuracy: 1.0000
  29/1688 [..............................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9935
  58/1688 [>.............................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9952
  86/1688 [>.............................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9960
 114/1688 [=>............................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9962
 142/1688 [=>............................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9963
 170/1688 [==>...........................] - ETA: 2s - loss: 0.0175 - accuracy: 0.9960
 198/1688 [==>...........................] - ETA: 2s - loss: 0.0169 - accuracy: 0.9957
 226/1688 [===>..........................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9952
 254/1688 [===>..........................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
 281/1688 [===>..........................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9958
 310/1688 [====>.........................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9957
 339/1688 [=====>........................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9953
 368/1688 [=====>........................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9952
 396/1688 [======>.......................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9952
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9949
 453/1688 [=======>......................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9948
 481/1688 [=======>......................] - ETA: 2s - loss: 0.0168 - accuracy: 0.9949
 509/1688 [========>.....................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9947
 537/1688 [========>.....................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9944
 565/1688 [=========>....................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9942
 593/1688 [=========>....................] - ETA: 1s - loss: 0.0182 - accuracy: 0.9941
 622/1688 [==========>...................] - ETA: 1s - loss: 0.0184 - accuracy: 0.9939
 651/1688 [==========>...................] - ETA: 1s - loss: 0.0186 - accuracy: 0.9939
 680/1688 [===========>..................] - ETA: 1s - loss: 0.0188 - accuracy: 0.9938
 708/1688 [===========>..................] - ETA: 1s - loss: 0.0186 - accuracy: 0.9939
 736/1688 [============>.................] - ETA: 1s - loss: 0.0184 - accuracy: 0.9940
 764/1688 [============>.................] - ETA: 1s - loss: 0.0180 - accuracy: 0.9940
 792/1688 [=============>................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9941
 820/1688 [=============>................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9941
 848/1688 [==============>...............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9941
 876/1688 [==============>...............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9942
 905/1688 [===============>..............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9943
 932/1688 [===============>..............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9942
 961/1688 [================>.............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9941
 990/1688 [================>.............] - ETA: 1s - loss: 0.0180 - accuracy: 0.9940
1019/1688 [=================>............] - ETA: 1s - loss: 0.0182 - accuracy: 0.9940
1048/1688 [=================>............] - ETA: 1s - loss: 0.0183 - accuracy: 0.9939
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0184 - accuracy: 0.9938
1106/1688 [==================>...........] - ETA: 1s - loss: 0.0182 - accuracy: 0.9938
1134/1688 [===================>..........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9937
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9936
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9936
1218/1688 [====================>.........] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0184 - accuracy: 0.9937
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9936
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0185 - accuracy: 0.9936
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0184 - accuracy: 0.9936
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0184 - accuracy: 0.9936
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0185 - accuracy: 0.9935
1416/1688 [========================>.....] - ETA: 0s - loss: 0.0187 - accuracy: 0.9935
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0190 - accuracy: 0.9934
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0192 - accuracy: 0.9933
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0196 - accuracy: 0.9933
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0194 - accuracy: 0.9933
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0195 - accuracy: 0.9933
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0194 - accuracy: 0.9933
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0196 - accuracy: 0.9933
1642/1688 [============================>.] - ETA: 0s - loss: 0.0195 - accuracy: 0.9933
1670/1688 [============================>.] - ETA: 0s - loss: 0.0195 - accuracy: 0.9933
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0195 - accuracy: 0.9933 - val_loss: 0.0601 - val_accuracy: 0.9850
Epoch 9/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0033 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9912
  62/1688 [>.............................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9914
  91/1688 [>.............................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9921
 120/1688 [=>............................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9935
 148/1688 [=>............................] - ETA: 2s - loss: 0.0168 - accuracy: 0.9943
 176/1688 [==>...........................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9949
 204/1688 [==>...........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9953
 232/1688 [===>..........................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9958
 260/1688 [===>..........................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9959
 288/1688 [====>.........................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9959
 317/1688 [====>.........................] - ETA: 2s - loss: 0.0127 - accuracy: 0.9960
 345/1688 [=====>........................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9961
 374/1688 [=====>........................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9959
 402/1688 [======>.......................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9961
 430/1688 [======>.......................] - ETA: 2s - loss: 0.0121 - accuracy: 0.9961
 458/1688 [=======>......................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9960
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0123 - accuracy: 0.9959
 515/1688 [========>.....................] - ETA: 2s - loss: 0.0127 - accuracy: 0.9956
 543/1688 [========>.....................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9955
 571/1688 [=========>....................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
 599/1688 [=========>....................] - ETA: 1s - loss: 0.0138 - accuracy: 0.9952
 627/1688 [==========>...................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9950
 656/1688 [==========>...................] - ETA: 1s - loss: 0.0147 - accuracy: 0.9950
 685/1688 [===========>..................] - ETA: 1s - loss: 0.0152 - accuracy: 0.9947
 713/1688 [===========>..................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9947
 742/1688 [============>.................] - ETA: 1s - loss: 0.0155 - accuracy: 0.9946
 770/1688 [============>.................] - ETA: 1s - loss: 0.0161 - accuracy: 0.9944
 799/1688 [=============>................] - ETA: 1s - loss: 0.0162 - accuracy: 0.9944
 827/1688 [=============>................] - ETA: 1s - loss: 0.0164 - accuracy: 0.9944
 855/1688 [==============>...............] - ETA: 1s - loss: 0.0167 - accuracy: 0.9942
 884/1688 [==============>...............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9943
 914/1688 [===============>..............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9943
 944/1688 [===============>..............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9942
 972/1688 [================>.............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9942
1001/1688 [================>.............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9941
1029/1688 [=================>............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9940
1057/1688 [=================>............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9939
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9939
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0176 - accuracy: 0.9939
1143/1688 [===================>..........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9939
1171/1688 [===================>..........] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1229/1688 [====================>.........] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1315/1688 [======================>.......] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0176 - accuracy: 0.9939
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0176 - accuracy: 0.9940
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0178 - accuracy: 0.9938
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0177 - accuracy: 0.9938
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0175 - accuracy: 0.9939
1568/1688 [==========================>...] - ETA: 0s - loss: 0.0175 - accuracy: 0.9939
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1653/1688 [============================>.] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1681/1688 [============================>.] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0183 - accuracy: 0.9937 - val_loss: 0.0753 - val_accuracy: 0.9833
Epoch 10/10

   1/1688 [..............................] - ETA: 3s - loss: 1.9658e-04 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0093 - accuracy: 0.9958    
  58/1688 [>.............................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9968
  87/1688 [>.............................] - ETA: 2s - loss: 0.0107 - accuracy: 0.9968
 116/1688 [=>............................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9968
 143/1688 [=>............................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9967
 171/1688 [==>...........................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9971
 200/1688 [==>...........................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9970
 228/1688 [===>..........................] - ETA: 2s - loss: 0.0094 - accuracy: 0.9971
 256/1688 [===>..........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9968
 285/1688 [====>.........................] - ETA: 2s - loss: 0.0102 - accuracy: 0.9967
 313/1688 [====>.........................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9964
 341/1688 [=====>........................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9964
 370/1688 [=====>........................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9965
 398/1688 [======>.......................] - ETA: 2s - loss: 0.0104 - accuracy: 0.9966
 426/1688 [======>.......................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9966
 454/1688 [=======>......................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9963
 483/1688 [=======>......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9961
 512/1688 [========>.....................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9960
 541/1688 [========>.....................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9958
 570/1688 [=========>....................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9958
 598/1688 [=========>....................] - ETA: 1s - loss: 0.0125 - accuracy: 0.9957
 626/1688 [==========>...................] - ETA: 1s - loss: 0.0126 - accuracy: 0.9956
 655/1688 [==========>...................] - ETA: 1s - loss: 0.0127 - accuracy: 0.9956
 683/1688 [===========>..................] - ETA: 1s - loss: 0.0125 - accuracy: 0.9957
 711/1688 [===========>..................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
 739/1688 [============>.................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
 768/1688 [============>.................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
 796/1688 [=============>................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9952
 825/1688 [=============>................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9953
 855/1688 [==============>...............] - ETA: 1s - loss: 0.0133 - accuracy: 0.9953
 884/1688 [==============>...............] - ETA: 1s - loss: 0.0130 - accuracy: 0.9954
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0132 - accuracy: 0.9953
 938/1688 [===============>..............] - ETA: 1s - loss: 0.0131 - accuracy: 0.9953
 967/1688 [================>.............] - ETA: 1s - loss: 0.0131 - accuracy: 0.9953
 995/1688 [================>.............] - ETA: 1s - loss: 0.0130 - accuracy: 0.9954
1023/1688 [=================>............] - ETA: 1s - loss: 0.0131 - accuracy: 0.9953
1051/1688 [=================>............] - ETA: 1s - loss: 0.0132 - accuracy: 0.9952
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0133 - accuracy: 0.9951
1107/1688 [==================>...........] - ETA: 1s - loss: 0.0137 - accuracy: 0.9950
1135/1688 [===================>..........] - ETA: 0s - loss: 0.0137 - accuracy: 0.9950
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0138 - accuracy: 0.9949
1192/1688 [====================>.........] - ETA: 0s - loss: 0.0141 - accuracy: 0.9949
1221/1688 [====================>.........] - ETA: 0s - loss: 0.0142 - accuracy: 0.9949
1249/1688 [=====================>........] - ETA: 0s - loss: 0.0141 - accuracy: 0.9949
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0142 - accuracy: 0.9949
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0142 - accuracy: 0.9948
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0142 - accuracy: 0.9948
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0141 - accuracy: 0.9949
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0142 - accuracy: 0.9950
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0143 - accuracy: 0.9950
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0144 - accuracy: 0.9949
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0144 - accuracy: 0.9949
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0146 - accuracy: 0.9948
1649/1688 [============================>.] - ETA: 0s - loss: 0.0149 - accuracy: 0.9947
1678/1688 [============================>.] - ETA: 0s - loss: 0.0152 - accuracy: 0.9946
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0151 - accuracy: 0.9946 - val_loss: 0.0711 - val_accuracy: 0.9838
Test accuracy: 0.9842000007629395

4. Model quantization

We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.

For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).

The quantized model is a Keras model where the neural layers (Conv2D, Dense) and the ReLU layers are replaced with custom CNN2SNN quantized layers (QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function folds the batch normalization layers into the corresponding neural layer. The new weights are computed according to this folding operation.

from cnn2snn import quantize

model_quantized = quantize(model_keras,
                           input_weight_quantization=8,
                           weight_quantization=4,
                           activ_quantization=4)
model_quantized.summary()

Out:

Model: "sequential_48"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 conv2d (QuantizedConv2D)    (None, 13, 13, 32)        320

 re_lu (QuantizedReLU)       (None, 13, 13, 32)        0

 separable_conv2d (Quantized  (None, 7, 7, 64)         2400
 SeparableConv2D)

 re_lu_1 (QuantizedReLU)     (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (QuantizedDense)      (None, 10)                31370

=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________

Check the quantized model accuracy.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])

Out:

Test accuracy after 8-4-4 quantization: 0.8953999876976013

Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.

Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.

Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.

from cnn2snn import quantize_layer

model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])

# To recover the original model accuracy, a quantization-aware training phase
# is required.

Out:

Test accuracy after low bitwidth quantization: 0.11349999904632568

5. Model fine tuning (quantization-aware training)

This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.

Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])

Out:

Epoch 1/5

   1/1688 [..............................] - ETA: 17:11 - loss: 2.3145 - accuracy: 0.0625
  21/1688 [..............................] - ETA: 4s - loss: 2.3020 - accuracy: 0.0893   
  41/1688 [..............................] - ETA: 4s - loss: 2.2981 - accuracy: 0.0983
  61/1688 [>.............................] - ETA: 4s - loss: 2.3024 - accuracy: 0.1071
  81/1688 [>.............................] - ETA: 4s - loss: 2.3132 - accuracy: 0.1100
 101/1688 [>.............................] - ETA: 4s - loss: 2.3191 - accuracy: 0.1281
 122/1688 [=>............................] - ETA: 4s - loss: 2.2941 - accuracy: 0.1601
 142/1688 [=>............................] - ETA: 3s - loss: 2.2519 - accuracy: 0.2168
 162/1688 [=>............................] - ETA: 3s - loss: 2.2677 - accuracy: 0.2731
 181/1688 [==>...........................] - ETA: 3s - loss: 2.2224 - accuracy: 0.3225
 201/1688 [==>...........................] - ETA: 3s - loss: 2.1318 - accuracy: 0.3752
 221/1688 [==>...........................] - ETA: 3s - loss: 2.0567 - accuracy: 0.4164
 241/1688 [===>..........................] - ETA: 3s - loss: 1.9687 - accuracy: 0.4557
 261/1688 [===>..........................] - ETA: 3s - loss: 1.8924 - accuracy: 0.4892
 281/1688 [===>..........................] - ETA: 3s - loss: 1.8334 - accuracy: 0.5180
 302/1688 [====>.........................] - ETA: 3s - loss: 1.7959 - accuracy: 0.5435
 322/1688 [====>.........................] - ETA: 3s - loss: 1.7547 - accuracy: 0.5656
 342/1688 [=====>........................] - ETA: 3s - loss: 1.7165 - accuracy: 0.5850
 362/1688 [=====>........................] - ETA: 3s - loss: 1.7029 - accuracy: 0.6024
 382/1688 [=====>........................] - ETA: 3s - loss: 1.6739 - accuracy: 0.6180
 402/1688 [======>.......................] - ETA: 3s - loss: 1.6314 - accuracy: 0.6328
 422/1688 [======>.......................] - ETA: 3s - loss: 1.5961 - accuracy: 0.6467
 442/1688 [======>.......................] - ETA: 3s - loss: 1.5635 - accuracy: 0.6589
 462/1688 [=======>......................] - ETA: 3s - loss: 1.5372 - accuracy: 0.6707
 482/1688 [=======>......................] - ETA: 3s - loss: 1.5189 - accuracy: 0.6809
 502/1688 [=======>......................] - ETA: 3s - loss: 1.4906 - accuracy: 0.6911
 523/1688 [========>.....................] - ETA: 2s - loss: 1.4735 - accuracy: 0.7003
 544/1688 [========>.....................] - ETA: 2s - loss: 1.4433 - accuracy: 0.7092
 564/1688 [=========>....................] - ETA: 2s - loss: 1.4293 - accuracy: 0.7173
 584/1688 [=========>....................] - ETA: 2s - loss: 1.4115 - accuracy: 0.7247
 604/1688 [=========>....................] - ETA: 2s - loss: 1.3960 - accuracy: 0.7320
 624/1688 [==========>...................] - ETA: 2s - loss: 1.3868 - accuracy: 0.7381
 644/1688 [==========>...................] - ETA: 2s - loss: 1.3730 - accuracy: 0.7443
 664/1688 [==========>...................] - ETA: 2s - loss: 1.3562 - accuracy: 0.7500
 684/1688 [===========>..................] - ETA: 2s - loss: 1.3373 - accuracy: 0.7555
 704/1688 [===========>..................] - ETA: 2s - loss: 1.3154 - accuracy: 0.7610
 724/1688 [===========>..................] - ETA: 2s - loss: 1.3121 - accuracy: 0.7656
 744/1688 [============>.................] - ETA: 2s - loss: 1.3090 - accuracy: 0.7702
 764/1688 [============>.................] - ETA: 2s - loss: 1.3034 - accuracy: 0.7744
 784/1688 [============>.................] - ETA: 2s - loss: 1.2915 - accuracy: 0.7787
 804/1688 [=============>................] - ETA: 2s - loss: 1.2876 - accuracy: 0.7825
 825/1688 [=============>................] - ETA: 2s - loss: 1.2719 - accuracy: 0.7868
 845/1688 [==============>...............] - ETA: 2s - loss: 1.2632 - accuracy: 0.7902
 865/1688 [==============>...............] - ETA: 2s - loss: 1.2580 - accuracy: 0.7934
 885/1688 [==============>...............] - ETA: 2s - loss: 1.2475 - accuracy: 0.7966
 906/1688 [===============>..............] - ETA: 1s - loss: 1.2366 - accuracy: 0.8001
 926/1688 [===============>..............] - ETA: 1s - loss: 1.2336 - accuracy: 0.8032
 946/1688 [===============>..............] - ETA: 1s - loss: 1.2305 - accuracy: 0.8060
 967/1688 [================>.............] - ETA: 1s - loss: 1.2250 - accuracy: 0.8088
 987/1688 [================>.............] - ETA: 1s - loss: 1.2157 - accuracy: 0.8113
1007/1688 [================>.............] - ETA: 1s - loss: 1.2075 - accuracy: 0.8138
1027/1688 [=================>............] - ETA: 1s - loss: 1.2021 - accuracy: 0.8164
1047/1688 [=================>............] - ETA: 1s - loss: 1.1930 - accuracy: 0.8189
1067/1688 [=================>............] - ETA: 1s - loss: 1.1841 - accuracy: 0.8214
1087/1688 [==================>...........] - ETA: 1s - loss: 1.1747 - accuracy: 0.8238
1107/1688 [==================>...........] - ETA: 1s - loss: 1.1688 - accuracy: 0.8261
1127/1688 [===================>..........] - ETA: 1s - loss: 1.1581 - accuracy: 0.8284
1147/1688 [===================>..........] - ETA: 1s - loss: 1.1557 - accuracy: 0.8305
1168/1688 [===================>..........] - ETA: 1s - loss: 1.1525 - accuracy: 0.8326
1188/1688 [====================>.........] - ETA: 1s - loss: 1.1479 - accuracy: 0.8346
1208/1688 [====================>.........] - ETA: 1s - loss: 1.1409 - accuracy: 0.8366
1228/1688 [====================>.........] - ETA: 1s - loss: 1.1346 - accuracy: 0.8384
1248/1688 [=====================>........] - ETA: 1s - loss: 1.1243 - accuracy: 0.8404
1268/1688 [=====================>........] - ETA: 1s - loss: 1.1208 - accuracy: 0.8423
1288/1688 [=====================>........] - ETA: 1s - loss: 1.1216 - accuracy: 0.8439
1308/1688 [======================>.......] - ETA: 0s - loss: 1.1266 - accuracy: 0.8453
1328/1688 [======================>.......] - ETA: 0s - loss: 1.1259 - accuracy: 0.8466
1348/1688 [======================>.......] - ETA: 0s - loss: 1.1257 - accuracy: 0.8479
1368/1688 [=======================>......] - ETA: 0s - loss: 1.1213 - accuracy: 0.8493
1388/1688 [=======================>......] - ETA: 0s - loss: 1.1161 - accuracy: 0.8507
1408/1688 [========================>.....] - ETA: 0s - loss: 1.1116 - accuracy: 0.8521
1428/1688 [========================>.....] - ETA: 0s - loss: 1.1113 - accuracy: 0.8532
1448/1688 [========================>.....] - ETA: 0s - loss: 1.1020 - accuracy: 0.8546
1468/1688 [=========================>....] - ETA: 0s - loss: 1.1043 - accuracy: 0.8556
1488/1688 [=========================>....] - ETA: 0s - loss: 1.1047 - accuracy: 0.8566
1508/1688 [=========================>....] - ETA: 0s - loss: 1.0998 - accuracy: 0.8580
1528/1688 [==========================>...] - ETA: 0s - loss: 1.0984 - accuracy: 0.8591
1548/1688 [==========================>...] - ETA: 0s - loss: 1.0942 - accuracy: 0.8604
1569/1688 [==========================>...] - ETA: 0s - loss: 1.0947 - accuracy: 0.8615
1589/1688 [===========================>..] - ETA: 0s - loss: 1.0924 - accuracy: 0.8625
1609/1688 [===========================>..] - ETA: 0s - loss: 1.0858 - accuracy: 0.8636
1630/1688 [===========================>..] - ETA: 0s - loss: 1.0832 - accuracy: 0.8647
1650/1688 [============================>.] - ETA: 0s - loss: 1.0830 - accuracy: 0.8657
1670/1688 [============================>.] - ETA: 0s - loss: 1.0794 - accuracy: 0.8668
1688/1688 [==============================] - 5s 3ms/step - loss: 1.0765 - accuracy: 0.8678 - val_loss: 0.8484 - val_accuracy: 0.9598
Epoch 2/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0000e+00 - accuracy: 1.0000
  22/1688 [..............................] - ETA: 4s - loss: 0.7210 - accuracy: 0.9588    
  42/1688 [..............................] - ETA: 4s - loss: 0.8106 - accuracy: 0.9583
  62/1688 [>.............................] - ETA: 4s - loss: 0.6682 - accuracy: 0.9627
  82/1688 [>.............................] - ETA: 4s - loss: 0.7188 - accuracy: 0.9615
 102/1688 [>.............................] - ETA: 4s - loss: 0.7400 - accuracy: 0.9599
 123/1688 [=>............................] - ETA: 3s - loss: 0.7327 - accuracy: 0.9596
 143/1688 [=>............................] - ETA: 3s - loss: 0.7188 - accuracy: 0.9583
 163/1688 [=>............................] - ETA: 3s - loss: 0.7522 - accuracy: 0.9576
 183/1688 [==>...........................] - ETA: 3s - loss: 0.7663 - accuracy: 0.9570
 203/1688 [==>...........................] - ETA: 3s - loss: 0.7656 - accuracy: 0.9577
 223/1688 [==>...........................] - ETA: 3s - loss: 0.7771 - accuracy: 0.9577
 243/1688 [===>..........................] - ETA: 3s - loss: 0.7791 - accuracy: 0.9573
 263/1688 [===>..........................] - ETA: 3s - loss: 0.7750 - accuracy: 0.9573
 283/1688 [====>.........................] - ETA: 3s - loss: 0.7925 - accuracy: 0.9563
 303/1688 [====>.........................] - ETA: 3s - loss: 0.7715 - accuracy: 0.9570
 323/1688 [====>.........................] - ETA: 3s - loss: 0.7725 - accuracy: 0.9576
 343/1688 [=====>........................] - ETA: 3s - loss: 0.7645 - accuracy: 0.9583
 363/1688 [=====>........................] - ETA: 3s - loss: 0.7737 - accuracy: 0.9576
 383/1688 [=====>........................] - ETA: 3s - loss: 0.7691 - accuracy: 0.9577
 404/1688 [======>.......................] - ETA: 3s - loss: 0.7777 - accuracy: 0.9570
 424/1688 [======>.......................] - ETA: 3s - loss: 0.7673 - accuracy: 0.9575
 444/1688 [======>.......................] - ETA: 3s - loss: 0.7614 - accuracy: 0.9578
 464/1688 [=======>......................] - ETA: 3s - loss: 0.7652 - accuracy: 0.9580
 484/1688 [=======>......................] - ETA: 3s - loss: 0.7495 - accuracy: 0.9585
 504/1688 [=======>......................] - ETA: 3s - loss: 0.7369 - accuracy: 0.9586
 524/1688 [========>.....................] - ETA: 2s - loss: 0.7409 - accuracy: 0.9585
 545/1688 [========>.....................] - ETA: 2s - loss: 0.7471 - accuracy: 0.9582
 565/1688 [=========>....................] - ETA: 2s - loss: 0.7492 - accuracy: 0.9582
 586/1688 [=========>....................] - ETA: 2s - loss: 0.7390 - accuracy: 0.9587
 606/1688 [=========>....................] - ETA: 2s - loss: 0.7392 - accuracy: 0.9587
 626/1688 [==========>...................] - ETA: 2s - loss: 0.7301 - accuracy: 0.9588
 646/1688 [==========>...................] - ETA: 2s - loss: 0.7410 - accuracy: 0.9588
 666/1688 [==========>...................] - ETA: 2s - loss: 0.7344 - accuracy: 0.9587
 687/1688 [===========>..................] - ETA: 2s - loss: 0.7415 - accuracy: 0.9582
 707/1688 [===========>..................] - ETA: 2s - loss: 0.7463 - accuracy: 0.9583
 727/1688 [===========>..................] - ETA: 2s - loss: 0.7462 - accuracy: 0.9583
 748/1688 [============>.................] - ETA: 2s - loss: 0.7415 - accuracy: 0.9584
 768/1688 [============>.................] - ETA: 2s - loss: 0.7448 - accuracy: 0.9585
 788/1688 [=============>................] - ETA: 2s - loss: 0.7505 - accuracy: 0.9582
 808/1688 [=============>................] - ETA: 2s - loss: 0.7479 - accuracy: 0.9582
 828/1688 [=============>................] - ETA: 2s - loss: 0.7507 - accuracy: 0.9582
 848/1688 [==============>...............] - ETA: 2s - loss: 0.7478 - accuracy: 0.9584
 868/1688 [==============>...............] - ETA: 2s - loss: 0.7410 - accuracy: 0.9588
 888/1688 [==============>...............] - ETA: 2s - loss: 0.7427 - accuracy: 0.9588
 908/1688 [===============>..............] - ETA: 1s - loss: 0.7432 - accuracy: 0.9589
 928/1688 [===============>..............] - ETA: 1s - loss: 0.7418 - accuracy: 0.9589
 949/1688 [===============>..............] - ETA: 1s - loss: 0.7402 - accuracy: 0.9590
 969/1688 [================>.............] - ETA: 1s - loss: 0.7371 - accuracy: 0.9592
 989/1688 [================>.............] - ETA: 1s - loss: 0.7388 - accuracy: 0.9592
1009/1688 [================>.............] - ETA: 1s - loss: 0.7371 - accuracy: 0.9591
1029/1688 [=================>............] - ETA: 1s - loss: 0.7390 - accuracy: 0.9592
1049/1688 [=================>............] - ETA: 1s - loss: 0.7354 - accuracy: 0.9595
1069/1688 [=================>............] - ETA: 1s - loss: 0.7329 - accuracy: 0.9596
1089/1688 [==================>...........] - ETA: 1s - loss: 0.7297 - accuracy: 0.9598
1109/1688 [==================>...........] - ETA: 1s - loss: 0.7314 - accuracy: 0.9599
1129/1688 [===================>..........] - ETA: 1s - loss: 0.7303 - accuracy: 0.9599
1149/1688 [===================>..........] - ETA: 1s - loss: 0.7285 - accuracy: 0.9599
1169/1688 [===================>..........] - ETA: 1s - loss: 0.7220 - accuracy: 0.9602
1189/1688 [====================>.........] - ETA: 1s - loss: 0.7174 - accuracy: 0.9603
1209/1688 [====================>.........] - ETA: 1s - loss: 0.7144 - accuracy: 0.9603
1230/1688 [====================>.........] - ETA: 1s - loss: 0.7150 - accuracy: 0.9602
1250/1688 [=====================>........] - ETA: 1s - loss: 0.7074 - accuracy: 0.9604
1270/1688 [=====================>........] - ETA: 1s - loss: 0.7127 - accuracy: 0.9603
1290/1688 [=====================>........] - ETA: 1s - loss: 0.7091 - accuracy: 0.9605
1310/1688 [======================>.......] - ETA: 0s - loss: 0.7057 - accuracy: 0.9606
1330/1688 [======================>.......] - ETA: 0s - loss: 0.7055 - accuracy: 0.9606
1350/1688 [======================>.......] - ETA: 0s - loss: 0.7041 - accuracy: 0.9607
1371/1688 [=======================>......] - ETA: 0s - loss: 0.7011 - accuracy: 0.9609
1391/1688 [=======================>......] - ETA: 0s - loss: 0.7076 - accuracy: 0.9608
1411/1688 [========================>.....] - ETA: 0s - loss: 0.7098 - accuracy: 0.9606
1431/1688 [========================>.....] - ETA: 0s - loss: 0.7112 - accuracy: 0.9607
1451/1688 [========================>.....] - ETA: 0s - loss: 0.7084 - accuracy: 0.9607
1471/1688 [=========================>....] - ETA: 0s - loss: 0.7063 - accuracy: 0.9608
1491/1688 [=========================>....] - ETA: 0s - loss: 0.7080 - accuracy: 0.9609
1511/1688 [=========================>....] - ETA: 0s - loss: 0.7091 - accuracy: 0.9610
1531/1688 [==========================>...] - ETA: 0s - loss: 0.7078 - accuracy: 0.9611
1551/1688 [==========================>...] - ETA: 0s - loss: 0.7059 - accuracy: 0.9611
1572/1688 [==========================>...] - ETA: 0s - loss: 0.7064 - accuracy: 0.9611
1592/1688 [===========================>..] - ETA: 0s - loss: 0.7072 - accuracy: 0.9611
1612/1688 [===========================>..] - ETA: 0s - loss: 0.7077 - accuracy: 0.9610
1632/1688 [============================>.] - ETA: 0s - loss: 0.7058 - accuracy: 0.9611
1652/1688 [============================>.] - ETA: 0s - loss: 0.7018 - accuracy: 0.9613
1673/1688 [============================>.] - ETA: 0s - loss: 0.7016 - accuracy: 0.9613
1688/1688 [==============================] - 5s 3ms/step - loss: 0.7009 - accuracy: 0.9613 - val_loss: 0.7160 - val_accuracy: 0.9697
Epoch 3/5

   1/1688 [..............................] - ETA: 4s - loss: 0.1883 - accuracy: 0.9688
  22/1688 [..............................] - ETA: 4s - loss: 0.6565 - accuracy: 0.9688
  42/1688 [..............................] - ETA: 4s - loss: 0.5602 - accuracy: 0.9643
  62/1688 [>.............................] - ETA: 4s - loss: 0.5310 - accuracy: 0.9672
  82/1688 [>.............................] - ETA: 4s - loss: 0.5457 - accuracy: 0.9672
 102/1688 [>.............................] - ETA: 4s - loss: 0.5406 - accuracy: 0.9688
 122/1688 [=>............................] - ETA: 3s - loss: 0.5385 - accuracy: 0.9685
 142/1688 [=>............................] - ETA: 3s - loss: 0.5114 - accuracy: 0.9690
 162/1688 [=>............................] - ETA: 3s - loss: 0.5002 - accuracy: 0.9697
 182/1688 [==>...........................] - ETA: 3s - loss: 0.5502 - accuracy: 0.9688
 202/1688 [==>...........................] - ETA: 3s - loss: 0.5488 - accuracy: 0.9678
 223/1688 [==>...........................] - ETA: 3s - loss: 0.5342 - accuracy: 0.9678
 243/1688 [===>..........................] - ETA: 3s - loss: 0.5491 - accuracy: 0.9672
 263/1688 [===>..........................] - ETA: 3s - loss: 0.5358 - accuracy: 0.9677
 283/1688 [====>.........................] - ETA: 3s - loss: 0.5583 - accuracy: 0.9671
 303/1688 [====>.........................] - ETA: 3s - loss: 0.5584 - accuracy: 0.9667
 323/1688 [====>.........................] - ETA: 3s - loss: 0.5719 - accuracy: 0.9665
 342/1688 [=====>........................] - ETA: 3s - loss: 0.5684 - accuracy: 0.9666
 362/1688 [=====>........................] - ETA: 3s - loss: 0.5695 - accuracy: 0.9672
 382/1688 [=====>........................] - ETA: 3s - loss: 0.5527 - accuracy: 0.9678
 402/1688 [======>.......................] - ETA: 3s - loss: 0.5531 - accuracy: 0.9681
 422/1688 [======>.......................] - ETA: 3s - loss: 0.5607 - accuracy: 0.9681
 442/1688 [======>.......................] - ETA: 3s - loss: 0.5655 - accuracy: 0.9679
 462/1688 [=======>......................] - ETA: 3s - loss: 0.5697 - accuracy: 0.9680
 483/1688 [=======>......................] - ETA: 3s - loss: 0.5859 - accuracy: 0.9674
 503/1688 [=======>......................] - ETA: 3s - loss: 0.5946 - accuracy: 0.9676
 523/1688 [========>.....................] - ETA: 2s - loss: 0.5937 - accuracy: 0.9679
 543/1688 [========>.....................] - ETA: 2s - loss: 0.5862 - accuracy: 0.9682
 563/1688 [=========>....................] - ETA: 2s - loss: 0.5923 - accuracy: 0.9684
 584/1688 [=========>....................] - ETA: 2s - loss: 0.6046 - accuracy: 0.9681
 604/1688 [=========>....................] - ETA: 2s - loss: 0.6073 - accuracy: 0.9679
 624/1688 [==========>...................] - ETA: 2s - loss: 0.6082 - accuracy: 0.9679
 644/1688 [==========>...................] - ETA: 2s - loss: 0.6189 - accuracy: 0.9676
 664/1688 [==========>...................] - ETA: 2s - loss: 0.6172 - accuracy: 0.9676
 684/1688 [===========>..................] - ETA: 2s - loss: 0.6253 - accuracy: 0.9673
 704/1688 [===========>..................] - ETA: 2s - loss: 0.6308 - accuracy: 0.9674
 724/1688 [===========>..................] - ETA: 2s - loss: 0.6236 - accuracy: 0.9676
 744/1688 [============>.................] - ETA: 2s - loss: 0.6237 - accuracy: 0.9675
 765/1688 [============>.................] - ETA: 2s - loss: 0.6229 - accuracy: 0.9676
 785/1688 [============>.................] - ETA: 2s - loss: 0.6275 - accuracy: 0.9677
 805/1688 [=============>................] - ETA: 2s - loss: 0.6241 - accuracy: 0.9679
 825/1688 [=============>................] - ETA: 2s - loss: 0.6205 - accuracy: 0.9681
 845/1688 [==============>...............] - ETA: 2s - loss: 0.6210 - accuracy: 0.9681
 865/1688 [==============>...............] - ETA: 2s - loss: 0.6233 - accuracy: 0.9682
 885/1688 [==============>...............] - ETA: 2s - loss: 0.6181 - accuracy: 0.9684
 905/1688 [===============>..............] - ETA: 1s - loss: 0.6216 - accuracy: 0.9682
 925/1688 [===============>..............] - ETA: 1s - loss: 0.6211 - accuracy: 0.9682
 945/1688 [===============>..............] - ETA: 1s - loss: 0.6159 - accuracy: 0.9684
 965/1688 [================>.............] - ETA: 1s - loss: 0.6169 - accuracy: 0.9683
 985/1688 [================>.............] - ETA: 1s - loss: 0.6179 - accuracy: 0.9683
1005/1688 [================>.............] - ETA: 1s - loss: 0.6253 - accuracy: 0.9680
1025/1688 [=================>............] - ETA: 1s - loss: 0.6314 - accuracy: 0.9678
1045/1688 [=================>............] - ETA: 1s - loss: 0.6342 - accuracy: 0.9678
1065/1688 [=================>............] - ETA: 1s - loss: 0.6315 - accuracy: 0.9680
1085/1688 [==================>...........] - ETA: 1s - loss: 0.6302 - accuracy: 0.9680
1105/1688 [==================>...........] - ETA: 1s - loss: 0.6385 - accuracy: 0.9679
1125/1688 [==================>...........] - ETA: 1s - loss: 0.6459 - accuracy: 0.9677
1146/1688 [===================>..........] - ETA: 1s - loss: 0.6430 - accuracy: 0.9679
1166/1688 [===================>..........] - ETA: 1s - loss: 0.6495 - accuracy: 0.9677
1186/1688 [====================>.........] - ETA: 1s - loss: 0.6494 - accuracy: 0.9676
1206/1688 [====================>.........] - ETA: 1s - loss: 0.6465 - accuracy: 0.9676
1226/1688 [====================>.........] - ETA: 1s - loss: 0.6417 - accuracy: 0.9677
1246/1688 [=====================>........] - ETA: 1s - loss: 0.6439 - accuracy: 0.9676
1266/1688 [=====================>........] - ETA: 1s - loss: 0.6480 - accuracy: 0.9674
1286/1688 [=====================>........] - ETA: 1s - loss: 0.6442 - accuracy: 0.9675
1306/1688 [======================>.......] - ETA: 0s - loss: 0.6476 - accuracy: 0.9675
1326/1688 [======================>.......] - ETA: 0s - loss: 0.6460 - accuracy: 0.9675
1346/1688 [======================>.......] - ETA: 0s - loss: 0.6489 - accuracy: 0.9674
1366/1688 [=======================>......] - ETA: 0s - loss: 0.6493 - accuracy: 0.9675
1386/1688 [=======================>......] - ETA: 0s - loss: 0.6466 - accuracy: 0.9676
1406/1688 [=======================>......] - ETA: 0s - loss: 0.6501 - accuracy: 0.9675
1426/1688 [========================>.....] - ETA: 0s - loss: 0.6493 - accuracy: 0.9674
1446/1688 [========================>.....] - ETA: 0s - loss: 0.6502 - accuracy: 0.9673
1466/1688 [=========================>....] - ETA: 0s - loss: 0.6509 - accuracy: 0.9672
1486/1688 [=========================>....] - ETA: 0s - loss: 0.6528 - accuracy: 0.9670
1506/1688 [=========================>....] - ETA: 0s - loss: 0.6503 - accuracy: 0.9672
1526/1688 [==========================>...] - ETA: 0s - loss: 0.6504 - accuracy: 0.9672
1546/1688 [==========================>...] - ETA: 0s - loss: 0.6509 - accuracy: 0.9672
1566/1688 [==========================>...] - ETA: 0s - loss: 0.6546 - accuracy: 0.9670
1586/1688 [===========================>..] - ETA: 0s - loss: 0.6569 - accuracy: 0.9669
1606/1688 [===========================>..] - ETA: 0s - loss: 0.6601 - accuracy: 0.9668
1626/1688 [===========================>..] - ETA: 0s - loss: 0.6647 - accuracy: 0.9666
1646/1688 [============================>.] - ETA: 0s - loss: 0.6631 - accuracy: 0.9667
1666/1688 [============================>.] - ETA: 0s - loss: 0.6586 - accuracy: 0.9668
1686/1688 [============================>.] - ETA: 0s - loss: 0.6577 - accuracy: 0.9669
1688/1688 [==============================] - 5s 3ms/step - loss: 0.6580 - accuracy: 0.9669 - val_loss: 0.7273 - val_accuracy: 0.9708
Epoch 4/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0000e+00 - accuracy: 1.0000
  21/1688 [..............................] - ETA: 4s - loss: 0.7101 - accuracy: 0.9702    
  41/1688 [..............................] - ETA: 4s - loss: 0.7737 - accuracy: 0.9665
  61/1688 [>.............................] - ETA: 4s - loss: 0.6542 - accuracy: 0.9677
  82/1688 [>.............................] - ETA: 4s - loss: 0.6235 - accuracy: 0.9680
 102/1688 [>.............................] - ETA: 4s - loss: 0.5701 - accuracy: 0.9697
 122/1688 [=>............................] - ETA: 3s - loss: 0.5706 - accuracy: 0.9695
 142/1688 [=>............................] - ETA: 3s - loss: 0.5961 - accuracy: 0.9692
 162/1688 [=>............................] - ETA: 3s - loss: 0.5793 - accuracy: 0.9695
 182/1688 [==>...........................] - ETA: 3s - loss: 0.6050 - accuracy: 0.9691
 202/1688 [==>...........................] - ETA: 3s - loss: 0.6062 - accuracy: 0.9689
 222/1688 [==>...........................] - ETA: 3s - loss: 0.6109 - accuracy: 0.9688
 241/1688 [===>..........................] - ETA: 3s - loss: 0.6041 - accuracy: 0.9690
 261/1688 [===>..........................] - ETA: 3s - loss: 0.5810 - accuracy: 0.9697
 281/1688 [===>..........................] - ETA: 3s - loss: 0.5797 - accuracy: 0.9694
 301/1688 [====>.........................] - ETA: 3s - loss: 0.5510 - accuracy: 0.9703
 321/1688 [====>.........................] - ETA: 3s - loss: 0.5677 - accuracy: 0.9702
 341/1688 [=====>........................] - ETA: 3s - loss: 0.5712 - accuracy: 0.9706
 361/1688 [=====>........................] - ETA: 3s - loss: 0.5571 - accuracy: 0.9713
 381/1688 [=====>........................] - ETA: 3s - loss: 0.5470 - accuracy: 0.9719
 401/1688 [======>.......................] - ETA: 3s - loss: 0.5459 - accuracy: 0.9721
 421/1688 [======>.......................] - ETA: 3s - loss: 0.5494 - accuracy: 0.9720
 441/1688 [======>.......................] - ETA: 3s - loss: 0.5408 - accuracy: 0.9719
 461/1688 [=======>......................] - ETA: 3s - loss: 0.5455 - accuracy: 0.9717
 481/1688 [=======>......................] - ETA: 3s - loss: 0.5447 - accuracy: 0.9712
 501/1688 [=======>......................] - ETA: 3s - loss: 0.5340 - accuracy: 0.9715
 521/1688 [========>.....................] - ETA: 2s - loss: 0.5364 - accuracy: 0.9713
 541/1688 [========>.....................] - ETA: 2s - loss: 0.5374 - accuracy: 0.9712
 561/1688 [========>.....................] - ETA: 2s - loss: 0.5348 - accuracy: 0.9716
 581/1688 [=========>....................] - ETA: 2s - loss: 0.5349 - accuracy: 0.9717
 601/1688 [=========>....................] - ETA: 2s - loss: 0.5400 - accuracy: 0.9716
 621/1688 [==========>...................] - ETA: 2s - loss: 0.5517 - accuracy: 0.9711
 641/1688 [==========>...................] - ETA: 2s - loss: 0.5569 - accuracy: 0.9709
 661/1688 [==========>...................] - ETA: 2s - loss: 0.5580 - accuracy: 0.9707
 681/1688 [===========>..................] - ETA: 2s - loss: 0.5579 - accuracy: 0.9709
 702/1688 [===========>..................] - ETA: 2s - loss: 0.5561 - accuracy: 0.9708
 722/1688 [===========>..................] - ETA: 2s - loss: 0.5559 - accuracy: 0.9708
 742/1688 [============>.................] - ETA: 2s - loss: 0.5497 - accuracy: 0.9711
 762/1688 [============>.................] - ETA: 2s - loss: 0.5616 - accuracy: 0.9708
 782/1688 [============>.................] - ETA: 2s - loss: 0.5651 - accuracy: 0.9707
 803/1688 [=============>................] - ETA: 2s - loss: 0.5705 - accuracy: 0.9707
 823/1688 [=============>................] - ETA: 2s - loss: 0.5713 - accuracy: 0.9706
 843/1688 [=============>................] - ETA: 2s - loss: 0.5787 - accuracy: 0.9701
 863/1688 [==============>...............] - ETA: 2s - loss: 0.5855 - accuracy: 0.9701
 883/1688 [==============>...............] - ETA: 2s - loss: 0.5849 - accuracy: 0.9700
 903/1688 [===============>..............] - ETA: 1s - loss: 0.5806 - accuracy: 0.9702
 923/1688 [===============>..............] - ETA: 1s - loss: 0.5867 - accuracy: 0.9700
 943/1688 [===============>..............] - ETA: 1s - loss: 0.5951 - accuracy: 0.9699
 963/1688 [================>.............] - ETA: 1s - loss: 0.5948 - accuracy: 0.9699
 983/1688 [================>.............] - ETA: 1s - loss: 0.5947 - accuracy: 0.9700
1003/1688 [================>.............] - ETA: 1s - loss: 0.5970 - accuracy: 0.9701
1022/1688 [=================>............] - ETA: 1s - loss: 0.5953 - accuracy: 0.9701
1042/1688 [=================>............] - ETA: 1s - loss: 0.6017 - accuracy: 0.9698
1062/1688 [=================>............] - ETA: 1s - loss: 0.5992 - accuracy: 0.9701
1083/1688 [==================>...........] - ETA: 1s - loss: 0.5968 - accuracy: 0.9702
1103/1688 [==================>...........] - ETA: 1s - loss: 0.5974 - accuracy: 0.9702
1123/1688 [==================>...........] - ETA: 1s - loss: 0.5977 - accuracy: 0.9702
1143/1688 [===================>..........] - ETA: 1s - loss: 0.6052 - accuracy: 0.9699
1163/1688 [===================>..........] - ETA: 1s - loss: 0.6051 - accuracy: 0.9699
1183/1688 [====================>.........] - ETA: 1s - loss: 0.6107 - accuracy: 0.9697
1203/1688 [====================>.........] - ETA: 1s - loss: 0.6073 - accuracy: 0.9697
1223/1688 [====================>.........] - ETA: 1s - loss: 0.6120 - accuracy: 0.9696
1243/1688 [=====================>........] - ETA: 1s - loss: 0.6125 - accuracy: 0.9696
1263/1688 [=====================>........] - ETA: 1s - loss: 0.6135 - accuracy: 0.9695
1283/1688 [=====================>........] - ETA: 1s - loss: 0.6175 - accuracy: 0.9694
1303/1688 [======================>.......] - ETA: 0s - loss: 0.6144 - accuracy: 0.9695
1323/1688 [======================>.......] - ETA: 0s - loss: 0.6148 - accuracy: 0.9696
1343/1688 [======================>.......] - ETA: 0s - loss: 0.6157 - accuracy: 0.9695
1363/1688 [=======================>......] - ETA: 0s - loss: 0.6201 - accuracy: 0.9693
1383/1688 [=======================>......] - ETA: 0s - loss: 0.6193 - accuracy: 0.9694
1403/1688 [=======================>......] - ETA: 0s - loss: 0.6170 - accuracy: 0.9694
1423/1688 [========================>.....] - ETA: 0s - loss: 0.6144 - accuracy: 0.9693
1443/1688 [========================>.....] - ETA: 0s - loss: 0.6148 - accuracy: 0.9692
1463/1688 [=========================>....] - ETA: 0s - loss: 0.6142 - accuracy: 0.9693
1484/1688 [=========================>....] - ETA: 0s - loss: 0.6156 - accuracy: 0.9693
1504/1688 [=========================>....] - ETA: 0s - loss: 0.6158 - accuracy: 0.9693
1524/1688 [==========================>...] - ETA: 0s - loss: 0.6208 - accuracy: 0.9692
1544/1688 [==========================>...] - ETA: 0s - loss: 0.6213 - accuracy: 0.9692
1564/1688 [==========================>...] - ETA: 0s - loss: 0.6219 - accuracy: 0.9691
1585/1688 [===========================>..] - ETA: 0s - loss: 0.6211 - accuracy: 0.9691
1605/1688 [===========================>..] - ETA: 0s - loss: 0.6211 - accuracy: 0.9691
1625/1688 [===========================>..] - ETA: 0s - loss: 0.6242 - accuracy: 0.9690
1645/1688 [============================>.] - ETA: 0s - loss: 0.6311 - accuracy: 0.9688
1665/1688 [============================>.] - ETA: 0s - loss: 0.6314 - accuracy: 0.9688
1685/1688 [============================>.] - ETA: 0s - loss: 0.6326 - accuracy: 0.9688
1688/1688 [==============================] - 5s 3ms/step - loss: 0.6345 - accuracy: 0.9688 - val_loss: 0.6984 - val_accuracy: 0.9717
Epoch 5/5

   1/1688 [..............................] - ETA: 4s - loss: 1.5806e-04 - accuracy: 1.0000
  21/1688 [..............................] - ETA: 4s - loss: 0.5264 - accuracy: 0.9747    
  41/1688 [..............................] - ETA: 4s - loss: 0.6207 - accuracy: 0.9733
  61/1688 [>.............................] - ETA: 4s - loss: 0.5021 - accuracy: 0.9780
  81/1688 [>.............................] - ETA: 4s - loss: 0.5711 - accuracy: 0.9711
 101/1688 [>.............................] - ETA: 4s - loss: 0.5065 - accuracy: 0.9743
 121/1688 [=>............................] - ETA: 4s - loss: 0.5417 - accuracy: 0.9744
 141/1688 [=>............................] - ETA: 3s - loss: 0.5170 - accuracy: 0.9752
 161/1688 [=>............................] - ETA: 3s - loss: 0.5035 - accuracy: 0.9748
 181/1688 [==>...........................] - ETA: 3s - loss: 0.4922 - accuracy: 0.9748
 201/1688 [==>...........................] - ETA: 3s - loss: 0.4713 - accuracy: 0.9748
 222/1688 [==>...........................] - ETA: 3s - loss: 0.4636 - accuracy: 0.9752
 243/1688 [===>..........................] - ETA: 3s - loss: 0.4929 - accuracy: 0.9748
 263/1688 [===>..........................] - ETA: 3s - loss: 0.4987 - accuracy: 0.9747
 283/1688 [====>.........................] - ETA: 3s - loss: 0.5059 - accuracy: 0.9738
 303/1688 [====>.........................] - ETA: 3s - loss: 0.5240 - accuracy: 0.9732
 323/1688 [====>.........................] - ETA: 3s - loss: 0.5227 - accuracy: 0.9731
 342/1688 [=====>........................] - ETA: 3s - loss: 0.5162 - accuracy: 0.9730
 362/1688 [=====>........................] - ETA: 3s - loss: 0.5090 - accuracy: 0.9734
 382/1688 [=====>........................] - ETA: 3s - loss: 0.5113 - accuracy: 0.9734
 403/1688 [======>.......................] - ETA: 3s - loss: 0.5183 - accuracy: 0.9732
 423/1688 [======>.......................] - ETA: 3s - loss: 0.5266 - accuracy: 0.9729
 443/1688 [======>.......................] - ETA: 3s - loss: 0.5215 - accuracy: 0.9732
 464/1688 [=======>......................] - ETA: 3s - loss: 0.5308 - accuracy: 0.9734
 484/1688 [=======>......................] - ETA: 3s - loss: 0.5384 - accuracy: 0.9732
 504/1688 [=======>......................] - ETA: 3s - loss: 0.5355 - accuracy: 0.9733
 524/1688 [========>.....................] - ETA: 2s - loss: 0.5413 - accuracy: 0.9732
 544/1688 [========>.....................] - ETA: 2s - loss: 0.5410 - accuracy: 0.9733
 565/1688 [=========>....................] - ETA: 2s - loss: 0.5405 - accuracy: 0.9732
 586/1688 [=========>....................] - ETA: 2s - loss: 0.5306 - accuracy: 0.9733
 606/1688 [=========>....................] - ETA: 2s - loss: 0.5276 - accuracy: 0.9734
 626/1688 [==========>...................] - ETA: 2s - loss: 0.5283 - accuracy: 0.9735
 646/1688 [==========>...................] - ETA: 2s - loss: 0.5247 - accuracy: 0.9736
 666/1688 [==========>...................] - ETA: 2s - loss: 0.5252 - accuracy: 0.9738
 686/1688 [===========>..................] - ETA: 2s - loss: 0.5225 - accuracy: 0.9739
 706/1688 [===========>..................] - ETA: 2s - loss: 0.5299 - accuracy: 0.9736
 726/1688 [===========>..................] - ETA: 2s - loss: 0.5331 - accuracy: 0.9734
 746/1688 [============>.................] - ETA: 2s - loss: 0.5317 - accuracy: 0.9733
 766/1688 [============>.................] - ETA: 2s - loss: 0.5275 - accuracy: 0.9734
 786/1688 [============>.................] - ETA: 2s - loss: 0.5238 - accuracy: 0.9734
 806/1688 [=============>................] - ETA: 2s - loss: 0.5333 - accuracy: 0.9729
 827/1688 [=============>................] - ETA: 2s - loss: 0.5468 - accuracy: 0.9726
 847/1688 [==============>...............] - ETA: 2s - loss: 0.5618 - accuracy: 0.9721
 868/1688 [==============>...............] - ETA: 2s - loss: 0.5632 - accuracy: 0.9721
 889/1688 [==============>...............] - ETA: 2s - loss: 0.5590 - accuracy: 0.9721
 909/1688 [===============>..............] - ETA: 1s - loss: 0.5585 - accuracy: 0.9721
 928/1688 [===============>..............] - ETA: 1s - loss: 0.5602 - accuracy: 0.9718
 948/1688 [===============>..............] - ETA: 1s - loss: 0.5541 - accuracy: 0.9721
 968/1688 [================>.............] - ETA: 1s - loss: 0.5552 - accuracy: 0.9721
 988/1688 [================>.............] - ETA: 1s - loss: 0.5525 - accuracy: 0.9721
1008/1688 [================>.............] - ETA: 1s - loss: 0.5542 - accuracy: 0.9720
1028/1688 [=================>............] - ETA: 1s - loss: 0.5558 - accuracy: 0.9719
1048/1688 [=================>............] - ETA: 1s - loss: 0.5554 - accuracy: 0.9721
1068/1688 [=================>............] - ETA: 1s - loss: 0.5527 - accuracy: 0.9721
1088/1688 [==================>...........] - ETA: 1s - loss: 0.5503 - accuracy: 0.9721
1107/1688 [==================>...........] - ETA: 1s - loss: 0.5514 - accuracy: 0.9721
1127/1688 [===================>..........] - ETA: 1s - loss: 0.5566 - accuracy: 0.9718
1147/1688 [===================>..........] - ETA: 1s - loss: 0.5583 - accuracy: 0.9719
1167/1688 [===================>..........] - ETA: 1s - loss: 0.5633 - accuracy: 0.9717
1187/1688 [====================>.........] - ETA: 1s - loss: 0.5650 - accuracy: 0.9716
1207/1688 [====================>.........] - ETA: 1s - loss: 0.5669 - accuracy: 0.9716
1227/1688 [====================>.........] - ETA: 1s - loss: 0.5643 - accuracy: 0.9718
1247/1688 [=====================>........] - ETA: 1s - loss: 0.5622 - accuracy: 0.9719
1267/1688 [=====================>........] - ETA: 1s - loss: 0.5582 - accuracy: 0.9719
1287/1688 [=====================>........] - ETA: 1s - loss: 0.5570 - accuracy: 0.9717
1307/1688 [======================>.......] - ETA: 0s - loss: 0.5523 - accuracy: 0.9719
1327/1688 [======================>.......] - ETA: 0s - loss: 0.5546 - accuracy: 0.9718
1347/1688 [======================>.......] - ETA: 0s - loss: 0.5543 - accuracy: 0.9719
1367/1688 [=======================>......] - ETA: 0s - loss: 0.5565 - accuracy: 0.9719
1387/1688 [=======================>......] - ETA: 0s - loss: 0.5557 - accuracy: 0.9719
1407/1688 [========================>.....] - ETA: 0s - loss: 0.5529 - accuracy: 0.9720
1427/1688 [========================>.....] - ETA: 0s - loss: 0.5534 - accuracy: 0.9721
1447/1688 [========================>.....] - ETA: 0s - loss: 0.5565 - accuracy: 0.9721
1467/1688 [=========================>....] - ETA: 0s - loss: 0.5613 - accuracy: 0.9719
1487/1688 [=========================>....] - ETA: 0s - loss: 0.5610 - accuracy: 0.9719
1507/1688 [=========================>....] - ETA: 0s - loss: 0.5610 - accuracy: 0.9717
1527/1688 [==========================>...] - ETA: 0s - loss: 0.5581 - accuracy: 0.9718
1548/1688 [==========================>...] - ETA: 0s - loss: 0.5606 - accuracy: 0.9717
1568/1688 [==========================>...] - ETA: 0s - loss: 0.5638 - accuracy: 0.9716
1588/1688 [===========================>..] - ETA: 0s - loss: 0.5644 - accuracy: 0.9715
1608/1688 [===========================>..] - ETA: 0s - loss: 0.5652 - accuracy: 0.9715
1628/1688 [===========================>..] - ETA: 0s - loss: 0.5656 - accuracy: 0.9714
1648/1688 [============================>.] - ETA: 0s - loss: 0.5692 - accuracy: 0.9713
1668/1688 [============================>.] - ETA: 0s - loss: 0.5708 - accuracy: 0.9713
1688/1688 [==============================] - ETA: 0s - loss: 0.5696 - accuracy: 0.9714
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5696 - accuracy: 0.9714 - val_loss: 0.8509 - val_accuracy: 0.9707
Test accuracy after fine tuning: 0.96670001745224

6. Model conversion

After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida software simulator.

Note

One needs to supply the coefficients used to rescale the input dataset before the training - here input_scaling.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()

accuracy = model_akida.evaluate(raw_x_test, raw_y_test)
print('Test accuracy after conversion:', accuracy)

# For non-regression purpose
assert accuracy > 0.96

Out:

                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          3
______________________________________________

                  SW/conv2d-dense (Software)
______________________________________________________________
Layer (type)                  Output shape  Kernel shape
==============================================================
conv2d (InputConv.)           [13, 13, 32]  (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.)  [7, 7, 64]    (3, 3, 32, 1)
______________________________________________________________
                                            (1, 1, 32, 64)
______________________________________________________________
dense (Fully.)                [1, 1, 10]    (1, 1, 3136, 10)
______________________________________________________________

Test accuracy after conversion: 0.9660999774932861

Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).

Total running time of the script: ( 0 minutes 59.799 seconds)

Gallery generated by Sphinx-Gallery