CNN conversion flow tutorial

This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.

The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.

Note

Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.

MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.

../../_images/cnn2snn_flow_small.jpg

1. Load and reshape MNIST dataset

After loading, we make 2 transformations on the dataset:

  1. Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.

Note

At this point, we’ll set aside the raw data for testing our converted model in the Akida runtime later.

  1. Rescale the 8-bit loaded data to the range 0-to-1 for training.

Note

Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.

This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.

Also note that we store the scaling values input_scaling for use when preparing the model for the Akida neuromorphic IP. The implementation of the Akida neural network allows us to completely skip the rescaling step (i.e. the Akida model should be fed with the raw 8-bit values) but that does require information about what scaling was applied prior to training - see below for more details.

from tensorflow import keras

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)

# Set aside raw test data for use with Akida runtime later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test

# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a

2. Model definition

Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.

An appropriate model for MNIST (inspired by this example) might look something like the following:

model_keras = keras.models.Sequential([
    keras.layers.Conv2D(
        filters=32, kernel_size=3, input_shape=(28, 28, 1), strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.SeparableConv2D(
        filters=64, kernel_size=3, padding='same', strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 conv2d (Conv2D)             (None, 13, 13, 32)        320

 batch_normalization (BatchN  (None, 13, 13, 32)       128
 ormalization)

 re_lu (ReLU)                (None, 13, 13, 32)        0

 separable_conv2d (Separable  (None, 7, 7, 64)         2400
 Conv2D)

 batch_normalization_1 (Batc  (None, 7, 7, 64)         256
 hNormalization)

 re_lu_1 (ReLU)              (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (Dense)               (None, 10)                31370

=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________

The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):

  • Standard Conv2D and Dense layers are supported

  • Hidden layers must be followed by a ReLU layer.

  • BatchNormalization must always happen before activations.

  • Convolutional blocks can optionally be followed by a MaxPooling.

The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).

from cnn2snn import check_model_compatibility

print("Model compatible for Akida conversion:",
      check_model_compatibility(model_keras))
Model compatible for Akida conversion: True

3. Model training

Before going any further, train the model and get its performance. The created model should have achieved a test accuracy over 98% after 10 epochs.

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)

score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Epoch 1/10

   1/1688 [..............................] - ETA: 12:58 - loss: 2.9708 - accuracy: 0.0312
  30/1688 [..............................] - ETA: 2s - loss: 1.2146 - accuracy: 0.6115   
  56/1688 [..............................] - ETA: 3s - loss: 0.8667 - accuracy: 0.7310
  84/1688 [>.............................] - ETA: 2s - loss: 0.7189 - accuracy: 0.7809
 112/1688 [>.............................] - ETA: 2s - loss: 0.6293 - accuracy: 0.8083
 140/1688 [=>............................] - ETA: 2s - loss: 0.5766 - accuracy: 0.8254
 168/1688 [=>............................] - ETA: 2s - loss: 0.5195 - accuracy: 0.8439
 196/1688 [==>...........................] - ETA: 2s - loss: 0.4819 - accuracy: 0.8546
 224/1688 [==>...........................] - ETA: 2s - loss: 0.4555 - accuracy: 0.8633
 252/1688 [===>..........................] - ETA: 2s - loss: 0.4291 - accuracy: 0.8713
 279/1688 [===>..........................] - ETA: 2s - loss: 0.4088 - accuracy: 0.8771
 306/1688 [====>.........................] - ETA: 2s - loss: 0.3915 - accuracy: 0.8824
 334/1688 [====>.........................] - ETA: 2s - loss: 0.3748 - accuracy: 0.8878
 360/1688 [=====>........................] - ETA: 2s - loss: 0.3611 - accuracy: 0.8908
 387/1688 [=====>........................] - ETA: 2s - loss: 0.3491 - accuracy: 0.8947
 415/1688 [======>.......................] - ETA: 2s - loss: 0.3364 - accuracy: 0.8984
 443/1688 [======>.......................] - ETA: 2s - loss: 0.3272 - accuracy: 0.9012
 471/1688 [=======>......................] - ETA: 2s - loss: 0.3165 - accuracy: 0.9047
 499/1688 [=======>......................] - ETA: 2s - loss: 0.3074 - accuracy: 0.9071
 527/1688 [========>.....................] - ETA: 2s - loss: 0.3000 - accuracy: 0.9096
 554/1688 [========>.....................] - ETA: 2s - loss: 0.2911 - accuracy: 0.9119
 582/1688 [=========>....................] - ETA: 2s - loss: 0.2846 - accuracy: 0.9140
 609/1688 [=========>....................] - ETA: 1s - loss: 0.2770 - accuracy: 0.9165
 636/1688 [==========>...................] - ETA: 1s - loss: 0.2726 - accuracy: 0.9180
 664/1688 [==========>...................] - ETA: 1s - loss: 0.2679 - accuracy: 0.9192
 692/1688 [===========>..................] - ETA: 1s - loss: 0.2609 - accuracy: 0.9214
 720/1688 [===========>..................] - ETA: 1s - loss: 0.2575 - accuracy: 0.9222
 749/1688 [============>.................] - ETA: 1s - loss: 0.2536 - accuracy: 0.9234
 777/1688 [============>.................] - ETA: 1s - loss: 0.2485 - accuracy: 0.9248
 806/1688 [=============>................] - ETA: 1s - loss: 0.2444 - accuracy: 0.9261
 834/1688 [=============>................] - ETA: 1s - loss: 0.2396 - accuracy: 0.9275
 862/1688 [==============>...............] - ETA: 1s - loss: 0.2358 - accuracy: 0.9287
 890/1688 [==============>...............] - ETA: 1s - loss: 0.2322 - accuracy: 0.9297
 918/1688 [===============>..............] - ETA: 1s - loss: 0.2277 - accuracy: 0.9311
 946/1688 [===============>..............] - ETA: 1s - loss: 0.2241 - accuracy: 0.9320
 973/1688 [================>.............] - ETA: 1s - loss: 0.2212 - accuracy: 0.9329
1001/1688 [================>.............] - ETA: 1s - loss: 0.2191 - accuracy: 0.9338
1028/1688 [=================>............] - ETA: 1s - loss: 0.2164 - accuracy: 0.9345
1056/1688 [=================>............] - ETA: 1s - loss: 0.2135 - accuracy: 0.9353
1085/1688 [==================>...........] - ETA: 1s - loss: 0.2105 - accuracy: 0.9361
1112/1688 [==================>...........] - ETA: 1s - loss: 0.2081 - accuracy: 0.9367
1141/1688 [===================>..........] - ETA: 1s - loss: 0.2054 - accuracy: 0.9375
1168/1688 [===================>..........] - ETA: 0s - loss: 0.2031 - accuracy: 0.9383
1196/1688 [====================>.........] - ETA: 0s - loss: 0.2008 - accuracy: 0.9390
1224/1688 [====================>.........] - ETA: 0s - loss: 0.1990 - accuracy: 0.9396
1251/1688 [=====================>........] - ETA: 0s - loss: 0.1965 - accuracy: 0.9404
1278/1688 [=====================>........] - ETA: 0s - loss: 0.1946 - accuracy: 0.9410
1306/1688 [======================>.......] - ETA: 0s - loss: 0.1929 - accuracy: 0.9414
1333/1688 [======================>.......] - ETA: 0s - loss: 0.1913 - accuracy: 0.9419
1361/1688 [=======================>......] - ETA: 0s - loss: 0.1907 - accuracy: 0.9422
1388/1688 [=======================>......] - ETA: 0s - loss: 0.1888 - accuracy: 0.9427
1416/1688 [========================>.....] - ETA: 0s - loss: 0.1866 - accuracy: 0.9434
1444/1688 [========================>.....] - ETA: 0s - loss: 0.1849 - accuracy: 0.9439
1472/1688 [=========================>....] - ETA: 0s - loss: 0.1836 - accuracy: 0.9443
1499/1688 [=========================>....] - ETA: 0s - loss: 0.1820 - accuracy: 0.9448
1527/1688 [==========================>...] - ETA: 0s - loss: 0.1803 - accuracy: 0.9452
1555/1688 [==========================>...] - ETA: 0s - loss: 0.1794 - accuracy: 0.9456
1583/1688 [===========================>..] - ETA: 0s - loss: 0.1776 - accuracy: 0.9461
1611/1688 [===========================>..] - ETA: 0s - loss: 0.1764 - accuracy: 0.9467
1640/1688 [============================>.] - ETA: 0s - loss: 0.1746 - accuracy: 0.9472
1668/1688 [============================>.] - ETA: 0s - loss: 0.1733 - accuracy: 0.9477
1688/1688 [==============================] - ETA: 0s - loss: 0.1723 - accuracy: 0.9479
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1723 - accuracy: 0.9479 - val_loss: 0.0930 - val_accuracy: 0.9732
Epoch 2/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0485 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0868 - accuracy: 0.9778
  60/1688 [>.............................] - ETA: 2s - loss: 0.0890 - accuracy: 0.9740
  87/1688 [>.............................] - ETA: 2s - loss: 0.0788 - accuracy: 0.9770
 115/1688 [=>............................] - ETA: 2s - loss: 0.0769 - accuracy: 0.9772
 143/1688 [=>............................] - ETA: 2s - loss: 0.0700 - accuracy: 0.9790
 171/1688 [==>...........................] - ETA: 2s - loss: 0.0710 - accuracy: 0.9786
 199/1688 [==>...........................] - ETA: 2s - loss: 0.0711 - accuracy: 0.9779
 226/1688 [===>..........................] - ETA: 2s - loss: 0.0691 - accuracy: 0.9786
 253/1688 [===>..........................] - ETA: 2s - loss: 0.0692 - accuracy: 0.9784
 280/1688 [===>..........................] - ETA: 2s - loss: 0.0683 - accuracy: 0.9792
 307/1688 [====>.........................] - ETA: 2s - loss: 0.0671 - accuracy: 0.9793
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0673 - accuracy: 0.9791
 362/1688 [=====>........................] - ETA: 2s - loss: 0.0676 - accuracy: 0.9789
 390/1688 [=====>........................] - ETA: 2s - loss: 0.0689 - accuracy: 0.9788
 418/1688 [======>.......................] - ETA: 2s - loss: 0.0696 - accuracy: 0.9785
 446/1688 [======>.......................] - ETA: 2s - loss: 0.0694 - accuracy: 0.9785
 475/1688 [=======>......................] - ETA: 2s - loss: 0.0704 - accuracy: 0.9782
 503/1688 [=======>......................] - ETA: 2s - loss: 0.0698 - accuracy: 0.9785
 530/1688 [========>.....................] - ETA: 2s - loss: 0.0680 - accuracy: 0.9790
 558/1688 [========>.....................] - ETA: 2s - loss: 0.0688 - accuracy: 0.9786
 587/1688 [=========>....................] - ETA: 2s - loss: 0.0688 - accuracy: 0.9784
 615/1688 [=========>....................] - ETA: 1s - loss: 0.0684 - accuracy: 0.9786
 642/1688 [==========>...................] - ETA: 1s - loss: 0.0680 - accuracy: 0.9788
 669/1688 [==========>...................] - ETA: 1s - loss: 0.0680 - accuracy: 0.9787
 697/1688 [===========>..................] - ETA: 1s - loss: 0.0685 - accuracy: 0.9784
 724/1688 [===========>..................] - ETA: 1s - loss: 0.0678 - accuracy: 0.9785
 752/1688 [============>.................] - ETA: 1s - loss: 0.0678 - accuracy: 0.9788
 779/1688 [============>.................] - ETA: 1s - loss: 0.0670 - accuracy: 0.9791
 807/1688 [=============>................] - ETA: 1s - loss: 0.0663 - accuracy: 0.9792
 835/1688 [=============>................] - ETA: 1s - loss: 0.0663 - accuracy: 0.9793
 863/1688 [==============>...............] - ETA: 1s - loss: 0.0671 - accuracy: 0.9791
 891/1688 [==============>...............] - ETA: 1s - loss: 0.0666 - accuracy: 0.9793
 919/1688 [===============>..............] - ETA: 1s - loss: 0.0673 - accuracy: 0.9791
 947/1688 [===============>..............] - ETA: 1s - loss: 0.0675 - accuracy: 0.9790
 975/1688 [================>.............] - ETA: 1s - loss: 0.0673 - accuracy: 0.9790
1003/1688 [================>.............] - ETA: 1s - loss: 0.0672 - accuracy: 0.9791
1031/1688 [=================>............] - ETA: 1s - loss: 0.0676 - accuracy: 0.9790
1058/1688 [=================>............] - ETA: 1s - loss: 0.0681 - accuracy: 0.9789
1086/1688 [==================>...........] - ETA: 1s - loss: 0.0681 - accuracy: 0.9788
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0676 - accuracy: 0.9790
1142/1688 [===================>..........] - ETA: 0s - loss: 0.0674 - accuracy: 0.9790
1169/1688 [===================>..........] - ETA: 0s - loss: 0.0672 - accuracy: 0.9790
1196/1688 [====================>.........] - ETA: 0s - loss: 0.0670 - accuracy: 0.9791
1224/1688 [====================>.........] - ETA: 0s - loss: 0.0671 - accuracy: 0.9792
1252/1688 [=====================>........] - ETA: 0s - loss: 0.0671 - accuracy: 0.9792
1279/1688 [=====================>........] - ETA: 0s - loss: 0.0670 - accuracy: 0.9793
1307/1688 [======================>.......] - ETA: 0s - loss: 0.0671 - accuracy: 0.9791
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0676 - accuracy: 0.9790
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0677 - accuracy: 0.9789
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0680 - accuracy: 0.9788
1417/1688 [========================>.....] - ETA: 0s - loss: 0.0684 - accuracy: 0.9787
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0686 - accuracy: 0.9788
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0687 - accuracy: 0.9787
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0689 - accuracy: 0.9786
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0692 - accuracy: 0.9784
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0697 - accuracy: 0.9782
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0698 - accuracy: 0.9782
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0694 - accuracy: 0.9783
1639/1688 [============================>.] - ETA: 0s - loss: 0.0700 - accuracy: 0.9784
1667/1688 [============================>.] - ETA: 0s - loss: 0.0704 - accuracy: 0.9783
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0705 - accuracy: 0.9783 - val_loss: 0.0572 - val_accuracy: 0.9852
Epoch 3/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0077 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0296 - accuracy: 0.9927
  57/1688 [>.............................] - ETA: 2s - loss: 0.0347 - accuracy: 0.9901
  85/1688 [>.............................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9864
 113/1688 [=>............................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9862
 141/1688 [=>............................] - ETA: 2s - loss: 0.0429 - accuracy: 0.9874
 169/1688 [==>...........................] - ETA: 2s - loss: 0.0434 - accuracy: 0.9874
 196/1688 [==>...........................] - ETA: 2s - loss: 0.0448 - accuracy: 0.9864
 223/1688 [==>...........................] - ETA: 2s - loss: 0.0484 - accuracy: 0.9850
 251/1688 [===>..........................] - ETA: 2s - loss: 0.0481 - accuracy: 0.9853
 279/1688 [===>..........................] - ETA: 2s - loss: 0.0493 - accuracy: 0.9850
 306/1688 [====>.........................] - ETA: 2s - loss: 0.0503 - accuracy: 0.9846
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0500 - accuracy: 0.9847
 362/1688 [=====>........................] - ETA: 2s - loss: 0.0489 - accuracy: 0.9853
 390/1688 [=====>........................] - ETA: 2s - loss: 0.0478 - accuracy: 0.9857
 418/1688 [======>.......................] - ETA: 2s - loss: 0.0488 - accuracy: 0.9854
 445/1688 [======>.......................] - ETA: 2s - loss: 0.0500 - accuracy: 0.9850
 473/1688 [=======>......................] - ETA: 2s - loss: 0.0488 - accuracy: 0.9852
 500/1688 [=======>......................] - ETA: 2s - loss: 0.0484 - accuracy: 0.9854
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0480 - accuracy: 0.9854
 556/1688 [========>.....................] - ETA: 2s - loss: 0.0488 - accuracy: 0.9852
 584/1688 [=========>....................] - ETA: 2s - loss: 0.0490 - accuracy: 0.9851
 612/1688 [=========>....................] - ETA: 1s - loss: 0.0502 - accuracy: 0.9846
 640/1688 [==========>...................] - ETA: 1s - loss: 0.0500 - accuracy: 0.9848
 668/1688 [==========>...................] - ETA: 1s - loss: 0.0503 - accuracy: 0.9844
 696/1688 [===========>..................] - ETA: 1s - loss: 0.0508 - accuracy: 0.9842
 724/1688 [===========>..................] - ETA: 1s - loss: 0.0505 - accuracy: 0.9844
 752/1688 [============>.................] - ETA: 1s - loss: 0.0504 - accuracy: 0.9843
 780/1688 [============>.................] - ETA: 1s - loss: 0.0514 - accuracy: 0.9839
 808/1688 [=============>................] - ETA: 1s - loss: 0.0516 - accuracy: 0.9838
 836/1688 [=============>................] - ETA: 1s - loss: 0.0513 - accuracy: 0.9838
 864/1688 [==============>...............] - ETA: 1s - loss: 0.0517 - accuracy: 0.9838
 891/1688 [==============>...............] - ETA: 1s - loss: 0.0513 - accuracy: 0.9838
 919/1688 [===============>..............] - ETA: 1s - loss: 0.0517 - accuracy: 0.9838
 947/1688 [===============>..............] - ETA: 1s - loss: 0.0520 - accuracy: 0.9835
 974/1688 [================>.............] - ETA: 1s - loss: 0.0517 - accuracy: 0.9836
1002/1688 [================>.............] - ETA: 1s - loss: 0.0514 - accuracy: 0.9838
1030/1688 [=================>............] - ETA: 1s - loss: 0.0517 - accuracy: 0.9836
1058/1688 [=================>............] - ETA: 1s - loss: 0.0522 - accuracy: 0.9835
1086/1688 [==================>...........] - ETA: 1s - loss: 0.0522 - accuracy: 0.9835
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0518 - accuracy: 0.9836
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0516 - accuracy: 0.9837
1169/1688 [===================>..........] - ETA: 0s - loss: 0.0515 - accuracy: 0.9837
1197/1688 [====================>.........] - ETA: 0s - loss: 0.0515 - accuracy: 0.9837
1225/1688 [====================>.........] - ETA: 0s - loss: 0.0515 - accuracy: 0.9837
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0514 - accuracy: 0.9837
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0514 - accuracy: 0.9837
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0518 - accuracy: 0.9836
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0522 - accuracy: 0.9834
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0523 - accuracy: 0.9834
1391/1688 [=======================>......] - ETA: 0s - loss: 0.0522 - accuracy: 0.9834
1419/1688 [========================>.....] - ETA: 0s - loss: 0.0524 - accuracy: 0.9834
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0526 - accuracy: 0.9833
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0527 - accuracy: 0.9832
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0530 - accuracy: 0.9832
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0529 - accuracy: 0.9831
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0529 - accuracy: 0.9832
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0528 - accuracy: 0.9832
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0528 - accuracy: 0.9831
1642/1688 [============================>.] - ETA: 0s - loss: 0.0526 - accuracy: 0.9832
1670/1688 [============================>.] - ETA: 0s - loss: 0.0524 - accuracy: 0.9832
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0525 - accuracy: 0.9832 - val_loss: 0.0646 - val_accuracy: 0.9812
Epoch 4/10

   1/1688 [..............................] - ETA: 3s - loss: 0.1505 - accuracy: 0.9375
  30/1688 [..............................] - ETA: 2s - loss: 0.0330 - accuracy: 0.9844
  58/1688 [>.............................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9887
  86/1688 [>.............................] - ETA: 2s - loss: 0.0354 - accuracy: 0.9869
 115/1688 [=>............................] - ETA: 2s - loss: 0.0340 - accuracy: 0.9875
 143/1688 [=>............................] - ETA: 2s - loss: 0.0358 - accuracy: 0.9871
 172/1688 [==>...........................] - ETA: 2s - loss: 0.0363 - accuracy: 0.9875
 199/1688 [==>...........................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9871
 227/1688 [===>..........................] - ETA: 2s - loss: 0.0389 - accuracy: 0.9866
 255/1688 [===>..........................] - ETA: 2s - loss: 0.0385 - accuracy: 0.9866
 283/1688 [====>.........................] - ETA: 2s - loss: 0.0387 - accuracy: 0.9869
 311/1688 [====>.........................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9868
 338/1688 [=====>........................] - ETA: 2s - loss: 0.0391 - accuracy: 0.9871
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0381 - accuracy: 0.9874
 393/1688 [=====>........................] - ETA: 2s - loss: 0.0392 - accuracy: 0.9877
 421/1688 [======>.......................] - ETA: 2s - loss: 0.0390 - accuracy: 0.9875
 449/1688 [======>.......................] - ETA: 2s - loss: 0.0388 - accuracy: 0.9875
 477/1688 [=======>......................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9880
 505/1688 [=======>......................] - ETA: 2s - loss: 0.0385 - accuracy: 0.9878
 533/1688 [========>.....................] - ETA: 2s - loss: 0.0392 - accuracy: 0.9876
 561/1688 [========>.....................] - ETA: 2s - loss: 0.0384 - accuracy: 0.9879
 589/1688 [=========>....................] - ETA: 1s - loss: 0.0383 - accuracy: 0.9877
 617/1688 [=========>....................] - ETA: 1s - loss: 0.0388 - accuracy: 0.9875
 644/1688 [==========>...................] - ETA: 1s - loss: 0.0382 - accuracy: 0.9878
 672/1688 [==========>...................] - ETA: 1s - loss: 0.0374 - accuracy: 0.9880
 700/1688 [===========>..................] - ETA: 1s - loss: 0.0377 - accuracy: 0.9879
 727/1688 [===========>..................] - ETA: 1s - loss: 0.0376 - accuracy: 0.9880
 755/1688 [============>.................] - ETA: 1s - loss: 0.0377 - accuracy: 0.9879
 783/1688 [============>.................] - ETA: 1s - loss: 0.0384 - accuracy: 0.9877
 810/1688 [=============>................] - ETA: 1s - loss: 0.0386 - accuracy: 0.9877
 838/1688 [=============>................] - ETA: 1s - loss: 0.0383 - accuracy: 0.9878
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0387 - accuracy: 0.9877
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0397 - accuracy: 0.9876
 922/1688 [===============>..............] - ETA: 1s - loss: 0.0394 - accuracy: 0.9877
 950/1688 [===============>..............] - ETA: 1s - loss: 0.0391 - accuracy: 0.9877
 978/1688 [================>.............] - ETA: 1s - loss: 0.0392 - accuracy: 0.9878
1005/1688 [================>.............] - ETA: 1s - loss: 0.0394 - accuracy: 0.9878
1033/1688 [=================>............] - ETA: 1s - loss: 0.0396 - accuracy: 0.9877
1061/1688 [=================>............] - ETA: 1s - loss: 0.0395 - accuracy: 0.9876
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0395 - accuracy: 0.9876
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0397 - accuracy: 0.9875
1144/1688 [===================>..........] - ETA: 0s - loss: 0.0395 - accuracy: 0.9876
1171/1688 [===================>..........] - ETA: 0s - loss: 0.0403 - accuracy: 0.9874
1199/1688 [====================>.........] - ETA: 0s - loss: 0.0403 - accuracy: 0.9873
1227/1688 [====================>.........] - ETA: 0s - loss: 0.0405 - accuracy: 0.9872
1255/1688 [=====================>........] - ETA: 0s - loss: 0.0407 - accuracy: 0.9871
1283/1688 [=====================>........] - ETA: 0s - loss: 0.0413 - accuracy: 0.9869
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0414 - accuracy: 0.9869
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0412 - accuracy: 0.9869
1368/1688 [=======================>......] - ETA: 0s - loss: 0.0409 - accuracy: 0.9871
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0408 - accuracy: 0.9871
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0408 - accuracy: 0.9872
1451/1688 [========================>.....] - ETA: 0s - loss: 0.0409 - accuracy: 0.9871
1479/1688 [=========================>....] - ETA: 0s - loss: 0.0408 - accuracy: 0.9872
1507/1688 [=========================>....] - ETA: 0s - loss: 0.0408 - accuracy: 0.9871
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0407 - accuracy: 0.9872
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0416 - accuracy: 0.9870
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0415 - accuracy: 0.9870
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0415 - accuracy: 0.9870
1645/1688 [============================>.] - ETA: 0s - loss: 0.0414 - accuracy: 0.9870
1672/1688 [============================>.] - ETA: 0s - loss: 0.0421 - accuracy: 0.9868
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0420 - accuracy: 0.9869 - val_loss: 0.0615 - val_accuracy: 0.9843
Epoch 5/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0024 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9927
  58/1688 [>.............................] - ETA: 2s - loss: 0.0241 - accuracy: 0.9914
  86/1688 [>.............................] - ETA: 2s - loss: 0.0295 - accuracy: 0.9902
 114/1688 [=>............................] - ETA: 2s - loss: 0.0354 - accuracy: 0.9882
 142/1688 [=>............................] - ETA: 2s - loss: 0.0336 - accuracy: 0.9886
 170/1688 [==>...........................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9895
 198/1688 [==>...........................] - ETA: 2s - loss: 0.0322 - accuracy: 0.9893
 226/1688 [===>..........................] - ETA: 2s - loss: 0.0331 - accuracy: 0.9888
 253/1688 [===>..........................] - ETA: 2s - loss: 0.0328 - accuracy: 0.9889
 280/1688 [===>..........................] - ETA: 2s - loss: 0.0324 - accuracy: 0.9892
 308/1688 [====>.........................] - ETA: 2s - loss: 0.0318 - accuracy: 0.9897
 335/1688 [====>.........................] - ETA: 2s - loss: 0.0322 - accuracy: 0.9892
 363/1688 [=====>........................] - ETA: 2s - loss: 0.0315 - accuracy: 0.9894
 391/1688 [=====>........................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9895
 419/1688 [======>.......................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9898
 447/1688 [======>.......................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9899
 475/1688 [=======>......................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9896
 504/1688 [=======>......................] - ETA: 2s - loss: 0.0318 - accuracy: 0.9898
 532/1688 [========>.....................] - ETA: 2s - loss: 0.0331 - accuracy: 0.9893
 560/1688 [========>.....................] - ETA: 2s - loss: 0.0329 - accuracy: 0.9891
 588/1688 [=========>....................] - ETA: 2s - loss: 0.0328 - accuracy: 0.9891
 615/1688 [=========>....................] - ETA: 1s - loss: 0.0329 - accuracy: 0.9889
 643/1688 [==========>...................] - ETA: 1s - loss: 0.0329 - accuracy: 0.9888
 671/1688 [==========>...................] - ETA: 1s - loss: 0.0329 - accuracy: 0.9887
 699/1688 [===========>..................] - ETA: 1s - loss: 0.0330 - accuracy: 0.9888
 726/1688 [===========>..................] - ETA: 1s - loss: 0.0338 - accuracy: 0.9887
 753/1688 [============>.................] - ETA: 1s - loss: 0.0332 - accuracy: 0.9889
 781/1688 [============>.................] - ETA: 1s - loss: 0.0334 - accuracy: 0.9888
 809/1688 [=============>................] - ETA: 1s - loss: 0.0335 - accuracy: 0.9887
 838/1688 [=============>................] - ETA: 1s - loss: 0.0328 - accuracy: 0.9890
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0324 - accuracy: 0.9891
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0330 - accuracy: 0.9890
 921/1688 [===============>..............] - ETA: 1s - loss: 0.0325 - accuracy: 0.9891
 950/1688 [===============>..............] - ETA: 1s - loss: 0.0323 - accuracy: 0.9892
 978/1688 [================>.............] - ETA: 1s - loss: 0.0329 - accuracy: 0.9890
1006/1688 [================>.............] - ETA: 1s - loss: 0.0327 - accuracy: 0.9890
1034/1688 [=================>............] - ETA: 1s - loss: 0.0331 - accuracy: 0.9887
1062/1688 [=================>............] - ETA: 1s - loss: 0.0328 - accuracy: 0.9888
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0329 - accuracy: 0.9888
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0326 - accuracy: 0.9889
1144/1688 [===================>..........] - ETA: 0s - loss: 0.0328 - accuracy: 0.9888
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0326 - accuracy: 0.9888
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0324 - accuracy: 0.9889
1228/1688 [====================>.........] - ETA: 0s - loss: 0.0326 - accuracy: 0.9888
1255/1688 [=====================>........] - ETA: 0s - loss: 0.0329 - accuracy: 0.9888
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0333 - accuracy: 0.9887
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0334 - accuracy: 0.9887
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0334 - accuracy: 0.9887
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0335 - accuracy: 0.9887
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0337 - accuracy: 0.9886
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0336 - accuracy: 0.9887
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0339 - accuracy: 0.9887
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0341 - accuracy: 0.9885
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0341 - accuracy: 0.9886
1532/1688 [==========================>...] - ETA: 0s - loss: 0.0341 - accuracy: 0.9886
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0345 - accuracy: 0.9886
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0345 - accuracy: 0.9886
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0345 - accuracy: 0.9886
1642/1688 [============================>.] - ETA: 0s - loss: 0.0350 - accuracy: 0.9884
1670/1688 [============================>.] - ETA: 0s - loss: 0.0349 - accuracy: 0.9885
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0349 - accuracy: 0.9884 - val_loss: 0.0606 - val_accuracy: 0.9848
Epoch 6/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0169 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0273 - accuracy: 0.9906
  57/1688 [>.............................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9901
  85/1688 [>.............................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9901
 113/1688 [=>............................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9900
 141/1688 [=>............................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9916
 169/1688 [==>...........................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9909
 197/1688 [==>...........................] - ETA: 2s - loss: 0.0270 - accuracy: 0.9918
 224/1688 [==>...........................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9914
 251/1688 [===>..........................] - ETA: 2s - loss: 0.0270 - accuracy: 0.9913
 278/1688 [===>..........................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9916
 306/1688 [====>.........................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9917
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0252 - accuracy: 0.9918
 362/1688 [=====>........................] - ETA: 2s - loss: 0.0256 - accuracy: 0.9915
 389/1688 [=====>........................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9916
 417/1688 [======>.......................] - ETA: 2s - loss: 0.0272 - accuracy: 0.9914
 444/1688 [======>.......................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9912
 472/1688 [=======>......................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9915
 500/1688 [=======>......................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9914
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9914
 556/1688 [========>.....................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9914
 585/1688 [=========>....................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9914
 613/1688 [=========>....................] - ETA: 1s - loss: 0.0268 - accuracy: 0.9915
 641/1688 [==========>...................] - ETA: 1s - loss: 0.0266 - accuracy: 0.9916
 669/1688 [==========>...................] - ETA: 1s - loss: 0.0264 - accuracy: 0.9916
 697/1688 [===========>..................] - ETA: 1s - loss: 0.0273 - accuracy: 0.9913
 725/1688 [===========>..................] - ETA: 1s - loss: 0.0273 - accuracy: 0.9913
 754/1688 [============>.................] - ETA: 1s - loss: 0.0273 - accuracy: 0.9913
 782/1688 [============>.................] - ETA: 1s - loss: 0.0282 - accuracy: 0.9912
 809/1688 [=============>................] - ETA: 1s - loss: 0.0285 - accuracy: 0.9911
 836/1688 [=============>................] - ETA: 1s - loss: 0.0286 - accuracy: 0.9911
 864/1688 [==============>...............] - ETA: 1s - loss: 0.0288 - accuracy: 0.9910
 892/1688 [==============>...............] - ETA: 1s - loss: 0.0286 - accuracy: 0.9910
 919/1688 [===============>..............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9910
 948/1688 [===============>..............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9911
 976/1688 [================>.............] - ETA: 1s - loss: 0.0286 - accuracy: 0.9910
1003/1688 [================>.............] - ETA: 1s - loss: 0.0285 - accuracy: 0.9909
1031/1688 [=================>............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9911
1059/1688 [=================>............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9910
1087/1688 [==================>...........] - ETA: 1s - loss: 0.0284 - accuracy: 0.9910
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0285 - accuracy: 0.9909
1142/1688 [===================>..........] - ETA: 0s - loss: 0.0283 - accuracy: 0.9909
1170/1688 [===================>..........] - ETA: 0s - loss: 0.0286 - accuracy: 0.9908
1197/1688 [====================>.........] - ETA: 0s - loss: 0.0287 - accuracy: 0.9907
1225/1688 [====================>.........] - ETA: 0s - loss: 0.0288 - accuracy: 0.9907
1252/1688 [=====================>........] - ETA: 0s - loss: 0.0287 - accuracy: 0.9907
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0287 - accuracy: 0.9907
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0288 - accuracy: 0.9906
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0287 - accuracy: 0.9906
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0285 - accuracy: 0.9906
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0287 - accuracy: 0.9906
1417/1688 [========================>.....] - ETA: 0s - loss: 0.0286 - accuracy: 0.9906
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0287 - accuracy: 0.9906
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0288 - accuracy: 0.9907
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0287 - accuracy: 0.9907
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0287 - accuracy: 0.9906
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1642/1688 [============================>.] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1670/1688 [============================>.] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0289 - accuracy: 0.9904 - val_loss: 0.0776 - val_accuracy: 0.9822
Epoch 7/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0104 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9875
  58/1688 [>.............................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9903
  86/1688 [>.............................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9909
 114/1688 [=>............................] - ETA: 2s - loss: 0.0308 - accuracy: 0.9901
 142/1688 [=>............................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9910
 170/1688 [==>...........................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9917
 198/1688 [==>...........................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9927
 226/1688 [===>..........................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9932
 254/1688 [===>..........................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9937
 282/1688 [====>.........................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9936
 310/1688 [====>.........................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9937
 338/1688 [=====>........................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9934
 365/1688 [=====>........................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9935
 393/1688 [=====>........................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9933
 421/1688 [======>.......................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9929
 449/1688 [======>.......................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9930
 477/1688 [=======>......................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9929
 505/1688 [=======>......................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9929
 532/1688 [========>.....................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9929
 560/1688 [========>.....................] - ETA: 2s - loss: 0.0225 - accuracy: 0.9930
 588/1688 [=========>....................] - ETA: 2s - loss: 0.0221 - accuracy: 0.9931
 616/1688 [=========>....................] - ETA: 1s - loss: 0.0220 - accuracy: 0.9932
 643/1688 [==========>...................] - ETA: 1s - loss: 0.0217 - accuracy: 0.9931
 671/1688 [==========>...................] - ETA: 1s - loss: 0.0218 - accuracy: 0.9930
 699/1688 [===========>..................] - ETA: 1s - loss: 0.0218 - accuracy: 0.9929
 726/1688 [===========>..................] - ETA: 1s - loss: 0.0216 - accuracy: 0.9929
 753/1688 [============>.................] - ETA: 1s - loss: 0.0217 - accuracy: 0.9930
 781/1688 [============>.................] - ETA: 1s - loss: 0.0216 - accuracy: 0.9931
 809/1688 [=============>................] - ETA: 1s - loss: 0.0216 - accuracy: 0.9931
 836/1688 [=============>................] - ETA: 1s - loss: 0.0214 - accuracy: 0.9931
 864/1688 [==============>...............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9931
 892/1688 [==============>...............] - ETA: 1s - loss: 0.0221 - accuracy: 0.9928
 918/1688 [===============>..............] - ETA: 1s - loss: 0.0229 - accuracy: 0.9925
 946/1688 [===============>..............] - ETA: 1s - loss: 0.0227 - accuracy: 0.9925
 973/1688 [================>.............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9923
1001/1688 [================>.............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9923
1028/1688 [=================>............] - ETA: 1s - loss: 0.0229 - accuracy: 0.9924
1056/1688 [=================>............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9923
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0239 - accuracy: 0.9920
1111/1688 [==================>...........] - ETA: 1s - loss: 0.0239 - accuracy: 0.9918
1138/1688 [===================>..........] - ETA: 1s - loss: 0.0239 - accuracy: 0.9918
1166/1688 [===================>..........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9920
1194/1688 [====================>.........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9920
1221/1688 [====================>.........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9920
1248/1688 [=====================>........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9921
1276/1688 [=====================>........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9922
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0235 - accuracy: 0.9922
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0241 - accuracy: 0.9920
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0241 - accuracy: 0.9920
1416/1688 [========================>.....] - ETA: 0s - loss: 0.0240 - accuracy: 0.9920
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0248 - accuracy: 0.9918
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0248 - accuracy: 0.9918
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0252 - accuracy: 0.9916
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0253 - accuracy: 0.9917
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0252 - accuracy: 0.9917
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0254 - accuracy: 0.9917
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0255 - accuracy: 0.9917
1638/1688 [============================>.] - ETA: 0s - loss: 0.0255 - accuracy: 0.9917
1666/1688 [============================>.] - ETA: 0s - loss: 0.0255 - accuracy: 0.9916
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0255 - accuracy: 0.9916 - val_loss: 0.0587 - val_accuracy: 0.9858
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 5.4804e-04 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9940    
  59/1688 [>.............................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9936
  87/1688 [>.............................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9935
 115/1688 [=>............................] - ETA: 2s - loss: 0.0223 - accuracy: 0.9943
 143/1688 [=>............................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9950
 171/1688 [==>...........................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9947
 200/1688 [==>...........................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9950
 228/1688 [===>..........................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9948
 255/1688 [===>..........................] - ETA: 2s - loss: 0.0177 - accuracy: 0.9950
 283/1688 [====>.........................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9946
 310/1688 [====>.........................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9947
 338/1688 [=====>........................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9950
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9950
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9947
 421/1688 [======>.......................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9949
 449/1688 [======>.......................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9948
 478/1688 [=======>......................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9948
 507/1688 [========>.....................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9946
 535/1688 [========>.....................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9945
 562/1688 [========>.....................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9943
 590/1688 [=========>....................] - ETA: 1s - loss: 0.0174 - accuracy: 0.9941
 617/1688 [=========>....................] - ETA: 1s - loss: 0.0173 - accuracy: 0.9939
 644/1688 [==========>...................] - ETA: 1s - loss: 0.0173 - accuracy: 0.9939
 672/1688 [==========>...................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9937
 700/1688 [===========>..................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9935
 729/1688 [===========>..................] - ETA: 1s - loss: 0.0180 - accuracy: 0.9934
 757/1688 [============>.................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9936
 785/1688 [============>.................] - ETA: 1s - loss: 0.0181 - accuracy: 0.9936
 812/1688 [=============>................] - ETA: 1s - loss: 0.0183 - accuracy: 0.9935
 841/1688 [=============>................] - ETA: 1s - loss: 0.0190 - accuracy: 0.9933
 869/1688 [==============>...............] - ETA: 1s - loss: 0.0188 - accuracy: 0.9933
 897/1688 [==============>...............] - ETA: 1s - loss: 0.0191 - accuracy: 0.9932
 924/1688 [===============>..............] - ETA: 1s - loss: 0.0192 - accuracy: 0.9932
 952/1688 [===============>..............] - ETA: 1s - loss: 0.0192 - accuracy: 0.9933
 980/1688 [================>.............] - ETA: 1s - loss: 0.0191 - accuracy: 0.9933
1009/1688 [================>.............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9932
1037/1688 [=================>............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9932
1065/1688 [=================>............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9932
1093/1688 [==================>...........] - ETA: 1s - loss: 0.0197 - accuracy: 0.9931
1121/1688 [==================>...........] - ETA: 1s - loss: 0.0196 - accuracy: 0.9932
1148/1688 [===================>..........] - ETA: 0s - loss: 0.0195 - accuracy: 0.9932
1176/1688 [===================>..........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9931
1204/1688 [====================>.........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9930
1231/1688 [====================>.........] - ETA: 0s - loss: 0.0197 - accuracy: 0.9931
1260/1688 [=====================>........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9930
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0197 - accuracy: 0.9930
1315/1688 [======================>.......] - ETA: 0s - loss: 0.0199 - accuracy: 0.9930
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0199 - accuracy: 0.9930
1371/1688 [=======================>......] - ETA: 0s - loss: 0.0199 - accuracy: 0.9930
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0199 - accuracy: 0.9930
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0198 - accuracy: 0.9930
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0202 - accuracy: 0.9929
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0201 - accuracy: 0.9930
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0202 - accuracy: 0.9929
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0204 - accuracy: 0.9929
1567/1688 [==========================>...] - ETA: 0s - loss: 0.0205 - accuracy: 0.9928
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0206 - accuracy: 0.9928
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0207 - accuracy: 0.9927
1651/1688 [============================>.] - ETA: 0s - loss: 0.0207 - accuracy: 0.9928
1678/1688 [============================>.] - ETA: 0s - loss: 0.0209 - accuracy: 0.9927
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0210 - accuracy: 0.9926 - val_loss: 0.0543 - val_accuracy: 0.9867
Epoch 9/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0081 - accuracy: 1.0000
  30/1688 [..............................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9906
  58/1688 [>.............................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9935
  86/1688 [>.............................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9938
 114/1688 [=>............................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9940
 142/1688 [=>............................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9941
 169/1688 [==>...........................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9941
 197/1688 [==>...........................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9946
 226/1688 [===>..........................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9946
 254/1688 [===>..........................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9950
 282/1688 [====>.........................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9953
 311/1688 [====>.........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9954
 339/1688 [=====>........................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9955
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9956
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9958
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9958
 450/1688 [======>.......................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9956
 478/1688 [=======>......................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9955
 505/1688 [=======>......................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9954
 533/1688 [========>.....................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
 560/1688 [========>.....................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9953
 589/1688 [=========>....................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9949
 617/1688 [=========>....................] - ETA: 1s - loss: 0.0144 - accuracy: 0.9949
 644/1688 [==========>...................] - ETA: 1s - loss: 0.0149 - accuracy: 0.9948
 672/1688 [==========>...................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9949
 700/1688 [===========>..................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9950
 728/1688 [===========>..................] - ETA: 1s - loss: 0.0146 - accuracy: 0.9949
 755/1688 [============>.................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9947
 783/1688 [============>.................] - ETA: 1s - loss: 0.0148 - accuracy: 0.9947
 811/1688 [=============>................] - ETA: 1s - loss: 0.0152 - accuracy: 0.9945
 839/1688 [=============>................] - ETA: 1s - loss: 0.0154 - accuracy: 0.9944
 867/1688 [==============>...............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9945
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0157 - accuracy: 0.9943
 922/1688 [===============>..............] - ETA: 1s - loss: 0.0157 - accuracy: 0.9943
 950/1688 [===============>..............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9942
 978/1688 [================>.............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9940
1006/1688 [================>.............] - ETA: 1s - loss: 0.0168 - accuracy: 0.9940
1033/1688 [=================>............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9939
1061/1688 [=================>............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9938
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0173 - accuracy: 0.9938
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0177 - accuracy: 0.9937
1146/1688 [===================>..........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9938
1174/1688 [===================>..........] - ETA: 0s - loss: 0.0175 - accuracy: 0.9938
1202/1688 [====================>.........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9937
1230/1688 [====================>.........] - ETA: 0s - loss: 0.0175 - accuracy: 0.9938
1259/1688 [=====================>........] - ETA: 0s - loss: 0.0175 - accuracy: 0.9938
1287/1688 [=====================>........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9938
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0177 - accuracy: 0.9938
1342/1688 [======================>.......] - ETA: 0s - loss: 0.0178 - accuracy: 0.9938
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0184 - accuracy: 0.9937
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0183 - accuracy: 0.9938
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0185 - accuracy: 0.9938
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0186 - accuracy: 0.9937
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0186 - accuracy: 0.9937
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0185 - accuracy: 0.9937
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0184 - accuracy: 0.9937
1647/1688 [============================>.] - ETA: 0s - loss: 0.0184 - accuracy: 0.9938
1675/1688 [============================>.] - ETA: 0s - loss: 0.0183 - accuracy: 0.9938
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0183 - accuracy: 0.9938 - val_loss: 0.0653 - val_accuracy: 0.9852
Epoch 10/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0072 - accuracy: 1.0000
  29/1688 [..............................] - ETA: 3s - loss: 0.0063 - accuracy: 0.9989
  56/1688 [..............................] - ETA: 3s - loss: 0.0104 - accuracy: 0.9967
  84/1688 [>.............................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9963
 111/1688 [>.............................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9966
 139/1688 [=>............................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9973
 167/1688 [=>............................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9970
 194/1688 [==>...........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9963
 222/1688 [==>...........................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9965
 250/1688 [===>..........................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9964
 277/1688 [===>..........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9958
 305/1688 [====>.........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9955
 333/1688 [====>.........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9953
 361/1688 [=====>........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9953
 389/1688 [=====>........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9953
 417/1688 [======>.......................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9954
 445/1688 [======>.......................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9956
 473/1688 [=======>......................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9952
 501/1688 [=======>......................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9950
 528/1688 [========>.....................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9949
 556/1688 [========>.....................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9951
 583/1688 [=========>....................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9952
 611/1688 [=========>....................] - ETA: 1s - loss: 0.0147 - accuracy: 0.9951
 638/1688 [==========>...................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9952
 665/1688 [==========>...................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9952
 692/1688 [===========>..................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9953
 720/1688 [===========>..................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9954
 748/1688 [============>.................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9954
 776/1688 [============>.................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9954
 803/1688 [=============>................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9953
 831/1688 [=============>................] - ETA: 1s - loss: 0.0146 - accuracy: 0.9952
 859/1688 [==============>...............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9951
 886/1688 [==============>...............] - ETA: 1s - loss: 0.0150 - accuracy: 0.9950
 914/1688 [===============>..............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9950
 942/1688 [===============>..............] - ETA: 1s - loss: 0.0149 - accuracy: 0.9949
 970/1688 [================>.............] - ETA: 1s - loss: 0.0149 - accuracy: 0.9949
 997/1688 [================>.............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9950
1025/1688 [=================>............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9950
1053/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9950
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0146 - accuracy: 0.9950
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0147 - accuracy: 0.9950
1137/1688 [===================>..........] - ETA: 1s - loss: 0.0149 - accuracy: 0.9951
1164/1688 [===================>..........] - ETA: 0s - loss: 0.0148 - accuracy: 0.9951
1191/1688 [====================>.........] - ETA: 0s - loss: 0.0152 - accuracy: 0.9951
1219/1688 [====================>.........] - ETA: 0s - loss: 0.0153 - accuracy: 0.9950
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0155 - accuracy: 0.9949
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0157 - accuracy: 0.9948
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1328/1688 [======================>.......] - ETA: 0s - loss: 0.0161 - accuracy: 0.9947
1356/1688 [=======================>......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1412/1688 [========================>.....] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0159 - accuracy: 0.9947
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9948
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0158 - accuracy: 0.9948
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0161 - accuracy: 0.9947
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0162 - accuracy: 0.9947
1635/1688 [============================>.] - ETA: 0s - loss: 0.0161 - accuracy: 0.9948
1663/1688 [============================>.] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0160 - accuracy: 0.9948 - val_loss: 0.0575 - val_accuracy: 0.9858
Test accuracy: 0.9869999885559082

4. Model quantization

We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.

For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).

The quantized model is a Keras model where the neural layers (Conv2D, Dense) and the ReLU layers are replaced with custom CNN2SNN quantized layers (QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function folds the batch normalization layers into the corresponding neural layer. The new weights are computed according to this folding operation.

from cnn2snn import quantize

model_quantized = quantize(model_keras,
                           input_weight_quantization=8,
                           weight_quantization=4,
                           activ_quantization=4)
model_quantized.summary()
Model: "sequential_50"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 conv2d (QuantizedConv2D)    (None, 13, 13, 32)        320

 re_lu (QuantizedReLU)       (None, 13, 13, 32)        0

 separable_conv2d (Quantized  (None, 7, 7, 64)         2400
 SeparableConv2D)

 re_lu_1 (QuantizedReLU)     (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (QuantizedDense)      (None, 10)                31370

=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________

Check the quantized model accuracy.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])
Test accuracy after 8-4-4 quantization: 0.9611999988555908

Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.

Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.

Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.

from cnn2snn import quantize_layer

model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer='adam',
    metrics=['accuracy'])

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])

# To recover the original model accuracy, a quantization-aware training phase
# is required.
Test accuracy after low bitwidth quantization: 0.10300000011920929

5. Model fine tuning (quantization-aware training)

This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.

Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)

score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])
Epoch 1/5

   1/1688 [..............................] - ETA: 19:37 - loss: 2.2347 - accuracy: 0.0938
  20/1688 [..............................] - ETA: 4s - loss: 2.2945 - accuracy: 0.1109   
  39/1688 [..............................] - ETA: 4s - loss: 2.3163 - accuracy: 0.1090
  58/1688 [>.............................] - ETA: 4s - loss: 2.4929 - accuracy: 0.1207
  77/1688 [>.............................] - ETA: 4s - loss: 2.5421 - accuracy: 0.1266
  96/1688 [>.............................] - ETA: 4s - loss: 2.5645 - accuracy: 0.1520
 115/1688 [=>............................] - ETA: 4s - loss: 2.5267 - accuracy: 0.1853
 134/1688 [=>............................] - ETA: 4s - loss: 2.4826 - accuracy: 0.2229
 153/1688 [=>............................] - ETA: 4s - loss: 2.4248 - accuracy: 0.2717
 172/1688 [==>...........................] - ETA: 4s - loss: 2.3861 - accuracy: 0.3216
 192/1688 [==>...........................] - ETA: 4s - loss: 2.3230 - accuracy: 0.3696
 212/1688 [==>...........................] - ETA: 3s - loss: 2.2486 - accuracy: 0.4099
 231/1688 [===>..........................] - ETA: 3s - loss: 2.1576 - accuracy: 0.4455
 250/1688 [===>..........................] - ETA: 3s - loss: 2.0862 - accuracy: 0.4782
 270/1688 [===>..........................] - ETA: 3s - loss: 2.0286 - accuracy: 0.5059
 289/1688 [====>.........................] - ETA: 3s - loss: 1.9879 - accuracy: 0.5295
 308/1688 [====>.........................] - ETA: 3s - loss: 1.9025 - accuracy: 0.5527
 327/1688 [====>.........................] - ETA: 3s - loss: 1.8582 - accuracy: 0.5723
 347/1688 [=====>........................] - ETA: 3s - loss: 1.7904 - accuracy: 0.5916
 366/1688 [=====>........................] - ETA: 3s - loss: 1.7319 - accuracy: 0.6090
 385/1688 [=====>........................] - ETA: 3s - loss: 1.6902 - accuracy: 0.6231
 404/1688 [======>.......................] - ETA: 3s - loss: 1.6608 - accuracy: 0.6372
 423/1688 [======>.......................] - ETA: 3s - loss: 1.6265 - accuracy: 0.6496
 442/1688 [======>.......................] - ETA: 3s - loss: 1.5865 - accuracy: 0.6616
 461/1688 [=======>......................] - ETA: 3s - loss: 1.5597 - accuracy: 0.6723
 480/1688 [=======>......................] - ETA: 3s - loss: 1.5399 - accuracy: 0.6818
 499/1688 [=======>......................] - ETA: 3s - loss: 1.5167 - accuracy: 0.6908
 519/1688 [========>.....................] - ETA: 3s - loss: 1.5037 - accuracy: 0.6986
 538/1688 [========>.....................] - ETA: 3s - loss: 1.4793 - accuracy: 0.7072
 557/1688 [========>.....................] - ETA: 3s - loss: 1.4608 - accuracy: 0.7143
 576/1688 [=========>....................] - ETA: 2s - loss: 1.4501 - accuracy: 0.7211
 595/1688 [=========>....................] - ETA: 2s - loss: 1.4209 - accuracy: 0.7282
 614/1688 [=========>....................] - ETA: 2s - loss: 1.4042 - accuracy: 0.7341
 634/1688 [==========>...................] - ETA: 2s - loss: 1.3734 - accuracy: 0.7413
 653/1688 [==========>...................] - ETA: 2s - loss: 1.3640 - accuracy: 0.7468
 672/1688 [==========>...................] - ETA: 2s - loss: 1.3531 - accuracy: 0.7521
 691/1688 [===========>..................] - ETA: 2s - loss: 1.3387 - accuracy: 0.7573
 710/1688 [===========>..................] - ETA: 2s - loss: 1.3210 - accuracy: 0.7623
 729/1688 [===========>..................] - ETA: 2s - loss: 1.3065 - accuracy: 0.7673
 748/1688 [============>.................] - ETA: 2s - loss: 1.2914 - accuracy: 0.7718
 767/1688 [============>.................] - ETA: 2s - loss: 1.2665 - accuracy: 0.7766
 786/1688 [============>.................] - ETA: 2s - loss: 1.2464 - accuracy: 0.7806
 805/1688 [=============>................] - ETA: 2s - loss: 1.2333 - accuracy: 0.7845
 824/1688 [=============>................] - ETA: 2s - loss: 1.2239 - accuracy: 0.7877
 843/1688 [=============>................] - ETA: 2s - loss: 1.2113 - accuracy: 0.7917
 863/1688 [==============>...............] - ETA: 2s - loss: 1.2073 - accuracy: 0.7950
 882/1688 [==============>...............] - ETA: 2s - loss: 1.1953 - accuracy: 0.7984
 901/1688 [===============>..............] - ETA: 2s - loss: 1.1854 - accuracy: 0.8013
 921/1688 [===============>..............] - ETA: 2s - loss: 1.1744 - accuracy: 0.8045
 940/1688 [===============>..............] - ETA: 1s - loss: 1.1609 - accuracy: 0.8076
 959/1688 [================>.............] - ETA: 1s - loss: 1.1543 - accuracy: 0.8101
 978/1688 [================>.............] - ETA: 1s - loss: 1.1436 - accuracy: 0.8128
 998/1688 [================>.............] - ETA: 1s - loss: 1.1411 - accuracy: 0.8152
1017/1688 [=================>............] - ETA: 1s - loss: 1.1304 - accuracy: 0.8178
1036/1688 [=================>............] - ETA: 1s - loss: 1.1231 - accuracy: 0.8200
1055/1688 [=================>............] - ETA: 1s - loss: 1.1139 - accuracy: 0.8224
1074/1688 [==================>...........] - ETA: 1s - loss: 1.1043 - accuracy: 0.8248
1094/1688 [==================>...........] - ETA: 1s - loss: 1.0941 - accuracy: 0.8271
1113/1688 [==================>...........] - ETA: 1s - loss: 1.0957 - accuracy: 0.8289
1132/1688 [===================>..........] - ETA: 1s - loss: 1.0895 - accuracy: 0.8309
1152/1688 [===================>..........] - ETA: 1s - loss: 1.0827 - accuracy: 0.8330
1171/1688 [===================>..........] - ETA: 1s - loss: 1.0725 - accuracy: 0.8349
1191/1688 [====================>.........] - ETA: 1s - loss: 1.0693 - accuracy: 0.8370
1210/1688 [====================>.........] - ETA: 1s - loss: 1.0663 - accuracy: 0.8387
1229/1688 [====================>.........] - ETA: 1s - loss: 1.0595 - accuracy: 0.8406
1248/1688 [=====================>........] - ETA: 1s - loss: 1.0537 - accuracy: 0.8423
1267/1688 [=====================>........] - ETA: 1s - loss: 1.0456 - accuracy: 0.8441
1286/1688 [=====================>........] - ETA: 1s - loss: 1.0388 - accuracy: 0.8459
1306/1688 [======================>.......] - ETA: 1s - loss: 1.0349 - accuracy: 0.8473
1325/1688 [======================>.......] - ETA: 0s - loss: 1.0290 - accuracy: 0.8489
1345/1688 [======================>.......] - ETA: 0s - loss: 1.0210 - accuracy: 0.8506
1364/1688 [=======================>......] - ETA: 0s - loss: 1.0127 - accuracy: 0.8523
1383/1688 [=======================>......] - ETA: 0s - loss: 1.0119 - accuracy: 0.8537
1403/1688 [=======================>......] - ETA: 0s - loss: 1.0065 - accuracy: 0.8552
1423/1688 [========================>.....] - ETA: 0s - loss: 1.0009 - accuracy: 0.8567
1442/1688 [========================>.....] - ETA: 0s - loss: 1.0007 - accuracy: 0.8578
1461/1688 [========================>.....] - ETA: 0s - loss: 0.9940 - accuracy: 0.8589
1481/1688 [=========================>....] - ETA: 0s - loss: 0.9911 - accuracy: 0.8603
1501/1688 [=========================>....] - ETA: 0s - loss: 0.9871 - accuracy: 0.8615
1520/1688 [==========================>...] - ETA: 0s - loss: 0.9819 - accuracy: 0.8626
1539/1688 [==========================>...] - ETA: 0s - loss: 0.9781 - accuracy: 0.8638
1558/1688 [==========================>...] - ETA: 0s - loss: 0.9752 - accuracy: 0.8650
1578/1688 [===========================>..] - ETA: 0s - loss: 0.9727 - accuracy: 0.8661
1597/1688 [===========================>..] - ETA: 0s - loss: 0.9680 - accuracy: 0.8673
1616/1688 [===========================>..] - ETA: 0s - loss: 0.9656 - accuracy: 0.8684
1635/1688 [============================>.] - ETA: 0s - loss: 0.9638 - accuracy: 0.8692
1654/1688 [============================>.] - ETA: 0s - loss: 0.9607 - accuracy: 0.8703
1673/1688 [============================>.] - ETA: 0s - loss: 0.9578 - accuracy: 0.8712
1688/1688 [==============================] - 5s 3ms/step - loss: 0.9555 - accuracy: 0.8720 - val_loss: 0.6048 - val_accuracy: 0.9688
Epoch 2/5

   1/1688 [..............................] - ETA: 4s - loss: 1.5083 - accuracy: 0.9375
  20/1688 [..............................] - ETA: 4s - loss: 0.6111 - accuracy: 0.9594
  40/1688 [..............................] - ETA: 4s - loss: 0.5613 - accuracy: 0.9641
  60/1688 [>.............................] - ETA: 4s - loss: 0.6398 - accuracy: 0.9625
  79/1688 [>.............................] - ETA: 4s - loss: 0.6193 - accuracy: 0.9628
  98/1688 [>.............................] - ETA: 4s - loss: 0.5962 - accuracy: 0.9636
 118/1688 [=>............................] - ETA: 4s - loss: 0.6058 - accuracy: 0.9637
 138/1688 [=>............................] - ETA: 4s - loss: 0.5662 - accuracy: 0.9660
 157/1688 [=>............................] - ETA: 4s - loss: 0.6118 - accuracy: 0.9648
 176/1688 [==>...........................] - ETA: 4s - loss: 0.6004 - accuracy: 0.9645
 195/1688 [==>...........................] - ETA: 3s - loss: 0.5792 - accuracy: 0.9655
 215/1688 [==>...........................] - ETA: 3s - loss: 0.5774 - accuracy: 0.9648
 235/1688 [===>..........................] - ETA: 3s - loss: 0.5580 - accuracy: 0.9656
 255/1688 [===>..........................] - ETA: 3s - loss: 0.5632 - accuracy: 0.9648
 274/1688 [===>..........................] - ETA: 3s - loss: 0.5570 - accuracy: 0.9646
 293/1688 [====>.........................] - ETA: 3s - loss: 0.5789 - accuracy: 0.9641
 312/1688 [====>.........................] - ETA: 3s - loss: 0.5827 - accuracy: 0.9648
 332/1688 [====>.........................] - ETA: 3s - loss: 0.5772 - accuracy: 0.9646
 351/1688 [=====>........................] - ETA: 3s - loss: 0.5750 - accuracy: 0.9651
 370/1688 [=====>........................] - ETA: 3s - loss: 0.5881 - accuracy: 0.9642
 389/1688 [=====>........................] - ETA: 3s - loss: 0.5911 - accuracy: 0.9641
 408/1688 [======>.......................] - ETA: 3s - loss: 0.5807 - accuracy: 0.9642
 427/1688 [======>.......................] - ETA: 3s - loss: 0.5936 - accuracy: 0.9633
 446/1688 [======>.......................] - ETA: 3s - loss: 0.5977 - accuracy: 0.9629
 466/1688 [=======>......................] - ETA: 3s - loss: 0.5941 - accuracy: 0.9630
 485/1688 [=======>......................] - ETA: 3s - loss: 0.5954 - accuracy: 0.9631
 505/1688 [=======>......................] - ETA: 3s - loss: 0.6155 - accuracy: 0.9629
 524/1688 [========>.....................] - ETA: 3s - loss: 0.6036 - accuracy: 0.9632
 544/1688 [========>.....................] - ETA: 3s - loss: 0.5977 - accuracy: 0.9631
 563/1688 [=========>....................] - ETA: 2s - loss: 0.5968 - accuracy: 0.9631
 583/1688 [=========>....................] - ETA: 2s - loss: 0.5920 - accuracy: 0.9632
 602/1688 [=========>....................] - ETA: 2s - loss: 0.5888 - accuracy: 0.9632
 621/1688 [==========>...................] - ETA: 2s - loss: 0.5914 - accuracy: 0.9629
 641/1688 [==========>...................] - ETA: 2s - loss: 0.5852 - accuracy: 0.9632
 660/1688 [==========>...................] - ETA: 2s - loss: 0.5863 - accuracy: 0.9633
 679/1688 [===========>..................] - ETA: 2s - loss: 0.5847 - accuracy: 0.9634
 698/1688 [===========>..................] - ETA: 2s - loss: 0.5882 - accuracy: 0.9633
 717/1688 [===========>..................] - ETA: 2s - loss: 0.5993 - accuracy: 0.9631
 737/1688 [============>.................] - ETA: 2s - loss: 0.6049 - accuracy: 0.9628
 756/1688 [============>.................] - ETA: 2s - loss: 0.6026 - accuracy: 0.9628
 775/1688 [============>.................] - ETA: 2s - loss: 0.6049 - accuracy: 0.9629
 794/1688 [=============>................] - ETA: 2s - loss: 0.6030 - accuracy: 0.9629
 814/1688 [=============>................] - ETA: 2s - loss: 0.6055 - accuracy: 0.9627
 833/1688 [=============>................] - ETA: 2s - loss: 0.6079 - accuracy: 0.9626
 852/1688 [==============>...............] - ETA: 2s - loss: 0.6032 - accuracy: 0.9628
 871/1688 [==============>...............] - ETA: 2s - loss: 0.5959 - accuracy: 0.9630
 890/1688 [==============>...............] - ETA: 2s - loss: 0.6026 - accuracy: 0.9630
 909/1688 [===============>..............] - ETA: 2s - loss: 0.5983 - accuracy: 0.9634
 928/1688 [===============>..............] - ETA: 2s - loss: 0.5958 - accuracy: 0.9636
 947/1688 [===============>..............] - ETA: 1s - loss: 0.5915 - accuracy: 0.9636
 966/1688 [================>.............] - ETA: 1s - loss: 0.5915 - accuracy: 0.9635
 986/1688 [================>.............] - ETA: 1s - loss: 0.5914 - accuracy: 0.9636
1005/1688 [================>.............] - ETA: 1s - loss: 0.5912 - accuracy: 0.9637
1024/1688 [=================>............] - ETA: 1s - loss: 0.5945 - accuracy: 0.9635
1043/1688 [=================>............] - ETA: 1s - loss: 0.5952 - accuracy: 0.9634
1062/1688 [=================>............] - ETA: 1s - loss: 0.6008 - accuracy: 0.9634
1081/1688 [==================>...........] - ETA: 1s - loss: 0.5977 - accuracy: 0.9636
1100/1688 [==================>...........] - ETA: 1s - loss: 0.5950 - accuracy: 0.9637
1119/1688 [==================>...........] - ETA: 1s - loss: 0.5928 - accuracy: 0.9638
1138/1688 [===================>..........] - ETA: 1s - loss: 0.5960 - accuracy: 0.9637
1157/1688 [===================>..........] - ETA: 1s - loss: 0.5995 - accuracy: 0.9636
1177/1688 [===================>..........] - ETA: 1s - loss: 0.5990 - accuracy: 0.9638
1196/1688 [====================>.........] - ETA: 1s - loss: 0.6021 - accuracy: 0.9637
1215/1688 [====================>.........] - ETA: 1s - loss: 0.6027 - accuracy: 0.9637
1234/1688 [====================>.........] - ETA: 1s - loss: 0.6042 - accuracy: 0.9637
1253/1688 [=====================>........] - ETA: 1s - loss: 0.6012 - accuracy: 0.9638
1272/1688 [=====================>........] - ETA: 1s - loss: 0.5987 - accuracy: 0.9639
1291/1688 [=====================>........] - ETA: 1s - loss: 0.6057 - accuracy: 0.9637
1310/1688 [======================>.......] - ETA: 1s - loss: 0.6089 - accuracy: 0.9637
1329/1688 [======================>.......] - ETA: 0s - loss: 0.6071 - accuracy: 0.9637
1348/1688 [======================>.......] - ETA: 0s - loss: 0.6074 - accuracy: 0.9638
1367/1688 [=======================>......] - ETA: 0s - loss: 0.6119 - accuracy: 0.9638
1386/1688 [=======================>......] - ETA: 0s - loss: 0.6080 - accuracy: 0.9639
1405/1688 [=======================>......] - ETA: 0s - loss: 0.6102 - accuracy: 0.9638
1424/1688 [========================>.....] - ETA: 0s - loss: 0.6093 - accuracy: 0.9639
1443/1688 [========================>.....] - ETA: 0s - loss: 0.6063 - accuracy: 0.9640
1462/1688 [========================>.....] - ETA: 0s - loss: 0.6073 - accuracy: 0.9640
1482/1688 [=========================>....] - ETA: 0s - loss: 0.6064 - accuracy: 0.9641
1501/1688 [=========================>....] - ETA: 0s - loss: 0.6076 - accuracy: 0.9641
1520/1688 [==========================>...] - ETA: 0s - loss: 0.6089 - accuracy: 0.9640
1540/1688 [==========================>...] - ETA: 0s - loss: 0.6057 - accuracy: 0.9641
1559/1688 [==========================>...] - ETA: 0s - loss: 0.6027 - accuracy: 0.9643
1578/1688 [===========================>..] - ETA: 0s - loss: 0.6037 - accuracy: 0.9643
1597/1688 [===========================>..] - ETA: 0s - loss: 0.6014 - accuracy: 0.9644
1616/1688 [===========================>..] - ETA: 0s - loss: 0.5999 - accuracy: 0.9646
1635/1688 [============================>.] - ETA: 0s - loss: 0.6035 - accuracy: 0.9646
1654/1688 [============================>.] - ETA: 0s - loss: 0.6014 - accuracy: 0.9647
1673/1688 [============================>.] - ETA: 0s - loss: 0.6014 - accuracy: 0.9647
1688/1688 [==============================] - 5s 3ms/step - loss: 0.6004 - accuracy: 0.9648 - val_loss: 0.7541 - val_accuracy: 0.9695
Epoch 3/5

   1/1688 [..............................] - ETA: 4s - loss: 0.7020 - accuracy: 0.9688
  21/1688 [..............................] - ETA: 4s - loss: 0.5888 - accuracy: 0.9673
  41/1688 [..............................] - ETA: 4s - loss: 0.5305 - accuracy: 0.9703
  60/1688 [>.............................] - ETA: 4s - loss: 0.5717 - accuracy: 0.9661
  79/1688 [>.............................] - ETA: 4s - loss: 0.5375 - accuracy: 0.9672
  98/1688 [>.............................] - ETA: 4s - loss: 0.5005 - accuracy: 0.9688
 117/1688 [=>............................] - ETA: 4s - loss: 0.5018 - accuracy: 0.9693
 136/1688 [=>............................] - ETA: 4s - loss: 0.5114 - accuracy: 0.9694
 155/1688 [=>............................] - ETA: 4s - loss: 0.5463 - accuracy: 0.9675
 174/1688 [==>...........................] - ETA: 4s - loss: 0.5436 - accuracy: 0.9682
 193/1688 [==>...........................] - ETA: 3s - loss: 0.5427 - accuracy: 0.9688
 212/1688 [==>...........................] - ETA: 3s - loss: 0.5275 - accuracy: 0.9692
 231/1688 [===>..........................] - ETA: 3s - loss: 0.5120 - accuracy: 0.9696
 251/1688 [===>..........................] - ETA: 3s - loss: 0.5072 - accuracy: 0.9699
 270/1688 [===>..........................] - ETA: 3s - loss: 0.4963 - accuracy: 0.9703
 289/1688 [====>.........................] - ETA: 3s - loss: 0.5085 - accuracy: 0.9699
 308/1688 [====>.........................] - ETA: 3s - loss: 0.4982 - accuracy: 0.9702
 328/1688 [====>.........................] - ETA: 3s - loss: 0.5091 - accuracy: 0.9696
 347/1688 [=====>........................] - ETA: 3s - loss: 0.5169 - accuracy: 0.9695
 367/1688 [=====>........................] - ETA: 3s - loss: 0.5249 - accuracy: 0.9699
 386/1688 [=====>........................] - ETA: 3s - loss: 0.5282 - accuracy: 0.9700
 405/1688 [======>.......................] - ETA: 3s - loss: 0.5303 - accuracy: 0.9698
 424/1688 [======>.......................] - ETA: 3s - loss: 0.5276 - accuracy: 0.9703
 444/1688 [======>.......................] - ETA: 3s - loss: 0.5221 - accuracy: 0.9704
 463/1688 [=======>......................] - ETA: 3s - loss: 0.5368 - accuracy: 0.9700
 482/1688 [=======>......................] - ETA: 3s - loss: 0.5390 - accuracy: 0.9699
 501/1688 [=======>......................] - ETA: 3s - loss: 0.5527 - accuracy: 0.9696
 520/1688 [========>.....................] - ETA: 3s - loss: 0.5720 - accuracy: 0.9694
 539/1688 [========>.....................] - ETA: 3s - loss: 0.5585 - accuracy: 0.9697
 558/1688 [========>.....................] - ETA: 3s - loss: 0.5479 - accuracy: 0.9700
 577/1688 [=========>....................] - ETA: 2s - loss: 0.5407 - accuracy: 0.9704
 597/1688 [=========>....................] - ETA: 2s - loss: 0.5579 - accuracy: 0.9701
 616/1688 [=========>....................] - ETA: 2s - loss: 0.5578 - accuracy: 0.9701
 635/1688 [==========>...................] - ETA: 2s - loss: 0.5513 - accuracy: 0.9703
 654/1688 [==========>...................] - ETA: 2s - loss: 0.5494 - accuracy: 0.9707
 673/1688 [==========>...................] - ETA: 2s - loss: 0.5501 - accuracy: 0.9704
 692/1688 [===========>..................] - ETA: 2s - loss: 0.5451 - accuracy: 0.9704
 711/1688 [===========>..................] - ETA: 2s - loss: 0.5380 - accuracy: 0.9705
 731/1688 [===========>..................] - ETA: 2s - loss: 0.5400 - accuracy: 0.9703
 750/1688 [============>.................] - ETA: 2s - loss: 0.5374 - accuracy: 0.9704
 769/1688 [============>.................] - ETA: 2s - loss: 0.5352 - accuracy: 0.9707
 788/1688 [=============>................] - ETA: 2s - loss: 0.5371 - accuracy: 0.9706
 807/1688 [=============>................] - ETA: 2s - loss: 0.5377 - accuracy: 0.9707
 826/1688 [=============>................] - ETA: 2s - loss: 0.5385 - accuracy: 0.9708
 845/1688 [==============>...............] - ETA: 2s - loss: 0.5377 - accuracy: 0.9708
 864/1688 [==============>...............] - ETA: 2s - loss: 0.5414 - accuracy: 0.9706
 884/1688 [==============>...............] - ETA: 2s - loss: 0.5410 - accuracy: 0.9705
 903/1688 [===============>..............] - ETA: 2s - loss: 0.5391 - accuracy: 0.9705
 922/1688 [===============>..............] - ETA: 2s - loss: 0.5480 - accuracy: 0.9704
 941/1688 [===============>..............] - ETA: 1s - loss: 0.5502 - accuracy: 0.9704
 960/1688 [================>.............] - ETA: 1s - loss: 0.5568 - accuracy: 0.9703
 979/1688 [================>.............] - ETA: 1s - loss: 0.5562 - accuracy: 0.9702
 999/1688 [================>.............] - ETA: 1s - loss: 0.5558 - accuracy: 0.9701
1018/1688 [=================>............] - ETA: 1s - loss: 0.5567 - accuracy: 0.9702
1037/1688 [=================>............] - ETA: 1s - loss: 0.5526 - accuracy: 0.9701
1056/1688 [=================>............] - ETA: 1s - loss: 0.5515 - accuracy: 0.9700
1075/1688 [==================>...........] - ETA: 1s - loss: 0.5564 - accuracy: 0.9700
1095/1688 [==================>...........] - ETA: 1s - loss: 0.5627 - accuracy: 0.9699
1114/1688 [==================>...........] - ETA: 1s - loss: 0.5603 - accuracy: 0.9699
1133/1688 [===================>..........] - ETA: 1s - loss: 0.5580 - accuracy: 0.9700
1153/1688 [===================>..........] - ETA: 1s - loss: 0.5599 - accuracy: 0.9701
1173/1688 [===================>..........] - ETA: 1s - loss: 0.5649 - accuracy: 0.9700
1192/1688 [====================>.........] - ETA: 1s - loss: 0.5662 - accuracy: 0.9700
1211/1688 [====================>.........] - ETA: 1s - loss: 0.5664 - accuracy: 0.9700
1230/1688 [====================>.........] - ETA: 1s - loss: 0.5661 - accuracy: 0.9700
1249/1688 [=====================>........] - ETA: 1s - loss: 0.5669 - accuracy: 0.9700
1268/1688 [=====================>........] - ETA: 1s - loss: 0.5648 - accuracy: 0.9702
1287/1688 [=====================>........] - ETA: 1s - loss: 0.5683 - accuracy: 0.9700
1306/1688 [======================>.......] - ETA: 1s - loss: 0.5660 - accuracy: 0.9700
1325/1688 [======================>.......] - ETA: 0s - loss: 0.5658 - accuracy: 0.9699
1344/1688 [======================>.......] - ETA: 0s - loss: 0.5636 - accuracy: 0.9701
1363/1688 [=======================>......] - ETA: 0s - loss: 0.5719 - accuracy: 0.9700
1382/1688 [=======================>......] - ETA: 0s - loss: 0.5735 - accuracy: 0.9700
1401/1688 [=======================>......] - ETA: 0s - loss: 0.5758 - accuracy: 0.9700
1420/1688 [========================>.....] - ETA: 0s - loss: 0.5740 - accuracy: 0.9701
1439/1688 [========================>.....] - ETA: 0s - loss: 0.5777 - accuracy: 0.9701
1458/1688 [========================>.....] - ETA: 0s - loss: 0.5776 - accuracy: 0.9702
1478/1688 [=========================>....] - ETA: 0s - loss: 0.5812 - accuracy: 0.9701
1497/1688 [=========================>....] - ETA: 0s - loss: 0.5791 - accuracy: 0.9701
1516/1688 [=========================>....] - ETA: 0s - loss: 0.5765 - accuracy: 0.9702
1535/1688 [==========================>...] - ETA: 0s - loss: 0.5788 - accuracy: 0.9702
1554/1688 [==========================>...] - ETA: 0s - loss: 0.5791 - accuracy: 0.9701
1573/1688 [==========================>...] - ETA: 0s - loss: 0.5800 - accuracy: 0.9701
1593/1688 [===========================>..] - ETA: 0s - loss: 0.5806 - accuracy: 0.9701
1612/1688 [===========================>..] - ETA: 0s - loss: 0.5808 - accuracy: 0.9702
1632/1688 [============================>.] - ETA: 0s - loss: 0.5778 - accuracy: 0.9703
1651/1688 [============================>.] - ETA: 0s - loss: 0.5788 - accuracy: 0.9702
1670/1688 [============================>.] - ETA: 0s - loss: 0.5763 - accuracy: 0.9702
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5746 - accuracy: 0.9702 - val_loss: 0.6071 - val_accuracy: 0.9743
Epoch 4/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0535 - accuracy: 0.9688
  21/1688 [..............................] - ETA: 4s - loss: 0.3352 - accuracy: 0.9821
  41/1688 [..............................] - ETA: 4s - loss: 0.3516 - accuracy: 0.9794
  60/1688 [>.............................] - ETA: 4s - loss: 0.4238 - accuracy: 0.9776
  79/1688 [>.............................] - ETA: 4s - loss: 0.4734 - accuracy: 0.9751
  98/1688 [>.............................] - ETA: 4s - loss: 0.5166 - accuracy: 0.9735
 117/1688 [=>............................] - ETA: 4s - loss: 0.4890 - accuracy: 0.9752
 136/1688 [=>............................] - ETA: 4s - loss: 0.5029 - accuracy: 0.9752
 155/1688 [=>............................] - ETA: 4s - loss: 0.4879 - accuracy: 0.9750
 174/1688 [==>...........................] - ETA: 4s - loss: 0.4574 - accuracy: 0.9758
 193/1688 [==>...........................] - ETA: 3s - loss: 0.4320 - accuracy: 0.9767
 212/1688 [==>...........................] - ETA: 3s - loss: 0.4257 - accuracy: 0.9760
 231/1688 [===>..........................] - ETA: 3s - loss: 0.4216 - accuracy: 0.9759
 251/1688 [===>..........................] - ETA: 3s - loss: 0.4344 - accuracy: 0.9755
 270/1688 [===>..........................] - ETA: 3s - loss: 0.4318 - accuracy: 0.9753
 289/1688 [====>.........................] - ETA: 3s - loss: 0.4263 - accuracy: 0.9759
 308/1688 [====>.........................] - ETA: 3s - loss: 0.4193 - accuracy: 0.9760
 327/1688 [====>.........................] - ETA: 3s - loss: 0.4376 - accuracy: 0.9753
 346/1688 [=====>........................] - ETA: 3s - loss: 0.4550 - accuracy: 0.9751
 365/1688 [=====>........................] - ETA: 3s - loss: 0.4606 - accuracy: 0.9748
 384/1688 [=====>........................] - ETA: 3s - loss: 0.4572 - accuracy: 0.9748
 403/1688 [======>.......................] - ETA: 3s - loss: 0.4602 - accuracy: 0.9746
 422/1688 [======>.......................] - ETA: 3s - loss: 0.4591 - accuracy: 0.9745
 442/1688 [======>.......................] - ETA: 3s - loss: 0.4612 - accuracy: 0.9745
 461/1688 [=======>......................] - ETA: 3s - loss: 0.4706 - accuracy: 0.9741
 481/1688 [=======>......................] - ETA: 3s - loss: 0.4637 - accuracy: 0.9743
 500/1688 [=======>......................] - ETA: 3s - loss: 0.4670 - accuracy: 0.9742
 519/1688 [========>.....................] - ETA: 3s - loss: 0.4718 - accuracy: 0.9736
 539/1688 [========>.....................] - ETA: 3s - loss: 0.4683 - accuracy: 0.9739
 558/1688 [========>.....................] - ETA: 2s - loss: 0.4920 - accuracy: 0.9732
 577/1688 [=========>....................] - ETA: 2s - loss: 0.4933 - accuracy: 0.9726
 596/1688 [=========>....................] - ETA: 2s - loss: 0.4910 - accuracy: 0.9728
 615/1688 [=========>....................] - ETA: 2s - loss: 0.4891 - accuracy: 0.9727
 634/1688 [==========>...................] - ETA: 2s - loss: 0.4889 - accuracy: 0.9726
 653/1688 [==========>...................] - ETA: 2s - loss: 0.4916 - accuracy: 0.9727
 672/1688 [==========>...................] - ETA: 2s - loss: 0.4923 - accuracy: 0.9727
 691/1688 [===========>..................] - ETA: 2s - loss: 0.4933 - accuracy: 0.9727
 711/1688 [===========>..................] - ETA: 2s - loss: 0.4914 - accuracy: 0.9727
 730/1688 [===========>..................] - ETA: 2s - loss: 0.4955 - accuracy: 0.9726
 750/1688 [============>.................] - ETA: 2s - loss: 0.4967 - accuracy: 0.9726
 769/1688 [============>.................] - ETA: 2s - loss: 0.4934 - accuracy: 0.9728
 789/1688 [=============>................] - ETA: 2s - loss: 0.4941 - accuracy: 0.9727
 808/1688 [=============>................] - ETA: 2s - loss: 0.4924 - accuracy: 0.9728
 827/1688 [=============>................] - ETA: 2s - loss: 0.5065 - accuracy: 0.9724
 846/1688 [==============>...............] - ETA: 2s - loss: 0.5048 - accuracy: 0.9725
 865/1688 [==============>...............] - ETA: 2s - loss: 0.5109 - accuracy: 0.9724
 884/1688 [==============>...............] - ETA: 2s - loss: 0.5093 - accuracy: 0.9724
 903/1688 [===============>..............] - ETA: 2s - loss: 0.5149 - accuracy: 0.9723
 923/1688 [===============>..............] - ETA: 2s - loss: 0.5104 - accuracy: 0.9724
 942/1688 [===============>..............] - ETA: 1s - loss: 0.5126 - accuracy: 0.9722
 961/1688 [================>.............] - ETA: 1s - loss: 0.5183 - accuracy: 0.9720
 981/1688 [================>.............] - ETA: 1s - loss: 0.5149 - accuracy: 0.9721
1000/1688 [================>.............] - ETA: 1s - loss: 0.5189 - accuracy: 0.9720
1019/1688 [=================>............] - ETA: 1s - loss: 0.5166 - accuracy: 0.9721
1038/1688 [=================>............] - ETA: 1s - loss: 0.5189 - accuracy: 0.9721
1057/1688 [=================>............] - ETA: 1s - loss: 0.5189 - accuracy: 0.9722
1076/1688 [==================>...........] - ETA: 1s - loss: 0.5193 - accuracy: 0.9722
1096/1688 [==================>...........] - ETA: 1s - loss: 0.5241 - accuracy: 0.9720
1115/1688 [==================>...........] - ETA: 1s - loss: 0.5187 - accuracy: 0.9722
1134/1688 [===================>..........] - ETA: 1s - loss: 0.5222 - accuracy: 0.9721
1153/1688 [===================>..........] - ETA: 1s - loss: 0.5241 - accuracy: 0.9720
1173/1688 [===================>..........] - ETA: 1s - loss: 0.5271 - accuracy: 0.9719
1192/1688 [====================>.........] - ETA: 1s - loss: 0.5300 - accuracy: 0.9719
1211/1688 [====================>.........] - ETA: 1s - loss: 0.5296 - accuracy: 0.9720
1230/1688 [====================>.........] - ETA: 1s - loss: 0.5268 - accuracy: 0.9720
1249/1688 [=====================>........] - ETA: 1s - loss: 0.5245 - accuracy: 0.9722
1268/1688 [=====================>........] - ETA: 1s - loss: 0.5239 - accuracy: 0.9722
1287/1688 [=====================>........] - ETA: 1s - loss: 0.5231 - accuracy: 0.9722
1306/1688 [======================>.......] - ETA: 1s - loss: 0.5292 - accuracy: 0.9722
1325/1688 [======================>.......] - ETA: 0s - loss: 0.5331 - accuracy: 0.9721
1344/1688 [======================>.......] - ETA: 0s - loss: 0.5340 - accuracy: 0.9720
1363/1688 [=======================>......] - ETA: 0s - loss: 0.5317 - accuracy: 0.9721
1382/1688 [=======================>......] - ETA: 0s - loss: 0.5297 - accuracy: 0.9720
1402/1688 [=======================>......] - ETA: 0s - loss: 0.5283 - accuracy: 0.9721
1421/1688 [========================>.....] - ETA: 0s - loss: 0.5257 - accuracy: 0.9721
1441/1688 [========================>.....] - ETA: 0s - loss: 0.5262 - accuracy: 0.9723
1460/1688 [========================>.....] - ETA: 0s - loss: 0.5260 - accuracy: 0.9725
1479/1688 [=========================>....] - ETA: 0s - loss: 0.5294 - accuracy: 0.9724
1499/1688 [=========================>....] - ETA: 0s - loss: 0.5304 - accuracy: 0.9725
1518/1688 [=========================>....] - ETA: 0s - loss: 0.5279 - accuracy: 0.9725
1537/1688 [==========================>...] - ETA: 0s - loss: 0.5287 - accuracy: 0.9725
1556/1688 [==========================>...] - ETA: 0s - loss: 0.5274 - accuracy: 0.9726
1575/1688 [==========================>...] - ETA: 0s - loss: 0.5282 - accuracy: 0.9725
1594/1688 [===========================>..] - ETA: 0s - loss: 0.5266 - accuracy: 0.9725
1613/1688 [===========================>..] - ETA: 0s - loss: 0.5274 - accuracy: 0.9725
1632/1688 [============================>.] - ETA: 0s - loss: 0.5245 - accuracy: 0.9726
1651/1688 [============================>.] - ETA: 0s - loss: 0.5285 - accuracy: 0.9725
1670/1688 [============================>.] - ETA: 0s - loss: 0.5284 - accuracy: 0.9726
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5284 - accuracy: 0.9726 - val_loss: 0.7504 - val_accuracy: 0.9713
Epoch 5/5

   1/1688 [..............................] - ETA: 4s - loss: 0.0121 - accuracy: 1.0000
  20/1688 [..............................] - ETA: 4s - loss: 0.4761 - accuracy: 0.9750
  40/1688 [..............................] - ETA: 4s - loss: 0.7435 - accuracy: 0.9742
  59/1688 [>.............................] - ETA: 4s - loss: 0.6715 - accuracy: 0.9746
  79/1688 [>.............................] - ETA: 4s - loss: 0.6209 - accuracy: 0.9751
  98/1688 [>.............................] - ETA: 4s - loss: 0.6231 - accuracy: 0.9748
 117/1688 [=>............................] - ETA: 4s - loss: 0.5633 - accuracy: 0.9760
 136/1688 [=>............................] - ETA: 4s - loss: 0.5346 - accuracy: 0.9756
 155/1688 [=>............................] - ETA: 4s - loss: 0.5332 - accuracy: 0.9764
 174/1688 [==>...........................] - ETA: 4s - loss: 0.5133 - accuracy: 0.9772
 193/1688 [==>...........................] - ETA: 3s - loss: 0.5337 - accuracy: 0.9772
 213/1688 [==>...........................] - ETA: 3s - loss: 0.4992 - accuracy: 0.9778
 233/1688 [===>..........................] - ETA: 3s - loss: 0.4724 - accuracy: 0.9783
 253/1688 [===>..........................] - ETA: 3s - loss: 0.4689 - accuracy: 0.9781
 272/1688 [===>..........................] - ETA: 3s - loss: 0.4720 - accuracy: 0.9781
 291/1688 [====>.........................] - ETA: 3s - loss: 0.4757 - accuracy: 0.9781
 310/1688 [====>.........................] - ETA: 3s - loss: 0.4701 - accuracy: 0.9784
 330/1688 [====>.........................] - ETA: 3s - loss: 0.4607 - accuracy: 0.9781
 349/1688 [=====>........................] - ETA: 3s - loss: 0.4492 - accuracy: 0.9780
 368/1688 [=====>........................] - ETA: 3s - loss: 0.4524 - accuracy: 0.9778
 387/1688 [=====>........................] - ETA: 3s - loss: 0.4523 - accuracy: 0.9780
 406/1688 [======>.......................] - ETA: 3s - loss: 0.4556 - accuracy: 0.9781
 425/1688 [======>.......................] - ETA: 3s - loss: 0.4736 - accuracy: 0.9782
 445/1688 [======>.......................] - ETA: 3s - loss: 0.4882 - accuracy: 0.9780
 464/1688 [=======>......................] - ETA: 3s - loss: 0.4954 - accuracy: 0.9774
 483/1688 [=======>......................] - ETA: 3s - loss: 0.4931 - accuracy: 0.9775
 502/1688 [=======>......................] - ETA: 3s - loss: 0.4891 - accuracy: 0.9775
 521/1688 [========>.....................] - ETA: 3s - loss: 0.4832 - accuracy: 0.9773
 540/1688 [========>.....................] - ETA: 3s - loss: 0.4886 - accuracy: 0.9768
 560/1688 [========>.....................] - ETA: 3s - loss: 0.4906 - accuracy: 0.9768
 579/1688 [=========>....................] - ETA: 2s - loss: 0.4896 - accuracy: 0.9770
 598/1688 [=========>....................] - ETA: 2s - loss: 0.4886 - accuracy: 0.9770
 617/1688 [=========>....................] - ETA: 2s - loss: 0.4888 - accuracy: 0.9769
 636/1688 [==========>...................] - ETA: 2s - loss: 0.4961 - accuracy: 0.9765
 655/1688 [==========>...................] - ETA: 2s - loss: 0.4945 - accuracy: 0.9762
 675/1688 [==========>...................] - ETA: 2s - loss: 0.4967 - accuracy: 0.9761
 694/1688 [===========>..................] - ETA: 2s - loss: 0.5044 - accuracy: 0.9760
 714/1688 [===========>..................] - ETA: 2s - loss: 0.5058 - accuracy: 0.9758
 733/1688 [============>.................] - ETA: 2s - loss: 0.5093 - accuracy: 0.9755
 752/1688 [============>.................] - ETA: 2s - loss: 0.5073 - accuracy: 0.9753
 771/1688 [============>.................] - ETA: 2s - loss: 0.5057 - accuracy: 0.9754
 790/1688 [=============>................] - ETA: 2s - loss: 0.5093 - accuracy: 0.9753
 809/1688 [=============>................] - ETA: 2s - loss: 0.5033 - accuracy: 0.9755
 828/1688 [=============>................] - ETA: 2s - loss: 0.5103 - accuracy: 0.9753
 847/1688 [==============>...............] - ETA: 2s - loss: 0.5068 - accuracy: 0.9754
 866/1688 [==============>...............] - ETA: 2s - loss: 0.5110 - accuracy: 0.9752
 885/1688 [==============>...............] - ETA: 2s - loss: 0.5179 - accuracy: 0.9749
 904/1688 [===============>..............] - ETA: 2s - loss: 0.5216 - accuracy: 0.9749
 923/1688 [===============>..............] - ETA: 2s - loss: 0.5251 - accuracy: 0.9748
 943/1688 [===============>..............] - ETA: 1s - loss: 0.5208 - accuracy: 0.9750
 961/1688 [================>.............] - ETA: 1s - loss: 0.5181 - accuracy: 0.9751
 981/1688 [================>.............] - ETA: 1s - loss: 0.5213 - accuracy: 0.9752
1001/1688 [================>.............] - ETA: 1s - loss: 0.5159 - accuracy: 0.9754
1020/1688 [=================>............] - ETA: 1s - loss: 0.5171 - accuracy: 0.9753
1039/1688 [=================>............] - ETA: 1s - loss: 0.5214 - accuracy: 0.9752
1058/1688 [=================>............] - ETA: 1s - loss: 0.5327 - accuracy: 0.9749
1077/1688 [==================>...........] - ETA: 1s - loss: 0.5342 - accuracy: 0.9748
1096/1688 [==================>...........] - ETA: 1s - loss: 0.5334 - accuracy: 0.9748
1115/1688 [==================>...........] - ETA: 1s - loss: 0.5386 - accuracy: 0.9746
1135/1688 [===================>..........] - ETA: 1s - loss: 0.5340 - accuracy: 0.9748
1155/1688 [===================>..........] - ETA: 1s - loss: 0.5346 - accuracy: 0.9747
1174/1688 [===================>..........] - ETA: 1s - loss: 0.5349 - accuracy: 0.9746
1193/1688 [====================>.........] - ETA: 1s - loss: 0.5366 - accuracy: 0.9745
1212/1688 [====================>.........] - ETA: 1s - loss: 0.5432 - accuracy: 0.9743
1231/1688 [====================>.........] - ETA: 1s - loss: 0.5428 - accuracy: 0.9743
1251/1688 [=====================>........] - ETA: 1s - loss: 0.5419 - accuracy: 0.9743
1270/1688 [=====================>........] - ETA: 1s - loss: 0.5380 - accuracy: 0.9745
1289/1688 [=====================>........] - ETA: 1s - loss: 0.5331 - accuracy: 0.9746
1308/1688 [======================>.......] - ETA: 1s - loss: 0.5311 - accuracy: 0.9746
1327/1688 [======================>.......] - ETA: 0s - loss: 0.5352 - accuracy: 0.9744
1346/1688 [======================>.......] - ETA: 0s - loss: 0.5351 - accuracy: 0.9745
1365/1688 [=======================>......] - ETA: 0s - loss: 0.5354 - accuracy: 0.9745
1384/1688 [=======================>......] - ETA: 0s - loss: 0.5327 - accuracy: 0.9746
1404/1688 [=======================>......] - ETA: 0s - loss: 0.5314 - accuracy: 0.9746
1424/1688 [========================>.....] - ETA: 0s - loss: 0.5276 - accuracy: 0.9748
1443/1688 [========================>.....] - ETA: 0s - loss: 0.5310 - accuracy: 0.9746
1462/1688 [========================>.....] - ETA: 0s - loss: 0.5347 - accuracy: 0.9746
1481/1688 [=========================>....] - ETA: 0s - loss: 0.5355 - accuracy: 0.9746
1500/1688 [=========================>....] - ETA: 0s - loss: 0.5357 - accuracy: 0.9746
1520/1688 [==========================>...] - ETA: 0s - loss: 0.5334 - accuracy: 0.9747
1539/1688 [==========================>...] - ETA: 0s - loss: 0.5320 - accuracy: 0.9746
1559/1688 [==========================>...] - ETA: 0s - loss: 0.5321 - accuracy: 0.9746
1578/1688 [===========================>..] - ETA: 0s - loss: 0.5338 - accuracy: 0.9747
1597/1688 [===========================>..] - ETA: 0s - loss: 0.5316 - accuracy: 0.9748
1616/1688 [===========================>..] - ETA: 0s - loss: 0.5307 - accuracy: 0.9748
1636/1688 [============================>.] - ETA: 0s - loss: 0.5310 - accuracy: 0.9746
1655/1688 [============================>.] - ETA: 0s - loss: 0.5272 - accuracy: 0.9747
1674/1688 [============================>.] - ETA: 0s - loss: 0.5277 - accuracy: 0.9747
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5298 - accuracy: 0.9746 - val_loss: 0.8022 - val_accuracy: 0.9705
Test accuracy after fine tuning: 0.9660000205039978

6. Model conversion

After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida software simulator.

Note

One needs to supply the coefficients used to rescale the input dataset before the training - here input_scaling.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()

accuracy = model_akida.evaluate(raw_x_test, raw_y_test)
print('Test accuracy after conversion:', accuracy)

# For non-regression purpose
assert accuracy > 0.96
                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          3
______________________________________________

______________________________________________________________
Layer (type)                  Output shape  Kernel shape

================= SW/conv2d-dense (Software) =================

conv2d (InputConv.)           [13, 13, 32]  (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.)  [7, 7, 64]    (3, 3, 32, 1)
______________________________________________________________
                                            (1, 1, 32, 64)
______________________________________________________________
dense (Fully.)                [1, 1, 10]    (1, 1, 3136, 10)
______________________________________________________________
Test accuracy after conversion: 0.964900016784668

Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).

Total running time of the script: ( 1 minutes 2.601 seconds)

Gallery generated by Sphinx-Gallery