Note
Click here to download the full example code
CNN conversion flow tutorial
This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.
The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.
Note
Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.
MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.
1. Load and reshape MNIST dataset
After loading, we make 2 transformations on the dataset:
Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.
Note
At this point, we’ll set aside the raw data for testing our converted model in the Akida Execution Engine later.
Rescale the 8-bit loaded data to the range 0-to-1 for training.
Note
Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.
This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.
Also note that we store the scaling values input_scaling for
use when preparing the model for the Akida Execution Engine. The
implementation of the Akida neural network allows us to completely
skip the rescaling step (i.e. the Akida model should be fed with
the raw 8-bit values) but that does require information about what
scaling was applied prior to training - see below for more details.
import tensorflow as tf
from tensorflow import keras
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)
# Set aside raw test data for use with Akida Execution Engine later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test
# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a
2. Model definition
Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.
An appropriate model for MNIST (inspired by this example) might look something like the following:
model_keras = keras.models.Sequential([
keras.layers.Conv2D(filters=32, kernel_size=3, input_shape=(28, 28, 1)),
keras.layers.MaxPool2D(),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.SeparableConv2D(filters=64, kernel_size=3, padding='same'),
keras.layers.MaxPool2D(padding='same'),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Out:
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 26, 26, 32) 320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32) 0
_________________________________________________________________
batch_normalization (BatchNo (None, 13, 13, 32) 128
_________________________________________________________________
re_lu (ReLU) (None, 13, 13, 32) 0
_________________________________________________________________
separable_conv2d (SeparableC (None, 13, 13, 64) 2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64) 0
_________________________________________________________________
batch_normalization_1 (Batch (None, 7, 7, 64) 256
_________________________________________________________________
re_lu_1 (ReLU) (None, 7, 7, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 3136) 0
_________________________________________________________________
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________
The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):
Standard Conv2D and Dense layers are supported
Hidden layers must be followed by a ReLU layer.
BatchNormalization must always happen before activations.
Convolutional blocks can optionally be followed by a MaxPooling.
The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).
from cnn2snn import check_model_compatibility
print("Model compatible for Akida conversion:",
check_model_compatibility(model_keras))
Out:
Model compatible for Akida conversion: True
3. Model training
Before going any further, train the model and get its performance. The created model should have achieved a test accuracy a little over 99% after 10 epochs.
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])
Out:
Epoch 1/10
1/1688 [..............................] - ETA: 11:45 - loss: 2.9876 - accuracy: 0.0625
31/1688 [..............................] - ETA: 2s - loss: 1.0383 - accuracy: 0.6845
61/1688 [>.............................] - ETA: 2s - loss: 0.6852 - accuracy: 0.7961
91/1688 [>.............................] - ETA: 2s - loss: 0.5450 - accuracy: 0.8365
122/1688 [=>............................] - ETA: 2s - loss: 0.4669 - accuracy: 0.8604
152/1688 [=>............................] - ETA: 2s - loss: 0.4098 - accuracy: 0.8793
183/1688 [==>...........................] - ETA: 2s - loss: 0.3704 - accuracy: 0.8911
213/1688 [==>...........................] - ETA: 2s - loss: 0.3434 - accuracy: 0.8982
243/1688 [===>..........................] - ETA: 2s - loss: 0.3158 - accuracy: 0.9068
273/1688 [===>..........................] - ETA: 2s - loss: 0.2963 - accuracy: 0.9125
303/1688 [====>.........................] - ETA: 2s - loss: 0.2785 - accuracy: 0.9178
333/1688 [====>.........................] - ETA: 2s - loss: 0.2678 - accuracy: 0.9209
364/1688 [=====>........................] - ETA: 2s - loss: 0.2554 - accuracy: 0.9244
394/1688 [======>.......................] - ETA: 2s - loss: 0.2431 - accuracy: 0.9274
423/1688 [======>.......................] - ETA: 2s - loss: 0.2334 - accuracy: 0.9302
454/1688 [=======>......................] - ETA: 2s - loss: 0.2238 - accuracy: 0.9330
484/1688 [=======>......................] - ETA: 2s - loss: 0.2169 - accuracy: 0.9349
515/1688 [========>.....................] - ETA: 1s - loss: 0.2097 - accuracy: 0.9367
545/1688 [========>.....................] - ETA: 1s - loss: 0.2026 - accuracy: 0.9387
575/1688 [=========>....................] - ETA: 1s - loss: 0.1963 - accuracy: 0.9405
605/1688 [=========>....................] - ETA: 1s - loss: 0.1912 - accuracy: 0.9421
635/1688 [==========>...................] - ETA: 1s - loss: 0.1881 - accuracy: 0.9437
665/1688 [==========>...................] - ETA: 1s - loss: 0.1844 - accuracy: 0.9448
695/1688 [===========>..................] - ETA: 1s - loss: 0.1796 - accuracy: 0.9460
725/1688 [===========>..................] - ETA: 1s - loss: 0.1758 - accuracy: 0.9472
756/1688 [============>.................] - ETA: 1s - loss: 0.1722 - accuracy: 0.9484
787/1688 [============>.................] - ETA: 1s - loss: 0.1691 - accuracy: 0.9492
816/1688 [=============>................] - ETA: 1s - loss: 0.1663 - accuracy: 0.9499
846/1688 [==============>...............] - ETA: 1s - loss: 0.1647 - accuracy: 0.9504
876/1688 [==============>...............] - ETA: 1s - loss: 0.1618 - accuracy: 0.9511
907/1688 [===============>..............] - ETA: 1s - loss: 0.1589 - accuracy: 0.9521
937/1688 [===============>..............] - ETA: 1s - loss: 0.1565 - accuracy: 0.9528
967/1688 [================>.............] - ETA: 1s - loss: 0.1538 - accuracy: 0.9535
997/1688 [================>.............] - ETA: 1s - loss: 0.1515 - accuracy: 0.9541
1027/1688 [=================>............] - ETA: 1s - loss: 0.1491 - accuracy: 0.9548
1057/1688 [=================>............] - ETA: 1s - loss: 0.1471 - accuracy: 0.9553
1087/1688 [==================>...........] - ETA: 1s - loss: 0.1457 - accuracy: 0.9559
1117/1688 [==================>...........] - ETA: 0s - loss: 0.1446 - accuracy: 0.9561
1147/1688 [===================>..........] - ETA: 0s - loss: 0.1421 - accuracy: 0.9568
1177/1688 [===================>..........] - ETA: 0s - loss: 0.1404 - accuracy: 0.9573
1208/1688 [====================>.........] - ETA: 0s - loss: 0.1394 - accuracy: 0.9576
1238/1688 [=====================>........] - ETA: 0s - loss: 0.1377 - accuracy: 0.9581
1269/1688 [=====================>........] - ETA: 0s - loss: 0.1362 - accuracy: 0.9585
1300/1688 [======================>.......] - ETA: 0s - loss: 0.1346 - accuracy: 0.9590
1331/1688 [======================>.......] - ETA: 0s - loss: 0.1323 - accuracy: 0.9597
1361/1688 [=======================>......] - ETA: 0s - loss: 0.1305 - accuracy: 0.9602
1392/1688 [=======================>......] - ETA: 0s - loss: 0.1289 - accuracy: 0.9607
1422/1688 [========================>.....] - ETA: 0s - loss: 0.1276 - accuracy: 0.9612
1452/1688 [========================>.....] - ETA: 0s - loss: 0.1265 - accuracy: 0.9614
1482/1688 [=========================>....] - ETA: 0s - loss: 0.1249 - accuracy: 0.9620
1512/1688 [=========================>....] - ETA: 0s - loss: 0.1239 - accuracy: 0.9623
1541/1688 [==========================>...] - ETA: 0s - loss: 0.1225 - accuracy: 0.9628
1571/1688 [==========================>...] - ETA: 0s - loss: 0.1212 - accuracy: 0.9632
1602/1688 [===========================>..] - ETA: 0s - loss: 0.1207 - accuracy: 0.9634
1631/1688 [===========================>..] - ETA: 0s - loss: 0.1196 - accuracy: 0.9637
1661/1688 [============================>.] - ETA: 0s - loss: 0.1186 - accuracy: 0.9640
1688/1688 [==============================] - ETA: 0s - loss: 0.1177 - accuracy: 0.9643
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1177 - accuracy: 0.9643 - val_loss: 0.0792 - val_accuracy: 0.9803
Epoch 2/10
1/1688 [..............................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9688
33/1688 [..............................] - ETA: 2s - loss: 0.0471 - accuracy: 0.9839
63/1688 [>.............................] - ETA: 2s - loss: 0.0415 - accuracy: 0.9851
94/1688 [>.............................] - ETA: 2s - loss: 0.0405 - accuracy: 0.9864
124/1688 [=>............................] - ETA: 2s - loss: 0.0384 - accuracy: 0.9869
154/1688 [=>............................] - ETA: 2s - loss: 0.0402 - accuracy: 0.9868
184/1688 [==>...........................] - ETA: 2s - loss: 0.0419 - accuracy: 0.9861
215/1688 [==>...........................] - ETA: 2s - loss: 0.0389 - accuracy: 0.9872
245/1688 [===>..........................] - ETA: 2s - loss: 0.0421 - accuracy: 0.9869
276/1688 [===>..........................] - ETA: 2s - loss: 0.0428 - accuracy: 0.9865
299/1688 [====>.........................] - ETA: 2s - loss: 0.0415 - accuracy: 0.9868
328/1688 [====>.........................] - ETA: 2s - loss: 0.0445 - accuracy: 0.9856
356/1688 [=====>........................] - ETA: 2s - loss: 0.0441 - accuracy: 0.9860
385/1688 [=====>........................] - ETA: 2s - loss: 0.0441 - accuracy: 0.9860
415/1688 [======>.......................] - ETA: 2s - loss: 0.0431 - accuracy: 0.9861
443/1688 [======>.......................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9859
473/1688 [=======>......................] - ETA: 2s - loss: 0.0461 - accuracy: 0.9857
503/1688 [=======>......................] - ETA: 2s - loss: 0.0461 - accuracy: 0.9855
533/1688 [========>.....................] - ETA: 1s - loss: 0.0462 - accuracy: 0.9853
563/1688 [=========>....................] - ETA: 1s - loss: 0.0463 - accuracy: 0.9853
593/1688 [=========>....................] - ETA: 1s - loss: 0.0463 - accuracy: 0.9852
623/1688 [==========>...................] - ETA: 1s - loss: 0.0467 - accuracy: 0.9850
653/1688 [==========>...................] - ETA: 1s - loss: 0.0471 - accuracy: 0.9849
683/1688 [===========>..................] - ETA: 1s - loss: 0.0466 - accuracy: 0.9850
713/1688 [===========>..................] - ETA: 1s - loss: 0.0478 - accuracy: 0.9849
743/1688 [============>.................] - ETA: 1s - loss: 0.0474 - accuracy: 0.9851
774/1688 [============>.................] - ETA: 1s - loss: 0.0482 - accuracy: 0.9847
805/1688 [=============>................] - ETA: 1s - loss: 0.0487 - accuracy: 0.9847
835/1688 [=============>................] - ETA: 1s - loss: 0.0493 - accuracy: 0.9845
865/1688 [==============>...............] - ETA: 1s - loss: 0.0487 - accuracy: 0.9847
896/1688 [==============>...............] - ETA: 1s - loss: 0.0484 - accuracy: 0.9847
926/1688 [===============>..............] - ETA: 1s - loss: 0.0485 - accuracy: 0.9848
956/1688 [===============>..............] - ETA: 1s - loss: 0.0487 - accuracy: 0.9848
986/1688 [================>.............] - ETA: 1s - loss: 0.0484 - accuracy: 0.9849
1017/1688 [=================>............] - ETA: 1s - loss: 0.0492 - accuracy: 0.9846
1047/1688 [=================>............] - ETA: 1s - loss: 0.0490 - accuracy: 0.9848
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0490 - accuracy: 0.9848
1107/1688 [==================>...........] - ETA: 0s - loss: 0.0489 - accuracy: 0.9848
1137/1688 [===================>..........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9847
1167/1688 [===================>..........] - ETA: 0s - loss: 0.0496 - accuracy: 0.9845
1197/1688 [====================>.........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9845
1227/1688 [====================>.........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9845
1258/1688 [=====================>........] - ETA: 0s - loss: 0.0494 - accuracy: 0.9845
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9846
1318/1688 [======================>.......] - ETA: 0s - loss: 0.0492 - accuracy: 0.9846
1348/1688 [======================>.......] - ETA: 0s - loss: 0.0491 - accuracy: 0.9847
1377/1688 [=======================>......] - ETA: 0s - loss: 0.0489 - accuracy: 0.9847
1407/1688 [========================>.....] - ETA: 0s - loss: 0.0491 - accuracy: 0.9846
1436/1688 [========================>.....] - ETA: 0s - loss: 0.0488 - accuracy: 0.9846
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0486 - accuracy: 0.9847
1496/1688 [=========================>....] - ETA: 0s - loss: 0.0487 - accuracy: 0.9847
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0488 - accuracy: 0.9846
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0487 - accuracy: 0.9847
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0487 - accuracy: 0.9847
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0487 - accuracy: 0.9847
1645/1688 [============================>.] - ETA: 0s - loss: 0.0486 - accuracy: 0.9847
1676/1688 [============================>.] - ETA: 0s - loss: 0.0482 - accuracy: 0.9848
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0482 - accuracy: 0.9849 - val_loss: 0.0450 - val_accuracy: 0.9877
Epoch 3/10
1/1688 [..............................] - ETA: 2s - loss: 0.0015 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9915
63/1688 [>.............................] - ETA: 2s - loss: 0.0316 - accuracy: 0.9906
93/1688 [>.............................] - ETA: 2s - loss: 0.0283 - accuracy: 0.9909
122/1688 [=>............................] - ETA: 2s - loss: 0.0250 - accuracy: 0.9921
152/1688 [=>............................] - ETA: 2s - loss: 0.0259 - accuracy: 0.9922
182/1688 [==>...........................] - ETA: 2s - loss: 0.0266 - accuracy: 0.9914
212/1688 [==>...........................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9912
242/1688 [===>..........................] - ETA: 2s - loss: 0.0290 - accuracy: 0.9910
272/1688 [===>..........................] - ETA: 2s - loss: 0.0285 - accuracy: 0.9909
302/1688 [====>.........................] - ETA: 2s - loss: 0.0287 - accuracy: 0.9908
332/1688 [====>.........................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9909
362/1688 [=====>........................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9909
392/1688 [=====>........................] - ETA: 2s - loss: 0.0308 - accuracy: 0.9904
422/1688 [======>.......................] - ETA: 2s - loss: 0.0313 - accuracy: 0.9899
452/1688 [=======>......................] - ETA: 2s - loss: 0.0316 - accuracy: 0.9897
482/1688 [=======>......................] - ETA: 2s - loss: 0.0318 - accuracy: 0.9896
512/1688 [========>.....................] - ETA: 1s - loss: 0.0328 - accuracy: 0.9895
542/1688 [========>.....................] - ETA: 1s - loss: 0.0326 - accuracy: 0.9893
572/1688 [=========>....................] - ETA: 1s - loss: 0.0337 - accuracy: 0.9893
603/1688 [=========>....................] - ETA: 1s - loss: 0.0337 - accuracy: 0.9893
633/1688 [==========>...................] - ETA: 1s - loss: 0.0343 - accuracy: 0.9892
663/1688 [==========>...................] - ETA: 1s - loss: 0.0350 - accuracy: 0.9891
693/1688 [===========>..................] - ETA: 1s - loss: 0.0353 - accuracy: 0.9888
723/1688 [===========>..................] - ETA: 1s - loss: 0.0353 - accuracy: 0.9888
753/1688 [============>.................] - ETA: 1s - loss: 0.0351 - accuracy: 0.9888
783/1688 [============>.................] - ETA: 1s - loss: 0.0349 - accuracy: 0.9889
812/1688 [=============>................] - ETA: 1s - loss: 0.0347 - accuracy: 0.9890
843/1688 [=============>................] - ETA: 1s - loss: 0.0343 - accuracy: 0.9891
873/1688 [==============>...............] - ETA: 1s - loss: 0.0342 - accuracy: 0.9891
903/1688 [===============>..............] - ETA: 1s - loss: 0.0341 - accuracy: 0.9891
933/1688 [===============>..............] - ETA: 1s - loss: 0.0341 - accuracy: 0.9892
963/1688 [================>.............] - ETA: 1s - loss: 0.0342 - accuracy: 0.9891
994/1688 [================>.............] - ETA: 1s - loss: 0.0349 - accuracy: 0.9890
1025/1688 [=================>............] - ETA: 1s - loss: 0.0351 - accuracy: 0.9888
1055/1688 [=================>............] - ETA: 1s - loss: 0.0353 - accuracy: 0.9886
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0353 - accuracy: 0.9886
1115/1688 [==================>...........] - ETA: 0s - loss: 0.0350 - accuracy: 0.9887
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1175/1688 [===================>..........] - ETA: 0s - loss: 0.0349 - accuracy: 0.9889
1205/1688 [====================>.........] - ETA: 0s - loss: 0.0345 - accuracy: 0.9890
1235/1688 [====================>.........] - ETA: 0s - loss: 0.0345 - accuracy: 0.9889
1265/1688 [=====================>........] - ETA: 0s - loss: 0.0344 - accuracy: 0.9889
1295/1688 [======================>.......] - ETA: 0s - loss: 0.0344 - accuracy: 0.9889
1325/1688 [======================>.......] - ETA: 0s - loss: 0.0345 - accuracy: 0.9888
1355/1688 [=======================>......] - ETA: 0s - loss: 0.0344 - accuracy: 0.9888
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0347 - accuracy: 0.9889
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0345 - accuracy: 0.9889
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0352 - accuracy: 0.9888
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0352 - accuracy: 0.9888
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0351 - accuracy: 0.9888
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0348 - accuracy: 0.9888
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0347 - accuracy: 0.9889
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1654/1688 [============================>.] - ETA: 0s - loss: 0.0347 - accuracy: 0.9888
1684/1688 [============================>.] - ETA: 0s - loss: 0.0348 - accuracy: 0.9888
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0349 - accuracy: 0.9888 - val_loss: 0.0496 - val_accuracy: 0.9873
Epoch 4/10
1/1688 [..............................] - ETA: 3s - loss: 0.0038 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0297 - accuracy: 0.9896
63/1688 [>.............................] - ETA: 2s - loss: 0.0319 - accuracy: 0.9886
93/1688 [>.............................] - ETA: 2s - loss: 0.0352 - accuracy: 0.9869
123/1688 [=>............................] - ETA: 2s - loss: 0.0299 - accuracy: 0.9886
154/1688 [=>............................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9890
184/1688 [==>...........................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9898
214/1688 [==>...........................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9902
245/1688 [===>..........................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9903
275/1688 [===>..........................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9899
305/1688 [====>.........................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9895
335/1688 [====>.........................] - ETA: 2s - loss: 0.0262 - accuracy: 0.9901
365/1688 [=====>........................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9906
396/1688 [======>.......................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9911
427/1688 [======>.......................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9913
457/1688 [=======>......................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9912
488/1688 [=======>......................] - ETA: 2s - loss: 0.0243 - accuracy: 0.9914
519/1688 [========>.....................] - ETA: 1s - loss: 0.0238 - accuracy: 0.9918
549/1688 [========>.....................] - ETA: 1s - loss: 0.0244 - accuracy: 0.9917
579/1688 [=========>....................] - ETA: 1s - loss: 0.0252 - accuracy: 0.9916
609/1688 [=========>....................] - ETA: 1s - loss: 0.0256 - accuracy: 0.9916
639/1688 [==========>...................] - ETA: 1s - loss: 0.0259 - accuracy: 0.9914
668/1688 [==========>...................] - ETA: 1s - loss: 0.0267 - accuracy: 0.9912
698/1688 [===========>..................] - ETA: 1s - loss: 0.0262 - accuracy: 0.9914
728/1688 [===========>..................] - ETA: 1s - loss: 0.0257 - accuracy: 0.9916
758/1688 [============>.................] - ETA: 1s - loss: 0.0258 - accuracy: 0.9915
789/1688 [=============>................] - ETA: 1s - loss: 0.0253 - accuracy: 0.9918
819/1688 [=============>................] - ETA: 1s - loss: 0.0256 - accuracy: 0.9917
850/1688 [==============>...............] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
880/1688 [==============>...............] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
909/1688 [===============>..............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9915
939/1688 [===============>..............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9915
969/1688 [================>.............] - ETA: 1s - loss: 0.0253 - accuracy: 0.9916
999/1688 [================>.............] - ETA: 1s - loss: 0.0250 - accuracy: 0.9916
1030/1688 [=================>............] - ETA: 1s - loss: 0.0251 - accuracy: 0.9916
1060/1688 [=================>............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9916
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
1121/1688 [==================>...........] - ETA: 0s - loss: 0.0257 - accuracy: 0.9915
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0259 - accuracy: 0.9914
1181/1688 [===================>..........] - ETA: 0s - loss: 0.0265 - accuracy: 0.9912
1211/1688 [====================>.........] - ETA: 0s - loss: 0.0266 - accuracy: 0.9913
1242/1688 [=====================>........] - ETA: 0s - loss: 0.0267 - accuracy: 0.9912
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0265 - accuracy: 0.9912
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0261 - accuracy: 0.9913
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0258 - accuracy: 0.9914
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0254 - accuracy: 0.9915
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1422/1688 [========================>.....] - ETA: 0s - loss: 0.0256 - accuracy: 0.9914
1452/1688 [========================>.....] - ETA: 0s - loss: 0.0256 - accuracy: 0.9913
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0256 - accuracy: 0.9913
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0254 - accuracy: 0.9914
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1635/1688 [============================>.] - ETA: 0s - loss: 0.0255 - accuracy: 0.9913
1664/1688 [============================>.] - ETA: 0s - loss: 0.0254 - accuracy: 0.9913
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0255 - accuracy: 0.9914 - val_loss: 0.0499 - val_accuracy: 0.9875
Epoch 5/10
1/1688 [..............................] - ETA: 3s - loss: 0.0020 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0233 - accuracy: 0.9912
63/1688 [>.............................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9916
93/1688 [>.............................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9929
123/1688 [=>............................] - ETA: 2s - loss: 0.0198 - accuracy: 0.9921
153/1688 [=>............................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9922
183/1688 [==>...........................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9925
213/1688 [==>...........................] - ETA: 2s - loss: 0.0187 - accuracy: 0.9931
243/1688 [===>..........................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9931
273/1688 [===>..........................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9928
303/1688 [====>.........................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9931
334/1688 [====>.........................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9934
364/1688 [=====>........................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9936
394/1688 [======>.......................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9938
425/1688 [======>.......................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9939
455/1688 [=======>......................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9940
485/1688 [=======>......................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9937
515/1688 [========>.....................] - ETA: 1s - loss: 0.0194 - accuracy: 0.9936
545/1688 [========>.....................] - ETA: 1s - loss: 0.0193 - accuracy: 0.9936
575/1688 [=========>....................] - ETA: 1s - loss: 0.0191 - accuracy: 0.9937
605/1688 [=========>....................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9935
634/1688 [==========>...................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9935
665/1688 [==========>...................] - ETA: 1s - loss: 0.0195 - accuracy: 0.9935
695/1688 [===========>..................] - ETA: 1s - loss: 0.0199 - accuracy: 0.9933
725/1688 [===========>..................] - ETA: 1s - loss: 0.0201 - accuracy: 0.9933
755/1688 [============>.................] - ETA: 1s - loss: 0.0202 - accuracy: 0.9932
785/1688 [============>.................] - ETA: 1s - loss: 0.0202 - accuracy: 0.9932
815/1688 [=============>................] - ETA: 1s - loss: 0.0208 - accuracy: 0.9929
845/1688 [==============>...............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9930
873/1688 [==============>...............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9930
904/1688 [===============>..............] - ETA: 1s - loss: 0.0208 - accuracy: 0.9929
934/1688 [===============>..............] - ETA: 1s - loss: 0.0205 - accuracy: 0.9930
964/1688 [================>.............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9931
994/1688 [================>.............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9929
1024/1688 [=================>............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9927
1055/1688 [=================>............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9927
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9927
1115/1688 [==================>...........] - ETA: 0s - loss: 0.0208 - accuracy: 0.9929
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0207 - accuracy: 0.9929
1176/1688 [===================>..........] - ETA: 0s - loss: 0.0209 - accuracy: 0.9929
1206/1688 [====================>.........] - ETA: 0s - loss: 0.0208 - accuracy: 0.9928
1236/1688 [====================>.........] - ETA: 0s - loss: 0.0206 - accuracy: 0.9929
1266/1688 [=====================>........] - ETA: 0s - loss: 0.0209 - accuracy: 0.9928
1296/1688 [======================>.......] - ETA: 0s - loss: 0.0210 - accuracy: 0.9927
1325/1688 [======================>.......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9927
1355/1688 [=======================>......] - ETA: 0s - loss: 0.0211 - accuracy: 0.9927
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0212 - accuracy: 0.9927
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9928
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0210 - accuracy: 0.9928
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0214 - accuracy: 0.9927
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0219 - accuracy: 0.9925
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0216 - accuracy: 0.9927
1627/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1657/1688 [============================>.] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1687/1688 [============================>.] - ETA: 0s - loss: 0.0224 - accuracy: 0.9924
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0224 - accuracy: 0.9924 - val_loss: 0.0493 - val_accuracy: 0.9877
Epoch 6/10
1/1688 [..............................] - ETA: 2s - loss: 2.0065e-04 - accuracy: 1.0000
34/1688 [..............................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9945
64/1688 [>.............................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9937
94/1688 [>.............................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9947
125/1688 [=>............................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9942
155/1688 [=>............................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9948
185/1688 [==>...........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9949
215/1688 [==>...........................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9951
245/1688 [===>..........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9950
275/1688 [===>..........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9949
305/1688 [====>.........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9951
334/1688 [====>.........................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9954
363/1688 [=====>........................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9954
394/1688 [======>.......................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9955
424/1688 [======>.......................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9955
454/1688 [=======>......................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9955
484/1688 [=======>......................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9955
513/1688 [========>.....................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
543/1688 [========>.....................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9955
573/1688 [=========>....................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
603/1688 [=========>....................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9952
633/1688 [==========>...................] - ETA: 1s - loss: 0.0136 - accuracy: 0.9952
662/1688 [==========>...................] - ETA: 1s - loss: 0.0134 - accuracy: 0.9953
692/1688 [===========>..................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9954
722/1688 [===========>..................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9955
752/1688 [============>.................] - ETA: 1s - loss: 0.0129 - accuracy: 0.9955
782/1688 [============>.................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9954
812/1688 [=============>................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
843/1688 [=============>................] - ETA: 1s - loss: 0.0130 - accuracy: 0.9955
873/1688 [==============>...............] - ETA: 1s - loss: 0.0133 - accuracy: 0.9955
903/1688 [===============>..............] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
932/1688 [===============>..............] - ETA: 1s - loss: 0.0134 - accuracy: 0.9955
962/1688 [================>.............] - ETA: 1s - loss: 0.0137 - accuracy: 0.9955
993/1688 [================>.............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9955
1023/1688 [=================>............] - ETA: 1s - loss: 0.0139 - accuracy: 0.9954
1053/1688 [=================>............] - ETA: 1s - loss: 0.0140 - accuracy: 0.9953
1083/1688 [==================>...........] - ETA: 1s - loss: 0.0143 - accuracy: 0.9952
1113/1688 [==================>...........] - ETA: 0s - loss: 0.0143 - accuracy: 0.9953
1142/1688 [===================>..........] - ETA: 0s - loss: 0.0144 - accuracy: 0.9953
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0150 - accuracy: 0.9951
1203/1688 [====================>.........] - ETA: 0s - loss: 0.0155 - accuracy: 0.9950
1234/1688 [====================>.........] - ETA: 0s - loss: 0.0157 - accuracy: 0.9949
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1294/1688 [=====================>........] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1324/1688 [======================>.......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1355/1688 [=======================>......] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1385/1688 [=======================>......] - ETA: 0s - loss: 0.0162 - accuracy: 0.9947
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0162 - accuracy: 0.9947
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0165 - accuracy: 0.9947
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0170 - accuracy: 0.9946
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0170 - accuracy: 0.9946
1564/1688 [==========================>...] - ETA: 0s - loss: 0.0171 - accuracy: 0.9946
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0170 - accuracy: 0.9947
1625/1688 [===========================>..] - ETA: 0s - loss: 0.0170 - accuracy: 0.9947
1655/1688 [============================>.] - ETA: 0s - loss: 0.0170 - accuracy: 0.9947
1685/1688 [============================>.] - ETA: 0s - loss: 0.0170 - accuracy: 0.9946
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0170 - accuracy: 0.9946 - val_loss: 0.0490 - val_accuracy: 0.9890
Epoch 7/10
1/1688 [..............................] - ETA: 3s - loss: 6.6795e-04 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0048 - accuracy: 1.0000
61/1688 [>.............................] - ETA: 2s - loss: 0.0047 - accuracy: 0.9995
91/1688 [>.............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9986
121/1688 [=>............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9985
152/1688 [=>............................] - ETA: 2s - loss: 0.0058 - accuracy: 0.9984
181/1688 [==>...........................] - ETA: 2s - loss: 0.0058 - accuracy: 0.9981
211/1688 [==>...........................] - ETA: 2s - loss: 0.0072 - accuracy: 0.9979
241/1688 [===>..........................] - ETA: 2s - loss: 0.0074 - accuracy: 0.9979
271/1688 [===>..........................] - ETA: 2s - loss: 0.0074 - accuracy: 0.9979
301/1688 [====>.........................] - ETA: 2s - loss: 0.0084 - accuracy: 0.9975
331/1688 [====>.........................] - ETA: 2s - loss: 0.0085 - accuracy: 0.9973
361/1688 [=====>........................] - ETA: 2s - loss: 0.0082 - accuracy: 0.9973
391/1688 [=====>........................] - ETA: 2s - loss: 0.0088 - accuracy: 0.9972
421/1688 [======>.......................] - ETA: 2s - loss: 0.0091 - accuracy: 0.9970
452/1688 [=======>......................] - ETA: 2s - loss: 0.0089 - accuracy: 0.9970
482/1688 [=======>......................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9968
512/1688 [========>.....................] - ETA: 1s - loss: 0.0100 - accuracy: 0.9966
542/1688 [========>.....................] - ETA: 1s - loss: 0.0105 - accuracy: 0.9964
572/1688 [=========>....................] - ETA: 1s - loss: 0.0107 - accuracy: 0.9963
603/1688 [=========>....................] - ETA: 1s - loss: 0.0113 - accuracy: 0.9961
633/1688 [==========>...................] - ETA: 1s - loss: 0.0118 - accuracy: 0.9959
663/1688 [==========>...................] - ETA: 1s - loss: 0.0117 - accuracy: 0.9958
694/1688 [===========>..................] - ETA: 1s - loss: 0.0117 - accuracy: 0.9958
724/1688 [===========>..................] - ETA: 1s - loss: 0.0121 - accuracy: 0.9957
755/1688 [============>.................] - ETA: 1s - loss: 0.0122 - accuracy: 0.9956
785/1688 [============>.................] - ETA: 1s - loss: 0.0120 - accuracy: 0.9956
816/1688 [=============>................] - ETA: 1s - loss: 0.0118 - accuracy: 0.9957
846/1688 [==============>...............] - ETA: 1s - loss: 0.0116 - accuracy: 0.9958
877/1688 [==============>...............] - ETA: 1s - loss: 0.0114 - accuracy: 0.9959
908/1688 [===============>..............] - ETA: 1s - loss: 0.0114 - accuracy: 0.9959
937/1688 [===============>..............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9959
967/1688 [================>.............] - ETA: 1s - loss: 0.0111 - accuracy: 0.9960
998/1688 [================>.............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9960
1028/1688 [=================>............] - ETA: 1s - loss: 0.0116 - accuracy: 0.9959
1059/1688 [=================>............] - ETA: 1s - loss: 0.0119 - accuracy: 0.9958
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0119 - accuracy: 0.9958
1119/1688 [==================>...........] - ETA: 0s - loss: 0.0120 - accuracy: 0.9957
1150/1688 [===================>..........] - ETA: 0s - loss: 0.0121 - accuracy: 0.9956
1180/1688 [===================>..........] - ETA: 0s - loss: 0.0120 - accuracy: 0.9956
1210/1688 [====================>.........] - ETA: 0s - loss: 0.0119 - accuracy: 0.9956
1240/1688 [=====================>........] - ETA: 0s - loss: 0.0117 - accuracy: 0.9957
1270/1688 [=====================>........] - ETA: 0s - loss: 0.0116 - accuracy: 0.9958
1299/1688 [======================>.......] - ETA: 0s - loss: 0.0116 - accuracy: 0.9958
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0116 - accuracy: 0.9958
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0119 - accuracy: 0.9956
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0120 - accuracy: 0.9956
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0120 - accuracy: 0.9956
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9956
1479/1688 [=========================>....] - ETA: 0s - loss: 0.0120 - accuracy: 0.9955
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0121 - accuracy: 0.9955
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0122 - accuracy: 0.9954
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0125 - accuracy: 0.9954
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0125 - accuracy: 0.9954
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0128 - accuracy: 0.9952
1661/1688 [============================>.] - ETA: 0s - loss: 0.0127 - accuracy: 0.9953
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0128 - accuracy: 0.9951 - val_loss: 0.0794 - val_accuracy: 0.9837
Epoch 8/10
1/1688 [..............................] - ETA: 3s - loss: 0.0106 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9951
62/1688 [>.............................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9965
92/1688 [>.............................] - ETA: 2s - loss: 0.0084 - accuracy: 0.9973
121/1688 [=>............................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9979
152/1688 [=>............................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9984
182/1688 [==>...........................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9986
212/1688 [==>...........................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9984
242/1688 [===>..........................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
273/1688 [===>..........................] - ETA: 2s - loss: 0.0070 - accuracy: 0.9974
302/1688 [====>.........................] - ETA: 2s - loss: 0.0087 - accuracy: 0.9966
332/1688 [====>.........................] - ETA: 2s - loss: 0.0090 - accuracy: 0.9965
362/1688 [=====>........................] - ETA: 2s - loss: 0.0091 - accuracy: 0.9965
392/1688 [=====>........................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9963
422/1688 [======>.......................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9961
452/1688 [=======>......................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9961
482/1688 [=======>......................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9963
512/1688 [========>.....................] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
542/1688 [========>.....................] - ETA: 1s - loss: 0.0093 - accuracy: 0.9965
572/1688 [=========>....................] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
603/1688 [=========>....................] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
633/1688 [==========>...................] - ETA: 1s - loss: 0.0095 - accuracy: 0.9964
664/1688 [==========>...................] - ETA: 1s - loss: 0.0097 - accuracy: 0.9963
694/1688 [===========>..................] - ETA: 1s - loss: 0.0097 - accuracy: 0.9964
724/1688 [===========>..................] - ETA: 1s - loss: 0.0099 - accuracy: 0.9965
754/1688 [============>.................] - ETA: 1s - loss: 0.0098 - accuracy: 0.9965
785/1688 [============>.................] - ETA: 1s - loss: 0.0099 - accuracy: 0.9965
815/1688 [=============>................] - ETA: 1s - loss: 0.0097 - accuracy: 0.9966
844/1688 [==============>...............] - ETA: 1s - loss: 0.0103 - accuracy: 0.9963
875/1688 [==============>...............] - ETA: 1s - loss: 0.0105 - accuracy: 0.9962
905/1688 [===============>..............] - ETA: 1s - loss: 0.0104 - accuracy: 0.9963
935/1688 [===============>..............] - ETA: 1s - loss: 0.0104 - accuracy: 0.9963
965/1688 [================>.............] - ETA: 1s - loss: 0.0105 - accuracy: 0.9963
995/1688 [================>.............] - ETA: 1s - loss: 0.0110 - accuracy: 0.9961
1025/1688 [=================>............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9961
1055/1688 [=================>............] - ETA: 1s - loss: 0.0111 - accuracy: 0.9961
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0111 - accuracy: 0.9961
1115/1688 [==================>...........] - ETA: 0s - loss: 0.0110 - accuracy: 0.9961
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0109 - accuracy: 0.9962
1175/1688 [===================>..........] - ETA: 0s - loss: 0.0109 - accuracy: 0.9962
1205/1688 [====================>.........] - ETA: 0s - loss: 0.0110 - accuracy: 0.9962
1234/1688 [====================>.........] - ETA: 0s - loss: 0.0109 - accuracy: 0.9962
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0109 - accuracy: 0.9962
1293/1688 [=====================>........] - ETA: 0s - loss: 0.0110 - accuracy: 0.9962
1324/1688 [======================>.......] - ETA: 0s - loss: 0.0112 - accuracy: 0.9961
1354/1688 [=======================>......] - ETA: 0s - loss: 0.0115 - accuracy: 0.9961
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0117 - accuracy: 0.9959
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0121 - accuracy: 0.9958
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0118 - accuracy: 0.9958
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1564/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1653/1688 [============================>.] - ETA: 0s - loss: 0.0119 - accuracy: 0.9958
1683/1688 [============================>.] - ETA: 0s - loss: 0.0118 - accuracy: 0.9958
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0118 - accuracy: 0.9958 - val_loss: 0.0493 - val_accuracy: 0.9898
Epoch 9/10
1/1688 [..............................] - ETA: 2s - loss: 0.0040 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0031 - accuracy: 0.9991
64/1688 [>.............................] - ETA: 2s - loss: 0.0027 - accuracy: 0.9995
94/1688 [>.............................] - ETA: 2s - loss: 0.0037 - accuracy: 0.9990
124/1688 [=>............................] - ETA: 2s - loss: 0.0040 - accuracy: 0.9990
154/1688 [=>............................] - ETA: 2s - loss: 0.0051 - accuracy: 0.9986
185/1688 [==>...........................] - ETA: 2s - loss: 0.0076 - accuracy: 0.9978
215/1688 [==>...........................] - ETA: 2s - loss: 0.0080 - accuracy: 0.9975
245/1688 [===>..........................] - ETA: 2s - loss: 0.0080 - accuracy: 0.9976
275/1688 [===>..........................] - ETA: 2s - loss: 0.0080 - accuracy: 0.9974
305/1688 [====>.........................] - ETA: 2s - loss: 0.0085 - accuracy: 0.9973
336/1688 [====>.........................] - ETA: 2s - loss: 0.0087 - accuracy: 0.9973
366/1688 [=====>........................] - ETA: 2s - loss: 0.0082 - accuracy: 0.9975
397/1688 [======>.......................] - ETA: 2s - loss: 0.0083 - accuracy: 0.9974
427/1688 [======>.......................] - ETA: 2s - loss: 0.0088 - accuracy: 0.9971
457/1688 [=======>......................] - ETA: 2s - loss: 0.0087 - accuracy: 0.9971
487/1688 [=======>......................] - ETA: 2s - loss: 0.0085 - accuracy: 0.9971
516/1688 [========>.....................] - ETA: 1s - loss: 0.0086 - accuracy: 0.9972
547/1688 [========>.....................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9971
577/1688 [=========>....................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9971
607/1688 [=========>....................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9971
638/1688 [==========>...................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9971
669/1688 [==========>...................] - ETA: 1s - loss: 0.0091 - accuracy: 0.9971
699/1688 [===========>..................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9971
729/1688 [===========>..................] - ETA: 1s - loss: 0.0091 - accuracy: 0.9970
758/1688 [============>.................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9971
789/1688 [=============>................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9971
819/1688 [=============>................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9971
850/1688 [==============>...............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9969
880/1688 [==============>...............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9969
909/1688 [===============>..............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9969
939/1688 [===============>..............] - ETA: 1s - loss: 0.0094 - accuracy: 0.9969
969/1688 [================>.............] - ETA: 1s - loss: 0.0096 - accuracy: 0.9969
1000/1688 [================>.............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9968
1030/1688 [=================>............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9968
1060/1688 [=================>............] - ETA: 1s - loss: 0.0097 - accuracy: 0.9968
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0096 - accuracy: 0.9968
1120/1688 [==================>...........] - ETA: 0s - loss: 0.0098 - accuracy: 0.9967
1151/1688 [===================>..........] - ETA: 0s - loss: 0.0100 - accuracy: 0.9967
1182/1688 [====================>.........] - ETA: 0s - loss: 0.0099 - accuracy: 0.9967
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0099 - accuracy: 0.9968
1242/1688 [=====================>........] - ETA: 0s - loss: 0.0098 - accuracy: 0.9968
1273/1688 [=====================>........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9968
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0098 - accuracy: 0.9968
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0097 - accuracy: 0.9967
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0097 - accuracy: 0.9967
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0098 - accuracy: 0.9966
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0097 - accuracy: 0.9966
1455/1688 [========================>.....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9966
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9966
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9966
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0099 - accuracy: 0.9965
1635/1688 [============================>.] - ETA: 0s - loss: 0.0100 - accuracy: 0.9965
1665/1688 [============================>.] - ETA: 0s - loss: 0.0100 - accuracy: 0.9965
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0100 - accuracy: 0.9965 - val_loss: 0.0507 - val_accuracy: 0.9893
Epoch 10/10
1/1688 [..............................] - ETA: 2s - loss: 5.2356e-04 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0046 - accuracy: 0.9991
64/1688 [>.............................] - ETA: 2s - loss: 0.0046 - accuracy: 0.9990
94/1688 [>.............................] - ETA: 2s - loss: 0.0050 - accuracy: 0.9990
124/1688 [=>............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9987
154/1688 [=>............................] - ETA: 2s - loss: 0.0051 - accuracy: 0.9986
184/1688 [==>...........................] - ETA: 2s - loss: 0.0055 - accuracy: 0.9981
214/1688 [==>...........................] - ETA: 2s - loss: 0.0064 - accuracy: 0.9978
244/1688 [===>..........................] - ETA: 2s - loss: 0.0065 - accuracy: 0.9978
275/1688 [===>..........................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9977
305/1688 [====>.........................] - ETA: 2s - loss: 0.0067 - accuracy: 0.9978
336/1688 [====>.........................] - ETA: 2s - loss: 0.0064 - accuracy: 0.9980
367/1688 [=====>........................] - ETA: 2s - loss: 0.0064 - accuracy: 0.9980
397/1688 [======>.......................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9982
427/1688 [======>.......................] - ETA: 2s - loss: 0.0060 - accuracy: 0.9982
457/1688 [=======>......................] - ETA: 2s - loss: 0.0057 - accuracy: 0.9983
487/1688 [=======>......................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9983
517/1688 [========>.....................] - ETA: 1s - loss: 0.0055 - accuracy: 0.9984
548/1688 [========>.....................] - ETA: 1s - loss: 0.0055 - accuracy: 0.9984
578/1688 [=========>....................] - ETA: 1s - loss: 0.0056 - accuracy: 0.9984
609/1688 [=========>....................] - ETA: 1s - loss: 0.0054 - accuracy: 0.9985
640/1688 [==========>...................] - ETA: 1s - loss: 0.0060 - accuracy: 0.9983
670/1688 [==========>...................] - ETA: 1s - loss: 0.0067 - accuracy: 0.9981
701/1688 [===========>..................] - ETA: 1s - loss: 0.0071 - accuracy: 0.9980
731/1688 [===========>..................] - ETA: 1s - loss: 0.0075 - accuracy: 0.9979
761/1688 [============>.................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9980
791/1688 [=============>................] - ETA: 1s - loss: 0.0074 - accuracy: 0.9979
821/1688 [=============>................] - ETA: 1s - loss: 0.0077 - accuracy: 0.9978
851/1688 [==============>...............] - ETA: 1s - loss: 0.0080 - accuracy: 0.9976
881/1688 [==============>...............] - ETA: 1s - loss: 0.0079 - accuracy: 0.9976
911/1688 [===============>..............] - ETA: 1s - loss: 0.0081 - accuracy: 0.9975
941/1688 [===============>..............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9975
972/1688 [================>.............] - ETA: 1s - loss: 0.0081 - accuracy: 0.9976
1002/1688 [================>.............] - ETA: 1s - loss: 0.0081 - accuracy: 0.9975
1032/1688 [=================>............] - ETA: 1s - loss: 0.0080 - accuracy: 0.9975
1062/1688 [=================>............] - ETA: 1s - loss: 0.0080 - accuracy: 0.9975
1092/1688 [==================>...........] - ETA: 1s - loss: 0.0081 - accuracy: 0.9974
1123/1688 [==================>...........] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1152/1688 [===================>..........] - ETA: 0s - loss: 0.0080 - accuracy: 0.9974
1182/1688 [====================>.........] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1212/1688 [====================>.........] - ETA: 0s - loss: 0.0083 - accuracy: 0.9974
1243/1688 [=====================>........] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1273/1688 [=====================>........] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1333/1688 [======================>.......] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0082 - accuracy: 0.9973
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0084 - accuracy: 0.9973
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0084 - accuracy: 0.9972
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0085 - accuracy: 0.9972
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0085 - accuracy: 0.9973
1636/1688 [============================>.] - ETA: 0s - loss: 0.0085 - accuracy: 0.9973
1666/1688 [============================>.] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0088 - accuracy: 0.9972 - val_loss: 0.0524 - val_accuracy: 0.9895
Test score: 0.04670824855566025
Test accuracy: 0.9882000088691711
4. Model quantization
We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.
For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).
The quantized model is a Keras model where the neural layers (Conv2D, Dense)
and the ReLU layers are replaced with custom CNN2SNN quantized layers
(QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions
can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function folds the batch normalization layers into
the corresponding neural layer. The new weights are computed
according to this folding operation.
from cnn2snn import quantize
model_quantized = quantize(model_keras,
input_weight_quantization=8,
weight_quantization=4,
activ_quantization=4)
model_quantized.summary()
Out:
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (QuantizedConv2D) (None, 26, 26, 32) 320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32) 0
_________________________________________________________________
re_lu (ActivationDiscreteRel (None, 13, 13, 32) 0
_________________________________________________________________
separable_conv2d (QuantizedS (None, 13, 13, 64) 2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64) 0
_________________________________________________________________
re_lu_1 (ActivationDiscreteR (None, 7, 7, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 3136) 0
_________________________________________________________________
dense (QuantizedDense) (None, 10) 31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________
Check the quantized model accuracy.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])
Out:
Test accuracy after 8-4-4 quantization: 0.9843999743461609
Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.
Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.
Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.
from cnn2snn import quantize_layer
model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])
# To recover the original model accuracy, a quantization-aware training phase
# is required.
Out:
Test accuracy after low bitwidth quantization: 0.6850000023841858
5. Model fine tuning (quantization-aware training)
This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.
Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])
Out:
Epoch 1/5
1/1688 [..............................] - ETA: 16:13 - loss: 0.7107 - accuracy: 0.7188
22/1688 [..............................] - ETA: 4s - loss: 0.5075 - accuracy: 0.8239
43/1688 [..............................] - ETA: 4s - loss: 0.3858 - accuracy: 0.8743
64/1688 [>.............................] - ETA: 3s - loss: 0.3116 - accuracy: 0.9023
85/1688 [>.............................] - ETA: 3s - loss: 0.2666 - accuracy: 0.9173
106/1688 [>.............................] - ETA: 3s - loss: 0.2376 - accuracy: 0.9263
127/1688 [=>............................] - ETA: 3s - loss: 0.2171 - accuracy: 0.9323
148/1688 [=>............................] - ETA: 3s - loss: 0.1984 - accuracy: 0.9383
169/1688 [==>...........................] - ETA: 3s - loss: 0.1871 - accuracy: 0.9406
190/1688 [==>...........................] - ETA: 3s - loss: 0.1748 - accuracy: 0.9441
212/1688 [==>...........................] - ETA: 3s - loss: 0.1656 - accuracy: 0.9472
233/1688 [===>..........................] - ETA: 3s - loss: 0.1567 - accuracy: 0.9505
254/1688 [===>..........................] - ETA: 3s - loss: 0.1482 - accuracy: 0.9531
275/1688 [===>..........................] - ETA: 3s - loss: 0.1402 - accuracy: 0.9555
296/1688 [====>.........................] - ETA: 3s - loss: 0.1339 - accuracy: 0.9577
317/1688 [====>.........................] - ETA: 3s - loss: 0.1289 - accuracy: 0.9590
338/1688 [=====>........................] - ETA: 3s - loss: 0.1247 - accuracy: 0.9598
359/1688 [=====>........................] - ETA: 3s - loss: 0.1194 - accuracy: 0.9616
380/1688 [=====>........................] - ETA: 3s - loss: 0.1163 - accuracy: 0.9623
402/1688 [======>.......................] - ETA: 3s - loss: 0.1131 - accuracy: 0.9632
423/1688 [======>.......................] - ETA: 3s - loss: 0.1105 - accuracy: 0.9637
444/1688 [======>.......................] - ETA: 3s - loss: 0.1073 - accuracy: 0.9648
465/1688 [=======>......................] - ETA: 2s - loss: 0.1045 - accuracy: 0.9659
486/1688 [=======>......................] - ETA: 2s - loss: 0.1021 - accuracy: 0.9668
507/1688 [========>.....................] - ETA: 2s - loss: 0.1002 - accuracy: 0.9675
528/1688 [========>.....................] - ETA: 2s - loss: 0.0976 - accuracy: 0.9682
549/1688 [========>.....................] - ETA: 2s - loss: 0.0953 - accuracy: 0.9689
570/1688 [=========>....................] - ETA: 2s - loss: 0.0934 - accuracy: 0.9693
591/1688 [=========>....................] - ETA: 2s - loss: 0.0916 - accuracy: 0.9699
612/1688 [=========>....................] - ETA: 2s - loss: 0.0904 - accuracy: 0.9701
633/1688 [==========>...................] - ETA: 2s - loss: 0.0891 - accuracy: 0.9705
654/1688 [==========>...................] - ETA: 2s - loss: 0.0875 - accuracy: 0.9709
675/1688 [==========>...................] - ETA: 2s - loss: 0.0860 - accuracy: 0.9713
696/1688 [===========>..................] - ETA: 2s - loss: 0.0843 - accuracy: 0.9719
717/1688 [===========>..................] - ETA: 2s - loss: 0.0830 - accuracy: 0.9724
738/1688 [============>.................] - ETA: 2s - loss: 0.0816 - accuracy: 0.9729
759/1688 [============>.................] - ETA: 2s - loss: 0.0808 - accuracy: 0.9732
780/1688 [============>.................] - ETA: 2s - loss: 0.0797 - accuracy: 0.9736
802/1688 [=============>................] - ETA: 2s - loss: 0.0788 - accuracy: 0.9740
824/1688 [=============>................] - ETA: 2s - loss: 0.0774 - accuracy: 0.9744
845/1688 [==============>...............] - ETA: 2s - loss: 0.0767 - accuracy: 0.9749
866/1688 [==============>...............] - ETA: 1s - loss: 0.0755 - accuracy: 0.9753
887/1688 [==============>...............] - ETA: 1s - loss: 0.0744 - accuracy: 0.9756
909/1688 [===============>..............] - ETA: 1s - loss: 0.0740 - accuracy: 0.9757
930/1688 [===============>..............] - ETA: 1s - loss: 0.0730 - accuracy: 0.9760
951/1688 [===============>..............] - ETA: 1s - loss: 0.0721 - accuracy: 0.9764
972/1688 [================>.............] - ETA: 1s - loss: 0.0715 - accuracy: 0.9766
993/1688 [================>.............] - ETA: 1s - loss: 0.0707 - accuracy: 0.9769
1014/1688 [=================>............] - ETA: 1s - loss: 0.0702 - accuracy: 0.9770
1036/1688 [=================>............] - ETA: 1s - loss: 0.0693 - accuracy: 0.9772
1057/1688 [=================>............] - ETA: 1s - loss: 0.0685 - accuracy: 0.9775
1078/1688 [==================>...........] - ETA: 1s - loss: 0.0677 - accuracy: 0.9777
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0672 - accuracy: 0.9779
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0665 - accuracy: 0.9782
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0661 - accuracy: 0.9784
1162/1688 [===================>..........] - ETA: 1s - loss: 0.0655 - accuracy: 0.9785
1183/1688 [====================>.........] - ETA: 1s - loss: 0.0654 - accuracy: 0.9785
1204/1688 [====================>.........] - ETA: 1s - loss: 0.0653 - accuracy: 0.9786
1225/1688 [====================>.........] - ETA: 1s - loss: 0.0648 - accuracy: 0.9787
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0645 - accuracy: 0.9788
1268/1688 [=====================>........] - ETA: 1s - loss: 0.0641 - accuracy: 0.9790
1290/1688 [=====================>........] - ETA: 0s - loss: 0.0638 - accuracy: 0.9792
1311/1688 [======================>.......] - ETA: 0s - loss: 0.0636 - accuracy: 0.9792
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0630 - accuracy: 0.9794
1353/1688 [=======================>......] - ETA: 0s - loss: 0.0624 - accuracy: 0.9796
1374/1688 [=======================>......] - ETA: 0s - loss: 0.0619 - accuracy: 0.9797
1396/1688 [=======================>......] - ETA: 0s - loss: 0.0615 - accuracy: 0.9799
1418/1688 [========================>.....] - ETA: 0s - loss: 0.0608 - accuracy: 0.9801
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0603 - accuracy: 0.9803
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0597 - accuracy: 0.9805
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0592 - accuracy: 0.9806
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0587 - accuracy: 0.9808
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0584 - accuracy: 0.9808
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0583 - accuracy: 0.9809
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0578 - accuracy: 0.9811
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0575 - accuracy: 0.9812
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0571 - accuracy: 0.9813
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0572 - accuracy: 0.9813
1651/1688 [============================>.] - ETA: 0s - loss: 0.0569 - accuracy: 0.9814
1673/1688 [============================>.] - ETA: 0s - loss: 0.0569 - accuracy: 0.9814
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0567 - accuracy: 0.9814 - val_loss: 0.0608 - val_accuracy: 0.9838
Epoch 2/5
1/1688 [..............................] - ETA: 4s - loss: 0.1173 - accuracy: 0.9688
23/1688 [..............................] - ETA: 3s - loss: 0.0318 - accuracy: 0.9878
44/1688 [..............................] - ETA: 3s - loss: 0.0346 - accuracy: 0.9886
65/1688 [>.............................] - ETA: 3s - loss: 0.0333 - accuracy: 0.9894
86/1688 [>.............................] - ETA: 3s - loss: 0.0324 - accuracy: 0.9895
107/1688 [>.............................] - ETA: 3s - loss: 0.0292 - accuracy: 0.9907
129/1688 [=>............................] - ETA: 3s - loss: 0.0315 - accuracy: 0.9898
151/1688 [=>............................] - ETA: 3s - loss: 0.0308 - accuracy: 0.9897
172/1688 [==>...........................] - ETA: 3s - loss: 0.0297 - accuracy: 0.9900
193/1688 [==>...........................] - ETA: 3s - loss: 0.0278 - accuracy: 0.9901
214/1688 [==>...........................] - ETA: 3s - loss: 0.0277 - accuracy: 0.9902
235/1688 [===>..........................] - ETA: 3s - loss: 0.0275 - accuracy: 0.9904
256/1688 [===>..........................] - ETA: 3s - loss: 0.0278 - accuracy: 0.9904
277/1688 [===>..........................] - ETA: 3s - loss: 0.0267 - accuracy: 0.9907
298/1688 [====>.........................] - ETA: 3s - loss: 0.0268 - accuracy: 0.9908
319/1688 [====>.........................] - ETA: 3s - loss: 0.0281 - accuracy: 0.9901
340/1688 [=====>........................] - ETA: 3s - loss: 0.0288 - accuracy: 0.9899
361/1688 [=====>........................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9901
382/1688 [=====>........................] - ETA: 3s - loss: 0.0276 - accuracy: 0.9903
403/1688 [======>.......................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9901
424/1688 [======>.......................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9901
446/1688 [======>.......................] - ETA: 3s - loss: 0.0276 - accuracy: 0.9903
467/1688 [=======>......................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9904
488/1688 [=======>......................] - ETA: 2s - loss: 0.0280 - accuracy: 0.9903
510/1688 [========>.....................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9902
531/1688 [========>.....................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9900
552/1688 [========>.....................] - ETA: 2s - loss: 0.0280 - accuracy: 0.9900
574/1688 [=========>....................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9901
595/1688 [=========>....................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9901
616/1688 [=========>....................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9901
637/1688 [==========>...................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9899
658/1688 [==========>...................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9901
679/1688 [===========>..................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9899
700/1688 [===========>..................] - ETA: 2s - loss: 0.0286 - accuracy: 0.9899
721/1688 [===========>..................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9900
743/1688 [============>.................] - ETA: 2s - loss: 0.0283 - accuracy: 0.9902
764/1688 [============>.................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9902
785/1688 [============>.................] - ETA: 2s - loss: 0.0283 - accuracy: 0.9902
806/1688 [=============>................] - ETA: 2s - loss: 0.0286 - accuracy: 0.9902
827/1688 [=============>................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9903
848/1688 [==============>...............] - ETA: 2s - loss: 0.0281 - accuracy: 0.9903
869/1688 [==============>...............] - ETA: 1s - loss: 0.0281 - accuracy: 0.9903
890/1688 [==============>...............] - ETA: 1s - loss: 0.0282 - accuracy: 0.9902
911/1688 [===============>..............] - ETA: 1s - loss: 0.0280 - accuracy: 0.9903
932/1688 [===============>..............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9903
953/1688 [===============>..............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9902
974/1688 [================>.............] - ETA: 1s - loss: 0.0284 - accuracy: 0.9902
996/1688 [================>.............] - ETA: 1s - loss: 0.0280 - accuracy: 0.9903
1017/1688 [=================>............] - ETA: 1s - loss: 0.0278 - accuracy: 0.9903
1039/1688 [=================>............] - ETA: 1s - loss: 0.0275 - accuracy: 0.9904
1060/1688 [=================>............] - ETA: 1s - loss: 0.0275 - accuracy: 0.9904
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0279 - accuracy: 0.9903
1102/1688 [==================>...........] - ETA: 1s - loss: 0.0279 - accuracy: 0.9903
1123/1688 [==================>...........] - ETA: 1s - loss: 0.0285 - accuracy: 0.9902
1144/1688 [===================>..........] - ETA: 1s - loss: 0.0292 - accuracy: 0.9900
1166/1688 [===================>..........] - ETA: 1s - loss: 0.0292 - accuracy: 0.9899
1187/1688 [====================>.........] - ETA: 1s - loss: 0.0291 - accuracy: 0.9899
1208/1688 [====================>.........] - ETA: 1s - loss: 0.0288 - accuracy: 0.9900
1229/1688 [====================>.........] - ETA: 1s - loss: 0.0288 - accuracy: 0.9900
1250/1688 [=====================>........] - ETA: 1s - loss: 0.0291 - accuracy: 0.9900
1271/1688 [=====================>........] - ETA: 1s - loss: 0.0290 - accuracy: 0.9900
1292/1688 [=====================>........] - ETA: 0s - loss: 0.0289 - accuracy: 0.9900
1313/1688 [======================>.......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9900
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9901
1356/1688 [=======================>......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9902
1377/1688 [=======================>......] - ETA: 0s - loss: 0.0286 - accuracy: 0.9903
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0285 - accuracy: 0.9903
1419/1688 [========================>.....] - ETA: 0s - loss: 0.0285 - accuracy: 0.9902
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0286 - accuracy: 0.9902
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9903
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9903
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9902
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0283 - accuracy: 0.9902
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0282 - accuracy: 0.9903
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0282 - accuracy: 0.9902
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0283 - accuracy: 0.9902
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0282 - accuracy: 0.9903
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0281 - accuracy: 0.9903
1650/1688 [============================>.] - ETA: 0s - loss: 0.0280 - accuracy: 0.9903
1671/1688 [============================>.] - ETA: 0s - loss: 0.0280 - accuracy: 0.9903
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0280 - accuracy: 0.9903 - val_loss: 0.0534 - val_accuracy: 0.9863
Epoch 3/5
1/1688 [..............................] - ETA: 4s - loss: 0.0284 - accuracy: 0.9688
23/1688 [..............................] - ETA: 3s - loss: 0.0303 - accuracy: 0.9918
44/1688 [..............................] - ETA: 3s - loss: 0.0258 - accuracy: 0.9929
66/1688 [>.............................] - ETA: 3s - loss: 0.0262 - accuracy: 0.9920
87/1688 [>.............................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9928
108/1688 [>.............................] - ETA: 3s - loss: 0.0224 - accuracy: 0.9931
129/1688 [=>............................] - ETA: 3s - loss: 0.0218 - accuracy: 0.9932
151/1688 [=>............................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9934
172/1688 [==>...........................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9940
193/1688 [==>...........................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9945
214/1688 [==>...........................] - ETA: 3s - loss: 0.0175 - accuracy: 0.9950
235/1688 [===>..........................] - ETA: 3s - loss: 0.0176 - accuracy: 0.9949
256/1688 [===>..........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9949
277/1688 [===>..........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9945
298/1688 [====>.........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9947
319/1688 [====>.........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9944
338/1688 [=====>........................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9944
359/1688 [=====>........................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9936
381/1688 [=====>........................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9935
402/1688 [======>.......................] - ETA: 3s - loss: 0.0206 - accuracy: 0.9934
423/1688 [======>.......................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9934
444/1688 [======>.......................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9933
465/1688 [=======>......................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9931
486/1688 [=======>......................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9931
508/1688 [========>.....................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9932
529/1688 [========>.....................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9932
550/1688 [========>.....................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9930
571/1688 [=========>....................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9930
592/1688 [=========>....................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9930
613/1688 [=========>....................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9930
634/1688 [==========>...................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9931
655/1688 [==========>...................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9930
676/1688 [===========>..................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9929
698/1688 [===========>..................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9930
719/1688 [===========>..................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9930
740/1688 [============>.................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9931
761/1688 [============>.................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9931
782/1688 [============>.................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9930
803/1688 [=============>................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9930
824/1688 [=============>................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9929
845/1688 [==============>...............] - ETA: 2s - loss: 0.0212 - accuracy: 0.9930
866/1688 [==============>...............] - ETA: 1s - loss: 0.0213 - accuracy: 0.9930
887/1688 [==============>...............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
908/1688 [===============>..............] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
929/1688 [===============>..............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
950/1688 [===============>..............] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
971/1688 [================>.............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
993/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
1015/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
1036/1688 [=================>............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1057/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0223 - accuracy: 0.9927
1098/1688 [==================>...........] - ETA: 1s - loss: 0.0226 - accuracy: 0.9926
1119/1688 [==================>...........] - ETA: 1s - loss: 0.0227 - accuracy: 0.9926
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0229 - accuracy: 0.9926
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0229 - accuracy: 0.9925
1182/1688 [====================>.........] - ETA: 1s - loss: 0.0231 - accuracy: 0.9925
1203/1688 [====================>.........] - ETA: 1s - loss: 0.0230 - accuracy: 0.9925
1224/1688 [====================>.........] - ETA: 1s - loss: 0.0230 - accuracy: 0.9925
1245/1688 [=====================>........] - ETA: 1s - loss: 0.0233 - accuracy: 0.9924
1266/1688 [=====================>........] - ETA: 1s - loss: 0.0233 - accuracy: 0.9924
1287/1688 [=====================>........] - ETA: 0s - loss: 0.0233 - accuracy: 0.9924
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0235 - accuracy: 0.9923
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0234 - accuracy: 0.9923
1351/1688 [=======================>......] - ETA: 0s - loss: 0.0233 - accuracy: 0.9923
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0234 - accuracy: 0.9922
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0233 - accuracy: 0.9923
1436/1688 [========================>.....] - ETA: 0s - loss: 0.0235 - accuracy: 0.9922
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0235 - accuracy: 0.9921
1478/1688 [=========================>....] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0232 - accuracy: 0.9923
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0234 - accuracy: 0.9922
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0234 - accuracy: 0.9922
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0235 - accuracy: 0.9922
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0233 - accuracy: 0.9923
1647/1688 [============================>.] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1668/1688 [============================>.] - ETA: 0s - loss: 0.0230 - accuracy: 0.9923
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0229 - accuracy: 0.9924 - val_loss: 0.0564 - val_accuracy: 0.9883
Epoch 4/5
1/1688 [..............................] - ETA: 4s - loss: 0.0055 - accuracy: 1.0000
23/1688 [..............................] - ETA: 3s - loss: 0.0107 - accuracy: 0.9973
45/1688 [..............................] - ETA: 3s - loss: 0.0218 - accuracy: 0.9931
66/1688 [>.............................] - ETA: 3s - loss: 0.0212 - accuracy: 0.9938
87/1688 [>.............................] - ETA: 3s - loss: 0.0186 - accuracy: 0.9946
109/1688 [>.............................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9948
130/1688 [=>............................] - ETA: 3s - loss: 0.0161 - accuracy: 0.9952
151/1688 [=>............................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9950
172/1688 [==>...........................] - ETA: 3s - loss: 0.0155 - accuracy: 0.9947
193/1688 [==>...........................] - ETA: 3s - loss: 0.0167 - accuracy: 0.9943
214/1688 [==>...........................] - ETA: 3s - loss: 0.0171 - accuracy: 0.9942
235/1688 [===>..........................] - ETA: 3s - loss: 0.0172 - accuracy: 0.9940
256/1688 [===>..........................] - ETA: 3s - loss: 0.0179 - accuracy: 0.9939
277/1688 [===>..........................] - ETA: 3s - loss: 0.0189 - accuracy: 0.9931
298/1688 [====>.........................] - ETA: 3s - loss: 0.0196 - accuracy: 0.9927
319/1688 [====>.........................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9929
340/1688 [=====>........................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9928
361/1688 [=====>........................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9926
382/1688 [=====>........................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9926
403/1688 [======>.......................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9922
424/1688 [======>.......................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9921
445/1688 [======>.......................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9923
466/1688 [=======>......................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9921
487/1688 [=======>......................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9922
508/1688 [========>.....................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9922
529/1688 [========>.....................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9920
550/1688 [========>.....................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9920
571/1688 [=========>....................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9923
592/1688 [=========>....................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9924
613/1688 [=========>....................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9926
634/1688 [==========>...................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9926
655/1688 [==========>...................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9926
676/1688 [===========>..................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9926
697/1688 [===========>..................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9926
718/1688 [===========>..................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9926
739/1688 [============>.................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9926
760/1688 [============>.................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9926
781/1688 [============>.................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9926
802/1688 [=============>................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9926
823/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9927
844/1688 [==============>...............] - ETA: 2s - loss: 0.0204 - accuracy: 0.9926
865/1688 [==============>...............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9927
886/1688 [==============>...............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9928
907/1688 [===============>..............] - ETA: 1s - loss: 0.0199 - accuracy: 0.9929
928/1688 [===============>..............] - ETA: 1s - loss: 0.0198 - accuracy: 0.9928
948/1688 [===============>..............] - ETA: 1s - loss: 0.0197 - accuracy: 0.9929
969/1688 [================>.............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9929
990/1688 [================>.............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9929
1011/1688 [================>.............] - ETA: 1s - loss: 0.0199 - accuracy: 0.9929
1032/1688 [=================>............] - ETA: 1s - loss: 0.0199 - accuracy: 0.9928
1053/1688 [=================>............] - ETA: 1s - loss: 0.0197 - accuracy: 0.9929
1074/1688 [==================>...........] - ETA: 1s - loss: 0.0199 - accuracy: 0.9927
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0198 - accuracy: 0.9928
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0199 - accuracy: 0.9929
1137/1688 [===================>..........] - ETA: 1s - loss: 0.0198 - accuracy: 0.9929
1158/1688 [===================>..........] - ETA: 1s - loss: 0.0197 - accuracy: 0.9929
1179/1688 [===================>..........] - ETA: 1s - loss: 0.0196 - accuracy: 0.9929
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0196 - accuracy: 0.9929
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0195 - accuracy: 0.9929
1243/1688 [=====================>........] - ETA: 1s - loss: 0.0195 - accuracy: 0.9929
1264/1688 [=====================>........] - ETA: 1s - loss: 0.0195 - accuracy: 0.9930
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0194 - accuracy: 0.9930
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0193 - accuracy: 0.9930
1327/1688 [======================>.......] - ETA: 0s - loss: 0.0195 - accuracy: 0.9930
1348/1688 [======================>.......] - ETA: 0s - loss: 0.0197 - accuracy: 0.9929
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0198 - accuracy: 0.9929
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0197 - accuracy: 0.9929
1411/1688 [========================>.....] - ETA: 0s - loss: 0.0199 - accuracy: 0.9928
1432/1688 [========================>.....] - ETA: 0s - loss: 0.0199 - accuracy: 0.9928
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0198 - accuracy: 0.9928
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0202 - accuracy: 0.9927
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0202 - accuracy: 0.9927
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0201 - accuracy: 0.9928
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0204 - accuracy: 0.9927
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0205 - accuracy: 0.9927
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0206 - accuracy: 0.9926
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0209 - accuracy: 0.9925
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0211 - accuracy: 0.9923
1642/1688 [============================>.] - ETA: 0s - loss: 0.0211 - accuracy: 0.9924
1663/1688 [============================>.] - ETA: 0s - loss: 0.0209 - accuracy: 0.9924
1684/1688 [============================>.] - ETA: 0s - loss: 0.0210 - accuracy: 0.9924
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0210 - accuracy: 0.9924 - val_loss: 0.0558 - val_accuracy: 0.9892
Epoch 5/5
1/1688 [..............................] - ETA: 4s - loss: 0.0463 - accuracy: 0.9688
23/1688 [..............................] - ETA: 3s - loss: 0.0238 - accuracy: 0.9932
44/1688 [..............................] - ETA: 3s - loss: 0.0230 - accuracy: 0.9943
65/1688 [>.............................] - ETA: 3s - loss: 0.0229 - accuracy: 0.9933
87/1688 [>.............................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9932
108/1688 [>.............................] - ETA: 3s - loss: 0.0237 - accuracy: 0.9928
129/1688 [=>............................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9930
150/1688 [=>............................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9925
171/1688 [==>...........................] - ETA: 3s - loss: 0.0231 - accuracy: 0.9929
192/1688 [==>...........................] - ETA: 3s - loss: 0.0220 - accuracy: 0.9932
213/1688 [==>...........................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9931
234/1688 [===>..........................] - ETA: 3s - loss: 0.0201 - accuracy: 0.9936
255/1688 [===>..........................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9940
276/1688 [===>..........................] - ETA: 3s - loss: 0.0197 - accuracy: 0.9935
297/1688 [====>.........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9939
318/1688 [====>.........................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9939
339/1688 [=====>........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9937
360/1688 [=====>........................] - ETA: 3s - loss: 0.0183 - accuracy: 0.9938
381/1688 [=====>........................] - ETA: 3s - loss: 0.0181 - accuracy: 0.9938
402/1688 [======>.......................] - ETA: 3s - loss: 0.0180 - accuracy: 0.9939
424/1688 [======>.......................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9941
445/1688 [======>.......................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9940
466/1688 [=======>......................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9938
487/1688 [=======>......................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9938
508/1688 [========>.....................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9938
529/1688 [========>.....................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9940
550/1688 [========>.....................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9939
571/1688 [=========>....................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9939
591/1688 [=========>....................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9940
612/1688 [=========>....................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9939
633/1688 [==========>...................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9939
654/1688 [==========>...................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9937
675/1688 [==========>...................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9936
696/1688 [===========>..................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9936
717/1688 [===========>..................] - ETA: 2s - loss: 0.0194 - accuracy: 0.9936
738/1688 [============>.................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9936
759/1688 [============>.................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9936
780/1688 [============>.................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9937
801/1688 [=============>................] - ETA: 2s - loss: 0.0198 - accuracy: 0.9933
822/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9932
843/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9931
864/1688 [==============>...............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9931
885/1688 [==============>...............] - ETA: 1s - loss: 0.0207 - accuracy: 0.9930
906/1688 [===============>..............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
927/1688 [===============>..............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
948/1688 [===============>..............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9929
969/1688 [================>.............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9930
990/1688 [================>.............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
1011/1688 [================>.............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9931
1032/1688 [=================>............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9930
1054/1688 [=================>............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9930
1075/1688 [==================>...........] - ETA: 1s - loss: 0.0209 - accuracy: 0.9931
1096/1688 [==================>...........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9930
1118/1688 [==================>...........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9930
1139/1688 [===================>..........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9930
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9930
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9929
1202/1688 [====================>.........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9930
1223/1688 [====================>.........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
1244/1688 [=====================>........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9927
1265/1688 [=====================>........] - ETA: 1s - loss: 0.0219 - accuracy: 0.9927
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0219 - accuracy: 0.9927
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0219 - accuracy: 0.9926
1329/1688 [======================>.......] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1350/1688 [======================>.......] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0215 - accuracy: 0.9927
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1435/1688 [========================>.....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1498/1688 [=========================>....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0215 - accuracy: 0.9927
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0219 - accuracy: 0.9926
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9926
1603/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1645/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1666/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1687/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9926
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0216 - accuracy: 0.9926 - val_loss: 0.0595 - val_accuracy: 0.9877
Test accuracy after fine tuning: 0.98580002784729
6. Model conversion
After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida Execution Engine.
Note
One needs to supply the coefficients used to rescale the input
dataset before the training - here input_scaling.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()
results = model_akida.predict(raw_x_test)
accuracy = (raw_y_test == results).mean()
print('Test accuracy after conversion:', accuracy)
# For non-regression purpose
assert accuracy > 0.97
Out:
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 3
______________________________________________
SW/conv2d-dense (Software)
______________________________________________________________
Layer (type) Output shape Kernel shape
==============================================================
conv2d (InputConv.) [13, 13, 32] (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.) [7, 7, 64] (3, 3, 32, 1)
______________________________________________________________
(1, 1, 32, 64)
______________________________________________________________
dense (Fully.) [1, 1, 10] (1, 1, 3136, 10)
______________________________________________________________
Test accuracy after conversion: 0.9857
Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).
Total running time of the script: ( 0 minutes 57.164 seconds)