Note
Click here to download the full example code
CNN conversion flow tutorial¶
This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.
The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.
Note
Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.
MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.
1. Load and reshape MNIST dataset¶
After loading, we make 2 transformations on the dataset:
Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.
Note
At this point, we’ll set aside the raw data for testing our converted model in the Akida Execution Engine later.
Rescale the 8-bit loaded data to the range 0-to-1 for training.
Note
Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.
This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.
Also note that we store the scaling values input_scaling for
use when preparing the model for the Akida Execution Engine. The
implementation of the Akida neural network allows us to completely
skip the rescaling step (i.e. the Akida model should be fed with
the raw 8-bit values) but that does require information about what
scaling was applied prior to training - see below for more details.
import tensorflow as tf
from tensorflow import keras
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)
# Set aside raw test data for use with Akida Execution Engine later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test
# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a
2. Model definition¶
Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.
An appropriate model for MNIST (inspired by this example) might look something like the following:
model_keras = keras.models.Sequential([
keras.layers.Conv2D(filters=32, kernel_size=3, input_shape=(28, 28, 1)),
keras.layers.MaxPool2D(),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.SeparableConv2D(filters=64, kernel_size=3, padding='same'),
keras.layers.MaxPool2D(padding='same'),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Out:
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 26, 26, 32) 320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32) 0
_________________________________________________________________
batch_normalization (BatchNo (None, 13, 13, 32) 128
_________________________________________________________________
re_lu (ReLU) (None, 13, 13, 32) 0
_________________________________________________________________
separable_conv2d (SeparableC (None, 13, 13, 64) 2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64) 0
_________________________________________________________________
batch_normalization_1 (Batch (None, 7, 7, 64) 256
_________________________________________________________________
re_lu_1 (ReLU) (None, 7, 7, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 3136) 0
_________________________________________________________________
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________
The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):
Standard Conv2D and Dense layers are supported
Hidden layers must be followed by a ReLU layer.
BatchNormalization must always happen before activations.
Convolutional blocks can optionally be followed by a MaxPooling.
The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).
from cnn2snn import check_model_compatibility
print("Model compatible for Akida conversion:",
check_model_compatibility(model_keras))
Out:
Model compatible for Akida conversion: True
3. Model training¶
Before going any further, train the model and get its performance. The created model should have achieved a test accuracy a little over 99% after 10 epochs.
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])
Out:
Epoch 1/10
1/1688 [..............................] - ETA: 10:35 - loss: 2.8026 - accuracy: 0.0625
30/1688 [..............................] - ETA: 2s - loss: 1.8196 - accuracy: 0.4397
59/1688 [>.............................] - ETA: 2s - loss: 1.3904 - accuracy: 0.5719
89/1688 [>.............................] - ETA: 2s - loss: 1.1593 - accuracy: 0.6441
119/1688 [=>............................] - ETA: 2s - loss: 1.0128 - accuracy: 0.6893
148/1688 [=>............................] - ETA: 2s - loss: 0.9124 - accuracy: 0.7204
177/1688 [==>...........................] - ETA: 2s - loss: 0.8361 - accuracy: 0.7442
206/1688 [==>...........................] - ETA: 2s - loss: 0.7748 - accuracy: 0.7634
235/1688 [===>..........................] - ETA: 2s - loss: 0.7243 - accuracy: 0.7791
265/1688 [===>..........................] - ETA: 2s - loss: 0.6807 - accuracy: 0.7926
294/1688 [====>.........................] - ETA: 2s - loss: 0.6446 - accuracy: 0.8038
324/1688 [====>.........................] - ETA: 2s - loss: 0.6125 - accuracy: 0.8137
353/1688 [=====>........................] - ETA: 2s - loss: 0.5853 - accuracy: 0.8222
382/1688 [=====>........................] - ETA: 2s - loss: 0.5612 - accuracy: 0.8296
411/1688 [======>.......................] - ETA: 2s - loss: 0.5399 - accuracy: 0.8362
440/1688 [======>.......................] - ETA: 2s - loss: 0.5207 - accuracy: 0.8421
469/1688 [=======>......................] - ETA: 2s - loss: 0.5033 - accuracy: 0.8474
498/1688 [=======>......................] - ETA: 2s - loss: 0.4874 - accuracy: 0.8523
528/1688 [========>.....................] - ETA: 2s - loss: 0.4724 - accuracy: 0.8569
557/1688 [========>.....................] - ETA: 1s - loss: 0.4591 - accuracy: 0.8609
587/1688 [=========>....................] - ETA: 1s - loss: 0.4463 - accuracy: 0.8647
616/1688 [=========>....................] - ETA: 1s - loss: 0.4349 - accuracy: 0.8682
645/1688 [==========>...................] - ETA: 1s - loss: 0.4243 - accuracy: 0.8714
675/1688 [==========>...................] - ETA: 1s - loss: 0.4141 - accuracy: 0.8745
704/1688 [===========>..................] - ETA: 1s - loss: 0.4048 - accuracy: 0.8773
734/1688 [============>.................] - ETA: 1s - loss: 0.3958 - accuracy: 0.8800
764/1688 [============>.................] - ETA: 1s - loss: 0.3874 - accuracy: 0.8826
793/1688 [=============>................] - ETA: 1s - loss: 0.3798 - accuracy: 0.8849
822/1688 [=============>................] - ETA: 1s - loss: 0.3725 - accuracy: 0.8871
851/1688 [==============>...............] - ETA: 1s - loss: 0.3657 - accuracy: 0.8892
881/1688 [==============>...............] - ETA: 1s - loss: 0.3589 - accuracy: 0.8912
910/1688 [===============>..............] - ETA: 1s - loss: 0.3527 - accuracy: 0.8931
939/1688 [===============>..............] - ETA: 1s - loss: 0.3468 - accuracy: 0.8949
968/1688 [================>.............] - ETA: 1s - loss: 0.3412 - accuracy: 0.8966
997/1688 [================>.............] - ETA: 1s - loss: 0.3358 - accuracy: 0.8982
1026/1688 [=================>............] - ETA: 1s - loss: 0.3306 - accuracy: 0.8998
1055/1688 [=================>............] - ETA: 1s - loss: 0.3257 - accuracy: 0.9013
1084/1688 [==================>...........] - ETA: 1s - loss: 0.3210 - accuracy: 0.9027
1114/1688 [==================>...........] - ETA: 0s - loss: 0.3163 - accuracy: 0.9042
1144/1688 [===================>..........] - ETA: 0s - loss: 0.3118 - accuracy: 0.9055
1173/1688 [===================>..........] - ETA: 0s - loss: 0.3076 - accuracy: 0.9068
1202/1688 [====================>.........] - ETA: 0s - loss: 0.3035 - accuracy: 0.9080
1232/1688 [====================>.........] - ETA: 0s - loss: 0.2995 - accuracy: 0.9093
1262/1688 [=====================>........] - ETA: 0s - loss: 0.2957 - accuracy: 0.9104
1291/1688 [=====================>........] - ETA: 0s - loss: 0.2921 - accuracy: 0.9115
1320/1688 [======================>.......] - ETA: 0s - loss: 0.2886 - accuracy: 0.9126
1350/1688 [======================>.......] - ETA: 0s - loss: 0.2852 - accuracy: 0.9136
1380/1688 [=======================>......] - ETA: 0s - loss: 0.2819 - accuracy: 0.9146
1409/1688 [========================>.....] - ETA: 0s - loss: 0.2788 - accuracy: 0.9155
1438/1688 [========================>.....] - ETA: 0s - loss: 0.2758 - accuracy: 0.9165
1467/1688 [=========================>....] - ETA: 0s - loss: 0.2729 - accuracy: 0.9173
1496/1688 [=========================>....] - ETA: 0s - loss: 0.2701 - accuracy: 0.9182
1525/1688 [==========================>...] - ETA: 0s - loss: 0.2673 - accuracy: 0.9190
1554/1688 [==========================>...] - ETA: 0s - loss: 0.2647 - accuracy: 0.9198
1583/1688 [===========================>..] - ETA: 0s - loss: 0.2621 - accuracy: 0.9206
1613/1688 [===========================>..] - ETA: 0s - loss: 0.2595 - accuracy: 0.9214
1642/1688 [============================>.] - ETA: 0s - loss: 0.2571 - accuracy: 0.9221
1672/1688 [============================>.] - ETA: 0s - loss: 0.2547 - accuracy: 0.9228
1688/1688 [==============================] - ETA: 0s - loss: 0.2534 - accuracy: 0.9232
1688/1688 [==============================] - 4s 2ms/step - loss: 0.2533 - accuracy: 0.9232 - val_loss: 0.0706 - val_accuracy: 0.9793
Epoch 2/10
1/1688 [..............................] - ETA: 2s - loss: 0.0573 - accuracy: 0.9688
32/1688 [..............................] - ETA: 2s - loss: 0.0696 - accuracy: 0.9707
61/1688 [>.............................] - ETA: 2s - loss: 0.0634 - accuracy: 0.9739
90/1688 [>.............................] - ETA: 2s - loss: 0.0594 - accuracy: 0.9763
119/1688 [=>............................] - ETA: 2s - loss: 0.0577 - accuracy: 0.9776
148/1688 [=>............................] - ETA: 2s - loss: 0.0564 - accuracy: 0.9785
178/1688 [==>...........................] - ETA: 2s - loss: 0.0551 - accuracy: 0.9794
207/1688 [==>...........................] - ETA: 2s - loss: 0.0539 - accuracy: 0.9801
236/1688 [===>..........................] - ETA: 2s - loss: 0.0527 - accuracy: 0.9808
265/1688 [===>..........................] - ETA: 2s - loss: 0.0518 - accuracy: 0.9812
294/1688 [====>.........................] - ETA: 2s - loss: 0.0512 - accuracy: 0.9815
323/1688 [====>.........................] - ETA: 2s - loss: 0.0509 - accuracy: 0.9817
353/1688 [=====>........................] - ETA: 2s - loss: 0.0508 - accuracy: 0.9819
383/1688 [=====>........................] - ETA: 2s - loss: 0.0507 - accuracy: 0.9820
412/1688 [======>.......................] - ETA: 2s - loss: 0.0506 - accuracy: 0.9821
442/1688 [======>.......................] - ETA: 2s - loss: 0.0504 - accuracy: 0.9822
471/1688 [=======>......................] - ETA: 2s - loss: 0.0502 - accuracy: 0.9823
500/1688 [=======>......................] - ETA: 2s - loss: 0.0500 - accuracy: 0.9825
529/1688 [========>.....................] - ETA: 2s - loss: 0.0498 - accuracy: 0.9826
558/1688 [========>.....................] - ETA: 1s - loss: 0.0497 - accuracy: 0.9827
587/1688 [=========>....................] - ETA: 1s - loss: 0.0495 - accuracy: 0.9828
616/1688 [=========>....................] - ETA: 1s - loss: 0.0493 - accuracy: 0.9829
645/1688 [==========>...................] - ETA: 1s - loss: 0.0491 - accuracy: 0.9831
674/1688 [==========>...................] - ETA: 1s - loss: 0.0490 - accuracy: 0.9832
703/1688 [===========>..................] - ETA: 1s - loss: 0.0488 - accuracy: 0.9833
733/1688 [============>.................] - ETA: 1s - loss: 0.0487 - accuracy: 0.9834
763/1688 [============>.................] - ETA: 1s - loss: 0.0486 - accuracy: 0.9834
793/1688 [=============>................] - ETA: 1s - loss: 0.0485 - accuracy: 0.9835
823/1688 [=============>................] - ETA: 1s - loss: 0.0484 - accuracy: 0.9836
852/1688 [==============>...............] - ETA: 1s - loss: 0.0483 - accuracy: 0.9837
882/1688 [==============>...............] - ETA: 1s - loss: 0.0482 - accuracy: 0.9837
911/1688 [===============>..............] - ETA: 1s - loss: 0.0482 - accuracy: 0.9838
940/1688 [===============>..............] - ETA: 1s - loss: 0.0482 - accuracy: 0.9838
969/1688 [================>.............] - ETA: 1s - loss: 0.0481 - accuracy: 0.9839
998/1688 [================>.............] - ETA: 1s - loss: 0.0481 - accuracy: 0.9839
1027/1688 [=================>............] - ETA: 1s - loss: 0.0480 - accuracy: 0.9839
1056/1688 [=================>............] - ETA: 1s - loss: 0.0480 - accuracy: 0.9840
1086/1688 [==================>...........] - ETA: 1s - loss: 0.0480 - accuracy: 0.9840
1116/1688 [==================>...........] - ETA: 0s - loss: 0.0480 - accuracy: 0.9840
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0480 - accuracy: 0.9841
1175/1688 [===================>..........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9841
1204/1688 [====================>.........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9841
1233/1688 [====================>.........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9841
1262/1688 [=====================>........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9841
1292/1688 [=====================>........] - ETA: 0s - loss: 0.0479 - accuracy: 0.9842
1322/1688 [======================>.......] - ETA: 0s - loss: 0.0479 - accuracy: 0.9842
1351/1688 [=======================>......] - ETA: 0s - loss: 0.0478 - accuracy: 0.9842
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0478 - accuracy: 0.9842
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0478 - accuracy: 0.9842
1439/1688 [========================>.....] - ETA: 0s - loss: 0.0478 - accuracy: 0.9842
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0478 - accuracy: 0.9843
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1644/1688 [============================>.] - ETA: 0s - loss: 0.0477 - accuracy: 0.9843
1673/1688 [============================>.] - ETA: 0s - loss: 0.0476 - accuracy: 0.9844
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0476 - accuracy: 0.9844 - val_loss: 0.0597 - val_accuracy: 0.9857
Epoch 3/10
1/1688 [..............................] - ETA: 3s - loss: 9.3639e-04 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9940
60/1688 [>.............................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9930
89/1688 [>.............................] - ETA: 2s - loss: 0.0259 - accuracy: 0.9925
118/1688 [=>............................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9922
148/1688 [=>............................] - ETA: 2s - loss: 0.0290 - accuracy: 0.9922
178/1688 [==>...........................] - ETA: 2s - loss: 0.0294 - accuracy: 0.9920
207/1688 [==>...........................] - ETA: 2s - loss: 0.0298 - accuracy: 0.9918
236/1688 [===>..........................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9916
265/1688 [===>..........................] - ETA: 2s - loss: 0.0304 - accuracy: 0.9915
294/1688 [====>.........................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9914
323/1688 [====>.........................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9913
352/1688 [=====>........................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9913
382/1688 [=====>........................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9913
411/1688 [======>.......................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9913
440/1688 [======>.......................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9912
470/1688 [=======>......................] - ETA: 2s - loss: 0.0307 - accuracy: 0.9912
499/1688 [=======>......................] - ETA: 2s - loss: 0.0307 - accuracy: 0.9911
528/1688 [========>.....................] - ETA: 2s - loss: 0.0308 - accuracy: 0.9911
557/1688 [========>.....................] - ETA: 1s - loss: 0.0308 - accuracy: 0.9910
586/1688 [=========>....................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9909
616/1688 [=========>....................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9909
645/1688 [==========>...................] - ETA: 1s - loss: 0.0310 - accuracy: 0.9908
674/1688 [==========>...................] - ETA: 1s - loss: 0.0311 - accuracy: 0.9908
703/1688 [===========>..................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9907
732/1688 [============>.................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9907
762/1688 [============>.................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9907
791/1688 [=============>................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9907
820/1688 [=============>................] - ETA: 1s - loss: 0.0312 - accuracy: 0.9906
849/1688 [==============>...............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9906
879/1688 [==============>...............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9906
909/1688 [===============>..............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9906
939/1688 [===============>..............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9906
968/1688 [================>.............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9906
997/1688 [================>.............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9906
1026/1688 [=================>............] - ETA: 1s - loss: 0.0310 - accuracy: 0.9906
1055/1688 [=================>............] - ETA: 1s - loss: 0.0310 - accuracy: 0.9906
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0310 - accuracy: 0.9906
1114/1688 [==================>...........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9906
1143/1688 [===================>..........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9906
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9906
1202/1688 [====================>.........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9906
1231/1688 [====================>.........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9905
1260/1688 [=====================>........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9905
1289/1688 [=====================>........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9905
1318/1688 [======================>.......] - ETA: 0s - loss: 0.0311 - accuracy: 0.9905
1347/1688 [======================>.......] - ETA: 0s - loss: 0.0311 - accuracy: 0.9905
1376/1688 [=======================>......] - ETA: 0s - loss: 0.0312 - accuracy: 0.9905
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0312 - accuracy: 0.9905
1435/1688 [========================>.....] - ETA: 0s - loss: 0.0312 - accuracy: 0.9904
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0312 - accuracy: 0.9904
1493/1688 [=========================>....] - ETA: 0s - loss: 0.0313 - accuracy: 0.9904
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0313 - accuracy: 0.9904
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0313 - accuracy: 0.9904
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0314 - accuracy: 0.9904
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0314 - accuracy: 0.9904
1639/1688 [============================>.] - ETA: 0s - loss: 0.0314 - accuracy: 0.9904
1668/1688 [============================>.] - ETA: 0s - loss: 0.0314 - accuracy: 0.9904
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0315 - accuracy: 0.9903 - val_loss: 0.0483 - val_accuracy: 0.9870
Epoch 4/10
1/1688 [..............................] - ETA: 2s - loss: 0.0030 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9922
60/1688 [>.............................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9921
88/1688 [>.............................] - ETA: 2s - loss: 0.0251 - accuracy: 0.9916
117/1688 [=>............................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9914
146/1688 [=>............................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9913
175/1688 [==>...........................] - ETA: 2s - loss: 0.0255 - accuracy: 0.9914
204/1688 [==>...........................] - ETA: 2s - loss: 0.0250 - accuracy: 0.9915
234/1688 [===>..........................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9917
263/1688 [===>..........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9918
292/1688 [====>.........................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9920
321/1688 [====>.........................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9921
350/1688 [=====>........................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9923
379/1688 [=====>........................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9924
408/1688 [======>.......................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9925
437/1688 [======>.......................] - ETA: 2s - loss: 0.0224 - accuracy: 0.9926
466/1688 [=======>......................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9926
495/1688 [=======>......................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9927
524/1688 [========>.....................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9927
553/1688 [========>.....................] - ETA: 1s - loss: 0.0219 - accuracy: 0.9928
582/1688 [=========>....................] - ETA: 1s - loss: 0.0218 - accuracy: 0.9928
611/1688 [=========>....................] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
639/1688 [==========>...................] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
668/1688 [==========>...................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
697/1688 [===========>..................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
727/1688 [===========>..................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
756/1688 [============>.................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
785/1688 [============>.................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
814/1688 [=============>................] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
844/1688 [==============>...............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
873/1688 [==============>...............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
902/1688 [===============>..............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
931/1688 [===============>..............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
960/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9928
989/1688 [================>.............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1018/1688 [=================>............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1048/1688 [=================>............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
1106/1688 [==================>...........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9928
1135/1688 [===================>..........] - ETA: 0s - loss: 0.0218 - accuracy: 0.9928
1165/1688 [===================>..........] - ETA: 0s - loss: 0.0218 - accuracy: 0.9928
1195/1688 [====================>.........] - ETA: 0s - loss: 0.0219 - accuracy: 0.9927
1225/1688 [====================>.........] - ETA: 0s - loss: 0.0219 - accuracy: 0.9927
1254/1688 [=====================>........] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1283/1688 [=====================>........] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0220 - accuracy: 0.9927
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9927
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9927
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0222 - accuracy: 0.9927
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0222 - accuracy: 0.9926
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0222 - accuracy: 0.9926
1486/1688 [=========================>....] - ETA: 0s - loss: 0.0223 - accuracy: 0.9926
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0223 - accuracy: 0.9926
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0224 - accuracy: 0.9926
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0224 - accuracy: 0.9926
1603/1688 [===========================>..] - ETA: 0s - loss: 0.0225 - accuracy: 0.9926
1632/1688 [============================>.] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1661/1688 [============================>.] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0226 - accuracy: 0.9925 - val_loss: 0.0524 - val_accuracy: 0.9870
Epoch 5/10
1/1688 [..............................] - ETA: 3s - loss: 0.0098 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9972
59/1688 [>.............................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9961
88/1688 [>.............................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9958
118/1688 [=>............................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9956
147/1688 [=>............................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9954
176/1688 [==>...........................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9953
205/1688 [==>...........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9953
235/1688 [===>..........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9952
264/1688 [===>..........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9952
293/1688 [====>.........................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9951
323/1688 [====>.........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9951
352/1688 [=====>........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9951
381/1688 [=====>........................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9951
410/1688 [======>.......................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9951
439/1688 [======>.......................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9951
468/1688 [=======>......................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9951
497/1688 [=======>......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
526/1688 [========>.....................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
555/1688 [========>.....................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9951
584/1688 [=========>....................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
614/1688 [=========>....................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
643/1688 [==========>...................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
672/1688 [==========>...................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
702/1688 [===========>..................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
731/1688 [===========>..................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9951
760/1688 [============>.................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9950
789/1688 [=============>................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9950
818/1688 [=============>................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9950
847/1688 [==============>...............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9950
876/1688 [==============>...............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9950
905/1688 [===============>..............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9950
934/1688 [===============>..............] - ETA: 1s - loss: 0.0145 - accuracy: 0.9949
963/1688 [================>.............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9949
992/1688 [================>.............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9949
1022/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9949
1051/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9949
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0148 - accuracy: 0.9948
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0149 - accuracy: 0.9948
1138/1688 [===================>..........] - ETA: 0s - loss: 0.0150 - accuracy: 0.9948
1166/1688 [===================>..........] - ETA: 0s - loss: 0.0151 - accuracy: 0.9948
1195/1688 [====================>.........] - ETA: 0s - loss: 0.0152 - accuracy: 0.9948
1224/1688 [====================>.........] - ETA: 0s - loss: 0.0153 - accuracy: 0.9947
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0154 - accuracy: 0.9947
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0155 - accuracy: 0.9947
1311/1688 [======================>.......] - ETA: 0s - loss: 0.0156 - accuracy: 0.9947
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0157 - accuracy: 0.9946
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0158 - accuracy: 0.9946
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9946
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0160 - accuracy: 0.9945
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0160 - accuracy: 0.9945
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0161 - accuracy: 0.9945
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0162 - accuracy: 0.9945
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9944
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0164 - accuracy: 0.9944
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0164 - accuracy: 0.9944
1634/1688 [============================>.] - ETA: 0s - loss: 0.0165 - accuracy: 0.9944
1663/1688 [============================>.] - ETA: 0s - loss: 0.0166 - accuracy: 0.9943
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0167 - accuracy: 0.9943 - val_loss: 0.0754 - val_accuracy: 0.9817
Epoch 6/10
1/1688 [..............................] - ETA: 2s - loss: 0.0020 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9942
62/1688 [>.............................] - ETA: 2s - loss: 0.0273 - accuracy: 0.9937
91/1688 [>.............................] - ETA: 2s - loss: 0.0265 - accuracy: 0.9934
120/1688 [=>............................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9932
149/1688 [=>............................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9932
178/1688 [==>...........................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9933
207/1688 [==>...........................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9934
236/1688 [===>..........................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9935
266/1688 [===>..........................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9936
296/1688 [====>.........................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9937
325/1688 [====>.........................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9938
354/1688 [=====>........................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9939
383/1688 [=====>........................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9940
412/1688 [======>.......................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9941
441/1688 [======>.......................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9941
470/1688 [=======>......................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9942
499/1688 [=======>......................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9943
528/1688 [========>.....................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9943
558/1688 [========>.....................] - ETA: 1s - loss: 0.0186 - accuracy: 0.9944
587/1688 [=========>....................] - ETA: 1s - loss: 0.0185 - accuracy: 0.9944
616/1688 [=========>....................] - ETA: 1s - loss: 0.0183 - accuracy: 0.9944
645/1688 [==========>...................] - ETA: 1s - loss: 0.0182 - accuracy: 0.9944
674/1688 [==========>...................] - ETA: 1s - loss: 0.0181 - accuracy: 0.9945
703/1688 [===========>..................] - ETA: 1s - loss: 0.0180 - accuracy: 0.9945
732/1688 [============>.................] - ETA: 1s - loss: 0.0179 - accuracy: 0.9945
762/1688 [============>.................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9945
791/1688 [=============>................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9946
821/1688 [=============>................] - ETA: 1s - loss: 0.0176 - accuracy: 0.9946
850/1688 [==============>...............] - ETA: 1s - loss: 0.0175 - accuracy: 0.9946
879/1688 [==============>...............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9946
908/1688 [===============>..............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9947
936/1688 [===============>..............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9947
965/1688 [================>.............] - ETA: 1s - loss: 0.0171 - accuracy: 0.9947
995/1688 [================>.............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9947
1024/1688 [=================>............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9947
1053/1688 [=================>............] - ETA: 1s - loss: 0.0169 - accuracy: 0.9947
1082/1688 [==================>...........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
1111/1688 [==================>...........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
1140/1688 [===================>..........] - ETA: 0s - loss: 0.0167 - accuracy: 0.9948
1169/1688 [===================>..........] - ETA: 0s - loss: 0.0167 - accuracy: 0.9948
1198/1688 [====================>.........] - ETA: 0s - loss: 0.0167 - accuracy: 0.9948
1228/1688 [====================>.........] - ETA: 0s - loss: 0.0166 - accuracy: 0.9948
1258/1688 [=====================>........] - ETA: 0s - loss: 0.0166 - accuracy: 0.9948
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1317/1688 [======================>.......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1346/1688 [======================>.......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1375/1688 [=======================>......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1404/1688 [=======================>......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9948
1433/1688 [========================>.....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1638/1688 [============================>.] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1668/1688 [============================>.] - ETA: 0s - loss: 0.0164 - accuracy: 0.9948
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0164 - accuracy: 0.9948 - val_loss: 0.0481 - val_accuracy: 0.9887
Epoch 7/10
1/1688 [..............................] - ETA: 3s - loss: 0.0089 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9959
61/1688 [>.............................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9958
91/1688 [>.............................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9961
121/1688 [=>............................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9963
151/1688 [=>............................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9964
180/1688 [==>...........................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9965
210/1688 [==>...........................] - ETA: 2s - loss: 0.0103 - accuracy: 0.9965
240/1688 [===>..........................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9967
269/1688 [===>..........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9967
298/1688 [====>.........................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9968
327/1688 [====>.........................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9968
356/1688 [=====>........................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9968
386/1688 [=====>........................] - ETA: 2s - loss: 0.0096 - accuracy: 0.9968
416/1688 [======>.......................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9968
446/1688 [======>.......................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9968
476/1688 [=======>......................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9968
505/1688 [=======>......................] - ETA: 2s - loss: 0.0100 - accuracy: 0.9967
534/1688 [========>.....................] - ETA: 1s - loss: 0.0100 - accuracy: 0.9967
564/1688 [=========>....................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9967
593/1688 [=========>....................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9966
622/1688 [==========>...................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9966
651/1688 [==========>...................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9966
680/1688 [===========>..................] - ETA: 1s - loss: 0.0103 - accuracy: 0.9965
709/1688 [===========>..................] - ETA: 1s - loss: 0.0104 - accuracy: 0.9965
738/1688 [============>.................] - ETA: 1s - loss: 0.0105 - accuracy: 0.9965
767/1688 [============>.................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9964
797/1688 [=============>................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9964
827/1688 [=============>................] - ETA: 1s - loss: 0.0107 - accuracy: 0.9964
856/1688 [==============>...............] - ETA: 1s - loss: 0.0108 - accuracy: 0.9964
885/1688 [==============>...............] - ETA: 1s - loss: 0.0108 - accuracy: 0.9963
914/1688 [===============>..............] - ETA: 1s - loss: 0.0109 - accuracy: 0.9963
944/1688 [===============>..............] - ETA: 1s - loss: 0.0110 - accuracy: 0.9963
973/1688 [================>.............] - ETA: 1s - loss: 0.0111 - accuracy: 0.9963
1002/1688 [================>.............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9962
1031/1688 [=================>............] - ETA: 1s - loss: 0.0112 - accuracy: 0.9962
1060/1688 [=================>............] - ETA: 1s - loss: 0.0113 - accuracy: 0.9962
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0114 - accuracy: 0.9961
1118/1688 [==================>...........] - ETA: 0s - loss: 0.0115 - accuracy: 0.9961
1147/1688 [===================>..........] - ETA: 0s - loss: 0.0116 - accuracy: 0.9961
1177/1688 [===================>..........] - ETA: 0s - loss: 0.0117 - accuracy: 0.9961
1206/1688 [====================>.........] - ETA: 0s - loss: 0.0118 - accuracy: 0.9960
1235/1688 [====================>.........] - ETA: 0s - loss: 0.0119 - accuracy: 0.9960
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0119 - accuracy: 0.9960
1293/1688 [=====================>........] - ETA: 0s - loss: 0.0120 - accuracy: 0.9960
1322/1688 [======================>.......] - ETA: 0s - loss: 0.0120 - accuracy: 0.9960
1351/1688 [=======================>......] - ETA: 0s - loss: 0.0121 - accuracy: 0.9959
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0122 - accuracy: 0.9959
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0122 - accuracy: 0.9959
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0123 - accuracy: 0.9959
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0123 - accuracy: 0.9959
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0124 - accuracy: 0.9958
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0124 - accuracy: 0.9958
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0125 - accuracy: 0.9958
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0125 - accuracy: 0.9958
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0125 - accuracy: 0.9958
1643/1688 [============================>.] - ETA: 0s - loss: 0.0126 - accuracy: 0.9958
1672/1688 [============================>.] - ETA: 0s - loss: 0.0126 - accuracy: 0.9957
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0127 - accuracy: 0.9957 - val_loss: 0.0547 - val_accuracy: 0.9880
Epoch 8/10
1/1688 [..............................] - ETA: 3s - loss: 0.0013 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0291 - accuracy: 0.9966
59/1688 [>.............................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9966
88/1688 [>.............................] - ETA: 2s - loss: 0.0185 - accuracy: 0.9966
117/1688 [=>............................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9967
146/1688 [=>............................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9968
175/1688 [==>...........................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9969
204/1688 [==>...........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9969
233/1688 [===>..........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9969
262/1688 [===>..........................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9970
292/1688 [====>.........................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9970
322/1688 [====>.........................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9970
351/1688 [=====>........................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9970
380/1688 [=====>........................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9971
410/1688 [======>.......................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9971
439/1688 [======>.......................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9971
468/1688 [=======>......................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9971
497/1688 [=======>......................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9971
526/1688 [========>.....................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9971
555/1688 [========>.....................] - ETA: 1s - loss: 0.0108 - accuracy: 0.9972
584/1688 [=========>....................] - ETA: 1s - loss: 0.0106 - accuracy: 0.9972
613/1688 [=========>....................] - ETA: 1s - loss: 0.0105 - accuracy: 0.9972
642/1688 [==========>...................] - ETA: 1s - loss: 0.0104 - accuracy: 0.9972
672/1688 [==========>...................] - ETA: 1s - loss: 0.0103 - accuracy: 0.9972
701/1688 [===========>..................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9972
730/1688 [===========>..................] - ETA: 1s - loss: 0.0102 - accuracy: 0.9972
758/1688 [============>.................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9972
787/1688 [============>.................] - ETA: 1s - loss: 0.0101 - accuracy: 0.9972
816/1688 [=============>................] - ETA: 1s - loss: 0.0100 - accuracy: 0.9972
845/1688 [==============>...............] - ETA: 1s - loss: 0.0100 - accuracy: 0.9972
874/1688 [==============>...............] - ETA: 1s - loss: 0.0100 - accuracy: 0.9972
903/1688 [===============>..............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9972
933/1688 [===============>..............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9972
963/1688 [================>.............] - ETA: 1s - loss: 0.0099 - accuracy: 0.9971
992/1688 [================>.............] - ETA: 1s - loss: 0.0098 - accuracy: 0.9971
1021/1688 [=================>............] - ETA: 1s - loss: 0.0098 - accuracy: 0.9971
1049/1688 [=================>............] - ETA: 1s - loss: 0.0098 - accuracy: 0.9971
1078/1688 [==================>...........] - ETA: 1s - loss: 0.0098 - accuracy: 0.9971
1107/1688 [==================>...........] - ETA: 1s - loss: 0.0097 - accuracy: 0.9971
1136/1688 [===================>..........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9971
1165/1688 [===================>..........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9971
1194/1688 [====================>.........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9971
1223/1688 [====================>.........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9971
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1283/1688 [=====================>........] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9971
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9970
1636/1688 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9970
1665/1688 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9970
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0096 - accuracy: 0.9970 - val_loss: 0.0478 - val_accuracy: 0.9893
Epoch 9/10
1/1688 [..............................] - ETA: 2s - loss: 4.8947e-04 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0052 - accuracy: 0.9970
61/1688 [>.............................] - ETA: 2s - loss: 0.0051 - accuracy: 0.9977
90/1688 [>.............................] - ETA: 2s - loss: 0.0054 - accuracy: 0.9978
119/1688 [=>............................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9978
148/1688 [=>............................] - ETA: 2s - loss: 0.0057 - accuracy: 0.9978
177/1688 [==>...........................] - ETA: 2s - loss: 0.0058 - accuracy: 0.9979
207/1688 [==>...........................] - ETA: 2s - loss: 0.0059 - accuracy: 0.9978
236/1688 [===>..........................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9978
266/1688 [===>..........................] - ETA: 2s - loss: 0.0063 - accuracy: 0.9977
295/1688 [====>.........................] - ETA: 2s - loss: 0.0064 - accuracy: 0.9977
324/1688 [====>.........................] - ETA: 2s - loss: 0.0065 - accuracy: 0.9976
354/1688 [=====>........................] - ETA: 2s - loss: 0.0066 - accuracy: 0.9976
383/1688 [=====>........................] - ETA: 2s - loss: 0.0067 - accuracy: 0.9976
412/1688 [======>.......................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9976
441/1688 [======>.......................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9976
470/1688 [=======>......................] - ETA: 2s - loss: 0.0070 - accuracy: 0.9975
499/1688 [=======>......................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9975
528/1688 [========>.....................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9975
557/1688 [========>.....................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9975
587/1688 [=========>....................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9975
616/1688 [=========>....................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
645/1688 [==========>...................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
674/1688 [==========>...................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
703/1688 [===========>..................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
732/1688 [============>.................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
761/1688 [============>.................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9975
790/1688 [=============>................] - ETA: 1s - loss: 0.0074 - accuracy: 0.9975
819/1688 [=============>................] - ETA: 1s - loss: 0.0074 - accuracy: 0.9975
848/1688 [==============>...............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9975
877/1688 [==============>...............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9975
906/1688 [===============>..............] - ETA: 1s - loss: 0.0075 - accuracy: 0.9975
935/1688 [===============>..............] - ETA: 1s - loss: 0.0075 - accuracy: 0.9975
964/1688 [================>.............] - ETA: 1s - loss: 0.0075 - accuracy: 0.9975
993/1688 [================>.............] - ETA: 1s - loss: 0.0076 - accuracy: 0.9975
1022/1688 [=================>............] - ETA: 1s - loss: 0.0076 - accuracy: 0.9975
1051/1688 [=================>............] - ETA: 1s - loss: 0.0076 - accuracy: 0.9975
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0077 - accuracy: 0.9975
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0077 - accuracy: 0.9975
1138/1688 [===================>..........] - ETA: 0s - loss: 0.0077 - accuracy: 0.9975
1167/1688 [===================>..........] - ETA: 0s - loss: 0.0077 - accuracy: 0.9975
1196/1688 [====================>.........] - ETA: 0s - loss: 0.0078 - accuracy: 0.9975
1226/1688 [====================>.........] - ETA: 0s - loss: 0.0078 - accuracy: 0.9975
1256/1688 [=====================>........] - ETA: 0s - loss: 0.0078 - accuracy: 0.9974
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0079 - accuracy: 0.9974
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0079 - accuracy: 0.9974
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0080 - accuracy: 0.9974
1374/1688 [=======================>......] - ETA: 0s - loss: 0.0080 - accuracy: 0.9974
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0080 - accuracy: 0.9974
1433/1688 [========================>.....] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1462/1688 [========================>.....] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0081 - accuracy: 0.9974
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0082 - accuracy: 0.9974
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0083 - accuracy: 0.9974
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1640/1688 [============================>.] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1669/1688 [============================>.] - ETA: 0s - loss: 0.0084 - accuracy: 0.9973
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0084 - accuracy: 0.9973 - val_loss: 0.0639 - val_accuracy: 0.9845
Epoch 10/10
1/1688 [..............................] - ETA: 3s - loss: 0.0489 - accuracy: 0.9688
32/1688 [..............................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9895
61/1688 [>.............................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9921
91/1688 [>.............................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9932
121/1688 [=>............................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9941
150/1688 [=>............................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9947
179/1688 [==>...........................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9951
208/1688 [==>...........................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9955
237/1688 [===>..........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9958
266/1688 [===>..........................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9960
296/1688 [====>.........................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9961
325/1688 [====>.........................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9962
354/1688 [=====>........................] - ETA: 2s - loss: 0.0104 - accuracy: 0.9963
383/1688 [=====>........................] - ETA: 2s - loss: 0.0103 - accuracy: 0.9964
412/1688 [======>.......................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9965
441/1688 [======>.......................] - ETA: 2s - loss: 0.0100 - accuracy: 0.9966
470/1688 [=======>......................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9966
499/1688 [=======>......................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9967
528/1688 [========>.....................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9967
557/1688 [========>.....................] - ETA: 1s - loss: 0.0094 - accuracy: 0.9968
585/1688 [=========>....................] - ETA: 1s - loss: 0.0093 - accuracy: 0.9968
614/1688 [=========>....................] - ETA: 1s - loss: 0.0092 - accuracy: 0.9968
643/1688 [==========>...................] - ETA: 1s - loss: 0.0092 - accuracy: 0.9969
672/1688 [==========>...................] - ETA: 1s - loss: 0.0091 - accuracy: 0.9969
702/1688 [===========>..................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9969
731/1688 [===========>..................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9969
760/1688 [============>.................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9969
788/1688 [=============>................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9969
817/1688 [=============>................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9969
846/1688 [==============>...............] - ETA: 1s - loss: 0.0089 - accuracy: 0.9970
875/1688 [==============>...............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
904/1688 [===============>..............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
933/1688 [===============>..............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
962/1688 [================>.............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
991/1688 [================>.............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1021/1688 [=================>............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1050/1688 [=================>............] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0088 - accuracy: 0.9970
1139/1688 [===================>..........] - ETA: 0s - loss: 0.0088 - accuracy: 0.9970
1168/1688 [===================>..........] - ETA: 0s - loss: 0.0088 - accuracy: 0.9970
1198/1688 [====================>.........] - ETA: 0s - loss: 0.0088 - accuracy: 0.9970
1227/1688 [====================>.........] - ETA: 0s - loss: 0.0088 - accuracy: 0.9970
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1316/1688 [======================>.......] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1346/1688 [======================>.......] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1375/1688 [=======================>......] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1434/1688 [========================>.....] - ETA: 0s - loss: 0.0089 - accuracy: 0.9970
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0090 - accuracy: 0.9970
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0090 - accuracy: 0.9970
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0090 - accuracy: 0.9970
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0090 - accuracy: 0.9970
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0091 - accuracy: 0.9970
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0091 - accuracy: 0.9970
1638/1688 [============================>.] - ETA: 0s - loss: 0.0091 - accuracy: 0.9970
1667/1688 [============================>.] - ETA: 0s - loss: 0.0091 - accuracy: 0.9970
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0091 - accuracy: 0.9969 - val_loss: 0.0432 - val_accuracy: 0.9907
Test score: 0.04563219100236893
Test accuracy: 0.9890000224113464
4. Model quantization¶
We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.
For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).
The quantized model is a Keras model where the neural layers (Conv2D, Dense)
and the ReLU layers are replaced with custom CNN2SNN quantized layers
(QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions
can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function folds the batch normalization layers into
the corresponding neural layer. The new weights are computed
according to this folding operation.
from cnn2snn import quantize
model_quantized = quantize(model_keras,
input_weight_quantization=8,
weight_quantization=4,
activ_quantization=4)
model_quantized.summary()
Out:
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (QuantizedConv2D) (None, 26, 26, 32) 320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32) 0
_________________________________________________________________
re_lu (ActivationDiscreteRel (None, 13, 13, 32) 0
_________________________________________________________________
separable_conv2d (QuantizedS (None, 13, 13, 64) 2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64) 0
_________________________________________________________________
re_lu_1 (ActivationDiscreteR (None, 7, 7, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 3136) 0
_________________________________________________________________
dense (QuantizedDense) (None, 10) 31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________
Check the quantized model accuracy.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])
Out:
Test accuracy after 8-4-4 quantization: 0.9866999983787537
Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.
Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.
Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.
from cnn2snn import quantize_layer
model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])
# To recover the original model accuracy, a quantization-aware training phase
# is required.
Out:
Test accuracy after low bitwidth quantization: 0.7461000084877014
5. Model fine tuning (quantization-aware training)¶
This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.
Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])
Out:
Epoch 1/5
1/1688 [..............................] - ETA: 18:55 - loss: 0.5271 - accuracy: 0.7812
21/1688 [..............................] - ETA: 4s - loss: 0.4219 - accuracy: 0.8586
41/1688 [..............................] - ETA: 4s - loss: 0.3135 - accuracy: 0.9017
61/1688 [>.............................] - ETA: 4s - loss: 0.2557 - accuracy: 0.9232
81/1688 [>.............................] - ETA: 4s - loss: 0.2172 - accuracy: 0.9363
101/1688 [>.............................] - ETA: 4s - loss: 0.1944 - accuracy: 0.9418
121/1688 [=>............................] - ETA: 4s - loss: 0.1734 - accuracy: 0.9486
141/1688 [=>............................] - ETA: 3s - loss: 0.1555 - accuracy: 0.9546
161/1688 [=>............................] - ETA: 3s - loss: 0.1457 - accuracy: 0.9571
181/1688 [==>...........................] - ETA: 3s - loss: 0.1387 - accuracy: 0.9584
201/1688 [==>...........................] - ETA: 3s - loss: 0.1301 - accuracy: 0.9611
221/1688 [==>...........................] - ETA: 3s - loss: 0.1228 - accuracy: 0.9632
241/1688 [===>..........................] - ETA: 3s - loss: 0.1167 - accuracy: 0.9651
261/1688 [===>..........................] - ETA: 3s - loss: 0.1141 - accuracy: 0.9658
281/1688 [===>..........................] - ETA: 3s - loss: 0.1104 - accuracy: 0.9667
301/1688 [====>.........................] - ETA: 3s - loss: 0.1079 - accuracy: 0.9674
321/1688 [====>.........................] - ETA: 3s - loss: 0.1055 - accuracy: 0.9679
341/1688 [=====>........................] - ETA: 3s - loss: 0.1021 - accuracy: 0.9690
361/1688 [=====>........................] - ETA: 3s - loss: 0.0991 - accuracy: 0.9697
381/1688 [=====>........................] - ETA: 3s - loss: 0.0964 - accuracy: 0.9704
401/1688 [======>.......................] - ETA: 3s - loss: 0.0945 - accuracy: 0.9709
421/1688 [======>.......................] - ETA: 3s - loss: 0.0919 - accuracy: 0.9715
441/1688 [======>.......................] - ETA: 3s - loss: 0.0895 - accuracy: 0.9722
461/1688 [=======>......................] - ETA: 3s - loss: 0.0873 - accuracy: 0.9726
481/1688 [=======>......................] - ETA: 3s - loss: 0.0846 - accuracy: 0.9736
500/1688 [=======>......................] - ETA: 3s - loss: 0.0822 - accuracy: 0.9743
520/1688 [========>.....................] - ETA: 2s - loss: 0.0802 - accuracy: 0.9749
540/1688 [========>.....................] - ETA: 2s - loss: 0.0793 - accuracy: 0.9752
560/1688 [========>.....................] - ETA: 2s - loss: 0.0779 - accuracy: 0.9757
580/1688 [=========>....................] - ETA: 2s - loss: 0.0770 - accuracy: 0.9759
600/1688 [=========>....................] - ETA: 2s - loss: 0.0757 - accuracy: 0.9764
620/1688 [==========>...................] - ETA: 2s - loss: 0.0742 - accuracy: 0.9769
640/1688 [==========>...................] - ETA: 2s - loss: 0.0736 - accuracy: 0.9770
660/1688 [==========>...................] - ETA: 2s - loss: 0.0730 - accuracy: 0.9772
680/1688 [===========>..................] - ETA: 2s - loss: 0.0718 - accuracy: 0.9776
700/1688 [===========>..................] - ETA: 2s - loss: 0.0709 - accuracy: 0.9779
720/1688 [===========>..................] - ETA: 2s - loss: 0.0695 - accuracy: 0.9783
740/1688 [============>.................] - ETA: 2s - loss: 0.0682 - accuracy: 0.9785
760/1688 [============>.................] - ETA: 2s - loss: 0.0673 - accuracy: 0.9789
780/1688 [============>.................] - ETA: 2s - loss: 0.0666 - accuracy: 0.9790
800/1688 [=============>................] - ETA: 2s - loss: 0.0656 - accuracy: 0.9793
820/1688 [=============>................] - ETA: 2s - loss: 0.0649 - accuracy: 0.9795
840/1688 [=============>................] - ETA: 2s - loss: 0.0645 - accuracy: 0.9795
860/1688 [==============>...............] - ETA: 2s - loss: 0.0637 - accuracy: 0.9797
880/1688 [==============>...............] - ETA: 2s - loss: 0.0633 - accuracy: 0.9798
900/1688 [==============>...............] - ETA: 2s - loss: 0.0627 - accuracy: 0.9800
920/1688 [===============>..............] - ETA: 1s - loss: 0.0620 - accuracy: 0.9803
940/1688 [===============>..............] - ETA: 1s - loss: 0.0611 - accuracy: 0.9805
960/1688 [================>.............] - ETA: 1s - loss: 0.0602 - accuracy: 0.9807
980/1688 [================>.............] - ETA: 1s - loss: 0.0597 - accuracy: 0.9808
1000/1688 [================>.............] - ETA: 1s - loss: 0.0592 - accuracy: 0.9809
1020/1688 [=================>............] - ETA: 1s - loss: 0.0588 - accuracy: 0.9811
1040/1688 [=================>............] - ETA: 1s - loss: 0.0586 - accuracy: 0.9812
1060/1688 [=================>............] - ETA: 1s - loss: 0.0581 - accuracy: 0.9814
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0575 - accuracy: 0.9816
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0574 - accuracy: 0.9816
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0571 - accuracy: 0.9817
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0567 - accuracy: 0.9818
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0566 - accuracy: 0.9817
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0560 - accuracy: 0.9819
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0560 - accuracy: 0.9821
1220/1688 [====================>.........] - ETA: 1s - loss: 0.0555 - accuracy: 0.9822
1240/1688 [=====================>........] - ETA: 1s - loss: 0.0552 - accuracy: 0.9823
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0547 - accuracy: 0.9824
1280/1688 [=====================>........] - ETA: 1s - loss: 0.0542 - accuracy: 0.9826
1300/1688 [======================>.......] - ETA: 0s - loss: 0.0539 - accuracy: 0.9826
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0534 - accuracy: 0.9828
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0531 - accuracy: 0.9829
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0529 - accuracy: 0.9829
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0527 - accuracy: 0.9830
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0525 - accuracy: 0.9830
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0520 - accuracy: 0.9831
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0515 - accuracy: 0.9833
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0511 - accuracy: 0.9834
1480/1688 [=========================>....] - ETA: 0s - loss: 0.0507 - accuracy: 0.9835
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0504 - accuracy: 0.9836
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0501 - accuracy: 0.9837
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0497 - accuracy: 0.9838
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0496 - accuracy: 0.9839
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0492 - accuracy: 0.9840
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0492 - accuracy: 0.9839
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0489 - accuracy: 0.9840
1640/1688 [============================>.] - ETA: 0s - loss: 0.0486 - accuracy: 0.9841
1660/1688 [============================>.] - ETA: 0s - loss: 0.0485 - accuracy: 0.9841
1680/1688 [============================>.] - ETA: 0s - loss: 0.0481 - accuracy: 0.9843
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0480 - accuracy: 0.9844 - val_loss: 0.0479 - val_accuracy: 0.9877
Epoch 2/5
1/1688 [..............................] - ETA: 4s - loss: 0.0017 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.0281 - accuracy: 0.9940
41/1688 [..............................] - ETA: 4s - loss: 0.0288 - accuracy: 0.9916
61/1688 [>.............................] - ETA: 4s - loss: 0.0259 - accuracy: 0.9913
81/1688 [>.............................] - ETA: 4s - loss: 0.0228 - accuracy: 0.9923
101/1688 [>.............................] - ETA: 4s - loss: 0.0235 - accuracy: 0.9920
121/1688 [=>............................] - ETA: 4s - loss: 0.0236 - accuracy: 0.9917
141/1688 [=>............................] - ETA: 3s - loss: 0.0252 - accuracy: 0.9916
161/1688 [=>............................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9920
181/1688 [==>...........................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9922
201/1688 [==>...........................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9919
221/1688 [==>...........................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9924
241/1688 [===>..........................] - ETA: 3s - loss: 0.0237 - accuracy: 0.9922
261/1688 [===>..........................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9925
281/1688 [===>..........................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9922
301/1688 [====>.........................] - ETA: 3s - loss: 0.0244 - accuracy: 0.9919
321/1688 [====>.........................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9920
341/1688 [=====>........................] - ETA: 3s - loss: 0.0244 - accuracy: 0.9919
361/1688 [=====>........................] - ETA: 3s - loss: 0.0248 - accuracy: 0.9917
381/1688 [=====>........................] - ETA: 3s - loss: 0.0252 - accuracy: 0.9916
401/1688 [======>.......................] - ETA: 3s - loss: 0.0250 - accuracy: 0.9918
421/1688 [======>.......................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9920
441/1688 [======>.......................] - ETA: 3s - loss: 0.0242 - accuracy: 0.9920
461/1688 [=======>......................] - ETA: 3s - loss: 0.0244 - accuracy: 0.9919
481/1688 [=======>......................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9920
501/1688 [=======>......................] - ETA: 3s - loss: 0.0241 - accuracy: 0.9921
521/1688 [========>.....................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9921
541/1688 [========>.....................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9921
561/1688 [========>.....................] - ETA: 2s - loss: 0.0248 - accuracy: 0.9918
581/1688 [=========>....................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9918
601/1688 [=========>....................] - ETA: 2s - loss: 0.0246 - accuracy: 0.9918
621/1688 [==========>...................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9919
641/1688 [==========>...................] - ETA: 2s - loss: 0.0239 - accuracy: 0.9922
661/1688 [==========>...................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9922
681/1688 [===========>..................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9922
701/1688 [===========>..................] - ETA: 2s - loss: 0.0236 - accuracy: 0.9923
721/1688 [===========>..................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9924
741/1688 [============>.................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9924
761/1688 [============>.................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9923
781/1688 [============>.................] - ETA: 2s - loss: 0.0239 - accuracy: 0.9922
801/1688 [=============>................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9922
821/1688 [=============>................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9922
841/1688 [=============>................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9922
860/1688 [==============>...............] - ETA: 2s - loss: 0.0239 - accuracy: 0.9922
880/1688 [==============>...............] - ETA: 2s - loss: 0.0242 - accuracy: 0.9922
900/1688 [==============>...............] - ETA: 2s - loss: 0.0243 - accuracy: 0.9921
920/1688 [===============>..............] - ETA: 1s - loss: 0.0245 - accuracy: 0.9921
940/1688 [===============>..............] - ETA: 1s - loss: 0.0246 - accuracy: 0.9921
960/1688 [================>.............] - ETA: 1s - loss: 0.0244 - accuracy: 0.9922
980/1688 [================>.............] - ETA: 1s - loss: 0.0242 - accuracy: 0.9923
1000/1688 [================>.............] - ETA: 1s - loss: 0.0243 - accuracy: 0.9922
1020/1688 [=================>............] - ETA: 1s - loss: 0.0242 - accuracy: 0.9923
1040/1688 [=================>............] - ETA: 1s - loss: 0.0242 - accuracy: 0.9922
1060/1688 [=================>............] - ETA: 1s - loss: 0.0243 - accuracy: 0.9922
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0242 - accuracy: 0.9921
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0243 - accuracy: 0.9920
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0243 - accuracy: 0.9920
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0245 - accuracy: 0.9919
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0244 - accuracy: 0.9919
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0244 - accuracy: 0.9920
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0244 - accuracy: 0.9920
1220/1688 [====================>.........] - ETA: 1s - loss: 0.0245 - accuracy: 0.9920
1240/1688 [=====================>........] - ETA: 1s - loss: 0.0245 - accuracy: 0.9921
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0246 - accuracy: 0.9920
1280/1688 [=====================>........] - ETA: 1s - loss: 0.0246 - accuracy: 0.9919
1300/1688 [======================>.......] - ETA: 0s - loss: 0.0244 - accuracy: 0.9920
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0243 - accuracy: 0.9921
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0242 - accuracy: 0.9921
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0246 - accuracy: 0.9920
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0245 - accuracy: 0.9919
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0246 - accuracy: 0.9919
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0248 - accuracy: 0.9918
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0250 - accuracy: 0.9918
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0252 - accuracy: 0.9918
1480/1688 [=========================>....] - ETA: 0s - loss: 0.0252 - accuracy: 0.9918
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0251 - accuracy: 0.9918
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0250 - accuracy: 0.9919
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0249 - accuracy: 0.9919
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0249 - accuracy: 0.9919
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0248 - accuracy: 0.9919
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0247 - accuracy: 0.9920
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0246 - accuracy: 0.9920
1640/1688 [============================>.] - ETA: 0s - loss: 0.0246 - accuracy: 0.9919
1660/1688 [============================>.] - ETA: 0s - loss: 0.0246 - accuracy: 0.9919
1680/1688 [============================>.] - ETA: 0s - loss: 0.0247 - accuracy: 0.9919
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0247 - accuracy: 0.9919 - val_loss: 0.0577 - val_accuracy: 0.9845
Epoch 3/5
1/1688 [..............................] - ETA: 4s - loss: 0.0016 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.0271 - accuracy: 0.9926
41/1688 [..............................] - ETA: 4s - loss: 0.0209 - accuracy: 0.9931
61/1688 [>.............................] - ETA: 4s - loss: 0.0220 - accuracy: 0.9913
81/1688 [>.............................] - ETA: 4s - loss: 0.0190 - accuracy: 0.9927
101/1688 [>.............................] - ETA: 4s - loss: 0.0181 - accuracy: 0.9929
121/1688 [=>............................] - ETA: 4s - loss: 0.0183 - accuracy: 0.9933
141/1688 [=>............................] - ETA: 3s - loss: 0.0177 - accuracy: 0.9936
161/1688 [=>............................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9932
181/1688 [==>...........................] - ETA: 3s - loss: 0.0194 - accuracy: 0.9933
201/1688 [==>...........................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9928
221/1688 [==>...........................] - ETA: 3s - loss: 0.0222 - accuracy: 0.9924
241/1688 [===>..........................] - ETA: 3s - loss: 0.0215 - accuracy: 0.9926
261/1688 [===>..........................] - ETA: 3s - loss: 0.0225 - accuracy: 0.9921
281/1688 [===>..........................] - ETA: 3s - loss: 0.0231 - accuracy: 0.9920
301/1688 [====>.........................] - ETA: 3s - loss: 0.0229 - accuracy: 0.9919
321/1688 [====>.........................] - ETA: 3s - loss: 0.0227 - accuracy: 0.9919
341/1688 [=====>........................] - ETA: 3s - loss: 0.0226 - accuracy: 0.9918
361/1688 [=====>........................] - ETA: 3s - loss: 0.0222 - accuracy: 0.9920
381/1688 [=====>........................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9922
401/1688 [======>.......................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9921
421/1688 [======>.......................] - ETA: 3s - loss: 0.0229 - accuracy: 0.9920
441/1688 [======>.......................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9922
461/1688 [=======>......................] - ETA: 3s - loss: 0.0221 - accuracy: 0.9923
481/1688 [=======>......................] - ETA: 3s - loss: 0.0221 - accuracy: 0.9923
501/1688 [=======>......................] - ETA: 3s - loss: 0.0220 - accuracy: 0.9923
521/1688 [========>.....................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9923
541/1688 [========>.....................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9923
561/1688 [========>.....................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9923
581/1688 [=========>....................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9924
601/1688 [=========>....................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9924
621/1688 [==========>...................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9924
641/1688 [==========>...................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9926
661/1688 [==========>...................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9926
681/1688 [===========>..................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9926
701/1688 [===========>..................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9925
721/1688 [===========>..................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9924
741/1688 [============>.................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9924
761/1688 [============>.................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9925
781/1688 [============>.................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9925
801/1688 [=============>................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9925
821/1688 [=============>................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9925
841/1688 [=============>................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9926
861/1688 [==============>...............] - ETA: 2s - loss: 0.0210 - accuracy: 0.9927
881/1688 [==============>...............] - ETA: 2s - loss: 0.0210 - accuracy: 0.9928
901/1688 [===============>..............] - ETA: 2s - loss: 0.0212 - accuracy: 0.9927
921/1688 [===============>..............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9927
941/1688 [===============>..............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9927
961/1688 [================>.............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9926
981/1688 [================>.............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9926
1001/1688 [================>.............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9927
1021/1688 [=================>............] - ETA: 1s - loss: 0.0210 - accuracy: 0.9927
1041/1688 [=================>............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9926
1061/1688 [=================>............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9926
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0210 - accuracy: 0.9926
1101/1688 [==================>...........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9925
1121/1688 [==================>...........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9925
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9926
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9926
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0214 - accuracy: 0.9925
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9924
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0217 - accuracy: 0.9924
1241/1688 [=====================>........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9924
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9924
1281/1688 [=====================>........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9924
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1321/1688 [======================>.......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0211 - accuracy: 0.9925
1361/1688 [=======================>......] - ETA: 0s - loss: 0.0214 - accuracy: 0.9924
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0214 - accuracy: 0.9924
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9924
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0212 - accuracy: 0.9924
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0212 - accuracy: 0.9925
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0212 - accuracy: 0.9925
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0214 - accuracy: 0.9924
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0214 - accuracy: 0.9925
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0216 - accuracy: 0.9925
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9925
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9925
1640/1688 [============================>.] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1660/1688 [============================>.] - ETA: 0s - loss: 0.0218 - accuracy: 0.9924
1680/1688 [============================>.] - ETA: 0s - loss: 0.0218 - accuracy: 0.9924
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0220 - accuracy: 0.9923 - val_loss: 0.0596 - val_accuracy: 0.9840
Epoch 4/5
1/1688 [..............................] - ETA: 5s - loss: 0.0060 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.0253 - accuracy: 0.9911
41/1688 [..............................] - ETA: 4s - loss: 0.0208 - accuracy: 0.9924
61/1688 [>.............................] - ETA: 4s - loss: 0.0194 - accuracy: 0.9933
81/1688 [>.............................] - ETA: 4s - loss: 0.0216 - accuracy: 0.9931
101/1688 [>.............................] - ETA: 4s - loss: 0.0190 - accuracy: 0.9941
121/1688 [=>............................] - ETA: 4s - loss: 0.0196 - accuracy: 0.9941
141/1688 [=>............................] - ETA: 3s - loss: 0.0202 - accuracy: 0.9936
161/1688 [=>............................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9930
181/1688 [==>...........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9926
201/1688 [==>...........................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9927
221/1688 [==>...........................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9929
241/1688 [===>..........................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9931
261/1688 [===>..........................] - ETA: 3s - loss: 0.0213 - accuracy: 0.9934
281/1688 [===>..........................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9932
301/1688 [====>.........................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9931
321/1688 [====>.........................] - ETA: 3s - loss: 0.0221 - accuracy: 0.9930
341/1688 [=====>........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9929
361/1688 [=====>........................] - ETA: 3s - loss: 0.0227 - accuracy: 0.9926
381/1688 [=====>........................] - ETA: 3s - loss: 0.0228 - accuracy: 0.9926
401/1688 [======>.......................] - ETA: 3s - loss: 0.0233 - accuracy: 0.9923
421/1688 [======>.......................] - ETA: 3s - loss: 0.0226 - accuracy: 0.9925
441/1688 [======>.......................] - ETA: 3s - loss: 0.0223 - accuracy: 0.9925
461/1688 [=======>......................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9927
481/1688 [=======>......................] - ETA: 3s - loss: 0.0220 - accuracy: 0.9926
501/1688 [=======>......................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9926
521/1688 [========>.....................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9925
541/1688 [========>.....................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9925
561/1688 [========>.....................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9926
581/1688 [=========>....................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9928
601/1688 [=========>....................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9929
621/1688 [==========>...................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9928
641/1688 [==========>...................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9926
661/1688 [==========>...................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9927
681/1688 [===========>..................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9927
701/1688 [===========>..................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9926
721/1688 [===========>..................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9925
741/1688 [============>.................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9923
761/1688 [============>.................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9924
781/1688 [============>.................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9923
801/1688 [=============>................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9922
821/1688 [=============>................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9922
841/1688 [=============>................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9922
861/1688 [==============>...............] - ETA: 2s - loss: 0.0213 - accuracy: 0.9923
881/1688 [==============>...............] - ETA: 2s - loss: 0.0212 - accuracy: 0.9923
901/1688 [===============>..............] - ETA: 2s - loss: 0.0211 - accuracy: 0.9924
921/1688 [===============>..............] - ETA: 1s - loss: 0.0208 - accuracy: 0.9925
941/1688 [===============>..............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9926
961/1688 [================>.............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9927
981/1688 [================>.............] - ETA: 1s - loss: 0.0205 - accuracy: 0.9928
1001/1688 [================>.............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9929
1021/1688 [=================>............] - ETA: 1s - loss: 0.0205 - accuracy: 0.9929
1041/1688 [=================>............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9929
1061/1688 [=================>............] - ETA: 1s - loss: 0.0203 - accuracy: 0.9930
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0203 - accuracy: 0.9929
1101/1688 [==================>...........] - ETA: 1s - loss: 0.0202 - accuracy: 0.9929
1121/1688 [==================>...........] - ETA: 1s - loss: 0.0205 - accuracy: 0.9929
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0204 - accuracy: 0.9929
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0204 - accuracy: 0.9928
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0205 - accuracy: 0.9928
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0205 - accuracy: 0.9928
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0206 - accuracy: 0.9928
1241/1688 [=====================>........] - ETA: 1s - loss: 0.0208 - accuracy: 0.9926
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0206 - accuracy: 0.9927
1281/1688 [=====================>........] - ETA: 1s - loss: 0.0206 - accuracy: 0.9927
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0206 - accuracy: 0.9927
1321/1688 [======================>.......] - ETA: 0s - loss: 0.0206 - accuracy: 0.9926
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0207 - accuracy: 0.9926
1361/1688 [=======================>......] - ETA: 0s - loss: 0.0205 - accuracy: 0.9927
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0204 - accuracy: 0.9928
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0207 - accuracy: 0.9928
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9927
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9928
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9927
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0210 - accuracy: 0.9927
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0212 - accuracy: 0.9926
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0212 - accuracy: 0.9925
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0214 - accuracy: 0.9925
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0215 - accuracy: 0.9924
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0215 - accuracy: 0.9924
1601/1688 [===========================>..] - ETA: 0s - loss: 0.0215 - accuracy: 0.9924
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0213 - accuracy: 0.9925
1641/1688 [============================>.] - ETA: 0s - loss: 0.0215 - accuracy: 0.9925
1661/1688 [============================>.] - ETA: 0s - loss: 0.0214 - accuracy: 0.9925
1681/1688 [============================>.] - ETA: 0s - loss: 0.0212 - accuracy: 0.9925
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0212 - accuracy: 0.9926 - val_loss: 0.0532 - val_accuracy: 0.9873
Epoch 5/5
1/1688 [..............................] - ETA: 5s - loss: 2.5233e-04 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9955
41/1688 [..............................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9962
61/1688 [>.............................] - ETA: 4s - loss: 0.0139 - accuracy: 0.9964
81/1688 [>.............................] - ETA: 4s - loss: 0.0156 - accuracy: 0.9954
101/1688 [>.............................] - ETA: 4s - loss: 0.0147 - accuracy: 0.9954
121/1688 [=>............................] - ETA: 4s - loss: 0.0163 - accuracy: 0.9954
141/1688 [=>............................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9953
161/1688 [=>............................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9953
181/1688 [==>...........................] - ETA: 3s - loss: 0.0159 - accuracy: 0.9953
201/1688 [==>...........................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9952
221/1688 [==>...........................] - ETA: 3s - loss: 0.0171 - accuracy: 0.9949
241/1688 [===>..........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9949
261/1688 [===>..........................] - ETA: 3s - loss: 0.0167 - accuracy: 0.9951
281/1688 [===>..........................] - ETA: 3s - loss: 0.0175 - accuracy: 0.9948
301/1688 [====>.........................] - ETA: 3s - loss: 0.0172 - accuracy: 0.9949
321/1688 [====>.........................] - ETA: 3s - loss: 0.0175 - accuracy: 0.9946
341/1688 [=====>........................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9947
361/1688 [=====>........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9947
381/1688 [=====>........................] - ETA: 3s - loss: 0.0166 - accuracy: 0.9950
401/1688 [======>.......................] - ETA: 3s - loss: 0.0162 - accuracy: 0.9952
421/1688 [======>.......................] - ETA: 3s - loss: 0.0159 - accuracy: 0.9952
441/1688 [======>.......................] - ETA: 3s - loss: 0.0156 - accuracy: 0.9953
461/1688 [=======>......................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9954
481/1688 [=======>......................] - ETA: 3s - loss: 0.0151 - accuracy: 0.9954
501/1688 [=======>......................] - ETA: 3s - loss: 0.0151 - accuracy: 0.9953
521/1688 [========>.....................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9952
541/1688 [========>.....................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9951
561/1688 [========>.....................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9952
581/1688 [=========>....................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9952
601/1688 [=========>....................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9952
621/1688 [==========>...................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9954
641/1688 [==========>...................] - ETA: 2s - loss: 0.0158 - accuracy: 0.9950
661/1688 [==========>...................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9949
681/1688 [===========>..................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9949
701/1688 [===========>..................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9948
721/1688 [===========>..................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9947
740/1688 [============>.................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9947
760/1688 [============>.................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9947
780/1688 [============>.................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9948
800/1688 [=============>................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9949
820/1688 [=============>................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9950
840/1688 [=============>................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9949
860/1688 [==============>...............] - ETA: 2s - loss: 0.0156 - accuracy: 0.9948
880/1688 [==============>...............] - ETA: 2s - loss: 0.0154 - accuracy: 0.9949
900/1688 [==============>...............] - ETA: 2s - loss: 0.0161 - accuracy: 0.9947
920/1688 [===============>..............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9947
940/1688 [===============>..............] - ETA: 1s - loss: 0.0160 - accuracy: 0.9947
960/1688 [================>.............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9947
980/1688 [================>.............] - ETA: 1s - loss: 0.0163 - accuracy: 0.9946
1000/1688 [================>.............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9947
1020/1688 [=================>............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9946
1040/1688 [=================>............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9947
1060/1688 [=================>............] - ETA: 1s - loss: 0.0162 - accuracy: 0.9947
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0163 - accuracy: 0.9947
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0163 - accuracy: 0.9947
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0165 - accuracy: 0.9946
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9946
1160/1688 [===================>..........] - ETA: 1s - loss: 0.0167 - accuracy: 0.9946
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9945
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9945
1220/1688 [====================>.........] - ETA: 1s - loss: 0.0170 - accuracy: 0.9944
1240/1688 [=====================>........] - ETA: 1s - loss: 0.0171 - accuracy: 0.9945
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0174 - accuracy: 0.9945
1280/1688 [=====================>........] - ETA: 1s - loss: 0.0178 - accuracy: 0.9943
1300/1688 [======================>.......] - ETA: 0s - loss: 0.0181 - accuracy: 0.9942
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0180 - accuracy: 0.9942
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0182 - accuracy: 0.9941
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0183 - accuracy: 0.9941
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0182 - accuracy: 0.9942
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0184 - accuracy: 0.9942
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0182 - accuracy: 0.9942
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0181 - accuracy: 0.9943
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0180 - accuracy: 0.9942
1480/1688 [=========================>....] - ETA: 0s - loss: 0.0180 - accuracy: 0.9942
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0179 - accuracy: 0.9943
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0180 - accuracy: 0.9942
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0181 - accuracy: 0.9942
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0182 - accuracy: 0.9942
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0182 - accuracy: 0.9941
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0181 - accuracy: 0.9941
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0184 - accuracy: 0.9940
1640/1688 [============================>.] - ETA: 0s - loss: 0.0184 - accuracy: 0.9940
1660/1688 [============================>.] - ETA: 0s - loss: 0.0184 - accuracy: 0.9940
1680/1688 [============================>.] - ETA: 0s - loss: 0.0185 - accuracy: 0.9940
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0186 - accuracy: 0.9940 - val_loss: 0.0461 - val_accuracy: 0.9883
Test accuracy after fine tuning: 0.987500011920929
6. Model conversion¶
After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida Execution Engine.
Note
One needs to supply the coefficients used to rescale the input
dataset before the training - here input_scaling.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()
results = model_akida.predict(raw_x_test)
accuracy = (raw_y_test == results).mean()
print('Test accuracy after conversion:', accuracy)
# For non-regression purpose
assert accuracy > 0.97
Out:
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 3
______________________________________________
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(a, dtype, copy=False, order=order)
SW/conv2d-dense (Software)
______________________________________________________________
Layer (type) Output shape Kernel shape
==============================================================
conv2d (InputConv.) [13, 13, 32] (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.) [7, 7, 64] (3, 3, 32, 1)
______________________________________________________________
(1, 1, 32, 64)
______________________________________________________________
dense (Fully.) [1, 1, 10] (1, 1, 3136, 10)
______________________________________________________________
Test accuracy after conversion: 0.9876
Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).
Total running time of the script: ( 1 minutes 2.414 seconds)