Note
Click here to download the full example code
CNN conversion flow tutorial
This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.
The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.
Note
Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.
MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.
1. Load and reshape MNIST dataset
After loading, we make 2 transformations on the dataset:
Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.
Note
At this point, we’ll set aside the raw data for testing our converted model in the Akida runtime later.
Rescale the 8-bit loaded data to the range 0-to-1 for training.
Note
Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.
This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.
Also note that we store the scaling values input_scaling for
use when preparing the model for the Akida neuromorphic IP. The
implementation of the Akida neural network allows us to completely
skip the rescaling step (i.e. the Akida model should be fed with
the raw 8-bit values) but that does require information about what
scaling was applied prior to training - see below for more details.
from tensorflow import keras
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)
# Set aside raw test data for use with Akida runtime later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test
# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a
2. Model definition
Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.
An appropriate model for MNIST (inspired by this example) might look something like the following:
model_keras = keras.models.Sequential([
keras.layers.Conv2D(
filters=32, kernel_size=3, input_shape=(28, 28, 1), strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.SeparableConv2D(
filters=64, kernel_size=3, padding='same', strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 13, 13, 32) 320
batch_normalization (BatchN (None, 13, 13, 32) 128
ormalization)
re_lu (ReLU) (None, 13, 13, 32) 0
separable_conv2d (Separable (None, 7, 7, 64) 2400
Conv2D)
batch_normalization_1 (Batc (None, 7, 7, 64) 256
hNormalization)
re_lu_1 (ReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________
The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):
Standard Conv2D and Dense layers are supported
Hidden layers must be followed by a ReLU layer.
BatchNormalization must always happen before activations.
Convolutional blocks can optionally be followed by a MaxPooling.
The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).
from cnn2snn import check_model_compatibility
print("Model compatible for Akida conversion:",
check_model_compatibility(model_keras))
Model compatible for Akida conversion: True
3. Model training
Before going any further, train the model and get its performance. The created model should have achieved a test accuracy over 98% after 10 epochs.
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Epoch 1/10
1/1688 [..............................] - ETA: 12:58 - loss: 2.9708 - accuracy: 0.0312
30/1688 [..............................] - ETA: 2s - loss: 1.2146 - accuracy: 0.6115
56/1688 [..............................] - ETA: 3s - loss: 0.8667 - accuracy: 0.7310
84/1688 [>.............................] - ETA: 2s - loss: 0.7189 - accuracy: 0.7809
112/1688 [>.............................] - ETA: 2s - loss: 0.6293 - accuracy: 0.8083
140/1688 [=>............................] - ETA: 2s - loss: 0.5766 - accuracy: 0.8254
168/1688 [=>............................] - ETA: 2s - loss: 0.5195 - accuracy: 0.8439
196/1688 [==>...........................] - ETA: 2s - loss: 0.4819 - accuracy: 0.8546
224/1688 [==>...........................] - ETA: 2s - loss: 0.4555 - accuracy: 0.8633
252/1688 [===>..........................] - ETA: 2s - loss: 0.4291 - accuracy: 0.8713
279/1688 [===>..........................] - ETA: 2s - loss: 0.4088 - accuracy: 0.8771
306/1688 [====>.........................] - ETA: 2s - loss: 0.3915 - accuracy: 0.8824
334/1688 [====>.........................] - ETA: 2s - loss: 0.3748 - accuracy: 0.8878
360/1688 [=====>........................] - ETA: 2s - loss: 0.3611 - accuracy: 0.8908
387/1688 [=====>........................] - ETA: 2s - loss: 0.3491 - accuracy: 0.8947
415/1688 [======>.......................] - ETA: 2s - loss: 0.3364 - accuracy: 0.8984
443/1688 [======>.......................] - ETA: 2s - loss: 0.3272 - accuracy: 0.9012
471/1688 [=======>......................] - ETA: 2s - loss: 0.3165 - accuracy: 0.9047
499/1688 [=======>......................] - ETA: 2s - loss: 0.3074 - accuracy: 0.9071
527/1688 [========>.....................] - ETA: 2s - loss: 0.3000 - accuracy: 0.9096
554/1688 [========>.....................] - ETA: 2s - loss: 0.2911 - accuracy: 0.9119
582/1688 [=========>....................] - ETA: 2s - loss: 0.2846 - accuracy: 0.9140
609/1688 [=========>....................] - ETA: 1s - loss: 0.2770 - accuracy: 0.9165
636/1688 [==========>...................] - ETA: 1s - loss: 0.2726 - accuracy: 0.9180
664/1688 [==========>...................] - ETA: 1s - loss: 0.2679 - accuracy: 0.9192
692/1688 [===========>..................] - ETA: 1s - loss: 0.2609 - accuracy: 0.9214
720/1688 [===========>..................] - ETA: 1s - loss: 0.2575 - accuracy: 0.9222
749/1688 [============>.................] - ETA: 1s - loss: 0.2536 - accuracy: 0.9234
777/1688 [============>.................] - ETA: 1s - loss: 0.2485 - accuracy: 0.9248
806/1688 [=============>................] - ETA: 1s - loss: 0.2444 - accuracy: 0.9261
834/1688 [=============>................] - ETA: 1s - loss: 0.2396 - accuracy: 0.9275
862/1688 [==============>...............] - ETA: 1s - loss: 0.2358 - accuracy: 0.9287
890/1688 [==============>...............] - ETA: 1s - loss: 0.2322 - accuracy: 0.9297
918/1688 [===============>..............] - ETA: 1s - loss: 0.2277 - accuracy: 0.9311
946/1688 [===============>..............] - ETA: 1s - loss: 0.2241 - accuracy: 0.9320
973/1688 [================>.............] - ETA: 1s - loss: 0.2212 - accuracy: 0.9329
1001/1688 [================>.............] - ETA: 1s - loss: 0.2191 - accuracy: 0.9338
1028/1688 [=================>............] - ETA: 1s - loss: 0.2164 - accuracy: 0.9345
1056/1688 [=================>............] - ETA: 1s - loss: 0.2135 - accuracy: 0.9353
1085/1688 [==================>...........] - ETA: 1s - loss: 0.2105 - accuracy: 0.9361
1112/1688 [==================>...........] - ETA: 1s - loss: 0.2081 - accuracy: 0.9367
1141/1688 [===================>..........] - ETA: 1s - loss: 0.2054 - accuracy: 0.9375
1168/1688 [===================>..........] - ETA: 0s - loss: 0.2031 - accuracy: 0.9383
1196/1688 [====================>.........] - ETA: 0s - loss: 0.2008 - accuracy: 0.9390
1224/1688 [====================>.........] - ETA: 0s - loss: 0.1990 - accuracy: 0.9396
1251/1688 [=====================>........] - ETA: 0s - loss: 0.1965 - accuracy: 0.9404
1278/1688 [=====================>........] - ETA: 0s - loss: 0.1946 - accuracy: 0.9410
1306/1688 [======================>.......] - ETA: 0s - loss: 0.1929 - accuracy: 0.9414
1333/1688 [======================>.......] - ETA: 0s - loss: 0.1913 - accuracy: 0.9419
1361/1688 [=======================>......] - ETA: 0s - loss: 0.1907 - accuracy: 0.9422
1388/1688 [=======================>......] - ETA: 0s - loss: 0.1888 - accuracy: 0.9427
1416/1688 [========================>.....] - ETA: 0s - loss: 0.1866 - accuracy: 0.9434
1444/1688 [========================>.....] - ETA: 0s - loss: 0.1849 - accuracy: 0.9439
1472/1688 [=========================>....] - ETA: 0s - loss: 0.1836 - accuracy: 0.9443
1499/1688 [=========================>....] - ETA: 0s - loss: 0.1820 - accuracy: 0.9448
1527/1688 [==========================>...] - ETA: 0s - loss: 0.1803 - accuracy: 0.9452
1555/1688 [==========================>...] - ETA: 0s - loss: 0.1794 - accuracy: 0.9456
1583/1688 [===========================>..] - ETA: 0s - loss: 0.1776 - accuracy: 0.9461
1611/1688 [===========================>..] - ETA: 0s - loss: 0.1764 - accuracy: 0.9467
1640/1688 [============================>.] - ETA: 0s - loss: 0.1746 - accuracy: 0.9472
1668/1688 [============================>.] - ETA: 0s - loss: 0.1733 - accuracy: 0.9477
1688/1688 [==============================] - ETA: 0s - loss: 0.1723 - accuracy: 0.9479
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1723 - accuracy: 0.9479 - val_loss: 0.0930 - val_accuracy: 0.9732
Epoch 2/10
1/1688 [..............................] - ETA: 3s - loss: 0.0485 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0868 - accuracy: 0.9778
60/1688 [>.............................] - ETA: 2s - loss: 0.0890 - accuracy: 0.9740
87/1688 [>.............................] - ETA: 2s - loss: 0.0788 - accuracy: 0.9770
115/1688 [=>............................] - ETA: 2s - loss: 0.0769 - accuracy: 0.9772
143/1688 [=>............................] - ETA: 2s - loss: 0.0700 - accuracy: 0.9790
171/1688 [==>...........................] - ETA: 2s - loss: 0.0710 - accuracy: 0.9786
199/1688 [==>...........................] - ETA: 2s - loss: 0.0711 - accuracy: 0.9779
226/1688 [===>..........................] - ETA: 2s - loss: 0.0691 - accuracy: 0.9786
253/1688 [===>..........................] - ETA: 2s - loss: 0.0692 - accuracy: 0.9784
280/1688 [===>..........................] - ETA: 2s - loss: 0.0683 - accuracy: 0.9792
307/1688 [====>.........................] - ETA: 2s - loss: 0.0671 - accuracy: 0.9793
335/1688 [====>.........................] - ETA: 2s - loss: 0.0673 - accuracy: 0.9791
362/1688 [=====>........................] - ETA: 2s - loss: 0.0676 - accuracy: 0.9789
390/1688 [=====>........................] - ETA: 2s - loss: 0.0689 - accuracy: 0.9788
418/1688 [======>.......................] - ETA: 2s - loss: 0.0696 - accuracy: 0.9785
446/1688 [======>.......................] - ETA: 2s - loss: 0.0694 - accuracy: 0.9785
475/1688 [=======>......................] - ETA: 2s - loss: 0.0704 - accuracy: 0.9782
503/1688 [=======>......................] - ETA: 2s - loss: 0.0698 - accuracy: 0.9785
530/1688 [========>.....................] - ETA: 2s - loss: 0.0680 - accuracy: 0.9790
558/1688 [========>.....................] - ETA: 2s - loss: 0.0688 - accuracy: 0.9786
587/1688 [=========>....................] - ETA: 2s - loss: 0.0688 - accuracy: 0.9784
615/1688 [=========>....................] - ETA: 1s - loss: 0.0684 - accuracy: 0.9786
642/1688 [==========>...................] - ETA: 1s - loss: 0.0680 - accuracy: 0.9788
669/1688 [==========>...................] - ETA: 1s - loss: 0.0680 - accuracy: 0.9787
697/1688 [===========>..................] - ETA: 1s - loss: 0.0685 - accuracy: 0.9784
724/1688 [===========>..................] - ETA: 1s - loss: 0.0678 - accuracy: 0.9785
752/1688 [============>.................] - ETA: 1s - loss: 0.0678 - accuracy: 0.9788
779/1688 [============>.................] - ETA: 1s - loss: 0.0670 - accuracy: 0.9791
807/1688 [=============>................] - ETA: 1s - loss: 0.0663 - accuracy: 0.9792
835/1688 [=============>................] - ETA: 1s - loss: 0.0663 - accuracy: 0.9793
863/1688 [==============>...............] - ETA: 1s - loss: 0.0671 - accuracy: 0.9791
891/1688 [==============>...............] - ETA: 1s - loss: 0.0666 - accuracy: 0.9793
919/1688 [===============>..............] - ETA: 1s - loss: 0.0673 - accuracy: 0.9791
947/1688 [===============>..............] - ETA: 1s - loss: 0.0675 - accuracy: 0.9790
975/1688 [================>.............] - ETA: 1s - loss: 0.0673 - accuracy: 0.9790
1003/1688 [================>.............] - ETA: 1s - loss: 0.0672 - accuracy: 0.9791
1031/1688 [=================>............] - ETA: 1s - loss: 0.0676 - accuracy: 0.9790
1058/1688 [=================>............] - ETA: 1s - loss: 0.0681 - accuracy: 0.9789
1086/1688 [==================>...........] - ETA: 1s - loss: 0.0681 - accuracy: 0.9788
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0676 - accuracy: 0.9790
1142/1688 [===================>..........] - ETA: 0s - loss: 0.0674 - accuracy: 0.9790
1169/1688 [===================>..........] - ETA: 0s - loss: 0.0672 - accuracy: 0.9790
1196/1688 [====================>.........] - ETA: 0s - loss: 0.0670 - accuracy: 0.9791
1224/1688 [====================>.........] - ETA: 0s - loss: 0.0671 - accuracy: 0.9792
1252/1688 [=====================>........] - ETA: 0s - loss: 0.0671 - accuracy: 0.9792
1279/1688 [=====================>........] - ETA: 0s - loss: 0.0670 - accuracy: 0.9793
1307/1688 [======================>.......] - ETA: 0s - loss: 0.0671 - accuracy: 0.9791
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0676 - accuracy: 0.9790
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0677 - accuracy: 0.9789
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0680 - accuracy: 0.9788
1417/1688 [========================>.....] - ETA: 0s - loss: 0.0684 - accuracy: 0.9787
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0686 - accuracy: 0.9788
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0687 - accuracy: 0.9787
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0689 - accuracy: 0.9786
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0692 - accuracy: 0.9784
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0697 - accuracy: 0.9782
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0698 - accuracy: 0.9782
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0694 - accuracy: 0.9783
1639/1688 [============================>.] - ETA: 0s - loss: 0.0700 - accuracy: 0.9784
1667/1688 [============================>.] - ETA: 0s - loss: 0.0704 - accuracy: 0.9783
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0705 - accuracy: 0.9783 - val_loss: 0.0572 - val_accuracy: 0.9852
Epoch 3/10
1/1688 [..............................] - ETA: 3s - loss: 0.0077 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0296 - accuracy: 0.9927
57/1688 [>.............................] - ETA: 2s - loss: 0.0347 - accuracy: 0.9901
85/1688 [>.............................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9864
113/1688 [=>............................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9862
141/1688 [=>............................] - ETA: 2s - loss: 0.0429 - accuracy: 0.9874
169/1688 [==>...........................] - ETA: 2s - loss: 0.0434 - accuracy: 0.9874
196/1688 [==>...........................] - ETA: 2s - loss: 0.0448 - accuracy: 0.9864
223/1688 [==>...........................] - ETA: 2s - loss: 0.0484 - accuracy: 0.9850
251/1688 [===>..........................] - ETA: 2s - loss: 0.0481 - accuracy: 0.9853
279/1688 [===>..........................] - ETA: 2s - loss: 0.0493 - accuracy: 0.9850
306/1688 [====>.........................] - ETA: 2s - loss: 0.0503 - accuracy: 0.9846
334/1688 [====>.........................] - ETA: 2s - loss: 0.0500 - accuracy: 0.9847
362/1688 [=====>........................] - ETA: 2s - loss: 0.0489 - accuracy: 0.9853
390/1688 [=====>........................] - ETA: 2s - loss: 0.0478 - accuracy: 0.9857
418/1688 [======>.......................] - ETA: 2s - loss: 0.0488 - accuracy: 0.9854
445/1688 [======>.......................] - ETA: 2s - loss: 0.0500 - accuracy: 0.9850
473/1688 [=======>......................] - ETA: 2s - loss: 0.0488 - accuracy: 0.9852
500/1688 [=======>......................] - ETA: 2s - loss: 0.0484 - accuracy: 0.9854
528/1688 [========>.....................] - ETA: 2s - loss: 0.0480 - accuracy: 0.9854
556/1688 [========>.....................] - ETA: 2s - loss: 0.0488 - accuracy: 0.9852
584/1688 [=========>....................] - ETA: 2s - loss: 0.0490 - accuracy: 0.9851
612/1688 [=========>....................] - ETA: 1s - loss: 0.0502 - accuracy: 0.9846
640/1688 [==========>...................] - ETA: 1s - loss: 0.0500 - accuracy: 0.9848
668/1688 [==========>...................] - ETA: 1s - loss: 0.0503 - accuracy: 0.9844
696/1688 [===========>..................] - ETA: 1s - loss: 0.0508 - accuracy: 0.9842
724/1688 [===========>..................] - ETA: 1s - loss: 0.0505 - accuracy: 0.9844
752/1688 [============>.................] - ETA: 1s - loss: 0.0504 - accuracy: 0.9843
780/1688 [============>.................] - ETA: 1s - loss: 0.0514 - accuracy: 0.9839
808/1688 [=============>................] - ETA: 1s - loss: 0.0516 - accuracy: 0.9838
836/1688 [=============>................] - ETA: 1s - loss: 0.0513 - accuracy: 0.9838
864/1688 [==============>...............] - ETA: 1s - loss: 0.0517 - accuracy: 0.9838
891/1688 [==============>...............] - ETA: 1s - loss: 0.0513 - accuracy: 0.9838
919/1688 [===============>..............] - ETA: 1s - loss: 0.0517 - accuracy: 0.9838
947/1688 [===============>..............] - ETA: 1s - loss: 0.0520 - accuracy: 0.9835
974/1688 [================>.............] - ETA: 1s - loss: 0.0517 - accuracy: 0.9836
1002/1688 [================>.............] - ETA: 1s - loss: 0.0514 - accuracy: 0.9838
1030/1688 [=================>............] - ETA: 1s - loss: 0.0517 - accuracy: 0.9836
1058/1688 [=================>............] - ETA: 1s - loss: 0.0522 - accuracy: 0.9835
1086/1688 [==================>...........] - ETA: 1s - loss: 0.0522 - accuracy: 0.9835
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0518 - accuracy: 0.9836
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0516 - accuracy: 0.9837
1169/1688 [===================>..........] - ETA: 0s - loss: 0.0515 - accuracy: 0.9837
1197/1688 [====================>.........] - ETA: 0s - loss: 0.0515 - accuracy: 0.9837
1225/1688 [====================>.........] - ETA: 0s - loss: 0.0515 - accuracy: 0.9837
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0514 - accuracy: 0.9837
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0514 - accuracy: 0.9837
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0518 - accuracy: 0.9836
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0522 - accuracy: 0.9834
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0523 - accuracy: 0.9834
1391/1688 [=======================>......] - ETA: 0s - loss: 0.0522 - accuracy: 0.9834
1419/1688 [========================>.....] - ETA: 0s - loss: 0.0524 - accuracy: 0.9834
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0526 - accuracy: 0.9833
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0527 - accuracy: 0.9832
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0530 - accuracy: 0.9832
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0529 - accuracy: 0.9831
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0529 - accuracy: 0.9832
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0528 - accuracy: 0.9832
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0528 - accuracy: 0.9831
1642/1688 [============================>.] - ETA: 0s - loss: 0.0526 - accuracy: 0.9832
1670/1688 [============================>.] - ETA: 0s - loss: 0.0524 - accuracy: 0.9832
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0525 - accuracy: 0.9832 - val_loss: 0.0646 - val_accuracy: 0.9812
Epoch 4/10
1/1688 [..............................] - ETA: 3s - loss: 0.1505 - accuracy: 0.9375
30/1688 [..............................] - ETA: 2s - loss: 0.0330 - accuracy: 0.9844
58/1688 [>.............................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9887
86/1688 [>.............................] - ETA: 2s - loss: 0.0354 - accuracy: 0.9869
115/1688 [=>............................] - ETA: 2s - loss: 0.0340 - accuracy: 0.9875
143/1688 [=>............................] - ETA: 2s - loss: 0.0358 - accuracy: 0.9871
172/1688 [==>...........................] - ETA: 2s - loss: 0.0363 - accuracy: 0.9875
199/1688 [==>...........................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9871
227/1688 [===>..........................] - ETA: 2s - loss: 0.0389 - accuracy: 0.9866
255/1688 [===>..........................] - ETA: 2s - loss: 0.0385 - accuracy: 0.9866
283/1688 [====>.........................] - ETA: 2s - loss: 0.0387 - accuracy: 0.9869
311/1688 [====>.........................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9868
338/1688 [=====>........................] - ETA: 2s - loss: 0.0391 - accuracy: 0.9871
366/1688 [=====>........................] - ETA: 2s - loss: 0.0381 - accuracy: 0.9874
393/1688 [=====>........................] - ETA: 2s - loss: 0.0392 - accuracy: 0.9877
421/1688 [======>.......................] - ETA: 2s - loss: 0.0390 - accuracy: 0.9875
449/1688 [======>.......................] - ETA: 2s - loss: 0.0388 - accuracy: 0.9875
477/1688 [=======>......................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9880
505/1688 [=======>......................] - ETA: 2s - loss: 0.0385 - accuracy: 0.9878
533/1688 [========>.....................] - ETA: 2s - loss: 0.0392 - accuracy: 0.9876
561/1688 [========>.....................] - ETA: 2s - loss: 0.0384 - accuracy: 0.9879
589/1688 [=========>....................] - ETA: 1s - loss: 0.0383 - accuracy: 0.9877
617/1688 [=========>....................] - ETA: 1s - loss: 0.0388 - accuracy: 0.9875
644/1688 [==========>...................] - ETA: 1s - loss: 0.0382 - accuracy: 0.9878
672/1688 [==========>...................] - ETA: 1s - loss: 0.0374 - accuracy: 0.9880
700/1688 [===========>..................] - ETA: 1s - loss: 0.0377 - accuracy: 0.9879
727/1688 [===========>..................] - ETA: 1s - loss: 0.0376 - accuracy: 0.9880
755/1688 [============>.................] - ETA: 1s - loss: 0.0377 - accuracy: 0.9879
783/1688 [============>.................] - ETA: 1s - loss: 0.0384 - accuracy: 0.9877
810/1688 [=============>................] - ETA: 1s - loss: 0.0386 - accuracy: 0.9877
838/1688 [=============>................] - ETA: 1s - loss: 0.0383 - accuracy: 0.9878
866/1688 [==============>...............] - ETA: 1s - loss: 0.0387 - accuracy: 0.9877
894/1688 [==============>...............] - ETA: 1s - loss: 0.0397 - accuracy: 0.9876
922/1688 [===============>..............] - ETA: 1s - loss: 0.0394 - accuracy: 0.9877
950/1688 [===============>..............] - ETA: 1s - loss: 0.0391 - accuracy: 0.9877
978/1688 [================>.............] - ETA: 1s - loss: 0.0392 - accuracy: 0.9878
1005/1688 [================>.............] - ETA: 1s - loss: 0.0394 - accuracy: 0.9878
1033/1688 [=================>............] - ETA: 1s - loss: 0.0396 - accuracy: 0.9877
1061/1688 [=================>............] - ETA: 1s - loss: 0.0395 - accuracy: 0.9876
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0395 - accuracy: 0.9876
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0397 - accuracy: 0.9875
1144/1688 [===================>..........] - ETA: 0s - loss: 0.0395 - accuracy: 0.9876
1171/1688 [===================>..........] - ETA: 0s - loss: 0.0403 - accuracy: 0.9874
1199/1688 [====================>.........] - ETA: 0s - loss: 0.0403 - accuracy: 0.9873
1227/1688 [====================>.........] - ETA: 0s - loss: 0.0405 - accuracy: 0.9872
1255/1688 [=====================>........] - ETA: 0s - loss: 0.0407 - accuracy: 0.9871
1283/1688 [=====================>........] - ETA: 0s - loss: 0.0413 - accuracy: 0.9869
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0414 - accuracy: 0.9869
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0412 - accuracy: 0.9869
1368/1688 [=======================>......] - ETA: 0s - loss: 0.0409 - accuracy: 0.9871
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0408 - accuracy: 0.9871
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0408 - accuracy: 0.9872
1451/1688 [========================>.....] - ETA: 0s - loss: 0.0409 - accuracy: 0.9871
1479/1688 [=========================>....] - ETA: 0s - loss: 0.0408 - accuracy: 0.9872
1507/1688 [=========================>....] - ETA: 0s - loss: 0.0408 - accuracy: 0.9871
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0407 - accuracy: 0.9872
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0416 - accuracy: 0.9870
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0415 - accuracy: 0.9870
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0415 - accuracy: 0.9870
1645/1688 [============================>.] - ETA: 0s - loss: 0.0414 - accuracy: 0.9870
1672/1688 [============================>.] - ETA: 0s - loss: 0.0421 - accuracy: 0.9868
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0420 - accuracy: 0.9869 - val_loss: 0.0615 - val_accuracy: 0.9843
Epoch 5/10
1/1688 [..............................] - ETA: 3s - loss: 0.0024 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9927
58/1688 [>.............................] - ETA: 2s - loss: 0.0241 - accuracy: 0.9914
86/1688 [>.............................] - ETA: 2s - loss: 0.0295 - accuracy: 0.9902
114/1688 [=>............................] - ETA: 2s - loss: 0.0354 - accuracy: 0.9882
142/1688 [=>............................] - ETA: 2s - loss: 0.0336 - accuracy: 0.9886
170/1688 [==>...........................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9895
198/1688 [==>...........................] - ETA: 2s - loss: 0.0322 - accuracy: 0.9893
226/1688 [===>..........................] - ETA: 2s - loss: 0.0331 - accuracy: 0.9888
253/1688 [===>..........................] - ETA: 2s - loss: 0.0328 - accuracy: 0.9889
280/1688 [===>..........................] - ETA: 2s - loss: 0.0324 - accuracy: 0.9892
308/1688 [====>.........................] - ETA: 2s - loss: 0.0318 - accuracy: 0.9897
335/1688 [====>.........................] - ETA: 2s - loss: 0.0322 - accuracy: 0.9892
363/1688 [=====>........................] - ETA: 2s - loss: 0.0315 - accuracy: 0.9894
391/1688 [=====>........................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9895
419/1688 [======>.......................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9898
447/1688 [======>.......................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9899
475/1688 [=======>......................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9896
504/1688 [=======>......................] - ETA: 2s - loss: 0.0318 - accuracy: 0.9898
532/1688 [========>.....................] - ETA: 2s - loss: 0.0331 - accuracy: 0.9893
560/1688 [========>.....................] - ETA: 2s - loss: 0.0329 - accuracy: 0.9891
588/1688 [=========>....................] - ETA: 2s - loss: 0.0328 - accuracy: 0.9891
615/1688 [=========>....................] - ETA: 1s - loss: 0.0329 - accuracy: 0.9889
643/1688 [==========>...................] - ETA: 1s - loss: 0.0329 - accuracy: 0.9888
671/1688 [==========>...................] - ETA: 1s - loss: 0.0329 - accuracy: 0.9887
699/1688 [===========>..................] - ETA: 1s - loss: 0.0330 - accuracy: 0.9888
726/1688 [===========>..................] - ETA: 1s - loss: 0.0338 - accuracy: 0.9887
753/1688 [============>.................] - ETA: 1s - loss: 0.0332 - accuracy: 0.9889
781/1688 [============>.................] - ETA: 1s - loss: 0.0334 - accuracy: 0.9888
809/1688 [=============>................] - ETA: 1s - loss: 0.0335 - accuracy: 0.9887
838/1688 [=============>................] - ETA: 1s - loss: 0.0328 - accuracy: 0.9890
866/1688 [==============>...............] - ETA: 1s - loss: 0.0324 - accuracy: 0.9891
894/1688 [==============>...............] - ETA: 1s - loss: 0.0330 - accuracy: 0.9890
921/1688 [===============>..............] - ETA: 1s - loss: 0.0325 - accuracy: 0.9891
950/1688 [===============>..............] - ETA: 1s - loss: 0.0323 - accuracy: 0.9892
978/1688 [================>.............] - ETA: 1s - loss: 0.0329 - accuracy: 0.9890
1006/1688 [================>.............] - ETA: 1s - loss: 0.0327 - accuracy: 0.9890
1034/1688 [=================>............] - ETA: 1s - loss: 0.0331 - accuracy: 0.9887
1062/1688 [=================>............] - ETA: 1s - loss: 0.0328 - accuracy: 0.9888
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0329 - accuracy: 0.9888
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0326 - accuracy: 0.9889
1144/1688 [===================>..........] - ETA: 0s - loss: 0.0328 - accuracy: 0.9888
1173/1688 [===================>..........] - ETA: 0s - loss: 0.0326 - accuracy: 0.9888
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0324 - accuracy: 0.9889
1228/1688 [====================>.........] - ETA: 0s - loss: 0.0326 - accuracy: 0.9888
1255/1688 [=====================>........] - ETA: 0s - loss: 0.0329 - accuracy: 0.9888
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0333 - accuracy: 0.9887
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0334 - accuracy: 0.9887
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0334 - accuracy: 0.9887
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0335 - accuracy: 0.9887
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0337 - accuracy: 0.9886
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0336 - accuracy: 0.9887
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0339 - accuracy: 0.9887
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0341 - accuracy: 0.9885
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0341 - accuracy: 0.9886
1532/1688 [==========================>...] - ETA: 0s - loss: 0.0341 - accuracy: 0.9886
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0345 - accuracy: 0.9886
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0345 - accuracy: 0.9886
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0345 - accuracy: 0.9886
1642/1688 [============================>.] - ETA: 0s - loss: 0.0350 - accuracy: 0.9884
1670/1688 [============================>.] - ETA: 0s - loss: 0.0349 - accuracy: 0.9885
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0349 - accuracy: 0.9884 - val_loss: 0.0606 - val_accuracy: 0.9848
Epoch 6/10
1/1688 [..............................] - ETA: 3s - loss: 0.0169 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0273 - accuracy: 0.9906
57/1688 [>.............................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9901
85/1688 [>.............................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9901
113/1688 [=>............................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9900
141/1688 [=>............................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9916
169/1688 [==>...........................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9909
197/1688 [==>...........................] - ETA: 2s - loss: 0.0270 - accuracy: 0.9918
224/1688 [==>...........................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9914
251/1688 [===>..........................] - ETA: 2s - loss: 0.0270 - accuracy: 0.9913
278/1688 [===>..........................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9916
306/1688 [====>.........................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9917
334/1688 [====>.........................] - ETA: 2s - loss: 0.0252 - accuracy: 0.9918
362/1688 [=====>........................] - ETA: 2s - loss: 0.0256 - accuracy: 0.9915
389/1688 [=====>........................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9916
417/1688 [======>.......................] - ETA: 2s - loss: 0.0272 - accuracy: 0.9914
444/1688 [======>.......................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9912
472/1688 [=======>......................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9915
500/1688 [=======>......................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9914
528/1688 [========>.....................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9914
556/1688 [========>.....................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9914
585/1688 [=========>....................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9914
613/1688 [=========>....................] - ETA: 1s - loss: 0.0268 - accuracy: 0.9915
641/1688 [==========>...................] - ETA: 1s - loss: 0.0266 - accuracy: 0.9916
669/1688 [==========>...................] - ETA: 1s - loss: 0.0264 - accuracy: 0.9916
697/1688 [===========>..................] - ETA: 1s - loss: 0.0273 - accuracy: 0.9913
725/1688 [===========>..................] - ETA: 1s - loss: 0.0273 - accuracy: 0.9913
754/1688 [============>.................] - ETA: 1s - loss: 0.0273 - accuracy: 0.9913
782/1688 [============>.................] - ETA: 1s - loss: 0.0282 - accuracy: 0.9912
809/1688 [=============>................] - ETA: 1s - loss: 0.0285 - accuracy: 0.9911
836/1688 [=============>................] - ETA: 1s - loss: 0.0286 - accuracy: 0.9911
864/1688 [==============>...............] - ETA: 1s - loss: 0.0288 - accuracy: 0.9910
892/1688 [==============>...............] - ETA: 1s - loss: 0.0286 - accuracy: 0.9910
919/1688 [===============>..............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9910
948/1688 [===============>..............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9911
976/1688 [================>.............] - ETA: 1s - loss: 0.0286 - accuracy: 0.9910
1003/1688 [================>.............] - ETA: 1s - loss: 0.0285 - accuracy: 0.9909
1031/1688 [=================>............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9911
1059/1688 [=================>............] - ETA: 1s - loss: 0.0283 - accuracy: 0.9910
1087/1688 [==================>...........] - ETA: 1s - loss: 0.0284 - accuracy: 0.9910
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0285 - accuracy: 0.9909
1142/1688 [===================>..........] - ETA: 0s - loss: 0.0283 - accuracy: 0.9909
1170/1688 [===================>..........] - ETA: 0s - loss: 0.0286 - accuracy: 0.9908
1197/1688 [====================>.........] - ETA: 0s - loss: 0.0287 - accuracy: 0.9907
1225/1688 [====================>.........] - ETA: 0s - loss: 0.0288 - accuracy: 0.9907
1252/1688 [=====================>........] - ETA: 0s - loss: 0.0287 - accuracy: 0.9907
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0287 - accuracy: 0.9907
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0288 - accuracy: 0.9906
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0287 - accuracy: 0.9906
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0285 - accuracy: 0.9906
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0287 - accuracy: 0.9906
1417/1688 [========================>.....] - ETA: 0s - loss: 0.0286 - accuracy: 0.9906
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0287 - accuracy: 0.9906
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0288 - accuracy: 0.9907
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0287 - accuracy: 0.9907
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0287 - accuracy: 0.9906
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1642/1688 [============================>.] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1670/1688 [============================>.] - ETA: 0s - loss: 0.0288 - accuracy: 0.9905
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0289 - accuracy: 0.9904 - val_loss: 0.0776 - val_accuracy: 0.9822
Epoch 7/10
1/1688 [..............................] - ETA: 3s - loss: 0.0104 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9875
58/1688 [>.............................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9903
86/1688 [>.............................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9909
114/1688 [=>............................] - ETA: 2s - loss: 0.0308 - accuracy: 0.9901
142/1688 [=>............................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9910
170/1688 [==>...........................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9917
198/1688 [==>...........................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9927
226/1688 [===>..........................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9932
254/1688 [===>..........................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9937
282/1688 [====>.........................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9936
310/1688 [====>.........................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9937
338/1688 [=====>........................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9934
365/1688 [=====>........................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9935
393/1688 [=====>........................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9933
421/1688 [======>.......................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9929
449/1688 [======>.......................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9930
477/1688 [=======>......................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9929
505/1688 [=======>......................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9929
532/1688 [========>.....................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9929
560/1688 [========>.....................] - ETA: 2s - loss: 0.0225 - accuracy: 0.9930
588/1688 [=========>....................] - ETA: 2s - loss: 0.0221 - accuracy: 0.9931
616/1688 [=========>....................] - ETA: 1s - loss: 0.0220 - accuracy: 0.9932
643/1688 [==========>...................] - ETA: 1s - loss: 0.0217 - accuracy: 0.9931
671/1688 [==========>...................] - ETA: 1s - loss: 0.0218 - accuracy: 0.9930
699/1688 [===========>..................] - ETA: 1s - loss: 0.0218 - accuracy: 0.9929
726/1688 [===========>..................] - ETA: 1s - loss: 0.0216 - accuracy: 0.9929
753/1688 [============>.................] - ETA: 1s - loss: 0.0217 - accuracy: 0.9930
781/1688 [============>.................] - ETA: 1s - loss: 0.0216 - accuracy: 0.9931
809/1688 [=============>................] - ETA: 1s - loss: 0.0216 - accuracy: 0.9931
836/1688 [=============>................] - ETA: 1s - loss: 0.0214 - accuracy: 0.9931
864/1688 [==============>...............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9931
892/1688 [==============>...............] - ETA: 1s - loss: 0.0221 - accuracy: 0.9928
918/1688 [===============>..............] - ETA: 1s - loss: 0.0229 - accuracy: 0.9925
946/1688 [===============>..............] - ETA: 1s - loss: 0.0227 - accuracy: 0.9925
973/1688 [================>.............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9923
1001/1688 [================>.............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9923
1028/1688 [=================>............] - ETA: 1s - loss: 0.0229 - accuracy: 0.9924
1056/1688 [=================>............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9923
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0239 - accuracy: 0.9920
1111/1688 [==================>...........] - ETA: 1s - loss: 0.0239 - accuracy: 0.9918
1138/1688 [===================>..........] - ETA: 1s - loss: 0.0239 - accuracy: 0.9918
1166/1688 [===================>..........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9920
1194/1688 [====================>.........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9920
1221/1688 [====================>.........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9920
1248/1688 [=====================>........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9921
1276/1688 [=====================>........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9922
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0235 - accuracy: 0.9922
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0241 - accuracy: 0.9920
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0241 - accuracy: 0.9920
1416/1688 [========================>.....] - ETA: 0s - loss: 0.0240 - accuracy: 0.9920
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0248 - accuracy: 0.9918
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0248 - accuracy: 0.9918
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0252 - accuracy: 0.9916
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0253 - accuracy: 0.9917
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0252 - accuracy: 0.9917
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0254 - accuracy: 0.9917
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0255 - accuracy: 0.9917
1638/1688 [============================>.] - ETA: 0s - loss: 0.0255 - accuracy: 0.9917
1666/1688 [============================>.] - ETA: 0s - loss: 0.0255 - accuracy: 0.9916
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0255 - accuracy: 0.9916 - val_loss: 0.0587 - val_accuracy: 0.9858
Epoch 8/10
1/1688 [..............................] - ETA: 3s - loss: 5.4804e-04 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9940
59/1688 [>.............................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9936
87/1688 [>.............................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9935
115/1688 [=>............................] - ETA: 2s - loss: 0.0223 - accuracy: 0.9943
143/1688 [=>............................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9950
171/1688 [==>...........................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9947
200/1688 [==>...........................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9950
228/1688 [===>..........................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9948
255/1688 [===>..........................] - ETA: 2s - loss: 0.0177 - accuracy: 0.9950
283/1688 [====>.........................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9946
310/1688 [====>.........................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9947
338/1688 [=====>........................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9950
366/1688 [=====>........................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9950
394/1688 [======>.......................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9947
421/1688 [======>.......................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9949
449/1688 [======>.......................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9948
478/1688 [=======>......................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9948
507/1688 [========>.....................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9946
535/1688 [========>.....................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9945
562/1688 [========>.....................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9943
590/1688 [=========>....................] - ETA: 1s - loss: 0.0174 - accuracy: 0.9941
617/1688 [=========>....................] - ETA: 1s - loss: 0.0173 - accuracy: 0.9939
644/1688 [==========>...................] - ETA: 1s - loss: 0.0173 - accuracy: 0.9939
672/1688 [==========>...................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9937
700/1688 [===========>..................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9935
729/1688 [===========>..................] - ETA: 1s - loss: 0.0180 - accuracy: 0.9934
757/1688 [============>.................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9936
785/1688 [============>.................] - ETA: 1s - loss: 0.0181 - accuracy: 0.9936
812/1688 [=============>................] - ETA: 1s - loss: 0.0183 - accuracy: 0.9935
841/1688 [=============>................] - ETA: 1s - loss: 0.0190 - accuracy: 0.9933
869/1688 [==============>...............] - ETA: 1s - loss: 0.0188 - accuracy: 0.9933
897/1688 [==============>...............] - ETA: 1s - loss: 0.0191 - accuracy: 0.9932
924/1688 [===============>..............] - ETA: 1s - loss: 0.0192 - accuracy: 0.9932
952/1688 [===============>..............] - ETA: 1s - loss: 0.0192 - accuracy: 0.9933
980/1688 [================>.............] - ETA: 1s - loss: 0.0191 - accuracy: 0.9933
1009/1688 [================>.............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9932
1037/1688 [=================>............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9932
1065/1688 [=================>............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9932
1093/1688 [==================>...........] - ETA: 1s - loss: 0.0197 - accuracy: 0.9931
1121/1688 [==================>...........] - ETA: 1s - loss: 0.0196 - accuracy: 0.9932
1148/1688 [===================>..........] - ETA: 0s - loss: 0.0195 - accuracy: 0.9932
1176/1688 [===================>..........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9931
1204/1688 [====================>.........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9930
1231/1688 [====================>.........] - ETA: 0s - loss: 0.0197 - accuracy: 0.9931
1260/1688 [=====================>........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9930
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0197 - accuracy: 0.9930
1315/1688 [======================>.......] - ETA: 0s - loss: 0.0199 - accuracy: 0.9930
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0199 - accuracy: 0.9930
1371/1688 [=======================>......] - ETA: 0s - loss: 0.0199 - accuracy: 0.9930
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0199 - accuracy: 0.9930
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0198 - accuracy: 0.9930
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0202 - accuracy: 0.9929
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0201 - accuracy: 0.9930
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0202 - accuracy: 0.9929
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0204 - accuracy: 0.9929
1567/1688 [==========================>...] - ETA: 0s - loss: 0.0205 - accuracy: 0.9928
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0206 - accuracy: 0.9928
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0207 - accuracy: 0.9927
1651/1688 [============================>.] - ETA: 0s - loss: 0.0207 - accuracy: 0.9928
1678/1688 [============================>.] - ETA: 0s - loss: 0.0209 - accuracy: 0.9927
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0210 - accuracy: 0.9926 - val_loss: 0.0543 - val_accuracy: 0.9867
Epoch 9/10
1/1688 [..............................] - ETA: 3s - loss: 0.0081 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9906
58/1688 [>.............................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9935
86/1688 [>.............................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9938
114/1688 [=>............................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9940
142/1688 [=>............................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9941
169/1688 [==>...........................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9941
197/1688 [==>...........................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9946
226/1688 [===>..........................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9946
254/1688 [===>..........................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9950
282/1688 [====>.........................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9953
311/1688 [====>.........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9954
339/1688 [=====>........................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9955
366/1688 [=====>........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9956
394/1688 [======>.......................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9958
422/1688 [======>.......................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9958
450/1688 [======>.......................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9956
478/1688 [=======>......................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9955
505/1688 [=======>......................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9954
533/1688 [========>.....................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
560/1688 [========>.....................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9953
589/1688 [=========>....................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9949
617/1688 [=========>....................] - ETA: 1s - loss: 0.0144 - accuracy: 0.9949
644/1688 [==========>...................] - ETA: 1s - loss: 0.0149 - accuracy: 0.9948
672/1688 [==========>...................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9949
700/1688 [===========>..................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9950
728/1688 [===========>..................] - ETA: 1s - loss: 0.0146 - accuracy: 0.9949
755/1688 [============>.................] - ETA: 1s - loss: 0.0150 - accuracy: 0.9947
783/1688 [============>.................] - ETA: 1s - loss: 0.0148 - accuracy: 0.9947
811/1688 [=============>................] - ETA: 1s - loss: 0.0152 - accuracy: 0.9945
839/1688 [=============>................] - ETA: 1s - loss: 0.0154 - accuracy: 0.9944
867/1688 [==============>...............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9945
894/1688 [==============>...............] - ETA: 1s - loss: 0.0157 - accuracy: 0.9943
922/1688 [===============>..............] - ETA: 1s - loss: 0.0157 - accuracy: 0.9943
950/1688 [===============>..............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9942
978/1688 [================>.............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9940
1006/1688 [================>.............] - ETA: 1s - loss: 0.0168 - accuracy: 0.9940
1033/1688 [=================>............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9939
1061/1688 [=================>............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9938
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0173 - accuracy: 0.9938
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0177 - accuracy: 0.9937
1146/1688 [===================>..........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9938
1174/1688 [===================>..........] - ETA: 0s - loss: 0.0175 - accuracy: 0.9938
1202/1688 [====================>.........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9937
1230/1688 [====================>.........] - ETA: 0s - loss: 0.0175 - accuracy: 0.9938
1259/1688 [=====================>........] - ETA: 0s - loss: 0.0175 - accuracy: 0.9938
1287/1688 [=====================>........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9938
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0177 - accuracy: 0.9938
1342/1688 [======================>.......] - ETA: 0s - loss: 0.0178 - accuracy: 0.9938
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0184 - accuracy: 0.9937
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0183 - accuracy: 0.9938
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0185 - accuracy: 0.9938
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0186 - accuracy: 0.9937
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0186 - accuracy: 0.9937
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0185 - accuracy: 0.9937
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0184 - accuracy: 0.9937
1647/1688 [============================>.] - ETA: 0s - loss: 0.0184 - accuracy: 0.9938
1675/1688 [============================>.] - ETA: 0s - loss: 0.0183 - accuracy: 0.9938
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0183 - accuracy: 0.9938 - val_loss: 0.0653 - val_accuracy: 0.9852
Epoch 10/10
1/1688 [..............................] - ETA: 3s - loss: 0.0072 - accuracy: 1.0000
29/1688 [..............................] - ETA: 3s - loss: 0.0063 - accuracy: 0.9989
56/1688 [..............................] - ETA: 3s - loss: 0.0104 - accuracy: 0.9967
84/1688 [>.............................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9963
111/1688 [>.............................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9966
139/1688 [=>............................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9973
167/1688 [=>............................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9970
194/1688 [==>...........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9963
222/1688 [==>...........................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9965
250/1688 [===>..........................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9964
277/1688 [===>..........................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9958
305/1688 [====>.........................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9955
333/1688 [====>.........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9953
361/1688 [=====>........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9953
389/1688 [=====>........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9953
417/1688 [======>.......................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9954
445/1688 [======>.......................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9956
473/1688 [=======>......................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9952
501/1688 [=======>......................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9950
528/1688 [========>.....................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9949
556/1688 [========>.....................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9951
583/1688 [=========>....................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9952
611/1688 [=========>....................] - ETA: 1s - loss: 0.0147 - accuracy: 0.9951
638/1688 [==========>...................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9952
665/1688 [==========>...................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9952
692/1688 [===========>..................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9953
720/1688 [===========>..................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9954
748/1688 [============>.................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9954
776/1688 [============>.................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9954
803/1688 [=============>................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9953
831/1688 [=============>................] - ETA: 1s - loss: 0.0146 - accuracy: 0.9952
859/1688 [==============>...............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9951
886/1688 [==============>...............] - ETA: 1s - loss: 0.0150 - accuracy: 0.9950
914/1688 [===============>..............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9950
942/1688 [===============>..............] - ETA: 1s - loss: 0.0149 - accuracy: 0.9949
970/1688 [================>.............] - ETA: 1s - loss: 0.0149 - accuracy: 0.9949
997/1688 [================>.............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9950
1025/1688 [=================>............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9950
1053/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9950
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0146 - accuracy: 0.9950
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0147 - accuracy: 0.9950
1137/1688 [===================>..........] - ETA: 1s - loss: 0.0149 - accuracy: 0.9951
1164/1688 [===================>..........] - ETA: 0s - loss: 0.0148 - accuracy: 0.9951
1191/1688 [====================>.........] - ETA: 0s - loss: 0.0152 - accuracy: 0.9951
1219/1688 [====================>.........] - ETA: 0s - loss: 0.0153 - accuracy: 0.9950
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0155 - accuracy: 0.9949
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0157 - accuracy: 0.9948
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1328/1688 [======================>.......] - ETA: 0s - loss: 0.0161 - accuracy: 0.9947
1356/1688 [=======================>......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1412/1688 [========================>.....] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0159 - accuracy: 0.9947
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0159 - accuracy: 0.9948
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9948
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0158 - accuracy: 0.9948
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0161 - accuracy: 0.9947
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0162 - accuracy: 0.9947
1635/1688 [============================>.] - ETA: 0s - loss: 0.0161 - accuracy: 0.9948
1663/1688 [============================>.] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0160 - accuracy: 0.9948 - val_loss: 0.0575 - val_accuracy: 0.9858
Test accuracy: 0.9869999885559082
4. Model quantization
We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.
For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).
The quantized model is a Keras model where the neural layers (Conv2D, Dense)
and the ReLU layers are replaced with custom CNN2SNN quantized layers
(QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions
can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function folds the batch normalization layers into
the corresponding neural layer. The new weights are computed
according to this folding operation.
from cnn2snn import quantize
model_quantized = quantize(model_keras,
input_weight_quantization=8,
weight_quantization=4,
activ_quantization=4)
model_quantized.summary()
Model: "sequential_50"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (QuantizedConv2D) (None, 13, 13, 32) 320
re_lu (QuantizedReLU) (None, 13, 13, 32) 0
separable_conv2d (Quantized (None, 7, 7, 64) 2400
SeparableConv2D)
re_lu_1 (QuantizedReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (QuantizedDense) (None, 10) 31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________
Check the quantized model accuracy.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])
Test accuracy after 8-4-4 quantization: 0.9611999988555908
Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.
Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.
Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.
from cnn2snn import quantize_layer
model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])
# To recover the original model accuracy, a quantization-aware training phase
# is required.
Test accuracy after low bitwidth quantization: 0.10300000011920929
5. Model fine tuning (quantization-aware training)
This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.
Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])
Epoch 1/5
1/1688 [..............................] - ETA: 19:37 - loss: 2.2347 - accuracy: 0.0938
20/1688 [..............................] - ETA: 4s - loss: 2.2945 - accuracy: 0.1109
39/1688 [..............................] - ETA: 4s - loss: 2.3163 - accuracy: 0.1090
58/1688 [>.............................] - ETA: 4s - loss: 2.4929 - accuracy: 0.1207
77/1688 [>.............................] - ETA: 4s - loss: 2.5421 - accuracy: 0.1266
96/1688 [>.............................] - ETA: 4s - loss: 2.5645 - accuracy: 0.1520
115/1688 [=>............................] - ETA: 4s - loss: 2.5267 - accuracy: 0.1853
134/1688 [=>............................] - ETA: 4s - loss: 2.4826 - accuracy: 0.2229
153/1688 [=>............................] - ETA: 4s - loss: 2.4248 - accuracy: 0.2717
172/1688 [==>...........................] - ETA: 4s - loss: 2.3861 - accuracy: 0.3216
192/1688 [==>...........................] - ETA: 4s - loss: 2.3230 - accuracy: 0.3696
212/1688 [==>...........................] - ETA: 3s - loss: 2.2486 - accuracy: 0.4099
231/1688 [===>..........................] - ETA: 3s - loss: 2.1576 - accuracy: 0.4455
250/1688 [===>..........................] - ETA: 3s - loss: 2.0862 - accuracy: 0.4782
270/1688 [===>..........................] - ETA: 3s - loss: 2.0286 - accuracy: 0.5059
289/1688 [====>.........................] - ETA: 3s - loss: 1.9879 - accuracy: 0.5295
308/1688 [====>.........................] - ETA: 3s - loss: 1.9025 - accuracy: 0.5527
327/1688 [====>.........................] - ETA: 3s - loss: 1.8582 - accuracy: 0.5723
347/1688 [=====>........................] - ETA: 3s - loss: 1.7904 - accuracy: 0.5916
366/1688 [=====>........................] - ETA: 3s - loss: 1.7319 - accuracy: 0.6090
385/1688 [=====>........................] - ETA: 3s - loss: 1.6902 - accuracy: 0.6231
404/1688 [======>.......................] - ETA: 3s - loss: 1.6608 - accuracy: 0.6372
423/1688 [======>.......................] - ETA: 3s - loss: 1.6265 - accuracy: 0.6496
442/1688 [======>.......................] - ETA: 3s - loss: 1.5865 - accuracy: 0.6616
461/1688 [=======>......................] - ETA: 3s - loss: 1.5597 - accuracy: 0.6723
480/1688 [=======>......................] - ETA: 3s - loss: 1.5399 - accuracy: 0.6818
499/1688 [=======>......................] - ETA: 3s - loss: 1.5167 - accuracy: 0.6908
519/1688 [========>.....................] - ETA: 3s - loss: 1.5037 - accuracy: 0.6986
538/1688 [========>.....................] - ETA: 3s - loss: 1.4793 - accuracy: 0.7072
557/1688 [========>.....................] - ETA: 3s - loss: 1.4608 - accuracy: 0.7143
576/1688 [=========>....................] - ETA: 2s - loss: 1.4501 - accuracy: 0.7211
595/1688 [=========>....................] - ETA: 2s - loss: 1.4209 - accuracy: 0.7282
614/1688 [=========>....................] - ETA: 2s - loss: 1.4042 - accuracy: 0.7341
634/1688 [==========>...................] - ETA: 2s - loss: 1.3734 - accuracy: 0.7413
653/1688 [==========>...................] - ETA: 2s - loss: 1.3640 - accuracy: 0.7468
672/1688 [==========>...................] - ETA: 2s - loss: 1.3531 - accuracy: 0.7521
691/1688 [===========>..................] - ETA: 2s - loss: 1.3387 - accuracy: 0.7573
710/1688 [===========>..................] - ETA: 2s - loss: 1.3210 - accuracy: 0.7623
729/1688 [===========>..................] - ETA: 2s - loss: 1.3065 - accuracy: 0.7673
748/1688 [============>.................] - ETA: 2s - loss: 1.2914 - accuracy: 0.7718
767/1688 [============>.................] - ETA: 2s - loss: 1.2665 - accuracy: 0.7766
786/1688 [============>.................] - ETA: 2s - loss: 1.2464 - accuracy: 0.7806
805/1688 [=============>................] - ETA: 2s - loss: 1.2333 - accuracy: 0.7845
824/1688 [=============>................] - ETA: 2s - loss: 1.2239 - accuracy: 0.7877
843/1688 [=============>................] - ETA: 2s - loss: 1.2113 - accuracy: 0.7917
863/1688 [==============>...............] - ETA: 2s - loss: 1.2073 - accuracy: 0.7950
882/1688 [==============>...............] - ETA: 2s - loss: 1.1953 - accuracy: 0.7984
901/1688 [===============>..............] - ETA: 2s - loss: 1.1854 - accuracy: 0.8013
921/1688 [===============>..............] - ETA: 2s - loss: 1.1744 - accuracy: 0.8045
940/1688 [===============>..............] - ETA: 1s - loss: 1.1609 - accuracy: 0.8076
959/1688 [================>.............] - ETA: 1s - loss: 1.1543 - accuracy: 0.8101
978/1688 [================>.............] - ETA: 1s - loss: 1.1436 - accuracy: 0.8128
998/1688 [================>.............] - ETA: 1s - loss: 1.1411 - accuracy: 0.8152
1017/1688 [=================>............] - ETA: 1s - loss: 1.1304 - accuracy: 0.8178
1036/1688 [=================>............] - ETA: 1s - loss: 1.1231 - accuracy: 0.8200
1055/1688 [=================>............] - ETA: 1s - loss: 1.1139 - accuracy: 0.8224
1074/1688 [==================>...........] - ETA: 1s - loss: 1.1043 - accuracy: 0.8248
1094/1688 [==================>...........] - ETA: 1s - loss: 1.0941 - accuracy: 0.8271
1113/1688 [==================>...........] - ETA: 1s - loss: 1.0957 - accuracy: 0.8289
1132/1688 [===================>..........] - ETA: 1s - loss: 1.0895 - accuracy: 0.8309
1152/1688 [===================>..........] - ETA: 1s - loss: 1.0827 - accuracy: 0.8330
1171/1688 [===================>..........] - ETA: 1s - loss: 1.0725 - accuracy: 0.8349
1191/1688 [====================>.........] - ETA: 1s - loss: 1.0693 - accuracy: 0.8370
1210/1688 [====================>.........] - ETA: 1s - loss: 1.0663 - accuracy: 0.8387
1229/1688 [====================>.........] - ETA: 1s - loss: 1.0595 - accuracy: 0.8406
1248/1688 [=====================>........] - ETA: 1s - loss: 1.0537 - accuracy: 0.8423
1267/1688 [=====================>........] - ETA: 1s - loss: 1.0456 - accuracy: 0.8441
1286/1688 [=====================>........] - ETA: 1s - loss: 1.0388 - accuracy: 0.8459
1306/1688 [======================>.......] - ETA: 1s - loss: 1.0349 - accuracy: 0.8473
1325/1688 [======================>.......] - ETA: 0s - loss: 1.0290 - accuracy: 0.8489
1345/1688 [======================>.......] - ETA: 0s - loss: 1.0210 - accuracy: 0.8506
1364/1688 [=======================>......] - ETA: 0s - loss: 1.0127 - accuracy: 0.8523
1383/1688 [=======================>......] - ETA: 0s - loss: 1.0119 - accuracy: 0.8537
1403/1688 [=======================>......] - ETA: 0s - loss: 1.0065 - accuracy: 0.8552
1423/1688 [========================>.....] - ETA: 0s - loss: 1.0009 - accuracy: 0.8567
1442/1688 [========================>.....] - ETA: 0s - loss: 1.0007 - accuracy: 0.8578
1461/1688 [========================>.....] - ETA: 0s - loss: 0.9940 - accuracy: 0.8589
1481/1688 [=========================>....] - ETA: 0s - loss: 0.9911 - accuracy: 0.8603
1501/1688 [=========================>....] - ETA: 0s - loss: 0.9871 - accuracy: 0.8615
1520/1688 [==========================>...] - ETA: 0s - loss: 0.9819 - accuracy: 0.8626
1539/1688 [==========================>...] - ETA: 0s - loss: 0.9781 - accuracy: 0.8638
1558/1688 [==========================>...] - ETA: 0s - loss: 0.9752 - accuracy: 0.8650
1578/1688 [===========================>..] - ETA: 0s - loss: 0.9727 - accuracy: 0.8661
1597/1688 [===========================>..] - ETA: 0s - loss: 0.9680 - accuracy: 0.8673
1616/1688 [===========================>..] - ETA: 0s - loss: 0.9656 - accuracy: 0.8684
1635/1688 [============================>.] - ETA: 0s - loss: 0.9638 - accuracy: 0.8692
1654/1688 [============================>.] - ETA: 0s - loss: 0.9607 - accuracy: 0.8703
1673/1688 [============================>.] - ETA: 0s - loss: 0.9578 - accuracy: 0.8712
1688/1688 [==============================] - 5s 3ms/step - loss: 0.9555 - accuracy: 0.8720 - val_loss: 0.6048 - val_accuracy: 0.9688
Epoch 2/5
1/1688 [..............................] - ETA: 4s - loss: 1.5083 - accuracy: 0.9375
20/1688 [..............................] - ETA: 4s - loss: 0.6111 - accuracy: 0.9594
40/1688 [..............................] - ETA: 4s - loss: 0.5613 - accuracy: 0.9641
60/1688 [>.............................] - ETA: 4s - loss: 0.6398 - accuracy: 0.9625
79/1688 [>.............................] - ETA: 4s - loss: 0.6193 - accuracy: 0.9628
98/1688 [>.............................] - ETA: 4s - loss: 0.5962 - accuracy: 0.9636
118/1688 [=>............................] - ETA: 4s - loss: 0.6058 - accuracy: 0.9637
138/1688 [=>............................] - ETA: 4s - loss: 0.5662 - accuracy: 0.9660
157/1688 [=>............................] - ETA: 4s - loss: 0.6118 - accuracy: 0.9648
176/1688 [==>...........................] - ETA: 4s - loss: 0.6004 - accuracy: 0.9645
195/1688 [==>...........................] - ETA: 3s - loss: 0.5792 - accuracy: 0.9655
215/1688 [==>...........................] - ETA: 3s - loss: 0.5774 - accuracy: 0.9648
235/1688 [===>..........................] - ETA: 3s - loss: 0.5580 - accuracy: 0.9656
255/1688 [===>..........................] - ETA: 3s - loss: 0.5632 - accuracy: 0.9648
274/1688 [===>..........................] - ETA: 3s - loss: 0.5570 - accuracy: 0.9646
293/1688 [====>.........................] - ETA: 3s - loss: 0.5789 - accuracy: 0.9641
312/1688 [====>.........................] - ETA: 3s - loss: 0.5827 - accuracy: 0.9648
332/1688 [====>.........................] - ETA: 3s - loss: 0.5772 - accuracy: 0.9646
351/1688 [=====>........................] - ETA: 3s - loss: 0.5750 - accuracy: 0.9651
370/1688 [=====>........................] - ETA: 3s - loss: 0.5881 - accuracy: 0.9642
389/1688 [=====>........................] - ETA: 3s - loss: 0.5911 - accuracy: 0.9641
408/1688 [======>.......................] - ETA: 3s - loss: 0.5807 - accuracy: 0.9642
427/1688 [======>.......................] - ETA: 3s - loss: 0.5936 - accuracy: 0.9633
446/1688 [======>.......................] - ETA: 3s - loss: 0.5977 - accuracy: 0.9629
466/1688 [=======>......................] - ETA: 3s - loss: 0.5941 - accuracy: 0.9630
485/1688 [=======>......................] - ETA: 3s - loss: 0.5954 - accuracy: 0.9631
505/1688 [=======>......................] - ETA: 3s - loss: 0.6155 - accuracy: 0.9629
524/1688 [========>.....................] - ETA: 3s - loss: 0.6036 - accuracy: 0.9632
544/1688 [========>.....................] - ETA: 3s - loss: 0.5977 - accuracy: 0.9631
563/1688 [=========>....................] - ETA: 2s - loss: 0.5968 - accuracy: 0.9631
583/1688 [=========>....................] - ETA: 2s - loss: 0.5920 - accuracy: 0.9632
602/1688 [=========>....................] - ETA: 2s - loss: 0.5888 - accuracy: 0.9632
621/1688 [==========>...................] - ETA: 2s - loss: 0.5914 - accuracy: 0.9629
641/1688 [==========>...................] - ETA: 2s - loss: 0.5852 - accuracy: 0.9632
660/1688 [==========>...................] - ETA: 2s - loss: 0.5863 - accuracy: 0.9633
679/1688 [===========>..................] - ETA: 2s - loss: 0.5847 - accuracy: 0.9634
698/1688 [===========>..................] - ETA: 2s - loss: 0.5882 - accuracy: 0.9633
717/1688 [===========>..................] - ETA: 2s - loss: 0.5993 - accuracy: 0.9631
737/1688 [============>.................] - ETA: 2s - loss: 0.6049 - accuracy: 0.9628
756/1688 [============>.................] - ETA: 2s - loss: 0.6026 - accuracy: 0.9628
775/1688 [============>.................] - ETA: 2s - loss: 0.6049 - accuracy: 0.9629
794/1688 [=============>................] - ETA: 2s - loss: 0.6030 - accuracy: 0.9629
814/1688 [=============>................] - ETA: 2s - loss: 0.6055 - accuracy: 0.9627
833/1688 [=============>................] - ETA: 2s - loss: 0.6079 - accuracy: 0.9626
852/1688 [==============>...............] - ETA: 2s - loss: 0.6032 - accuracy: 0.9628
871/1688 [==============>...............] - ETA: 2s - loss: 0.5959 - accuracy: 0.9630
890/1688 [==============>...............] - ETA: 2s - loss: 0.6026 - accuracy: 0.9630
909/1688 [===============>..............] - ETA: 2s - loss: 0.5983 - accuracy: 0.9634
928/1688 [===============>..............] - ETA: 2s - loss: 0.5958 - accuracy: 0.9636
947/1688 [===============>..............] - ETA: 1s - loss: 0.5915 - accuracy: 0.9636
966/1688 [================>.............] - ETA: 1s - loss: 0.5915 - accuracy: 0.9635
986/1688 [================>.............] - ETA: 1s - loss: 0.5914 - accuracy: 0.9636
1005/1688 [================>.............] - ETA: 1s - loss: 0.5912 - accuracy: 0.9637
1024/1688 [=================>............] - ETA: 1s - loss: 0.5945 - accuracy: 0.9635
1043/1688 [=================>............] - ETA: 1s - loss: 0.5952 - accuracy: 0.9634
1062/1688 [=================>............] - ETA: 1s - loss: 0.6008 - accuracy: 0.9634
1081/1688 [==================>...........] - ETA: 1s - loss: 0.5977 - accuracy: 0.9636
1100/1688 [==================>...........] - ETA: 1s - loss: 0.5950 - accuracy: 0.9637
1119/1688 [==================>...........] - ETA: 1s - loss: 0.5928 - accuracy: 0.9638
1138/1688 [===================>..........] - ETA: 1s - loss: 0.5960 - accuracy: 0.9637
1157/1688 [===================>..........] - ETA: 1s - loss: 0.5995 - accuracy: 0.9636
1177/1688 [===================>..........] - ETA: 1s - loss: 0.5990 - accuracy: 0.9638
1196/1688 [====================>.........] - ETA: 1s - loss: 0.6021 - accuracy: 0.9637
1215/1688 [====================>.........] - ETA: 1s - loss: 0.6027 - accuracy: 0.9637
1234/1688 [====================>.........] - ETA: 1s - loss: 0.6042 - accuracy: 0.9637
1253/1688 [=====================>........] - ETA: 1s - loss: 0.6012 - accuracy: 0.9638
1272/1688 [=====================>........] - ETA: 1s - loss: 0.5987 - accuracy: 0.9639
1291/1688 [=====================>........] - ETA: 1s - loss: 0.6057 - accuracy: 0.9637
1310/1688 [======================>.......] - ETA: 1s - loss: 0.6089 - accuracy: 0.9637
1329/1688 [======================>.......] - ETA: 0s - loss: 0.6071 - accuracy: 0.9637
1348/1688 [======================>.......] - ETA: 0s - loss: 0.6074 - accuracy: 0.9638
1367/1688 [=======================>......] - ETA: 0s - loss: 0.6119 - accuracy: 0.9638
1386/1688 [=======================>......] - ETA: 0s - loss: 0.6080 - accuracy: 0.9639
1405/1688 [=======================>......] - ETA: 0s - loss: 0.6102 - accuracy: 0.9638
1424/1688 [========================>.....] - ETA: 0s - loss: 0.6093 - accuracy: 0.9639
1443/1688 [========================>.....] - ETA: 0s - loss: 0.6063 - accuracy: 0.9640
1462/1688 [========================>.....] - ETA: 0s - loss: 0.6073 - accuracy: 0.9640
1482/1688 [=========================>....] - ETA: 0s - loss: 0.6064 - accuracy: 0.9641
1501/1688 [=========================>....] - ETA: 0s - loss: 0.6076 - accuracy: 0.9641
1520/1688 [==========================>...] - ETA: 0s - loss: 0.6089 - accuracy: 0.9640
1540/1688 [==========================>...] - ETA: 0s - loss: 0.6057 - accuracy: 0.9641
1559/1688 [==========================>...] - ETA: 0s - loss: 0.6027 - accuracy: 0.9643
1578/1688 [===========================>..] - ETA: 0s - loss: 0.6037 - accuracy: 0.9643
1597/1688 [===========================>..] - ETA: 0s - loss: 0.6014 - accuracy: 0.9644
1616/1688 [===========================>..] - ETA: 0s - loss: 0.5999 - accuracy: 0.9646
1635/1688 [============================>.] - ETA: 0s - loss: 0.6035 - accuracy: 0.9646
1654/1688 [============================>.] - ETA: 0s - loss: 0.6014 - accuracy: 0.9647
1673/1688 [============================>.] - ETA: 0s - loss: 0.6014 - accuracy: 0.9647
1688/1688 [==============================] - 5s 3ms/step - loss: 0.6004 - accuracy: 0.9648 - val_loss: 0.7541 - val_accuracy: 0.9695
Epoch 3/5
1/1688 [..............................] - ETA: 4s - loss: 0.7020 - accuracy: 0.9688
21/1688 [..............................] - ETA: 4s - loss: 0.5888 - accuracy: 0.9673
41/1688 [..............................] - ETA: 4s - loss: 0.5305 - accuracy: 0.9703
60/1688 [>.............................] - ETA: 4s - loss: 0.5717 - accuracy: 0.9661
79/1688 [>.............................] - ETA: 4s - loss: 0.5375 - accuracy: 0.9672
98/1688 [>.............................] - ETA: 4s - loss: 0.5005 - accuracy: 0.9688
117/1688 [=>............................] - ETA: 4s - loss: 0.5018 - accuracy: 0.9693
136/1688 [=>............................] - ETA: 4s - loss: 0.5114 - accuracy: 0.9694
155/1688 [=>............................] - ETA: 4s - loss: 0.5463 - accuracy: 0.9675
174/1688 [==>...........................] - ETA: 4s - loss: 0.5436 - accuracy: 0.9682
193/1688 [==>...........................] - ETA: 3s - loss: 0.5427 - accuracy: 0.9688
212/1688 [==>...........................] - ETA: 3s - loss: 0.5275 - accuracy: 0.9692
231/1688 [===>..........................] - ETA: 3s - loss: 0.5120 - accuracy: 0.9696
251/1688 [===>..........................] - ETA: 3s - loss: 0.5072 - accuracy: 0.9699
270/1688 [===>..........................] - ETA: 3s - loss: 0.4963 - accuracy: 0.9703
289/1688 [====>.........................] - ETA: 3s - loss: 0.5085 - accuracy: 0.9699
308/1688 [====>.........................] - ETA: 3s - loss: 0.4982 - accuracy: 0.9702
328/1688 [====>.........................] - ETA: 3s - loss: 0.5091 - accuracy: 0.9696
347/1688 [=====>........................] - ETA: 3s - loss: 0.5169 - accuracy: 0.9695
367/1688 [=====>........................] - ETA: 3s - loss: 0.5249 - accuracy: 0.9699
386/1688 [=====>........................] - ETA: 3s - loss: 0.5282 - accuracy: 0.9700
405/1688 [======>.......................] - ETA: 3s - loss: 0.5303 - accuracy: 0.9698
424/1688 [======>.......................] - ETA: 3s - loss: 0.5276 - accuracy: 0.9703
444/1688 [======>.......................] - ETA: 3s - loss: 0.5221 - accuracy: 0.9704
463/1688 [=======>......................] - ETA: 3s - loss: 0.5368 - accuracy: 0.9700
482/1688 [=======>......................] - ETA: 3s - loss: 0.5390 - accuracy: 0.9699
501/1688 [=======>......................] - ETA: 3s - loss: 0.5527 - accuracy: 0.9696
520/1688 [========>.....................] - ETA: 3s - loss: 0.5720 - accuracy: 0.9694
539/1688 [========>.....................] - ETA: 3s - loss: 0.5585 - accuracy: 0.9697
558/1688 [========>.....................] - ETA: 3s - loss: 0.5479 - accuracy: 0.9700
577/1688 [=========>....................] - ETA: 2s - loss: 0.5407 - accuracy: 0.9704
597/1688 [=========>....................] - ETA: 2s - loss: 0.5579 - accuracy: 0.9701
616/1688 [=========>....................] - ETA: 2s - loss: 0.5578 - accuracy: 0.9701
635/1688 [==========>...................] - ETA: 2s - loss: 0.5513 - accuracy: 0.9703
654/1688 [==========>...................] - ETA: 2s - loss: 0.5494 - accuracy: 0.9707
673/1688 [==========>...................] - ETA: 2s - loss: 0.5501 - accuracy: 0.9704
692/1688 [===========>..................] - ETA: 2s - loss: 0.5451 - accuracy: 0.9704
711/1688 [===========>..................] - ETA: 2s - loss: 0.5380 - accuracy: 0.9705
731/1688 [===========>..................] - ETA: 2s - loss: 0.5400 - accuracy: 0.9703
750/1688 [============>.................] - ETA: 2s - loss: 0.5374 - accuracy: 0.9704
769/1688 [============>.................] - ETA: 2s - loss: 0.5352 - accuracy: 0.9707
788/1688 [=============>................] - ETA: 2s - loss: 0.5371 - accuracy: 0.9706
807/1688 [=============>................] - ETA: 2s - loss: 0.5377 - accuracy: 0.9707
826/1688 [=============>................] - ETA: 2s - loss: 0.5385 - accuracy: 0.9708
845/1688 [==============>...............] - ETA: 2s - loss: 0.5377 - accuracy: 0.9708
864/1688 [==============>...............] - ETA: 2s - loss: 0.5414 - accuracy: 0.9706
884/1688 [==============>...............] - ETA: 2s - loss: 0.5410 - accuracy: 0.9705
903/1688 [===============>..............] - ETA: 2s - loss: 0.5391 - accuracy: 0.9705
922/1688 [===============>..............] - ETA: 2s - loss: 0.5480 - accuracy: 0.9704
941/1688 [===============>..............] - ETA: 1s - loss: 0.5502 - accuracy: 0.9704
960/1688 [================>.............] - ETA: 1s - loss: 0.5568 - accuracy: 0.9703
979/1688 [================>.............] - ETA: 1s - loss: 0.5562 - accuracy: 0.9702
999/1688 [================>.............] - ETA: 1s - loss: 0.5558 - accuracy: 0.9701
1018/1688 [=================>............] - ETA: 1s - loss: 0.5567 - accuracy: 0.9702
1037/1688 [=================>............] - ETA: 1s - loss: 0.5526 - accuracy: 0.9701
1056/1688 [=================>............] - ETA: 1s - loss: 0.5515 - accuracy: 0.9700
1075/1688 [==================>...........] - ETA: 1s - loss: 0.5564 - accuracy: 0.9700
1095/1688 [==================>...........] - ETA: 1s - loss: 0.5627 - accuracy: 0.9699
1114/1688 [==================>...........] - ETA: 1s - loss: 0.5603 - accuracy: 0.9699
1133/1688 [===================>..........] - ETA: 1s - loss: 0.5580 - accuracy: 0.9700
1153/1688 [===================>..........] - ETA: 1s - loss: 0.5599 - accuracy: 0.9701
1173/1688 [===================>..........] - ETA: 1s - loss: 0.5649 - accuracy: 0.9700
1192/1688 [====================>.........] - ETA: 1s - loss: 0.5662 - accuracy: 0.9700
1211/1688 [====================>.........] - ETA: 1s - loss: 0.5664 - accuracy: 0.9700
1230/1688 [====================>.........] - ETA: 1s - loss: 0.5661 - accuracy: 0.9700
1249/1688 [=====================>........] - ETA: 1s - loss: 0.5669 - accuracy: 0.9700
1268/1688 [=====================>........] - ETA: 1s - loss: 0.5648 - accuracy: 0.9702
1287/1688 [=====================>........] - ETA: 1s - loss: 0.5683 - accuracy: 0.9700
1306/1688 [======================>.......] - ETA: 1s - loss: 0.5660 - accuracy: 0.9700
1325/1688 [======================>.......] - ETA: 0s - loss: 0.5658 - accuracy: 0.9699
1344/1688 [======================>.......] - ETA: 0s - loss: 0.5636 - accuracy: 0.9701
1363/1688 [=======================>......] - ETA: 0s - loss: 0.5719 - accuracy: 0.9700
1382/1688 [=======================>......] - ETA: 0s - loss: 0.5735 - accuracy: 0.9700
1401/1688 [=======================>......] - ETA: 0s - loss: 0.5758 - accuracy: 0.9700
1420/1688 [========================>.....] - ETA: 0s - loss: 0.5740 - accuracy: 0.9701
1439/1688 [========================>.....] - ETA: 0s - loss: 0.5777 - accuracy: 0.9701
1458/1688 [========================>.....] - ETA: 0s - loss: 0.5776 - accuracy: 0.9702
1478/1688 [=========================>....] - ETA: 0s - loss: 0.5812 - accuracy: 0.9701
1497/1688 [=========================>....] - ETA: 0s - loss: 0.5791 - accuracy: 0.9701
1516/1688 [=========================>....] - ETA: 0s - loss: 0.5765 - accuracy: 0.9702
1535/1688 [==========================>...] - ETA: 0s - loss: 0.5788 - accuracy: 0.9702
1554/1688 [==========================>...] - ETA: 0s - loss: 0.5791 - accuracy: 0.9701
1573/1688 [==========================>...] - ETA: 0s - loss: 0.5800 - accuracy: 0.9701
1593/1688 [===========================>..] - ETA: 0s - loss: 0.5806 - accuracy: 0.9701
1612/1688 [===========================>..] - ETA: 0s - loss: 0.5808 - accuracy: 0.9702
1632/1688 [============================>.] - ETA: 0s - loss: 0.5778 - accuracy: 0.9703
1651/1688 [============================>.] - ETA: 0s - loss: 0.5788 - accuracy: 0.9702
1670/1688 [============================>.] - ETA: 0s - loss: 0.5763 - accuracy: 0.9702
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5746 - accuracy: 0.9702 - val_loss: 0.6071 - val_accuracy: 0.9743
Epoch 4/5
1/1688 [..............................] - ETA: 4s - loss: 0.0535 - accuracy: 0.9688
21/1688 [..............................] - ETA: 4s - loss: 0.3352 - accuracy: 0.9821
41/1688 [..............................] - ETA: 4s - loss: 0.3516 - accuracy: 0.9794
60/1688 [>.............................] - ETA: 4s - loss: 0.4238 - accuracy: 0.9776
79/1688 [>.............................] - ETA: 4s - loss: 0.4734 - accuracy: 0.9751
98/1688 [>.............................] - ETA: 4s - loss: 0.5166 - accuracy: 0.9735
117/1688 [=>............................] - ETA: 4s - loss: 0.4890 - accuracy: 0.9752
136/1688 [=>............................] - ETA: 4s - loss: 0.5029 - accuracy: 0.9752
155/1688 [=>............................] - ETA: 4s - loss: 0.4879 - accuracy: 0.9750
174/1688 [==>...........................] - ETA: 4s - loss: 0.4574 - accuracy: 0.9758
193/1688 [==>...........................] - ETA: 3s - loss: 0.4320 - accuracy: 0.9767
212/1688 [==>...........................] - ETA: 3s - loss: 0.4257 - accuracy: 0.9760
231/1688 [===>..........................] - ETA: 3s - loss: 0.4216 - accuracy: 0.9759
251/1688 [===>..........................] - ETA: 3s - loss: 0.4344 - accuracy: 0.9755
270/1688 [===>..........................] - ETA: 3s - loss: 0.4318 - accuracy: 0.9753
289/1688 [====>.........................] - ETA: 3s - loss: 0.4263 - accuracy: 0.9759
308/1688 [====>.........................] - ETA: 3s - loss: 0.4193 - accuracy: 0.9760
327/1688 [====>.........................] - ETA: 3s - loss: 0.4376 - accuracy: 0.9753
346/1688 [=====>........................] - ETA: 3s - loss: 0.4550 - accuracy: 0.9751
365/1688 [=====>........................] - ETA: 3s - loss: 0.4606 - accuracy: 0.9748
384/1688 [=====>........................] - ETA: 3s - loss: 0.4572 - accuracy: 0.9748
403/1688 [======>.......................] - ETA: 3s - loss: 0.4602 - accuracy: 0.9746
422/1688 [======>.......................] - ETA: 3s - loss: 0.4591 - accuracy: 0.9745
442/1688 [======>.......................] - ETA: 3s - loss: 0.4612 - accuracy: 0.9745
461/1688 [=======>......................] - ETA: 3s - loss: 0.4706 - accuracy: 0.9741
481/1688 [=======>......................] - ETA: 3s - loss: 0.4637 - accuracy: 0.9743
500/1688 [=======>......................] - ETA: 3s - loss: 0.4670 - accuracy: 0.9742
519/1688 [========>.....................] - ETA: 3s - loss: 0.4718 - accuracy: 0.9736
539/1688 [========>.....................] - ETA: 3s - loss: 0.4683 - accuracy: 0.9739
558/1688 [========>.....................] - ETA: 2s - loss: 0.4920 - accuracy: 0.9732
577/1688 [=========>....................] - ETA: 2s - loss: 0.4933 - accuracy: 0.9726
596/1688 [=========>....................] - ETA: 2s - loss: 0.4910 - accuracy: 0.9728
615/1688 [=========>....................] - ETA: 2s - loss: 0.4891 - accuracy: 0.9727
634/1688 [==========>...................] - ETA: 2s - loss: 0.4889 - accuracy: 0.9726
653/1688 [==========>...................] - ETA: 2s - loss: 0.4916 - accuracy: 0.9727
672/1688 [==========>...................] - ETA: 2s - loss: 0.4923 - accuracy: 0.9727
691/1688 [===========>..................] - ETA: 2s - loss: 0.4933 - accuracy: 0.9727
711/1688 [===========>..................] - ETA: 2s - loss: 0.4914 - accuracy: 0.9727
730/1688 [===========>..................] - ETA: 2s - loss: 0.4955 - accuracy: 0.9726
750/1688 [============>.................] - ETA: 2s - loss: 0.4967 - accuracy: 0.9726
769/1688 [============>.................] - ETA: 2s - loss: 0.4934 - accuracy: 0.9728
789/1688 [=============>................] - ETA: 2s - loss: 0.4941 - accuracy: 0.9727
808/1688 [=============>................] - ETA: 2s - loss: 0.4924 - accuracy: 0.9728
827/1688 [=============>................] - ETA: 2s - loss: 0.5065 - accuracy: 0.9724
846/1688 [==============>...............] - ETA: 2s - loss: 0.5048 - accuracy: 0.9725
865/1688 [==============>...............] - ETA: 2s - loss: 0.5109 - accuracy: 0.9724
884/1688 [==============>...............] - ETA: 2s - loss: 0.5093 - accuracy: 0.9724
903/1688 [===============>..............] - ETA: 2s - loss: 0.5149 - accuracy: 0.9723
923/1688 [===============>..............] - ETA: 2s - loss: 0.5104 - accuracy: 0.9724
942/1688 [===============>..............] - ETA: 1s - loss: 0.5126 - accuracy: 0.9722
961/1688 [================>.............] - ETA: 1s - loss: 0.5183 - accuracy: 0.9720
981/1688 [================>.............] - ETA: 1s - loss: 0.5149 - accuracy: 0.9721
1000/1688 [================>.............] - ETA: 1s - loss: 0.5189 - accuracy: 0.9720
1019/1688 [=================>............] - ETA: 1s - loss: 0.5166 - accuracy: 0.9721
1038/1688 [=================>............] - ETA: 1s - loss: 0.5189 - accuracy: 0.9721
1057/1688 [=================>............] - ETA: 1s - loss: 0.5189 - accuracy: 0.9722
1076/1688 [==================>...........] - ETA: 1s - loss: 0.5193 - accuracy: 0.9722
1096/1688 [==================>...........] - ETA: 1s - loss: 0.5241 - accuracy: 0.9720
1115/1688 [==================>...........] - ETA: 1s - loss: 0.5187 - accuracy: 0.9722
1134/1688 [===================>..........] - ETA: 1s - loss: 0.5222 - accuracy: 0.9721
1153/1688 [===================>..........] - ETA: 1s - loss: 0.5241 - accuracy: 0.9720
1173/1688 [===================>..........] - ETA: 1s - loss: 0.5271 - accuracy: 0.9719
1192/1688 [====================>.........] - ETA: 1s - loss: 0.5300 - accuracy: 0.9719
1211/1688 [====================>.........] - ETA: 1s - loss: 0.5296 - accuracy: 0.9720
1230/1688 [====================>.........] - ETA: 1s - loss: 0.5268 - accuracy: 0.9720
1249/1688 [=====================>........] - ETA: 1s - loss: 0.5245 - accuracy: 0.9722
1268/1688 [=====================>........] - ETA: 1s - loss: 0.5239 - accuracy: 0.9722
1287/1688 [=====================>........] - ETA: 1s - loss: 0.5231 - accuracy: 0.9722
1306/1688 [======================>.......] - ETA: 1s - loss: 0.5292 - accuracy: 0.9722
1325/1688 [======================>.......] - ETA: 0s - loss: 0.5331 - accuracy: 0.9721
1344/1688 [======================>.......] - ETA: 0s - loss: 0.5340 - accuracy: 0.9720
1363/1688 [=======================>......] - ETA: 0s - loss: 0.5317 - accuracy: 0.9721
1382/1688 [=======================>......] - ETA: 0s - loss: 0.5297 - accuracy: 0.9720
1402/1688 [=======================>......] - ETA: 0s - loss: 0.5283 - accuracy: 0.9721
1421/1688 [========================>.....] - ETA: 0s - loss: 0.5257 - accuracy: 0.9721
1441/1688 [========================>.....] - ETA: 0s - loss: 0.5262 - accuracy: 0.9723
1460/1688 [========================>.....] - ETA: 0s - loss: 0.5260 - accuracy: 0.9725
1479/1688 [=========================>....] - ETA: 0s - loss: 0.5294 - accuracy: 0.9724
1499/1688 [=========================>....] - ETA: 0s - loss: 0.5304 - accuracy: 0.9725
1518/1688 [=========================>....] - ETA: 0s - loss: 0.5279 - accuracy: 0.9725
1537/1688 [==========================>...] - ETA: 0s - loss: 0.5287 - accuracy: 0.9725
1556/1688 [==========================>...] - ETA: 0s - loss: 0.5274 - accuracy: 0.9726
1575/1688 [==========================>...] - ETA: 0s - loss: 0.5282 - accuracy: 0.9725
1594/1688 [===========================>..] - ETA: 0s - loss: 0.5266 - accuracy: 0.9725
1613/1688 [===========================>..] - ETA: 0s - loss: 0.5274 - accuracy: 0.9725
1632/1688 [============================>.] - ETA: 0s - loss: 0.5245 - accuracy: 0.9726
1651/1688 [============================>.] - ETA: 0s - loss: 0.5285 - accuracy: 0.9725
1670/1688 [============================>.] - ETA: 0s - loss: 0.5284 - accuracy: 0.9726
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5284 - accuracy: 0.9726 - val_loss: 0.7504 - val_accuracy: 0.9713
Epoch 5/5
1/1688 [..............................] - ETA: 4s - loss: 0.0121 - accuracy: 1.0000
20/1688 [..............................] - ETA: 4s - loss: 0.4761 - accuracy: 0.9750
40/1688 [..............................] - ETA: 4s - loss: 0.7435 - accuracy: 0.9742
59/1688 [>.............................] - ETA: 4s - loss: 0.6715 - accuracy: 0.9746
79/1688 [>.............................] - ETA: 4s - loss: 0.6209 - accuracy: 0.9751
98/1688 [>.............................] - ETA: 4s - loss: 0.6231 - accuracy: 0.9748
117/1688 [=>............................] - ETA: 4s - loss: 0.5633 - accuracy: 0.9760
136/1688 [=>............................] - ETA: 4s - loss: 0.5346 - accuracy: 0.9756
155/1688 [=>............................] - ETA: 4s - loss: 0.5332 - accuracy: 0.9764
174/1688 [==>...........................] - ETA: 4s - loss: 0.5133 - accuracy: 0.9772
193/1688 [==>...........................] - ETA: 3s - loss: 0.5337 - accuracy: 0.9772
213/1688 [==>...........................] - ETA: 3s - loss: 0.4992 - accuracy: 0.9778
233/1688 [===>..........................] - ETA: 3s - loss: 0.4724 - accuracy: 0.9783
253/1688 [===>..........................] - ETA: 3s - loss: 0.4689 - accuracy: 0.9781
272/1688 [===>..........................] - ETA: 3s - loss: 0.4720 - accuracy: 0.9781
291/1688 [====>.........................] - ETA: 3s - loss: 0.4757 - accuracy: 0.9781
310/1688 [====>.........................] - ETA: 3s - loss: 0.4701 - accuracy: 0.9784
330/1688 [====>.........................] - ETA: 3s - loss: 0.4607 - accuracy: 0.9781
349/1688 [=====>........................] - ETA: 3s - loss: 0.4492 - accuracy: 0.9780
368/1688 [=====>........................] - ETA: 3s - loss: 0.4524 - accuracy: 0.9778
387/1688 [=====>........................] - ETA: 3s - loss: 0.4523 - accuracy: 0.9780
406/1688 [======>.......................] - ETA: 3s - loss: 0.4556 - accuracy: 0.9781
425/1688 [======>.......................] - ETA: 3s - loss: 0.4736 - accuracy: 0.9782
445/1688 [======>.......................] - ETA: 3s - loss: 0.4882 - accuracy: 0.9780
464/1688 [=======>......................] - ETA: 3s - loss: 0.4954 - accuracy: 0.9774
483/1688 [=======>......................] - ETA: 3s - loss: 0.4931 - accuracy: 0.9775
502/1688 [=======>......................] - ETA: 3s - loss: 0.4891 - accuracy: 0.9775
521/1688 [========>.....................] - ETA: 3s - loss: 0.4832 - accuracy: 0.9773
540/1688 [========>.....................] - ETA: 3s - loss: 0.4886 - accuracy: 0.9768
560/1688 [========>.....................] - ETA: 3s - loss: 0.4906 - accuracy: 0.9768
579/1688 [=========>....................] - ETA: 2s - loss: 0.4896 - accuracy: 0.9770
598/1688 [=========>....................] - ETA: 2s - loss: 0.4886 - accuracy: 0.9770
617/1688 [=========>....................] - ETA: 2s - loss: 0.4888 - accuracy: 0.9769
636/1688 [==========>...................] - ETA: 2s - loss: 0.4961 - accuracy: 0.9765
655/1688 [==========>...................] - ETA: 2s - loss: 0.4945 - accuracy: 0.9762
675/1688 [==========>...................] - ETA: 2s - loss: 0.4967 - accuracy: 0.9761
694/1688 [===========>..................] - ETA: 2s - loss: 0.5044 - accuracy: 0.9760
714/1688 [===========>..................] - ETA: 2s - loss: 0.5058 - accuracy: 0.9758
733/1688 [============>.................] - ETA: 2s - loss: 0.5093 - accuracy: 0.9755
752/1688 [============>.................] - ETA: 2s - loss: 0.5073 - accuracy: 0.9753
771/1688 [============>.................] - ETA: 2s - loss: 0.5057 - accuracy: 0.9754
790/1688 [=============>................] - ETA: 2s - loss: 0.5093 - accuracy: 0.9753
809/1688 [=============>................] - ETA: 2s - loss: 0.5033 - accuracy: 0.9755
828/1688 [=============>................] - ETA: 2s - loss: 0.5103 - accuracy: 0.9753
847/1688 [==============>...............] - ETA: 2s - loss: 0.5068 - accuracy: 0.9754
866/1688 [==============>...............] - ETA: 2s - loss: 0.5110 - accuracy: 0.9752
885/1688 [==============>...............] - ETA: 2s - loss: 0.5179 - accuracy: 0.9749
904/1688 [===============>..............] - ETA: 2s - loss: 0.5216 - accuracy: 0.9749
923/1688 [===============>..............] - ETA: 2s - loss: 0.5251 - accuracy: 0.9748
943/1688 [===============>..............] - ETA: 1s - loss: 0.5208 - accuracy: 0.9750
961/1688 [================>.............] - ETA: 1s - loss: 0.5181 - accuracy: 0.9751
981/1688 [================>.............] - ETA: 1s - loss: 0.5213 - accuracy: 0.9752
1001/1688 [================>.............] - ETA: 1s - loss: 0.5159 - accuracy: 0.9754
1020/1688 [=================>............] - ETA: 1s - loss: 0.5171 - accuracy: 0.9753
1039/1688 [=================>............] - ETA: 1s - loss: 0.5214 - accuracy: 0.9752
1058/1688 [=================>............] - ETA: 1s - loss: 0.5327 - accuracy: 0.9749
1077/1688 [==================>...........] - ETA: 1s - loss: 0.5342 - accuracy: 0.9748
1096/1688 [==================>...........] - ETA: 1s - loss: 0.5334 - accuracy: 0.9748
1115/1688 [==================>...........] - ETA: 1s - loss: 0.5386 - accuracy: 0.9746
1135/1688 [===================>..........] - ETA: 1s - loss: 0.5340 - accuracy: 0.9748
1155/1688 [===================>..........] - ETA: 1s - loss: 0.5346 - accuracy: 0.9747
1174/1688 [===================>..........] - ETA: 1s - loss: 0.5349 - accuracy: 0.9746
1193/1688 [====================>.........] - ETA: 1s - loss: 0.5366 - accuracy: 0.9745
1212/1688 [====================>.........] - ETA: 1s - loss: 0.5432 - accuracy: 0.9743
1231/1688 [====================>.........] - ETA: 1s - loss: 0.5428 - accuracy: 0.9743
1251/1688 [=====================>........] - ETA: 1s - loss: 0.5419 - accuracy: 0.9743
1270/1688 [=====================>........] - ETA: 1s - loss: 0.5380 - accuracy: 0.9745
1289/1688 [=====================>........] - ETA: 1s - loss: 0.5331 - accuracy: 0.9746
1308/1688 [======================>.......] - ETA: 1s - loss: 0.5311 - accuracy: 0.9746
1327/1688 [======================>.......] - ETA: 0s - loss: 0.5352 - accuracy: 0.9744
1346/1688 [======================>.......] - ETA: 0s - loss: 0.5351 - accuracy: 0.9745
1365/1688 [=======================>......] - ETA: 0s - loss: 0.5354 - accuracy: 0.9745
1384/1688 [=======================>......] - ETA: 0s - loss: 0.5327 - accuracy: 0.9746
1404/1688 [=======================>......] - ETA: 0s - loss: 0.5314 - accuracy: 0.9746
1424/1688 [========================>.....] - ETA: 0s - loss: 0.5276 - accuracy: 0.9748
1443/1688 [========================>.....] - ETA: 0s - loss: 0.5310 - accuracy: 0.9746
1462/1688 [========================>.....] - ETA: 0s - loss: 0.5347 - accuracy: 0.9746
1481/1688 [=========================>....] - ETA: 0s - loss: 0.5355 - accuracy: 0.9746
1500/1688 [=========================>....] - ETA: 0s - loss: 0.5357 - accuracy: 0.9746
1520/1688 [==========================>...] - ETA: 0s - loss: 0.5334 - accuracy: 0.9747
1539/1688 [==========================>...] - ETA: 0s - loss: 0.5320 - accuracy: 0.9746
1559/1688 [==========================>...] - ETA: 0s - loss: 0.5321 - accuracy: 0.9746
1578/1688 [===========================>..] - ETA: 0s - loss: 0.5338 - accuracy: 0.9747
1597/1688 [===========================>..] - ETA: 0s - loss: 0.5316 - accuracy: 0.9748
1616/1688 [===========================>..] - ETA: 0s - loss: 0.5307 - accuracy: 0.9748
1636/1688 [============================>.] - ETA: 0s - loss: 0.5310 - accuracy: 0.9746
1655/1688 [============================>.] - ETA: 0s - loss: 0.5272 - accuracy: 0.9747
1674/1688 [============================>.] - ETA: 0s - loss: 0.5277 - accuracy: 0.9747
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5298 - accuracy: 0.9746 - val_loss: 0.8022 - val_accuracy: 0.9705
Test accuracy after fine tuning: 0.9660000205039978
6. Model conversion
After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida software simulator.
Note
One needs to supply the coefficients used to rescale the input
dataset before the training - here input_scaling.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()
accuracy = model_akida.evaluate(raw_x_test, raw_y_test)
print('Test accuracy after conversion:', accuracy)
# For non-regression purpose
assert accuracy > 0.96
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 3
______________________________________________
______________________________________________________________
Layer (type) Output shape Kernel shape
================= SW/conv2d-dense (Software) =================
conv2d (InputConv.) [13, 13, 32] (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.) [7, 7, 64] (3, 3, 32, 1)
______________________________________________________________
(1, 1, 32, 64)
______________________________________________________________
dense (Fully.) [1, 1, 10] (1, 1, 3136, 10)
______________________________________________________________
Test accuracy after conversion: 0.964900016784668
Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).
Total running time of the script: ( 1 minutes 2.601 seconds)