Note
Click here to download the full example code
CNN conversion flow tutorial¶
This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.
The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.
Note
Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.
MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.
1. Load and reshape MNIST dataset¶
After loading, we make 2 transformations on the dataset:
Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.
Note
At this point, we’ll set aside the raw data for testing our converted model in the Akida Execution Engine later.
Rescale the 8-bit loaded data to the range 0-to-1 for training.
Note
Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.
This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.
Also note that we store the scaling values input_scaling for
use when preparing the model for the Akida Execution Engine. The
implementation of the Akida neural network allows us to completely
skip the rescaling step (i.e. the Akida model should be fed with
the raw 8-bit values) but that does require information about what
scaling was applied prior to training - see below for more details.
import tensorflow as tf
from tensorflow import keras
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)
# Set aside raw test data for use with Akida Execution Engine later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test
# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a
2. Model definition¶
Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.
An appropriate model for MNIST (inspired by this example) might look something like the following:
model_keras = keras.models.Sequential([
keras.layers.Conv2D(filters=32, kernel_size=3, input_shape=(28, 28, 1)),
keras.layers.MaxPool2D(),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.SeparableConv2D(filters=64, kernel_size=3, padding='same'),
keras.layers.MaxPool2D(padding='same'),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Out:
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 26, 26, 32) 320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32) 0
_________________________________________________________________
batch_normalization (BatchNo (None, 13, 13, 32) 128
_________________________________________________________________
re_lu (ReLU) (None, 13, 13, 32) 0
_________________________________________________________________
separable_conv2d (SeparableC (None, 13, 13, 64) 2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64) 0
_________________________________________________________________
batch_normalization_1 (Batch (None, 7, 7, 64) 256
_________________________________________________________________
re_lu_1 (ReLU) (None, 7, 7, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 3136) 0
_________________________________________________________________
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________
The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):
Standard Conv2D and Dense layers are supported
Hidden layers must be followed by a ReLU layer.
BatchNormalization must always happen before activations.
Convolutional blocks can optionally be followed by a MaxPooling.
The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).
from cnn2snn import check_model_compatibility
print("Model compatible for Akida conversion:",
check_model_compatibility(model_keras))
Out:
Model compatible for Akida conversion: True
3. Model training¶
Before going any further, train the model and get its performance. The created model should have achieved a test accuracy a little over 99% after 10 epochs.
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])
Out:
Epoch 1/10
1/1688 [..............................] - ETA: 10:10 - loss: 3.3138 - accuracy: 0.0625
30/1688 [..............................] - ETA: 2s - loss: 1.7556 - accuracy: 0.4486
59/1688 [>.............................] - ETA: 2s - loss: 1.3453 - accuracy: 0.5822
89/1688 [>.............................] - ETA: 2s - loss: 1.1229 - accuracy: 0.6538
119/1688 [=>............................] - ETA: 2s - loss: 0.9790 - accuracy: 0.6995
149/1688 [=>............................] - ETA: 2s - loss: 0.8779 - accuracy: 0.7315
179/1688 [==>...........................] - ETA: 2s - loss: 0.8024 - accuracy: 0.7555
209/1688 [==>...........................] - ETA: 2s - loss: 0.7433 - accuracy: 0.7742
238/1688 [===>..........................] - ETA: 2s - loss: 0.6966 - accuracy: 0.7888
268/1688 [===>..........................] - ETA: 2s - loss: 0.6562 - accuracy: 0.8014
297/1688 [====>.........................] - ETA: 2s - loss: 0.6226 - accuracy: 0.8118
327/1688 [====>.........................] - ETA: 2s - loss: 0.5923 - accuracy: 0.8212
356/1688 [=====>........................] - ETA: 2s - loss: 0.5668 - accuracy: 0.8291
386/1688 [=====>........................] - ETA: 2s - loss: 0.5435 - accuracy: 0.8363
416/1688 [======>.......................] - ETA: 2s - loss: 0.5226 - accuracy: 0.8427
446/1688 [======>.......................] - ETA: 2s - loss: 0.5039 - accuracy: 0.8484
475/1688 [=======>......................] - ETA: 2s - loss: 0.4876 - accuracy: 0.8535
505/1688 [=======>......................] - ETA: 2s - loss: 0.4721 - accuracy: 0.8582
534/1688 [========>.....................] - ETA: 1s - loss: 0.4584 - accuracy: 0.8623
564/1688 [=========>....................] - ETA: 1s - loss: 0.4453 - accuracy: 0.8663
593/1688 [=========>....................] - ETA: 1s - loss: 0.4336 - accuracy: 0.8698
623/1688 [==========>...................] - ETA: 1s - loss: 0.4224 - accuracy: 0.8732
653/1688 [==========>...................] - ETA: 1s - loss: 0.4119 - accuracy: 0.8763
683/1688 [===========>..................] - ETA: 1s - loss: 0.4022 - accuracy: 0.8793
713/1688 [===========>..................] - ETA: 1s - loss: 0.3930 - accuracy: 0.8820
744/1688 [============>.................] - ETA: 1s - loss: 0.3842 - accuracy: 0.8847
773/1688 [============>.................] - ETA: 1s - loss: 0.3764 - accuracy: 0.8871
803/1688 [=============>................] - ETA: 1s - loss: 0.3688 - accuracy: 0.8893
833/1688 [=============>................] - ETA: 1s - loss: 0.3616 - accuracy: 0.8915
862/1688 [==============>...............] - ETA: 1s - loss: 0.3550 - accuracy: 0.8935
891/1688 [==============>...............] - ETA: 1s - loss: 0.3488 - accuracy: 0.8954
921/1688 [===============>..............] - ETA: 1s - loss: 0.3426 - accuracy: 0.8972
950/1688 [===============>..............] - ETA: 1s - loss: 0.3370 - accuracy: 0.8989
980/1688 [================>.............] - ETA: 1s - loss: 0.3315 - accuracy: 0.9005
1010/1688 [================>.............] - ETA: 1s - loss: 0.3262 - accuracy: 0.9021
1040/1688 [=================>............] - ETA: 1s - loss: 0.3211 - accuracy: 0.9036
1070/1688 [==================>...........] - ETA: 1s - loss: 0.3163 - accuracy: 0.9051
1099/1688 [==================>...........] - ETA: 1s - loss: 0.3118 - accuracy: 0.9064
1129/1688 [===================>..........] - ETA: 0s - loss: 0.3073 - accuracy: 0.9078
1159/1688 [===================>..........] - ETA: 0s - loss: 0.3030 - accuracy: 0.9090
1189/1688 [====================>.........] - ETA: 0s - loss: 0.2989 - accuracy: 0.9103
1219/1688 [====================>.........] - ETA: 0s - loss: 0.2950 - accuracy: 0.9114
1249/1688 [=====================>........] - ETA: 0s - loss: 0.2912 - accuracy: 0.9126
1278/1688 [=====================>........] - ETA: 0s - loss: 0.2877 - accuracy: 0.9136
1308/1688 [======================>.......] - ETA: 0s - loss: 0.2842 - accuracy: 0.9146
1338/1688 [======================>.......] - ETA: 0s - loss: 0.2808 - accuracy: 0.9157
1367/1688 [=======================>......] - ETA: 0s - loss: 0.2777 - accuracy: 0.9166
1397/1688 [=======================>......] - ETA: 0s - loss: 0.2745 - accuracy: 0.9176
1427/1688 [========================>.....] - ETA: 0s - loss: 0.2715 - accuracy: 0.9185
1456/1688 [========================>.....] - ETA: 0s - loss: 0.2686 - accuracy: 0.9193
1485/1688 [=========================>....] - ETA: 0s - loss: 0.2659 - accuracy: 0.9202
1514/1688 [=========================>....] - ETA: 0s - loss: 0.2632 - accuracy: 0.9210
1544/1688 [==========================>...] - ETA: 0s - loss: 0.2605 - accuracy: 0.9218
1574/1688 [==========================>...] - ETA: 0s - loss: 0.2579 - accuracy: 0.9226
1604/1688 [===========================>..] - ETA: 0s - loss: 0.2553 - accuracy: 0.9233
1634/1688 [============================>.] - ETA: 0s - loss: 0.2529 - accuracy: 0.9241
1664/1688 [============================>.] - ETA: 0s - loss: 0.2505 - accuracy: 0.9248
1688/1688 [==============================] - ETA: 0s - loss: 0.2487 - accuracy: 0.9253
1688/1688 [==============================] - 4s 2ms/step - loss: 0.2486 - accuracy: 0.9253 - val_loss: 0.0547 - val_accuracy: 0.9853
Epoch 2/10
1/1688 [..............................] - ETA: 3s - loss: 0.0060 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0425 - accuracy: 0.9876
62/1688 [>.............................] - ETA: 2s - loss: 0.0430 - accuracy: 0.9870
92/1688 [>.............................] - ETA: 2s - loss: 0.0425 - accuracy: 0.9869
122/1688 [=>............................] - ETA: 2s - loss: 0.0423 - accuracy: 0.9869
152/1688 [=>............................] - ETA: 2s - loss: 0.0436 - accuracy: 0.9866
181/1688 [==>...........................] - ETA: 2s - loss: 0.0444 - accuracy: 0.9866
210/1688 [==>...........................] - ETA: 2s - loss: 0.0448 - accuracy: 0.9867
240/1688 [===>..........................] - ETA: 2s - loss: 0.0449 - accuracy: 0.9868
270/1688 [===>..........................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9868
299/1688 [====>.........................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9868
328/1688 [====>.........................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9868
358/1688 [=====>........................] - ETA: 2s - loss: 0.0451 - accuracy: 0.9868
387/1688 [=====>........................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9867
416/1688 [======>.......................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9867
445/1688 [======>.......................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9866
475/1688 [=======>......................] - ETA: 2s - loss: 0.0454 - accuracy: 0.9866
504/1688 [=======>......................] - ETA: 2s - loss: 0.0455 - accuracy: 0.9866
533/1688 [========>.....................] - ETA: 1s - loss: 0.0455 - accuracy: 0.9865
563/1688 [=========>....................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9865
593/1688 [=========>....................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9865
623/1688 [==========>...................] - ETA: 1s - loss: 0.0458 - accuracy: 0.9865
652/1688 [==========>...................] - ETA: 1s - loss: 0.0459 - accuracy: 0.9864
682/1688 [===========>..................] - ETA: 1s - loss: 0.0460 - accuracy: 0.9864
711/1688 [===========>..................] - ETA: 1s - loss: 0.0462 - accuracy: 0.9864
741/1688 [============>.................] - ETA: 1s - loss: 0.0463 - accuracy: 0.9863
771/1688 [============>.................] - ETA: 1s - loss: 0.0464 - accuracy: 0.9863
801/1688 [=============>................] - ETA: 1s - loss: 0.0465 - accuracy: 0.9863
831/1688 [=============>................] - ETA: 1s - loss: 0.0465 - accuracy: 0.9862
860/1688 [==============>...............] - ETA: 1s - loss: 0.0466 - accuracy: 0.9862
890/1688 [==============>...............] - ETA: 1s - loss: 0.0467 - accuracy: 0.9862
921/1688 [===============>..............] - ETA: 1s - loss: 0.0468 - accuracy: 0.9861
951/1688 [===============>..............] - ETA: 1s - loss: 0.0468 - accuracy: 0.9861
981/1688 [================>.............] - ETA: 1s - loss: 0.0469 - accuracy: 0.9861
1011/1688 [================>.............] - ETA: 1s - loss: 0.0469 - accuracy: 0.9861
1040/1688 [=================>............] - ETA: 1s - loss: 0.0470 - accuracy: 0.9860
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0470 - accuracy: 0.9860
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0470 - accuracy: 0.9860
1129/1688 [===================>..........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1159/1688 [===================>..........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1189/1688 [====================>.........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1219/1688 [====================>.........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1248/1688 [=====================>........] - ETA: 0s - loss: 0.0471 - accuracy: 0.9859
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0472 - accuracy: 0.9858
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0472 - accuracy: 0.9858
1338/1688 [======================>.......] - ETA: 0s - loss: 0.0473 - accuracy: 0.9858
1368/1688 [=======================>......] - ETA: 0s - loss: 0.0473 - accuracy: 0.9858
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0473 - accuracy: 0.9858
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0473 - accuracy: 0.9857
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0473 - accuracy: 0.9857
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0474 - accuracy: 0.9857
1637/1688 [============================>.] - ETA: 0s - loss: 0.0474 - accuracy: 0.9856
1667/1688 [============================>.] - ETA: 0s - loss: 0.0474 - accuracy: 0.9856
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0474 - accuracy: 0.9856 - val_loss: 0.0561 - val_accuracy: 0.9835
Epoch 3/10
1/1688 [..............................] - ETA: 3s - loss: 3.8257e-04 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9933
61/1688 [>.............................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9923
91/1688 [>.............................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9923
120/1688 [=>............................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9925
150/1688 [=>............................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9927
180/1688 [==>...........................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9928
210/1688 [==>...........................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9927
239/1688 [===>..........................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9926
269/1688 [===>..........................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9925
299/1688 [====>.........................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9923
328/1688 [====>.........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9922
358/1688 [=====>........................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9921
388/1688 [=====>........................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9920
417/1688 [======>.......................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9919
447/1688 [======>.......................] - ETA: 2s - loss: 0.0251 - accuracy: 0.9918
477/1688 [=======>......................] - ETA: 2s - loss: 0.0253 - accuracy: 0.9918
507/1688 [========>.....................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9917
537/1688 [========>.....................] - ETA: 1s - loss: 0.0256 - accuracy: 0.9917
567/1688 [=========>....................] - ETA: 1s - loss: 0.0257 - accuracy: 0.9916
597/1688 [=========>....................] - ETA: 1s - loss: 0.0258 - accuracy: 0.9916
626/1688 [==========>...................] - ETA: 1s - loss: 0.0260 - accuracy: 0.9915
656/1688 [==========>...................] - ETA: 1s - loss: 0.0261 - accuracy: 0.9915
686/1688 [===========>..................] - ETA: 1s - loss: 0.0263 - accuracy: 0.9915
716/1688 [===========>..................] - ETA: 1s - loss: 0.0264 - accuracy: 0.9914
745/1688 [============>.................] - ETA: 1s - loss: 0.0266 - accuracy: 0.9914
774/1688 [============>.................] - ETA: 1s - loss: 0.0267 - accuracy: 0.9914
804/1688 [=============>................] - ETA: 1s - loss: 0.0269 - accuracy: 0.9913
833/1688 [=============>................] - ETA: 1s - loss: 0.0270 - accuracy: 0.9913
863/1688 [==============>...............] - ETA: 1s - loss: 0.0272 - accuracy: 0.9912
893/1688 [==============>...............] - ETA: 1s - loss: 0.0273 - accuracy: 0.9912
923/1688 [===============>..............] - ETA: 1s - loss: 0.0275 - accuracy: 0.9911
953/1688 [===============>..............] - ETA: 1s - loss: 0.0276 - accuracy: 0.9911
982/1688 [================>.............] - ETA: 1s - loss: 0.0277 - accuracy: 0.9911
1011/1688 [================>.............] - ETA: 1s - loss: 0.0278 - accuracy: 0.9911
1041/1688 [=================>............] - ETA: 1s - loss: 0.0279 - accuracy: 0.9910
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0280 - accuracy: 0.9910
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0281 - accuracy: 0.9910
1130/1688 [===================>..........] - ETA: 0s - loss: 0.0282 - accuracy: 0.9909
1160/1688 [===================>..........] - ETA: 0s - loss: 0.0283 - accuracy: 0.9909
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0284 - accuracy: 0.9909
1220/1688 [====================>.........] - ETA: 0s - loss: 0.0285 - accuracy: 0.9908
1250/1688 [=====================>........] - ETA: 0s - loss: 0.0286 - accuracy: 0.9908
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0287 - accuracy: 0.9908
1310/1688 [======================>.......] - ETA: 0s - loss: 0.0287 - accuracy: 0.9908
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0288 - accuracy: 0.9907
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9907
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0290 - accuracy: 0.9907
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0290 - accuracy: 0.9907
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0291 - accuracy: 0.9907
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0292 - accuracy: 0.9906
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0292 - accuracy: 0.9906
1546/1688 [==========================>...] - ETA: 0s - loss: 0.0293 - accuracy: 0.9906
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0293 - accuracy: 0.9906
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0294 - accuracy: 0.9905
1635/1688 [============================>.] - ETA: 0s - loss: 0.0295 - accuracy: 0.9905
1665/1688 [============================>.] - ETA: 0s - loss: 0.0295 - accuracy: 0.9905
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0296 - accuracy: 0.9905 - val_loss: 0.0468 - val_accuracy: 0.9863
Epoch 4/10
1/1688 [..............................] - ETA: 2s - loss: 0.0101 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9908
61/1688 [>.............................] - ETA: 2s - loss: 0.0284 - accuracy: 0.9896
91/1688 [>.............................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9900
121/1688 [=>............................] - ETA: 2s - loss: 0.0262 - accuracy: 0.9905
151/1688 [=>............................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9908
180/1688 [==>...........................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9910
209/1688 [==>...........................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9912
239/1688 [===>..........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9913
269/1688 [===>..........................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9914
298/1688 [====>.........................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9915
328/1688 [====>.........................] - ETA: 2s - loss: 0.0236 - accuracy: 0.9916
358/1688 [=====>........................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9917
388/1688 [=====>........................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9918
418/1688 [======>.......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9918
447/1688 [======>.......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9919
476/1688 [=======>......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9919
505/1688 [=======>......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9920
535/1688 [========>.....................] - ETA: 1s - loss: 0.0233 - accuracy: 0.9920
564/1688 [=========>....................] - ETA: 1s - loss: 0.0233 - accuracy: 0.9921
593/1688 [=========>....................] - ETA: 1s - loss: 0.0232 - accuracy: 0.9921
622/1688 [==========>...................] - ETA: 1s - loss: 0.0231 - accuracy: 0.9922
651/1688 [==========>...................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9922
680/1688 [===========>..................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9923
709/1688 [===========>..................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9923
739/1688 [============>.................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9923
768/1688 [============>.................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9924
798/1688 [=============>................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9924
827/1688 [=============>................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9924
857/1688 [==============>...............] - ETA: 1s - loss: 0.0230 - accuracy: 0.9924
888/1688 [==============>...............] - ETA: 1s - loss: 0.0230 - accuracy: 0.9924
917/1688 [===============>..............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9924
947/1688 [===============>..............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9924
977/1688 [================>.............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9924
1007/1688 [================>.............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9924
1037/1688 [=================>............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9923
1067/1688 [=================>............] - ETA: 1s - loss: 0.0234 - accuracy: 0.9923
1097/1688 [==================>...........] - ETA: 1s - loss: 0.0234 - accuracy: 0.9923
1127/1688 [===================>..........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9923
1156/1688 [===================>..........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9923
1186/1688 [====================>.........] - ETA: 0s - loss: 0.0235 - accuracy: 0.9923
1216/1688 [====================>.........] - ETA: 0s - loss: 0.0236 - accuracy: 0.9923
1245/1688 [=====================>........] - ETA: 0s - loss: 0.0236 - accuracy: 0.9923
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0236 - accuracy: 0.9922
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0237 - accuracy: 0.9922
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9922
1452/1688 [========================>.....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9922
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9922
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9922
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0239 - accuracy: 0.9922
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0239 - accuracy: 0.9922
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0239 - accuracy: 0.9922
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0240 - accuracy: 0.9922
1659/1688 [============================>.] - ETA: 0s - loss: 0.0240 - accuracy: 0.9921
1688/1688 [==============================] - ETA: 0s - loss: 0.0241 - accuracy: 0.9921
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0241 - accuracy: 0.9921 - val_loss: 0.0664 - val_accuracy: 0.9842
Epoch 5/10
1/1688 [..............................] - ETA: 2s - loss: 0.0053 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9965
60/1688 [>.............................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9964
90/1688 [>.............................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9963
119/1688 [=>............................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9963
149/1688 [=>............................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9964
179/1688 [==>...........................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9964
209/1688 [==>...........................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9962
239/1688 [===>..........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9961
269/1688 [===>..........................] - ETA: 2s - loss: 0.0121 - accuracy: 0.9959
299/1688 [====>.........................] - ETA: 2s - loss: 0.0126 - accuracy: 0.9957
329/1688 [====>.........................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9955
358/1688 [=====>........................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9954
388/1688 [=====>........................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9952
417/1688 [======>.......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
447/1688 [======>.......................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9950
477/1688 [=======>......................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9949
507/1688 [========>.....................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9947
537/1688 [========>.....................] - ETA: 1s - loss: 0.0154 - accuracy: 0.9946
567/1688 [=========>....................] - ETA: 1s - loss: 0.0157 - accuracy: 0.9946
597/1688 [=========>....................] - ETA: 1s - loss: 0.0159 - accuracy: 0.9945
627/1688 [==========>...................] - ETA: 1s - loss: 0.0161 - accuracy: 0.9944
656/1688 [==========>...................] - ETA: 1s - loss: 0.0163 - accuracy: 0.9944
685/1688 [===========>..................] - ETA: 1s - loss: 0.0164 - accuracy: 0.9943
715/1688 [===========>..................] - ETA: 1s - loss: 0.0165 - accuracy: 0.9943
744/1688 [============>.................] - ETA: 1s - loss: 0.0166 - accuracy: 0.9943
774/1688 [============>.................] - ETA: 1s - loss: 0.0167 - accuracy: 0.9943
804/1688 [=============>................] - ETA: 1s - loss: 0.0168 - accuracy: 0.9942
834/1688 [=============>................] - ETA: 1s - loss: 0.0169 - accuracy: 0.9942
865/1688 [==============>...............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9942
894/1688 [==============>...............] - ETA: 1s - loss: 0.0171 - accuracy: 0.9941
923/1688 [===============>..............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9941
952/1688 [===============>..............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9941
981/1688 [================>.............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9941
1010/1688 [================>.............] - ETA: 1s - loss: 0.0175 - accuracy: 0.9940
1040/1688 [=================>............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9940
1069/1688 [=================>............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9940
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0177 - accuracy: 0.9940
1129/1688 [===================>..........] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1159/1688 [===================>..........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1189/1688 [====================>.........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1219/1688 [====================>.........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1248/1688 [=====================>........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1338/1688 [======================>.......] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1397/1688 [=======================>......] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0181 - accuracy: 0.9938
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0181 - accuracy: 0.9938
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0181 - accuracy: 0.9938
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1633/1688 [============================>.] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1663/1688 [============================>.] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0183 - accuracy: 0.9938 - val_loss: 0.0430 - val_accuracy: 0.9890
Epoch 6/10
1/1688 [..............................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9688
32/1688 [..............................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9953
62/1688 [>.............................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9956
92/1688 [>.............................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9956
121/1688 [=>............................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9958
151/1688 [=>............................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9960
180/1688 [==>...........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9961
209/1688 [==>...........................] - ETA: 2s - loss: 0.0114 - accuracy: 0.9962
239/1688 [===>..........................] - ETA: 2s - loss: 0.0114 - accuracy: 0.9963
269/1688 [===>..........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9963
298/1688 [====>.........................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9963
328/1688 [====>.........................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9963
358/1688 [=====>........................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9963
388/1688 [=====>........................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9963
417/1688 [======>.......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9963
447/1688 [======>.......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9963
469/1688 [=======>......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9963
499/1688 [=======>......................] - ETA: 2s - loss: 0.0120 - accuracy: 0.9963
529/1688 [========>.....................] - ETA: 2s - loss: 0.0121 - accuracy: 0.9963
558/1688 [========>.....................] - ETA: 1s - loss: 0.0122 - accuracy: 0.9962
587/1688 [=========>....................] - ETA: 1s - loss: 0.0122 - accuracy: 0.9962
617/1688 [=========>....................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
647/1688 [==========>...................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
677/1688 [===========>..................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
707/1688 [===========>..................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
736/1688 [============>.................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
766/1688 [============>.................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9962
796/1688 [=============>................] - ETA: 1s - loss: 0.0124 - accuracy: 0.9962
826/1688 [=============>................] - ETA: 1s - loss: 0.0124 - accuracy: 0.9962
856/1688 [==============>...............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
886/1688 [==============>...............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
915/1688 [===============>..............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
945/1688 [===============>..............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
974/1688 [================>.............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
1004/1688 [================>.............] - ETA: 1s - loss: 0.0124 - accuracy: 0.9961
1034/1688 [=================>............] - ETA: 1s - loss: 0.0125 - accuracy: 0.9961
1064/1688 [=================>............] - ETA: 1s - loss: 0.0125 - accuracy: 0.9961
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0125 - accuracy: 0.9960
1125/1688 [==================>...........] - ETA: 0s - loss: 0.0125 - accuracy: 0.9960
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0125 - accuracy: 0.9960
1185/1688 [====================>.........] - ETA: 0s - loss: 0.0126 - accuracy: 0.9960
1214/1688 [====================>.........] - ETA: 0s - loss: 0.0126 - accuracy: 0.9960
1244/1688 [=====================>........] - ETA: 0s - loss: 0.0126 - accuracy: 0.9960
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0127 - accuracy: 0.9959
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0127 - accuracy: 0.9959
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0127 - accuracy: 0.9959
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0128 - accuracy: 0.9959
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0128 - accuracy: 0.9959
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0129 - accuracy: 0.9959
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0129 - accuracy: 0.9958
1483/1688 [=========================>....] - ETA: 0s - loss: 0.0129 - accuracy: 0.9958
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0130 - accuracy: 0.9958
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0130 - accuracy: 0.9958
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0131 - accuracy: 0.9958
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0131 - accuracy: 0.9958
1632/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9957
1661/1688 [============================>.] - ETA: 0s - loss: 0.0132 - accuracy: 0.9957
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0132 - accuracy: 0.9957 - val_loss: 0.0523 - val_accuracy: 0.9873
Epoch 7/10
1/1688 [..............................] - ETA: 2s - loss: 0.0027 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0026 - accuracy: 0.9995
64/1688 [>.............................] - ETA: 2s - loss: 0.0033 - accuracy: 0.9991
94/1688 [>.............................] - ETA: 2s - loss: 0.0039 - accuracy: 0.9987
124/1688 [=>............................] - ETA: 2s - loss: 0.0047 - accuracy: 0.9983
154/1688 [=>............................] - ETA: 2s - loss: 0.0052 - accuracy: 0.9981
184/1688 [==>...........................] - ETA: 2s - loss: 0.0056 - accuracy: 0.9979
214/1688 [==>...........................] - ETA: 2s - loss: 0.0058 - accuracy: 0.9977
244/1688 [===>..........................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9976
274/1688 [===>..........................] - ETA: 2s - loss: 0.0066 - accuracy: 0.9974
304/1688 [====>.........................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9971
333/1688 [====>.........................] - ETA: 2s - loss: 0.0075 - accuracy: 0.9970
362/1688 [=====>........................] - ETA: 2s - loss: 0.0078 - accuracy: 0.9969
392/1688 [=====>........................] - ETA: 2s - loss: 0.0080 - accuracy: 0.9968
422/1688 [======>.......................] - ETA: 2s - loss: 0.0082 - accuracy: 0.9967
451/1688 [=======>......................] - ETA: 2s - loss: 0.0083 - accuracy: 0.9967
480/1688 [=======>......................] - ETA: 2s - loss: 0.0084 - accuracy: 0.9966
509/1688 [========>.....................] - ETA: 2s - loss: 0.0085 - accuracy: 0.9966
539/1688 [========>.....................] - ETA: 1s - loss: 0.0086 - accuracy: 0.9966
569/1688 [=========>....................] - ETA: 1s - loss: 0.0087 - accuracy: 0.9965
599/1688 [=========>....................] - ETA: 1s - loss: 0.0087 - accuracy: 0.9965
628/1688 [==========>...................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9965
658/1688 [==========>...................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9965
687/1688 [===========>..................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9965
717/1688 [===========>..................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9965
747/1688 [============>.................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9965
776/1688 [============>.................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9965
806/1688 [=============>................] - ETA: 1s - loss: 0.0090 - accuracy: 0.9965
836/1688 [=============>................] - ETA: 1s - loss: 0.0091 - accuracy: 0.9965
865/1688 [==============>...............] - ETA: 1s - loss: 0.0091 - accuracy: 0.9965
894/1688 [==============>...............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9965
924/1688 [===============>..............] - ETA: 1s - loss: 0.0092 - accuracy: 0.9965
953/1688 [===============>..............] - ETA: 1s - loss: 0.0093 - accuracy: 0.9964
983/1688 [================>.............] - ETA: 1s - loss: 0.0093 - accuracy: 0.9964
1013/1688 [=================>............] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
1043/1688 [=================>............] - ETA: 1s - loss: 0.0094 - accuracy: 0.9964
1073/1688 [==================>...........] - ETA: 1s - loss: 0.0095 - accuracy: 0.9964
1103/1688 [==================>...........] - ETA: 1s - loss: 0.0095 - accuracy: 0.9964
1133/1688 [===================>..........] - ETA: 0s - loss: 0.0096 - accuracy: 0.9964
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0096 - accuracy: 0.9963
1193/1688 [====================>.........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9963
1222/1688 [====================>.........] - ETA: 0s - loss: 0.0097 - accuracy: 0.9963
1252/1688 [=====================>........] - ETA: 0s - loss: 0.0098 - accuracy: 0.9963
1281/1688 [=====================>........] - ETA: 0s - loss: 0.0098 - accuracy: 0.9963
1310/1688 [======================>.......] - ETA: 0s - loss: 0.0099 - accuracy: 0.9963
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0099 - accuracy: 0.9963
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0100 - accuracy: 0.9963
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0100 - accuracy: 0.9963
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0101 - accuracy: 0.9962
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0101 - accuracy: 0.9962
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0102 - accuracy: 0.9962
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0102 - accuracy: 0.9962
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0103 - accuracy: 0.9962
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9962
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0104 - accuracy: 0.9962
1637/1688 [============================>.] - ETA: 0s - loss: 0.0104 - accuracy: 0.9962
1667/1688 [============================>.] - ETA: 0s - loss: 0.0105 - accuracy: 0.9961
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0105 - accuracy: 0.9961 - val_loss: 0.0502 - val_accuracy: 0.9883
Epoch 8/10
1/1688 [..............................] - ETA: 3s - loss: 2.6362e-04 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9969
61/1688 [>.............................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9968
91/1688 [>.............................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9964
120/1688 [=>............................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9961
150/1688 [=>............................] - ETA: 2s - loss: 0.0120 - accuracy: 0.9961
180/1688 [==>...........................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9962
210/1688 [==>...........................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9963
239/1688 [===>..........................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9964
268/1688 [===>..........................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9965
297/1688 [====>.........................] - ETA: 2s - loss: 0.0104 - accuracy: 0.9966
326/1688 [====>.........................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9967
355/1688 [=====>........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9967
385/1688 [=====>........................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9968
415/1688 [======>.......................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9969
446/1688 [======>.......................] - ETA: 2s - loss: 0.0093 - accuracy: 0.9970
475/1688 [=======>......................] - ETA: 2s - loss: 0.0091 - accuracy: 0.9970
504/1688 [=======>......................] - ETA: 2s - loss: 0.0090 - accuracy: 0.9971
534/1688 [========>.....................] - ETA: 1s - loss: 0.0089 - accuracy: 0.9971
564/1688 [=========>....................] - ETA: 1s - loss: 0.0088 - accuracy: 0.9972
594/1688 [=========>....................] - ETA: 1s - loss: 0.0087 - accuracy: 0.9972
624/1688 [==========>...................] - ETA: 1s - loss: 0.0086 - accuracy: 0.9972
653/1688 [==========>...................] - ETA: 1s - loss: 0.0085 - accuracy: 0.9972
683/1688 [===========>..................] - ETA: 1s - loss: 0.0084 - accuracy: 0.9973
713/1688 [===========>..................] - ETA: 1s - loss: 0.0084 - accuracy: 0.9973
743/1688 [============>.................] - ETA: 1s - loss: 0.0083 - accuracy: 0.9973
772/1688 [============>.................] - ETA: 1s - loss: 0.0083 - accuracy: 0.9973
801/1688 [=============>................] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
830/1688 [=============>................] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
860/1688 [==============>...............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
890/1688 [==============>...............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
920/1688 [===============>..............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
950/1688 [===============>..............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
980/1688 [================>.............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1009/1688 [================>.............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1039/1688 [=================>............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1069/1688 [=================>............] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1098/1688 [==================>...........] - ETA: 1s - loss: 0.0082 - accuracy: 0.9973
1128/1688 [===================>..........] - ETA: 0s - loss: 0.0082 - accuracy: 0.9973
1158/1688 [===================>..........] - ETA: 0s - loss: 0.0082 - accuracy: 0.9973
1188/1688 [====================>.........] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1217/1688 [====================>.........] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0083 - accuracy: 0.9973
1276/1688 [=====================>........] - ETA: 0s - loss: 0.0084 - accuracy: 0.9973
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0084 - accuracy: 0.9972
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0084 - accuracy: 0.9972
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0085 - accuracy: 0.9972
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0085 - accuracy: 0.9972
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1454/1688 [========================>.....] - ETA: 0s - loss: 0.0086 - accuracy: 0.9972
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0087 - accuracy: 0.9972
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0087 - accuracy: 0.9971
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0087 - accuracy: 0.9971
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0088 - accuracy: 0.9971
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0088 - accuracy: 0.9971
1632/1688 [============================>.] - ETA: 0s - loss: 0.0089 - accuracy: 0.9971
1662/1688 [============================>.] - ETA: 0s - loss: 0.0089 - accuracy: 0.9971
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0089 - accuracy: 0.9970 - val_loss: 0.0454 - val_accuracy: 0.9905
Epoch 9/10
1/1688 [..............................] - ETA: 2s - loss: 3.8184e-06 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0052 - accuracy: 0.9985
61/1688 [>.............................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9976
91/1688 [>.............................] - ETA: 2s - loss: 0.0073 - accuracy: 0.9972
120/1688 [=>............................] - ETA: 2s - loss: 0.0073 - accuracy: 0.9971
150/1688 [=>............................] - ETA: 2s - loss: 0.0072 - accuracy: 0.9971
179/1688 [==>...........................] - ETA: 2s - loss: 0.0073 - accuracy: 0.9971
209/1688 [==>...........................] - ETA: 2s - loss: 0.0072 - accuracy: 0.9972
238/1688 [===>..........................] - ETA: 2s - loss: 0.0071 - accuracy: 0.9973
268/1688 [===>..........................] - ETA: 2s - loss: 0.0070 - accuracy: 0.9973
298/1688 [====>.........................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9974
327/1688 [====>.........................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9974
357/1688 [=====>........................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9975
386/1688 [=====>........................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9975
416/1688 [======>.......................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9975
446/1688 [======>.......................] - ETA: 2s - loss: 0.0068 - accuracy: 0.9975
476/1688 [=======>......................] - ETA: 2s - loss: 0.0069 - accuracy: 0.9974
505/1688 [=======>......................] - ETA: 2s - loss: 0.0070 - accuracy: 0.9974
534/1688 [========>.....................] - ETA: 1s - loss: 0.0070 - accuracy: 0.9974
563/1688 [=========>....................] - ETA: 1s - loss: 0.0071 - accuracy: 0.9974
592/1688 [=========>....................] - ETA: 1s - loss: 0.0071 - accuracy: 0.9973
622/1688 [==========>...................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
652/1688 [==========>...................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
682/1688 [===========>..................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
712/1688 [===========>..................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
741/1688 [============>.................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
770/1688 [============>.................] - ETA: 1s - loss: 0.0072 - accuracy: 0.9973
799/1688 [=============>................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9973
829/1688 [=============>................] - ETA: 1s - loss: 0.0073 - accuracy: 0.9973
858/1688 [==============>...............] - ETA: 1s - loss: 0.0073 - accuracy: 0.9973
888/1688 [==============>...............] - ETA: 1s - loss: 0.0073 - accuracy: 0.9972
918/1688 [===============>..............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
947/1688 [===============>..............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
977/1688 [================>.............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
1006/1688 [================>.............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
1035/1688 [=================>............] - ETA: 1s - loss: 0.0074 - accuracy: 0.9972
1065/1688 [=================>............] - ETA: 1s - loss: 0.0075 - accuracy: 0.9972
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0075 - accuracy: 0.9972
1124/1688 [==================>...........] - ETA: 0s - loss: 0.0075 - accuracy: 0.9972
1154/1688 [===================>..........] - ETA: 0s - loss: 0.0076 - accuracy: 0.9972
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0076 - accuracy: 0.9971
1214/1688 [====================>.........] - ETA: 0s - loss: 0.0076 - accuracy: 0.9971
1244/1688 [=====================>........] - ETA: 0s - loss: 0.0077 - accuracy: 0.9971
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0077 - accuracy: 0.9971
1304/1688 [======================>.......] - ETA: 0s - loss: 0.0077 - accuracy: 0.9971
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0078 - accuracy: 0.9971
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0078 - accuracy: 0.9971
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0078 - accuracy: 0.9971
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0079 - accuracy: 0.9971
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0079 - accuracy: 0.9971
1483/1688 [=========================>....] - ETA: 0s - loss: 0.0080 - accuracy: 0.9971
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0080 - accuracy: 0.9970
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0081 - accuracy: 0.9970
1573/1688 [==========================>...] - ETA: 0s - loss: 0.0081 - accuracy: 0.9970
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0081 - accuracy: 0.9970
1632/1688 [============================>.] - ETA: 0s - loss: 0.0082 - accuracy: 0.9970
1662/1688 [============================>.] - ETA: 0s - loss: 0.0082 - accuracy: 0.9970
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0082 - accuracy: 0.9970 - val_loss: 0.0587 - val_accuracy: 0.9882
Epoch 10/10
1/1688 [..............................] - ETA: 3s - loss: 0.0014 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0049 - accuracy: 0.9980
61/1688 [>.............................] - ETA: 2s - loss: 0.0052 - accuracy: 0.9975
91/1688 [>.............................] - ETA: 2s - loss: 0.0049 - accuracy: 0.9976
121/1688 [=>............................] - ETA: 2s - loss: 0.0046 - accuracy: 0.9977
151/1688 [=>............................] - ETA: 2s - loss: 0.0046 - accuracy: 0.9978
181/1688 [==>...........................] - ETA: 2s - loss: 0.0045 - accuracy: 0.9979
211/1688 [==>...........................] - ETA: 2s - loss: 0.0044 - accuracy: 0.9979
240/1688 [===>..........................] - ETA: 2s - loss: 0.0043 - accuracy: 0.9980
270/1688 [===>..........................] - ETA: 2s - loss: 0.0043 - accuracy: 0.9981
300/1688 [====>.........................] - ETA: 2s - loss: 0.0042 - accuracy: 0.9981
329/1688 [====>.........................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9982
359/1688 [=====>........................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9982
389/1688 [=====>........................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
418/1688 [======>.......................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
448/1688 [======>.......................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
477/1688 [=======>......................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
507/1688 [========>.....................] - ETA: 2s - loss: 0.0041 - accuracy: 0.9983
537/1688 [========>.....................] - ETA: 1s - loss: 0.0041 - accuracy: 0.9983
567/1688 [=========>....................] - ETA: 1s - loss: 0.0041 - accuracy: 0.9983
597/1688 [=========>....................] - ETA: 1s - loss: 0.0041 - accuracy: 0.9983
626/1688 [==========>...................] - ETA: 1s - loss: 0.0041 - accuracy: 0.9983
656/1688 [==========>...................] - ETA: 1s - loss: 0.0042 - accuracy: 0.9983
686/1688 [===========>..................] - ETA: 1s - loss: 0.0042 - accuracy: 0.9983
716/1688 [===========>..................] - ETA: 1s - loss: 0.0043 - accuracy: 0.9983
746/1688 [============>.................] - ETA: 1s - loss: 0.0043 - accuracy: 0.9983
776/1688 [============>.................] - ETA: 1s - loss: 0.0044 - accuracy: 0.9983
806/1688 [=============>................] - ETA: 1s - loss: 0.0044 - accuracy: 0.9983
836/1688 [=============>................] - ETA: 1s - loss: 0.0045 - accuracy: 0.9982
866/1688 [==============>...............] - ETA: 1s - loss: 0.0045 - accuracy: 0.9982
895/1688 [==============>...............] - ETA: 1s - loss: 0.0046 - accuracy: 0.9982
924/1688 [===============>..............] - ETA: 1s - loss: 0.0046 - accuracy: 0.9982
953/1688 [===============>..............] - ETA: 1s - loss: 0.0047 - accuracy: 0.9982
982/1688 [================>.............] - ETA: 1s - loss: 0.0047 - accuracy: 0.9982
1012/1688 [================>.............] - ETA: 1s - loss: 0.0048 - accuracy: 0.9982
1042/1688 [=================>............] - ETA: 1s - loss: 0.0048 - accuracy: 0.9982
1072/1688 [==================>...........] - ETA: 1s - loss: 0.0049 - accuracy: 0.9982
1102/1688 [==================>...........] - ETA: 1s - loss: 0.0049 - accuracy: 0.9981
1132/1688 [===================>..........] - ETA: 0s - loss: 0.0050 - accuracy: 0.9981
1161/1688 [===================>..........] - ETA: 0s - loss: 0.0050 - accuracy: 0.9981
1191/1688 [====================>.........] - ETA: 0s - loss: 0.0051 - accuracy: 0.9981
1221/1688 [====================>.........] - ETA: 0s - loss: 0.0052 - accuracy: 0.9981
1250/1688 [=====================>........] - ETA: 0s - loss: 0.0052 - accuracy: 0.9980
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0053 - accuracy: 0.9980
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0054 - accuracy: 0.9980
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0054 - accuracy: 0.9980
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0055 - accuracy: 0.9980
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0055 - accuracy: 0.9979
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0056 - accuracy: 0.9979
1458/1688 [========================>.....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9979
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0057 - accuracy: 0.9979
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0058 - accuracy: 0.9979
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9979
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9979
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9979
1635/1688 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9978
1665/1688 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9978
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0060 - accuracy: 0.9978 - val_loss: 0.0483 - val_accuracy: 0.9902
Test score: 0.052414294332265854
Test accuracy: 0.9876999855041504
4. Model quantization¶
We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.
For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).
The quantized model is a Keras model where the neural layers (Conv2D, Dense)
and the ReLU layers are replaced with custom CNN2SNN quantized layers
(QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions
can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function folds the batch normalization layers into
the corresponding neural layer. The new weights are computed
according to this folding operation.
from cnn2snn import quantize
model_quantized = quantize(model_keras,
input_weight_quantization=8,
weight_quantization=4,
activ_quantization=4)
model_quantized.summary()
Out:
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (QuantizedConv2D) (None, 26, 26, 32) 320
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 13, 13, 32) 0
_________________________________________________________________
re_lu (ActivationDiscreteRel (None, 13, 13, 32) 0
_________________________________________________________________
separable_conv2d (QuantizedS (None, 13, 13, 64) 2400
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64) 0
_________________________________________________________________
re_lu_1 (ActivationDiscreteR (None, 7, 7, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 3136) 0
_________________________________________________________________
dense (QuantizedDense) (None, 10) 31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________
Check the quantized model accuracy.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])
Out:
Test accuracy after 8-4-4 quantization: 0.9846000075340271
Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.
Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.
Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.
from cnn2snn import quantize_layer
model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])
# To recover the original model accuracy, a quantization-aware training phase
# is required.
Out:
Test accuracy after low bitwidth quantization: 0.7494999766349792
5. Model fine tuning (quantization-aware training)¶
This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.
Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])
Out:
Epoch 1/5
1/1688 [..............................] - ETA: 18:51 - loss: 0.8136 - accuracy: 0.7188
22/1688 [..............................] - ETA: 4s - loss: 0.5149 - accuracy: 0.8438
43/1688 [..............................] - ETA: 4s - loss: 0.3889 - accuracy: 0.8844
63/1688 [>.............................] - ETA: 4s - loss: 0.3238 - accuracy: 0.9038
84/1688 [>.............................] - ETA: 3s - loss: 0.2795 - accuracy: 0.9200
105/1688 [>.............................] - ETA: 3s - loss: 0.2472 - accuracy: 0.9301
126/1688 [=>............................] - ETA: 3s - loss: 0.2279 - accuracy: 0.9355
146/1688 [=>............................] - ETA: 3s - loss: 0.2143 - accuracy: 0.9390
167/1688 [=>............................] - ETA: 3s - loss: 0.1976 - accuracy: 0.9435
188/1688 [==>...........................] - ETA: 3s - loss: 0.1830 - accuracy: 0.9475
209/1688 [==>...........................] - ETA: 3s - loss: 0.1706 - accuracy: 0.9508
230/1688 [===>..........................] - ETA: 3s - loss: 0.1609 - accuracy: 0.9538
250/1688 [===>..........................] - ETA: 3s - loss: 0.1524 - accuracy: 0.9559
271/1688 [===>..........................] - ETA: 3s - loss: 0.1450 - accuracy: 0.9580
291/1688 [====>.........................] - ETA: 3s - loss: 0.1386 - accuracy: 0.9597
311/1688 [====>.........................] - ETA: 3s - loss: 0.1334 - accuracy: 0.9608
331/1688 [====>.........................] - ETA: 3s - loss: 0.1296 - accuracy: 0.9619
352/1688 [=====>........................] - ETA: 3s - loss: 0.1253 - accuracy: 0.9632
372/1688 [=====>........................] - ETA: 3s - loss: 0.1213 - accuracy: 0.9644
393/1688 [=====>........................] - ETA: 3s - loss: 0.1178 - accuracy: 0.9652
413/1688 [======>.......................] - ETA: 3s - loss: 0.1145 - accuracy: 0.9662
433/1688 [======>.......................] - ETA: 3s - loss: 0.1119 - accuracy: 0.9666
453/1688 [=======>......................] - ETA: 3s - loss: 0.1086 - accuracy: 0.9676
473/1688 [=======>......................] - ETA: 3s - loss: 0.1068 - accuracy: 0.9680
493/1688 [=======>......................] - ETA: 2s - loss: 0.1044 - accuracy: 0.9688
513/1688 [========>.....................] - ETA: 2s - loss: 0.1017 - accuracy: 0.9696
533/1688 [========>.....................] - ETA: 2s - loss: 0.0998 - accuracy: 0.9702
553/1688 [========>.....................] - ETA: 2s - loss: 0.0979 - accuracy: 0.9706
573/1688 [=========>....................] - ETA: 2s - loss: 0.0960 - accuracy: 0.9711
594/1688 [=========>....................] - ETA: 2s - loss: 0.0939 - accuracy: 0.9716
614/1688 [=========>....................] - ETA: 2s - loss: 0.0926 - accuracy: 0.9719
634/1688 [==========>...................] - ETA: 2s - loss: 0.0924 - accuracy: 0.9717
654/1688 [==========>...................] - ETA: 2s - loss: 0.0912 - accuracy: 0.9721
675/1688 [==========>...................] - ETA: 2s - loss: 0.0896 - accuracy: 0.9726
695/1688 [===========>..................] - ETA: 2s - loss: 0.0882 - accuracy: 0.9732
715/1688 [===========>..................] - ETA: 2s - loss: 0.0866 - accuracy: 0.9736
736/1688 [============>.................] - ETA: 2s - loss: 0.0856 - accuracy: 0.9738
757/1688 [============>.................] - ETA: 2s - loss: 0.0842 - accuracy: 0.9741
778/1688 [============>.................] - ETA: 2s - loss: 0.0834 - accuracy: 0.9742
798/1688 [=============>................] - ETA: 2s - loss: 0.0821 - accuracy: 0.9746
818/1688 [=============>................] - ETA: 2s - loss: 0.0808 - accuracy: 0.9750
839/1688 [=============>................] - ETA: 2s - loss: 0.0797 - accuracy: 0.9753
859/1688 [==============>...............] - ETA: 2s - loss: 0.0784 - accuracy: 0.9757
879/1688 [==============>...............] - ETA: 2s - loss: 0.0774 - accuracy: 0.9759
899/1688 [==============>...............] - ETA: 1s - loss: 0.0764 - accuracy: 0.9762
920/1688 [===============>..............] - ETA: 1s - loss: 0.0758 - accuracy: 0.9762
941/1688 [===============>..............] - ETA: 1s - loss: 0.0751 - accuracy: 0.9764
961/1688 [================>.............] - ETA: 1s - loss: 0.0742 - accuracy: 0.9767
981/1688 [================>.............] - ETA: 1s - loss: 0.0734 - accuracy: 0.9769
1002/1688 [================>.............] - ETA: 1s - loss: 0.0728 - accuracy: 0.9770
1022/1688 [=================>............] - ETA: 1s - loss: 0.0721 - accuracy: 0.9771
1042/1688 [=================>............] - ETA: 1s - loss: 0.0716 - accuracy: 0.9772
1063/1688 [=================>............] - ETA: 1s - loss: 0.0711 - accuracy: 0.9775
1083/1688 [==================>...........] - ETA: 1s - loss: 0.0705 - accuracy: 0.9776
1103/1688 [==================>...........] - ETA: 1s - loss: 0.0699 - accuracy: 0.9778
1123/1688 [==================>...........] - ETA: 1s - loss: 0.0694 - accuracy: 0.9779
1144/1688 [===================>..........] - ETA: 1s - loss: 0.0691 - accuracy: 0.9780
1164/1688 [===================>..........] - ETA: 1s - loss: 0.0686 - accuracy: 0.9781
1185/1688 [====================>.........] - ETA: 1s - loss: 0.0681 - accuracy: 0.9783
1206/1688 [====================>.........] - ETA: 1s - loss: 0.0677 - accuracy: 0.9784
1226/1688 [====================>.........] - ETA: 1s - loss: 0.0670 - accuracy: 0.9787
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0664 - accuracy: 0.9789
1268/1688 [=====================>........] - ETA: 1s - loss: 0.0660 - accuracy: 0.9790
1288/1688 [=====================>........] - ETA: 1s - loss: 0.0654 - accuracy: 0.9793
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0648 - accuracy: 0.9795
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0642 - accuracy: 0.9797
1350/1688 [======================>.......] - ETA: 0s - loss: 0.0637 - accuracy: 0.9798
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0632 - accuracy: 0.9799
1390/1688 [=======================>......] - ETA: 0s - loss: 0.0626 - accuracy: 0.9801
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0620 - accuracy: 0.9803
1430/1688 [========================>.....] - ETA: 0s - loss: 0.0616 - accuracy: 0.9804
1450/1688 [========================>.....] - ETA: 0s - loss: 0.0610 - accuracy: 0.9805
1470/1688 [=========================>....] - ETA: 0s - loss: 0.0605 - accuracy: 0.9807
1490/1688 [=========================>....] - ETA: 0s - loss: 0.0604 - accuracy: 0.9808
1510/1688 [=========================>....] - ETA: 0s - loss: 0.0598 - accuracy: 0.9809
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0592 - accuracy: 0.9812
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0589 - accuracy: 0.9812
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0586 - accuracy: 0.9813
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0585 - accuracy: 0.9812
1611/1688 [===========================>..] - ETA: 0s - loss: 0.0582 - accuracy: 0.9812
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0578 - accuracy: 0.9814
1651/1688 [============================>.] - ETA: 0s - loss: 0.0576 - accuracy: 0.9814
1671/1688 [============================>.] - ETA: 0s - loss: 0.0578 - accuracy: 0.9814
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0575 - accuracy: 0.9814 - val_loss: 0.0611 - val_accuracy: 0.9843
Epoch 2/5
1/1688 [..............................] - ETA: 4s - loss: 0.0625 - accuracy: 0.9688
22/1688 [..............................] - ETA: 4s - loss: 0.0196 - accuracy: 0.9915
42/1688 [..............................] - ETA: 4s - loss: 0.0253 - accuracy: 0.9896
63/1688 [>.............................] - ETA: 4s - loss: 0.0245 - accuracy: 0.9896
83/1688 [>.............................] - ETA: 4s - loss: 0.0274 - accuracy: 0.9876
103/1688 [>.............................] - ETA: 3s - loss: 0.0278 - accuracy: 0.9882
123/1688 [=>............................] - ETA: 3s - loss: 0.0305 - accuracy: 0.9876
144/1688 [=>............................] - ETA: 3s - loss: 0.0287 - accuracy: 0.9887
164/1688 [=>............................] - ETA: 3s - loss: 0.0275 - accuracy: 0.9897
184/1688 [==>...........................] - ETA: 3s - loss: 0.0280 - accuracy: 0.9895
204/1688 [==>...........................] - ETA: 3s - loss: 0.0272 - accuracy: 0.9899
225/1688 [==>...........................] - ETA: 3s - loss: 0.0299 - accuracy: 0.9893
246/1688 [===>..........................] - ETA: 3s - loss: 0.0291 - accuracy: 0.9896
267/1688 [===>..........................] - ETA: 3s - loss: 0.0297 - accuracy: 0.9892
287/1688 [====>.........................] - ETA: 3s - loss: 0.0297 - accuracy: 0.9892
308/1688 [====>.........................] - ETA: 3s - loss: 0.0290 - accuracy: 0.9893
328/1688 [====>.........................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9896
348/1688 [=====>........................] - ETA: 3s - loss: 0.0293 - accuracy: 0.9893
369/1688 [=====>........................] - ETA: 3s - loss: 0.0289 - accuracy: 0.9896
389/1688 [=====>........................] - ETA: 3s - loss: 0.0284 - accuracy: 0.9899
409/1688 [======>.......................] - ETA: 3s - loss: 0.0289 - accuracy: 0.9895
429/1688 [======>.......................] - ETA: 3s - loss: 0.0287 - accuracy: 0.9895
449/1688 [======>.......................] - ETA: 3s - loss: 0.0289 - accuracy: 0.9894
470/1688 [=======>......................] - ETA: 3s - loss: 0.0291 - accuracy: 0.9894
490/1688 [=======>......................] - ETA: 3s - loss: 0.0298 - accuracy: 0.9893
511/1688 [========>.....................] - ETA: 2s - loss: 0.0297 - accuracy: 0.9894
531/1688 [========>.....................] - ETA: 2s - loss: 0.0295 - accuracy: 0.9895
551/1688 [========>.....................] - ETA: 2s - loss: 0.0291 - accuracy: 0.9897
571/1688 [=========>....................] - ETA: 2s - loss: 0.0298 - accuracy: 0.9895
591/1688 [=========>....................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9894
611/1688 [=========>....................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9893
631/1688 [==========>...................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9893
652/1688 [==========>...................] - ETA: 2s - loss: 0.0309 - accuracy: 0.9894
672/1688 [==========>...................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9894
692/1688 [===========>..................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9893
713/1688 [===========>..................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9893
733/1688 [============>.................] - ETA: 2s - loss: 0.0310 - accuracy: 0.9895
754/1688 [============>.................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9894
774/1688 [============>.................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9892
794/1688 [=============>................] - ETA: 2s - loss: 0.0314 - accuracy: 0.9893
814/1688 [=============>................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9894
834/1688 [=============>................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9896
855/1688 [==============>...............] - ETA: 2s - loss: 0.0314 - accuracy: 0.9895
875/1688 [==============>...............] - ETA: 2s - loss: 0.0313 - accuracy: 0.9895
895/1688 [==============>...............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9895
915/1688 [===============>..............] - ETA: 1s - loss: 0.0310 - accuracy: 0.9896
936/1688 [===============>..............] - ETA: 1s - loss: 0.0306 - accuracy: 0.9897
956/1688 [===============>..............] - ETA: 1s - loss: 0.0304 - accuracy: 0.9897
977/1688 [================>.............] - ETA: 1s - loss: 0.0302 - accuracy: 0.9898
997/1688 [================>.............] - ETA: 1s - loss: 0.0304 - accuracy: 0.9898
1018/1688 [=================>............] - ETA: 1s - loss: 0.0301 - accuracy: 0.9899
1038/1688 [=================>............] - ETA: 1s - loss: 0.0299 - accuracy: 0.9900
1058/1688 [=================>............] - ETA: 1s - loss: 0.0298 - accuracy: 0.9900
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0299 - accuracy: 0.9900
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0300 - accuracy: 0.9899
1119/1688 [==================>...........] - ETA: 1s - loss: 0.0300 - accuracy: 0.9899
1139/1688 [===================>..........] - ETA: 1s - loss: 0.0299 - accuracy: 0.9900
1159/1688 [===================>..........] - ETA: 1s - loss: 0.0296 - accuracy: 0.9901
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0294 - accuracy: 0.9901
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0294 - accuracy: 0.9901
1220/1688 [====================>.........] - ETA: 1s - loss: 0.0291 - accuracy: 0.9901
1241/1688 [=====================>........] - ETA: 1s - loss: 0.0292 - accuracy: 0.9902
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0292 - accuracy: 0.9902
1281/1688 [=====================>........] - ETA: 1s - loss: 0.0289 - accuracy: 0.9903
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0290 - accuracy: 0.9903
1322/1688 [======================>.......] - ETA: 0s - loss: 0.0293 - accuracy: 0.9903
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0293 - accuracy: 0.9903
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0293 - accuracy: 0.9903
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0291 - accuracy: 0.9904
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9904
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0289 - accuracy: 0.9903
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0288 - accuracy: 0.9904
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0291 - accuracy: 0.9903
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0292 - accuracy: 0.9902
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0294 - accuracy: 0.9901
1529/1688 [==========================>...] - ETA: 0s - loss: 0.0294 - accuracy: 0.9901
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0295 - accuracy: 0.9901
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0294 - accuracy: 0.9901
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0292 - accuracy: 0.9902
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0293 - accuracy: 0.9902
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0292 - accuracy: 0.9902
1651/1688 [============================>.] - ETA: 0s - loss: 0.0293 - accuracy: 0.9902
1671/1688 [============================>.] - ETA: 0s - loss: 0.0294 - accuracy: 0.9902
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0293 - accuracy: 0.9902 - val_loss: 0.0620 - val_accuracy: 0.9835
Epoch 3/5
1/1688 [..............................] - ETA: 4s - loss: 0.0024 - accuracy: 1.0000
22/1688 [..............................] - ETA: 4s - loss: 0.0204 - accuracy: 0.9943
42/1688 [..............................] - ETA: 4s - loss: 0.0227 - accuracy: 0.9918
63/1688 [>.............................] - ETA: 4s - loss: 0.0233 - accuracy: 0.9921
83/1688 [>.............................] - ETA: 4s - loss: 0.0206 - accuracy: 0.9921
103/1688 [>.............................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9924
124/1688 [=>............................] - ETA: 3s - loss: 0.0199 - accuracy: 0.9932
145/1688 [=>............................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9922
166/1688 [=>............................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9921
186/1688 [==>...........................] - ETA: 3s - loss: 0.0222 - accuracy: 0.9916
206/1688 [==>...........................] - ETA: 3s - loss: 0.0227 - accuracy: 0.9914
226/1688 [===>..........................] - ETA: 3s - loss: 0.0231 - accuracy: 0.9913
247/1688 [===>..........................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9905
267/1688 [===>..........................] - ETA: 3s - loss: 0.0247 - accuracy: 0.9905
287/1688 [====>.........................] - ETA: 3s - loss: 0.0242 - accuracy: 0.9907
307/1688 [====>.........................] - ETA: 3s - loss: 0.0234 - accuracy: 0.9911
327/1688 [====>.........................] - ETA: 3s - loss: 0.0225 - accuracy: 0.9916
348/1688 [=====>........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9920
368/1688 [=====>........................] - ETA: 3s - loss: 0.0215 - accuracy: 0.9922
388/1688 [=====>........................] - ETA: 3s - loss: 0.0210 - accuracy: 0.9923
408/1688 [======>.......................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9922
428/1688 [======>.......................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9920
448/1688 [======>.......................] - ETA: 3s - loss: 0.0212 - accuracy: 0.9921
469/1688 [=======>......................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9921
489/1688 [=======>......................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9924
510/1688 [========>.....................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9924
530/1688 [========>.....................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9923
550/1688 [========>.....................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9923
571/1688 [=========>....................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9922
591/1688 [=========>....................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9921
611/1688 [=========>....................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9920
631/1688 [==========>...................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9922
652/1688 [==========>...................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9920
672/1688 [==========>...................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9921
692/1688 [===========>..................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9920
713/1688 [===========>..................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9921
733/1688 [============>.................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9920
754/1688 [============>.................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9921
774/1688 [============>.................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9921
794/1688 [=============>................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9921
814/1688 [=============>................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9921
834/1688 [=============>................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9923
855/1688 [==============>...............] - ETA: 2s - loss: 0.0210 - accuracy: 0.9923
875/1688 [==============>...............] - ETA: 2s - loss: 0.0210 - accuracy: 0.9923
896/1688 [==============>...............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9924
917/1688 [===============>..............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9923
937/1688 [===============>..............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9924
957/1688 [================>.............] - ETA: 1s - loss: 0.0212 - accuracy: 0.9924
977/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9924
998/1688 [================>.............] - ETA: 1s - loss: 0.0217 - accuracy: 0.9923
1018/1688 [=================>............] - ETA: 1s - loss: 0.0218 - accuracy: 0.9923
1038/1688 [=================>............] - ETA: 1s - loss: 0.0219 - accuracy: 0.9922
1058/1688 [=================>............] - ETA: 1s - loss: 0.0222 - accuracy: 0.9922
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0220 - accuracy: 0.9923
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0219 - accuracy: 0.9923
1119/1688 [==================>...........] - ETA: 1s - loss: 0.0219 - accuracy: 0.9922
1139/1688 [===================>..........] - ETA: 1s - loss: 0.0220 - accuracy: 0.9922
1159/1688 [===================>..........] - ETA: 1s - loss: 0.0222 - accuracy: 0.9921
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0223 - accuracy: 0.9921
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0223 - accuracy: 0.9921
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0225 - accuracy: 0.9921
1242/1688 [=====================>........] - ETA: 1s - loss: 0.0229 - accuracy: 0.9920
1263/1688 [=====================>........] - ETA: 1s - loss: 0.0230 - accuracy: 0.9920
1283/1688 [=====================>........] - ETA: 1s - loss: 0.0232 - accuracy: 0.9919
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0234 - accuracy: 0.9918
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0236 - accuracy: 0.9917
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0239 - accuracy: 0.9916
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9917
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9916
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9916
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0239 - accuracy: 0.9916
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0241 - accuracy: 0.9916
1546/1688 [==========================>...] - ETA: 0s - loss: 0.0243 - accuracy: 0.9915
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0246 - accuracy: 0.9914
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0250 - accuracy: 0.9912
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0253 - accuracy: 0.9912
1628/1688 [===========================>..] - ETA: 0s - loss: 0.0254 - accuracy: 0.9912
1649/1688 [============================>.] - ETA: 0s - loss: 0.0253 - accuracy: 0.9912
1670/1688 [============================>.] - ETA: 0s - loss: 0.0256 - accuracy: 0.9912
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0258 - accuracy: 0.9911 - val_loss: 0.0540 - val_accuracy: 0.9872
Epoch 4/5
1/1688 [..............................] - ETA: 4s - loss: 9.5040e-04 - accuracy: 1.0000
22/1688 [..............................] - ETA: 4s - loss: 0.0211 - accuracy: 0.9929
42/1688 [..............................] - ETA: 4s - loss: 0.0268 - accuracy: 0.9903
63/1688 [>.............................] - ETA: 4s - loss: 0.0305 - accuracy: 0.9901
84/1688 [>.............................] - ETA: 4s - loss: 0.0246 - accuracy: 0.9922
104/1688 [>.............................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9928
125/1688 [=>............................] - ETA: 3s - loss: 0.0215 - accuracy: 0.9935
146/1688 [=>............................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9940
167/1688 [=>............................] - ETA: 3s - loss: 0.0186 - accuracy: 0.9944
187/1688 [==>...........................] - ETA: 3s - loss: 0.0184 - accuracy: 0.9943
207/1688 [==>...........................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9940
227/1688 [===>..........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9942
248/1688 [===>..........................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9937
268/1688 [===>..........................] - ETA: 3s - loss: 0.0194 - accuracy: 0.9939
289/1688 [====>.........................] - ETA: 3s - loss: 0.0194 - accuracy: 0.9941
309/1688 [====>.........................] - ETA: 3s - loss: 0.0188 - accuracy: 0.9943
330/1688 [====>.........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9942
350/1688 [=====>........................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9941
370/1688 [=====>........................] - ETA: 3s - loss: 0.0188 - accuracy: 0.9943
390/1688 [=====>........................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9945
410/1688 [======>.......................] - ETA: 3s - loss: 0.0178 - accuracy: 0.9947
431/1688 [======>.......................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9943
451/1688 [=======>......................] - ETA: 3s - loss: 0.0184 - accuracy: 0.9942
472/1688 [=======>......................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9944
492/1688 [=======>......................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9940
512/1688 [========>.....................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9940
532/1688 [========>.....................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9939
553/1688 [========>.....................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9937
573/1688 [=========>....................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9935
594/1688 [=========>....................] - ETA: 2s - loss: 0.0194 - accuracy: 0.9935
614/1688 [=========>....................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9936
634/1688 [==========>...................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9935
655/1688 [==========>...................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9934
675/1688 [==========>...................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9933
696/1688 [===========>..................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9934
716/1688 [===========>..................] - ETA: 2s - loss: 0.0198 - accuracy: 0.9935
737/1688 [============>.................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9936
757/1688 [============>.................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9936
777/1688 [============>.................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9935
797/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9934
817/1688 [=============>................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9932
838/1688 [=============>................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9932
858/1688 [==============>...............] - ETA: 2s - loss: 0.0214 - accuracy: 0.9930
878/1688 [==============>...............] - ETA: 2s - loss: 0.0212 - accuracy: 0.9931
898/1688 [==============>...............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
918/1688 [===============>..............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9930
938/1688 [===============>..............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
958/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
978/1688 [================>.............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
998/1688 [================>.............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9930
1018/1688 [=================>............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9929
1038/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
1059/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9929
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9928
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9929
1120/1688 [==================>...........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9929
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0214 - accuracy: 0.9928
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9927
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9927
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0215 - accuracy: 0.9927
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0214 - accuracy: 0.9927
1242/1688 [=====================>........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9928
1262/1688 [=====================>........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9929
1282/1688 [=====================>........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9928
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0215 - accuracy: 0.9927
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1365/1688 [=======================>......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1385/1688 [=======================>......] - ETA: 0s - loss: 0.0222 - accuracy: 0.9924
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0227 - accuracy: 0.9923
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0225 - accuracy: 0.9923
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0225 - accuracy: 0.9923
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0225 - accuracy: 0.9923
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0225 - accuracy: 0.9923
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0228 - accuracy: 0.9922
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0227 - accuracy: 0.9922
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0229 - accuracy: 0.9922
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0228 - accuracy: 0.9921
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0227 - accuracy: 0.9922
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0228 - accuracy: 0.9921
1650/1688 [============================>.] - ETA: 0s - loss: 0.0230 - accuracy: 0.9921
1671/1688 [============================>.] - ETA: 0s - loss: 0.0229 - accuracy: 0.9921
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0231 - accuracy: 0.9921 - val_loss: 0.0561 - val_accuracy: 0.9873
Epoch 5/5
1/1688 [..............................] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.0230 - accuracy: 0.9911
41/1688 [..............................] - ETA: 4s - loss: 0.0160 - accuracy: 0.9947
62/1688 [>.............................] - ETA: 4s - loss: 0.0149 - accuracy: 0.9950
82/1688 [>.............................] - ETA: 4s - loss: 0.0175 - accuracy: 0.9935
102/1688 [>.............................] - ETA: 4s - loss: 0.0178 - accuracy: 0.9936
123/1688 [=>............................] - ETA: 3s - loss: 0.0162 - accuracy: 0.9942
143/1688 [=>............................] - ETA: 3s - loss: 0.0165 - accuracy: 0.9934
163/1688 [=>............................] - ETA: 3s - loss: 0.0164 - accuracy: 0.9933
183/1688 [==>...........................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9925
203/1688 [==>...........................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9929
223/1688 [==>...........................] - ETA: 3s - loss: 0.0186 - accuracy: 0.9927
243/1688 [===>..........................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9927
263/1688 [===>..........................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9928
283/1688 [====>.........................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9927
303/1688 [====>.........................] - ETA: 3s - loss: 0.0210 - accuracy: 0.9927
324/1688 [====>.........................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9929
344/1688 [=====>........................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9932
364/1688 [=====>........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9934
384/1688 [=====>........................] - ETA: 3s - loss: 0.0184 - accuracy: 0.9933
404/1688 [======>.......................] - ETA: 3s - loss: 0.0189 - accuracy: 0.9933
425/1688 [======>.......................] - ETA: 3s - loss: 0.0195 - accuracy: 0.9929
445/1688 [======>.......................] - ETA: 3s - loss: 0.0189 - accuracy: 0.9931
466/1688 [=======>......................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9932
487/1688 [=======>......................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9931
507/1688 [========>.....................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9931
528/1688 [========>.....................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9933
548/1688 [========>.....................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9932
568/1688 [=========>....................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9930
589/1688 [=========>....................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9929
610/1688 [=========>....................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9928
630/1688 [==========>...................] - ETA: 2s - loss: 0.0197 - accuracy: 0.9930
651/1688 [==========>...................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9929
671/1688 [==========>...................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9929
691/1688 [===========>..................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9928
711/1688 [===========>..................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9927
731/1688 [===========>..................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9926
751/1688 [============>.................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9926
771/1688 [============>.................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9925
791/1688 [=============>................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9924
811/1688 [=============>................] - ETA: 2s - loss: 0.0216 - accuracy: 0.9923
831/1688 [=============>................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9923
851/1688 [==============>...............] - ETA: 2s - loss: 0.0214 - accuracy: 0.9924
871/1688 [==============>...............] - ETA: 2s - loss: 0.0211 - accuracy: 0.9924
892/1688 [==============>...............] - ETA: 2s - loss: 0.0208 - accuracy: 0.9924
912/1688 [===============>..............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9925
933/1688 [===============>..............] - ETA: 1s - loss: 0.0209 - accuracy: 0.9925
953/1688 [===============>..............] - ETA: 1s - loss: 0.0207 - accuracy: 0.9925
973/1688 [================>.............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9924
993/1688 [================>.............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9923
1013/1688 [=================>............] - ETA: 1s - loss: 0.0217 - accuracy: 0.9922
1033/1688 [=================>............] - ETA: 1s - loss: 0.0215 - accuracy: 0.9923
1053/1688 [=================>............] - ETA: 1s - loss: 0.0216 - accuracy: 0.9923
1073/1688 [==================>...........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9923
1093/1688 [==================>...........] - ETA: 1s - loss: 0.0213 - accuracy: 0.9923
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9923
1134/1688 [===================>..........] - ETA: 1s - loss: 0.0210 - accuracy: 0.9924
1154/1688 [===================>..........] - ETA: 1s - loss: 0.0210 - accuracy: 0.9925
1175/1688 [===================>..........] - ETA: 1s - loss: 0.0208 - accuracy: 0.9925
1196/1688 [====================>.........] - ETA: 1s - loss: 0.0208 - accuracy: 0.9925
1216/1688 [====================>.........] - ETA: 1s - loss: 0.0208 - accuracy: 0.9925
1237/1688 [====================>.........] - ETA: 1s - loss: 0.0209 - accuracy: 0.9925
1257/1688 [=====================>........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9924
1277/1688 [=====================>........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9924
1297/1688 [======================>.......] - ETA: 0s - loss: 0.0214 - accuracy: 0.9923
1317/1688 [======================>.......] - ETA: 0s - loss: 0.0212 - accuracy: 0.9923
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0212 - accuracy: 0.9924
1358/1688 [=======================>......] - ETA: 0s - loss: 0.0214 - accuracy: 0.9923
1378/1688 [=======================>......] - ETA: 0s - loss: 0.0213 - accuracy: 0.9924
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0216 - accuracy: 0.9922
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9921
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9921
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9922
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0218 - accuracy: 0.9922
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0220 - accuracy: 0.9921
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0223 - accuracy: 0.9920
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0222 - accuracy: 0.9920
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0220 - accuracy: 0.9921
1642/1688 [============================>.] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1662/1688 [============================>.] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1683/1688 [============================>.] - ETA: 0s - loss: 0.0219 - accuracy: 0.9921
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0219 - accuracy: 0.9921 - val_loss: 0.0633 - val_accuracy: 0.9855
Test accuracy after fine tuning: 0.986299991607666
6. Model conversion¶
After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida Execution Engine.
Note
One needs to supply the coefficients used to rescale the input
dataset before the training - here input_scaling.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()
results = model_akida.predict(raw_x_test)
accuracy = (raw_y_test == results).mean()
print('Test accuracy after conversion:', accuracy)
# For non-regression purpose
assert accuracy > 0.97
Out:
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 3
______________________________________________
SW/conv2d-dense (Software)
________________________________________________________________________________________
Layer (type) Output shape Kernel shape
========================================================================================
conv2d (InputConvolutional) [13, 13, 32] (3, 3, 1, 32)
________________________________________________________________________________________
separable_conv2d (SeparableConvolutional) [7, 7, 64] (3, 3, 32, 1), (1, 1, 32, 64)
________________________________________________________________________________________
dense (FullyConnected) [1, 1, 10] (1, 1, 3136, 10)
________________________________________________________________________________________
Test accuracy after conversion: 0.986
Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).
Total running time of the script: ( 1 minutes 1.440 seconds)