Note
Click here to download the full example code
CNN conversion flow tutorial
This tutorial illustrates how to use the CNN2SNN toolkit to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps. You can refer to our CNN2SNN toolkit user guide for further explanation.
The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.
Note
Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.
MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.
1. Load and reshape MNIST dataset
After loading, we make 2 transformations on the dataset:
Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.
Note
At this point, we’ll set aside the raw data for testing our converted model in the Akida runtime later.
Rescale the 8-bit loaded data to the range 0-to-1 for training.
Note
Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some reading can be found here.
This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.
Also note that we store the scaling values input_scaling for
use when preparing the model for the Akida neuromorphic IP. The
implementation of the Akida neural network allows us to completely
skip the rescaling step (i.e. the Akida model should be fed with
the raw 8-bit values) but that does require information about what
scaling was applied prior to training - see below for more details.
from tensorflow import keras
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)
# Set aside raw test data for use with Akida runtime later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test
# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a
2. Model definition
Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.
An appropriate model for MNIST (inspired by this example) might look something like the following:
model_keras = keras.models.Sequential([
keras.layers.Conv2D(
filters=32, kernel_size=3, input_shape=(28, 28, 1), strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.SeparableConv2D(
filters=64, kernel_size=3, padding='same', strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Out:
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 13, 13, 32) 320
batch_normalization (BatchN (None, 13, 13, 32) 128
ormalization)
re_lu (ReLU) (None, 13, 13, 32) 0
separable_conv2d (Separable (None, 7, 7, 64) 2400
Conv2D)
batch_normalization_1 (Batc (None, 7, 7, 64) 256
hNormalization)
re_lu_1 (ReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34,474
Trainable params: 34,282
Non-trainable params: 192
_________________________________________________________________
The model defined above is compatible for conversion into an Akida model, i.e. the model doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):
Standard Conv2D and Dense layers are supported
Hidden layers must be followed by a ReLU layer.
BatchNormalization must always happen before activations.
Convolutional blocks can optionally be followed by a MaxPooling.
The CNN2SNN toolkit provides the check_model_compatibility function to ensure that the model can be converted into an Akida model. If the model is not fully compatible, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).
from cnn2snn import check_model_compatibility
print("Model compatible for Akida conversion:",
check_model_compatibility(model_keras))
Out:
Model compatible for Akida conversion: True
3. Model training
Before going any further, train the model and get its performance. The created model should have achieved a test accuracy over 98% after 10 epochs.
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Out:
Epoch 1/10
1/1688 [..............................] - ETA: 11:51 - loss: 2.7239 - accuracy: 0.0938
28/1688 [..............................] - ETA: 3s - loss: 1.3301 - accuracy: 0.5938
56/1688 [..............................] - ETA: 2s - loss: 0.9390 - accuracy: 0.7109
84/1688 [>.............................] - ETA: 2s - loss: 0.7748 - accuracy: 0.7638
113/1688 [=>............................] - ETA: 2s - loss: 0.6793 - accuracy: 0.7929
142/1688 [=>............................] - ETA: 2s - loss: 0.6182 - accuracy: 0.8134
171/1688 [==>...........................] - ETA: 2s - loss: 0.5609 - accuracy: 0.8300
199/1688 [==>...........................] - ETA: 2s - loss: 0.5208 - accuracy: 0.8430
228/1688 [===>..........................] - ETA: 2s - loss: 0.4973 - accuracy: 0.8509
256/1688 [===>..........................] - ETA: 2s - loss: 0.4710 - accuracy: 0.8586
284/1688 [====>.........................] - ETA: 2s - loss: 0.4531 - accuracy: 0.8637
312/1688 [====>.........................] - ETA: 2s - loss: 0.4338 - accuracy: 0.8694
340/1688 [=====>........................] - ETA: 2s - loss: 0.4129 - accuracy: 0.8753
368/1688 [=====>........................] - ETA: 2s - loss: 0.3958 - accuracy: 0.8809
397/1688 [======>.......................] - ETA: 2s - loss: 0.3807 - accuracy: 0.8852
426/1688 [======>.......................] - ETA: 2s - loss: 0.3723 - accuracy: 0.8878
454/1688 [=======>......................] - ETA: 2s - loss: 0.3604 - accuracy: 0.8918
482/1688 [=======>......................] - ETA: 2s - loss: 0.3483 - accuracy: 0.8959
511/1688 [========>.....................] - ETA: 2s - loss: 0.3396 - accuracy: 0.8987
540/1688 [========>.....................] - ETA: 2s - loss: 0.3321 - accuracy: 0.9008
568/1688 [=========>....................] - ETA: 2s - loss: 0.3227 - accuracy: 0.9031
597/1688 [=========>....................] - ETA: 1s - loss: 0.3135 - accuracy: 0.9062
626/1688 [==========>...................] - ETA: 1s - loss: 0.3059 - accuracy: 0.9087
654/1688 [==========>...................] - ETA: 1s - loss: 0.2988 - accuracy: 0.9105
683/1688 [===========>..................] - ETA: 1s - loss: 0.2919 - accuracy: 0.9126
712/1688 [===========>..................] - ETA: 1s - loss: 0.2848 - accuracy: 0.9147
740/1688 [============>.................] - ETA: 1s - loss: 0.2795 - accuracy: 0.9165
769/1688 [============>.................] - ETA: 1s - loss: 0.2738 - accuracy: 0.9178
798/1688 [=============>................] - ETA: 1s - loss: 0.2690 - accuracy: 0.9191
826/1688 [=============>................] - ETA: 1s - loss: 0.2638 - accuracy: 0.9209
853/1688 [==============>...............] - ETA: 1s - loss: 0.2584 - accuracy: 0.9224
881/1688 [==============>...............] - ETA: 1s - loss: 0.2540 - accuracy: 0.9235
910/1688 [===============>..............] - ETA: 1s - loss: 0.2493 - accuracy: 0.9248
938/1688 [===============>..............] - ETA: 1s - loss: 0.2458 - accuracy: 0.9258
966/1688 [================>.............] - ETA: 1s - loss: 0.2422 - accuracy: 0.9268
994/1688 [================>.............] - ETA: 1s - loss: 0.2380 - accuracy: 0.9281
1022/1688 [=================>............] - ETA: 1s - loss: 0.2344 - accuracy: 0.9291
1050/1688 [=================>............] - ETA: 1s - loss: 0.2310 - accuracy: 0.9299
1078/1688 [==================>...........] - ETA: 1s - loss: 0.2273 - accuracy: 0.9311
1107/1688 [==================>...........] - ETA: 1s - loss: 0.2237 - accuracy: 0.9321
1135/1688 [===================>..........] - ETA: 0s - loss: 0.2214 - accuracy: 0.9327
1163/1688 [===================>..........] - ETA: 0s - loss: 0.2195 - accuracy: 0.9333
1192/1688 [====================>.........] - ETA: 0s - loss: 0.2163 - accuracy: 0.9342
1221/1688 [====================>.........] - ETA: 0s - loss: 0.2135 - accuracy: 0.9351
1250/1688 [=====================>........] - ETA: 0s - loss: 0.2103 - accuracy: 0.9361
1279/1688 [=====================>........] - ETA: 0s - loss: 0.2079 - accuracy: 0.9367
1307/1688 [======================>.......] - ETA: 0s - loss: 0.2051 - accuracy: 0.9377
1335/1688 [======================>.......] - ETA: 0s - loss: 0.2022 - accuracy: 0.9386
1363/1688 [=======================>......] - ETA: 0s - loss: 0.2000 - accuracy: 0.9391
1391/1688 [=======================>......] - ETA: 0s - loss: 0.1986 - accuracy: 0.9396
1420/1688 [========================>.....] - ETA: 0s - loss: 0.1959 - accuracy: 0.9404
1448/1688 [========================>.....] - ETA: 0s - loss: 0.1933 - accuracy: 0.9411
1476/1688 [=========================>....] - ETA: 0s - loss: 0.1917 - accuracy: 0.9416
1505/1688 [=========================>....] - ETA: 0s - loss: 0.1901 - accuracy: 0.9421
1533/1688 [==========================>...] - ETA: 0s - loss: 0.1886 - accuracy: 0.9426
1561/1688 [==========================>...] - ETA: 0s - loss: 0.1866 - accuracy: 0.9431
1590/1688 [===========================>..] - ETA: 0s - loss: 0.1846 - accuracy: 0.9438
1619/1688 [===========================>..] - ETA: 0s - loss: 0.1839 - accuracy: 0.9439
1647/1688 [============================>.] - ETA: 0s - loss: 0.1823 - accuracy: 0.9443
1676/1688 [============================>.] - ETA: 0s - loss: 0.1812 - accuracy: 0.9446
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1805 - accuracy: 0.9448 - val_loss: 0.1017 - val_accuracy: 0.9705
Epoch 2/10
1/1688 [..............................] - ETA: 3s - loss: 0.0141 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0691 - accuracy: 0.9758
59/1688 [>.............................] - ETA: 2s - loss: 0.0758 - accuracy: 0.9783
89/1688 [>.............................] - ETA: 2s - loss: 0.0867 - accuracy: 0.9754
119/1688 [=>............................] - ETA: 2s - loss: 0.0814 - accuracy: 0.9761
148/1688 [=>............................] - ETA: 2s - loss: 0.0814 - accuracy: 0.9753
177/1688 [==>...........................] - ETA: 2s - loss: 0.0853 - accuracy: 0.9748
205/1688 [==>...........................] - ETA: 2s - loss: 0.0814 - accuracy: 0.9753
233/1688 [===>..........................] - ETA: 2s - loss: 0.0826 - accuracy: 0.9753
262/1688 [===>..........................] - ETA: 2s - loss: 0.0801 - accuracy: 0.9759
290/1688 [====>.........................] - ETA: 2s - loss: 0.0773 - accuracy: 0.9767
319/1688 [====>.........................] - ETA: 2s - loss: 0.0763 - accuracy: 0.9769
347/1688 [=====>........................] - ETA: 2s - loss: 0.0753 - accuracy: 0.9771
375/1688 [=====>........................] - ETA: 2s - loss: 0.0746 - accuracy: 0.9771
402/1688 [======>.......................] - ETA: 2s - loss: 0.0741 - accuracy: 0.9771
431/1688 [======>.......................] - ETA: 2s - loss: 0.0756 - accuracy: 0.9765
459/1688 [=======>......................] - ETA: 2s - loss: 0.0789 - accuracy: 0.9756
489/1688 [=======>......................] - ETA: 2s - loss: 0.0799 - accuracy: 0.9750
517/1688 [========>.....................] - ETA: 2s - loss: 0.0797 - accuracy: 0.9750
545/1688 [========>.....................] - ETA: 2s - loss: 0.0793 - accuracy: 0.9752
574/1688 [=========>....................] - ETA: 1s - loss: 0.0788 - accuracy: 0.9754
602/1688 [=========>....................] - ETA: 1s - loss: 0.0778 - accuracy: 0.9758
631/1688 [==========>...................] - ETA: 1s - loss: 0.0767 - accuracy: 0.9762
659/1688 [==========>...................] - ETA: 1s - loss: 0.0774 - accuracy: 0.9762
688/1688 [===========>..................] - ETA: 1s - loss: 0.0769 - accuracy: 0.9763
716/1688 [===========>..................] - ETA: 1s - loss: 0.0762 - accuracy: 0.9766
745/1688 [============>.................] - ETA: 1s - loss: 0.0755 - accuracy: 0.9768
774/1688 [============>.................] - ETA: 1s - loss: 0.0755 - accuracy: 0.9769
801/1688 [=============>................] - ETA: 1s - loss: 0.0748 - accuracy: 0.9770
829/1688 [=============>................] - ETA: 1s - loss: 0.0753 - accuracy: 0.9769
857/1688 [==============>...............] - ETA: 1s - loss: 0.0755 - accuracy: 0.9768
885/1688 [==============>...............] - ETA: 1s - loss: 0.0762 - accuracy: 0.9764
913/1688 [===============>..............] - ETA: 1s - loss: 0.0762 - accuracy: 0.9763
942/1688 [===============>..............] - ETA: 1s - loss: 0.0764 - accuracy: 0.9761
970/1688 [================>.............] - ETA: 1s - loss: 0.0761 - accuracy: 0.9763
999/1688 [================>.............] - ETA: 1s - loss: 0.0753 - accuracy: 0.9765
1027/1688 [=================>............] - ETA: 1s - loss: 0.0753 - accuracy: 0.9766
1056/1688 [=================>............] - ETA: 1s - loss: 0.0753 - accuracy: 0.9765
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0751 - accuracy: 0.9766
1113/1688 [==================>...........] - ETA: 1s - loss: 0.0750 - accuracy: 0.9765
1141/1688 [===================>..........] - ETA: 0s - loss: 0.0748 - accuracy: 0.9767
1169/1688 [===================>..........] - ETA: 0s - loss: 0.0744 - accuracy: 0.9769
1197/1688 [====================>.........] - ETA: 0s - loss: 0.0746 - accuracy: 0.9768
1225/1688 [====================>.........] - ETA: 0s - loss: 0.0741 - accuracy: 0.9769
1254/1688 [=====================>........] - ETA: 0s - loss: 0.0741 - accuracy: 0.9770
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0742 - accuracy: 0.9770
1310/1688 [======================>.......] - ETA: 0s - loss: 0.0741 - accuracy: 0.9769
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0749 - accuracy: 0.9767
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0746 - accuracy: 0.9768
1395/1688 [=======================>......] - ETA: 0s - loss: 0.0741 - accuracy: 0.9770
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0738 - accuracy: 0.9771
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0729 - accuracy: 0.9774
1481/1688 [=========================>....] - ETA: 0s - loss: 0.0730 - accuracy: 0.9774
1510/1688 [=========================>....] - ETA: 0s - loss: 0.0726 - accuracy: 0.9775
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0726 - accuracy: 0.9776
1567/1688 [==========================>...] - ETA: 0s - loss: 0.0727 - accuracy: 0.9775
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0724 - accuracy: 0.9775
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0723 - accuracy: 0.9775
1653/1688 [============================>.] - ETA: 0s - loss: 0.0726 - accuracy: 0.9775
1682/1688 [============================>.] - ETA: 0s - loss: 0.0723 - accuracy: 0.9776
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0725 - accuracy: 0.9775 - val_loss: 0.0751 - val_accuracy: 0.9782
Epoch 3/10
1/1688 [..............................] - ETA: 2s - loss: 0.0191 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0580 - accuracy: 0.9771
57/1688 [>.............................] - ETA: 2s - loss: 0.0508 - accuracy: 0.9836
86/1688 [>.............................] - ETA: 2s - loss: 0.0470 - accuracy: 0.9851
114/1688 [=>............................] - ETA: 2s - loss: 0.0498 - accuracy: 0.9855
142/1688 [=>............................] - ETA: 2s - loss: 0.0505 - accuracy: 0.9855
171/1688 [==>...........................] - ETA: 2s - loss: 0.0507 - accuracy: 0.9854
199/1688 [==>...........................] - ETA: 2s - loss: 0.0518 - accuracy: 0.9846
227/1688 [===>..........................] - ETA: 2s - loss: 0.0493 - accuracy: 0.9851
255/1688 [===>..........................] - ETA: 2s - loss: 0.0477 - accuracy: 0.9857
283/1688 [====>.........................] - ETA: 2s - loss: 0.0481 - accuracy: 0.9856
311/1688 [====>.........................] - ETA: 2s - loss: 0.0511 - accuracy: 0.9845
339/1688 [=====>........................] - ETA: 2s - loss: 0.0489 - accuracy: 0.9851
367/1688 [=====>........................] - ETA: 2s - loss: 0.0480 - accuracy: 0.9854
396/1688 [======>.......................] - ETA: 2s - loss: 0.0467 - accuracy: 0.9858
424/1688 [======>.......................] - ETA: 2s - loss: 0.0463 - accuracy: 0.9860
452/1688 [=======>......................] - ETA: 2s - loss: 0.0459 - accuracy: 0.9858
480/1688 [=======>......................] - ETA: 2s - loss: 0.0459 - accuracy: 0.9859
507/1688 [========>.....................] - ETA: 2s - loss: 0.0455 - accuracy: 0.9861
535/1688 [========>.....................] - ETA: 2s - loss: 0.0456 - accuracy: 0.9860
563/1688 [=========>....................] - ETA: 2s - loss: 0.0465 - accuracy: 0.9860
591/1688 [=========>....................] - ETA: 1s - loss: 0.0466 - accuracy: 0.9859
619/1688 [==========>...................] - ETA: 1s - loss: 0.0467 - accuracy: 0.9858
646/1688 [==========>...................] - ETA: 1s - loss: 0.0470 - accuracy: 0.9857
674/1688 [==========>...................] - ETA: 1s - loss: 0.0479 - accuracy: 0.9855
703/1688 [===========>..................] - ETA: 1s - loss: 0.0475 - accuracy: 0.9853
732/1688 [============>.................] - ETA: 1s - loss: 0.0481 - accuracy: 0.9850
760/1688 [============>.................] - ETA: 1s - loss: 0.0481 - accuracy: 0.9850
789/1688 [=============>................] - ETA: 1s - loss: 0.0493 - accuracy: 0.9848
818/1688 [=============>................] - ETA: 1s - loss: 0.0495 - accuracy: 0.9846
847/1688 [==============>...............] - ETA: 1s - loss: 0.0499 - accuracy: 0.9842
875/1688 [==============>...............] - ETA: 1s - loss: 0.0505 - accuracy: 0.9839
903/1688 [===============>..............] - ETA: 1s - loss: 0.0501 - accuracy: 0.9840
930/1688 [===============>..............] - ETA: 1s - loss: 0.0497 - accuracy: 0.9841
958/1688 [================>.............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9841
987/1688 [================>.............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9841
1015/1688 [=================>............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9841
1043/1688 [=================>............] - ETA: 1s - loss: 0.0507 - accuracy: 0.9840
1071/1688 [==================>...........] - ETA: 1s - loss: 0.0508 - accuracy: 0.9838
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0505 - accuracy: 0.9839
1127/1688 [===================>..........] - ETA: 1s - loss: 0.0511 - accuracy: 0.9838
1155/1688 [===================>..........] - ETA: 0s - loss: 0.0509 - accuracy: 0.9838
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0508 - accuracy: 0.9839
1213/1688 [====================>.........] - ETA: 0s - loss: 0.0508 - accuracy: 0.9839
1241/1688 [=====================>........] - ETA: 0s - loss: 0.0509 - accuracy: 0.9839
1269/1688 [=====================>........] - ETA: 0s - loss: 0.0504 - accuracy: 0.9842
1297/1688 [======================>.......] - ETA: 0s - loss: 0.0507 - accuracy: 0.9841
1325/1688 [======================>.......] - ETA: 0s - loss: 0.0507 - accuracy: 0.9841
1353/1688 [=======================>......] - ETA: 0s - loss: 0.0507 - accuracy: 0.9839
1382/1688 [=======================>......] - ETA: 0s - loss: 0.0511 - accuracy: 0.9839
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0517 - accuracy: 0.9837
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0521 - accuracy: 0.9836
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0524 - accuracy: 0.9836
1494/1688 [=========================>....] - ETA: 0s - loss: 0.0519 - accuracy: 0.9838
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0518 - accuracy: 0.9838
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0516 - accuracy: 0.9838
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0515 - accuracy: 0.9838
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0520 - accuracy: 0.9836
1635/1688 [============================>.] - ETA: 0s - loss: 0.0520 - accuracy: 0.9837
1663/1688 [============================>.] - ETA: 0s - loss: 0.0523 - accuracy: 0.9836
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0520 - accuracy: 0.9836 - val_loss: 0.0637 - val_accuracy: 0.9833
Epoch 4/10
1/1688 [..............................] - ETA: 3s - loss: 0.0853 - accuracy: 0.9688
30/1688 [..............................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9865
59/1688 [>.............................] - ETA: 2s - loss: 0.0394 - accuracy: 0.9894
88/1688 [>.............................] - ETA: 2s - loss: 0.0391 - accuracy: 0.9879
117/1688 [=>............................] - ETA: 2s - loss: 0.0362 - accuracy: 0.9885
146/1688 [=>............................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9876
175/1688 [==>...........................] - ETA: 2s - loss: 0.0367 - accuracy: 0.9887
203/1688 [==>...........................] - ETA: 2s - loss: 0.0356 - accuracy: 0.9886
232/1688 [===>..........................] - ETA: 2s - loss: 0.0372 - accuracy: 0.9881
261/1688 [===>..........................] - ETA: 2s - loss: 0.0384 - accuracy: 0.9880
289/1688 [====>.........................] - ETA: 2s - loss: 0.0392 - accuracy: 0.9880
317/1688 [====>.........................] - ETA: 2s - loss: 0.0385 - accuracy: 0.9883
346/1688 [=====>........................] - ETA: 2s - loss: 0.0379 - accuracy: 0.9886
374/1688 [=====>........................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9886
403/1688 [======>.......................] - ETA: 2s - loss: 0.0389 - accuracy: 0.9881
432/1688 [======>.......................] - ETA: 2s - loss: 0.0394 - accuracy: 0.9878
461/1688 [=======>......................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9876
489/1688 [=======>......................] - ETA: 2s - loss: 0.0397 - accuracy: 0.9877
517/1688 [========>.....................] - ETA: 2s - loss: 0.0393 - accuracy: 0.9878
546/1688 [========>.....................] - ETA: 2s - loss: 0.0395 - accuracy: 0.9876
575/1688 [=========>....................] - ETA: 1s - loss: 0.0399 - accuracy: 0.9874
603/1688 [=========>....................] - ETA: 1s - loss: 0.0404 - accuracy: 0.9873
632/1688 [==========>...................] - ETA: 1s - loss: 0.0406 - accuracy: 0.9869
661/1688 [==========>...................] - ETA: 1s - loss: 0.0414 - accuracy: 0.9866
689/1688 [===========>..................] - ETA: 1s - loss: 0.0418 - accuracy: 0.9864
718/1688 [===========>..................] - ETA: 1s - loss: 0.0417 - accuracy: 0.9865
747/1688 [============>.................] - ETA: 1s - loss: 0.0410 - accuracy: 0.9868
776/1688 [============>.................] - ETA: 1s - loss: 0.0413 - accuracy: 0.9868
804/1688 [=============>................] - ETA: 1s - loss: 0.0416 - accuracy: 0.9867
832/1688 [=============>................] - ETA: 1s - loss: 0.0418 - accuracy: 0.9867
861/1688 [==============>...............] - ETA: 1s - loss: 0.0426 - accuracy: 0.9866
890/1688 [==============>...............] - ETA: 1s - loss: 0.0421 - accuracy: 0.9868
918/1688 [===============>..............] - ETA: 1s - loss: 0.0427 - accuracy: 0.9867
946/1688 [===============>..............] - ETA: 1s - loss: 0.0424 - accuracy: 0.9867
974/1688 [================>.............] - ETA: 1s - loss: 0.0421 - accuracy: 0.9868
1002/1688 [================>.............] - ETA: 1s - loss: 0.0417 - accuracy: 0.9869
1030/1688 [=================>............] - ETA: 1s - loss: 0.0414 - accuracy: 0.9870
1058/1688 [=================>............] - ETA: 1s - loss: 0.0411 - accuracy: 0.9871
1087/1688 [==================>...........] - ETA: 1s - loss: 0.0410 - accuracy: 0.9871
1115/1688 [==================>...........] - ETA: 1s - loss: 0.0407 - accuracy: 0.9872
1143/1688 [===================>..........] - ETA: 0s - loss: 0.0405 - accuracy: 0.9872
1172/1688 [===================>..........] - ETA: 0s - loss: 0.0405 - accuracy: 0.9872
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0409 - accuracy: 0.9871
1229/1688 [====================>.........] - ETA: 0s - loss: 0.0412 - accuracy: 0.9869
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0410 - accuracy: 0.9869
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0414 - accuracy: 0.9869
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0416 - accuracy: 0.9868
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0414 - accuracy: 0.9869
1371/1688 [=======================>......] - ETA: 0s - loss: 0.0415 - accuracy: 0.9869
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0417 - accuracy: 0.9867
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0417 - accuracy: 0.9867
1456/1688 [========================>.....] - ETA: 0s - loss: 0.0417 - accuracy: 0.9867
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0416 - accuracy: 0.9867
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0421 - accuracy: 0.9866
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0422 - accuracy: 0.9865
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0420 - accuracy: 0.9866
1598/1688 [===========================>..] - ETA: 0s - loss: 0.0429 - accuracy: 0.9864
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0430 - accuracy: 0.9864
1655/1688 [============================>.] - ETA: 0s - loss: 0.0430 - accuracy: 0.9863
1684/1688 [============================>.] - ETA: 0s - loss: 0.0429 - accuracy: 0.9863
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0428 - accuracy: 0.9863 - val_loss: 0.0564 - val_accuracy: 0.9855
Epoch 5/10
1/1688 [..............................] - ETA: 3s - loss: 0.0470 - accuracy: 0.9688
31/1688 [..............................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9940
60/1688 [>.............................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9906
88/1688 [>.............................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9908
116/1688 [=>............................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9908
143/1688 [=>............................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9910
172/1688 [==>...........................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9907
201/1688 [==>...........................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9905
229/1688 [===>..........................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9907
258/1688 [===>..........................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9908
286/1688 [====>.........................] - ETA: 2s - loss: 0.0274 - accuracy: 0.9908
314/1688 [====>.........................] - ETA: 2s - loss: 0.0271 - accuracy: 0.9908
342/1688 [=====>........................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9906
371/1688 [=====>........................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9906
399/1688 [======>.......................] - ETA: 2s - loss: 0.0273 - accuracy: 0.9908
427/1688 [======>.......................] - ETA: 2s - loss: 0.0268 - accuracy: 0.9912
455/1688 [=======>......................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9913
484/1688 [=======>......................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9915
513/1688 [========>.....................] - ETA: 2s - loss: 0.0269 - accuracy: 0.9913
541/1688 [========>.....................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9909
569/1688 [=========>....................] - ETA: 2s - loss: 0.0276 - accuracy: 0.9910
598/1688 [=========>....................] - ETA: 1s - loss: 0.0276 - accuracy: 0.9910
626/1688 [==========>...................] - ETA: 1s - loss: 0.0284 - accuracy: 0.9907
654/1688 [==========>...................] - ETA: 1s - loss: 0.0287 - accuracy: 0.9907
682/1688 [===========>..................] - ETA: 1s - loss: 0.0283 - accuracy: 0.9907
710/1688 [===========>..................] - ETA: 1s - loss: 0.0283 - accuracy: 0.9905
738/1688 [============>.................] - ETA: 1s - loss: 0.0287 - accuracy: 0.9904
766/1688 [============>.................] - ETA: 1s - loss: 0.0292 - accuracy: 0.9900
794/1688 [=============>................] - ETA: 1s - loss: 0.0303 - accuracy: 0.9897
823/1688 [=============>................] - ETA: 1s - loss: 0.0309 - accuracy: 0.9896
851/1688 [==============>...............] - ETA: 1s - loss: 0.0313 - accuracy: 0.9895
879/1688 [==============>...............] - ETA: 1s - loss: 0.0314 - accuracy: 0.9894
907/1688 [===============>..............] - ETA: 1s - loss: 0.0317 - accuracy: 0.9894
936/1688 [===============>..............] - ETA: 1s - loss: 0.0313 - accuracy: 0.9894
964/1688 [================>.............] - ETA: 1s - loss: 0.0312 - accuracy: 0.9895
993/1688 [================>.............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9896
1021/1688 [=================>............] - ETA: 1s - loss: 0.0309 - accuracy: 0.9897
1048/1688 [=================>............] - ETA: 1s - loss: 0.0306 - accuracy: 0.9899
1076/1688 [==================>...........] - ETA: 1s - loss: 0.0308 - accuracy: 0.9897
1104/1688 [==================>...........] - ETA: 1s - loss: 0.0310 - accuracy: 0.9896
1133/1688 [===================>..........] - ETA: 0s - loss: 0.0311 - accuracy: 0.9896
1162/1688 [===================>..........] - ETA: 0s - loss: 0.0310 - accuracy: 0.9896
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0308 - accuracy: 0.9896
1218/1688 [====================>.........] - ETA: 0s - loss: 0.0306 - accuracy: 0.9896
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0309 - accuracy: 0.9896
1275/1688 [=====================>........] - ETA: 0s - loss: 0.0306 - accuracy: 0.9897
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0312 - accuracy: 0.9895
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0309 - accuracy: 0.9896
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0312 - accuracy: 0.9895
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0315 - accuracy: 0.9895
1416/1688 [========================>.....] - ETA: 0s - loss: 0.0316 - accuracy: 0.9895
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0319 - accuracy: 0.9895
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0322 - accuracy: 0.9894
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0323 - accuracy: 0.9894
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0327 - accuracy: 0.9893
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0329 - accuracy: 0.9892
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0332 - accuracy: 0.9891
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0332 - accuracy: 0.9892
1644/1688 [============================>.] - ETA: 0s - loss: 0.0332 - accuracy: 0.9891
1673/1688 [============================>.] - ETA: 0s - loss: 0.0331 - accuracy: 0.9891
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0332 - accuracy: 0.9891 - val_loss: 0.0600 - val_accuracy: 0.9848
Epoch 6/10
1/1688 [..............................] - ETA: 3s - loss: 0.0193 - accuracy: 1.0000
31/1688 [..............................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9950
60/1688 [>.............................] - ETA: 2s - loss: 0.0186 - accuracy: 0.9943
89/1688 [>.............................] - ETA: 2s - loss: 0.0190 - accuracy: 0.9944
117/1688 [=>............................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9931
145/1688 [=>............................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9925
173/1688 [==>...........................] - ETA: 2s - loss: 0.0222 - accuracy: 0.9924
202/1688 [==>...........................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9918
230/1688 [===>..........................] - ETA: 2s - loss: 0.0240 - accuracy: 0.9917
259/1688 [===>..........................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9919
287/1688 [====>.........................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9921
315/1688 [====>.........................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9918
344/1688 [=====>........................] - ETA: 2s - loss: 0.0243 - accuracy: 0.9917
373/1688 [=====>........................] - ETA: 2s - loss: 0.0243 - accuracy: 0.9916
402/1688 [======>.......................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9917
430/1688 [======>.......................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9915
459/1688 [=======>......................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9918
487/1688 [=======>......................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9919
515/1688 [========>.....................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9915
543/1688 [========>.....................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9918
570/1688 [=========>....................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9921
597/1688 [=========>....................] - ETA: 1s - loss: 0.0231 - accuracy: 0.9921
625/1688 [==========>...................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9922
654/1688 [==========>...................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9923
682/1688 [===========>..................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9920
710/1688 [===========>..................] - ETA: 1s - loss: 0.0245 - accuracy: 0.9918
738/1688 [============>.................] - ETA: 1s - loss: 0.0246 - accuracy: 0.9918
767/1688 [============>.................] - ETA: 1s - loss: 0.0245 - accuracy: 0.9917
795/1688 [=============>................] - ETA: 1s - loss: 0.0246 - accuracy: 0.9917
823/1688 [=============>................] - ETA: 1s - loss: 0.0249 - accuracy: 0.9917
851/1688 [==============>...............] - ETA: 1s - loss: 0.0252 - accuracy: 0.9917
879/1688 [==============>...............] - ETA: 1s - loss: 0.0256 - accuracy: 0.9915
908/1688 [===============>..............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9915
936/1688 [===============>..............] - ETA: 1s - loss: 0.0254 - accuracy: 0.9915
964/1688 [================>.............] - ETA: 1s - loss: 0.0257 - accuracy: 0.9914
992/1688 [================>.............] - ETA: 1s - loss: 0.0257 - accuracy: 0.9914
1021/1688 [=================>............] - ETA: 1s - loss: 0.0258 - accuracy: 0.9914
1050/1688 [=================>............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9915
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0252 - accuracy: 0.9916
1107/1688 [==================>...........] - ETA: 1s - loss: 0.0252 - accuracy: 0.9916
1135/1688 [===================>..........] - ETA: 0s - loss: 0.0254 - accuracy: 0.9915
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0257 - accuracy: 0.9915
1191/1688 [====================>.........] - ETA: 0s - loss: 0.0260 - accuracy: 0.9914
1219/1688 [====================>.........] - ETA: 0s - loss: 0.0261 - accuracy: 0.9913
1247/1688 [=====================>........] - ETA: 0s - loss: 0.0263 - accuracy: 0.9913
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0262 - accuracy: 0.9913
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0266 - accuracy: 0.9911
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0269 - accuracy: 0.9909
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9909
1387/1688 [=======================>......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9909
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0274 - accuracy: 0.9907
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0279 - accuracy: 0.9907
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0277 - accuracy: 0.9908
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0281 - accuracy: 0.9906
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0283 - accuracy: 0.9905
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0293 - accuracy: 0.9902
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0295 - accuracy: 0.9901
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0298 - accuracy: 0.9900
1641/1688 [============================>.] - ETA: 0s - loss: 0.0300 - accuracy: 0.9900
1669/1688 [============================>.] - ETA: 0s - loss: 0.0298 - accuracy: 0.9901
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0298 - accuracy: 0.9901 - val_loss: 0.0498 - val_accuracy: 0.9863
Epoch 7/10
1/1688 [..............................] - ETA: 3s - loss: 0.0012 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0229 - accuracy: 0.9922
60/1688 [>.............................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9932
89/1688 [>.............................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9947
118/1688 [=>............................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9950
147/1688 [=>............................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9945
175/1688 [==>...........................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9943
204/1688 [==>...........................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9946
232/1688 [===>..........................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9950
261/1688 [===>..........................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9952
289/1688 [====>.........................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9948
318/1688 [====>.........................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9947
346/1688 [=====>........................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9947
374/1688 [=====>........................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9944
401/1688 [======>.......................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9938
430/1688 [======>.......................] - ETA: 2s - loss: 0.0193 - accuracy: 0.9936
459/1688 [=======>......................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9936
487/1688 [=======>......................] - ETA: 2s - loss: 0.0194 - accuracy: 0.9937
516/1688 [========>.....................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9934
543/1688 [========>.....................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9932
572/1688 [=========>....................] - ETA: 1s - loss: 0.0222 - accuracy: 0.9927
600/1688 [=========>....................] - ETA: 1s - loss: 0.0232 - accuracy: 0.9923
629/1688 [==========>...................] - ETA: 1s - loss: 0.0231 - accuracy: 0.9923
657/1688 [==========>...................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9924
685/1688 [===========>..................] - ETA: 1s - loss: 0.0234 - accuracy: 0.9923
714/1688 [===========>..................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9922
743/1688 [============>.................] - ETA: 1s - loss: 0.0238 - accuracy: 0.9922
771/1688 [============>.................] - ETA: 1s - loss: 0.0234 - accuracy: 0.9923
799/1688 [=============>................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9921
827/1688 [=============>................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9921
855/1688 [==============>...............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9923
883/1688 [==============>...............] - ETA: 1s - loss: 0.0231 - accuracy: 0.9924
911/1688 [===============>..............] - ETA: 1s - loss: 0.0230 - accuracy: 0.9924
940/1688 [===============>..............] - ETA: 1s - loss: 0.0229 - accuracy: 0.9925
968/1688 [================>.............] - ETA: 1s - loss: 0.0227 - accuracy: 0.9926
996/1688 [================>.............] - ETA: 1s - loss: 0.0224 - accuracy: 0.9927
1025/1688 [=================>............] - ETA: 1s - loss: 0.0222 - accuracy: 0.9927
1053/1688 [=================>............] - ETA: 1s - loss: 0.0223 - accuracy: 0.9927
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0224 - accuracy: 0.9926
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0224 - accuracy: 0.9926
1137/1688 [===================>..........] - ETA: 0s - loss: 0.0222 - accuracy: 0.9926
1165/1688 [===================>..........] - ETA: 0s - loss: 0.0224 - accuracy: 0.9925
1193/1688 [====================>.........] - ETA: 0s - loss: 0.0226 - accuracy: 0.9925
1221/1688 [====================>.........] - ETA: 0s - loss: 0.0226 - accuracy: 0.9924
1249/1688 [=====================>........] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0231 - accuracy: 0.9923
1391/1688 [=======================>......] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1419/1688 [========================>.....] - ETA: 0s - loss: 0.0235 - accuracy: 0.9921
1448/1688 [========================>.....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9920
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9920
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0237 - accuracy: 0.9920
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0238 - accuracy: 0.9920
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0240 - accuracy: 0.9919
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0240 - accuracy: 0.9919
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0241 - accuracy: 0.9918
1646/1688 [============================>.] - ETA: 0s - loss: 0.0241 - accuracy: 0.9918
1674/1688 [============================>.] - ETA: 0s - loss: 0.0242 - accuracy: 0.9918
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0243 - accuracy: 0.9918 - val_loss: 0.0545 - val_accuracy: 0.9878
Epoch 8/10
1/1688 [..............................] - ETA: 3s - loss: 0.0141 - accuracy: 1.0000
29/1688 [..............................] - ETA: 2s - loss: 0.0237 - accuracy: 0.9935
58/1688 [>.............................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9952
86/1688 [>.............................] - ETA: 2s - loss: 0.0174 - accuracy: 0.9960
114/1688 [=>............................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9962
142/1688 [=>............................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9963
170/1688 [==>...........................] - ETA: 2s - loss: 0.0175 - accuracy: 0.9960
198/1688 [==>...........................] - ETA: 2s - loss: 0.0169 - accuracy: 0.9957
226/1688 [===>..........................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9952
254/1688 [===>..........................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
281/1688 [===>..........................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9958
310/1688 [====>.........................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9957
339/1688 [=====>........................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9953
368/1688 [=====>........................] - ETA: 2s - loss: 0.0157 - accuracy: 0.9952
396/1688 [======>.......................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9952
425/1688 [======>.......................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9949
453/1688 [=======>......................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9948
481/1688 [=======>......................] - ETA: 2s - loss: 0.0168 - accuracy: 0.9949
509/1688 [========>.....................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9947
537/1688 [========>.....................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9944
565/1688 [=========>....................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9942
593/1688 [=========>....................] - ETA: 1s - loss: 0.0182 - accuracy: 0.9941
622/1688 [==========>...................] - ETA: 1s - loss: 0.0184 - accuracy: 0.9939
651/1688 [==========>...................] - ETA: 1s - loss: 0.0186 - accuracy: 0.9939
680/1688 [===========>..................] - ETA: 1s - loss: 0.0188 - accuracy: 0.9938
708/1688 [===========>..................] - ETA: 1s - loss: 0.0186 - accuracy: 0.9939
736/1688 [============>.................] - ETA: 1s - loss: 0.0184 - accuracy: 0.9940
764/1688 [============>.................] - ETA: 1s - loss: 0.0180 - accuracy: 0.9940
792/1688 [=============>................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9941
820/1688 [=============>................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9941
848/1688 [==============>...............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9941
876/1688 [==============>...............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9942
905/1688 [===============>..............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9943
932/1688 [===============>..............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9942
961/1688 [================>.............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9941
990/1688 [================>.............] - ETA: 1s - loss: 0.0180 - accuracy: 0.9940
1019/1688 [=================>............] - ETA: 1s - loss: 0.0182 - accuracy: 0.9940
1048/1688 [=================>............] - ETA: 1s - loss: 0.0183 - accuracy: 0.9939
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0184 - accuracy: 0.9938
1106/1688 [==================>...........] - ETA: 1s - loss: 0.0182 - accuracy: 0.9938
1134/1688 [===================>..........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9937
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9936
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9936
1218/1688 [====================>.........] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0184 - accuracy: 0.9937
1274/1688 [=====================>........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9936
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0185 - accuracy: 0.9936
1331/1688 [======================>.......] - ETA: 0s - loss: 0.0184 - accuracy: 0.9936
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0184 - accuracy: 0.9936
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0185 - accuracy: 0.9935
1416/1688 [========================>.....] - ETA: 0s - loss: 0.0187 - accuracy: 0.9935
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0190 - accuracy: 0.9934
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0192 - accuracy: 0.9933
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0196 - accuracy: 0.9933
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0194 - accuracy: 0.9933
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0195 - accuracy: 0.9933
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0194 - accuracy: 0.9933
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0196 - accuracy: 0.9933
1642/1688 [============================>.] - ETA: 0s - loss: 0.0195 - accuracy: 0.9933
1670/1688 [============================>.] - ETA: 0s - loss: 0.0195 - accuracy: 0.9933
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0195 - accuracy: 0.9933 - val_loss: 0.0601 - val_accuracy: 0.9850
Epoch 9/10
1/1688 [..............................] - ETA: 3s - loss: 0.0033 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9912
62/1688 [>.............................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9914
91/1688 [>.............................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9921
120/1688 [=>............................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9935
148/1688 [=>............................] - ETA: 2s - loss: 0.0168 - accuracy: 0.9943
176/1688 [==>...........................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9949
204/1688 [==>...........................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9953
232/1688 [===>..........................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9958
260/1688 [===>..........................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9959
288/1688 [====>.........................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9959
317/1688 [====>.........................] - ETA: 2s - loss: 0.0127 - accuracy: 0.9960
345/1688 [=====>........................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9961
374/1688 [=====>........................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9959
402/1688 [======>.......................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9961
430/1688 [======>.......................] - ETA: 2s - loss: 0.0121 - accuracy: 0.9961
458/1688 [=======>......................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9960
486/1688 [=======>......................] - ETA: 2s - loss: 0.0123 - accuracy: 0.9959
515/1688 [========>.....................] - ETA: 2s - loss: 0.0127 - accuracy: 0.9956
543/1688 [========>.....................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9955
571/1688 [=========>....................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9955
599/1688 [=========>....................] - ETA: 1s - loss: 0.0138 - accuracy: 0.9952
627/1688 [==========>...................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9950
656/1688 [==========>...................] - ETA: 1s - loss: 0.0147 - accuracy: 0.9950
685/1688 [===========>..................] - ETA: 1s - loss: 0.0152 - accuracy: 0.9947
713/1688 [===========>..................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9947
742/1688 [============>.................] - ETA: 1s - loss: 0.0155 - accuracy: 0.9946
770/1688 [============>.................] - ETA: 1s - loss: 0.0161 - accuracy: 0.9944
799/1688 [=============>................] - ETA: 1s - loss: 0.0162 - accuracy: 0.9944
827/1688 [=============>................] - ETA: 1s - loss: 0.0164 - accuracy: 0.9944
855/1688 [==============>...............] - ETA: 1s - loss: 0.0167 - accuracy: 0.9942
884/1688 [==============>...............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9943
914/1688 [===============>..............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9943
944/1688 [===============>..............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9942
972/1688 [================>.............] - ETA: 1s - loss: 0.0170 - accuracy: 0.9942
1001/1688 [================>.............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9941
1029/1688 [=================>............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9940
1057/1688 [=================>............] - ETA: 1s - loss: 0.0176 - accuracy: 0.9939
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9939
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0176 - accuracy: 0.9939
1143/1688 [===================>..........] - ETA: 0s - loss: 0.0176 - accuracy: 0.9939
1171/1688 [===================>..........] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1200/1688 [====================>.........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1229/1688 [====================>.........] - ETA: 0s - loss: 0.0179 - accuracy: 0.9939
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1315/1688 [======================>.......] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0176 - accuracy: 0.9939
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0176 - accuracy: 0.9940
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0178 - accuracy: 0.9939
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0178 - accuracy: 0.9938
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0177 - accuracy: 0.9938
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0175 - accuracy: 0.9939
1568/1688 [==========================>...] - ETA: 0s - loss: 0.0175 - accuracy: 0.9939
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0177 - accuracy: 0.9939
1653/1688 [============================>.] - ETA: 0s - loss: 0.0180 - accuracy: 0.9938
1681/1688 [============================>.] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0183 - accuracy: 0.9937 - val_loss: 0.0753 - val_accuracy: 0.9833
Epoch 10/10
1/1688 [..............................] - ETA: 3s - loss: 1.9658e-04 - accuracy: 1.0000
30/1688 [..............................] - ETA: 2s - loss: 0.0093 - accuracy: 0.9958
58/1688 [>.............................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9968
87/1688 [>.............................] - ETA: 2s - loss: 0.0107 - accuracy: 0.9968
116/1688 [=>............................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9968
143/1688 [=>............................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9967
171/1688 [==>...........................] - ETA: 2s - loss: 0.0095 - accuracy: 0.9971
200/1688 [==>...........................] - ETA: 2s - loss: 0.0097 - accuracy: 0.9970
228/1688 [===>..........................] - ETA: 2s - loss: 0.0094 - accuracy: 0.9971
256/1688 [===>..........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9968
285/1688 [====>.........................] - ETA: 2s - loss: 0.0102 - accuracy: 0.9967
313/1688 [====>.........................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9964
341/1688 [=====>........................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9964
370/1688 [=====>........................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9965
398/1688 [======>.......................] - ETA: 2s - loss: 0.0104 - accuracy: 0.9966
426/1688 [======>.......................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9966
454/1688 [=======>......................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9963
483/1688 [=======>......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9961
512/1688 [========>.....................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9960
541/1688 [========>.....................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9958
570/1688 [=========>....................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9958
598/1688 [=========>....................] - ETA: 1s - loss: 0.0125 - accuracy: 0.9957
626/1688 [==========>...................] - ETA: 1s - loss: 0.0126 - accuracy: 0.9956
655/1688 [==========>...................] - ETA: 1s - loss: 0.0127 - accuracy: 0.9956
683/1688 [===========>..................] - ETA: 1s - loss: 0.0125 - accuracy: 0.9957
711/1688 [===========>..................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
739/1688 [============>.................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
768/1688 [============>.................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
796/1688 [=============>................] - ETA: 1s - loss: 0.0135 - accuracy: 0.9952
825/1688 [=============>................] - ETA: 1s - loss: 0.0132 - accuracy: 0.9953
855/1688 [==============>...............] - ETA: 1s - loss: 0.0133 - accuracy: 0.9953
884/1688 [==============>...............] - ETA: 1s - loss: 0.0130 - accuracy: 0.9954
911/1688 [===============>..............] - ETA: 1s - loss: 0.0132 - accuracy: 0.9953
938/1688 [===============>..............] - ETA: 1s - loss: 0.0131 - accuracy: 0.9953
967/1688 [================>.............] - ETA: 1s - loss: 0.0131 - accuracy: 0.9953
995/1688 [================>.............] - ETA: 1s - loss: 0.0130 - accuracy: 0.9954
1023/1688 [=================>............] - ETA: 1s - loss: 0.0131 - accuracy: 0.9953
1051/1688 [=================>............] - ETA: 1s - loss: 0.0132 - accuracy: 0.9952
1079/1688 [==================>...........] - ETA: 1s - loss: 0.0133 - accuracy: 0.9951
1107/1688 [==================>...........] - ETA: 1s - loss: 0.0137 - accuracy: 0.9950
1135/1688 [===================>..........] - ETA: 0s - loss: 0.0137 - accuracy: 0.9950
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0138 - accuracy: 0.9949
1192/1688 [====================>.........] - ETA: 0s - loss: 0.0141 - accuracy: 0.9949
1221/1688 [====================>.........] - ETA: 0s - loss: 0.0142 - accuracy: 0.9949
1249/1688 [=====================>........] - ETA: 0s - loss: 0.0141 - accuracy: 0.9949
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0142 - accuracy: 0.9949
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0142 - accuracy: 0.9948
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0142 - accuracy: 0.9948
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0141 - accuracy: 0.9949
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1420/1688 [========================>.....] - ETA: 0s - loss: 0.0142 - accuracy: 0.9950
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0143 - accuracy: 0.9950
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0144 - accuracy: 0.9949
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0143 - accuracy: 0.9949
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0144 - accuracy: 0.9949
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0146 - accuracy: 0.9948
1649/1688 [============================>.] - ETA: 0s - loss: 0.0149 - accuracy: 0.9947
1678/1688 [============================>.] - ETA: 0s - loss: 0.0152 - accuracy: 0.9946
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0151 - accuracy: 0.9946 - val_loss: 0.0711 - val_accuracy: 0.9838
Test accuracy: 0.9842000007629395
4. Model quantization
We can now turn to quantization to get a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.
For this, we just have to quantize the Keras model using the quantize function. Here, we decide to quantize to the maximum allowed bitwidths for the first layer weights (8-bit), the subsequent layer weights (4-bit) and the activations (4-bit).
The quantized model is a Keras model where the neural layers (Conv2D, Dense)
and the ReLU layers are replaced with custom CNN2SNN quantized layers
(QuantizedConv2D, QuantizedDense, QuantizedReLU). All Keras API functions
can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function folds the batch normalization layers into
the corresponding neural layer. The new weights are computed
according to this folding operation.
from cnn2snn import quantize
model_quantized = quantize(model_keras,
input_weight_quantization=8,
weight_quantization=4,
activ_quantization=4)
model_quantized.summary()
Out:
Model: "sequential_48"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (QuantizedConv2D) (None, 13, 13, 32) 320
re_lu (QuantizedReLU) (None, 13, 13, 32) 0
separable_conv2d (Quantized (None, 7, 7, 64) 2400
SeparableConv2D)
re_lu_1 (QuantizedReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (QuantizedDense) (None, 10) 31370
=================================================================
Total params: 34,090
Trainable params: 34,090
Non-trainable params: 0
_________________________________________________________________
Check the quantized model accuracy.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after 8-4-4 quantization:', score[1])
Out:
Test accuracy after 8-4-4 quantization: 0.8953999876976013
Since we used the maximum allowed bitwidths for weights and activations, the accuracy of the quantized model is equivalent to the one of the base model, but for lower bitwidth, the quantization usually introduces a performance drop.
Let’s try to quantize specific layers to a lower bitwidth. The CNN2SNN toolkit provides the quantize_layer function: each layer can be individually quantized.
Here, we quantize the “re_lu_1” layer to binary activations (bitwidth=1) and the “dense” layer with 2-bit weights.
from cnn2snn import quantize_layer
model_quantized = quantize_layer(model_quantized, "re_lu_1", bitwidth=1)
model_quantized = quantize_layer(model_quantized, "dense", bitwidth=2)
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer='adam',
metrics=['accuracy'])
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after low bitwidth quantization:', score[1])
# To recover the original model accuracy, a quantization-aware training phase
# is required.
Out:
Test accuracy after low bitwidth quantization: 0.11349999904632568
5. Model fine tuning (quantization-aware training)
This quantization-aware training (fine tuning) allows to cover the performance drop due to the quantization step.
Note that since this step is a fine tuning, the number of epochs can be lowered, compared to the training from scratch of the standard model.
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
score = model_quantized.evaluate(x_test, y_test, verbose=0)
print('Test accuracy after fine tuning:', score[1])
Out:
Epoch 1/5
1/1688 [..............................] - ETA: 17:11 - loss: 2.3145 - accuracy: 0.0625
21/1688 [..............................] - ETA: 4s - loss: 2.3020 - accuracy: 0.0893
41/1688 [..............................] - ETA: 4s - loss: 2.2981 - accuracy: 0.0983
61/1688 [>.............................] - ETA: 4s - loss: 2.3024 - accuracy: 0.1071
81/1688 [>.............................] - ETA: 4s - loss: 2.3132 - accuracy: 0.1100
101/1688 [>.............................] - ETA: 4s - loss: 2.3191 - accuracy: 0.1281
122/1688 [=>............................] - ETA: 4s - loss: 2.2941 - accuracy: 0.1601
142/1688 [=>............................] - ETA: 3s - loss: 2.2519 - accuracy: 0.2168
162/1688 [=>............................] - ETA: 3s - loss: 2.2677 - accuracy: 0.2731
181/1688 [==>...........................] - ETA: 3s - loss: 2.2224 - accuracy: 0.3225
201/1688 [==>...........................] - ETA: 3s - loss: 2.1318 - accuracy: 0.3752
221/1688 [==>...........................] - ETA: 3s - loss: 2.0567 - accuracy: 0.4164
241/1688 [===>..........................] - ETA: 3s - loss: 1.9687 - accuracy: 0.4557
261/1688 [===>..........................] - ETA: 3s - loss: 1.8924 - accuracy: 0.4892
281/1688 [===>..........................] - ETA: 3s - loss: 1.8334 - accuracy: 0.5180
302/1688 [====>.........................] - ETA: 3s - loss: 1.7959 - accuracy: 0.5435
322/1688 [====>.........................] - ETA: 3s - loss: 1.7547 - accuracy: 0.5656
342/1688 [=====>........................] - ETA: 3s - loss: 1.7165 - accuracy: 0.5850
362/1688 [=====>........................] - ETA: 3s - loss: 1.7029 - accuracy: 0.6024
382/1688 [=====>........................] - ETA: 3s - loss: 1.6739 - accuracy: 0.6180
402/1688 [======>.......................] - ETA: 3s - loss: 1.6314 - accuracy: 0.6328
422/1688 [======>.......................] - ETA: 3s - loss: 1.5961 - accuracy: 0.6467
442/1688 [======>.......................] - ETA: 3s - loss: 1.5635 - accuracy: 0.6589
462/1688 [=======>......................] - ETA: 3s - loss: 1.5372 - accuracy: 0.6707
482/1688 [=======>......................] - ETA: 3s - loss: 1.5189 - accuracy: 0.6809
502/1688 [=======>......................] - ETA: 3s - loss: 1.4906 - accuracy: 0.6911
523/1688 [========>.....................] - ETA: 2s - loss: 1.4735 - accuracy: 0.7003
544/1688 [========>.....................] - ETA: 2s - loss: 1.4433 - accuracy: 0.7092
564/1688 [=========>....................] - ETA: 2s - loss: 1.4293 - accuracy: 0.7173
584/1688 [=========>....................] - ETA: 2s - loss: 1.4115 - accuracy: 0.7247
604/1688 [=========>....................] - ETA: 2s - loss: 1.3960 - accuracy: 0.7320
624/1688 [==========>...................] - ETA: 2s - loss: 1.3868 - accuracy: 0.7381
644/1688 [==========>...................] - ETA: 2s - loss: 1.3730 - accuracy: 0.7443
664/1688 [==========>...................] - ETA: 2s - loss: 1.3562 - accuracy: 0.7500
684/1688 [===========>..................] - ETA: 2s - loss: 1.3373 - accuracy: 0.7555
704/1688 [===========>..................] - ETA: 2s - loss: 1.3154 - accuracy: 0.7610
724/1688 [===========>..................] - ETA: 2s - loss: 1.3121 - accuracy: 0.7656
744/1688 [============>.................] - ETA: 2s - loss: 1.3090 - accuracy: 0.7702
764/1688 [============>.................] - ETA: 2s - loss: 1.3034 - accuracy: 0.7744
784/1688 [============>.................] - ETA: 2s - loss: 1.2915 - accuracy: 0.7787
804/1688 [=============>................] - ETA: 2s - loss: 1.2876 - accuracy: 0.7825
825/1688 [=============>................] - ETA: 2s - loss: 1.2719 - accuracy: 0.7868
845/1688 [==============>...............] - ETA: 2s - loss: 1.2632 - accuracy: 0.7902
865/1688 [==============>...............] - ETA: 2s - loss: 1.2580 - accuracy: 0.7934
885/1688 [==============>...............] - ETA: 2s - loss: 1.2475 - accuracy: 0.7966
906/1688 [===============>..............] - ETA: 1s - loss: 1.2366 - accuracy: 0.8001
926/1688 [===============>..............] - ETA: 1s - loss: 1.2336 - accuracy: 0.8032
946/1688 [===============>..............] - ETA: 1s - loss: 1.2305 - accuracy: 0.8060
967/1688 [================>.............] - ETA: 1s - loss: 1.2250 - accuracy: 0.8088
987/1688 [================>.............] - ETA: 1s - loss: 1.2157 - accuracy: 0.8113
1007/1688 [================>.............] - ETA: 1s - loss: 1.2075 - accuracy: 0.8138
1027/1688 [=================>............] - ETA: 1s - loss: 1.2021 - accuracy: 0.8164
1047/1688 [=================>............] - ETA: 1s - loss: 1.1930 - accuracy: 0.8189
1067/1688 [=================>............] - ETA: 1s - loss: 1.1841 - accuracy: 0.8214
1087/1688 [==================>...........] - ETA: 1s - loss: 1.1747 - accuracy: 0.8238
1107/1688 [==================>...........] - ETA: 1s - loss: 1.1688 - accuracy: 0.8261
1127/1688 [===================>..........] - ETA: 1s - loss: 1.1581 - accuracy: 0.8284
1147/1688 [===================>..........] - ETA: 1s - loss: 1.1557 - accuracy: 0.8305
1168/1688 [===================>..........] - ETA: 1s - loss: 1.1525 - accuracy: 0.8326
1188/1688 [====================>.........] - ETA: 1s - loss: 1.1479 - accuracy: 0.8346
1208/1688 [====================>.........] - ETA: 1s - loss: 1.1409 - accuracy: 0.8366
1228/1688 [====================>.........] - ETA: 1s - loss: 1.1346 - accuracy: 0.8384
1248/1688 [=====================>........] - ETA: 1s - loss: 1.1243 - accuracy: 0.8404
1268/1688 [=====================>........] - ETA: 1s - loss: 1.1208 - accuracy: 0.8423
1288/1688 [=====================>........] - ETA: 1s - loss: 1.1216 - accuracy: 0.8439
1308/1688 [======================>.......] - ETA: 0s - loss: 1.1266 - accuracy: 0.8453
1328/1688 [======================>.......] - ETA: 0s - loss: 1.1259 - accuracy: 0.8466
1348/1688 [======================>.......] - ETA: 0s - loss: 1.1257 - accuracy: 0.8479
1368/1688 [=======================>......] - ETA: 0s - loss: 1.1213 - accuracy: 0.8493
1388/1688 [=======================>......] - ETA: 0s - loss: 1.1161 - accuracy: 0.8507
1408/1688 [========================>.....] - ETA: 0s - loss: 1.1116 - accuracy: 0.8521
1428/1688 [========================>.....] - ETA: 0s - loss: 1.1113 - accuracy: 0.8532
1448/1688 [========================>.....] - ETA: 0s - loss: 1.1020 - accuracy: 0.8546
1468/1688 [=========================>....] - ETA: 0s - loss: 1.1043 - accuracy: 0.8556
1488/1688 [=========================>....] - ETA: 0s - loss: 1.1047 - accuracy: 0.8566
1508/1688 [=========================>....] - ETA: 0s - loss: 1.0998 - accuracy: 0.8580
1528/1688 [==========================>...] - ETA: 0s - loss: 1.0984 - accuracy: 0.8591
1548/1688 [==========================>...] - ETA: 0s - loss: 1.0942 - accuracy: 0.8604
1569/1688 [==========================>...] - ETA: 0s - loss: 1.0947 - accuracy: 0.8615
1589/1688 [===========================>..] - ETA: 0s - loss: 1.0924 - accuracy: 0.8625
1609/1688 [===========================>..] - ETA: 0s - loss: 1.0858 - accuracy: 0.8636
1630/1688 [===========================>..] - ETA: 0s - loss: 1.0832 - accuracy: 0.8647
1650/1688 [============================>.] - ETA: 0s - loss: 1.0830 - accuracy: 0.8657
1670/1688 [============================>.] - ETA: 0s - loss: 1.0794 - accuracy: 0.8668
1688/1688 [==============================] - 5s 3ms/step - loss: 1.0765 - accuracy: 0.8678 - val_loss: 0.8484 - val_accuracy: 0.9598
Epoch 2/5
1/1688 [..............................] - ETA: 4s - loss: 0.0000e+00 - accuracy: 1.0000
22/1688 [..............................] - ETA: 4s - loss: 0.7210 - accuracy: 0.9588
42/1688 [..............................] - ETA: 4s - loss: 0.8106 - accuracy: 0.9583
62/1688 [>.............................] - ETA: 4s - loss: 0.6682 - accuracy: 0.9627
82/1688 [>.............................] - ETA: 4s - loss: 0.7188 - accuracy: 0.9615
102/1688 [>.............................] - ETA: 4s - loss: 0.7400 - accuracy: 0.9599
123/1688 [=>............................] - ETA: 3s - loss: 0.7327 - accuracy: 0.9596
143/1688 [=>............................] - ETA: 3s - loss: 0.7188 - accuracy: 0.9583
163/1688 [=>............................] - ETA: 3s - loss: 0.7522 - accuracy: 0.9576
183/1688 [==>...........................] - ETA: 3s - loss: 0.7663 - accuracy: 0.9570
203/1688 [==>...........................] - ETA: 3s - loss: 0.7656 - accuracy: 0.9577
223/1688 [==>...........................] - ETA: 3s - loss: 0.7771 - accuracy: 0.9577
243/1688 [===>..........................] - ETA: 3s - loss: 0.7791 - accuracy: 0.9573
263/1688 [===>..........................] - ETA: 3s - loss: 0.7750 - accuracy: 0.9573
283/1688 [====>.........................] - ETA: 3s - loss: 0.7925 - accuracy: 0.9563
303/1688 [====>.........................] - ETA: 3s - loss: 0.7715 - accuracy: 0.9570
323/1688 [====>.........................] - ETA: 3s - loss: 0.7725 - accuracy: 0.9576
343/1688 [=====>........................] - ETA: 3s - loss: 0.7645 - accuracy: 0.9583
363/1688 [=====>........................] - ETA: 3s - loss: 0.7737 - accuracy: 0.9576
383/1688 [=====>........................] - ETA: 3s - loss: 0.7691 - accuracy: 0.9577
404/1688 [======>.......................] - ETA: 3s - loss: 0.7777 - accuracy: 0.9570
424/1688 [======>.......................] - ETA: 3s - loss: 0.7673 - accuracy: 0.9575
444/1688 [======>.......................] - ETA: 3s - loss: 0.7614 - accuracy: 0.9578
464/1688 [=======>......................] - ETA: 3s - loss: 0.7652 - accuracy: 0.9580
484/1688 [=======>......................] - ETA: 3s - loss: 0.7495 - accuracy: 0.9585
504/1688 [=======>......................] - ETA: 3s - loss: 0.7369 - accuracy: 0.9586
524/1688 [========>.....................] - ETA: 2s - loss: 0.7409 - accuracy: 0.9585
545/1688 [========>.....................] - ETA: 2s - loss: 0.7471 - accuracy: 0.9582
565/1688 [=========>....................] - ETA: 2s - loss: 0.7492 - accuracy: 0.9582
586/1688 [=========>....................] - ETA: 2s - loss: 0.7390 - accuracy: 0.9587
606/1688 [=========>....................] - ETA: 2s - loss: 0.7392 - accuracy: 0.9587
626/1688 [==========>...................] - ETA: 2s - loss: 0.7301 - accuracy: 0.9588
646/1688 [==========>...................] - ETA: 2s - loss: 0.7410 - accuracy: 0.9588
666/1688 [==========>...................] - ETA: 2s - loss: 0.7344 - accuracy: 0.9587
687/1688 [===========>..................] - ETA: 2s - loss: 0.7415 - accuracy: 0.9582
707/1688 [===========>..................] - ETA: 2s - loss: 0.7463 - accuracy: 0.9583
727/1688 [===========>..................] - ETA: 2s - loss: 0.7462 - accuracy: 0.9583
748/1688 [============>.................] - ETA: 2s - loss: 0.7415 - accuracy: 0.9584
768/1688 [============>.................] - ETA: 2s - loss: 0.7448 - accuracy: 0.9585
788/1688 [=============>................] - ETA: 2s - loss: 0.7505 - accuracy: 0.9582
808/1688 [=============>................] - ETA: 2s - loss: 0.7479 - accuracy: 0.9582
828/1688 [=============>................] - ETA: 2s - loss: 0.7507 - accuracy: 0.9582
848/1688 [==============>...............] - ETA: 2s - loss: 0.7478 - accuracy: 0.9584
868/1688 [==============>...............] - ETA: 2s - loss: 0.7410 - accuracy: 0.9588
888/1688 [==============>...............] - ETA: 2s - loss: 0.7427 - accuracy: 0.9588
908/1688 [===============>..............] - ETA: 1s - loss: 0.7432 - accuracy: 0.9589
928/1688 [===============>..............] - ETA: 1s - loss: 0.7418 - accuracy: 0.9589
949/1688 [===============>..............] - ETA: 1s - loss: 0.7402 - accuracy: 0.9590
969/1688 [================>.............] - ETA: 1s - loss: 0.7371 - accuracy: 0.9592
989/1688 [================>.............] - ETA: 1s - loss: 0.7388 - accuracy: 0.9592
1009/1688 [================>.............] - ETA: 1s - loss: 0.7371 - accuracy: 0.9591
1029/1688 [=================>............] - ETA: 1s - loss: 0.7390 - accuracy: 0.9592
1049/1688 [=================>............] - ETA: 1s - loss: 0.7354 - accuracy: 0.9595
1069/1688 [=================>............] - ETA: 1s - loss: 0.7329 - accuracy: 0.9596
1089/1688 [==================>...........] - ETA: 1s - loss: 0.7297 - accuracy: 0.9598
1109/1688 [==================>...........] - ETA: 1s - loss: 0.7314 - accuracy: 0.9599
1129/1688 [===================>..........] - ETA: 1s - loss: 0.7303 - accuracy: 0.9599
1149/1688 [===================>..........] - ETA: 1s - loss: 0.7285 - accuracy: 0.9599
1169/1688 [===================>..........] - ETA: 1s - loss: 0.7220 - accuracy: 0.9602
1189/1688 [====================>.........] - ETA: 1s - loss: 0.7174 - accuracy: 0.9603
1209/1688 [====================>.........] - ETA: 1s - loss: 0.7144 - accuracy: 0.9603
1230/1688 [====================>.........] - ETA: 1s - loss: 0.7150 - accuracy: 0.9602
1250/1688 [=====================>........] - ETA: 1s - loss: 0.7074 - accuracy: 0.9604
1270/1688 [=====================>........] - ETA: 1s - loss: 0.7127 - accuracy: 0.9603
1290/1688 [=====================>........] - ETA: 1s - loss: 0.7091 - accuracy: 0.9605
1310/1688 [======================>.......] - ETA: 0s - loss: 0.7057 - accuracy: 0.9606
1330/1688 [======================>.......] - ETA: 0s - loss: 0.7055 - accuracy: 0.9606
1350/1688 [======================>.......] - ETA: 0s - loss: 0.7041 - accuracy: 0.9607
1371/1688 [=======================>......] - ETA: 0s - loss: 0.7011 - accuracy: 0.9609
1391/1688 [=======================>......] - ETA: 0s - loss: 0.7076 - accuracy: 0.9608
1411/1688 [========================>.....] - ETA: 0s - loss: 0.7098 - accuracy: 0.9606
1431/1688 [========================>.....] - ETA: 0s - loss: 0.7112 - accuracy: 0.9607
1451/1688 [========================>.....] - ETA: 0s - loss: 0.7084 - accuracy: 0.9607
1471/1688 [=========================>....] - ETA: 0s - loss: 0.7063 - accuracy: 0.9608
1491/1688 [=========================>....] - ETA: 0s - loss: 0.7080 - accuracy: 0.9609
1511/1688 [=========================>....] - ETA: 0s - loss: 0.7091 - accuracy: 0.9610
1531/1688 [==========================>...] - ETA: 0s - loss: 0.7078 - accuracy: 0.9611
1551/1688 [==========================>...] - ETA: 0s - loss: 0.7059 - accuracy: 0.9611
1572/1688 [==========================>...] - ETA: 0s - loss: 0.7064 - accuracy: 0.9611
1592/1688 [===========================>..] - ETA: 0s - loss: 0.7072 - accuracy: 0.9611
1612/1688 [===========================>..] - ETA: 0s - loss: 0.7077 - accuracy: 0.9610
1632/1688 [============================>.] - ETA: 0s - loss: 0.7058 - accuracy: 0.9611
1652/1688 [============================>.] - ETA: 0s - loss: 0.7018 - accuracy: 0.9613
1673/1688 [============================>.] - ETA: 0s - loss: 0.7016 - accuracy: 0.9613
1688/1688 [==============================] - 5s 3ms/step - loss: 0.7009 - accuracy: 0.9613 - val_loss: 0.7160 - val_accuracy: 0.9697
Epoch 3/5
1/1688 [..............................] - ETA: 4s - loss: 0.1883 - accuracy: 0.9688
22/1688 [..............................] - ETA: 4s - loss: 0.6565 - accuracy: 0.9688
42/1688 [..............................] - ETA: 4s - loss: 0.5602 - accuracy: 0.9643
62/1688 [>.............................] - ETA: 4s - loss: 0.5310 - accuracy: 0.9672
82/1688 [>.............................] - ETA: 4s - loss: 0.5457 - accuracy: 0.9672
102/1688 [>.............................] - ETA: 4s - loss: 0.5406 - accuracy: 0.9688
122/1688 [=>............................] - ETA: 3s - loss: 0.5385 - accuracy: 0.9685
142/1688 [=>............................] - ETA: 3s - loss: 0.5114 - accuracy: 0.9690
162/1688 [=>............................] - ETA: 3s - loss: 0.5002 - accuracy: 0.9697
182/1688 [==>...........................] - ETA: 3s - loss: 0.5502 - accuracy: 0.9688
202/1688 [==>...........................] - ETA: 3s - loss: 0.5488 - accuracy: 0.9678
223/1688 [==>...........................] - ETA: 3s - loss: 0.5342 - accuracy: 0.9678
243/1688 [===>..........................] - ETA: 3s - loss: 0.5491 - accuracy: 0.9672
263/1688 [===>..........................] - ETA: 3s - loss: 0.5358 - accuracy: 0.9677
283/1688 [====>.........................] - ETA: 3s - loss: 0.5583 - accuracy: 0.9671
303/1688 [====>.........................] - ETA: 3s - loss: 0.5584 - accuracy: 0.9667
323/1688 [====>.........................] - ETA: 3s - loss: 0.5719 - accuracy: 0.9665
342/1688 [=====>........................] - ETA: 3s - loss: 0.5684 - accuracy: 0.9666
362/1688 [=====>........................] - ETA: 3s - loss: 0.5695 - accuracy: 0.9672
382/1688 [=====>........................] - ETA: 3s - loss: 0.5527 - accuracy: 0.9678
402/1688 [======>.......................] - ETA: 3s - loss: 0.5531 - accuracy: 0.9681
422/1688 [======>.......................] - ETA: 3s - loss: 0.5607 - accuracy: 0.9681
442/1688 [======>.......................] - ETA: 3s - loss: 0.5655 - accuracy: 0.9679
462/1688 [=======>......................] - ETA: 3s - loss: 0.5697 - accuracy: 0.9680
483/1688 [=======>......................] - ETA: 3s - loss: 0.5859 - accuracy: 0.9674
503/1688 [=======>......................] - ETA: 3s - loss: 0.5946 - accuracy: 0.9676
523/1688 [========>.....................] - ETA: 2s - loss: 0.5937 - accuracy: 0.9679
543/1688 [========>.....................] - ETA: 2s - loss: 0.5862 - accuracy: 0.9682
563/1688 [=========>....................] - ETA: 2s - loss: 0.5923 - accuracy: 0.9684
584/1688 [=========>....................] - ETA: 2s - loss: 0.6046 - accuracy: 0.9681
604/1688 [=========>....................] - ETA: 2s - loss: 0.6073 - accuracy: 0.9679
624/1688 [==========>...................] - ETA: 2s - loss: 0.6082 - accuracy: 0.9679
644/1688 [==========>...................] - ETA: 2s - loss: 0.6189 - accuracy: 0.9676
664/1688 [==========>...................] - ETA: 2s - loss: 0.6172 - accuracy: 0.9676
684/1688 [===========>..................] - ETA: 2s - loss: 0.6253 - accuracy: 0.9673
704/1688 [===========>..................] - ETA: 2s - loss: 0.6308 - accuracy: 0.9674
724/1688 [===========>..................] - ETA: 2s - loss: 0.6236 - accuracy: 0.9676
744/1688 [============>.................] - ETA: 2s - loss: 0.6237 - accuracy: 0.9675
765/1688 [============>.................] - ETA: 2s - loss: 0.6229 - accuracy: 0.9676
785/1688 [============>.................] - ETA: 2s - loss: 0.6275 - accuracy: 0.9677
805/1688 [=============>................] - ETA: 2s - loss: 0.6241 - accuracy: 0.9679
825/1688 [=============>................] - ETA: 2s - loss: 0.6205 - accuracy: 0.9681
845/1688 [==============>...............] - ETA: 2s - loss: 0.6210 - accuracy: 0.9681
865/1688 [==============>...............] - ETA: 2s - loss: 0.6233 - accuracy: 0.9682
885/1688 [==============>...............] - ETA: 2s - loss: 0.6181 - accuracy: 0.9684
905/1688 [===============>..............] - ETA: 1s - loss: 0.6216 - accuracy: 0.9682
925/1688 [===============>..............] - ETA: 1s - loss: 0.6211 - accuracy: 0.9682
945/1688 [===============>..............] - ETA: 1s - loss: 0.6159 - accuracy: 0.9684
965/1688 [================>.............] - ETA: 1s - loss: 0.6169 - accuracy: 0.9683
985/1688 [================>.............] - ETA: 1s - loss: 0.6179 - accuracy: 0.9683
1005/1688 [================>.............] - ETA: 1s - loss: 0.6253 - accuracy: 0.9680
1025/1688 [=================>............] - ETA: 1s - loss: 0.6314 - accuracy: 0.9678
1045/1688 [=================>............] - ETA: 1s - loss: 0.6342 - accuracy: 0.9678
1065/1688 [=================>............] - ETA: 1s - loss: 0.6315 - accuracy: 0.9680
1085/1688 [==================>...........] - ETA: 1s - loss: 0.6302 - accuracy: 0.9680
1105/1688 [==================>...........] - ETA: 1s - loss: 0.6385 - accuracy: 0.9679
1125/1688 [==================>...........] - ETA: 1s - loss: 0.6459 - accuracy: 0.9677
1146/1688 [===================>..........] - ETA: 1s - loss: 0.6430 - accuracy: 0.9679
1166/1688 [===================>..........] - ETA: 1s - loss: 0.6495 - accuracy: 0.9677
1186/1688 [====================>.........] - ETA: 1s - loss: 0.6494 - accuracy: 0.9676
1206/1688 [====================>.........] - ETA: 1s - loss: 0.6465 - accuracy: 0.9676
1226/1688 [====================>.........] - ETA: 1s - loss: 0.6417 - accuracy: 0.9677
1246/1688 [=====================>........] - ETA: 1s - loss: 0.6439 - accuracy: 0.9676
1266/1688 [=====================>........] - ETA: 1s - loss: 0.6480 - accuracy: 0.9674
1286/1688 [=====================>........] - ETA: 1s - loss: 0.6442 - accuracy: 0.9675
1306/1688 [======================>.......] - ETA: 0s - loss: 0.6476 - accuracy: 0.9675
1326/1688 [======================>.......] - ETA: 0s - loss: 0.6460 - accuracy: 0.9675
1346/1688 [======================>.......] - ETA: 0s - loss: 0.6489 - accuracy: 0.9674
1366/1688 [=======================>......] - ETA: 0s - loss: 0.6493 - accuracy: 0.9675
1386/1688 [=======================>......] - ETA: 0s - loss: 0.6466 - accuracy: 0.9676
1406/1688 [=======================>......] - ETA: 0s - loss: 0.6501 - accuracy: 0.9675
1426/1688 [========================>.....] - ETA: 0s - loss: 0.6493 - accuracy: 0.9674
1446/1688 [========================>.....] - ETA: 0s - loss: 0.6502 - accuracy: 0.9673
1466/1688 [=========================>....] - ETA: 0s - loss: 0.6509 - accuracy: 0.9672
1486/1688 [=========================>....] - ETA: 0s - loss: 0.6528 - accuracy: 0.9670
1506/1688 [=========================>....] - ETA: 0s - loss: 0.6503 - accuracy: 0.9672
1526/1688 [==========================>...] - ETA: 0s - loss: 0.6504 - accuracy: 0.9672
1546/1688 [==========================>...] - ETA: 0s - loss: 0.6509 - accuracy: 0.9672
1566/1688 [==========================>...] - ETA: 0s - loss: 0.6546 - accuracy: 0.9670
1586/1688 [===========================>..] - ETA: 0s - loss: 0.6569 - accuracy: 0.9669
1606/1688 [===========================>..] - ETA: 0s - loss: 0.6601 - accuracy: 0.9668
1626/1688 [===========================>..] - ETA: 0s - loss: 0.6647 - accuracy: 0.9666
1646/1688 [============================>.] - ETA: 0s - loss: 0.6631 - accuracy: 0.9667
1666/1688 [============================>.] - ETA: 0s - loss: 0.6586 - accuracy: 0.9668
1686/1688 [============================>.] - ETA: 0s - loss: 0.6577 - accuracy: 0.9669
1688/1688 [==============================] - 5s 3ms/step - loss: 0.6580 - accuracy: 0.9669 - val_loss: 0.7273 - val_accuracy: 0.9708
Epoch 4/5
1/1688 [..............................] - ETA: 4s - loss: 0.0000e+00 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.7101 - accuracy: 0.9702
41/1688 [..............................] - ETA: 4s - loss: 0.7737 - accuracy: 0.9665
61/1688 [>.............................] - ETA: 4s - loss: 0.6542 - accuracy: 0.9677
82/1688 [>.............................] - ETA: 4s - loss: 0.6235 - accuracy: 0.9680
102/1688 [>.............................] - ETA: 4s - loss: 0.5701 - accuracy: 0.9697
122/1688 [=>............................] - ETA: 3s - loss: 0.5706 - accuracy: 0.9695
142/1688 [=>............................] - ETA: 3s - loss: 0.5961 - accuracy: 0.9692
162/1688 [=>............................] - ETA: 3s - loss: 0.5793 - accuracy: 0.9695
182/1688 [==>...........................] - ETA: 3s - loss: 0.6050 - accuracy: 0.9691
202/1688 [==>...........................] - ETA: 3s - loss: 0.6062 - accuracy: 0.9689
222/1688 [==>...........................] - ETA: 3s - loss: 0.6109 - accuracy: 0.9688
241/1688 [===>..........................] - ETA: 3s - loss: 0.6041 - accuracy: 0.9690
261/1688 [===>..........................] - ETA: 3s - loss: 0.5810 - accuracy: 0.9697
281/1688 [===>..........................] - ETA: 3s - loss: 0.5797 - accuracy: 0.9694
301/1688 [====>.........................] - ETA: 3s - loss: 0.5510 - accuracy: 0.9703
321/1688 [====>.........................] - ETA: 3s - loss: 0.5677 - accuracy: 0.9702
341/1688 [=====>........................] - ETA: 3s - loss: 0.5712 - accuracy: 0.9706
361/1688 [=====>........................] - ETA: 3s - loss: 0.5571 - accuracy: 0.9713
381/1688 [=====>........................] - ETA: 3s - loss: 0.5470 - accuracy: 0.9719
401/1688 [======>.......................] - ETA: 3s - loss: 0.5459 - accuracy: 0.9721
421/1688 [======>.......................] - ETA: 3s - loss: 0.5494 - accuracy: 0.9720
441/1688 [======>.......................] - ETA: 3s - loss: 0.5408 - accuracy: 0.9719
461/1688 [=======>......................] - ETA: 3s - loss: 0.5455 - accuracy: 0.9717
481/1688 [=======>......................] - ETA: 3s - loss: 0.5447 - accuracy: 0.9712
501/1688 [=======>......................] - ETA: 3s - loss: 0.5340 - accuracy: 0.9715
521/1688 [========>.....................] - ETA: 2s - loss: 0.5364 - accuracy: 0.9713
541/1688 [========>.....................] - ETA: 2s - loss: 0.5374 - accuracy: 0.9712
561/1688 [========>.....................] - ETA: 2s - loss: 0.5348 - accuracy: 0.9716
581/1688 [=========>....................] - ETA: 2s - loss: 0.5349 - accuracy: 0.9717
601/1688 [=========>....................] - ETA: 2s - loss: 0.5400 - accuracy: 0.9716
621/1688 [==========>...................] - ETA: 2s - loss: 0.5517 - accuracy: 0.9711
641/1688 [==========>...................] - ETA: 2s - loss: 0.5569 - accuracy: 0.9709
661/1688 [==========>...................] - ETA: 2s - loss: 0.5580 - accuracy: 0.9707
681/1688 [===========>..................] - ETA: 2s - loss: 0.5579 - accuracy: 0.9709
702/1688 [===========>..................] - ETA: 2s - loss: 0.5561 - accuracy: 0.9708
722/1688 [===========>..................] - ETA: 2s - loss: 0.5559 - accuracy: 0.9708
742/1688 [============>.................] - ETA: 2s - loss: 0.5497 - accuracy: 0.9711
762/1688 [============>.................] - ETA: 2s - loss: 0.5616 - accuracy: 0.9708
782/1688 [============>.................] - ETA: 2s - loss: 0.5651 - accuracy: 0.9707
803/1688 [=============>................] - ETA: 2s - loss: 0.5705 - accuracy: 0.9707
823/1688 [=============>................] - ETA: 2s - loss: 0.5713 - accuracy: 0.9706
843/1688 [=============>................] - ETA: 2s - loss: 0.5787 - accuracy: 0.9701
863/1688 [==============>...............] - ETA: 2s - loss: 0.5855 - accuracy: 0.9701
883/1688 [==============>...............] - ETA: 2s - loss: 0.5849 - accuracy: 0.9700
903/1688 [===============>..............] - ETA: 1s - loss: 0.5806 - accuracy: 0.9702
923/1688 [===============>..............] - ETA: 1s - loss: 0.5867 - accuracy: 0.9700
943/1688 [===============>..............] - ETA: 1s - loss: 0.5951 - accuracy: 0.9699
963/1688 [================>.............] - ETA: 1s - loss: 0.5948 - accuracy: 0.9699
983/1688 [================>.............] - ETA: 1s - loss: 0.5947 - accuracy: 0.9700
1003/1688 [================>.............] - ETA: 1s - loss: 0.5970 - accuracy: 0.9701
1022/1688 [=================>............] - ETA: 1s - loss: 0.5953 - accuracy: 0.9701
1042/1688 [=================>............] - ETA: 1s - loss: 0.6017 - accuracy: 0.9698
1062/1688 [=================>............] - ETA: 1s - loss: 0.5992 - accuracy: 0.9701
1083/1688 [==================>...........] - ETA: 1s - loss: 0.5968 - accuracy: 0.9702
1103/1688 [==================>...........] - ETA: 1s - loss: 0.5974 - accuracy: 0.9702
1123/1688 [==================>...........] - ETA: 1s - loss: 0.5977 - accuracy: 0.9702
1143/1688 [===================>..........] - ETA: 1s - loss: 0.6052 - accuracy: 0.9699
1163/1688 [===================>..........] - ETA: 1s - loss: 0.6051 - accuracy: 0.9699
1183/1688 [====================>.........] - ETA: 1s - loss: 0.6107 - accuracy: 0.9697
1203/1688 [====================>.........] - ETA: 1s - loss: 0.6073 - accuracy: 0.9697
1223/1688 [====================>.........] - ETA: 1s - loss: 0.6120 - accuracy: 0.9696
1243/1688 [=====================>........] - ETA: 1s - loss: 0.6125 - accuracy: 0.9696
1263/1688 [=====================>........] - ETA: 1s - loss: 0.6135 - accuracy: 0.9695
1283/1688 [=====================>........] - ETA: 1s - loss: 0.6175 - accuracy: 0.9694
1303/1688 [======================>.......] - ETA: 0s - loss: 0.6144 - accuracy: 0.9695
1323/1688 [======================>.......] - ETA: 0s - loss: 0.6148 - accuracy: 0.9696
1343/1688 [======================>.......] - ETA: 0s - loss: 0.6157 - accuracy: 0.9695
1363/1688 [=======================>......] - ETA: 0s - loss: 0.6201 - accuracy: 0.9693
1383/1688 [=======================>......] - ETA: 0s - loss: 0.6193 - accuracy: 0.9694
1403/1688 [=======================>......] - ETA: 0s - loss: 0.6170 - accuracy: 0.9694
1423/1688 [========================>.....] - ETA: 0s - loss: 0.6144 - accuracy: 0.9693
1443/1688 [========================>.....] - ETA: 0s - loss: 0.6148 - accuracy: 0.9692
1463/1688 [=========================>....] - ETA: 0s - loss: 0.6142 - accuracy: 0.9693
1484/1688 [=========================>....] - ETA: 0s - loss: 0.6156 - accuracy: 0.9693
1504/1688 [=========================>....] - ETA: 0s - loss: 0.6158 - accuracy: 0.9693
1524/1688 [==========================>...] - ETA: 0s - loss: 0.6208 - accuracy: 0.9692
1544/1688 [==========================>...] - ETA: 0s - loss: 0.6213 - accuracy: 0.9692
1564/1688 [==========================>...] - ETA: 0s - loss: 0.6219 - accuracy: 0.9691
1585/1688 [===========================>..] - ETA: 0s - loss: 0.6211 - accuracy: 0.9691
1605/1688 [===========================>..] - ETA: 0s - loss: 0.6211 - accuracy: 0.9691
1625/1688 [===========================>..] - ETA: 0s - loss: 0.6242 - accuracy: 0.9690
1645/1688 [============================>.] - ETA: 0s - loss: 0.6311 - accuracy: 0.9688
1665/1688 [============================>.] - ETA: 0s - loss: 0.6314 - accuracy: 0.9688
1685/1688 [============================>.] - ETA: 0s - loss: 0.6326 - accuracy: 0.9688
1688/1688 [==============================] - 5s 3ms/step - loss: 0.6345 - accuracy: 0.9688 - val_loss: 0.6984 - val_accuracy: 0.9717
Epoch 5/5
1/1688 [..............................] - ETA: 4s - loss: 1.5806e-04 - accuracy: 1.0000
21/1688 [..............................] - ETA: 4s - loss: 0.5264 - accuracy: 0.9747
41/1688 [..............................] - ETA: 4s - loss: 0.6207 - accuracy: 0.9733
61/1688 [>.............................] - ETA: 4s - loss: 0.5021 - accuracy: 0.9780
81/1688 [>.............................] - ETA: 4s - loss: 0.5711 - accuracy: 0.9711
101/1688 [>.............................] - ETA: 4s - loss: 0.5065 - accuracy: 0.9743
121/1688 [=>............................] - ETA: 4s - loss: 0.5417 - accuracy: 0.9744
141/1688 [=>............................] - ETA: 3s - loss: 0.5170 - accuracy: 0.9752
161/1688 [=>............................] - ETA: 3s - loss: 0.5035 - accuracy: 0.9748
181/1688 [==>...........................] - ETA: 3s - loss: 0.4922 - accuracy: 0.9748
201/1688 [==>...........................] - ETA: 3s - loss: 0.4713 - accuracy: 0.9748
222/1688 [==>...........................] - ETA: 3s - loss: 0.4636 - accuracy: 0.9752
243/1688 [===>..........................] - ETA: 3s - loss: 0.4929 - accuracy: 0.9748
263/1688 [===>..........................] - ETA: 3s - loss: 0.4987 - accuracy: 0.9747
283/1688 [====>.........................] - ETA: 3s - loss: 0.5059 - accuracy: 0.9738
303/1688 [====>.........................] - ETA: 3s - loss: 0.5240 - accuracy: 0.9732
323/1688 [====>.........................] - ETA: 3s - loss: 0.5227 - accuracy: 0.9731
342/1688 [=====>........................] - ETA: 3s - loss: 0.5162 - accuracy: 0.9730
362/1688 [=====>........................] - ETA: 3s - loss: 0.5090 - accuracy: 0.9734
382/1688 [=====>........................] - ETA: 3s - loss: 0.5113 - accuracy: 0.9734
403/1688 [======>.......................] - ETA: 3s - loss: 0.5183 - accuracy: 0.9732
423/1688 [======>.......................] - ETA: 3s - loss: 0.5266 - accuracy: 0.9729
443/1688 [======>.......................] - ETA: 3s - loss: 0.5215 - accuracy: 0.9732
464/1688 [=======>......................] - ETA: 3s - loss: 0.5308 - accuracy: 0.9734
484/1688 [=======>......................] - ETA: 3s - loss: 0.5384 - accuracy: 0.9732
504/1688 [=======>......................] - ETA: 3s - loss: 0.5355 - accuracy: 0.9733
524/1688 [========>.....................] - ETA: 2s - loss: 0.5413 - accuracy: 0.9732
544/1688 [========>.....................] - ETA: 2s - loss: 0.5410 - accuracy: 0.9733
565/1688 [=========>....................] - ETA: 2s - loss: 0.5405 - accuracy: 0.9732
586/1688 [=========>....................] - ETA: 2s - loss: 0.5306 - accuracy: 0.9733
606/1688 [=========>....................] - ETA: 2s - loss: 0.5276 - accuracy: 0.9734
626/1688 [==========>...................] - ETA: 2s - loss: 0.5283 - accuracy: 0.9735
646/1688 [==========>...................] - ETA: 2s - loss: 0.5247 - accuracy: 0.9736
666/1688 [==========>...................] - ETA: 2s - loss: 0.5252 - accuracy: 0.9738
686/1688 [===========>..................] - ETA: 2s - loss: 0.5225 - accuracy: 0.9739
706/1688 [===========>..................] - ETA: 2s - loss: 0.5299 - accuracy: 0.9736
726/1688 [===========>..................] - ETA: 2s - loss: 0.5331 - accuracy: 0.9734
746/1688 [============>.................] - ETA: 2s - loss: 0.5317 - accuracy: 0.9733
766/1688 [============>.................] - ETA: 2s - loss: 0.5275 - accuracy: 0.9734
786/1688 [============>.................] - ETA: 2s - loss: 0.5238 - accuracy: 0.9734
806/1688 [=============>................] - ETA: 2s - loss: 0.5333 - accuracy: 0.9729
827/1688 [=============>................] - ETA: 2s - loss: 0.5468 - accuracy: 0.9726
847/1688 [==============>...............] - ETA: 2s - loss: 0.5618 - accuracy: 0.9721
868/1688 [==============>...............] - ETA: 2s - loss: 0.5632 - accuracy: 0.9721
889/1688 [==============>...............] - ETA: 2s - loss: 0.5590 - accuracy: 0.9721
909/1688 [===============>..............] - ETA: 1s - loss: 0.5585 - accuracy: 0.9721
928/1688 [===============>..............] - ETA: 1s - loss: 0.5602 - accuracy: 0.9718
948/1688 [===============>..............] - ETA: 1s - loss: 0.5541 - accuracy: 0.9721
968/1688 [================>.............] - ETA: 1s - loss: 0.5552 - accuracy: 0.9721
988/1688 [================>.............] - ETA: 1s - loss: 0.5525 - accuracy: 0.9721
1008/1688 [================>.............] - ETA: 1s - loss: 0.5542 - accuracy: 0.9720
1028/1688 [=================>............] - ETA: 1s - loss: 0.5558 - accuracy: 0.9719
1048/1688 [=================>............] - ETA: 1s - loss: 0.5554 - accuracy: 0.9721
1068/1688 [=================>............] - ETA: 1s - loss: 0.5527 - accuracy: 0.9721
1088/1688 [==================>...........] - ETA: 1s - loss: 0.5503 - accuracy: 0.9721
1107/1688 [==================>...........] - ETA: 1s - loss: 0.5514 - accuracy: 0.9721
1127/1688 [===================>..........] - ETA: 1s - loss: 0.5566 - accuracy: 0.9718
1147/1688 [===================>..........] - ETA: 1s - loss: 0.5583 - accuracy: 0.9719
1167/1688 [===================>..........] - ETA: 1s - loss: 0.5633 - accuracy: 0.9717
1187/1688 [====================>.........] - ETA: 1s - loss: 0.5650 - accuracy: 0.9716
1207/1688 [====================>.........] - ETA: 1s - loss: 0.5669 - accuracy: 0.9716
1227/1688 [====================>.........] - ETA: 1s - loss: 0.5643 - accuracy: 0.9718
1247/1688 [=====================>........] - ETA: 1s - loss: 0.5622 - accuracy: 0.9719
1267/1688 [=====================>........] - ETA: 1s - loss: 0.5582 - accuracy: 0.9719
1287/1688 [=====================>........] - ETA: 1s - loss: 0.5570 - accuracy: 0.9717
1307/1688 [======================>.......] - ETA: 0s - loss: 0.5523 - accuracy: 0.9719
1327/1688 [======================>.......] - ETA: 0s - loss: 0.5546 - accuracy: 0.9718
1347/1688 [======================>.......] - ETA: 0s - loss: 0.5543 - accuracy: 0.9719
1367/1688 [=======================>......] - ETA: 0s - loss: 0.5565 - accuracy: 0.9719
1387/1688 [=======================>......] - ETA: 0s - loss: 0.5557 - accuracy: 0.9719
1407/1688 [========================>.....] - ETA: 0s - loss: 0.5529 - accuracy: 0.9720
1427/1688 [========================>.....] - ETA: 0s - loss: 0.5534 - accuracy: 0.9721
1447/1688 [========================>.....] - ETA: 0s - loss: 0.5565 - accuracy: 0.9721
1467/1688 [=========================>....] - ETA: 0s - loss: 0.5613 - accuracy: 0.9719
1487/1688 [=========================>....] - ETA: 0s - loss: 0.5610 - accuracy: 0.9719
1507/1688 [=========================>....] - ETA: 0s - loss: 0.5610 - accuracy: 0.9717
1527/1688 [==========================>...] - ETA: 0s - loss: 0.5581 - accuracy: 0.9718
1548/1688 [==========================>...] - ETA: 0s - loss: 0.5606 - accuracy: 0.9717
1568/1688 [==========================>...] - ETA: 0s - loss: 0.5638 - accuracy: 0.9716
1588/1688 [===========================>..] - ETA: 0s - loss: 0.5644 - accuracy: 0.9715
1608/1688 [===========================>..] - ETA: 0s - loss: 0.5652 - accuracy: 0.9715
1628/1688 [===========================>..] - ETA: 0s - loss: 0.5656 - accuracy: 0.9714
1648/1688 [============================>.] - ETA: 0s - loss: 0.5692 - accuracy: 0.9713
1668/1688 [============================>.] - ETA: 0s - loss: 0.5708 - accuracy: 0.9713
1688/1688 [==============================] - ETA: 0s - loss: 0.5696 - accuracy: 0.9714
1688/1688 [==============================] - 5s 3ms/step - loss: 0.5696 - accuracy: 0.9714 - val_loss: 0.8509 - val_accuracy: 0.9707
Test accuracy after fine tuning: 0.96670001745224
6. Model conversion
After having obtained a quantized model with satisfactory performance, it can be converted to a model suitable to be used in the Akida NSoC in inference mode. The convert function returns a model in Akida format, ready for the Akida NSoC or the Akida software simulator.
Note
One needs to supply the coefficients used to rescale the input
dataset before the training - here input_scaling.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized, input_scaling=input_scaling)
model_akida.summary()
accuracy = model_akida.evaluate(raw_x_test, raw_y_test)
print('Test accuracy after conversion:', accuracy)
# For non-regression purpose
assert accuracy > 0.96
Out:
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 3
______________________________________________
SW/conv2d-dense (Software)
______________________________________________________________
Layer (type) Output shape Kernel shape
==============================================================
conv2d (InputConv.) [13, 13, 32] (3, 3, 1, 32)
______________________________________________________________
separable_conv2d (Sep.Conv.) [7, 7, 64] (3, 3, 32, 1)
______________________________________________________________
(1, 1, 32, 64)
______________________________________________________________
dense (Fully.) [1, 1, 10] (1, 1, 3136, 10)
______________________________________________________________
Test accuracy after conversion: 0.9660999774932861
Depending on the number of samples you run, you should find a performance of around 98% (better results can be achieved using more epochs for training).
Total running time of the script: ( 0 minutes 59.799 seconds)