Note
Go to the end to download the full example code
Global Akida workflow
Using the MNIST dataset, this example shows the definition and training of a keras floating point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original keras floating point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.
Note
Please refer to the TensorFlow tf.keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.
The MNIST example below is light enough so that a GPU is not needed for training.
Global Akida workflow
1. Create and train
1.1. Load and reshape MNIST dataset
import numpy as np
import matplotlib.cm as cm
import matplotlib.pyplot as plt
from keras.datasets import mnist
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, width, height, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
axarr[i].set_title('Class %d' % y_test[i])
plt.show()
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
8192/11490434 [..............................] - ETA: 0s
24576/11490434 [..............................] - ETA: 1:13
40960/11490434 [..............................] - ETA: 1:01
57344/11490434 [..............................] - ETA: 59s
81920/11490434 [..............................] - ETA: 50s
106496/11490434 [..............................] - ETA: 44s
131072/11490434 [..............................] - ETA: 40s
172032/11490434 [..............................] - ETA: 34s
221184/11490434 [..............................] - ETA: 29s
278528/11490434 [..............................] - ETA: 25s
352256/11490434 [..............................] - ETA: 21s
450560/11490434 [>.............................] - ETA: 17s
573440/11490434 [>.............................] - ETA: 14s
720896/11490434 [>.............................] - ETA: 12s
909312/11490434 [=>............................] - ETA: 10s
1146880/11490434 [=>............................] - ETA: 8s
1458176/11490434 [==>...........................] - ETA: 6s
1835008/11490434 [===>..........................] - ETA: 5s
2326528/11490434 [=====>........................] - ETA: 4s
2883584/11490434 [======>.......................] - ETA: 3s
3473408/11490434 [========>.....................] - ETA: 2s
4063232/11490434 [=========>....................] - ETA: 2s
4653056/11490434 [===========>..................] - ETA: 1s
5242880/11490434 [============>.................] - ETA: 1s
5832704/11490434 [==============>...............] - ETA: 1s
6414336/11490434 [===============>..............] - ETA: 1s
6963200/11490434 [=================>............] - ETA: 0s
7520256/11490434 [==================>...........] - ETA: 0s
7782400/11490434 [===================>..........] - ETA: 0s
10821632/11490434 [===========================>..] - ETA: 0s
11288576/11490434 [============================>.] - ETA: 0s
11490434/11490434 [==============================] - 2s 0us/step
1.2. Model definition
Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard Keras CNN model.
import keras
model_keras = keras.models.Sequential([
keras.layers.Rescaling(1. / 255, input_shape=(28, 28, 1)),
keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
# Separable layer
keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
rescaling (Rescaling) (None, 28, 28, 1) 0
conv2d (Conv2D) (None, 13, 13, 32) 320
batch_normalization (BatchN (None, 13, 13, 32) 128
ormalization)
re_lu (ReLU) (None, 13, 13, 32) 0
depthwise_conv2d (Depthwise (None, 7, 7, 32) 320
Conv2D)
conv2d_1 (Conv2D) (None, 7, 7, 64) 2112
batch_normalization_1 (Batc (None, 7, 7, 64) 256
hNormalization)
re_lu_1 (ReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34,506
Trainable params: 34,314
Non-trainable params: 192
_________________________________________________________________
1.3. Model training
Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.
from keras.optimizers import Adam
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=Adam(learning_rate=1e-3),
metrics=['accuracy'])
_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
Epoch 1/10
1/1688 [..............................] - ETA: 22:19 - loss: 2.7690 - accuracy: 0.1250
30/1688 [..............................] - ETA: 2s - loss: 1.2310 - accuracy: 0.6083
60/1688 [>.............................] - ETA: 2s - loss: 0.8957 - accuracy: 0.7198
91/1688 [>.............................] - ETA: 2s - loss: 0.7277 - accuracy: 0.7734
121/1688 [=>............................] - ETA: 2s - loss: 0.6204 - accuracy: 0.8086
152/1688 [=>............................] - ETA: 2s - loss: 0.5547 - accuracy: 0.8316
182/1688 [==>...........................] - ETA: 2s - loss: 0.5135 - accuracy: 0.8444
212/1688 [==>...........................] - ETA: 2s - loss: 0.4793 - accuracy: 0.8561
242/1688 [===>..........................] - ETA: 2s - loss: 0.4492 - accuracy: 0.8651
273/1688 [===>..........................] - ETA: 2s - loss: 0.4201 - accuracy: 0.8740
304/1688 [====>.........................] - ETA: 2s - loss: 0.3958 - accuracy: 0.8811
335/1688 [====>.........................] - ETA: 2s - loss: 0.3810 - accuracy: 0.8855
366/1688 [=====>........................] - ETA: 2s - loss: 0.3649 - accuracy: 0.8911
397/1688 [======>.......................] - ETA: 2s - loss: 0.3543 - accuracy: 0.8938
427/1688 [======>.......................] - ETA: 2s - loss: 0.3402 - accuracy: 0.8978
457/1688 [=======>......................] - ETA: 2s - loss: 0.3292 - accuracy: 0.9008
487/1688 [=======>......................] - ETA: 2s - loss: 0.3156 - accuracy: 0.9052
520/1688 [========>.....................] - ETA: 1s - loss: 0.3046 - accuracy: 0.9088
550/1688 [========>.....................] - ETA: 1s - loss: 0.2936 - accuracy: 0.9120
580/1688 [=========>....................] - ETA: 1s - loss: 0.2868 - accuracy: 0.9140
611/1688 [=========>....................] - ETA: 1s - loss: 0.2803 - accuracy: 0.9162
642/1688 [==========>...................] - ETA: 1s - loss: 0.2740 - accuracy: 0.9182
672/1688 [==========>...................] - ETA: 1s - loss: 0.2682 - accuracy: 0.9198
702/1688 [===========>..................] - ETA: 1s - loss: 0.2618 - accuracy: 0.9216
732/1688 [============>.................] - ETA: 1s - loss: 0.2569 - accuracy: 0.9230
763/1688 [============>.................] - ETA: 1s - loss: 0.2523 - accuracy: 0.9243
793/1688 [=============>................] - ETA: 1s - loss: 0.2459 - accuracy: 0.9263
824/1688 [=============>................] - ETA: 1s - loss: 0.2415 - accuracy: 0.9274
854/1688 [==============>...............] - ETA: 1s - loss: 0.2363 - accuracy: 0.9288
884/1688 [==============>...............] - ETA: 1s - loss: 0.2325 - accuracy: 0.9298
914/1688 [===============>..............] - ETA: 1s - loss: 0.2283 - accuracy: 0.9310
945/1688 [===============>..............] - ETA: 1s - loss: 0.2252 - accuracy: 0.9320
975/1688 [================>.............] - ETA: 1s - loss: 0.2217 - accuracy: 0.9331
1005/1688 [================>.............] - ETA: 1s - loss: 0.2180 - accuracy: 0.9343
1035/1688 [=================>............] - ETA: 1s - loss: 0.2145 - accuracy: 0.9352
1065/1688 [=================>............] - ETA: 1s - loss: 0.2108 - accuracy: 0.9363
1095/1688 [==================>...........] - ETA: 0s - loss: 0.2076 - accuracy: 0.9374
1125/1688 [==================>...........] - ETA: 0s - loss: 0.2044 - accuracy: 0.9383
1156/1688 [===================>..........] - ETA: 0s - loss: 0.2023 - accuracy: 0.9391
1186/1688 [====================>.........] - ETA: 0s - loss: 0.1993 - accuracy: 0.9400
1216/1688 [====================>.........] - ETA: 0s - loss: 0.1967 - accuracy: 0.9409
1246/1688 [=====================>........] - ETA: 0s - loss: 0.1940 - accuracy: 0.9418
1276/1688 [=====================>........] - ETA: 0s - loss: 0.1916 - accuracy: 0.9425
1306/1688 [======================>.......] - ETA: 0s - loss: 0.1902 - accuracy: 0.9433
1337/1688 [======================>.......] - ETA: 0s - loss: 0.1881 - accuracy: 0.9438
1368/1688 [=======================>......] - ETA: 0s - loss: 0.1860 - accuracy: 0.9445
1399/1688 [=======================>......] - ETA: 0s - loss: 0.1843 - accuracy: 0.9450
1428/1688 [========================>.....] - ETA: 0s - loss: 0.1827 - accuracy: 0.9454
1458/1688 [========================>.....] - ETA: 0s - loss: 0.1809 - accuracy: 0.9460
1489/1688 [=========================>....] - ETA: 0s - loss: 0.1792 - accuracy: 0.9466
1520/1688 [==========================>...] - ETA: 0s - loss: 0.1771 - accuracy: 0.9472
1551/1688 [==========================>...] - ETA: 0s - loss: 0.1752 - accuracy: 0.9477
1581/1688 [===========================>..] - ETA: 0s - loss: 0.1733 - accuracy: 0.9483
1611/1688 [===========================>..] - ETA: 0s - loss: 0.1719 - accuracy: 0.9487
1641/1688 [============================>.] - ETA: 0s - loss: 0.1699 - accuracy: 0.9492
1672/1688 [============================>.] - ETA: 0s - loss: 0.1682 - accuracy: 0.9497
1688/1688 [==============================] - ETA: 0s - loss: 0.1671 - accuracy: 0.9500
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1671 - accuracy: 0.9500 - val_loss: 0.0851 - val_accuracy: 0.9760
Epoch 2/10
1/1688 [..............................] - ETA: 2s - loss: 0.0182 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0821 - accuracy: 0.9736
63/1688 [>.............................] - ETA: 2s - loss: 0.0691 - accuracy: 0.9802
94/1688 [>.............................] - ETA: 2s - loss: 0.0665 - accuracy: 0.9797
124/1688 [=>............................] - ETA: 2s - loss: 0.0660 - accuracy: 0.9808
155/1688 [=>............................] - ETA: 2s - loss: 0.0680 - accuracy: 0.9810
186/1688 [==>...........................] - ETA: 2s - loss: 0.0672 - accuracy: 0.9814
217/1688 [==>...........................] - ETA: 2s - loss: 0.0648 - accuracy: 0.9819
247/1688 [===>..........................] - ETA: 2s - loss: 0.0664 - accuracy: 0.9810
278/1688 [===>..........................] - ETA: 2s - loss: 0.0697 - accuracy: 0.9794
307/1688 [====>.........................] - ETA: 2s - loss: 0.0697 - accuracy: 0.9793
337/1688 [====>.........................] - ETA: 2s - loss: 0.0699 - accuracy: 0.9792
367/1688 [=====>........................] - ETA: 2s - loss: 0.0682 - accuracy: 0.9796
398/1688 [======>.......................] - ETA: 2s - loss: 0.0675 - accuracy: 0.9796
429/1688 [======>.......................] - ETA: 2s - loss: 0.0670 - accuracy: 0.9797
459/1688 [=======>......................] - ETA: 2s - loss: 0.0676 - accuracy: 0.9794
489/1688 [=======>......................] - ETA: 2s - loss: 0.0681 - accuracy: 0.9794
519/1688 [========>.....................] - ETA: 1s - loss: 0.0682 - accuracy: 0.9797
550/1688 [========>.....................] - ETA: 1s - loss: 0.0676 - accuracy: 0.9799
580/1688 [=========>....................] - ETA: 1s - loss: 0.0682 - accuracy: 0.9795
611/1688 [=========>....................] - ETA: 1s - loss: 0.0680 - accuracy: 0.9794
641/1688 [==========>...................] - ETA: 1s - loss: 0.0670 - accuracy: 0.9795
671/1688 [==========>...................] - ETA: 1s - loss: 0.0662 - accuracy: 0.9796
701/1688 [===========>..................] - ETA: 1s - loss: 0.0672 - accuracy: 0.9793
731/1688 [===========>..................] - ETA: 1s - loss: 0.0668 - accuracy: 0.9794
761/1688 [============>.................] - ETA: 1s - loss: 0.0678 - accuracy: 0.9790
792/1688 [=============>................] - ETA: 1s - loss: 0.0681 - accuracy: 0.9789
823/1688 [=============>................] - ETA: 1s - loss: 0.0676 - accuracy: 0.9791
853/1688 [==============>...............] - ETA: 1s - loss: 0.0673 - accuracy: 0.9792
883/1688 [==============>...............] - ETA: 1s - loss: 0.0669 - accuracy: 0.9793
913/1688 [===============>..............] - ETA: 1s - loss: 0.0670 - accuracy: 0.9792
943/1688 [===============>..............] - ETA: 1s - loss: 0.0681 - accuracy: 0.9789
973/1688 [================>.............] - ETA: 1s - loss: 0.0679 - accuracy: 0.9789
1003/1688 [================>.............] - ETA: 1s - loss: 0.0680 - accuracy: 0.9789
1034/1688 [=================>............] - ETA: 1s - loss: 0.0681 - accuracy: 0.9789
1065/1688 [=================>............] - ETA: 1s - loss: 0.0679 - accuracy: 0.9790
1095/1688 [==================>...........] - ETA: 0s - loss: 0.0672 - accuracy: 0.9793
1125/1688 [==================>...........] - ETA: 0s - loss: 0.0679 - accuracy: 0.9791
1156/1688 [===================>..........] - ETA: 0s - loss: 0.0679 - accuracy: 0.9792
1187/1688 [====================>.........] - ETA: 0s - loss: 0.0679 - accuracy: 0.9792
1218/1688 [====================>.........] - ETA: 0s - loss: 0.0676 - accuracy: 0.9792
1249/1688 [=====================>........] - ETA: 0s - loss: 0.0677 - accuracy: 0.9792
1279/1688 [=====================>........] - ETA: 0s - loss: 0.0681 - accuracy: 0.9791
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0679 - accuracy: 0.9792
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0677 - accuracy: 0.9792
1371/1688 [=======================>......] - ETA: 0s - loss: 0.0670 - accuracy: 0.9795
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0666 - accuracy: 0.9796
1432/1688 [========================>.....] - ETA: 0s - loss: 0.0665 - accuracy: 0.9796
1462/1688 [========================>.....] - ETA: 0s - loss: 0.0667 - accuracy: 0.9793
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0661 - accuracy: 0.9795
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0658 - accuracy: 0.9795
1552/1688 [==========================>...] - ETA: 0s - loss: 0.0658 - accuracy: 0.9795
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0663 - accuracy: 0.9794
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0666 - accuracy: 0.9793
1643/1688 [============================>.] - ETA: 0s - loss: 0.0665 - accuracy: 0.9793
1673/1688 [============================>.] - ETA: 0s - loss: 0.0672 - accuracy: 0.9792
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0673 - accuracy: 0.9792 - val_loss: 0.0705 - val_accuracy: 0.9817
Epoch 3/10
1/1688 [..............................] - ETA: 2s - loss: 0.1274 - accuracy: 0.9688
32/1688 [..............................] - ETA: 2s - loss: 0.0540 - accuracy: 0.9795
62/1688 [>.............................] - ETA: 2s - loss: 0.0491 - accuracy: 0.9839
93/1688 [>.............................] - ETA: 2s - loss: 0.0486 - accuracy: 0.9839
124/1688 [=>............................] - ETA: 2s - loss: 0.0440 - accuracy: 0.9856
155/1688 [=>............................] - ETA: 2s - loss: 0.0446 - accuracy: 0.9855
185/1688 [==>...........................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9856
216/1688 [==>...........................] - ETA: 2s - loss: 0.0458 - accuracy: 0.9848
246/1688 [===>..........................] - ETA: 2s - loss: 0.0472 - accuracy: 0.9841
277/1688 [===>..........................] - ETA: 2s - loss: 0.0465 - accuracy: 0.9843
307/1688 [====>.........................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9848
336/1688 [====>.........................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9851
366/1688 [=====>........................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9853
397/1688 [======>.......................] - ETA: 2s - loss: 0.0448 - accuracy: 0.9854
427/1688 [======>.......................] - ETA: 2s - loss: 0.0458 - accuracy: 0.9849
458/1688 [=======>......................] - ETA: 2s - loss: 0.0447 - accuracy: 0.9851
488/1688 [=======>......................] - ETA: 2s - loss: 0.0456 - accuracy: 0.9849
519/1688 [========>.....................] - ETA: 1s - loss: 0.0441 - accuracy: 0.9855
549/1688 [========>.....................] - ETA: 1s - loss: 0.0441 - accuracy: 0.9858
580/1688 [=========>....................] - ETA: 1s - loss: 0.0445 - accuracy: 0.9857
611/1688 [=========>....................] - ETA: 1s - loss: 0.0454 - accuracy: 0.9854
642/1688 [==========>...................] - ETA: 1s - loss: 0.0463 - accuracy: 0.9852
672/1688 [==========>...................] - ETA: 1s - loss: 0.0463 - accuracy: 0.9851
703/1688 [===========>..................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9853
733/1688 [============>.................] - ETA: 1s - loss: 0.0457 - accuracy: 0.9852
763/1688 [============>.................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9851
795/1688 [=============>................] - ETA: 1s - loss: 0.0457 - accuracy: 0.9850
825/1688 [=============>................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9850
856/1688 [==============>...............] - ETA: 1s - loss: 0.0470 - accuracy: 0.9847
887/1688 [==============>...............] - ETA: 1s - loss: 0.0475 - accuracy: 0.9846
917/1688 [===============>..............] - ETA: 1s - loss: 0.0472 - accuracy: 0.9848
948/1688 [===============>..............] - ETA: 1s - loss: 0.0471 - accuracy: 0.9849
979/1688 [================>.............] - ETA: 1s - loss: 0.0478 - accuracy: 0.9846
1010/1688 [================>.............] - ETA: 1s - loss: 0.0483 - accuracy: 0.9845
1040/1688 [=================>............] - ETA: 1s - loss: 0.0484 - accuracy: 0.9845
1071/1688 [==================>...........] - ETA: 1s - loss: 0.0486 - accuracy: 0.9845
1101/1688 [==================>...........] - ETA: 0s - loss: 0.0492 - accuracy: 0.9844
1132/1688 [===================>..........] - ETA: 0s - loss: 0.0489 - accuracy: 0.9845
1163/1688 [===================>..........] - ETA: 0s - loss: 0.0489 - accuracy: 0.9845
1193/1688 [====================>.........] - ETA: 0s - loss: 0.0490 - accuracy: 0.9844
1224/1688 [====================>.........] - ETA: 0s - loss: 0.0487 - accuracy: 0.9845
1255/1688 [=====================>........] - ETA: 0s - loss: 0.0493 - accuracy: 0.9843
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0492 - accuracy: 0.9844
1317/1688 [======================>.......] - ETA: 0s - loss: 0.0492 - accuracy: 0.9844
1350/1688 [======================>.......] - ETA: 0s - loss: 0.0492 - accuracy: 0.9844
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0494 - accuracy: 0.9845
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0495 - accuracy: 0.9844
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0493 - accuracy: 0.9844
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0495 - accuracy: 0.9844
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0494 - accuracy: 0.9844
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0497 - accuracy: 0.9842
1564/1688 [==========================>...] - ETA: 0s - loss: 0.0497 - accuracy: 0.9842
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0494 - accuracy: 0.9843
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0497 - accuracy: 0.9842
1654/1688 [============================>.] - ETA: 0s - loss: 0.0496 - accuracy: 0.9842
1685/1688 [============================>.] - ETA: 0s - loss: 0.0497 - accuracy: 0.9842
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0496 - accuracy: 0.9842 - val_loss: 0.0573 - val_accuracy: 0.9830
Epoch 4/10
1/1688 [..............................] - ETA: 2s - loss: 0.0396 - accuracy: 0.9688
35/1688 [..............................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9902
66/1688 [>.............................] - ETA: 2s - loss: 0.0229 - accuracy: 0.9910
97/1688 [>.............................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9919
128/1688 [=>............................] - ETA: 2s - loss: 0.0253 - accuracy: 0.9919
159/1688 [=>............................] - ETA: 2s - loss: 0.0309 - accuracy: 0.9902
189/1688 [==>...........................] - ETA: 2s - loss: 0.0329 - accuracy: 0.9896
219/1688 [==>...........................] - ETA: 2s - loss: 0.0315 - accuracy: 0.9900
249/1688 [===>..........................] - ETA: 2s - loss: 0.0316 - accuracy: 0.9895
279/1688 [===>..........................] - ETA: 2s - loss: 0.0312 - accuracy: 0.9895
310/1688 [====>.........................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9897
340/1688 [=====>........................] - ETA: 2s - loss: 0.0314 - accuracy: 0.9896
371/1688 [=====>........................] - ETA: 2s - loss: 0.0333 - accuracy: 0.9893
401/1688 [======>.......................] - ETA: 2s - loss: 0.0349 - accuracy: 0.9888
432/1688 [======>.......................] - ETA: 2s - loss: 0.0350 - accuracy: 0.9886
463/1688 [=======>......................] - ETA: 2s - loss: 0.0356 - accuracy: 0.9887
494/1688 [=======>......................] - ETA: 1s - loss: 0.0368 - accuracy: 0.9882
524/1688 [========>.....................] - ETA: 1s - loss: 0.0370 - accuracy: 0.9883
555/1688 [========>.....................] - ETA: 1s - loss: 0.0365 - accuracy: 0.9886
586/1688 [=========>....................] - ETA: 1s - loss: 0.0368 - accuracy: 0.9886
617/1688 [=========>....................] - ETA: 1s - loss: 0.0371 - accuracy: 0.9888
648/1688 [==========>...................] - ETA: 1s - loss: 0.0371 - accuracy: 0.9888
679/1688 [===========>..................] - ETA: 1s - loss: 0.0375 - accuracy: 0.9887
709/1688 [===========>..................] - ETA: 1s - loss: 0.0379 - accuracy: 0.9886
739/1688 [============>.................] - ETA: 1s - loss: 0.0379 - accuracy: 0.9887
770/1688 [============>.................] - ETA: 1s - loss: 0.0380 - accuracy: 0.9886
800/1688 [=============>................] - ETA: 1s - loss: 0.0378 - accuracy: 0.9886
830/1688 [=============>................] - ETA: 1s - loss: 0.0378 - accuracy: 0.9886
860/1688 [==============>...............] - ETA: 1s - loss: 0.0376 - accuracy: 0.9887
891/1688 [==============>...............] - ETA: 1s - loss: 0.0373 - accuracy: 0.9887
921/1688 [===============>..............] - ETA: 1s - loss: 0.0373 - accuracy: 0.9886
951/1688 [===============>..............] - ETA: 1s - loss: 0.0373 - accuracy: 0.9885
982/1688 [================>.............] - ETA: 1s - loss: 0.0372 - accuracy: 0.9884
1013/1688 [=================>............] - ETA: 1s - loss: 0.0372 - accuracy: 0.9884
1043/1688 [=================>............] - ETA: 1s - loss: 0.0378 - accuracy: 0.9883
1074/1688 [==================>...........] - ETA: 1s - loss: 0.0376 - accuracy: 0.9882
1104/1688 [==================>...........] - ETA: 0s - loss: 0.0373 - accuracy: 0.9883
1134/1688 [===================>..........] - ETA: 0s - loss: 0.0377 - accuracy: 0.9882
1164/1688 [===================>..........] - ETA: 0s - loss: 0.0377 - accuracy: 0.9882
1194/1688 [====================>.........] - ETA: 0s - loss: 0.0379 - accuracy: 0.9882
1224/1688 [====================>.........] - ETA: 0s - loss: 0.0382 - accuracy: 0.9881
1254/1688 [=====================>........] - ETA: 0s - loss: 0.0388 - accuracy: 0.9880
1285/1688 [=====================>........] - ETA: 0s - loss: 0.0388 - accuracy: 0.9880
1315/1688 [======================>.......] - ETA: 0s - loss: 0.0392 - accuracy: 0.9880
1347/1688 [======================>.......] - ETA: 0s - loss: 0.0394 - accuracy: 0.9879
1379/1688 [=======================>......] - ETA: 0s - loss: 0.0391 - accuracy: 0.9880
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0392 - accuracy: 0.9880
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0394 - accuracy: 0.9880
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0394 - accuracy: 0.9880
1502/1688 [=========================>....] - ETA: 0s - loss: 0.0396 - accuracy: 0.9879
1532/1688 [==========================>...] - ETA: 0s - loss: 0.0394 - accuracy: 0.9879
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0393 - accuracy: 0.9878
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0394 - accuracy: 0.9878
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0400 - accuracy: 0.9877
1653/1688 [============================>.] - ETA: 0s - loss: 0.0400 - accuracy: 0.9876
1683/1688 [============================>.] - ETA: 0s - loss: 0.0402 - accuracy: 0.9876
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0401 - accuracy: 0.9877 - val_loss: 0.0569 - val_accuracy: 0.9868
Epoch 5/10
1/1688 [..............................] - ETA: 3s - loss: 0.0039 - accuracy: 1.0000
32/1688 [..............................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9932
62/1688 [>.............................] - ETA: 2s - loss: 0.0270 - accuracy: 0.9904
92/1688 [>.............................] - ETA: 2s - loss: 0.0295 - accuracy: 0.9905
122/1688 [=>............................] - ETA: 2s - loss: 0.0294 - accuracy: 0.9910
153/1688 [=>............................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9920
183/1688 [==>...........................] - ETA: 2s - loss: 0.0256 - accuracy: 0.9921
214/1688 [==>...........................] - ETA: 2s - loss: 0.0273 - accuracy: 0.9917
245/1688 [===>..........................] - ETA: 2s - loss: 0.0291 - accuracy: 0.9912
275/1688 [===>..........................] - ETA: 2s - loss: 0.0290 - accuracy: 0.9910
306/1688 [====>.........................] - ETA: 2s - loss: 0.0297 - accuracy: 0.9910
337/1688 [====>.........................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9909
367/1688 [=====>........................] - ETA: 2s - loss: 0.0304 - accuracy: 0.9906
398/1688 [======>.......................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9902
428/1688 [======>.......................] - ETA: 2s - loss: 0.0313 - accuracy: 0.9902
458/1688 [=======>......................] - ETA: 2s - loss: 0.0309 - accuracy: 0.9902
490/1688 [=======>......................] - ETA: 1s - loss: 0.0311 - accuracy: 0.9902
520/1688 [========>.....................] - ETA: 1s - loss: 0.0308 - accuracy: 0.9902
551/1688 [========>.....................] - ETA: 1s - loss: 0.0304 - accuracy: 0.9903
581/1688 [=========>....................] - ETA: 1s - loss: 0.0304 - accuracy: 0.9903
612/1688 [=========>....................] - ETA: 1s - loss: 0.0304 - accuracy: 0.9902
642/1688 [==========>...................] - ETA: 1s - loss: 0.0301 - accuracy: 0.9903
672/1688 [==========>...................] - ETA: 1s - loss: 0.0298 - accuracy: 0.9903
702/1688 [===========>..................] - ETA: 1s - loss: 0.0296 - accuracy: 0.9904
733/1688 [============>.................] - ETA: 1s - loss: 0.0293 - accuracy: 0.9905
763/1688 [============>.................] - ETA: 1s - loss: 0.0302 - accuracy: 0.9903
793/1688 [=============>................] - ETA: 1s - loss: 0.0316 - accuracy: 0.9898
823/1688 [=============>................] - ETA: 1s - loss: 0.0316 - accuracy: 0.9896
853/1688 [==============>...............] - ETA: 1s - loss: 0.0319 - accuracy: 0.9894
883/1688 [==============>...............] - ETA: 1s - loss: 0.0317 - accuracy: 0.9895
914/1688 [===============>..............] - ETA: 1s - loss: 0.0313 - accuracy: 0.9897
945/1688 [===============>..............] - ETA: 1s - loss: 0.0319 - accuracy: 0.9894
975/1688 [================>.............] - ETA: 1s - loss: 0.0320 - accuracy: 0.9893
1005/1688 [================>.............] - ETA: 1s - loss: 0.0321 - accuracy: 0.9893
1036/1688 [=================>............] - ETA: 1s - loss: 0.0317 - accuracy: 0.9894
1066/1688 [=================>............] - ETA: 1s - loss: 0.0318 - accuracy: 0.9894
1096/1688 [==================>...........] - ETA: 0s - loss: 0.0319 - accuracy: 0.9894
1126/1688 [===================>..........] - ETA: 0s - loss: 0.0319 - accuracy: 0.9893
1157/1688 [===================>..........] - ETA: 0s - loss: 0.0322 - accuracy: 0.9892
1187/1688 [====================>.........] - ETA: 0s - loss: 0.0324 - accuracy: 0.9892
1217/1688 [====================>.........] - ETA: 0s - loss: 0.0324 - accuracy: 0.9892
1247/1688 [=====================>........] - ETA: 0s - loss: 0.0323 - accuracy: 0.9892
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0321 - accuracy: 0.9893
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0323 - accuracy: 0.9892
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0319 - accuracy: 0.9893
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0328 - accuracy: 0.9892
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0329 - accuracy: 0.9891
1431/1688 [========================>.....] - ETA: 0s - loss: 0.0327 - accuracy: 0.9892
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0326 - accuracy: 0.9892
1491/1688 [=========================>....] - ETA: 0s - loss: 0.0324 - accuracy: 0.9893
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0328 - accuracy: 0.9892
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0328 - accuracy: 0.9891
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0329 - accuracy: 0.9890
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0329 - accuracy: 0.9890
1643/1688 [============================>.] - ETA: 0s - loss: 0.0332 - accuracy: 0.9889
1673/1688 [============================>.] - ETA: 0s - loss: 0.0332 - accuracy: 0.9889
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0335 - accuracy: 0.9888 - val_loss: 0.0564 - val_accuracy: 0.9860
Epoch 6/10
1/1688 [..............................] - ETA: 2s - loss: 0.0097 - accuracy: 1.0000
34/1688 [..............................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9963
65/1688 [>.............................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9942
96/1688 [>.............................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9941
127/1688 [=>............................] - ETA: 2s - loss: 0.0167 - accuracy: 0.9938
157/1688 [=>............................] - ETA: 2s - loss: 0.0187 - accuracy: 0.9942
187/1688 [==>...........................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9942
218/1688 [==>...........................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9941
249/1688 [===>..........................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9936
280/1688 [===>..........................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9935
311/1688 [====>.........................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9934
341/1688 [=====>........................] - ETA: 2s - loss: 0.0217 - accuracy: 0.9932
371/1688 [=====>........................] - ETA: 2s - loss: 0.0223 - accuracy: 0.9931
401/1688 [======>.......................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9926
431/1688 [======>.......................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9928
462/1688 [=======>......................] - ETA: 2s - loss: 0.0224 - accuracy: 0.9928
492/1688 [=======>......................] - ETA: 1s - loss: 0.0224 - accuracy: 0.9928
522/1688 [========>.....................] - ETA: 1s - loss: 0.0219 - accuracy: 0.9931
552/1688 [========>.....................] - ETA: 1s - loss: 0.0218 - accuracy: 0.9930
582/1688 [=========>....................] - ETA: 1s - loss: 0.0235 - accuracy: 0.9925
612/1688 [=========>....................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9923
642/1688 [==========>...................] - ETA: 1s - loss: 0.0235 - accuracy: 0.9923
673/1688 [==========>...................] - ETA: 1s - loss: 0.0235 - accuracy: 0.9922
703/1688 [===========>..................] - ETA: 1s - loss: 0.0241 - accuracy: 0.9921
733/1688 [============>.................] - ETA: 1s - loss: 0.0246 - accuracy: 0.9921
764/1688 [============>.................] - ETA: 1s - loss: 0.0243 - accuracy: 0.9922
795/1688 [=============>................] - ETA: 1s - loss: 0.0255 - accuracy: 0.9917
825/1688 [=============>................] - ETA: 1s - loss: 0.0257 - accuracy: 0.9915
856/1688 [==============>...............] - ETA: 1s - loss: 0.0261 - accuracy: 0.9913
886/1688 [==============>...............] - ETA: 1s - loss: 0.0264 - accuracy: 0.9912
917/1688 [===============>..............] - ETA: 1s - loss: 0.0264 - accuracy: 0.9913
947/1688 [===============>..............] - ETA: 1s - loss: 0.0264 - accuracy: 0.9914
977/1688 [================>.............] - ETA: 1s - loss: 0.0263 - accuracy: 0.9914
1007/1688 [================>.............] - ETA: 1s - loss: 0.0266 - accuracy: 0.9913
1038/1688 [=================>............] - ETA: 1s - loss: 0.0270 - accuracy: 0.9911
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0270 - accuracy: 0.9911
1100/1688 [==================>...........] - ETA: 0s - loss: 0.0269 - accuracy: 0.9911
1130/1688 [===================>..........] - ETA: 0s - loss: 0.0270 - accuracy: 0.9911
1160/1688 [===================>..........] - ETA: 0s - loss: 0.0273 - accuracy: 0.9911
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0273 - accuracy: 0.9911
1220/1688 [====================>.........] - ETA: 0s - loss: 0.0273 - accuracy: 0.9911
1251/1688 [=====================>........] - ETA: 0s - loss: 0.0273 - accuracy: 0.9911
1281/1688 [=====================>........] - ETA: 0s - loss: 0.0272 - accuracy: 0.9911
1311/1688 [======================>.......] - ETA: 0s - loss: 0.0276 - accuracy: 0.9910
1342/1688 [======================>.......] - ETA: 0s - loss: 0.0278 - accuracy: 0.9910
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0279 - accuracy: 0.9909
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0283 - accuracy: 0.9908
1434/1688 [========================>.....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9908
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9907
1496/1688 [=========================>....] - ETA: 0s - loss: 0.0282 - accuracy: 0.9908
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0281 - accuracy: 0.9908
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0281 - accuracy: 0.9908
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0282 - accuracy: 0.9908
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0278 - accuracy: 0.9909
1648/1688 [============================>.] - ETA: 0s - loss: 0.0278 - accuracy: 0.9909
1678/1688 [============================>.] - ETA: 0s - loss: 0.0279 - accuracy: 0.9909
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0281 - accuracy: 0.9909 - val_loss: 0.0577 - val_accuracy: 0.9842
Epoch 7/10
1/1688 [..............................] - ETA: 2s - loss: 0.0027 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9915
65/1688 [>.............................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9923
95/1688 [>.............................] - ETA: 2s - loss: 0.0234 - accuracy: 0.9924
125/1688 [=>............................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9927
156/1688 [=>............................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9936
186/1688 [==>...........................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9936
216/1688 [==>...........................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9938
247/1688 [===>..........................] - ETA: 2s - loss: 0.0186 - accuracy: 0.9941
277/1688 [===>..........................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9946
308/1688 [====>.........................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9947
338/1688 [=====>........................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9946
368/1688 [=====>........................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9945
398/1688 [======>.......................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9945
428/1688 [======>.......................] - ETA: 2s - loss: 0.0177 - accuracy: 0.9945
459/1688 [=======>......................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9944
490/1688 [=======>......................] - ETA: 1s - loss: 0.0175 - accuracy: 0.9946
521/1688 [========>.....................] - ETA: 1s - loss: 0.0175 - accuracy: 0.9945
551/1688 [========>.....................] - ETA: 1s - loss: 0.0176 - accuracy: 0.9944
581/1688 [=========>....................] - ETA: 1s - loss: 0.0181 - accuracy: 0.9944
612/1688 [=========>....................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9946
642/1688 [==========>...................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9945
673/1688 [==========>...................] - ETA: 1s - loss: 0.0184 - accuracy: 0.9942
704/1688 [===========>..................] - ETA: 1s - loss: 0.0185 - accuracy: 0.9941
734/1688 [============>.................] - ETA: 1s - loss: 0.0188 - accuracy: 0.9941
764/1688 [============>.................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9937
795/1688 [=============>................] - ETA: 1s - loss: 0.0200 - accuracy: 0.9937
826/1688 [=============>................] - ETA: 1s - loss: 0.0198 - accuracy: 0.9938
857/1688 [==============>...............] - ETA: 1s - loss: 0.0200 - accuracy: 0.9937
887/1688 [==============>...............] - ETA: 1s - loss: 0.0196 - accuracy: 0.9937
917/1688 [===============>..............] - ETA: 1s - loss: 0.0199 - accuracy: 0.9937
948/1688 [===============>..............] - ETA: 1s - loss: 0.0200 - accuracy: 0.9935
979/1688 [================>.............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9934
1010/1688 [================>.............] - ETA: 1s - loss: 0.0211 - accuracy: 0.9933
1040/1688 [=================>............] - ETA: 1s - loss: 0.0214 - accuracy: 0.9932
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0216 - accuracy: 0.9931
1100/1688 [==================>...........] - ETA: 0s - loss: 0.0215 - accuracy: 0.9931
1131/1688 [===================>..........] - ETA: 0s - loss: 0.0212 - accuracy: 0.9932
1162/1688 [===================>..........] - ETA: 0s - loss: 0.0213 - accuracy: 0.9932
1193/1688 [====================>.........] - ETA: 0s - loss: 0.0210 - accuracy: 0.9933
1223/1688 [====================>.........] - ETA: 0s - loss: 0.0212 - accuracy: 0.9934
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0214 - accuracy: 0.9932
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0217 - accuracy: 0.9931
1313/1688 [======================>.......] - ETA: 0s - loss: 0.0216 - accuracy: 0.9931
1344/1688 [======================>.......] - ETA: 0s - loss: 0.0217 - accuracy: 0.9930
1375/1688 [=======================>......] - ETA: 0s - loss: 0.0216 - accuracy: 0.9930
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0215 - accuracy: 0.9931
1435/1688 [========================>.....] - ETA: 0s - loss: 0.0217 - accuracy: 0.9930
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0218 - accuracy: 0.9930
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0220 - accuracy: 0.9929
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0222 - accuracy: 0.9929
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0226 - accuracy: 0.9928
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0228 - accuracy: 0.9927
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0227 - accuracy: 0.9927
1645/1688 [============================>.] - ETA: 0s - loss: 0.0226 - accuracy: 0.9928
1676/1688 [============================>.] - ETA: 0s - loss: 0.0227 - accuracy: 0.9927
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0227 - accuracy: 0.9927 - val_loss: 0.0675 - val_accuracy: 0.9840
Epoch 8/10
1/1688 [..............................] - ETA: 2s - loss: 0.0872 - accuracy: 0.9688
33/1688 [..............................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9943
63/1688 [>.............................] - ETA: 2s - loss: 0.0185 - accuracy: 0.9926
93/1688 [>.............................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9929
123/1688 [=>............................] - ETA: 2s - loss: 0.0191 - accuracy: 0.9924
153/1688 [=>............................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9935
183/1688 [==>...........................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9939
212/1688 [==>...........................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9941
241/1688 [===>..........................] - ETA: 2s - loss: 0.0168 - accuracy: 0.9936
271/1688 [===>..........................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9933
302/1688 [====>.........................] - ETA: 2s - loss: 0.0175 - accuracy: 0.9931
333/1688 [====>.........................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9931
362/1688 [=====>........................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9934
393/1688 [=====>........................] - ETA: 2s - loss: 0.0167 - accuracy: 0.9935
424/1688 [======>.......................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9934
454/1688 [=======>......................] - ETA: 2s - loss: 0.0169 - accuracy: 0.9937
484/1688 [=======>......................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9937
515/1688 [========>.....................] - ETA: 1s - loss: 0.0171 - accuracy: 0.9937
545/1688 [========>.....................] - ETA: 1s - loss: 0.0176 - accuracy: 0.9936
577/1688 [=========>....................] - ETA: 1s - loss: 0.0181 - accuracy: 0.9937
609/1688 [=========>....................] - ETA: 1s - loss: 0.0176 - accuracy: 0.9939
640/1688 [==========>...................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9937
671/1688 [==========>...................] - ETA: 1s - loss: 0.0180 - accuracy: 0.9936
701/1688 [===========>..................] - ETA: 1s - loss: 0.0180 - accuracy: 0.9935
734/1688 [============>.................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9936
765/1688 [============>.................] - ETA: 1s - loss: 0.0176 - accuracy: 0.9938
796/1688 [=============>................] - ETA: 1s - loss: 0.0172 - accuracy: 0.9939
826/1688 [=============>................] - ETA: 1s - loss: 0.0172 - accuracy: 0.9939
856/1688 [==============>...............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9939
886/1688 [==============>...............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9938
916/1688 [===============>..............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9939
946/1688 [===============>..............] - ETA: 1s - loss: 0.0173 - accuracy: 0.9938
976/1688 [================>.............] - ETA: 1s - loss: 0.0175 - accuracy: 0.9939
1006/1688 [================>.............] - ETA: 1s - loss: 0.0180 - accuracy: 0.9937
1037/1688 [=================>............] - ETA: 1s - loss: 0.0179 - accuracy: 0.9937
1067/1688 [=================>............] - ETA: 1s - loss: 0.0182 - accuracy: 0.9937
1098/1688 [==================>...........] - ETA: 0s - loss: 0.0183 - accuracy: 0.9937
1128/1688 [===================>..........] - ETA: 0s - loss: 0.0181 - accuracy: 0.9938
1159/1688 [===================>..........] - ETA: 0s - loss: 0.0183 - accuracy: 0.9938
1190/1688 [====================>.........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9937
1220/1688 [====================>.........] - ETA: 0s - loss: 0.0190 - accuracy: 0.9937
1251/1688 [=====================>........] - ETA: 0s - loss: 0.0190 - accuracy: 0.9937
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0191 - accuracy: 0.9936
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0190 - accuracy: 0.9936
1342/1688 [======================>.......] - ETA: 0s - loss: 0.0189 - accuracy: 0.9936
1373/1688 [=======================>......] - ETA: 0s - loss: 0.0192 - accuracy: 0.9936
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0192 - accuracy: 0.9935
1434/1688 [========================>.....] - ETA: 0s - loss: 0.0191 - accuracy: 0.9935
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0192 - accuracy: 0.9935
1494/1688 [=========================>....] - ETA: 0s - loss: 0.0193 - accuracy: 0.9935
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0195 - accuracy: 0.9933
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0198 - accuracy: 0.9932
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0198 - accuracy: 0.9932
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0197 - accuracy: 0.9932
1646/1688 [============================>.] - ETA: 0s - loss: 0.0196 - accuracy: 0.9932
1677/1688 [============================>.] - ETA: 0s - loss: 0.0197 - accuracy: 0.9932
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0196 - accuracy: 0.9933 - val_loss: 0.0700 - val_accuracy: 0.9827
Epoch 9/10
1/1688 [..............................] - ETA: 2s - loss: 0.0030 - accuracy: 1.0000
33/1688 [..............................] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
63/1688 [>.............................] - ETA: 2s - loss: 0.0101 - accuracy: 0.9980
94/1688 [>.............................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9953
124/1688 [=>............................] - ETA: 2s - loss: 0.0171 - accuracy: 0.9945
154/1688 [=>............................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9941
185/1688 [==>...........................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9951
215/1688 [==>...........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9955
245/1688 [===>..........................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9958
275/1688 [===>..........................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9960
305/1688 [====>.........................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9956
336/1688 [====>.........................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9957
367/1688 [=====>........................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9959
398/1688 [======>.......................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9957
428/1688 [======>.......................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9953
459/1688 [=======>......................] - ETA: 2s - loss: 0.0156 - accuracy: 0.9953
490/1688 [=======>......................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9953
520/1688 [========>.....................] - ETA: 1s - loss: 0.0160 - accuracy: 0.9950
550/1688 [========>.....................] - ETA: 1s - loss: 0.0164 - accuracy: 0.9948
581/1688 [=========>....................] - ETA: 1s - loss: 0.0163 - accuracy: 0.9948
611/1688 [=========>....................] - ETA: 1s - loss: 0.0162 - accuracy: 0.9949
640/1688 [==========>...................] - ETA: 1s - loss: 0.0166 - accuracy: 0.9949
671/1688 [==========>...................] - ETA: 1s - loss: 0.0164 - accuracy: 0.9949
701/1688 [===========>..................] - ETA: 1s - loss: 0.0168 - accuracy: 0.9947
732/1688 [============>.................] - ETA: 1s - loss: 0.0169 - accuracy: 0.9947
762/1688 [============>.................] - ETA: 1s - loss: 0.0166 - accuracy: 0.9948
793/1688 [=============>................] - ETA: 1s - loss: 0.0163 - accuracy: 0.9948
824/1688 [=============>................] - ETA: 1s - loss: 0.0163 - accuracy: 0.9948
855/1688 [==============>...............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9949
886/1688 [==============>...............] - ETA: 1s - loss: 0.0158 - accuracy: 0.9949
916/1688 [===============>..............] - ETA: 1s - loss: 0.0156 - accuracy: 0.9950
946/1688 [===============>..............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9950
976/1688 [================>.............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9947
1007/1688 [================>.............] - ETA: 1s - loss: 0.0160 - accuracy: 0.9948
1037/1688 [=================>............] - ETA: 1s - loss: 0.0162 - accuracy: 0.9947
1068/1688 [=================>............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9946
1098/1688 [==================>...........] - ETA: 0s - loss: 0.0165 - accuracy: 0.9946
1128/1688 [===================>..........] - ETA: 0s - loss: 0.0164 - accuracy: 0.9946
1158/1688 [===================>..........] - ETA: 0s - loss: 0.0169 - accuracy: 0.9945
1189/1688 [====================>.........] - ETA: 0s - loss: 0.0169 - accuracy: 0.9945
1220/1688 [====================>.........] - ETA: 0s - loss: 0.0171 - accuracy: 0.9944
1251/1688 [=====================>........] - ETA: 0s - loss: 0.0171 - accuracy: 0.9944
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0172 - accuracy: 0.9943
1312/1688 [======================>.......] - ETA: 0s - loss: 0.0174 - accuracy: 0.9942
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0176 - accuracy: 0.9942
1374/1688 [=======================>......] - ETA: 0s - loss: 0.0181 - accuracy: 0.9940
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0186 - accuracy: 0.9939
1436/1688 [========================>.....] - ETA: 0s - loss: 0.0185 - accuracy: 0.9939
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0187 - accuracy: 0.9938
1498/1688 [=========================>....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9938
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0184 - accuracy: 0.9939
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0184 - accuracy: 0.9939
1588/1688 [===========================>..] - ETA: 0s - loss: 0.0184 - accuracy: 0.9940
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0183 - accuracy: 0.9940
1649/1688 [============================>.] - ETA: 0s - loss: 0.0183 - accuracy: 0.9940
1680/1688 [============================>.] - ETA: 0s - loss: 0.0181 - accuracy: 0.9940
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0182 - accuracy: 0.9940 - val_loss: 0.0572 - val_accuracy: 0.9858
Epoch 10/10
1/1688 [..............................] - ETA: 2s - loss: 0.0236 - accuracy: 0.9688
34/1688 [..............................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9954
64/1688 [>.............................] - ETA: 2s - loss: 0.0088 - accuracy: 0.9966
94/1688 [>.............................] - ETA: 2s - loss: 0.0116 - accuracy: 0.9957
124/1688 [=>............................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9955
154/1688 [=>............................] - ETA: 2s - loss: 0.0126 - accuracy: 0.9951
184/1688 [==>...........................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9952
215/1688 [==>...........................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9953
245/1688 [===>..........................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9953
276/1688 [===>..........................] - ETA: 2s - loss: 0.0122 - accuracy: 0.9954
306/1688 [====>.........................] - ETA: 2s - loss: 0.0121 - accuracy: 0.9953
336/1688 [====>.........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9957
366/1688 [=====>........................] - ETA: 2s - loss: 0.0115 - accuracy: 0.9957
396/1688 [======>.......................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9958
426/1688 [======>.......................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9957
456/1688 [=======>......................] - ETA: 2s - loss: 0.0118 - accuracy: 0.9958
486/1688 [=======>......................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9958
516/1688 [========>.....................] - ETA: 1s - loss: 0.0117 - accuracy: 0.9958
547/1688 [========>.....................] - ETA: 1s - loss: 0.0119 - accuracy: 0.9958
577/1688 [=========>....................] - ETA: 1s - loss: 0.0137 - accuracy: 0.9955
607/1688 [=========>....................] - ETA: 1s - loss: 0.0138 - accuracy: 0.9954
638/1688 [==========>...................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9953
668/1688 [==========>...................] - ETA: 1s - loss: 0.0139 - accuracy: 0.9953
700/1688 [===========>..................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9951
731/1688 [===========>..................] - ETA: 1s - loss: 0.0148 - accuracy: 0.9948
762/1688 [============>.................] - ETA: 1s - loss: 0.0151 - accuracy: 0.9947
792/1688 [=============>................] - ETA: 1s - loss: 0.0157 - accuracy: 0.9945
822/1688 [=============>................] - ETA: 1s - loss: 0.0157 - accuracy: 0.9946
852/1688 [==============>...............] - ETA: 1s - loss: 0.0155 - accuracy: 0.9946
882/1688 [==============>...............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9947
912/1688 [===============>..............] - ETA: 1s - loss: 0.0151 - accuracy: 0.9947
943/1688 [===============>..............] - ETA: 1s - loss: 0.0150 - accuracy: 0.9947
973/1688 [================>.............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9948
1003/1688 [================>.............] - ETA: 1s - loss: 0.0149 - accuracy: 0.9948
1033/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9949
1063/1688 [=================>............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9949
1093/1688 [==================>...........] - ETA: 0s - loss: 0.0146 - accuracy: 0.9949
1124/1688 [==================>...........] - ETA: 0s - loss: 0.0147 - accuracy: 0.9949
1154/1688 [===================>..........] - ETA: 0s - loss: 0.0151 - accuracy: 0.9948
1184/1688 [====================>.........] - ETA: 0s - loss: 0.0151 - accuracy: 0.9948
1215/1688 [====================>.........] - ETA: 0s - loss: 0.0151 - accuracy: 0.9949
1246/1688 [=====================>........] - ETA: 0s - loss: 0.0150 - accuracy: 0.9949
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0150 - accuracy: 0.9949
1308/1688 [======================>.......] - ETA: 0s - loss: 0.0149 - accuracy: 0.9950
1338/1688 [======================>.......] - ETA: 0s - loss: 0.0149 - accuracy: 0.9950
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0149 - accuracy: 0.9950
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0151 - accuracy: 0.9949
1430/1688 [========================>.....] - ETA: 0s - loss: 0.0151 - accuracy: 0.9949
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0151 - accuracy: 0.9949
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0151 - accuracy: 0.9948
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0150 - accuracy: 0.9949
1552/1688 [==========================>...] - ETA: 0s - loss: 0.0154 - accuracy: 0.9948
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0154 - accuracy: 0.9948
1612/1688 [===========================>..] - ETA: 0s - loss: 0.0153 - accuracy: 0.9949
1642/1688 [============================>.] - ETA: 0s - loss: 0.0153 - accuracy: 0.9949
1673/1688 [============================>.] - ETA: 0s - loss: 0.0153 - accuracy: 0.9949
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0153 - accuracy: 0.9949 - val_loss: 0.0596 - val_accuracy: 0.9873
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Test accuracy: 0.9851999878883362
2. Quantize
2.1. 8-bit quantization
An Akida accelerator processes 8, 4, 2, or 1 bit integer activations and weights. Therefore, the floating point Keras model must be quantized in preparation to run on an Akida accelerator.
The QuantizeML quantize function can be used to quantize a Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating point Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.
The quantization process results in a Keras model with custom QuantizeML quantized layers substituted for the original Keras layers.
All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.
Note
The quantize function applies several transformations to
the original model. For example, it folds the batch normalization layers into the
corresponding neural layers. The new weights are computed according to this folding
operation.
from quantizeml.models import quantize, QuantizationParams
qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
model_quantized.summary()
/usr/local/lib/python3.8/dist-packages/quantizeml/models/quantize.py:400: UserWarning: Quantizing per-axis with random calibration samples is not accurate. Set QuantizationParams.per_tensor_activations=True when calibrating with random samples.
warnings.warn("Quantizing per-axis with random calibration samples is not accurate.\
1/1024 [..............................] - ETA: 2:31
58/1024 [>.............................] - ETA: 0s
117/1024 [==>...........................] - ETA: 0s
175/1024 [====>.........................] - ETA: 0s
234/1024 [=====>........................] - ETA: 0s
293/1024 [=======>......................] - ETA: 0s
351/1024 [=========>....................] - ETA: 0s
409/1024 [==========>...................] - ETA: 0s
469/1024 [============>.................] - ETA: 0s
527/1024 [==============>...............] - ETA: 0s
585/1024 [================>.............] - ETA: 0s
643/1024 [=================>............] - ETA: 0s
701/1024 [===================>..........] - ETA: 0s
760/1024 [=====================>........] - ETA: 0s
820/1024 [=======================>......] - ETA: 0s
878/1024 [========================>.....] - ETA: 0s
937/1024 [==========================>...] - ETA: 0s
995/1024 [============================>.] - ETA: 0s
1024/1024 [==============================] - 1s 862us/step
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
rescaling_input (InputLayer [(None, 28, 28, 1)] 0
)
rescaling (QuantizedRescali (None, 28, 28, 1) 0
ng)
conv2d (QuantizedConv2D) (None, 13, 13, 32) 320
re_lu (QuantizedReLU) (None, 13, 13, 32) 64
depthwise_conv2d (Quantized (None, 7, 7, 32) 384
DepthwiseConv2D)
conv2d_1 (QuantizedConv2D) (None, 7, 7, 64) 2112
re_lu_1 (QuantizedReLU) (None, 7, 7, 64) 128
flatten (QuantizedFlatten) (None, 3136) 0
dense (QuantizedDense) (None, 10) 31370
dequantizer (Dequantizer) [(None, 10)] 0
=================================================================
Total params: 34,378
Trainable params: 34,122
Non-trainable params: 256
_________________________________________________________________
Note
Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.
Check the quantized model accuracy.
def compile_evaluate(model):
""" Compiles and evaluates the model, then return accuracy score. """
model.compile(metrics=['accuracy'])
return model.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 8-bit quantization: 0.9812999963760376
2.2. Effect of calibration
The previous call to quantize was made with random samples for calibration
(default parameters). While the observed drop in accuracy is minimal, that is
around 1%, it can be worse on more complex models. Therefore, it is advised to
use a set of real samples from the training set for calibration during a call
to quantize.
Note that this remains a calibration step rather than a training step in that
no output labels are required. Furthermore, any relevant data could be used for
calibration. The recommended settings for calibration that are widely used to
obtain the zoo performance are:
1024 samples
a batch size of 100
2 epochs
model_quantized = quantize(model_keras, qparams=qparams,
samples=x_train, num_samples=1024, batch_size=100, epochs=2)
1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step
1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 983us/step
Check the accuracy for the quantized and calibrated model.
print('Test accuracy after calibration:', compile_evaluate(model_quantized))
Test accuracy after calibration: 0.9835000038146973
Calibrating with real samples on this model recovers the initial float accuracy.
2.3. 4-bit quantization
The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.
The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.
qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
samples=x_train, num_samples=1024, batch_size=100, epochs=2)
1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 979us/step
1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 941us/step
Check the 4-bit quantized accuracy.
print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 4-bit quantization: 0.9799000024795532
2.4. Model fine tuning (Quantization Aware Training)
When a model suffers from an accuracy drop after quantization, fine tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.
Note that since this is a fine tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=Adam(learning_rate=1e-4),
metrics=['accuracy'])
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
Epoch 1/5
1/1688 [..............................] - ETA: 1:05:43 - loss: 0.0155 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0515 - accuracy: 0.9880
25/1688 [..............................] - ETA: 7s - loss: 0.0356 - accuracy: 0.9900
37/1688 [..............................] - ETA: 7s - loss: 0.0309 - accuracy: 0.9916
49/1688 [..............................] - ETA: 7s - loss: 0.0295 - accuracy: 0.9923
61/1688 [>.............................] - ETA: 7s - loss: 0.0262 - accuracy: 0.9939
73/1688 [>.............................] - ETA: 7s - loss: 0.0298 - accuracy: 0.9914
85/1688 [>.............................] - ETA: 7s - loss: 0.0290 - accuracy: 0.9915
97/1688 [>.............................] - ETA: 6s - loss: 0.0285 - accuracy: 0.9913
109/1688 [>.............................] - ETA: 6s - loss: 0.0269 - accuracy: 0.9920
121/1688 [=>............................] - ETA: 6s - loss: 0.0259 - accuracy: 0.9917
133/1688 [=>............................] - ETA: 6s - loss: 0.0250 - accuracy: 0.9922
145/1688 [=>............................] - ETA: 6s - loss: 0.0243 - accuracy: 0.9922
157/1688 [=>............................] - ETA: 6s - loss: 0.0235 - accuracy: 0.9928
169/1688 [==>...........................] - ETA: 6s - loss: 0.0234 - accuracy: 0.9926
181/1688 [==>...........................] - ETA: 6s - loss: 0.0231 - accuracy: 0.9927
193/1688 [==>...........................] - ETA: 6s - loss: 0.0229 - accuracy: 0.9927
205/1688 [==>...........................] - ETA: 6s - loss: 0.0221 - accuracy: 0.9931
217/1688 [==>...........................] - ETA: 6s - loss: 0.0218 - accuracy: 0.9932
229/1688 [===>..........................] - ETA: 6s - loss: 0.0212 - accuracy: 0.9934
241/1688 [===>..........................] - ETA: 6s - loss: 0.0211 - accuracy: 0.9934
253/1688 [===>..........................] - ETA: 6s - loss: 0.0206 - accuracy: 0.9936
265/1688 [===>..........................] - ETA: 6s - loss: 0.0203 - accuracy: 0.9937
277/1688 [===>..........................] - ETA: 6s - loss: 0.0202 - accuracy: 0.9938
289/1688 [====>.........................] - ETA: 6s - loss: 0.0204 - accuracy: 0.9935
301/1688 [====>.........................] - ETA: 6s - loss: 0.0201 - accuracy: 0.9937
313/1688 [====>.........................] - ETA: 6s - loss: 0.0198 - accuracy: 0.9937
325/1688 [====>.........................] - ETA: 5s - loss: 0.0200 - accuracy: 0.9938
337/1688 [====>.........................] - ETA: 5s - loss: 0.0199 - accuracy: 0.9939
349/1688 [=====>........................] - ETA: 5s - loss: 0.0202 - accuracy: 0.9938
361/1688 [=====>........................] - ETA: 5s - loss: 0.0198 - accuracy: 0.9939
373/1688 [=====>........................] - ETA: 5s - loss: 0.0198 - accuracy: 0.9938
385/1688 [=====>........................] - ETA: 5s - loss: 0.0197 - accuracy: 0.9938
397/1688 [======>.......................] - ETA: 5s - loss: 0.0194 - accuracy: 0.9939
409/1688 [======>.......................] - ETA: 5s - loss: 0.0196 - accuracy: 0.9939
421/1688 [======>.......................] - ETA: 5s - loss: 0.0198 - accuracy: 0.9938
433/1688 [======>.......................] - ETA: 5s - loss: 0.0197 - accuracy: 0.9939
445/1688 [======>.......................] - ETA: 5s - loss: 0.0198 - accuracy: 0.9937
457/1688 [=======>......................] - ETA: 5s - loss: 0.0195 - accuracy: 0.9938
469/1688 [=======>......................] - ETA: 5s - loss: 0.0194 - accuracy: 0.9939
481/1688 [=======>......................] - ETA: 5s - loss: 0.0193 - accuracy: 0.9940
493/1688 [=======>......................] - ETA: 5s - loss: 0.0197 - accuracy: 0.9937
505/1688 [=======>......................] - ETA: 5s - loss: 0.0195 - accuracy: 0.9938
517/1688 [========>.....................] - ETA: 5s - loss: 0.0193 - accuracy: 0.9939
529/1688 [========>.....................] - ETA: 5s - loss: 0.0191 - accuracy: 0.9939
541/1688 [========>.....................] - ETA: 5s - loss: 0.0191 - accuracy: 0.9939
553/1688 [========>.....................] - ETA: 4s - loss: 0.0190 - accuracy: 0.9940
565/1688 [=========>....................] - ETA: 4s - loss: 0.0192 - accuracy: 0.9939
577/1688 [=========>....................] - ETA: 4s - loss: 0.0193 - accuracy: 0.9939
589/1688 [=========>....................] - ETA: 4s - loss: 0.0192 - accuracy: 0.9939
601/1688 [=========>....................] - ETA: 4s - loss: 0.0191 - accuracy: 0.9940
613/1688 [=========>....................] - ETA: 4s - loss: 0.0190 - accuracy: 0.9940
625/1688 [==========>...................] - ETA: 4s - loss: 0.0190 - accuracy: 0.9941
637/1688 [==========>...................] - ETA: 4s - loss: 0.0192 - accuracy: 0.9940
649/1688 [==========>...................] - ETA: 4s - loss: 0.0191 - accuracy: 0.9940
661/1688 [==========>...................] - ETA: 4s - loss: 0.0190 - accuracy: 0.9940
673/1688 [==========>...................] - ETA: 4s - loss: 0.0189 - accuracy: 0.9940
685/1688 [===========>..................] - ETA: 4s - loss: 0.0187 - accuracy: 0.9941
697/1688 [===========>..................] - ETA: 4s - loss: 0.0186 - accuracy: 0.9941
709/1688 [===========>..................] - ETA: 4s - loss: 0.0186 - accuracy: 0.9941
721/1688 [===========>..................] - ETA: 4s - loss: 0.0185 - accuracy: 0.9942
733/1688 [============>.................] - ETA: 4s - loss: 0.0187 - accuracy: 0.9941
745/1688 [============>.................] - ETA: 4s - loss: 0.0186 - accuracy: 0.9942
757/1688 [============>.................] - ETA: 4s - loss: 0.0186 - accuracy: 0.9941
769/1688 [============>.................] - ETA: 4s - loss: 0.0185 - accuracy: 0.9942
781/1688 [============>.................] - ETA: 3s - loss: 0.0185 - accuracy: 0.9942
793/1688 [=============>................] - ETA: 3s - loss: 0.0184 - accuracy: 0.9942
805/1688 [=============>................] - ETA: 3s - loss: 0.0183 - accuracy: 0.9943
817/1688 [=============>................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9943
829/1688 [=============>................] - ETA: 3s - loss: 0.0181 - accuracy: 0.9943
840/1688 [=============>................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9943
852/1688 [==============>...............] - ETA: 3s - loss: 0.0181 - accuracy: 0.9943
864/1688 [==============>...............] - ETA: 3s - loss: 0.0181 - accuracy: 0.9943
876/1688 [==============>...............] - ETA: 3s - loss: 0.0180 - accuracy: 0.9944
888/1688 [==============>...............] - ETA: 3s - loss: 0.0179 - accuracy: 0.9944
900/1688 [==============>...............] - ETA: 3s - loss: 0.0178 - accuracy: 0.9945
912/1688 [===============>..............] - ETA: 3s - loss: 0.0179 - accuracy: 0.9944
924/1688 [===============>..............] - ETA: 3s - loss: 0.0180 - accuracy: 0.9944
936/1688 [===============>..............] - ETA: 3s - loss: 0.0179 - accuracy: 0.9945
948/1688 [===============>..............] - ETA: 3s - loss: 0.0178 - accuracy: 0.9945
960/1688 [================>.............] - ETA: 3s - loss: 0.0177 - accuracy: 0.9946
972/1688 [================>.............] - ETA: 3s - loss: 0.0177 - accuracy: 0.9946
984/1688 [================>.............] - ETA: 3s - loss: 0.0177 - accuracy: 0.9946
996/1688 [================>.............] - ETA: 3s - loss: 0.0177 - accuracy: 0.9945
1008/1688 [================>.............] - ETA: 2s - loss: 0.0178 - accuracy: 0.9945
1020/1688 [=================>............] - ETA: 2s - loss: 0.0177 - accuracy: 0.9946
1032/1688 [=================>............] - ETA: 2s - loss: 0.0177 - accuracy: 0.9946
1044/1688 [=================>............] - ETA: 2s - loss: 0.0177 - accuracy: 0.9946
1056/1688 [=================>............] - ETA: 2s - loss: 0.0176 - accuracy: 0.9946
1068/1688 [=================>............] - ETA: 2s - loss: 0.0175 - accuracy: 0.9947
1080/1688 [==================>...........] - ETA: 2s - loss: 0.0174 - accuracy: 0.9947
1092/1688 [==================>...........] - ETA: 2s - loss: 0.0174 - accuracy: 0.9947
1104/1688 [==================>...........] - ETA: 2s - loss: 0.0173 - accuracy: 0.9947
1116/1688 [==================>...........] - ETA: 2s - loss: 0.0173 - accuracy: 0.9947
1128/1688 [===================>..........] - ETA: 2s - loss: 0.0174 - accuracy: 0.9946
1140/1688 [===================>..........] - ETA: 2s - loss: 0.0173 - accuracy: 0.9947
1152/1688 [===================>..........] - ETA: 2s - loss: 0.0173 - accuracy: 0.9947
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0172 - accuracy: 0.9947
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0172 - accuracy: 0.9947
1188/1688 [====================>.........] - ETA: 2s - loss: 0.0172 - accuracy: 0.9947
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0171 - accuracy: 0.9947
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0171 - accuracy: 0.9948
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0170 - accuracy: 0.9948
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0169 - accuracy: 0.9948
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9949
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9949
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0167 - accuracy: 0.9949
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0166 - accuracy: 0.9950
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0166 - accuracy: 0.9950
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0165 - accuracy: 0.9950
1320/1688 [======================>.......] - ETA: 1s - loss: 0.0165 - accuracy: 0.9950
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0165 - accuracy: 0.9950
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0164 - accuracy: 0.9951
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0163 - accuracy: 0.9951
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0162 - accuracy: 0.9952
1380/1688 [=======================>......] - ETA: 1s - loss: 0.0161 - accuracy: 0.9952
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0161 - accuracy: 0.9952
1404/1688 [=======================>......] - ETA: 1s - loss: 0.0161 - accuracy: 0.9953
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0160 - accuracy: 0.9953
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0160 - accuracy: 0.9953
1440/1688 [========================>.....] - ETA: 1s - loss: 0.0160 - accuracy: 0.9953
1452/1688 [========================>.....] - ETA: 1s - loss: 0.0159 - accuracy: 0.9954
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0159 - accuracy: 0.9954
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9954
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9954
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9954
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9953
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0159 - accuracy: 0.9953
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0158 - accuracy: 0.9953
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0160 - accuracy: 0.9953
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0160 - accuracy: 0.9953
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0159 - accuracy: 0.9953
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0159 - accuracy: 0.9953
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0159 - accuracy: 0.9953
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0159 - accuracy: 0.9953
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0159 - accuracy: 0.9953
1632/1688 [============================>.] - ETA: 0s - loss: 0.0159 - accuracy: 0.9953
1644/1688 [============================>.] - ETA: 0s - loss: 0.0158 - accuracy: 0.9953
1656/1688 [============================>.] - ETA: 0s - loss: 0.0158 - accuracy: 0.9953
1668/1688 [============================>.] - ETA: 0s - loss: 0.0158 - accuracy: 0.9953
1680/1688 [============================>.] - ETA: 0s - loss: 0.0157 - accuracy: 0.9953
1688/1688 [==============================] - 11s 5ms/step - loss: 0.0158 - accuracy: 0.9953 - val_loss: 0.0494 - val_accuracy: 0.9880
Epoch 2/5
1/1688 [..............................] - ETA: 7s - loss: 0.0040 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0103 - accuracy: 0.9928
25/1688 [..............................] - ETA: 7s - loss: 0.0103 - accuracy: 0.9937
37/1688 [..............................] - ETA: 7s - loss: 0.0105 - accuracy: 0.9949
49/1688 [..............................] - ETA: 7s - loss: 0.0108 - accuracy: 0.9949
61/1688 [>.............................] - ETA: 7s - loss: 0.0108 - accuracy: 0.9954
73/1688 [>.............................] - ETA: 7s - loss: 0.0114 - accuracy: 0.9953
85/1688 [>.............................] - ETA: 7s - loss: 0.0110 - accuracy: 0.9960
97/1688 [>.............................] - ETA: 6s - loss: 0.0115 - accuracy: 0.9958
109/1688 [>.............................] - ETA: 6s - loss: 0.0110 - accuracy: 0.9963
121/1688 [=>............................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9964
133/1688 [=>............................] - ETA: 6s - loss: 0.0112 - accuracy: 0.9962
145/1688 [=>............................] - ETA: 6s - loss: 0.0111 - accuracy: 0.9963
157/1688 [=>............................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9966
169/1688 [==>...........................] - ETA: 6s - loss: 0.0110 - accuracy: 0.9965
181/1688 [==>...........................] - ETA: 6s - loss: 0.0113 - accuracy: 0.9965
193/1688 [==>...........................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9964
205/1688 [==>...........................] - ETA: 6s - loss: 0.0121 - accuracy: 0.9963
217/1688 [==>...........................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9965
229/1688 [===>..........................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9963
241/1688 [===>..........................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9964
253/1688 [===>..........................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9963
265/1688 [===>..........................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9965
277/1688 [===>..........................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9965
289/1688 [====>.........................] - ETA: 6s - loss: 0.0115 - accuracy: 0.9965
301/1688 [====>.........................] - ETA: 6s - loss: 0.0113 - accuracy: 0.9967
313/1688 [====>.........................] - ETA: 6s - loss: 0.0114 - accuracy: 0.9967
325/1688 [====>.........................] - ETA: 5s - loss: 0.0115 - accuracy: 0.9967
337/1688 [====>.........................] - ETA: 5s - loss: 0.0115 - accuracy: 0.9968
349/1688 [=====>........................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9969
361/1688 [=====>........................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9969
373/1688 [=====>........................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9969
385/1688 [=====>........................] - ETA: 5s - loss: 0.0115 - accuracy: 0.9968
397/1688 [======>.......................] - ETA: 5s - loss: 0.0115 - accuracy: 0.9968
409/1688 [======>.......................] - ETA: 5s - loss: 0.0118 - accuracy: 0.9966
421/1688 [======>.......................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9967
433/1688 [======>.......................] - ETA: 5s - loss: 0.0118 - accuracy: 0.9968
445/1688 [======>.......................] - ETA: 5s - loss: 0.0116 - accuracy: 0.9968
457/1688 [=======>......................] - ETA: 5s - loss: 0.0120 - accuracy: 0.9967
469/1688 [=======>......................] - ETA: 5s - loss: 0.0123 - accuracy: 0.9967
481/1688 [=======>......................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9968
493/1688 [=======>......................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9968
505/1688 [=======>......................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9968
517/1688 [========>.....................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9967
529/1688 [========>.....................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9968
541/1688 [========>.....................] - ETA: 5s - loss: 0.0123 - accuracy: 0.9966
553/1688 [========>.....................] - ETA: 4s - loss: 0.0122 - accuracy: 0.9967
565/1688 [=========>....................] - ETA: 4s - loss: 0.0122 - accuracy: 0.9967
577/1688 [=========>....................] - ETA: 4s - loss: 0.0121 - accuracy: 0.9967
589/1688 [=========>....................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9968
601/1688 [=========>....................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9968
613/1688 [=========>....................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9968
625/1688 [==========>...................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9969
637/1688 [==========>...................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9969
649/1688 [==========>...................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9969
661/1688 [==========>...................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9969
672/1688 [==========>...................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9969
684/1688 [===========>..................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9969
696/1688 [===========>..................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9969
708/1688 [===========>..................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9968
720/1688 [===========>..................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9967
732/1688 [============>.................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9968
744/1688 [============>.................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9968
756/1688 [============>.................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9969
768/1688 [============>.................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9968
780/1688 [============>.................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9969
792/1688 [=============>................] - ETA: 3s - loss: 0.0117 - accuracy: 0.9969
804/1688 [=============>................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9969
816/1688 [=============>................] - ETA: 3s - loss: 0.0119 - accuracy: 0.9969
828/1688 [=============>................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9969
840/1688 [=============>................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9970
852/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9970
864/1688 [==============>...............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9970
876/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9970
888/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9969
900/1688 [==============>...............] - ETA: 3s - loss: 0.0118 - accuracy: 0.9969
912/1688 [===============>..............] - ETA: 3s - loss: 0.0118 - accuracy: 0.9969
924/1688 [===============>..............] - ETA: 3s - loss: 0.0119 - accuracy: 0.9969
936/1688 [===============>..............] - ETA: 3s - loss: 0.0119 - accuracy: 0.9969
948/1688 [===============>..............] - ETA: 3s - loss: 0.0119 - accuracy: 0.9969
960/1688 [================>.............] - ETA: 3s - loss: 0.0122 - accuracy: 0.9968
972/1688 [================>.............] - ETA: 3s - loss: 0.0121 - accuracy: 0.9969
984/1688 [================>.............] - ETA: 3s - loss: 0.0121 - accuracy: 0.9969
996/1688 [================>.............] - ETA: 3s - loss: 0.0120 - accuracy: 0.9969
1008/1688 [================>.............] - ETA: 2s - loss: 0.0120 - accuracy: 0.9970
1020/1688 [=================>............] - ETA: 2s - loss: 0.0119 - accuracy: 0.9970
1032/1688 [=================>............] - ETA: 2s - loss: 0.0119 - accuracy: 0.9970
1044/1688 [=================>............] - ETA: 2s - loss: 0.0119 - accuracy: 0.9970
1056/1688 [=================>............] - ETA: 2s - loss: 0.0120 - accuracy: 0.9970
1068/1688 [=================>............] - ETA: 2s - loss: 0.0119 - accuracy: 0.9970
1080/1688 [==================>...........] - ETA: 2s - loss: 0.0119 - accuracy: 0.9970
1092/1688 [==================>...........] - ETA: 2s - loss: 0.0119 - accuracy: 0.9970
1104/1688 [==================>...........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9970
1116/1688 [==================>...........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9970
1128/1688 [===================>..........] - ETA: 2s - loss: 0.0119 - accuracy: 0.9970
1140/1688 [===================>..........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9970
1152/1688 [===================>..........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9970
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0120 - accuracy: 0.9970
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0121 - accuracy: 0.9969
1188/1688 [====================>.........] - ETA: 2s - loss: 0.0121 - accuracy: 0.9969
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0121 - accuracy: 0.9969
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0122 - accuracy: 0.9969
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0123 - accuracy: 0.9969
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0124 - accuracy: 0.9968
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0123 - accuracy: 0.9968
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0123 - accuracy: 0.9969
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0123 - accuracy: 0.9969
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0122 - accuracy: 0.9969
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0122 - accuracy: 0.9969
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0121 - accuracy: 0.9969
1320/1688 [======================>.......] - ETA: 1s - loss: 0.0121 - accuracy: 0.9969
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0121 - accuracy: 0.9969
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9969
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9970
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9970
1380/1688 [=======================>......] - ETA: 1s - loss: 0.0120 - accuracy: 0.9970
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9970
1404/1688 [=======================>......] - ETA: 1s - loss: 0.0119 - accuracy: 0.9970
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0120 - accuracy: 0.9969
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0120 - accuracy: 0.9969
1440/1688 [========================>.....] - ETA: 1s - loss: 0.0120 - accuracy: 0.9969
1452/1688 [========================>.....] - ETA: 1s - loss: 0.0119 - accuracy: 0.9970
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0119 - accuracy: 0.9969
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0118 - accuracy: 0.9970
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0118 - accuracy: 0.9970
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0118 - accuracy: 0.9969
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0118 - accuracy: 0.9969
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9969
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9969
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0119 - accuracy: 0.9970
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0118 - accuracy: 0.9970
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0118 - accuracy: 0.9970
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0117 - accuracy: 0.9970
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0118 - accuracy: 0.9970
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0117 - accuracy: 0.9970
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0117 - accuracy: 0.9970
1632/1688 [============================>.] - ETA: 0s - loss: 0.0117 - accuracy: 0.9970
1644/1688 [============================>.] - ETA: 0s - loss: 0.0117 - accuracy: 0.9970
1656/1688 [============================>.] - ETA: 0s - loss: 0.0118 - accuracy: 0.9970
1668/1688 [============================>.] - ETA: 0s - loss: 0.0117 - accuracy: 0.9970
1680/1688 [============================>.] - ETA: 0s - loss: 0.0117 - accuracy: 0.9970
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0117 - accuracy: 0.9970 - val_loss: 0.0506 - val_accuracy: 0.9887
Epoch 3/5
1/1688 [..............................] - ETA: 7s - loss: 0.0318 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0149 - accuracy: 0.9976
25/1688 [..............................] - ETA: 7s - loss: 0.0109 - accuracy: 0.9987
37/1688 [..............................] - ETA: 7s - loss: 0.0116 - accuracy: 0.9983
49/1688 [..............................] - ETA: 7s - loss: 0.0114 - accuracy: 0.9981
61/1688 [>.............................] - ETA: 7s - loss: 0.0108 - accuracy: 0.9985
73/1688 [>.............................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9987
85/1688 [>.............................] - ETA: 7s - loss: 0.0112 - accuracy: 0.9982
97/1688 [>.............................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9977
109/1688 [>.............................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9977
121/1688 [=>............................] - ETA: 6s - loss: 0.0114 - accuracy: 0.9977
133/1688 [=>............................] - ETA: 6s - loss: 0.0112 - accuracy: 0.9979
145/1688 [=>............................] - ETA: 6s - loss: 0.0107 - accuracy: 0.9981
157/1688 [=>............................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9980
169/1688 [==>...........................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9980
181/1688 [==>...........................] - ETA: 6s - loss: 0.0106 - accuracy: 0.9979
193/1688 [==>...........................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9979
205/1688 [==>...........................] - ETA: 6s - loss: 0.0107 - accuracy: 0.9979
217/1688 [==>...........................] - ETA: 6s - loss: 0.0114 - accuracy: 0.9976
229/1688 [===>..........................] - ETA: 6s - loss: 0.0112 - accuracy: 0.9975
241/1688 [===>..........................] - ETA: 6s - loss: 0.0110 - accuracy: 0.9977
253/1688 [===>..........................] - ETA: 6s - loss: 0.0108 - accuracy: 0.9977
265/1688 [===>..........................] - ETA: 6s - loss: 0.0108 - accuracy: 0.9976
277/1688 [===>..........................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9975
289/1688 [====>.........................] - ETA: 6s - loss: 0.0113 - accuracy: 0.9974
301/1688 [====>.........................] - ETA: 6s - loss: 0.0110 - accuracy: 0.9975
313/1688 [====>.........................] - ETA: 6s - loss: 0.0110 - accuracy: 0.9974
325/1688 [====>.........................] - ETA: 5s - loss: 0.0108 - accuracy: 0.9974
337/1688 [====>.........................] - ETA: 5s - loss: 0.0108 - accuracy: 0.9974
349/1688 [=====>........................] - ETA: 5s - loss: 0.0107 - accuracy: 0.9974
361/1688 [=====>........................] - ETA: 5s - loss: 0.0108 - accuracy: 0.9975
373/1688 [=====>........................] - ETA: 5s - loss: 0.0108 - accuracy: 0.9976
385/1688 [=====>........................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9974
397/1688 [======>.......................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9972
409/1688 [======>.......................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9973
421/1688 [======>.......................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9973
433/1688 [======>.......................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9974
445/1688 [======>.......................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9974
457/1688 [=======>......................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9974
469/1688 [=======>......................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9973
481/1688 [=======>......................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9974
493/1688 [=======>......................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9974
505/1688 [=======>......................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9975
517/1688 [========>.....................] - ETA: 5s - loss: 0.0109 - accuracy: 0.9975
529/1688 [========>.....................] - ETA: 5s - loss: 0.0107 - accuracy: 0.9975
541/1688 [========>.....................] - ETA: 5s - loss: 0.0107 - accuracy: 0.9975
553/1688 [========>.....................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9975
565/1688 [=========>....................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9975
577/1688 [=========>....................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9975
589/1688 [=========>....................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9975
601/1688 [=========>....................] - ETA: 4s - loss: 0.0105 - accuracy: 0.9975
613/1688 [=========>....................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9973
625/1688 [==========>...................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9973
637/1688 [==========>...................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9974
649/1688 [==========>...................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9974
661/1688 [==========>...................] - ETA: 4s - loss: 0.0108 - accuracy: 0.9973
673/1688 [==========>...................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9974
685/1688 [===========>..................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9974
697/1688 [===========>..................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9974
709/1688 [===========>..................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9974
721/1688 [===========>..................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9974
733/1688 [============>.................] - ETA: 4s - loss: 0.0105 - accuracy: 0.9974
745/1688 [============>.................] - ETA: 4s - loss: 0.0105 - accuracy: 0.9974
757/1688 [============>.................] - ETA: 4s - loss: 0.0106 - accuracy: 0.9973
769/1688 [============>.................] - ETA: 4s - loss: 0.0107 - accuracy: 0.9973
781/1688 [============>.................] - ETA: 3s - loss: 0.0106 - accuracy: 0.9974
793/1688 [=============>................] - ETA: 3s - loss: 0.0106 - accuracy: 0.9974
805/1688 [=============>................] - ETA: 3s - loss: 0.0107 - accuracy: 0.9972
817/1688 [=============>................] - ETA: 3s - loss: 0.0107 - accuracy: 0.9972
829/1688 [=============>................] - ETA: 3s - loss: 0.0107 - accuracy: 0.9972
841/1688 [=============>................] - ETA: 3s - loss: 0.0108 - accuracy: 0.9972
853/1688 [==============>...............] - ETA: 3s - loss: 0.0107 - accuracy: 0.9973
865/1688 [==============>...............] - ETA: 3s - loss: 0.0107 - accuracy: 0.9973
877/1688 [==============>...............] - ETA: 3s - loss: 0.0107 - accuracy: 0.9973
889/1688 [==============>...............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
901/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
913/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
925/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
937/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
949/1688 [===============>..............] - ETA: 3s - loss: 0.0106 - accuracy: 0.9973
961/1688 [================>.............] - ETA: 3s - loss: 0.0105 - accuracy: 0.9973
973/1688 [================>.............] - ETA: 3s - loss: 0.0107 - accuracy: 0.9973
985/1688 [================>.............] - ETA: 3s - loss: 0.0108 - accuracy: 0.9973
997/1688 [================>.............] - ETA: 3s - loss: 0.0108 - accuracy: 0.9972
1009/1688 [================>.............] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1021/1688 [=================>............] - ETA: 2s - loss: 0.0107 - accuracy: 0.9972
1033/1688 [=================>............] - ETA: 2s - loss: 0.0107 - accuracy: 0.9972
1045/1688 [=================>............] - ETA: 2s - loss: 0.0107 - accuracy: 0.9973
1057/1688 [=================>............] - ETA: 2s - loss: 0.0107 - accuracy: 0.9973
1069/1688 [=================>............] - ETA: 2s - loss: 0.0107 - accuracy: 0.9973
1081/1688 [==================>...........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9972
1093/1688 [==================>...........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9972
1105/1688 [==================>...........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9971
1117/1688 [==================>...........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1129/1688 [===================>..........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9972
1141/1688 [===================>..........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1153/1688 [===================>..........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9971
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9971
1188/1688 [====================>.........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9971
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1224/1688 [====================>.........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9972
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0108 - accuracy: 0.9972
1248/1688 [=====================>........] - ETA: 1s - loss: 0.0108 - accuracy: 0.9972
1260/1688 [=====================>........] - ETA: 1s - loss: 0.0107 - accuracy: 0.9972
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0107 - accuracy: 0.9972
1284/1688 [=====================>........] - ETA: 1s - loss: 0.0107 - accuracy: 0.9972
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9972
1308/1688 [======================>.......] - ETA: 1s - loss: 0.0108 - accuracy: 0.9972
1320/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1380/1688 [=======================>......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1404/1688 [=======================>......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1440/1688 [========================>.....] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1452/1688 [========================>.....] - ETA: 1s - loss: 0.0107 - accuracy: 0.9973
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0107 - accuracy: 0.9973
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0107 - accuracy: 0.9973
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0106 - accuracy: 0.9973
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0106 - accuracy: 0.9973
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0106 - accuracy: 0.9973
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0106 - accuracy: 0.9973
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0106 - accuracy: 0.9973
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0107 - accuracy: 0.9972
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0107 - accuracy: 0.9972
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0107 - accuracy: 0.9973
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0106 - accuracy: 0.9973
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0106 - accuracy: 0.9972
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0107 - accuracy: 0.9972
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0107 - accuracy: 0.9972
1632/1688 [============================>.] - ETA: 0s - loss: 0.0107 - accuracy: 0.9972
1644/1688 [============================>.] - ETA: 0s - loss: 0.0107 - accuracy: 0.9972
1656/1688 [============================>.] - ETA: 0s - loss: 0.0107 - accuracy: 0.9972
1668/1688 [============================>.] - ETA: 0s - loss: 0.0108 - accuracy: 0.9972
1680/1688 [============================>.] - ETA: 0s - loss: 0.0107 - accuracy: 0.9972
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0107 - accuracy: 0.9972 - val_loss: 0.0469 - val_accuracy: 0.9888
Epoch 4/5
1/1688 [..............................] - ETA: 7s - loss: 0.0024 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0051 - accuracy: 1.0000
25/1688 [..............................] - ETA: 7s - loss: 0.0079 - accuracy: 0.9987
37/1688 [..............................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9992
49/1688 [..............................] - ETA: 7s - loss: 0.0084 - accuracy: 0.9987
61/1688 [>.............................] - ETA: 7s - loss: 0.0075 - accuracy: 0.9990
73/1688 [>.............................] - ETA: 7s - loss: 0.0088 - accuracy: 0.9983
85/1688 [>.............................] - ETA: 7s - loss: 0.0090 - accuracy: 0.9982
97/1688 [>.............................] - ETA: 6s - loss: 0.0085 - accuracy: 0.9981
109/1688 [>.............................] - ETA: 6s - loss: 0.0080 - accuracy: 0.9983
121/1688 [=>............................] - ETA: 6s - loss: 0.0078 - accuracy: 0.9985
133/1688 [=>............................] - ETA: 6s - loss: 0.0077 - accuracy: 0.9984
145/1688 [=>............................] - ETA: 6s - loss: 0.0078 - accuracy: 0.9983
157/1688 [=>............................] - ETA: 6s - loss: 0.0077 - accuracy: 0.9984
169/1688 [==>...........................] - ETA: 6s - loss: 0.0078 - accuracy: 0.9983
181/1688 [==>...........................] - ETA: 6s - loss: 0.0074 - accuracy: 0.9984
193/1688 [==>...........................] - ETA: 6s - loss: 0.0073 - accuracy: 0.9985
205/1688 [==>...........................] - ETA: 6s - loss: 0.0078 - accuracy: 0.9982
217/1688 [==>...........................] - ETA: 6s - loss: 0.0080 - accuracy: 0.9980
229/1688 [===>..........................] - ETA: 6s - loss: 0.0079 - accuracy: 0.9981
241/1688 [===>..........................] - ETA: 6s - loss: 0.0082 - accuracy: 0.9979
253/1688 [===>..........................] - ETA: 6s - loss: 0.0083 - accuracy: 0.9979
265/1688 [===>..........................] - ETA: 6s - loss: 0.0081 - accuracy: 0.9980
277/1688 [===>..........................] - ETA: 6s - loss: 0.0081 - accuracy: 0.9980
289/1688 [====>.........................] - ETA: 6s - loss: 0.0082 - accuracy: 0.9979
301/1688 [====>.........................] - ETA: 6s - loss: 0.0081 - accuracy: 0.9980
313/1688 [====>.........................] - ETA: 6s - loss: 0.0079 - accuracy: 0.9981
325/1688 [====>.........................] - ETA: 5s - loss: 0.0081 - accuracy: 0.9981
337/1688 [====>.........................] - ETA: 5s - loss: 0.0081 - accuracy: 0.9981
349/1688 [=====>........................] - ETA: 5s - loss: 0.0080 - accuracy: 0.9982
361/1688 [=====>........................] - ETA: 5s - loss: 0.0084 - accuracy: 0.9981
373/1688 [=====>........................] - ETA: 5s - loss: 0.0082 - accuracy: 0.9982
385/1688 [=====>........................] - ETA: 5s - loss: 0.0081 - accuracy: 0.9982
397/1688 [======>.......................] - ETA: 5s - loss: 0.0083 - accuracy: 0.9982
409/1688 [======>.......................] - ETA: 5s - loss: 0.0085 - accuracy: 0.9981
421/1688 [======>.......................] - ETA: 5s - loss: 0.0086 - accuracy: 0.9980
433/1688 [======>.......................] - ETA: 5s - loss: 0.0086 - accuracy: 0.9980
445/1688 [======>.......................] - ETA: 5s - loss: 0.0085 - accuracy: 0.9980
457/1688 [=======>......................] - ETA: 5s - loss: 0.0083 - accuracy: 0.9981
469/1688 [=======>......................] - ETA: 5s - loss: 0.0085 - accuracy: 0.9980
481/1688 [=======>......................] - ETA: 5s - loss: 0.0086 - accuracy: 0.9980
493/1688 [=======>......................] - ETA: 5s - loss: 0.0085 - accuracy: 0.9980
504/1688 [=======>......................] - ETA: 5s - loss: 0.0085 - accuracy: 0.9980
516/1688 [========>.....................] - ETA: 5s - loss: 0.0084 - accuracy: 0.9981
528/1688 [========>.....................] - ETA: 5s - loss: 0.0087 - accuracy: 0.9980
540/1688 [========>.....................] - ETA: 5s - loss: 0.0088 - accuracy: 0.9980
552/1688 [========>.....................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9980
564/1688 [=========>....................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9981
576/1688 [=========>....................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9980
588/1688 [=========>....................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9980
600/1688 [=========>....................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9980
612/1688 [=========>....................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9980
624/1688 [==========>...................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9979
636/1688 [==========>...................] - ETA: 4s - loss: 0.0090 - accuracy: 0.9979
648/1688 [==========>...................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9979
660/1688 [==========>...................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9980
672/1688 [==========>...................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9980
684/1688 [===========>..................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9980
696/1688 [===========>..................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9979
708/1688 [===========>..................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9979
720/1688 [===========>..................] - ETA: 4s - loss: 0.0088 - accuracy: 0.9980
731/1688 [===========>..................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9979
743/1688 [============>.................] - ETA: 4s - loss: 0.0090 - accuracy: 0.9979
755/1688 [============>.................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9979
767/1688 [============>.................] - ETA: 4s - loss: 0.0089 - accuracy: 0.9979
779/1688 [============>.................] - ETA: 3s - loss: 0.0089 - accuracy: 0.9979
791/1688 [=============>................] - ETA: 3s - loss: 0.0090 - accuracy: 0.9979
803/1688 [=============>................] - ETA: 3s - loss: 0.0090 - accuracy: 0.9979
815/1688 [=============>................] - ETA: 3s - loss: 0.0092 - accuracy: 0.9978
827/1688 [=============>................] - ETA: 3s - loss: 0.0092 - accuracy: 0.9977
839/1688 [=============>................] - ETA: 3s - loss: 0.0092 - accuracy: 0.9978
851/1688 [==============>...............] - ETA: 3s - loss: 0.0092 - accuracy: 0.9978
863/1688 [==============>...............] - ETA: 3s - loss: 0.0091 - accuracy: 0.9978
875/1688 [==============>...............] - ETA: 3s - loss: 0.0091 - accuracy: 0.9978
887/1688 [==============>...............] - ETA: 3s - loss: 0.0093 - accuracy: 0.9977
899/1688 [==============>...............] - ETA: 3s - loss: 0.0093 - accuracy: 0.9978
911/1688 [===============>..............] - ETA: 3s - loss: 0.0094 - accuracy: 0.9977
923/1688 [===============>..............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9977
935/1688 [===============>..............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9977
947/1688 [===============>..............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9977
959/1688 [================>.............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9977
971/1688 [================>.............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9977
983/1688 [================>.............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9976
995/1688 [================>.............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9976
1007/1688 [================>.............] - ETA: 2s - loss: 0.0095 - accuracy: 0.9976
1019/1688 [=================>............] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1031/1688 [=================>............] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1043/1688 [=================>............] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1055/1688 [=================>............] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1067/1688 [=================>............] - ETA: 2s - loss: 0.0096 - accuracy: 0.9976
1079/1688 [==================>...........] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1091/1688 [==================>...........] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1103/1688 [==================>...........] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1115/1688 [==================>...........] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1127/1688 [===================>..........] - ETA: 2s - loss: 0.0094 - accuracy: 0.9977
1139/1688 [===================>..........] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1151/1688 [===================>..........] - ETA: 2s - loss: 0.0095 - accuracy: 0.9977
1163/1688 [===================>..........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9976
1175/1688 [===================>..........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9976
1187/1688 [====================>.........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9976
1199/1688 [====================>.........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9976
1211/1688 [====================>.........] - ETA: 2s - loss: 0.0098 - accuracy: 0.9976
1223/1688 [====================>.........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9976
1235/1688 [====================>.........] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1259/1688 [=====================>........] - ETA: 1s - loss: 0.0096 - accuracy: 0.9976
1271/1688 [=====================>........] - ETA: 1s - loss: 0.0096 - accuracy: 0.9976
1283/1688 [=====================>........] - ETA: 1s - loss: 0.0096 - accuracy: 0.9977
1295/1688 [======================>.......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1307/1688 [======================>.......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1319/1688 [======================>.......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1331/1688 [======================>.......] - ETA: 1s - loss: 0.0098 - accuracy: 0.9976
1343/1688 [======================>.......] - ETA: 1s - loss: 0.0098 - accuracy: 0.9976
1355/1688 [=======================>......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1379/1688 [=======================>......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1391/1688 [=======================>......] - ETA: 1s - loss: 0.0098 - accuracy: 0.9975
1403/1688 [=======================>......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9975
1415/1688 [========================>.....] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1427/1688 [========================>.....] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1439/1688 [========================>.....] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0097 - accuracy: 0.9976
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0097 - accuracy: 0.9976
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0097 - accuracy: 0.9976
1487/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9976
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9976
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0096 - accuracy: 0.9976
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0097 - accuracy: 0.9976
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0097 - accuracy: 0.9976
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0097 - accuracy: 0.9976
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0097 - accuracy: 0.9975
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9975
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9975
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0098 - accuracy: 0.9975
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0098 - accuracy: 0.9975
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0098 - accuracy: 0.9975
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0098 - accuracy: 0.9974
1643/1688 [============================>.] - ETA: 0s - loss: 0.0098 - accuracy: 0.9975
1655/1688 [============================>.] - ETA: 0s - loss: 0.0098 - accuracy: 0.9975
1667/1688 [============================>.] - ETA: 0s - loss: 0.0098 - accuracy: 0.9975
1679/1688 [============================>.] - ETA: 0s - loss: 0.0098 - accuracy: 0.9975
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0098 - accuracy: 0.9975 - val_loss: 0.0470 - val_accuracy: 0.9887
Epoch 5/5
1/1688 [..............................] - ETA: 7s - loss: 0.0045 - accuracy: 1.0000
13/1688 [..............................] - ETA: 7s - loss: 0.0065 - accuracy: 1.0000
25/1688 [..............................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9987
37/1688 [..............................] - ETA: 7s - loss: 0.0107 - accuracy: 0.9966
49/1688 [..............................] - ETA: 7s - loss: 0.0094 - accuracy: 0.9968
61/1688 [>.............................] - ETA: 7s - loss: 0.0090 - accuracy: 0.9974
73/1688 [>.............................] - ETA: 7s - loss: 0.0086 - accuracy: 0.9974
85/1688 [>.............................] - ETA: 7s - loss: 0.0091 - accuracy: 0.9971
97/1688 [>.............................] - ETA: 7s - loss: 0.0089 - accuracy: 0.9971
109/1688 [>.............................] - ETA: 6s - loss: 0.0086 - accuracy: 0.9974
121/1688 [=>............................] - ETA: 6s - loss: 0.0091 - accuracy: 0.9974
133/1688 [=>............................] - ETA: 6s - loss: 0.0090 - accuracy: 0.9974
145/1688 [=>............................] - ETA: 6s - loss: 0.0088 - accuracy: 0.9974
157/1688 [=>............................] - ETA: 6s - loss: 0.0085 - accuracy: 0.9976
169/1688 [==>...........................] - ETA: 6s - loss: 0.0082 - accuracy: 0.9978
181/1688 [==>...........................] - ETA: 6s - loss: 0.0083 - accuracy: 0.9976
193/1688 [==>...........................] - ETA: 6s - loss: 0.0083 - accuracy: 0.9977
205/1688 [==>...........................] - ETA: 6s - loss: 0.0081 - accuracy: 0.9977
217/1688 [==>...........................] - ETA: 6s - loss: 0.0079 - accuracy: 0.9977
229/1688 [===>..........................] - ETA: 6s - loss: 0.0083 - accuracy: 0.9977
241/1688 [===>..........................] - ETA: 6s - loss: 0.0082 - accuracy: 0.9977
253/1688 [===>..........................] - ETA: 6s - loss: 0.0080 - accuracy: 0.9978
265/1688 [===>..........................] - ETA: 6s - loss: 0.0080 - accuracy: 0.9979
277/1688 [===>..........................] - ETA: 6s - loss: 0.0080 - accuracy: 0.9979
289/1688 [====>.........................] - ETA: 6s - loss: 0.0079 - accuracy: 0.9979
301/1688 [====>.........................] - ETA: 6s - loss: 0.0079 - accuracy: 0.9980
313/1688 [====>.........................] - ETA: 6s - loss: 0.0081 - accuracy: 0.9980
325/1688 [====>.........................] - ETA: 5s - loss: 0.0080 - accuracy: 0.9980
337/1688 [====>.........................] - ETA: 5s - loss: 0.0081 - accuracy: 0.9980
349/1688 [=====>........................] - ETA: 5s - loss: 0.0081 - accuracy: 0.9979
361/1688 [=====>........................] - ETA: 5s - loss: 0.0083 - accuracy: 0.9980
373/1688 [=====>........................] - ETA: 5s - loss: 0.0082 - accuracy: 0.9980
385/1688 [=====>........................] - ETA: 5s - loss: 0.0084 - accuracy: 0.9979
397/1688 [======>.......................] - ETA: 5s - loss: 0.0083 - accuracy: 0.9980
409/1688 [======>.......................] - ETA: 5s - loss: 0.0082 - accuracy: 0.9979
421/1688 [======>.......................] - ETA: 5s - loss: 0.0082 - accuracy: 0.9980
433/1688 [======>.......................] - ETA: 5s - loss: 0.0083 - accuracy: 0.9979
445/1688 [======>.......................] - ETA: 5s - loss: 0.0084 - accuracy: 0.9978
457/1688 [=======>......................] - ETA: 5s - loss: 0.0083 - accuracy: 0.9979
469/1688 [=======>......................] - ETA: 5s - loss: 0.0082 - accuracy: 0.9979
481/1688 [=======>......................] - ETA: 5s - loss: 0.0083 - accuracy: 0.9979
493/1688 [=======>......................] - ETA: 5s - loss: 0.0083 - accuracy: 0.9978
505/1688 [=======>......................] - ETA: 5s - loss: 0.0084 - accuracy: 0.9978
517/1688 [========>.....................] - ETA: 5s - loss: 0.0085 - accuracy: 0.9977
529/1688 [========>.....................] - ETA: 5s - loss: 0.0084 - accuracy: 0.9978
541/1688 [========>.....................] - ETA: 5s - loss: 0.0084 - accuracy: 0.9977
553/1688 [========>.....................] - ETA: 4s - loss: 0.0083 - accuracy: 0.9977
565/1688 [=========>....................] - ETA: 4s - loss: 0.0084 - accuracy: 0.9977
577/1688 [=========>....................] - ETA: 4s - loss: 0.0084 - accuracy: 0.9977
589/1688 [=========>....................] - ETA: 4s - loss: 0.0084 - accuracy: 0.9978
601/1688 [=========>....................] - ETA: 4s - loss: 0.0083 - accuracy: 0.9978
613/1688 [=========>....................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9979
625/1688 [==========>...................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9979
637/1688 [==========>...................] - ETA: 4s - loss: 0.0081 - accuracy: 0.9979
649/1688 [==========>...................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9979
661/1688 [==========>...................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9978
673/1688 [==========>...................] - ETA: 4s - loss: 0.0083 - accuracy: 0.9978
685/1688 [===========>..................] - ETA: 4s - loss: 0.0083 - accuracy: 0.9978
697/1688 [===========>..................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9978
709/1688 [===========>..................] - ETA: 4s - loss: 0.0083 - accuracy: 0.9978
721/1688 [===========>..................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9978
733/1688 [============>.................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9979
745/1688 [============>.................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9979
757/1688 [============>.................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9979
769/1688 [============>.................] - ETA: 4s - loss: 0.0082 - accuracy: 0.9978
781/1688 [============>.................] - ETA: 3s - loss: 0.0081 - accuracy: 0.9979
793/1688 [=============>................] - ETA: 3s - loss: 0.0082 - accuracy: 0.9979
805/1688 [=============>................] - ETA: 3s - loss: 0.0082 - accuracy: 0.9979
817/1688 [=============>................] - ETA: 3s - loss: 0.0082 - accuracy: 0.9979
829/1688 [=============>................] - ETA: 3s - loss: 0.0081 - accuracy: 0.9979
841/1688 [=============>................] - ETA: 3s - loss: 0.0082 - accuracy: 0.9978
853/1688 [==============>...............] - ETA: 3s - loss: 0.0083 - accuracy: 0.9978
865/1688 [==============>...............] - ETA: 3s - loss: 0.0083 - accuracy: 0.9978
877/1688 [==============>...............] - ETA: 3s - loss: 0.0084 - accuracy: 0.9978
889/1688 [==============>...............] - ETA: 3s - loss: 0.0083 - accuracy: 0.9978
901/1688 [===============>..............] - ETA: 3s - loss: 0.0083 - accuracy: 0.9978
913/1688 [===============>..............] - ETA: 3s - loss: 0.0083 - accuracy: 0.9978
925/1688 [===============>..............] - ETA: 3s - loss: 0.0083 - accuracy: 0.9978
937/1688 [===============>..............] - ETA: 3s - loss: 0.0083 - accuracy: 0.9978
949/1688 [===============>..............] - ETA: 3s - loss: 0.0084 - accuracy: 0.9978
961/1688 [================>.............] - ETA: 3s - loss: 0.0084 - accuracy: 0.9978
973/1688 [================>.............] - ETA: 3s - loss: 0.0084 - accuracy: 0.9978
985/1688 [================>.............] - ETA: 3s - loss: 0.0084 - accuracy: 0.9978
997/1688 [================>.............] - ETA: 3s - loss: 0.0085 - accuracy: 0.9977
1009/1688 [================>.............] - ETA: 2s - loss: 0.0085 - accuracy: 0.9977
1021/1688 [=================>............] - ETA: 2s - loss: 0.0086 - accuracy: 0.9976
1033/1688 [=================>............] - ETA: 2s - loss: 0.0085 - accuracy: 0.9976
1045/1688 [=================>............] - ETA: 2s - loss: 0.0086 - accuracy: 0.9976
1057/1688 [=================>............] - ETA: 2s - loss: 0.0087 - accuracy: 0.9976
1069/1688 [=================>............] - ETA: 2s - loss: 0.0087 - accuracy: 0.9976
1081/1688 [==================>...........] - ETA: 2s - loss: 0.0087 - accuracy: 0.9976
1093/1688 [==================>...........] - ETA: 2s - loss: 0.0087 - accuracy: 0.9976
1105/1688 [==================>...........] - ETA: 2s - loss: 0.0088 - accuracy: 0.9975
1117/1688 [==================>...........] - ETA: 2s - loss: 0.0088 - accuracy: 0.9975
1129/1688 [===================>..........] - ETA: 2s - loss: 0.0088 - accuracy: 0.9975
1141/1688 [===================>..........] - ETA: 2s - loss: 0.0088 - accuracy: 0.9975
1153/1688 [===================>..........] - ETA: 2s - loss: 0.0088 - accuracy: 0.9975
1165/1688 [===================>..........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9975
1177/1688 [===================>..........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9975
1189/1688 [====================>.........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9974
1201/1688 [====================>.........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9975
1213/1688 [====================>.........] - ETA: 2s - loss: 0.0089 - accuracy: 0.9975
1225/1688 [====================>.........] - ETA: 2s - loss: 0.0090 - accuracy: 0.9974
1237/1688 [====================>.........] - ETA: 1s - loss: 0.0091 - accuracy: 0.9974
1249/1688 [=====================>........] - ETA: 1s - loss: 0.0091 - accuracy: 0.9974
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0090 - accuracy: 0.9974
1273/1688 [=====================>........] - ETA: 1s - loss: 0.0090 - accuracy: 0.9975
1285/1688 [=====================>........] - ETA: 1s - loss: 0.0090 - accuracy: 0.9975
1297/1688 [======================>.......] - ETA: 1s - loss: 0.0091 - accuracy: 0.9975
1309/1688 [======================>.......] - ETA: 1s - loss: 0.0091 - accuracy: 0.9975
1321/1688 [======================>.......] - ETA: 1s - loss: 0.0092 - accuracy: 0.9974
1333/1688 [======================>.......] - ETA: 1s - loss: 0.0094 - accuracy: 0.9973
1345/1688 [======================>.......] - ETA: 1s - loss: 0.0094 - accuracy: 0.9973
1357/1688 [=======================>......] - ETA: 1s - loss: 0.0093 - accuracy: 0.9974
1369/1688 [=======================>......] - ETA: 1s - loss: 0.0093 - accuracy: 0.9974
1381/1688 [=======================>......] - ETA: 1s - loss: 0.0093 - accuracy: 0.9974
1393/1688 [=======================>......] - ETA: 1s - loss: 0.0092 - accuracy: 0.9974
1405/1688 [=======================>......] - ETA: 1s - loss: 0.0092 - accuracy: 0.9974
1417/1688 [========================>.....] - ETA: 1s - loss: 0.0092 - accuracy: 0.9974
1429/1688 [========================>.....] - ETA: 1s - loss: 0.0092 - accuracy: 0.9974
1441/1688 [========================>.....] - ETA: 1s - loss: 0.0092 - accuracy: 0.9975
1452/1688 [========================>.....] - ETA: 1s - loss: 0.0092 - accuracy: 0.9975
1464/1688 [=========================>....] - ETA: 0s - loss: 0.0092 - accuracy: 0.9975
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0092 - accuracy: 0.9975
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0091 - accuracy: 0.9975
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0091 - accuracy: 0.9975
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0091 - accuracy: 0.9975
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0091 - accuracy: 0.9975
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0091 - accuracy: 0.9975
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0091 - accuracy: 0.9975
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0091 - accuracy: 0.9975
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0090 - accuracy: 0.9975
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1632/1688 [============================>.] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1644/1688 [============================>.] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1656/1688 [============================>.] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1668/1688 [============================>.] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1680/1688 [============================>.] - ETA: 0s - loss: 0.0090 - accuracy: 0.9976
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0090 - accuracy: 0.9976 - val_loss: 0.0472 - val_accuracy: 0.9893
<keras.callbacks.History object at 0x7f23de852eb0>
score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine tuning:', score)
Test accuracy after fine tuning: 0.9861000180244446
3. Convert
3.1 Convert to Akida model
When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.
As with Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized)
model_akida.summary()
WARNING:tensorflow:5 out of the last 5 calls to <function NonTrackVariable.set_var at 0x7f23ceafc1f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:6 out of the last 6 calls to <function NonTrackVariable.set_var at 0x7f23ceafc700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 5
______________________________________________
__________________________________________________________________
Layer (type) Output shape Kernel shape
=============== SW/conv2d-dequantizer_2 (Software) ===============
conv2d (InputConv2D) [13, 13, 32] (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D) [7, 7, 32] (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D) [7, 7, 64] (1, 1, 32, 64)
__________________________________________________________________
dense (Dense2D) [1, 1, 10] (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer) [1, 1, 10] N/A
__________________________________________________________________
3.2. Check performance
accuracy = model_akida.evaluate(x_test, y_test)
print('Test accuracy after conversion:', accuracy)
# For non-regression purposes
assert accuracy > 0.96
Test accuracy after conversion: 0.9848999977111816
3.3 Show predictions for a single image
Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.
# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])
f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()
print(outputs.squeeze())
Input Label: 7
[ -8.843261 -6.970094 -5.442708 -1.5962703 -13.384637 -3.2714906
-21.486294 6.415907 -4.8704467 -0.6497679]
Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.
In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.
4. GXNOR/MNIST
A more robust model called GXNOR/MNIST is provided in the model zoo It is inspired from the GXNOR-Net paper. It comes with its pretrained 2/2/1 helper for which the float training was done for 20 epochs, then the model was then gradually quantized following: 4/4/4 –> 4/4/2 –> 2/2/2 –> 2/2/1 with a 15 epochs QAT step at each quantization stage.
Total running time of the script: (1 minutes 38.482 seconds)