Note
Go to the end to download the full example code.
Global Akida workflow
Using the MNIST dataset, this example shows the definition and training of a TF-Keras floating-point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original TF-Keras floating-point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.
Note
Please refer to the TensorFlow tf_keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.
The MNIST example below is light enough so that a GPU is not needed for training.
Global Akida workflow
1. Create and train
1.1. Load and reshape MNIST dataset
import numpy as np
import tensorflow as tf
import matplotlib.cm as cm
import matplotlib.pyplot as plt
from tf_keras.datasets import mnist
# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, height, width, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
axarr[i].set_title('Class %d' % y_test[i])
plt.show()
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
8192/11490434 [..............................] - ETA: 0s
49152/11490434 [..............................] - ETA: 22s
81920/11490434 [..............................] - ETA: 25s
163840/11490434 [..............................] - ETA: 16s
245760/11490434 [..............................] - ETA: 13s
360448/11490434 [..............................] - ETA: 10s
573440/11490434 [>.............................] - ETA: 7s
835584/11490434 [=>............................] - ETA: 5s
1294336/11490434 [==>...........................] - ETA: 3s
1851392/11490434 [===>..........................] - ETA: 2s
2449408/11490434 [=====>........................] - ETA: 2s
3039232/11490434 [======>.......................] - ETA: 1s
3620864/11490434 [========>.....................] - ETA: 1s
4202496/11490434 [=========>....................] - ETA: 1s
4784128/11490434 [===========>..................] - ETA: 1s
5373952/11490434 [=============>................] - ETA: 0s
5963776/11490434 [==============>...............] - ETA: 0s
6553600/11490434 [================>.............] - ETA: 0s
7143424/11490434 [=================>............] - ETA: 0s
7733248/11490434 [===================>..........] - ETA: 0s
8323072/11490434 [====================>.........] - ETA: 0s
8912896/11490434 [======================>.......] - ETA: 0s
9502720/11490434 [=======================>......] - ETA: 0s
10059776/11490434 [=========================>....] - ETA: 0s
10649600/11490434 [==========================>...] - ETA: 0s
11206656/11490434 [============================>.] - ETA: 0s
11490434/11490434 [==============================] - 1s 0us/step
1.2. Model definition
Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard TF-Keras CNN model.
import tf_keras as keras
model_keras = keras.models.Sequential([
keras.layers.Input(shape=(28, 28, 1), name="input", dtype=tf.uint8),
keras.layers.Rescaling(1. / 255),
keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
# Separable layer
keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
keras.layers.BatchNormalization(),
keras.layers.ReLU(),
keras.layers.Flatten(),
keras.layers.Dense(10)
], 'mnistnet')
model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
rescaling (Rescaling) (None, 28, 28, 1) 0
conv2d (Conv2D) (None, 13, 13, 32) 320
batch_normalization (Batch (None, 13, 13, 32) 128
Normalization)
re_lu (ReLU) (None, 13, 13, 32) 0
depthwise_conv2d (Depthwis (None, 7, 7, 32) 320
eConv2D)
conv2d_1 (Conv2D) (None, 7, 7, 64) 2112
batch_normalization_1 (Bat (None, 7, 7, 64) 256
chNormalization)
re_lu_1 (ReLU) (None, 7, 7, 64) 0
flatten (Flatten) (None, 3136) 0
dense (Dense) (None, 10) 31370
=================================================================
Total params: 34506 (134.79 KB)
Trainable params: 34314 (134.04 KB)
Non-trainable params: 192 (768.00 Byte)
_________________________________________________________________
1.3. Model training
Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.
from tf_keras.optimizers import Adam
model_keras.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=Adam(learning_rate=1e-3),
metrics=['accuracy'])
_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
Epoch 1/10
1/1688 [..............................] - ETA: 1:16:37 - loss: 2.5945 - accuracy: 0.0938
22/1688 [..............................] - ETA: 4s - loss: 1.4327 - accuracy: 0.5213
45/1688 [..............................] - ETA: 3s - loss: 1.0474 - accuracy: 0.6653
67/1688 [>.............................] - ETA: 3s - loss: 0.8523 - accuracy: 0.7346
89/1688 [>.............................] - ETA: 3s - loss: 0.7447 - accuracy: 0.7697
112/1688 [>.............................] - ETA: 3s - loss: 0.6703 - accuracy: 0.7949
134/1688 [=>............................] - ETA: 3s - loss: 0.6122 - accuracy: 0.8144
154/1688 [=>............................] - ETA: 3s - loss: 0.5699 - accuracy: 0.8275
176/1688 [==>...........................] - ETA: 3s - loss: 0.5384 - accuracy: 0.8372
198/1688 [==>...........................] - ETA: 3s - loss: 0.5078 - accuracy: 0.8475
220/1688 [==>...........................] - ETA: 3s - loss: 0.4853 - accuracy: 0.8544
242/1688 [===>..........................] - ETA: 3s - loss: 0.4672 - accuracy: 0.8595
264/1688 [===>..........................] - ETA: 3s - loss: 0.4489 - accuracy: 0.8646
287/1688 [====>.........................] - ETA: 3s - loss: 0.4308 - accuracy: 0.8710
309/1688 [====>.........................] - ETA: 3s - loss: 0.4196 - accuracy: 0.8744
331/1688 [====>.........................] - ETA: 3s - loss: 0.4032 - accuracy: 0.8790
354/1688 [=====>........................] - ETA: 3s - loss: 0.3878 - accuracy: 0.8833
376/1688 [=====>........................] - ETA: 3s - loss: 0.3763 - accuracy: 0.8874
398/1688 [======>.......................] - ETA: 2s - loss: 0.3641 - accuracy: 0.8913
420/1688 [======>.......................] - ETA: 2s - loss: 0.3561 - accuracy: 0.8940
442/1688 [======>.......................] - ETA: 2s - loss: 0.3478 - accuracy: 0.8957
464/1688 [=======>......................] - ETA: 2s - loss: 0.3391 - accuracy: 0.8980
486/1688 [=======>......................] - ETA: 2s - loss: 0.3308 - accuracy: 0.9002
509/1688 [========>.....................] - ETA: 2s - loss: 0.3244 - accuracy: 0.9016
532/1688 [========>.....................] - ETA: 2s - loss: 0.3182 - accuracy: 0.9038
554/1688 [========>.....................] - ETA: 2s - loss: 0.3142 - accuracy: 0.9053
575/1688 [=========>....................] - ETA: 2s - loss: 0.3073 - accuracy: 0.9072
597/1688 [=========>....................] - ETA: 2s - loss: 0.3020 - accuracy: 0.9090
619/1688 [==========>...................] - ETA: 2s - loss: 0.2972 - accuracy: 0.9101
641/1688 [==========>...................] - ETA: 2s - loss: 0.2927 - accuracy: 0.9116
663/1688 [==========>...................] - ETA: 2s - loss: 0.2872 - accuracy: 0.9131
685/1688 [===========>..................] - ETA: 2s - loss: 0.2816 - accuracy: 0.9147
708/1688 [===========>..................] - ETA: 2s - loss: 0.2767 - accuracy: 0.9162
731/1688 [===========>..................] - ETA: 2s - loss: 0.2722 - accuracy: 0.9176
754/1688 [============>.................] - ETA: 2s - loss: 0.2674 - accuracy: 0.9189
776/1688 [============>.................] - ETA: 2s - loss: 0.2641 - accuracy: 0.9200
798/1688 [=============>................] - ETA: 2s - loss: 0.2601 - accuracy: 0.9213
821/1688 [=============>................] - ETA: 1s - loss: 0.2565 - accuracy: 0.9223
844/1688 [==============>...............] - ETA: 1s - loss: 0.2542 - accuracy: 0.9232
866/1688 [==============>...............] - ETA: 1s - loss: 0.2511 - accuracy: 0.9240
888/1688 [==============>...............] - ETA: 1s - loss: 0.2487 - accuracy: 0.9248
909/1688 [===============>..............] - ETA: 1s - loss: 0.2453 - accuracy: 0.9257
931/1688 [===============>..............] - ETA: 1s - loss: 0.2429 - accuracy: 0.9266
953/1688 [===============>..............] - ETA: 1s - loss: 0.2406 - accuracy: 0.9273
975/1688 [================>.............] - ETA: 1s - loss: 0.2375 - accuracy: 0.9282
997/1688 [================>.............] - ETA: 1s - loss: 0.2351 - accuracy: 0.9290
1019/1688 [=================>............] - ETA: 1s - loss: 0.2325 - accuracy: 0.9297
1041/1688 [=================>............] - ETA: 1s - loss: 0.2298 - accuracy: 0.9303
1064/1688 [=================>............] - ETA: 1s - loss: 0.2274 - accuracy: 0.9310
1086/1688 [==================>...........] - ETA: 1s - loss: 0.2251 - accuracy: 0.9317
1108/1688 [==================>...........] - ETA: 1s - loss: 0.2229 - accuracy: 0.9323
1131/1688 [===================>..........] - ETA: 1s - loss: 0.2206 - accuracy: 0.9329
1153/1688 [===================>..........] - ETA: 1s - loss: 0.2191 - accuracy: 0.9333
1175/1688 [===================>..........] - ETA: 1s - loss: 0.2173 - accuracy: 0.9337
1197/1688 [====================>.........] - ETA: 1s - loss: 0.2155 - accuracy: 0.9343
1219/1688 [====================>.........] - ETA: 1s - loss: 0.2133 - accuracy: 0.9349
1241/1688 [=====================>........] - ETA: 1s - loss: 0.2111 - accuracy: 0.9356
1263/1688 [=====================>........] - ETA: 0s - loss: 0.2094 - accuracy: 0.9361
1286/1688 [=====================>........] - ETA: 0s - loss: 0.2077 - accuracy: 0.9367
1308/1688 [======================>.......] - ETA: 0s - loss: 0.2062 - accuracy: 0.9371
1330/1688 [======================>.......] - ETA: 0s - loss: 0.2046 - accuracy: 0.9376
1352/1688 [=======================>......] - ETA: 0s - loss: 0.2028 - accuracy: 0.9381
1375/1688 [=======================>......] - ETA: 0s - loss: 0.2010 - accuracy: 0.9387
1397/1688 [=======================>......] - ETA: 0s - loss: 0.1996 - accuracy: 0.9393
1419/1688 [========================>.....] - ETA: 0s - loss: 0.1978 - accuracy: 0.9398
1441/1688 [========================>.....] - ETA: 0s - loss: 0.1962 - accuracy: 0.9403
1463/1688 [=========================>....] - ETA: 0s - loss: 0.1945 - accuracy: 0.9409
1485/1688 [=========================>....] - ETA: 0s - loss: 0.1928 - accuracy: 0.9414
1507/1688 [=========================>....] - ETA: 0s - loss: 0.1912 - accuracy: 0.9419
1529/1688 [==========================>...] - ETA: 0s - loss: 0.1893 - accuracy: 0.9425
1551/1688 [==========================>...] - ETA: 0s - loss: 0.1878 - accuracy: 0.9429
1573/1688 [==========================>...] - ETA: 0s - loss: 0.1872 - accuracy: 0.9432
1596/1688 [===========================>..] - ETA: 0s - loss: 0.1865 - accuracy: 0.9434
1618/1688 [===========================>..] - ETA: 0s - loss: 0.1849 - accuracy: 0.9439
1642/1688 [============================>.] - ETA: 0s - loss: 0.1836 - accuracy: 0.9444
1664/1688 [============================>.] - ETA: 0s - loss: 0.1830 - accuracy: 0.9447
1686/1688 [============================>.] - ETA: 0s - loss: 0.1820 - accuracy: 0.9450
1688/1688 [==============================] - 7s 3ms/step - loss: 0.1819 - accuracy: 0.9451 - val_loss: 0.0700 - val_accuracy: 0.9788
Epoch 2/10
1/1688 [..............................] - ETA: 3s - loss: 0.2731 - accuracy: 0.9062
23/1688 [..............................] - ETA: 3s - loss: 0.0724 - accuracy: 0.9783
45/1688 [..............................] - ETA: 3s - loss: 0.0676 - accuracy: 0.9792
67/1688 [>.............................] - ETA: 3s - loss: 0.0679 - accuracy: 0.9790
89/1688 [>.............................] - ETA: 3s - loss: 0.0678 - accuracy: 0.9796
111/1688 [>.............................] - ETA: 3s - loss: 0.0663 - accuracy: 0.9806
133/1688 [=>............................] - ETA: 3s - loss: 0.0782 - accuracy: 0.9786
155/1688 [=>............................] - ETA: 3s - loss: 0.0814 - accuracy: 0.9780
178/1688 [==>...........................] - ETA: 3s - loss: 0.0821 - accuracy: 0.9775
200/1688 [==>...........................] - ETA: 3s - loss: 0.0800 - accuracy: 0.9775
222/1688 [==>...........................] - ETA: 3s - loss: 0.0762 - accuracy: 0.9787
244/1688 [===>..........................] - ETA: 3s - loss: 0.0763 - accuracy: 0.9782
266/1688 [===>..........................] - ETA: 3s - loss: 0.0757 - accuracy: 0.9779
288/1688 [====>.........................] - ETA: 3s - loss: 0.0762 - accuracy: 0.9772
310/1688 [====>.........................] - ETA: 3s - loss: 0.0761 - accuracy: 0.9773
332/1688 [====>.........................] - ETA: 3s - loss: 0.0744 - accuracy: 0.9777
354/1688 [=====>........................] - ETA: 3s - loss: 0.0735 - accuracy: 0.9780
376/1688 [=====>........................] - ETA: 3s - loss: 0.0755 - accuracy: 0.9771
399/1688 [======>.......................] - ETA: 2s - loss: 0.0744 - accuracy: 0.9774
421/1688 [======>.......................] - ETA: 2s - loss: 0.0744 - accuracy: 0.9774
443/1688 [======>.......................] - ETA: 2s - loss: 0.0759 - accuracy: 0.9768
466/1688 [=======>......................] - ETA: 2s - loss: 0.0777 - accuracy: 0.9763
489/1688 [=======>......................] - ETA: 2s - loss: 0.0761 - accuracy: 0.9769
511/1688 [========>.....................] - ETA: 2s - loss: 0.0754 - accuracy: 0.9773
534/1688 [========>.....................] - ETA: 2s - loss: 0.0746 - accuracy: 0.9775
557/1688 [========>.....................] - ETA: 2s - loss: 0.0742 - accuracy: 0.9776
579/1688 [=========>....................] - ETA: 2s - loss: 0.0733 - accuracy: 0.9778
602/1688 [=========>....................] - ETA: 2s - loss: 0.0734 - accuracy: 0.9776
624/1688 [==========>...................] - ETA: 2s - loss: 0.0735 - accuracy: 0.9774
646/1688 [==========>...................] - ETA: 2s - loss: 0.0728 - accuracy: 0.9776
669/1688 [==========>...................] - ETA: 2s - loss: 0.0737 - accuracy: 0.9775
692/1688 [===========>..................] - ETA: 2s - loss: 0.0727 - accuracy: 0.9778
714/1688 [===========>..................] - ETA: 2s - loss: 0.0720 - accuracy: 0.9782
737/1688 [============>.................] - ETA: 2s - loss: 0.0722 - accuracy: 0.9781
759/1688 [============>.................] - ETA: 2s - loss: 0.0723 - accuracy: 0.9779
782/1688 [============>.................] - ETA: 2s - loss: 0.0725 - accuracy: 0.9779
805/1688 [=============>................] - ETA: 2s - loss: 0.0727 - accuracy: 0.9778
827/1688 [=============>................] - ETA: 1s - loss: 0.0732 - accuracy: 0.9777
849/1688 [==============>...............] - ETA: 1s - loss: 0.0729 - accuracy: 0.9777
870/1688 [==============>...............] - ETA: 1s - loss: 0.0730 - accuracy: 0.9776
892/1688 [==============>...............] - ETA: 1s - loss: 0.0732 - accuracy: 0.9776
914/1688 [===============>..............] - ETA: 1s - loss: 0.0732 - accuracy: 0.9776
936/1688 [===============>..............] - ETA: 1s - loss: 0.0731 - accuracy: 0.9777
959/1688 [================>.............] - ETA: 1s - loss: 0.0723 - accuracy: 0.9780
981/1688 [================>.............] - ETA: 1s - loss: 0.0721 - accuracy: 0.9781
1003/1688 [================>.............] - ETA: 1s - loss: 0.0723 - accuracy: 0.9779
1025/1688 [=================>............] - ETA: 1s - loss: 0.0722 - accuracy: 0.9778
1048/1688 [=================>............] - ETA: 1s - loss: 0.0720 - accuracy: 0.9778
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0716 - accuracy: 0.9779
1092/1688 [==================>...........] - ETA: 1s - loss: 0.0716 - accuracy: 0.9777
1115/1688 [==================>...........] - ETA: 1s - loss: 0.0715 - accuracy: 0.9777
1137/1688 [===================>..........] - ETA: 1s - loss: 0.0719 - accuracy: 0.9776
1159/1688 [===================>..........] - ETA: 1s - loss: 0.0716 - accuracy: 0.9776
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0713 - accuracy: 0.9777
1203/1688 [====================>.........] - ETA: 1s - loss: 0.0713 - accuracy: 0.9777
1225/1688 [====================>.........] - ETA: 1s - loss: 0.0710 - accuracy: 0.9779
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0706 - accuracy: 0.9780
1269/1688 [=====================>........] - ETA: 0s - loss: 0.0708 - accuracy: 0.9780
1291/1688 [=====================>........] - ETA: 0s - loss: 0.0707 - accuracy: 0.9780
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0714 - accuracy: 0.9778
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0716 - accuracy: 0.9778
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0718 - accuracy: 0.9776
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0725 - accuracy: 0.9775
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0728 - accuracy: 0.9773
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0726 - accuracy: 0.9774
1448/1688 [========================>.....] - ETA: 0s - loss: 0.0724 - accuracy: 0.9775
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0719 - accuracy: 0.9777
1493/1688 [=========================>....] - ETA: 0s - loss: 0.0721 - accuracy: 0.9777
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0728 - accuracy: 0.9776
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0727 - accuracy: 0.9776
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0726 - accuracy: 0.9776
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0723 - accuracy: 0.9777
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0721 - accuracy: 0.9778
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0721 - accuracy: 0.9777
1648/1688 [============================>.] - ETA: 0s - loss: 0.0720 - accuracy: 0.9778
1671/1688 [============================>.] - ETA: 0s - loss: 0.0717 - accuracy: 0.9779
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0721 - accuracy: 0.9778 - val_loss: 0.0816 - val_accuracy: 0.9755
Epoch 3/10
1/1688 [..............................] - ETA: 4s - loss: 0.1030 - accuracy: 0.9688
23/1688 [..............................] - ETA: 3s - loss: 0.0557 - accuracy: 0.9837
45/1688 [..............................] - ETA: 3s - loss: 0.0555 - accuracy: 0.9819
67/1688 [>.............................] - ETA: 3s - loss: 0.0501 - accuracy: 0.9841
89/1688 [>.............................] - ETA: 3s - loss: 0.0479 - accuracy: 0.9846
111/1688 [>.............................] - ETA: 3s - loss: 0.0487 - accuracy: 0.9848
133/1688 [=>............................] - ETA: 3s - loss: 0.0504 - accuracy: 0.9840
156/1688 [=>............................] - ETA: 3s - loss: 0.0550 - accuracy: 0.9832
179/1688 [==>...........................] - ETA: 3s - loss: 0.0571 - accuracy: 0.9822
201/1688 [==>...........................] - ETA: 3s - loss: 0.0589 - accuracy: 0.9815
224/1688 [==>...........................] - ETA: 3s - loss: 0.0600 - accuracy: 0.9807
246/1688 [===>..........................] - ETA: 3s - loss: 0.0584 - accuracy: 0.9811
268/1688 [===>..........................] - ETA: 3s - loss: 0.0559 - accuracy: 0.9820
289/1688 [====>.........................] - ETA: 3s - loss: 0.0556 - accuracy: 0.9819
311/1688 [====>.........................] - ETA: 3s - loss: 0.0544 - accuracy: 0.9825
333/1688 [====>.........................] - ETA: 3s - loss: 0.0533 - accuracy: 0.9825
355/1688 [=====>........................] - ETA: 3s - loss: 0.0536 - accuracy: 0.9827
377/1688 [=====>........................] - ETA: 3s - loss: 0.0553 - accuracy: 0.9823
399/1688 [======>.......................] - ETA: 2s - loss: 0.0556 - accuracy: 0.9823
421/1688 [======>.......................] - ETA: 2s - loss: 0.0552 - accuracy: 0.9826
444/1688 [======>.......................] - ETA: 2s - loss: 0.0549 - accuracy: 0.9825
466/1688 [=======>......................] - ETA: 2s - loss: 0.0554 - accuracy: 0.9826
488/1688 [=======>......................] - ETA: 2s - loss: 0.0542 - accuracy: 0.9829
510/1688 [========>.....................] - ETA: 2s - loss: 0.0533 - accuracy: 0.9833
533/1688 [========>.....................] - ETA: 2s - loss: 0.0540 - accuracy: 0.9831
555/1688 [========>.....................] - ETA: 2s - loss: 0.0536 - accuracy: 0.9832
578/1688 [=========>....................] - ETA: 2s - loss: 0.0536 - accuracy: 0.9833
600/1688 [=========>....................] - ETA: 2s - loss: 0.0533 - accuracy: 0.9833
622/1688 [==========>...................] - ETA: 2s - loss: 0.0531 - accuracy: 0.9833
645/1688 [==========>...................] - ETA: 2s - loss: 0.0532 - accuracy: 0.9833
667/1688 [==========>...................] - ETA: 2s - loss: 0.0536 - accuracy: 0.9831
689/1688 [===========>..................] - ETA: 2s - loss: 0.0526 - accuracy: 0.9834
711/1688 [===========>..................] - ETA: 2s - loss: 0.0526 - accuracy: 0.9833
734/1688 [============>.................] - ETA: 2s - loss: 0.0521 - accuracy: 0.9835
757/1688 [============>.................] - ETA: 2s - loss: 0.0528 - accuracy: 0.9835
781/1688 [============>.................] - ETA: 2s - loss: 0.0534 - accuracy: 0.9832
804/1688 [=============>................] - ETA: 2s - loss: 0.0533 - accuracy: 0.9834
826/1688 [=============>................] - ETA: 1s - loss: 0.0534 - accuracy: 0.9834
848/1688 [==============>...............] - ETA: 1s - loss: 0.0537 - accuracy: 0.9833
871/1688 [==============>...............] - ETA: 1s - loss: 0.0548 - accuracy: 0.9831
892/1688 [==============>...............] - ETA: 1s - loss: 0.0543 - accuracy: 0.9833
914/1688 [===============>..............] - ETA: 1s - loss: 0.0534 - accuracy: 0.9836
936/1688 [===============>..............] - ETA: 1s - loss: 0.0537 - accuracy: 0.9836
958/1688 [================>.............] - ETA: 1s - loss: 0.0542 - accuracy: 0.9835
980/1688 [================>.............] - ETA: 1s - loss: 0.0541 - accuracy: 0.9835
1002/1688 [================>.............] - ETA: 1s - loss: 0.0543 - accuracy: 0.9834
1025/1688 [=================>............] - ETA: 1s - loss: 0.0545 - accuracy: 0.9834
1047/1688 [=================>............] - ETA: 1s - loss: 0.0550 - accuracy: 0.9833
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0547 - accuracy: 0.9834
1093/1688 [==================>...........] - ETA: 1s - loss: 0.0560 - accuracy: 0.9832
1115/1688 [==================>...........] - ETA: 1s - loss: 0.0559 - accuracy: 0.9831
1137/1688 [===================>..........] - ETA: 1s - loss: 0.0565 - accuracy: 0.9829
1159/1688 [===================>..........] - ETA: 1s - loss: 0.0564 - accuracy: 0.9829
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0563 - accuracy: 0.9829
1202/1688 [====================>.........] - ETA: 1s - loss: 0.0562 - accuracy: 0.9829
1224/1688 [====================>.........] - ETA: 1s - loss: 0.0566 - accuracy: 0.9827
1246/1688 [=====================>........] - ETA: 1s - loss: 0.0565 - accuracy: 0.9827
1268/1688 [=====================>........] - ETA: 0s - loss: 0.0564 - accuracy: 0.9827
1290/1688 [=====================>........] - ETA: 0s - loss: 0.0563 - accuracy: 0.9827
1313/1688 [======================>.......] - ETA: 0s - loss: 0.0561 - accuracy: 0.9828
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0565 - accuracy: 0.9826
1358/1688 [=======================>......] - ETA: 0s - loss: 0.0566 - accuracy: 0.9825
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0564 - accuracy: 0.9827
1402/1688 [=======================>......] - ETA: 0s - loss: 0.0566 - accuracy: 0.9826
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0564 - accuracy: 0.9826
1448/1688 [========================>.....] - ETA: 0s - loss: 0.0566 - accuracy: 0.9826
1470/1688 [=========================>....] - ETA: 0s - loss: 0.0565 - accuracy: 0.9825
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0564 - accuracy: 0.9826
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0565 - accuracy: 0.9825
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0564 - accuracy: 0.9825
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0563 - accuracy: 0.9825
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0561 - accuracy: 0.9825
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0561 - accuracy: 0.9826
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0563 - accuracy: 0.9826
1648/1688 [============================>.] - ETA: 0s - loss: 0.0562 - accuracy: 0.9826
1670/1688 [============================>.] - ETA: 0s - loss: 0.0565 - accuracy: 0.9824
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0563 - accuracy: 0.9825 - val_loss: 0.0602 - val_accuracy: 0.9845
Epoch 4/10
1/1688 [..............................] - ETA: 3s - loss: 0.0189 - accuracy: 1.0000
23/1688 [..............................] - ETA: 3s - loss: 0.0256 - accuracy: 0.9905
45/1688 [..............................] - ETA: 3s - loss: 0.0312 - accuracy: 0.9882
67/1688 [>.............................] - ETA: 3s - loss: 0.0382 - accuracy: 0.9888
89/1688 [>.............................] - ETA: 3s - loss: 0.0367 - accuracy: 0.9884
112/1688 [>.............................] - ETA: 3s - loss: 0.0400 - accuracy: 0.9872
135/1688 [=>............................] - ETA: 3s - loss: 0.0419 - accuracy: 0.9863
158/1688 [=>............................] - ETA: 3s - loss: 0.0422 - accuracy: 0.9867
181/1688 [==>...........................] - ETA: 3s - loss: 0.0425 - accuracy: 0.9862
203/1688 [==>...........................] - ETA: 3s - loss: 0.0426 - accuracy: 0.9861
225/1688 [==>...........................] - ETA: 3s - loss: 0.0413 - accuracy: 0.9867
247/1688 [===>..........................] - ETA: 3s - loss: 0.0428 - accuracy: 0.9860
270/1688 [===>..........................] - ETA: 3s - loss: 0.0434 - accuracy: 0.9858
293/1688 [====>.........................] - ETA: 3s - loss: 0.0454 - accuracy: 0.9850
315/1688 [====>.........................] - ETA: 3s - loss: 0.0444 - accuracy: 0.9851
337/1688 [====>.........................] - ETA: 3s - loss: 0.0463 - accuracy: 0.9843
359/1688 [=====>........................] - ETA: 3s - loss: 0.0450 - accuracy: 0.9847
382/1688 [=====>........................] - ETA: 2s - loss: 0.0446 - accuracy: 0.9849
404/1688 [======>.......................] - ETA: 2s - loss: 0.0438 - accuracy: 0.9851
425/1688 [======>.......................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9849
447/1688 [======>.......................] - ETA: 2s - loss: 0.0447 - accuracy: 0.9850
470/1688 [=======>......................] - ETA: 2s - loss: 0.0439 - accuracy: 0.9853
491/1688 [=======>......................] - ETA: 2s - loss: 0.0435 - accuracy: 0.9854
513/1688 [========>.....................] - ETA: 2s - loss: 0.0429 - accuracy: 0.9856
535/1688 [========>.....................] - ETA: 2s - loss: 0.0430 - accuracy: 0.9856
558/1688 [========>.....................] - ETA: 2s - loss: 0.0426 - accuracy: 0.9855
579/1688 [=========>....................] - ETA: 2s - loss: 0.0426 - accuracy: 0.9855
601/1688 [=========>....................] - ETA: 2s - loss: 0.0426 - accuracy: 0.9853
623/1688 [==========>...................] - ETA: 2s - loss: 0.0427 - accuracy: 0.9853
645/1688 [==========>...................] - ETA: 2s - loss: 0.0428 - accuracy: 0.9853
668/1688 [==========>...................] - ETA: 2s - loss: 0.0429 - accuracy: 0.9852
691/1688 [===========>..................] - ETA: 2s - loss: 0.0428 - accuracy: 0.9853
713/1688 [===========>..................] - ETA: 2s - loss: 0.0429 - accuracy: 0.9853
735/1688 [============>.................] - ETA: 2s - loss: 0.0427 - accuracy: 0.9854
758/1688 [============>.................] - ETA: 2s - loss: 0.0422 - accuracy: 0.9855
780/1688 [============>.................] - ETA: 2s - loss: 0.0425 - accuracy: 0.9855
802/1688 [=============>................] - ETA: 2s - loss: 0.0422 - accuracy: 0.9855
825/1688 [=============>................] - ETA: 1s - loss: 0.0418 - accuracy: 0.9857
847/1688 [==============>...............] - ETA: 1s - loss: 0.0417 - accuracy: 0.9858
869/1688 [==============>...............] - ETA: 1s - loss: 0.0419 - accuracy: 0.9858
891/1688 [==============>...............] - ETA: 1s - loss: 0.0426 - accuracy: 0.9855
913/1688 [===============>..............] - ETA: 1s - loss: 0.0429 - accuracy: 0.9854
935/1688 [===============>..............] - ETA: 1s - loss: 0.0433 - accuracy: 0.9853
957/1688 [================>.............] - ETA: 1s - loss: 0.0432 - accuracy: 0.9853
979/1688 [================>.............] - ETA: 1s - loss: 0.0429 - accuracy: 0.9855
1001/1688 [================>.............] - ETA: 1s - loss: 0.0431 - accuracy: 0.9853
1023/1688 [=================>............] - ETA: 1s - loss: 0.0430 - accuracy: 0.9855
1046/1688 [=================>............] - ETA: 1s - loss: 0.0437 - accuracy: 0.9852
1069/1688 [=================>............] - ETA: 1s - loss: 0.0437 - accuracy: 0.9852
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0436 - accuracy: 0.9853
1113/1688 [==================>...........] - ETA: 1s - loss: 0.0435 - accuracy: 0.9854
1135/1688 [===================>..........] - ETA: 1s - loss: 0.0436 - accuracy: 0.9854
1157/1688 [===================>..........] - ETA: 1s - loss: 0.0440 - accuracy: 0.9853
1179/1688 [===================>..........] - ETA: 1s - loss: 0.0440 - accuracy: 0.9854
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0441 - accuracy: 0.9854
1223/1688 [====================>.........] - ETA: 1s - loss: 0.0439 - accuracy: 0.9854
1245/1688 [=====================>........] - ETA: 1s - loss: 0.0435 - accuracy: 0.9856
1268/1688 [=====================>........] - ETA: 0s - loss: 0.0434 - accuracy: 0.9856
1291/1688 [=====================>........] - ETA: 0s - loss: 0.0437 - accuracy: 0.9856
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0435 - accuracy: 0.9857
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0435 - accuracy: 0.9857
1358/1688 [=======================>......] - ETA: 0s - loss: 0.0434 - accuracy: 0.9857
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0433 - accuracy: 0.9856
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0431 - accuracy: 0.9857
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0430 - accuracy: 0.9858
1448/1688 [========================>.....] - ETA: 0s - loss: 0.0433 - accuracy: 0.9858
1470/1688 [=========================>....] - ETA: 0s - loss: 0.0431 - accuracy: 0.9859
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0436 - accuracy: 0.9858
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0437 - accuracy: 0.9856
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0441 - accuracy: 0.9855
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0444 - accuracy: 0.9855
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0445 - accuracy: 0.9854
1603/1688 [===========================>..] - ETA: 0s - loss: 0.0446 - accuracy: 0.9854
1625/1688 [===========================>..] - ETA: 0s - loss: 0.0445 - accuracy: 0.9854
1648/1688 [============================>.] - ETA: 0s - loss: 0.0444 - accuracy: 0.9854
1671/1688 [============================>.] - ETA: 0s - loss: 0.0444 - accuracy: 0.9854
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0448 - accuracy: 0.9853 - val_loss: 0.0678 - val_accuracy: 0.9818
Epoch 5/10
1/1688 [..............................] - ETA: 4s - loss: 0.1496 - accuracy: 0.9688
22/1688 [..............................] - ETA: 4s - loss: 0.0378 - accuracy: 0.9858
44/1688 [..............................] - ETA: 3s - loss: 0.0295 - accuracy: 0.9886
67/1688 [>.............................] - ETA: 3s - loss: 0.0283 - accuracy: 0.9902
89/1688 [>.............................] - ETA: 3s - loss: 0.0281 - accuracy: 0.9902
111/1688 [>.............................] - ETA: 3s - loss: 0.0278 - accuracy: 0.9901
134/1688 [=>............................] - ETA: 3s - loss: 0.0276 - accuracy: 0.9904
155/1688 [=>............................] - ETA: 3s - loss: 0.0294 - accuracy: 0.9897
176/1688 [==>...........................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9901
197/1688 [==>...........................] - ETA: 3s - loss: 0.0283 - accuracy: 0.9905
218/1688 [==>...........................] - ETA: 3s - loss: 0.0280 - accuracy: 0.9907
239/1688 [===>..........................] - ETA: 3s - loss: 0.0302 - accuracy: 0.9901
260/1688 [===>..........................] - ETA: 3s - loss: 0.0314 - accuracy: 0.9897
282/1688 [====>.........................] - ETA: 3s - loss: 0.0303 - accuracy: 0.9900
304/1688 [====>.........................] - ETA: 3s - loss: 0.0314 - accuracy: 0.9900
326/1688 [====>.........................] - ETA: 3s - loss: 0.0330 - accuracy: 0.9897
349/1688 [=====>........................] - ETA: 3s - loss: 0.0324 - accuracy: 0.9902
371/1688 [=====>........................] - ETA: 3s - loss: 0.0322 - accuracy: 0.9901
393/1688 [=====>........................] - ETA: 3s - loss: 0.0316 - accuracy: 0.9905
415/1688 [======>.......................] - ETA: 2s - loss: 0.0316 - accuracy: 0.9904
438/1688 [======>.......................] - ETA: 2s - loss: 0.0309 - accuracy: 0.9907
460/1688 [=======>......................] - ETA: 2s - loss: 0.0307 - accuracy: 0.9907
483/1688 [=======>......................] - ETA: 2s - loss: 0.0303 - accuracy: 0.9909
506/1688 [=======>......................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9908
528/1688 [========>.....................] - ETA: 2s - loss: 0.0297 - accuracy: 0.9909
550/1688 [========>.....................] - ETA: 2s - loss: 0.0303 - accuracy: 0.9907
573/1688 [=========>....................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9907
596/1688 [=========>....................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9905
618/1688 [=========>....................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9904
640/1688 [==========>...................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9903
662/1688 [==========>...................] - ETA: 2s - loss: 0.0307 - accuracy: 0.9904
684/1688 [===========>..................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9904
706/1688 [===========>..................] - ETA: 2s - loss: 0.0313 - accuracy: 0.9902
728/1688 [===========>..................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9900
750/1688 [============>.................] - ETA: 2s - loss: 0.0321 - accuracy: 0.9899
773/1688 [============>.................] - ETA: 2s - loss: 0.0330 - accuracy: 0.9897
796/1688 [=============>................] - ETA: 2s - loss: 0.0334 - accuracy: 0.9895
819/1688 [=============>................] - ETA: 2s - loss: 0.0337 - accuracy: 0.9894
841/1688 [=============>................] - ETA: 1s - loss: 0.0340 - accuracy: 0.9893
863/1688 [==============>...............] - ETA: 1s - loss: 0.0338 - accuracy: 0.9894
886/1688 [==============>...............] - ETA: 1s - loss: 0.0338 - accuracy: 0.9894
908/1688 [===============>..............] - ETA: 1s - loss: 0.0337 - accuracy: 0.9895
930/1688 [===============>..............] - ETA: 1s - loss: 0.0341 - accuracy: 0.9893
953/1688 [===============>..............] - ETA: 1s - loss: 0.0342 - accuracy: 0.9893
976/1688 [================>.............] - ETA: 1s - loss: 0.0345 - accuracy: 0.9893
998/1688 [================>.............] - ETA: 1s - loss: 0.0348 - accuracy: 0.9892
1020/1688 [=================>............] - ETA: 1s - loss: 0.0351 - accuracy: 0.9891
1042/1688 [=================>............] - ETA: 1s - loss: 0.0350 - accuracy: 0.9890
1064/1688 [=================>............] - ETA: 1s - loss: 0.0349 - accuracy: 0.9889
1087/1688 [==================>...........] - ETA: 1s - loss: 0.0350 - accuracy: 0.9889
1109/1688 [==================>...........] - ETA: 1s - loss: 0.0353 - accuracy: 0.9889
1131/1688 [===================>..........] - ETA: 1s - loss: 0.0356 - accuracy: 0.9888
1153/1688 [===================>..........] - ETA: 1s - loss: 0.0361 - accuracy: 0.9886
1176/1688 [===================>..........] - ETA: 1s - loss: 0.0360 - accuracy: 0.9886
1199/1688 [====================>.........] - ETA: 1s - loss: 0.0357 - accuracy: 0.9888
1221/1688 [====================>.........] - ETA: 1s - loss: 0.0357 - accuracy: 0.9888
1243/1688 [=====================>........] - ETA: 1s - loss: 0.0356 - accuracy: 0.9888
1265/1688 [=====================>........] - ETA: 0s - loss: 0.0355 - accuracy: 0.9888
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0355 - accuracy: 0.9888
1311/1688 [======================>.......] - ETA: 0s - loss: 0.0356 - accuracy: 0.9887
1334/1688 [======================>.......] - ETA: 0s - loss: 0.0357 - accuracy: 0.9887
1357/1688 [=======================>......] - ETA: 0s - loss: 0.0356 - accuracy: 0.9886
1379/1688 [=======================>......] - ETA: 0s - loss: 0.0359 - accuracy: 0.9886
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0359 - accuracy: 0.9886
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0359 - accuracy: 0.9885
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0361 - accuracy: 0.9885
1466/1688 [=========================>....] - ETA: 0s - loss: 0.0358 - accuracy: 0.9886
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0358 - accuracy: 0.9886
1510/1688 [=========================>....] - ETA: 0s - loss: 0.0356 - accuracy: 0.9887
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0356 - accuracy: 0.9888
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0359 - accuracy: 0.9887
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0361 - accuracy: 0.9887
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0364 - accuracy: 0.9887
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0364 - accuracy: 0.9886
1644/1688 [============================>.] - ETA: 0s - loss: 0.0363 - accuracy: 0.9886
1666/1688 [============================>.] - ETA: 0s - loss: 0.0362 - accuracy: 0.9887
1688/1688 [==============================] - ETA: 0s - loss: 0.0365 - accuracy: 0.9886
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0365 - accuracy: 0.9886 - val_loss: 0.0545 - val_accuracy: 0.9850
Epoch 6/10
1/1688 [..............................] - ETA: 4s - loss: 0.0014 - accuracy: 1.0000
22/1688 [..............................] - ETA: 3s - loss: 0.0244 - accuracy: 0.9915
44/1688 [..............................] - ETA: 3s - loss: 0.0305 - accuracy: 0.9893
66/1688 [>.............................] - ETA: 3s - loss: 0.0282 - accuracy: 0.9896
87/1688 [>.............................] - ETA: 3s - loss: 0.0240 - accuracy: 0.9914
110/1688 [>.............................] - ETA: 3s - loss: 0.0244 - accuracy: 0.9909
132/1688 [=>............................] - ETA: 3s - loss: 0.0252 - accuracy: 0.9910
155/1688 [=>............................] - ETA: 3s - loss: 0.0234 - accuracy: 0.9919
177/1688 [==>...........................] - ETA: 3s - loss: 0.0240 - accuracy: 0.9919
199/1688 [==>...........................] - ETA: 3s - loss: 0.0242 - accuracy: 0.9923
221/1688 [==>...........................] - ETA: 3s - loss: 0.0242 - accuracy: 0.9925
242/1688 [===>..........................] - ETA: 3s - loss: 0.0235 - accuracy: 0.9926
264/1688 [===>..........................] - ETA: 3s - loss: 0.0241 - accuracy: 0.9927
287/1688 [====>.........................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9928
309/1688 [====>.........................] - ETA: 3s - loss: 0.0249 - accuracy: 0.9925
331/1688 [====>.........................] - ETA: 3s - loss: 0.0254 - accuracy: 0.9922
352/1688 [=====>........................] - ETA: 3s - loss: 0.0258 - accuracy: 0.9921
374/1688 [=====>........................] - ETA: 3s - loss: 0.0260 - accuracy: 0.9918
397/1688 [======>.......................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9917
419/1688 [======>.......................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9917
441/1688 [======>.......................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9916
463/1688 [=======>......................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9918
485/1688 [=======>......................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9918
507/1688 [========>.....................] - ETA: 2s - loss: 0.0258 - accuracy: 0.9918
528/1688 [========>.....................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9920
550/1688 [========>.....................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9919
573/1688 [=========>....................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9919
595/1688 [=========>....................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9919
617/1688 [=========>....................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9918
639/1688 [==========>...................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9916
661/1688 [==========>...................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9916
683/1688 [===========>..................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9916
705/1688 [===========>..................] - ETA: 2s - loss: 0.0268 - accuracy: 0.9914
727/1688 [===========>..................] - ETA: 2s - loss: 0.0267 - accuracy: 0.9914
749/1688 [============>.................] - ETA: 2s - loss: 0.0269 - accuracy: 0.9912
771/1688 [============>.................] - ETA: 2s - loss: 0.0266 - accuracy: 0.9914
793/1688 [=============>................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9914
816/1688 [=============>................] - ETA: 2s - loss: 0.0262 - accuracy: 0.9915
838/1688 [=============>................] - ETA: 1s - loss: 0.0260 - accuracy: 0.9916
860/1688 [==============>...............] - ETA: 1s - loss: 0.0261 - accuracy: 0.9916
883/1688 [==============>...............] - ETA: 1s - loss: 0.0264 - accuracy: 0.9915
905/1688 [===============>..............] - ETA: 1s - loss: 0.0265 - accuracy: 0.9914
927/1688 [===============>..............] - ETA: 1s - loss: 0.0269 - accuracy: 0.9913
950/1688 [===============>..............] - ETA: 1s - loss: 0.0274 - accuracy: 0.9911
972/1688 [================>.............] - ETA: 1s - loss: 0.0278 - accuracy: 0.9910
994/1688 [================>.............] - ETA: 1s - loss: 0.0278 - accuracy: 0.9909
1017/1688 [=================>............] - ETA: 1s - loss: 0.0278 - accuracy: 0.9910
1040/1688 [=================>............] - ETA: 1s - loss: 0.0275 - accuracy: 0.9911
1064/1688 [=================>............] - ETA: 1s - loss: 0.0273 - accuracy: 0.9912
1086/1688 [==================>...........] - ETA: 1s - loss: 0.0274 - accuracy: 0.9913
1108/1688 [==================>...........] - ETA: 1s - loss: 0.0274 - accuracy: 0.9912
1131/1688 [===================>..........] - ETA: 1s - loss: 0.0273 - accuracy: 0.9912
1153/1688 [===================>..........] - ETA: 1s - loss: 0.0281 - accuracy: 0.9910
1175/1688 [===================>..........] - ETA: 1s - loss: 0.0281 - accuracy: 0.9909
1197/1688 [====================>.........] - ETA: 1s - loss: 0.0284 - accuracy: 0.9909
1219/1688 [====================>.........] - ETA: 1s - loss: 0.0284 - accuracy: 0.9909
1240/1688 [=====================>........] - ETA: 1s - loss: 0.0285 - accuracy: 0.9909
1263/1688 [=====================>........] - ETA: 0s - loss: 0.0285 - accuracy: 0.9909
1287/1688 [=====================>........] - ETA: 0s - loss: 0.0286 - accuracy: 0.9909
1309/1688 [======================>.......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9908
1332/1688 [======================>.......] - ETA: 0s - loss: 0.0290 - accuracy: 0.9907
1355/1688 [=======================>......] - ETA: 0s - loss: 0.0289 - accuracy: 0.9908
1377/1688 [=======================>......] - ETA: 0s - loss: 0.0296 - accuracy: 0.9905
1399/1688 [=======================>......] - ETA: 0s - loss: 0.0296 - accuracy: 0.9905
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0296 - accuracy: 0.9905
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0300 - accuracy: 0.9904
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0304 - accuracy: 0.9903
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0308 - accuracy: 0.9901
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0309 - accuracy: 0.9900
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0309 - accuracy: 0.9901
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0308 - accuracy: 0.9901
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0308 - accuracy: 0.9900
1601/1688 [===========================>..] - ETA: 0s - loss: 0.0312 - accuracy: 0.9900
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0312 - accuracy: 0.9900
1647/1688 [============================>.] - ETA: 0s - loss: 0.0311 - accuracy: 0.9900
1669/1688 [============================>.] - ETA: 0s - loss: 0.0313 - accuracy: 0.9899
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0312 - accuracy: 0.9900 - val_loss: 0.0678 - val_accuracy: 0.9842
Epoch 7/10
1/1688 [..............................] - ETA: 4s - loss: 0.0168 - accuracy: 1.0000
23/1688 [..............................] - ETA: 3s - loss: 0.0297 - accuracy: 0.9932
46/1688 [..............................] - ETA: 3s - loss: 0.0260 - accuracy: 0.9939
69/1688 [>.............................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9946
91/1688 [>.............................] - ETA: 3s - loss: 0.0206 - accuracy: 0.9935
113/1688 [=>............................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9931
136/1688 [=>............................] - ETA: 3s - loss: 0.0206 - accuracy: 0.9931
159/1688 [=>............................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9923
181/1688 [==>...........................] - ETA: 3s - loss: 0.0206 - accuracy: 0.9926
203/1688 [==>...........................] - ETA: 3s - loss: 0.0208 - accuracy: 0.9925
225/1688 [==>...........................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9926
247/1688 [===>..........................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9927
269/1688 [===>..........................] - ETA: 3s - loss: 0.0194 - accuracy: 0.9930
291/1688 [====>.........................] - ETA: 3s - loss: 0.0196 - accuracy: 0.9931
313/1688 [====>.........................] - ETA: 3s - loss: 0.0213 - accuracy: 0.9926
335/1688 [====>.........................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9925
358/1688 [=====>........................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9926
381/1688 [=====>........................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9925
404/1688 [======>.......................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9926
426/1688 [======>.......................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9928
448/1688 [======>.......................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9930
470/1688 [=======>......................] - ETA: 2s - loss: 0.0210 - accuracy: 0.9928
492/1688 [=======>......................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9928
514/1688 [========>.....................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9929
536/1688 [========>.....................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9928
558/1688 [========>.....................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9928
580/1688 [=========>....................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9927
602/1688 [=========>....................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9927
625/1688 [==========>...................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9927
647/1688 [==========>...................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9926
670/1688 [==========>...................] - ETA: 2s - loss: 0.0209 - accuracy: 0.9928
692/1688 [===========>..................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9929
714/1688 [===========>..................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9927
736/1688 [============>.................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9927
758/1688 [============>.................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9928
781/1688 [============>.................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9928
803/1688 [=============>................] - ETA: 2s - loss: 0.0214 - accuracy: 0.9928
825/1688 [=============>................] - ETA: 1s - loss: 0.0214 - accuracy: 0.9928
847/1688 [==============>...............] - ETA: 1s - loss: 0.0218 - accuracy: 0.9926
869/1688 [==============>...............] - ETA: 1s - loss: 0.0220 - accuracy: 0.9924
892/1688 [==============>...............] - ETA: 1s - loss: 0.0221 - accuracy: 0.9925
915/1688 [===============>..............] - ETA: 1s - loss: 0.0223 - accuracy: 0.9925
937/1688 [===============>..............] - ETA: 1s - loss: 0.0226 - accuracy: 0.9924
959/1688 [================>.............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9922
982/1688 [================>.............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9922
1004/1688 [================>.............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9922
1026/1688 [=================>............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9921
1048/1688 [=================>............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9921
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0235 - accuracy: 0.9921
1092/1688 [==================>...........] - ETA: 1s - loss: 0.0238 - accuracy: 0.9920
1115/1688 [==================>...........] - ETA: 1s - loss: 0.0237 - accuracy: 0.9919
1136/1688 [===================>..........] - ETA: 1s - loss: 0.0240 - accuracy: 0.9918
1159/1688 [===================>..........] - ETA: 1s - loss: 0.0240 - accuracy: 0.9918
1181/1688 [===================>..........] - ETA: 1s - loss: 0.0241 - accuracy: 0.9917
1204/1688 [====================>.........] - ETA: 1s - loss: 0.0246 - accuracy: 0.9916
1226/1688 [====================>.........] - ETA: 1s - loss: 0.0249 - accuracy: 0.9915
1249/1688 [=====================>........] - ETA: 1s - loss: 0.0254 - accuracy: 0.9913
1271/1688 [=====================>........] - ETA: 0s - loss: 0.0254 - accuracy: 0.9912
1292/1688 [=====================>........] - ETA: 0s - loss: 0.0253 - accuracy: 0.9913
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0253 - accuracy: 0.9912
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0251 - accuracy: 0.9913
1358/1688 [=======================>......] - ETA: 0s - loss: 0.0253 - accuracy: 0.9913
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0254 - accuracy: 0.9912
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0253 - accuracy: 0.9913
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0255 - accuracy: 0.9912
1446/1688 [========================>.....] - ETA: 0s - loss: 0.0255 - accuracy: 0.9913
1468/1688 [=========================>....] - ETA: 0s - loss: 0.0255 - accuracy: 0.9913
1490/1688 [=========================>....] - ETA: 0s - loss: 0.0256 - accuracy: 0.9912
1513/1688 [=========================>....] - ETA: 0s - loss: 0.0258 - accuracy: 0.9911
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0256 - accuracy: 0.9913
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0255 - accuracy: 0.9913
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0257 - accuracy: 0.9913
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0255 - accuracy: 0.9914
1625/1688 [===========================>..] - ETA: 0s - loss: 0.0256 - accuracy: 0.9913
1647/1688 [============================>.] - ETA: 0s - loss: 0.0257 - accuracy: 0.9913
1667/1688 [============================>.] - ETA: 0s - loss: 0.0256 - accuracy: 0.9913
1685/1688 [============================>.] - ETA: 0s - loss: 0.0258 - accuracy: 0.9912
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0258 - accuracy: 0.9912 - val_loss: 0.0524 - val_accuracy: 0.9877
Epoch 8/10
1/1688 [..............................] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
24/1688 [..............................] - ETA: 3s - loss: 0.0284 - accuracy: 0.9909
46/1688 [..............................] - ETA: 3s - loss: 0.0213 - accuracy: 0.9925
68/1688 [>.............................] - ETA: 3s - loss: 0.0221 - accuracy: 0.9922
91/1688 [>.............................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9935
113/1688 [=>............................] - ETA: 3s - loss: 0.0184 - accuracy: 0.9936
135/1688 [=>............................] - ETA: 3s - loss: 0.0164 - accuracy: 0.9944
157/1688 [=>............................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9950
180/1688 [==>...........................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9955
202/1688 [==>...........................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9955
224/1688 [==>...........................] - ETA: 3s - loss: 0.0151 - accuracy: 0.9953
246/1688 [===>..........................] - ETA: 3s - loss: 0.0155 - accuracy: 0.9949
268/1688 [===>..........................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9949
290/1688 [====>.........................] - ETA: 3s - loss: 0.0156 - accuracy: 0.9950
312/1688 [====>.........................] - ETA: 3s - loss: 0.0158 - accuracy: 0.9950
334/1688 [====>.........................] - ETA: 3s - loss: 0.0157 - accuracy: 0.9951
356/1688 [=====>........................] - ETA: 3s - loss: 0.0156 - accuracy: 0.9952
377/1688 [=====>........................] - ETA: 3s - loss: 0.0159 - accuracy: 0.9950
399/1688 [======>.......................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9949
422/1688 [======>.......................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9949
445/1688 [======>.......................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9949
468/1688 [=======>......................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9949
490/1688 [=======>......................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9949
512/1688 [========>.....................] - ETA: 2s - loss: 0.0159 - accuracy: 0.9949
535/1688 [========>.....................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9950
558/1688 [========>.....................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9948
581/1688 [=========>....................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9947
602/1688 [=========>....................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9947
624/1688 [==========>...................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9947
646/1688 [==========>...................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9947
668/1688 [==========>...................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9946
690/1688 [===========>..................] - ETA: 2s - loss: 0.0168 - accuracy: 0.9944
712/1688 [===========>..................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9942
734/1688 [============>.................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9940
756/1688 [============>.................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9940
779/1688 [============>.................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9941
801/1688 [=============>................] - ETA: 2s - loss: 0.0183 - accuracy: 0.9940
824/1688 [=============>................] - ETA: 1s - loss: 0.0187 - accuracy: 0.9939
847/1688 [==============>...............] - ETA: 1s - loss: 0.0185 - accuracy: 0.9940
869/1688 [==============>...............] - ETA: 1s - loss: 0.0183 - accuracy: 0.9941
893/1688 [==============>...............] - ETA: 1s - loss: 0.0182 - accuracy: 0.9940
915/1688 [===============>..............] - ETA: 1s - loss: 0.0184 - accuracy: 0.9939
937/1688 [===============>..............] - ETA: 1s - loss: 0.0187 - accuracy: 0.9936
960/1688 [================>.............] - ETA: 1s - loss: 0.0187 - accuracy: 0.9936
981/1688 [================>.............] - ETA: 1s - loss: 0.0189 - accuracy: 0.9936
1004/1688 [================>.............] - ETA: 1s - loss: 0.0190 - accuracy: 0.9936
1027/1688 [=================>............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9934
1049/1688 [=================>............] - ETA: 1s - loss: 0.0202 - accuracy: 0.9933
1071/1688 [==================>...........] - ETA: 1s - loss: 0.0202 - accuracy: 0.9933
1094/1688 [==================>...........] - ETA: 1s - loss: 0.0201 - accuracy: 0.9934
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0200 - accuracy: 0.9934
1138/1688 [===================>..........] - ETA: 1s - loss: 0.0202 - accuracy: 0.9934
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0204 - accuracy: 0.9934
1183/1688 [====================>.........] - ETA: 1s - loss: 0.0203 - accuracy: 0.9934
1205/1688 [====================>.........] - ETA: 1s - loss: 0.0202 - accuracy: 0.9934
1228/1688 [====================>.........] - ETA: 1s - loss: 0.0203 - accuracy: 0.9933
1250/1688 [=====================>........] - ETA: 1s - loss: 0.0202 - accuracy: 0.9933
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0203 - accuracy: 0.9933
1295/1688 [======================>.......] - ETA: 0s - loss: 0.0203 - accuracy: 0.9933
1317/1688 [======================>.......] - ETA: 0s - loss: 0.0203 - accuracy: 0.9932
1340/1688 [======================>.......] - ETA: 0s - loss: 0.0205 - accuracy: 0.9931
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0205 - accuracy: 0.9931
1385/1688 [=======================>......] - ETA: 0s - loss: 0.0206 - accuracy: 0.9931
1407/1688 [========================>.....] - ETA: 0s - loss: 0.0205 - accuracy: 0.9931
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0206 - accuracy: 0.9930
1451/1688 [========================>.....] - ETA: 0s - loss: 0.0207 - accuracy: 0.9930
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0206 - accuracy: 0.9930
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0208 - accuracy: 0.9930
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0209 - accuracy: 0.9930
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0210 - accuracy: 0.9929
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0210 - accuracy: 0.9929
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0209 - accuracy: 0.9929
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0211 - accuracy: 0.9928
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0211 - accuracy: 0.9929
1648/1688 [============================>.] - ETA: 0s - loss: 0.0211 - accuracy: 0.9929
1672/1688 [============================>.] - ETA: 0s - loss: 0.0211 - accuracy: 0.9929
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0210 - accuracy: 0.9929 - val_loss: 0.0661 - val_accuracy: 0.9852
Epoch 9/10
1/1688 [..............................] - ETA: 4s - loss: 0.0031 - accuracy: 1.0000
23/1688 [..............................] - ETA: 3s - loss: 0.0283 - accuracy: 0.9932
45/1688 [..............................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9944
67/1688 [>.............................] - ETA: 3s - loss: 0.0180 - accuracy: 0.9953
89/1688 [>.............................] - ETA: 3s - loss: 0.0200 - accuracy: 0.9937
112/1688 [>.............................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9936
134/1688 [=>............................] - ETA: 3s - loss: 0.0176 - accuracy: 0.9937
156/1688 [=>............................] - ETA: 3s - loss: 0.0169 - accuracy: 0.9938
178/1688 [==>...........................] - ETA: 3s - loss: 0.0163 - accuracy: 0.9940
201/1688 [==>...........................] - ETA: 3s - loss: 0.0149 - accuracy: 0.9947
224/1688 [==>...........................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9950
246/1688 [===>..........................] - ETA: 3s - loss: 0.0144 - accuracy: 0.9950
269/1688 [===>..........................] - ETA: 3s - loss: 0.0138 - accuracy: 0.9952
291/1688 [====>.........................] - ETA: 3s - loss: 0.0136 - accuracy: 0.9954
314/1688 [====>.........................] - ETA: 3s - loss: 0.0135 - accuracy: 0.9955
336/1688 [====>.........................] - ETA: 3s - loss: 0.0133 - accuracy: 0.9956
358/1688 [=====>........................] - ETA: 3s - loss: 0.0131 - accuracy: 0.9957
381/1688 [=====>........................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9957
404/1688 [======>.......................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9956
426/1688 [======>.......................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9955
448/1688 [======>.......................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9952
470/1688 [=======>......................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9951
493/1688 [=======>......................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9952
516/1688 [========>.....................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9954
539/1688 [========>.....................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9953
561/1688 [========>.....................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9953
583/1688 [=========>....................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9953
605/1688 [=========>....................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9952
627/1688 [==========>...................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9953
649/1688 [==========>...................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9950
671/1688 [==========>...................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9952
693/1688 [===========>..................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9950
715/1688 [===========>..................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9950
737/1688 [============>.................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9950
760/1688 [============>.................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9950
781/1688 [============>.................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9951
803/1688 [=============>................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9952
825/1688 [=============>................] - ETA: 1s - loss: 0.0139 - accuracy: 0.9952
847/1688 [==============>...............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9951
869/1688 [==============>...............] - ETA: 1s - loss: 0.0145 - accuracy: 0.9950
891/1688 [==============>...............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9949
913/1688 [===============>..............] - ETA: 1s - loss: 0.0148 - accuracy: 0.9949
935/1688 [===============>..............] - ETA: 1s - loss: 0.0150 - accuracy: 0.9948
958/1688 [================>.............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9947
980/1688 [================>.............] - ETA: 1s - loss: 0.0157 - accuracy: 0.9945
1002/1688 [================>.............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9944
1024/1688 [=================>............] - ETA: 1s - loss: 0.0164 - accuracy: 0.9944
1046/1688 [=================>............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9943
1069/1688 [=================>............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9944
1092/1688 [==================>...........] - ETA: 1s - loss: 0.0167 - accuracy: 0.9944
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0170 - accuracy: 0.9942
1136/1688 [===================>..........] - ETA: 1s - loss: 0.0173 - accuracy: 0.9942
1158/1688 [===================>..........] - ETA: 1s - loss: 0.0172 - accuracy: 0.9941
1180/1688 [===================>..........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9940
1203/1688 [====================>.........] - ETA: 1s - loss: 0.0178 - accuracy: 0.9941
1225/1688 [====================>.........] - ETA: 1s - loss: 0.0178 - accuracy: 0.9940
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0180 - accuracy: 0.9939
1269/1688 [=====================>........] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1292/1688 [=====================>........] - ETA: 0s - loss: 0.0185 - accuracy: 0.9937
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0183 - accuracy: 0.9938
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0184 - accuracy: 0.9938
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0183 - accuracy: 0.9938
1382/1688 [=======================>......] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1404/1688 [=======================>......] - ETA: 0s - loss: 0.0182 - accuracy: 0.9938
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0187 - accuracy: 0.9936
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9936
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0190 - accuracy: 0.9935
1493/1688 [=========================>....] - ETA: 0s - loss: 0.0193 - accuracy: 0.9934
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0193 - accuracy: 0.9935
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0193 - accuracy: 0.9935
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0192 - accuracy: 0.9935
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0193 - accuracy: 0.9935
1603/1688 [===========================>..] - ETA: 0s - loss: 0.0193 - accuracy: 0.9935
1625/1688 [===========================>..] - ETA: 0s - loss: 0.0194 - accuracy: 0.9935
1648/1688 [============================>.] - ETA: 0s - loss: 0.0194 - accuracy: 0.9935
1671/1688 [============================>.] - ETA: 0s - loss: 0.0192 - accuracy: 0.9935
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0192 - accuracy: 0.9936 - val_loss: 0.0506 - val_accuracy: 0.9872
Epoch 10/10
1/1688 [..............................] - ETA: 4s - loss: 0.0046 - accuracy: 1.0000
23/1688 [..............................] - ETA: 3s - loss: 0.0088 - accuracy: 0.9973
46/1688 [..............................] - ETA: 3s - loss: 0.0080 - accuracy: 0.9973
68/1688 [>.............................] - ETA: 3s - loss: 0.0092 - accuracy: 0.9968
90/1688 [>.............................] - ETA: 3s - loss: 0.0095 - accuracy: 0.9969
113/1688 [=>............................] - ETA: 3s - loss: 0.0091 - accuracy: 0.9970
135/1688 [=>............................] - ETA: 3s - loss: 0.0095 - accuracy: 0.9970
158/1688 [=>............................] - ETA: 3s - loss: 0.0096 - accuracy: 0.9968
180/1688 [==>...........................] - ETA: 3s - loss: 0.0094 - accuracy: 0.9969
202/1688 [==>...........................] - ETA: 3s - loss: 0.0101 - accuracy: 0.9966
225/1688 [==>...........................] - ETA: 3s - loss: 0.0100 - accuracy: 0.9967
248/1688 [===>..........................] - ETA: 3s - loss: 0.0102 - accuracy: 0.9965
270/1688 [===>..........................] - ETA: 3s - loss: 0.0101 - accuracy: 0.9964
292/1688 [====>.........................] - ETA: 3s - loss: 0.0102 - accuracy: 0.9965
315/1688 [====>.........................] - ETA: 3s - loss: 0.0107 - accuracy: 0.9963
338/1688 [=====>........................] - ETA: 3s - loss: 0.0110 - accuracy: 0.9961
360/1688 [=====>........................] - ETA: 3s - loss: 0.0113 - accuracy: 0.9961
382/1688 [=====>........................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9962
404/1688 [======>.......................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9962
427/1688 [======>.......................] - ETA: 2s - loss: 0.0113 - accuracy: 0.9962
449/1688 [======>.......................] - ETA: 2s - loss: 0.0117 - accuracy: 0.9960
472/1688 [=======>......................] - ETA: 2s - loss: 0.0126 - accuracy: 0.9958
494/1688 [=======>......................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9956
516/1688 [========>.....................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9955
538/1688 [========>.....................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9957
561/1688 [========>.....................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9958
583/1688 [=========>....................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9958
605/1688 [=========>....................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9958
627/1688 [==========>...................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9959
649/1688 [==========>...................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9958
672/1688 [==========>...................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9959
694/1688 [===========>..................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9959
716/1688 [===========>..................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9959
738/1688 [============>.................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9959
760/1688 [============>.................] - ETA: 2s - loss: 0.0127 - accuracy: 0.9960
782/1688 [============>.................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9958
803/1688 [=============>................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9958
825/1688 [=============>................] - ETA: 1s - loss: 0.0131 - accuracy: 0.9957
847/1688 [==============>...............] - ETA: 1s - loss: 0.0135 - accuracy: 0.9955
869/1688 [==============>...............] - ETA: 1s - loss: 0.0134 - accuracy: 0.9955
891/1688 [==============>...............] - ETA: 1s - loss: 0.0134 - accuracy: 0.9955
913/1688 [===============>..............] - ETA: 1s - loss: 0.0135 - accuracy: 0.9956
935/1688 [===============>..............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9955
957/1688 [================>.............] - ETA: 1s - loss: 0.0137 - accuracy: 0.9954
978/1688 [================>.............] - ETA: 1s - loss: 0.0138 - accuracy: 0.9954
1000/1688 [================>.............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9955
1023/1688 [=================>............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9955
1046/1688 [=================>............] - ETA: 1s - loss: 0.0138 - accuracy: 0.9955
1068/1688 [=================>............] - ETA: 1s - loss: 0.0141 - accuracy: 0.9953
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0141 - accuracy: 0.9954
1113/1688 [==================>...........] - ETA: 1s - loss: 0.0141 - accuracy: 0.9954
1135/1688 [===================>..........] - ETA: 1s - loss: 0.0144 - accuracy: 0.9953
1157/1688 [===================>..........] - ETA: 1s - loss: 0.0145 - accuracy: 0.9953
1179/1688 [===================>..........] - ETA: 1s - loss: 0.0148 - accuracy: 0.9953
1202/1688 [====================>.........] - ETA: 1s - loss: 0.0147 - accuracy: 0.9952
1224/1688 [====================>.........] - ETA: 1s - loss: 0.0149 - accuracy: 0.9952
1247/1688 [=====================>........] - ETA: 1s - loss: 0.0150 - accuracy: 0.9952
1269/1688 [=====================>........] - ETA: 0s - loss: 0.0149 - accuracy: 0.9952
1292/1688 [=====================>........] - ETA: 0s - loss: 0.0149 - accuracy: 0.9952
1314/1688 [======================>.......] - ETA: 0s - loss: 0.0149 - accuracy: 0.9951
1337/1688 [======================>.......] - ETA: 0s - loss: 0.0150 - accuracy: 0.9951
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0151 - accuracy: 0.9951
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0151 - accuracy: 0.9951
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0152 - accuracy: 0.9951
1427/1688 [========================>.....] - ETA: 0s - loss: 0.0153 - accuracy: 0.9951
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0153 - accuracy: 0.9950
1471/1688 [=========================>....] - ETA: 0s - loss: 0.0155 - accuracy: 0.9950
1493/1688 [=========================>....] - ETA: 0s - loss: 0.0157 - accuracy: 0.9949
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0157 - accuracy: 0.9948
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0162 - accuracy: 0.9947
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0163 - accuracy: 0.9947
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0161 - accuracy: 0.9947
1651/1688 [============================>.] - ETA: 0s - loss: 0.0162 - accuracy: 0.9947
1674/1688 [============================>.] - ETA: 0s - loss: 0.0164 - accuracy: 0.9946
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0164 - accuracy: 0.9946 - val_loss: 0.0619 - val_accuracy: 0.9862
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Test accuracy: 0.9853000044822693
2. Quantize
2.1. 8-bit quantization
An Akida accelerator processes 8 or 4-bit integer activations and weights. Therefore, the floating-point TF-Keras model must be quantized in preparation to run on an Akida accelerator.
The QuantizeML quantize function can be used to quantize a TF-Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating-point TF-Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.
The quantization process results in a TF-Keras model with custom QuantizeML quantized layers substituted for the original TF-Keras layers.
All TF-Keras API functions can be applied on this new model: summary(), compile(),
fit(). etc.
Note
The quantize function applies several transformations to
the original model. For example, it folds the batch normalization layers into the
corresponding neural layers. The new weights are computed according to this folding
operation.
from quantizeml.models import quantize, QuantizationParams
qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
/usr/local/lib/python3.11/dist-packages/quantizeml/models/quantize.py:577: UserWarning: Quantizing per-axis with random calibration samples is not accurate. Set QuantizationParams.per_tensor_activations=True when calibrating with random samples. Continuing execution.
warnings.warn("Quantizing per-axis with random calibration samples is not accurate. "
1/1024 [..............................] - ETA: 2:32
51/1024 [>.............................] - ETA: 0s
103/1024 [==>...........................] - ETA: 0s
154/1024 [===>..........................] - ETA: 0s
206/1024 [=====>........................] - ETA: 0s
258/1024 [======>.......................] - ETA: 0s
309/1024 [========>.....................] - ETA: 0s
360/1024 [=========>....................] - ETA: 0s
412/1024 [===========>..................] - ETA: 0s
464/1024 [============>.................] - ETA: 0s
516/1024 [==============>...............] - ETA: 0s
567/1024 [===============>..............] - ETA: 0s
618/1024 [=================>............] - ETA: 0s
670/1024 [==================>...........] - ETA: 0s
721/1024 [====================>.........] - ETA: 0s
772/1024 [=====================>........] - ETA: 0s
824/1024 [=======================>......] - ETA: 0s
876/1024 [========================>.....] - ETA: 0s
928/1024 [==========================>...] - ETA: 0s
981/1024 [===========================>..] - ETA: 0s
1024/1024 [==============================] - 1s 977us/step
model_quantized.summary()
Model: "mnistnet"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input (InputLayer) [(None, 28, 28, 1)] 0
rescaling (QuantizedRescal (None, 28, 28, 1) 0
ing)
conv2d (QuantizedConv2D) (None, 13, 13, 32) 320
re_lu (QuantizedReLU) (None, 13, 13, 32) 64
depthwise_conv2d (Quantize (None, 7, 7, 32) 384
dDepthwiseConv2D)
conv2d_1 (QuantizedConv2D) (None, 7, 7, 64) 2112
re_lu_1 (QuantizedReLU) (None, 7, 7, 64) 128
flatten (QuantizedFlatten) (None, 3136) 0
dense (QuantizedDense) (None, 10) 31370
dequantizer (Dequantizer) (None, 10) 0
=================================================================
Total params: 34378 (134.29 KB)
Trainable params: 34122 (133.29 KB)
Non-trainable params: 256 (1.00 KB)
_________________________________________________________________
Note
Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.
Check the quantized model accuracy.
def compile_evaluate(model):
""" Compiles and evaluates the model, then return accuracy score. """
model.compile(metrics=['accuracy'])
return model.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 8-bit quantization: 0.9776999950408936
2.2. Effect of calibration
The previous call to quantize was made with random samples for calibration
(default parameters). While the observed drop in accuracy is minimal, that is
around 1%, it can be worse on more complex models. Therefore, it is advised to
use a set of real samples from the training set for calibration during a call
to quantize.
Note that this remains a calibration step rather than a training step in that
no output labels are required. Furthermore, any relevant data could be used for
calibration. The recommended settings for calibration that are widely used to
obtain the zoo performance are:
1024 samples
a batch size of 100
2 epochs
model_quantized = quantize(model_keras, qparams=qparams,
samples=x_train, num_samples=1024, batch_size=100, epochs=2)
1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step
1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 1ms/step
Check the accuracy for the quantized and calibrated model.
print('Test accuracy after calibration:', compile_evaluate(model_quantized))
Test accuracy after calibration: 0.9782000184059143
Calibrating with real samples on this model recovers the initial float accuracy.
2.3. 4-bit quantization
The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating-point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.
The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.
qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
samples=x_train, num_samples=1024, batch_size=100, epochs=2)
1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step
1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 1ms/step
Check the 4-bit quantized accuracy.
print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 4-bit quantization: 0.9739000201225281
2.4. Model fine-tuning (Quantization Aware Training)
When a model suffers from an accuracy drop after quantization, fine-tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.
Note that since this is a fine-tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.
model_quantized.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=Adam(learning_rate=1e-4),
metrics=['accuracy'])
model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
Epoch 1/5
1/1688 [..............................] - ETA: 1:48:12 - loss: 0.1671 - accuracy: 0.9375
9/1688 [..............................] - ETA: 10s - loss: 0.0534 - accuracy: 0.9792
18/1688 [..............................] - ETA: 10s - loss: 0.0461 - accuracy: 0.9809
27/1688 [..............................] - ETA: 10s - loss: 0.0450 - accuracy: 0.9826
36/1688 [..............................] - ETA: 10s - loss: 0.0429 - accuracy: 0.9826
45/1688 [..............................] - ETA: 9s - loss: 0.0388 - accuracy: 0.9847
53/1688 [..............................] - ETA: 9s - loss: 0.0403 - accuracy: 0.9853
61/1688 [>.............................] - ETA: 9s - loss: 0.0391 - accuracy: 0.9857
70/1688 [>.............................] - ETA: 9s - loss: 0.0388 - accuracy: 0.9857
79/1688 [>.............................] - ETA: 9s - loss: 0.0391 - accuracy: 0.9846
88/1688 [>.............................] - ETA: 9s - loss: 0.0378 - accuracy: 0.9851
96/1688 [>.............................] - ETA: 9s - loss: 0.0363 - accuracy: 0.9860
104/1688 [>.............................] - ETA: 9s - loss: 0.0354 - accuracy: 0.9868
113/1688 [=>............................] - ETA: 9s - loss: 0.0340 - accuracy: 0.9873
121/1688 [=>............................] - ETA: 9s - loss: 0.0332 - accuracy: 0.9876
130/1688 [=>............................] - ETA: 9s - loss: 0.0322 - accuracy: 0.9882
139/1688 [=>............................] - ETA: 9s - loss: 0.0326 - accuracy: 0.9885
148/1688 [=>............................] - ETA: 9s - loss: 0.0336 - accuracy: 0.9884
156/1688 [=>............................] - ETA: 9s - loss: 0.0329 - accuracy: 0.9886
165/1688 [=>............................] - ETA: 9s - loss: 0.0319 - accuracy: 0.9892
174/1688 [==>...........................] - ETA: 9s - loss: 0.0320 - accuracy: 0.9892
183/1688 [==>...........................] - ETA: 9s - loss: 0.0315 - accuracy: 0.9891
192/1688 [==>...........................] - ETA: 9s - loss: 0.0308 - accuracy: 0.9894
200/1688 [==>...........................] - ETA: 9s - loss: 0.0303 - accuracy: 0.9895
209/1688 [==>...........................] - ETA: 9s - loss: 0.0298 - accuracy: 0.9900
218/1688 [==>...........................] - ETA: 9s - loss: 0.0298 - accuracy: 0.9898
227/1688 [===>..........................] - ETA: 8s - loss: 0.0296 - accuracy: 0.9900
236/1688 [===>..........................] - ETA: 8s - loss: 0.0298 - accuracy: 0.9897
245/1688 [===>..........................] - ETA: 8s - loss: 0.0291 - accuracy: 0.9901
254/1688 [===>..........................] - ETA: 8s - loss: 0.0286 - accuracy: 0.9903
263/1688 [===>..........................] - ETA: 8s - loss: 0.0288 - accuracy: 0.9901
271/1688 [===>..........................] - ETA: 8s - loss: 0.0285 - accuracy: 0.9903
279/1688 [===>..........................] - ETA: 8s - loss: 0.0283 - accuracy: 0.9904
288/1688 [====>.........................] - ETA: 8s - loss: 0.0284 - accuracy: 0.9903
297/1688 [====>.........................] - ETA: 8s - loss: 0.0284 - accuracy: 0.9903
306/1688 [====>.........................] - ETA: 8s - loss: 0.0288 - accuracy: 0.9902
315/1688 [====>.........................] - ETA: 8s - loss: 0.0283 - accuracy: 0.9905
324/1688 [====>.........................] - ETA: 8s - loss: 0.0280 - accuracy: 0.9905
333/1688 [====>.........................] - ETA: 8s - loss: 0.0274 - accuracy: 0.9908
342/1688 [=====>........................] - ETA: 8s - loss: 0.0269 - accuracy: 0.9910
350/1688 [=====>........................] - ETA: 8s - loss: 0.0272 - accuracy: 0.9909
359/1688 [=====>........................] - ETA: 8s - loss: 0.0272 - accuracy: 0.9909
368/1688 [=====>........................] - ETA: 8s - loss: 0.0272 - accuracy: 0.9909
376/1688 [=====>........................] - ETA: 8s - loss: 0.0268 - accuracy: 0.9911
385/1688 [=====>........................] - ETA: 7s - loss: 0.0267 - accuracy: 0.9912
394/1688 [======>.......................] - ETA: 7s - loss: 0.0264 - accuracy: 0.9914
403/1688 [======>.......................] - ETA: 7s - loss: 0.0264 - accuracy: 0.9914
412/1688 [======>.......................] - ETA: 7s - loss: 0.0262 - accuracy: 0.9916
421/1688 [======>.......................] - ETA: 7s - loss: 0.0259 - accuracy: 0.9918
429/1688 [======>.......................] - ETA: 7s - loss: 0.0257 - accuracy: 0.9918
437/1688 [======>.......................] - ETA: 7s - loss: 0.0254 - accuracy: 0.9920
446/1688 [======>.......................] - ETA: 7s - loss: 0.0252 - accuracy: 0.9921
455/1688 [=======>......................] - ETA: 7s - loss: 0.0250 - accuracy: 0.9921
463/1688 [=======>......................] - ETA: 7s - loss: 0.0251 - accuracy: 0.9920
472/1688 [=======>......................] - ETA: 7s - loss: 0.0249 - accuracy: 0.9920
481/1688 [=======>......................] - ETA: 7s - loss: 0.0248 - accuracy: 0.9920
490/1688 [=======>......................] - ETA: 7s - loss: 0.0247 - accuracy: 0.9921
499/1688 [=======>......................] - ETA: 7s - loss: 0.0246 - accuracy: 0.9922
508/1688 [========>.....................] - ETA: 7s - loss: 0.0249 - accuracy: 0.9920
517/1688 [========>.....................] - ETA: 7s - loss: 0.0248 - accuracy: 0.9920
525/1688 [========>.....................] - ETA: 7s - loss: 0.0248 - accuracy: 0.9921
534/1688 [========>.....................] - ETA: 7s - loss: 0.0246 - accuracy: 0.9921
543/1688 [========>.....................] - ETA: 7s - loss: 0.0248 - accuracy: 0.9919
552/1688 [========>.....................] - ETA: 6s - loss: 0.0245 - accuracy: 0.9921
561/1688 [========>.....................] - ETA: 6s - loss: 0.0244 - accuracy: 0.9921
570/1688 [=========>....................] - ETA: 6s - loss: 0.0246 - accuracy: 0.9920
579/1688 [=========>....................] - ETA: 6s - loss: 0.0249 - accuracy: 0.9919
588/1688 [=========>....................] - ETA: 6s - loss: 0.0249 - accuracy: 0.9919
597/1688 [=========>....................] - ETA: 6s - loss: 0.0249 - accuracy: 0.9918
606/1688 [=========>....................] - ETA: 6s - loss: 0.0248 - accuracy: 0.9920
615/1688 [=========>....................] - ETA: 6s - loss: 0.0247 - accuracy: 0.9920
624/1688 [==========>...................] - ETA: 6s - loss: 0.0249 - accuracy: 0.9919
633/1688 [==========>...................] - ETA: 6s - loss: 0.0251 - accuracy: 0.9918
642/1688 [==========>...................] - ETA: 6s - loss: 0.0251 - accuracy: 0.9919
651/1688 [==========>...................] - ETA: 6s - loss: 0.0250 - accuracy: 0.9919
660/1688 [==========>...................] - ETA: 6s - loss: 0.0250 - accuracy: 0.9920
668/1688 [==========>...................] - ETA: 6s - loss: 0.0249 - accuracy: 0.9921
677/1688 [===========>..................] - ETA: 6s - loss: 0.0247 - accuracy: 0.9922
686/1688 [===========>..................] - ETA: 6s - loss: 0.0248 - accuracy: 0.9921
695/1688 [===========>..................] - ETA: 6s - loss: 0.0247 - accuracy: 0.9922
704/1688 [===========>..................] - ETA: 6s - loss: 0.0247 - accuracy: 0.9922
713/1688 [===========>..................] - ETA: 5s - loss: 0.0246 - accuracy: 0.9923
722/1688 [===========>..................] - ETA: 5s - loss: 0.0244 - accuracy: 0.9923
731/1688 [===========>..................] - ETA: 5s - loss: 0.0243 - accuracy: 0.9924
740/1688 [============>.................] - ETA: 5s - loss: 0.0242 - accuracy: 0.9925
749/1688 [============>.................] - ETA: 5s - loss: 0.0242 - accuracy: 0.9925
758/1688 [============>.................] - ETA: 5s - loss: 0.0244 - accuracy: 0.9925
767/1688 [============>.................] - ETA: 5s - loss: 0.0243 - accuracy: 0.9925
776/1688 [============>.................] - ETA: 5s - loss: 0.0241 - accuracy: 0.9926
785/1688 [============>.................] - ETA: 5s - loss: 0.0240 - accuracy: 0.9927
794/1688 [=============>................] - ETA: 5s - loss: 0.0239 - accuracy: 0.9927
802/1688 [=============>................] - ETA: 5s - loss: 0.0238 - accuracy: 0.9928
811/1688 [=============>................] - ETA: 5s - loss: 0.0240 - accuracy: 0.9926
820/1688 [=============>................] - ETA: 5s - loss: 0.0239 - accuracy: 0.9926
829/1688 [=============>................] - ETA: 5s - loss: 0.0240 - accuracy: 0.9926
838/1688 [=============>................] - ETA: 5s - loss: 0.0239 - accuracy: 0.9927
847/1688 [==============>...............] - ETA: 5s - loss: 0.0238 - accuracy: 0.9927
856/1688 [==============>...............] - ETA: 5s - loss: 0.0238 - accuracy: 0.9927
865/1688 [==============>...............] - ETA: 5s - loss: 0.0239 - accuracy: 0.9927
874/1688 [==============>...............] - ETA: 4s - loss: 0.0238 - accuracy: 0.9927
882/1688 [==============>...............] - ETA: 4s - loss: 0.0237 - accuracy: 0.9927
891/1688 [==============>...............] - ETA: 4s - loss: 0.0236 - accuracy: 0.9928
899/1688 [==============>...............] - ETA: 4s - loss: 0.0237 - accuracy: 0.9928
908/1688 [===============>..............] - ETA: 4s - loss: 0.0236 - accuracy: 0.9928
917/1688 [===============>..............] - ETA: 4s - loss: 0.0236 - accuracy: 0.9929
926/1688 [===============>..............] - ETA: 4s - loss: 0.0234 - accuracy: 0.9929
935/1688 [===============>..............] - ETA: 4s - loss: 0.0238 - accuracy: 0.9927
944/1688 [===============>..............] - ETA: 4s - loss: 0.0237 - accuracy: 0.9928
952/1688 [===============>..............] - ETA: 4s - loss: 0.0236 - accuracy: 0.9928
961/1688 [================>.............] - ETA: 4s - loss: 0.0235 - accuracy: 0.9929
970/1688 [================>.............] - ETA: 4s - loss: 0.0235 - accuracy: 0.9929
979/1688 [================>.............] - ETA: 4s - loss: 0.0236 - accuracy: 0.9928
988/1688 [================>.............] - ETA: 4s - loss: 0.0235 - accuracy: 0.9929
997/1688 [================>.............] - ETA: 4s - loss: 0.0234 - accuracy: 0.9929
1006/1688 [================>.............] - ETA: 4s - loss: 0.0233 - accuracy: 0.9929
1015/1688 [=================>............] - ETA: 4s - loss: 0.0233 - accuracy: 0.9929
1024/1688 [=================>............] - ETA: 4s - loss: 0.0233 - accuracy: 0.9930
1033/1688 [=================>............] - ETA: 3s - loss: 0.0232 - accuracy: 0.9930
1042/1688 [=================>............] - ETA: 3s - loss: 0.0234 - accuracy: 0.9929
1051/1688 [=================>............] - ETA: 3s - loss: 0.0234 - accuracy: 0.9929
1060/1688 [=================>............] - ETA: 3s - loss: 0.0233 - accuracy: 0.9930
1068/1688 [=================>............] - ETA: 3s - loss: 0.0233 - accuracy: 0.9930
1077/1688 [==================>...........] - ETA: 3s - loss: 0.0232 - accuracy: 0.9930
1086/1688 [==================>...........] - ETA: 3s - loss: 0.0232 - accuracy: 0.9930
1095/1688 [==================>...........] - ETA: 3s - loss: 0.0230 - accuracy: 0.9930
1104/1688 [==================>...........] - ETA: 3s - loss: 0.0229 - accuracy: 0.9931
1113/1688 [==================>...........] - ETA: 3s - loss: 0.0229 - accuracy: 0.9931
1121/1688 [==================>...........] - ETA: 3s - loss: 0.0228 - accuracy: 0.9932
1130/1688 [===================>..........] - ETA: 3s - loss: 0.0227 - accuracy: 0.9932
1139/1688 [===================>..........] - ETA: 3s - loss: 0.0227 - accuracy: 0.9932
1147/1688 [===================>..........] - ETA: 3s - loss: 0.0227 - accuracy: 0.9932
1155/1688 [===================>..........] - ETA: 3s - loss: 0.0227 - accuracy: 0.9932
1164/1688 [===================>..........] - ETA: 3s - loss: 0.0227 - accuracy: 0.9932
1173/1688 [===================>..........] - ETA: 3s - loss: 0.0227 - accuracy: 0.9932
1182/1688 [====================>.........] - ETA: 3s - loss: 0.0227 - accuracy: 0.9932
1191/1688 [====================>.........] - ETA: 3s - loss: 0.0228 - accuracy: 0.9931
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0229 - accuracy: 0.9931
1208/1688 [====================>.........] - ETA: 2s - loss: 0.0228 - accuracy: 0.9931
1217/1688 [====================>.........] - ETA: 2s - loss: 0.0227 - accuracy: 0.9932
1226/1688 [====================>.........] - ETA: 2s - loss: 0.0226 - accuracy: 0.9932
1235/1688 [====================>.........] - ETA: 2s - loss: 0.0226 - accuracy: 0.9932
1244/1688 [=====================>........] - ETA: 2s - loss: 0.0226 - accuracy: 0.9932
1253/1688 [=====================>........] - ETA: 2s - loss: 0.0225 - accuracy: 0.9932
1262/1688 [=====================>........] - ETA: 2s - loss: 0.0226 - accuracy: 0.9932
1271/1688 [=====================>........] - ETA: 2s - loss: 0.0225 - accuracy: 0.9933
1280/1688 [=====================>........] - ETA: 2s - loss: 0.0224 - accuracy: 0.9933
1288/1688 [=====================>........] - ETA: 2s - loss: 0.0225 - accuracy: 0.9933
1297/1688 [======================>.......] - ETA: 2s - loss: 0.0224 - accuracy: 0.9933
1306/1688 [======================>.......] - ETA: 2s - loss: 0.0224 - accuracy: 0.9933
1315/1688 [======================>.......] - ETA: 2s - loss: 0.0223 - accuracy: 0.9934
1324/1688 [======================>.......] - ETA: 2s - loss: 0.0224 - accuracy: 0.9933
1333/1688 [======================>.......] - ETA: 2s - loss: 0.0224 - accuracy: 0.9933
1342/1688 [======================>.......] - ETA: 2s - loss: 0.0224 - accuracy: 0.9933
1351/1688 [=======================>......] - ETA: 2s - loss: 0.0224 - accuracy: 0.9933
1359/1688 [=======================>......] - ETA: 2s - loss: 0.0224 - accuracy: 0.9933
1368/1688 [=======================>......] - ETA: 1s - loss: 0.0224 - accuracy: 0.9933
1377/1688 [=======================>......] - ETA: 1s - loss: 0.0226 - accuracy: 0.9932
1386/1688 [=======================>......] - ETA: 1s - loss: 0.0226 - accuracy: 0.9932
1395/1688 [=======================>......] - ETA: 1s - loss: 0.0225 - accuracy: 0.9933
1404/1688 [=======================>......] - ETA: 1s - loss: 0.0225 - accuracy: 0.9933
1413/1688 [========================>.....] - ETA: 1s - loss: 0.0224 - accuracy: 0.9933
1422/1688 [========================>.....] - ETA: 1s - loss: 0.0224 - accuracy: 0.9933
1431/1688 [========================>.....] - ETA: 1s - loss: 0.0224 - accuracy: 0.9933
1440/1688 [========================>.....] - ETA: 1s - loss: 0.0223 - accuracy: 0.9933
1449/1688 [========================>.....] - ETA: 1s - loss: 0.0223 - accuracy: 0.9934
1458/1688 [========================>.....] - ETA: 1s - loss: 0.0223 - accuracy: 0.9934
1467/1688 [=========================>....] - ETA: 1s - loss: 0.0222 - accuracy: 0.9934
1476/1688 [=========================>....] - ETA: 1s - loss: 0.0223 - accuracy: 0.9934
1485/1688 [=========================>....] - ETA: 1s - loss: 0.0222 - accuracy: 0.9935
1494/1688 [=========================>....] - ETA: 1s - loss: 0.0223 - accuracy: 0.9935
1503/1688 [=========================>....] - ETA: 1s - loss: 0.0223 - accuracy: 0.9935
1512/1688 [=========================>....] - ETA: 1s - loss: 0.0222 - accuracy: 0.9935
1521/1688 [==========================>...] - ETA: 1s - loss: 0.0222 - accuracy: 0.9935
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0222 - accuracy: 0.9935
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0222 - accuracy: 0.9935
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0221 - accuracy: 0.9936
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0221 - accuracy: 0.9936
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0220 - accuracy: 0.9936
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0220 - accuracy: 0.9936
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0219 - accuracy: 0.9936
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9937
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9937
1611/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9937
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0218 - accuracy: 0.9937
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9937
1638/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9937
1646/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9938
1654/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9937
1663/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9937
1672/1688 [============================>.] - ETA: 0s - loss: 0.0215 - accuracy: 0.9938
1680/1688 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9937
1688/1688 [==============================] - 16s 7ms/step - loss: 0.0216 - accuracy: 0.9937 - val_loss: 0.0528 - val_accuracy: 0.9855
Epoch 2/5
1/1688 [..............................] - ETA: 11s - loss: 0.0114 - accuracy: 1.0000
10/1688 [..............................] - ETA: 10s - loss: 0.0083 - accuracy: 0.9969
18/1688 [..............................] - ETA: 10s - loss: 0.0112 - accuracy: 0.9965
27/1688 [..............................] - ETA: 10s - loss: 0.0144 - accuracy: 0.9965
36/1688 [..............................] - ETA: 10s - loss: 0.0125 - accuracy: 0.9974
45/1688 [..............................] - ETA: 10s - loss: 0.0138 - accuracy: 0.9965
53/1688 [..............................] - ETA: 10s - loss: 0.0124 - accuracy: 0.9971
61/1688 [>.............................] - ETA: 10s - loss: 0.0117 - accuracy: 0.9974
69/1688 [>.............................] - ETA: 10s - loss: 0.0119 - accuracy: 0.9973
78/1688 [>.............................] - ETA: 9s - loss: 0.0126 - accuracy: 0.9968
87/1688 [>.............................] - ETA: 9s - loss: 0.0140 - accuracy: 0.9960
96/1688 [>.............................] - ETA: 9s - loss: 0.0134 - accuracy: 0.9964
105/1688 [>.............................] - ETA: 9s - loss: 0.0139 - accuracy: 0.9961
114/1688 [=>............................] - ETA: 9s - loss: 0.0144 - accuracy: 0.9962
123/1688 [=>............................] - ETA: 9s - loss: 0.0150 - accuracy: 0.9954
131/1688 [=>............................] - ETA: 9s - loss: 0.0151 - accuracy: 0.9955
140/1688 [=>............................] - ETA: 9s - loss: 0.0155 - accuracy: 0.9955
149/1688 [=>............................] - ETA: 9s - loss: 0.0160 - accuracy: 0.9952
158/1688 [=>............................] - ETA: 9s - loss: 0.0162 - accuracy: 0.9951
167/1688 [=>............................] - ETA: 9s - loss: 0.0158 - accuracy: 0.9953
176/1688 [==>...........................] - ETA: 9s - loss: 0.0156 - accuracy: 0.9954
185/1688 [==>...........................] - ETA: 9s - loss: 0.0159 - accuracy: 0.9954
194/1688 [==>...........................] - ETA: 9s - loss: 0.0161 - accuracy: 0.9953
202/1688 [==>...........................] - ETA: 9s - loss: 0.0162 - accuracy: 0.9950
211/1688 [==>...........................] - ETA: 9s - loss: 0.0163 - accuracy: 0.9951
220/1688 [==>...........................] - ETA: 8s - loss: 0.0160 - accuracy: 0.9953
228/1688 [===>..........................] - ETA: 8s - loss: 0.0163 - accuracy: 0.9952
237/1688 [===>..........................] - ETA: 8s - loss: 0.0161 - accuracy: 0.9954
246/1688 [===>..........................] - ETA: 8s - loss: 0.0158 - accuracy: 0.9956
255/1688 [===>..........................] - ETA: 8s - loss: 0.0157 - accuracy: 0.9956
264/1688 [===>..........................] - ETA: 8s - loss: 0.0156 - accuracy: 0.9957
273/1688 [===>..........................] - ETA: 8s - loss: 0.0161 - accuracy: 0.9957
282/1688 [====>.........................] - ETA: 8s - loss: 0.0160 - accuracy: 0.9957
290/1688 [====>.........................] - ETA: 8s - loss: 0.0160 - accuracy: 0.9957
299/1688 [====>.........................] - ETA: 8s - loss: 0.0162 - accuracy: 0.9955
308/1688 [====>.........................] - ETA: 8s - loss: 0.0161 - accuracy: 0.9955
317/1688 [====>.........................] - ETA: 8s - loss: 0.0159 - accuracy: 0.9956
326/1688 [====>.........................] - ETA: 8s - loss: 0.0159 - accuracy: 0.9955
334/1688 [====>.........................] - ETA: 8s - loss: 0.0160 - accuracy: 0.9955
343/1688 [=====>........................] - ETA: 8s - loss: 0.0160 - accuracy: 0.9954
352/1688 [=====>........................] - ETA: 8s - loss: 0.0159 - accuracy: 0.9955
361/1688 [=====>........................] - ETA: 8s - loss: 0.0157 - accuracy: 0.9956
369/1688 [=====>........................] - ETA: 8s - loss: 0.0155 - accuracy: 0.9957
378/1688 [=====>........................] - ETA: 8s - loss: 0.0155 - accuracy: 0.9957
387/1688 [=====>........................] - ETA: 7s - loss: 0.0155 - accuracy: 0.9957
396/1688 [======>.......................] - ETA: 7s - loss: 0.0160 - accuracy: 0.9957
405/1688 [======>.......................] - ETA: 7s - loss: 0.0162 - accuracy: 0.9955
414/1688 [======>.......................] - ETA: 7s - loss: 0.0165 - accuracy: 0.9952
423/1688 [======>.......................] - ETA: 7s - loss: 0.0165 - accuracy: 0.9952
432/1688 [======>.......................] - ETA: 7s - loss: 0.0164 - accuracy: 0.9953
441/1688 [======>.......................] - ETA: 7s - loss: 0.0172 - accuracy: 0.9951
450/1688 [======>.......................] - ETA: 7s - loss: 0.0170 - accuracy: 0.9952
459/1688 [=======>......................] - ETA: 7s - loss: 0.0172 - accuracy: 0.9951
468/1688 [=======>......................] - ETA: 7s - loss: 0.0172 - accuracy: 0.9951
477/1688 [=======>......................] - ETA: 7s - loss: 0.0176 - accuracy: 0.9950
485/1688 [=======>......................] - ETA: 7s - loss: 0.0176 - accuracy: 0.9949
494/1688 [=======>......................] - ETA: 7s - loss: 0.0176 - accuracy: 0.9949
502/1688 [=======>......................] - ETA: 7s - loss: 0.0175 - accuracy: 0.9949
511/1688 [========>.....................] - ETA: 7s - loss: 0.0176 - accuracy: 0.9949
519/1688 [========>.....................] - ETA: 7s - loss: 0.0175 - accuracy: 0.9949
528/1688 [========>.....................] - ETA: 7s - loss: 0.0175 - accuracy: 0.9949
537/1688 [========>.....................] - ETA: 7s - loss: 0.0173 - accuracy: 0.9949
546/1688 [========>.....................] - ETA: 6s - loss: 0.0175 - accuracy: 0.9949
555/1688 [========>.....................] - ETA: 6s - loss: 0.0175 - accuracy: 0.9949
563/1688 [=========>....................] - ETA: 6s - loss: 0.0176 - accuracy: 0.9948
572/1688 [=========>....................] - ETA: 6s - loss: 0.0174 - accuracy: 0.9949
581/1688 [=========>....................] - ETA: 6s - loss: 0.0175 - accuracy: 0.9948
590/1688 [=========>....................] - ETA: 6s - loss: 0.0174 - accuracy: 0.9949
599/1688 [=========>....................] - ETA: 6s - loss: 0.0173 - accuracy: 0.9949
608/1688 [=========>....................] - ETA: 6s - loss: 0.0173 - accuracy: 0.9949
616/1688 [=========>....................] - ETA: 6s - loss: 0.0173 - accuracy: 0.9949
625/1688 [==========>...................] - ETA: 6s - loss: 0.0171 - accuracy: 0.9950
633/1688 [==========>...................] - ETA: 6s - loss: 0.0172 - accuracy: 0.9950
642/1688 [==========>...................] - ETA: 6s - loss: 0.0171 - accuracy: 0.9950
651/1688 [==========>...................] - ETA: 6s - loss: 0.0171 - accuracy: 0.9950
660/1688 [==========>...................] - ETA: 6s - loss: 0.0170 - accuracy: 0.9950
669/1688 [==========>...................] - ETA: 6s - loss: 0.0169 - accuracy: 0.9950
678/1688 [===========>..................] - ETA: 6s - loss: 0.0168 - accuracy: 0.9950
686/1688 [===========>..................] - ETA: 6s - loss: 0.0169 - accuracy: 0.9949
695/1688 [===========>..................] - ETA: 6s - loss: 0.0169 - accuracy: 0.9950
704/1688 [===========>..................] - ETA: 6s - loss: 0.0170 - accuracy: 0.9949
713/1688 [===========>..................] - ETA: 5s - loss: 0.0168 - accuracy: 0.9950
722/1688 [===========>..................] - ETA: 5s - loss: 0.0168 - accuracy: 0.9951
731/1688 [===========>..................] - ETA: 5s - loss: 0.0167 - accuracy: 0.9951
740/1688 [============>.................] - ETA: 5s - loss: 0.0167 - accuracy: 0.9951
749/1688 [============>.................] - ETA: 5s - loss: 0.0167 - accuracy: 0.9952
758/1688 [============>.................] - ETA: 5s - loss: 0.0166 - accuracy: 0.9952
767/1688 [============>.................] - ETA: 5s - loss: 0.0167 - accuracy: 0.9952
776/1688 [============>.................] - ETA: 5s - loss: 0.0166 - accuracy: 0.9952
785/1688 [============>.................] - ETA: 5s - loss: 0.0167 - accuracy: 0.9951
794/1688 [=============>................] - ETA: 5s - loss: 0.0166 - accuracy: 0.9951
802/1688 [=============>................] - ETA: 5s - loss: 0.0167 - accuracy: 0.9951
811/1688 [=============>................] - ETA: 5s - loss: 0.0166 - accuracy: 0.9951
820/1688 [=============>................] - ETA: 5s - loss: 0.0165 - accuracy: 0.9952
829/1688 [=============>................] - ETA: 5s - loss: 0.0164 - accuracy: 0.9952
838/1688 [=============>................] - ETA: 5s - loss: 0.0164 - accuracy: 0.9952
847/1688 [==============>...............] - ETA: 5s - loss: 0.0164 - accuracy: 0.9952
856/1688 [==============>...............] - ETA: 5s - loss: 0.0164 - accuracy: 0.9953
865/1688 [==============>...............] - ETA: 5s - loss: 0.0164 - accuracy: 0.9952
874/1688 [==============>...............] - ETA: 4s - loss: 0.0165 - accuracy: 0.9952
882/1688 [==============>...............] - ETA: 4s - loss: 0.0164 - accuracy: 0.9953
891/1688 [==============>...............] - ETA: 4s - loss: 0.0164 - accuracy: 0.9953
899/1688 [==============>...............] - ETA: 4s - loss: 0.0164 - accuracy: 0.9953
908/1688 [===============>..............] - ETA: 4s - loss: 0.0164 - accuracy: 0.9953
917/1688 [===============>..............] - ETA: 4s - loss: 0.0164 - accuracy: 0.9953
926/1688 [===============>..............] - ETA: 4s - loss: 0.0163 - accuracy: 0.9953
935/1688 [===============>..............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9954
944/1688 [===============>..............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9953
953/1688 [===============>..............] - ETA: 4s - loss: 0.0163 - accuracy: 0.9953
962/1688 [================>.............] - ETA: 4s - loss: 0.0163 - accuracy: 0.9953
971/1688 [================>.............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9953
980/1688 [================>.............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9954
989/1688 [================>.............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9954
998/1688 [================>.............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9954
1007/1688 [================>.............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9954
1016/1688 [=================>............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9954
1025/1688 [=================>............] - ETA: 4s - loss: 0.0162 - accuracy: 0.9954
1034/1688 [=================>............] - ETA: 3s - loss: 0.0164 - accuracy: 0.9954
1042/1688 [=================>............] - ETA: 3s - loss: 0.0165 - accuracy: 0.9954
1051/1688 [=================>............] - ETA: 3s - loss: 0.0166 - accuracy: 0.9952
1060/1688 [=================>............] - ETA: 3s - loss: 0.0166 - accuracy: 0.9953
1069/1688 [=================>............] - ETA: 3s - loss: 0.0165 - accuracy: 0.9953
1078/1688 [==================>...........] - ETA: 3s - loss: 0.0165 - accuracy: 0.9953
1087/1688 [==================>...........] - ETA: 3s - loss: 0.0164 - accuracy: 0.9953
1096/1688 [==================>...........] - ETA: 3s - loss: 0.0164 - accuracy: 0.9954
1105/1688 [==================>...........] - ETA: 3s - loss: 0.0163 - accuracy: 0.9954
1114/1688 [==================>...........] - ETA: 3s - loss: 0.0163 - accuracy: 0.9955
1123/1688 [==================>...........] - ETA: 3s - loss: 0.0162 - accuracy: 0.9955
1132/1688 [===================>..........] - ETA: 3s - loss: 0.0163 - accuracy: 0.9955
1141/1688 [===================>..........] - ETA: 3s - loss: 0.0163 - accuracy: 0.9954
1150/1688 [===================>..........] - ETA: 3s - loss: 0.0163 - accuracy: 0.9954
1158/1688 [===================>..........] - ETA: 3s - loss: 0.0163 - accuracy: 0.9954
1167/1688 [===================>..........] - ETA: 3s - loss: 0.0163 - accuracy: 0.9954
1176/1688 [===================>..........] - ETA: 3s - loss: 0.0163 - accuracy: 0.9955
1184/1688 [====================>.........] - ETA: 3s - loss: 0.0162 - accuracy: 0.9955
1193/1688 [====================>.........] - ETA: 3s - loss: 0.0162 - accuracy: 0.9955
1202/1688 [====================>.........] - ETA: 2s - loss: 0.0161 - accuracy: 0.9955
1211/1688 [====================>.........] - ETA: 2s - loss: 0.0161 - accuracy: 0.9955
1220/1688 [====================>.........] - ETA: 2s - loss: 0.0161 - accuracy: 0.9955
1229/1688 [====================>.........] - ETA: 2s - loss: 0.0160 - accuracy: 0.9956
1238/1688 [=====================>........] - ETA: 2s - loss: 0.0160 - accuracy: 0.9956
1246/1688 [=====================>........] - ETA: 2s - loss: 0.0160 - accuracy: 0.9956
1254/1688 [=====================>........] - ETA: 2s - loss: 0.0160 - accuracy: 0.9956
1263/1688 [=====================>........] - ETA: 2s - loss: 0.0160 - accuracy: 0.9956
1272/1688 [=====================>........] - ETA: 2s - loss: 0.0160 - accuracy: 0.9955
1281/1688 [=====================>........] - ETA: 2s - loss: 0.0160 - accuracy: 0.9956
1289/1688 [=====================>........] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1297/1688 [======================>.......] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1305/1688 [======================>.......] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1314/1688 [======================>.......] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1322/1688 [======================>.......] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1331/1688 [======================>.......] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1339/1688 [======================>.......] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1348/1688 [======================>.......] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1357/1688 [=======================>......] - ETA: 2s - loss: 0.0159 - accuracy: 0.9956
1366/1688 [=======================>......] - ETA: 1s - loss: 0.0160 - accuracy: 0.9956
1375/1688 [=======================>......] - ETA: 1s - loss: 0.0160 - accuracy: 0.9956
1384/1688 [=======================>......] - ETA: 1s - loss: 0.0160 - accuracy: 0.9956
1393/1688 [=======================>......] - ETA: 1s - loss: 0.0161 - accuracy: 0.9955
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0161 - accuracy: 0.9955
1409/1688 [========================>.....] - ETA: 1s - loss: 0.0160 - accuracy: 0.9955
1418/1688 [========================>.....] - ETA: 1s - loss: 0.0163 - accuracy: 0.9955
1427/1688 [========================>.....] - ETA: 1s - loss: 0.0163 - accuracy: 0.9955
1436/1688 [========================>.....] - ETA: 1s - loss: 0.0163 - accuracy: 0.9955
1444/1688 [========================>.....] - ETA: 1s - loss: 0.0162 - accuracy: 0.9955
1453/1688 [========================>.....] - ETA: 1s - loss: 0.0163 - accuracy: 0.9954
1462/1688 [========================>.....] - ETA: 1s - loss: 0.0163 - accuracy: 0.9954
1471/1688 [=========================>....] - ETA: 1s - loss: 0.0163 - accuracy: 0.9955
1480/1688 [=========================>....] - ETA: 1s - loss: 0.0162 - accuracy: 0.9955
1489/1688 [=========================>....] - ETA: 1s - loss: 0.0162 - accuracy: 0.9955
1498/1688 [=========================>....] - ETA: 1s - loss: 0.0162 - accuracy: 0.9955
1507/1688 [=========================>....] - ETA: 1s - loss: 0.0161 - accuracy: 0.9955
1516/1688 [=========================>....] - ETA: 1s - loss: 0.0161 - accuracy: 0.9955
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9955
1542/1688 [==========================>...] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1551/1688 [==========================>...] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9955
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9955
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0163 - accuracy: 0.9955
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0163 - accuracy: 0.9955
1640/1688 [============================>.] - ETA: 0s - loss: 0.0163 - accuracy: 0.9955
1649/1688 [============================>.] - ETA: 0s - loss: 0.0162 - accuracy: 0.9955
1657/1688 [============================>.] - ETA: 0s - loss: 0.0163 - accuracy: 0.9955
1666/1688 [============================>.] - ETA: 0s - loss: 0.0163 - accuracy: 0.9955
1675/1688 [============================>.] - ETA: 0s - loss: 0.0165 - accuracy: 0.9954
1683/1688 [============================>.] - ETA: 0s - loss: 0.0165 - accuracy: 0.9954
1688/1688 [==============================] - 11s 7ms/step - loss: 0.0165 - accuracy: 0.9954 - val_loss: 0.0530 - val_accuracy: 0.9862
Epoch 3/5
1/1688 [..............................] - ETA: 11s - loss: 3.3348e-04 - accuracy: 1.0000
9/1688 [..............................] - ETA: 10s - loss: 0.0211 - accuracy: 0.9931
18/1688 [..............................] - ETA: 10s - loss: 0.0213 - accuracy: 0.9931
26/1688 [..............................] - ETA: 10s - loss: 0.0185 - accuracy: 0.9940
35/1688 [..............................] - ETA: 10s - loss: 0.0191 - accuracy: 0.9937
44/1688 [..............................] - ETA: 10s - loss: 0.0188 - accuracy: 0.9936
53/1688 [..............................] - ETA: 10s - loss: 0.0165 - accuracy: 0.9947
61/1688 [>.............................] - ETA: 10s - loss: 0.0170 - accuracy: 0.9944
70/1688 [>.............................] - ETA: 10s - loss: 0.0158 - accuracy: 0.9951
79/1688 [>.............................] - ETA: 9s - loss: 0.0157 - accuracy: 0.9949
87/1688 [>.............................] - ETA: 9s - loss: 0.0150 - accuracy: 0.9953
96/1688 [>.............................] - ETA: 9s - loss: 0.0146 - accuracy: 0.9958
105/1688 [>.............................] - ETA: 9s - loss: 0.0149 - accuracy: 0.9955
113/1688 [=>............................] - ETA: 9s - loss: 0.0152 - accuracy: 0.9956
122/1688 [=>............................] - ETA: 9s - loss: 0.0154 - accuracy: 0.9954
131/1688 [=>............................] - ETA: 9s - loss: 0.0148 - accuracy: 0.9957
140/1688 [=>............................] - ETA: 9s - loss: 0.0143 - accuracy: 0.9960
149/1688 [=>............................] - ETA: 9s - loss: 0.0141 - accuracy: 0.9962
158/1688 [=>............................] - ETA: 9s - loss: 0.0138 - accuracy: 0.9962
167/1688 [=>............................] - ETA: 9s - loss: 0.0134 - accuracy: 0.9964
176/1688 [==>...........................] - ETA: 9s - loss: 0.0137 - accuracy: 0.9964
185/1688 [==>...........................] - ETA: 9s - loss: 0.0142 - accuracy: 0.9959
194/1688 [==>...........................] - ETA: 9s - loss: 0.0142 - accuracy: 0.9961
203/1688 [==>...........................] - ETA: 9s - loss: 0.0142 - accuracy: 0.9962
212/1688 [==>...........................] - ETA: 9s - loss: 0.0138 - accuracy: 0.9963
221/1688 [==>...........................] - ETA: 9s - loss: 0.0138 - accuracy: 0.9965
230/1688 [===>..........................] - ETA: 8s - loss: 0.0136 - accuracy: 0.9966
239/1688 [===>..........................] - ETA: 8s - loss: 0.0140 - accuracy: 0.9963
248/1688 [===>..........................] - ETA: 8s - loss: 0.0142 - accuracy: 0.9962
257/1688 [===>..........................] - ETA: 8s - loss: 0.0144 - accuracy: 0.9962
266/1688 [===>..........................] - ETA: 8s - loss: 0.0148 - accuracy: 0.9961
275/1688 [===>..........................] - ETA: 8s - loss: 0.0147 - accuracy: 0.9962
283/1688 [====>.........................] - ETA: 8s - loss: 0.0147 - accuracy: 0.9962
292/1688 [====>.........................] - ETA: 8s - loss: 0.0144 - accuracy: 0.9964
301/1688 [====>.........................] - ETA: 8s - loss: 0.0146 - accuracy: 0.9962
310/1688 [====>.........................] - ETA: 8s - loss: 0.0146 - accuracy: 0.9962
319/1688 [====>.........................] - ETA: 8s - loss: 0.0147 - accuracy: 0.9962
328/1688 [====>.........................] - ETA: 8s - loss: 0.0145 - accuracy: 0.9963
337/1688 [====>.........................] - ETA: 8s - loss: 0.0142 - accuracy: 0.9964
346/1688 [=====>........................] - ETA: 8s - loss: 0.0141 - accuracy: 0.9965
355/1688 [=====>........................] - ETA: 8s - loss: 0.0141 - accuracy: 0.9964
364/1688 [=====>........................] - ETA: 8s - loss: 0.0142 - accuracy: 0.9965
373/1688 [=====>........................] - ETA: 8s - loss: 0.0143 - accuracy: 0.9964
382/1688 [=====>........................] - ETA: 7s - loss: 0.0145 - accuracy: 0.9963
390/1688 [=====>........................] - ETA: 7s - loss: 0.0147 - accuracy: 0.9961
399/1688 [======>.......................] - ETA: 7s - loss: 0.0147 - accuracy: 0.9960
408/1688 [======>.......................] - ETA: 7s - loss: 0.0146 - accuracy: 0.9960
417/1688 [======>.......................] - ETA: 7s - loss: 0.0146 - accuracy: 0.9960
425/1688 [======>.......................] - ETA: 7s - loss: 0.0146 - accuracy: 0.9961
434/1688 [======>.......................] - ETA: 7s - loss: 0.0147 - accuracy: 0.9961
443/1688 [======>.......................] - ETA: 7s - loss: 0.0149 - accuracy: 0.9961
452/1688 [=======>......................] - ETA: 7s - loss: 0.0148 - accuracy: 0.9962
461/1688 [=======>......................] - ETA: 7s - loss: 0.0146 - accuracy: 0.9963
469/1688 [=======>......................] - ETA: 7s - loss: 0.0145 - accuracy: 0.9963
478/1688 [=======>......................] - ETA: 7s - loss: 0.0145 - accuracy: 0.9963
487/1688 [=======>......................] - ETA: 7s - loss: 0.0143 - accuracy: 0.9963
496/1688 [=======>......................] - ETA: 7s - loss: 0.0144 - accuracy: 0.9963
505/1688 [=======>......................] - ETA: 7s - loss: 0.0142 - accuracy: 0.9964
514/1688 [========>.....................] - ETA: 7s - loss: 0.0143 - accuracy: 0.9964
522/1688 [========>.....................] - ETA: 7s - loss: 0.0142 - accuracy: 0.9965
531/1688 [========>.....................] - ETA: 7s - loss: 0.0142 - accuracy: 0.9965
540/1688 [========>.....................] - ETA: 7s - loss: 0.0140 - accuracy: 0.9965
549/1688 [========>.....................] - ETA: 6s - loss: 0.0140 - accuracy: 0.9966
558/1688 [========>.....................] - ETA: 6s - loss: 0.0140 - accuracy: 0.9966
567/1688 [=========>....................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9965
576/1688 [=========>....................] - ETA: 6s - loss: 0.0146 - accuracy: 0.9964
585/1688 [=========>....................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9963
594/1688 [=========>....................] - ETA: 6s - loss: 0.0145 - accuracy: 0.9964
603/1688 [=========>....................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9964
612/1688 [=========>....................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9964
621/1688 [==========>...................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9965
630/1688 [==========>...................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9965
639/1688 [==========>...................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9965
647/1688 [==========>...................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9964
656/1688 [==========>...................] - ETA: 6s - loss: 0.0144 - accuracy: 0.9964
665/1688 [==========>...................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9964
674/1688 [==========>...................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9965
683/1688 [===========>..................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9965
692/1688 [===========>..................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9965
701/1688 [===========>..................] - ETA: 6s - loss: 0.0143 - accuracy: 0.9965
710/1688 [===========>..................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9965
719/1688 [===========>..................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9965
727/1688 [===========>..................] - ETA: 5s - loss: 0.0144 - accuracy: 0.9964
736/1688 [============>.................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9965
744/1688 [============>.................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9964
753/1688 [============>.................] - ETA: 5s - loss: 0.0144 - accuracy: 0.9963
762/1688 [============>.................] - ETA: 5s - loss: 0.0144 - accuracy: 0.9964
771/1688 [============>.................] - ETA: 5s - loss: 0.0144 - accuracy: 0.9964
779/1688 [============>.................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9964
788/1688 [=============>................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9964
797/1688 [=============>................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9964
806/1688 [=============>................] - ETA: 5s - loss: 0.0144 - accuracy: 0.9964
814/1688 [=============>................] - ETA: 5s - loss: 0.0143 - accuracy: 0.9965
823/1688 [=============>................] - ETA: 5s - loss: 0.0144 - accuracy: 0.9964
832/1688 [=============>................] - ETA: 5s - loss: 0.0146 - accuracy: 0.9963
841/1688 [=============>................] - ETA: 5s - loss: 0.0147 - accuracy: 0.9962
850/1688 [==============>...............] - ETA: 5s - loss: 0.0146 - accuracy: 0.9962
858/1688 [==============>...............] - ETA: 5s - loss: 0.0148 - accuracy: 0.9961
867/1688 [==============>...............] - ETA: 5s - loss: 0.0149 - accuracy: 0.9961
876/1688 [==============>...............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9961
885/1688 [==============>...............] - ETA: 4s - loss: 0.0148 - accuracy: 0.9961
894/1688 [==============>...............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9961
903/1688 [===============>..............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9961
912/1688 [===============>..............] - ETA: 4s - loss: 0.0148 - accuracy: 0.9961
921/1688 [===============>..............] - ETA: 4s - loss: 0.0150 - accuracy: 0.9960
930/1688 [===============>..............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9960
939/1688 [===============>..............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9960
948/1688 [===============>..............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9960
957/1688 [================>.............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9960
966/1688 [================>.............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9960
974/1688 [================>.............] - ETA: 4s - loss: 0.0148 - accuracy: 0.9961
982/1688 [================>.............] - ETA: 4s - loss: 0.0148 - accuracy: 0.9961
991/1688 [================>.............] - ETA: 4s - loss: 0.0148 - accuracy: 0.9961
1000/1688 [================>.............] - ETA: 4s - loss: 0.0149 - accuracy: 0.9961
1009/1688 [================>.............] - ETA: 4s - loss: 0.0148 - accuracy: 0.9961
1018/1688 [=================>............] - ETA: 4s - loss: 0.0150 - accuracy: 0.9960
1027/1688 [=================>............] - ETA: 4s - loss: 0.0150 - accuracy: 0.9960
1036/1688 [=================>............] - ETA: 3s - loss: 0.0149 - accuracy: 0.9960
1045/1688 [=================>............] - ETA: 3s - loss: 0.0150 - accuracy: 0.9960
1054/1688 [=================>............] - ETA: 3s - loss: 0.0149 - accuracy: 0.9960
1063/1688 [=================>............] - ETA: 3s - loss: 0.0149 - accuracy: 0.9961
1071/1688 [==================>...........] - ETA: 3s - loss: 0.0148 - accuracy: 0.9961
1080/1688 [==================>...........] - ETA: 3s - loss: 0.0148 - accuracy: 0.9961
1089/1688 [==================>...........] - ETA: 3s - loss: 0.0148 - accuracy: 0.9961
1098/1688 [==================>...........] - ETA: 3s - loss: 0.0148 - accuracy: 0.9961
1107/1688 [==================>...........] - ETA: 3s - loss: 0.0148 - accuracy: 0.9961
1115/1688 [==================>...........] - ETA: 3s - loss: 0.0147 - accuracy: 0.9961
1124/1688 [==================>...........] - ETA: 3s - loss: 0.0147 - accuracy: 0.9961
1133/1688 [===================>..........] - ETA: 3s - loss: 0.0147 - accuracy: 0.9961
1141/1688 [===================>..........] - ETA: 3s - loss: 0.0146 - accuracy: 0.9962
1150/1688 [===================>..........] - ETA: 3s - loss: 0.0146 - accuracy: 0.9961
1159/1688 [===================>..........] - ETA: 3s - loss: 0.0146 - accuracy: 0.9962
1167/1688 [===================>..........] - ETA: 3s - loss: 0.0146 - accuracy: 0.9962
1176/1688 [===================>..........] - ETA: 3s - loss: 0.0146 - accuracy: 0.9962
1185/1688 [====================>.........] - ETA: 3s - loss: 0.0145 - accuracy: 0.9962
1194/1688 [====================>.........] - ETA: 3s - loss: 0.0145 - accuracy: 0.9962
1203/1688 [====================>.........] - ETA: 2s - loss: 0.0144 - accuracy: 0.9962
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0145 - accuracy: 0.9962
1221/1688 [====================>.........] - ETA: 2s - loss: 0.0144 - accuracy: 0.9962
1229/1688 [====================>.........] - ETA: 2s - loss: 0.0144 - accuracy: 0.9962
1238/1688 [=====================>........] - ETA: 2s - loss: 0.0145 - accuracy: 0.9962
1247/1688 [=====================>........] - ETA: 2s - loss: 0.0145 - accuracy: 0.9962
1255/1688 [=====================>........] - ETA: 2s - loss: 0.0144 - accuracy: 0.9962
1263/1688 [=====================>........] - ETA: 2s - loss: 0.0144 - accuracy: 0.9962
1271/1688 [=====================>........] - ETA: 2s - loss: 0.0145 - accuracy: 0.9962
1279/1688 [=====================>........] - ETA: 2s - loss: 0.0144 - accuracy: 0.9962
1288/1688 [=====================>........] - ETA: 2s - loss: 0.0145 - accuracy: 0.9962
1297/1688 [======================>.......] - ETA: 2s - loss: 0.0146 - accuracy: 0.9961
1305/1688 [======================>.......] - ETA: 2s - loss: 0.0147 - accuracy: 0.9961
1313/1688 [======================>.......] - ETA: 2s - loss: 0.0147 - accuracy: 0.9961
1321/1688 [======================>.......] - ETA: 2s - loss: 0.0147 - accuracy: 0.9961
1329/1688 [======================>.......] - ETA: 2s - loss: 0.0147 - accuracy: 0.9961
1338/1688 [======================>.......] - ETA: 2s - loss: 0.0147 - accuracy: 0.9961
1347/1688 [======================>.......] - ETA: 2s - loss: 0.0147 - accuracy: 0.9961
1356/1688 [=======================>......] - ETA: 2s - loss: 0.0146 - accuracy: 0.9962
1365/1688 [=======================>......] - ETA: 1s - loss: 0.0146 - accuracy: 0.9962
1374/1688 [=======================>......] - ETA: 1s - loss: 0.0146 - accuracy: 0.9962
1383/1688 [=======================>......] - ETA: 1s - loss: 0.0147 - accuracy: 0.9961
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0148 - accuracy: 0.9960
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0148 - accuracy: 0.9960
1410/1688 [========================>.....] - ETA: 1s - loss: 0.0148 - accuracy: 0.9960
1419/1688 [========================>.....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9960
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9960
1437/1688 [========================>.....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9959
1446/1688 [========================>.....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9959
1455/1688 [========================>.....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9959
1464/1688 [=========================>....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9959
1473/1688 [=========================>....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9959
1482/1688 [=========================>....] - ETA: 1s - loss: 0.0148 - accuracy: 0.9960
1490/1688 [=========================>....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9959
1499/1688 [=========================>....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9959
1508/1688 [=========================>....] - ETA: 1s - loss: 0.0149 - accuracy: 0.9959
1517/1688 [=========================>....] - ETA: 1s - loss: 0.0148 - accuracy: 0.9959
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0148 - accuracy: 0.9959
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0149 - accuracy: 0.9959
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0150 - accuracy: 0.9959
1552/1688 [==========================>...] - ETA: 0s - loss: 0.0149 - accuracy: 0.9959
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0149 - accuracy: 0.9959
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0150 - accuracy: 0.9959
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0149 - accuracy: 0.9959
1588/1688 [===========================>..] - ETA: 0s - loss: 0.0149 - accuracy: 0.9959
1597/1688 [===========================>..] - ETA: 0s - loss: 0.0148 - accuracy: 0.9959
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0148 - accuracy: 0.9960
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0148 - accuracy: 0.9960
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0148 - accuracy: 0.9959
1632/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9959
1641/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9959
1650/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9959
1659/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9960
1668/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9960
1677/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9959
1686/1688 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9960
1688/1688 [==============================] - 11s 7ms/step - loss: 0.0148 - accuracy: 0.9960 - val_loss: 0.0541 - val_accuracy: 0.9858
Epoch 4/5
1/1688 [..............................] - ETA: 11s - loss: 0.0039 - accuracy: 1.0000
10/1688 [..............................] - ETA: 10s - loss: 0.0068 - accuracy: 1.0000
19/1688 [..............................] - ETA: 10s - loss: 0.0098 - accuracy: 0.9984
28/1688 [..............................] - ETA: 10s - loss: 0.0094 - accuracy: 0.9989
36/1688 [..............................] - ETA: 10s - loss: 0.0101 - accuracy: 0.9991
45/1688 [..............................] - ETA: 10s - loss: 0.0101 - accuracy: 0.9986
53/1688 [..............................] - ETA: 10s - loss: 0.0106 - accuracy: 0.9988
62/1688 [>.............................] - ETA: 10s - loss: 0.0097 - accuracy: 0.9990
71/1688 [>.............................] - ETA: 9s - loss: 0.0098 - accuracy: 0.9991
79/1688 [>.............................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9988
88/1688 [>.............................] - ETA: 9s - loss: 0.0102 - accuracy: 0.9989
96/1688 [>.............................] - ETA: 9s - loss: 0.0101 - accuracy: 0.9990
105/1688 [>.............................] - ETA: 9s - loss: 0.0107 - accuracy: 0.9985
114/1688 [=>............................] - ETA: 9s - loss: 0.0103 - accuracy: 0.9984
123/1688 [=>............................] - ETA: 9s - loss: 0.0113 - accuracy: 0.9977
132/1688 [=>............................] - ETA: 9s - loss: 0.0113 - accuracy: 0.9976
141/1688 [=>............................] - ETA: 9s - loss: 0.0122 - accuracy: 0.9969
150/1688 [=>............................] - ETA: 9s - loss: 0.0120 - accuracy: 0.9969
159/1688 [=>............................] - ETA: 9s - loss: 0.0117 - accuracy: 0.9971
168/1688 [=>............................] - ETA: 9s - loss: 0.0114 - accuracy: 0.9972
177/1688 [==>...........................] - ETA: 9s - loss: 0.0117 - accuracy: 0.9970
186/1688 [==>...........................] - ETA: 9s - loss: 0.0114 - accuracy: 0.9971
195/1688 [==>...........................] - ETA: 9s - loss: 0.0115 - accuracy: 0.9971
204/1688 [==>...........................] - ETA: 9s - loss: 0.0113 - accuracy: 0.9972
213/1688 [==>...........................] - ETA: 9s - loss: 0.0111 - accuracy: 0.9974
221/1688 [==>...........................] - ETA: 9s - loss: 0.0111 - accuracy: 0.9973
230/1688 [===>..........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9973
239/1688 [===>..........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9973
248/1688 [===>..........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9972
257/1688 [===>..........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9972
266/1688 [===>..........................] - ETA: 8s - loss: 0.0111 - accuracy: 0.9972
274/1688 [===>..........................] - ETA: 8s - loss: 0.0111 - accuracy: 0.9973
283/1688 [====>.........................] - ETA: 8s - loss: 0.0110 - accuracy: 0.9972
292/1688 [====>.........................] - ETA: 8s - loss: 0.0111 - accuracy: 0.9972
301/1688 [====>.........................] - ETA: 8s - loss: 0.0113 - accuracy: 0.9972
310/1688 [====>.........................] - ETA: 8s - loss: 0.0113 - accuracy: 0.9973
318/1688 [====>.........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9973
327/1688 [====>.........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9973
336/1688 [====>.........................] - ETA: 8s - loss: 0.0111 - accuracy: 0.9974
344/1688 [=====>........................] - ETA: 8s - loss: 0.0110 - accuracy: 0.9974
353/1688 [=====>........................] - ETA: 8s - loss: 0.0113 - accuracy: 0.9973
362/1688 [=====>........................] - ETA: 8s - loss: 0.0113 - accuracy: 0.9973
371/1688 [=====>........................] - ETA: 8s - loss: 0.0113 - accuracy: 0.9973
380/1688 [=====>........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9972
389/1688 [=====>........................] - ETA: 7s - loss: 0.0114 - accuracy: 0.9973
398/1688 [======>.......................] - ETA: 7s - loss: 0.0116 - accuracy: 0.9973
407/1688 [======>.......................] - ETA: 7s - loss: 0.0116 - accuracy: 0.9973
416/1688 [======>.......................] - ETA: 7s - loss: 0.0116 - accuracy: 0.9973
425/1688 [======>.......................] - ETA: 7s - loss: 0.0116 - accuracy: 0.9973
433/1688 [======>.......................] - ETA: 7s - loss: 0.0115 - accuracy: 0.9973
442/1688 [======>.......................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9970
451/1688 [=======>......................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9970
460/1688 [=======>......................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9970
469/1688 [=======>......................] - ETA: 7s - loss: 0.0120 - accuracy: 0.9971
478/1688 [=======>......................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9971
487/1688 [=======>......................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9970
496/1688 [=======>......................] - ETA: 7s - loss: 0.0120 - accuracy: 0.9970
505/1688 [=======>......................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9970
514/1688 [========>.....................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9969
523/1688 [========>.....................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9970
532/1688 [========>.....................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9969
541/1688 [========>.....................] - ETA: 7s - loss: 0.0125 - accuracy: 0.9968
550/1688 [========>.....................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9968
559/1688 [========>.....................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9968
568/1688 [=========>....................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9968
577/1688 [=========>....................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9967
586/1688 [=========>....................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9967
595/1688 [=========>....................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9967
604/1688 [=========>....................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9967
613/1688 [=========>....................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9966
622/1688 [==========>...................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9967
631/1688 [==========>...................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9967
640/1688 [==========>...................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9967
649/1688 [==========>...................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9967
657/1688 [==========>...................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9968
666/1688 [==========>...................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9968
675/1688 [==========>...................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9967
683/1688 [===========>..................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9966
692/1688 [===========>..................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9967
701/1688 [===========>..................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9966
710/1688 [===========>..................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9964
719/1688 [===========>..................] - ETA: 5s - loss: 0.0130 - accuracy: 0.9964
728/1688 [===========>..................] - ETA: 5s - loss: 0.0131 - accuracy: 0.9964
736/1688 [============>.................] - ETA: 5s - loss: 0.0130 - accuracy: 0.9964
745/1688 [============>.................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9964
754/1688 [============>.................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9964
763/1688 [============>.................] - ETA: 5s - loss: 0.0133 - accuracy: 0.9963
772/1688 [============>.................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9964
781/1688 [============>.................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9964
790/1688 [=============>................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9964
799/1688 [=============>................] - ETA: 5s - loss: 0.0134 - accuracy: 0.9963
807/1688 [=============>................] - ETA: 5s - loss: 0.0134 - accuracy: 0.9963
816/1688 [=============>................] - ETA: 5s - loss: 0.0134 - accuracy: 0.9963
825/1688 [=============>................] - ETA: 5s - loss: 0.0133 - accuracy: 0.9963
834/1688 [=============>................] - ETA: 5s - loss: 0.0133 - accuracy: 0.9964
843/1688 [=============>................] - ETA: 5s - loss: 0.0133 - accuracy: 0.9964
851/1688 [==============>...............] - ETA: 5s - loss: 0.0132 - accuracy: 0.9964
860/1688 [==============>...............] - ETA: 5s - loss: 0.0134 - accuracy: 0.9963
869/1688 [==============>...............] - ETA: 5s - loss: 0.0134 - accuracy: 0.9964
878/1688 [==============>...............] - ETA: 4s - loss: 0.0133 - accuracy: 0.9964
887/1688 [==============>...............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9964
896/1688 [==============>...............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9964
904/1688 [===============>..............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9964
913/1688 [===============>..............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9964
922/1688 [===============>..............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9964
931/1688 [===============>..............] - ETA: 4s - loss: 0.0133 - accuracy: 0.9965
940/1688 [===============>..............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9965
949/1688 [===============>..............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9964
958/1688 [================>.............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9965
967/1688 [================>.............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9965
976/1688 [================>.............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9965
985/1688 [================>.............] - ETA: 4s - loss: 0.0133 - accuracy: 0.9965
994/1688 [================>.............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9965
1002/1688 [================>.............] - ETA: 4s - loss: 0.0133 - accuracy: 0.9965
1011/1688 [================>.............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9965
1019/1688 [=================>............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9965
1028/1688 [=================>............] - ETA: 4s - loss: 0.0134 - accuracy: 0.9965
1036/1688 [=================>............] - ETA: 3s - loss: 0.0133 - accuracy: 0.9965
1045/1688 [=================>............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9964
1054/1688 [=================>............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9964
1063/1688 [=================>............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9964
1071/1688 [==================>...........] - ETA: 3s - loss: 0.0133 - accuracy: 0.9964
1080/1688 [==================>...........] - ETA: 3s - loss: 0.0133 - accuracy: 0.9964
1089/1688 [==================>...........] - ETA: 3s - loss: 0.0134 - accuracy: 0.9964
1098/1688 [==================>...........] - ETA: 3s - loss: 0.0133 - accuracy: 0.9964
1107/1688 [==================>...........] - ETA: 3s - loss: 0.0133 - accuracy: 0.9964
1116/1688 [==================>...........] - ETA: 3s - loss: 0.0134 - accuracy: 0.9964
1125/1688 [==================>...........] - ETA: 3s - loss: 0.0134 - accuracy: 0.9963
1134/1688 [===================>..........] - ETA: 3s - loss: 0.0134 - accuracy: 0.9964
1143/1688 [===================>..........] - ETA: 3s - loss: 0.0134 - accuracy: 0.9964
1152/1688 [===================>..........] - ETA: 3s - loss: 0.0133 - accuracy: 0.9964
1160/1688 [===================>..........] - ETA: 3s - loss: 0.0133 - accuracy: 0.9964
1169/1688 [===================>..........] - ETA: 3s - loss: 0.0133 - accuracy: 0.9964
1178/1688 [===================>..........] - ETA: 3s - loss: 0.0133 - accuracy: 0.9964
1187/1688 [====================>.........] - ETA: 3s - loss: 0.0132 - accuracy: 0.9964
1196/1688 [====================>.........] - ETA: 3s - loss: 0.0132 - accuracy: 0.9964
1204/1688 [====================>.........] - ETA: 2s - loss: 0.0132 - accuracy: 0.9964
1213/1688 [====================>.........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9964
1222/1688 [====================>.........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9964
1231/1688 [====================>.........] - ETA: 2s - loss: 0.0132 - accuracy: 0.9964
1240/1688 [=====================>........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9964
1249/1688 [=====================>........] - ETA: 2s - loss: 0.0133 - accuracy: 0.9964
1258/1688 [=====================>........] - ETA: 2s - loss: 0.0132 - accuracy: 0.9964
1266/1688 [=====================>........] - ETA: 2s - loss: 0.0132 - accuracy: 0.9964
1275/1688 [=====================>........] - ETA: 2s - loss: 0.0132 - accuracy: 0.9964
1284/1688 [=====================>........] - ETA: 2s - loss: 0.0132 - accuracy: 0.9964
1293/1688 [=====================>........] - ETA: 2s - loss: 0.0132 - accuracy: 0.9964
1301/1688 [======================>.......] - ETA: 2s - loss: 0.0132 - accuracy: 0.9964
1310/1688 [======================>.......] - ETA: 2s - loss: 0.0133 - accuracy: 0.9964
1319/1688 [======================>.......] - ETA: 2s - loss: 0.0134 - accuracy: 0.9964
1328/1688 [======================>.......] - ETA: 2s - loss: 0.0134 - accuracy: 0.9964
1337/1688 [======================>.......] - ETA: 2s - loss: 0.0134 - accuracy: 0.9964
1346/1688 [======================>.......] - ETA: 2s - loss: 0.0134 - accuracy: 0.9963
1355/1688 [=======================>......] - ETA: 2s - loss: 0.0134 - accuracy: 0.9963
1364/1688 [=======================>......] - ETA: 1s - loss: 0.0135 - accuracy: 0.9963
1372/1688 [=======================>......] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1381/1688 [=======================>......] - ETA: 1s - loss: 0.0135 - accuracy: 0.9963
1390/1688 [=======================>......] - ETA: 1s - loss: 0.0135 - accuracy: 0.9963
1399/1688 [=======================>......] - ETA: 1s - loss: 0.0135 - accuracy: 0.9963
1408/1688 [========================>.....] - ETA: 1s - loss: 0.0135 - accuracy: 0.9963
1417/1688 [========================>.....] - ETA: 1s - loss: 0.0135 - accuracy: 0.9963
1425/1688 [========================>.....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1434/1688 [========================>.....] - ETA: 1s - loss: 0.0135 - accuracy: 0.9963
1443/1688 [========================>.....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1460/1688 [========================>.....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1469/1688 [=========================>....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1478/1688 [=========================>....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1487/1688 [=========================>....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1496/1688 [=========================>....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9964
1505/1688 [=========================>....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9964
1514/1688 [=========================>....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1522/1688 [==========================>...] - ETA: 1s - loss: 0.0134 - accuracy: 0.9963
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0135 - accuracy: 0.9963
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0134 - accuracy: 0.9963
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0134 - accuracy: 0.9963
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0134 - accuracy: 0.9963
1567/1688 [==========================>...] - ETA: 0s - loss: 0.0134 - accuracy: 0.9964
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0133 - accuracy: 0.9964
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0133 - accuracy: 0.9964
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0133 - accuracy: 0.9964
1601/1688 [===========================>..] - ETA: 0s - loss: 0.0133 - accuracy: 0.9964
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0132 - accuracy: 0.9964
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0133 - accuracy: 0.9964
1627/1688 [===========================>..] - ETA: 0s - loss: 0.0133 - accuracy: 0.9964
1635/1688 [============================>.] - ETA: 0s - loss: 0.0134 - accuracy: 0.9964
1644/1688 [============================>.] - ETA: 0s - loss: 0.0134 - accuracy: 0.9964
1653/1688 [============================>.] - ETA: 0s - loss: 0.0134 - accuracy: 0.9964
1662/1688 [============================>.] - ETA: 0s - loss: 0.0135 - accuracy: 0.9964
1671/1688 [============================>.] - ETA: 0s - loss: 0.0135 - accuracy: 0.9964
1680/1688 [============================>.] - ETA: 0s - loss: 0.0135 - accuracy: 0.9964
1688/1688 [==============================] - 11s 7ms/step - loss: 0.0135 - accuracy: 0.9964 - val_loss: 0.0519 - val_accuracy: 0.9855
Epoch 5/5
1/1688 [..............................] - ETA: 10s - loss: 0.0068 - accuracy: 1.0000
10/1688 [..............................] - ETA: 10s - loss: 0.0086 - accuracy: 1.0000
19/1688 [..............................] - ETA: 10s - loss: 0.0131 - accuracy: 0.9951
28/1688 [..............................] - ETA: 10s - loss: 0.0125 - accuracy: 0.9955
37/1688 [..............................] - ETA: 10s - loss: 0.0113 - accuracy: 0.9958
46/1688 [..............................] - ETA: 9s - loss: 0.0123 - accuracy: 0.9946
55/1688 [..............................] - ETA: 9s - loss: 0.0126 - accuracy: 0.9955
64/1688 [>.............................] - ETA: 9s - loss: 0.0127 - accuracy: 0.9956
73/1688 [>.............................] - ETA: 9s - loss: 0.0123 - accuracy: 0.9961
82/1688 [>.............................] - ETA: 9s - loss: 0.0122 - accuracy: 0.9966
91/1688 [>.............................] - ETA: 9s - loss: 0.0118 - accuracy: 0.9969
100/1688 [>.............................] - ETA: 9s - loss: 0.0124 - accuracy: 0.9969
109/1688 [>.............................] - ETA: 9s - loss: 0.0121 - accuracy: 0.9971
118/1688 [=>............................] - ETA: 9s - loss: 0.0119 - accuracy: 0.9971
127/1688 [=>............................] - ETA: 9s - loss: 0.0122 - accuracy: 0.9968
136/1688 [=>............................] - ETA: 9s - loss: 0.0119 - accuracy: 0.9970
145/1688 [=>............................] - ETA: 9s - loss: 0.0117 - accuracy: 0.9972
154/1688 [=>............................] - ETA: 9s - loss: 0.0114 - accuracy: 0.9974
163/1688 [=>............................] - ETA: 9s - loss: 0.0114 - accuracy: 0.9973
172/1688 [==>...........................] - ETA: 9s - loss: 0.0109 - accuracy: 0.9975
181/1688 [==>...........................] - ETA: 9s - loss: 0.0107 - accuracy: 0.9976
190/1688 [==>...........................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9977
199/1688 [==>...........................] - ETA: 9s - loss: 0.0110 - accuracy: 0.9975
208/1688 [==>...........................] - ETA: 9s - loss: 0.0108 - accuracy: 0.9974
217/1688 [==>...........................] - ETA: 8s - loss: 0.0107 - accuracy: 0.9976
225/1688 [==>...........................] - ETA: 8s - loss: 0.0106 - accuracy: 0.9976
234/1688 [===>..........................] - ETA: 8s - loss: 0.0105 - accuracy: 0.9977
243/1688 [===>..........................] - ETA: 8s - loss: 0.0105 - accuracy: 0.9977
251/1688 [===>..........................] - ETA: 8s - loss: 0.0114 - accuracy: 0.9975
260/1688 [===>..........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9976
269/1688 [===>..........................] - ETA: 8s - loss: 0.0116 - accuracy: 0.9976
278/1688 [===>..........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9976
287/1688 [====>.........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9977
296/1688 [====>.........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9977
304/1688 [====>.........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9977
312/1688 [====>.........................] - ETA: 8s - loss: 0.0113 - accuracy: 0.9978
321/1688 [====>.........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9979
330/1688 [====>.........................] - ETA: 8s - loss: 0.0112 - accuracy: 0.9978
339/1688 [=====>........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9976
347/1688 [=====>........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9976
356/1688 [=====>........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9976
365/1688 [=====>........................] - ETA: 8s - loss: 0.0115 - accuracy: 0.9975
373/1688 [=====>........................] - ETA: 8s - loss: 0.0120 - accuracy: 0.9972
382/1688 [=====>........................] - ETA: 8s - loss: 0.0119 - accuracy: 0.9973
391/1688 [=====>........................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9972
400/1688 [======>.......................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9972
409/1688 [======>.......................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9972
418/1688 [======>.......................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9972
427/1688 [======>.......................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9971
436/1688 [======>.......................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9971
445/1688 [======>.......................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9972
454/1688 [=======>......................] - ETA: 7s - loss: 0.0120 - accuracy: 0.9972
463/1688 [=======>......................] - ETA: 7s - loss: 0.0123 - accuracy: 0.9970
472/1688 [=======>......................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9970
481/1688 [=======>......................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9970
490/1688 [=======>......................] - ETA: 7s - loss: 0.0125 - accuracy: 0.9969
499/1688 [=======>......................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9970
508/1688 [========>.....................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9970
517/1688 [========>.....................] - ETA: 7s - loss: 0.0125 - accuracy: 0.9970
526/1688 [========>.....................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9970
535/1688 [========>.....................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9970
544/1688 [========>.....................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9970
553/1688 [========>.....................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9971
561/1688 [========>.....................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9969
570/1688 [=========>....................] - ETA: 6s - loss: 0.0123 - accuracy: 0.9969
579/1688 [=========>....................] - ETA: 6s - loss: 0.0123 - accuracy: 0.9969
587/1688 [=========>....................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9970
596/1688 [=========>....................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9969
604/1688 [=========>....................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9969
613/1688 [=========>....................] - ETA: 6s - loss: 0.0123 - accuracy: 0.9968
622/1688 [==========>...................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9967
630/1688 [==========>...................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9967
638/1688 [==========>...................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9968
646/1688 [==========>...................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9967
654/1688 [==========>...................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9968
663/1688 [==========>...................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9968
672/1688 [==========>...................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9968
681/1688 [===========>..................] - ETA: 6s - loss: 0.0123 - accuracy: 0.9968
690/1688 [===========>..................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9968
699/1688 [===========>..................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9968
708/1688 [===========>..................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9968
717/1688 [===========>..................] - ETA: 5s - loss: 0.0126 - accuracy: 0.9968
726/1688 [===========>..................] - ETA: 5s - loss: 0.0127 - accuracy: 0.9968
734/1688 [============>.................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9968
743/1688 [============>.................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9968
752/1688 [============>.................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9967
761/1688 [============>.................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9967
770/1688 [============>.................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9967
779/1688 [============>.................] - ETA: 5s - loss: 0.0130 - accuracy: 0.9967
788/1688 [=============>................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9967
797/1688 [=============>................] - ETA: 5s - loss: 0.0130 - accuracy: 0.9967
805/1688 [=============>................] - ETA: 5s - loss: 0.0131 - accuracy: 0.9966
813/1688 [=============>................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9966
822/1688 [=============>................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9965
830/1688 [=============>................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9965
839/1688 [=============>................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9965
848/1688 [==============>...............] - ETA: 5s - loss: 0.0131 - accuracy: 0.9966
857/1688 [==============>...............] - ETA: 5s - loss: 0.0131 - accuracy: 0.9966
866/1688 [==============>...............] - ETA: 5s - loss: 0.0132 - accuracy: 0.9965
874/1688 [==============>...............] - ETA: 4s - loss: 0.0132 - accuracy: 0.9965
883/1688 [==============>...............] - ETA: 4s - loss: 0.0133 - accuracy: 0.9965
892/1688 [==============>...............] - ETA: 4s - loss: 0.0132 - accuracy: 0.9965
900/1688 [==============>...............] - ETA: 4s - loss: 0.0132 - accuracy: 0.9965
908/1688 [===============>..............] - ETA: 4s - loss: 0.0131 - accuracy: 0.9966
916/1688 [===============>..............] - ETA: 4s - loss: 0.0131 - accuracy: 0.9966
925/1688 [===============>..............] - ETA: 4s - loss: 0.0130 - accuracy: 0.9966
934/1688 [===============>..............] - ETA: 4s - loss: 0.0130 - accuracy: 0.9966
943/1688 [===============>..............] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
952/1688 [===============>..............] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
961/1688 [================>.............] - ETA: 4s - loss: 0.0132 - accuracy: 0.9965
969/1688 [================>.............] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
978/1688 [================>.............] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
986/1688 [================>.............] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
995/1688 [================>.............] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
1004/1688 [================>.............] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
1013/1688 [=================>............] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
1022/1688 [=================>............] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
1031/1688 [=================>............] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
1040/1688 [=================>............] - ETA: 3s - loss: 0.0130 - accuracy: 0.9965
1049/1688 [=================>............] - ETA: 3s - loss: 0.0129 - accuracy: 0.9965
1058/1688 [=================>............] - ETA: 3s - loss: 0.0129 - accuracy: 0.9966
1067/1688 [=================>............] - ETA: 3s - loss: 0.0128 - accuracy: 0.9965
1076/1688 [==================>...........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1085/1688 [==================>...........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1094/1688 [==================>...........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1103/1688 [==================>...........] - ETA: 3s - loss: 0.0129 - accuracy: 0.9966
1112/1688 [==================>...........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1121/1688 [==================>...........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1130/1688 [===================>..........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1139/1688 [===================>..........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1148/1688 [===================>..........] - ETA: 3s - loss: 0.0127 - accuracy: 0.9967
1157/1688 [===================>..........] - ETA: 3s - loss: 0.0127 - accuracy: 0.9967
1166/1688 [===================>..........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1175/1688 [===================>..........] - ETA: 3s - loss: 0.0127 - accuracy: 0.9966
1184/1688 [====================>.........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1193/1688 [====================>.........] - ETA: 3s - loss: 0.0128 - accuracy: 0.9966
1202/1688 [====================>.........] - ETA: 2s - loss: 0.0128 - accuracy: 0.9966
1210/1688 [====================>.........] - ETA: 2s - loss: 0.0128 - accuracy: 0.9966
1219/1688 [====================>.........] - ETA: 2s - loss: 0.0128 - accuracy: 0.9966
1227/1688 [====================>.........] - ETA: 2s - loss: 0.0128 - accuracy: 0.9966
1236/1688 [====================>.........] - ETA: 2s - loss: 0.0128 - accuracy: 0.9966
1245/1688 [=====================>........] - ETA: 2s - loss: 0.0130 - accuracy: 0.9965
1254/1688 [=====================>........] - ETA: 2s - loss: 0.0130 - accuracy: 0.9965
1263/1688 [=====================>........] - ETA: 2s - loss: 0.0130 - accuracy: 0.9965
1272/1688 [=====================>........] - ETA: 2s - loss: 0.0129 - accuracy: 0.9966
1280/1688 [=====================>........] - ETA: 2s - loss: 0.0130 - accuracy: 0.9965
1288/1688 [=====================>........] - ETA: 2s - loss: 0.0129 - accuracy: 0.9966
1297/1688 [======================>.......] - ETA: 2s - loss: 0.0128 - accuracy: 0.9966
1305/1688 [======================>.......] - ETA: 2s - loss: 0.0130 - accuracy: 0.9966
1313/1688 [======================>.......] - ETA: 2s - loss: 0.0131 - accuracy: 0.9965
1322/1688 [======================>.......] - ETA: 2s - loss: 0.0131 - accuracy: 0.9965
1331/1688 [======================>.......] - ETA: 2s - loss: 0.0131 - accuracy: 0.9965
1339/1688 [======================>.......] - ETA: 2s - loss: 0.0131 - accuracy: 0.9965
1348/1688 [======================>.......] - ETA: 2s - loss: 0.0131 - accuracy: 0.9965
1357/1688 [=======================>......] - ETA: 2s - loss: 0.0130 - accuracy: 0.9965
1365/1688 [=======================>......] - ETA: 1s - loss: 0.0131 - accuracy: 0.9965
1374/1688 [=======================>......] - ETA: 1s - loss: 0.0130 - accuracy: 0.9965
1383/1688 [=======================>......] - ETA: 1s - loss: 0.0131 - accuracy: 0.9965
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0132 - accuracy: 0.9964
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0132 - accuracy: 0.9964
1410/1688 [========================>.....] - ETA: 1s - loss: 0.0132 - accuracy: 0.9965
1418/1688 [========================>.....] - ETA: 1s - loss: 0.0132 - accuracy: 0.9965
1427/1688 [========================>.....] - ETA: 1s - loss: 0.0131 - accuracy: 0.9965
1436/1688 [========================>.....] - ETA: 1s - loss: 0.0131 - accuracy: 0.9965
1444/1688 [========================>.....] - ETA: 1s - loss: 0.0132 - accuracy: 0.9964
1453/1688 [========================>.....] - ETA: 1s - loss: 0.0132 - accuracy: 0.9964
1462/1688 [========================>.....] - ETA: 1s - loss: 0.0131 - accuracy: 0.9964
1471/1688 [=========================>....] - ETA: 1s - loss: 0.0132 - accuracy: 0.9964
1480/1688 [=========================>....] - ETA: 1s - loss: 0.0132 - accuracy: 0.9964
1489/1688 [=========================>....] - ETA: 1s - loss: 0.0132 - accuracy: 0.9964
1497/1688 [=========================>....] - ETA: 1s - loss: 0.0131 - accuracy: 0.9964
1505/1688 [=========================>....] - ETA: 1s - loss: 0.0131 - accuracy: 0.9964
1514/1688 [=========================>....] - ETA: 1s - loss: 0.0131 - accuracy: 0.9964
1523/1688 [==========================>...] - ETA: 1s - loss: 0.0131 - accuracy: 0.9965
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0131 - accuracy: 0.9965
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0131 - accuracy: 0.9965
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0131 - accuracy: 0.9965
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0132 - accuracy: 0.9964
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1611/1688 [===========================>..] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0130 - accuracy: 0.9965
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0130 - accuracy: 0.9965
1638/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1647/1688 [============================>.] - ETA: 0s - loss: 0.0130 - accuracy: 0.9964
1656/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1665/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1674/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1683/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9964
1688/1688 [==============================] - 11s 7ms/step - loss: 0.0131 - accuracy: 0.9964 - val_loss: 0.0563 - val_accuracy: 0.9853
<tf_keras.src.callbacks.History object at 0x75635a3500d0>
score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine-tuning:', score)
Test accuracy after fine-tuning: 0.9847000241279602
3. Convert
3.1 Convert to Akida model
When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.
As with TF-Keras, the summary() method provides a textual representation of the Akida model.
from cnn2snn import convert
model_akida = convert(model_quantized)
model_akida.summary()
Model Summary
______________________________________________
Input shape Output shape Sequences Layers
==============================================
[28, 28, 1] [1, 1, 10] 1 5
______________________________________________
__________________________________________________________________
Layer (type) Output shape Kernel shape
=============== SW/conv2d-dequantizer_2 (Software) ===============
conv2d (InputConv2D) [13, 13, 32] (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D) [7, 7, 32] (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D) [7, 7, 64] (1, 1, 32, 64)
__________________________________________________________________
dense (Dense1D) [1, 1, 10] (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer) [1, 1, 10] N/A
__________________________________________________________________
3.2. Check performance
accuracy = model_akida.evaluate(x_test, y_test.astype(np.int32))
print('Test accuracy after conversion:', accuracy)
# For non-regression purposes
assert accuracy > 0.96
Test accuracy after conversion: 0.9857000112533569
3.3 Show predictions for a single image
Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.
# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])
f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()
print(outputs.squeeze())
Input Label: 7
[ -7.467366 -7.3476696 -2.5231757 -1.295884 -10.602389 -6.829221
-18.014277 7.0449476 -4.90244 -0.7034977]
Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.
In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.
Total running time of the script: (2 minutes 14.699 seconds)