Global Akida workflow

Using the MNIST dataset, this example shows the definition and training of a TF-Keras floating point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original TF-Keras floating point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.

Note

Please refer to the TensorFlow tf_keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.

The MNIST example below is light enough so that a GPU is not needed for training.

Overall flow

Global Akida workflow

1. Create and train

1.1. Load and reshape MNIST dataset

import numpy as np

import matplotlib.cm as cm
import matplotlib.pyplot as plt

from tf_keras.datasets import mnist

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, width, height, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)

# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
    axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
    axarr[i].set_title('Class %d' % y_test[i])
plt.show()
Class 7, Class 2, Class 1, Class 0
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz

    8192/11490434 [..............................] - ETA: 0s
   16384/11490434 [..............................] - ETA: 43s
   49152/11490434 [..............................] - ETA: 35s
   81920/11490434 [..............................] - ETA: 33s
  147456/11490434 [..............................] - ETA: 23s
  212992/11490434 [..............................] - ETA: 19s
  327680/11490434 [..............................] - ETA: 14s
  475136/11490434 [>.............................] - ETA: 10s
  753664/11490434 [>.............................] - ETA: 7s 
 1130496/11490434 [=>............................] - ETA: 5s
 1720320/11490434 [===>..........................] - ETA: 3s
 2678784/11490434 [=====>........................] - ETA: 2s
 2973696/11490434 [======>.......................] - ETA: 2s
 3858432/11490434 [=========>....................] - ETA: 1s
 4612096/11490434 [===========>..................] - ETA: 1s
 4702208/11490434 [===========>..................] - ETA: 1s
 6160384/11490434 [===============>..............] - ETA: 0s
 6750208/11490434 [================>.............] - ETA: 0s
 7421952/11490434 [==================>...........] - ETA: 0s
 8060928/11490434 [====================>.........] - ETA: 0s
 8667136/11490434 [=====================>........] - ETA: 0s
 9273344/11490434 [=======================>......] - ETA: 0s
 9887744/11490434 [========================>.....] - ETA: 0s
10469376/11490434 [==========================>...] - ETA: 0s
11075584/11490434 [===========================>..] - ETA: 0s
11490434/11490434 [==============================] - 1s 0us/step

1.2. Model definition

Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard TF-Keras CNN model.

import tf_keras as keras

model_keras = keras.models.Sequential([
    keras.layers.Rescaling(1. / 255, input_shape=(28, 28, 1)),
    keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    # Separable layer
    keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
    keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling (Rescaling)       (None, 28, 28, 1)         0

 conv2d (Conv2D)             (None, 13, 13, 32)        320

 batch_normalization (Batch  (None, 13, 13, 32)        128
 Normalization)

 re_lu (ReLU)                (None, 13, 13, 32)        0

 depthwise_conv2d (Depthwis  (None, 7, 7, 32)          320
 eConv2D)

 conv2d_1 (Conv2D)           (None, 7, 7, 64)          2112

 batch_normalization_1 (Bat  (None, 7, 7, 64)          256
 chNormalization)

 re_lu_1 (ReLU)              (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (Dense)               (None, 10)                31370

=================================================================
Total params: 34506 (134.79 KB)
Trainable params: 34314 (134.04 KB)
Non-trainable params: 192 (768.00 Byte)
_________________________________________________________________

1.3. Model training

Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.

from tf_keras.optimizers import Adam

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-3),
    metrics=['accuracy'])

_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
Epoch 1/10

   1/1688 [..............................] - ETA: 1:14:10 - loss: 2.9487 - accuracy: 0.0000e+00
  21/1688 [..............................] - ETA: 4s - loss: 1.5952 - accuracy: 0.4673         
  43/1688 [..............................] - ETA: 3s - loss: 1.1236 - accuracy: 0.6432
  65/1688 [>.............................] - ETA: 3s - loss: 0.9032 - accuracy: 0.7159
  88/1688 [>.............................] - ETA: 3s - loss: 0.7725 - accuracy: 0.7592
 111/1688 [>.............................] - ETA: 3s - loss: 0.6787 - accuracy: 0.7891
 133/1688 [=>............................] - ETA: 3s - loss: 0.6124 - accuracy: 0.8127
 156/1688 [=>............................] - ETA: 3s - loss: 0.5699 - accuracy: 0.8269
 176/1688 [==>...........................] - ETA: 3s - loss: 0.5439 - accuracy: 0.8354
 197/1688 [==>...........................] - ETA: 3s - loss: 0.5140 - accuracy: 0.8441
 219/1688 [==>...........................] - ETA: 3s - loss: 0.4879 - accuracy: 0.8529
 241/1688 [===>..........................] - ETA: 3s - loss: 0.4720 - accuracy: 0.8575
 263/1688 [===>..........................] - ETA: 3s - loss: 0.4517 - accuracy: 0.8634
 285/1688 [====>.........................] - ETA: 3s - loss: 0.4333 - accuracy: 0.8685
 307/1688 [====>.........................] - ETA: 3s - loss: 0.4148 - accuracy: 0.8744
 331/1688 [====>.........................] - ETA: 3s - loss: 0.3969 - accuracy: 0.8798
 354/1688 [=====>........................] - ETA: 3s - loss: 0.3842 - accuracy: 0.8840
 376/1688 [=====>........................] - ETA: 3s - loss: 0.3765 - accuracy: 0.8866
 399/1688 [======>.......................] - ETA: 2s - loss: 0.3662 - accuracy: 0.8893
 421/1688 [======>.......................] - ETA: 2s - loss: 0.3566 - accuracy: 0.8921
 444/1688 [======>.......................] - ETA: 2s - loss: 0.3515 - accuracy: 0.8934
 467/1688 [=======>......................] - ETA: 2s - loss: 0.3440 - accuracy: 0.8954
 490/1688 [=======>......................] - ETA: 2s - loss: 0.3361 - accuracy: 0.8977
 513/1688 [========>.....................] - ETA: 2s - loss: 0.3272 - accuracy: 0.9003
 536/1688 [========>.....................] - ETA: 2s - loss: 0.3182 - accuracy: 0.9030
 560/1688 [========>.....................] - ETA: 2s - loss: 0.3110 - accuracy: 0.9054
 583/1688 [=========>....................] - ETA: 2s - loss: 0.3047 - accuracy: 0.9071
 605/1688 [=========>....................] - ETA: 2s - loss: 0.2987 - accuracy: 0.9089
 627/1688 [==========>...................] - ETA: 2s - loss: 0.2922 - accuracy: 0.9110
 650/1688 [==========>...................] - ETA: 2s - loss: 0.2876 - accuracy: 0.9125
 672/1688 [==========>...................] - ETA: 2s - loss: 0.2823 - accuracy: 0.9141
 694/1688 [===========>..................] - ETA: 2s - loss: 0.2775 - accuracy: 0.9155
 716/1688 [===========>..................] - ETA: 2s - loss: 0.2729 - accuracy: 0.9167
 739/1688 [============>.................] - ETA: 2s - loss: 0.2676 - accuracy: 0.9184
 762/1688 [============>.................] - ETA: 2s - loss: 0.2644 - accuracy: 0.9196
 785/1688 [============>.................] - ETA: 2s - loss: 0.2607 - accuracy: 0.9206
 807/1688 [=============>................] - ETA: 2s - loss: 0.2570 - accuracy: 0.9216
 829/1688 [=============>................] - ETA: 1s - loss: 0.2540 - accuracy: 0.9226
 851/1688 [==============>...............] - ETA: 1s - loss: 0.2502 - accuracy: 0.9235
 873/1688 [==============>...............] - ETA: 1s - loss: 0.2470 - accuracy: 0.9246
 897/1688 [==============>...............] - ETA: 1s - loss: 0.2435 - accuracy: 0.9257
 919/1688 [===============>..............] - ETA: 1s - loss: 0.2403 - accuracy: 0.9267
 941/1688 [===============>..............] - ETA: 1s - loss: 0.2369 - accuracy: 0.9277
 963/1688 [================>.............] - ETA: 1s - loss: 0.2344 - accuracy: 0.9285
 985/1688 [================>.............] - ETA: 1s - loss: 0.2320 - accuracy: 0.9292
1008/1688 [================>.............] - ETA: 1s - loss: 0.2292 - accuracy: 0.9300
1030/1688 [=================>............] - ETA: 1s - loss: 0.2270 - accuracy: 0.9307
1052/1688 [=================>............] - ETA: 1s - loss: 0.2244 - accuracy: 0.9316
1075/1688 [==================>...........] - ETA: 1s - loss: 0.2219 - accuracy: 0.9324
1097/1688 [==================>...........] - ETA: 1s - loss: 0.2201 - accuracy: 0.9331
1119/1688 [==================>...........] - ETA: 1s - loss: 0.2179 - accuracy: 0.9338
1142/1688 [===================>..........] - ETA: 1s - loss: 0.2160 - accuracy: 0.9343
1164/1688 [===================>..........] - ETA: 1s - loss: 0.2144 - accuracy: 0.9347
1186/1688 [====================>.........] - ETA: 1s - loss: 0.2121 - accuracy: 0.9355
1208/1688 [====================>.........] - ETA: 1s - loss: 0.2098 - accuracy: 0.9361
1230/1688 [====================>.........] - ETA: 1s - loss: 0.2076 - accuracy: 0.9368
1253/1688 [=====================>........] - ETA: 0s - loss: 0.2056 - accuracy: 0.9375
1276/1688 [=====================>........] - ETA: 0s - loss: 0.2040 - accuracy: 0.9383
1299/1688 [======================>.......] - ETA: 0s - loss: 0.2021 - accuracy: 0.9389
1321/1688 [======================>.......] - ETA: 0s - loss: 0.2006 - accuracy: 0.9394
1343/1688 [======================>.......] - ETA: 0s - loss: 0.1992 - accuracy: 0.9397
1365/1688 [=======================>......] - ETA: 0s - loss: 0.1975 - accuracy: 0.9403
1387/1688 [=======================>......] - ETA: 0s - loss: 0.1955 - accuracy: 0.9409
1410/1688 [========================>.....] - ETA: 0s - loss: 0.1939 - accuracy: 0.9414
1433/1688 [========================>.....] - ETA: 0s - loss: 0.1925 - accuracy: 0.9417
1455/1688 [========================>.....] - ETA: 0s - loss: 0.1913 - accuracy: 0.9421
1478/1688 [=========================>....] - ETA: 0s - loss: 0.1895 - accuracy: 0.9426
1501/1688 [=========================>....] - ETA: 0s - loss: 0.1880 - accuracy: 0.9430
1523/1688 [==========================>...] - ETA: 0s - loss: 0.1860 - accuracy: 0.9436
1545/1688 [==========================>...] - ETA: 0s - loss: 0.1842 - accuracy: 0.9442
1567/1688 [==========================>...] - ETA: 0s - loss: 0.1828 - accuracy: 0.9446
1590/1688 [===========================>..] - ETA: 0s - loss: 0.1813 - accuracy: 0.9449
1613/1688 [===========================>..] - ETA: 0s - loss: 0.1805 - accuracy: 0.9453
1635/1688 [============================>.] - ETA: 0s - loss: 0.1795 - accuracy: 0.9456
1657/1688 [============================>.] - ETA: 0s - loss: 0.1781 - accuracy: 0.9460
1679/1688 [============================>.] - ETA: 0s - loss: 0.1772 - accuracy: 0.9463
1688/1688 [==============================] - ETA: 0s - loss: 0.1766 - accuracy: 0.9465
1688/1688 [==============================] - 7s 3ms/step - loss: 0.1766 - accuracy: 0.9465 - val_loss: 0.0821 - val_accuracy: 0.9742
Epoch 2/10

   1/1688 [..............................] - ETA: 6s - loss: 0.0573 - accuracy: 1.0000
  18/1688 [..............................] - ETA: 4s - loss: 0.0653 - accuracy: 0.9809
  41/1688 [..............................] - ETA: 4s - loss: 0.0818 - accuracy: 0.9748
  63/1688 [>.............................] - ETA: 4s - loss: 0.0708 - accuracy: 0.9777
  86/1688 [>.............................] - ETA: 3s - loss: 0.0677 - accuracy: 0.9793
 108/1688 [>.............................] - ETA: 3s - loss: 0.0722 - accuracy: 0.9783
 131/1688 [=>............................] - ETA: 3s - loss: 0.0681 - accuracy: 0.9800
 153/1688 [=>............................] - ETA: 3s - loss: 0.0670 - accuracy: 0.9804
 176/1688 [==>...........................] - ETA: 3s - loss: 0.0699 - accuracy: 0.9799
 199/1688 [==>...........................] - ETA: 3s - loss: 0.0694 - accuracy: 0.9796
 222/1688 [==>...........................] - ETA: 3s - loss: 0.0710 - accuracy: 0.9778
 245/1688 [===>..........................] - ETA: 3s - loss: 0.0684 - accuracy: 0.9786
 267/1688 [===>..........................] - ETA: 3s - loss: 0.0677 - accuracy: 0.9789
 290/1688 [====>.........................] - ETA: 3s - loss: 0.0679 - accuracy: 0.9787
 312/1688 [====>.........................] - ETA: 3s - loss: 0.0665 - accuracy: 0.9790
 334/1688 [====>.........................] - ETA: 3s - loss: 0.0689 - accuracy: 0.9785
 357/1688 [=====>........................] - ETA: 3s - loss: 0.0683 - accuracy: 0.9786
 379/1688 [=====>........................] - ETA: 3s - loss: 0.0676 - accuracy: 0.9788
 401/1688 [======>.......................] - ETA: 2s - loss: 0.0673 - accuracy: 0.9788
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0666 - accuracy: 0.9790
 445/1688 [======>.......................] - ETA: 2s - loss: 0.0666 - accuracy: 0.9790
 468/1688 [=======>......................] - ETA: 2s - loss: 0.0659 - accuracy: 0.9792
 491/1688 [=======>......................] - ETA: 2s - loss: 0.0667 - accuracy: 0.9788
 514/1688 [========>.....................] - ETA: 2s - loss: 0.0671 - accuracy: 0.9787
 536/1688 [========>.....................] - ETA: 2s - loss: 0.0669 - accuracy: 0.9790
 558/1688 [========>.....................] - ETA: 2s - loss: 0.0665 - accuracy: 0.9792
 581/1688 [=========>....................] - ETA: 2s - loss: 0.0670 - accuracy: 0.9789
 604/1688 [=========>....................] - ETA: 2s - loss: 0.0678 - accuracy: 0.9788
 626/1688 [==========>...................] - ETA: 2s - loss: 0.0677 - accuracy: 0.9786
 649/1688 [==========>...................] - ETA: 2s - loss: 0.0672 - accuracy: 0.9787
 671/1688 [==========>...................] - ETA: 2s - loss: 0.0666 - accuracy: 0.9789
 694/1688 [===========>..................] - ETA: 2s - loss: 0.0667 - accuracy: 0.9790
 716/1688 [===========>..................] - ETA: 2s - loss: 0.0666 - accuracy: 0.9791
 739/1688 [============>.................] - ETA: 2s - loss: 0.0665 - accuracy: 0.9791
 761/1688 [============>.................] - ETA: 2s - loss: 0.0676 - accuracy: 0.9788
 783/1688 [============>.................] - ETA: 2s - loss: 0.0680 - accuracy: 0.9786
 805/1688 [=============>................] - ETA: 2s - loss: 0.0677 - accuracy: 0.9785
 827/1688 [=============>................] - ETA: 1s - loss: 0.0682 - accuracy: 0.9783
 849/1688 [==============>...............] - ETA: 1s - loss: 0.0682 - accuracy: 0.9784
 871/1688 [==============>...............] - ETA: 1s - loss: 0.0686 - accuracy: 0.9785
 893/1688 [==============>...............] - ETA: 1s - loss: 0.0684 - accuracy: 0.9787
 915/1688 [===============>..............] - ETA: 1s - loss: 0.0688 - accuracy: 0.9785
 938/1688 [===============>..............] - ETA: 1s - loss: 0.0680 - accuracy: 0.9786
 961/1688 [================>.............] - ETA: 1s - loss: 0.0679 - accuracy: 0.9786
 983/1688 [================>.............] - ETA: 1s - loss: 0.0677 - accuracy: 0.9787
1005/1688 [================>.............] - ETA: 1s - loss: 0.0680 - accuracy: 0.9786
1027/1688 [=================>............] - ETA: 1s - loss: 0.0678 - accuracy: 0.9787
1050/1688 [=================>............] - ETA: 1s - loss: 0.0679 - accuracy: 0.9786
1072/1688 [==================>...........] - ETA: 1s - loss: 0.0678 - accuracy: 0.9784
1094/1688 [==================>...........] - ETA: 1s - loss: 0.0674 - accuracy: 0.9786
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0677 - accuracy: 0.9784
1139/1688 [===================>..........] - ETA: 1s - loss: 0.0678 - accuracy: 0.9785
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0679 - accuracy: 0.9785
1184/1688 [====================>.........] - ETA: 1s - loss: 0.0674 - accuracy: 0.9786
1206/1688 [====================>.........] - ETA: 1s - loss: 0.0674 - accuracy: 0.9786
1228/1688 [====================>.........] - ETA: 1s - loss: 0.0674 - accuracy: 0.9788
1250/1688 [=====================>........] - ETA: 1s - loss: 0.0671 - accuracy: 0.9789
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0669 - accuracy: 0.9789
1295/1688 [======================>.......] - ETA: 0s - loss: 0.0670 - accuracy: 0.9790
1318/1688 [======================>.......] - ETA: 0s - loss: 0.0677 - accuracy: 0.9789
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0678 - accuracy: 0.9789
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0676 - accuracy: 0.9789
1385/1688 [=======================>......] - ETA: 0s - loss: 0.0678 - accuracy: 0.9788
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0676 - accuracy: 0.9788
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0678 - accuracy: 0.9787
1451/1688 [========================>.....] - ETA: 0s - loss: 0.0680 - accuracy: 0.9786
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0676 - accuracy: 0.9787
1496/1688 [=========================>....] - ETA: 0s - loss: 0.0676 - accuracy: 0.9787
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0679 - accuracy: 0.9786
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0680 - accuracy: 0.9786
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0680 - accuracy: 0.9786
1588/1688 [===========================>..] - ETA: 0s - loss: 0.0682 - accuracy: 0.9785
1611/1688 [===========================>..] - ETA: 0s - loss: 0.0682 - accuracy: 0.9785
1633/1688 [============================>.] - ETA: 0s - loss: 0.0683 - accuracy: 0.9785
1656/1688 [============================>.] - ETA: 0s - loss: 0.0686 - accuracy: 0.9785
1679/1688 [============================>.] - ETA: 0s - loss: 0.0686 - accuracy: 0.9784
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0687 - accuracy: 0.9785 - val_loss: 0.0683 - val_accuracy: 0.9797
Epoch 3/10

   1/1688 [..............................] - ETA: 4s - loss: 0.0048 - accuracy: 1.0000
  23/1688 [..............................] - ETA: 3s - loss: 0.0362 - accuracy: 0.9891
  46/1688 [..............................] - ETA: 3s - loss: 0.0444 - accuracy: 0.9837
  69/1688 [>.............................] - ETA: 3s - loss: 0.0402 - accuracy: 0.9864
  92/1688 [>.............................] - ETA: 3s - loss: 0.0385 - accuracy: 0.9861
 114/1688 [=>............................] - ETA: 3s - loss: 0.0427 - accuracy: 0.9855
 136/1688 [=>............................] - ETA: 3s - loss: 0.0439 - accuracy: 0.9855
 158/1688 [=>............................] - ETA: 3s - loss: 0.0416 - accuracy: 0.9864
 181/1688 [==>...........................] - ETA: 3s - loss: 0.0403 - accuracy: 0.9869
 204/1688 [==>...........................] - ETA: 3s - loss: 0.0383 - accuracy: 0.9877
 227/1688 [===>..........................] - ETA: 3s - loss: 0.0390 - accuracy: 0.9876
 249/1688 [===>..........................] - ETA: 3s - loss: 0.0395 - accuracy: 0.9869
 271/1688 [===>..........................] - ETA: 3s - loss: 0.0392 - accuracy: 0.9871
 293/1688 [====>.........................] - ETA: 3s - loss: 0.0399 - accuracy: 0.9869
 316/1688 [====>.........................] - ETA: 3s - loss: 0.0399 - accuracy: 0.9871
 339/1688 [=====>........................] - ETA: 3s - loss: 0.0400 - accuracy: 0.9873
 361/1688 [=====>........................] - ETA: 3s - loss: 0.0404 - accuracy: 0.9874
 383/1688 [=====>........................] - ETA: 2s - loss: 0.0397 - accuracy: 0.9877
 405/1688 [======>.......................] - ETA: 2s - loss: 0.0417 - accuracy: 0.9872
 428/1688 [======>.......................] - ETA: 2s - loss: 0.0436 - accuracy: 0.9866
 451/1688 [=======>......................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9861
 473/1688 [=======>......................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9864
 495/1688 [=======>......................] - ETA: 2s - loss: 0.0443 - accuracy: 0.9861
 518/1688 [========>.....................] - ETA: 2s - loss: 0.0438 - accuracy: 0.9863
 540/1688 [========>.....................] - ETA: 2s - loss: 0.0442 - accuracy: 0.9863
 562/1688 [========>.....................] - ETA: 2s - loss: 0.0448 - accuracy: 0.9859
 584/1688 [=========>....................] - ETA: 2s - loss: 0.0455 - accuracy: 0.9855
 606/1688 [=========>....................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9855
 628/1688 [==========>...................] - ETA: 2s - loss: 0.0448 - accuracy: 0.9856
 651/1688 [==========>...................] - ETA: 2s - loss: 0.0447 - accuracy: 0.9857
 673/1688 [==========>...................] - ETA: 2s - loss: 0.0439 - accuracy: 0.9859
 696/1688 [===========>..................] - ETA: 2s - loss: 0.0448 - accuracy: 0.9860
 719/1688 [===========>..................] - ETA: 2s - loss: 0.0452 - accuracy: 0.9858
 741/1688 [============>.................] - ETA: 2s - loss: 0.0453 - accuracy: 0.9857
 763/1688 [============>.................] - ETA: 2s - loss: 0.0454 - accuracy: 0.9858
 785/1688 [============>.................] - ETA: 2s - loss: 0.0461 - accuracy: 0.9856
 808/1688 [=============>................] - ETA: 2s - loss: 0.0466 - accuracy: 0.9855
 831/1688 [=============>................] - ETA: 1s - loss: 0.0464 - accuracy: 0.9854
 854/1688 [==============>...............] - ETA: 1s - loss: 0.0473 - accuracy: 0.9854
 876/1688 [==============>...............] - ETA: 1s - loss: 0.0474 - accuracy: 0.9854
 898/1688 [==============>...............] - ETA: 1s - loss: 0.0478 - accuracy: 0.9853
 920/1688 [===============>..............] - ETA: 1s - loss: 0.0482 - accuracy: 0.9851
 942/1688 [===============>..............] - ETA: 1s - loss: 0.0495 - accuracy: 0.9848
 965/1688 [================>.............] - ETA: 1s - loss: 0.0504 - accuracy: 0.9845
 987/1688 [================>.............] - ETA: 1s - loss: 0.0506 - accuracy: 0.9844
1010/1688 [================>.............] - ETA: 1s - loss: 0.0502 - accuracy: 0.9846
1033/1688 [=================>............] - ETA: 1s - loss: 0.0502 - accuracy: 0.9847
1055/1688 [=================>............] - ETA: 1s - loss: 0.0500 - accuracy: 0.9847
1077/1688 [==================>...........] - ETA: 1s - loss: 0.0499 - accuracy: 0.9849
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0501 - accuracy: 0.9847
1121/1688 [==================>...........] - ETA: 1s - loss: 0.0503 - accuracy: 0.9847
1145/1688 [===================>..........] - ETA: 1s - loss: 0.0501 - accuracy: 0.9847
1168/1688 [===================>..........] - ETA: 1s - loss: 0.0506 - accuracy: 0.9846
1190/1688 [====================>.........] - ETA: 1s - loss: 0.0503 - accuracy: 0.9846
1211/1688 [====================>.........] - ETA: 1s - loss: 0.0502 - accuracy: 0.9845
1234/1688 [====================>.........] - ETA: 1s - loss: 0.0500 - accuracy: 0.9846
1256/1688 [=====================>........] - ETA: 0s - loss: 0.0499 - accuracy: 0.9845
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0502 - accuracy: 0.9844
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0505 - accuracy: 0.9844
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0508 - accuracy: 0.9843
1345/1688 [======================>.......] - ETA: 0s - loss: 0.0510 - accuracy: 0.9842
1367/1688 [=======================>......] - ETA: 0s - loss: 0.0515 - accuracy: 0.9841
1388/1688 [=======================>......] - ETA: 0s - loss: 0.0513 - accuracy: 0.9842
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0514 - accuracy: 0.9843
1431/1688 [========================>.....] - ETA: 0s - loss: 0.0513 - accuracy: 0.9843
1452/1688 [========================>.....] - ETA: 0s - loss: 0.0509 - accuracy: 0.9844
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0508 - accuracy: 0.9844
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0509 - accuracy: 0.9844
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0508 - accuracy: 0.9844
1541/1688 [==========================>...] - ETA: 0s - loss: 0.0508 - accuracy: 0.9844
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0509 - accuracy: 0.9843
1586/1688 [===========================>..] - ETA: 0s - loss: 0.0515 - accuracy: 0.9842
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0515 - accuracy: 0.9843
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0518 - accuracy: 0.9842
1652/1688 [============================>.] - ETA: 0s - loss: 0.0520 - accuracy: 0.9841
1674/1688 [============================>.] - ETA: 0s - loss: 0.0518 - accuracy: 0.9841
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0520 - accuracy: 0.9840 - val_loss: 0.0559 - val_accuracy: 0.9835
Epoch 4/10

   1/1688 [..............................] - ETA: 4s - loss: 0.0032 - accuracy: 1.0000
  23/1688 [..............................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9946
  45/1688 [..............................] - ETA: 3s - loss: 0.0193 - accuracy: 0.9951
  67/1688 [>.............................] - ETA: 3s - loss: 0.0213 - accuracy: 0.9949
  89/1688 [>.............................] - ETA: 3s - loss: 0.0203 - accuracy: 0.9944
 111/1688 [>.............................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9932
 133/1688 [=>............................] - ETA: 3s - loss: 0.0243 - accuracy: 0.9930
 156/1688 [=>............................] - ETA: 3s - loss: 0.0262 - accuracy: 0.9920
 179/1688 [==>...........................] - ETA: 3s - loss: 0.0277 - accuracy: 0.9914
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0304 - accuracy: 0.9910
 224/1688 [==>...........................] - ETA: 3s - loss: 0.0327 - accuracy: 0.9901
 246/1688 [===>..........................] - ETA: 3s - loss: 0.0329 - accuracy: 0.9897
 268/1688 [===>..........................] - ETA: 3s - loss: 0.0334 - accuracy: 0.9899
 289/1688 [====>.........................] - ETA: 3s - loss: 0.0348 - accuracy: 0.9895
 311/1688 [====>.........................] - ETA: 3s - loss: 0.0349 - accuracy: 0.9893
 333/1688 [====>.........................] - ETA: 3s - loss: 0.0348 - accuracy: 0.9894
 355/1688 [=====>........................] - ETA: 3s - loss: 0.0340 - accuracy: 0.9896
 378/1688 [=====>........................] - ETA: 3s - loss: 0.0338 - accuracy: 0.9897
 400/1688 [======>.......................] - ETA: 2s - loss: 0.0334 - accuracy: 0.9899
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0342 - accuracy: 0.9896
 445/1688 [======>.......................] - ETA: 2s - loss: 0.0355 - accuracy: 0.9893
 467/1688 [=======>......................] - ETA: 2s - loss: 0.0362 - accuracy: 0.9891
 490/1688 [=======>......................] - ETA: 2s - loss: 0.0358 - accuracy: 0.9892
 513/1688 [========>.....................] - ETA: 2s - loss: 0.0362 - accuracy: 0.9889
 536/1688 [========>.....................] - ETA: 2s - loss: 0.0360 - accuracy: 0.9891
 559/1688 [========>.....................] - ETA: 2s - loss: 0.0365 - accuracy: 0.9890
 581/1688 [=========>....................] - ETA: 2s - loss: 0.0370 - accuracy: 0.9887
 604/1688 [=========>....................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9887
 626/1688 [==========>...................] - ETA: 2s - loss: 0.0378 - accuracy: 0.9887
 649/1688 [==========>...................] - ETA: 2s - loss: 0.0383 - accuracy: 0.9883
 671/1688 [==========>...................] - ETA: 2s - loss: 0.0392 - accuracy: 0.9883
 693/1688 [===========>..................] - ETA: 2s - loss: 0.0399 - accuracy: 0.9882
 716/1688 [===========>..................] - ETA: 2s - loss: 0.0404 - accuracy: 0.9881
 738/1688 [============>.................] - ETA: 2s - loss: 0.0407 - accuracy: 0.9880
 761/1688 [============>.................] - ETA: 2s - loss: 0.0413 - accuracy: 0.9876
 784/1688 [============>.................] - ETA: 2s - loss: 0.0411 - accuracy: 0.9877
 806/1688 [=============>................] - ETA: 2s - loss: 0.0410 - accuracy: 0.9878
 829/1688 [=============>................] - ETA: 1s - loss: 0.0410 - accuracy: 0.9878
 851/1688 [==============>...............] - ETA: 1s - loss: 0.0407 - accuracy: 0.9878
 874/1688 [==============>...............] - ETA: 1s - loss: 0.0406 - accuracy: 0.9879
 897/1688 [==============>...............] - ETA: 1s - loss: 0.0403 - accuracy: 0.9880
 920/1688 [===============>..............] - ETA: 1s - loss: 0.0406 - accuracy: 0.9878
 943/1688 [===============>..............] - ETA: 1s - loss: 0.0408 - accuracy: 0.9879
 966/1688 [================>.............] - ETA: 1s - loss: 0.0405 - accuracy: 0.9880
 988/1688 [================>.............] - ETA: 1s - loss: 0.0404 - accuracy: 0.9880
1010/1688 [================>.............] - ETA: 1s - loss: 0.0404 - accuracy: 0.9881
1033/1688 [=================>............] - ETA: 1s - loss: 0.0406 - accuracy: 0.9880
1056/1688 [=================>............] - ETA: 1s - loss: 0.0405 - accuracy: 0.9880
1078/1688 [==================>...........] - ETA: 1s - loss: 0.0403 - accuracy: 0.9880
1100/1688 [==================>...........] - ETA: 1s - loss: 0.0405 - accuracy: 0.9880
1123/1688 [==================>...........] - ETA: 1s - loss: 0.0405 - accuracy: 0.9880
1145/1688 [===================>..........] - ETA: 1s - loss: 0.0405 - accuracy: 0.9880
1168/1688 [===================>..........] - ETA: 1s - loss: 0.0403 - accuracy: 0.9881
1191/1688 [====================>.........] - ETA: 1s - loss: 0.0400 - accuracy: 0.9882
1212/1688 [====================>.........] - ETA: 1s - loss: 0.0398 - accuracy: 0.9882
1236/1688 [====================>.........] - ETA: 1s - loss: 0.0394 - accuracy: 0.9883
1259/1688 [=====================>........] - ETA: 0s - loss: 0.0394 - accuracy: 0.9883
1281/1688 [=====================>........] - ETA: 0s - loss: 0.0398 - accuracy: 0.9882
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0396 - accuracy: 0.9882
1325/1688 [======================>.......] - ETA: 0s - loss: 0.0396 - accuracy: 0.9883
1348/1688 [======================>.......] - ETA: 0s - loss: 0.0401 - accuracy: 0.9882
1370/1688 [=======================>......] - ETA: 0s - loss: 0.0404 - accuracy: 0.9881
1393/1688 [=======================>......] - ETA: 0s - loss: 0.0403 - accuracy: 0.9881
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0402 - accuracy: 0.9881
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0401 - accuracy: 0.9881
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0399 - accuracy: 0.9882
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0396 - accuracy: 0.9882
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0399 - accuracy: 0.9881
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0400 - accuracy: 0.9881
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0400 - accuracy: 0.9880
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0402 - accuracy: 0.9880
1592/1688 [===========================>..] - ETA: 0s - loss: 0.0405 - accuracy: 0.9878
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0406 - accuracy: 0.9877
1636/1688 [============================>.] - ETA: 0s - loss: 0.0406 - accuracy: 0.9877
1659/1688 [============================>.] - ETA: 0s - loss: 0.0406 - accuracy: 0.9877
1681/1688 [============================>.] - ETA: 0s - loss: 0.0411 - accuracy: 0.9875
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0410 - accuracy: 0.9875 - val_loss: 0.0633 - val_accuracy: 0.9825
Epoch 5/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0820 - accuracy: 0.9688
  23/1688 [..............................] - ETA: 3s - loss: 0.0339 - accuracy: 0.9878
  45/1688 [..............................] - ETA: 3s - loss: 0.0325 - accuracy: 0.9868
  67/1688 [>.............................] - ETA: 3s - loss: 0.0273 - accuracy: 0.9893
  89/1688 [>.............................] - ETA: 3s - loss: 0.0254 - accuracy: 0.9909
 111/1688 [>.............................] - ETA: 3s - loss: 0.0262 - accuracy: 0.9910
 133/1688 [=>............................] - ETA: 3s - loss: 0.0262 - accuracy: 0.9908
 156/1688 [=>............................] - ETA: 3s - loss: 0.0275 - accuracy: 0.9900
 178/1688 [==>...........................] - ETA: 3s - loss: 0.0275 - accuracy: 0.9898
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0276 - accuracy: 0.9899
 224/1688 [==>...........................] - ETA: 3s - loss: 0.0260 - accuracy: 0.9904
 247/1688 [===>..........................] - ETA: 3s - loss: 0.0251 - accuracy: 0.9908
 269/1688 [===>..........................] - ETA: 3s - loss: 0.0251 - accuracy: 0.9908
 291/1688 [====>.........................] - ETA: 3s - loss: 0.0255 - accuracy: 0.9908
 314/1688 [====>.........................] - ETA: 3s - loss: 0.0261 - accuracy: 0.9907
 336/1688 [====>.........................] - ETA: 3s - loss: 0.0263 - accuracy: 0.9908
 358/1688 [=====>........................] - ETA: 3s - loss: 0.0274 - accuracy: 0.9904
 380/1688 [=====>........................] - ETA: 3s - loss: 0.0285 - accuracy: 0.9902
 402/1688 [======>.......................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9904
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0285 - accuracy: 0.9903
 447/1688 [======>.......................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9904
 469/1688 [=======>......................] - ETA: 2s - loss: 0.0275 - accuracy: 0.9907
 492/1688 [=======>......................] - ETA: 2s - loss: 0.0286 - accuracy: 0.9903
 515/1688 [========>.....................] - ETA: 2s - loss: 0.0285 - accuracy: 0.9905
 538/1688 [========>.....................] - ETA: 2s - loss: 0.0291 - accuracy: 0.9904
 560/1688 [========>.....................] - ETA: 2s - loss: 0.0291 - accuracy: 0.9905
 583/1688 [=========>....................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9904
 605/1688 [=========>....................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9903
 627/1688 [==========>...................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9902
 650/1688 [==========>...................] - ETA: 2s - loss: 0.0299 - accuracy: 0.9901
 672/1688 [==========>...................] - ETA: 2s - loss: 0.0308 - accuracy: 0.9899
 694/1688 [===========>..................] - ETA: 2s - loss: 0.0309 - accuracy: 0.9899
 716/1688 [===========>..................] - ETA: 2s - loss: 0.0310 - accuracy: 0.9898
 739/1688 [============>.................] - ETA: 2s - loss: 0.0308 - accuracy: 0.9899
 761/1688 [============>.................] - ETA: 2s - loss: 0.0310 - accuracy: 0.9898
 783/1688 [============>.................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9899
 806/1688 [=============>................] - ETA: 2s - loss: 0.0315 - accuracy: 0.9895
 828/1688 [=============>................] - ETA: 1s - loss: 0.0317 - accuracy: 0.9894
 850/1688 [==============>...............] - ETA: 1s - loss: 0.0319 - accuracy: 0.9894
 873/1688 [==============>...............] - ETA: 1s - loss: 0.0322 - accuracy: 0.9893
 895/1688 [==============>...............] - ETA: 1s - loss: 0.0321 - accuracy: 0.9894
 917/1688 [===============>..............] - ETA: 1s - loss: 0.0327 - accuracy: 0.9892
 939/1688 [===============>..............] - ETA: 1s - loss: 0.0326 - accuracy: 0.9892
 962/1688 [================>.............] - ETA: 1s - loss: 0.0328 - accuracy: 0.9892
 984/1688 [================>.............] - ETA: 1s - loss: 0.0328 - accuracy: 0.9891
1006/1688 [================>.............] - ETA: 1s - loss: 0.0330 - accuracy: 0.9891
1028/1688 [=================>............] - ETA: 1s - loss: 0.0329 - accuracy: 0.9891
1050/1688 [=================>............] - ETA: 1s - loss: 0.0327 - accuracy: 0.9891
1072/1688 [==================>...........] - ETA: 1s - loss: 0.0325 - accuracy: 0.9892
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0322 - accuracy: 0.9893
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0325 - accuracy: 0.9893
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0325 - accuracy: 0.9893
1162/1688 [===================>..........] - ETA: 1s - loss: 0.0327 - accuracy: 0.9894
1185/1688 [====================>.........] - ETA: 1s - loss: 0.0326 - accuracy: 0.9894
1206/1688 [====================>.........] - ETA: 1s - loss: 0.0326 - accuracy: 0.9893
1228/1688 [====================>.........] - ETA: 1s - loss: 0.0326 - accuracy: 0.9894
1250/1688 [=====================>........] - ETA: 1s - loss: 0.0329 - accuracy: 0.9893
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0326 - accuracy: 0.9894
1294/1688 [=====================>........] - ETA: 0s - loss: 0.0327 - accuracy: 0.9894
1315/1688 [======================>.......] - ETA: 0s - loss: 0.0329 - accuracy: 0.9894
1338/1688 [======================>.......] - ETA: 0s - loss: 0.0332 - accuracy: 0.9893
1360/1688 [=======================>......] - ETA: 0s - loss: 0.0332 - accuracy: 0.9892
1382/1688 [=======================>......] - ETA: 0s - loss: 0.0331 - accuracy: 0.9893
1405/1688 [=======================>......] - ETA: 0s - loss: 0.0331 - accuracy: 0.9892
1426/1688 [========================>.....] - ETA: 0s - loss: 0.0332 - accuracy: 0.9892
1448/1688 [========================>.....] - ETA: 0s - loss: 0.0332 - accuracy: 0.9892
1470/1688 [=========================>....] - ETA: 0s - loss: 0.0332 - accuracy: 0.9892
1493/1688 [=========================>....] - ETA: 0s - loss: 0.0331 - accuracy: 0.9893
1515/1688 [=========================>....] - ETA: 0s - loss: 0.0333 - accuracy: 0.9893
1538/1688 [==========================>...] - ETA: 0s - loss: 0.0331 - accuracy: 0.9893
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0331 - accuracy: 0.9893
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0332 - accuracy: 0.9892
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0333 - accuracy: 0.9893
1628/1688 [===========================>..] - ETA: 0s - loss: 0.0334 - accuracy: 0.9892
1650/1688 [============================>.] - ETA: 0s - loss: 0.0335 - accuracy: 0.9892
1672/1688 [============================>.] - ETA: 0s - loss: 0.0333 - accuracy: 0.9892
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0336 - accuracy: 0.9891 - val_loss: 0.0535 - val_accuracy: 0.9858
Epoch 6/10

   1/1688 [..............................] - ETA: 4s - loss: 0.0053 - accuracy: 1.0000
  23/1688 [..............................] - ETA: 3s - loss: 0.0261 - accuracy: 0.9891
  45/1688 [..............................] - ETA: 3s - loss: 0.0266 - accuracy: 0.9896
  67/1688 [>.............................] - ETA: 3s - loss: 0.0270 - accuracy: 0.9916
  90/1688 [>.............................] - ETA: 3s - loss: 0.0249 - accuracy: 0.9924
 112/1688 [>.............................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9927
 135/1688 [=>............................] - ETA: 3s - loss: 0.0222 - accuracy: 0.9935
 157/1688 [=>............................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9940
 179/1688 [==>...........................] - ETA: 3s - loss: 0.0189 - accuracy: 0.9946
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0192 - accuracy: 0.9942
 223/1688 [==>...........................] - ETA: 3s - loss: 0.0201 - accuracy: 0.9938
 245/1688 [===>..........................] - ETA: 3s - loss: 0.0204 - accuracy: 0.9936
 266/1688 [===>..........................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9934
 289/1688 [====>.........................] - ETA: 3s - loss: 0.0208 - accuracy: 0.9936
 312/1688 [====>.........................] - ETA: 3s - loss: 0.0206 - accuracy: 0.9934
 334/1688 [====>.........................] - ETA: 3s - loss: 0.0206 - accuracy: 0.9931
 356/1688 [=====>........................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9928
 379/1688 [=====>........................] - ETA: 3s - loss: 0.0210 - accuracy: 0.9927
 401/1688 [======>.......................] - ETA: 2s - loss: 0.0215 - accuracy: 0.9927
 423/1688 [======>.......................] - ETA: 2s - loss: 0.0224 - accuracy: 0.9926
 446/1688 [======>.......................] - ETA: 2s - loss: 0.0220 - accuracy: 0.9927
 469/1688 [=======>......................] - ETA: 2s - loss: 0.0223 - accuracy: 0.9927
 490/1688 [=======>......................] - ETA: 2s - loss: 0.0225 - accuracy: 0.9927
 512/1688 [========>.....................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9924
 534/1688 [========>.....................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9925
 557/1688 [========>.....................] - ETA: 2s - loss: 0.0223 - accuracy: 0.9927
 579/1688 [=========>....................] - ETA: 2s - loss: 0.0221 - accuracy: 0.9927
 601/1688 [=========>....................] - ETA: 2s - loss: 0.0227 - accuracy: 0.9926
 623/1688 [==========>...................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9926
 645/1688 [==========>...................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9926
 668/1688 [==========>...................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9924
 690/1688 [===========>..................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9923
 713/1688 [===========>..................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9920
 735/1688 [============>.................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9918
 757/1688 [============>.................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9918
 779/1688 [============>.................] - ETA: 2s - loss: 0.0244 - accuracy: 0.9919
 801/1688 [=============>................] - ETA: 2s - loss: 0.0247 - accuracy: 0.9917
 824/1688 [=============>................] - ETA: 1s - loss: 0.0247 - accuracy: 0.9917
 847/1688 [==============>...............] - ETA: 1s - loss: 0.0244 - accuracy: 0.9919
 871/1688 [==============>...............] - ETA: 1s - loss: 0.0243 - accuracy: 0.9919
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0247 - accuracy: 0.9918
 916/1688 [===============>..............] - ETA: 1s - loss: 0.0247 - accuracy: 0.9918
 939/1688 [===============>..............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9916
 961/1688 [================>.............] - ETA: 1s - loss: 0.0259 - accuracy: 0.9914
 983/1688 [================>.............] - ETA: 1s - loss: 0.0259 - accuracy: 0.9914
1005/1688 [================>.............] - ETA: 1s - loss: 0.0259 - accuracy: 0.9913
1027/1688 [=================>............] - ETA: 1s - loss: 0.0261 - accuracy: 0.9912
1049/1688 [=================>............] - ETA: 1s - loss: 0.0260 - accuracy: 0.9913
1071/1688 [==================>...........] - ETA: 1s - loss: 0.0261 - accuracy: 0.9913
1093/1688 [==================>...........] - ETA: 1s - loss: 0.0263 - accuracy: 0.9913
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0262 - accuracy: 0.9913
1139/1688 [===================>..........] - ETA: 1s - loss: 0.0263 - accuracy: 0.9911
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0264 - accuracy: 0.9911
1183/1688 [====================>.........] - ETA: 1s - loss: 0.0270 - accuracy: 0.9910
1205/1688 [====================>.........] - ETA: 1s - loss: 0.0276 - accuracy: 0.9907
1228/1688 [====================>.........] - ETA: 1s - loss: 0.0274 - accuracy: 0.9908
1250/1688 [=====================>........] - ETA: 1s - loss: 0.0274 - accuracy: 0.9908
1272/1688 [=====================>........] - ETA: 0s - loss: 0.0273 - accuracy: 0.9909
1295/1688 [======================>.......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9910
1318/1688 [======================>.......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9910
1341/1688 [======================>.......] - ETA: 0s - loss: 0.0275 - accuracy: 0.9910
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0275 - accuracy: 0.9910
1387/1688 [=======================>......] - ETA: 0s - loss: 0.0276 - accuracy: 0.9909
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0278 - accuracy: 0.9909
1431/1688 [========================>.....] - ETA: 0s - loss: 0.0281 - accuracy: 0.9908
1453/1688 [========================>.....] - ETA: 0s - loss: 0.0282 - accuracy: 0.9908
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0281 - accuracy: 0.9907
1498/1688 [=========================>....] - ETA: 0s - loss: 0.0280 - accuracy: 0.9908
1521/1688 [==========================>...] - ETA: 0s - loss: 0.0282 - accuracy: 0.9907
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0282 - accuracy: 0.9907
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0284 - accuracy: 0.9906
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0283 - accuracy: 0.9906
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0284 - accuracy: 0.9906
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0284 - accuracy: 0.9905
1654/1688 [============================>.] - ETA: 0s - loss: 0.0285 - accuracy: 0.9905
1676/1688 [============================>.] - ETA: 0s - loss: 0.0286 - accuracy: 0.9905
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0285 - accuracy: 0.9905 - val_loss: 0.0573 - val_accuracy: 0.9858
Epoch 7/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0593 - accuracy: 0.9688
  23/1688 [..............................] - ETA: 3s - loss: 0.0346 - accuracy: 0.9878
  45/1688 [..............................] - ETA: 3s - loss: 0.0317 - accuracy: 0.9889
  68/1688 [>.............................] - ETA: 3s - loss: 0.0313 - accuracy: 0.9894
  90/1688 [>.............................] - ETA: 3s - loss: 0.0275 - accuracy: 0.9903
 112/1688 [>.............................] - ETA: 3s - loss: 0.0271 - accuracy: 0.9902
 134/1688 [=>............................] - ETA: 3s - loss: 0.0268 - accuracy: 0.9904
 157/1688 [=>............................] - ETA: 3s - loss: 0.0254 - accuracy: 0.9906
 179/1688 [==>...........................] - ETA: 3s - loss: 0.0241 - accuracy: 0.9913
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0232 - accuracy: 0.9916
 222/1688 [==>...........................] - ETA: 3s - loss: 0.0228 - accuracy: 0.9917
 245/1688 [===>..........................] - ETA: 3s - loss: 0.0218 - accuracy: 0.9921
 268/1688 [===>..........................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9924
 290/1688 [====>.........................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9925
 312/1688 [====>.........................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9924
 335/1688 [====>.........................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9926
 357/1688 [=====>........................] - ETA: 3s - loss: 0.0210 - accuracy: 0.9927
 380/1688 [=====>........................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9929
 402/1688 [======>.......................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9931
 424/1688 [======>.......................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9931
 446/1688 [======>.......................] - ETA: 2s - loss: 0.0208 - accuracy: 0.9931
 469/1688 [=======>......................] - ETA: 2s - loss: 0.0211 - accuracy: 0.9932
 492/1688 [=======>......................] - ETA: 2s - loss: 0.0218 - accuracy: 0.9930
 515/1688 [========>.....................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9931
 537/1688 [========>.....................] - ETA: 2s - loss: 0.0213 - accuracy: 0.9932
 560/1688 [========>.....................] - ETA: 2s - loss: 0.0219 - accuracy: 0.9931
 582/1688 [=========>....................] - ETA: 2s - loss: 0.0226 - accuracy: 0.9928
 603/1688 [=========>....................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9927
 625/1688 [==========>...................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9927
 648/1688 [==========>...................] - ETA: 2s - loss: 0.0229 - accuracy: 0.9928
 670/1688 [==========>...................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9926
 693/1688 [===========>..................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9927
 715/1688 [===========>..................] - ETA: 2s - loss: 0.0236 - accuracy: 0.9927
 738/1688 [============>.................] - ETA: 2s - loss: 0.0232 - accuracy: 0.9928
 760/1688 [============>.................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9929
 782/1688 [============>.................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9929
 804/1688 [=============>................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9928
 827/1688 [=============>................] - ETA: 1s - loss: 0.0230 - accuracy: 0.9928
 849/1688 [==============>...............] - ETA: 1s - loss: 0.0229 - accuracy: 0.9929
 872/1688 [==============>...............] - ETA: 1s - loss: 0.0232 - accuracy: 0.9928
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0234 - accuracy: 0.9927
 916/1688 [===============>..............] - ETA: 1s - loss: 0.0234 - accuracy: 0.9927
 938/1688 [===============>..............] - ETA: 1s - loss: 0.0236 - accuracy: 0.9926
 960/1688 [================>.............] - ETA: 1s - loss: 0.0238 - accuracy: 0.9926
 983/1688 [================>.............] - ETA: 1s - loss: 0.0239 - accuracy: 0.9926
1005/1688 [================>.............] - ETA: 1s - loss: 0.0236 - accuracy: 0.9927
1027/1688 [=================>............] - ETA: 1s - loss: 0.0233 - accuracy: 0.9928
1049/1688 [=================>............] - ETA: 1s - loss: 0.0235 - accuracy: 0.9926
1072/1688 [==================>...........] - ETA: 1s - loss: 0.0236 - accuracy: 0.9926
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0236 - accuracy: 0.9926
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0235 - accuracy: 0.9926
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0235 - accuracy: 0.9926
1163/1688 [===================>..........] - ETA: 1s - loss: 0.0235 - accuracy: 0.9927
1186/1688 [====================>.........] - ETA: 1s - loss: 0.0237 - accuracy: 0.9926
1208/1688 [====================>.........] - ETA: 1s - loss: 0.0237 - accuracy: 0.9925
1231/1688 [====================>.........] - ETA: 1s - loss: 0.0240 - accuracy: 0.9925
1253/1688 [=====================>........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9925
1275/1688 [=====================>........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9925
1298/1688 [======================>.......] - ETA: 0s - loss: 0.0238 - accuracy: 0.9925
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0241 - accuracy: 0.9924
1342/1688 [======================>.......] - ETA: 0s - loss: 0.0242 - accuracy: 0.9924
1364/1688 [=======================>......] - ETA: 0s - loss: 0.0243 - accuracy: 0.9923
1386/1688 [=======================>......] - ETA: 0s - loss: 0.0242 - accuracy: 0.9924
1408/1688 [========================>.....] - ETA: 0s - loss: 0.0241 - accuracy: 0.9924
1430/1688 [========================>.....] - ETA: 0s - loss: 0.0240 - accuracy: 0.9924
1451/1688 [========================>.....] - ETA: 0s - loss: 0.0240 - accuracy: 0.9924
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0239 - accuracy: 0.9924
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0240 - accuracy: 0.9924
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0238 - accuracy: 0.9925
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0236 - accuracy: 0.9925
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0236 - accuracy: 0.9926
1583/1688 [===========================>..] - ETA: 0s - loss: 0.0236 - accuracy: 0.9926
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0236 - accuracy: 0.9926
1628/1688 [===========================>..] - ETA: 0s - loss: 0.0237 - accuracy: 0.9925
1649/1688 [============================>.] - ETA: 0s - loss: 0.0237 - accuracy: 0.9925
1671/1688 [============================>.] - ETA: 0s - loss: 0.0236 - accuracy: 0.9925
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0237 - accuracy: 0.9924 - val_loss: 0.0571 - val_accuracy: 0.9855
Epoch 8/10

   1/1688 [..............................] - ETA: 4s - loss: 0.0023 - accuracy: 1.0000
  23/1688 [..............................] - ETA: 3s - loss: 0.0242 - accuracy: 0.9932
  45/1688 [..............................] - ETA: 3s - loss: 0.0197 - accuracy: 0.9931
  67/1688 [>.............................] - ETA: 3s - loss: 0.0162 - accuracy: 0.9939
  89/1688 [>.............................] - ETA: 3s - loss: 0.0146 - accuracy: 0.9944
 111/1688 [>.............................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9949
 133/1688 [=>............................] - ETA: 3s - loss: 0.0161 - accuracy: 0.9948
 156/1688 [=>............................] - ETA: 3s - loss: 0.0156 - accuracy: 0.9950
 178/1688 [==>...........................] - ETA: 3s - loss: 0.0153 - accuracy: 0.9953
 200/1688 [==>...........................] - ETA: 3s - loss: 0.0148 - accuracy: 0.9953
 222/1688 [==>...........................] - ETA: 3s - loss: 0.0151 - accuracy: 0.9952
 244/1688 [===>..........................] - ETA: 3s - loss: 0.0154 - accuracy: 0.9950
 267/1688 [===>..........................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9953
 289/1688 [====>.........................] - ETA: 3s - loss: 0.0145 - accuracy: 0.9954
 312/1688 [====>.........................] - ETA: 3s - loss: 0.0144 - accuracy: 0.9952
 334/1688 [====>.........................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9954
 357/1688 [=====>........................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9954
 380/1688 [=====>........................] - ETA: 3s - loss: 0.0142 - accuracy: 0.9953
 402/1688 [======>.......................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9953
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9954
 447/1688 [======>.......................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9955
 469/1688 [=======>......................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9954
 491/1688 [=======>......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9955
 513/1688 [========>.....................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9954
 535/1688 [========>.....................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9954
 556/1688 [========>.....................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9953
 578/1688 [=========>....................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9953
 600/1688 [=========>....................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9953
 623/1688 [==========>...................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9951
 646/1688 [==========>...................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9951
 668/1688 [==========>...................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9952
 691/1688 [===========>..................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9952
 714/1688 [===========>..................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9952
 736/1688 [============>.................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9953
 760/1688 [============>.................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9952
 782/1688 [============>.................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9951
 805/1688 [=============>................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9951
 827/1688 [=============>................] - ETA: 1s - loss: 0.0158 - accuracy: 0.9950
 849/1688 [==============>...............] - ETA: 1s - loss: 0.0157 - accuracy: 0.9951
 871/1688 [==============>...............] - ETA: 1s - loss: 0.0160 - accuracy: 0.9950
 893/1688 [==============>...............] - ETA: 1s - loss: 0.0158 - accuracy: 0.9951
 915/1688 [===============>..............] - ETA: 1s - loss: 0.0159 - accuracy: 0.9950
 937/1688 [===============>..............] - ETA: 1s - loss: 0.0157 - accuracy: 0.9951
 960/1688 [================>.............] - ETA: 1s - loss: 0.0158 - accuracy: 0.9950
 983/1688 [================>.............] - ETA: 1s - loss: 0.0162 - accuracy: 0.9949
1005/1688 [================>.............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9947
1027/1688 [=================>............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9946
1049/1688 [=================>............] - ETA: 1s - loss: 0.0174 - accuracy: 0.9945
1071/1688 [==================>...........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9944
1094/1688 [==================>...........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9943
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0176 - accuracy: 0.9942
1138/1688 [===================>..........] - ETA: 1s - loss: 0.0177 - accuracy: 0.9942
1161/1688 [===================>..........] - ETA: 1s - loss: 0.0181 - accuracy: 0.9941
1183/1688 [====================>.........] - ETA: 1s - loss: 0.0184 - accuracy: 0.9941
1205/1688 [====================>.........] - ETA: 1s - loss: 0.0184 - accuracy: 0.9941
1227/1688 [====================>.........] - ETA: 1s - loss: 0.0182 - accuracy: 0.9942
1249/1688 [=====================>........] - ETA: 1s - loss: 0.0182 - accuracy: 0.9942
1271/1688 [=====================>........] - ETA: 0s - loss: 0.0181 - accuracy: 0.9942
1294/1688 [=====================>........] - ETA: 0s - loss: 0.0179 - accuracy: 0.9943
1317/1688 [======================>.......] - ETA: 0s - loss: 0.0181 - accuracy: 0.9943
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0181 - accuracy: 0.9943
1361/1688 [=======================>......] - ETA: 0s - loss: 0.0181 - accuracy: 0.9942
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0183 - accuracy: 0.9941
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0184 - accuracy: 0.9941
1428/1688 [========================>.....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9940
1450/1688 [========================>.....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9939
1472/1688 [=========================>....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9939
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9939
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0187 - accuracy: 0.9939
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0186 - accuracy: 0.9938
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0187 - accuracy: 0.9938
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0188 - accuracy: 0.9938
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0189 - accuracy: 0.9938
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0190 - accuracy: 0.9936
1652/1688 [============================>.] - ETA: 0s - loss: 0.0189 - accuracy: 0.9936
1674/1688 [============================>.] - ETA: 0s - loss: 0.0192 - accuracy: 0.9936
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0191 - accuracy: 0.9936 - val_loss: 0.0642 - val_accuracy: 0.9833
Epoch 9/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0669 - accuracy: 0.9688
  24/1688 [..............................] - ETA: 3s - loss: 0.0283 - accuracy: 0.9909
  47/1688 [..............................] - ETA: 3s - loss: 0.0196 - accuracy: 0.9934
  69/1688 [>.............................] - ETA: 3s - loss: 0.0182 - accuracy: 0.9946
  91/1688 [>.............................] - ETA: 3s - loss: 0.0167 - accuracy: 0.9948
 113/1688 [=>............................] - ETA: 3s - loss: 0.0160 - accuracy: 0.9950
 136/1688 [=>............................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9949
 158/1688 [=>............................] - ETA: 3s - loss: 0.0147 - accuracy: 0.9951
 180/1688 [==>...........................] - ETA: 3s - loss: 0.0154 - accuracy: 0.9948
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0164 - accuracy: 0.9941
 222/1688 [==>...........................] - ETA: 3s - loss: 0.0158 - accuracy: 0.9942
 244/1688 [===>..........................] - ETA: 3s - loss: 0.0167 - accuracy: 0.9940
 266/1688 [===>..........................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9938
 288/1688 [====>.........................] - ETA: 3s - loss: 0.0177 - accuracy: 0.9939
 310/1688 [====>.........................] - ETA: 3s - loss: 0.0168 - accuracy: 0.9942
 332/1688 [====>.........................] - ETA: 3s - loss: 0.0173 - accuracy: 0.9938
 354/1688 [=====>........................] - ETA: 3s - loss: 0.0179 - accuracy: 0.9937
 376/1688 [=====>........................] - ETA: 3s - loss: 0.0178 - accuracy: 0.9938
 398/1688 [======>.......................] - ETA: 2s - loss: 0.0186 - accuracy: 0.9935
 420/1688 [======>.......................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9937
 442/1688 [======>.......................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9939
 464/1688 [=======>......................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9940
 486/1688 [=======>......................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9941
 508/1688 [========>.....................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9941
 531/1688 [========>.....................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9939
 553/1688 [========>.....................] - ETA: 2s - loss: 0.0177 - accuracy: 0.9941
 575/1688 [=========>....................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9942
 598/1688 [=========>....................] - ETA: 2s - loss: 0.0169 - accuracy: 0.9944
 621/1688 [==========>...................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9945
 643/1688 [==========>...................] - ETA: 2s - loss: 0.0166 - accuracy: 0.9945
 665/1688 [==========>...................] - ETA: 2s - loss: 0.0164 - accuracy: 0.9945
 687/1688 [===========>..................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9946
 709/1688 [===========>..................] - ETA: 2s - loss: 0.0163 - accuracy: 0.9946
 731/1688 [===========>..................] - ETA: 2s - loss: 0.0161 - accuracy: 0.9947
 753/1688 [============>.................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9947
 775/1688 [============>.................] - ETA: 2s - loss: 0.0162 - accuracy: 0.9947
 798/1688 [=============>................] - ETA: 2s - loss: 0.0160 - accuracy: 0.9948
 820/1688 [=============>................] - ETA: 2s - loss: 0.0165 - accuracy: 0.9947
 843/1688 [=============>................] - ETA: 1s - loss: 0.0166 - accuracy: 0.9947
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0164 - accuracy: 0.9947
 888/1688 [==============>...............] - ETA: 1s - loss: 0.0164 - accuracy: 0.9948
 910/1688 [===============>..............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9947
 933/1688 [===============>..............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9947
 955/1688 [===============>..............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9947
 978/1688 [================>.............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9947
1001/1688 [================>.............] - ETA: 1s - loss: 0.0164 - accuracy: 0.9947
1023/1688 [=================>............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9946
1045/1688 [=================>............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9946
1067/1688 [=================>............] - ETA: 1s - loss: 0.0172 - accuracy: 0.9944
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0172 - accuracy: 0.9944
1112/1688 [==================>...........] - ETA: 1s - loss: 0.0175 - accuracy: 0.9943
1135/1688 [===================>..........] - ETA: 1s - loss: 0.0174 - accuracy: 0.9943
1157/1688 [===================>..........] - ETA: 1s - loss: 0.0173 - accuracy: 0.9943
1179/1688 [===================>..........] - ETA: 1s - loss: 0.0171 - accuracy: 0.9944
1201/1688 [====================>.........] - ETA: 1s - loss: 0.0170 - accuracy: 0.9945
1224/1688 [====================>.........] - ETA: 1s - loss: 0.0172 - accuracy: 0.9943
1246/1688 [=====================>........] - ETA: 1s - loss: 0.0172 - accuracy: 0.9943
1268/1688 [=====================>........] - ETA: 0s - loss: 0.0171 - accuracy: 0.9943
1291/1688 [=====================>........] - ETA: 0s - loss: 0.0170 - accuracy: 0.9944
1313/1688 [======================>.......] - ETA: 0s - loss: 0.0170 - accuracy: 0.9944
1336/1688 [======================>.......] - ETA: 0s - loss: 0.0173 - accuracy: 0.9943
1357/1688 [=======================>......] - ETA: 0s - loss: 0.0173 - accuracy: 0.9943
1379/1688 [=======================>......] - ETA: 0s - loss: 0.0175 - accuracy: 0.9943
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0175 - accuracy: 0.9943
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0175 - accuracy: 0.9943
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0174 - accuracy: 0.9943
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0176 - accuracy: 0.9942
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0177 - accuracy: 0.9941
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0177 - accuracy: 0.9941
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0178 - accuracy: 0.9941
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0176 - accuracy: 0.9942
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0178 - accuracy: 0.9942
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0177 - accuracy: 0.9942
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0176 - accuracy: 0.9942
1646/1688 [============================>.] - ETA: 0s - loss: 0.0175 - accuracy: 0.9942
1668/1688 [============================>.] - ETA: 0s - loss: 0.0175 - accuracy: 0.9942
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0173 - accuracy: 0.9943 - val_loss: 0.0540 - val_accuracy: 0.9873
Epoch 10/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0053 - accuracy: 1.0000
  23/1688 [..............................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9932
  45/1688 [..............................] - ETA: 3s - loss: 0.0146 - accuracy: 0.9958
  67/1688 [>.............................] - ETA: 3s - loss: 0.0121 - accuracy: 0.9967
  90/1688 [>.............................] - ETA: 3s - loss: 0.0110 - accuracy: 0.9972
 112/1688 [>.............................] - ETA: 3s - loss: 0.0134 - accuracy: 0.9972
 134/1688 [=>............................] - ETA: 3s - loss: 0.0149 - accuracy: 0.9963
 156/1688 [=>............................] - ETA: 3s - loss: 0.0146 - accuracy: 0.9966
 178/1688 [==>...........................] - ETA: 3s - loss: 0.0166 - accuracy: 0.9956
 200/1688 [==>...........................] - ETA: 3s - loss: 0.0166 - accuracy: 0.9956
 223/1688 [==>...........................] - ETA: 3s - loss: 0.0164 - accuracy: 0.9957
 245/1688 [===>..........................] - ETA: 3s - loss: 0.0162 - accuracy: 0.9955
 267/1688 [===>..........................] - ETA: 3s - loss: 0.0153 - accuracy: 0.9958
 289/1688 [====>.........................] - ETA: 3s - loss: 0.0146 - accuracy: 0.9960
 311/1688 [====>.........................] - ETA: 3s - loss: 0.0140 - accuracy: 0.9963
 333/1688 [====>.........................] - ETA: 3s - loss: 0.0141 - accuracy: 0.9962
 355/1688 [=====>........................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9961
 377/1688 [=====>........................] - ETA: 3s - loss: 0.0143 - accuracy: 0.9959
 400/1688 [======>.......................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9954
 423/1688 [======>.......................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9954
 445/1688 [======>.......................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9956
 466/1688 [=======>......................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9954
 487/1688 [=======>......................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9956
 510/1688 [========>.....................] - ETA: 2s - loss: 0.0155 - accuracy: 0.9954
 533/1688 [========>.....................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9955
 555/1688 [========>.....................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9955
 577/1688 [=========>....................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9955
 599/1688 [=========>....................] - ETA: 2s - loss: 0.0151 - accuracy: 0.9955
 621/1688 [==========>...................] - ETA: 2s - loss: 0.0154 - accuracy: 0.9954
 644/1688 [==========>...................] - ETA: 2s - loss: 0.0153 - accuracy: 0.9954
 666/1688 [==========>...................] - ETA: 2s - loss: 0.0152 - accuracy: 0.9954
 688/1688 [===========>..................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9955
 710/1688 [===========>..................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9956
 733/1688 [============>.................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9956
 755/1688 [============>.................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9956
 778/1688 [============>.................] - ETA: 2s - loss: 0.0146 - accuracy: 0.9956
 801/1688 [=============>................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9954
 823/1688 [=============>................] - ETA: 1s - loss: 0.0145 - accuracy: 0.9955
 846/1688 [==============>...............] - ETA: 1s - loss: 0.0142 - accuracy: 0.9956
 868/1688 [==============>...............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9955
 891/1688 [==============>...............] - ETA: 1s - loss: 0.0142 - accuracy: 0.9956
 913/1688 [===============>..............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9954
 935/1688 [===============>..............] - ETA: 1s - loss: 0.0142 - accuracy: 0.9955
 958/1688 [================>.............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9955
 980/1688 [================>.............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9953
1002/1688 [================>.............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9953
1024/1688 [=================>............] - ETA: 1s - loss: 0.0145 - accuracy: 0.9952
1046/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9951
1069/1688 [=================>............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9951
1091/1688 [==================>...........] - ETA: 1s - loss: 0.0151 - accuracy: 0.9950
1113/1688 [==================>...........] - ETA: 1s - loss: 0.0150 - accuracy: 0.9949
1134/1688 [===================>..........] - ETA: 1s - loss: 0.0149 - accuracy: 0.9949
1156/1688 [===================>..........] - ETA: 1s - loss: 0.0149 - accuracy: 0.9949
1178/1688 [===================>..........] - ETA: 1s - loss: 0.0147 - accuracy: 0.9950
1200/1688 [====================>.........] - ETA: 1s - loss: 0.0145 - accuracy: 0.9951
1223/1688 [====================>.........] - ETA: 1s - loss: 0.0145 - accuracy: 0.9951
1246/1688 [=====================>........] - ETA: 1s - loss: 0.0145 - accuracy: 0.9950
1268/1688 [=====================>........] - ETA: 0s - loss: 0.0145 - accuracy: 0.9950
1291/1688 [=====================>........] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1313/1688 [======================>.......] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1335/1688 [======================>.......] - ETA: 0s - loss: 0.0146 - accuracy: 0.9950
1357/1688 [=======================>......] - ETA: 0s - loss: 0.0145 - accuracy: 0.9951
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0143 - accuracy: 0.9952
1402/1688 [=======================>......] - ETA: 0s - loss: 0.0144 - accuracy: 0.9952
1424/1688 [========================>.....] - ETA: 0s - loss: 0.0143 - accuracy: 0.9952
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0142 - accuracy: 0.9952
1469/1688 [=========================>....] - ETA: 0s - loss: 0.0141 - accuracy: 0.9952
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0141 - accuracy: 0.9952
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0140 - accuracy: 0.9953
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0139 - accuracy: 0.9953
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0141 - accuracy: 0.9952
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0141 - accuracy: 0.9952
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0141 - accuracy: 0.9951
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0144 - accuracy: 0.9951
1646/1688 [============================>.] - ETA: 0s - loss: 0.0145 - accuracy: 0.9951
1669/1688 [============================>.] - ETA: 0s - loss: 0.0145 - accuracy: 0.9951
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0147 - accuracy: 0.9950 - val_loss: 0.0549 - val_accuracy: 0.9878
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Test accuracy: 0.98580002784729

2. Quantize

2.1. 8-bit quantization

An Akida accelerator processes 8 or 4-bits integer activations and weights. Therefore, the floating point TF-Keras model must be quantized in preparation to run on an Akida accelerator.

The QuantizeML quantize function can be used to quantize a TF-Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating point TF-Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.

The quantization process results in a TF-Keras model with custom QuantizeML quantized layers substituted for the original TF-Keras layers. All TF-Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function applies several transformations to the original model. For example, it folds the batch normalization layers into the corresponding neural layers. The new weights are computed according to this folding operation.

from quantizeml.models import quantize, QuantizationParams

qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
/usr/local/lib/python3.11/dist-packages/quantizeml/models/quantize.py:489: UserWarning: Quantizing per-axis with random calibration samples is not accurate. Set QuantizationParams.per_tensor_activations=True when calibrating with random samples.
  warnings.warn("Quantizing per-axis with random calibration samples is not accurate. "

   1/1024 [..............................] - ETA: 2:28
  51/1024 [>.............................] - ETA: 0s  
 103/1024 [==>...........................] - ETA: 0s
 154/1024 [===>..........................] - ETA: 0s
 206/1024 [=====>........................] - ETA: 0s
 258/1024 [======>.......................] - ETA: 0s
 310/1024 [========>.....................] - ETA: 0s
 361/1024 [=========>....................] - ETA: 0s
 412/1024 [===========>..................] - ETA: 0s
 464/1024 [============>.................] - ETA: 0s
 516/1024 [==============>...............] - ETA: 0s
 568/1024 [===============>..............] - ETA: 0s
 620/1024 [=================>............] - ETA: 0s
 672/1024 [==================>...........] - ETA: 0s
 724/1024 [====================>.........] - ETA: 0s
 776/1024 [=====================>........] - ETA: 0s
 828/1024 [=======================>......] - ETA: 0s
 879/1024 [========================>.....] - ETA: 0s
 931/1024 [==========================>...] - ETA: 0s
 982/1024 [===========================>..] - ETA: 0s
1024/1024 [==============================] - 1s 977us/step
model_quantized.summary()
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling_input (InputLaye  [(None, 28, 28, 1)]       0
 r)

 rescaling (QuantizedRescal  (None, 28, 28, 1)         0
 ing)

 conv2d (QuantizedConv2D)    (None, 13, 13, 32)        320

 re_lu (QuantizedReLU)       (None, 13, 13, 32)        64

 depthwise_conv2d (Quantize  (None, 7, 7, 32)          384
 dDepthwiseConv2D)

 conv2d_1 (QuantizedConv2D)  (None, 7, 7, 64)          2112

 re_lu_1 (QuantizedReLU)     (None, 7, 7, 64)          128

 flatten (QuantizedFlatten)  (None, 3136)              0

 dense (QuantizedDense)      (None, 10)                31370

 dequantizer (Dequantizer)   (None, 10)                0

=================================================================
Total params: 34378 (134.29 KB)
Trainable params: 34122 (133.29 KB)
Non-trainable params: 256 (1.00 KB)
_________________________________________________________________

Note

Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.

Check the quantized model accuracy.

def compile_evaluate(model):
    """ Compiles and evaluates the model, then return accuracy score. """
    model.compile(metrics=['accuracy'])
    return model.evaluate(x_test, y_test, verbose=0)[1]


print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 8-bit quantization: 0.9790999889373779

2.2. Effect of calibration

The previous call to quantize was made with random samples for calibration (default parameters). While the observed drop in accuracy is minimal, that is around 1%, it can be worse on more complex models. Therefore, it is advised to use a set of real samples from the training set for calibration during a call to quantize. Note that this remains a calibration step rather than a training step in that no output labels are required. Furthermore, any relevant data could be used for calibration. The recommended settings for calibration that are widely used to obtain the zoo performance are:

  • 1024 samples

  • a batch size of 100

  • 2 epochs

model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)
 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 1ms/step

Check the accuracy for the quantized and calibrated model.

print('Test accuracy after calibration:', compile_evaluate(model_quantized))
Test accuracy after calibration: 0.9842000007629395

Calibrating with real samples on this model recovers the initial float accuracy.

2.3. 4-bit quantization

The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.

The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.

qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)
 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 1ms/step

Check the 4-bit quantized accuracy.

print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 4-bit quantization: 0.9814000129699707

2.4. Model fine tuning (Quantization Aware Training)

When a model suffers from an accuracy drop after quantization, fine tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.

Note that since this is a fine tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-4),
    metrics=['accuracy'])

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
Epoch 1/5

   1/1688 [..............................] - ETA: 1:48:25 - loss: 9.6093e-04 - accuracy: 1.0000
  10/1688 [..............................] - ETA: 10s - loss: 0.0144 - accuracy: 0.9969        
  19/1688 [..............................] - ETA: 10s - loss: 0.0151 - accuracy: 0.9951
  28/1688 [..............................] - ETA: 10s - loss: 0.0157 - accuracy: 0.9944
  37/1688 [..............................] - ETA: 10s - loss: 0.0173 - accuracy: 0.9941
  45/1688 [..............................] - ETA: 10s - loss: 0.0173 - accuracy: 0.9944
  54/1688 [..............................] - ETA: 10s - loss: 0.0158 - accuracy: 0.9954
  62/1688 [>.............................] - ETA: 10s - loss: 0.0162 - accuracy: 0.9950
  71/1688 [>.............................] - ETA: 10s - loss: 0.0177 - accuracy: 0.9947
  80/1688 [>.............................] - ETA: 10s - loss: 0.0176 - accuracy: 0.9941
  89/1688 [>.............................] - ETA: 9s - loss: 0.0164 - accuracy: 0.9947 
  98/1688 [>.............................] - ETA: 9s - loss: 0.0156 - accuracy: 0.9949
 106/1688 [>.............................] - ETA: 9s - loss: 0.0157 - accuracy: 0.9944
 115/1688 [=>............................] - ETA: 9s - loss: 0.0154 - accuracy: 0.9946
 124/1688 [=>............................] - ETA: 9s - loss: 0.0148 - accuracy: 0.9947
 133/1688 [=>............................] - ETA: 9s - loss: 0.0156 - accuracy: 0.9944
 142/1688 [=>............................] - ETA: 9s - loss: 0.0152 - accuracy: 0.9945
 151/1688 [=>............................] - ETA: 9s - loss: 0.0152 - accuracy: 0.9942
 160/1688 [=>............................] - ETA: 9s - loss: 0.0154 - accuracy: 0.9941
 169/1688 [==>...........................] - ETA: 9s - loss: 0.0158 - accuracy: 0.9939
 177/1688 [==>...........................] - ETA: 9s - loss: 0.0152 - accuracy: 0.9942
 186/1688 [==>...........................] - ETA: 9s - loss: 0.0161 - accuracy: 0.9938
 195/1688 [==>...........................] - ETA: 9s - loss: 0.0157 - accuracy: 0.9939
 204/1688 [==>...........................] - ETA: 9s - loss: 0.0154 - accuracy: 0.9940
 212/1688 [==>...........................] - ETA: 9s - loss: 0.0158 - accuracy: 0.9938
 221/1688 [==>...........................] - ETA: 9s - loss: 0.0153 - accuracy: 0.9941
 230/1688 [===>..........................] - ETA: 8s - loss: 0.0150 - accuracy: 0.9943
 239/1688 [===>..........................] - ETA: 8s - loss: 0.0146 - accuracy: 0.9944
 248/1688 [===>..........................] - ETA: 8s - loss: 0.0146 - accuracy: 0.9942
 257/1688 [===>..........................] - ETA: 8s - loss: 0.0144 - accuracy: 0.9944
 266/1688 [===>..........................] - ETA: 8s - loss: 0.0142 - accuracy: 0.9945
 275/1688 [===>..........................] - ETA: 8s - loss: 0.0146 - accuracy: 0.9944
 284/1688 [====>.........................] - ETA: 8s - loss: 0.0144 - accuracy: 0.9945
 293/1688 [====>.........................] - ETA: 8s - loss: 0.0141 - accuracy: 0.9947
 302/1688 [====>.........................] - ETA: 8s - loss: 0.0137 - accuracy: 0.9948
 310/1688 [====>.........................] - ETA: 8s - loss: 0.0134 - accuracy: 0.9950
 319/1688 [====>.........................] - ETA: 8s - loss: 0.0132 - accuracy: 0.9950
 328/1688 [====>.........................] - ETA: 8s - loss: 0.0131 - accuracy: 0.9950
 337/1688 [====>.........................] - ETA: 8s - loss: 0.0128 - accuracy: 0.9951
 346/1688 [=====>........................] - ETA: 8s - loss: 0.0126 - accuracy: 0.9952
 355/1688 [=====>........................] - ETA: 8s - loss: 0.0130 - accuracy: 0.9952
 363/1688 [=====>........................] - ETA: 8s - loss: 0.0131 - accuracy: 0.9951
 372/1688 [=====>........................] - ETA: 8s - loss: 0.0129 - accuracy: 0.9950
 381/1688 [=====>........................] - ETA: 8s - loss: 0.0128 - accuracy: 0.9951
 390/1688 [=====>........................] - ETA: 7s - loss: 0.0127 - accuracy: 0.9952
 399/1688 [======>.......................] - ETA: 7s - loss: 0.0126 - accuracy: 0.9952
 408/1688 [======>.......................] - ETA: 7s - loss: 0.0126 - accuracy: 0.9953
 417/1688 [======>.......................] - ETA: 7s - loss: 0.0125 - accuracy: 0.9953
 425/1688 [======>.......................] - ETA: 7s - loss: 0.0126 - accuracy: 0.9952
 434/1688 [======>.......................] - ETA: 7s - loss: 0.0127 - accuracy: 0.9952
 443/1688 [======>.......................] - ETA: 7s - loss: 0.0129 - accuracy: 0.9951
 452/1688 [=======>......................] - ETA: 7s - loss: 0.0127 - accuracy: 0.9952
 461/1688 [=======>......................] - ETA: 7s - loss: 0.0126 - accuracy: 0.9953
 470/1688 [=======>......................] - ETA: 7s - loss: 0.0127 - accuracy: 0.9953
 479/1688 [=======>......................] - ETA: 7s - loss: 0.0125 - accuracy: 0.9954
 487/1688 [=======>......................] - ETA: 7s - loss: 0.0123 - accuracy: 0.9954
 496/1688 [=======>......................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9955
 505/1688 [=======>......................] - ETA: 7s - loss: 0.0121 - accuracy: 0.9955
 513/1688 [========>.....................] - ETA: 7s - loss: 0.0120 - accuracy: 0.9956
 522/1688 [========>.....................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9954
 531/1688 [========>.....................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9954
 539/1688 [========>.....................] - ETA: 7s - loss: 0.0120 - accuracy: 0.9955
 547/1688 [========>.....................] - ETA: 7s - loss: 0.0119 - accuracy: 0.9955
 556/1688 [========>.....................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9956
 565/1688 [=========>....................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9956
 573/1688 [=========>....................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9955
 582/1688 [=========>....................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9955
 590/1688 [=========>....................] - ETA: 6s - loss: 0.0121 - accuracy: 0.9954
 599/1688 [=========>....................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9955
 608/1688 [=========>....................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9954
 617/1688 [=========>....................] - ETA: 6s - loss: 0.0120 - accuracy: 0.9954
 626/1688 [==========>...................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9955
 635/1688 [==========>...................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9956
 644/1688 [==========>...................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9956
 653/1688 [==========>...................] - ETA: 6s - loss: 0.0119 - accuracy: 0.9956
 662/1688 [==========>...................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9956
 671/1688 [==========>...................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9956
 679/1688 [===========>..................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9957
 688/1688 [===========>..................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9957
 697/1688 [===========>..................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9957
 705/1688 [===========>..................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9957
 714/1688 [===========>..................] - ETA: 5s - loss: 0.0116 - accuracy: 0.9958
 723/1688 [===========>..................] - ETA: 5s - loss: 0.0115 - accuracy: 0.9958
 732/1688 [============>.................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9959
 741/1688 [============>.................] - ETA: 5s - loss: 0.0115 - accuracy: 0.9959
 749/1688 [============>.................] - ETA: 5s - loss: 0.0114 - accuracy: 0.9959
 758/1688 [============>.................] - ETA: 5s - loss: 0.0113 - accuracy: 0.9959
 767/1688 [============>.................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9960
 776/1688 [============>.................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9960
 785/1688 [============>.................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9960
 794/1688 [=============>................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9960
 803/1688 [=============>................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9960
 812/1688 [=============>................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9960
 821/1688 [=============>................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9960
 829/1688 [=============>................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9960
 838/1688 [=============>................] - ETA: 5s - loss: 0.0111 - accuracy: 0.9960
 846/1688 [==============>...............] - ETA: 5s - loss: 0.0112 - accuracy: 0.9960
 855/1688 [==============>...............] - ETA: 5s - loss: 0.0112 - accuracy: 0.9960
 864/1688 [==============>...............] - ETA: 5s - loss: 0.0112 - accuracy: 0.9960
 873/1688 [==============>...............] - ETA: 5s - loss: 0.0111 - accuracy: 0.9960
 882/1688 [==============>...............] - ETA: 4s - loss: 0.0110 - accuracy: 0.9961
 891/1688 [==============>...............] - ETA: 4s - loss: 0.0110 - accuracy: 0.9960
 899/1688 [==============>...............] - ETA: 4s - loss: 0.0110 - accuracy: 0.9960
 907/1688 [===============>..............] - ETA: 4s - loss: 0.0110 - accuracy: 0.9961
 916/1688 [===============>..............] - ETA: 4s - loss: 0.0109 - accuracy: 0.9961
 925/1688 [===============>..............] - ETA: 4s - loss: 0.0109 - accuracy: 0.9961
 934/1688 [===============>..............] - ETA: 4s - loss: 0.0108 - accuracy: 0.9962
 942/1688 [===============>..............] - ETA: 4s - loss: 0.0108 - accuracy: 0.9962
 950/1688 [===============>..............] - ETA: 4s - loss: 0.0107 - accuracy: 0.9962
 959/1688 [================>.............] - ETA: 4s - loss: 0.0108 - accuracy: 0.9962
 968/1688 [================>.............] - ETA: 4s - loss: 0.0108 - accuracy: 0.9962
 976/1688 [================>.............] - ETA: 4s - loss: 0.0107 - accuracy: 0.9962
 985/1688 [================>.............] - ETA: 4s - loss: 0.0107 - accuracy: 0.9963
 994/1688 [================>.............] - ETA: 4s - loss: 0.0106 - accuracy: 0.9963
1003/1688 [================>.............] - ETA: 4s - loss: 0.0106 - accuracy: 0.9963
1012/1688 [================>.............] - ETA: 4s - loss: 0.0105 - accuracy: 0.9963
1021/1688 [=================>............] - ETA: 4s - loss: 0.0105 - accuracy: 0.9963
1030/1688 [=================>............] - ETA: 4s - loss: 0.0105 - accuracy: 0.9963
1039/1688 [=================>............] - ETA: 3s - loss: 0.0105 - accuracy: 0.9963
1047/1688 [=================>............] - ETA: 3s - loss: 0.0104 - accuracy: 0.9963
1056/1688 [=================>............] - ETA: 3s - loss: 0.0104 - accuracy: 0.9963
1065/1688 [=================>............] - ETA: 3s - loss: 0.0104 - accuracy: 0.9963
1074/1688 [==================>...........] - ETA: 3s - loss: 0.0104 - accuracy: 0.9964
1082/1688 [==================>...........] - ETA: 3s - loss: 0.0103 - accuracy: 0.9964
1091/1688 [==================>...........] - ETA: 3s - loss: 0.0104 - accuracy: 0.9964
1099/1688 [==================>...........] - ETA: 3s - loss: 0.0104 - accuracy: 0.9964
1108/1688 [==================>...........] - ETA: 3s - loss: 0.0103 - accuracy: 0.9964
1116/1688 [==================>...........] - ETA: 3s - loss: 0.0103 - accuracy: 0.9964
1125/1688 [==================>...........] - ETA: 3s - loss: 0.0103 - accuracy: 0.9964
1134/1688 [===================>..........] - ETA: 3s - loss: 0.0103 - accuracy: 0.9964
1142/1688 [===================>..........] - ETA: 3s - loss: 0.0102 - accuracy: 0.9964
1151/1688 [===================>..........] - ETA: 3s - loss: 0.0102 - accuracy: 0.9964
1160/1688 [===================>..........] - ETA: 3s - loss: 0.0102 - accuracy: 0.9964
1168/1688 [===================>..........] - ETA: 3s - loss: 0.0102 - accuracy: 0.9964
1177/1688 [===================>..........] - ETA: 3s - loss: 0.0104 - accuracy: 0.9964
1186/1688 [====================>.........] - ETA: 3s - loss: 0.0104 - accuracy: 0.9964
1195/1688 [====================>.........] - ETA: 3s - loss: 0.0104 - accuracy: 0.9963
1204/1688 [====================>.........] - ETA: 2s - loss: 0.0104 - accuracy: 0.9964
1213/1688 [====================>.........] - ETA: 2s - loss: 0.0103 - accuracy: 0.9964
1221/1688 [====================>.........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1230/1688 [====================>.........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1239/1688 [=====================>........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1247/1688 [=====================>........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1255/1688 [=====================>........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1264/1688 [=====================>........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1273/1688 [=====================>........] - ETA: 2s - loss: 0.0103 - accuracy: 0.9964
1281/1688 [=====================>........] - ETA: 2s - loss: 0.0103 - accuracy: 0.9964
1289/1688 [=====================>........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1297/1688 [======================>.......] - ETA: 2s - loss: 0.0101 - accuracy: 0.9964
1305/1688 [======================>.......] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1313/1688 [======================>.......] - ETA: 2s - loss: 0.0102 - accuracy: 0.9964
1322/1688 [======================>.......] - ETA: 2s - loss: 0.0101 - accuracy: 0.9964
1331/1688 [======================>.......] - ETA: 2s - loss: 0.0101 - accuracy: 0.9965
1340/1688 [======================>.......] - ETA: 2s - loss: 0.0101 - accuracy: 0.9964
1349/1688 [======================>.......] - ETA: 2s - loss: 0.0101 - accuracy: 0.9964
1358/1688 [=======================>......] - ETA: 2s - loss: 0.0101 - accuracy: 0.9964
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9964
1375/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9965
1384/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9965
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9964
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9965
1410/1688 [========================>.....] - ETA: 1s - loss: 0.0100 - accuracy: 0.9965
1419/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9964
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0100 - accuracy: 0.9964
1437/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9964
1446/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9964
1455/1688 [========================>.....] - ETA: 1s - loss: 0.0100 - accuracy: 0.9964
1464/1688 [=========================>....] - ETA: 1s - loss: 0.0100 - accuracy: 0.9964
1473/1688 [=========================>....] - ETA: 1s - loss: 0.0100 - accuracy: 0.9964
1482/1688 [=========================>....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9964
1491/1688 [=========================>....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9964
1500/1688 [=========================>....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9964
1509/1688 [=========================>....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9965
1518/1688 [=========================>....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9964
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0099 - accuracy: 0.9965
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1553/1688 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1588/1688 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1597/1688 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1606/1688 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9966
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9966
1633/1688 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1642/1688 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1650/1688 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9966
1659/1688 [============================>.] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1668/1688 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1676/1688 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9965
1684/1688 [============================>.] - ETA: 0s - loss: 0.0098 - accuracy: 0.9965
1688/1688 [==============================] - 16s 7ms/step - loss: 0.0098 - accuracy: 0.9965 - val_loss: 0.0473 - val_accuracy: 0.9887
Epoch 2/5

   1/1688 [..............................] - ETA: 10s - loss: 0.0015 - accuracy: 1.0000
   9/1688 [..............................] - ETA: 10s - loss: 0.0020 - accuracy: 1.0000
  17/1688 [..............................] - ETA: 10s - loss: 0.0055 - accuracy: 0.9982
  26/1688 [..............................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9988
  35/1688 [..............................] - ETA: 10s - loss: 0.0040 - accuracy: 0.9991
  44/1688 [..............................] - ETA: 10s - loss: 0.0037 - accuracy: 0.9993
  53/1688 [..............................] - ETA: 10s - loss: 0.0035 - accuracy: 0.9994
  61/1688 [>.............................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9990
  70/1688 [>.............................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9987
  79/1688 [>.............................] - ETA: 10s - loss: 0.0052 - accuracy: 0.9988
  88/1688 [>.............................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9989 
  97/1688 [>.............................] - ETA: 9s - loss: 0.0051 - accuracy: 0.9984
 106/1688 [>.............................] - ETA: 9s - loss: 0.0050 - accuracy: 0.9985
 115/1688 [=>............................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9986
 124/1688 [=>............................] - ETA: 9s - loss: 0.0051 - accuracy: 0.9985
 133/1688 [=>............................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9986
 141/1688 [=>............................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9987
 149/1688 [=>............................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9985
 158/1688 [=>............................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9984
 167/1688 [=>............................] - ETA: 9s - loss: 0.0051 - accuracy: 0.9985
 176/1688 [==>...........................] - ETA: 9s - loss: 0.0050 - accuracy: 0.9986
 185/1688 [==>...........................] - ETA: 9s - loss: 0.0056 - accuracy: 0.9985
 194/1688 [==>...........................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9986
 203/1688 [==>...........................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9986
 212/1688 [==>...........................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9985
 220/1688 [==>...........................] - ETA: 9s - loss: 0.0052 - accuracy: 0.9986
 228/1688 [===>..........................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9985
 237/1688 [===>..........................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9983
 246/1688 [===>..........................] - ETA: 8s - loss: 0.0056 - accuracy: 0.9982
 255/1688 [===>..........................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9983
 264/1688 [===>..........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9983
 273/1688 [===>..........................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9983
 282/1688 [====>.........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9983
 291/1688 [====>.........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9984
 300/1688 [====>.........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9983
 308/1688 [====>.........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9983
 317/1688 [====>.........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9982
 326/1688 [====>.........................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9983
 335/1688 [====>.........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9982
 344/1688 [=====>........................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9982
 353/1688 [=====>........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9982
 362/1688 [=====>........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9983
 371/1688 [=====>........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9982
 380/1688 [=====>........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9983
 389/1688 [=====>........................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9982
 398/1688 [======>.......................] - ETA: 7s - loss: 0.0056 - accuracy: 0.9982
 407/1688 [======>.......................] - ETA: 7s - loss: 0.0057 - accuracy: 0.9982
 415/1688 [======>.......................] - ETA: 7s - loss: 0.0058 - accuracy: 0.9981
 424/1688 [======>.......................] - ETA: 7s - loss: 0.0058 - accuracy: 0.9981
 433/1688 [======>.......................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9981
 441/1688 [======>.......................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9981
 449/1688 [======>.......................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9981
 457/1688 [=======>......................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9980
 466/1688 [=======>......................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9981
 475/1688 [=======>......................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9981
 484/1688 [=======>......................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9981
 493/1688 [=======>......................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9981
 501/1688 [=======>......................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9981
 509/1688 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9981
 518/1688 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9981
 527/1688 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9980
 535/1688 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9981
 544/1688 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9981
 553/1688 [========>.....................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9981
 562/1688 [========>.....................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 571/1688 [=========>....................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9981
 580/1688 [=========>....................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9982
 589/1688 [=========>....................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9981
 598/1688 [=========>....................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 606/1688 [=========>....................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9981
 614/1688 [=========>....................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 623/1688 [==========>...................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 632/1688 [==========>...................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 641/1688 [==========>...................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 650/1688 [==========>...................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 659/1688 [==========>...................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 668/1688 [==========>...................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 677/1688 [===========>..................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 685/1688 [===========>..................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 694/1688 [===========>..................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9982
 703/1688 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9981
 711/1688 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9981
 720/1688 [===========>..................] - ETA: 5s - loss: 0.0061 - accuracy: 0.9981
 729/1688 [===========>..................] - ETA: 5s - loss: 0.0061 - accuracy: 0.9981
 738/1688 [============>.................] - ETA: 5s - loss: 0.0061 - accuracy: 0.9981
 747/1688 [============>.................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 756/1688 [============>.................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9981
 765/1688 [============>.................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9981
 773/1688 [============>.................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9980
 782/1688 [============>.................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9980
 790/1688 [=============>................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 799/1688 [=============>................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 808/1688 [=============>................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 817/1688 [=============>................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 826/1688 [=============>................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 834/1688 [=============>................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 843/1688 [=============>................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 852/1688 [==============>...............] - ETA: 5s - loss: 0.0062 - accuracy: 0.9980
 861/1688 [==============>...............] - ETA: 5s - loss: 0.0063 - accuracy: 0.9980
 870/1688 [==============>...............] - ETA: 5s - loss: 0.0062 - accuracy: 0.9980
 879/1688 [==============>...............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9980
 888/1688 [==============>...............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9980
 897/1688 [==============>...............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9980
 906/1688 [===============>..............] - ETA: 4s - loss: 0.0063 - accuracy: 0.9980
 915/1688 [===============>..............] - ETA: 4s - loss: 0.0063 - accuracy: 0.9980
 924/1688 [===============>..............] - ETA: 4s - loss: 0.0063 - accuracy: 0.9980
 933/1688 [===============>..............] - ETA: 4s - loss: 0.0063 - accuracy: 0.9980
 942/1688 [===============>..............] - ETA: 4s - loss: 0.0063 - accuracy: 0.9980
 950/1688 [===============>..............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9980
 959/1688 [================>.............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9980
 968/1688 [================>.............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9981
 976/1688 [================>.............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9981
 985/1688 [================>.............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9981
 994/1688 [================>.............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9981
1003/1688 [================>.............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9981
1012/1688 [================>.............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9981
1020/1688 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9981
1029/1688 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9981
1038/1688 [=================>............] - ETA: 3s - loss: 0.0061 - accuracy: 0.9982
1047/1688 [=================>............] - ETA: 3s - loss: 0.0061 - accuracy: 0.9982
1056/1688 [=================>............] - ETA: 3s - loss: 0.0061 - accuracy: 0.9982
1065/1688 [=================>............] - ETA: 3s - loss: 0.0061 - accuracy: 0.9981
1074/1688 [==================>...........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9981
1083/1688 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9981
1091/1688 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9981
1100/1688 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9981
1109/1688 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9981
1117/1688 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9981
1125/1688 [==================>...........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1134/1688 [===================>..........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1143/1688 [===================>..........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1151/1688 [===================>..........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1160/1688 [===================>..........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1169/1688 [===================>..........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1178/1688 [===================>..........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1186/1688 [====================>.........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1195/1688 [====================>.........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9980
1204/1688 [====================>.........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9980
1213/1688 [====================>.........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9980
1221/1688 [====================>.........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1230/1688 [====================>.........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1239/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1247/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1256/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1265/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1274/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1282/1688 [=====================>........] - ETA: 2s - loss: 0.0060 - accuracy: 0.9981
1291/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1300/1688 [======================>.......] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1309/1688 [======================>.......] - ETA: 2s - loss: 0.0061 - accuracy: 0.9980
1318/1688 [======================>.......] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1327/1688 [======================>.......] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1336/1688 [======================>.......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9981
1345/1688 [======================>.......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9981
1354/1688 [=======================>......] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1362/1688 [=======================>......] - ETA: 2s - loss: 0.0061 - accuracy: 0.9981
1371/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1380/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1389/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1398/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1407/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1425/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1434/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1442/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1451/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1460/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1469/1688 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1477/1688 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1486/1688 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1495/1688 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1504/1688 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9982
1513/1688 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9981
1522/1688 [==========================>...] - ETA: 1s - loss: 0.0061 - accuracy: 0.9982
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9981
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9982
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9981
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9981
1574/1688 [==========================>...] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9981
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9981
1599/1688 [===========================>..] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1635/1688 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9981
1643/1688 [============================>.] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1652/1688 [============================>.] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1661/1688 [============================>.] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1670/1688 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9981
1679/1688 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9981
1688/1688 [==============================] - ETA: 0s - loss: 0.0062 - accuracy: 0.9981
1688/1688 [==============================] - 11s 7ms/step - loss: 0.0062 - accuracy: 0.9981 - val_loss: 0.0450 - val_accuracy: 0.9888
Epoch 3/5

   1/1688 [..............................] - ETA: 11s - loss: 0.0065 - accuracy: 1.0000
   9/1688 [..............................] - ETA: 10s - loss: 0.0134 - accuracy: 0.9931
  17/1688 [..............................] - ETA: 10s - loss: 0.0103 - accuracy: 0.9945
  26/1688 [..............................] - ETA: 10s - loss: 0.0086 - accuracy: 0.9964
  35/1688 [..............................] - ETA: 10s - loss: 0.0068 - accuracy: 0.9973
  44/1688 [..............................] - ETA: 10s - loss: 0.0074 - accuracy: 0.9972
  53/1688 [..............................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9971 
  62/1688 [>.............................] - ETA: 9s - loss: 0.0067 - accuracy: 0.9975
  71/1688 [>.............................] - ETA: 9s - loss: 0.0068 - accuracy: 0.9974
  80/1688 [>.............................] - ETA: 9s - loss: 0.0061 - accuracy: 0.9977
  89/1688 [>.............................] - ETA: 9s - loss: 0.0061 - accuracy: 0.9975
  97/1688 [>.............................] - ETA: 9s - loss: 0.0060 - accuracy: 0.9977
 106/1688 [>.............................] - ETA: 9s - loss: 0.0061 - accuracy: 0.9976
 114/1688 [=>............................] - ETA: 9s - loss: 0.0058 - accuracy: 0.9978
 123/1688 [=>............................] - ETA: 9s - loss: 0.0057 - accuracy: 0.9980
 132/1688 [=>............................] - ETA: 9s - loss: 0.0063 - accuracy: 0.9979
 141/1688 [=>............................] - ETA: 9s - loss: 0.0061 - accuracy: 0.9980
 150/1688 [=>............................] - ETA: 9s - loss: 0.0059 - accuracy: 0.9981
 158/1688 [=>............................] - ETA: 9s - loss: 0.0059 - accuracy: 0.9982
 167/1688 [=>............................] - ETA: 9s - loss: 0.0057 - accuracy: 0.9983
 175/1688 [==>...........................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9984
 183/1688 [==>...........................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9985
 192/1688 [==>...........................] - ETA: 9s - loss: 0.0059 - accuracy: 0.9984
 201/1688 [==>...........................] - ETA: 9s - loss: 0.0059 - accuracy: 0.9984
 209/1688 [==>...........................] - ETA: 9s - loss: 0.0058 - accuracy: 0.9985
 218/1688 [==>...........................] - ETA: 9s - loss: 0.0059 - accuracy: 0.9984
 227/1688 [===>..........................] - ETA: 8s - loss: 0.0059 - accuracy: 0.9983
 236/1688 [===>..........................] - ETA: 8s - loss: 0.0058 - accuracy: 0.9984
 245/1688 [===>..........................] - ETA: 8s - loss: 0.0056 - accuracy: 0.9985
 253/1688 [===>..........................] - ETA: 8s - loss: 0.0058 - accuracy: 0.9984
 262/1688 [===>..........................] - ETA: 8s - loss: 0.0057 - accuracy: 0.9984
 271/1688 [===>..........................] - ETA: 8s - loss: 0.0056 - accuracy: 0.9985
 280/1688 [===>..........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9985
 289/1688 [====>.........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9986
 298/1688 [====>.........................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9985
 307/1688 [====>.........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9986
 316/1688 [====>.........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9986
 325/1688 [====>.........................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9986
 334/1688 [====>.........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9986
 343/1688 [=====>........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9986
 352/1688 [=====>........................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9987
 361/1688 [=====>........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9987
 370/1688 [=====>........................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9987
 378/1688 [=====>........................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9988
 387/1688 [=====>........................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9988
 395/1688 [======>.......................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9988
 403/1688 [======>.......................] - ETA: 7s - loss: 0.0050 - accuracy: 0.9988
 411/1688 [======>.......................] - ETA: 7s - loss: 0.0056 - accuracy: 0.9987
 420/1688 [======>.......................] - ETA: 7s - loss: 0.0055 - accuracy: 0.9987
 429/1688 [======>.......................] - ETA: 7s - loss: 0.0055 - accuracy: 0.9988
 438/1688 [======>.......................] - ETA: 7s - loss: 0.0055 - accuracy: 0.9987
 446/1688 [======>.......................] - ETA: 7s - loss: 0.0055 - accuracy: 0.9987
 454/1688 [=======>......................] - ETA: 7s - loss: 0.0055 - accuracy: 0.9988
 463/1688 [=======>......................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9988
 471/1688 [=======>......................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9988
 480/1688 [=======>......................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9988
 488/1688 [=======>......................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9988
 497/1688 [=======>......................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9988
 506/1688 [=======>......................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9988
 515/1688 [========>.....................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9988
 523/1688 [========>.....................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9988
 532/1688 [========>.....................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9988
 541/1688 [========>.....................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9988
 550/1688 [========>.....................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
 558/1688 [========>.....................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
 567/1688 [=========>....................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
 576/1688 [=========>....................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 585/1688 [=========>....................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 594/1688 [=========>....................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 603/1688 [=========>....................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9990
 611/1688 [=========>....................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 620/1688 [==========>...................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 628/1688 [==========>...................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9990
 636/1688 [==========>...................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 644/1688 [==========>...................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9989
 653/1688 [==========>...................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9989
 661/1688 [==========>...................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9990
 669/1688 [==========>...................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9989
 678/1688 [===========>..................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9989
 687/1688 [===========>..................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9990
 696/1688 [===========>..................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9990
 705/1688 [===========>..................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9989
 713/1688 [===========>..................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 722/1688 [===========>..................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9990
 730/1688 [===========>..................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9990
 738/1688 [============>.................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 747/1688 [============>.................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 756/1688 [============>.................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 765/1688 [============>.................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 773/1688 [============>.................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 782/1688 [============>.................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 790/1688 [=============>................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 799/1688 [=============>................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 808/1688 [=============>................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 816/1688 [=============>................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 825/1688 [=============>................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 834/1688 [=============>................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9988
 843/1688 [=============>................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 852/1688 [==============>...............] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 861/1688 [==============>...............] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 870/1688 [==============>...............] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 878/1688 [==============>...............] - ETA: 4s - loss: 0.0049 - accuracy: 0.9989
 887/1688 [==============>...............] - ETA: 4s - loss: 0.0049 - accuracy: 0.9989
 896/1688 [==============>...............] - ETA: 4s - loss: 0.0049 - accuracy: 0.9989
 905/1688 [===============>..............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 913/1688 [===============>..............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 921/1688 [===============>..............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 930/1688 [===============>..............] - ETA: 4s - loss: 0.0051 - accuracy: 0.9987
 939/1688 [===============>..............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9987
 948/1688 [===============>..............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9987
 956/1688 [===============>..............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 964/1688 [================>.............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 972/1688 [================>.............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 981/1688 [================>.............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 990/1688 [================>.............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 999/1688 [================>.............] - ETA: 4s - loss: 0.0049 - accuracy: 0.9988
1008/1688 [================>.............] - ETA: 4s - loss: 0.0049 - accuracy: 0.9988
1017/1688 [=================>............] - ETA: 4s - loss: 0.0049 - accuracy: 0.9988
1025/1688 [=================>............] - ETA: 4s - loss: 0.0049 - accuracy: 0.9988
1034/1688 [=================>............] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
1043/1688 [=================>............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9988
1052/1688 [=================>............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9988
1060/1688 [=================>............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9988
1068/1688 [=================>............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9988
1077/1688 [==================>...........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1086/1688 [==================>...........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1095/1688 [==================>...........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1104/1688 [==================>...........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1113/1688 [==================>...........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1122/1688 [==================>...........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1131/1688 [===================>..........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1140/1688 [===================>..........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1148/1688 [===================>..........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1157/1688 [===================>..........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1165/1688 [===================>..........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1174/1688 [===================>..........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1183/1688 [====================>.........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1192/1688 [====================>.........] - ETA: 3s - loss: 0.0049 - accuracy: 0.9988
1201/1688 [====================>.........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1210/1688 [====================>.........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1219/1688 [====================>.........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1228/1688 [====================>.........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1237/1688 [====================>.........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1245/1688 [=====================>........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1254/1688 [=====================>........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1263/1688 [=====================>........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1272/1688 [=====================>........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1280/1688 [=====================>........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1289/1688 [=====================>........] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1298/1688 [======================>.......] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1307/1688 [======================>.......] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1315/1688 [======================>.......] - ETA: 2s - loss: 0.0050 - accuracy: 0.9987
1324/1688 [======================>.......] - ETA: 2s - loss: 0.0050 - accuracy: 0.9987
1333/1688 [======================>.......] - ETA: 2s - loss: 0.0049 - accuracy: 0.9988
1341/1688 [======================>.......] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1349/1688 [======================>.......] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1357/1688 [=======================>......] - ETA: 2s - loss: 0.0050 - accuracy: 0.9988
1365/1688 [=======================>......] - ETA: 1s - loss: 0.0049 - accuracy: 0.9988
1374/1688 [=======================>......] - ETA: 1s - loss: 0.0049 - accuracy: 0.9988
1383/1688 [=======================>......] - ETA: 1s - loss: 0.0049 - accuracy: 0.9988
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0049 - accuracy: 0.9988
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0049 - accuracy: 0.9988
1410/1688 [========================>.....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9988
1419/1688 [========================>.....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1436/1688 [========================>.....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1445/1688 [========================>.....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1453/1688 [========================>.....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1462/1688 [========================>.....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1470/1688 [=========================>....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1478/1688 [=========================>....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1487/1688 [=========================>....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1496/1688 [=========================>....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1505/1688 [=========================>....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1513/1688 [=========================>....] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1522/1688 [==========================>...] - ETA: 1s - loss: 0.0050 - accuracy: 0.9987
1531/1688 [==========================>...] - ETA: 0s - loss: 0.0050 - accuracy: 0.9987
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0050 - accuracy: 0.9987
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0050 - accuracy: 0.9987
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0050 - accuracy: 0.9987
1567/1688 [==========================>...] - ETA: 0s - loss: 0.0050 - accuracy: 0.9987
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9987
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9987
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9987
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9987
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9987
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0049 - accuracy: 0.9987
1628/1688 [===========================>..] - ETA: 0s - loss: 0.0050 - accuracy: 0.9987
1636/1688 [============================>.] - ETA: 0s - loss: 0.0050 - accuracy: 0.9986
1645/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9986
1653/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9986
1661/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9986
1669/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9986
1677/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9986
1686/1688 [============================>.] - ETA: 0s - loss: 0.0051 - accuracy: 0.9986
1688/1688 [==============================] - 11s 7ms/step - loss: 0.0051 - accuracy: 0.9986 - val_loss: 0.0454 - val_accuracy: 0.9880
Epoch 4/5

   1/1688 [..............................] - ETA: 11s - loss: 0.0016 - accuracy: 1.0000
   9/1688 [..............................] - ETA: 10s - loss: 0.0035 - accuracy: 1.0000
  18/1688 [..............................] - ETA: 10s - loss: 0.0043 - accuracy: 1.0000
  27/1688 [..............................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9988
  36/1688 [..............................] - ETA: 10s - loss: 0.0041 - accuracy: 0.9991
  45/1688 [..............................] - ETA: 10s - loss: 0.0042 - accuracy: 0.9986
  53/1688 [..............................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9988
  62/1688 [>.............................] - ETA: 9s - loss: 0.0042 - accuracy: 0.9990 
  71/1688 [>.............................] - ETA: 9s - loss: 0.0039 - accuracy: 0.9991
  79/1688 [>.............................] - ETA: 9s - loss: 0.0036 - accuracy: 0.9992
  88/1688 [>.............................] - ETA: 9s - loss: 0.0035 - accuracy: 0.9993
  97/1688 [>.............................] - ETA: 9s - loss: 0.0035 - accuracy: 0.9994
 106/1688 [>.............................] - ETA: 9s - loss: 0.0033 - accuracy: 0.9994
 115/1688 [=>............................] - ETA: 9s - loss: 0.0035 - accuracy: 0.9995
 124/1688 [=>............................] - ETA: 9s - loss: 0.0041 - accuracy: 0.9992
 133/1688 [=>............................] - ETA: 9s - loss: 0.0040 - accuracy: 0.9993
 142/1688 [=>............................] - ETA: 9s - loss: 0.0039 - accuracy: 0.9993
 151/1688 [=>............................] - ETA: 9s - loss: 0.0037 - accuracy: 0.9994
 160/1688 [=>............................] - ETA: 9s - loss: 0.0037 - accuracy: 0.9994
 168/1688 [=>............................] - ETA: 9s - loss: 0.0037 - accuracy: 0.9994
 177/1688 [==>...........................] - ETA: 9s - loss: 0.0036 - accuracy: 0.9995
 186/1688 [==>...........................] - ETA: 9s - loss: 0.0035 - accuracy: 0.9995
 194/1688 [==>...........................] - ETA: 9s - loss: 0.0034 - accuracy: 0.9995
 203/1688 [==>...........................] - ETA: 9s - loss: 0.0035 - accuracy: 0.9995
 212/1688 [==>...........................] - ETA: 9s - loss: 0.0034 - accuracy: 0.9996
 221/1688 [==>...........................] - ETA: 9s - loss: 0.0034 - accuracy: 0.9996
 230/1688 [===>..........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9996
 239/1688 [===>..........................] - ETA: 8s - loss: 0.0033 - accuracy: 0.9996
 248/1688 [===>..........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9996
 257/1688 [===>..........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9996
 265/1688 [===>..........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9995
 274/1688 [===>..........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9995
 283/1688 [====>.........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9996
 291/1688 [====>.........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9996
 300/1688 [====>.........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9996
 308/1688 [====>.........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9996
 317/1688 [====>.........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9996
 326/1688 [====>.........................] - ETA: 8s - loss: 0.0033 - accuracy: 0.9996
 335/1688 [====>.........................] - ETA: 8s - loss: 0.0033 - accuracy: 0.9996
 344/1688 [=====>........................] - ETA: 8s - loss: 0.0033 - accuracy: 0.9996
 353/1688 [=====>........................] - ETA: 8s - loss: 0.0033 - accuracy: 0.9996
 361/1688 [=====>........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9995
 369/1688 [=====>........................] - ETA: 8s - loss: 0.0033 - accuracy: 0.9995
 378/1688 [=====>........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9994
 387/1688 [=====>........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9994
 396/1688 [======>.......................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9993
 405/1688 [======>.......................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9993
 413/1688 [======>.......................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9992
 422/1688 [======>.......................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9993
 431/1688 [======>.......................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9993
 439/1688 [======>.......................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9993
 448/1688 [======>.......................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9993
 457/1688 [=======>......................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9992
 466/1688 [=======>......................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9993
 474/1688 [=======>......................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9993
 483/1688 [=======>......................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9993
 492/1688 [=======>......................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9993
 500/1688 [=======>......................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9992
 509/1688 [========>.....................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9993
 517/1688 [========>.....................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9993
 526/1688 [========>.....................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9993
 535/1688 [========>.....................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9993
 544/1688 [========>.....................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9993
 553/1688 [========>.....................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9993
 562/1688 [========>.....................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9992
 571/1688 [=========>....................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9992
 580/1688 [=========>....................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9992
 589/1688 [=========>....................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9993
 597/1688 [=========>....................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9993
 606/1688 [=========>....................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9992
 615/1688 [=========>....................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9992
 623/1688 [==========>...................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9992
 632/1688 [==========>...................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9993
 641/1688 [==========>...................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9993
 649/1688 [==========>...................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9992
 657/1688 [==========>...................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9992
 666/1688 [==========>...................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9992
 675/1688 [==========>...................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9993
 684/1688 [===========>..................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9993
 693/1688 [===========>..................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9993
 701/1688 [===========>..................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9993
 710/1688 [===========>..................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9993
 719/1688 [===========>..................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 728/1688 [===========>..................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 737/1688 [============>.................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 746/1688 [============>.................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 755/1688 [============>.................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 763/1688 [============>.................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 772/1688 [============>.................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 780/1688 [============>.................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 789/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 798/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 807/1688 [=============>................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9993
 816/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 825/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 834/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 843/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 852/1688 [==============>...............] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 861/1688 [==============>...............] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 870/1688 [==============>...............] - ETA: 5s - loss: 0.0039 - accuracy: 0.9993
 879/1688 [==============>...............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9993
 887/1688 [==============>...............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9993
 896/1688 [==============>...............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9993
 905/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 914/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 923/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9993
 932/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 941/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 950/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 959/1688 [================>.............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 968/1688 [================>.............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 977/1688 [================>.............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 985/1688 [================>.............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
 994/1688 [================>.............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9992
1003/1688 [================>.............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9992
1011/1688 [================>.............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9991
1020/1688 [=================>............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9991
1028/1688 [=================>............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9991
1037/1688 [=================>............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9991
1045/1688 [=================>............] - ETA: 3s - loss: 0.0041 - accuracy: 0.9991
1053/1688 [=================>............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1062/1688 [=================>............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1071/1688 [==================>...........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1080/1688 [==================>...........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1089/1688 [==================>...........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1097/1688 [==================>...........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1105/1688 [==================>...........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9991
1114/1688 [==================>...........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9991
1123/1688 [==================>...........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9991
1131/1688 [===================>..........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9991
1139/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1148/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1157/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9991
1166/1688 [===================>..........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9991
1175/1688 [===================>..........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9991
1183/1688 [====================>.........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9992
1192/1688 [====================>.........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9992
1201/1688 [====================>.........] - ETA: 3s - loss: 0.0041 - accuracy: 0.9992
1210/1688 [====================>.........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1219/1688 [====================>.........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9992
1228/1688 [====================>.........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1237/1688 [====================>.........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1246/1688 [=====================>........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1254/1688 [=====================>........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1262/1688 [=====================>........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1271/1688 [=====================>........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1279/1688 [=====================>........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1288/1688 [=====================>........] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1297/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1306/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1315/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1324/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1332/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1341/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9991
1349/1688 [======================>.......] - ETA: 2s - loss: 0.0040 - accuracy: 0.9991
1358/1688 [=======================>......] - ETA: 2s - loss: 0.0040 - accuracy: 0.9991
1366/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1374/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1383/1688 [=======================>......] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1392/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1400/1688 [=======================>......] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1408/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1417/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1425/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1434/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1443/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1452/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1461/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1470/1688 [=========================>....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9991
1479/1688 [=========================>....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1488/1688 [=========================>....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1497/1688 [=========================>....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1506/1688 [=========================>....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1514/1688 [=========================>....] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1522/1688 [==========================>...] - ETA: 1s - loss: 0.0040 - accuracy: 0.9991
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1547/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1564/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1573/1688 [==========================>...] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1600/1688 [===========================>..] - ETA: 0s - loss: 0.0040 - accuracy: 0.9991
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0042 - accuracy: 0.9991
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0042 - accuracy: 0.9991
1627/1688 [===========================>..] - ETA: 0s - loss: 0.0041 - accuracy: 0.9991
1635/1688 [============================>.] - ETA: 0s - loss: 0.0041 - accuracy: 0.9991
1644/1688 [============================>.] - ETA: 0s - loss: 0.0041 - accuracy: 0.9991
1652/1688 [============================>.] - ETA: 0s - loss: 0.0042 - accuracy: 0.9991
1661/1688 [============================>.] - ETA: 0s - loss: 0.0042 - accuracy: 0.9991
1669/1688 [============================>.] - ETA: 0s - loss: 0.0042 - accuracy: 0.9991
1678/1688 [============================>.] - ETA: 0s - loss: 0.0042 - accuracy: 0.9991
1687/1688 [============================>.] - ETA: 0s - loss: 0.0042 - accuracy: 0.9991
1688/1688 [==============================] - 11s 7ms/step - loss: 0.0042 - accuracy: 0.9991 - val_loss: 0.0478 - val_accuracy: 0.9885
Epoch 5/5

   1/1688 [..............................] - ETA: 10s - loss: 4.6126e-04 - accuracy: 1.0000
   9/1688 [..............................] - ETA: 10s - loss: 0.0043 - accuracy: 1.0000    
  17/1688 [..............................] - ETA: 10s - loss: 0.0053 - accuracy: 0.9982
  26/1688 [..............................] - ETA: 10s - loss: 0.0043 - accuracy: 0.9988
  35/1688 [..............................] - ETA: 10s - loss: 0.0039 - accuracy: 0.9991
  44/1688 [..............................] - ETA: 10s - loss: 0.0041 - accuracy: 0.9993
  52/1688 [..............................] - ETA: 10s - loss: 0.0037 - accuracy: 0.9994
  61/1688 [>.............................] - ETA: 10s - loss: 0.0034 - accuracy: 0.9995
  69/1688 [>.............................] - ETA: 10s - loss: 0.0032 - accuracy: 0.9995
  77/1688 [>.............................] - ETA: 10s - loss: 0.0030 - accuracy: 0.9996
  85/1688 [>.............................] - ETA: 10s - loss: 0.0032 - accuracy: 0.9996
  94/1688 [>.............................] - ETA: 9s - loss: 0.0032 - accuracy: 0.9997 
 103/1688 [>.............................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9997
 112/1688 [>.............................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9994
 121/1688 [=>............................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9995
 130/1688 [=>............................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9995
 139/1688 [=>............................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9996
 148/1688 [=>............................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9996
 157/1688 [=>............................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9996
 166/1688 [=>............................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9996
 175/1688 [==>...........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9996
 183/1688 [==>...........................] - ETA: 9s - loss: 0.0029 - accuracy: 0.9997
 192/1688 [==>...........................] - ETA: 9s - loss: 0.0028 - accuracy: 0.9997
 201/1688 [==>...........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9995
 209/1688 [==>...........................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9994
 217/1688 [==>...........................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9994
 226/1688 [===>..........................] - ETA: 9s - loss: 0.0032 - accuracy: 0.9993
 235/1688 [===>..........................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9993
 244/1688 [===>..........................] - ETA: 8s - loss: 0.0032 - accuracy: 0.9994
 253/1688 [===>..........................] - ETA: 8s - loss: 0.0032 - accuracy: 0.9994
 261/1688 [===>..........................] - ETA: 8s - loss: 0.0032 - accuracy: 0.9994
 269/1688 [===>..........................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9994
 278/1688 [===>..........................] - ETA: 8s - loss: 0.0032 - accuracy: 0.9993
 286/1688 [====>.........................] - ETA: 8s - loss: 0.0032 - accuracy: 0.9993
 295/1688 [====>.........................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9994
 304/1688 [====>.........................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9994
 313/1688 [====>.........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9994
 322/1688 [====>.........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9992
 330/1688 [====>.........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9992
 339/1688 [=====>........................] - ETA: 8s - loss: 0.0033 - accuracy: 0.9993
 347/1688 [=====>........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9992
 356/1688 [=====>........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9992
 365/1688 [=====>........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9992
 374/1688 [=====>........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9992
 383/1688 [=====>........................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9993
 391/1688 [=====>........................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9993
 400/1688 [======>.......................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9993
 409/1688 [======>.......................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9993
 418/1688 [======>.......................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9993
 426/1688 [======>.......................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9993
 435/1688 [======>.......................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9993
 443/1688 [======>.......................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9993
 452/1688 [=======>......................] - ETA: 7s - loss: 0.0034 - accuracy: 0.9992
 461/1688 [=======>......................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9992
 470/1688 [=======>......................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9992
 479/1688 [=======>......................] - ETA: 7s - loss: 0.0035 - accuracy: 0.9992
 488/1688 [=======>......................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9991
 497/1688 [=======>......................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9991
 505/1688 [=======>......................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9991
 514/1688 [========>.....................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9991
 523/1688 [========>.....................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9992
 532/1688 [========>.....................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9991
 540/1688 [========>.....................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9991
 549/1688 [========>.....................] - ETA: 6s - loss: 0.0037 - accuracy: 0.9991
 558/1688 [========>.....................] - ETA: 6s - loss: 0.0037 - accuracy: 0.9991
 566/1688 [=========>....................] - ETA: 6s - loss: 0.0037 - accuracy: 0.9991
 574/1688 [=========>....................] - ETA: 6s - loss: 0.0037 - accuracy: 0.9991
 583/1688 [=========>....................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9990
 592/1688 [=========>....................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9990
 601/1688 [=========>....................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 610/1688 [=========>....................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 619/1688 [==========>...................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 627/1688 [==========>...................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 636/1688 [==========>...................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 644/1688 [==========>...................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9990
 653/1688 [==========>...................] - ETA: 6s - loss: 0.0039 - accuracy: 0.9990
 662/1688 [==========>...................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 670/1688 [==========>...................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 679/1688 [===========>..................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 687/1688 [===========>..................] - ETA: 6s - loss: 0.0038 - accuracy: 0.9991
 696/1688 [===========>..................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9990
 705/1688 [===========>..................] - ETA: 6s - loss: 0.0040 - accuracy: 0.9990
 714/1688 [===========>..................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9990
 722/1688 [===========>..................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9990
 730/1688 [===========>..................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9990
 739/1688 [============>.................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9990
 747/1688 [============>.................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9990
 756/1688 [============>.................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9990
 765/1688 [============>.................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
 774/1688 [============>.................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9990
 783/1688 [============>.................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9990
 792/1688 [=============>................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9991
 801/1688 [=============>................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9991
 810/1688 [=============>................] - ETA: 5s - loss: 0.0040 - accuracy: 0.9991
 819/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
 828/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
 836/1688 [=============>................] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
 845/1688 [==============>...............] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
 854/1688 [==============>...............] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
 862/1688 [==============>...............] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
 870/1688 [==============>...............] - ETA: 5s - loss: 0.0039 - accuracy: 0.9991
 878/1688 [==============>...............] - ETA: 4s - loss: 0.0039 - accuracy: 0.9991
 887/1688 [==============>...............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9991
 895/1688 [==============>...............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9991
 903/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9991
 912/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9991
 921/1688 [===============>..............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9991
 930/1688 [===============>..............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9990
 938/1688 [===============>..............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9990
 947/1688 [===============>..............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9990
 956/1688 [===============>..............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9990
 964/1688 [================>.............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9990
 973/1688 [================>.............] - ETA: 4s - loss: 0.0040 - accuracy: 0.9990
 982/1688 [================>.............] - ETA: 4s - loss: 0.0041 - accuracy: 0.9990
 990/1688 [================>.............] - ETA: 4s - loss: 0.0042 - accuracy: 0.9989
 999/1688 [================>.............] - ETA: 4s - loss: 0.0044 - accuracy: 0.9989
1008/1688 [================>.............] - ETA: 4s - loss: 0.0044 - accuracy: 0.9989
1017/1688 [=================>............] - ETA: 4s - loss: 0.0044 - accuracy: 0.9989
1026/1688 [=================>............] - ETA: 4s - loss: 0.0043 - accuracy: 0.9989
1035/1688 [=================>............] - ETA: 4s - loss: 0.0043 - accuracy: 0.9989
1043/1688 [=================>............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9989
1052/1688 [=================>............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9989
1061/1688 [=================>............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9989
1070/1688 [==================>...........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9989
1079/1688 [==================>...........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9989
1087/1688 [==================>...........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9989
1095/1688 [==================>...........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9989
1104/1688 [==================>...........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9990
1112/1688 [==================>...........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1121/1688 [==================>...........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1130/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1139/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1148/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1157/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1166/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1175/1688 [===================>..........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1184/1688 [====================>.........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1193/1688 [====================>.........] - ETA: 3s - loss: 0.0042 - accuracy: 0.9990
1202/1688 [====================>.........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1211/1688 [====================>.........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1220/1688 [====================>.........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1228/1688 [====================>.........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1237/1688 [====================>.........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1246/1688 [=====================>........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1254/1688 [=====================>........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1262/1688 [=====================>........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1270/1688 [=====================>........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1279/1688 [=====================>........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1287/1688 [=====================>........] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1296/1688 [======================>.......] - ETA: 2s - loss: 0.0042 - accuracy: 0.9990
1305/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9990
1314/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9990
1323/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9990
1332/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9990
1341/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9990
1349/1688 [======================>.......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9990
1358/1688 [=======================>......] - ETA: 2s - loss: 0.0041 - accuracy: 0.9990
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1376/1688 [=======================>......] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1385/1688 [=======================>......] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1394/1688 [=======================>......] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1402/1688 [=======================>......] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1411/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1420/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1429/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1438/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1446/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1455/1688 [========================>.....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1464/1688 [=========================>....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1472/1688 [=========================>....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1481/1688 [=========================>....] - ETA: 1s - loss: 0.0042 - accuracy: 0.9990
1490/1688 [=========================>....] - ETA: 1s - loss: 0.0042 - accuracy: 0.9990
1499/1688 [=========================>....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1508/1688 [=========================>....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1516/1688 [=========================>....] - ETA: 1s - loss: 0.0041 - accuracy: 0.9990
1525/1688 [==========================>...] - ETA: 1s - loss: 0.0042 - accuracy: 0.9990
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1552/1688 [==========================>...] - ETA: 0s - loss: 0.0042 - accuracy: 0.9990
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1569/1688 [==========================>...] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1578/1688 [===========================>..] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1595/1688 [===========================>..] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1631/1688 [===========================>..] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1639/1688 [============================>.] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1648/1688 [============================>.] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1657/1688 [============================>.] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1666/1688 [============================>.] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1675/1688 [============================>.] - ETA: 0s - loss: 0.0041 - accuracy: 0.9990
1684/1688 [============================>.] - ETA: 0s - loss: 0.0042 - accuracy: 0.9990
1688/1688 [==============================] - 11s 7ms/step - loss: 0.0041 - accuracy: 0.9990 - val_loss: 0.0469 - val_accuracy: 0.9883

<tf_keras.src.callbacks.History object at 0x7145955a6390>
score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine tuning:', score)
Test accuracy after fine tuning: 0.9883000254631042

3. Convert

3.1 Convert to Akida model

When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.

As with TF-Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized)
model_akida.summary()
                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          5
______________________________________________

__________________________________________________________________
Layer (type)                        Output shape  Kernel shape

=============== SW/conv2d-dequantizer_2 (Software) ===============

conv2d (InputConv2D)                [13, 13, 32]  (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D)  [7, 7, 32]    (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D)                   [7, 7, 64]    (1, 1, 32, 64)
__________________________________________________________________
dense (Dense1D)                     [1, 1, 10]    (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer)         [1, 1, 10]    N/A
__________________________________________________________________

3.2. Check performance

accuracy = model_akida.evaluate(x_test, y_test.astype(np.int32))
print('Test accuracy after conversion:', accuracy)

# For non-regression purposes
assert accuracy > 0.96
Test accuracy after conversion: 0.9865000247955322

3.3 Show predictions for a single image

Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.

# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])

f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()

print(outputs.squeeze())
Class 7
Input Label: 7
[-17.85389    -10.923257    -7.0271673   -0.06355193 -14.291705
 -11.592661   -27.31079     12.217054    -8.211378    -1.4405913 ]

Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.

In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.

Total running time of the script: (2 minutes 15.062 seconds)

Gallery generated by Sphinx-Gallery