Global Akida workflow

Using the MNIST dataset, this example shows the definition and training of a keras floating point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original keras floating point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.

Note

Please refer to the TensorFlow tf.keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.

The MNIST example below is light enough so that a GPU is not needed for training.

Overall flow

Global Akida workflow

1. Create and train

1.1. Load and reshape MNIST dataset

import numpy as np

import matplotlib.cm as cm
import matplotlib.pyplot as plt

from keras.datasets import mnist

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, width, height, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)

# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
    axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
    axarr[i].set_title('Class %d' % y_test[i])
plt.show()
Class 7, Class 2, Class 1, Class 0
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz

    8192/11490434 [..............................] - ETA: 0s
   16384/11490434 [..............................] - ETA: 36s
   49152/11490434 [..............................] - ETA: 32s
   81920/11490434 [..............................] - ETA: 31s
  147456/11490434 [..............................] - ETA: 22s
  221184/11490434 [..............................] - ETA: 17s
  335872/11490434 [..............................] - ETA: 13s
  475136/11490434 [>.............................] - ETA: 10s
  745472/11490434 [>.............................] - ETA: 7s 
 1114112/11490434 [=>............................] - ETA: 5s
 1720320/11490434 [===>..........................] - ETA: 3s
 2629632/11490434 [=====>........................] - ETA: 2s
 3874816/11490434 [=========>....................] - ETA: 1s
 5275648/11490434 [============>.................] - ETA: 0s
 6586368/11490434 [================>.............] - ETA: 0s
 7831552/11490434 [===================>..........] - ETA: 0s
 9060352/11490434 [======================>.......] - ETA: 0s
10305536/11490434 [=========================>....] - ETA: 0s
11042816/11490434 [===========================>..] - ETA: 0s
11490434/11490434 [==============================] - 1s 0us/step

1.2. Model definition

Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard Keras CNN model.

import keras

model_keras = keras.models.Sequential([
    keras.layers.Rescaling(1. / 255, input_shape=(28, 28, 1)),
    keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    # Separable layer
    keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
    keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling (Rescaling)       (None, 28, 28, 1)         0

 conv2d (Conv2D)             (None, 13, 13, 32)        320

 batch_normalization (Batch  (None, 13, 13, 32)        128
 Normalization)

 re_lu (ReLU)                (None, 13, 13, 32)        0

 depthwise_conv2d (Depthwis  (None, 7, 7, 32)          320
 eConv2D)

 conv2d_1 (Conv2D)           (None, 7, 7, 64)          2112

 batch_normalization_1 (Bat  (None, 7, 7, 64)          256
 chNormalization)

 re_lu_1 (ReLU)              (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (Dense)               (None, 10)                31370

=================================================================
Total params: 34506 (134.79 KB)
Trainable params: 34314 (134.04 KB)
Non-trainable params: 192 (768.00 Byte)
_________________________________________________________________

1.3. Model training

Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.

from keras.optimizers import Adam

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-3),
    metrics=['accuracy'])

_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
Epoch 1/10

   1/1688 [..............................] - ETA: 1:03:08 - loss: 3.0761 - accuracy: 0.0938
  24/1688 [..............................] - ETA: 3s - loss: 1.4955 - accuracy: 0.5234     
  47/1688 [..............................] - ETA: 3s - loss: 1.0861 - accuracy: 0.6556
  70/1688 [>.............................] - ETA: 3s - loss: 0.8868 - accuracy: 0.7219
  93/1688 [>.............................] - ETA: 3s - loss: 0.7584 - accuracy: 0.7641
 115/1688 [=>............................] - ETA: 3s - loss: 0.6801 - accuracy: 0.7878
 136/1688 [=>............................] - ETA: 3s - loss: 0.6298 - accuracy: 0.8038
 158/1688 [=>............................] - ETA: 3s - loss: 0.5827 - accuracy: 0.8192
 180/1688 [==>...........................] - ETA: 3s - loss: 0.5451 - accuracy: 0.8326
 203/1688 [==>...........................] - ETA: 3s - loss: 0.5108 - accuracy: 0.8431
 226/1688 [===>..........................] - ETA: 3s - loss: 0.4851 - accuracy: 0.8508
 249/1688 [===>..........................] - ETA: 3s - loss: 0.4581 - accuracy: 0.8592
 272/1688 [===>..........................] - ETA: 3s - loss: 0.4379 - accuracy: 0.8649
 295/1688 [====>.........................] - ETA: 3s - loss: 0.4211 - accuracy: 0.8703
 318/1688 [====>.........................] - ETA: 3s - loss: 0.4105 - accuracy: 0.8740
 341/1688 [=====>........................] - ETA: 3s - loss: 0.3972 - accuracy: 0.8785
 364/1688 [=====>........................] - ETA: 2s - loss: 0.3828 - accuracy: 0.8824
 387/1688 [=====>........................] - ETA: 2s - loss: 0.3703 - accuracy: 0.8864
 410/1688 [======>.......................] - ETA: 2s - loss: 0.3579 - accuracy: 0.8909
 433/1688 [======>.......................] - ETA: 2s - loss: 0.3486 - accuracy: 0.8943
 456/1688 [=======>......................] - ETA: 2s - loss: 0.3401 - accuracy: 0.8969
 479/1688 [=======>......................] - ETA: 2s - loss: 0.3319 - accuracy: 0.8991
 502/1688 [=======>......................] - ETA: 2s - loss: 0.3232 - accuracy: 0.9018
 525/1688 [========>.....................] - ETA: 2s - loss: 0.3171 - accuracy: 0.9034
 548/1688 [========>.....................] - ETA: 2s - loss: 0.3111 - accuracy: 0.9051
 571/1688 [=========>....................] - ETA: 2s - loss: 0.3037 - accuracy: 0.9071
 594/1688 [=========>....................] - ETA: 2s - loss: 0.2969 - accuracy: 0.9095
 617/1688 [=========>....................] - ETA: 2s - loss: 0.2901 - accuracy: 0.9116
 640/1688 [==========>...................] - ETA: 2s - loss: 0.2854 - accuracy: 0.9133
 661/1688 [==========>...................] - ETA: 2s - loss: 0.2803 - accuracy: 0.9149
 682/1688 [===========>..................] - ETA: 2s - loss: 0.2753 - accuracy: 0.9165
 704/1688 [===========>..................] - ETA: 2s - loss: 0.2706 - accuracy: 0.9178
 726/1688 [===========>..................] - ETA: 2s - loss: 0.2668 - accuracy: 0.9190
 747/1688 [============>.................] - ETA: 2s - loss: 0.2626 - accuracy: 0.9203
 769/1688 [============>.................] - ETA: 2s - loss: 0.2584 - accuracy: 0.9215
 791/1688 [=============>................] - ETA: 2s - loss: 0.2543 - accuracy: 0.9230
 813/1688 [=============>................] - ETA: 1s - loss: 0.2508 - accuracy: 0.9242
 835/1688 [=============>................] - ETA: 1s - loss: 0.2475 - accuracy: 0.9251
 857/1688 [==============>...............] - ETA: 1s - loss: 0.2450 - accuracy: 0.9259
 879/1688 [==============>...............] - ETA: 1s - loss: 0.2423 - accuracy: 0.9266
 900/1688 [==============>...............] - ETA: 1s - loss: 0.2397 - accuracy: 0.9272
 921/1688 [===============>..............] - ETA: 1s - loss: 0.2372 - accuracy: 0.9280
 942/1688 [===============>..............] - ETA: 1s - loss: 0.2346 - accuracy: 0.9286
 964/1688 [================>.............] - ETA: 1s - loss: 0.2319 - accuracy: 0.9294
 985/1688 [================>.............] - ETA: 1s - loss: 0.2293 - accuracy: 0.9301
1006/1688 [================>.............] - ETA: 1s - loss: 0.2271 - accuracy: 0.9308
1027/1688 [=================>............] - ETA: 1s - loss: 0.2256 - accuracy: 0.9315
1048/1688 [=================>............] - ETA: 1s - loss: 0.2236 - accuracy: 0.9319
1069/1688 [=================>............] - ETA: 1s - loss: 0.2219 - accuracy: 0.9325
1090/1688 [==================>...........] - ETA: 1s - loss: 0.2193 - accuracy: 0.9333
1111/1688 [==================>...........] - ETA: 1s - loss: 0.2174 - accuracy: 0.9339
1132/1688 [===================>..........] - ETA: 1s - loss: 0.2154 - accuracy: 0.9343
1154/1688 [===================>..........] - ETA: 1s - loss: 0.2137 - accuracy: 0.9348
1176/1688 [===================>..........] - ETA: 1s - loss: 0.2114 - accuracy: 0.9355
1197/1688 [====================>.........] - ETA: 1s - loss: 0.2095 - accuracy: 0.9361
1218/1688 [====================>.........] - ETA: 1s - loss: 0.2079 - accuracy: 0.9365
1239/1688 [=====================>........] - ETA: 1s - loss: 0.2062 - accuracy: 0.9370
1259/1688 [=====================>........] - ETA: 0s - loss: 0.2041 - accuracy: 0.9377
1279/1688 [=====================>........] - ETA: 0s - loss: 0.2023 - accuracy: 0.9382
1299/1688 [======================>.......] - ETA: 0s - loss: 0.2009 - accuracy: 0.9387
1318/1688 [======================>.......] - ETA: 0s - loss: 0.1993 - accuracy: 0.9391
1338/1688 [======================>.......] - ETA: 0s - loss: 0.1983 - accuracy: 0.9395
1358/1688 [=======================>......] - ETA: 0s - loss: 0.1966 - accuracy: 0.9401
1378/1688 [=======================>......] - ETA: 0s - loss: 0.1952 - accuracy: 0.9404
1398/1688 [=======================>......] - ETA: 0s - loss: 0.1943 - accuracy: 0.9407
1418/1688 [========================>.....] - ETA: 0s - loss: 0.1928 - accuracy: 0.9411
1438/1688 [========================>.....] - ETA: 0s - loss: 0.1915 - accuracy: 0.9416
1458/1688 [========================>.....] - ETA: 0s - loss: 0.1905 - accuracy: 0.9420
1478/1688 [=========================>....] - ETA: 0s - loss: 0.1893 - accuracy: 0.9424
1498/1688 [=========================>....] - ETA: 0s - loss: 0.1877 - accuracy: 0.9429
1518/1688 [=========================>....] - ETA: 0s - loss: 0.1863 - accuracy: 0.9432
1538/1688 [==========================>...] - ETA: 0s - loss: 0.1856 - accuracy: 0.9435
1558/1688 [==========================>...] - ETA: 0s - loss: 0.1849 - accuracy: 0.9437
1578/1688 [===========================>..] - ETA: 0s - loss: 0.1836 - accuracy: 0.9440
1598/1688 [===========================>..] - ETA: 0s - loss: 0.1826 - accuracy: 0.9444
1618/1688 [===========================>..] - ETA: 0s - loss: 0.1813 - accuracy: 0.9447
1638/1688 [============================>.] - ETA: 0s - loss: 0.1806 - accuracy: 0.9451
1658/1688 [============================>.] - ETA: 0s - loss: 0.1793 - accuracy: 0.9455
1678/1688 [============================>.] - ETA: 0s - loss: 0.1784 - accuracy: 0.9458
1688/1688 [==============================] - ETA: 0s - loss: 0.1778 - accuracy: 0.9460
1688/1688 [==============================] - 7s 3ms/step - loss: 0.1778 - accuracy: 0.9460 - val_loss: 0.0858 - val_accuracy: 0.9763
Epoch 2/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0485 - accuracy: 1.0000
  24/1688 [..............................] - ETA: 3s - loss: 0.0752 - accuracy: 0.9779
  46/1688 [..............................] - ETA: 3s - loss: 0.0887 - accuracy: 0.9749
  69/1688 [>.............................] - ETA: 3s - loss: 0.0808 - accuracy: 0.9755
  92/1688 [>.............................] - ETA: 3s - loss: 0.0752 - accuracy: 0.9789
 115/1688 [=>............................] - ETA: 3s - loss: 0.0770 - accuracy: 0.9780
 137/1688 [=>............................] - ETA: 3s - loss: 0.0774 - accuracy: 0.9779
 160/1688 [=>............................] - ETA: 3s - loss: 0.0781 - accuracy: 0.9779
 183/1688 [==>...........................] - ETA: 3s - loss: 0.0777 - accuracy: 0.9775
 206/1688 [==>...........................] - ETA: 3s - loss: 0.0776 - accuracy: 0.9774
 229/1688 [===>..........................] - ETA: 3s - loss: 0.0793 - accuracy: 0.9763
 252/1688 [===>..........................] - ETA: 3s - loss: 0.0758 - accuracy: 0.9774
 275/1688 [===>..........................] - ETA: 3s - loss: 0.0769 - accuracy: 0.9766
 298/1688 [====>.........................] - ETA: 3s - loss: 0.0785 - accuracy: 0.9758
 321/1688 [====>.........................] - ETA: 3s - loss: 0.0792 - accuracy: 0.9755
 343/1688 [=====>........................] - ETA: 3s - loss: 0.0809 - accuracy: 0.9751
 366/1688 [=====>........................] - ETA: 2s - loss: 0.0795 - accuracy: 0.9752
 389/1688 [=====>........................] - ETA: 2s - loss: 0.0803 - accuracy: 0.9750
 412/1688 [======>.......................] - ETA: 2s - loss: 0.0801 - accuracy: 0.9752
 435/1688 [======>.......................] - ETA: 2s - loss: 0.0803 - accuracy: 0.9753
 457/1688 [=======>......................] - ETA: 2s - loss: 0.0809 - accuracy: 0.9751
 478/1688 [=======>......................] - ETA: 2s - loss: 0.0812 - accuracy: 0.9748
 499/1688 [=======>......................] - ETA: 2s - loss: 0.0803 - accuracy: 0.9749
 520/1688 [========>.....................] - ETA: 2s - loss: 0.0794 - accuracy: 0.9751
 541/1688 [========>.....................] - ETA: 2s - loss: 0.0777 - accuracy: 0.9755
 562/1688 [========>.....................] - ETA: 2s - loss: 0.0766 - accuracy: 0.9760
 584/1688 [=========>....................] - ETA: 2s - loss: 0.0759 - accuracy: 0.9761
 606/1688 [=========>....................] - ETA: 2s - loss: 0.0773 - accuracy: 0.9762
 627/1688 [==========>...................] - ETA: 2s - loss: 0.0763 - accuracy: 0.9763
 649/1688 [==========>...................] - ETA: 2s - loss: 0.0771 - accuracy: 0.9760
 670/1688 [==========>...................] - ETA: 2s - loss: 0.0763 - accuracy: 0.9763
 691/1688 [===========>..................] - ETA: 2s - loss: 0.0757 - accuracy: 0.9766
 712/1688 [===========>..................] - ETA: 2s - loss: 0.0757 - accuracy: 0.9767
 734/1688 [============>.................] - ETA: 2s - loss: 0.0752 - accuracy: 0.9768
 755/1688 [============>.................] - ETA: 2s - loss: 0.0750 - accuracy: 0.9768
 776/1688 [============>.................] - ETA: 2s - loss: 0.0749 - accuracy: 0.9767
 797/1688 [=============>................] - ETA: 2s - loss: 0.0741 - accuracy: 0.9768
 818/1688 [=============>................] - ETA: 2s - loss: 0.0741 - accuracy: 0.9767
 840/1688 [=============>................] - ETA: 1s - loss: 0.0734 - accuracy: 0.9768
 861/1688 [==============>...............] - ETA: 1s - loss: 0.0738 - accuracy: 0.9767
 883/1688 [==============>...............] - ETA: 1s - loss: 0.0736 - accuracy: 0.9767
 903/1688 [===============>..............] - ETA: 1s - loss: 0.0732 - accuracy: 0.9768
 923/1688 [===============>..............] - ETA: 1s - loss: 0.0734 - accuracy: 0.9768
 943/1688 [===============>..............] - ETA: 1s - loss: 0.0736 - accuracy: 0.9767
 963/1688 [================>.............] - ETA: 1s - loss: 0.0739 - accuracy: 0.9766
 983/1688 [================>.............] - ETA: 1s - loss: 0.0740 - accuracy: 0.9766
1003/1688 [================>.............] - ETA: 1s - loss: 0.0738 - accuracy: 0.9766
1024/1688 [=================>............] - ETA: 1s - loss: 0.0739 - accuracy: 0.9766
1044/1688 [=================>............] - ETA: 1s - loss: 0.0738 - accuracy: 0.9766
1064/1688 [=================>............] - ETA: 1s - loss: 0.0736 - accuracy: 0.9766
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0735 - accuracy: 0.9766
1104/1688 [==================>...........] - ETA: 1s - loss: 0.0739 - accuracy: 0.9763
1124/1688 [==================>...........] - ETA: 1s - loss: 0.0740 - accuracy: 0.9764
1144/1688 [===================>..........] - ETA: 1s - loss: 0.0736 - accuracy: 0.9765
1164/1688 [===================>..........] - ETA: 1s - loss: 0.0731 - accuracy: 0.9766
1184/1688 [====================>.........] - ETA: 1s - loss: 0.0729 - accuracy: 0.9767
1203/1688 [====================>.........] - ETA: 1s - loss: 0.0729 - accuracy: 0.9767
1223/1688 [====================>.........] - ETA: 1s - loss: 0.0729 - accuracy: 0.9767
1243/1688 [=====================>........] - ETA: 1s - loss: 0.0731 - accuracy: 0.9768
1262/1688 [=====================>........] - ETA: 1s - loss: 0.0729 - accuracy: 0.9767
1282/1688 [=====================>........] - ETA: 0s - loss: 0.0730 - accuracy: 0.9768
1302/1688 [======================>.......] - ETA: 0s - loss: 0.0728 - accuracy: 0.9768
1322/1688 [======================>.......] - ETA: 0s - loss: 0.0726 - accuracy: 0.9769
1342/1688 [======================>.......] - ETA: 0s - loss: 0.0726 - accuracy: 0.9769
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0722 - accuracy: 0.9770
1382/1688 [=======================>......] - ETA: 0s - loss: 0.0723 - accuracy: 0.9770
1401/1688 [=======================>......] - ETA: 0s - loss: 0.0725 - accuracy: 0.9770
1421/1688 [========================>.....] - ETA: 0s - loss: 0.0726 - accuracy: 0.9770
1441/1688 [========================>.....] - ETA: 0s - loss: 0.0724 - accuracy: 0.9770
1460/1688 [========================>.....] - ETA: 0s - loss: 0.0722 - accuracy: 0.9771
1480/1688 [=========================>....] - ETA: 0s - loss: 0.0721 - accuracy: 0.9771
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0721 - accuracy: 0.9771
1520/1688 [==========================>...] - ETA: 0s - loss: 0.0724 - accuracy: 0.9771
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0725 - accuracy: 0.9770
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0724 - accuracy: 0.9770
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0726 - accuracy: 0.9770
1598/1688 [===========================>..] - ETA: 0s - loss: 0.0723 - accuracy: 0.9772
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0721 - accuracy: 0.9773
1637/1688 [============================>.] - ETA: 0s - loss: 0.0723 - accuracy: 0.9773
1657/1688 [============================>.] - ETA: 0s - loss: 0.0720 - accuracy: 0.9773
1676/1688 [============================>.] - ETA: 0s - loss: 0.0721 - accuracy: 0.9772
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0720 - accuracy: 0.9772 - val_loss: 0.0892 - val_accuracy: 0.9742
Epoch 3/10

   1/1688 [..............................] - ETA: 4s - loss: 0.0168 - accuracy: 1.0000
  21/1688 [..............................] - ETA: 4s - loss: 0.0582 - accuracy: 0.9836
  41/1688 [..............................] - ETA: 4s - loss: 0.0591 - accuracy: 0.9832
  61/1688 [>.............................] - ETA: 4s - loss: 0.0526 - accuracy: 0.9836
  81/1688 [>.............................] - ETA: 4s - loss: 0.0495 - accuracy: 0.9861
 101/1688 [>.............................] - ETA: 4s - loss: 0.0484 - accuracy: 0.9858
 121/1688 [=>............................] - ETA: 4s - loss: 0.0506 - accuracy: 0.9848
 141/1688 [=>............................] - ETA: 3s - loss: 0.0495 - accuracy: 0.9849
 161/1688 [=>............................] - ETA: 3s - loss: 0.0493 - accuracy: 0.9851
 181/1688 [==>...........................] - ETA: 3s - loss: 0.0499 - accuracy: 0.9845
 200/1688 [==>...........................] - ETA: 3s - loss: 0.0499 - accuracy: 0.9845
 220/1688 [==>...........................] - ETA: 3s - loss: 0.0470 - accuracy: 0.9857
 240/1688 [===>..........................] - ETA: 3s - loss: 0.0493 - accuracy: 0.9849
 260/1688 [===>..........................] - ETA: 3s - loss: 0.0496 - accuracy: 0.9849
 280/1688 [===>..........................] - ETA: 3s - loss: 0.0491 - accuracy: 0.9852
 300/1688 [====>.........................] - ETA: 3s - loss: 0.0499 - accuracy: 0.9849
 320/1688 [====>.........................] - ETA: 3s - loss: 0.0494 - accuracy: 0.9848
 340/1688 [=====>........................] - ETA: 3s - loss: 0.0492 - accuracy: 0.9850
 360/1688 [=====>........................] - ETA: 3s - loss: 0.0503 - accuracy: 0.9847
 380/1688 [=====>........................] - ETA: 3s - loss: 0.0507 - accuracy: 0.9843
 400/1688 [======>.......................] - ETA: 3s - loss: 0.0509 - accuracy: 0.9841
 420/1688 [======>.......................] - ETA: 3s - loss: 0.0495 - accuracy: 0.9846
 440/1688 [======>.......................] - ETA: 3s - loss: 0.0485 - accuracy: 0.9848
 460/1688 [=======>......................] - ETA: 3s - loss: 0.0481 - accuracy: 0.9848
 479/1688 [=======>......................] - ETA: 3s - loss: 0.0489 - accuracy: 0.9845
 499/1688 [=======>......................] - ETA: 3s - loss: 0.0494 - accuracy: 0.9843
 519/1688 [========>.....................] - ETA: 3s - loss: 0.0499 - accuracy: 0.9843
 538/1688 [========>.....................] - ETA: 2s - loss: 0.0504 - accuracy: 0.9843
 558/1688 [========>.....................] - ETA: 2s - loss: 0.0504 - accuracy: 0.9842
 578/1688 [=========>....................] - ETA: 2s - loss: 0.0499 - accuracy: 0.9844
 598/1688 [=========>....................] - ETA: 2s - loss: 0.0499 - accuracy: 0.9844
 618/1688 [=========>....................] - ETA: 2s - loss: 0.0498 - accuracy: 0.9844
 638/1688 [==========>...................] - ETA: 2s - loss: 0.0494 - accuracy: 0.9846
 658/1688 [==========>...................] - ETA: 2s - loss: 0.0490 - accuracy: 0.9847
 677/1688 [===========>..................] - ETA: 2s - loss: 0.0487 - accuracy: 0.9848
 697/1688 [===========>..................] - ETA: 2s - loss: 0.0484 - accuracy: 0.9848
 717/1688 [===========>..................] - ETA: 2s - loss: 0.0478 - accuracy: 0.9849
 736/1688 [============>.................] - ETA: 2s - loss: 0.0484 - accuracy: 0.9847
 756/1688 [============>.................] - ETA: 2s - loss: 0.0484 - accuracy: 0.9845
 776/1688 [============>.................] - ETA: 2s - loss: 0.0483 - accuracy: 0.9845
 796/1688 [=============>................] - ETA: 2s - loss: 0.0494 - accuracy: 0.9842
 816/1688 [=============>................] - ETA: 2s - loss: 0.0497 - accuracy: 0.9840
 836/1688 [=============>................] - ETA: 2s - loss: 0.0502 - accuracy: 0.9839
 855/1688 [==============>...............] - ETA: 2s - loss: 0.0501 - accuracy: 0.9839
 875/1688 [==============>...............] - ETA: 2s - loss: 0.0496 - accuracy: 0.9841
 895/1688 [==============>...............] - ETA: 2s - loss: 0.0497 - accuracy: 0.9841
 915/1688 [===============>..............] - ETA: 1s - loss: 0.0500 - accuracy: 0.9839
 935/1688 [===============>..............] - ETA: 1s - loss: 0.0503 - accuracy: 0.9838
 955/1688 [===============>..............] - ETA: 1s - loss: 0.0502 - accuracy: 0.9837
 975/1688 [================>.............] - ETA: 1s - loss: 0.0504 - accuracy: 0.9837
 995/1688 [================>.............] - ETA: 1s - loss: 0.0508 - accuracy: 0.9836
1015/1688 [=================>............] - ETA: 1s - loss: 0.0507 - accuracy: 0.9837
1034/1688 [=================>............] - ETA: 1s - loss: 0.0506 - accuracy: 0.9838
1054/1688 [=================>............] - ETA: 1s - loss: 0.0509 - accuracy: 0.9837
1074/1688 [==================>...........] - ETA: 1s - loss: 0.0508 - accuracy: 0.9838
1094/1688 [==================>...........] - ETA: 1s - loss: 0.0512 - accuracy: 0.9838
1114/1688 [==================>...........] - ETA: 1s - loss: 0.0509 - accuracy: 0.9839
1134/1688 [===================>..........] - ETA: 1s - loss: 0.0511 - accuracy: 0.9839
1153/1688 [===================>..........] - ETA: 1s - loss: 0.0510 - accuracy: 0.9838
1173/1688 [===================>..........] - ETA: 1s - loss: 0.0509 - accuracy: 0.9838
1193/1688 [====================>.........] - ETA: 1s - loss: 0.0513 - accuracy: 0.9838
1213/1688 [====================>.........] - ETA: 1s - loss: 0.0512 - accuracy: 0.9837
1233/1688 [====================>.........] - ETA: 1s - loss: 0.0512 - accuracy: 0.9838
1253/1688 [=====================>........] - ETA: 1s - loss: 0.0513 - accuracy: 0.9837
1273/1688 [=====================>........] - ETA: 1s - loss: 0.0513 - accuracy: 0.9837
1293/1688 [=====================>........] - ETA: 1s - loss: 0.0519 - accuracy: 0.9836
1313/1688 [======================>.......] - ETA: 0s - loss: 0.0518 - accuracy: 0.9837
1333/1688 [======================>.......] - ETA: 0s - loss: 0.0518 - accuracy: 0.9837
1353/1688 [=======================>......] - ETA: 0s - loss: 0.0518 - accuracy: 0.9837
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0518 - accuracy: 0.9837
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0516 - accuracy: 0.9837
1411/1688 [========================>.....] - ETA: 0s - loss: 0.0520 - accuracy: 0.9836
1431/1688 [========================>.....] - ETA: 0s - loss: 0.0519 - accuracy: 0.9837
1451/1688 [========================>.....] - ETA: 0s - loss: 0.0519 - accuracy: 0.9837
1470/1688 [=========================>....] - ETA: 0s - loss: 0.0520 - accuracy: 0.9836
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0518 - accuracy: 0.9837
1508/1688 [=========================>....] - ETA: 0s - loss: 0.0518 - accuracy: 0.9837
1526/1688 [==========================>...] - ETA: 0s - loss: 0.0519 - accuracy: 0.9838
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0518 - accuracy: 0.9838
1562/1688 [==========================>...] - ETA: 0s - loss: 0.0519 - accuracy: 0.9838
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0522 - accuracy: 0.9837
1598/1688 [===========================>..] - ETA: 0s - loss: 0.0527 - accuracy: 0.9836
1616/1688 [===========================>..] - ETA: 0s - loss: 0.0526 - accuracy: 0.9836
1633/1688 [============================>.] - ETA: 0s - loss: 0.0526 - accuracy: 0.9836
1650/1688 [============================>.] - ETA: 0s - loss: 0.0528 - accuracy: 0.9835
1667/1688 [============================>.] - ETA: 0s - loss: 0.0527 - accuracy: 0.9836
1684/1688 [============================>.] - ETA: 0s - loss: 0.0526 - accuracy: 0.9837
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0525 - accuracy: 0.9837 - val_loss: 0.0564 - val_accuracy: 0.9843
Epoch 4/10

   1/1688 [..............................] - ETA: 4s - loss: 0.0018 - accuracy: 1.0000
  20/1688 [..............................] - ETA: 4s - loss: 0.0278 - accuracy: 0.9922
  39/1688 [..............................] - ETA: 4s - loss: 0.0392 - accuracy: 0.9904
  57/1688 [>.............................] - ETA: 4s - loss: 0.0388 - accuracy: 0.9885
  76/1688 [>.............................] - ETA: 4s - loss: 0.0401 - accuracy: 0.9873
  95/1688 [>.............................] - ETA: 4s - loss: 0.0419 - accuracy: 0.9862
 114/1688 [=>............................] - ETA: 4s - loss: 0.0405 - accuracy: 0.9860
 133/1688 [=>............................] - ETA: 4s - loss: 0.0418 - accuracy: 0.9854
 152/1688 [=>............................] - ETA: 4s - loss: 0.0424 - accuracy: 0.9852
 171/1688 [==>...........................] - ETA: 4s - loss: 0.0424 - accuracy: 0.9852
 190/1688 [==>...........................] - ETA: 4s - loss: 0.0424 - accuracy: 0.9850
 209/1688 [==>...........................] - ETA: 4s - loss: 0.0462 - accuracy: 0.9846
 228/1688 [===>..........................] - ETA: 3s - loss: 0.0464 - accuracy: 0.9848
 247/1688 [===>..........................] - ETA: 3s - loss: 0.0454 - accuracy: 0.9851
 266/1688 [===>..........................] - ETA: 3s - loss: 0.0440 - accuracy: 0.9857
 285/1688 [====>.........................] - ETA: 3s - loss: 0.0434 - accuracy: 0.9859
 304/1688 [====>.........................] - ETA: 3s - loss: 0.0421 - accuracy: 0.9861
 322/1688 [====>.........................] - ETA: 3s - loss: 0.0416 - accuracy: 0.9863
 341/1688 [=====>........................] - ETA: 3s - loss: 0.0403 - accuracy: 0.9866
 360/1688 [=====>........................] - ETA: 3s - loss: 0.0402 - accuracy: 0.9869
 379/1688 [=====>........................] - ETA: 3s - loss: 0.0400 - accuracy: 0.9868
 398/1688 [======>.......................] - ETA: 3s - loss: 0.0405 - accuracy: 0.9867
 417/1688 [======>.......................] - ETA: 3s - loss: 0.0411 - accuracy: 0.9866
 436/1688 [======>.......................] - ETA: 3s - loss: 0.0414 - accuracy: 0.9867
 455/1688 [=======>......................] - ETA: 3s - loss: 0.0404 - accuracy: 0.9871
 474/1688 [=======>......................] - ETA: 3s - loss: 0.0407 - accuracy: 0.9868
 493/1688 [=======>......................] - ETA: 3s - loss: 0.0400 - accuracy: 0.9871
 512/1688 [========>.....................] - ETA: 3s - loss: 0.0403 - accuracy: 0.9871
 531/1688 [========>.....................] - ETA: 3s - loss: 0.0405 - accuracy: 0.9868
 550/1688 [========>.....................] - ETA: 3s - loss: 0.0402 - accuracy: 0.9869
 569/1688 [=========>....................] - ETA: 3s - loss: 0.0401 - accuracy: 0.9871
 588/1688 [=========>....................] - ETA: 2s - loss: 0.0398 - accuracy: 0.9870
 607/1688 [=========>....................] - ETA: 2s - loss: 0.0398 - accuracy: 0.9871
 626/1688 [==========>...................] - ETA: 2s - loss: 0.0397 - accuracy: 0.9872
 645/1688 [==========>...................] - ETA: 2s - loss: 0.0403 - accuracy: 0.9871
 664/1688 [==========>...................] - ETA: 2s - loss: 0.0405 - accuracy: 0.9870
 683/1688 [===========>..................] - ETA: 2s - loss: 0.0401 - accuracy: 0.9871
 702/1688 [===========>..................] - ETA: 2s - loss: 0.0402 - accuracy: 0.9871
 721/1688 [===========>..................] - ETA: 2s - loss: 0.0400 - accuracy: 0.9872
 740/1688 [============>.................] - ETA: 2s - loss: 0.0400 - accuracy: 0.9872
 759/1688 [============>.................] - ETA: 2s - loss: 0.0398 - accuracy: 0.9873
 778/1688 [============>.................] - ETA: 2s - loss: 0.0397 - accuracy: 0.9874
 797/1688 [=============>................] - ETA: 2s - loss: 0.0402 - accuracy: 0.9873
 816/1688 [=============>................] - ETA: 2s - loss: 0.0400 - accuracy: 0.9873
 835/1688 [=============>................] - ETA: 2s - loss: 0.0399 - accuracy: 0.9872
 854/1688 [==============>...............] - ETA: 2s - loss: 0.0405 - accuracy: 0.9871
 873/1688 [==============>...............] - ETA: 2s - loss: 0.0405 - accuracy: 0.9871
 892/1688 [==============>...............] - ETA: 2s - loss: 0.0405 - accuracy: 0.9871
 911/1688 [===============>..............] - ETA: 2s - loss: 0.0401 - accuracy: 0.9873
 930/1688 [===============>..............] - ETA: 2s - loss: 0.0407 - accuracy: 0.9872
 949/1688 [===============>..............] - ETA: 2s - loss: 0.0411 - accuracy: 0.9870
 968/1688 [================>.............] - ETA: 1s - loss: 0.0414 - accuracy: 0.9870
 986/1688 [================>.............] - ETA: 1s - loss: 0.0415 - accuracy: 0.9868
1005/1688 [================>.............] - ETA: 1s - loss: 0.0417 - accuracy: 0.9867
1023/1688 [=================>............] - ETA: 1s - loss: 0.0415 - accuracy: 0.9866
1041/1688 [=================>............] - ETA: 1s - loss: 0.0417 - accuracy: 0.9864
1059/1688 [=================>............] - ETA: 1s - loss: 0.0417 - accuracy: 0.9865
1078/1688 [==================>...........] - ETA: 1s - loss: 0.0415 - accuracy: 0.9865
1097/1688 [==================>...........] - ETA: 1s - loss: 0.0414 - accuracy: 0.9866
1116/1688 [==================>...........] - ETA: 1s - loss: 0.0416 - accuracy: 0.9864
1135/1688 [===================>..........] - ETA: 1s - loss: 0.0420 - accuracy: 0.9863
1154/1688 [===================>..........] - ETA: 1s - loss: 0.0419 - accuracy: 0.9864
1173/1688 [===================>..........] - ETA: 1s - loss: 0.0422 - accuracy: 0.9862
1192/1688 [====================>.........] - ETA: 1s - loss: 0.0423 - accuracy: 0.9862
1211/1688 [====================>.........] - ETA: 1s - loss: 0.0428 - accuracy: 0.9861
1230/1688 [====================>.........] - ETA: 1s - loss: 0.0426 - accuracy: 0.9862
1249/1688 [=====================>........] - ETA: 1s - loss: 0.0428 - accuracy: 0.9861
1268/1688 [=====================>........] - ETA: 1s - loss: 0.0428 - accuracy: 0.9861
1286/1688 [=====================>........] - ETA: 1s - loss: 0.0427 - accuracy: 0.9862
1305/1688 [======================>.......] - ETA: 1s - loss: 0.0426 - accuracy: 0.9862
1324/1688 [======================>.......] - ETA: 0s - loss: 0.0422 - accuracy: 0.9863
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0422 - accuracy: 0.9864
1362/1688 [=======================>......] - ETA: 0s - loss: 0.0425 - accuracy: 0.9863
1381/1688 [=======================>......] - ETA: 0s - loss: 0.0424 - accuracy: 0.9863
1400/1688 [=======================>......] - ETA: 0s - loss: 0.0427 - accuracy: 0.9861
1419/1688 [========================>.....] - ETA: 0s - loss: 0.0427 - accuracy: 0.9861
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0429 - accuracy: 0.9861
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0434 - accuracy: 0.9861
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0433 - accuracy: 0.9861
1495/1688 [=========================>....] - ETA: 0s - loss: 0.0431 - accuracy: 0.9862
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0430 - accuracy: 0.9863
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0430 - accuracy: 0.9863
1552/1688 [==========================>...] - ETA: 0s - loss: 0.0429 - accuracy: 0.9863
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0429 - accuracy: 0.9864
1590/1688 [===========================>..] - ETA: 0s - loss: 0.0428 - accuracy: 0.9864
1609/1688 [===========================>..] - ETA: 0s - loss: 0.0428 - accuracy: 0.9864
1627/1688 [===========================>..] - ETA: 0s - loss: 0.0429 - accuracy: 0.9863
1646/1688 [============================>.] - ETA: 0s - loss: 0.0428 - accuracy: 0.9863
1665/1688 [============================>.] - ETA: 0s - loss: 0.0429 - accuracy: 0.9863
1687/1688 [============================>.] - ETA: 0s - loss: 0.0428 - accuracy: 0.9863
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0427 - accuracy: 0.9863 - val_loss: 0.0540 - val_accuracy: 0.9868
Epoch 5/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0028 - accuracy: 1.0000
  25/1688 [..............................] - ETA: 3s - loss: 0.0171 - accuracy: 0.9937
  48/1688 [..............................] - ETA: 3s - loss: 0.0252 - accuracy: 0.9909
  70/1688 [>.............................] - ETA: 3s - loss: 0.0294 - accuracy: 0.9902
  93/1688 [>.............................] - ETA: 3s - loss: 0.0280 - accuracy: 0.9899
 115/1688 [=>............................] - ETA: 3s - loss: 0.0286 - accuracy: 0.9908
 138/1688 [=>............................] - ETA: 3s - loss: 0.0269 - accuracy: 0.9916
 161/1688 [=>............................] - ETA: 3s - loss: 0.0257 - accuracy: 0.9922
 184/1688 [==>...........................] - ETA: 3s - loss: 0.0249 - accuracy: 0.9925
 207/1688 [==>...........................] - ETA: 3s - loss: 0.0252 - accuracy: 0.9925
 230/1688 [===>..........................] - ETA: 3s - loss: 0.0241 - accuracy: 0.9929
 253/1688 [===>..........................] - ETA: 3s - loss: 0.0239 - accuracy: 0.9930
 276/1688 [===>..........................] - ETA: 3s - loss: 0.0245 - accuracy: 0.9925
 299/1688 [====>.........................] - ETA: 3s - loss: 0.0255 - accuracy: 0.9924
 322/1688 [====>.........................] - ETA: 3s - loss: 0.0265 - accuracy: 0.9922
 346/1688 [=====>........................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9922
 369/1688 [=====>........................] - ETA: 2s - loss: 0.0270 - accuracy: 0.9918
 392/1688 [=====>........................] - ETA: 2s - loss: 0.0270 - accuracy: 0.9919
 415/1688 [======>.......................] - ETA: 2s - loss: 0.0278 - accuracy: 0.9918
 437/1688 [======>.......................] - ETA: 2s - loss: 0.0281 - accuracy: 0.9916
 459/1688 [=======>......................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9916
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0291 - accuracy: 0.9912
 504/1688 [=======>......................] - ETA: 2s - loss: 0.0292 - accuracy: 0.9912
 527/1688 [========>.....................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9912
 549/1688 [========>.....................] - ETA: 2s - loss: 0.0294 - accuracy: 0.9910
 572/1688 [=========>....................] - ETA: 2s - loss: 0.0293 - accuracy: 0.9910
 595/1688 [=========>....................] - ETA: 2s - loss: 0.0303 - accuracy: 0.9907
 616/1688 [=========>....................] - ETA: 2s - loss: 0.0307 - accuracy: 0.9905
 638/1688 [==========>...................] - ETA: 2s - loss: 0.0305 - accuracy: 0.9905
 659/1688 [==========>...................] - ETA: 2s - loss: 0.0304 - accuracy: 0.9905
 680/1688 [===========>..................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9904
 701/1688 [===========>..................] - ETA: 2s - loss: 0.0310 - accuracy: 0.9901
 723/1688 [===========>..................] - ETA: 2s - loss: 0.0320 - accuracy: 0.9900
 744/1688 [============>.................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9901
 765/1688 [============>.................] - ETA: 2s - loss: 0.0317 - accuracy: 0.9901
 786/1688 [============>.................] - ETA: 2s - loss: 0.0319 - accuracy: 0.9900
 806/1688 [=============>................] - ETA: 2s - loss: 0.0319 - accuracy: 0.9900
 826/1688 [=============>................] - ETA: 1s - loss: 0.0318 - accuracy: 0.9900
 845/1688 [==============>...............] - ETA: 1s - loss: 0.0317 - accuracy: 0.9899
 865/1688 [==============>...............] - ETA: 1s - loss: 0.0316 - accuracy: 0.9900
 885/1688 [==============>...............] - ETA: 1s - loss: 0.0313 - accuracy: 0.9901
 905/1688 [===============>..............] - ETA: 1s - loss: 0.0310 - accuracy: 0.9902
 925/1688 [===============>..............] - ETA: 1s - loss: 0.0311 - accuracy: 0.9902
 945/1688 [===============>..............] - ETA: 1s - loss: 0.0317 - accuracy: 0.9900
 965/1688 [================>.............] - ETA: 1s - loss: 0.0315 - accuracy: 0.9901
 985/1688 [================>.............] - ETA: 1s - loss: 0.0316 - accuracy: 0.9900
1005/1688 [================>.............] - ETA: 1s - loss: 0.0316 - accuracy: 0.9899
1025/1688 [=================>............] - ETA: 1s - loss: 0.0320 - accuracy: 0.9898
1044/1688 [=================>............] - ETA: 1s - loss: 0.0318 - accuracy: 0.9898
1064/1688 [=================>............] - ETA: 1s - loss: 0.0322 - accuracy: 0.9898
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0320 - accuracy: 0.9899
1104/1688 [==================>...........] - ETA: 1s - loss: 0.0326 - accuracy: 0.9898
1124/1688 [==================>...........] - ETA: 1s - loss: 0.0326 - accuracy: 0.9897
1144/1688 [===================>..........] - ETA: 1s - loss: 0.0326 - accuracy: 0.9898
1164/1688 [===================>..........] - ETA: 1s - loss: 0.0327 - accuracy: 0.9898
1184/1688 [====================>.........] - ETA: 1s - loss: 0.0329 - accuracy: 0.9898
1204/1688 [====================>.........] - ETA: 1s - loss: 0.0327 - accuracy: 0.9899
1224/1688 [====================>.........] - ETA: 1s - loss: 0.0324 - accuracy: 0.9899
1244/1688 [=====================>........] - ETA: 1s - loss: 0.0323 - accuracy: 0.9899
1264/1688 [=====================>........] - ETA: 1s - loss: 0.0322 - accuracy: 0.9899
1283/1688 [=====================>........] - ETA: 0s - loss: 0.0324 - accuracy: 0.9898
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0325 - accuracy: 0.9898
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0326 - accuracy: 0.9897
1343/1688 [======================>.......] - ETA: 0s - loss: 0.0325 - accuracy: 0.9898
1363/1688 [=======================>......] - ETA: 0s - loss: 0.0325 - accuracy: 0.9898
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0327 - accuracy: 0.9897
1403/1688 [=======================>......] - ETA: 0s - loss: 0.0329 - accuracy: 0.9897
1423/1688 [========================>.....] - ETA: 0s - loss: 0.0332 - accuracy: 0.9896
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0332 - accuracy: 0.9896
1463/1688 [=========================>....] - ETA: 0s - loss: 0.0331 - accuracy: 0.9896
1483/1688 [=========================>....] - ETA: 0s - loss: 0.0332 - accuracy: 0.9896
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0334 - accuracy: 0.9895
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0332 - accuracy: 0.9896
1543/1688 [==========================>...] - ETA: 0s - loss: 0.0332 - accuracy: 0.9895
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0331 - accuracy: 0.9895
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0331 - accuracy: 0.9895
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0331 - accuracy: 0.9894
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0335 - accuracy: 0.9894
1641/1688 [============================>.] - ETA: 0s - loss: 0.0335 - accuracy: 0.9893
1661/1688 [============================>.] - ETA: 0s - loss: 0.0337 - accuracy: 0.9892
1680/1688 [============================>.] - ETA: 0s - loss: 0.0337 - accuracy: 0.9892
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0338 - accuracy: 0.9891 - val_loss: 0.0636 - val_accuracy: 0.9815
Epoch 6/10

   1/1688 [..............................] - ETA: 4s - loss: 0.0300 - accuracy: 0.9688
  21/1688 [..............................] - ETA: 4s - loss: 0.0167 - accuracy: 0.9940
  41/1688 [..............................] - ETA: 4s - loss: 0.0173 - accuracy: 0.9931
  61/1688 [>.............................] - ETA: 4s - loss: 0.0214 - accuracy: 0.9928
  81/1688 [>.............................] - ETA: 4s - loss: 0.0198 - accuracy: 0.9938
 101/1688 [>.............................] - ETA: 4s - loss: 0.0193 - accuracy: 0.9935
 121/1688 [=>............................] - ETA: 4s - loss: 0.0206 - accuracy: 0.9930
 141/1688 [=>............................] - ETA: 4s - loss: 0.0217 - accuracy: 0.9927
 161/1688 [=>............................] - ETA: 3s - loss: 0.0206 - accuracy: 0.9932
 181/1688 [==>...........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9931
 201/1688 [==>...........................] - ETA: 3s - loss: 0.0211 - accuracy: 0.9933
 221/1688 [==>...........................] - ETA: 3s - loss: 0.0213 - accuracy: 0.9929
 241/1688 [===>..........................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9929
 261/1688 [===>..........................] - ETA: 3s - loss: 0.0212 - accuracy: 0.9929
 281/1688 [===>..........................] - ETA: 3s - loss: 0.0215 - accuracy: 0.9924
 301/1688 [====>.........................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9925
 321/1688 [====>.........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9923
 340/1688 [=====>........................] - ETA: 3s - loss: 0.0217 - accuracy: 0.9925
 358/1688 [=====>........................] - ETA: 3s - loss: 0.0215 - accuracy: 0.9926
 379/1688 [=====>........................] - ETA: 3s - loss: 0.0219 - accuracy: 0.9927
 401/1688 [======>.......................] - ETA: 3s - loss: 0.0225 - accuracy: 0.9924
 424/1688 [======>.......................] - ETA: 3s - loss: 0.0224 - accuracy: 0.9926
 447/1688 [======>.......................] - ETA: 3s - loss: 0.0224 - accuracy: 0.9925
 470/1688 [=======>......................] - ETA: 3s - loss: 0.0238 - accuracy: 0.9926
 493/1688 [=======>......................] - ETA: 3s - loss: 0.0241 - accuracy: 0.9924
 516/1688 [========>.....................] - ETA: 2s - loss: 0.0242 - accuracy: 0.9923
 539/1688 [========>.....................] - ETA: 2s - loss: 0.0246 - accuracy: 0.9922
 562/1688 [========>.....................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9923
 585/1688 [=========>....................] - ETA: 2s - loss: 0.0256 - accuracy: 0.9920
 608/1688 [=========>....................] - ETA: 2s - loss: 0.0260 - accuracy: 0.9919
 631/1688 [==========>...................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9918
 654/1688 [==========>...................] - ETA: 2s - loss: 0.0265 - accuracy: 0.9915
 677/1688 [===========>..................] - ETA: 2s - loss: 0.0266 - accuracy: 0.9914
 699/1688 [===========>..................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9916
 721/1688 [===========>..................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9915
 744/1688 [============>.................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9916
 767/1688 [============>.................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9916
 790/1688 [=============>................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9917
 812/1688 [=============>................] - ETA: 2s - loss: 0.0264 - accuracy: 0.9916
 835/1688 [=============>................] - ETA: 2s - loss: 0.0261 - accuracy: 0.9918
 857/1688 [==============>...............] - ETA: 1s - loss: 0.0258 - accuracy: 0.9919
 880/1688 [==============>...............] - ETA: 1s - loss: 0.0256 - accuracy: 0.9920
 902/1688 [===============>..............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9920
 924/1688 [===============>..............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9920
 947/1688 [===============>..............] - ETA: 1s - loss: 0.0255 - accuracy: 0.9919
 970/1688 [================>.............] - ETA: 1s - loss: 0.0259 - accuracy: 0.9919
 993/1688 [================>.............] - ETA: 1s - loss: 0.0261 - accuracy: 0.9918
1016/1688 [=================>............] - ETA: 1s - loss: 0.0262 - accuracy: 0.9918
1039/1688 [=================>............] - ETA: 1s - loss: 0.0260 - accuracy: 0.9919
1062/1688 [=================>............] - ETA: 1s - loss: 0.0260 - accuracy: 0.9918
1084/1688 [==================>...........] - ETA: 1s - loss: 0.0262 - accuracy: 0.9918
1106/1688 [==================>...........] - ETA: 1s - loss: 0.0263 - accuracy: 0.9917
1127/1688 [===================>..........] - ETA: 1s - loss: 0.0262 - accuracy: 0.9918
1146/1688 [===================>..........] - ETA: 1s - loss: 0.0261 - accuracy: 0.9918
1165/1688 [===================>..........] - ETA: 1s - loss: 0.0269 - accuracy: 0.9916
1185/1688 [====================>.........] - ETA: 1s - loss: 0.0269 - accuracy: 0.9916
1205/1688 [====================>.........] - ETA: 1s - loss: 0.0270 - accuracy: 0.9915
1225/1688 [====================>.........] - ETA: 1s - loss: 0.0269 - accuracy: 0.9915
1245/1688 [=====================>........] - ETA: 1s - loss: 0.0270 - accuracy: 0.9914
1265/1688 [=====================>........] - ETA: 1s - loss: 0.0270 - accuracy: 0.9914
1286/1688 [=====================>........] - ETA: 0s - loss: 0.0270 - accuracy: 0.9914
1306/1688 [======================>.......] - ETA: 0s - loss: 0.0269 - accuracy: 0.9914
1326/1688 [======================>.......] - ETA: 0s - loss: 0.0271 - accuracy: 0.9913
1346/1688 [======================>.......] - ETA: 0s - loss: 0.0273 - accuracy: 0.9913
1366/1688 [=======================>......] - ETA: 0s - loss: 0.0274 - accuracy: 0.9912
1386/1688 [=======================>......] - ETA: 0s - loss: 0.0277 - accuracy: 0.9910
1406/1688 [=======================>......] - ETA: 0s - loss: 0.0280 - accuracy: 0.9910
1425/1688 [========================>.....] - ETA: 0s - loss: 0.0283 - accuracy: 0.9909
1445/1688 [========================>.....] - ETA: 0s - loss: 0.0283 - accuracy: 0.9909
1465/1688 [=========================>....] - ETA: 0s - loss: 0.0284 - accuracy: 0.9909
1485/1688 [=========================>....] - ETA: 0s - loss: 0.0283 - accuracy: 0.9908
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0282 - accuracy: 0.9908
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0281 - accuracy: 0.9908
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0281 - accuracy: 0.9909
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0279 - accuracy: 0.9909
1585/1688 [===========================>..] - ETA: 0s - loss: 0.0280 - accuracy: 0.9909
1605/1688 [===========================>..] - ETA: 0s - loss: 0.0281 - accuracy: 0.9909
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0281 - accuracy: 0.9909
1646/1688 [============================>.] - ETA: 0s - loss: 0.0281 - accuracy: 0.9908
1665/1688 [============================>.] - ETA: 0s - loss: 0.0281 - accuracy: 0.9908
1685/1688 [============================>.] - ETA: 0s - loss: 0.0282 - accuracy: 0.9908
1688/1688 [==============================] - 4s 3ms/step - loss: 0.0282 - accuracy: 0.9908 - val_loss: 0.0586 - val_accuracy: 0.9828
Epoch 7/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0854 - accuracy: 0.9688
  24/1688 [..............................] - ETA: 3s - loss: 0.0230 - accuracy: 0.9909
  46/1688 [..............................] - ETA: 3s - loss: 0.0230 - accuracy: 0.9905
  69/1688 [>.............................] - ETA: 3s - loss: 0.0263 - accuracy: 0.9905
  91/1688 [>.............................] - ETA: 3s - loss: 0.0222 - accuracy: 0.9921
 114/1688 [=>............................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9923
 137/1688 [=>............................] - ETA: 3s - loss: 0.0199 - accuracy: 0.9927
 160/1688 [=>............................] - ETA: 3s - loss: 0.0179 - accuracy: 0.9937
 183/1688 [==>...........................] - ETA: 3s - loss: 0.0187 - accuracy: 0.9937
 205/1688 [==>...........................] - ETA: 3s - loss: 0.0183 - accuracy: 0.9937
 227/1688 [===>..........................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9935
 250/1688 [===>..........................] - ETA: 3s - loss: 0.0191 - accuracy: 0.9940
 272/1688 [===>..........................] - ETA: 3s - loss: 0.0183 - accuracy: 0.9940
 295/1688 [====>.........................] - ETA: 3s - loss: 0.0176 - accuracy: 0.9943
 318/1688 [====>.........................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9943
 341/1688 [=====>........................] - ETA: 3s - loss: 0.0176 - accuracy: 0.9939
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0170 - accuracy: 0.9942
 387/1688 [=====>........................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9939
 410/1688 [======>.......................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9939
 433/1688 [======>.......................] - ETA: 2s - loss: 0.0181 - accuracy: 0.9938
 456/1688 [=======>......................] - ETA: 2s - loss: 0.0179 - accuracy: 0.9938
 478/1688 [=======>......................] - ETA: 2s - loss: 0.0186 - accuracy: 0.9935
 501/1688 [=======>......................] - ETA: 2s - loss: 0.0185 - accuracy: 0.9935
 524/1688 [========>.....................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9933
 547/1688 [========>.....................] - ETA: 2s - loss: 0.0192 - accuracy: 0.9932
 570/1688 [=========>....................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9929
 592/1688 [=========>....................] - ETA: 2s - loss: 0.0199 - accuracy: 0.9929
 615/1688 [=========>....................] - ETA: 2s - loss: 0.0198 - accuracy: 0.9930
 638/1688 [==========>...................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9929
 660/1688 [==========>...................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9929
 683/1688 [===========>..................] - ETA: 2s - loss: 0.0205 - accuracy: 0.9928
 705/1688 [===========>..................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9928
 728/1688 [===========>..................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9929
 751/1688 [============>.................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9930
 774/1688 [============>.................] - ETA: 2s - loss: 0.0203 - accuracy: 0.9929
 797/1688 [=============>................] - ETA: 2s - loss: 0.0202 - accuracy: 0.9929
 820/1688 [=============>................] - ETA: 1s - loss: 0.0201 - accuracy: 0.9929
 842/1688 [=============>................] - ETA: 1s - loss: 0.0201 - accuracy: 0.9930
 865/1688 [==============>...............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9930
 888/1688 [==============>...............] - ETA: 1s - loss: 0.0200 - accuracy: 0.9930
 911/1688 [===============>..............] - ETA: 1s - loss: 0.0202 - accuracy: 0.9929
 934/1688 [===============>..............] - ETA: 1s - loss: 0.0202 - accuracy: 0.9929
 957/1688 [================>.............] - ETA: 1s - loss: 0.0201 - accuracy: 0.9930
 980/1688 [================>.............] - ETA: 1s - loss: 0.0204 - accuracy: 0.9929
1003/1688 [================>.............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9929
1026/1688 [=================>............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9929
1049/1688 [=================>............] - ETA: 1s - loss: 0.0206 - accuracy: 0.9929
1072/1688 [==================>...........] - ETA: 1s - loss: 0.0207 - accuracy: 0.9928
1094/1688 [==================>...........] - ETA: 1s - loss: 0.0207 - accuracy: 0.9927
1117/1688 [==================>...........] - ETA: 1s - loss: 0.0211 - accuracy: 0.9926
1140/1688 [===================>..........] - ETA: 1s - loss: 0.0210 - accuracy: 0.9926
1163/1688 [===================>..........] - ETA: 1s - loss: 0.0212 - accuracy: 0.9925
1186/1688 [====================>.........] - ETA: 1s - loss: 0.0214 - accuracy: 0.9926
1209/1688 [====================>.........] - ETA: 1s - loss: 0.0214 - accuracy: 0.9926
1232/1688 [====================>.........] - ETA: 1s - loss: 0.0219 - accuracy: 0.9925
1255/1688 [=====================>........] - ETA: 0s - loss: 0.0218 - accuracy: 0.9925
1277/1688 [=====================>........] - ETA: 0s - loss: 0.0221 - accuracy: 0.9925
1299/1688 [======================>.......] - ETA: 0s - loss: 0.0225 - accuracy: 0.9925
1319/1688 [======================>.......] - ETA: 0s - loss: 0.0228 - accuracy: 0.9923
1339/1688 [======================>.......] - ETA: 0s - loss: 0.0225 - accuracy: 0.9924
1359/1688 [=======================>......] - ETA: 0s - loss: 0.0226 - accuracy: 0.9924
1378/1688 [=======================>......] - ETA: 0s - loss: 0.0228 - accuracy: 0.9923
1398/1688 [=======================>......] - ETA: 0s - loss: 0.0230 - accuracy: 0.9923
1418/1688 [========================>.....] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1437/1688 [========================>.....] - ETA: 0s - loss: 0.0232 - accuracy: 0.9922
1457/1688 [========================>.....] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0233 - accuracy: 0.9922
1517/1688 [=========================>....] - ETA: 0s - loss: 0.0232 - accuracy: 0.9922
1537/1688 [==========================>...] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1597/1688 [===========================>..] - ETA: 0s - loss: 0.0232 - accuracy: 0.9921
1617/1688 [===========================>..] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1637/1688 [============================>.] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1657/1688 [============================>.] - ETA: 0s - loss: 0.0231 - accuracy: 0.9922
1676/1688 [============================>.] - ETA: 0s - loss: 0.0232 - accuracy: 0.9922
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0232 - accuracy: 0.9922 - val_loss: 0.0464 - val_accuracy: 0.9890
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0138 - accuracy: 1.0000
  24/1688 [..............................] - ETA: 3s - loss: 0.0062 - accuracy: 1.0000
  46/1688 [..............................] - ETA: 3s - loss: 0.0091 - accuracy: 0.9986
  69/1688 [>.............................] - ETA: 3s - loss: 0.0112 - accuracy: 0.9977
  91/1688 [>.............................] - ETA: 3s - loss: 0.0121 - accuracy: 0.9969
 114/1688 [=>............................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9970
 137/1688 [=>............................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9968
 159/1688 [=>............................] - ETA: 3s - loss: 0.0122 - accuracy: 0.9965
 181/1688 [==>...........................] - ETA: 3s - loss: 0.0114 - accuracy: 0.9967
 203/1688 [==>...........................] - ETA: 3s - loss: 0.0121 - accuracy: 0.9963
 225/1688 [==>...........................] - ETA: 3s - loss: 0.0121 - accuracy: 0.9961
 248/1688 [===>..........................] - ETA: 3s - loss: 0.0132 - accuracy: 0.9956
 271/1688 [===>..........................] - ETA: 3s - loss: 0.0124 - accuracy: 0.9960
 294/1688 [====>.........................] - ETA: 3s - loss: 0.0122 - accuracy: 0.9960
 317/1688 [====>.........................] - ETA: 3s - loss: 0.0121 - accuracy: 0.9960
 340/1688 [=====>........................] - ETA: 3s - loss: 0.0125 - accuracy: 0.9957
 363/1688 [=====>........................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9955
 386/1688 [=====>........................] - ETA: 2s - loss: 0.0131 - accuracy: 0.9953
 409/1688 [======>.......................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9953
 431/1688 [======>.......................] - ETA: 2s - loss: 0.0127 - accuracy: 0.9954
 454/1688 [=======>......................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9954
 477/1688 [=======>......................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9954
 500/1688 [=======>......................] - ETA: 2s - loss: 0.0134 - accuracy: 0.9951
 523/1688 [========>.....................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9949
 545/1688 [========>.....................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9949
 568/1688 [=========>....................] - ETA: 2s - loss: 0.0147 - accuracy: 0.9949
 591/1688 [=========>....................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9951
 614/1688 [=========>....................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9951
 637/1688 [==========>...................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9952
 660/1688 [==========>...................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9953
 683/1688 [===========>..................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9952
 706/1688 [===========>..................] - ETA: 2s - loss: 0.0145 - accuracy: 0.9951
 729/1688 [===========>..................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9949
 752/1688 [============>.................] - ETA: 2s - loss: 0.0148 - accuracy: 0.9951
 775/1688 [============>.................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9950
 798/1688 [=============>................] - ETA: 2s - loss: 0.0149 - accuracy: 0.9950
 821/1688 [=============>................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9950
 844/1688 [==============>...............] - ETA: 1s - loss: 0.0154 - accuracy: 0.9949
 866/1688 [==============>...............] - ETA: 1s - loss: 0.0158 - accuracy: 0.9949
 889/1688 [==============>...............] - ETA: 1s - loss: 0.0160 - accuracy: 0.9948
 912/1688 [===============>..............] - ETA: 1s - loss: 0.0162 - accuracy: 0.9948
 936/1688 [===============>..............] - ETA: 1s - loss: 0.0161 - accuracy: 0.9948
 958/1688 [================>.............] - ETA: 1s - loss: 0.0163 - accuracy: 0.9948
 981/1688 [================>.............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9948
1003/1688 [================>.............] - ETA: 1s - loss: 0.0166 - accuracy: 0.9947
1026/1688 [=================>............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9948
1049/1688 [=================>............] - ETA: 1s - loss: 0.0165 - accuracy: 0.9947
1072/1688 [==================>...........] - ETA: 1s - loss: 0.0164 - accuracy: 0.9947
1095/1688 [==================>...........] - ETA: 1s - loss: 0.0163 - accuracy: 0.9947
1118/1688 [==================>...........] - ETA: 1s - loss: 0.0163 - accuracy: 0.9948
1141/1688 [===================>..........] - ETA: 1s - loss: 0.0164 - accuracy: 0.9947
1164/1688 [===================>..........] - ETA: 1s - loss: 0.0168 - accuracy: 0.9946
1186/1688 [====================>.........] - ETA: 1s - loss: 0.0170 - accuracy: 0.9946
1209/1688 [====================>.........] - ETA: 1s - loss: 0.0170 - accuracy: 0.9946
1232/1688 [====================>.........] - ETA: 1s - loss: 0.0169 - accuracy: 0.9946
1255/1688 [=====================>........] - ETA: 0s - loss: 0.0173 - accuracy: 0.9945
1278/1688 [=====================>........] - ETA: 0s - loss: 0.0174 - accuracy: 0.9945
1301/1688 [======================>.......] - ETA: 0s - loss: 0.0179 - accuracy: 0.9943
1324/1688 [======================>.......] - ETA: 0s - loss: 0.0181 - accuracy: 0.9942
1346/1688 [======================>.......] - ETA: 0s - loss: 0.0185 - accuracy: 0.9941
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0189 - accuracy: 0.9940
1392/1688 [=======================>......] - ETA: 0s - loss: 0.0190 - accuracy: 0.9940
1415/1688 [========================>.....] - ETA: 0s - loss: 0.0190 - accuracy: 0.9940
1438/1688 [========================>.....] - ETA: 0s - loss: 0.0191 - accuracy: 0.9939
1461/1688 [========================>.....] - ETA: 0s - loss: 0.0190 - accuracy: 0.9939
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0189 - accuracy: 0.9939
1507/1688 [=========================>....] - ETA: 0s - loss: 0.0192 - accuracy: 0.9938
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0195 - accuracy: 0.9938
1553/1688 [==========================>...] - ETA: 0s - loss: 0.0196 - accuracy: 0.9938
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0202 - accuracy: 0.9936
1599/1688 [===========================>..] - ETA: 0s - loss: 0.0204 - accuracy: 0.9936
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0206 - accuracy: 0.9935
1644/1688 [============================>.] - ETA: 0s - loss: 0.0205 - accuracy: 0.9935
1667/1688 [============================>.] - ETA: 0s - loss: 0.0207 - accuracy: 0.9935
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0207 - accuracy: 0.9935 - val_loss: 0.0588 - val_accuracy: 0.9853
Epoch 9/10

   1/1688 [..............................] - ETA: 3s - loss: 3.4678e-04 - accuracy: 1.0000
  24/1688 [..............................] - ETA: 3s - loss: 0.0131 - accuracy: 0.9974    
  47/1688 [..............................] - ETA: 3s - loss: 0.0216 - accuracy: 0.9960
  70/1688 [>.............................] - ETA: 3s - loss: 0.0190 - accuracy: 0.9955
  93/1688 [>.............................] - ETA: 3s - loss: 0.0174 - accuracy: 0.9953
 116/1688 [=>............................] - ETA: 3s - loss: 0.0171 - accuracy: 0.9954
 139/1688 [=>............................] - ETA: 3s - loss: 0.0159 - accuracy: 0.9957
 162/1688 [=>............................] - ETA: 3s - loss: 0.0164 - accuracy: 0.9956
 184/1688 [==>...........................] - ETA: 3s - loss: 0.0168 - accuracy: 0.9952
 206/1688 [==>...........................] - ETA: 3s - loss: 0.0154 - accuracy: 0.9958
 229/1688 [===>..........................] - ETA: 3s - loss: 0.0158 - accuracy: 0.9955
 252/1688 [===>..........................] - ETA: 3s - loss: 0.0151 - accuracy: 0.9958
 275/1688 [===>..........................] - ETA: 3s - loss: 0.0152 - accuracy: 0.9957
 298/1688 [====>.........................] - ETA: 3s - loss: 0.0146 - accuracy: 0.9958
 321/1688 [====>.........................] - ETA: 3s - loss: 0.0142 - accuracy: 0.9958
 344/1688 [=====>........................] - ETA: 3s - loss: 0.0139 - accuracy: 0.9959
 367/1688 [=====>........................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9957
 390/1688 [=====>........................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9957
 413/1688 [======>.......................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9957
 436/1688 [======>.......................] - ETA: 2s - loss: 0.0137 - accuracy: 0.9959
 459/1688 [=======>......................] - ETA: 2s - loss: 0.0137 - accuracy: 0.9960
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9960
 504/1688 [=======>......................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9960
 527/1688 [========>.....................] - ETA: 2s - loss: 0.0137 - accuracy: 0.9958
 550/1688 [========>.....................] - ETA: 2s - loss: 0.0137 - accuracy: 0.9959
 573/1688 [=========>....................] - ETA: 2s - loss: 0.0135 - accuracy: 0.9959
 596/1688 [=========>....................] - ETA: 2s - loss: 0.0139 - accuracy: 0.9958
 619/1688 [==========>...................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9957
 641/1688 [==========>...................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9958
 663/1688 [==========>...................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9957
 686/1688 [===========>..................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9957
 709/1688 [===========>..................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9958
 732/1688 [============>.................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9957
 755/1688 [============>.................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9957
 778/1688 [============>.................] - ETA: 2s - loss: 0.0143 - accuracy: 0.9957
 802/1688 [=============>................] - ETA: 1s - loss: 0.0143 - accuracy: 0.9957
 825/1688 [=============>................] - ETA: 1s - loss: 0.0144 - accuracy: 0.9957
 848/1688 [==============>...............] - ETA: 1s - loss: 0.0143 - accuracy: 0.9957
 871/1688 [==============>...............] - ETA: 1s - loss: 0.0146 - accuracy: 0.9955
 894/1688 [==============>...............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9956
 917/1688 [===============>..............] - ETA: 1s - loss: 0.0145 - accuracy: 0.9955
 940/1688 [===============>..............] - ETA: 1s - loss: 0.0147 - accuracy: 0.9954
 963/1688 [================>.............] - ETA: 1s - loss: 0.0150 - accuracy: 0.9954
 985/1688 [================>.............] - ETA: 1s - loss: 0.0149 - accuracy: 0.9954
1008/1688 [================>.............] - ETA: 1s - loss: 0.0150 - accuracy: 0.9953
1031/1688 [=================>............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9952
1054/1688 [=================>............] - ETA: 1s - loss: 0.0152 - accuracy: 0.9953
1076/1688 [==================>...........] - ETA: 1s - loss: 0.0152 - accuracy: 0.9953
1099/1688 [==================>...........] - ETA: 1s - loss: 0.0152 - accuracy: 0.9953
1122/1688 [==================>...........] - ETA: 1s - loss: 0.0155 - accuracy: 0.9952
1145/1688 [===================>..........] - ETA: 1s - loss: 0.0154 - accuracy: 0.9952
1167/1688 [===================>..........] - ETA: 1s - loss: 0.0158 - accuracy: 0.9951
1190/1688 [====================>.........] - ETA: 1s - loss: 0.0159 - accuracy: 0.9950
1212/1688 [====================>.........] - ETA: 1s - loss: 0.0162 - accuracy: 0.9949
1234/1688 [====================>.........] - ETA: 1s - loss: 0.0161 - accuracy: 0.9950
1257/1688 [=====================>........] - ETA: 0s - loss: 0.0160 - accuracy: 0.9950
1280/1688 [=====================>........] - ETA: 0s - loss: 0.0160 - accuracy: 0.9950
1303/1688 [======================>.......] - ETA: 0s - loss: 0.0160 - accuracy: 0.9950
1326/1688 [======================>.......] - ETA: 0s - loss: 0.0160 - accuracy: 0.9949
1349/1688 [======================>.......] - ETA: 0s - loss: 0.0161 - accuracy: 0.9949
1372/1688 [=======================>......] - ETA: 0s - loss: 0.0160 - accuracy: 0.9949
1394/1688 [=======================>......] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1417/1688 [========================>.....] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948
1462/1688 [========================>.....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9947
1484/1688 [=========================>....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9947
1506/1688 [=========================>....] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948
1529/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948
1552/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948
1575/1688 [==========================>...] - ETA: 0s - loss: 0.0164 - accuracy: 0.9947
1598/1688 [===========================>..] - ETA: 0s - loss: 0.0163 - accuracy: 0.9947
1620/1688 [===========================>..] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948
1643/1688 [============================>.] - ETA: 0s - loss: 0.0161 - accuracy: 0.9948
1666/1688 [============================>.] - ETA: 0s - loss: 0.0160 - accuracy: 0.9948
1688/1688 [==============================] - 4s 2ms/step - loss: 0.0162 - accuracy: 0.9948 - val_loss: 0.0596 - val_accuracy: 0.9868
Epoch 10/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0272 - accuracy: 1.0000
  24/1688 [..............................] - ETA: 3s - loss: 0.0163 - accuracy: 0.9922
  46/1688 [..............................] - ETA: 3s - loss: 0.0141 - accuracy: 0.9939
  69/1688 [>.............................] - ETA: 3s - loss: 0.0105 - accuracy: 0.9959
  92/1688 [>.............................] - ETA: 3s - loss: 0.0109 - accuracy: 0.9963
 114/1688 [=>............................] - ETA: 3s - loss: 0.0105 - accuracy: 0.9964
 134/1688 [=>............................] - ETA: 3s - loss: 0.0112 - accuracy: 0.9967
 153/1688 [=>............................] - ETA: 3s - loss: 0.0109 - accuracy: 0.9969
 173/1688 [==>...........................] - ETA: 3s - loss: 0.0113 - accuracy: 0.9966
 193/1688 [==>...........................] - ETA: 3s - loss: 0.0113 - accuracy: 0.9964
 213/1688 [==>...........................] - ETA: 3s - loss: 0.0122 - accuracy: 0.9962
 233/1688 [===>..........................] - ETA: 3s - loss: 0.0121 - accuracy: 0.9961
 252/1688 [===>..........................] - ETA: 3s - loss: 0.0119 - accuracy: 0.9959
 272/1688 [===>..........................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9959
 292/1688 [====>.........................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9959
 312/1688 [====>.........................] - ETA: 3s - loss: 0.0114 - accuracy: 0.9961
 331/1688 [====>.........................] - ETA: 3s - loss: 0.0112 - accuracy: 0.9961
 351/1688 [=====>........................] - ETA: 3s - loss: 0.0109 - accuracy: 0.9963
 371/1688 [=====>........................] - ETA: 3s - loss: 0.0113 - accuracy: 0.9960
 391/1688 [=====>........................] - ETA: 3s - loss: 0.0111 - accuracy: 0.9962
 411/1688 [======>.......................] - ETA: 3s - loss: 0.0109 - accuracy: 0.9964
 431/1688 [======>.......................] - ETA: 3s - loss: 0.0112 - accuracy: 0.9960
 451/1688 [=======>......................] - ETA: 3s - loss: 0.0109 - accuracy: 0.9961
 471/1688 [=======>......................] - ETA: 3s - loss: 0.0113 - accuracy: 0.9961
 491/1688 [=======>......................] - ETA: 3s - loss: 0.0118 - accuracy: 0.9959
 511/1688 [========>.....................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9958
 531/1688 [========>.....................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9958
 551/1688 [========>.....................] - ETA: 2s - loss: 0.0127 - accuracy: 0.9956
 571/1688 [=========>....................] - ETA: 2s - loss: 0.0126 - accuracy: 0.9956
 591/1688 [=========>....................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9957
 611/1688 [=========>....................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9957
 631/1688 [==========>...................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9956
 651/1688 [==========>...................] - ETA: 2s - loss: 0.0125 - accuracy: 0.9956
 671/1688 [==========>...................] - ETA: 2s - loss: 0.0123 - accuracy: 0.9956
 691/1688 [===========>..................] - ETA: 2s - loss: 0.0123 - accuracy: 0.9956
 711/1688 [===========>..................] - ETA: 2s - loss: 0.0123 - accuracy: 0.9956
 731/1688 [===========>..................] - ETA: 2s - loss: 0.0124 - accuracy: 0.9956
 751/1688 [============>.................] - ETA: 2s - loss: 0.0128 - accuracy: 0.9956
 770/1688 [============>.................] - ETA: 2s - loss: 0.0129 - accuracy: 0.9956
 789/1688 [=============>................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9955
 809/1688 [=============>................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9954
 829/1688 [=============>................] - ETA: 2s - loss: 0.0133 - accuracy: 0.9954
 849/1688 [==============>...............] - ETA: 2s - loss: 0.0134 - accuracy: 0.9954
 869/1688 [==============>...............] - ETA: 2s - loss: 0.0134 - accuracy: 0.9954
 889/1688 [==============>...............] - ETA: 2s - loss: 0.0136 - accuracy: 0.9953
 909/1688 [===============>..............] - ETA: 1s - loss: 0.0137 - accuracy: 0.9952
 929/1688 [===============>..............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9953
 949/1688 [===============>..............] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
 969/1688 [================>.............] - ETA: 1s - loss: 0.0137 - accuracy: 0.9952
 989/1688 [================>.............] - ETA: 1s - loss: 0.0137 - accuracy: 0.9952
1009/1688 [================>.............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9953
1030/1688 [=================>............] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
1050/1688 [=================>............] - ETA: 1s - loss: 0.0136 - accuracy: 0.9953
1070/1688 [==================>...........] - ETA: 1s - loss: 0.0135 - accuracy: 0.9953
1090/1688 [==================>...........] - ETA: 1s - loss: 0.0139 - accuracy: 0.9952
1110/1688 [==================>...........] - ETA: 1s - loss: 0.0138 - accuracy: 0.9953
1130/1688 [===================>..........] - ETA: 1s - loss: 0.0145 - accuracy: 0.9952
1150/1688 [===================>..........] - ETA: 1s - loss: 0.0146 - accuracy: 0.9951
1170/1688 [===================>..........] - ETA: 1s - loss: 0.0148 - accuracy: 0.9951
1190/1688 [====================>.........] - ETA: 1s - loss: 0.0149 - accuracy: 0.9950
1210/1688 [====================>.........] - ETA: 1s - loss: 0.0151 - accuracy: 0.9949
1230/1688 [====================>.........] - ETA: 1s - loss: 0.0152 - accuracy: 0.9949
1250/1688 [=====================>........] - ETA: 1s - loss: 0.0152 - accuracy: 0.9948
1270/1688 [=====================>........] - ETA: 1s - loss: 0.0152 - accuracy: 0.9948
1290/1688 [=====================>........] - ETA: 1s - loss: 0.0152 - accuracy: 0.9948
1310/1688 [======================>.......] - ETA: 0s - loss: 0.0154 - accuracy: 0.9947
1330/1688 [======================>.......] - ETA: 0s - loss: 0.0156 - accuracy: 0.9946
1349/1688 [======================>.......] - ETA: 0s - loss: 0.0156 - accuracy: 0.9946
1369/1688 [=======================>......] - ETA: 0s - loss: 0.0157 - accuracy: 0.9945
1389/1688 [=======================>......] - ETA: 0s - loss: 0.0156 - accuracy: 0.9945
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0157 - accuracy: 0.9945
1429/1688 [========================>.....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9944
1449/1688 [========================>.....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9944
1469/1688 [=========================>....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9944
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0158 - accuracy: 0.9944
1508/1688 [=========================>....] - ETA: 0s - loss: 0.0157 - accuracy: 0.9944
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0157 - accuracy: 0.9945
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0157 - accuracy: 0.9945
1568/1688 [==========================>...] - ETA: 0s - loss: 0.0156 - accuracy: 0.9945
1588/1688 [===========================>..] - ETA: 0s - loss: 0.0156 - accuracy: 0.9945
1608/1688 [===========================>..] - ETA: 0s - loss: 0.0157 - accuracy: 0.9944
1628/1688 [===========================>..] - ETA: 0s - loss: 0.0157 - accuracy: 0.9944
1648/1688 [============================>.] - ETA: 0s - loss: 0.0158 - accuracy: 0.9944
1668/1688 [============================>.] - ETA: 0s - loss: 0.0158 - accuracy: 0.9944
1688/1688 [==============================] - ETA: 0s - loss: 0.0159 - accuracy: 0.9944
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0159 - accuracy: 0.9944 - val_loss: 0.0615 - val_accuracy: 0.9857
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Test accuracy: 0.9836999773979187

2. Quantize

2.1. 8-bit quantization

An Akida accelerator processes 8 or 4-bits integer activations and weights. Therefore, the floating point Keras model must be quantized in preparation to run on an Akida accelerator.

The QuantizeML quantize function can be used to quantize a Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating point Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.

The quantization process results in a Keras model with custom QuantizeML quantized layers substituted for the original Keras layers. All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function applies several transformations to the original model. For example, it folds the batch normalization layers into the corresponding neural layers. The new weights are computed according to this folding operation.

from quantizeml.models import quantize, QuantizationParams

qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
/usr/local/lib/python3.11/dist-packages/quantizeml/models/quantize.py:494: UserWarning: Quantizing per-axis with random calibration samples is not accurate. Set QuantizationParams.per_tensor_activations=True when calibrating with random samples.
  warnings.warn("Quantizing per-axis with random calibration samples is not accurate. "

   1/1024 [..............................] - ETA: 3:21
  55/1024 [>.............................] - ETA: 0s  
 114/1024 [==>...........................] - ETA: 0s
 144/1024 [===>..........................] - ETA: 1s
 201/1024 [====>.........................] - ETA: 1s
 259/1024 [======>.......................] - ETA: 1s
 316/1024 [========>.....................] - ETA: 1s
 374/1024 [=========>....................] - ETA: 0s
 433/1024 [===========>..................] - ETA: 0s
 491/1024 [=============>................] - ETA: 0s
 547/1024 [===============>..............] - ETA: 0s
 603/1024 [================>.............] - ETA: 0s
 660/1024 [==================>...........] - ETA: 0s
 715/1024 [===================>..........] - ETA: 0s
 773/1024 [=====================>........] - ETA: 0s
 830/1024 [=======================>......] - ETA: 0s
 887/1024 [========================>.....] - ETA: 0s
 944/1024 [==========================>...] - ETA: 0s
1000/1024 [============================>.] - ETA: 0s
1024/1024 [==============================] - 1s 1ms/step
model_quantized.summary()
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling_input (InputLaye  [(None, 28, 28, 1)]       0
 r)

 rescaling (QuantizedRescal  (None, 28, 28, 1)         0
 ing)

 conv2d (QuantizedConv2D)    (None, 13, 13, 32)        320

 re_lu (QuantizedReLU)       (None, 13, 13, 32)        64

 depthwise_conv2d (Quantize  (None, 7, 7, 32)          384
 dDepthwiseConv2D)

 conv2d_1 (QuantizedConv2D)  (None, 7, 7, 64)          2112

 re_lu_1 (QuantizedReLU)     (None, 7, 7, 64)          128

 flatten (QuantizedFlatten)  (None, 3136)              0

 dense (QuantizedDense)      (None, 10)                31370

 dequantizer (Dequantizer)   (None, 10)                0

=================================================================
Total params: 34378 (134.29 KB)
Trainable params: 34122 (133.29 KB)
Non-trainable params: 256 (1.00 KB)
_________________________________________________________________

Note

Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.

Check the quantized model accuracy.

def compile_evaluate(model):
    """ Compiles and evaluates the model, then return accuracy score. """
    model.compile(metrics=['accuracy'])
    return model.evaluate(x_test, y_test, verbose=0)[1]


print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 8-bit quantization: 0.9779000282287598

2.2. Effect of calibration

The previous call to quantize was made with random samples for calibration (default parameters). While the observed drop in accuracy is minimal, that is around 1%, it can be worse on more complex models. Therefore, it is advised to use a set of real samples from the training set for calibration during a call to quantize. Note that this remains a calibration step rather than a training step in that no output labels are required. Furthermore, any relevant data could be used for calibration. The recommended settings for calibration that are widely used to obtain the zoo performance are:

  • 1024 samples

  • a batch size of 100

  • 2 epochs

model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)
 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 987us/step

Check the accuracy for the quantized and calibrated model.

print('Test accuracy after calibration:', compile_evaluate(model_quantized))
Test accuracy after calibration: 0.9833999872207642

Calibrating with real samples on this model recovers the initial float accuracy.

2.3. 4-bit quantization

The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.

The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.

qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)
 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 974us/step

Check the 4-bit quantized accuracy.

print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 4-bit quantization: 0.9818000197410583

2.4. Model fine tuning (Quantization Aware Training)

When a model suffers from an accuracy drop after quantization, fine tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.

Note that since this is a fine tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-4),
    metrics=['accuracy'])

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
Epoch 1/5

   1/1688 [..............................] - ETA: 2:07:33 - loss: 0.0017 - accuracy: 1.0000
  11/1688 [..............................] - ETA: 8s - loss: 0.0125 - accuracy: 0.9915     
  22/1688 [..............................] - ETA: 8s - loss: 0.0194 - accuracy: 0.9915
  33/1688 [..............................] - ETA: 8s - loss: 0.0213 - accuracy: 0.9924
  43/1688 [..............................] - ETA: 8s - loss: 0.0203 - accuracy: 0.9927
  54/1688 [..............................] - ETA: 8s - loss: 0.0180 - accuracy: 0.9931
  65/1688 [>.............................] - ETA: 8s - loss: 0.0182 - accuracy: 0.9928
  76/1688 [>.............................] - ETA: 8s - loss: 0.0193 - accuracy: 0.9926
  86/1688 [>.............................] - ETA: 7s - loss: 0.0188 - accuracy: 0.9924
  96/1688 [>.............................] - ETA: 7s - loss: 0.0174 - accuracy: 0.9932
 107/1688 [>.............................] - ETA: 7s - loss: 0.0173 - accuracy: 0.9933
 118/1688 [=>............................] - ETA: 7s - loss: 0.0178 - accuracy: 0.9931
 128/1688 [=>............................] - ETA: 7s - loss: 0.0168 - accuracy: 0.9937
 139/1688 [=>............................] - ETA: 7s - loss: 0.0170 - accuracy: 0.9937
 150/1688 [=>............................] - ETA: 7s - loss: 0.0170 - accuracy: 0.9935
 160/1688 [=>............................] - ETA: 7s - loss: 0.0162 - accuracy: 0.9939
 170/1688 [==>...........................] - ETA: 7s - loss: 0.0159 - accuracy: 0.9939
 181/1688 [==>...........................] - ETA: 7s - loss: 0.0151 - accuracy: 0.9943
 191/1688 [==>...........................] - ETA: 7s - loss: 0.0144 - accuracy: 0.9946
 201/1688 [==>...........................] - ETA: 7s - loss: 0.0140 - accuracy: 0.9949
 211/1688 [==>...........................] - ETA: 7s - loss: 0.0138 - accuracy: 0.9950
 222/1688 [==>...........................] - ETA: 7s - loss: 0.0135 - accuracy: 0.9951
 232/1688 [===>..........................] - ETA: 7s - loss: 0.0135 - accuracy: 0.9953
 243/1688 [===>..........................] - ETA: 7s - loss: 0.0138 - accuracy: 0.9951
 254/1688 [===>..........................] - ETA: 7s - loss: 0.0137 - accuracy: 0.9953
 264/1688 [===>..........................] - ETA: 7s - loss: 0.0138 - accuracy: 0.9954
 275/1688 [===>..........................] - ETA: 7s - loss: 0.0134 - accuracy: 0.9956
 286/1688 [====>.........................] - ETA: 6s - loss: 0.0130 - accuracy: 0.9957
 296/1688 [====>.........................] - ETA: 6s - loss: 0.0128 - accuracy: 0.9958
 307/1688 [====>.........................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9958
 318/1688 [====>.........................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9958
 328/1688 [====>.........................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9958
 339/1688 [=====>........................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9959
 350/1688 [=====>........................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9958
 360/1688 [=====>........................] - ETA: 6s - loss: 0.0129 - accuracy: 0.9956
 370/1688 [=====>........................] - ETA: 6s - loss: 0.0129 - accuracy: 0.9957
 381/1688 [=====>........................] - ETA: 6s - loss: 0.0130 - accuracy: 0.9957
 392/1688 [=====>........................] - ETA: 6s - loss: 0.0129 - accuracy: 0.9957
 402/1688 [======>.......................] - ETA: 6s - loss: 0.0131 - accuracy: 0.9956
 413/1688 [======>.......................] - ETA: 6s - loss: 0.0130 - accuracy: 0.9957
 424/1688 [======>.......................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9958
 435/1688 [======>.......................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9958
 445/1688 [======>.......................] - ETA: 6s - loss: 0.0127 - accuracy: 0.9958
 455/1688 [=======>......................] - ETA: 6s - loss: 0.0125 - accuracy: 0.9959
 466/1688 [=======>......................] - ETA: 6s - loss: 0.0123 - accuracy: 0.9960
 477/1688 [=======>......................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9960
 487/1688 [=======>......................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9960
 497/1688 [=======>......................] - ETA: 5s - loss: 0.0120 - accuracy: 0.9960
 508/1688 [========>.....................] - ETA: 5s - loss: 0.0124 - accuracy: 0.9961
 519/1688 [========>.....................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9961
 530/1688 [========>.....................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9962
 540/1688 [========>.....................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9961
 551/1688 [========>.....................] - ETA: 5s - loss: 0.0120 - accuracy: 0.9962
 562/1688 [========>.....................] - ETA: 5s - loss: 0.0120 - accuracy: 0.9962
 573/1688 [=========>....................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9962
 584/1688 [=========>....................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9961
 594/1688 [=========>....................] - ETA: 5s - loss: 0.0120 - accuracy: 0.9962
 604/1688 [=========>....................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9962
 614/1688 [=========>....................] - ETA: 5s - loss: 0.0118 - accuracy: 0.9963
 625/1688 [==========>...................] - ETA: 5s - loss: 0.0116 - accuracy: 0.9963
 636/1688 [==========>...................] - ETA: 5s - loss: 0.0116 - accuracy: 0.9963
 646/1688 [==========>...................] - ETA: 5s - loss: 0.0116 - accuracy: 0.9963
 657/1688 [==========>...................] - ETA: 5s - loss: 0.0117 - accuracy: 0.9963
 668/1688 [==========>...................] - ETA: 5s - loss: 0.0117 - accuracy: 0.9963
 679/1688 [===========>..................] - ETA: 5s - loss: 0.0122 - accuracy: 0.9962
 690/1688 [===========>..................] - ETA: 4s - loss: 0.0121 - accuracy: 0.9962
 700/1688 [===========>..................] - ETA: 4s - loss: 0.0121 - accuracy: 0.9962
 710/1688 [===========>..................] - ETA: 4s - loss: 0.0121 - accuracy: 0.9962
 721/1688 [===========>..................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9962
 732/1688 [============>.................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9962
 743/1688 [============>.................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9963
 754/1688 [============>.................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9963
 764/1688 [============>.................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9963
 775/1688 [============>.................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9963
 785/1688 [============>.................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9963
 796/1688 [=============>................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9963
 806/1688 [=============>................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9963
 817/1688 [=============>................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9963
 828/1688 [=============>................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9963
 838/1688 [=============>................] - ETA: 4s - loss: 0.0117 - accuracy: 0.9963
 848/1688 [==============>...............] - ETA: 4s - loss: 0.0117 - accuracy: 0.9963
 859/1688 [==============>...............] - ETA: 4s - loss: 0.0116 - accuracy: 0.9964
 869/1688 [==============>...............] - ETA: 4s - loss: 0.0115 - accuracy: 0.9964
 879/1688 [==============>...............] - ETA: 4s - loss: 0.0117 - accuracy: 0.9963
 890/1688 [==============>...............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9963
 901/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9963
 912/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9963
 923/1688 [===============>..............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9963
 934/1688 [===============>..............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9964
 945/1688 [===============>..............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9964
 955/1688 [===============>..............] - ETA: 3s - loss: 0.0114 - accuracy: 0.9964
 966/1688 [================>.............] - ETA: 3s - loss: 0.0113 - accuracy: 0.9964
 977/1688 [================>.............] - ETA: 3s - loss: 0.0113 - accuracy: 0.9964
 988/1688 [================>.............] - ETA: 3s - loss: 0.0112 - accuracy: 0.9965
 998/1688 [================>.............] - ETA: 3s - loss: 0.0112 - accuracy: 0.9965
1008/1688 [================>.............] - ETA: 3s - loss: 0.0111 - accuracy: 0.9965
1018/1688 [=================>............] - ETA: 3s - loss: 0.0112 - accuracy: 0.9965
1028/1688 [=================>............] - ETA: 3s - loss: 0.0111 - accuracy: 0.9965
1039/1688 [=================>............] - ETA: 3s - loss: 0.0111 - accuracy: 0.9965
1050/1688 [=================>............] - ETA: 3s - loss: 0.0111 - accuracy: 0.9965
1060/1688 [=================>............] - ETA: 3s - loss: 0.0111 - accuracy: 0.9965
1071/1688 [==================>...........] - ETA: 3s - loss: 0.0111 - accuracy: 0.9965
1082/1688 [==================>...........] - ETA: 3s - loss: 0.0111 - accuracy: 0.9965
1093/1688 [==================>...........] - ETA: 2s - loss: 0.0110 - accuracy: 0.9965
1103/1688 [==================>...........] - ETA: 2s - loss: 0.0111 - accuracy: 0.9965
1114/1688 [==================>...........] - ETA: 2s - loss: 0.0111 - accuracy: 0.9964
1125/1688 [==================>...........] - ETA: 2s - loss: 0.0111 - accuracy: 0.9964
1136/1688 [===================>..........] - ETA: 2s - loss: 0.0110 - accuracy: 0.9965
1146/1688 [===================>..........] - ETA: 2s - loss: 0.0110 - accuracy: 0.9965
1157/1688 [===================>..........] - ETA: 2s - loss: 0.0110 - accuracy: 0.9965
1168/1688 [===================>..........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9965
1179/1688 [===================>..........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9965
1189/1688 [====================>.........] - ETA: 2s - loss: 0.0109 - accuracy: 0.9965
1200/1688 [====================>.........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9965
1210/1688 [====================>.........] - ETA: 2s - loss: 0.0108 - accuracy: 0.9965
1220/1688 [====================>.........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9965
1231/1688 [====================>.........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9966
1242/1688 [=====================>........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9966
1253/1688 [=====================>........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9966
1263/1688 [=====================>........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9965
1274/1688 [=====================>........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9965
1284/1688 [=====================>........] - ETA: 2s - loss: 0.0107 - accuracy: 0.9966
1295/1688 [======================>.......] - ETA: 1s - loss: 0.0107 - accuracy: 0.9966
1306/1688 [======================>.......] - ETA: 1s - loss: 0.0106 - accuracy: 0.9966
1316/1688 [======================>.......] - ETA: 1s - loss: 0.0106 - accuracy: 0.9966
1326/1688 [======================>.......] - ETA: 1s - loss: 0.0105 - accuracy: 0.9967
1337/1688 [======================>.......] - ETA: 1s - loss: 0.0105 - accuracy: 0.9967
1347/1688 [======================>.......] - ETA: 1s - loss: 0.0106 - accuracy: 0.9966
1357/1688 [=======================>......] - ETA: 1s - loss: 0.0105 - accuracy: 0.9966
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0105 - accuracy: 0.9966
1377/1688 [=======================>......] - ETA: 1s - loss: 0.0105 - accuracy: 0.9966
1388/1688 [=======================>......] - ETA: 1s - loss: 0.0106 - accuracy: 0.9966
1399/1688 [=======================>......] - ETA: 1s - loss: 0.0106 - accuracy: 0.9965
1410/1688 [========================>.....] - ETA: 1s - loss: 0.0106 - accuracy: 0.9965
1421/1688 [========================>.....] - ETA: 1s - loss: 0.0106 - accuracy: 0.9965
1431/1688 [========================>.....] - ETA: 1s - loss: 0.0105 - accuracy: 0.9965
1442/1688 [========================>.....] - ETA: 1s - loss: 0.0106 - accuracy: 0.9965
1453/1688 [========================>.....] - ETA: 1s - loss: 0.0105 - accuracy: 0.9965
1464/1688 [=========================>....] - ETA: 1s - loss: 0.0105 - accuracy: 0.9965
1475/1688 [=========================>....] - ETA: 1s - loss: 0.0105 - accuracy: 0.9965
1486/1688 [=========================>....] - ETA: 1s - loss: 0.0105 - accuracy: 0.9965
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0105 - accuracy: 0.9965
1508/1688 [=========================>....] - ETA: 0s - loss: 0.0105 - accuracy: 0.9965
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0105 - accuracy: 0.9965
1529/1688 [==========================>...] - ETA: 0s - loss: 0.0105 - accuracy: 0.9965
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0105 - accuracy: 0.9965
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0104 - accuracy: 0.9965
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0104 - accuracy: 0.9965
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0104 - accuracy: 0.9965
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9965
1592/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9966
1603/1688 [===========================>..] - ETA: 0s - loss: 0.0102 - accuracy: 0.9966
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9966
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0102 - accuracy: 0.9966
1633/1688 [============================>.] - ETA: 0s - loss: 0.0102 - accuracy: 0.9966
1644/1688 [============================>.] - ETA: 0s - loss: 0.0102 - accuracy: 0.9966
1654/1688 [============================>.] - ETA: 0s - loss: 0.0101 - accuracy: 0.9966
1665/1688 [============================>.] - ETA: 0s - loss: 0.0101 - accuracy: 0.9967
1676/1688 [============================>.] - ETA: 0s - loss: 0.0101 - accuracy: 0.9967
1686/1688 [============================>.] - ETA: 0s - loss: 0.0101 - accuracy: 0.9967
1688/1688 [==============================] - 15s 6ms/step - loss: 0.0101 - accuracy: 0.9967 - val_loss: 0.0523 - val_accuracy: 0.9882
Epoch 2/5

   1/1688 [..............................] - ETA: 8s - loss: 0.0019 - accuracy: 1.0000
  12/1688 [..............................] - ETA: 8s - loss: 0.0029 - accuracy: 1.0000
  23/1688 [..............................] - ETA: 8s - loss: 0.0036 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 8s - loss: 0.0091 - accuracy: 0.9981
  44/1688 [..............................] - ETA: 8s - loss: 0.0073 - accuracy: 0.9986
  54/1688 [..............................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9988
  65/1688 [>.............................] - ETA: 8s - loss: 0.0073 - accuracy: 0.9976
  76/1688 [>.............................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9979
  87/1688 [>.............................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9978
  98/1688 [>.............................] - ETA: 7s - loss: 0.0070 - accuracy: 0.9974
 108/1688 [>.............................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9977
 118/1688 [=>............................] - ETA: 7s - loss: 0.0065 - accuracy: 0.9979
 129/1688 [=>............................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9981
 139/1688 [=>............................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9982
 150/1688 [=>............................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9983
 161/1688 [=>............................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9981
 172/1688 [==>...........................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9982
 182/1688 [==>...........................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9983
 193/1688 [==>...........................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9984
 204/1688 [==>...........................] - ETA: 7s - loss: 0.0058 - accuracy: 0.9985
 215/1688 [==>...........................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9983
 226/1688 [===>..........................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9982
 237/1688 [===>..........................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9983
 248/1688 [===>..........................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9982
 258/1688 [===>..........................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9983
 269/1688 [===>..........................] - ETA: 7s - loss: 0.0063 - accuracy: 0.9983
 280/1688 [===>..........................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9983
 290/1688 [====>.........................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9984
 300/1688 [====>.........................] - ETA: 6s - loss: 0.0063 - accuracy: 0.9983
 310/1688 [====>.........................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9982
 320/1688 [====>.........................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9981
 330/1688 [====>.........................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9982
 341/1688 [=====>........................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9983
 351/1688 [=====>........................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9983
 361/1688 [=====>........................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9983
 372/1688 [=====>........................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9983
 383/1688 [=====>........................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9984
 393/1688 [=====>........................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9983
 403/1688 [======>.......................] - ETA: 6s - loss: 0.0065 - accuracy: 0.9982
 413/1688 [======>.......................] - ETA: 6s - loss: 0.0069 - accuracy: 0.9981
 423/1688 [======>.......................] - ETA: 6s - loss: 0.0069 - accuracy: 0.9981
 433/1688 [======>.......................] - ETA: 6s - loss: 0.0067 - accuracy: 0.9981
 444/1688 [======>.......................] - ETA: 6s - loss: 0.0067 - accuracy: 0.9981
 455/1688 [=======>......................] - ETA: 6s - loss: 0.0067 - accuracy: 0.9981
 466/1688 [=======>......................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9981
 476/1688 [=======>......................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9980
 487/1688 [=======>......................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9981
 497/1688 [=======>......................] - ETA: 5s - loss: 0.0068 - accuracy: 0.9981
 508/1688 [========>.....................] - ETA: 5s - loss: 0.0067 - accuracy: 0.9981
 518/1688 [========>.....................] - ETA: 5s - loss: 0.0067 - accuracy: 0.9981
 529/1688 [========>.....................] - ETA: 5s - loss: 0.0066 - accuracy: 0.9981
 540/1688 [========>.....................] - ETA: 5s - loss: 0.0066 - accuracy: 0.9981
 550/1688 [========>.....................] - ETA: 5s - loss: 0.0065 - accuracy: 0.9981
 561/1688 [========>.....................] - ETA: 5s - loss: 0.0065 - accuracy: 0.9981
 571/1688 [=========>....................] - ETA: 5s - loss: 0.0065 - accuracy: 0.9981
 582/1688 [=========>....................] - ETA: 5s - loss: 0.0065 - accuracy: 0.9981
 592/1688 [=========>....................] - ETA: 5s - loss: 0.0065 - accuracy: 0.9982
 603/1688 [=========>....................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9982
 614/1688 [=========>....................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9982
 625/1688 [==========>...................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9983
 636/1688 [==========>...................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9983
 646/1688 [==========>...................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9983
 656/1688 [==========>...................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9983
 667/1688 [==========>...................] - ETA: 5s - loss: 0.0064 - accuracy: 0.9983
 678/1688 [===========>..................] - ETA: 5s - loss: 0.0063 - accuracy: 0.9983
 688/1688 [===========>..................] - ETA: 4s - loss: 0.0063 - accuracy: 0.9983
 698/1688 [===========>..................] - ETA: 4s - loss: 0.0062 - accuracy: 0.9983
 708/1688 [===========>..................] - ETA: 4s - loss: 0.0062 - accuracy: 0.9983
 718/1688 [===========>..................] - ETA: 4s - loss: 0.0061 - accuracy: 0.9983
 728/1688 [===========>..................] - ETA: 4s - loss: 0.0061 - accuracy: 0.9984
 738/1688 [============>.................] - ETA: 4s - loss: 0.0061 - accuracy: 0.9983
 749/1688 [============>.................] - ETA: 4s - loss: 0.0063 - accuracy: 0.9983
 759/1688 [============>.................] - ETA: 4s - loss: 0.0062 - accuracy: 0.9983
 769/1688 [============>.................] - ETA: 4s - loss: 0.0062 - accuracy: 0.9983
 780/1688 [============>.................] - ETA: 4s - loss: 0.0063 - accuracy: 0.9983
 790/1688 [=============>................] - ETA: 4s - loss: 0.0063 - accuracy: 0.9983
 801/1688 [=============>................] - ETA: 4s - loss: 0.0064 - accuracy: 0.9983
 811/1688 [=============>................] - ETA: 4s - loss: 0.0064 - accuracy: 0.9983
 821/1688 [=============>................] - ETA: 4s - loss: 0.0064 - accuracy: 0.9983
 832/1688 [=============>................] - ETA: 4s - loss: 0.0063 - accuracy: 0.9983
 843/1688 [=============>................] - ETA: 4s - loss: 0.0063 - accuracy: 0.9983
 854/1688 [==============>...............] - ETA: 4s - loss: 0.0063 - accuracy: 0.9984
 864/1688 [==============>...............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9984
 874/1688 [==============>...............] - ETA: 4s - loss: 0.0063 - accuracy: 0.9983
 885/1688 [==============>...............] - ETA: 4s - loss: 0.0063 - accuracy: 0.9983
 896/1688 [==============>...............] - ETA: 3s - loss: 0.0063 - accuracy: 0.9983
 907/1688 [===============>..............] - ETA: 3s - loss: 0.0063 - accuracy: 0.9983
 917/1688 [===============>..............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
 928/1688 [===============>..............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
 939/1688 [===============>..............] - ETA: 3s - loss: 0.0061 - accuracy: 0.9984
 950/1688 [===============>..............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
 960/1688 [================>.............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
 970/1688 [================>.............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
 980/1688 [================>.............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
 990/1688 [================>.............] - ETA: 3s - loss: 0.0063 - accuracy: 0.9984
1000/1688 [================>.............] - ETA: 3s - loss: 0.0063 - accuracy: 0.9984
1010/1688 [================>.............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
1020/1688 [=================>............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
1031/1688 [=================>............] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
1041/1688 [=================>............] - ETA: 3s - loss: 0.0061 - accuracy: 0.9984
1052/1688 [=================>............] - ETA: 3s - loss: 0.0061 - accuracy: 0.9985
1063/1688 [=================>............] - ETA: 3s - loss: 0.0061 - accuracy: 0.9984
1073/1688 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9985
1084/1688 [==================>...........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9984
1095/1688 [==================>...........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1106/1688 [==================>...........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9984
1117/1688 [==================>...........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1127/1688 [===================>..........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9984
1137/1688 [===================>..........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1148/1688 [===================>..........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1159/1688 [===================>..........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1169/1688 [===================>..........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1180/1688 [===================>..........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1191/1688 [====================>.........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1201/1688 [====================>.........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1211/1688 [====================>.........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1221/1688 [====================>.........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1231/1688 [====================>.........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1242/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1252/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1262/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1272/1688 [=====================>........] - ETA: 2s - loss: 0.0061 - accuracy: 0.9985
1283/1688 [=====================>........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9985
1294/1688 [=====================>........] - ETA: 1s - loss: 0.0062 - accuracy: 0.9985
1305/1688 [======================>.......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1316/1688 [======================>.......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1326/1688 [======================>.......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1336/1688 [======================>.......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1347/1688 [======================>.......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1357/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1378/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1388/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1398/1688 [=======================>......] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1408/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1418/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1428/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1438/1688 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9985
1449/1688 [========================>.....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9985
1460/1688 [========================>.....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9985
1470/1688 [=========================>....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9985
1481/1688 [=========================>....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9985
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0059 - accuracy: 0.9986
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0060 - accuracy: 0.9985
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1524/1688 [==========================>...] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0059 - accuracy: 0.9985
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0060 - accuracy: 0.9985
1576/1688 [===========================>..] - ETA: 0s - loss: 0.0060 - accuracy: 0.9985
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9985
1597/1688 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9985
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9985
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9985
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9985
1639/1688 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9985
1650/1688 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9985
1661/1688 [============================>.] - ETA: 0s - loss: 0.0062 - accuracy: 0.9984
1672/1688 [============================>.] - ETA: 0s - loss: 0.0063 - accuracy: 0.9984
1682/1688 [============================>.] - ETA: 0s - loss: 0.0063 - accuracy: 0.9984
1688/1688 [==============================] - 9s 5ms/step - loss: 0.0063 - accuracy: 0.9984 - val_loss: 0.0499 - val_accuracy: 0.9892
Epoch 3/5

   1/1688 [..............................] - ETA: 8s - loss: 2.8948e-04 - accuracy: 1.0000
  12/1688 [..............................] - ETA: 8s - loss: 0.0026 - accuracy: 1.0000    
  22/1688 [..............................] - ETA: 8s - loss: 0.0034 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 8s - loss: 0.0036 - accuracy: 1.0000
  42/1688 [..............................] - ETA: 8s - loss: 0.0029 - accuracy: 1.0000
  52/1688 [..............................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9994
  62/1688 [>.............................] - ETA: 8s - loss: 0.0036 - accuracy: 0.9995
  73/1688 [>.............................] - ETA: 8s - loss: 0.0037 - accuracy: 0.9991
  84/1688 [>.............................] - ETA: 8s - loss: 0.0044 - accuracy: 0.9989
  95/1688 [>.............................] - ETA: 7s - loss: 0.0043 - accuracy: 0.9990
 105/1688 [>.............................] - ETA: 7s - loss: 0.0047 - accuracy: 0.9988
 115/1688 [=>............................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9989
 125/1688 [=>............................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9987
 136/1688 [=>............................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9989
 147/1688 [=>............................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9989
 157/1688 [=>............................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9990
 168/1688 [=>............................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9987
 179/1688 [==>...........................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9988
 190/1688 [==>...........................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9988
 201/1688 [==>...........................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9989
 212/1688 [==>...........................] - ETA: 7s - loss: 0.0050 - accuracy: 0.9988
 222/1688 [==>...........................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9987
 232/1688 [===>..........................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9988
 243/1688 [===>..........................] - ETA: 7s - loss: 0.0050 - accuracy: 0.9988
 253/1688 [===>..........................] - ETA: 7s - loss: 0.0049 - accuracy: 0.9989
 264/1688 [===>..........................] - ETA: 7s - loss: 0.0051 - accuracy: 0.9987
 275/1688 [===>..........................] - ETA: 7s - loss: 0.0050 - accuracy: 0.9987
 285/1688 [====>.........................] - ETA: 7s - loss: 0.0050 - accuracy: 0.9988
 296/1688 [====>.........................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9988
 307/1688 [====>.........................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9989
 318/1688 [====>.........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9988
 328/1688 [====>.........................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 339/1688 [=====>........................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 349/1688 [=====>........................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 360/1688 [=====>........................] - ETA: 6s - loss: 0.0049 - accuracy: 0.9990
 371/1688 [=====>........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
 381/1688 [=====>........................] - ETA: 6s - loss: 0.0052 - accuracy: 0.9989
 392/1688 [=====>........................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
 402/1688 [======>.......................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 412/1688 [======>.......................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 423/1688 [======>.......................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 434/1688 [======>.......................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
 444/1688 [======>.......................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9989
 454/1688 [=======>......................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 465/1688 [=======>......................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 476/1688 [=======>......................] - ETA: 6s - loss: 0.0050 - accuracy: 0.9989
 487/1688 [=======>......................] - ETA: 6s - loss: 0.0051 - accuracy: 0.9988
 498/1688 [=======>......................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9988
 508/1688 [========>.....................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9988
 519/1688 [========>.....................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9988
 529/1688 [========>.....................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9988
 539/1688 [========>.....................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9988
 550/1688 [========>.....................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9988
 560/1688 [========>.....................] - ETA: 5s - loss: 0.0051 - accuracy: 0.9988
 571/1688 [=========>....................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9988
 581/1688 [=========>....................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9988
 592/1688 [=========>....................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9988
 603/1688 [=========>....................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 613/1688 [=========>....................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9989
 624/1688 [==========>...................] - ETA: 5s - loss: 0.0050 - accuracy: 0.9988
 635/1688 [==========>...................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 646/1688 [==========>...................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 657/1688 [==========>...................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 667/1688 [==========>...................] - ETA: 5s - loss: 0.0049 - accuracy: 0.9989
 677/1688 [===========>..................] - ETA: 5s - loss: 0.0048 - accuracy: 0.9989
 687/1688 [===========>..................] - ETA: 5s - loss: 0.0048 - accuracy: 0.9989
 698/1688 [===========>..................] - ETA: 4s - loss: 0.0048 - accuracy: 0.9989
 709/1688 [===========>..................] - ETA: 4s - loss: 0.0049 - accuracy: 0.9988
 720/1688 [===========>..................] - ETA: 4s - loss: 0.0049 - accuracy: 0.9988
 730/1688 [===========>..................] - ETA: 4s - loss: 0.0049 - accuracy: 0.9988
 740/1688 [============>.................] - ETA: 4s - loss: 0.0049 - accuracy: 0.9989
 751/1688 [============>.................] - ETA: 4s - loss: 0.0049 - accuracy: 0.9989
 761/1688 [============>.................] - ETA: 4s - loss: 0.0049 - accuracy: 0.9989
 772/1688 [============>.................] - ETA: 4s - loss: 0.0049 - accuracy: 0.9989
 783/1688 [============>.................] - ETA: 4s - loss: 0.0049 - accuracy: 0.9989
 793/1688 [=============>................] - ETA: 4s - loss: 0.0048 - accuracy: 0.9989
 803/1688 [=============>................] - ETA: 4s - loss: 0.0050 - accuracy: 0.9989
 813/1688 [=============>................] - ETA: 4s - loss: 0.0050 - accuracy: 0.9988
 824/1688 [=============>................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9988
 835/1688 [=============>................] - ETA: 4s - loss: 0.0052 - accuracy: 0.9988
 845/1688 [==============>...............] - ETA: 4s - loss: 0.0052 - accuracy: 0.9988
 856/1688 [==============>...............] - ETA: 4s - loss: 0.0052 - accuracy: 0.9988
 866/1688 [==============>...............] - ETA: 4s - loss: 0.0051 - accuracy: 0.9988
 877/1688 [==============>...............] - ETA: 4s - loss: 0.0051 - accuracy: 0.9989
 888/1688 [==============>...............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
 898/1688 [==============>...............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9989
 909/1688 [===============>..............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
 920/1688 [===============>..............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
 930/1688 [===============>..............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9989
 940/1688 [===============>..............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9989
 950/1688 [===============>..............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
 960/1688 [================>.............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
 970/1688 [================>.............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
 981/1688 [================>.............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9989
 991/1688 [================>.............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9989
1001/1688 [================>.............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9988
1012/1688 [================>.............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9989
1023/1688 [=================>............] - ETA: 3s - loss: 0.0050 - accuracy: 0.9988
1034/1688 [=================>............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
1045/1688 [=================>............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
1056/1688 [=================>............] - ETA: 3s - loss: 0.0052 - accuracy: 0.9988
1067/1688 [=================>............] - ETA: 3s - loss: 0.0051 - accuracy: 0.9988
1077/1688 [==================>...........] - ETA: 3s - loss: 0.0052 - accuracy: 0.9988
1088/1688 [==================>...........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9988
1098/1688 [==================>...........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9987
1108/1688 [==================>...........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9988
1119/1688 [==================>...........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1129/1688 [===================>..........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1139/1688 [===================>..........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1150/1688 [===================>..........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1161/1688 [===================>..........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1171/1688 [===================>..........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1181/1688 [===================>..........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1191/1688 [====================>.........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1202/1688 [====================>.........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1212/1688 [====================>.........] - ETA: 2s - loss: 0.0051 - accuracy: 0.9988
1223/1688 [====================>.........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9987
1233/1688 [====================>.........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9988
1244/1688 [=====================>........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9987
1255/1688 [=====================>........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9987
1266/1688 [=====================>........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9987
1276/1688 [=====================>........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9987
1286/1688 [=====================>........] - ETA: 2s - loss: 0.0052 - accuracy: 0.9987
1296/1688 [======================>.......] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1307/1688 [======================>.......] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1317/1688 [======================>.......] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1327/1688 [======================>.......] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1338/1688 [======================>.......] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1348/1688 [======================>.......] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1358/1688 [=======================>......] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1369/1688 [=======================>......] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1379/1688 [=======================>......] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1389/1688 [=======================>......] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1400/1688 [=======================>......] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1410/1688 [========================>.....] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1421/1688 [========================>.....] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1432/1688 [========================>.....] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1442/1688 [========================>.....] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1453/1688 [========================>.....] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1464/1688 [=========================>....] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1475/1688 [=========================>....] - ETA: 1s - loss: 0.0052 - accuracy: 0.9987
1486/1688 [=========================>....] - ETA: 1s - loss: 0.0053 - accuracy: 0.9987
1497/1688 [=========================>....] - ETA: 0s - loss: 0.0053 - accuracy: 0.9987
1508/1688 [=========================>....] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1518/1688 [=========================>....] - ETA: 0s - loss: 0.0053 - accuracy: 0.9987
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0053 - accuracy: 0.9987
1539/1688 [==========================>...] - ETA: 0s - loss: 0.0053 - accuracy: 0.9987
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0053 - accuracy: 0.9987
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0053 - accuracy: 0.9987
1570/1688 [==========================>...] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0053 - accuracy: 0.9987
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1634/1688 [============================>.] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1645/1688 [============================>.] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1655/1688 [============================>.] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1665/1688 [============================>.] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1675/1688 [============================>.] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1685/1688 [============================>.] - ETA: 0s - loss: 0.0052 - accuracy: 0.9987
1688/1688 [==============================] - 9s 5ms/step - loss: 0.0052 - accuracy: 0.9987 - val_loss: 0.0502 - val_accuracy: 0.9885
Epoch 4/5

   1/1688 [..............................] - ETA: 8s - loss: 0.0014 - accuracy: 1.0000
  12/1688 [..............................] - ETA: 8s - loss: 0.0037 - accuracy: 1.0000
  23/1688 [..............................] - ETA: 8s - loss: 0.0033 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 8s - loss: 0.0030 - accuracy: 1.0000
  43/1688 [..............................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9993
  54/1688 [..............................] - ETA: 8s - loss: 0.0057 - accuracy: 0.9994
  65/1688 [>.............................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9995
  76/1688 [>.............................] - ETA: 8s - loss: 0.0049 - accuracy: 0.9996
  87/1688 [>.............................] - ETA: 7s - loss: 0.0044 - accuracy: 0.9996
  98/1688 [>.............................] - ETA: 7s - loss: 0.0044 - accuracy: 0.9997
 109/1688 [>.............................] - ETA: 7s - loss: 0.0041 - accuracy: 0.9997
 120/1688 [=>............................] - ETA: 7s - loss: 0.0044 - accuracy: 0.9995
 130/1688 [=>............................] - ETA: 7s - loss: 0.0042 - accuracy: 0.9995
 140/1688 [=>............................] - ETA: 7s - loss: 0.0042 - accuracy: 0.9996
 151/1688 [=>............................] - ETA: 7s - loss: 0.0040 - accuracy: 0.9996
 161/1688 [=>............................] - ETA: 7s - loss: 0.0041 - accuracy: 0.9996
 172/1688 [==>...........................] - ETA: 7s - loss: 0.0040 - accuracy: 0.9996
 183/1688 [==>...........................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9997
 193/1688 [==>...........................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9997
 204/1688 [==>...........................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9997
 215/1688 [==>...........................] - ETA: 7s - loss: 0.0041 - accuracy: 0.9996
 226/1688 [===>..........................] - ETA: 7s - loss: 0.0041 - accuracy: 0.9996
 236/1688 [===>..........................] - ETA: 7s - loss: 0.0044 - accuracy: 0.9995
 247/1688 [===>..........................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9994
 257/1688 [===>..........................] - ETA: 7s - loss: 0.0047 - accuracy: 0.9993
 268/1688 [===>..........................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9993
 278/1688 [===>..........................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9993
 289/1688 [====>.........................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9994
 299/1688 [====>.........................] - ETA: 6s - loss: 0.0043 - accuracy: 0.9994
 310/1688 [====>.........................] - ETA: 6s - loss: 0.0043 - accuracy: 0.9994
 320/1688 [====>.........................] - ETA: 6s - loss: 0.0042 - accuracy: 0.9994
 330/1688 [====>.........................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9993
 341/1688 [=====>........................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9994
 352/1688 [=====>........................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9993
 363/1688 [=====>........................] - ETA: 6s - loss: 0.0047 - accuracy: 0.9991
 374/1688 [=====>........................] - ETA: 6s - loss: 0.0046 - accuracy: 0.9992
 385/1688 [=====>........................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9992
 396/1688 [======>.......................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9992
 407/1688 [======>.......................] - ETA: 6s - loss: 0.0046 - accuracy: 0.9992
 418/1688 [======>.......................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9992
 429/1688 [======>.......................] - ETA: 6s - loss: 0.0044 - accuracy: 0.9992
 440/1688 [======>.......................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9991
 450/1688 [======>.......................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9991
 461/1688 [=======>......................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9991
 471/1688 [=======>......................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9991
 481/1688 [=======>......................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9991
 491/1688 [=======>......................] - ETA: 5s - loss: 0.0044 - accuracy: 0.9991
 501/1688 [=======>......................] - ETA: 5s - loss: 0.0043 - accuracy: 0.9991
 512/1688 [========>.....................] - ETA: 5s - loss: 0.0043 - accuracy: 0.9991
 522/1688 [========>.....................] - ETA: 5s - loss: 0.0043 - accuracy: 0.9992
 532/1688 [========>.....................] - ETA: 5s - loss: 0.0042 - accuracy: 0.9992
 542/1688 [========>.....................] - ETA: 5s - loss: 0.0042 - accuracy: 0.9992
 552/1688 [========>.....................] - ETA: 5s - loss: 0.0042 - accuracy: 0.9992
 562/1688 [========>.....................] - ETA: 5s - loss: 0.0042 - accuracy: 0.9992
 573/1688 [=========>....................] - ETA: 5s - loss: 0.0042 - accuracy: 0.9992
 584/1688 [=========>....................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9992
 595/1688 [=========>....................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9992
 606/1688 [=========>....................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9992
 616/1688 [=========>....................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9992
 626/1688 [==========>...................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9992
 637/1688 [==========>...................] - ETA: 5s - loss: 0.0042 - accuracy: 0.9992
 647/1688 [==========>...................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9992
 658/1688 [==========>...................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9992
 669/1688 [==========>...................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9993
 679/1688 [===========>..................] - ETA: 5s - loss: 0.0041 - accuracy: 0.9993
 690/1688 [===========>..................] - ETA: 4s - loss: 0.0041 - accuracy: 0.9993
 700/1688 [===========>..................] - ETA: 4s - loss: 0.0040 - accuracy: 0.9993
 711/1688 [===========>..................] - ETA: 4s - loss: 0.0041 - accuracy: 0.9993
 721/1688 [===========>..................] - ETA: 4s - loss: 0.0041 - accuracy: 0.9993
 732/1688 [============>.................] - ETA: 4s - loss: 0.0041 - accuracy: 0.9993
 742/1688 [============>.................] - ETA: 4s - loss: 0.0041 - accuracy: 0.9993
 753/1688 [============>.................] - ETA: 4s - loss: 0.0041 - accuracy: 0.9993
 764/1688 [============>.................] - ETA: 4s - loss: 0.0043 - accuracy: 0.9993
 774/1688 [============>.................] - ETA: 4s - loss: 0.0043 - accuracy: 0.9993
 785/1688 [============>.................] - ETA: 4s - loss: 0.0042 - accuracy: 0.9993
 795/1688 [=============>................] - ETA: 4s - loss: 0.0042 - accuracy: 0.9993
 805/1688 [=============>................] - ETA: 4s - loss: 0.0043 - accuracy: 0.9992
 815/1688 [=============>................] - ETA: 4s - loss: 0.0043 - accuracy: 0.9992
 825/1688 [=============>................] - ETA: 4s - loss: 0.0043 - accuracy: 0.9992
 835/1688 [=============>................] - ETA: 4s - loss: 0.0043 - accuracy: 0.9993
 846/1688 [==============>...............] - ETA: 4s - loss: 0.0043 - accuracy: 0.9992
 856/1688 [==============>...............] - ETA: 4s - loss: 0.0043 - accuracy: 0.9992
 867/1688 [==============>...............] - ETA: 4s - loss: 0.0044 - accuracy: 0.9992
 878/1688 [==============>...............] - ETA: 4s - loss: 0.0044 - accuracy: 0.9991
 889/1688 [==============>...............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9992
 899/1688 [==============>...............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9992
 910/1688 [===============>..............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9991
 920/1688 [===============>..............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9991
 930/1688 [===============>..............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9991
 940/1688 [===============>..............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9991
 951/1688 [===============>..............] - ETA: 3s - loss: 0.0043 - accuracy: 0.9991
 962/1688 [================>.............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
 973/1688 [================>.............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
 983/1688 [================>.............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
 994/1688 [================>.............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1005/1688 [================>.............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1016/1688 [=================>............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1027/1688 [=================>............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1037/1688 [=================>............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1048/1688 [=================>............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1059/1688 [=================>............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1069/1688 [=================>............] - ETA: 3s - loss: 0.0042 - accuracy: 0.9992
1080/1688 [==================>...........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9991
1090/1688 [==================>...........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1101/1688 [==================>...........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1112/1688 [==================>...........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1122/1688 [==================>...........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1132/1688 [===================>..........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1143/1688 [===================>..........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1154/1688 [===================>..........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1165/1688 [===================>..........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1175/1688 [===================>..........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9990
1185/1688 [====================>.........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1195/1688 [====================>.........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1206/1688 [====================>.........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1217/1688 [====================>.........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1228/1688 [====================>.........] - ETA: 2s - loss: 0.0043 - accuracy: 0.9991
1239/1688 [=====================>........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1249/1688 [=====================>........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1259/1688 [=====================>........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1270/1688 [=====================>........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1281/1688 [=====================>........] - ETA: 2s - loss: 0.0044 - accuracy: 0.9991
1292/1688 [=====================>........] - ETA: 1s - loss: 0.0044 - accuracy: 0.9991
1303/1688 [======================>.......] - ETA: 1s - loss: 0.0044 - accuracy: 0.9991
1313/1688 [======================>.......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1324/1688 [======================>.......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1334/1688 [======================>.......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1355/1688 [=======================>......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1365/1688 [=======================>......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1376/1688 [=======================>......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1387/1688 [=======================>......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9991
1397/1688 [=======================>......] - ETA: 1s - loss: 0.0045 - accuracy: 0.9991
1408/1688 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9991
1418/1688 [========================>.....] - ETA: 1s - loss: 0.0044 - accuracy: 0.9991
1429/1688 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9991
1439/1688 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1449/1688 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1460/1688 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1471/1688 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1481/1688 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9990
1492/1688 [=========================>....] - ETA: 0s - loss: 0.0045 - accuracy: 0.9990
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0045 - accuracy: 0.9990
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0045 - accuracy: 0.9991
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0045 - accuracy: 0.9990
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0044 - accuracy: 0.9990
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0044 - accuracy: 0.9990
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0044 - accuracy: 0.9990
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9990
1587/1688 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1597/1688 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1639/1688 [============================>.] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1650/1688 [============================>.] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1661/1688 [============================>.] - ETA: 0s - loss: 0.0044 - accuracy: 0.9991
1672/1688 [============================>.] - ETA: 0s - loss: 0.0043 - accuracy: 0.9991
1682/1688 [============================>.] - ETA: 0s - loss: 0.0043 - accuracy: 0.9991
1688/1688 [==============================] - 9s 5ms/step - loss: 0.0043 - accuracy: 0.9991 - val_loss: 0.0505 - val_accuracy: 0.9888
Epoch 5/5

   1/1688 [..............................] - ETA: 8s - loss: 0.0045 - accuracy: 1.0000
  12/1688 [..............................] - ETA: 8s - loss: 0.0034 - accuracy: 1.0000
  22/1688 [..............................] - ETA: 8s - loss: 0.0031 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 8s - loss: 0.0028 - accuracy: 1.0000
  43/1688 [..............................] - ETA: 8s - loss: 0.0025 - accuracy: 1.0000
  53/1688 [..............................] - ETA: 8s - loss: 0.0036 - accuracy: 0.9994
  63/1688 [>.............................] - ETA: 8s - loss: 0.0036 - accuracy: 0.9995
  73/1688 [>.............................] - ETA: 8s - loss: 0.0039 - accuracy: 0.9996
  83/1688 [>.............................] - ETA: 8s - loss: 0.0037 - accuracy: 0.9996
  94/1688 [>.............................] - ETA: 8s - loss: 0.0037 - accuracy: 0.9997
 105/1688 [>.............................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9997
 115/1688 [=>............................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9997
 126/1688 [=>............................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9998
 136/1688 [=>............................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9998
 147/1688 [=>............................] - ETA: 7s - loss: 0.0036 - accuracy: 0.9998
 158/1688 [=>............................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9996
 169/1688 [==>...........................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9994
 180/1688 [==>...........................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9993
 190/1688 [==>...........................] - ETA: 7s - loss: 0.0040 - accuracy: 0.9993
 200/1688 [==>...........................] - ETA: 7s - loss: 0.0040 - accuracy: 0.9994
 211/1688 [==>...........................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9994
 221/1688 [==>...........................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9994
 232/1688 [===>..........................] - ETA: 7s - loss: 0.0040 - accuracy: 0.9993
 242/1688 [===>..........................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9994
 253/1688 [===>..........................] - ETA: 7s - loss: 0.0039 - accuracy: 0.9994
 263/1688 [===>..........................] - ETA: 7s - loss: 0.0038 - accuracy: 0.9994
 274/1688 [===>..........................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9994
 284/1688 [====>.........................] - ETA: 7s - loss: 0.0037 - accuracy: 0.9994
 295/1688 [====>.........................] - ETA: 6s - loss: 0.0037 - accuracy: 0.9995
 306/1688 [====>.........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9995
 317/1688 [====>.........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9995
 327/1688 [====>.........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9995
 337/1688 [====>.........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9995
 347/1688 [=====>........................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9995
 357/1688 [=====>........................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9996
 367/1688 [=====>........................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9996
 378/1688 [=====>........................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9996
 389/1688 [=====>........................] - ETA: 6s - loss: 0.0036 - accuracy: 0.9994
 400/1688 [======>.......................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9995
 411/1688 [======>.......................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9995
 422/1688 [======>.......................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9994
 433/1688 [======>.......................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9994
 443/1688 [======>.......................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9994
 453/1688 [=======>......................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9994
 464/1688 [=======>......................] - ETA: 6s - loss: 0.0035 - accuracy: 0.9995
 474/1688 [=======>......................] - ETA: 6s - loss: 0.0034 - accuracy: 0.9995
 485/1688 [=======>......................] - ETA: 6s - loss: 0.0034 - accuracy: 0.9995
 495/1688 [=======>......................] - ETA: 5s - loss: 0.0034 - accuracy: 0.9994
 506/1688 [=======>......................] - ETA: 5s - loss: 0.0034 - accuracy: 0.9994
 516/1688 [========>.....................] - ETA: 5s - loss: 0.0034 - accuracy: 0.9995
 527/1688 [========>.....................] - ETA: 5s - loss: 0.0035 - accuracy: 0.9994
 537/1688 [========>.....................] - ETA: 5s - loss: 0.0035 - accuracy: 0.9994
 548/1688 [========>.....................] - ETA: 5s - loss: 0.0035 - accuracy: 0.9994
 559/1688 [========>.....................] - ETA: 5s - loss: 0.0035 - accuracy: 0.9994
 569/1688 [=========>....................] - ETA: 5s - loss: 0.0035 - accuracy: 0.9994
 580/1688 [=========>....................] - ETA: 5s - loss: 0.0036 - accuracy: 0.9994
 590/1688 [=========>....................] - ETA: 5s - loss: 0.0035 - accuracy: 0.9994
 600/1688 [=========>....................] - ETA: 5s - loss: 0.0035 - accuracy: 0.9994
 611/1688 [=========>....................] - ETA: 5s - loss: 0.0036 - accuracy: 0.9993
 621/1688 [==========>...................] - ETA: 5s - loss: 0.0036 - accuracy: 0.9993
 632/1688 [==========>...................] - ETA: 5s - loss: 0.0036 - accuracy: 0.9993
 643/1688 [==========>...................] - ETA: 5s - loss: 0.0036 - accuracy: 0.9993
 654/1688 [==========>...................] - ETA: 5s - loss: 0.0037 - accuracy: 0.9992
 664/1688 [==========>...................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9992
 674/1688 [==========>...................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9992
 685/1688 [===========>..................] - ETA: 5s - loss: 0.0038 - accuracy: 0.9992
 695/1688 [===========>..................] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
 705/1688 [===========>..................] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
 715/1688 [===========>..................] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
 725/1688 [===========>..................] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
 735/1688 [============>.................] - ETA: 4s - loss: 0.0038 - accuracy: 0.9992
 745/1688 [============>.................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9992
 755/1688 [============>.................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 765/1688 [============>.................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 775/1688 [============>.................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 785/1688 [============>.................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 796/1688 [=============>................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 806/1688 [=============>................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 817/1688 [=============>................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 827/1688 [=============>................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 838/1688 [=============>................] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 849/1688 [==============>...............] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 859/1688 [==============>...............] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 870/1688 [==============>...............] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 880/1688 [==============>...............] - ETA: 4s - loss: 0.0037 - accuracy: 0.9993
 890/1688 [==============>...............] - ETA: 3s - loss: 0.0037 - accuracy: 0.9993
 900/1688 [==============>...............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
 910/1688 [===============>..............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
 921/1688 [===============>..............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
 931/1688 [===============>..............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
 942/1688 [===============>..............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
 952/1688 [===============>..............] - ETA: 3s - loss: 0.0039 - accuracy: 0.9992
 962/1688 [================>.............] - ETA: 3s - loss: 0.0039 - accuracy: 0.9993
 973/1688 [================>.............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
 983/1688 [================>.............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
 994/1688 [================>.............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9992
1005/1688 [================>.............] - ETA: 3s - loss: 0.0039 - accuracy: 0.9993
1016/1688 [=================>............] - ETA: 3s - loss: 0.0039 - accuracy: 0.9992
1027/1688 [=================>............] - ETA: 3s - loss: 0.0039 - accuracy: 0.9992
1037/1688 [=================>............] - ETA: 3s - loss: 0.0039 - accuracy: 0.9992
1048/1688 [=================>............] - ETA: 3s - loss: 0.0039 - accuracy: 0.9993
1059/1688 [=================>............] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
1069/1688 [=================>............] - ETA: 3s - loss: 0.0039 - accuracy: 0.9993
1079/1688 [==================>...........] - ETA: 3s - loss: 0.0038 - accuracy: 0.9993
1090/1688 [==================>...........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9993
1100/1688 [==================>...........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9993
1111/1688 [==================>...........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9993
1122/1688 [==================>...........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9993
1133/1688 [===================>..........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9993
1144/1688 [===================>..........] - ETA: 2s - loss: 0.0039 - accuracy: 0.9993
1154/1688 [===================>..........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1175/1688 [===================>..........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1185/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1196/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1207/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1218/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1228/1688 [====================>.........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1238/1688 [=====================>........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1248/1688 [=====================>........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1258/1688 [=====================>........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1268/1688 [=====================>........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1278/1688 [=====================>........] - ETA: 2s - loss: 0.0038 - accuracy: 0.9993
1289/1688 [=====================>........] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1300/1688 [======================>.......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1311/1688 [======================>.......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1322/1688 [======================>.......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1332/1688 [======================>.......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1343/1688 [======================>.......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1353/1688 [=======================>......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1363/1688 [=======================>......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1373/1688 [=======================>......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1383/1688 [=======================>......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1394/1688 [=======================>......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1405/1688 [=======================>......] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1416/1688 [========================>.....] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1426/1688 [========================>.....] - ETA: 1s - loss: 0.0038 - accuracy: 0.9993
1437/1688 [========================>.....] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1447/1688 [========================>.....] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1457/1688 [========================>.....] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1468/1688 [=========================>....] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1478/1688 [=========================>....] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1488/1688 [=========================>....] - ETA: 1s - loss: 0.0038 - accuracy: 0.9992
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0038 - accuracy: 0.9992
1509/1688 [=========================>....] - ETA: 0s - loss: 0.0038 - accuracy: 0.9992
1519/1688 [=========================>....] - ETA: 0s - loss: 0.0038 - accuracy: 0.9992
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1540/1688 [==========================>...] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1550/1688 [==========================>...] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1581/1688 [===========================>..] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1635/1688 [============================>.] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1646/1688 [============================>.] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1656/1688 [============================>.] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1666/1688 [============================>.] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1677/1688 [============================>.] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1688/1688 [==============================] - ETA: 0s - loss: 0.0039 - accuracy: 0.9992
1688/1688 [==============================] - 9s 5ms/step - loss: 0.0039 - accuracy: 0.9992 - val_loss: 0.0509 - val_accuracy: 0.9885

<keras.src.callbacks.History object at 0x7fe414001490>
score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine tuning:', score)
Test accuracy after fine tuning: 0.9854999780654907

3. Convert

3.1 Convert to Akida model

When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized)
model_akida.summary()
                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          5
______________________________________________

__________________________________________________________________
Layer (type)                        Output shape  Kernel shape

=============== SW/conv2d-dequantizer_2 (Software) ===============

conv2d (InputConv2D)                [13, 13, 32]  (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D)  [7, 7, 32]    (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D)                   [7, 7, 64]    (1, 1, 32, 64)
__________________________________________________________________
dense (Dense1D)                     [1, 1, 10]    (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer)         [1, 1, 10]    N/A
__________________________________________________________________

3.2. Check performance

accuracy = model_akida.evaluate(x_test, y_test)
print('Test accuracy after conversion:', accuracy)

# For non-regression purposes
assert accuracy > 0.96
Test accuracy after conversion: 0.9839000105857849

3.3 Show predictions for a single image

Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.

# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])

f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()

print(outputs.squeeze())
Class 7
Input Label: 7
[-16.429174   -12.029522    -8.657636     0.73896545 -20.699781
  -9.316402   -34.37597     10.484481    -6.573269    -2.5740962 ]

Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.

In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.

Total running time of the script: (2 minutes 7.436 seconds)

Gallery generated by Sphinx-Gallery