Global Akida workflow

Using the MNIST dataset, this example shows the definition and training of a keras floating point model, its quantization to 8-bit with the help of calibration, its quantization to 4-bit using QAT and its conversion to Akida. Notice that the performance of the original keras floating point model is maintained throughout the Akida flow. Please refer to the Akida user guide for further information.

Note

Please refer to the TensorFlow tf.keras.models module for model creation/import details and the TensorFlow Guide for TensorFlow usage.

The MNIST example below is light enough so that a GPU is not needed for training.

Overall flow

Global Akida workflow

1. Create and train

1.1. Load and reshape MNIST dataset

import numpy as np

import matplotlib.cm as cm
import matplotlib.pyplot as plt

from keras.datasets import mnist

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Add a channels dimension to the image sets as Akida expects 4-D inputs (corresponding to
# (num_samples, width, height, channels). Note: MNIST is a grayscale dataset and is unusual
# in this respect - most image data already includes a channel dimension, and this step will
# not be necessary.
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)

# Display a few images from the test set
f, axarr = plt.subplots(1, 4)
for i in range(0, 4):
    axarr[i].imshow(x_test[i].reshape((28, 28)), cmap=cm.Greys_r)
    axarr[i].set_title('Class %d' % y_test[i])
plt.show()
Class 7, Class 2, Class 1, Class 0
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz

    8192/11490434 [..............................] - ETA: 0s
   24576/11490434 [..............................] - ETA: 1:12
   40960/11490434 [..............................] - ETA: 1:00
   57344/11490434 [..............................] - ETA: 58s 
   81920/11490434 [..............................] - ETA: 49s
  106496/11490434 [..............................] - ETA: 44s
  139264/11490434 [..............................] - ETA: 38s
  180224/11490434 [..............................] - ETA: 32s
  229376/11490434 [..............................] - ETA: 28s
  286720/11490434 [..............................] - ETA: 24s
  368640/11490434 [..............................] - ETA: 20s
  475136/11490434 [>.............................] - ETA: 17s
  606208/11490434 [>.............................] - ETA: 14s
  770048/11490434 [=>............................] - ETA: 11s
  974848/11490434 [=>............................] - ETA: 9s 
 1228800/11490434 [==>...........................] - ETA: 7s
 1556480/11490434 [===>..........................] - ETA: 6s
 1925120/11490434 [====>.........................] - ETA: 5s
 2465792/11490434 [=====>........................] - ETA: 4s
 3039232/11490434 [======>.......................] - ETA: 3s
 3629056/11490434 [========>.....................] - ETA: 2s
 4218880/11490434 [==========>...................] - ETA: 2s
 4816896/11490434 [===========>..................] - ETA: 1s
 5414912/11490434 [=============>................] - ETA: 1s
 6012928/11490434 [==============>...............] - ETA: 1s
 6610944/11490434 [================>.............] - ETA: 1s
 7192576/11490434 [=================>............] - ETA: 0s
 7774208/11490434 [===================>..........] - ETA: 0s
 8372224/11490434 [====================>.........] - ETA: 0s
 8962048/11490434 [======================>.......] - ETA: 0s
 9560064/11490434 [=======================>......] - ETA: 0s
10133504/11490434 [=========================>....] - ETA: 0s
10739712/11490434 [===========================>..] - ETA: 0s
11313152/11490434 [============================>.] - ETA: 0s
11490434/11490434 [==============================] - 2s 0us/step

1.2. Model definition

Note that at this stage, there is nothing specific to the Akida IP. The model constructed below, as inspired by this example, is a completely standard Keras CNN model.

import keras

model_keras = keras.models.Sequential([
    keras.layers.Rescaling(1. / 255, input_shape=(28, 28, 1)),
    keras.layers.Conv2D(filters=32, kernel_size=3, strides=2),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    # Separable layer
    keras.layers.DepthwiseConv2D(kernel_size=3, padding='same', strides=2),
    keras.layers.Conv2D(filters=64, kernel_size=1, padding='same'),
    keras.layers.BatchNormalization(),
    keras.layers.ReLU(),
    keras.layers.Flatten(),
    keras.layers.Dense(10)
], 'mnistnet')

model_keras.summary()
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling (Rescaling)       (None, 28, 28, 1)         0

 conv2d (Conv2D)             (None, 13, 13, 32)        320

 batch_normalization (BatchN  (None, 13, 13, 32)       128
 ormalization)

 re_lu (ReLU)                (None, 13, 13, 32)        0

 depthwise_conv2d (Depthwise  (None, 7, 7, 32)         320
 Conv2D)

 conv2d_1 (Conv2D)           (None, 7, 7, 64)          2112

 batch_normalization_1 (Batc  (None, 7, 7, 64)         256
 hNormalization)

 re_lu_1 (ReLU)              (None, 7, 7, 64)          0

 flatten (Flatten)           (None, 3136)              0

 dense (Dense)               (None, 10)                31370

=================================================================
Total params: 34,506
Trainable params: 34,314
Non-trainable params: 192
_________________________________________________________________

1.3. Model training

Given the model created above, train the model and check its accuracy. The model should achieve a test accuracy over 98% after 10 epochs.

from keras.optimizers import Adam

model_keras.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-3),
    metrics=['accuracy'])

_ = model_keras.fit(x_train, y_train, epochs=10, validation_split=0.1)
Epoch 1/10

   1/1688 [..............................] - ETA: 21:18 - loss: 2.4851 - accuracy: 0.2500
  30/1688 [..............................] - ETA: 2s - loss: 1.2134 - accuracy: 0.6135   
  59/1688 [>.............................] - ETA: 2s - loss: 0.8762 - accuracy: 0.7214
  89/1688 [>.............................] - ETA: 2s - loss: 0.7270 - accuracy: 0.7739
 119/1688 [=>............................] - ETA: 2s - loss: 0.6463 - accuracy: 0.7978
 149/1688 [=>............................] - ETA: 2s - loss: 0.5778 - accuracy: 0.8198
 178/1688 [==>...........................] - ETA: 2s - loss: 0.5368 - accuracy: 0.8329
 209/1688 [==>...........................] - ETA: 2s - loss: 0.4994 - accuracy: 0.8451
 238/1688 [===>..........................] - ETA: 2s - loss: 0.4770 - accuracy: 0.8522
 268/1688 [===>..........................] - ETA: 2s - loss: 0.4529 - accuracy: 0.8603
 298/1688 [====>.........................] - ETA: 2s - loss: 0.4324 - accuracy: 0.8662
 328/1688 [====>.........................] - ETA: 2s - loss: 0.4113 - accuracy: 0.8729
 357/1688 [=====>........................] - ETA: 2s - loss: 0.3935 - accuracy: 0.8778
 387/1688 [=====>........................] - ETA: 2s - loss: 0.3766 - accuracy: 0.8835
 417/1688 [======>.......................] - ETA: 2s - loss: 0.3625 - accuracy: 0.8881
 447/1688 [======>.......................] - ETA: 2s - loss: 0.3501 - accuracy: 0.8923
 477/1688 [=======>......................] - ETA: 2s - loss: 0.3362 - accuracy: 0.8969
 507/1688 [========>.....................] - ETA: 2s - loss: 0.3264 - accuracy: 0.9002
 537/1688 [========>.....................] - ETA: 1s - loss: 0.3161 - accuracy: 0.9034
 567/1688 [=========>....................] - ETA: 1s - loss: 0.3080 - accuracy: 0.9062
 597/1688 [=========>....................] - ETA: 1s - loss: 0.3009 - accuracy: 0.9086
 627/1688 [==========>...................] - ETA: 1s - loss: 0.2928 - accuracy: 0.9112
 657/1688 [==========>...................] - ETA: 1s - loss: 0.2848 - accuracy: 0.9136
 687/1688 [===========>..................] - ETA: 1s - loss: 0.2787 - accuracy: 0.9154
 717/1688 [===========>..................] - ETA: 1s - loss: 0.2719 - accuracy: 0.9174
 747/1688 [============>.................] - ETA: 1s - loss: 0.2674 - accuracy: 0.9188
 776/1688 [============>.................] - ETA: 1s - loss: 0.2619 - accuracy: 0.9204
 806/1688 [=============>................] - ETA: 1s - loss: 0.2562 - accuracy: 0.9221
 836/1688 [=============>................] - ETA: 1s - loss: 0.2502 - accuracy: 0.9240
 867/1688 [==============>...............] - ETA: 1s - loss: 0.2448 - accuracy: 0.9255
 898/1688 [==============>...............] - ETA: 1s - loss: 0.2406 - accuracy: 0.9267
 928/1688 [===============>..............] - ETA: 1s - loss: 0.2362 - accuracy: 0.9279
 957/1688 [================>.............] - ETA: 1s - loss: 0.2320 - accuracy: 0.9294
 987/1688 [================>.............] - ETA: 1s - loss: 0.2271 - accuracy: 0.9309
1017/1688 [=================>............] - ETA: 1s - loss: 0.2238 - accuracy: 0.9319
1047/1688 [=================>............] - ETA: 1s - loss: 0.2198 - accuracy: 0.9330
1077/1688 [==================>...........] - ETA: 1s - loss: 0.2169 - accuracy: 0.9338
1107/1688 [==================>...........] - ETA: 0s - loss: 0.2135 - accuracy: 0.9346
1137/1688 [===================>..........] - ETA: 0s - loss: 0.2117 - accuracy: 0.9352
1166/1688 [===================>..........] - ETA: 0s - loss: 0.2099 - accuracy: 0.9358
1196/1688 [====================>.........] - ETA: 0s - loss: 0.2069 - accuracy: 0.9367
1226/1688 [====================>.........] - ETA: 0s - loss: 0.2042 - accuracy: 0.9376
1256/1688 [=====================>........] - ETA: 0s - loss: 0.2014 - accuracy: 0.9383
1286/1688 [=====================>........] - ETA: 0s - loss: 0.1998 - accuracy: 0.9389
1316/1688 [======================>.......] - ETA: 0s - loss: 0.1974 - accuracy: 0.9396
1347/1688 [======================>.......] - ETA: 0s - loss: 0.1953 - accuracy: 0.9402
1378/1688 [=======================>......] - ETA: 0s - loss: 0.1925 - accuracy: 0.9412
1409/1688 [========================>.....] - ETA: 0s - loss: 0.1903 - accuracy: 0.9419
1439/1688 [========================>.....] - ETA: 0s - loss: 0.1895 - accuracy: 0.9422
1470/1688 [=========================>....] - ETA: 0s - loss: 0.1875 - accuracy: 0.9429
1500/1688 [=========================>....] - ETA: 0s - loss: 0.1855 - accuracy: 0.9435
1530/1688 [==========================>...] - ETA: 0s - loss: 0.1842 - accuracy: 0.9439
1560/1688 [==========================>...] - ETA: 0s - loss: 0.1825 - accuracy: 0.9445
1590/1688 [===========================>..] - ETA: 0s - loss: 0.1809 - accuracy: 0.9449
1620/1688 [===========================>..] - ETA: 0s - loss: 0.1792 - accuracy: 0.9454
1651/1688 [============================>.] - ETA: 0s - loss: 0.1779 - accuracy: 0.9458
1681/1688 [============================>.] - ETA: 0s - loss: 0.1763 - accuracy: 0.9463
1688/1688 [==============================] - ETA: 0s - loss: 0.1759 - accuracy: 0.9464
1688/1688 [==============================] - 4s 2ms/step - loss: 0.1759 - accuracy: 0.9464 - val_loss: 0.0766 - val_accuracy: 0.9803
Epoch 2/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0271 - accuracy: 1.0000
  33/1688 [..............................] - ETA: 2s - loss: 0.0751 - accuracy: 0.9792
  64/1688 [>.............................] - ETA: 2s - loss: 0.0691 - accuracy: 0.9785
  94/1688 [>.............................] - ETA: 2s - loss: 0.0782 - accuracy: 0.9757
 125/1688 [=>............................] - ETA: 2s - loss: 0.0767 - accuracy: 0.9762
 155/1688 [=>............................] - ETA: 2s - loss: 0.0693 - accuracy: 0.9792
 185/1688 [==>...........................] - ETA: 2s - loss: 0.0667 - accuracy: 0.9797
 216/1688 [==>...........................] - ETA: 2s - loss: 0.0678 - accuracy: 0.9792
 246/1688 [===>..........................] - ETA: 2s - loss: 0.0682 - accuracy: 0.9790
 276/1688 [===>..........................] - ETA: 2s - loss: 0.0686 - accuracy: 0.9788
 307/1688 [====>.........................] - ETA: 2s - loss: 0.0687 - accuracy: 0.9789
 337/1688 [====>.........................] - ETA: 2s - loss: 0.0703 - accuracy: 0.9783
 367/1688 [=====>........................] - ETA: 2s - loss: 0.0712 - accuracy: 0.9782
 397/1688 [======>.......................] - ETA: 2s - loss: 0.0735 - accuracy: 0.9777
 428/1688 [======>.......................] - ETA: 2s - loss: 0.0738 - accuracy: 0.9776
 458/1688 [=======>......................] - ETA: 2s - loss: 0.0758 - accuracy: 0.9776
 489/1688 [=======>......................] - ETA: 2s - loss: 0.0757 - accuracy: 0.9774
 519/1688 [========>.....................] - ETA: 1s - loss: 0.0764 - accuracy: 0.9771
 548/1688 [========>.....................] - ETA: 1s - loss: 0.0766 - accuracy: 0.9769
 578/1688 [=========>....................] - ETA: 1s - loss: 0.0772 - accuracy: 0.9766
 607/1688 [=========>....................] - ETA: 1s - loss: 0.0766 - accuracy: 0.9768
 637/1688 [==========>...................] - ETA: 1s - loss: 0.0767 - accuracy: 0.9766
 667/1688 [==========>...................] - ETA: 1s - loss: 0.0766 - accuracy: 0.9765
 697/1688 [===========>..................] - ETA: 1s - loss: 0.0757 - accuracy: 0.9766
 727/1688 [===========>..................] - ETA: 1s - loss: 0.0750 - accuracy: 0.9769
 756/1688 [============>.................] - ETA: 1s - loss: 0.0753 - accuracy: 0.9768
 787/1688 [============>.................] - ETA: 1s - loss: 0.0753 - accuracy: 0.9769
 817/1688 [=============>................] - ETA: 1s - loss: 0.0754 - accuracy: 0.9769
 846/1688 [==============>...............] - ETA: 1s - loss: 0.0757 - accuracy: 0.9767
 876/1688 [==============>...............] - ETA: 1s - loss: 0.0769 - accuracy: 0.9763
 906/1688 [===============>..............] - ETA: 1s - loss: 0.0765 - accuracy: 0.9763
 936/1688 [===============>..............] - ETA: 1s - loss: 0.0763 - accuracy: 0.9764
 966/1688 [================>.............] - ETA: 1s - loss: 0.0761 - accuracy: 0.9764
 996/1688 [================>.............] - ETA: 1s - loss: 0.0753 - accuracy: 0.9765
1026/1688 [=================>............] - ETA: 1s - loss: 0.0752 - accuracy: 0.9766
1056/1688 [=================>............] - ETA: 1s - loss: 0.0757 - accuracy: 0.9765
1085/1688 [==================>...........] - ETA: 1s - loss: 0.0756 - accuracy: 0.9766
1115/1688 [==================>...........] - ETA: 0s - loss: 0.0751 - accuracy: 0.9768
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0746 - accuracy: 0.9770
1175/1688 [===================>..........] - ETA: 0s - loss: 0.0747 - accuracy: 0.9770
1205/1688 [====================>.........] - ETA: 0s - loss: 0.0750 - accuracy: 0.9768
1235/1688 [====================>.........] - ETA: 0s - loss: 0.0756 - accuracy: 0.9766
1266/1688 [=====================>........] - ETA: 0s - loss: 0.0752 - accuracy: 0.9767
1297/1688 [======================>.......] - ETA: 0s - loss: 0.0747 - accuracy: 0.9768
1327/1688 [======================>.......] - ETA: 0s - loss: 0.0746 - accuracy: 0.9768
1357/1688 [=======================>......] - ETA: 0s - loss: 0.0744 - accuracy: 0.9769
1387/1688 [=======================>......] - ETA: 0s - loss: 0.0742 - accuracy: 0.9769
1417/1688 [========================>.....] - ETA: 0s - loss: 0.0736 - accuracy: 0.9771
1447/1688 [========================>.....] - ETA: 0s - loss: 0.0732 - accuracy: 0.9771
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0731 - accuracy: 0.9771
1508/1688 [=========================>....] - ETA: 0s - loss: 0.0732 - accuracy: 0.9771
1538/1688 [==========================>...] - ETA: 0s - loss: 0.0728 - accuracy: 0.9771
1568/1688 [==========================>...] - ETA: 0s - loss: 0.0724 - accuracy: 0.9772
1599/1688 [===========================>..] - ETA: 0s - loss: 0.0725 - accuracy: 0.9773
1630/1688 [===========================>..] - ETA: 0s - loss: 0.0724 - accuracy: 0.9772
1660/1688 [============================>.] - ETA: 0s - loss: 0.0721 - accuracy: 0.9773
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0720 - accuracy: 0.9773 - val_loss: 0.0645 - val_accuracy: 0.9812
Epoch 3/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0966 - accuracy: 0.9688
  33/1688 [..............................] - ETA: 2s - loss: 0.0602 - accuracy: 0.9811
  63/1688 [>.............................] - ETA: 2s - loss: 0.0546 - accuracy: 0.9821
  94/1688 [>.............................] - ETA: 2s - loss: 0.0661 - accuracy: 0.9794
 124/1688 [=>............................] - ETA: 2s - loss: 0.0634 - accuracy: 0.9798
 154/1688 [=>............................] - ETA: 2s - loss: 0.0604 - accuracy: 0.9807
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0592 - accuracy: 0.9814
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0579 - accuracy: 0.9815
 243/1688 [===>..........................] - ETA: 2s - loss: 0.0565 - accuracy: 0.9823
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0551 - accuracy: 0.9828
 302/1688 [====>.........................] - ETA: 2s - loss: 0.0563 - accuracy: 0.9830
 332/1688 [====>.........................] - ETA: 2s - loss: 0.0564 - accuracy: 0.9830
 362/1688 [=====>........................] - ETA: 2s - loss: 0.0545 - accuracy: 0.9835
 392/1688 [=====>........................] - ETA: 2s - loss: 0.0544 - accuracy: 0.9838
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0542 - accuracy: 0.9833
 452/1688 [=======>......................] - ETA: 2s - loss: 0.0546 - accuracy: 0.9830
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0545 - accuracy: 0.9831
 512/1688 [========>.....................] - ETA: 1s - loss: 0.0545 - accuracy: 0.9832
 541/1688 [========>.....................] - ETA: 1s - loss: 0.0550 - accuracy: 0.9831
 571/1688 [=========>....................] - ETA: 1s - loss: 0.0549 - accuracy: 0.9829
 602/1688 [=========>....................] - ETA: 1s - loss: 0.0539 - accuracy: 0.9830
 632/1688 [==========>...................] - ETA: 1s - loss: 0.0536 - accuracy: 0.9830
 662/1688 [==========>...................] - ETA: 1s - loss: 0.0531 - accuracy: 0.9833
 691/1688 [===========>..................] - ETA: 1s - loss: 0.0533 - accuracy: 0.9834
 721/1688 [===========>..................] - ETA: 1s - loss: 0.0534 - accuracy: 0.9834
 751/1688 [============>.................] - ETA: 1s - loss: 0.0531 - accuracy: 0.9834
 782/1688 [============>.................] - ETA: 1s - loss: 0.0527 - accuracy: 0.9836
 812/1688 [=============>................] - ETA: 1s - loss: 0.0530 - accuracy: 0.9834
 842/1688 [=============>................] - ETA: 1s - loss: 0.0531 - accuracy: 0.9833
 872/1688 [==============>...............] - ETA: 1s - loss: 0.0532 - accuracy: 0.9833
 902/1688 [===============>..............] - ETA: 1s - loss: 0.0537 - accuracy: 0.9832
 933/1688 [===============>..............] - ETA: 1s - loss: 0.0530 - accuracy: 0.9834
 963/1688 [================>.............] - ETA: 1s - loss: 0.0525 - accuracy: 0.9835
 993/1688 [================>.............] - ETA: 1s - loss: 0.0529 - accuracy: 0.9835
1023/1688 [=================>............] - ETA: 1s - loss: 0.0534 - accuracy: 0.9834
1053/1688 [=================>............] - ETA: 1s - loss: 0.0538 - accuracy: 0.9833
1083/1688 [==================>...........] - ETA: 1s - loss: 0.0537 - accuracy: 0.9833
1112/1688 [==================>...........] - ETA: 0s - loss: 0.0533 - accuracy: 0.9833
1142/1688 [===================>..........] - ETA: 0s - loss: 0.0537 - accuracy: 0.9833
1172/1688 [===================>..........] - ETA: 0s - loss: 0.0535 - accuracy: 0.9833
1203/1688 [====================>.........] - ETA: 0s - loss: 0.0530 - accuracy: 0.9834
1234/1688 [====================>.........] - ETA: 0s - loss: 0.0526 - accuracy: 0.9835
1263/1688 [=====================>........] - ETA: 0s - loss: 0.0520 - accuracy: 0.9838
1293/1688 [=====================>........] - ETA: 0s - loss: 0.0521 - accuracy: 0.9839
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0517 - accuracy: 0.9841
1352/1688 [=======================>......] - ETA: 0s - loss: 0.0520 - accuracy: 0.9841
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0518 - accuracy: 0.9841
1413/1688 [========================>.....] - ETA: 0s - loss: 0.0522 - accuracy: 0.9840
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0519 - accuracy: 0.9842
1473/1688 [=========================>....] - ETA: 0s - loss: 0.0521 - accuracy: 0.9842
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0520 - accuracy: 0.9842
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0521 - accuracy: 0.9840
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0521 - accuracy: 0.9841
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0522 - accuracy: 0.9840
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0519 - accuracy: 0.9841
1653/1688 [============================>.] - ETA: 0s - loss: 0.0520 - accuracy: 0.9840
1683/1688 [============================>.] - ETA: 0s - loss: 0.0520 - accuracy: 0.9840
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0519 - accuracy: 0.9840 - val_loss: 0.0790 - val_accuracy: 0.9783
Epoch 4/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0391 - accuracy: 0.9688
  34/1688 [..............................] - ETA: 2s - loss: 0.0573 - accuracy: 0.9807
  64/1688 [>.............................] - ETA: 2s - loss: 0.0428 - accuracy: 0.9854
  94/1688 [>.............................] - ETA: 2s - loss: 0.0371 - accuracy: 0.9870
 124/1688 [=>............................] - ETA: 2s - loss: 0.0377 - accuracy: 0.9866
 154/1688 [=>............................] - ETA: 2s - loss: 0.0450 - accuracy: 0.9856
 184/1688 [==>...........................] - ETA: 2s - loss: 0.0431 - accuracy: 0.9859
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0426 - accuracy: 0.9861
 243/1688 [===>..........................] - ETA: 2s - loss: 0.0418 - accuracy: 0.9868
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0401 - accuracy: 0.9873
 303/1688 [====>.........................] - ETA: 2s - loss: 0.0410 - accuracy: 0.9867
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0412 - accuracy: 0.9864
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0407 - accuracy: 0.9867
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0420 - accuracy: 0.9864
 424/1688 [======>.......................] - ETA: 2s - loss: 0.0429 - accuracy: 0.9860
 454/1688 [=======>......................] - ETA: 2s - loss: 0.0424 - accuracy: 0.9862
 484/1688 [=======>......................] - ETA: 2s - loss: 0.0443 - accuracy: 0.9856
 514/1688 [========>.....................] - ETA: 1s - loss: 0.0449 - accuracy: 0.9855
 544/1688 [========>.....................] - ETA: 1s - loss: 0.0445 - accuracy: 0.9857
 574/1688 [=========>....................] - ETA: 1s - loss: 0.0441 - accuracy: 0.9857
 603/1688 [=========>....................] - ETA: 1s - loss: 0.0443 - accuracy: 0.9857
 633/1688 [==========>...................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9853
 664/1688 [==========>...................] - ETA: 1s - loss: 0.0454 - accuracy: 0.9854
 694/1688 [===========>..................] - ETA: 1s - loss: 0.0450 - accuracy: 0.9855
 724/1688 [===========>..................] - ETA: 1s - loss: 0.0460 - accuracy: 0.9850
 754/1688 [============>.................] - ETA: 1s - loss: 0.0453 - accuracy: 0.9852
 783/1688 [============>.................] - ETA: 1s - loss: 0.0454 - accuracy: 0.9852
 813/1688 [=============>................] - ETA: 1s - loss: 0.0448 - accuracy: 0.9854
 842/1688 [=============>................] - ETA: 1s - loss: 0.0456 - accuracy: 0.9853
 872/1688 [==============>...............] - ETA: 1s - loss: 0.0455 - accuracy: 0.9853
 902/1688 [===============>..............] - ETA: 1s - loss: 0.0453 - accuracy: 0.9852
 932/1688 [===============>..............] - ETA: 1s - loss: 0.0450 - accuracy: 0.9853
 961/1688 [================>.............] - ETA: 1s - loss: 0.0449 - accuracy: 0.9852
 991/1688 [================>.............] - ETA: 1s - loss: 0.0448 - accuracy: 0.9851
1021/1688 [=================>............] - ETA: 1s - loss: 0.0446 - accuracy: 0.9852
1051/1688 [=================>............] - ETA: 1s - loss: 0.0441 - accuracy: 0.9854
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0440 - accuracy: 0.9854
1111/1688 [==================>...........] - ETA: 0s - loss: 0.0441 - accuracy: 0.9854
1141/1688 [===================>..........] - ETA: 0s - loss: 0.0442 - accuracy: 0.9853
1171/1688 [===================>..........] - ETA: 0s - loss: 0.0439 - accuracy: 0.9854
1201/1688 [====================>.........] - ETA: 0s - loss: 0.0437 - accuracy: 0.9855
1231/1688 [====================>.........] - ETA: 0s - loss: 0.0441 - accuracy: 0.9853
1262/1688 [=====================>........] - ETA: 0s - loss: 0.0440 - accuracy: 0.9854
1292/1688 [=====================>........] - ETA: 0s - loss: 0.0443 - accuracy: 0.9852
1321/1688 [======================>.......] - ETA: 0s - loss: 0.0441 - accuracy: 0.9854
1352/1688 [=======================>......] - ETA: 0s - loss: 0.0443 - accuracy: 0.9853
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0440 - accuracy: 0.9854
1413/1688 [========================>.....] - ETA: 0s - loss: 0.0440 - accuracy: 0.9854
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0446 - accuracy: 0.9853
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0448 - accuracy: 0.9851
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0446 - accuracy: 0.9852
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0444 - accuracy: 0.9852
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0443 - accuracy: 0.9853
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0440 - accuracy: 0.9854
1623/1688 [===========================>..] - ETA: 0s - loss: 0.0444 - accuracy: 0.9853
1653/1688 [============================>.] - ETA: 0s - loss: 0.0446 - accuracy: 0.9853
1684/1688 [============================>.] - ETA: 0s - loss: 0.0444 - accuracy: 0.9854
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0443 - accuracy: 0.9854 - val_loss: 0.0672 - val_accuracy: 0.9807
Epoch 5/10

   1/1688 [..............................] - ETA: 3s - loss: 8.7618e-04 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0207 - accuracy: 0.9919    
  61/1688 [>.............................] - ETA: 2s - loss: 0.0204 - accuracy: 0.9939
  91/1688 [>.............................] - ETA: 2s - loss: 0.0279 - accuracy: 0.9911
 121/1688 [=>............................] - ETA: 2s - loss: 0.0311 - accuracy: 0.9907
 151/1688 [=>............................] - ETA: 2s - loss: 0.0369 - accuracy: 0.9890
 181/1688 [==>...........................] - ETA: 2s - loss: 0.0371 - accuracy: 0.9891
 211/1688 [==>...........................] - ETA: 2s - loss: 0.0356 - accuracy: 0.9890
 241/1688 [===>..........................] - ETA: 2s - loss: 0.0327 - accuracy: 0.9900
 271/1688 [===>..........................] - ETA: 2s - loss: 0.0371 - accuracy: 0.9886
 301/1688 [====>.........................] - ETA: 2s - loss: 0.0375 - accuracy: 0.9884
 331/1688 [====>.........................] - ETA: 2s - loss: 0.0359 - accuracy: 0.9888
 361/1688 [=====>........................] - ETA: 2s - loss: 0.0372 - accuracy: 0.9887
 391/1688 [=====>........................] - ETA: 2s - loss: 0.0356 - accuracy: 0.9890
 421/1688 [======>.......................] - ETA: 2s - loss: 0.0355 - accuracy: 0.9890
 451/1688 [=======>......................] - ETA: 2s - loss: 0.0353 - accuracy: 0.9893
 480/1688 [=======>......................] - ETA: 2s - loss: 0.0347 - accuracy: 0.9893
 509/1688 [========>.....................] - ETA: 2s - loss: 0.0343 - accuracy: 0.9893
 539/1688 [========>.....................] - ETA: 1s - loss: 0.0346 - accuracy: 0.9890
 569/1688 [=========>....................] - ETA: 1s - loss: 0.0344 - accuracy: 0.9891
 599/1688 [=========>....................] - ETA: 1s - loss: 0.0346 - accuracy: 0.9890
 629/1688 [==========>...................] - ETA: 1s - loss: 0.0346 - accuracy: 0.9890
 659/1688 [==========>...................] - ETA: 1s - loss: 0.0347 - accuracy: 0.9889
 690/1688 [===========>..................] - ETA: 1s - loss: 0.0360 - accuracy: 0.9885
 720/1688 [===========>..................] - ETA: 1s - loss: 0.0354 - accuracy: 0.9887
 751/1688 [============>.................] - ETA: 1s - loss: 0.0352 - accuracy: 0.9886
 781/1688 [============>.................] - ETA: 1s - loss: 0.0349 - accuracy: 0.9888
 811/1688 [=============>................] - ETA: 1s - loss: 0.0346 - accuracy: 0.9889
 841/1688 [=============>................] - ETA: 1s - loss: 0.0342 - accuracy: 0.9890
 871/1688 [==============>...............] - ETA: 1s - loss: 0.0338 - accuracy: 0.9892
 901/1688 [===============>..............] - ETA: 1s - loss: 0.0341 - accuracy: 0.9892
 931/1688 [===============>..............] - ETA: 1s - loss: 0.0352 - accuracy: 0.9890
 961/1688 [================>.............] - ETA: 1s - loss: 0.0354 - accuracy: 0.9890
 991/1688 [================>.............] - ETA: 1s - loss: 0.0354 - accuracy: 0.9889
1022/1688 [=================>............] - ETA: 1s - loss: 0.0351 - accuracy: 0.9890
1052/1688 [=================>............] - ETA: 1s - loss: 0.0350 - accuracy: 0.9890
1083/1688 [==================>...........] - ETA: 1s - loss: 0.0350 - accuracy: 0.9889
1113/1688 [==================>...........] - ETA: 0s - loss: 0.0347 - accuracy: 0.9889
1144/1688 [===================>..........] - ETA: 0s - loss: 0.0348 - accuracy: 0.9888
1174/1688 [===================>..........] - ETA: 0s - loss: 0.0350 - accuracy: 0.9887
1204/1688 [====================>.........] - ETA: 0s - loss: 0.0346 - accuracy: 0.9888
1234/1688 [====================>.........] - ETA: 0s - loss: 0.0350 - accuracy: 0.9887
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0351 - accuracy: 0.9886
1295/1688 [======================>.......] - ETA: 0s - loss: 0.0349 - accuracy: 0.9886
1326/1688 [======================>.......] - ETA: 0s - loss: 0.0353 - accuracy: 0.9885
1354/1688 [=======================>......] - ETA: 0s - loss: 0.0355 - accuracy: 0.9884
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0356 - accuracy: 0.9883
1413/1688 [========================>.....] - ETA: 0s - loss: 0.0356 - accuracy: 0.9883
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0354 - accuracy: 0.9884
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0356 - accuracy: 0.9883
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0355 - accuracy: 0.9883
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0356 - accuracy: 0.9883
1563/1688 [==========================>...] - ETA: 0s - loss: 0.0355 - accuracy: 0.9883
1592/1688 [===========================>..] - ETA: 0s - loss: 0.0356 - accuracy: 0.9883
1622/1688 [===========================>..] - ETA: 0s - loss: 0.0359 - accuracy: 0.9882
1652/1688 [============================>.] - ETA: 0s - loss: 0.0356 - accuracy: 0.9883
1681/1688 [============================>.] - ETA: 0s - loss: 0.0356 - accuracy: 0.9883
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0358 - accuracy: 0.9883 - val_loss: 0.0809 - val_accuracy: 0.9778
Epoch 6/10

   1/1688 [..............................] - ETA: 3s - loss: 0.1163 - accuracy: 0.9688
  32/1688 [..............................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9902
  63/1688 [>.............................] - ETA: 2s - loss: 0.0288 - accuracy: 0.9901
  93/1688 [>.............................] - ETA: 2s - loss: 0.0254 - accuracy: 0.9916
 123/1688 [=>............................] - ETA: 2s - loss: 0.0249 - accuracy: 0.9919
 152/1688 [=>............................] - ETA: 2s - loss: 0.0257 - accuracy: 0.9922
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0263 - accuracy: 0.9923
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0277 - accuracy: 0.9916
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0282 - accuracy: 0.9913
 272/1688 [===>..........................] - ETA: 2s - loss: 0.0297 - accuracy: 0.9908
 302/1688 [====>.........................] - ETA: 2s - loss: 0.0302 - accuracy: 0.9910
 332/1688 [====>.........................] - ETA: 2s - loss: 0.0306 - accuracy: 0.9908
 362/1688 [=====>........................] - ETA: 2s - loss: 0.0301 - accuracy: 0.9908
 392/1688 [=====>........................] - ETA: 2s - loss: 0.0299 - accuracy: 0.9909
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0295 - accuracy: 0.9912
 452/1688 [=======>......................] - ETA: 2s - loss: 0.0288 - accuracy: 0.9913
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0295 - accuracy: 0.9911
 512/1688 [========>.....................] - ETA: 1s - loss: 0.0288 - accuracy: 0.9911
 542/1688 [========>.....................] - ETA: 1s - loss: 0.0285 - accuracy: 0.9911
 572/1688 [=========>....................] - ETA: 1s - loss: 0.0278 - accuracy: 0.9913
 602/1688 [=========>....................] - ETA: 1s - loss: 0.0274 - accuracy: 0.9913
 632/1688 [==========>...................] - ETA: 1s - loss: 0.0275 - accuracy: 0.9912
 662/1688 [==========>...................] - ETA: 1s - loss: 0.0281 - accuracy: 0.9912
 692/1688 [===========>..................] - ETA: 1s - loss: 0.0282 - accuracy: 0.9911
 722/1688 [===========>..................] - ETA: 1s - loss: 0.0283 - accuracy: 0.9911
 752/1688 [============>.................] - ETA: 1s - loss: 0.0281 - accuracy: 0.9911
 782/1688 [============>.................] - ETA: 1s - loss: 0.0277 - accuracy: 0.9913
 812/1688 [=============>................] - ETA: 1s - loss: 0.0277 - accuracy: 0.9913
 842/1688 [=============>................] - ETA: 1s - loss: 0.0277 - accuracy: 0.9913
 872/1688 [==============>...............] - ETA: 1s - loss: 0.0274 - accuracy: 0.9914
 902/1688 [===============>..............] - ETA: 1s - loss: 0.0272 - accuracy: 0.9914
 932/1688 [===============>..............] - ETA: 1s - loss: 0.0272 - accuracy: 0.9914
 961/1688 [================>.............] - ETA: 1s - loss: 0.0272 - accuracy: 0.9913
 991/1688 [================>.............] - ETA: 1s - loss: 0.0277 - accuracy: 0.9912
1021/1688 [=================>............] - ETA: 1s - loss: 0.0281 - accuracy: 0.9911
1051/1688 [=================>............] - ETA: 1s - loss: 0.0287 - accuracy: 0.9909
1081/1688 [==================>...........] - ETA: 1s - loss: 0.0289 - accuracy: 0.9908
1111/1688 [==================>...........] - ETA: 0s - loss: 0.0292 - accuracy: 0.9906
1141/1688 [===================>..........] - ETA: 0s - loss: 0.0296 - accuracy: 0.9904
1170/1688 [===================>..........] - ETA: 0s - loss: 0.0294 - accuracy: 0.9904
1199/1688 [====================>.........] - ETA: 0s - loss: 0.0296 - accuracy: 0.9903
1229/1688 [====================>.........] - ETA: 0s - loss: 0.0297 - accuracy: 0.9903
1259/1688 [=====================>........] - ETA: 0s - loss: 0.0298 - accuracy: 0.9902
1289/1688 [=====================>........] - ETA: 0s - loss: 0.0302 - accuracy: 0.9901
1319/1688 [======================>.......] - ETA: 0s - loss: 0.0302 - accuracy: 0.9901
1349/1688 [======================>.......] - ETA: 0s - loss: 0.0301 - accuracy: 0.9902
1379/1688 [=======================>......] - ETA: 0s - loss: 0.0303 - accuracy: 0.9901
1409/1688 [========================>.....] - ETA: 0s - loss: 0.0305 - accuracy: 0.9900
1439/1688 [========================>.....] - ETA: 0s - loss: 0.0306 - accuracy: 0.9899
1469/1688 [=========================>....] - ETA: 0s - loss: 0.0308 - accuracy: 0.9899
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0310 - accuracy: 0.9898
1529/1688 [==========================>...] - ETA: 0s - loss: 0.0311 - accuracy: 0.9899
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0311 - accuracy: 0.9899
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0313 - accuracy: 0.9898
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0312 - accuracy: 0.9899
1649/1688 [============================>.] - ETA: 0s - loss: 0.0313 - accuracy: 0.9899
1679/1688 [============================>.] - ETA: 0s - loss: 0.0314 - accuracy: 0.9899
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0314 - accuracy: 0.9899 - val_loss: 0.0618 - val_accuracy: 0.9827
Epoch 7/10

   1/1688 [..............................] - ETA: 2s - loss: 0.0079 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0195 - accuracy: 0.9941
  62/1688 [>.............................] - ETA: 2s - loss: 0.0198 - accuracy: 0.9934
  92/1688 [>.............................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9949
 122/1688 [=>............................] - ETA: 2s - loss: 0.0182 - accuracy: 0.9951
 152/1688 [=>............................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9949
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0189 - accuracy: 0.9943
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0201 - accuracy: 0.9938
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0206 - accuracy: 0.9939
 272/1688 [===>..........................] - ETA: 2s - loss: 0.0212 - accuracy: 0.9931
 301/1688 [====>.........................] - ETA: 2s - loss: 0.0228 - accuracy: 0.9927
 331/1688 [====>.........................] - ETA: 2s - loss: 0.0238 - accuracy: 0.9922
 361/1688 [=====>........................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9925
 391/1688 [=====>........................] - ETA: 2s - loss: 0.0231 - accuracy: 0.9923
 422/1688 [======>.......................] - ETA: 2s - loss: 0.0230 - accuracy: 0.9923
 452/1688 [=======>......................] - ETA: 2s - loss: 0.0235 - accuracy: 0.9925
 482/1688 [=======>......................] - ETA: 2s - loss: 0.0227 - accuracy: 0.9928
 512/1688 [========>.....................] - ETA: 1s - loss: 0.0228 - accuracy: 0.9927
 542/1688 [========>.....................] - ETA: 1s - loss: 0.0228 - accuracy: 0.9929
 572/1688 [=========>....................] - ETA: 1s - loss: 0.0229 - accuracy: 0.9928
 601/1688 [=========>....................] - ETA: 1s - loss: 0.0235 - accuracy: 0.9926
 631/1688 [==========>...................] - ETA: 1s - loss: 0.0241 - accuracy: 0.9922
 661/1688 [==========>...................] - ETA: 1s - loss: 0.0237 - accuracy: 0.9922
 691/1688 [===========>..................] - ETA: 1s - loss: 0.0235 - accuracy: 0.9922
 720/1688 [===========>..................] - ETA: 1s - loss: 0.0234 - accuracy: 0.9922
 750/1688 [============>.................] - ETA: 1s - loss: 0.0236 - accuracy: 0.9921
 780/1688 [============>.................] - ETA: 1s - loss: 0.0234 - accuracy: 0.9921
 810/1688 [=============>................] - ETA: 1s - loss: 0.0233 - accuracy: 0.9922
 839/1688 [=============>................] - ETA: 1s - loss: 0.0232 - accuracy: 0.9922
 869/1688 [==============>...............] - ETA: 1s - loss: 0.0237 - accuracy: 0.9920
 899/1688 [==============>...............] - ETA: 1s - loss: 0.0236 - accuracy: 0.9920
 928/1688 [===============>..............] - ETA: 1s - loss: 0.0236 - accuracy: 0.9920
 959/1688 [================>.............] - ETA: 1s - loss: 0.0237 - accuracy: 0.9920
 988/1688 [================>.............] - ETA: 1s - loss: 0.0240 - accuracy: 0.9920
1018/1688 [=================>............] - ETA: 1s - loss: 0.0241 - accuracy: 0.9919
1048/1688 [=================>............] - ETA: 1s - loss: 0.0241 - accuracy: 0.9919
1078/1688 [==================>...........] - ETA: 1s - loss: 0.0239 - accuracy: 0.9919
1108/1688 [==================>...........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9918
1138/1688 [===================>..........] - ETA: 0s - loss: 0.0237 - accuracy: 0.9919
1168/1688 [===================>..........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9918
1198/1688 [====================>.........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9918
1228/1688 [====================>.........] - ETA: 0s - loss: 0.0238 - accuracy: 0.9918
1258/1688 [=====================>........] - ETA: 0s - loss: 0.0239 - accuracy: 0.9918
1288/1688 [=====================>........] - ETA: 0s - loss: 0.0244 - accuracy: 0.9916
1317/1688 [======================>.......] - ETA: 0s - loss: 0.0244 - accuracy: 0.9916
1347/1688 [======================>.......] - ETA: 0s - loss: 0.0244 - accuracy: 0.9915
1377/1688 [=======================>......] - ETA: 0s - loss: 0.0242 - accuracy: 0.9916
1407/1688 [========================>.....] - ETA: 0s - loss: 0.0242 - accuracy: 0.9915
1437/1688 [========================>.....] - ETA: 0s - loss: 0.0243 - accuracy: 0.9915
1467/1688 [=========================>....] - ETA: 0s - loss: 0.0243 - accuracy: 0.9915
1498/1688 [=========================>....] - ETA: 0s - loss: 0.0244 - accuracy: 0.9915
1528/1688 [==========================>...] - ETA: 0s - loss: 0.0246 - accuracy: 0.9915
1558/1688 [==========================>...] - ETA: 0s - loss: 0.0244 - accuracy: 0.9915
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0246 - accuracy: 0.9914
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0249 - accuracy: 0.9913
1650/1688 [============================>.] - ETA: 0s - loss: 0.0250 - accuracy: 0.9912
1680/1688 [============================>.] - ETA: 0s - loss: 0.0252 - accuracy: 0.9912
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0253 - accuracy: 0.9912 - val_loss: 0.0591 - val_accuracy: 0.9842
Epoch 8/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0238 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0196 - accuracy: 0.9932
  62/1688 [>.............................] - ETA: 2s - loss: 0.0188 - accuracy: 0.9940
  92/1688 [>.............................] - ETA: 2s - loss: 0.0173 - accuracy: 0.9942
 121/1688 [=>............................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9938
 152/1688 [=>............................] - ETA: 2s - loss: 0.0200 - accuracy: 0.9932
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0184 - accuracy: 0.9938
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9943
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0168 - accuracy: 0.9944
 272/1688 [===>..........................] - ETA: 2s - loss: 0.0172 - accuracy: 0.9944
 301/1688 [====>.........................] - ETA: 2s - loss: 0.0167 - accuracy: 0.9945
 331/1688 [====>.........................] - ETA: 2s - loss: 0.0176 - accuracy: 0.9941
 361/1688 [=====>........................] - ETA: 2s - loss: 0.0167 - accuracy: 0.9945
 391/1688 [=====>........................] - ETA: 2s - loss: 0.0180 - accuracy: 0.9942
 421/1688 [======>.......................] - ETA: 2s - loss: 0.0178 - accuracy: 0.9943
 451/1688 [=======>......................] - ETA: 2s - loss: 0.0175 - accuracy: 0.9943
 481/1688 [=======>......................] - ETA: 2s - loss: 0.0175 - accuracy: 0.9942
 511/1688 [========>.....................] - ETA: 1s - loss: 0.0178 - accuracy: 0.9941
 542/1688 [========>.....................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9942
 572/1688 [=========>....................] - ETA: 1s - loss: 0.0175 - accuracy: 0.9942
 601/1688 [=========>....................] - ETA: 1s - loss: 0.0175 - accuracy: 0.9942
 631/1688 [==========>...................] - ETA: 1s - loss: 0.0177 - accuracy: 0.9941
 661/1688 [==========>...................] - ETA: 1s - loss: 0.0174 - accuracy: 0.9942
 690/1688 [===========>..................] - ETA: 1s - loss: 0.0174 - accuracy: 0.9942
 720/1688 [===========>..................] - ETA: 1s - loss: 0.0179 - accuracy: 0.9941
 750/1688 [============>.................] - ETA: 1s - loss: 0.0184 - accuracy: 0.9940
 779/1688 [============>.................] - ETA: 1s - loss: 0.0186 - accuracy: 0.9939
 809/1688 [=============>................] - ETA: 1s - loss: 0.0192 - accuracy: 0.9937
 838/1688 [=============>................] - ETA: 1s - loss: 0.0197 - accuracy: 0.9935
 868/1688 [==============>...............] - ETA: 1s - loss: 0.0194 - accuracy: 0.9936
 899/1688 [==============>...............] - ETA: 1s - loss: 0.0192 - accuracy: 0.9937
 929/1688 [===============>..............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9936
 959/1688 [================>.............] - ETA: 1s - loss: 0.0194 - accuracy: 0.9935
 989/1688 [================>.............] - ETA: 1s - loss: 0.0195 - accuracy: 0.9935
1019/1688 [=================>............] - ETA: 1s - loss: 0.0193 - accuracy: 0.9935
1049/1688 [=================>............] - ETA: 1s - loss: 0.0197 - accuracy: 0.9933
1080/1688 [==================>...........] - ETA: 1s - loss: 0.0197 - accuracy: 0.9933
1110/1688 [==================>...........] - ETA: 0s - loss: 0.0198 - accuracy: 0.9933
1140/1688 [===================>..........] - ETA: 0s - loss: 0.0201 - accuracy: 0.9932
1171/1688 [===================>..........] - ETA: 0s - loss: 0.0209 - accuracy: 0.9929
1201/1688 [====================>.........] - ETA: 0s - loss: 0.0210 - accuracy: 0.9929
1233/1688 [====================>.........] - ETA: 0s - loss: 0.0210 - accuracy: 0.9928
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0208 - accuracy: 0.9929
1293/1688 [=====================>........] - ETA: 0s - loss: 0.0208 - accuracy: 0.9929
1323/1688 [======================>.......] - ETA: 0s - loss: 0.0208 - accuracy: 0.9929
1353/1688 [=======================>......] - ETA: 0s - loss: 0.0210 - accuracy: 0.9927
1384/1688 [=======================>......] - ETA: 0s - loss: 0.0210 - accuracy: 0.9928
1414/1688 [========================>.....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9928
1444/1688 [========================>.....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9928
1475/1688 [=========================>....] - ETA: 0s - loss: 0.0214 - accuracy: 0.9928
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0213 - accuracy: 0.9928
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0212 - accuracy: 0.9928
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0213 - accuracy: 0.9928
1594/1688 [===========================>..] - ETA: 0s - loss: 0.0214 - accuracy: 0.9926
1624/1688 [===========================>..] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1654/1688 [============================>.] - ETA: 0s - loss: 0.0217 - accuracy: 0.9926
1684/1688 [============================>.] - ETA: 0s - loss: 0.0219 - accuracy: 0.9926
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0219 - accuracy: 0.9926 - val_loss: 0.0506 - val_accuracy: 0.9875
Epoch 9/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0031 - accuracy: 1.0000
  32/1688 [..............................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9990
  61/1688 [>.............................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9985
  90/1688 [>.............................] - ETA: 2s - loss: 0.0087 - accuracy: 0.9990
 120/1688 [=>............................] - ETA: 2s - loss: 0.0081 - accuracy: 0.9987
 151/1688 [=>............................] - ETA: 2s - loss: 0.0094 - accuracy: 0.9979
 183/1688 [==>...........................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9974
 213/1688 [==>...........................] - ETA: 2s - loss: 0.0099 - accuracy: 0.9978
 243/1688 [===>..........................] - ETA: 2s - loss: 0.0105 - accuracy: 0.9973
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0106 - accuracy: 0.9973
 303/1688 [====>.........................] - ETA: 2s - loss: 0.0107 - accuracy: 0.9973
 333/1688 [====>.........................] - ETA: 2s - loss: 0.0109 - accuracy: 0.9972
 363/1688 [=====>........................] - ETA: 2s - loss: 0.0110 - accuracy: 0.9972
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9971
 425/1688 [======>.......................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9974
 454/1688 [=======>......................] - ETA: 2s - loss: 0.0111 - accuracy: 0.9972
 484/1688 [=======>......................] - ETA: 2s - loss: 0.0108 - accuracy: 0.9974
 514/1688 [========>.....................] - ETA: 1s - loss: 0.0111 - accuracy: 0.9971
 545/1688 [========>.....................] - ETA: 1s - loss: 0.0113 - accuracy: 0.9970
 575/1688 [=========>....................] - ETA: 1s - loss: 0.0112 - accuracy: 0.9970
 605/1688 [=========>....................] - ETA: 1s - loss: 0.0117 - accuracy: 0.9967
 635/1688 [==========>...................] - ETA: 1s - loss: 0.0116 - accuracy: 0.9966
 664/1688 [==========>...................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9964
 693/1688 [===========>..................] - ETA: 1s - loss: 0.0123 - accuracy: 0.9963
 723/1688 [===========>..................] - ETA: 1s - loss: 0.0126 - accuracy: 0.9962
 750/1688 [============>.................] - ETA: 1s - loss: 0.0133 - accuracy: 0.9960
 778/1688 [============>.................] - ETA: 1s - loss: 0.0138 - accuracy: 0.9958
 806/1688 [=============>................] - ETA: 1s - loss: 0.0138 - accuracy: 0.9958
 834/1688 [=============>................] - ETA: 1s - loss: 0.0140 - accuracy: 0.9957
 862/1688 [==============>...............] - ETA: 1s - loss: 0.0140 - accuracy: 0.9956
 890/1688 [==============>...............] - ETA: 1s - loss: 0.0144 - accuracy: 0.9954
 919/1688 [===============>..............] - ETA: 1s - loss: 0.0150 - accuracy: 0.9952
 947/1688 [===============>..............] - ETA: 1s - loss: 0.0151 - accuracy: 0.9951
 975/1688 [================>.............] - ETA: 1s - loss: 0.0155 - accuracy: 0.9950
1003/1688 [================>.............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9950
1032/1688 [=================>............] - ETA: 1s - loss: 0.0155 - accuracy: 0.9949
1061/1688 [=================>............] - ETA: 1s - loss: 0.0158 - accuracy: 0.9948
1089/1688 [==================>...........] - ETA: 1s - loss: 0.0160 - accuracy: 0.9947
1117/1688 [==================>...........] - ETA: 0s - loss: 0.0163 - accuracy: 0.9946
1145/1688 [===================>..........] - ETA: 0s - loss: 0.0168 - accuracy: 0.9944
1175/1688 [===================>..........] - ETA: 0s - loss: 0.0170 - accuracy: 0.9944
1205/1688 [====================>.........] - ETA: 0s - loss: 0.0168 - accuracy: 0.9945
1234/1688 [====================>.........] - ETA: 0s - loss: 0.0166 - accuracy: 0.9945
1264/1688 [=====================>........] - ETA: 0s - loss: 0.0166 - accuracy: 0.9945
1294/1688 [=====================>........] - ETA: 0s - loss: 0.0165 - accuracy: 0.9945
1324/1688 [======================>.......] - ETA: 0s - loss: 0.0168 - accuracy: 0.9945
1354/1688 [=======================>......] - ETA: 0s - loss: 0.0171 - accuracy: 0.9943
1383/1688 [=======================>......] - ETA: 0s - loss: 0.0173 - accuracy: 0.9943
1413/1688 [========================>.....] - ETA: 0s - loss: 0.0175 - accuracy: 0.9942
1443/1688 [========================>.....] - ETA: 0s - loss: 0.0179 - accuracy: 0.9942
1474/1688 [=========================>....] - ETA: 0s - loss: 0.0182 - accuracy: 0.9940
1505/1688 [=========================>....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9940
1535/1688 [==========================>...] - ETA: 0s - loss: 0.0185 - accuracy: 0.9940
1565/1688 [==========================>...] - ETA: 0s - loss: 0.0185 - accuracy: 0.9940
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0185 - accuracy: 0.9940
1626/1688 [===========================>..] - ETA: 0s - loss: 0.0188 - accuracy: 0.9939
1656/1688 [============================>.] - ETA: 0s - loss: 0.0192 - accuracy: 0.9938
1686/1688 [============================>.] - ETA: 0s - loss: 0.0193 - accuracy: 0.9938
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0192 - accuracy: 0.9938 - val_loss: 0.0634 - val_accuracy: 0.9848
Epoch 10/10

   1/1688 [..............................] - ETA: 3s - loss: 0.0022 - accuracy: 1.0000
  31/1688 [..............................] - ETA: 2s - loss: 0.0098 - accuracy: 0.9940
  61/1688 [>.............................] - ETA: 2s - loss: 0.0100 - accuracy: 0.9949
  91/1688 [>.............................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9945
 122/1688 [=>............................] - ETA: 2s - loss: 0.0150 - accuracy: 0.9936
 152/1688 [=>............................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9940
 182/1688 [==>...........................] - ETA: 2s - loss: 0.0130 - accuracy: 0.9945
 212/1688 [==>...........................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9951
 242/1688 [===>..........................] - ETA: 2s - loss: 0.0112 - accuracy: 0.9955
 273/1688 [===>..........................] - ETA: 2s - loss: 0.0119 - accuracy: 0.9955
 303/1688 [====>.........................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9952
 334/1688 [====>.........................] - ETA: 2s - loss: 0.0138 - accuracy: 0.9952
 364/1688 [=====>........................] - ETA: 2s - loss: 0.0136 - accuracy: 0.9953
 394/1688 [======>.......................] - ETA: 2s - loss: 0.0140 - accuracy: 0.9952
 424/1688 [======>.......................] - ETA: 2s - loss: 0.0141 - accuracy: 0.9952
 454/1688 [=======>......................] - ETA: 2s - loss: 0.0144 - accuracy: 0.9952
 484/1688 [=======>......................] - ETA: 2s - loss: 0.0142 - accuracy: 0.9953
 514/1688 [========>.....................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9951
 544/1688 [========>.....................] - ETA: 1s - loss: 0.0142 - accuracy: 0.9952
 574/1688 [=========>....................] - ETA: 1s - loss: 0.0141 - accuracy: 0.9953
 604/1688 [=========>....................] - ETA: 1s - loss: 0.0146 - accuracy: 0.9951
 634/1688 [==========>...................] - ETA: 1s - loss: 0.0147 - accuracy: 0.9950
 665/1688 [==========>...................] - ETA: 1s - loss: 0.0151 - accuracy: 0.9950
 695/1688 [===========>..................] - ETA: 1s - loss: 0.0151 - accuracy: 0.9950
 725/1688 [===========>..................] - ETA: 1s - loss: 0.0156 - accuracy: 0.9949
 755/1688 [============>.................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9950
 785/1688 [============>.................] - ETA: 1s - loss: 0.0152 - accuracy: 0.9950
 815/1688 [=============>................] - ETA: 1s - loss: 0.0153 - accuracy: 0.9950
 845/1688 [==============>...............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9949
 875/1688 [==============>...............] - ETA: 1s - loss: 0.0154 - accuracy: 0.9948
 904/1688 [===============>..............] - ETA: 1s - loss: 0.0154 - accuracy: 0.9948
 934/1688 [===============>..............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9949
 963/1688 [================>.............] - ETA: 1s - loss: 0.0151 - accuracy: 0.9949
 993/1688 [================>.............] - ETA: 1s - loss: 0.0152 - accuracy: 0.9948
1023/1688 [=================>............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9947
1052/1688 [=================>............] - ETA: 1s - loss: 0.0153 - accuracy: 0.9947
1082/1688 [==================>...........] - ETA: 1s - loss: 0.0152 - accuracy: 0.9947
1112/1688 [==================>...........] - ETA: 0s - loss: 0.0152 - accuracy: 0.9947
1142/1688 [===================>..........] - ETA: 0s - loss: 0.0157 - accuracy: 0.9945
1172/1688 [===================>..........] - ETA: 0s - loss: 0.0158 - accuracy: 0.9944
1202/1688 [====================>.........] - ETA: 0s - loss: 0.0159 - accuracy: 0.9944
1233/1688 [====================>.........] - ETA: 0s - loss: 0.0161 - accuracy: 0.9943
1262/1688 [=====================>........] - ETA: 0s - loss: 0.0164 - accuracy: 0.9942
1291/1688 [=====================>........] - ETA: 0s - loss: 0.0163 - accuracy: 0.9942
1320/1688 [======================>.......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9941
1350/1688 [======================>.......] - ETA: 0s - loss: 0.0166 - accuracy: 0.9941
1380/1688 [=======================>......] - ETA: 0s - loss: 0.0165 - accuracy: 0.9942
1410/1688 [========================>.....] - ETA: 0s - loss: 0.0163 - accuracy: 0.9942
1440/1688 [========================>.....] - ETA: 0s - loss: 0.0164 - accuracy: 0.9941
1470/1688 [=========================>....] - ETA: 0s - loss: 0.0162 - accuracy: 0.9942
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0163 - accuracy: 0.9941
1530/1688 [==========================>...] - ETA: 0s - loss: 0.0163 - accuracy: 0.9941
1559/1688 [==========================>...] - ETA: 0s - loss: 0.0164 - accuracy: 0.9940
1589/1688 [===========================>..] - ETA: 0s - loss: 0.0164 - accuracy: 0.9940
1619/1688 [===========================>..] - ETA: 0s - loss: 0.0168 - accuracy: 0.9939
1648/1688 [============================>.] - ETA: 0s - loss: 0.0170 - accuracy: 0.9939
1678/1688 [============================>.] - ETA: 0s - loss: 0.0169 - accuracy: 0.9939
1688/1688 [==============================] - 3s 2ms/step - loss: 0.0170 - accuracy: 0.9939 - val_loss: 0.0520 - val_accuracy: 0.9860
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test accuracy:', score[1])
Test accuracy: 0.9847999811172485

2. Quantize

2.1. 8-bit quantization

An Akida accelerator processes 8, 4, 2, or 1 bit integer activations and weights. Therefore, the floating point Keras model must be quantized in preparation to run on an Akida accelerator.

The QuantizeML quantize function can be used to quantize a Keras model for Akida. For this step in this example, an “8/8/8” quantization scheme will be applied to the floating point Keras model to produce 8-bit weights in the first layer, 8-bit weights in all other layers, and 8-bit activations.

The quantization process results in a Keras model with custom QuantizeML quantized layers substituted for the original Keras layers. All Keras API functions can be applied on this new model: summary(), compile(), fit(). etc.

Note

The quantize function applies several transformations to the original model. For example, it folds the batch normalization layers into the corresponding neural layers. The new weights are computed according to this folding operation.

from quantizeml.models import quantize, QuantizationParams

qparams = QuantizationParams(input_weight_bits=8, weight_bits=8, activation_bits=8)
model_quantized = quantize(model_keras, qparams=qparams)
model_quantized.summary()
/usr/local/lib/python3.8/dist-packages/quantizeml/models/quantize.py:400: UserWarning: Quantizing per-axis with random calibration samples is not accurate.                       Set QuantizationParams.per_tensor_activations=True when calibrating with                        random samples.
  warnings.warn("Quantizing per-axis with random calibration samples is not accurate.\

   1/1024 [..............................] - ETA: 2:29
  58/1024 [>.............................] - ETA: 0s  
 115/1024 [==>...........................] - ETA: 0s
 173/1024 [====>.........................] - ETA: 0s
 230/1024 [=====>........................] - ETA: 0s
 287/1024 [=======>......................] - ETA: 0s
 346/1024 [=========>....................] - ETA: 0s
 403/1024 [==========>...................] - ETA: 0s
 461/1024 [============>.................] - ETA: 0s
 518/1024 [==============>...............] - ETA: 0s
 575/1024 [===============>..............] - ETA: 0s
 633/1024 [=================>............] - ETA: 0s
 691/1024 [===================>..........] - ETA: 0s
 750/1024 [====================>.........] - ETA: 0s
 809/1024 [======================>.......] - ETA: 0s
 866/1024 [========================>.....] - ETA: 0s
 923/1024 [==========================>...] - ETA: 0s
 980/1024 [===========================>..] - ETA: 0s
1024/1024 [==============================] - 1s 877us/step
Model: "mnistnet"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 rescaling_input (InputLayer  [(None, 28, 28, 1)]      0
 )

 rescaling (QuantizedRescali  (None, 28, 28, 1)        0
 ng)

 conv2d (QuantizedConv2D)    (None, 13, 13, 32)        320

 re_lu (QuantizedReLU)       (None, 13, 13, 32)        64

 depthwise_conv2d (Quantized  (None, 7, 7, 32)         384
 DepthwiseConv2D)

 conv2d_1 (QuantizedConv2D)  (None, 7, 7, 64)          2112

 re_lu_1 (QuantizedReLU)     (None, 7, 7, 64)          128

 flatten (QuantizedFlatten)  (None, 3136)              0

 dense (QuantizedDense)      (None, 10)                31370

 dequantizer (Dequantizer)   [(None, 10)]              0

=================================================================
Total params: 34,378
Trainable params: 34,122
Non-trainable params: 256
_________________________________________________________________

Note

Note that the number of parameters for the floating and quantized models differs, a consequence of the BatchNormalization folding and the additional parameters added for quantization. For further details, please refer to their respective summary.

Check the quantized model accuracy.

def compile_evaluate(model):
    """ Compiles and evaluates the model, then return accuracy score. """
    model.compile(metrics=['accuracy'])
    return model.evaluate(x_test, y_test, verbose=0)[1]


print('Test accuracy after 8-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 8-bit quantization: 0.9800999760627747

2.2. Effect of calibration

The previous call to quantize was made with random samples for calibration (default parameters). While the observed drop in accuracy is minimal, that is around 1%, it can be worse on more complex models. Therefore, it is advised to use a set of real samples from the training set for calibration during a call to quantize. Note that this remains a calibration step rather than a training step in that no output labels are required. Furthermore, any relevant data could be used for calibration. The recommended settings for calibration that are widely used to obtain the zoo performance are:

  • 1024 samples

  • a batch size of 100

  • 2 epochs

model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)
 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 1ms/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 987us/step

Check the accuracy for the quantized and calibrated model.

print('Test accuracy after calibration:', compile_evaluate(model_quantized))
Test accuracy after calibration: 0.9786999821662903

Calibrating with real samples on this model recovers the initial float accuracy.

2.3. 4-bit quantization

The accuracy of the 8/8/8 quantized model is equal to that of the Keras floating point model. In some cases, a smaller memory size for the model is required. This can be accomplished through quantization of the model to smaller bitwidths.

The model will now be quantized to 8/4/4, that is 8-bit weights in the first layer with 4-bit weights and activations in all other layers. Such a quantization scheme will usually introduce a performance drop.

qparams = QuantizationParams(input_weight_bits=8, weight_bits=4, activation_bits=4)
model_quantized = quantize(model_keras, qparams=qparams,
                           samples=x_train, num_samples=1024, batch_size=100, epochs=2)
 1/11 [=>............................] - ETA: 1s
11/11 [==============================] - 0s 980us/step

 1/11 [=>............................] - ETA: 0s
11/11 [==============================] - 0s 1ms/step

Check the 4-bit quantized accuracy.

print('Test accuracy after 4-bit quantization:', compile_evaluate(model_quantized))
Test accuracy after 4-bit quantization: 0.9733999967575073

2.4. Model fine tuning (Quantization Aware Training)

When a model suffers from an accuracy drop after quantization, fine tuning or Quantization Aware Training (QAT) may recover some or all of the original performance.

Note that since this is a fine tuning step, both the number of epochs and learning rate are expected to be lower than during the initial float training.

model_quantized.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=Adam(learning_rate=1e-4),
    metrics=['accuracy'])

model_quantized.fit(x_train, y_train, epochs=5, validation_split=0.1)
Epoch 1/5

   1/1688 [..............................] - ETA: 1:05:23 - loss: 0.0426 - accuracy: 0.9688
  13/1688 [..............................] - ETA: 7s - loss: 0.0395 - accuracy: 0.9832     
  25/1688 [..............................] - ETA: 7s - loss: 0.0354 - accuracy: 0.9875
  36/1688 [..............................] - ETA: 7s - loss: 0.0376 - accuracy: 0.9861
  47/1688 [..............................] - ETA: 7s - loss: 0.0388 - accuracy: 0.9860
  59/1688 [>.............................] - ETA: 7s - loss: 0.0380 - accuracy: 0.9873
  71/1688 [>.............................] - ETA: 7s - loss: 0.0422 - accuracy: 0.9842
  82/1688 [>.............................] - ETA: 7s - loss: 0.0396 - accuracy: 0.9855
  93/1688 [>.............................] - ETA: 7s - loss: 0.0384 - accuracy: 0.9856
 105/1688 [>.............................] - ETA: 7s - loss: 0.0363 - accuracy: 0.9866
 116/1688 [=>............................] - ETA: 7s - loss: 0.0360 - accuracy: 0.9868
 128/1688 [=>............................] - ETA: 7s - loss: 0.0359 - accuracy: 0.9871
 139/1688 [=>............................] - ETA: 7s - loss: 0.0355 - accuracy: 0.9872
 150/1688 [=>............................] - ETA: 6s - loss: 0.0338 - accuracy: 0.9879
 161/1688 [=>............................] - ETA: 6s - loss: 0.0328 - accuracy: 0.9884
 172/1688 [==>...........................] - ETA: 6s - loss: 0.0317 - accuracy: 0.9887
 183/1688 [==>...........................] - ETA: 6s - loss: 0.0306 - accuracy: 0.9892
 194/1688 [==>...........................] - ETA: 6s - loss: 0.0294 - accuracy: 0.9899
 206/1688 [==>...........................] - ETA: 6s - loss: 0.0286 - accuracy: 0.9903
 218/1688 [==>...........................] - ETA: 6s - loss: 0.0281 - accuracy: 0.9905
 230/1688 [===>..........................] - ETA: 6s - loss: 0.0274 - accuracy: 0.9909
 241/1688 [===>..........................] - ETA: 6s - loss: 0.0270 - accuracy: 0.9911
 253/1688 [===>..........................] - ETA: 6s - loss: 0.0265 - accuracy: 0.9914
 265/1688 [===>..........................] - ETA: 6s - loss: 0.0259 - accuracy: 0.9917
 276/1688 [===>..........................] - ETA: 6s - loss: 0.0260 - accuracy: 0.9916
 287/1688 [====>.........................] - ETA: 6s - loss: 0.0259 - accuracy: 0.9918
 298/1688 [====>.........................] - ETA: 6s - loss: 0.0259 - accuracy: 0.9919
 309/1688 [====>.........................] - ETA: 6s - loss: 0.0259 - accuracy: 0.9919
 320/1688 [====>.........................] - ETA: 6s - loss: 0.0254 - accuracy: 0.9921
 332/1688 [====>.........................] - ETA: 6s - loss: 0.0249 - accuracy: 0.9924
 343/1688 [=====>........................] - ETA: 6s - loss: 0.0244 - accuracy: 0.9926
 354/1688 [=====>........................] - ETA: 6s - loss: 0.0243 - accuracy: 0.9926
 366/1688 [=====>........................] - ETA: 6s - loss: 0.0244 - accuracy: 0.9927
 377/1688 [=====>........................] - ETA: 5s - loss: 0.0243 - accuracy: 0.9926
 389/1688 [=====>........................] - ETA: 5s - loss: 0.0240 - accuracy: 0.9929
 400/1688 [======>.......................] - ETA: 5s - loss: 0.0239 - accuracy: 0.9928
 411/1688 [======>.......................] - ETA: 5s - loss: 0.0238 - accuracy: 0.9929
 422/1688 [======>.......................] - ETA: 5s - loss: 0.0240 - accuracy: 0.9927
 433/1688 [======>.......................] - ETA: 5s - loss: 0.0239 - accuracy: 0.9926
 444/1688 [======>.......................] - ETA: 5s - loss: 0.0234 - accuracy: 0.9928
 456/1688 [=======>......................] - ETA: 5s - loss: 0.0233 - accuracy: 0.9929
 468/1688 [=======>......................] - ETA: 5s - loss: 0.0230 - accuracy: 0.9930
 479/1688 [=======>......................] - ETA: 5s - loss: 0.0232 - accuracy: 0.9930
 491/1688 [=======>......................] - ETA: 5s - loss: 0.0234 - accuracy: 0.9928
 502/1688 [=======>......................] - ETA: 5s - loss: 0.0233 - accuracy: 0.9929
 513/1688 [========>.....................] - ETA: 5s - loss: 0.0232 - accuracy: 0.9930
 524/1688 [========>.....................] - ETA: 5s - loss: 0.0232 - accuracy: 0.9930
 535/1688 [========>.....................] - ETA: 5s - loss: 0.0229 - accuracy: 0.9931
 546/1688 [========>.....................] - ETA: 5s - loss: 0.0229 - accuracy: 0.9930
 557/1688 [========>.....................] - ETA: 5s - loss: 0.0227 - accuracy: 0.9930
 569/1688 [=========>....................] - ETA: 5s - loss: 0.0225 - accuracy: 0.9931
 580/1688 [=========>....................] - ETA: 5s - loss: 0.0225 - accuracy: 0.9931
 591/1688 [=========>....................] - ETA: 5s - loss: 0.0224 - accuracy: 0.9931
 602/1688 [=========>....................] - ETA: 4s - loss: 0.0223 - accuracy: 0.9931
 613/1688 [=========>....................] - ETA: 4s - loss: 0.0222 - accuracy: 0.9932
 624/1688 [==========>...................] - ETA: 4s - loss: 0.0222 - accuracy: 0.9931
 635/1688 [==========>...................] - ETA: 4s - loss: 0.0223 - accuracy: 0.9930
 647/1688 [==========>...................] - ETA: 4s - loss: 0.0221 - accuracy: 0.9931
 659/1688 [==========>...................] - ETA: 4s - loss: 0.0220 - accuracy: 0.9931
 671/1688 [==========>...................] - ETA: 4s - loss: 0.0218 - accuracy: 0.9932
 682/1688 [===========>..................] - ETA: 4s - loss: 0.0216 - accuracy: 0.9933
 693/1688 [===========>..................] - ETA: 4s - loss: 0.0215 - accuracy: 0.9934
 704/1688 [===========>..................] - ETA: 4s - loss: 0.0214 - accuracy: 0.9934
 716/1688 [===========>..................] - ETA: 4s - loss: 0.0214 - accuracy: 0.9934
 728/1688 [===========>..................] - ETA: 4s - loss: 0.0215 - accuracy: 0.9933
 739/1688 [============>.................] - ETA: 4s - loss: 0.0214 - accuracy: 0.9934
 750/1688 [============>.................] - ETA: 4s - loss: 0.0213 - accuracy: 0.9935
 761/1688 [============>.................] - ETA: 4s - loss: 0.0212 - accuracy: 0.9935
 772/1688 [============>.................] - ETA: 4s - loss: 0.0210 - accuracy: 0.9936
 783/1688 [============>.................] - ETA: 4s - loss: 0.0208 - accuracy: 0.9937
 795/1688 [=============>................] - ETA: 4s - loss: 0.0207 - accuracy: 0.9937
 807/1688 [=============>................] - ETA: 4s - loss: 0.0209 - accuracy: 0.9936
 819/1688 [=============>................] - ETA: 3s - loss: 0.0209 - accuracy: 0.9937
 830/1688 [=============>................] - ETA: 3s - loss: 0.0208 - accuracy: 0.9937
 842/1688 [=============>................] - ETA: 3s - loss: 0.0207 - accuracy: 0.9938
 853/1688 [==============>...............] - ETA: 3s - loss: 0.0207 - accuracy: 0.9938
 865/1688 [==============>...............] - ETA: 3s - loss: 0.0206 - accuracy: 0.9939
 877/1688 [==============>...............] - ETA: 3s - loss: 0.0204 - accuracy: 0.9939
 888/1688 [==============>...............] - ETA: 3s - loss: 0.0205 - accuracy: 0.9939
 899/1688 [==============>...............] - ETA: 3s - loss: 0.0205 - accuracy: 0.9939
 910/1688 [===============>..............] - ETA: 3s - loss: 0.0205 - accuracy: 0.9939
 921/1688 [===============>..............] - ETA: 3s - loss: 0.0205 - accuracy: 0.9939
 932/1688 [===============>..............] - ETA: 3s - loss: 0.0204 - accuracy: 0.9939
 944/1688 [===============>..............] - ETA: 3s - loss: 0.0204 - accuracy: 0.9940
 956/1688 [===============>..............] - ETA: 3s - loss: 0.0203 - accuracy: 0.9940
 968/1688 [================>.............] - ETA: 3s - loss: 0.0203 - accuracy: 0.9940
 979/1688 [================>.............] - ETA: 3s - loss: 0.0203 - accuracy: 0.9940
 991/1688 [================>.............] - ETA: 3s - loss: 0.0202 - accuracy: 0.9941
1002/1688 [================>.............] - ETA: 3s - loss: 0.0202 - accuracy: 0.9940
1014/1688 [=================>............] - ETA: 3s - loss: 0.0201 - accuracy: 0.9941
1026/1688 [=================>............] - ETA: 3s - loss: 0.0202 - accuracy: 0.9941
1037/1688 [=================>............] - ETA: 2s - loss: 0.0202 - accuracy: 0.9941
1049/1688 [=================>............] - ETA: 2s - loss: 0.0201 - accuracy: 0.9942
1060/1688 [=================>............] - ETA: 2s - loss: 0.0202 - accuracy: 0.9941
1071/1688 [==================>...........] - ETA: 2s - loss: 0.0201 - accuracy: 0.9941
1082/1688 [==================>...........] - ETA: 2s - loss: 0.0201 - accuracy: 0.9941
1093/1688 [==================>...........] - ETA: 2s - loss: 0.0200 - accuracy: 0.9942
1104/1688 [==================>...........] - ETA: 2s - loss: 0.0199 - accuracy: 0.9942
1115/1688 [==================>...........] - ETA: 2s - loss: 0.0199 - accuracy: 0.9941
1126/1688 [===================>..........] - ETA: 2s - loss: 0.0199 - accuracy: 0.9941
1137/1688 [===================>..........] - ETA: 2s - loss: 0.0198 - accuracy: 0.9942
1148/1688 [===================>..........] - ETA: 2s - loss: 0.0198 - accuracy: 0.9942
1159/1688 [===================>..........] - ETA: 2s - loss: 0.0197 - accuracy: 0.9943
1170/1688 [===================>..........] - ETA: 2s - loss: 0.0197 - accuracy: 0.9943
1181/1688 [===================>..........] - ETA: 2s - loss: 0.0198 - accuracy: 0.9942
1193/1688 [====================>.........] - ETA: 2s - loss: 0.0197 - accuracy: 0.9943
1204/1688 [====================>.........] - ETA: 2s - loss: 0.0196 - accuracy: 0.9943
1215/1688 [====================>.........] - ETA: 2s - loss: 0.0197 - accuracy: 0.9943
1227/1688 [====================>.........] - ETA: 2s - loss: 0.0195 - accuracy: 0.9943
1239/1688 [=====================>........] - ETA: 2s - loss: 0.0194 - accuracy: 0.9944
1251/1688 [=====================>........] - ETA: 1s - loss: 0.0194 - accuracy: 0.9944
1263/1688 [=====================>........] - ETA: 1s - loss: 0.0194 - accuracy: 0.9944
1275/1688 [=====================>........] - ETA: 1s - loss: 0.0193 - accuracy: 0.9944
1287/1688 [=====================>........] - ETA: 1s - loss: 0.0194 - accuracy: 0.9944
1298/1688 [======================>.......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9944
1310/1688 [======================>.......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9944
1321/1688 [======================>.......] - ETA: 1s - loss: 0.0192 - accuracy: 0.9944
1333/1688 [======================>.......] - ETA: 1s - loss: 0.0191 - accuracy: 0.9944
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0191 - accuracy: 0.9944
1355/1688 [=======================>......] - ETA: 1s - loss: 0.0191 - accuracy: 0.9944
1366/1688 [=======================>......] - ETA: 1s - loss: 0.0190 - accuracy: 0.9944
1377/1688 [=======================>......] - ETA: 1s - loss: 0.0190 - accuracy: 0.9945
1388/1688 [=======================>......] - ETA: 1s - loss: 0.0190 - accuracy: 0.9945
1400/1688 [=======================>......] - ETA: 1s - loss: 0.0189 - accuracy: 0.9945
1411/1688 [========================>.....] - ETA: 1s - loss: 0.0189 - accuracy: 0.9945
1422/1688 [========================>.....] - ETA: 1s - loss: 0.0189 - accuracy: 0.9945
1433/1688 [========================>.....] - ETA: 1s - loss: 0.0189 - accuracy: 0.9945
1444/1688 [========================>.....] - ETA: 1s - loss: 0.0189 - accuracy: 0.9945
1455/1688 [========================>.....] - ETA: 1s - loss: 0.0190 - accuracy: 0.9945
1466/1688 [=========================>....] - ETA: 1s - loss: 0.0189 - accuracy: 0.9945
1477/1688 [=========================>....] - ETA: 0s - loss: 0.0189 - accuracy: 0.9945
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0188 - accuracy: 0.9946
1500/1688 [=========================>....] - ETA: 0s - loss: 0.0187 - accuracy: 0.9946
1511/1688 [=========================>....] - ETA: 0s - loss: 0.0186 - accuracy: 0.9946
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0186 - accuracy: 0.9947
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0185 - accuracy: 0.9947
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0185 - accuracy: 0.9947
1557/1688 [==========================>...] - ETA: 0s - loss: 0.0184 - accuracy: 0.9947
1568/1688 [==========================>...] - ETA: 0s - loss: 0.0184 - accuracy: 0.9948
1580/1688 [===========================>..] - ETA: 0s - loss: 0.0183 - accuracy: 0.9948
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0183 - accuracy: 0.9948
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0182 - accuracy: 0.9948
1613/1688 [===========================>..] - ETA: 0s - loss: 0.0182 - accuracy: 0.9948
1625/1688 [===========================>..] - ETA: 0s - loss: 0.0182 - accuracy: 0.9948
1636/1688 [============================>.] - ETA: 0s - loss: 0.0182 - accuracy: 0.9948
1647/1688 [============================>.] - ETA: 0s - loss: 0.0181 - accuracy: 0.9948
1658/1688 [============================>.] - ETA: 0s - loss: 0.0181 - accuracy: 0.9949
1669/1688 [============================>.] - ETA: 0s - loss: 0.0181 - accuracy: 0.9949
1681/1688 [============================>.] - ETA: 0s - loss: 0.0181 - accuracy: 0.9949
1688/1688 [==============================] - 11s 5ms/step - loss: 0.0180 - accuracy: 0.9949 - val_loss: 0.0483 - val_accuracy: 0.9853
Epoch 2/5

   1/1688 [..............................] - ETA: 7s - loss: 0.0040 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0150 - accuracy: 0.9928
  24/1688 [..............................] - ETA: 7s - loss: 0.0109 - accuracy: 0.9961
  36/1688 [..............................] - ETA: 7s - loss: 0.0129 - accuracy: 0.9965
  47/1688 [..............................] - ETA: 7s - loss: 0.0140 - accuracy: 0.9953
  58/1688 [>.............................] - ETA: 7s - loss: 0.0125 - accuracy: 0.9962
  70/1688 [>.............................] - ETA: 7s - loss: 0.0115 - accuracy: 0.9969
  81/1688 [>.............................] - ETA: 7s - loss: 0.0108 - accuracy: 0.9973
  92/1688 [>.............................] - ETA: 7s - loss: 0.0112 - accuracy: 0.9969
 104/1688 [>.............................] - ETA: 7s - loss: 0.0114 - accuracy: 0.9970
 115/1688 [=>............................] - ETA: 7s - loss: 0.0112 - accuracy: 0.9973
 127/1688 [=>............................] - ETA: 7s - loss: 0.0116 - accuracy: 0.9973
 138/1688 [=>............................] - ETA: 7s - loss: 0.0116 - accuracy: 0.9973
 150/1688 [=>............................] - ETA: 7s - loss: 0.0114 - accuracy: 0.9973
 161/1688 [=>............................] - ETA: 6s - loss: 0.0114 - accuracy: 0.9973
 173/1688 [==>...........................] - ETA: 6s - loss: 0.0111 - accuracy: 0.9973
 184/1688 [==>...........................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9969
 195/1688 [==>...........................] - ETA: 6s - loss: 0.0118 - accuracy: 0.9971
 206/1688 [==>...........................] - ETA: 6s - loss: 0.0123 - accuracy: 0.9971
 217/1688 [==>...........................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9973
 228/1688 [===>..........................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9973
 240/1688 [===>..........................] - ETA: 6s - loss: 0.0122 - accuracy: 0.9973
 251/1688 [===>..........................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9973
 263/1688 [===>..........................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9971
 274/1688 [===>..........................] - ETA: 6s - loss: 0.0124 - accuracy: 0.9973
 285/1688 [====>.........................] - ETA: 6s - loss: 0.0126 - accuracy: 0.9971
 296/1688 [====>.........................] - ETA: 6s - loss: 0.0129 - accuracy: 0.9968
 307/1688 [====>.........................] - ETA: 6s - loss: 0.0129 - accuracy: 0.9968
 318/1688 [====>.........................] - ETA: 6s - loss: 0.0130 - accuracy: 0.9968
 329/1688 [====>.........................] - ETA: 6s - loss: 0.0130 - accuracy: 0.9968
 340/1688 [=====>........................] - ETA: 6s - loss: 0.0131 - accuracy: 0.9968
 352/1688 [=====>........................] - ETA: 6s - loss: 0.0136 - accuracy: 0.9966
 363/1688 [=====>........................] - ETA: 6s - loss: 0.0134 - accuracy: 0.9967
 374/1688 [=====>........................] - ETA: 6s - loss: 0.0134 - accuracy: 0.9967
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0133 - accuracy: 0.9967
 396/1688 [======>.......................] - ETA: 5s - loss: 0.0134 - accuracy: 0.9965
 407/1688 [======>.......................] - ETA: 5s - loss: 0.0133 - accuracy: 0.9965
 418/1688 [======>.......................] - ETA: 5s - loss: 0.0133 - accuracy: 0.9966
 430/1688 [======>.......................] - ETA: 5s - loss: 0.0133 - accuracy: 0.9966
 441/1688 [======>.......................] - ETA: 5s - loss: 0.0131 - accuracy: 0.9967
 452/1688 [=======>......................] - ETA: 5s - loss: 0.0130 - accuracy: 0.9967
 464/1688 [=======>......................] - ETA: 5s - loss: 0.0128 - accuracy: 0.9968
 475/1688 [=======>......................] - ETA: 5s - loss: 0.0130 - accuracy: 0.9966
 486/1688 [=======>......................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9965
 497/1688 [=======>......................] - ETA: 5s - loss: 0.0131 - accuracy: 0.9966
 509/1688 [========>.....................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9966
 520/1688 [========>.....................] - ETA: 5s - loss: 0.0132 - accuracy: 0.9965
 531/1688 [========>.....................] - ETA: 5s - loss: 0.0131 - accuracy: 0.9966
 542/1688 [========>.....................] - ETA: 5s - loss: 0.0130 - accuracy: 0.9967
 553/1688 [========>.....................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9967
 565/1688 [=========>....................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9967
 577/1688 [=========>....................] - ETA: 5s - loss: 0.0129 - accuracy: 0.9968
 588/1688 [=========>....................] - ETA: 5s - loss: 0.0128 - accuracy: 0.9968
 599/1688 [=========>....................] - ETA: 4s - loss: 0.0130 - accuracy: 0.9966
 611/1688 [=========>....................] - ETA: 4s - loss: 0.0129 - accuracy: 0.9966
 622/1688 [==========>...................] - ETA: 4s - loss: 0.0129 - accuracy: 0.9966
 633/1688 [==========>...................] - ETA: 4s - loss: 0.0128 - accuracy: 0.9966
 644/1688 [==========>...................] - ETA: 4s - loss: 0.0128 - accuracy: 0.9966
 655/1688 [==========>...................] - ETA: 4s - loss: 0.0127 - accuracy: 0.9966
 666/1688 [==========>...................] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
 677/1688 [===========>..................] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
 688/1688 [===========>..................] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
 700/1688 [===========>..................] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
 712/1688 [===========>..................] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
 724/1688 [===========>..................] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
 736/1688 [============>.................] - ETA: 4s - loss: 0.0132 - accuracy: 0.9964
 747/1688 [============>.................] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
 759/1688 [============>.................] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
 770/1688 [============>.................] - ETA: 4s - loss: 0.0131 - accuracy: 0.9964
 781/1688 [============>.................] - ETA: 4s - loss: 0.0130 - accuracy: 0.9965
 792/1688 [=============>................] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
 803/1688 [=============>................] - ETA: 4s - loss: 0.0131 - accuracy: 0.9965
 814/1688 [=============>................] - ETA: 3s - loss: 0.0131 - accuracy: 0.9965
 825/1688 [=============>................] - ETA: 3s - loss: 0.0131 - accuracy: 0.9965
 836/1688 [=============>................] - ETA: 3s - loss: 0.0132 - accuracy: 0.9963
 848/1688 [==============>...............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9962
 859/1688 [==============>...............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9961
 870/1688 [==============>...............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9962
 881/1688 [==============>...............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9962
 892/1688 [==============>...............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9962
 903/1688 [===============>..............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9962
 914/1688 [===============>..............] - ETA: 3s - loss: 0.0133 - accuracy: 0.9962
 926/1688 [===============>..............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9962
 937/1688 [===============>..............] - ETA: 3s - loss: 0.0133 - accuracy: 0.9962
 948/1688 [===============>..............] - ETA: 3s - loss: 0.0134 - accuracy: 0.9962
 959/1688 [================>.............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9961
 970/1688 [================>.............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9961
 981/1688 [================>.............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9962
 992/1688 [================>.............] - ETA: 3s - loss: 0.0135 - accuracy: 0.9962
1004/1688 [================>.............] - ETA: 3s - loss: 0.0136 - accuracy: 0.9961
1016/1688 [=================>............] - ETA: 3s - loss: 0.0137 - accuracy: 0.9961
1027/1688 [=================>............] - ETA: 3s - loss: 0.0137 - accuracy: 0.9960
1038/1688 [=================>............] - ETA: 2s - loss: 0.0138 - accuracy: 0.9960
1049/1688 [=================>............] - ETA: 2s - loss: 0.0138 - accuracy: 0.9959
1060/1688 [=================>............] - ETA: 2s - loss: 0.0137 - accuracy: 0.9959
1071/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9960
1083/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9960
1094/1688 [==================>...........] - ETA: 2s - loss: 0.0136 - accuracy: 0.9960
1106/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9960
1117/1688 [==================>...........] - ETA: 2s - loss: 0.0137 - accuracy: 0.9960
1128/1688 [===================>..........] - ETA: 2s - loss: 0.0136 - accuracy: 0.9960
1139/1688 [===================>..........] - ETA: 2s - loss: 0.0136 - accuracy: 0.9960
1150/1688 [===================>..........] - ETA: 2s - loss: 0.0136 - accuracy: 0.9961
1161/1688 [===================>..........] - ETA: 2s - loss: 0.0136 - accuracy: 0.9960
1173/1688 [===================>..........] - ETA: 2s - loss: 0.0136 - accuracy: 0.9961
1184/1688 [====================>.........] - ETA: 2s - loss: 0.0135 - accuracy: 0.9961
1195/1688 [====================>.........] - ETA: 2s - loss: 0.0136 - accuracy: 0.9961
1206/1688 [====================>.........] - ETA: 2s - loss: 0.0136 - accuracy: 0.9961
1217/1688 [====================>.........] - ETA: 2s - loss: 0.0135 - accuracy: 0.9961
1228/1688 [====================>.........] - ETA: 2s - loss: 0.0135 - accuracy: 0.9961
1239/1688 [=====================>........] - ETA: 2s - loss: 0.0135 - accuracy: 0.9961
1250/1688 [=====================>........] - ETA: 2s - loss: 0.0135 - accuracy: 0.9961
1261/1688 [=====================>........] - ETA: 1s - loss: 0.0135 - accuracy: 0.9961
1272/1688 [=====================>........] - ETA: 1s - loss: 0.0135 - accuracy: 0.9961
1283/1688 [=====================>........] - ETA: 1s - loss: 0.0134 - accuracy: 0.9962
1294/1688 [=====================>........] - ETA: 1s - loss: 0.0134 - accuracy: 0.9962
1305/1688 [======================>.......] - ETA: 1s - loss: 0.0134 - accuracy: 0.9961
1316/1688 [======================>.......] - ETA: 1s - loss: 0.0134 - accuracy: 0.9962
1327/1688 [======================>.......] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1339/1688 [======================>.......] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1351/1688 [=======================>......] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1362/1688 [=======================>......] - ETA: 1s - loss: 0.0134 - accuracy: 0.9962
1374/1688 [=======================>......] - ETA: 1s - loss: 0.0134 - accuracy: 0.9962
1385/1688 [=======================>......] - ETA: 1s - loss: 0.0134 - accuracy: 0.9962
1397/1688 [=======================>......] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1409/1688 [========================>.....] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1421/1688 [========================>.....] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1432/1688 [========================>.....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9962
1443/1688 [========================>.....] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1454/1688 [========================>.....] - ETA: 1s - loss: 0.0133 - accuracy: 0.9962
1465/1688 [=========================>....] - ETA: 1s - loss: 0.0134 - accuracy: 0.9962
1476/1688 [=========================>....] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1488/1688 [=========================>....] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1499/1688 [=========================>....] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1510/1688 [=========================>....] - ETA: 0s - loss: 0.0132 - accuracy: 0.9962
1522/1688 [==========================>...] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1533/1688 [==========================>...] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1544/1688 [==========================>...] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1555/1688 [==========================>...] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1566/1688 [==========================>...] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1577/1688 [===========================>..] - ETA: 0s - loss: 0.0132 - accuracy: 0.9962
1588/1688 [===========================>..] - ETA: 0s - loss: 0.0133 - accuracy: 0.9962
1599/1688 [===========================>..] - ETA: 0s - loss: 0.0132 - accuracy: 0.9962
1610/1688 [===========================>..] - ETA: 0s - loss: 0.0132 - accuracy: 0.9962
1621/1688 [===========================>..] - ETA: 0s - loss: 0.0132 - accuracy: 0.9963
1632/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9963
1643/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9963
1655/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9963
1666/1688 [============================>.] - ETA: 0s - loss: 0.0131 - accuracy: 0.9963
1678/1688 [============================>.] - ETA: 0s - loss: 0.0132 - accuracy: 0.9963
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0132 - accuracy: 0.9962 - val_loss: 0.0455 - val_accuracy: 0.9862
Epoch 3/5

   1/1688 [..............................] - ETA: 8s - loss: 0.0219 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0178 - accuracy: 0.9952
  25/1688 [..............................] - ETA: 7s - loss: 0.0110 - accuracy: 0.9975
  37/1688 [..............................] - ETA: 7s - loss: 0.0100 - accuracy: 0.9983
  48/1688 [..............................] - ETA: 7s - loss: 0.0093 - accuracy: 0.9987
  59/1688 [>.............................] - ETA: 7s - loss: 0.0085 - accuracy: 0.9989
  70/1688 [>.............................] - ETA: 7s - loss: 0.0083 - accuracy: 0.9991
  81/1688 [>.............................] - ETA: 7s - loss: 0.0080 - accuracy: 0.9992
  92/1688 [>.............................] - ETA: 7s - loss: 0.0076 - accuracy: 0.9993
 103/1688 [>.............................] - ETA: 7s - loss: 0.0077 - accuracy: 0.9994
 114/1688 [=>............................] - ETA: 7s - loss: 0.0076 - accuracy: 0.9995
 125/1688 [=>............................] - ETA: 7s - loss: 0.0077 - accuracy: 0.9995
 136/1688 [=>............................] - ETA: 7s - loss: 0.0089 - accuracy: 0.9991
 147/1688 [=>............................] - ETA: 7s - loss: 0.0099 - accuracy: 0.9987
 158/1688 [=>............................] - ETA: 7s - loss: 0.0098 - accuracy: 0.9986
 170/1688 [==>...........................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9983
 182/1688 [==>...........................] - ETA: 6s - loss: 0.0102 - accuracy: 0.9985
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9984
 204/1688 [==>...........................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9983
 215/1688 [==>...........................] - ETA: 6s - loss: 0.0105 - accuracy: 0.9983
 226/1688 [===>..........................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9983
 237/1688 [===>..........................] - ETA: 6s - loss: 0.0103 - accuracy: 0.9983
 248/1688 [===>..........................] - ETA: 6s - loss: 0.0103 - accuracy: 0.9982
 259/1688 [===>..........................] - ETA: 6s - loss: 0.0102 - accuracy: 0.9982
 271/1688 [===>..........................] - ETA: 6s - loss: 0.0100 - accuracy: 0.9983
 282/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9983
 294/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9984
 306/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9984
 317/1688 [====>.........................] - ETA: 6s - loss: 0.0100 - accuracy: 0.9983
 328/1688 [====>.........................] - ETA: 6s - loss: 0.0104 - accuracy: 0.9982
 339/1688 [=====>........................] - ETA: 6s - loss: 0.0107 - accuracy: 0.9981
 350/1688 [=====>........................] - ETA: 6s - loss: 0.0110 - accuracy: 0.9979
 362/1688 [=====>........................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9980
 374/1688 [=====>........................] - ETA: 6s - loss: 0.0108 - accuracy: 0.9981
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0106 - accuracy: 0.9981
 396/1688 [======>.......................] - ETA: 5s - loss: 0.0106 - accuracy: 0.9981
 407/1688 [======>.......................] - ETA: 5s - loss: 0.0106 - accuracy: 0.9981
 418/1688 [======>.......................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9981
 429/1688 [======>.......................] - ETA: 5s - loss: 0.0106 - accuracy: 0.9980
 440/1688 [======>.......................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9980
 451/1688 [=======>......................] - ETA: 5s - loss: 0.0106 - accuracy: 0.9980
 462/1688 [=======>......................] - ETA: 5s - loss: 0.0107 - accuracy: 0.9979
 473/1688 [=======>......................] - ETA: 5s - loss: 0.0106 - accuracy: 0.9980
 485/1688 [=======>......................] - ETA: 5s - loss: 0.0107 - accuracy: 0.9979
 495/1688 [=======>......................] - ETA: 5s - loss: 0.0110 - accuracy: 0.9977
 507/1688 [========>.....................] - ETA: 5s - loss: 0.0112 - accuracy: 0.9976
 519/1688 [========>.....................] - ETA: 5s - loss: 0.0117 - accuracy: 0.9974
 530/1688 [========>.....................] - ETA: 5s - loss: 0.0118 - accuracy: 0.9974
 541/1688 [========>.....................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9974
 552/1688 [========>.....................] - ETA: 5s - loss: 0.0118 - accuracy: 0.9975
 564/1688 [=========>....................] - ETA: 5s - loss: 0.0119 - accuracy: 0.9975
 575/1688 [=========>....................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9973
 586/1688 [=========>....................] - ETA: 5s - loss: 0.0121 - accuracy: 0.9974
 597/1688 [=========>....................] - ETA: 4s - loss: 0.0121 - accuracy: 0.9974
 608/1688 [=========>....................] - ETA: 4s - loss: 0.0121 - accuracy: 0.9974
 619/1688 [==========>...................] - ETA: 4s - loss: 0.0121 - accuracy: 0.9974
 630/1688 [==========>...................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9974
 642/1688 [==========>...................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9975
 653/1688 [==========>...................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9975
 664/1688 [==========>...................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9975
 675/1688 [==========>...................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9975
 686/1688 [===========>..................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9974
 698/1688 [===========>..................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9974
 710/1688 [===========>..................] - ETA: 4s - loss: 0.0120 - accuracy: 0.9973
 721/1688 [===========>..................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9973
 732/1688 [============>.................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9973
 744/1688 [============>.................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9974
 755/1688 [============>.................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9974
 767/1688 [============>.................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9974
 778/1688 [============>.................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9973
 789/1688 [=============>................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9973
 800/1688 [=============>................] - ETA: 4s - loss: 0.0119 - accuracy: 0.9973
 811/1688 [=============>................] - ETA: 4s - loss: 0.0118 - accuracy: 0.9973
 822/1688 [=============>................] - ETA: 3s - loss: 0.0119 - accuracy: 0.9972
 833/1688 [=============>................] - ETA: 3s - loss: 0.0119 - accuracy: 0.9973
 845/1688 [==============>...............] - ETA: 3s - loss: 0.0118 - accuracy: 0.9973
 856/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9973
 868/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9973
 879/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9973
 891/1688 [==============>...............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9973
 903/1688 [===============>..............] - ETA: 3s - loss: 0.0117 - accuracy: 0.9974
 914/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9974
 926/1688 [===============>..............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9974
 938/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9974
 949/1688 [===============>..............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9974
 961/1688 [================>.............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9974
 972/1688 [================>.............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9974
 983/1688 [================>.............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9974
 995/1688 [================>.............] - ETA: 3s - loss: 0.0116 - accuracy: 0.9974
1006/1688 [================>.............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9974
1017/1688 [=================>............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9974
1028/1688 [=================>............] - ETA: 3s - loss: 0.0115 - accuracy: 0.9974
1039/1688 [=================>............] - ETA: 2s - loss: 0.0115 - accuracy: 0.9975
1050/1688 [=================>............] - ETA: 2s - loss: 0.0115 - accuracy: 0.9974
1061/1688 [=================>............] - ETA: 2s - loss: 0.0116 - accuracy: 0.9974
1072/1688 [==================>...........] - ETA: 2s - loss: 0.0116 - accuracy: 0.9974
1084/1688 [==================>...........] - ETA: 2s - loss: 0.0116 - accuracy: 0.9974
1096/1688 [==================>...........] - ETA: 2s - loss: 0.0115 - accuracy: 0.9974
1107/1688 [==================>...........] - ETA: 2s - loss: 0.0115 - accuracy: 0.9974
1118/1688 [==================>...........] - ETA: 2s - loss: 0.0115 - accuracy: 0.9974
1130/1688 [===================>..........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9975
1142/1688 [===================>..........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9975
1153/1688 [===================>..........] - ETA: 2s - loss: 0.0115 - accuracy: 0.9975
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9975
1176/1688 [===================>..........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9975
1187/1688 [====================>.........] - ETA: 2s - loss: 0.0113 - accuracy: 0.9975
1199/1688 [====================>.........] - ETA: 2s - loss: 0.0114 - accuracy: 0.9975
1210/1688 [====================>.........] - ETA: 2s - loss: 0.0113 - accuracy: 0.9975
1222/1688 [====================>.........] - ETA: 2s - loss: 0.0113 - accuracy: 0.9975
1233/1688 [====================>.........] - ETA: 2s - loss: 0.0112 - accuracy: 0.9975
1244/1688 [=====================>........] - ETA: 2s - loss: 0.0112 - accuracy: 0.9975
1255/1688 [=====================>........] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1266/1688 [=====================>........] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1277/1688 [=====================>........] - ETA: 1s - loss: 0.0113 - accuracy: 0.9975
1289/1688 [=====================>........] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1300/1688 [======================>.......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1311/1688 [======================>.......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1322/1688 [======================>.......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1333/1688 [======================>.......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1345/1688 [======================>.......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9974
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9974
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9974
1378/1688 [=======================>......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1389/1688 [=======================>......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1412/1688 [========================>.....] - ETA: 1s - loss: 0.0112 - accuracy: 0.9975
1424/1688 [========================>.....] - ETA: 1s - loss: 0.0112 - accuracy: 0.9974
1435/1688 [========================>.....] - ETA: 1s - loss: 0.0112 - accuracy: 0.9974
1447/1688 [========================>.....] - ETA: 1s - loss: 0.0112 - accuracy: 0.9974
1458/1688 [========================>.....] - ETA: 1s - loss: 0.0112 - accuracy: 0.9974
1470/1688 [=========================>....] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1482/1688 [=========================>....] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1493/1688 [=========================>....] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1504/1688 [=========================>....] - ETA: 0s - loss: 0.0111 - accuracy: 0.9974
1516/1688 [=========================>....] - ETA: 0s - loss: 0.0113 - accuracy: 0.9974
1527/1688 [==========================>...] - ETA: 0s - loss: 0.0112 - accuracy: 0.9973
1538/1688 [==========================>...] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1549/1688 [==========================>...] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1561/1688 [==========================>...] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1572/1688 [==========================>...] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1584/1688 [===========================>..] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1596/1688 [===========================>..] - ETA: 0s - loss: 0.0112 - accuracy: 0.9974
1607/1688 [===========================>..] - ETA: 0s - loss: 0.0113 - accuracy: 0.9973
1618/1688 [===========================>..] - ETA: 0s - loss: 0.0113 - accuracy: 0.9973
1629/1688 [===========================>..] - ETA: 0s - loss: 0.0113 - accuracy: 0.9973
1641/1688 [============================>.] - ETA: 0s - loss: 0.0113 - accuracy: 0.9973
1653/1688 [============================>.] - ETA: 0s - loss: 0.0113 - accuracy: 0.9973
1664/1688 [============================>.] - ETA: 0s - loss: 0.0113 - accuracy: 0.9973
1675/1688 [============================>.] - ETA: 0s - loss: 0.0113 - accuracy: 0.9973
1686/1688 [============================>.] - ETA: 0s - loss: 0.0113 - accuracy: 0.9973
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0113 - accuracy: 0.9973 - val_loss: 0.0432 - val_accuracy: 0.9862
Epoch 4/5

   1/1688 [..............................] - ETA: 8s - loss: 0.0076 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0061 - accuracy: 1.0000
  24/1688 [..............................] - ETA: 7s - loss: 0.0119 - accuracy: 0.9974
  35/1688 [..............................] - ETA: 7s - loss: 0.0114 - accuracy: 0.9973
  46/1688 [..............................] - ETA: 7s - loss: 0.0110 - accuracy: 0.9973
  57/1688 [>.............................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9967
  68/1688 [>.............................] - ETA: 7s - loss: 0.0125 - accuracy: 0.9968
  79/1688 [>.............................] - ETA: 7s - loss: 0.0123 - accuracy: 0.9964
  90/1688 [>.............................] - ETA: 7s - loss: 0.0120 - accuracy: 0.9969
 101/1688 [>.............................] - ETA: 7s - loss: 0.0119 - accuracy: 0.9969
 112/1688 [>.............................] - ETA: 7s - loss: 0.0128 - accuracy: 0.9967
 123/1688 [=>............................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9970
 135/1688 [=>............................] - ETA: 7s - loss: 0.0124 - accuracy: 0.9968
 146/1688 [=>............................] - ETA: 7s - loss: 0.0122 - accuracy: 0.9968
 158/1688 [=>............................] - ETA: 7s - loss: 0.0119 - accuracy: 0.9970
 169/1688 [==>...........................] - ETA: 6s - loss: 0.0117 - accuracy: 0.9972
 181/1688 [==>...........................] - ETA: 6s - loss: 0.0116 - accuracy: 0.9974
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0112 - accuracy: 0.9974
 205/1688 [==>...........................] - ETA: 6s - loss: 0.0111 - accuracy: 0.9974
 216/1688 [==>...........................] - ETA: 6s - loss: 0.0115 - accuracy: 0.9971
 227/1688 [===>..........................] - ETA: 6s - loss: 0.0116 - accuracy: 0.9971
 238/1688 [===>..........................] - ETA: 6s - loss: 0.0112 - accuracy: 0.9972
 249/1688 [===>..........................] - ETA: 6s - loss: 0.0114 - accuracy: 0.9971
 261/1688 [===>..........................] - ETA: 6s - loss: 0.0112 - accuracy: 0.9971
 272/1688 [===>..........................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9972
 283/1688 [====>.........................] - ETA: 6s - loss: 0.0110 - accuracy: 0.9972
 295/1688 [====>.........................] - ETA: 6s - loss: 0.0108 - accuracy: 0.9974
 306/1688 [====>.........................] - ETA: 6s - loss: 0.0107 - accuracy: 0.9974
 317/1688 [====>.........................] - ETA: 6s - loss: 0.0105 - accuracy: 0.9975
 328/1688 [====>.........................] - ETA: 6s - loss: 0.0105 - accuracy: 0.9976
 339/1688 [=====>........................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9976
 351/1688 [=====>........................] - ETA: 6s - loss: 0.0109 - accuracy: 0.9976
 362/1688 [=====>........................] - ETA: 6s - loss: 0.0107 - accuracy: 0.9977
 373/1688 [=====>........................] - ETA: 6s - loss: 0.0107 - accuracy: 0.9977
 385/1688 [=====>........................] - ETA: 5s - loss: 0.0105 - accuracy: 0.9977
 396/1688 [======>.......................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9978
 408/1688 [======>.......................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9978
 419/1688 [======>.......................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9978
 431/1688 [======>.......................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9978
 443/1688 [======>.......................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9977
 454/1688 [=======>......................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9977
 466/1688 [=======>......................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9977
 477/1688 [=======>......................] - ETA: 5s - loss: 0.0104 - accuracy: 0.9976
 488/1688 [=======>......................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9976
 499/1688 [=======>......................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9976
 511/1688 [========>.....................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9975
 522/1688 [========>.....................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9975
 533/1688 [========>.....................] - ETA: 5s - loss: 0.0103 - accuracy: 0.9975
 544/1688 [========>.....................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9976
 555/1688 [========>.....................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9976
 566/1688 [=========>....................] - ETA: 5s - loss: 0.0102 - accuracy: 0.9976
 578/1688 [=========>....................] - ETA: 5s - loss: 0.0101 - accuracy: 0.9976
 589/1688 [=========>....................] - ETA: 5s - loss: 0.0101 - accuracy: 0.9977
 600/1688 [=========>....................] - ETA: 4s - loss: 0.0100 - accuracy: 0.9977
 611/1688 [=========>....................] - ETA: 4s - loss: 0.0100 - accuracy: 0.9976
 622/1688 [==========>...................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 634/1688 [==========>...................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 645/1688 [==========>...................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9977
 656/1688 [==========>...................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 667/1688 [==========>...................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 679/1688 [===========>..................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 690/1688 [===========>..................] - ETA: 4s - loss: 0.0103 - accuracy: 0.9976
 702/1688 [===========>..................] - ETA: 4s - loss: 0.0103 - accuracy: 0.9976
 713/1688 [===========>..................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 724/1688 [===========>..................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9977
 735/1688 [============>.................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9977
 746/1688 [============>.................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 757/1688 [============>.................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 769/1688 [============>.................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9976
 780/1688 [============>.................] - ETA: 4s - loss: 0.0102 - accuracy: 0.9976
 792/1688 [=============>................] - ETA: 4s - loss: 0.0101 - accuracy: 0.9976
 803/1688 [=============>................] - ETA: 4s - loss: 0.0100 - accuracy: 0.9976
 814/1688 [=============>................] - ETA: 4s - loss: 0.0100 - accuracy: 0.9976
 826/1688 [=============>................] - ETA: 3s - loss: 0.0099 - accuracy: 0.9976
 837/1688 [=============>................] - ETA: 3s - loss: 0.0100 - accuracy: 0.9976
 848/1688 [==============>...............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9976
 859/1688 [==============>...............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9976
 870/1688 [==============>...............] - ETA: 3s - loss: 0.0100 - accuracy: 0.9976
 881/1688 [==============>...............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9976
 892/1688 [==============>...............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9977
 904/1688 [===============>..............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9976
 916/1688 [===============>..............] - ETA: 3s - loss: 0.0100 - accuracy: 0.9976
 927/1688 [===============>..............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9977
 938/1688 [===============>..............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9977
 949/1688 [===============>..............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9977
 960/1688 [================>.............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9977
 971/1688 [================>.............] - ETA: 3s - loss: 0.0099 - accuracy: 0.9977
 982/1688 [================>.............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9978
 993/1688 [================>.............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9978
1005/1688 [================>.............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9978
1016/1688 [=================>............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9978
1027/1688 [=================>............] - ETA: 3s - loss: 0.0098 - accuracy: 0.9978
1039/1688 [=================>............] - ETA: 2s - loss: 0.0098 - accuracy: 0.9978
1051/1688 [=================>............] - ETA: 2s - loss: 0.0098 - accuracy: 0.9979
1062/1688 [=================>............] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1073/1688 [==================>...........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1084/1688 [==================>...........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1095/1688 [==================>...........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1107/1688 [==================>...........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1119/1688 [==================>...........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1130/1688 [===================>..........] - ETA: 2s - loss: 0.0098 - accuracy: 0.9978
1141/1688 [===================>..........] - ETA: 2s - loss: 0.0099 - accuracy: 0.9978
1152/1688 [===================>..........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9977
1163/1688 [===================>..........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9977
1175/1688 [===================>..........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9977
1186/1688 [====================>.........] - ETA: 2s - loss: 0.0100 - accuracy: 0.9977
1197/1688 [====================>.........] - ETA: 2s - loss: 0.0101 - accuracy: 0.9977
1209/1688 [====================>.........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9977
1220/1688 [====================>.........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9977
1231/1688 [====================>.........] - ETA: 2s - loss: 0.0102 - accuracy: 0.9977
1242/1688 [=====================>........] - ETA: 2s - loss: 0.0101 - accuracy: 0.9977
1254/1688 [=====================>........] - ETA: 1s - loss: 0.0101 - accuracy: 0.9977
1265/1688 [=====================>........] - ETA: 1s - loss: 0.0101 - accuracy: 0.9977
1277/1688 [=====================>........] - ETA: 1s - loss: 0.0102 - accuracy: 0.9976
1288/1688 [=====================>........] - ETA: 1s - loss: 0.0102 - accuracy: 0.9976
1300/1688 [======================>.......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1311/1688 [======================>.......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1322/1688 [======================>.......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1333/1688 [======================>.......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9977
1344/1688 [======================>.......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9977
1355/1688 [=======================>......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1366/1688 [=======================>......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1377/1688 [=======================>......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1388/1688 [=======================>......] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1399/1688 [=======================>......] - ETA: 1s - loss: 0.0102 - accuracy: 0.9976
1410/1688 [========================>.....] - ETA: 1s - loss: 0.0102 - accuracy: 0.9976
1421/1688 [========================>.....] - ETA: 1s - loss: 0.0102 - accuracy: 0.9975
1432/1688 [========================>.....] - ETA: 1s - loss: 0.0102 - accuracy: 0.9976
1443/1688 [========================>.....] - ETA: 1s - loss: 0.0102 - accuracy: 0.9976
1455/1688 [========================>.....] - ETA: 1s - loss: 0.0103 - accuracy: 0.9975
1466/1688 [=========================>....] - ETA: 1s - loss: 0.0103 - accuracy: 0.9975
1478/1688 [=========================>....] - ETA: 0s - loss: 0.0103 - accuracy: 0.9975
1489/1688 [=========================>....] - ETA: 0s - loss: 0.0105 - accuracy: 0.9975
1501/1688 [=========================>....] - ETA: 0s - loss: 0.0105 - accuracy: 0.9974
1512/1688 [=========================>....] - ETA: 0s - loss: 0.0105 - accuracy: 0.9974
1523/1688 [==========================>...] - ETA: 0s - loss: 0.0105 - accuracy: 0.9975
1534/1688 [==========================>...] - ETA: 0s - loss: 0.0105 - accuracy: 0.9974
1545/1688 [==========================>...] - ETA: 0s - loss: 0.0106 - accuracy: 0.9974
1556/1688 [==========================>...] - ETA: 0s - loss: 0.0106 - accuracy: 0.9974
1568/1688 [==========================>...] - ETA: 0s - loss: 0.0106 - accuracy: 0.9974
1579/1688 [===========================>..] - ETA: 0s - loss: 0.0106 - accuracy: 0.9974
1591/1688 [===========================>..] - ETA: 0s - loss: 0.0106 - accuracy: 0.9974
1602/1688 [===========================>..] - ETA: 0s - loss: 0.0106 - accuracy: 0.9974
1614/1688 [===========================>..] - ETA: 0s - loss: 0.0105 - accuracy: 0.9974
1625/1688 [===========================>..] - ETA: 0s - loss: 0.0105 - accuracy: 0.9974
1636/1688 [============================>.] - ETA: 0s - loss: 0.0104 - accuracy: 0.9974
1648/1688 [============================>.] - ETA: 0s - loss: 0.0104 - accuracy: 0.9975
1659/1688 [============================>.] - ETA: 0s - loss: 0.0104 - accuracy: 0.9975
1670/1688 [============================>.] - ETA: 0s - loss: 0.0104 - accuracy: 0.9974
1681/1688 [============================>.] - ETA: 0s - loss: 0.0105 - accuracy: 0.9974
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0104 - accuracy: 0.9974 - val_loss: 0.0441 - val_accuracy: 0.9860
Epoch 5/5

   1/1688 [..............................] - ETA: 7s - loss: 6.8218e-04 - accuracy: 1.0000
  13/1688 [..............................] - ETA: 7s - loss: 0.0068 - accuracy: 1.0000    
  24/1688 [..............................] - ETA: 7s - loss: 0.0063 - accuracy: 1.0000
  35/1688 [..............................] - ETA: 7s - loss: 0.0060 - accuracy: 1.0000
  46/1688 [..............................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9993
  57/1688 [>.............................] - ETA: 7s - loss: 0.0087 - accuracy: 0.9984
  68/1688 [>.............................] - ETA: 7s - loss: 0.0078 - accuracy: 0.9986
  79/1688 [>.............................] - ETA: 7s - loss: 0.0079 - accuracy: 0.9984
  90/1688 [>.............................] - ETA: 7s - loss: 0.0094 - accuracy: 0.9983
 101/1688 [>.............................] - ETA: 7s - loss: 0.0087 - accuracy: 0.9985
 112/1688 [>.............................] - ETA: 7s - loss: 0.0097 - accuracy: 0.9980
 123/1688 [=>............................] - ETA: 7s - loss: 0.0098 - accuracy: 0.9980
 134/1688 [=>............................] - ETA: 7s - loss: 0.0098 - accuracy: 0.9979
 146/1688 [=>............................] - ETA: 7s - loss: 0.0097 - accuracy: 0.9981
 158/1688 [=>............................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9978
 170/1688 [==>...........................] - ETA: 6s - loss: 0.0100 - accuracy: 0.9978
 182/1688 [==>...........................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9978
 193/1688 [==>...........................] - ETA: 6s - loss: 0.0103 - accuracy: 0.9977
 204/1688 [==>...........................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9977
 216/1688 [==>...........................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9978
 227/1688 [===>..........................] - ETA: 6s - loss: 0.0100 - accuracy: 0.9979
 238/1688 [===>..........................] - ETA: 6s - loss: 0.0097 - accuracy: 0.9980
 249/1688 [===>..........................] - ETA: 6s - loss: 0.0095 - accuracy: 0.9981
 260/1688 [===>..........................] - ETA: 6s - loss: 0.0096 - accuracy: 0.9981
 271/1688 [===>..........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9980
 282/1688 [====>.........................] - ETA: 6s - loss: 0.0099 - accuracy: 0.9980
 293/1688 [====>.........................] - ETA: 6s - loss: 0.0099 - accuracy: 0.9980
 304/1688 [====>.........................] - ETA: 6s - loss: 0.0097 - accuracy: 0.9979
 315/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9978
 326/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9979
 337/1688 [====>.........................] - ETA: 6s - loss: 0.0098 - accuracy: 0.9980
 348/1688 [=====>........................] - ETA: 6s - loss: 0.0097 - accuracy: 0.9980
 359/1688 [=====>........................] - ETA: 6s - loss: 0.0096 - accuracy: 0.9981
 370/1688 [=====>........................] - ETA: 6s - loss: 0.0095 - accuracy: 0.9981
 382/1688 [=====>........................] - ETA: 5s - loss: 0.0094 - accuracy: 0.9982
 393/1688 [=====>........................] - ETA: 5s - loss: 0.0095 - accuracy: 0.9983
 404/1688 [======>.......................] - ETA: 5s - loss: 0.0094 - accuracy: 0.9982
 416/1688 [======>.......................] - ETA: 5s - loss: 0.0093 - accuracy: 0.9983
 427/1688 [======>.......................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9983
 438/1688 [======>.......................] - ETA: 5s - loss: 0.0093 - accuracy: 0.9983
 449/1688 [======>.......................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9983
 460/1688 [=======>......................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9984
 472/1688 [=======>......................] - ETA: 5s - loss: 0.0091 - accuracy: 0.9984
 483/1688 [=======>......................] - ETA: 5s - loss: 0.0091 - accuracy: 0.9984
 494/1688 [=======>......................] - ETA: 5s - loss: 0.0091 - accuracy: 0.9984
 505/1688 [=======>......................] - ETA: 5s - loss: 0.0090 - accuracy: 0.9984
 516/1688 [========>.....................] - ETA: 5s - loss: 0.0091 - accuracy: 0.9983
 527/1688 [========>.....................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9983
 538/1688 [========>.....................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9983
 549/1688 [========>.....................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9983
 560/1688 [========>.....................] - ETA: 5s - loss: 0.0091 - accuracy: 0.9983
 572/1688 [=========>....................] - ETA: 5s - loss: 0.0091 - accuracy: 0.9983
 583/1688 [=========>....................] - ETA: 5s - loss: 0.0092 - accuracy: 0.9982
 594/1688 [=========>....................] - ETA: 5s - loss: 0.0093 - accuracy: 0.9981
 605/1688 [=========>....................] - ETA: 4s - loss: 0.0093 - accuracy: 0.9981
 616/1688 [=========>....................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 628/1688 [==========>...................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 639/1688 [==========>...................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 650/1688 [==========>...................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 661/1688 [==========>...................] - ETA: 4s - loss: 0.0095 - accuracy: 0.9981
 672/1688 [==========>...................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 683/1688 [===========>..................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
 694/1688 [===========>..................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
 705/1688 [===========>..................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
 717/1688 [===========>..................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9979
 729/1688 [===========>..................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
 741/1688 [============>.................] - ETA: 4s - loss: 0.0098 - accuracy: 0.9979
 752/1688 [============>.................] - ETA: 4s - loss: 0.0097 - accuracy: 0.9980
 763/1688 [============>.................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 774/1688 [============>.................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 785/1688 [============>.................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 797/1688 [=============>................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 809/1688 [=============>................] - ETA: 4s - loss: 0.0096 - accuracy: 0.9980
 820/1688 [=============>................] - ETA: 3s - loss: 0.0096 - accuracy: 0.9980
 831/1688 [=============>................] - ETA: 3s - loss: 0.0095 - accuracy: 0.9980
 842/1688 [=============>................] - ETA: 3s - loss: 0.0096 - accuracy: 0.9979
 854/1688 [==============>...............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9980
 865/1688 [==============>...............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9980
 877/1688 [==============>...............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9980
 888/1688 [==============>...............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9980
 900/1688 [==============>...............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9980
 912/1688 [===============>..............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9979
 923/1688 [===============>..............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9979
 934/1688 [===============>..............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9979
 945/1688 [===============>..............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9979
 956/1688 [===============>..............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9979
 967/1688 [================>.............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9979
 979/1688 [================>.............] - ETA: 3s - loss: 0.0097 - accuracy: 0.9979
 991/1688 [================>.............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9979
1002/1688 [================>.............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9979
1014/1688 [=================>............] - ETA: 3s - loss: 0.0096 - accuracy: 0.9979
1026/1688 [=================>............] - ETA: 3s - loss: 0.0095 - accuracy: 0.9980
1037/1688 [=================>............] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1048/1688 [=================>............] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1060/1688 [=================>............] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1072/1688 [==================>...........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1083/1688 [==================>...........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1094/1688 [==================>...........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1106/1688 [==================>...........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1118/1688 [==================>...........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9978
1130/1688 [===================>..........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1141/1688 [===================>..........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1152/1688 [===================>..........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1164/1688 [===================>..........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1175/1688 [===================>..........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1186/1688 [====================>.........] - ETA: 2s - loss: 0.0096 - accuracy: 0.9979
1198/1688 [====================>.........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9979
1209/1688 [====================>.........] - ETA: 2s - loss: 0.0098 - accuracy: 0.9978
1220/1688 [====================>.........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9978
1232/1688 [====================>.........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9978
1243/1688 [=====================>........] - ETA: 2s - loss: 0.0097 - accuracy: 0.9978
1254/1688 [=====================>........] - ETA: 1s - loss: 0.0097 - accuracy: 0.9978
1265/1688 [=====================>........] - ETA: 1s - loss: 0.0097 - accuracy: 0.9979
1276/1688 [=====================>........] - ETA: 1s - loss: 0.0096 - accuracy: 0.9979
1287/1688 [=====================>........] - ETA: 1s - loss: 0.0096 - accuracy: 0.9978
1298/1688 [======================>.......] - ETA: 1s - loss: 0.0096 - accuracy: 0.9978
1310/1688 [======================>.......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9978
1322/1688 [======================>.......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9978
1334/1688 [======================>.......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9978
1345/1688 [======================>.......] - ETA: 1s - loss: 0.0096 - accuracy: 0.9978
1356/1688 [=======================>......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9978
1367/1688 [=======================>......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9978
1378/1688 [=======================>......] - ETA: 1s - loss: 0.0097 - accuracy: 0.9978
1390/1688 [=======================>......] - ETA: 1s - loss: 0.0098 - accuracy: 0.9978
1401/1688 [=======================>......] - ETA: 1s - loss: 0.0100 - accuracy: 0.9977
1412/1688 [========================>.....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9977
1424/1688 [========================>.....] - ETA: 1s - loss: 0.0100 - accuracy: 0.9977
1435/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1446/1688 [========================>.....] - ETA: 1s - loss: 0.0101 - accuracy: 0.9976
1457/1688 [========================>.....] - ETA: 1s - loss: 0.0103 - accuracy: 0.9976
1468/1688 [=========================>....] - ETA: 1s - loss: 0.0102 - accuracy: 0.9976
1479/1688 [=========================>....] - ETA: 0s - loss: 0.0102 - accuracy: 0.9976
1491/1688 [=========================>....] - ETA: 0s - loss: 0.0102 - accuracy: 0.9976
1503/1688 [=========================>....] - ETA: 0s - loss: 0.0103 - accuracy: 0.9975
1514/1688 [=========================>....] - ETA: 0s - loss: 0.0103 - accuracy: 0.9975
1525/1688 [==========================>...] - ETA: 0s - loss: 0.0103 - accuracy: 0.9976
1536/1688 [==========================>...] - ETA: 0s - loss: 0.0103 - accuracy: 0.9976
1548/1688 [==========================>...] - ETA: 0s - loss: 0.0103 - accuracy: 0.9976
1560/1688 [==========================>...] - ETA: 0s - loss: 0.0104 - accuracy: 0.9975
1571/1688 [==========================>...] - ETA: 0s - loss: 0.0104 - accuracy: 0.9975
1582/1688 [===========================>..] - ETA: 0s - loss: 0.0104 - accuracy: 0.9976
1593/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9976
1604/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9976
1615/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9976
1627/1688 [===========================>..] - ETA: 0s - loss: 0.0103 - accuracy: 0.9976
1638/1688 [============================>.] - ETA: 0s - loss: 0.0102 - accuracy: 0.9976
1649/1688 [============================>.] - ETA: 0s - loss: 0.0102 - accuracy: 0.9976
1661/1688 [============================>.] - ETA: 0s - loss: 0.0102 - accuracy: 0.9976
1672/1688 [============================>.] - ETA: 0s - loss: 0.0102 - accuracy: 0.9976
1684/1688 [============================>.] - ETA: 0s - loss: 0.0103 - accuracy: 0.9976
1688/1688 [==============================] - 8s 5ms/step - loss: 0.0102 - accuracy: 0.9976 - val_loss: 0.0448 - val_accuracy: 0.9862

<keras.callbacks.History object at 0x7f62b703f850>
score = model_quantized.evaluate(x_test, y_test, verbose=0)[1]
print('Test accuracy after fine tuning:', score)
Test accuracy after fine tuning: 0.9855999946594238

3. Convert

3.1 Convert to Akida model

When the quantized model produces satisfactory performance, it can be converted to the native Akida format. The convert function returns a model in Akida format ready for inference.

As with Keras, the summary() method provides a textual representation of the Akida model.

from cnn2snn import convert

model_akida = convert(model_quantized)
model_akida.summary()
WARNING:tensorflow:5 out of the last 5 calls to <function NonTrackVariable.set_var at 0x7f62a79e5670> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function NonTrackVariable.set_var at 0x7f62a7a02040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
                Model Summary
______________________________________________
Input shape  Output shape  Sequences  Layers
==============================================
[28, 28, 1]  [1, 1, 10]    1          5
______________________________________________

__________________________________________________________________
Layer (type)                        Output shape  Kernel shape

=============== SW/conv2d-dequantizer_2 (Software) ===============

conv2d (InputConv2D)                [13, 13, 32]  (3, 3, 1, 32)
__________________________________________________________________
depthwise_conv2d (DepthwiseConv2D)  [7, 7, 32]    (3, 3, 32, 1)
__________________________________________________________________
conv2d_1 (Conv2D)                   [7, 7, 64]    (1, 1, 32, 64)
__________________________________________________________________
dense (Dense2D)                     [1, 1, 10]    (3136, 10)
__________________________________________________________________
dequantizer_2 (Dequantizer)         [1, 1, 10]    N/A
__________________________________________________________________

3.2. Check performance

accuracy = model_akida.evaluate(x_test, y_test)
print('Test accuracy after conversion:', accuracy)

# For non-regression purposes
assert accuracy > 0.96
Test accuracy after conversion: 0.9861000180244446

3.3 Show predictions for a single image

Display one of the test images, such as the first image in the dataset from above, to visualize the output of the model.

# Test a single example
sample_image = 0
image = x_test[sample_image]
outputs = model_akida.predict(image.reshape(1, 28, 28, 1))
print('Input Label: %i' % y_test[sample_image])

f, axarr = plt.subplots(1, 2)
axarr[0].imshow(x_test[sample_image].reshape((28, 28)), cmap=cm.Greys_r)
axarr[0].set_title('Class %d' % y_test[sample_image])
axarr[1].bar(range(10), outputs.squeeze())
axarr[1].set_xticks(range(10))
plt.show()

print(outputs.squeeze())
Class 7
Input Label: 7
[ -7.690133   -8.094707   -3.5707319  -1.7342007 -12.817631   -5.7350273
 -19.938774    6.819388   -3.9958827  -0.5754613]

Consider the output from the model above. As is typical in backprop-trained models, the final layer is a Dense layer with one neuron for each of the 10 classes in the dataset. The goal of training is to maximize the response of the neuron corresponding to the label of each training sample while minimizing the responses of the other neurons.

In the bar chart above, you can see the outputs from all 10 neurons. It is easy to see that neuron 7 responds much more strongly than the others. The first sample is indeed a number 7.

4. GXNOR/MNIST

A more robust model called GXNOR/MNIST is provided in the model zoo It is inspired from the GXNOR-Net paper. It comes with its pretrained 2/2/1 helper for which the float training was done for 20 epochs, then the model was then gradually quantized following: 4/4/4 –> 4/4/2 –> 2/2/2 –> 2/2/1 with a 15 epochs QAT step at each quantization stage.

Total running time of the script: (1 minutes 40.543 seconds)

Gallery generated by Sphinx-Gallery