CNN conversion flow tutorial

The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.

Note

Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.

CNN2SNN tool allows you to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps.

Note

MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.

../_images/cnn2snn_flow_small.png

1. System configuration

1.1 Load CNN2SNN tool dependencies

# System imports
import os
import sys
import numpy as np
from sklearn.metrics import accuracy_score
from tempfile import TemporaryDirectory

# TensorFlow imports
import tensorflow.keras.backend as K
from tensorflow.keras.callbacks import LearningRateScheduler
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Conv2D, Dense, BatchNormalization, MaxPooling2D, Activation, ReLU, Flatten, Input
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical

1.2 Load and reshape MNIST dataset

After loading, we make 3 transformations on the dataset:

  1. Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.

Note

At this point, we’ll set aside the raw data for testing our converted model in the Akida Execution Engine later

  1. Rescale the 8-bit loaded data to the range 0-to-1 for training.

Note

This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.

Also note that we store the scaling values input_scaling for use when preparing the model for the Akida Execution Engine. The implementation of the Akida neural network allows us to completely skip the rescaling step (i.e. the Akida model should be fed with the raw 8-bit values) but that does require information about what scaling was applied prior to training - see below for more details.

3. Transform the loaded labels from a scalar representation (single integer value per sample) to a one-hot vector representation, appropriate for use with the squared hinge loss function used in the current model.

Note

Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some interesting reading can be found here.

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)

# Set aside raw test data for use with Akida Execution Engine later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test

# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a

# Transform scalar labels to one-hot representation, scaled to +/- 1 appropriate for squared hinge loss function
y_train = to_categorical(y_train, 10) * 2 - 1
y_test = to_categorical(y_test, 10) * 2 - 1

1.3 Set training parameters

Set some training parameters used across the different training sessions:

# Set dataset relative training parameters
epochs = 5
batch_size = 128

# Set the learning rate parameters
lr_start = 1e-3
lr_end = 1e-4
lr_decay = (lr_end / lr_start)**(1. / epochs)

2. Model creation and performance check

2.1 Model creation

Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.

An appropriate model for MNIST (inspired by this paper) might look something like the following:

img_input = Input(shape=(28, 28, 1))
x = Conv2D(filters=32,
           kernel_size=(5, 5),
           padding='same',
           use_bias=False,
           data_format='channels_last')(img_input)
x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(x)
x = BatchNormalization()(x)
x = ReLU(6.)(x)

x = Conv2D(filters=32, kernel_size=(5, 5), padding='same', use_bias=False)(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(x)
x = BatchNormalization()(x)
x = ReLU(6.)(x)

x = Conv2D(filters=32, kernel_size=(5, 5), padding='same', use_bias=False)(x)
x = BatchNormalization()(x)
x = ReLU(6.)(x)

x = Flatten()(x)
x = Dense(512, use_bias=False)(x)
x = BatchNormalization()(x)
x = ReLU(6.)(x)
x = Dense(10, use_bias=False)(x)

model_keras = Model(img_input, x, name='mnistnet')

opt = Adam(lr=lr_start)
model_keras.compile(loss='squared_hinge', optimizer=opt, metrics=['accuracy'])
model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_6 (InputLayer)         [(None, 28, 28, 1)]       0
_________________________________________________________________
conv2d (Conv2D)              (None, 28, 28, 32)        800
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 32)        0
_________________________________________________________________
batch_normalization (BatchNo (None, 14, 14, 32)        128
_________________________________________________________________
re_lu (ReLU)                 (None, 14, 14, 32)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 14, 14, 32)        25600
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 32)          0
_________________________________________________________________
batch_normalization_1 (Batch (None, 7, 7, 32)          128
_________________________________________________________________
re_lu_1 (ReLU)               (None, 7, 7, 32)          0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 7, 7, 32)          25600
_________________________________________________________________
batch_normalization_2 (Batch (None, 7, 7, 32)          128
_________________________________________________________________
re_lu_2 (ReLU)               (None, 7, 7, 32)          0
_________________________________________________________________
flatten_1 (Flatten)          (None, 1568)              0
_________________________________________________________________
dense (Dense)                (None, 512)               802816
_________________________________________________________________
batch_normalization_3 (Batch (None, 512)               2048
_________________________________________________________________
re_lu_3 (ReLU)               (None, 512)               0
_________________________________________________________________
dense_1 (Dense)              (None, 10)                5120
=================================================================
Total params: 862,368
Trainable params: 861,152
Non-trainable params: 1,216
_________________________________________________________________

Note

Adam optimizer is commonly used, more details can be found here.

2.2 Performance check

Before going any further, check the current model performance as a benchmark for CNN2SNN conversion. The created model should achieve a test accuracy a little over 99% after 5 epochs:

callbacks = []
lr_scheduler = LearningRateScheduler(lambda e: lr_start * lr_decay**e)
callbacks.append(lr_scheduler)
history = model_keras.fit(x_train,
                          y_train,
                          batch_size=batch_size,
                          epochs=epochs,
                          verbose=1,
                          validation_data=(x_test, y_test),
                          callbacks=callbacks)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/5

  1/469 [..............................] - ETA: 0s - loss: 2.0924 - accuracy: 0.0547
  4/469 [..............................] - ETA: 8s - loss: 1.2178 - accuracy: 0.4766
  7/469 [..............................] - ETA: 9s - loss: 0.8239 - accuracy: 0.6250
 10/469 [..............................] - ETA: 9s - loss: 0.6138 - accuracy: 0.7055
 13/469 [..............................] - ETA: 10s - loss: 0.4951 - accuracy: 0.7506
 16/469 [>.............................] - ETA: 10s - loss: 0.4176 - accuracy: 0.7808
 19/469 [>.............................] - ETA: 10s - loss: 0.3609 - accuracy: 0.8063
 22/469 [>.............................] - ETA: 10s - loss: 0.3193 - accuracy: 0.8242
 25/469 [>.............................] - ETA: 10s - loss: 0.2879 - accuracy: 0.8375
 28/469 [>.............................] - ETA: 10s - loss: 0.2634 - accuracy: 0.8477
 31/469 [>.............................] - ETA: 10s - loss: 0.2418 - accuracy: 0.8584
 34/469 [=>............................] - ETA: 10s - loss: 0.2255 - accuracy: 0.8663
 37/469 [=>............................] - ETA: 10s - loss: 0.2120 - accuracy: 0.8714
 40/469 [=>............................] - ETA: 10s - loss: 0.1983 - accuracy: 0.8783
 43/469 [=>............................] - ETA: 9s - loss: 0.1867 - accuracy: 0.8852 
 46/469 [=>............................] - ETA: 9s - loss: 0.1777 - accuracy: 0.8888
 49/469 [==>...........................] - ETA: 9s - loss: 0.1686 - accuracy: 0.8937
 52/469 [==>...........................] - ETA: 9s - loss: 0.1616 - accuracy: 0.8974
 55/469 [==>...........................] - ETA: 9s - loss: 0.1540 - accuracy: 0.9017
 58/469 [==>...........................] - ETA: 9s - loss: 0.1471 - accuracy: 0.9054
 61/469 [==>...........................] - ETA: 9s - loss: 0.1412 - accuracy: 0.9087
 64/469 [===>..........................] - ETA: 9s - loss: 0.1359 - accuracy: 0.9115
 67/469 [===>..........................] - ETA: 9s - loss: 0.1312 - accuracy: 0.9141
 70/469 [===>..........................] - ETA: 9s - loss: 0.1266 - accuracy: 0.9169
 73/469 [===>..........................] - ETA: 9s - loss: 0.1222 - accuracy: 0.9195
 76/469 [===>..........................] - ETA: 9s - loss: 0.1180 - accuracy: 0.9221
 79/469 [====>.........................] - ETA: 9s - loss: 0.1141 - accuracy: 0.9244
 82/469 [====>.........................] - ETA: 9s - loss: 0.1113 - accuracy: 0.9257
 85/469 [====>.........................] - ETA: 9s - loss: 0.1083 - accuracy: 0.9272
 88/469 [====>.........................] - ETA: 8s - loss: 0.1055 - accuracy: 0.9288
 91/469 [====>.........................] - ETA: 8s - loss: 0.1028 - accuracy: 0.9304
 94/469 [=====>........................] - ETA: 8s - loss: 0.1004 - accuracy: 0.9313
 97/469 [=====>........................] - ETA: 8s - loss: 0.0982 - accuracy: 0.9326
100/469 [=====>........................] - ETA: 8s - loss: 0.0960 - accuracy: 0.9340
103/469 [=====>........................] - ETA: 8s - loss: 0.0941 - accuracy: 0.9352
106/469 [=====>........................] - ETA: 8s - loss: 0.0923 - accuracy: 0.9360
109/469 [=====>........................] - ETA: 8s - loss: 0.0903 - accuracy: 0.9371
112/469 [======>.......................] - ETA: 8s - loss: 0.0886 - accuracy: 0.9381
115/469 [======>.......................] - ETA: 8s - loss: 0.0866 - accuracy: 0.9393
118/469 [======>.......................] - ETA: 8s - loss: 0.0849 - accuracy: 0.9405
121/469 [======>.......................] - ETA: 8s - loss: 0.0831 - accuracy: 0.9414
124/469 [======>.......................] - ETA: 8s - loss: 0.0817 - accuracy: 0.9421
127/469 [=======>......................] - ETA: 8s - loss: 0.0801 - accuracy: 0.9431
130/469 [=======>......................] - ETA: 8s - loss: 0.0787 - accuracy: 0.9441
133/469 [=======>......................] - ETA: 7s - loss: 0.0774 - accuracy: 0.9448
136/469 [=======>......................] - ETA: 7s - loss: 0.0761 - accuracy: 0.9455
139/469 [=======>......................] - ETA: 7s - loss: 0.0750 - accuracy: 0.9462
142/469 [========>.....................] - ETA: 7s - loss: 0.0741 - accuracy: 0.9469
145/469 [========>.....................] - ETA: 7s - loss: 0.0729 - accuracy: 0.9475
148/469 [========>.....................] - ETA: 7s - loss: 0.0719 - accuracy: 0.9482
151/469 [========>.....................] - ETA: 7s - loss: 0.0709 - accuracy: 0.9486
154/469 [========>.....................] - ETA: 7s - loss: 0.0701 - accuracy: 0.9491
157/469 [=========>....................] - ETA: 7s - loss: 0.0690 - accuracy: 0.9497
160/469 [=========>....................] - ETA: 7s - loss: 0.0683 - accuracy: 0.9500
163/469 [=========>....................] - ETA: 7s - loss: 0.0673 - accuracy: 0.9507
166/469 [=========>....................] - ETA: 7s - loss: 0.0664 - accuracy: 0.9513
169/469 [=========>....................] - ETA: 7s - loss: 0.0656 - accuracy: 0.9517
172/469 [==========>...................] - ETA: 7s - loss: 0.0647 - accuracy: 0.9523
175/469 [==========>...................] - ETA: 6s - loss: 0.0639 - accuracy: 0.9527
178/469 [==========>...................] - ETA: 6s - loss: 0.0631 - accuracy: 0.9533
180/469 [==========>...................] - ETA: 6s - loss: 0.0626 - accuracy: 0.9534
182/469 [==========>...................] - ETA: 6s - loss: 0.0621 - accuracy: 0.9537
184/469 [==========>...................] - ETA: 6s - loss: 0.0616 - accuracy: 0.9541
186/469 [==========>...................] - ETA: 6s - loss: 0.0611 - accuracy: 0.9544
188/469 [===========>..................] - ETA: 6s - loss: 0.0606 - accuracy: 0.9548
190/469 [===========>..................] - ETA: 6s - loss: 0.0601 - accuracy: 0.9551
192/469 [===========>..................] - ETA: 6s - loss: 0.0596 - accuracy: 0.9554
195/469 [===========>..................] - ETA: 6s - loss: 0.0590 - accuracy: 0.9558
198/469 [===========>..................] - ETA: 6s - loss: 0.0583 - accuracy: 0.9562
201/469 [===========>..................] - ETA: 6s - loss: 0.0577 - accuracy: 0.9566
204/469 [============>.................] - ETA: 6s - loss: 0.0572 - accuracy: 0.9569
207/469 [============>.................] - ETA: 6s - loss: 0.0566 - accuracy: 0.9574
210/469 [============>.................] - ETA: 6s - loss: 0.0560 - accuracy: 0.9576
213/469 [============>.................] - ETA: 6s - loss: 0.0555 - accuracy: 0.9579
216/469 [============>.................] - ETA: 6s - loss: 0.0549 - accuracy: 0.9583
219/469 [=============>................] - ETA: 6s - loss: 0.0544 - accuracy: 0.9587
222/469 [=============>................] - ETA: 5s - loss: 0.0539 - accuracy: 0.9591
225/469 [=============>................] - ETA: 5s - loss: 0.0534 - accuracy: 0.9594
228/469 [=============>................] - ETA: 5s - loss: 0.0529 - accuracy: 0.9596
231/469 [=============>................] - ETA: 5s - loss: 0.0525 - accuracy: 0.9598
234/469 [=============>................] - ETA: 5s - loss: 0.0520 - accuracy: 0.9601
237/469 [==============>...............] - ETA: 5s - loss: 0.0516 - accuracy: 0.9603
240/469 [==============>...............] - ETA: 5s - loss: 0.0512 - accuracy: 0.9607
243/469 [==============>...............] - ETA: 5s - loss: 0.0508 - accuracy: 0.9608
246/469 [==============>...............] - ETA: 5s - loss: 0.0503 - accuracy: 0.9611
249/469 [==============>...............] - ETA: 5s - loss: 0.0499 - accuracy: 0.9614
252/469 [===============>..............] - ETA: 5s - loss: 0.0495 - accuracy: 0.9617
255/469 [===============>..............] - ETA: 5s - loss: 0.0491 - accuracy: 0.9617
258/469 [===============>..............] - ETA: 5s - loss: 0.0488 - accuracy: 0.9619
261/469 [===============>..............] - ETA: 4s - loss: 0.0484 - accuracy: 0.9621
264/469 [===============>..............] - ETA: 4s - loss: 0.0481 - accuracy: 0.9622
267/469 [================>.............] - ETA: 4s - loss: 0.0479 - accuracy: 0.9624
270/469 [================>.............] - ETA: 4s - loss: 0.0474 - accuracy: 0.9628
273/469 [================>.............] - ETA: 4s - loss: 0.0470 - accuracy: 0.9631
276/469 [================>.............] - ETA: 4s - loss: 0.0467 - accuracy: 0.9632
279/469 [================>.............] - ETA: 4s - loss: 0.0465 - accuracy: 0.9634
282/469 [=================>............] - ETA: 4s - loss: 0.0461 - accuracy: 0.9636
285/469 [=================>............] - ETA: 4s - loss: 0.0457 - accuracy: 0.9638
288/469 [=================>............] - ETA: 4s - loss: 0.0454 - accuracy: 0.9639
291/469 [=================>............] - ETA: 4s - loss: 0.0451 - accuracy: 0.9641
294/469 [=================>............] - ETA: 4s - loss: 0.0447 - accuracy: 0.9642
297/469 [=================>............] - ETA: 4s - loss: 0.0444 - accuracy: 0.9645
300/469 [==================>...........] - ETA: 4s - loss: 0.0441 - accuracy: 0.9647
303/469 [==================>...........] - ETA: 3s - loss: 0.0438 - accuracy: 0.9649
306/469 [==================>...........] - ETA: 3s - loss: 0.0435 - accuracy: 0.9652
309/469 [==================>...........] - ETA: 3s - loss: 0.0433 - accuracy: 0.9653
312/469 [==================>...........] - ETA: 3s - loss: 0.0430 - accuracy: 0.9654
315/469 [===================>..........] - ETA: 3s - loss: 0.0428 - accuracy: 0.9656
318/469 [===================>..........] - ETA: 3s - loss: 0.0426 - accuracy: 0.9657
321/469 [===================>..........] - ETA: 3s - loss: 0.0423 - accuracy: 0.9659
324/469 [===================>..........] - ETA: 3s - loss: 0.0420 - accuracy: 0.9661
327/469 [===================>..........] - ETA: 3s - loss: 0.0418 - accuracy: 0.9664
330/469 [====================>.........] - ETA: 3s - loss: 0.0416 - accuracy: 0.9665
333/469 [====================>.........] - ETA: 3s - loss: 0.0413 - accuracy: 0.9666
336/469 [====================>.........] - ETA: 3s - loss: 0.0411 - accuracy: 0.9667
339/469 [====================>.........] - ETA: 3s - loss: 0.0408 - accuracy: 0.9669
342/469 [====================>.........] - ETA: 3s - loss: 0.0406 - accuracy: 0.9669
345/469 [=====================>........] - ETA: 2s - loss: 0.0405 - accuracy: 0.9670
348/469 [=====================>........] - ETA: 2s - loss: 0.0403 - accuracy: 0.9672
351/469 [=====================>........] - ETA: 2s - loss: 0.0400 - accuracy: 0.9673
354/469 [=====================>........] - ETA: 2s - loss: 0.0398 - accuracy: 0.9674
357/469 [=====================>........] - ETA: 2s - loss: 0.0396 - accuracy: 0.9676
360/469 [======================>.......] - ETA: 2s - loss: 0.0394 - accuracy: 0.9677
363/469 [======================>.......] - ETA: 2s - loss: 0.0391 - accuracy: 0.9678
366/469 [======================>.......] - ETA: 2s - loss: 0.0389 - accuracy: 0.9680
369/469 [======================>.......] - ETA: 2s - loss: 0.0387 - accuracy: 0.9681
372/469 [======================>.......] - ETA: 2s - loss: 0.0385 - accuracy: 0.9683
375/469 [======================>.......] - ETA: 2s - loss: 0.0383 - accuracy: 0.9685
378/469 [=======================>......] - ETA: 2s - loss: 0.0380 - accuracy: 0.9686
381/469 [=======================>......] - ETA: 2s - loss: 0.0378 - accuracy: 0.9688
384/469 [=======================>......] - ETA: 2s - loss: 0.0376 - accuracy: 0.9689
387/469 [=======================>......] - ETA: 1s - loss: 0.0374 - accuracy: 0.9691
390/469 [=======================>......] - ETA: 1s - loss: 0.0372 - accuracy: 0.9692
393/469 [========================>.....] - ETA: 1s - loss: 0.0370 - accuracy: 0.9693
396/469 [========================>.....] - ETA: 1s - loss: 0.0369 - accuracy: 0.9693
399/469 [========================>.....] - ETA: 1s - loss: 0.0367 - accuracy: 0.9695
402/469 [========================>.....] - ETA: 1s - loss: 0.0366 - accuracy: 0.9695
405/469 [========================>.....] - ETA: 1s - loss: 0.0364 - accuracy: 0.9697
407/469 [=========================>....] - ETA: 1s - loss: 0.0362 - accuracy: 0.9698
409/469 [=========================>....] - ETA: 1s - loss: 0.0361 - accuracy: 0.9699
411/469 [=========================>....] - ETA: 1s - loss: 0.0360 - accuracy: 0.9700
413/469 [=========================>....] - ETA: 1s - loss: 0.0359 - accuracy: 0.9701
415/469 [=========================>....] - ETA: 1s - loss: 0.0358 - accuracy: 0.9701
417/469 [=========================>....] - ETA: 1s - loss: 0.0357 - accuracy: 0.9702
419/469 [=========================>....] - ETA: 1s - loss: 0.0356 - accuracy: 0.9702
422/469 [=========================>....] - ETA: 1s - loss: 0.0355 - accuracy: 0.9703
425/469 [==========================>...] - ETA: 1s - loss: 0.0353 - accuracy: 0.9703
427/469 [==========================>...] - ETA: 1s - loss: 0.0352 - accuracy: 0.9704
430/469 [==========================>...] - ETA: 0s - loss: 0.0351 - accuracy: 0.9705
433/469 [==========================>...] - ETA: 0s - loss: 0.0349 - accuracy: 0.9706
436/469 [==========================>...] - ETA: 0s - loss: 0.0347 - accuracy: 0.9707
439/469 [===========================>..] - ETA: 0s - loss: 0.0346 - accuracy: 0.9708
442/469 [===========================>..] - ETA: 0s - loss: 0.0344 - accuracy: 0.9709
445/469 [===========================>..] - ETA: 0s - loss: 0.0342 - accuracy: 0.9711
448/469 [===========================>..] - ETA: 0s - loss: 0.0341 - accuracy: 0.9712
451/469 [===========================>..] - ETA: 0s - loss: 0.0340 - accuracy: 0.9712
454/469 [============================>.] - ETA: 0s - loss: 0.0338 - accuracy: 0.9714
457/469 [============================>.] - ETA: 0s - loss: 0.0336 - accuracy: 0.9715
460/469 [============================>.] - ETA: 0s - loss: 0.0334 - accuracy: 0.9716
463/469 [============================>.] - ETA: 0s - loss: 0.0333 - accuracy: 0.9717
466/469 [============================>.] - ETA: 0s - loss: 0.0332 - accuracy: 0.9718
469/469 [==============================] - ETA: 0s - loss: 0.0331 - accuracy: 0.9719
469/469 [==============================] - 12s 25ms/step - loss: 0.0331 - accuracy: 0.9719 - val_loss: 0.0281 - val_accuracy: 0.9683 - lr: 0.0010
Epoch 2/5

  1/469 [..............................] - ETA: 0s - loss: 0.0489 - accuracy: 0.9844
  4/469 [..............................] - ETA: 8s - loss: 0.0225 - accuracy: 0.9883
  7/469 [..............................] - ETA: 9s - loss: 0.0185 - accuracy: 0.9888
 10/469 [..............................] - ETA: 9s - loss: 0.0153 - accuracy: 0.9898
 13/469 [..............................] - ETA: 9s - loss: 0.0135 - accuracy: 0.9910
 16/469 [>.............................] - ETA: 10s - loss: 0.0126 - accuracy: 0.9907
 19/469 [>.............................] - ETA: 10s - loss: 0.0117 - accuracy: 0.9914
 22/469 [>.............................] - ETA: 10s - loss: 0.0108 - accuracy: 0.9922
 25/469 [>.............................] - ETA: 10s - loss: 0.0108 - accuracy: 0.9919
 28/469 [>.............................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9914
 31/469 [>.............................] - ETA: 10s - loss: 0.0102 - accuracy: 0.9919
 34/469 [=>............................] - ETA: 10s - loss: 0.0095 - accuracy: 0.9924
 37/469 [=>............................] - ETA: 10s - loss: 0.0093 - accuracy: 0.9920
 40/469 [=>............................] - ETA: 10s - loss: 0.0088 - accuracy: 0.9924
 43/469 [=>............................] - ETA: 9s - loss: 0.0084 - accuracy: 0.9927 
 46/469 [=>............................] - ETA: 9s - loss: 0.0083 - accuracy: 0.9927
 49/469 [==>...........................] - ETA: 9s - loss: 0.0081 - accuracy: 0.9928
 52/469 [==>...........................] - ETA: 9s - loss: 0.0080 - accuracy: 0.9929
 55/469 [==>...........................] - ETA: 9s - loss: 0.0079 - accuracy: 0.9929
 58/469 [==>...........................] - ETA: 9s - loss: 0.0079 - accuracy: 0.9926
 61/469 [==>...........................] - ETA: 9s - loss: 0.0078 - accuracy: 0.9926
 64/469 [===>..........................] - ETA: 9s - loss: 0.0078 - accuracy: 0.9924
 67/469 [===>..........................] - ETA: 9s - loss: 0.0077 - accuracy: 0.9924
 70/469 [===>..........................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9923
 73/469 [===>..........................] - ETA: 9s - loss: 0.0074 - accuracy: 0.9926
 76/469 [===>..........................] - ETA: 9s - loss: 0.0074 - accuracy: 0.9924
 79/469 [====>.........................] - ETA: 9s - loss: 0.0073 - accuracy: 0.9925
 82/469 [====>.........................] - ETA: 9s - loss: 0.0072 - accuracy: 0.9926
 85/469 [====>.........................] - ETA: 9s - loss: 0.0070 - accuracy: 0.9927
 88/469 [====>.........................] - ETA: 8s - loss: 0.0070 - accuracy: 0.9927
 91/469 [====>.........................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9929
 94/469 [=====>........................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9929
 97/469 [=====>........................] - ETA: 8s - loss: 0.0070 - accuracy: 0.9924
100/469 [=====>........................] - ETA: 8s - loss: 0.0071 - accuracy: 0.9925
103/469 [=====>........................] - ETA: 8s - loss: 0.0070 - accuracy: 0.9924
106/469 [=====>........................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9926
109/469 [=====>........................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9925
112/469 [======>.......................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9924
115/469 [======>.......................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9925
118/469 [======>.......................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9925
121/469 [======>.......................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9924
124/469 [======>.......................] - ETA: 8s - loss: 0.0069 - accuracy: 0.9924
127/469 [=======>......................] - ETA: 8s - loss: 0.0070 - accuracy: 0.9922
130/469 [=======>......................] - ETA: 8s - loss: 0.0070 - accuracy: 0.9921
133/469 [=======>......................] - ETA: 7s - loss: 0.0069 - accuracy: 0.9922
136/469 [=======>......................] - ETA: 7s - loss: 0.0069 - accuracy: 0.9922
138/469 [=======>......................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9922
140/469 [=======>......................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9922
142/469 [========>.....................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9922
144/469 [========>.....................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9924
146/469 [========>.....................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9922
148/469 [========>.....................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9923
150/469 [========>.....................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9924
152/469 [========>.....................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9925
154/469 [========>.....................] - ETA: 7s - loss: 0.0066 - accuracy: 0.9926
157/469 [=========>....................] - ETA: 7s - loss: 0.0066 - accuracy: 0.9926
160/469 [=========>....................] - ETA: 7s - loss: 0.0065 - accuracy: 0.9927
163/469 [=========>....................] - ETA: 7s - loss: 0.0065 - accuracy: 0.9928
166/469 [=========>....................] - ETA: 7s - loss: 0.0065 - accuracy: 0.9927
169/469 [=========>....................] - ETA: 7s - loss: 0.0065 - accuracy: 0.9926
172/469 [==========>...................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9928
175/469 [==========>...................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9928
178/469 [==========>...................] - ETA: 7s - loss: 0.0065 - accuracy: 0.9927
181/469 [==========>...................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9928
184/469 [==========>...................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9928
187/469 [==========>...................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9929
190/469 [===========>..................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9929
193/469 [===========>..................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9929
196/469 [===========>..................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9929
199/469 [===========>..................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9930
202/469 [===========>..................] - ETA: 6s - loss: 0.0064 - accuracy: 0.9930
205/469 [============>.................] - ETA: 6s - loss: 0.0063 - accuracy: 0.9930
208/469 [============>.................] - ETA: 6s - loss: 0.0063 - accuracy: 0.9929
211/469 [============>.................] - ETA: 6s - loss: 0.0063 - accuracy: 0.9930
214/469 [============>.................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9931
217/469 [============>.................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9931
220/469 [=============>................] - ETA: 5s - loss: 0.0062 - accuracy: 0.9932
223/469 [=============>................] - ETA: 5s - loss: 0.0062 - accuracy: 0.9932
226/469 [=============>................] - ETA: 5s - loss: 0.0062 - accuracy: 0.9933
229/469 [=============>................] - ETA: 5s - loss: 0.0061 - accuracy: 0.9933
232/469 [=============>................] - ETA: 5s - loss: 0.0062 - accuracy: 0.9933
235/469 [==============>...............] - ETA: 5s - loss: 0.0061 - accuracy: 0.9934
238/469 [==============>...............] - ETA: 5s - loss: 0.0062 - accuracy: 0.9933
241/469 [==============>...............] - ETA: 5s - loss: 0.0062 - accuracy: 0.9933
244/469 [==============>...............] - ETA: 5s - loss: 0.0061 - accuracy: 0.9934
247/469 [==============>...............] - ETA: 5s - loss: 0.0061 - accuracy: 0.9934
250/469 [==============>...............] - ETA: 5s - loss: 0.0062 - accuracy: 0.9933
253/469 [===============>..............] - ETA: 5s - loss: 0.0062 - accuracy: 0.9932
256/469 [===============>..............] - ETA: 5s - loss: 0.0062 - accuracy: 0.9933
259/469 [===============>..............] - ETA: 5s - loss: 0.0062 - accuracy: 0.9932
262/469 [===============>..............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9932
265/469 [===============>..............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9933
268/469 [================>.............] - ETA: 4s - loss: 0.0062 - accuracy: 0.9933
271/469 [================>.............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9933
274/469 [================>.............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9933
277/469 [================>.............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9933
280/469 [================>.............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9934
283/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9934
286/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9934
289/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9934
292/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9934
295/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9934
298/469 [==================>...........] - ETA: 4s - loss: 0.0061 - accuracy: 0.9933
301/469 [==================>...........] - ETA: 4s - loss: 0.0061 - accuracy: 0.9934
304/469 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9933
307/469 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9934
310/469 [==================>...........] - ETA: 3s - loss: 0.0060 - accuracy: 0.9934
313/469 [===================>..........] - ETA: 3s - loss: 0.0060 - accuracy: 0.9934
316/469 [===================>..........] - ETA: 3s - loss: 0.0060 - accuracy: 0.9935
319/469 [===================>..........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9934
322/469 [===================>..........] - ETA: 3s - loss: 0.0060 - accuracy: 0.9934
325/469 [===================>..........] - ETA: 3s - loss: 0.0060 - accuracy: 0.9935
328/469 [===================>..........] - ETA: 3s - loss: 0.0060 - accuracy: 0.9935
331/469 [====================>.........] - ETA: 3s - loss: 0.0060 - accuracy: 0.9934
334/469 [====================>.........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9934
337/469 [====================>.........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9934
340/469 [====================>.........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9933
343/469 [====================>.........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9933
346/469 [=====================>........] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
349/469 [=====================>........] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
352/469 [=====================>........] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
355/469 [=====================>........] - ETA: 2s - loss: 0.0060 - accuracy: 0.9934
358/469 [=====================>........] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
361/469 [======================>.......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9934
364/469 [======================>.......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9934
366/469 [======================>.......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9934
368/469 [======================>.......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9934
370/469 [======================>.......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
372/469 [======================>.......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
374/469 [======================>.......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
376/469 [=======================>......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
378/469 [=======================>......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
380/469 [=======================>......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
383/469 [=======================>......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
386/469 [=======================>......] - ETA: 2s - loss: 0.0060 - accuracy: 0.9933
389/469 [=======================>......] - ETA: 1s - loss: 0.0060 - accuracy: 0.9933
392/469 [========================>.....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
395/469 [========================>.....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9933
398/469 [========================>.....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
401/469 [========================>.....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
404/469 [========================>.....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
407/469 [=========================>....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
410/469 [=========================>....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
413/469 [=========================>....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
416/469 [=========================>....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
419/469 [=========================>....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
422/469 [=========================>....] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
425/469 [==========================>...] - ETA: 1s - loss: 0.0060 - accuracy: 0.9932
428/469 [==========================>...] - ETA: 0s - loss: 0.0060 - accuracy: 0.9932
431/469 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
434/469 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
437/469 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
440/469 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
443/469 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
446/469 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
449/469 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
452/469 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
455/469 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9932
458/469 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9931
461/469 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9931
464/469 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9931
467/469 [============================>.] - ETA: 0s - loss: 0.0061 - accuracy: 0.9931
469/469 [==============================] - 12s 25ms/step - loss: 0.0061 - accuracy: 0.9930 - val_loss: 0.0095 - val_accuracy: 0.9897 - lr: 6.3096e-04
Epoch 3/5

  1/469 [..............................] - ETA: 0s - loss: 0.0028 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 0.0018 - accuracy: 1.0000
  7/469 [..............................] - ETA: 9s - loss: 0.0024 - accuracy: 1.0000
 10/469 [..............................] - ETA: 9s - loss: 0.0041 - accuracy: 0.9969
 13/469 [..............................] - ETA: 10s - loss: 0.0046 - accuracy: 0.9958
 16/469 [>.............................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9946
 19/469 [>.............................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9955
 22/469 [>.............................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9957
 25/469 [>.............................] - ETA: 10s - loss: 0.0040 - accuracy: 0.9962
 28/469 [>.............................] - ETA: 10s - loss: 0.0042 - accuracy: 0.9964
 31/469 [>.............................] - ETA: 10s - loss: 0.0040 - accuracy: 0.9967
 34/469 [=>............................] - ETA: 10s - loss: 0.0038 - accuracy: 0.9970
 37/469 [=>............................] - ETA: 10s - loss: 0.0039 - accuracy: 0.9964
 40/469 [=>............................] - ETA: 10s - loss: 0.0038 - accuracy: 0.9965
 43/469 [=>............................] - ETA: 9s - loss: 0.0037 - accuracy: 0.9967 
 46/469 [=>............................] - ETA: 9s - loss: 0.0035 - accuracy: 0.9969
 49/469 [==>...........................] - ETA: 9s - loss: 0.0034 - accuracy: 0.9971
 52/469 [==>...........................] - ETA: 9s - loss: 0.0033 - accuracy: 0.9973
 55/469 [==>...........................] - ETA: 9s - loss: 0.0033 - accuracy: 0.9973
 58/469 [==>...........................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9974
 61/469 [==>...........................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9976
 64/469 [===>..........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9977
 67/469 [===>..........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9976
 70/469 [===>..........................] - ETA: 9s - loss: 0.0029 - accuracy: 0.9977
 73/469 [===>..........................] - ETA: 9s - loss: 0.0029 - accuracy: 0.9978
 76/469 [===>..........................] - ETA: 9s - loss: 0.0028 - accuracy: 0.9978
 79/469 [====>.........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9978
 82/469 [====>.........................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9978
 85/469 [====>.........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9978
 88/469 [====>.........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9979
 91/469 [====>.........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9979
 94/469 [=====>........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9978
 97/469 [=====>........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9978
100/469 [=====>........................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9977
103/469 [=====>........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9978
105/469 [=====>........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9978
107/469 [=====>........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9978
109/469 [=====>........................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9978
111/469 [======>.......................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9977
113/469 [======>.......................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9976
115/469 [======>.......................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9976
117/469 [======>.......................] - ETA: 8s - loss: 0.0029 - accuracy: 0.9976
120/469 [======>.......................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9976
123/469 [======>.......................] - ETA: 8s - loss: 0.0029 - accuracy: 0.9976
126/469 [=======>......................] - ETA: 8s - loss: 0.0029 - accuracy: 0.9976
129/469 [=======>......................] - ETA: 8s - loss: 0.0029 - accuracy: 0.9976
132/469 [=======>......................] - ETA: 8s - loss: 0.0028 - accuracy: 0.9976
135/469 [=======>......................] - ETA: 8s - loss: 0.0028 - accuracy: 0.9977
138/469 [=======>......................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9977
141/469 [========>.....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9978
144/469 [========>.....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9978
147/469 [========>.....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9978
150/469 [========>.....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9978
153/469 [========>.....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9979
156/469 [========>.....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9978
159/469 [=========>....................] - ETA: 7s - loss: 0.0027 - accuracy: 0.9978
162/469 [=========>....................] - ETA: 7s - loss: 0.0027 - accuracy: 0.9978
165/469 [=========>....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9977
168/469 [=========>....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9977
171/469 [=========>....................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9977
174/469 [==========>...................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9977
177/469 [==========>...................] - ETA: 7s - loss: 0.0028 - accuracy: 0.9977
180/469 [==========>...................] - ETA: 6s - loss: 0.0028 - accuracy: 0.9977
183/469 [==========>...................] - ETA: 6s - loss: 0.0028 - accuracy: 0.9978
186/469 [==========>...................] - ETA: 6s - loss: 0.0028 - accuracy: 0.9978
189/469 [===========>..................] - ETA: 6s - loss: 0.0027 - accuracy: 0.9978
192/469 [===========>..................] - ETA: 6s - loss: 0.0028 - accuracy: 0.9978
195/469 [===========>..................] - ETA: 6s - loss: 0.0028 - accuracy: 0.9978
198/469 [===========>..................] - ETA: 6s - loss: 0.0028 - accuracy: 0.9978
201/469 [===========>..................] - ETA: 6s - loss: 0.0028 - accuracy: 0.9979
204/469 [============>.................] - ETA: 6s - loss: 0.0027 - accuracy: 0.9979
207/469 [============>.................] - ETA: 6s - loss: 0.0027 - accuracy: 0.9979
210/469 [============>.................] - ETA: 6s - loss: 0.0027 - accuracy: 0.9979
213/469 [============>.................] - ETA: 6s - loss: 0.0027 - accuracy: 0.9979
216/469 [============>.................] - ETA: 6s - loss: 0.0027 - accuracy: 0.9979
219/469 [=============>................] - ETA: 5s - loss: 0.0027 - accuracy: 0.9980
222/469 [=============>................] - ETA: 5s - loss: 0.0027 - accuracy: 0.9979
225/469 [=============>................] - ETA: 5s - loss: 0.0027 - accuracy: 0.9979
228/469 [=============>................] - ETA: 5s - loss: 0.0027 - accuracy: 0.9978
231/469 [=============>................] - ETA: 5s - loss: 0.0027 - accuracy: 0.9978
234/469 [=============>................] - ETA: 5s - loss: 0.0027 - accuracy: 0.9978
237/469 [==============>...............] - ETA: 5s - loss: 0.0027 - accuracy: 0.9978
240/469 [==============>...............] - ETA: 5s - loss: 0.0027 - accuracy: 0.9979
243/469 [==============>...............] - ETA: 5s - loss: 0.0027 - accuracy: 0.9979
246/469 [==============>...............] - ETA: 5s - loss: 0.0027 - accuracy: 0.9979
249/469 [==============>...............] - ETA: 5s - loss: 0.0027 - accuracy: 0.9979
252/469 [===============>..............] - ETA: 5s - loss: 0.0027 - accuracy: 0.9979
255/469 [===============>..............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9979
258/469 [===============>..............] - ETA: 5s - loss: 0.0027 - accuracy: 0.9979
261/469 [===============>..............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
264/469 [===============>..............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
267/469 [================>.............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
270/469 [================>.............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
273/469 [================>.............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
276/469 [================>.............] - ETA: 4s - loss: 0.0027 - accuracy: 0.9979
279/469 [================>.............] - ETA: 4s - loss: 0.0027 - accuracy: 0.9979
282/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
285/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
288/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
291/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
294/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
297/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
300/469 [==================>...........] - ETA: 4s - loss: 0.0026 - accuracy: 0.9979
303/469 [==================>...........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9979
306/469 [==================>...........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9980
309/469 [==================>...........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9980
312/469 [==================>...........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9980
315/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9979
318/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9980
321/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9980
324/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9980
327/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9979
329/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9979
331/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9979
333/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9979
335/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9979
337/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9979
339/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9978
341/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9978
343/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9978
345/469 [=====================>........] - ETA: 2s - loss: 0.0026 - accuracy: 0.9978
348/469 [=====================>........] - ETA: 2s - loss: 0.0026 - accuracy: 0.9978
351/469 [=====================>........] - ETA: 2s - loss: 0.0026 - accuracy: 0.9978
354/469 [=====================>........] - ETA: 2s - loss: 0.0027 - accuracy: 0.9978
357/469 [=====================>........] - ETA: 2s - loss: 0.0027 - accuracy: 0.9978
360/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9978
363/469 [======================>.......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9978
366/469 [======================>.......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9978
369/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9978
372/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9978
375/469 [======================>.......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9978
378/469 [=======================>......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9978
381/469 [=======================>......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9978
384/469 [=======================>......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9977
387/469 [=======================>......] - ETA: 1s - loss: 0.0027 - accuracy: 0.9977
389/469 [=======================>......] - ETA: 1s - loss: 0.0027 - accuracy: 0.9977
392/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
395/469 [========================>.....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9977
398/469 [========================>.....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9977
401/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9977
404/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
407/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
410/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
413/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
416/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
419/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
422/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
425/469 [==========================>...] - ETA: 1s - loss: 0.0028 - accuracy: 0.9976
428/469 [==========================>...] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
431/469 [==========================>...] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
434/469 [==========================>...] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
437/469 [==========================>...] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
440/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
443/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
446/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
449/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9977
452/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
455/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
458/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
461/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9975
464/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9975
467/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9976
469/469 [==============================] - 12s 25ms/step - loss: 0.0028 - accuracy: 0.9976 - val_loss: 0.0065 - val_accuracy: 0.9920 - lr: 3.9811e-04
Epoch 4/5

  1/469 [..............................] - ETA: 0s - loss: 0.0028 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 0.0012 - accuracy: 1.0000
  7/469 [..............................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9989
 10/469 [..............................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9992
 13/469 [..............................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9994
 16/469 [>.............................] - ETA: 10s - loss: 0.0015 - accuracy: 0.9990
 19/469 [>.............................] - ETA: 10s - loss: 0.0015 - accuracy: 0.9992
 22/469 [>.............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9993
 25/469 [>.............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9994
 28/469 [>.............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9994
 31/469 [>.............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9992
 34/469 [=>............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9991
 37/469 [=>............................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9992 
 40/469 [=>............................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9992
 43/469 [=>............................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9991
 46/469 [=>............................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9992
 49/469 [==>...........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 52/469 [==>...........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 55/469 [==>...........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9993
 58/469 [==>...........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9993
 61/469 [==>...........................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9991
 64/469 [===>..........................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9991
 67/469 [===>..........................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9992
 69/469 [===>..........................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9992
 71/469 [===>..........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 73/469 [===>..........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9993
 75/469 [===>..........................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9992
 77/469 [===>..........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 79/469 [====>.........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 81/469 [====>.........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 83/469 [====>.........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 86/469 [====>.........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9993
 89/469 [====>.........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 92/469 [====>.........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9992
 95/469 [=====>........................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9993
 98/469 [=====>........................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
101/469 [=====>........................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9991
104/469 [=====>........................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
107/469 [=====>........................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
110/469 [======>.......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9991
113/469 [======>.......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
116/469 [======>.......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
119/469 [======>.......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9991
122/469 [======>.......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
125/469 [======>.......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
128/469 [=======>......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
131/469 [=======>......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
134/469 [=======>......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9992
137/469 [=======>......................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9992
140/469 [=======>......................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9990
143/469 [========>.....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9990
146/469 [========>.....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9990
149/469 [========>.....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
152/469 [========>.....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
155/469 [========>.....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9991
158/469 [=========>....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
161/469 [=========>....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
164/469 [=========>....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
167/469 [=========>....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
170/469 [=========>....................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
173/469 [==========>...................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
176/469 [==========>...................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9991
179/469 [==========>...................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
182/469 [==========>...................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9991
185/469 [==========>...................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9992
188/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9991
191/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9991
194/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9991
197/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9991
200/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9991
203/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9992
206/469 [============>.................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9992
209/469 [============>.................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9992
212/469 [============>.................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9992
215/469 [============>.................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9992
218/469 [============>.................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9992
221/469 [=============>................] - ETA: 5s - loss: 0.0013 - accuracy: 0.9992
224/469 [=============>................] - ETA: 5s - loss: 0.0013 - accuracy: 0.9992
227/469 [=============>................] - ETA: 5s - loss: 0.0014 - accuracy: 0.9992
230/469 [=============>................] - ETA: 5s - loss: 0.0014 - accuracy: 0.9992
233/469 [=============>................] - ETA: 5s - loss: 0.0013 - accuracy: 0.9992
236/469 [==============>...............] - ETA: 5s - loss: 0.0014 - accuracy: 0.9991
239/469 [==============>...............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9992
242/469 [==============>...............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9992
245/469 [==============>...............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9992
248/469 [==============>...............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9992
251/469 [===============>..............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9992
254/469 [===============>..............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9991
257/469 [===============>..............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9991
260/469 [===============>..............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9991
263/469 [===============>..............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
266/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
269/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
272/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
275/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
278/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
281/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
284/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
287/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
290/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
293/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
295/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
297/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
299/469 [==================>...........] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
301/469 [==================>...........] - ETA: 4s - loss: 0.0013 - accuracy: 0.9991
303/469 [==================>...........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
305/469 [==================>...........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
307/469 [==================>...........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
309/469 [==================>...........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
312/469 [==================>...........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
315/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
318/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
321/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
324/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
327/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
330/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9991
333/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9992
336/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9992
339/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9992
342/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9992
345/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
348/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
351/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
354/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
357/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
360/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
363/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
366/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
369/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
372/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
375/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
378/469 [=======================>......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9992
381/469 [=======================>......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
384/469 [=======================>......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9991
387/469 [=======================>......] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
390/469 [=======================>......] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
393/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
396/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
399/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
402/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
405/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
408/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
411/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
414/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
417/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
420/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
423/469 [==========================>...] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
426/469 [==========================>...] - ETA: 1s - loss: 0.0013 - accuracy: 0.9992
429/469 [==========================>...] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
432/469 [==========================>...] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
435/469 [==========================>...] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
438/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
441/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
444/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
447/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
450/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9991
453/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
456/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9991
459/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9991
462/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
465/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
468/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9992
469/469 [==============================] - 12s 25ms/step - loss: 0.0013 - accuracy: 0.9992 - val_loss: 0.0055 - val_accuracy: 0.9936 - lr: 2.5119e-04
Epoch 5/5

  1/469 [..............................] - ETA: 0s - loss: 3.4480e-05 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 4.3297e-04 - accuracy: 1.0000
  7/469 [..............................] - ETA: 9s - loss: 8.6377e-04 - accuracy: 1.0000
 10/469 [..............................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9984    
 13/469 [..............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9988
 16/469 [>.............................] - ETA: 10s - loss: 0.0011 - accuracy: 0.9990
 19/469 [>.............................] - ETA: 10s - loss: 0.0011 - accuracy: 0.9992
 22/469 [>.............................] - ETA: 10s - loss: 0.0011 - accuracy: 0.9993
 25/469 [>.............................] - ETA: 10s - loss: 0.0011 - accuracy: 0.9994
 28/469 [>.............................] - ETA: 10s - loss: 9.8207e-04 - accuracy: 0.9994
 31/469 [>.............................] - ETA: 10s - loss: 9.8068e-04 - accuracy: 0.9995
 33/469 [=>............................] - ETA: 10s - loss: 9.3531e-04 - accuracy: 0.9995
 35/469 [=>............................] - ETA: 10s - loss: 8.9825e-04 - accuracy: 0.9996
 37/469 [=>............................] - ETA: 10s - loss: 8.6097e-04 - accuracy: 0.9996
 39/469 [=>............................] - ETA: 10s - loss: 8.3200e-04 - accuracy: 0.9996
 41/469 [=>............................] - ETA: 10s - loss: 8.0358e-04 - accuracy: 0.9996
 43/469 [=>............................] - ETA: 10s - loss: 7.6700e-04 - accuracy: 0.9996
 45/469 [=>............................] - ETA: 10s - loss: 7.3563e-04 - accuracy: 0.9997
 47/469 [==>...........................] - ETA: 10s - loss: 7.2027e-04 - accuracy: 0.9997
 50/469 [==>...........................] - ETA: 10s - loss: 7.0804e-04 - accuracy: 0.9997
 53/469 [==>...........................] - ETA: 10s - loss: 8.4489e-04 - accuracy: 0.9994
 56/469 [==>...........................] - ETA: 10s - loss: 8.5795e-04 - accuracy: 0.9994
 59/469 [==>...........................] - ETA: 10s - loss: 8.3176e-04 - accuracy: 0.9995
 62/469 [==>...........................] - ETA: 9s - loss: 8.3841e-04 - accuracy: 0.9995 
 65/469 [===>..........................] - ETA: 9s - loss: 8.3327e-04 - accuracy: 0.9995
 68/469 [===>..........................] - ETA: 9s - loss: 8.1761e-04 - accuracy: 0.9995
 71/469 [===>..........................] - ETA: 9s - loss: 8.0423e-04 - accuracy: 0.9996
 74/469 [===>..........................] - ETA: 9s - loss: 7.7612e-04 - accuracy: 0.9996
 77/469 [===>..........................] - ETA: 9s - loss: 7.8002e-04 - accuracy: 0.9996
 80/469 [====>.........................] - ETA: 9s - loss: 7.6271e-04 - accuracy: 0.9996
 83/469 [====>.........................] - ETA: 9s - loss: 7.5840e-04 - accuracy: 0.9996
 86/469 [====>.........................] - ETA: 9s - loss: 7.5167e-04 - accuracy: 0.9996
 89/469 [====>.........................] - ETA: 9s - loss: 7.6103e-04 - accuracy: 0.9996
 92/469 [====>.........................] - ETA: 9s - loss: 7.5848e-04 - accuracy: 0.9997
 95/469 [=====>........................] - ETA: 9s - loss: 7.4671e-04 - accuracy: 0.9997
 98/469 [=====>........................] - ETA: 8s - loss: 7.4579e-04 - accuracy: 0.9997
101/469 [=====>........................] - ETA: 8s - loss: 7.6697e-04 - accuracy: 0.9997
104/469 [=====>........................] - ETA: 8s - loss: 7.5540e-04 - accuracy: 0.9997
107/469 [=====>........................] - ETA: 8s - loss: 7.5053e-04 - accuracy: 0.9997
110/469 [======>.......................] - ETA: 8s - loss: 7.4574e-04 - accuracy: 0.9997
113/469 [======>.......................] - ETA: 8s - loss: 7.3405e-04 - accuracy: 0.9997
116/469 [======>.......................] - ETA: 8s - loss: 7.3741e-04 - accuracy: 0.9997
119/469 [======>.......................] - ETA: 8s - loss: 7.1999e-04 - accuracy: 0.9997
122/469 [======>.......................] - ETA: 8s - loss: 7.1785e-04 - accuracy: 0.9997
125/469 [======>.......................] - ETA: 8s - loss: 7.1468e-04 - accuracy: 0.9998
128/469 [=======>......................] - ETA: 8s - loss: 7.0790e-04 - accuracy: 0.9998
131/469 [=======>......................] - ETA: 8s - loss: 7.1067e-04 - accuracy: 0.9997
134/469 [=======>......................] - ETA: 8s - loss: 7.1672e-04 - accuracy: 0.9997
137/469 [=======>......................] - ETA: 7s - loss: 7.1332e-04 - accuracy: 0.9997
140/469 [=======>......................] - ETA: 7s - loss: 7.1051e-04 - accuracy: 0.9997
143/469 [========>.....................] - ETA: 7s - loss: 6.9945e-04 - accuracy: 0.9997
146/469 [========>.....................] - ETA: 7s - loss: 7.0049e-04 - accuracy: 0.9997
149/469 [========>.....................] - ETA: 7s - loss: 6.9092e-04 - accuracy: 0.9997
152/469 [========>.....................] - ETA: 7s - loss: 6.8699e-04 - accuracy: 0.9997
155/469 [========>.....................] - ETA: 7s - loss: 6.8416e-04 - accuracy: 0.9997
158/469 [=========>....................] - ETA: 7s - loss: 6.9852e-04 - accuracy: 0.9998
161/469 [=========>....................] - ETA: 7s - loss: 6.9135e-04 - accuracy: 0.9998
164/469 [=========>....................] - ETA: 7s - loss: 7.0218e-04 - accuracy: 0.9998
167/469 [=========>....................] - ETA: 7s - loss: 7.0148e-04 - accuracy: 0.9998
170/469 [=========>....................] - ETA: 7s - loss: 7.0800e-04 - accuracy: 0.9998
173/469 [==========>...................] - ETA: 7s - loss: 7.0320e-04 - accuracy: 0.9998
176/469 [==========>...................] - ETA: 7s - loss: 6.9318e-04 - accuracy: 0.9998
179/469 [==========>...................] - ETA: 6s - loss: 7.0319e-04 - accuracy: 0.9998
182/469 [==========>...................] - ETA: 6s - loss: 7.1851e-04 - accuracy: 0.9997
185/469 [==========>...................] - ETA: 6s - loss: 7.3390e-04 - accuracy: 0.9997
188/469 [===========>..................] - ETA: 6s - loss: 7.3234e-04 - accuracy: 0.9997
191/469 [===========>..................] - ETA: 6s - loss: 7.2992e-04 - accuracy: 0.9997
194/469 [===========>..................] - ETA: 6s - loss: 7.2441e-04 - accuracy: 0.9997
197/469 [===========>..................] - ETA: 6s - loss: 7.1771e-04 - accuracy: 0.9997
200/469 [===========>..................] - ETA: 6s - loss: 7.1193e-04 - accuracy: 0.9997
203/469 [===========>..................] - ETA: 6s - loss: 7.0690e-04 - accuracy: 0.9997
206/469 [============>.................] - ETA: 6s - loss: 7.0206e-04 - accuracy: 0.9997
209/469 [============>.................] - ETA: 6s - loss: 6.9620e-04 - accuracy: 0.9997
212/469 [============>.................] - ETA: 6s - loss: 6.9721e-04 - accuracy: 0.9997
215/469 [============>.................] - ETA: 6s - loss: 6.9641e-04 - accuracy: 0.9997
218/469 [============>.................] - ETA: 6s - loss: 6.8950e-04 - accuracy: 0.9997
221/469 [=============>................] - ETA: 5s - loss: 6.8202e-04 - accuracy: 0.9998
224/469 [=============>................] - ETA: 5s - loss: 6.7576e-04 - accuracy: 0.9998
227/469 [=============>................] - ETA: 5s - loss: 6.7227e-04 - accuracy: 0.9998
230/469 [=============>................] - ETA: 5s - loss: 6.6825e-04 - accuracy: 0.9998
233/469 [=============>................] - ETA: 5s - loss: 6.6321e-04 - accuracy: 0.9998
236/469 [==============>...............] - ETA: 5s - loss: 6.5826e-04 - accuracy: 0.9998
239/469 [==============>...............] - ETA: 5s - loss: 6.5685e-04 - accuracy: 0.9998
242/469 [==============>...............] - ETA: 5s - loss: 6.5890e-04 - accuracy: 0.9998
245/469 [==============>...............] - ETA: 5s - loss: 6.5922e-04 - accuracy: 0.9998
248/469 [==============>...............] - ETA: 5s - loss: 6.5515e-04 - accuracy: 0.9998
251/469 [===============>..............] - ETA: 5s - loss: 6.5259e-04 - accuracy: 0.9998
254/469 [===============>..............] - ETA: 5s - loss: 6.4830e-04 - accuracy: 0.9998
257/469 [===============>..............] - ETA: 5s - loss: 6.4870e-04 - accuracy: 0.9998
260/469 [===============>..............] - ETA: 5s - loss: 6.4889e-04 - accuracy: 0.9998
262/469 [===============>..............] - ETA: 4s - loss: 6.4779e-04 - accuracy: 0.9998
264/469 [===============>..............] - ETA: 4s - loss: 6.4636e-04 - accuracy: 0.9998
266/469 [================>.............] - ETA: 4s - loss: 6.4596e-04 - accuracy: 0.9998
268/469 [================>.............] - ETA: 4s - loss: 6.4497e-04 - accuracy: 0.9998
270/469 [================>.............] - ETA: 4s - loss: 6.4457e-04 - accuracy: 0.9998
272/469 [================>.............] - ETA: 4s - loss: 6.4319e-04 - accuracy: 0.9998
274/469 [================>.............] - ETA: 4s - loss: 6.4226e-04 - accuracy: 0.9998
277/469 [================>.............] - ETA: 4s - loss: 6.3756e-04 - accuracy: 0.9998
280/469 [================>.............] - ETA: 4s - loss: 6.4071e-04 - accuracy: 0.9998
283/469 [=================>............] - ETA: 4s - loss: 6.4229e-04 - accuracy: 0.9998
286/469 [=================>............] - ETA: 4s - loss: 6.4295e-04 - accuracy: 0.9998
289/469 [=================>............] - ETA: 4s - loss: 6.4792e-04 - accuracy: 0.9998
292/469 [=================>............] - ETA: 4s - loss: 6.4387e-04 - accuracy: 0.9998
295/469 [=================>............] - ETA: 4s - loss: 6.3899e-04 - accuracy: 0.9998
298/469 [==================>...........] - ETA: 4s - loss: 6.3810e-04 - accuracy: 0.9998
301/469 [==================>...........] - ETA: 4s - loss: 6.3585e-04 - accuracy: 0.9998
304/469 [==================>...........] - ETA: 3s - loss: 6.3490e-04 - accuracy: 0.9998
307/469 [==================>...........] - ETA: 3s - loss: 6.3088e-04 - accuracy: 0.9998
310/469 [==================>...........] - ETA: 3s - loss: 6.3149e-04 - accuracy: 0.9998
313/469 [===================>..........] - ETA: 3s - loss: 6.3425e-04 - accuracy: 0.9998
316/469 [===================>..........] - ETA: 3s - loss: 6.2959e-04 - accuracy: 0.9998
319/469 [===================>..........] - ETA: 3s - loss: 6.2618e-04 - accuracy: 0.9998
322/469 [===================>..........] - ETA: 3s - loss: 6.2963e-04 - accuracy: 0.9998
325/469 [===================>..........] - ETA: 3s - loss: 6.2864e-04 - accuracy: 0.9998
328/469 [===================>..........] - ETA: 3s - loss: 6.2553e-04 - accuracy: 0.9998
331/469 [====================>.........] - ETA: 3s - loss: 6.2560e-04 - accuracy: 0.9998
334/469 [====================>.........] - ETA: 3s - loss: 6.2502e-04 - accuracy: 0.9998
337/469 [====================>.........] - ETA: 3s - loss: 6.2053e-04 - accuracy: 0.9998
340/469 [====================>.........] - ETA: 3s - loss: 6.2199e-04 - accuracy: 0.9998
343/469 [====================>.........] - ETA: 3s - loss: 6.2924e-04 - accuracy: 0.9998
346/469 [=====================>........] - ETA: 2s - loss: 6.2741e-04 - accuracy: 0.9998
349/469 [=====================>........] - ETA: 2s - loss: 6.2700e-04 - accuracy: 0.9998
352/469 [=====================>........] - ETA: 2s - loss: 6.2745e-04 - accuracy: 0.9998
355/469 [=====================>........] - ETA: 2s - loss: 6.2347e-04 - accuracy: 0.9998
358/469 [=====================>........] - ETA: 2s - loss: 6.2147e-04 - accuracy: 0.9998
361/469 [======================>.......] - ETA: 2s - loss: 6.1856e-04 - accuracy: 0.9998
364/469 [======================>.......] - ETA: 2s - loss: 6.2604e-04 - accuracy: 0.9998
367/469 [======================>.......] - ETA: 2s - loss: 6.3809e-04 - accuracy: 0.9998
370/469 [======================>.......] - ETA: 2s - loss: 6.3656e-04 - accuracy: 0.9998
373/469 [======================>.......] - ETA: 2s - loss: 6.3661e-04 - accuracy: 0.9998
376/469 [=======================>......] - ETA: 2s - loss: 6.4715e-04 - accuracy: 0.9998
379/469 [=======================>......] - ETA: 2s - loss: 6.4644e-04 - accuracy: 0.9998
382/469 [=======================>......] - ETA: 2s - loss: 6.4536e-04 - accuracy: 0.9998
385/469 [=======================>......] - ETA: 2s - loss: 6.4305e-04 - accuracy: 0.9998
388/469 [=======================>......] - ETA: 1s - loss: 6.5054e-04 - accuracy: 0.9998
391/469 [========================>.....] - ETA: 1s - loss: 6.5489e-04 - accuracy: 0.9998
394/469 [========================>.....] - ETA: 1s - loss: 6.5260e-04 - accuracy: 0.9998
397/469 [========================>.....] - ETA: 1s - loss: 6.4890e-04 - accuracy: 0.9998
400/469 [========================>.....] - ETA: 1s - loss: 6.4651e-04 - accuracy: 0.9998
403/469 [========================>.....] - ETA: 1s - loss: 6.4515e-04 - accuracy: 0.9998
406/469 [========================>.....] - ETA: 1s - loss: 6.4935e-04 - accuracy: 0.9997
409/469 [=========================>....] - ETA: 1s - loss: 6.5037e-04 - accuracy: 0.9998
412/469 [=========================>....] - ETA: 1s - loss: 6.5697e-04 - accuracy: 0.9998
415/469 [=========================>....] - ETA: 1s - loss: 6.5483e-04 - accuracy: 0.9998
418/469 [=========================>....] - ETA: 1s - loss: 6.5238e-04 - accuracy: 0.9998
421/469 [=========================>....] - ETA: 1s - loss: 6.5680e-04 - accuracy: 0.9998
424/469 [==========================>...] - ETA: 1s - loss: 6.5435e-04 - accuracy: 0.9998
427/469 [==========================>...] - ETA: 1s - loss: 6.5277e-04 - accuracy: 0.9998
430/469 [==========================>...] - ETA: 0s - loss: 6.5053e-04 - accuracy: 0.9998
433/469 [==========================>...] - ETA: 0s - loss: 6.5009e-04 - accuracy: 0.9998
436/469 [==========================>...] - ETA: 0s - loss: 6.4986e-04 - accuracy: 0.9998
439/469 [===========================>..] - ETA: 0s - loss: 6.4622e-04 - accuracy: 0.9998
442/469 [===========================>..] - ETA: 0s - loss: 6.4829e-04 - accuracy: 0.9998
445/469 [===========================>..] - ETA: 0s - loss: 6.4871e-04 - accuracy: 0.9998
448/469 [===========================>..] - ETA: 0s - loss: 6.6277e-04 - accuracy: 0.9997
451/469 [===========================>..] - ETA: 0s - loss: 6.5997e-04 - accuracy: 0.9997
454/469 [============================>.] - ETA: 0s - loss: 6.5636e-04 - accuracy: 0.9997
457/469 [============================>.] - ETA: 0s - loss: 6.5603e-04 - accuracy: 0.9997
460/469 [============================>.] - ETA: 0s - loss: 6.5547e-04 - accuracy: 0.9997
463/469 [============================>.] - ETA: 0s - loss: 6.5581e-04 - accuracy: 0.9997
466/469 [============================>.] - ETA: 0s - loss: 6.5830e-04 - accuracy: 0.9997
469/469 [==============================] - ETA: 0s - loss: 6.5584e-04 - accuracy: 0.9997
469/469 [==============================] - 12s 25ms/step - loss: 6.5584e-04 - accuracy: 0.9997 - val_loss: 0.0055 - val_accuracy: 0.9932 - lr: 1.5849e-04
Test score: 0.00546363927423954
Test accuracy: 0.9932000041007996

3. Model Akida-compatibility check and changes

3.1 Compatibility check

The first step is to ensure that the model as defined doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):

  • Standard Conv2D and Dense layers are supported (note that there is currently no support for skip, recursive and parallel layers).

  • Each of these trainable core layers except for the last one must be followed by an Activation layer.

  • All blocks can optionally include a BatchNormalization layer.

  • Convolutional blocks can optionally include a MaxPooling type layer.

Note

This configuration of layers (Conv/Dense + BatchNormalization + Activation) constitutes the basic building block of Akida-compatible models and is widely used in deep learning.

If the model defined is not fully compatible with the Akida NSoC, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).

3.2 Model adaptation

As noted above, the basic building blocks of Akida compatible models actually comprise a trio of layers: Conv/Dense + BatchNormalization + Activation (with, optionally, pooling). The CNN2SNN tool provides a set of functions that simplify using these building blocks, and subsequently enable easy application of Brainchip’s custom quantization functions.

from akida_models.quantization_blocks import conv_block, dense_block

The following code illustrates how to express the MNIST model defined above using the functions provided by Brainchip. A couple of points to avoid confusion when you look through it:

  • The weight_quantization in each block isn’t used here, but will be used later to apply a quantization method to the model weights.

  • The block_id is just used for naming the layers, and will be good practice in enabling reloading of partially trained models in more advanced training cases.

  • Note that in the final block, we set the nonlinearity activ_quantization to None. In that case, the block has no Activation layer, and the output is simply the output from the BatchNormalization layer.

# Removes all the nodes left over from the previous model and free memory
K.clear_session()

# Define the model.
# The commented code shows the sets of layers in the original definition
# that are being replaced by the provided conv_block and dense_blocks here

img_input = Input(shape=(28, 28, 1))

# x = Conv2D(filters=32,
#            kernel_size=(5, 5),
#            padding='same',
#            use_bias=False,
#            data_format='channels_last')(img_input)
# x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(x)
# x = BatchNormalization()(x)
# x = ReLU(6.)(x)
x = conv_block(img_input,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_0',
               pooling='max',
               add_batchnorm=True)

# x = Conv2D(filters=32,
#            kernel_size=(5, 5),
#            padding='same',
#            use_bias=False)(x)
# x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(x)
# x = BatchNormalization()(x)
# x = ReLU(6.)(x)
x = conv_block(x,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_1',
               pooling='max',
               add_batchnorm=True)

# x = Conv2D(filters=32,
#            kernel_size=(5, 5),
#            padding='same',
#            use_bias=False)(x)
# x = BatchNormalization()(x)
# x = ReLU(6.)(x)
x = conv_block(x,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_2',
               add_batchnorm=True)

x = Flatten()(x)

# x = Dense(512,
#           use_bias=False)(x)
# x = BatchNormalization()(x)
# x = ReLU(6.)(x)
x = dense_block(x,
                units=512,
                use_bias=False,
                name='dense_2',
                add_batchnorm=True)

# x = Dense(10,
#           use_bias=False)(x)
# x = BatchNormalization()(x)
x = dense_block(x,
                units=10,
                use_bias=False,
                name='dense_3',
                activ_quantization=None)

model_keras = Model(img_input, x, name='mnistnet')

opt = Adam(lr=lr_start)
model_keras.compile(loss='squared_hinge', optimizer=opt, metrics=['accuracy'])
model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(None, 28, 28, 1)]       0
_________________________________________________________________
conv_0 (QuantizedConv2D)     (None, 28, 28, 32)        800
_________________________________________________________________
conv_0_maxpool (MaxPooling2D (None, 14, 14, 32)        0
_________________________________________________________________
conv_0_BN (BatchNormalizatio (None, 14, 14, 32)        128
_________________________________________________________________
conv_0_relu (ReLU)           (None, 14, 14, 32)        0
_________________________________________________________________
conv_1 (QuantizedConv2D)     (None, 14, 14, 32)        25600
_________________________________________________________________
conv_1_maxpool (MaxPooling2D (None, 7, 7, 32)          0
_________________________________________________________________
conv_1_BN (BatchNormalizatio (None, 7, 7, 32)          128
_________________________________________________________________
conv_1_relu (ReLU)           (None, 7, 7, 32)          0
_________________________________________________________________
conv_2 (QuantizedConv2D)     (None, 7, 7, 32)          25600
_________________________________________________________________
conv_2_BN (BatchNormalizatio (None, 7, 7, 32)          128
_________________________________________________________________
conv_2_relu (ReLU)           (None, 7, 7, 32)          0
_________________________________________________________________
flatten (Flatten)            (None, 1568)              0
_________________________________________________________________
dense_2 (QuantizedDense)     (None, 512)               802816
_________________________________________________________________
dense_2_BN (BatchNormalizati (None, 512)               2048
_________________________________________________________________
dense_2_relu (ReLU)          (None, 512)               0
_________________________________________________________________
dense_3 (QuantizedDense)     (None, 10)                5120
=================================================================
Total params: 862,368
Trainable params: 861,152
Non-trainable params: 1,216
_________________________________________________________________

3.3 Performance check

Check modifed model performance:

callbacks = []
lr_scheduler = LearningRateScheduler(lambda e: lr_start * lr_decay**e)
callbacks.append(lr_scheduler)
history = model_keras.fit(x_train,
                          y_train,
                          batch_size=batch_size,
                          epochs=epochs,
                          verbose=1,
                          validation_data=(x_test, y_test),
                          callbacks=callbacks)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/5

  1/469 [..............................] - ETA: 0s - loss: 2.1788 - accuracy: 0.0547
  4/469 [..............................] - ETA: 8s - loss: 1.3871 - accuracy: 0.4590
  7/469 [..............................] - ETA: 9s - loss: 0.9733 - accuracy: 0.5882
 10/469 [..............................] - ETA: 9s - loss: 0.7368 - accuracy: 0.6672
 13/469 [..............................] - ETA: 10s - loss: 0.5927 - accuracy: 0.7145
 15/469 [..............................] - ETA: 10s - loss: 0.5239 - accuracy: 0.7406
 18/469 [>.............................] - ETA: 10s - loss: 0.4475 - accuracy: 0.7743
 21/469 [>.............................] - ETA: 10s - loss: 0.3907 - accuracy: 0.7980
 24/469 [>.............................] - ETA: 10s - loss: 0.3495 - accuracy: 0.8138
 27/469 [>.............................] - ETA: 10s - loss: 0.3159 - accuracy: 0.8290
 30/469 [>.............................] - ETA: 10s - loss: 0.2888 - accuracy: 0.8419
 33/469 [=>............................] - ETA: 10s - loss: 0.2667 - accuracy: 0.8513
 36/469 [=>............................] - ETA: 10s - loss: 0.2476 - accuracy: 0.8594
 39/469 [=>............................] - ETA: 10s - loss: 0.2318 - accuracy: 0.8662
 42/469 [=>............................] - ETA: 9s - loss: 0.2175 - accuracy: 0.8739 
 45/469 [=>............................] - ETA: 9s - loss: 0.2058 - accuracy: 0.8788
 48/469 [==>...........................] - ETA: 9s - loss: 0.1945 - accuracy: 0.8848
 51/469 [==>...........................] - ETA: 9s - loss: 0.1853 - accuracy: 0.8897
 54/469 [==>...........................] - ETA: 9s - loss: 0.1767 - accuracy: 0.8940
 57/469 [==>...........................] - ETA: 9s - loss: 0.1689 - accuracy: 0.8980
 60/469 [==>...........................] - ETA: 9s - loss: 0.1617 - accuracy: 0.9016
 63/469 [===>..........................] - ETA: 9s - loss: 0.1553 - accuracy: 0.9054
 66/469 [===>..........................] - ETA: 9s - loss: 0.1494 - accuracy: 0.9087
 69/469 [===>..........................] - ETA: 9s - loss: 0.1451 - accuracy: 0.9107
 72/469 [===>..........................] - ETA: 9s - loss: 0.1402 - accuracy: 0.9134
 75/469 [===>..........................] - ETA: 9s - loss: 0.1362 - accuracy: 0.9152
 78/469 [===>..........................] - ETA: 9s - loss: 0.1320 - accuracy: 0.9177
 81/469 [====>.........................] - ETA: 9s - loss: 0.1278 - accuracy: 0.9198
 84/469 [====>.........................] - ETA: 9s - loss: 0.1244 - accuracy: 0.9212
 87/469 [====>.........................] - ETA: 9s - loss: 0.1211 - accuracy: 0.9230
 90/469 [====>.........................] - ETA: 8s - loss: 0.1180 - accuracy: 0.9245
 93/469 [====>.........................] - ETA: 8s - loss: 0.1151 - accuracy: 0.9259
 96/469 [=====>........................] - ETA: 8s - loss: 0.1123 - accuracy: 0.9273
 99/469 [=====>........................] - ETA: 8s - loss: 0.1099 - accuracy: 0.9283
102/469 [=====>........................] - ETA: 8s - loss: 0.1075 - accuracy: 0.9295
105/469 [=====>........................] - ETA: 8s - loss: 0.1049 - accuracy: 0.9312
108/469 [=====>........................] - ETA: 8s - loss: 0.1025 - accuracy: 0.9326
111/469 [======>.......................] - ETA: 8s - loss: 0.1006 - accuracy: 0.9336
114/469 [======>.......................] - ETA: 8s - loss: 0.0987 - accuracy: 0.9348
117/469 [======>.......................] - ETA: 8s - loss: 0.0966 - accuracy: 0.9358
120/469 [======>.......................] - ETA: 8s - loss: 0.0950 - accuracy: 0.9366
123/469 [======>.......................] - ETA: 8s - loss: 0.0932 - accuracy: 0.9376
126/469 [=======>......................] - ETA: 8s - loss: 0.0915 - accuracy: 0.9386
129/469 [=======>......................] - ETA: 8s - loss: 0.0899 - accuracy: 0.9395
132/469 [=======>......................] - ETA: 8s - loss: 0.0884 - accuracy: 0.9402
135/469 [=======>......................] - ETA: 7s - loss: 0.0868 - accuracy: 0.9409
138/469 [=======>......................] - ETA: 7s - loss: 0.0853 - accuracy: 0.9417
141/469 [========>.....................] - ETA: 7s - loss: 0.0839 - accuracy: 0.9425
144/469 [========>.....................] - ETA: 7s - loss: 0.0823 - accuracy: 0.9436
147/469 [========>.....................] - ETA: 7s - loss: 0.0809 - accuracy: 0.9445
150/469 [========>.....................] - ETA: 7s - loss: 0.0797 - accuracy: 0.9449
153/469 [========>.....................] - ETA: 7s - loss: 0.0783 - accuracy: 0.9458
156/469 [========>.....................] - ETA: 7s - loss: 0.0771 - accuracy: 0.9463
158/469 [=========>....................] - ETA: 7s - loss: 0.0763 - accuracy: 0.9469
160/469 [=========>....................] - ETA: 7s - loss: 0.0756 - accuracy: 0.9474
162/469 [=========>....................] - ETA: 7s - loss: 0.0748 - accuracy: 0.9477
164/469 [=========>....................] - ETA: 7s - loss: 0.0742 - accuracy: 0.9480
166/469 [=========>....................] - ETA: 7s - loss: 0.0735 - accuracy: 0.9484
168/469 [=========>....................] - ETA: 7s - loss: 0.0728 - accuracy: 0.9488
170/469 [=========>....................] - ETA: 7s - loss: 0.0721 - accuracy: 0.9493
172/469 [==========>...................] - ETA: 7s - loss: 0.0716 - accuracy: 0.9494
175/469 [==========>...................] - ETA: 7s - loss: 0.0706 - accuracy: 0.9499
178/469 [==========>...................] - ETA: 7s - loss: 0.0697 - accuracy: 0.9504
181/469 [==========>...................] - ETA: 6s - loss: 0.0688 - accuracy: 0.9510
184/469 [==========>...................] - ETA: 6s - loss: 0.0679 - accuracy: 0.9515
187/469 [==========>...................] - ETA: 6s - loss: 0.0672 - accuracy: 0.9519
190/469 [===========>..................] - ETA: 6s - loss: 0.0664 - accuracy: 0.9523
193/469 [===========>..................] - ETA: 6s - loss: 0.0657 - accuracy: 0.9526
196/469 [===========>..................] - ETA: 6s - loss: 0.0649 - accuracy: 0.9531
199/469 [===========>..................] - ETA: 6s - loss: 0.0641 - accuracy: 0.9536
202/469 [===========>..................] - ETA: 6s - loss: 0.0635 - accuracy: 0.9541
205/469 [============>.................] - ETA: 6s - loss: 0.0628 - accuracy: 0.9545
208/469 [============>.................] - ETA: 6s - loss: 0.0621 - accuracy: 0.9549
211/469 [============>.................] - ETA: 6s - loss: 0.0614 - accuracy: 0.9552
214/469 [============>.................] - ETA: 6s - loss: 0.0608 - accuracy: 0.9555
217/469 [============>.................] - ETA: 6s - loss: 0.0602 - accuracy: 0.9559
220/469 [=============>................] - ETA: 5s - loss: 0.0596 - accuracy: 0.9562
223/469 [=============>................] - ETA: 5s - loss: 0.0590 - accuracy: 0.9565
226/469 [=============>................] - ETA: 5s - loss: 0.0585 - accuracy: 0.9567
229/469 [=============>................] - ETA: 5s - loss: 0.0579 - accuracy: 0.9571
232/469 [=============>................] - ETA: 5s - loss: 0.0574 - accuracy: 0.9573
235/469 [==============>...............] - ETA: 5s - loss: 0.0569 - accuracy: 0.9577
238/469 [==============>...............] - ETA: 5s - loss: 0.0564 - accuracy: 0.9580
241/469 [==============>...............] - ETA: 5s - loss: 0.0559 - accuracy: 0.9584
244/469 [==============>...............] - ETA: 5s - loss: 0.0554 - accuracy: 0.9586
247/469 [==============>...............] - ETA: 5s - loss: 0.0550 - accuracy: 0.9588
250/469 [==============>...............] - ETA: 5s - loss: 0.0544 - accuracy: 0.9592
253/469 [===============>..............] - ETA: 5s - loss: 0.0540 - accuracy: 0.9595
256/469 [===============>..............] - ETA: 5s - loss: 0.0536 - accuracy: 0.9596
259/469 [===============>..............] - ETA: 5s - loss: 0.0532 - accuracy: 0.9599
262/469 [===============>..............] - ETA: 4s - loss: 0.0528 - accuracy: 0.9600
265/469 [===============>..............] - ETA: 4s - loss: 0.0523 - accuracy: 0.9603
268/469 [================>.............] - ETA: 4s - loss: 0.0519 - accuracy: 0.9605
271/469 [================>.............] - ETA: 4s - loss: 0.0516 - accuracy: 0.9608
274/469 [================>.............] - ETA: 4s - loss: 0.0512 - accuracy: 0.9609
277/469 [================>.............] - ETA: 4s - loss: 0.0508 - accuracy: 0.9612
280/469 [================>.............] - ETA: 4s - loss: 0.0504 - accuracy: 0.9615
283/469 [=================>............] - ETA: 4s - loss: 0.0500 - accuracy: 0.9618
286/469 [=================>............] - ETA: 4s - loss: 0.0497 - accuracy: 0.9620
289/469 [=================>............] - ETA: 4s - loss: 0.0493 - accuracy: 0.9622
292/469 [=================>............] - ETA: 4s - loss: 0.0489 - accuracy: 0.9625
295/469 [=================>............] - ETA: 4s - loss: 0.0485 - accuracy: 0.9627
298/469 [==================>...........] - ETA: 4s - loss: 0.0482 - accuracy: 0.9630
301/469 [==================>...........] - ETA: 4s - loss: 0.0478 - accuracy: 0.9632
304/469 [==================>...........] - ETA: 3s - loss: 0.0474 - accuracy: 0.9635
307/469 [==================>...........] - ETA: 3s - loss: 0.0471 - accuracy: 0.9638
310/469 [==================>...........] - ETA: 3s - loss: 0.0468 - accuracy: 0.9640
313/469 [===================>..........] - ETA: 3s - loss: 0.0464 - accuracy: 0.9642
316/469 [===================>..........] - ETA: 3s - loss: 0.0461 - accuracy: 0.9644
319/469 [===================>..........] - ETA: 3s - loss: 0.0458 - accuracy: 0.9646
322/469 [===================>..........] - ETA: 3s - loss: 0.0455 - accuracy: 0.9648
325/469 [===================>..........] - ETA: 3s - loss: 0.0452 - accuracy: 0.9650
328/469 [===================>..........] - ETA: 3s - loss: 0.0449 - accuracy: 0.9652
331/469 [====================>.........] - ETA: 3s - loss: 0.0447 - accuracy: 0.9654
334/469 [====================>.........] - ETA: 3s - loss: 0.0444 - accuracy: 0.9655
337/469 [====================>.........] - ETA: 3s - loss: 0.0441 - accuracy: 0.9657
340/469 [====================>.........] - ETA: 3s - loss: 0.0438 - accuracy: 0.9659
343/469 [====================>.........] - ETA: 3s - loss: 0.0436 - accuracy: 0.9660
346/469 [=====================>........] - ETA: 2s - loss: 0.0434 - accuracy: 0.9662
349/469 [=====================>........] - ETA: 2s - loss: 0.0431 - accuracy: 0.9664
352/469 [=====================>........] - ETA: 2s - loss: 0.0428 - accuracy: 0.9666
355/469 [=====================>........] - ETA: 2s - loss: 0.0425 - accuracy: 0.9667
358/469 [=====================>........] - ETA: 2s - loss: 0.0423 - accuracy: 0.9668
361/469 [======================>.......] - ETA: 2s - loss: 0.0420 - accuracy: 0.9670
364/469 [======================>.......] - ETA: 2s - loss: 0.0418 - accuracy: 0.9672
367/469 [======================>.......] - ETA: 2s - loss: 0.0416 - accuracy: 0.9673
370/469 [======================>.......] - ETA: 2s - loss: 0.0413 - accuracy: 0.9675
373/469 [======================>.......] - ETA: 2s - loss: 0.0410 - accuracy: 0.9677
376/469 [=======================>......] - ETA: 2s - loss: 0.0408 - accuracy: 0.9678
379/469 [=======================>......] - ETA: 2s - loss: 0.0406 - accuracy: 0.9679
382/469 [=======================>......] - ETA: 2s - loss: 0.0403 - accuracy: 0.9681
384/469 [=======================>......] - ETA: 2s - loss: 0.0402 - accuracy: 0.9682
386/469 [=======================>......] - ETA: 1s - loss: 0.0400 - accuracy: 0.9684
388/469 [=======================>......] - ETA: 1s - loss: 0.0399 - accuracy: 0.9685
390/469 [=======================>......] - ETA: 1s - loss: 0.0397 - accuracy: 0.9685
392/469 [========================>.....] - ETA: 1s - loss: 0.0396 - accuracy: 0.9687
394/469 [========================>.....] - ETA: 1s - loss: 0.0394 - accuracy: 0.9688
396/469 [========================>.....] - ETA: 1s - loss: 0.0393 - accuracy: 0.9688
398/469 [========================>.....] - ETA: 1s - loss: 0.0392 - accuracy: 0.9689
401/469 [========================>.....] - ETA: 1s - loss: 0.0390 - accuracy: 0.9691
404/469 [========================>.....] - ETA: 1s - loss: 0.0387 - accuracy: 0.9692
407/469 [=========================>....] - ETA: 1s - loss: 0.0386 - accuracy: 0.9693
410/469 [=========================>....] - ETA: 1s - loss: 0.0384 - accuracy: 0.9694
413/469 [=========================>....] - ETA: 1s - loss: 0.0382 - accuracy: 0.9695
416/469 [=========================>....] - ETA: 1s - loss: 0.0380 - accuracy: 0.9697
419/469 [=========================>....] - ETA: 1s - loss: 0.0378 - accuracy: 0.9699
422/469 [=========================>....] - ETA: 1s - loss: 0.0376 - accuracy: 0.9700
425/469 [==========================>...] - ETA: 1s - loss: 0.0375 - accuracy: 0.9701
428/469 [==========================>...] - ETA: 0s - loss: 0.0373 - accuracy: 0.9702
431/469 [==========================>...] - ETA: 0s - loss: 0.0371 - accuracy: 0.9703
434/469 [==========================>...] - ETA: 0s - loss: 0.0369 - accuracy: 0.9704
437/469 [==========================>...] - ETA: 0s - loss: 0.0367 - accuracy: 0.9706
440/469 [===========================>..] - ETA: 0s - loss: 0.0366 - accuracy: 0.9706
443/469 [===========================>..] - ETA: 0s - loss: 0.0364 - accuracy: 0.9708
446/469 [===========================>..] - ETA: 0s - loss: 0.0362 - accuracy: 0.9709
449/469 [===========================>..] - ETA: 0s - loss: 0.0360 - accuracy: 0.9711
452/469 [===========================>..] - ETA: 0s - loss: 0.0359 - accuracy: 0.9712
455/469 [============================>.] - ETA: 0s - loss: 0.0357 - accuracy: 0.9713
458/469 [============================>.] - ETA: 0s - loss: 0.0355 - accuracy: 0.9715
461/469 [============================>.] - ETA: 0s - loss: 0.0353 - accuracy: 0.9717
464/469 [============================>.] - ETA: 0s - loss: 0.0351 - accuracy: 0.9718
467/469 [============================>.] - ETA: 0s - loss: 0.0350 - accuracy: 0.9719
469/469 [==============================] - 12s 25ms/step - loss: 0.0349 - accuracy: 0.9720 - val_loss: 0.0775 - val_accuracy: 0.9032 - lr: 0.0010
Epoch 2/5

  1/469 [..............................] - ETA: 0s - loss: 0.0024 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 0.0057 - accuracy: 0.9941
  7/469 [..............................] - ETA: 9s - loss: 0.0067 - accuracy: 0.9933
 10/469 [..............................] - ETA: 9s - loss: 0.0083 - accuracy: 0.9906
 13/469 [..............................] - ETA: 10s - loss: 0.0096 - accuracy: 0.9892
 16/469 [>.............................] - ETA: 10s - loss: 0.0102 - accuracy: 0.9893
 19/469 [>.............................] - ETA: 10s - loss: 0.0098 - accuracy: 0.9897
 22/469 [>.............................] - ETA: 10s - loss: 0.0090 - accuracy: 0.9911
 25/469 [>.............................] - ETA: 10s - loss: 0.0087 - accuracy: 0.9912
 28/469 [>.............................] - ETA: 10s - loss: 0.0086 - accuracy: 0.9914
 31/469 [>.............................] - ETA: 10s - loss: 0.0086 - accuracy: 0.9914
 34/469 [=>............................] - ETA: 10s - loss: 0.0086 - accuracy: 0.9917
 37/469 [=>............................] - ETA: 10s - loss: 0.0083 - accuracy: 0.9922
 40/469 [=>............................] - ETA: 9s - loss: 0.0080 - accuracy: 0.9926 
 43/469 [=>............................] - ETA: 9s - loss: 0.0082 - accuracy: 0.9926
 46/469 [=>............................] - ETA: 9s - loss: 0.0078 - accuracy: 0.9930
 49/469 [==>...........................] - ETA: 9s - loss: 0.0075 - accuracy: 0.9931
 52/469 [==>...........................] - ETA: 9s - loss: 0.0074 - accuracy: 0.9931
 55/469 [==>...........................] - ETA: 9s - loss: 0.0073 - accuracy: 0.9930
 58/469 [==>...........................] - ETA: 9s - loss: 0.0072 - accuracy: 0.9933
 61/469 [==>...........................] - ETA: 9s - loss: 0.0072 - accuracy: 0.9933
 64/469 [===>..........................] - ETA: 9s - loss: 0.0071 - accuracy: 0.9930
 67/469 [===>..........................] - ETA: 9s - loss: 0.0070 - accuracy: 0.9931
 70/469 [===>..........................] - ETA: 9s - loss: 0.0068 - accuracy: 0.9933
 73/469 [===>..........................] - ETA: 9s - loss: 0.0068 - accuracy: 0.9934
 76/469 [===>..........................] - ETA: 9s - loss: 0.0069 - accuracy: 0.9933
 79/469 [====>.........................] - ETA: 9s - loss: 0.0068 - accuracy: 0.9934
 82/469 [====>.........................] - ETA: 9s - loss: 0.0069 - accuracy: 0.9931
 85/469 [====>.........................] - ETA: 9s - loss: 0.0069 - accuracy: 0.9932
 88/469 [====>.........................] - ETA: 8s - loss: 0.0068 - accuracy: 0.9933
 91/469 [====>.........................] - ETA: 8s - loss: 0.0068 - accuracy: 0.9932
 94/469 [=====>........................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9934
 97/469 [=====>........................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9934
100/469 [=====>........................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9936
103/469 [=====>........................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9936
106/469 [=====>........................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9931
109/469 [=====>........................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9932
112/469 [======>.......................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9932
115/469 [======>.......................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9933
118/469 [======>.......................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9933
120/469 [======>.......................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9932
122/469 [======>.......................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9930
124/469 [======>.......................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9930
126/469 [=======>......................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9931
128/469 [=======>......................] - ETA: 8s - loss: 0.0065 - accuracy: 0.9930
130/469 [=======>......................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9931
132/469 [=======>......................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9931
135/469 [=======>......................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9931
138/469 [=======>......................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9930
141/469 [========>.....................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9930
144/469 [========>.....................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9931
147/469 [========>.....................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9930
150/469 [========>.....................] - ETA: 7s - loss: 0.0063 - accuracy: 0.9931
153/469 [========>.....................] - ETA: 7s - loss: 0.0063 - accuracy: 0.9931
156/469 [========>.....................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9929
159/469 [=========>....................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9929
162/469 [=========>....................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9930
165/469 [=========>....................] - ETA: 7s - loss: 0.0064 - accuracy: 0.9929
168/469 [=========>....................] - ETA: 7s - loss: 0.0063 - accuracy: 0.9930
171/469 [=========>....................] - ETA: 7s - loss: 0.0063 - accuracy: 0.9929
174/469 [==========>...................] - ETA: 7s - loss: 0.0063 - accuracy: 0.9929
177/469 [==========>...................] - ETA: 7s - loss: 0.0063 - accuracy: 0.9929
180/469 [==========>...................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9931
183/469 [==========>...................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9932
186/469 [==========>...................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9932
189/469 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9931
192/469 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9931
195/469 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9931
198/469 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9932
201/469 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9932
204/469 [============>.................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9932
207/469 [============>.................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9932
210/469 [============>.................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9933
213/469 [============>.................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9933
216/469 [============>.................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9934
219/469 [=============>................] - ETA: 5s - loss: 0.0060 - accuracy: 0.9933
222/469 [=============>................] - ETA: 5s - loss: 0.0060 - accuracy: 0.9933
225/469 [=============>................] - ETA: 5s - loss: 0.0060 - accuracy: 0.9933
228/469 [=============>................] - ETA: 5s - loss: 0.0060 - accuracy: 0.9933
231/469 [=============>................] - ETA: 5s - loss: 0.0060 - accuracy: 0.9933
234/469 [=============>................] - ETA: 5s - loss: 0.0060 - accuracy: 0.9934
237/469 [==============>...............] - ETA: 5s - loss: 0.0060 - accuracy: 0.9934
240/469 [==============>...............] - ETA: 5s - loss: 0.0060 - accuracy: 0.9934
243/469 [==============>...............] - ETA: 5s - loss: 0.0060 - accuracy: 0.9934
246/469 [==============>...............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9935
249/469 [==============>...............] - ETA: 5s - loss: 0.0060 - accuracy: 0.9934
252/469 [===============>..............] - ETA: 5s - loss: 0.0060 - accuracy: 0.9935
255/469 [===============>..............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9935
258/469 [===============>..............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9936
261/469 [===============>..............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
264/469 [===============>..............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
267/469 [================>.............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
270/469 [================>.............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
273/469 [================>.............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
276/469 [================>.............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
279/469 [================>.............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9937
282/469 [=================>............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
285/469 [=================>............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
288/469 [=================>............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9937
291/469 [=================>............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
294/469 [=================>............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9936
297/469 [=================>............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9937
300/469 [==================>...........] - ETA: 4s - loss: 0.0059 - accuracy: 0.9937
303/469 [==================>...........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
306/469 [==================>...........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
309/469 [==================>...........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
312/469 [==================>...........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
315/469 [===================>..........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
318/469 [===================>..........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9938
321/469 [===================>..........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9938
324/469 [===================>..........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9938
327/469 [===================>..........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
330/469 [====================>.........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
333/469 [====================>.........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
336/469 [====================>.........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
339/469 [====================>.........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
342/469 [====================>.........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9936
345/469 [=====================>........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9936
347/469 [=====================>........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9936
349/469 [=====================>........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9936
351/469 [=====================>........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9937
353/469 [=====================>........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9937
355/469 [=====================>........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9937
357/469 [=====================>........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9937
359/469 [=====================>........] - ETA: 2s - loss: 0.0059 - accuracy: 0.9937
362/469 [======================>.......] - ETA: 2s - loss: 0.0059 - accuracy: 0.9937
365/469 [======================>.......] - ETA: 2s - loss: 0.0059 - accuracy: 0.9937
368/469 [======================>.......] - ETA: 2s - loss: 0.0059 - accuracy: 0.9937
371/469 [======================>.......] - ETA: 2s - loss: 0.0059 - accuracy: 0.9938
374/469 [======================>.......] - ETA: 2s - loss: 0.0058 - accuracy: 0.9938
377/469 [=======================>......] - ETA: 2s - loss: 0.0058 - accuracy: 0.9938
380/469 [=======================>......] - ETA: 2s - loss: 0.0058 - accuracy: 0.9939
383/469 [=======================>......] - ETA: 2s - loss: 0.0058 - accuracy: 0.9939
386/469 [=======================>......] - ETA: 1s - loss: 0.0058 - accuracy: 0.9939
389/469 [=======================>......] - ETA: 1s - loss: 0.0058 - accuracy: 0.9939
392/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9939
395/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9939
398/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9938
401/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9939
404/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9939
407/469 [=========================>....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9939
410/469 [=========================>....] - ETA: 1s - loss: 0.0059 - accuracy: 0.9939
413/469 [=========================>....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9939
416/469 [=========================>....] - ETA: 1s - loss: 0.0059 - accuracy: 0.9939
419/469 [=========================>....] - ETA: 1s - loss: 0.0059 - accuracy: 0.9939
422/469 [=========================>....] - ETA: 1s - loss: 0.0059 - accuracy: 0.9939
425/469 [==========================>...] - ETA: 1s - loss: 0.0059 - accuracy: 0.9939
428/469 [==========================>...] - ETA: 0s - loss: 0.0059 - accuracy: 0.9938
431/469 [==========================>...] - ETA: 0s - loss: 0.0059 - accuracy: 0.9939
434/469 [==========================>...] - ETA: 0s - loss: 0.0059 - accuracy: 0.9939
437/469 [==========================>...] - ETA: 0s - loss: 0.0059 - accuracy: 0.9939
440/469 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9939
443/469 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9939
446/469 [===========================>..] - ETA: 0s - loss: 0.0058 - accuracy: 0.9939
449/469 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9939
452/469 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9938
455/469 [============================>.] - ETA: 0s - loss: 0.0059 - accuracy: 0.9938
458/469 [============================>.] - ETA: 0s - loss: 0.0059 - accuracy: 0.9938
461/469 [============================>.] - ETA: 0s - loss: 0.0059 - accuracy: 0.9938
464/469 [============================>.] - ETA: 0s - loss: 0.0059 - accuracy: 0.9939
467/469 [============================>.] - ETA: 0s - loss: 0.0059 - accuracy: 0.9938
469/469 [==============================] - 12s 25ms/step - loss: 0.0059 - accuracy: 0.9938 - val_loss: 0.0113 - val_accuracy: 0.9880 - lr: 6.3096e-04
Epoch 3/5

  1/469 [..............................] - ETA: 0s - loss: 0.0119 - accuracy: 0.9844
  4/469 [..............................] - ETA: 8s - loss: 0.0085 - accuracy: 0.9902
  7/469 [..............................] - ETA: 9s - loss: 0.0067 - accuracy: 0.9922
 10/469 [..............................] - ETA: 9s - loss: 0.0051 - accuracy: 0.9945
 13/469 [..............................] - ETA: 10s - loss: 0.0052 - accuracy: 0.9952
 16/469 [>.............................] - ETA: 10s - loss: 0.0053 - accuracy: 0.9951
 19/469 [>.............................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9951
 22/469 [>.............................] - ETA: 10s - loss: 0.0046 - accuracy: 0.9957
 25/469 [>.............................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9962
 28/469 [>.............................] - ETA: 10s - loss: 0.0045 - accuracy: 0.9958
 31/469 [>.............................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9957
 34/469 [=>............................] - ETA: 10s - loss: 0.0043 - accuracy: 0.9956
 37/469 [=>............................] - ETA: 9s - loss: 0.0042 - accuracy: 0.9956 
 40/469 [=>............................] - ETA: 9s - loss: 0.0041 - accuracy: 0.9957
 43/469 [=>............................] - ETA: 9s - loss: 0.0042 - accuracy: 0.9956
 46/469 [=>............................] - ETA: 9s - loss: 0.0041 - accuracy: 0.9958
 49/469 [==>...........................] - ETA: 9s - loss: 0.0040 - accuracy: 0.9959
 52/469 [==>...........................] - ETA: 9s - loss: 0.0043 - accuracy: 0.9955
 55/469 [==>...........................] - ETA: 9s - loss: 0.0043 - accuracy: 0.9953
 58/469 [==>...........................] - ETA: 9s - loss: 0.0042 - accuracy: 0.9954
 61/469 [==>...........................] - ETA: 9s - loss: 0.0041 - accuracy: 0.9955
 64/469 [===>..........................] - ETA: 9s - loss: 0.0041 - accuracy: 0.9955
 67/469 [===>..........................] - ETA: 9s - loss: 0.0041 - accuracy: 0.9956
 70/469 [===>..........................] - ETA: 9s - loss: 0.0039 - accuracy: 0.9958
 73/469 [===>..........................] - ETA: 9s - loss: 0.0038 - accuracy: 0.9959
 76/469 [===>..........................] - ETA: 9s - loss: 0.0038 - accuracy: 0.9959
 79/469 [====>.........................] - ETA: 9s - loss: 0.0038 - accuracy: 0.9959
 81/469 [====>.........................] - ETA: 9s - loss: 0.0037 - accuracy: 0.9959
 83/469 [====>.........................] - ETA: 9s - loss: 0.0037 - accuracy: 0.9960
 85/469 [====>.........................] - ETA: 9s - loss: 0.0037 - accuracy: 0.9960
 87/469 [====>.........................] - ETA: 9s - loss: 0.0037 - accuracy: 0.9960
 89/469 [====>.........................] - ETA: 9s - loss: 0.0036 - accuracy: 0.9960
 91/469 [====>.........................] - ETA: 9s - loss: 0.0036 - accuracy: 0.9960
 93/469 [====>.........................] - ETA: 9s - loss: 0.0036 - accuracy: 0.9960
 95/469 [=====>........................] - ETA: 9s - loss: 0.0036 - accuracy: 0.9961
 97/469 [=====>........................] - ETA: 8s - loss: 0.0036 - accuracy: 0.9961
100/469 [=====>........................] - ETA: 8s - loss: 0.0036 - accuracy: 0.9961
103/469 [=====>........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9962
106/469 [=====>........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9962
109/469 [=====>........................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9963
112/469 [======>.......................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9962
115/469 [======>.......................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9963
118/469 [======>.......................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9964
121/469 [======>.......................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9964
124/469 [======>.......................] - ETA: 8s - loss: 0.0035 - accuracy: 0.9964
127/469 [=======>......................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9964
130/469 [=======>......................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9965
133/469 [=======>......................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9965
136/469 [=======>......................] - ETA: 8s - loss: 0.0034 - accuracy: 0.9964
139/469 [=======>......................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9964
142/469 [========>.....................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9965
145/469 [========>.....................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9965
148/469 [========>.....................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9965
151/469 [========>.....................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9966
154/469 [========>.....................] - ETA: 7s - loss: 0.0033 - accuracy: 0.9967
157/469 [=========>....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
160/469 [=========>....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
163/469 [=========>....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
166/469 [=========>....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9968
169/469 [=========>....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
172/469 [==========>...................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
175/469 [==========>...................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9968
178/469 [==========>...................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9968
181/469 [==========>...................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9969
184/469 [==========>...................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9969
187/469 [==========>...................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9970
190/469 [===========>..................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9970
193/469 [===========>..................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9970
196/469 [===========>..................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9969
199/469 [===========>..................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9969
202/469 [===========>..................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9970
205/469 [============>.................] - ETA: 6s - loss: 0.0029 - accuracy: 0.9970
208/469 [============>.................] - ETA: 6s - loss: 0.0029 - accuracy: 0.9970
211/469 [============>.................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9970
214/469 [============>.................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9970
217/469 [============>.................] - ETA: 6s - loss: 0.0030 - accuracy: 0.9969
220/469 [=============>................] - ETA: 5s - loss: 0.0030 - accuracy: 0.9969
223/469 [=============>................] - ETA: 5s - loss: 0.0030 - accuracy: 0.9970
226/469 [=============>................] - ETA: 5s - loss: 0.0030 - accuracy: 0.9970
229/469 [=============>................] - ETA: 5s - loss: 0.0030 - accuracy: 0.9969
232/469 [=============>................] - ETA: 5s - loss: 0.0030 - accuracy: 0.9970
235/469 [==============>...............] - ETA: 5s - loss: 0.0030 - accuracy: 0.9970
238/469 [==============>...............] - ETA: 5s - loss: 0.0029 - accuracy: 0.9970
241/469 [==============>...............] - ETA: 5s - loss: 0.0030 - accuracy: 0.9970
244/469 [==============>...............] - ETA: 5s - loss: 0.0029 - accuracy: 0.9970
247/469 [==============>...............] - ETA: 5s - loss: 0.0030 - accuracy: 0.9970
250/469 [==============>...............] - ETA: 5s - loss: 0.0029 - accuracy: 0.9970
253/469 [===============>..............] - ETA: 5s - loss: 0.0029 - accuracy: 0.9971
256/469 [===============>..............] - ETA: 5s - loss: 0.0030 - accuracy: 0.9970
259/469 [===============>..............] - ETA: 5s - loss: 0.0029 - accuracy: 0.9971
262/469 [===============>..............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9970
265/469 [===============>..............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9970
268/469 [================>.............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
271/469 [================>.............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
274/469 [================>.............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
277/469 [================>.............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
280/469 [================>.............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
283/469 [=================>............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
286/469 [=================>............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
289/469 [=================>............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
292/469 [=================>............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9972
295/469 [=================>............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
298/469 [==================>...........] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
301/469 [==================>...........] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
304/469 [==================>...........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
307/469 [==================>...........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9970
309/469 [==================>...........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9970
311/469 [==================>...........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
313/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
315/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9970
317/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9970
319/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
321/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
323/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
326/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
329/469 [====================>.........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
332/469 [====================>.........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
335/469 [====================>.........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
338/469 [====================>.........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
341/469 [====================>.........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
344/469 [=====================>........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
347/469 [=====================>........] - ETA: 2s - loss: 0.0029 - accuracy: 0.9972
350/469 [=====================>........] - ETA: 2s - loss: 0.0029 - accuracy: 0.9972
353/469 [=====================>........] - ETA: 2s - loss: 0.0029 - accuracy: 0.9972
356/469 [=====================>........] - ETA: 2s - loss: 0.0029 - accuracy: 0.9972
359/469 [=====================>........] - ETA: 2s - loss: 0.0029 - accuracy: 0.9972
362/469 [======================>.......] - ETA: 2s - loss: 0.0029 - accuracy: 0.9972
365/469 [======================>.......] - ETA: 2s - loss: 0.0029 - accuracy: 0.9972
368/469 [======================>.......] - ETA: 2s - loss: 0.0029 - accuracy: 0.9971
371/469 [======================>.......] - ETA: 2s - loss: 0.0029 - accuracy: 0.9971
374/469 [======================>.......] - ETA: 2s - loss: 0.0029 - accuracy: 0.9971
377/469 [=======================>......] - ETA: 2s - loss: 0.0029 - accuracy: 0.9971
380/469 [=======================>......] - ETA: 2s - loss: 0.0029 - accuracy: 0.9971
383/469 [=======================>......] - ETA: 2s - loss: 0.0029 - accuracy: 0.9971
386/469 [=======================>......] - ETA: 1s - loss: 0.0029 - accuracy: 0.9971
389/469 [=======================>......] - ETA: 1s - loss: 0.0029 - accuracy: 0.9971
392/469 [========================>.....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
395/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9972
398/469 [========================>.....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
401/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9972
404/469 [========================>.....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
407/469 [=========================>....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
410/469 [=========================>....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
413/469 [=========================>....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
416/469 [=========================>....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
419/469 [=========================>....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
422/469 [=========================>....] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
425/469 [==========================>...] - ETA: 1s - loss: 0.0029 - accuracy: 0.9972
428/469 [==========================>...] - ETA: 0s - loss: 0.0029 - accuracy: 0.9972
431/469 [==========================>...] - ETA: 0s - loss: 0.0029 - accuracy: 0.9972
434/469 [==========================>...] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
437/469 [==========================>...] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
440/469 [===========================>..] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
443/469 [===========================>..] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
446/469 [===========================>..] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
449/469 [===========================>..] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
452/469 [===========================>..] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
455/469 [============================>.] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
458/469 [============================>.] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
461/469 [============================>.] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
464/469 [============================>.] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
467/469 [============================>.] - ETA: 0s - loss: 0.0029 - accuracy: 0.9971
469/469 [==============================] - 12s 25ms/step - loss: 0.0029 - accuracy: 0.9970 - val_loss: 0.0059 - val_accuracy: 0.9933 - lr: 3.9811e-04
Epoch 4/5

  1/469 [..............................] - ETA: 0s - loss: 2.2360e-04 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 7.6295e-04 - accuracy: 1.0000
  7/469 [..............................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989    
 10/469 [..............................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9984
 13/469 [..............................] - ETA: 10s - loss: 0.0016 - accuracy: 0.9988
 16/469 [>.............................] - ETA: 10s - loss: 0.0020 - accuracy: 0.9976
 19/469 [>.............................] - ETA: 10s - loss: 0.0022 - accuracy: 0.9971
 22/469 [>.............................] - ETA: 10s - loss: 0.0021 - accuracy: 0.9975
 25/469 [>.............................] - ETA: 10s - loss: 0.0021 - accuracy: 0.9975
 28/469 [>.............................] - ETA: 10s - loss: 0.0021 - accuracy: 0.9975
 31/469 [>.............................] - ETA: 10s - loss: 0.0021 - accuracy: 0.9975
 34/469 [=>............................] - ETA: 10s - loss: 0.0020 - accuracy: 0.9975
 37/469 [=>............................] - ETA: 10s - loss: 0.0020 - accuracy: 0.9977
 40/469 [=>............................] - ETA: 10s - loss: 0.0019 - accuracy: 0.9979
 43/469 [=>............................] - ETA: 9s - loss: 0.0020 - accuracy: 0.9976 
 45/469 [=>............................] - ETA: 9s - loss: 0.0021 - accuracy: 0.9974
 47/469 [==>...........................] - ETA: 9s - loss: 0.0020 - accuracy: 0.9975
 49/469 [==>...........................] - ETA: 10s - loss: 0.0019 - accuracy: 0.9976
 51/469 [==>...........................] - ETA: 10s - loss: 0.0019 - accuracy: 0.9975
 53/469 [==>...........................] - ETA: 10s - loss: 0.0020 - accuracy: 0.9975
 55/469 [==>...........................] - ETA: 10s - loss: 0.0019 - accuracy: 0.9976
 57/469 [==>...........................] - ETA: 10s - loss: 0.0019 - accuracy: 0.9977
 59/469 [==>...........................] - ETA: 10s - loss: 0.0018 - accuracy: 0.9977
 61/469 [==>...........................] - ETA: 10s - loss: 0.0018 - accuracy: 0.9978
 64/469 [===>..........................] - ETA: 9s - loss: 0.0018 - accuracy: 0.9978 
 67/469 [===>..........................] - ETA: 9s - loss: 0.0018 - accuracy: 0.9977
 70/469 [===>..........................] - ETA: 9s - loss: 0.0018 - accuracy: 0.9978
 73/469 [===>..........................] - ETA: 9s - loss: 0.0017 - accuracy: 0.9979
 76/469 [===>..........................] - ETA: 9s - loss: 0.0017 - accuracy: 0.9978
 79/469 [====>.........................] - ETA: 9s - loss: 0.0017 - accuracy: 0.9979
 82/469 [====>.........................] - ETA: 9s - loss: 0.0017 - accuracy: 0.9979
 85/469 [====>.........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9980
 88/469 [====>.........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9980
 91/469 [====>.........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9980
 94/469 [=====>........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9981
 97/469 [=====>........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9981
100/469 [=====>........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9981
103/469 [=====>........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9982
106/469 [=====>........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9982
109/469 [=====>........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9981
112/469 [======>.......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9982
115/469 [======>.......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9982
118/469 [======>.......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9983
121/469 [======>.......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9983
124/469 [======>.......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9984
127/469 [=======>......................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9983
130/469 [=======>......................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9984
133/469 [=======>......................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9984
136/469 [=======>......................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9984
139/469 [=======>......................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9985
142/469 [========>.....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9985
145/469 [========>.....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9985
148/469 [========>.....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9985
151/469 [========>.....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9985
154/469 [========>.....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9985
157/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
160/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
163/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
166/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
169/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9987
172/469 [==========>...................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
175/469 [==========>...................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
178/469 [==========>...................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
181/469 [==========>...................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
184/469 [==========>...................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
187/469 [==========>...................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
190/469 [===========>..................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
193/469 [===========>..................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
196/469 [===========>..................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
199/469 [===========>..................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
202/469 [===========>..................] - ETA: 6s - loss: 0.0016 - accuracy: 0.9986
205/469 [============>.................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
208/469 [============>.................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9985
211/469 [============>.................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9986
214/469 [============>.................] - ETA: 6s - loss: 0.0015 - accuracy: 0.9985
217/469 [============>.................] - ETA: 6s - loss: 0.0016 - accuracy: 0.9985
220/469 [=============>................] - ETA: 5s - loss: 0.0016 - accuracy: 0.9985
223/469 [=============>................] - ETA: 5s - loss: 0.0016 - accuracy: 0.9985
226/469 [=============>................] - ETA: 5s - loss: 0.0016 - accuracy: 0.9985
229/469 [=============>................] - ETA: 5s - loss: 0.0016 - accuracy: 0.9985
232/469 [=============>................] - ETA: 5s - loss: 0.0016 - accuracy: 0.9985
235/469 [==============>...............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9985
238/469 [==============>...............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9985
241/469 [==============>...............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9984
244/469 [==============>...............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9984
247/469 [==============>...............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9984
250/469 [==============>...............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9984
253/469 [===============>..............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9984
256/469 [===============>..............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9984
259/469 [===============>..............] - ETA: 5s - loss: 0.0016 - accuracy: 0.9984
262/469 [===============>..............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
265/469 [===============>..............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
268/469 [================>.............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9985
271/469 [================>.............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9985
273/469 [================>.............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9985
275/469 [================>.............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9985
277/469 [================>.............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9985
279/469 [================>.............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9985
281/469 [================>.............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9985
283/469 [=================>............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
285/469 [=================>............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
287/469 [=================>............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
290/469 [=================>............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
293/469 [=================>............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
296/469 [=================>............] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
299/469 [==================>...........] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
302/469 [==================>...........] - ETA: 4s - loss: 0.0016 - accuracy: 0.9984
305/469 [==================>...........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
308/469 [==================>...........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9983
311/469 [==================>...........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9983
314/469 [===================>..........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9983
317/469 [===================>..........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9983
320/469 [===================>..........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
323/469 [===================>..........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
326/469 [===================>..........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
329/469 [====================>.........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
332/469 [====================>.........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
335/469 [====================>.........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
338/469 [====================>.........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
341/469 [====================>.........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
344/469 [=====================>........] - ETA: 3s - loss: 0.0016 - accuracy: 0.9984
347/469 [=====================>........] - ETA: 2s - loss: 0.0016 - accuracy: 0.9984
350/469 [=====================>........] - ETA: 2s - loss: 0.0016 - accuracy: 0.9984
353/469 [=====================>........] - ETA: 2s - loss: 0.0016 - accuracy: 0.9984
356/469 [=====================>........] - ETA: 2s - loss: 0.0016 - accuracy: 0.9984
359/469 [=====================>........] - ETA: 2s - loss: 0.0016 - accuracy: 0.9985
362/469 [======================>.......] - ETA: 2s - loss: 0.0016 - accuracy: 0.9985
365/469 [======================>.......] - ETA: 2s - loss: 0.0016 - accuracy: 0.9985
368/469 [======================>.......] - ETA: 2s - loss: 0.0015 - accuracy: 0.9985
371/469 [======================>.......] - ETA: 2s - loss: 0.0015 - accuracy: 0.9985
374/469 [======================>.......] - ETA: 2s - loss: 0.0016 - accuracy: 0.9985
377/469 [=======================>......] - ETA: 2s - loss: 0.0016 - accuracy: 0.9985
380/469 [=======================>......] - ETA: 2s - loss: 0.0016 - accuracy: 0.9985
383/469 [=======================>......] - ETA: 2s - loss: 0.0016 - accuracy: 0.9985
386/469 [=======================>......] - ETA: 2s - loss: 0.0016 - accuracy: 0.9985
389/469 [=======================>......] - ETA: 1s - loss: 0.0016 - accuracy: 0.9985
392/469 [========================>.....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
395/469 [========================>.....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
398/469 [========================>.....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
401/469 [========================>.....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
404/469 [========================>.....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
407/469 [=========================>....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
410/469 [=========================>....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
413/469 [=========================>....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
416/469 [=========================>....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
419/469 [=========================>....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
422/469 [=========================>....] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
425/469 [==========================>...] - ETA: 1s - loss: 0.0015 - accuracy: 0.9985
428/469 [==========================>...] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
431/469 [==========================>...] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
434/469 [==========================>...] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
437/469 [==========================>...] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
440/469 [===========================>..] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
443/469 [===========================>..] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
446/469 [===========================>..] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
449/469 [===========================>..] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
452/469 [===========================>..] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
455/469 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
458/469 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
461/469 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
464/469 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
467/469 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 0.9985
469/469 [==============================] - 12s 25ms/step - loss: 0.0015 - accuracy: 0.9985 - val_loss: 0.0052 - val_accuracy: 0.9936 - lr: 2.5119e-04
Epoch 5/5

  1/469 [..............................] - ETA: 0s - loss: 4.3587e-04 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 9.4846e-04 - accuracy: 1.0000
  7/469 [..............................] - ETA: 9s - loss: 7.5825e-04 - accuracy: 1.0000
 10/469 [..............................] - ETA: 9s - loss: 7.2174e-04 - accuracy: 1.0000
 12/469 [..............................] - ETA: 10s - loss: 8.1688e-04 - accuracy: 1.0000
 14/469 [..............................] - ETA: 10s - loss: 9.8662e-04 - accuracy: 1.0000
 16/469 [>.............................] - ETA: 10s - loss: 0.0010 - accuracy: 0.9995    
 18/469 [>.............................] - ETA: 11s - loss: 0.0010 - accuracy: 0.9996
 20/469 [>.............................] - ETA: 11s - loss: 9.4239e-04 - accuracy: 0.9996
 22/469 [>.............................] - ETA: 11s - loss: 8.6230e-04 - accuracy: 0.9996
 24/469 [>.............................] - ETA: 11s - loss: 8.4679e-04 - accuracy: 0.9997
 27/469 [>.............................] - ETA: 11s - loss: 7.8998e-04 - accuracy: 0.9997
 30/469 [>.............................] - ETA: 10s - loss: 7.8213e-04 - accuracy: 0.9997
 33/469 [=>............................] - ETA: 10s - loss: 7.7886e-04 - accuracy: 0.9998
 36/469 [=>............................] - ETA: 10s - loss: 7.5603e-04 - accuracy: 0.9998
 39/469 [=>............................] - ETA: 10s - loss: 7.4258e-04 - accuracy: 0.9998
 42/469 [=>............................] - ETA: 10s - loss: 7.0392e-04 - accuracy: 0.9998
 45/469 [=>............................] - ETA: 10s - loss: 7.8091e-04 - accuracy: 0.9997
 48/469 [==>...........................] - ETA: 10s - loss: 7.5338e-04 - accuracy: 0.9997
 51/469 [==>...........................] - ETA: 10s - loss: 8.0682e-04 - accuracy: 0.9997
 54/469 [==>...........................] - ETA: 10s - loss: 8.1093e-04 - accuracy: 0.9997
 57/469 [==>...........................] - ETA: 10s - loss: 8.1136e-04 - accuracy: 0.9996
 60/469 [==>...........................] - ETA: 9s - loss: 8.0527e-04 - accuracy: 0.9996 
 63/469 [===>..........................] - ETA: 9s - loss: 8.0647e-04 - accuracy: 0.9996
 66/469 [===>..........................] - ETA: 9s - loss: 7.9292e-04 - accuracy: 0.9996
 69/469 [===>..........................] - ETA: 9s - loss: 7.7425e-04 - accuracy: 0.9997
 72/469 [===>..........................] - ETA: 9s - loss: 7.4949e-04 - accuracy: 0.9997
 75/469 [===>..........................] - ETA: 9s - loss: 7.4096e-04 - accuracy: 0.9997
 78/469 [===>..........................] - ETA: 9s - loss: 7.3884e-04 - accuracy: 0.9997
 81/469 [====>.........................] - ETA: 9s - loss: 7.3486e-04 - accuracy: 0.9997
 84/469 [====>.........................] - ETA: 9s - loss: 7.2167e-04 - accuracy: 0.9997
 87/469 [====>.........................] - ETA: 9s - loss: 7.0868e-04 - accuracy: 0.9997
 90/469 [====>.........................] - ETA: 9s - loss: 7.1567e-04 - accuracy: 0.9997
 93/469 [====>.........................] - ETA: 9s - loss: 7.1675e-04 - accuracy: 0.9997
 96/469 [=====>........................] - ETA: 9s - loss: 7.3983e-04 - accuracy: 0.9997
 99/469 [=====>........................] - ETA: 8s - loss: 7.3524e-04 - accuracy: 0.9997
102/469 [=====>........................] - ETA: 8s - loss: 7.3673e-04 - accuracy: 0.9997
105/469 [=====>........................] - ETA: 8s - loss: 7.2636e-04 - accuracy: 0.9997
108/469 [=====>........................] - ETA: 8s - loss: 7.1440e-04 - accuracy: 0.9997
111/469 [======>.......................] - ETA: 8s - loss: 7.0127e-04 - accuracy: 0.9997
114/469 [======>.......................] - ETA: 8s - loss: 7.0022e-04 - accuracy: 0.9997
117/469 [======>.......................] - ETA: 8s - loss: 6.8420e-04 - accuracy: 0.9997
120/469 [======>.......................] - ETA: 8s - loss: 6.7045e-04 - accuracy: 0.9997
123/469 [======>.......................] - ETA: 8s - loss: 6.6702e-04 - accuracy: 0.9997
126/469 [=======>......................] - ETA: 8s - loss: 6.6886e-04 - accuracy: 0.9998
129/469 [=======>......................] - ETA: 8s - loss: 6.6809e-04 - accuracy: 0.9998
132/469 [=======>......................] - ETA: 8s - loss: 6.7074e-04 - accuracy: 0.9998
135/469 [=======>......................] - ETA: 8s - loss: 6.7606e-04 - accuracy: 0.9997
138/469 [=======>......................] - ETA: 7s - loss: 6.7481e-04 - accuracy: 0.9997
141/469 [========>.....................] - ETA: 7s - loss: 6.6793e-04 - accuracy: 0.9997
144/469 [========>.....................] - ETA: 7s - loss: 6.6406e-04 - accuracy: 0.9997
147/469 [========>.....................] - ETA: 7s - loss: 6.9413e-04 - accuracy: 0.9997
150/469 [========>.....................] - ETA: 7s - loss: 6.9114e-04 - accuracy: 0.9997
153/469 [========>.....................] - ETA: 7s - loss: 6.8980e-04 - accuracy: 0.9997
156/469 [========>.....................] - ETA: 7s - loss: 6.9033e-04 - accuracy: 0.9997
159/469 [=========>....................] - ETA: 7s - loss: 6.8284e-04 - accuracy: 0.9997
162/469 [=========>....................] - ETA: 7s - loss: 6.7423e-04 - accuracy: 0.9997
165/469 [=========>....................] - ETA: 7s - loss: 6.7158e-04 - accuracy: 0.9997
168/469 [=========>....................] - ETA: 7s - loss: 6.6787e-04 - accuracy: 0.9997
171/469 [=========>....................] - ETA: 7s - loss: 6.6566e-04 - accuracy: 0.9997
174/469 [==========>...................] - ETA: 7s - loss: 6.6908e-04 - accuracy: 0.9997
177/469 [==========>...................] - ETA: 7s - loss: 6.6559e-04 - accuracy: 0.9997
180/469 [==========>...................] - ETA: 6s - loss: 6.7581e-04 - accuracy: 0.9997
183/469 [==========>...................] - ETA: 6s - loss: 6.7597e-04 - accuracy: 0.9997
186/469 [==========>...................] - ETA: 6s - loss: 6.8062e-04 - accuracy: 0.9997
189/469 [===========>..................] - ETA: 6s - loss: 6.8325e-04 - accuracy: 0.9997
192/469 [===========>..................] - ETA: 6s - loss: 6.8836e-04 - accuracy: 0.9997
195/469 [===========>..................] - ETA: 6s - loss: 6.8901e-04 - accuracy: 0.9997
198/469 [===========>..................] - ETA: 6s - loss: 6.8466e-04 - accuracy: 0.9997
201/469 [===========>..................] - ETA: 6s - loss: 6.7890e-04 - accuracy: 0.9997
204/469 [============>.................] - ETA: 6s - loss: 6.7328e-04 - accuracy: 0.9997
207/469 [============>.................] - ETA: 6s - loss: 6.6924e-04 - accuracy: 0.9997
210/469 [============>.................] - ETA: 6s - loss: 6.6278e-04 - accuracy: 0.9997
213/469 [============>.................] - ETA: 6s - loss: 6.6720e-04 - accuracy: 0.9997
216/469 [============>.................] - ETA: 6s - loss: 6.6825e-04 - accuracy: 0.9997
219/469 [=============>................] - ETA: 6s - loss: 6.8178e-04 - accuracy: 0.9996
222/469 [=============>................] - ETA: 5s - loss: 6.8636e-04 - accuracy: 0.9996
225/469 [=============>................] - ETA: 5s - loss: 6.8238e-04 - accuracy: 0.9997
228/469 [=============>................] - ETA: 5s - loss: 6.7735e-04 - accuracy: 0.9997
231/469 [=============>................] - ETA: 5s - loss: 6.7799e-04 - accuracy: 0.9997
234/469 [=============>................] - ETA: 5s - loss: 6.7170e-04 - accuracy: 0.9997
236/469 [==============>...............] - ETA: 5s - loss: 6.6868e-04 - accuracy: 0.9997
238/469 [==============>...............] - ETA: 5s - loss: 6.6518e-04 - accuracy: 0.9997
240/469 [==============>...............] - ETA: 5s - loss: 6.6038e-04 - accuracy: 0.9997
242/469 [==============>...............] - ETA: 5s - loss: 6.5914e-04 - accuracy: 0.9997
244/469 [==============>...............] - ETA: 5s - loss: 6.5450e-04 - accuracy: 0.9997
246/469 [==============>...............] - ETA: 5s - loss: 6.7029e-04 - accuracy: 0.9997
248/469 [==============>...............] - ETA: 5s - loss: 6.6650e-04 - accuracy: 0.9997
250/469 [==============>...............] - ETA: 5s - loss: 6.6117e-04 - accuracy: 0.9997
253/469 [===============>..............] - ETA: 5s - loss: 6.6585e-04 - accuracy: 0.9997
256/469 [===============>..............] - ETA: 5s - loss: 6.6180e-04 - accuracy: 0.9997
259/469 [===============>..............] - ETA: 5s - loss: 6.7197e-04 - accuracy: 0.9997
262/469 [===============>..............] - ETA: 5s - loss: 6.8036e-04 - accuracy: 0.9997
265/469 [===============>..............] - ETA: 4s - loss: 6.7522e-04 - accuracy: 0.9997
268/469 [================>.............] - ETA: 4s - loss: 6.7895e-04 - accuracy: 0.9997
271/469 [================>.............] - ETA: 4s - loss: 6.7739e-04 - accuracy: 0.9997
274/469 [================>.............] - ETA: 4s - loss: 6.7565e-04 - accuracy: 0.9997
277/469 [================>.............] - ETA: 4s - loss: 6.8407e-04 - accuracy: 0.9997
280/469 [================>.............] - ETA: 4s - loss: 6.7815e-04 - accuracy: 0.9997
283/469 [=================>............] - ETA: 4s - loss: 6.7553e-04 - accuracy: 0.9997
286/469 [=================>............] - ETA: 4s - loss: 6.7124e-04 - accuracy: 0.9997
289/469 [=================>............] - ETA: 4s - loss: 6.7730e-04 - accuracy: 0.9997
292/469 [=================>............] - ETA: 4s - loss: 6.8710e-04 - accuracy: 0.9997
295/469 [=================>............] - ETA: 4s - loss: 6.8384e-04 - accuracy: 0.9997
298/469 [==================>...........] - ETA: 4s - loss: 6.8528e-04 - accuracy: 0.9997
301/469 [==================>...........] - ETA: 4s - loss: 6.9393e-04 - accuracy: 0.9997
304/469 [==================>...........] - ETA: 3s - loss: 6.9389e-04 - accuracy: 0.9997
307/469 [==================>...........] - ETA: 3s - loss: 7.0025e-04 - accuracy: 0.9997
310/469 [==================>...........] - ETA: 3s - loss: 6.9862e-04 - accuracy: 0.9997
313/469 [===================>..........] - ETA: 3s - loss: 6.9392e-04 - accuracy: 0.9997
316/469 [===================>..........] - ETA: 3s - loss: 6.9732e-04 - accuracy: 0.9997
319/469 [===================>..........] - ETA: 3s - loss: 6.9405e-04 - accuracy: 0.9997
322/469 [===================>..........] - ETA: 3s - loss: 6.9568e-04 - accuracy: 0.9997
325/469 [===================>..........] - ETA: 3s - loss: 6.9309e-04 - accuracy: 0.9997
328/469 [===================>..........] - ETA: 3s - loss: 6.8960e-04 - accuracy: 0.9997
331/469 [====================>.........] - ETA: 3s - loss: 6.9181e-04 - accuracy: 0.9997
334/469 [====================>.........] - ETA: 3s - loss: 6.9607e-04 - accuracy: 0.9997
337/469 [====================>.........] - ETA: 3s - loss: 6.9844e-04 - accuracy: 0.9997
340/469 [====================>.........] - ETA: 3s - loss: 6.9708e-04 - accuracy: 0.9997
343/469 [====================>.........] - ETA: 3s - loss: 6.9399e-04 - accuracy: 0.9997
346/469 [=====================>........] - ETA: 2s - loss: 6.9396e-04 - accuracy: 0.9997
349/469 [=====================>........] - ETA: 2s - loss: 6.9137e-04 - accuracy: 0.9997
352/469 [=====================>........] - ETA: 2s - loss: 6.8777e-04 - accuracy: 0.9997
355/469 [=====================>........] - ETA: 2s - loss: 6.8687e-04 - accuracy: 0.9997
358/469 [=====================>........] - ETA: 2s - loss: 6.8590e-04 - accuracy: 0.9997
361/469 [======================>.......] - ETA: 2s - loss: 6.8836e-04 - accuracy: 0.9997
364/469 [======================>.......] - ETA: 2s - loss: 6.8724e-04 - accuracy: 0.9997
367/469 [======================>.......] - ETA: 2s - loss: 6.8965e-04 - accuracy: 0.9997
370/469 [======================>.......] - ETA: 2s - loss: 6.8890e-04 - accuracy: 0.9997
373/469 [======================>.......] - ETA: 2s - loss: 6.8396e-04 - accuracy: 0.9997
376/469 [=======================>......] - ETA: 2s - loss: 6.8332e-04 - accuracy: 0.9997
379/469 [=======================>......] - ETA: 2s - loss: 6.8266e-04 - accuracy: 0.9997
382/469 [=======================>......] - ETA: 2s - loss: 6.7879e-04 - accuracy: 0.9997
385/469 [=======================>......] - ETA: 2s - loss: 6.7883e-04 - accuracy: 0.9997
388/469 [=======================>......] - ETA: 1s - loss: 6.7570e-04 - accuracy: 0.9997
391/469 [========================>.....] - ETA: 1s - loss: 6.7100e-04 - accuracy: 0.9997
394/469 [========================>.....] - ETA: 1s - loss: 6.6730e-04 - accuracy: 0.9997
397/469 [========================>.....] - ETA: 1s - loss: 6.7145e-04 - accuracy: 0.9997
400/469 [========================>.....] - ETA: 1s - loss: 6.8172e-04 - accuracy: 0.9997
403/469 [========================>.....] - ETA: 1s - loss: 6.7731e-04 - accuracy: 0.9997
406/469 [========================>.....] - ETA: 1s - loss: 6.8076e-04 - accuracy: 0.9997
409/469 [=========================>....] - ETA: 1s - loss: 6.7736e-04 - accuracy: 0.9997
412/469 [=========================>....] - ETA: 1s - loss: 6.7845e-04 - accuracy: 0.9997
415/469 [=========================>....] - ETA: 1s - loss: 6.7891e-04 - accuracy: 0.9997
418/469 [=========================>....] - ETA: 1s - loss: 6.8055e-04 - accuracy: 0.9997
421/469 [=========================>....] - ETA: 1s - loss: 6.7961e-04 - accuracy: 0.9997
424/469 [==========================>...] - ETA: 1s - loss: 6.8158e-04 - accuracy: 0.9997
427/469 [==========================>...] - ETA: 1s - loss: 6.8038e-04 - accuracy: 0.9997
430/469 [==========================>...] - ETA: 0s - loss: 6.8825e-04 - accuracy: 0.9997
433/469 [==========================>...] - ETA: 0s - loss: 6.9420e-04 - accuracy: 0.9997
436/469 [==========================>...] - ETA: 0s - loss: 7.0031e-04 - accuracy: 0.9997
439/469 [===========================>..] - ETA: 0s - loss: 7.0212e-04 - accuracy: 0.9997
442/469 [===========================>..] - ETA: 0s - loss: 7.0181e-04 - accuracy: 0.9997
445/469 [===========================>..] - ETA: 0s - loss: 6.9815e-04 - accuracy: 0.9997
448/469 [===========================>..] - ETA: 0s - loss: 6.9475e-04 - accuracy: 0.9997
451/469 [===========================>..] - ETA: 0s - loss: 7.1117e-04 - accuracy: 0.9997
454/469 [============================>.] - ETA: 0s - loss: 7.1029e-04 - accuracy: 0.9997
457/469 [============================>.] - ETA: 0s - loss: 7.1014e-04 - accuracy: 0.9997
460/469 [============================>.] - ETA: 0s - loss: 7.1461e-04 - accuracy: 0.9997
462/469 [============================>.] - ETA: 0s - loss: 7.1561e-04 - accuracy: 0.9997
464/469 [============================>.] - ETA: 0s - loss: 7.1669e-04 - accuracy: 0.9997
466/469 [============================>.] - ETA: 0s - loss: 7.2473e-04 - accuracy: 0.9997
468/469 [============================>.] - ETA: 0s - loss: 7.2496e-04 - accuracy: 0.9997
469/469 [==============================] - 12s 25ms/step - loss: 7.2380e-04 - accuracy: 0.9997 - val_loss: 0.0049 - val_accuracy: 0.9935 - lr: 1.5849e-04
Test score: 0.004906964022666216
Test accuracy: 0.9934999942779541

Saving the model weights as mnistnet_act_fp_wgt_fp.hdf5 to reload them as init weights for the quantization step:

Note

This is not mandatory but helps with training speed.

temp_dir = TemporaryDirectory()
model_keras.save_weights(
    os.path.join(temp_dir.name, 'mnistnet_act_fp_wgt_fp.hdf5'))

4. Model quantization and training

4.1 Quantize the model

We can now turn to training a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.

For this, we just have to change very slightly the definition of the model used above, changing just the values of weight_quantization and activ_quantization used for the blocks (but still with no output nonlinearity for the final block). Additionally, we’ll initialise the model using the set of pre-trained weights that we just saved (not so important here, but for more complex datasets can make a huge difference both to the accuracy level ultimately achieved and to the speed of convergence).

Note that, for more challenging datasets, it may also be useful to make stepwise changes towards a fully quantized model - e.g. by first training with only the activations quantized, re-saving, and then adding the quantized weights. Additionally, the toolkit documentation describes how one can go further, optimizing the degree of sparsity in the model to reduce computational cost while maintaining accuracy. In this first example however, we’ll stick to a one step conversion (mainly because the MNIST dataset simply isn’t complex enough to see the benefit of the advanced techniques).

# Removes all the nodes left over from the previous model and free memory
K.clear_session()

img_input = Input(shape=(28, 28, 1))
x = conv_block(img_input,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_0',
               weight_quantization=2,
               activ_quantization=1,
               pooling='max',
               add_batchnorm=True)
x = conv_block(x,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_1',
               weight_quantization=2,
               activ_quantization=1,
               pooling='max',
               add_batchnorm=True)
x = conv_block(x,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_2',
               weight_quantization=2,
               activ_quantization=1,
               add_batchnorm=True)
x = Flatten()(x)
x = dense_block(x,
                units=512,
                use_bias=False,
                name='dense_2',
                weight_quantization=2,
                activ_quantization=1,
                add_batchnorm=True)
x = dense_block(x,
                units=10,
                use_bias=False,
                name='dense_3',
                weight_quantization=2,
                activ_quantization=None)

model_keras = Model(img_input, x, name='mnistnet_quantized')
lr_start = 1e-3
opt = Adam(lr=lr_start)
model_keras.compile(loss='squared_hinge', optimizer=opt, metrics=['accuracy'])
model_keras.summary()

# Reload previously computed weights as init weights for the quantization step
load_status = model_keras.load_weights(
    os.path.join(temp_dir.name, 'mnistnet_act_fp_wgt_fp.hdf5'))

Out:

Model: "mnistnet_quantized"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(None, 28, 28, 1)]       0
_________________________________________________________________
conv_0 (QuantizedConv2D)     (None, 28, 28, 32)        800
_________________________________________________________________
conv_0_maxpool (MaxPooling2D (None, 14, 14, 32)        0
_________________________________________________________________
conv_0_BN (BatchNormalizatio (None, 14, 14, 32)        128
_________________________________________________________________
conv_0_relu (ActivationDiscr (None, 14, 14, 32)        0
_________________________________________________________________
conv_1 (QuantizedConv2D)     (None, 14, 14, 32)        25600
_________________________________________________________________
conv_1_maxpool (MaxPooling2D (None, 7, 7, 32)          0
_________________________________________________________________
conv_1_BN (BatchNormalizatio (None, 7, 7, 32)          128
_________________________________________________________________
conv_1_relu (ActivationDiscr (None, 7, 7, 32)          0
_________________________________________________________________
conv_2 (QuantizedConv2D)     (None, 7, 7, 32)          25600
_________________________________________________________________
conv_2_BN (BatchNormalizatio (None, 7, 7, 32)          128
_________________________________________________________________
conv_2_relu (ActivationDiscr (None, 7, 7, 32)          0
_________________________________________________________________
flatten (Flatten)            (None, 1568)              0
_________________________________________________________________
dense_2 (QuantizedDense)     (None, 512)               802816
_________________________________________________________________
dense_2_BN (BatchNormalizati (None, 512)               2048
_________________________________________________________________
dense_2_relu (ActivationDisc (None, 512)               0
_________________________________________________________________
dense_3 (QuantizedDense)     (None, 10)                5120
=================================================================
Total params: 862,368
Trainable params: 861,152
Non-trainable params: 1,216
_________________________________________________________________

4.2 Performance check

Re-train and save the quantized model:

callbacks = []
lr_scheduler = LearningRateScheduler(lambda e: lr_start * lr_decay**e)
callbacks.append(lr_scheduler)
history = model_keras.fit(x_train,
                          y_train,
                          batch_size=batch_size,
                          epochs=epochs,
                          verbose=1,
                          validation_data=(x_test, y_test),
                          callbacks=callbacks)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/5

  1/469 [..............................] - ETA: 0s - loss: 0.1741 - accuracy: 0.7891
  3/469 [..............................] - ETA: 8s - loss: 0.1309 - accuracy: 0.8359
  5/469 [..............................] - ETA: 10s - loss: 0.1083 - accuracy: 0.8703
  7/469 [..............................] - ETA: 11s - loss: 0.0961 - accuracy: 0.8817
  9/469 [..............................] - ETA: 11s - loss: 0.0909 - accuracy: 0.8880
 11/469 [..............................] - ETA: 11s - loss: 0.0854 - accuracy: 0.8984
 13/469 [..............................] - ETA: 12s - loss: 0.0784 - accuracy: 0.9081
 15/469 [..............................] - ETA: 12s - loss: 0.0725 - accuracy: 0.9172
 17/469 [>.............................] - ETA: 12s - loss: 0.0695 - accuracy: 0.9196
 19/469 [>.............................] - ETA: 12s - loss: 0.0667 - accuracy: 0.9231
 21/469 [>.............................] - ETA: 12s - loss: 0.0628 - accuracy: 0.9278
 23/469 [>.............................] - ETA: 12s - loss: 0.0603 - accuracy: 0.9307
 25/469 [>.............................] - ETA: 12s - loss: 0.0581 - accuracy: 0.9331
 27/469 [>.............................] - ETA: 12s - loss: 0.0568 - accuracy: 0.9343
 29/469 [>.............................] - ETA: 12s - loss: 0.0558 - accuracy: 0.9351
 31/469 [>.............................] - ETA: 12s - loss: 0.0536 - accuracy: 0.9375
 33/469 [=>............................] - ETA: 12s - loss: 0.0522 - accuracy: 0.9384
 35/469 [=>............................] - ETA: 12s - loss: 0.0504 - accuracy: 0.9406
 37/469 [=>............................] - ETA: 11s - loss: 0.0500 - accuracy: 0.9415
 39/469 [=>............................] - ETA: 11s - loss: 0.0497 - accuracy: 0.9419
 41/469 [=>............................] - ETA: 11s - loss: 0.0486 - accuracy: 0.9436
 43/469 [=>............................] - ETA: 11s - loss: 0.0478 - accuracy: 0.9446
 45/469 [=>............................] - ETA: 11s - loss: 0.0468 - accuracy: 0.9457
 47/469 [==>...........................] - ETA: 11s - loss: 0.0463 - accuracy: 0.9463
 49/469 [==>...........................] - ETA: 11s - loss: 0.0453 - accuracy: 0.9474
 51/469 [==>...........................] - ETA: 11s - loss: 0.0443 - accuracy: 0.9484
 53/469 [==>...........................] - ETA: 11s - loss: 0.0436 - accuracy: 0.9491
 55/469 [==>...........................] - ETA: 11s - loss: 0.0425 - accuracy: 0.9504
 57/469 [==>...........................] - ETA: 11s - loss: 0.0425 - accuracy: 0.9505
 59/469 [==>...........................] - ETA: 11s - loss: 0.0418 - accuracy: 0.9511
 61/469 [==>...........................] - ETA: 11s - loss: 0.0412 - accuracy: 0.9522
 63/469 [===>..........................] - ETA: 11s - loss: 0.0408 - accuracy: 0.9525
 65/469 [===>..........................] - ETA: 11s - loss: 0.0402 - accuracy: 0.9530
 67/469 [===>..........................] - ETA: 11s - loss: 0.0397 - accuracy: 0.9538
 69/469 [===>..........................] - ETA: 11s - loss: 0.0391 - accuracy: 0.9544
 71/469 [===>..........................] - ETA: 11s - loss: 0.0387 - accuracy: 0.9547
 73/469 [===>..........................] - ETA: 11s - loss: 0.0384 - accuracy: 0.9551
 75/469 [===>..........................] - ETA: 11s - loss: 0.0381 - accuracy: 0.9553
 77/469 [===>..........................] - ETA: 11s - loss: 0.0376 - accuracy: 0.9560
 79/469 [====>.........................] - ETA: 10s - loss: 0.0373 - accuracy: 0.9562
 81/469 [====>.........................] - ETA: 10s - loss: 0.0368 - accuracy: 0.9568
 83/469 [====>.........................] - ETA: 10s - loss: 0.0365 - accuracy: 0.9571
 85/469 [====>.........................] - ETA: 10s - loss: 0.0361 - accuracy: 0.9574
 87/469 [====>.........................] - ETA: 10s - loss: 0.0356 - accuracy: 0.9581
 89/469 [====>.........................] - ETA: 10s - loss: 0.0352 - accuracy: 0.9585
 91/469 [====>.........................] - ETA: 10s - loss: 0.0348 - accuracy: 0.9589
 93/469 [====>.........................] - ETA: 10s - loss: 0.0344 - accuracy: 0.9595
 95/469 [=====>........................] - ETA: 10s - loss: 0.0346 - accuracy: 0.9593
 97/469 [=====>........................] - ETA: 10s - loss: 0.0344 - accuracy: 0.9595
 99/469 [=====>........................] - ETA: 10s - loss: 0.0345 - accuracy: 0.9589
101/469 [=====>........................] - ETA: 10s - loss: 0.0341 - accuracy: 0.9595
103/469 [=====>........................] - ETA: 10s - loss: 0.0339 - accuracy: 0.9596
105/469 [=====>........................] - ETA: 10s - loss: 0.0336 - accuracy: 0.9601
107/469 [=====>........................] - ETA: 10s - loss: 0.0334 - accuracy: 0.9604
109/469 [=====>........................] - ETA: 10s - loss: 0.0334 - accuracy: 0.9604
111/469 [======>.......................] - ETA: 10s - loss: 0.0331 - accuracy: 0.9608
113/469 [======>.......................] - ETA: 10s - loss: 0.0329 - accuracy: 0.9613
115/469 [======>.......................] - ETA: 10s - loss: 0.0326 - accuracy: 0.9615
117/469 [======>.......................] - ETA: 10s - loss: 0.0324 - accuracy: 0.9619
119/469 [======>.......................] - ETA: 10s - loss: 0.0322 - accuracy: 0.9621
121/469 [======>.......................] - ETA: 10s - loss: 0.0319 - accuracy: 0.9624
123/469 [======>.......................] - ETA: 9s - loss: 0.0317 - accuracy: 0.9627 
125/469 [======>.......................] - ETA: 9s - loss: 0.0314 - accuracy: 0.9631
127/469 [=======>......................] - ETA: 9s - loss: 0.0312 - accuracy: 0.9634
129/469 [=======>......................] - ETA: 9s - loss: 0.0311 - accuracy: 0.9635
131/469 [=======>......................] - ETA: 9s - loss: 0.0308 - accuracy: 0.9637
133/469 [=======>......................] - ETA: 9s - loss: 0.0307 - accuracy: 0.9639
135/469 [=======>......................] - ETA: 9s - loss: 0.0303 - accuracy: 0.9644
137/469 [=======>......................] - ETA: 9s - loss: 0.0301 - accuracy: 0.9645
139/469 [=======>......................] - ETA: 9s - loss: 0.0300 - accuracy: 0.9647
141/469 [========>.....................] - ETA: 9s - loss: 0.0299 - accuracy: 0.9649
143/469 [========>.....................] - ETA: 9s - loss: 0.0297 - accuracy: 0.9652
145/469 [========>.....................] - ETA: 9s - loss: 0.0296 - accuracy: 0.9652
147/469 [========>.....................] - ETA: 9s - loss: 0.0295 - accuracy: 0.9655
149/469 [========>.....................] - ETA: 9s - loss: 0.0293 - accuracy: 0.9657
151/469 [========>.....................] - ETA: 9s - loss: 0.0292 - accuracy: 0.9659
153/469 [========>.....................] - ETA: 9s - loss: 0.0290 - accuracy: 0.9661
155/469 [========>.....................] - ETA: 9s - loss: 0.0289 - accuracy: 0.9663
157/469 [=========>....................] - ETA: 8s - loss: 0.0288 - accuracy: 0.9663
159/469 [=========>....................] - ETA: 8s - loss: 0.0286 - accuracy: 0.9666
161/469 [=========>....................] - ETA: 8s - loss: 0.0286 - accuracy: 0.9666
163/469 [=========>....................] - ETA: 8s - loss: 0.0284 - accuracy: 0.9668
165/469 [=========>....................] - ETA: 8s - loss: 0.0283 - accuracy: 0.9669
167/469 [=========>....................] - ETA: 8s - loss: 0.0283 - accuracy: 0.9668
169/469 [=========>....................] - ETA: 8s - loss: 0.0282 - accuracy: 0.9669
171/469 [=========>....................] - ETA: 8s - loss: 0.0281 - accuracy: 0.9670
173/469 [==========>...................] - ETA: 8s - loss: 0.0279 - accuracy: 0.9671
175/469 [==========>...................] - ETA: 8s - loss: 0.0278 - accuracy: 0.9675
177/469 [==========>...................] - ETA: 8s - loss: 0.0276 - accuracy: 0.9676
179/469 [==========>...................] - ETA: 8s - loss: 0.0275 - accuracy: 0.9677
181/469 [==========>...................] - ETA: 8s - loss: 0.0276 - accuracy: 0.9677
183/469 [==========>...................] - ETA: 8s - loss: 0.0274 - accuracy: 0.9679
185/469 [==========>...................] - ETA: 8s - loss: 0.0274 - accuracy: 0.9680
187/469 [==========>...................] - ETA: 8s - loss: 0.0273 - accuracy: 0.9681
189/469 [===========>..................] - ETA: 8s - loss: 0.0273 - accuracy: 0.9680
191/469 [===========>..................] - ETA: 7s - loss: 0.0272 - accuracy: 0.9681
193/469 [===========>..................] - ETA: 7s - loss: 0.0270 - accuracy: 0.9683
195/469 [===========>..................] - ETA: 7s - loss: 0.0269 - accuracy: 0.9684
197/469 [===========>..................] - ETA: 7s - loss: 0.0268 - accuracy: 0.9686
199/469 [===========>..................] - ETA: 7s - loss: 0.0266 - accuracy: 0.9688
201/469 [===========>..................] - ETA: 7s - loss: 0.0265 - accuracy: 0.9689
203/469 [===========>..................] - ETA: 7s - loss: 0.0265 - accuracy: 0.9689
205/469 [============>.................] - ETA: 7s - loss: 0.0263 - accuracy: 0.9691
207/469 [============>.................] - ETA: 7s - loss: 0.0262 - accuracy: 0.9692
209/469 [============>.................] - ETA: 7s - loss: 0.0260 - accuracy: 0.9695
211/469 [============>.................] - ETA: 7s - loss: 0.0261 - accuracy: 0.9695
213/469 [============>.................] - ETA: 7s - loss: 0.0260 - accuracy: 0.9696
215/469 [============>.................] - ETA: 7s - loss: 0.0258 - accuracy: 0.9698
217/469 [============>.................] - ETA: 7s - loss: 0.0257 - accuracy: 0.9699
219/469 [=============>................] - ETA: 7s - loss: 0.0256 - accuracy: 0.9700
221/469 [=============>................] - ETA: 7s - loss: 0.0256 - accuracy: 0.9700
223/469 [=============>................] - ETA: 7s - loss: 0.0255 - accuracy: 0.9701
225/469 [=============>................] - ETA: 6s - loss: 0.0254 - accuracy: 0.9703
227/469 [=============>................] - ETA: 6s - loss: 0.0253 - accuracy: 0.9704
229/469 [=============>................] - ETA: 6s - loss: 0.0252 - accuracy: 0.9705
231/469 [=============>................] - ETA: 6s - loss: 0.0251 - accuracy: 0.9705
233/469 [=============>................] - ETA: 6s - loss: 0.0250 - accuracy: 0.9707
235/469 [==============>...............] - ETA: 6s - loss: 0.0249 - accuracy: 0.9709
237/469 [==============>...............] - ETA: 6s - loss: 0.0248 - accuracy: 0.9710
239/469 [==============>...............] - ETA: 6s - loss: 0.0248 - accuracy: 0.9711
241/469 [==============>...............] - ETA: 6s - loss: 0.0247 - accuracy: 0.9711
243/469 [==============>...............] - ETA: 6s - loss: 0.0246 - accuracy: 0.9712
245/469 [==============>...............] - ETA: 6s - loss: 0.0244 - accuracy: 0.9714
247/469 [==============>...............] - ETA: 6s - loss: 0.0243 - accuracy: 0.9716
249/469 [==============>...............] - ETA: 6s - loss: 0.0243 - accuracy: 0.9718
251/469 [===============>..............] - ETA: 6s - loss: 0.0242 - accuracy: 0.9720
253/469 [===============>..............] - ETA: 6s - loss: 0.0241 - accuracy: 0.9721
255/469 [===============>..............] - ETA: 6s - loss: 0.0241 - accuracy: 0.9721
257/469 [===============>..............] - ETA: 6s - loss: 0.0240 - accuracy: 0.9722
259/469 [===============>..............] - ETA: 6s - loss: 0.0240 - accuracy: 0.9720
261/469 [===============>..............] - ETA: 5s - loss: 0.0241 - accuracy: 0.9720
263/469 [===============>..............] - ETA: 5s - loss: 0.0240 - accuracy: 0.9721
265/469 [===============>..............] - ETA: 5s - loss: 0.0239 - accuracy: 0.9721
267/469 [================>.............] - ETA: 5s - loss: 0.0239 - accuracy: 0.9722
269/469 [================>.............] - ETA: 5s - loss: 0.0238 - accuracy: 0.9723
271/469 [================>.............] - ETA: 5s - loss: 0.0237 - accuracy: 0.9725
273/469 [================>.............] - ETA: 5s - loss: 0.0237 - accuracy: 0.9725
275/469 [================>.............] - ETA: 5s - loss: 0.0236 - accuracy: 0.9726
277/469 [================>.............] - ETA: 5s - loss: 0.0235 - accuracy: 0.9726
279/469 [================>.............] - ETA: 5s - loss: 0.0235 - accuracy: 0.9727
281/469 [================>.............] - ETA: 5s - loss: 0.0234 - accuracy: 0.9728
283/469 [=================>............] - ETA: 5s - loss: 0.0234 - accuracy: 0.9728
285/469 [=================>............] - ETA: 5s - loss: 0.0233 - accuracy: 0.9729
287/469 [=================>............] - ETA: 5s - loss: 0.0232 - accuracy: 0.9731
289/469 [=================>............] - ETA: 5s - loss: 0.0232 - accuracy: 0.9731
291/469 [=================>............] - ETA: 5s - loss: 0.0231 - accuracy: 0.9732
293/469 [=================>............] - ETA: 5s - loss: 0.0230 - accuracy: 0.9733
295/469 [=================>............] - ETA: 5s - loss: 0.0229 - accuracy: 0.9734
297/469 [=================>............] - ETA: 4s - loss: 0.0229 - accuracy: 0.9735
299/469 [==================>...........] - ETA: 4s - loss: 0.0228 - accuracy: 0.9734
301/469 [==================>...........] - ETA: 4s - loss: 0.0229 - accuracy: 0.9733
303/469 [==================>...........] - ETA: 4s - loss: 0.0229 - accuracy: 0.9734
305/469 [==================>...........] - ETA: 4s - loss: 0.0228 - accuracy: 0.9735
307/469 [==================>...........] - ETA: 4s - loss: 0.0227 - accuracy: 0.9736
309/469 [==================>...........] - ETA: 4s - loss: 0.0227 - accuracy: 0.9736
311/469 [==================>...........] - ETA: 4s - loss: 0.0227 - accuracy: 0.9736
313/469 [===================>..........] - ETA: 4s - loss: 0.0227 - accuracy: 0.9736
315/469 [===================>..........] - ETA: 4s - loss: 0.0227 - accuracy: 0.9736
317/469 [===================>..........] - ETA: 4s - loss: 0.0226 - accuracy: 0.9737
319/469 [===================>..........] - ETA: 4s - loss: 0.0225 - accuracy: 0.9737
321/469 [===================>..........] - ETA: 4s - loss: 0.0226 - accuracy: 0.9737
323/469 [===================>..........] - ETA: 4s - loss: 0.0225 - accuracy: 0.9738
325/469 [===================>..........] - ETA: 4s - loss: 0.0224 - accuracy: 0.9738
327/469 [===================>..........] - ETA: 4s - loss: 0.0224 - accuracy: 0.9738
329/469 [====================>.........] - ETA: 4s - loss: 0.0224 - accuracy: 0.9738
331/469 [====================>.........] - ETA: 3s - loss: 0.0224 - accuracy: 0.9738
333/469 [====================>.........] - ETA: 3s - loss: 0.0224 - accuracy: 0.9738
335/469 [====================>.........] - ETA: 3s - loss: 0.0224 - accuracy: 0.9738
337/469 [====================>.........] - ETA: 3s - loss: 0.0223 - accuracy: 0.9739
339/469 [====================>.........] - ETA: 3s - loss: 0.0222 - accuracy: 0.9740
341/469 [====================>.........] - ETA: 3s - loss: 0.0222 - accuracy: 0.9740
343/469 [====================>.........] - ETA: 3s - loss: 0.0222 - accuracy: 0.9740
345/469 [=====================>........] - ETA: 3s - loss: 0.0222 - accuracy: 0.9740
347/469 [=====================>........] - ETA: 3s - loss: 0.0221 - accuracy: 0.9740
349/469 [=====================>........] - ETA: 3s - loss: 0.0221 - accuracy: 0.9741
351/469 [=====================>........] - ETA: 3s - loss: 0.0220 - accuracy: 0.9742
353/469 [=====================>........] - ETA: 3s - loss: 0.0219 - accuracy: 0.9743
355/469 [=====================>........] - ETA: 3s - loss: 0.0219 - accuracy: 0.9743
357/469 [=====================>........] - ETA: 3s - loss: 0.0218 - accuracy: 0.9743
359/469 [=====================>........] - ETA: 3s - loss: 0.0218 - accuracy: 0.9744
361/469 [======================>.......] - ETA: 3s - loss: 0.0218 - accuracy: 0.9744
363/469 [======================>.......] - ETA: 3s - loss: 0.0217 - accuracy: 0.9746
365/469 [======================>.......] - ETA: 2s - loss: 0.0216 - accuracy: 0.9746
367/469 [======================>.......] - ETA: 2s - loss: 0.0216 - accuracy: 0.9747
369/469 [======================>.......] - ETA: 2s - loss: 0.0215 - accuracy: 0.9748
371/469 [======================>.......] - ETA: 2s - loss: 0.0214 - accuracy: 0.9749
373/469 [======================>.......] - ETA: 2s - loss: 0.0214 - accuracy: 0.9748
375/469 [======================>.......] - ETA: 2s - loss: 0.0214 - accuracy: 0.9748
377/469 [=======================>......] - ETA: 2s - loss: 0.0214 - accuracy: 0.9749
379/469 [=======================>......] - ETA: 2s - loss: 0.0213 - accuracy: 0.9750
381/469 [=======================>......] - ETA: 2s - loss: 0.0213 - accuracy: 0.9749
383/469 [=======================>......] - ETA: 2s - loss: 0.0212 - accuracy: 0.9750
385/469 [=======================>......] - ETA: 2s - loss: 0.0212 - accuracy: 0.9750
387/469 [=======================>......] - ETA: 2s - loss: 0.0212 - accuracy: 0.9750
389/469 [=======================>......] - ETA: 2s - loss: 0.0212 - accuracy: 0.9751
391/469 [========================>.....] - ETA: 2s - loss: 0.0212 - accuracy: 0.9751
393/469 [========================>.....] - ETA: 2s - loss: 0.0211 - accuracy: 0.9752
395/469 [========================>.....] - ETA: 2s - loss: 0.0210 - accuracy: 0.9753
397/469 [========================>.....] - ETA: 2s - loss: 0.0210 - accuracy: 0.9753
399/469 [========================>.....] - ETA: 2s - loss: 0.0210 - accuracy: 0.9753
401/469 [========================>.....] - ETA: 1s - loss: 0.0210 - accuracy: 0.9753
403/469 [========================>.....] - ETA: 1s - loss: 0.0210 - accuracy: 0.9753
405/469 [========================>.....] - ETA: 1s - loss: 0.0210 - accuracy: 0.9754
407/469 [=========================>....] - ETA: 1s - loss: 0.0209 - accuracy: 0.9755
409/469 [=========================>....] - ETA: 1s - loss: 0.0209 - accuracy: 0.9755
411/469 [=========================>....] - ETA: 1s - loss: 0.0208 - accuracy: 0.9756
413/469 [=========================>....] - ETA: 1s - loss: 0.0208 - accuracy: 0.9756
415/469 [=========================>....] - ETA: 1s - loss: 0.0208 - accuracy: 0.9756
417/469 [=========================>....] - ETA: 1s - loss: 0.0207 - accuracy: 0.9757
419/469 [=========================>....] - ETA: 1s - loss: 0.0207 - accuracy: 0.9758
421/469 [=========================>....] - ETA: 1s - loss: 0.0206 - accuracy: 0.9758
423/469 [==========================>...] - ETA: 1s - loss: 0.0206 - accuracy: 0.9758
425/469 [==========================>...] - ETA: 1s - loss: 0.0206 - accuracy: 0.9759
427/469 [==========================>...] - ETA: 1s - loss: 0.0205 - accuracy: 0.9758
429/469 [==========================>...] - ETA: 1s - loss: 0.0205 - accuracy: 0.9759
431/469 [==========================>...] - ETA: 1s - loss: 0.0205 - accuracy: 0.9759
433/469 [==========================>...] - ETA: 1s - loss: 0.0204 - accuracy: 0.9759
435/469 [==========================>...] - ETA: 0s - loss: 0.0204 - accuracy: 0.9759
437/469 [==========================>...] - ETA: 0s - loss: 0.0204 - accuracy: 0.9760
439/469 [===========================>..] - ETA: 0s - loss: 0.0204 - accuracy: 0.9760
441/469 [===========================>..] - ETA: 0s - loss: 0.0203 - accuracy: 0.9760
443/469 [===========================>..] - ETA: 0s - loss: 0.0203 - accuracy: 0.9761
445/469 [===========================>..] - ETA: 0s - loss: 0.0203 - accuracy: 0.9761
447/469 [===========================>..] - ETA: 0s - loss: 0.0202 - accuracy: 0.9761
449/469 [===========================>..] - ETA: 0s - loss: 0.0202 - accuracy: 0.9762
451/469 [===========================>..] - ETA: 0s - loss: 0.0201 - accuracy: 0.9762
453/469 [===========================>..] - ETA: 0s - loss: 0.0201 - accuracy: 0.9763
455/469 [============================>.] - ETA: 0s - loss: 0.0201 - accuracy: 0.9763
457/469 [============================>.] - ETA: 0s - loss: 0.0201 - accuracy: 0.9763
459/469 [============================>.] - ETA: 0s - loss: 0.0200 - accuracy: 0.9764
461/469 [============================>.] - ETA: 0s - loss: 0.0200 - accuracy: 0.9764
463/469 [============================>.] - ETA: 0s - loss: 0.0200 - accuracy: 0.9765
465/469 [============================>.] - ETA: 0s - loss: 0.0199 - accuracy: 0.9765
467/469 [============================>.] - ETA: 0s - loss: 0.0199 - accuracy: 0.9765
469/469 [==============================] - ETA: 0s - loss: 0.0199 - accuracy: 0.9766
469/469 [==============================] - 14s 30ms/step - loss: 0.0199 - accuracy: 0.9766 - val_loss: 0.0155 - val_accuracy: 0.9819 - lr: 0.0010
Epoch 2/5

  1/469 [..............................] - ETA: 0s - loss: 0.0042 - accuracy: 1.0000
  3/469 [..............................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9896
  5/469 [..............................] - ETA: 10s - loss: 0.0098 - accuracy: 0.9875
  7/469 [..............................] - ETA: 11s - loss: 0.0100 - accuracy: 0.9844
  9/469 [..............................] - ETA: 11s - loss: 0.0095 - accuracy: 0.9844
 11/469 [..............................] - ETA: 11s - loss: 0.0090 - accuracy: 0.9858
 13/469 [..............................] - ETA: 11s - loss: 0.0091 - accuracy: 0.9850
 15/469 [..............................] - ETA: 12s - loss: 0.0097 - accuracy: 0.9839
 17/469 [>.............................] - ETA: 12s - loss: 0.0103 - accuracy: 0.9844
 19/469 [>.............................] - ETA: 12s - loss: 0.0099 - accuracy: 0.9848
 21/469 [>.............................] - ETA: 12s - loss: 0.0106 - accuracy: 0.9851
 23/469 [>.............................] - ETA: 12s - loss: 0.0103 - accuracy: 0.9857
 25/469 [>.............................] - ETA: 12s - loss: 0.0108 - accuracy: 0.9853
 27/469 [>.............................] - ETA: 12s - loss: 0.0105 - accuracy: 0.9858
 29/469 [>.............................] - ETA: 12s - loss: 0.0101 - accuracy: 0.9865
 31/469 [>.............................] - ETA: 12s - loss: 0.0104 - accuracy: 0.9861
 33/469 [=>............................] - ETA: 12s - loss: 0.0104 - accuracy: 0.9863
 35/469 [=>............................] - ETA: 11s - loss: 0.0102 - accuracy: 0.9868
 37/469 [=>............................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9865
 39/469 [=>............................] - ETA: 11s - loss: 0.0102 - accuracy: 0.9870
 41/469 [=>............................] - ETA: 11s - loss: 0.0101 - accuracy: 0.9874
 43/469 [=>............................] - ETA: 11s - loss: 0.0105 - accuracy: 0.9869
 45/469 [=>............................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9868
 47/469 [==>...........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9869
 49/469 [==>...........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9868
 51/469 [==>...........................] - ETA: 11s - loss: 0.0105 - accuracy: 0.9864
 53/469 [==>...........................] - ETA: 11s - loss: 0.0106 - accuracy: 0.9863
 55/469 [==>...........................] - ETA: 11s - loss: 0.0105 - accuracy: 0.9862
 57/469 [==>...........................] - ETA: 11s - loss: 0.0106 - accuracy: 0.9862
 59/469 [==>...........................] - ETA: 11s - loss: 0.0107 - accuracy: 0.9861
 61/469 [==>...........................] - ETA: 11s - loss: 0.0106 - accuracy: 0.9863
 63/469 [===>..........................] - ETA: 11s - loss: 0.0105 - accuracy: 0.9865
 65/469 [===>..........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9868
 67/469 [===>..........................] - ETA: 11s - loss: 0.0105 - accuracy: 0.9867
 69/469 [===>..........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9869
 71/469 [===>..........................] - ETA: 11s - loss: 0.0106 - accuracy: 0.9866
 73/469 [===>..........................] - ETA: 11s - loss: 0.0105 - accuracy: 0.9867
 75/469 [===>..........................] - ETA: 11s - loss: 0.0105 - accuracy: 0.9867
 77/469 [===>..........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9869
 79/469 [====>.........................] - ETA: 10s - loss: 0.0108 - accuracy: 0.9867
 81/469 [====>.........................] - ETA: 10s - loss: 0.0109 - accuracy: 0.9865
 83/469 [====>.........................] - ETA: 10s - loss: 0.0110 - accuracy: 0.9864
 85/469 [====>.........................] - ETA: 10s - loss: 0.0111 - accuracy: 0.9863
 87/469 [====>.........................] - ETA: 10s - loss: 0.0112 - accuracy: 0.9861
 89/469 [====>.........................] - ETA: 10s - loss: 0.0111 - accuracy: 0.9861
 91/469 [====>.........................] - ETA: 10s - loss: 0.0110 - accuracy: 0.9863
 93/469 [====>.........................] - ETA: 10s - loss: 0.0109 - accuracy: 0.9865
 95/469 [=====>........................] - ETA: 10s - loss: 0.0111 - accuracy: 0.9863
 97/469 [=====>........................] - ETA: 10s - loss: 0.0111 - accuracy: 0.9865
 99/469 [=====>........................] - ETA: 10s - loss: 0.0111 - accuracy: 0.9865
101/469 [=====>........................] - ETA: 10s - loss: 0.0110 - accuracy: 0.9866
103/469 [=====>........................] - ETA: 10s - loss: 0.0109 - accuracy: 0.9867
105/469 [=====>........................] - ETA: 10s - loss: 0.0108 - accuracy: 0.9870
107/469 [=====>........................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9871
109/469 [=====>........................] - ETA: 10s - loss: 0.0108 - accuracy: 0.9871
111/469 [======>.......................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9871
113/469 [======>.......................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9870
115/469 [======>.......................] - ETA: 9s - loss: 0.0109 - accuracy: 0.9870 
117/469 [======>.......................] - ETA: 9s - loss: 0.0111 - accuracy: 0.9870
119/469 [======>.......................] - ETA: 9s - loss: 0.0110 - accuracy: 0.9870
121/469 [======>.......................] - ETA: 9s - loss: 0.0109 - accuracy: 0.9872
123/469 [======>.......................] - ETA: 9s - loss: 0.0109 - accuracy: 0.9872
125/469 [======>.......................] - ETA: 9s - loss: 0.0109 - accuracy: 0.9871
127/469 [=======>......................] - ETA: 9s - loss: 0.0109 - accuracy: 0.9872
129/469 [=======>......................] - ETA: 9s - loss: 0.0108 - accuracy: 0.9872
131/469 [=======>......................] - ETA: 9s - loss: 0.0108 - accuracy: 0.9872
133/469 [=======>......................] - ETA: 9s - loss: 0.0108 - accuracy: 0.9871
135/469 [=======>......................] - ETA: 9s - loss: 0.0108 - accuracy: 0.9871
137/469 [=======>......................] - ETA: 9s - loss: 0.0107 - accuracy: 0.9872
139/469 [=======>......................] - ETA: 9s - loss: 0.0107 - accuracy: 0.9872
141/469 [========>.....................] - ETA: 9s - loss: 0.0106 - accuracy: 0.9873
143/469 [========>.....................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9873
145/469 [========>.....................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9874
147/469 [========>.....................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9875
149/469 [========>.....................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9873
151/469 [========>.....................] - ETA: 9s - loss: 0.0104 - accuracy: 0.9874
153/469 [========>.....................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9874
155/469 [========>.....................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9875
157/469 [=========>....................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9875
159/469 [=========>....................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9875
161/469 [=========>....................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9873
163/469 [=========>....................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9873
165/469 [=========>....................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9874
167/469 [=========>....................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9874
169/469 [=========>....................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9875
171/469 [=========>....................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9873
173/469 [==========>...................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9874
175/469 [==========>...................] - ETA: 8s - loss: 0.0102 - accuracy: 0.9875
177/469 [==========>...................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9874
179/469 [==========>...................] - ETA: 8s - loss: 0.0102 - accuracy: 0.9875
181/469 [==========>...................] - ETA: 8s - loss: 0.0102 - accuracy: 0.9875
183/469 [==========>...................] - ETA: 8s - loss: 0.0101 - accuracy: 0.9876
185/469 [==========>...................] - ETA: 8s - loss: 0.0102 - accuracy: 0.9875
187/469 [==========>...................] - ETA: 8s - loss: 0.0102 - accuracy: 0.9875
189/469 [===========>..................] - ETA: 8s - loss: 0.0101 - accuracy: 0.9875
191/469 [===========>..................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9876
193/469 [===========>..................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9875
195/469 [===========>..................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9875
197/469 [===========>..................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9875
199/469 [===========>..................] - ETA: 7s - loss: 0.0100 - accuracy: 0.9876
201/469 [===========>..................] - ETA: 7s - loss: 0.0100 - accuracy: 0.9877
203/469 [===========>..................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9876
205/469 [============>.................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9874
207/469 [============>.................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9875
209/469 [============>.................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9875
211/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9875
213/469 [============>.................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9874
215/469 [============>.................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9875
217/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9875
219/469 [=============>................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9875
221/469 [=============>................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9875
223/469 [=============>................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9875
225/469 [=============>................] - ETA: 6s - loss: 0.0102 - accuracy: 0.9875
227/469 [=============>................] - ETA: 6s - loss: 0.0102 - accuracy: 0.9876
229/469 [=============>................] - ETA: 6s - loss: 0.0102 - accuracy: 0.9877
231/469 [=============>................] - ETA: 6s - loss: 0.0102 - accuracy: 0.9876
233/469 [=============>................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9877
235/469 [==============>...............] - ETA: 6s - loss: 0.0101 - accuracy: 0.9877
237/469 [==============>...............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9876
239/469 [==============>...............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9875
241/469 [==============>...............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9876
243/469 [==============>...............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9876
245/469 [==============>...............] - ETA: 6s - loss: 0.0101 - accuracy: 0.9876
247/469 [==============>...............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9876
249/469 [==============>...............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9875
251/469 [===============>..............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9875
253/469 [===============>..............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9875
255/469 [===============>..............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9875
257/469 [===============>..............] - ETA: 6s - loss: 0.0102 - accuracy: 0.9875
259/469 [===============>..............] - ETA: 6s - loss: 0.0101 - accuracy: 0.9876
261/469 [===============>..............] - ETA: 5s - loss: 0.0101 - accuracy: 0.9877
263/469 [===============>..............] - ETA: 5s - loss: 0.0101 - accuracy: 0.9876
265/469 [===============>..............] - ETA: 5s - loss: 0.0101 - accuracy: 0.9877
267/469 [================>.............] - ETA: 5s - loss: 0.0101 - accuracy: 0.9877
269/469 [================>.............] - ETA: 5s - loss: 0.0101 - accuracy: 0.9877
271/469 [================>.............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9877
273/469 [================>.............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9877
275/469 [================>.............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9877
277/469 [================>.............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9877
279/469 [================>.............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9877
281/469 [================>.............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9877
283/469 [=================>............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9877
285/469 [=================>............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9877
287/469 [=================>............] - ETA: 5s - loss: 0.0102 - accuracy: 0.9878
289/469 [=================>............] - ETA: 5s - loss: 0.0101 - accuracy: 0.9878
291/469 [=================>............] - ETA: 5s - loss: 0.0101 - accuracy: 0.9878
293/469 [=================>............] - ETA: 5s - loss: 0.0101 - accuracy: 0.9879
295/469 [=================>............] - ETA: 4s - loss: 0.0100 - accuracy: 0.9879
297/469 [=================>............] - ETA: 4s - loss: 0.0100 - accuracy: 0.9880
299/469 [==================>...........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9879
301/469 [==================>...........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9880
303/469 [==================>...........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9879
305/469 [==================>...........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9879
307/469 [==================>...........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9879
309/469 [==================>...........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9879
311/469 [==================>...........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9880
313/469 [===================>..........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9879
315/469 [===================>..........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9880
317/469 [===================>..........] - ETA: 4s - loss: 0.0100 - accuracy: 0.9880
319/469 [===================>..........] - ETA: 4s - loss: 0.0099 - accuracy: 0.9880
321/469 [===================>..........] - ETA: 4s - loss: 0.0099 - accuracy: 0.9881
323/469 [===================>..........] - ETA: 4s - loss: 0.0099 - accuracy: 0.9881
325/469 [===================>..........] - ETA: 4s - loss: 0.0099 - accuracy: 0.9881
327/469 [===================>..........] - ETA: 4s - loss: 0.0099 - accuracy: 0.9881
329/469 [====================>.........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9881
331/469 [====================>.........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9881
333/469 [====================>.........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
335/469 [====================>.........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9881
337/469 [====================>.........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
339/469 [====================>.........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
341/469 [====================>.........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
343/469 [====================>.........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
345/469 [=====================>........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
347/469 [=====================>........] - ETA: 3s - loss: 0.0100 - accuracy: 0.9880
349/469 [=====================>........] - ETA: 3s - loss: 0.0100 - accuracy: 0.9880
351/469 [=====================>........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9881
353/469 [=====================>........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9881
355/469 [=====================>........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9881
357/469 [=====================>........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
359/469 [=====================>........] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
361/469 [======================>.......] - ETA: 3s - loss: 0.0099 - accuracy: 0.9880
363/469 [======================>.......] - ETA: 3s - loss: 0.0100 - accuracy: 0.9879
365/469 [======================>.......] - ETA: 2s - loss: 0.0099 - accuracy: 0.9880
367/469 [======================>.......] - ETA: 2s - loss: 0.0099 - accuracy: 0.9880
369/469 [======================>.......] - ETA: 2s - loss: 0.0099 - accuracy: 0.9880
371/469 [======================>.......] - ETA: 2s - loss: 0.0099 - accuracy: 0.9880
373/469 [======================>.......] - ETA: 2s - loss: 0.0099 - accuracy: 0.9881
375/469 [======================>.......] - ETA: 2s - loss: 0.0099 - accuracy: 0.9880
377/469 [=======================>......] - ETA: 2s - loss: 0.0099 - accuracy: 0.9881
379/469 [=======================>......] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
381/469 [=======================>......] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
383/469 [=======================>......] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
385/469 [=======================>......] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
387/469 [=======================>......] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
389/469 [=======================>......] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
391/469 [========================>.....] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
393/469 [========================>.....] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
395/469 [========================>.....] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
397/469 [========================>.....] - ETA: 2s - loss: 0.0098 - accuracy: 0.9881
399/469 [========================>.....] - ETA: 2s - loss: 0.0098 - accuracy: 0.9880
401/469 [========================>.....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9880
403/469 [========================>.....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9880
405/469 [========================>.....] - ETA: 1s - loss: 0.0099 - accuracy: 0.9880
407/469 [=========================>....] - ETA: 1s - loss: 0.0098 - accuracy: 0.9880
409/469 [=========================>....] - ETA: 1s - loss: 0.0098 - accuracy: 0.9881
411/469 [=========================>....] - ETA: 1s - loss: 0.0098 - accuracy: 0.9881
413/469 [=========================>....] - ETA: 1s - loss: 0.0098 - accuracy: 0.9881
415/469 [=========================>....] - ETA: 1s - loss: 0.0098 - accuracy: 0.9880
417/469 [=========================>....] - ETA: 1s - loss: 0.0098 - accuracy: 0.9880
419/469 [=========================>....] - ETA: 1s - loss: 0.0098 - accuracy: 0.9880
421/469 [=========================>....] - ETA: 1s - loss: 0.0098 - accuracy: 0.9880
423/469 [==========================>...] - ETA: 1s - loss: 0.0098 - accuracy: 0.9881
425/469 [==========================>...] - ETA: 1s - loss: 0.0098 - accuracy: 0.9880
427/469 [==========================>...] - ETA: 1s - loss: 0.0098 - accuracy: 0.9881
429/469 [==========================>...] - ETA: 1s - loss: 0.0097 - accuracy: 0.9881
431/469 [==========================>...] - ETA: 1s - loss: 0.0098 - accuracy: 0.9881
433/469 [==========================>...] - ETA: 1s - loss: 0.0098 - accuracy: 0.9881
435/469 [==========================>...] - ETA: 0s - loss: 0.0098 - accuracy: 0.9881
437/469 [==========================>...] - ETA: 0s - loss: 0.0097 - accuracy: 0.9882
439/469 [===========================>..] - ETA: 0s - loss: 0.0098 - accuracy: 0.9881
441/469 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9882
443/469 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9882
445/469 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
447/469 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
449/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9883
451/469 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
453/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9883
455/469 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
457/469 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
459/469 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
461/469 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
463/469 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
465/469 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
467/469 [============================>.] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
469/469 [==============================] - ETA: 0s - loss: 0.0097 - accuracy: 0.9883
469/469 [==============================] - 14s 30ms/step - loss: 0.0097 - accuracy: 0.9883 - val_loss: 0.0136 - val_accuracy: 0.9822 - lr: 6.3096e-04
Epoch 3/5

  1/469 [..............................] - ETA: 0s - loss: 8.7029e-04 - accuracy: 1.0000
  3/469 [..............................] - ETA: 8s - loss: 0.0028 - accuracy: 1.0000    
  5/469 [..............................] - ETA: 10s - loss: 0.0039 - accuracy: 0.9969
  7/469 [..............................] - ETA: 11s - loss: 0.0051 - accuracy: 0.9967
  9/469 [..............................] - ETA: 11s - loss: 0.0049 - accuracy: 0.9965
 11/469 [..............................] - ETA: 11s - loss: 0.0049 - accuracy: 0.9957
 13/469 [..............................] - ETA: 11s - loss: 0.0050 - accuracy: 0.9952
 15/469 [..............................] - ETA: 12s - loss: 0.0048 - accuracy: 0.9953
 17/469 [>.............................] - ETA: 12s - loss: 0.0049 - accuracy: 0.9949
 19/469 [>.............................] - ETA: 12s - loss: 0.0052 - accuracy: 0.9942
 21/469 [>.............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9933
 23/469 [>.............................] - ETA: 12s - loss: 0.0059 - accuracy: 0.9929
 25/469 [>.............................] - ETA: 12s - loss: 0.0064 - accuracy: 0.9916
 27/469 [>.............................] - ETA: 12s - loss: 0.0066 - accuracy: 0.9913
 29/469 [>.............................] - ETA: 12s - loss: 0.0066 - accuracy: 0.9914
 31/469 [>.............................] - ETA: 12s - loss: 0.0066 - accuracy: 0.9912
 33/469 [=>............................] - ETA: 11s - loss: 0.0066 - accuracy: 0.9910
 35/469 [=>............................] - ETA: 11s - loss: 0.0065 - accuracy: 0.9911
 37/469 [=>............................] - ETA: 11s - loss: 0.0066 - accuracy: 0.9911
 39/469 [=>............................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9914
 41/469 [=>............................] - ETA: 11s - loss: 0.0063 - accuracy: 0.9914
 43/469 [=>............................] - ETA: 11s - loss: 0.0063 - accuracy: 0.9915
 45/469 [=>............................] - ETA: 11s - loss: 0.0066 - accuracy: 0.9915
 47/469 [==>...........................] - ETA: 11s - loss: 0.0065 - accuracy: 0.9917
 49/469 [==>...........................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9919
 51/469 [==>...........................] - ETA: 11s - loss: 0.0065 - accuracy: 0.9917
 53/469 [==>...........................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9917
 55/469 [==>...........................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9918
 57/469 [==>...........................] - ETA: 11s - loss: 0.0063 - accuracy: 0.9919
 59/469 [==>...........................] - ETA: 11s - loss: 0.0062 - accuracy: 0.9921
 61/469 [==>...........................] - ETA: 11s - loss: 0.0062 - accuracy: 0.9923
 63/469 [===>..........................] - ETA: 11s - loss: 0.0063 - accuracy: 0.9921
 65/469 [===>..........................] - ETA: 11s - loss: 0.0065 - accuracy: 0.9918
 67/469 [===>..........................] - ETA: 11s - loss: 0.0065 - accuracy: 0.9921
 69/469 [===>..........................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9922
 71/469 [===>..........................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9921
 73/469 [===>..........................] - ETA: 11s - loss: 0.0063 - accuracy: 0.9922
 75/469 [===>..........................] - ETA: 11s - loss: 0.0062 - accuracy: 0.9924
 77/469 [===>..........................] - ETA: 11s - loss: 0.0062 - accuracy: 0.9924
 79/469 [====>.........................] - ETA: 11s - loss: 0.0063 - accuracy: 0.9923
 81/469 [====>.........................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9922
 83/469 [====>.........................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9924
 85/469 [====>.........................] - ETA: 11s - loss: 0.0063 - accuracy: 0.9925
 87/469 [====>.........................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9925
 89/469 [====>.........................] - ETA: 10s - loss: 0.0064 - accuracy: 0.9924
 91/469 [====>.........................] - ETA: 10s - loss: 0.0064 - accuracy: 0.9924
 93/469 [====>.........................] - ETA: 10s - loss: 0.0064 - accuracy: 0.9924
 95/469 [=====>........................] - ETA: 10s - loss: 0.0065 - accuracy: 0.9924
 97/469 [=====>........................] - ETA: 10s - loss: 0.0066 - accuracy: 0.9922
 99/469 [=====>........................] - ETA: 10s - loss: 0.0066 - accuracy: 0.9923
101/469 [=====>........................] - ETA: 10s - loss: 0.0066 - accuracy: 0.9922
103/469 [=====>........................] - ETA: 10s - loss: 0.0068 - accuracy: 0.9920
105/469 [=====>........................] - ETA: 10s - loss: 0.0069 - accuracy: 0.9918
107/469 [=====>........................] - ETA: 10s - loss: 0.0069 - accuracy: 0.9917
109/469 [=====>........................] - ETA: 10s - loss: 0.0068 - accuracy: 0.9918
111/469 [======>.......................] - ETA: 10s - loss: 0.0069 - accuracy: 0.9916
113/469 [======>.......................] - ETA: 10s - loss: 0.0068 - accuracy: 0.9918
115/469 [======>.......................] - ETA: 10s - loss: 0.0067 - accuracy: 0.9919
117/469 [======>.......................] - ETA: 10s - loss: 0.0067 - accuracy: 0.9918
119/469 [======>.......................] - ETA: 10s - loss: 0.0067 - accuracy: 0.9919
121/469 [======>.......................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9920 
123/469 [======>.......................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9921
125/469 [======>.......................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9920
127/469 [=======>......................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9919
129/469 [=======>......................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9920
131/469 [=======>......................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9921
133/469 [=======>......................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9920
135/469 [=======>......................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9920
137/469 [=======>......................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9920
139/469 [=======>......................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9919
141/469 [========>.....................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9919
143/469 [========>.....................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9919
145/469 [========>.....................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9919
147/469 [========>.....................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9920
149/469 [========>.....................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9919
151/469 [========>.....................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9919
153/469 [========>.....................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9918
155/469 [========>.....................] - ETA: 9s - loss: 0.0067 - accuracy: 0.9917
157/469 [=========>....................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9917
159/469 [=========>....................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9918
161/469 [=========>....................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9918
163/469 [=========>....................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9919
165/469 [=========>....................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9918
167/469 [=========>....................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9917
169/469 [=========>....................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9918
171/469 [=========>....................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9917
173/469 [==========>...................] - ETA: 8s - loss: 0.0066 - accuracy: 0.9917
175/469 [==========>...................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9916
177/469 [==========>...................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9917
179/469 [==========>...................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9916
181/469 [==========>...................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9917
183/469 [==========>...................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9916
185/469 [==========>...................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9916
187/469 [==========>...................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9916
189/469 [===========>..................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9916
191/469 [===========>..................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9916
193/469 [===========>..................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9917
195/469 [===========>..................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9916
197/469 [===========>..................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9916
199/469 [===========>..................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9916
201/469 [===========>..................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9916
203/469 [===========>..................] - ETA: 7s - loss: 0.0067 - accuracy: 0.9915
205/469 [============>.................] - ETA: 7s - loss: 0.0069 - accuracy: 0.9913
207/469 [============>.................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9914
209/469 [============>.................] - ETA: 7s - loss: 0.0069 - accuracy: 0.9913
211/469 [============>.................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9914
213/469 [============>.................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9914
215/469 [============>.................] - ETA: 7s - loss: 0.0069 - accuracy: 0.9913
217/469 [============>.................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9914
219/469 [=============>................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9913
221/469 [=============>................] - ETA: 7s - loss: 0.0069 - accuracy: 0.9913
223/469 [=============>................] - ETA: 7s - loss: 0.0068 - accuracy: 0.9913
225/469 [=============>................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
227/469 [=============>................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9913
229/469 [=============>................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
231/469 [=============>................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
233/469 [=============>................] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
235/469 [==============>...............] - ETA: 6s - loss: 0.0067 - accuracy: 0.9915
237/469 [==============>...............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
239/469 [==============>...............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
241/469 [==============>...............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9913
243/469 [==============>...............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
245/469 [==============>...............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
247/469 [==============>...............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
249/469 [==============>...............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
251/469 [===============>..............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
253/469 [===============>..............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
255/469 [===============>..............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9915
257/469 [===============>..............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
259/469 [===============>..............] - ETA: 6s - loss: 0.0068 - accuracy: 0.9914
261/469 [===============>..............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9914
263/469 [===============>..............] - ETA: 5s - loss: 0.0067 - accuracy: 0.9915
265/469 [===============>..............] - ETA: 5s - loss: 0.0067 - accuracy: 0.9915
267/469 [================>.............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9915
269/469 [================>.............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9915
271/469 [================>.............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9915
273/469 [================>.............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9915
275/469 [================>.............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9916
277/469 [================>.............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9915
279/469 [================>.............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9915
281/469 [================>.............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9915
283/469 [=================>............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9916
285/469 [=================>............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9916
287/469 [=================>............] - ETA: 5s - loss: 0.0068 - accuracy: 0.9916
289/469 [=================>............] - ETA: 5s - loss: 0.0067 - accuracy: 0.9917
291/469 [=================>............] - ETA: 5s - loss: 0.0067 - accuracy: 0.9917
293/469 [=================>............] - ETA: 5s - loss: 0.0067 - accuracy: 0.9917
295/469 [=================>............] - ETA: 5s - loss: 0.0067 - accuracy: 0.9917
297/469 [=================>............] - ETA: 4s - loss: 0.0068 - accuracy: 0.9917
299/469 [==================>...........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9917
301/469 [==================>...........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
303/469 [==================>...........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
305/469 [==================>...........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
307/469 [==================>...........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
309/469 [==================>...........] - ETA: 4s - loss: 0.0068 - accuracy: 0.9918
311/469 [==================>...........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
313/469 [===================>..........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
315/469 [===================>..........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
317/469 [===================>..........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
319/469 [===================>..........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
321/469 [===================>..........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
323/469 [===================>..........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
325/469 [===================>..........] - ETA: 4s - loss: 0.0067 - accuracy: 0.9918
327/469 [===================>..........] - ETA: 4s - loss: 0.0068 - accuracy: 0.9916
329/469 [====================>.........] - ETA: 4s - loss: 0.0068 - accuracy: 0.9916
331/469 [====================>.........] - ETA: 3s - loss: 0.0068 - accuracy: 0.9916
333/469 [====================>.........] - ETA: 3s - loss: 0.0068 - accuracy: 0.9916
335/469 [====================>.........] - ETA: 3s - loss: 0.0068 - accuracy: 0.9916
337/469 [====================>.........] - ETA: 3s - loss: 0.0068 - accuracy: 0.9916
339/469 [====================>.........] - ETA: 3s - loss: 0.0068 - accuracy: 0.9917
341/469 [====================>.........] - ETA: 3s - loss: 0.0068 - accuracy: 0.9916
343/469 [====================>.........] - ETA: 3s - loss: 0.0068 - accuracy: 0.9916
345/469 [=====================>........] - ETA: 3s - loss: 0.0068 - accuracy: 0.9916
347/469 [=====================>........] - ETA: 3s - loss: 0.0069 - accuracy: 0.9916
349/469 [=====================>........] - ETA: 3s - loss: 0.0069 - accuracy: 0.9916
351/469 [=====================>........] - ETA: 3s - loss: 0.0069 - accuracy: 0.9916
353/469 [=====================>........] - ETA: 3s - loss: 0.0069 - accuracy: 0.9915
355/469 [=====================>........] - ETA: 3s - loss: 0.0069 - accuracy: 0.9915
357/469 [=====================>........] - ETA: 3s - loss: 0.0069 - accuracy: 0.9915
359/469 [=====================>........] - ETA: 3s - loss: 0.0069 - accuracy: 0.9915
361/469 [======================>.......] - ETA: 3s - loss: 0.0069 - accuracy: 0.9915
363/469 [======================>.......] - ETA: 3s - loss: 0.0069 - accuracy: 0.9915
365/469 [======================>.......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9915
367/469 [======================>.......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
369/469 [======================>.......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
371/469 [======================>.......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9915
373/469 [======================>.......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9915
375/469 [======================>.......] - ETA: 2s - loss: 0.0069 - accuracy: 0.9915
377/469 [=======================>......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
379/469 [=======================>......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
381/469 [=======================>......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
383/469 [=======================>......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
385/469 [=======================>......] - ETA: 2s - loss: 0.0069 - accuracy: 0.9914
387/469 [=======================>......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
389/469 [=======================>......] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
391/469 [========================>.....] - ETA: 2s - loss: 0.0070 - accuracy: 0.9914
393/469 [========================>.....] - ETA: 2s - loss: 0.0069 - accuracy: 0.9914
395/469 [========================>.....] - ETA: 2s - loss: 0.0069 - accuracy: 0.9914
397/469 [========================>.....] - ETA: 2s - loss: 0.0069 - accuracy: 0.9914
399/469 [========================>.....] - ETA: 2s - loss: 0.0069 - accuracy: 0.9915
401/469 [========================>.....] - ETA: 1s - loss: 0.0069 - accuracy: 0.9915
403/469 [========================>.....] - ETA: 1s - loss: 0.0069 - accuracy: 0.9915
405/469 [========================>.....] - ETA: 1s - loss: 0.0069 - accuracy: 0.9914
407/469 [=========================>....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
409/469 [=========================>....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
411/469 [=========================>....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
413/469 [=========================>....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
415/469 [=========================>....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
417/469 [=========================>....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
419/469 [=========================>....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
421/469 [=========================>....] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
423/469 [==========================>...] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
425/469 [==========================>...] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
427/469 [==========================>...] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
429/469 [==========================>...] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
431/469 [==========================>...] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
433/469 [==========================>...] - ETA: 1s - loss: 0.0070 - accuracy: 0.9914
435/469 [==========================>...] - ETA: 0s - loss: 0.0070 - accuracy: 0.9914
437/469 [==========================>...] - ETA: 0s - loss: 0.0070 - accuracy: 0.9914
439/469 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9915
441/469 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9915
443/469 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9915
445/469 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9915
447/469 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9915
449/469 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9915
451/469 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9915
453/469 [===========================>..] - ETA: 0s - loss: 0.0069 - accuracy: 0.9916
455/469 [============================>.] - ETA: 0s - loss: 0.0069 - accuracy: 0.9916
457/469 [============================>.] - ETA: 0s - loss: 0.0069 - accuracy: 0.9916
459/469 [============================>.] - ETA: 0s - loss: 0.0069 - accuracy: 0.9916
461/469 [============================>.] - ETA: 0s - loss: 0.0068 - accuracy: 0.9916
463/469 [============================>.] - ETA: 0s - loss: 0.0069 - accuracy: 0.9916
465/469 [============================>.] - ETA: 0s - loss: 0.0069 - accuracy: 0.9916
467/469 [============================>.] - ETA: 0s - loss: 0.0069 - accuracy: 0.9916
469/469 [==============================] - ETA: 0s - loss: 0.0069 - accuracy: 0.9916
469/469 [==============================] - 14s 30ms/step - loss: 0.0069 - accuracy: 0.9916 - val_loss: 0.0118 - val_accuracy: 0.9858 - lr: 3.9811e-04
Epoch 4/5

  1/469 [..............................] - ETA: 0s - loss: 0.0106 - accuracy: 0.9922
  3/469 [..............................] - ETA: 8s - loss: 0.0060 - accuracy: 0.9948
  5/469 [..............................] - ETA: 10s - loss: 0.0072 - accuracy: 0.9922
  7/469 [..............................] - ETA: 11s - loss: 0.0065 - accuracy: 0.9922
  9/469 [..............................] - ETA: 11s - loss: 0.0064 - accuracy: 0.9922
 11/469 [..............................] - ETA: 11s - loss: 0.0062 - accuracy: 0.9929
 13/469 [..............................] - ETA: 11s - loss: 0.0053 - accuracy: 0.9940
 15/469 [..............................] - ETA: 12s - loss: 0.0054 - accuracy: 0.9937
 17/469 [>.............................] - ETA: 12s - loss: 0.0054 - accuracy: 0.9936
 19/469 [>.............................] - ETA: 12s - loss: 0.0054 - accuracy: 0.9930
 21/469 [>.............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9929
 23/469 [>.............................] - ETA: 12s - loss: 0.0054 - accuracy: 0.9929
 25/469 [>.............................] - ETA: 12s - loss: 0.0056 - accuracy: 0.9928
 27/469 [>.............................] - ETA: 12s - loss: 0.0056 - accuracy: 0.9931
 29/469 [>.............................] - ETA: 12s - loss: 0.0057 - accuracy: 0.9930
 31/469 [>.............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9934
 33/469 [=>............................] - ETA: 12s - loss: 0.0058 - accuracy: 0.9929
 35/469 [=>............................] - ETA: 12s - loss: 0.0056 - accuracy: 0.9931
 37/469 [=>............................] - ETA: 11s - loss: 0.0054 - accuracy: 0.9935
 39/469 [=>............................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9934
 41/469 [=>............................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9933
 43/469 [=>............................] - ETA: 11s - loss: 0.0057 - accuracy: 0.9933
 45/469 [=>............................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9936
 47/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9937
 49/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9938
 51/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9939
 53/469 [==>...........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9941
 55/469 [==>...........................] - ETA: 11s - loss: 0.0054 - accuracy: 0.9942
 57/469 [==>...........................] - ETA: 11s - loss: 0.0053 - accuracy: 0.9942
 59/469 [==>...........................] - ETA: 11s - loss: 0.0053 - accuracy: 0.9943
 61/469 [==>...........................] - ETA: 11s - loss: 0.0052 - accuracy: 0.9945
 63/469 [===>..........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9939
 65/469 [===>..........................] - ETA: 11s - loss: 0.0058 - accuracy: 0.9936
 67/469 [===>..........................] - ETA: 11s - loss: 0.0057 - accuracy: 0.9936
 69/469 [===>..........................] - ETA: 11s - loss: 0.0058 - accuracy: 0.9935
 71/469 [===>..........................] - ETA: 11s - loss: 0.0058 - accuracy: 0.9935
 73/469 [===>..........................] - ETA: 11s - loss: 0.0058 - accuracy: 0.9936
 75/469 [===>..........................] - ETA: 11s - loss: 0.0058 - accuracy: 0.9935
 77/469 [===>..........................] - ETA: 11s - loss: 0.0057 - accuracy: 0.9937
 79/469 [====>.........................] - ETA: 10s - loss: 0.0057 - accuracy: 0.9937
 81/469 [====>.........................] - ETA: 10s - loss: 0.0057 - accuracy: 0.9936
 83/469 [====>.........................] - ETA: 10s - loss: 0.0057 - accuracy: 0.9936
 85/469 [====>.........................] - ETA: 10s - loss: 0.0057 - accuracy: 0.9936
 87/469 [====>.........................] - ETA: 10s - loss: 0.0056 - accuracy: 0.9936
 89/469 [====>.........................] - ETA: 10s - loss: 0.0056 - accuracy: 0.9937
 91/469 [====>.........................] - ETA: 10s - loss: 0.0055 - accuracy: 0.9937
 93/469 [====>.........................] - ETA: 10s - loss: 0.0055 - accuracy: 0.9938
 95/469 [=====>........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9939
 97/469 [=====>........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9940
 99/469 [=====>........................] - ETA: 10s - loss: 0.0055 - accuracy: 0.9939
101/469 [=====>........................] - ETA: 10s - loss: 0.0055 - accuracy: 0.9938
103/469 [=====>........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9939
105/469 [=====>........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9939
107/469 [=====>........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9939
109/469 [=====>........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9938
111/469 [======>.......................] - ETA: 10s - loss: 0.0055 - accuracy: 0.9938
113/469 [======>.......................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9938
115/469 [======>.......................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9939 
117/469 [======>.......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9939
119/469 [======>.......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
121/469 [======>.......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
123/469 [======>.......................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9939
125/469 [======>.......................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9940
127/469 [=======>......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
129/469 [=======>......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
131/469 [=======>......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
133/469 [=======>......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9939
135/469 [=======>......................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9938
137/469 [=======>......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9939
139/469 [=======>......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
141/469 [========>.....................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
143/469 [========>.....................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
145/469 [========>.....................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
147/469 [========>.....................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9939
149/469 [========>.....................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9939
151/469 [========>.....................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9940
153/469 [========>.....................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9939
155/469 [========>.....................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9940
157/469 [=========>....................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9939
159/469 [=========>....................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9939
161/469 [=========>....................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9939
163/469 [=========>....................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9939
165/469 [=========>....................] - ETA: 8s - loss: 0.0055 - accuracy: 0.9938
167/469 [=========>....................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9939
169/469 [=========>....................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9939
171/469 [=========>....................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9940
173/469 [==========>...................] - ETA: 8s - loss: 0.0054 - accuracy: 0.9941
175/469 [==========>...................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9941
177/469 [==========>...................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9941
179/469 [==========>...................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9942
181/469 [==========>...................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9942
183/469 [==========>...................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9942
185/469 [==========>...................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9941
187/469 [==========>...................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9942
189/469 [===========>..................] - ETA: 8s - loss: 0.0053 - accuracy: 0.9941
191/469 [===========>..................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9942
193/469 [===========>..................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9940
195/469 [===========>..................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9940
197/469 [===========>..................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9940
199/469 [===========>..................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9940
201/469 [===========>..................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9940
203/469 [===========>..................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9939
205/469 [============>.................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9939
207/469 [============>.................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9938
209/469 [============>.................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9939
211/469 [============>.................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9939
213/469 [============>.................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9939
215/469 [============>.................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9940
217/469 [============>.................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9940
219/469 [=============>................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9940
221/469 [=============>................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9938
223/469 [=============>................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9939
225/469 [=============>................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9939
227/469 [=============>................] - ETA: 6s - loss: 0.0053 - accuracy: 0.9939
229/469 [=============>................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9938
231/469 [=============>................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9937
233/469 [=============>................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9937
235/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9937
237/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9937
239/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9937
241/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9936
243/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9936
245/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9937
247/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9936
249/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9936
251/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9936
253/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9936
255/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9936
257/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9936
259/469 [===============>..............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9936
261/469 [===============>..............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9936
263/469 [===============>..............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9936
265/469 [===============>..............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9935
267/469 [================>.............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9935
269/469 [================>.............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9934
271/469 [================>.............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9934
273/469 [================>.............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9934
275/469 [================>.............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9935
277/469 [================>.............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9934
279/469 [================>.............] - ETA: 5s - loss: 0.0056 - accuracy: 0.9933
281/469 [================>.............] - ETA: 5s - loss: 0.0056 - accuracy: 0.9933
283/469 [=================>............] - ETA: 5s - loss: 0.0057 - accuracy: 0.9932
285/469 [=================>............] - ETA: 5s - loss: 0.0057 - accuracy: 0.9932
287/469 [=================>............] - ETA: 5s - loss: 0.0057 - accuracy: 0.9932
289/469 [=================>............] - ETA: 5s - loss: 0.0056 - accuracy: 0.9932
291/469 [=================>............] - ETA: 5s - loss: 0.0056 - accuracy: 0.9933
293/469 [=================>............] - ETA: 5s - loss: 0.0057 - accuracy: 0.9932
295/469 [=================>............] - ETA: 4s - loss: 0.0057 - accuracy: 0.9932
297/469 [=================>............] - ETA: 4s - loss: 0.0057 - accuracy: 0.9932
299/469 [==================>...........] - ETA: 4s - loss: 0.0058 - accuracy: 0.9931
301/469 [==================>...........] - ETA: 4s - loss: 0.0058 - accuracy: 0.9931
303/469 [==================>...........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9932
305/469 [==================>...........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9932
307/469 [==================>...........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9932
309/469 [==================>...........] - ETA: 4s - loss: 0.0058 - accuracy: 0.9931
311/469 [==================>...........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9931
313/469 [===================>..........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9932
315/469 [===================>..........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9932
317/469 [===================>..........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9931
319/469 [===================>..........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9931
321/469 [===================>..........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9931
323/469 [===================>..........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9931
325/469 [===================>..........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9931
327/469 [===================>..........] - ETA: 4s - loss: 0.0057 - accuracy: 0.9930
329/469 [====================>.........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
331/469 [====================>.........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
333/469 [====================>.........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9930
335/469 [====================>.........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
337/469 [====================>.........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
339/469 [====================>.........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
341/469 [====================>.........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
343/469 [====================>.........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
345/469 [=====================>........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
347/469 [=====================>........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
349/469 [=====================>........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
351/469 [=====================>........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
353/469 [=====================>........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
355/469 [=====================>........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
357/469 [=====================>........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
359/469 [=====================>........] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
361/469 [======================>.......] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
363/469 [======================>.......] - ETA: 3s - loss: 0.0057 - accuracy: 0.9931
365/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9931
367/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9931
369/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9931
371/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9931
373/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9931
375/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9931
377/469 [=======================>......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9931
379/469 [=======================>......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
381/469 [=======================>......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
383/469 [=======================>......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
385/469 [=======================>......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
387/469 [=======================>......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9933
389/469 [=======================>......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9933
391/469 [========================>.....] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
393/469 [========================>.....] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
395/469 [========================>.....] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
397/469 [========================>.....] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
399/469 [========================>.....] - ETA: 2s - loss: 0.0057 - accuracy: 0.9932
401/469 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
403/469 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9931
405/469 [========================>.....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
407/469 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
409/469 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
411/469 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
413/469 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
415/469 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
417/469 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
419/469 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
421/469 [=========================>....] - ETA: 1s - loss: 0.0057 - accuracy: 0.9932
423/469 [==========================>...] - ETA: 1s - loss: 0.0057 - accuracy: 0.9931
425/469 [==========================>...] - ETA: 1s - loss: 0.0057 - accuracy: 0.9931
427/469 [==========================>...] - ETA: 1s - loss: 0.0058 - accuracy: 0.9931
429/469 [==========================>...] - ETA: 1s - loss: 0.0057 - accuracy: 0.9931
431/469 [==========================>...] - ETA: 1s - loss: 0.0057 - accuracy: 0.9931
433/469 [==========================>...] - ETA: 1s - loss: 0.0057 - accuracy: 0.9931
435/469 [==========================>...] - ETA: 0s - loss: 0.0057 - accuracy: 0.9930
437/469 [==========================>...] - ETA: 0s - loss: 0.0057 - accuracy: 0.9930
439/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
441/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9930
443/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
445/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
447/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
449/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
451/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
453/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
455/469 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
457/469 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
459/469 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9932
461/469 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
463/469 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
465/469 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
467/469 [============================>.] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
469/469 [==============================] - ETA: 0s - loss: 0.0057 - accuracy: 0.9931
469/469 [==============================] - 14s 30ms/step - loss: 0.0057 - accuracy: 0.9931 - val_loss: 0.0102 - val_accuracy: 0.9886 - lr: 2.5119e-04
Epoch 5/5

  1/469 [..............................] - ETA: 0s - loss: 0.0087 - accuracy: 0.9922
  3/469 [..............................] - ETA: 8s - loss: 0.0041 - accuracy: 0.9974
  5/469 [..............................] - ETA: 10s - loss: 0.0039 - accuracy: 0.9969
  7/469 [..............................] - ETA: 11s - loss: 0.0044 - accuracy: 0.9944
  9/469 [..............................] - ETA: 11s - loss: 0.0045 - accuracy: 0.9939
 11/469 [..............................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9929
 13/469 [..............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9934
 15/469 [..............................] - ETA: 12s - loss: 0.0054 - accuracy: 0.9937
 17/469 [>.............................] - ETA: 12s - loss: 0.0053 - accuracy: 0.9940
 19/469 [>.............................] - ETA: 12s - loss: 0.0057 - accuracy: 0.9930
 21/469 [>.............................] - ETA: 12s - loss: 0.0054 - accuracy: 0.9937
 23/469 [>.............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9939
 25/469 [>.............................] - ETA: 12s - loss: 0.0059 - accuracy: 0.9931
 27/469 [>.............................] - ETA: 12s - loss: 0.0061 - accuracy: 0.9925
 29/469 [>.............................] - ETA: 12s - loss: 0.0058 - accuracy: 0.9930
 31/469 [>.............................] - ETA: 12s - loss: 0.0057 - accuracy: 0.9929
 33/469 [=>............................] - ETA: 12s - loss: 0.0056 - accuracy: 0.9929
 35/469 [=>............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9931
 37/469 [=>............................] - ETA: 12s - loss: 0.0056 - accuracy: 0.9932
 39/469 [=>............................] - ETA: 12s - loss: 0.0054 - accuracy: 0.9936
 41/469 [=>............................] - ETA: 12s - loss: 0.0053 - accuracy: 0.9937
 43/469 [=>............................] - ETA: 12s - loss: 0.0053 - accuracy: 0.9938
 45/469 [=>............................] - ETA: 12s - loss: 0.0051 - accuracy: 0.9941
 47/469 [==>...........................] - ETA: 12s - loss: 0.0051 - accuracy: 0.9942
 49/469 [==>...........................] - ETA: 12s - loss: 0.0051 - accuracy: 0.9941
 51/469 [==>...........................] - ETA: 12s - loss: 0.0050 - accuracy: 0.9942
 53/469 [==>...........................] - ETA: 12s - loss: 0.0049 - accuracy: 0.9944
 55/469 [==>...........................] - ETA: 12s - loss: 0.0048 - accuracy: 0.9945
 57/469 [==>...........................] - ETA: 11s - loss: 0.0048 - accuracy: 0.9945
 59/469 [==>...........................] - ETA: 11s - loss: 0.0048 - accuracy: 0.9944
 61/469 [==>...........................] - ETA: 11s - loss: 0.0048 - accuracy: 0.9945
 63/469 [===>..........................] - ETA: 11s - loss: 0.0049 - accuracy: 0.9943
 65/469 [===>..........................] - ETA: 11s - loss: 0.0049 - accuracy: 0.9945
 67/469 [===>..........................] - ETA: 11s - loss: 0.0049 - accuracy: 0.9944
 69/469 [===>..........................] - ETA: 11s - loss: 0.0050 - accuracy: 0.9943
 71/469 [===>..........................] - ETA: 11s - loss: 0.0051 - accuracy: 0.9941
 73/469 [===>..........................] - ETA: 11s - loss: 0.0051 - accuracy: 0.9940
 75/469 [===>..........................] - ETA: 11s - loss: 0.0051 - accuracy: 0.9940
 77/469 [===>..........................] - ETA: 11s - loss: 0.0052 - accuracy: 0.9939
 79/469 [====>.........................] - ETA: 11s - loss: 0.0051 - accuracy: 0.9941
 81/469 [====>.........................] - ETA: 11s - loss: 0.0051 - accuracy: 0.9940
 83/469 [====>.........................] - ETA: 11s - loss: 0.0051 - accuracy: 0.9942
 85/469 [====>.........................] - ETA: 11s - loss: 0.0051 - accuracy: 0.9941
 87/469 [====>.........................] - ETA: 11s - loss: 0.0050 - accuracy: 0.9943
 89/469 [====>.........................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9943
 91/469 [====>.........................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9942
 93/469 [====>.........................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9941
 95/469 [=====>........................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9942
 97/469 [=====>........................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9943
 99/469 [=====>........................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9944
101/469 [=====>........................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9943
103/469 [=====>........................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9943
105/469 [=====>........................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9940
107/469 [=====>........................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9940
109/469 [=====>........................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9941
111/469 [======>.......................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9941
113/469 [======>.......................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9941
115/469 [======>.......................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9941
117/469 [======>.......................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9941
119/469 [======>.......................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9941
121/469 [======>.......................] - ETA: 10s - loss: 0.0050 - accuracy: 0.9940
123/469 [======>.......................] - ETA: 9s - loss: 0.0050 - accuracy: 0.9940 
125/469 [======>.......................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9941
127/469 [=======>......................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9942
129/469 [=======>......................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9941
131/469 [=======>......................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9941
133/469 [=======>......................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9940
135/469 [=======>......................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9941
137/469 [=======>......................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9941
139/469 [=======>......................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9941
141/469 [========>.....................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9941
143/469 [========>.....................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9942
145/469 [========>.....................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9941
147/469 [========>.....................] - ETA: 9s - loss: 0.0049 - accuracy: 0.9942
149/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9942
151/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9943
153/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9943
155/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9943
157/469 [=========>....................] - ETA: 8s - loss: 0.0048 - accuracy: 0.9943
159/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9943
161/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9944
163/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9944
165/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9945
167/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9945
169/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9945
171/469 [=========>....................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9945
173/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9945
175/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9946
177/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9947
179/469 [==========>...................] - ETA: 8s - loss: 0.0045 - accuracy: 0.9947
181/469 [==========>...................] - ETA: 8s - loss: 0.0045 - accuracy: 0.9947
183/469 [==========>...................] - ETA: 8s - loss: 0.0045 - accuracy: 0.9947
185/469 [==========>...................] - ETA: 8s - loss: 0.0045 - accuracy: 0.9947
187/469 [==========>...................] - ETA: 8s - loss: 0.0045 - accuracy: 0.9947
189/469 [===========>..................] - ETA: 8s - loss: 0.0045 - accuracy: 0.9947
191/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9946
193/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9945
195/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9946
197/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9946
199/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
201/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9946
203/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9946
205/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
207/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
209/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
211/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
213/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
215/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
217/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
219/469 [=============>................] - ETA: 7s - loss: 0.0044 - accuracy: 0.9948
221/469 [=============>................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
223/469 [=============>................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9947
225/469 [=============>................] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
227/469 [=============>................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9947
229/469 [=============>................] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
231/469 [=============>................] - ETA: 6s - loss: 0.0045 - accuracy: 0.9947
233/469 [=============>................] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
235/469 [==============>...............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
237/469 [==============>...............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
239/469 [==============>...............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
241/469 [==============>...............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
243/469 [==============>...............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
245/469 [==============>...............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9946
247/469 [==============>...............] - ETA: 6s - loss: 0.0045 - accuracy: 0.9947
249/469 [==============>...............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
251/469 [===============>..............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9947
253/469 [===============>..............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9948
255/469 [===============>..............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9948
257/469 [===============>..............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9948
259/469 [===============>..............] - ETA: 6s - loss: 0.0044 - accuracy: 0.9949
261/469 [===============>..............] - ETA: 5s - loss: 0.0044 - accuracy: 0.9949
263/469 [===============>..............] - ETA: 5s - loss: 0.0044 - accuracy: 0.9949
265/469 [===============>..............] - ETA: 5s - loss: 0.0044 - accuracy: 0.9949
267/469 [================>.............] - ETA: 5s - loss: 0.0044 - accuracy: 0.9949
269/469 [================>.............] - ETA: 5s - loss: 0.0044 - accuracy: 0.9949
271/469 [================>.............] - ETA: 5s - loss: 0.0044 - accuracy: 0.9949
273/469 [================>.............] - ETA: 5s - loss: 0.0044 - accuracy: 0.9949
275/469 [================>.............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
277/469 [================>.............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
279/469 [================>.............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
281/469 [================>.............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
283/469 [=================>............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9951
285/469 [=================>............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
287/469 [=================>............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
289/469 [=================>............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9951
291/469 [=================>............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
293/469 [=================>............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
295/469 [=================>............] - ETA: 5s - loss: 0.0043 - accuracy: 0.9950
297/469 [=================>............] - ETA: 4s - loss: 0.0043 - accuracy: 0.9950
299/469 [==================>...........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
301/469 [==================>...........] - ETA: 4s - loss: 0.0042 - accuracy: 0.9951
303/469 [==================>...........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9950
305/469 [==================>...........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
307/469 [==================>...........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9950
309/469 [==================>...........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9950
311/469 [==================>...........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9950
313/469 [===================>..........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9950
315/469 [===================>..........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
317/469 [===================>..........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
319/469 [===================>..........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
321/469 [===================>..........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
323/469 [===================>..........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
325/469 [===================>..........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
327/469 [===================>..........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
329/469 [====================>.........] - ETA: 4s - loss: 0.0043 - accuracy: 0.9951
331/469 [====================>.........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
333/469 [====================>.........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
335/469 [====================>.........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
337/469 [====================>.........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
339/469 [====================>.........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
341/469 [====================>.........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
343/469 [====================>.........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
345/469 [=====================>........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
347/469 [=====================>........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9950
349/469 [=====================>........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
351/469 [=====================>........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
353/469 [=====================>........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
355/469 [=====================>........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
357/469 [=====================>........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
359/469 [=====================>........] - ETA: 3s - loss: 0.0043 - accuracy: 0.9951
361/469 [======================>.......] - ETA: 3s - loss: 0.0043 - accuracy: 0.9952
363/469 [======================>.......] - ETA: 3s - loss: 0.0043 - accuracy: 0.9952
365/469 [======================>.......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
367/469 [======================>.......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
369/469 [======================>.......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
371/469 [======================>.......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
373/469 [======================>.......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
375/469 [======================>.......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
377/469 [=======================>......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
379/469 [=======================>......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
381/469 [=======================>......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9950
383/469 [=======================>......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9950
385/469 [=======================>......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9950
387/469 [=======================>......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9950
389/469 [=======================>......] - ETA: 2s - loss: 0.0043 - accuracy: 0.9950
391/469 [========================>.....] - ETA: 2s - loss: 0.0044 - accuracy: 0.9950
393/469 [========================>.....] - ETA: 2s - loss: 0.0043 - accuracy: 0.9950
395/469 [========================>.....] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
397/469 [========================>.....] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
399/469 [========================>.....] - ETA: 2s - loss: 0.0043 - accuracy: 0.9951
401/469 [========================>.....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9951
403/469 [========================>.....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9951
405/469 [========================>.....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9951
407/469 [=========================>....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9950
409/469 [=========================>....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9950
411/469 [=========================>....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9950
413/469 [=========================>....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9949
415/469 [=========================>....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9950
417/469 [=========================>....] - ETA: 1s - loss: 0.0043 - accuracy: 0.9950
419/469 [=========================>....] - ETA: 1s - loss: 0.0044 - accuracy: 0.9949
421/469 [=========================>....] - ETA: 1s - loss: 0.0044 - accuracy: 0.9950
423/469 [==========================>...] - ETA: 1s - loss: 0.0044 - accuracy: 0.9950
425/469 [==========================>...] - ETA: 1s - loss: 0.0044 - accuracy: 0.9949
427/469 [==========================>...] - ETA: 1s - loss: 0.0044 - accuracy: 0.9949
429/469 [==========================>...] - ETA: 1s - loss: 0.0044 - accuracy: 0.9949
431/469 [==========================>...] - ETA: 1s - loss: 0.0044 - accuracy: 0.9949
433/469 [==========================>...] - ETA: 1s - loss: 0.0044 - accuracy: 0.9949
435/469 [==========================>...] - ETA: 0s - loss: 0.0044 - accuracy: 0.9950
437/469 [==========================>...] - ETA: 0s - loss: 0.0044 - accuracy: 0.9949
439/469 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9949
441/469 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9949
443/469 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9949
445/469 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9948
447/469 [===========================>..] - ETA: 0s - loss: 0.0044 - accuracy: 0.9948
449/469 [===========================>..] - ETA: 0s - loss: 0.0045 - accuracy: 0.9948
451/469 [===========================>..] - ETA: 0s - loss: 0.0045 - accuracy: 0.9948
453/469 [===========================>..] - ETA: 0s - loss: 0.0045 - accuracy: 0.9948
455/469 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9947
457/469 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9947
459/469 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9947
461/469 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9947
463/469 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9947
465/469 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9947
467/469 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9947
469/469 [==============================] - ETA: 0s - loss: 0.0045 - accuracy: 0.9947
469/469 [==============================] - 14s 30ms/step - loss: 0.0045 - accuracy: 0.9947 - val_loss: 0.0080 - val_accuracy: 0.9897 - lr: 1.5849e-04
Test score: 0.008030521683394909
Test accuracy: 0.989799976348877

5. Convert trained model for Akida and test

5.1 Final conversion

Convert the quantized model to a version suitable to be used in the Akida NSoC in inference mode:

Note

One needs to supply the coefficients used to rescale the input dataset before the training - here input_scaling.

As with Keras, the summary() method provides a textual representation of the model.

from cnn2snn import convert

model_akida = convert(model_keras, input_scaling=input_scaling)
model_akida.summary()

Out:

                        Model Summary
______________________________________________________________
Layer (type)                 Output shape  Kernel shape
==============================================================
conv_0 (InputConvolutional)  [14, 14, 32]  (5, 5, 1, 32)
______________________________________________________________
conv_1 (Convolutional)       [7, 7, 32]    (5, 5, 32, 32)
______________________________________________________________
conv_2 (Convolutional)       [7, 7, 32]    (5, 5, 32, 32)
______________________________________________________________
dense_2 (FullyConnected)     [1, 1, 512]   (1, 1, 1568, 512)
______________________________________________________________
dense_3 (FullyConnected)     [1, 1, 10]    (1, 1, 512, 10)
______________________________________________________________
Input shape: 28, 28, 1
Backend type: Software - 1.8.4

5.2 Performances check with the Akida Execution Engine

num_samples = 1000

results = model_akida.predict(raw_x_test[:num_samples])
accuracy = accuracy_score(raw_y_test[:num_samples], results[:num_samples])

print("Accuracy: " + "{0:.2f}".format(100 * accuracy) + "%")

# For non-regression purpose
assert accuracy > 0.95

# Print model statistics
print("Model statistics")
stats = model_akida.get_statistics()
model_akida.predict(raw_x_test[:20])
for _, stat in stats.items():
    print(stat)

Out:

Accuracy: 99.00%
Model statistics
Layer (type)                  output sparsity
conv_0 (InputConvolutional)   0.86
Layer (type)                  input sparsity      output sparsity     ops
conv_1 (Convolutional)        0.86                0.79                679880
Layer (type)                  input sparsity      output sparsity     ops
conv_2 (Convolutional)        0.79                0.70                267520
Layer (type)                  input sparsity      output sparsity     ops
dense_2 (FullyConnected)      0.70                0.69                243814
Layer (type)                  input sparsity      output sparsity     ops
dense_3 (FullyConnected)      0.69                0.00                1600

Depending on the number of samples you run, you should find a performance of around 99% (better results can be achieved using more epochs for training).

Note

Akida-compatible model first layer type is InputConvolutional and holds underlying data to spike conversion (please refer to Akida Execution Engine documentation for more details).

Total running time of the script: ( 3 minutes 14.417 seconds)

Gallery generated by Sphinx-Gallery