CNN conversion flow tutorial

The CNN2SNN tool is based on Keras, TensorFlow high-level API for building and training deep learning models.

Note

Please refer to TensorFlow tf.keras.models module for model creation/import details and TensorFlow Guide for details of how TensorFlow works.

CNN2SNN tool allows you to convert CNN networks to SNN networks compatible with the Akida NSoC in a few steps.

Note

MNIST example below is light enough so you do not need a GPU to run the CNN2SNN tool.

../_images/cnn2snn_flow_small.png

1. System configuration

1.1 Load CNN2SNN tool dependencies

# System imports
import os
import sys
import numpy as np
from sklearn.metrics import accuracy_score
from tempfile import TemporaryDirectory

# TensorFlow imports
import tensorflow.keras.backend as K
from tensorflow.keras.callbacks import LearningRateScheduler
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Conv2D, Dense, BatchNormalization, MaxPooling2D, Activation, ReLU, Flatten, Input
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical

1.2 Load and reshape MNIST dataset

After loading, we make 3 transformations on the dataset:

  1. Reshape the sample content data (x values) into a num_samples x width x height x channels matrix.

Note

At this point, we’ll set aside the raw data for testing our converted model in the Akida Execution Engine later

  1. Rescale the 8-bit loaded data to the range 0-to-1 for training.

Note

This shift makes almost no difference in the current example, but for some datasets rescaling the absolute values (and also shifting to zero-mean) can make a really major difference.

Also note that we store the scaling values input_scaling for use when preparing the model for the Akida Execution Engine. The implementation of the Akida neural network allows us to completely skip the rescaling step (i.e. the Akida model should be fed with the raw 8-bit values) but that does require information about what scaling was applied prior to training - see below for more details.

3. Transform the loaded labels from a scalar representation (single integer value per sample) to a one-hot vector representation, appropriate for use with the squared hinge loss function used in the current model.

Note

Input data normalization is a common step dealing with CNN (rationale is to keep data in a range that works with selected optimizers, some interesting reading can be found here.

# Load MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Reshape x-data
x_train = x_train.reshape(60000, 28, 28, 1)
x_test = x_test.reshape(10000, 28, 28, 1)

# Set aside raw test data for use with Akida Execution Engine later
raw_x_test = x_test.astype('uint8')
raw_y_test = y_test

# Rescale x-data
a = 255
b = 0
input_scaling = (a, b)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train = (x_train - b) / a
x_test = (x_test - b) / a

# Transform scalar labels to one-hot representation, scaled to +/- 1 appropriate for squared hinge loss function
y_train = to_categorical(y_train, 10) * 2 - 1
y_test = to_categorical(y_test, 10) * 2 - 1

1.3 Set training parameters

Set some training parameters used across the different training sessions:

# Set dataset relative training parameters
epochs = 5
batch_size = 128

# Set the learning rate parameters
lr_start = 1e-3
lr_end = 1e-4
lr_decay = (lr_end / lr_start)**(1. / epochs)

2. Model creation and performance check

2.1 Model creation

Note that at this stage, there is nothing specific to the Akida NSoC. This start point is very much a completely standard CNN as defined within Keras.

An appropriate model for MNIST (inspired by this paper) might look something like the following:

img_input = Input(shape=(28, 28, 1))
x = Conv2D(filters=32,
           kernel_size=(5, 5),
           padding='same',
           use_bias=False,
           data_format='channels_last')(img_input)
x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(x)
x = BatchNormalization()(x)
x = ReLU(6.)(x)

x = Conv2D(filters=32, kernel_size=(5, 5), padding='same', use_bias=False)(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(x)
x = BatchNormalization()(x)
x = ReLU(6.)(x)

x = Conv2D(filters=32, kernel_size=(5, 5), padding='same', use_bias=False)(x)
x = BatchNormalization()(x)
x = ReLU(6.)(x)

x = Flatten()(x)
x = Dense(512, use_bias=False)(x)
x = BatchNormalization()(x)
x = ReLU(6.)(x)
x = Dense(10, use_bias=False)(x)

model_keras = Model(img_input, x, name='mnistnet')

opt = Adam(lr=lr_start)
model_keras.compile(loss='squared_hinge', optimizer=opt, metrics=['accuracy'])
model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_2 (InputLayer)         [(None, 28, 28, 1)]       0
_________________________________________________________________
conv2d (Conv2D)              (None, 28, 28, 32)        800
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 32)        0
_________________________________________________________________
batch_normalization (BatchNo (None, 14, 14, 32)        128
_________________________________________________________________
re_lu (ReLU)                 (None, 14, 14, 32)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 14, 14, 32)        25600
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 32)          0
_________________________________________________________________
batch_normalization_1 (Batch (None, 7, 7, 32)          128
_________________________________________________________________
re_lu_1 (ReLU)               (None, 7, 7, 32)          0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 7, 7, 32)          25600
_________________________________________________________________
batch_normalization_2 (Batch (None, 7, 7, 32)          128
_________________________________________________________________
re_lu_2 (ReLU)               (None, 7, 7, 32)          0
_________________________________________________________________
flatten (Flatten)            (None, 1568)              0
_________________________________________________________________
dense (Dense)                (None, 512)               802816
_________________________________________________________________
batch_normalization_3 (Batch (None, 512)               2048
_________________________________________________________________
re_lu_3 (ReLU)               (None, 512)               0
_________________________________________________________________
dense_1 (Dense)              (None, 10)                5120
=================================================================
Total params: 862,368
Trainable params: 861,152
Non-trainable params: 1,216
_________________________________________________________________

Note

Adam optimizer is commonly used, more details can be found here.

2.2 Performance check

Before going any further, check the current model performance as a benchmark for CNN2SNN conversion. The created model should achieve a test accuracy a little over 99% after 5 epochs:

callbacks = []
lr_scheduler = LearningRateScheduler(lambda e: lr_start * lr_decay**e)
callbacks.append(lr_scheduler)
history = model_keras.fit(x_train,
                          y_train,
                          batch_size=batch_size,
                          epochs=epochs,
                          verbose=1,
                          validation_data=(x_test, y_test),
                          callbacks=callbacks)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/5

  1/469 [..............................] - ETA: 0s - loss: 2.3636 - accuracy: 0.0938
  4/469 [..............................] - ETA: 8s - loss: 1.4297 - accuracy: 0.4883
  7/469 [..............................] - ETA: 9s - loss: 0.9811 - accuracy: 0.6060
 10/469 [..............................] - ETA: 10s - loss: 0.7422 - accuracy: 0.6797
 12/469 [..............................] - ETA: 10s - loss: 0.6360 - accuracy: 0.7135
 15/469 [..............................] - ETA: 10s - loss: 0.5241 - accuracy: 0.7531
 18/469 [>.............................] - ETA: 10s - loss: 0.4470 - accuracy: 0.7817
 21/469 [>.............................] - ETA: 10s - loss: 0.3928 - accuracy: 0.8025
 24/469 [>.............................] - ETA: 10s - loss: 0.3508 - accuracy: 0.8197
 27/469 [>.............................] - ETA: 10s - loss: 0.3170 - accuracy: 0.8342
 30/469 [>.............................] - ETA: 10s - loss: 0.2906 - accuracy: 0.8451
 33/469 [=>............................] - ETA: 10s - loss: 0.2683 - accuracy: 0.8546
 36/469 [=>............................] - ETA: 10s - loss: 0.2484 - accuracy: 0.8646
 39/469 [=>............................] - ETA: 10s - loss: 0.2341 - accuracy: 0.8698
 42/469 [=>............................] - ETA: 10s - loss: 0.2200 - accuracy: 0.8761
 45/469 [=>............................] - ETA: 10s - loss: 0.2076 - accuracy: 0.8814
 48/469 [==>...........................] - ETA: 9s - loss: 0.1966 - accuracy: 0.8870 
 51/469 [==>...........................] - ETA: 9s - loss: 0.1869 - accuracy: 0.8915
 54/469 [==>...........................] - ETA: 9s - loss: 0.1785 - accuracy: 0.8955
 57/469 [==>...........................] - ETA: 9s - loss: 0.1704 - accuracy: 0.8995
 60/469 [==>...........................] - ETA: 9s - loss: 0.1630 - accuracy: 0.9033
 63/469 [===>..........................] - ETA: 9s - loss: 0.1563 - accuracy: 0.9067
 66/469 [===>..........................] - ETA: 9s - loss: 0.1504 - accuracy: 0.9097
 69/469 [===>..........................] - ETA: 9s - loss: 0.1448 - accuracy: 0.9125
 72/469 [===>..........................] - ETA: 9s - loss: 0.1400 - accuracy: 0.9153
 75/469 [===>..........................] - ETA: 9s - loss: 0.1353 - accuracy: 0.9176
 78/469 [===>..........................] - ETA: 9s - loss: 0.1311 - accuracy: 0.9195
 81/469 [====>.........................] - ETA: 9s - loss: 0.1274 - accuracy: 0.9213
 84/469 [====>.........................] - ETA: 9s - loss: 0.1240 - accuracy: 0.9232
 87/469 [====>.........................] - ETA: 9s - loss: 0.1204 - accuracy: 0.9253
 90/469 [====>.........................] - ETA: 9s - loss: 0.1170 - accuracy: 0.9274
 93/469 [====>.........................] - ETA: 9s - loss: 0.1137 - accuracy: 0.9294
 96/469 [=====>........................] - ETA: 8s - loss: 0.1109 - accuracy: 0.9306
 99/469 [=====>........................] - ETA: 8s - loss: 0.1082 - accuracy: 0.9321
102/469 [=====>........................] - ETA: 8s - loss: 0.1056 - accuracy: 0.9331
105/469 [=====>........................] - ETA: 8s - loss: 0.1031 - accuracy: 0.9344
108/469 [=====>........................] - ETA: 8s - loss: 0.1010 - accuracy: 0.9356
111/469 [======>.......................] - ETA: 8s - loss: 0.0988 - accuracy: 0.9368
114/469 [======>.......................] - ETA: 8s - loss: 0.0969 - accuracy: 0.9377
117/469 [======>.......................] - ETA: 8s - loss: 0.0947 - accuracy: 0.9390
120/469 [======>.......................] - ETA: 8s - loss: 0.0929 - accuracy: 0.9398
123/469 [======>.......................] - ETA: 8s - loss: 0.0910 - accuracy: 0.9409
126/469 [=======>......................] - ETA: 8s - loss: 0.0893 - accuracy: 0.9415
129/469 [=======>......................] - ETA: 8s - loss: 0.0877 - accuracy: 0.9423
131/469 [=======>......................] - ETA: 8s - loss: 0.0867 - accuracy: 0.9429
134/469 [=======>......................] - ETA: 8s - loss: 0.0853 - accuracy: 0.9436
137/469 [=======>......................] - ETA: 7s - loss: 0.0839 - accuracy: 0.9445
140/469 [=======>......................] - ETA: 7s - loss: 0.0826 - accuracy: 0.9449
143/469 [========>.....................] - ETA: 7s - loss: 0.0811 - accuracy: 0.9458
146/469 [========>.....................] - ETA: 7s - loss: 0.0801 - accuracy: 0.9463
149/469 [========>.....................] - ETA: 7s - loss: 0.0788 - accuracy: 0.9472
152/469 [========>.....................] - ETA: 7s - loss: 0.0776 - accuracy: 0.9479
155/469 [========>.....................] - ETA: 7s - loss: 0.0765 - accuracy: 0.9485
158/469 [=========>....................] - ETA: 7s - loss: 0.0754 - accuracy: 0.9490
161/469 [=========>....................] - ETA: 7s - loss: 0.0743 - accuracy: 0.9496
164/469 [=========>....................] - ETA: 7s - loss: 0.0732 - accuracy: 0.9504
167/469 [=========>....................] - ETA: 7s - loss: 0.0724 - accuracy: 0.9510
170/469 [=========>....................] - ETA: 7s - loss: 0.0715 - accuracy: 0.9514
173/469 [==========>...................] - ETA: 7s - loss: 0.0707 - accuracy: 0.9516
176/469 [==========>...................] - ETA: 7s - loss: 0.0697 - accuracy: 0.9521
179/469 [==========>...................] - ETA: 6s - loss: 0.0688 - accuracy: 0.9524
182/469 [==========>...................] - ETA: 6s - loss: 0.0681 - accuracy: 0.9530
185/469 [==========>...................] - ETA: 6s - loss: 0.0672 - accuracy: 0.9535
188/469 [===========>..................] - ETA: 6s - loss: 0.0664 - accuracy: 0.9540
191/469 [===========>..................] - ETA: 6s - loss: 0.0655 - accuracy: 0.9546
194/469 [===========>..................] - ETA: 6s - loss: 0.0647 - accuracy: 0.9549
197/469 [===========>..................] - ETA: 6s - loss: 0.0642 - accuracy: 0.9551
200/469 [===========>..................] - ETA: 6s - loss: 0.0635 - accuracy: 0.9555
203/469 [===========>..................] - ETA: 6s - loss: 0.0628 - accuracy: 0.9560
206/469 [============>.................] - ETA: 6s - loss: 0.0622 - accuracy: 0.9563
209/469 [============>.................] - ETA: 6s - loss: 0.0615 - accuracy: 0.9568
212/469 [============>.................] - ETA: 6s - loss: 0.0610 - accuracy: 0.9571
215/469 [============>.................] - ETA: 6s - loss: 0.0603 - accuracy: 0.9574
218/469 [============>.................] - ETA: 6s - loss: 0.0597 - accuracy: 0.9577
221/469 [=============>................] - ETA: 5s - loss: 0.0591 - accuracy: 0.9581
224/469 [=============>................] - ETA: 5s - loss: 0.0586 - accuracy: 0.9585
227/469 [=============>................] - ETA: 5s - loss: 0.0581 - accuracy: 0.9588
230/469 [=============>................] - ETA: 5s - loss: 0.0577 - accuracy: 0.9590
233/469 [=============>................] - ETA: 5s - loss: 0.0571 - accuracy: 0.9593
236/469 [==============>...............] - ETA: 5s - loss: 0.0566 - accuracy: 0.9596
238/469 [==============>...............] - ETA: 5s - loss: 0.0562 - accuracy: 0.9599
241/469 [==============>...............] - ETA: 5s - loss: 0.0558 - accuracy: 0.9601
244/469 [==============>...............] - ETA: 5s - loss: 0.0553 - accuracy: 0.9603
247/469 [==============>...............] - ETA: 5s - loss: 0.0548 - accuracy: 0.9605
250/469 [==============>...............] - ETA: 5s - loss: 0.0542 - accuracy: 0.9608
253/469 [===============>..............] - ETA: 5s - loss: 0.0538 - accuracy: 0.9611
256/469 [===============>..............] - ETA: 5s - loss: 0.0533 - accuracy: 0.9615
259/469 [===============>..............] - ETA: 5s - loss: 0.0529 - accuracy: 0.9616
262/469 [===============>..............] - ETA: 4s - loss: 0.0525 - accuracy: 0.9618
265/469 [===============>..............] - ETA: 4s - loss: 0.0520 - accuracy: 0.9621
268/469 [================>.............] - ETA: 4s - loss: 0.0516 - accuracy: 0.9623
271/469 [================>.............] - ETA: 4s - loss: 0.0512 - accuracy: 0.9626
274/469 [================>.............] - ETA: 4s - loss: 0.0508 - accuracy: 0.9627
277/469 [================>.............] - ETA: 4s - loss: 0.0505 - accuracy: 0.9630
280/469 [================>.............] - ETA: 4s - loss: 0.0501 - accuracy: 0.9633
283/469 [=================>............] - ETA: 4s - loss: 0.0497 - accuracy: 0.9635
286/469 [=================>............] - ETA: 4s - loss: 0.0494 - accuracy: 0.9636
289/469 [=================>............] - ETA: 4s - loss: 0.0491 - accuracy: 0.9637
292/469 [=================>............] - ETA: 4s - loss: 0.0488 - accuracy: 0.9639
295/469 [=================>............] - ETA: 4s - loss: 0.0484 - accuracy: 0.9641
298/469 [==================>...........] - ETA: 4s - loss: 0.0480 - accuracy: 0.9644
301/469 [==================>...........] - ETA: 4s - loss: 0.0478 - accuracy: 0.9645
304/469 [==================>...........] - ETA: 3s - loss: 0.0475 - accuracy: 0.9648
307/469 [==================>...........] - ETA: 3s - loss: 0.0471 - accuracy: 0.9650
310/469 [==================>...........] - ETA: 3s - loss: 0.0468 - accuracy: 0.9652
313/469 [===================>..........] - ETA: 3s - loss: 0.0465 - accuracy: 0.9653
316/469 [===================>..........] - ETA: 3s - loss: 0.0462 - accuracy: 0.9655
319/469 [===================>..........] - ETA: 3s - loss: 0.0459 - accuracy: 0.9657
322/469 [===================>..........] - ETA: 3s - loss: 0.0456 - accuracy: 0.9660
325/469 [===================>..........] - ETA: 3s - loss: 0.0453 - accuracy: 0.9662
328/469 [===================>..........] - ETA: 3s - loss: 0.0450 - accuracy: 0.9663
331/469 [====================>.........] - ETA: 3s - loss: 0.0447 - accuracy: 0.9664
334/469 [====================>.........] - ETA: 3s - loss: 0.0444 - accuracy: 0.9666
337/469 [====================>.........] - ETA: 3s - loss: 0.0441 - accuracy: 0.9668
340/469 [====================>.........] - ETA: 3s - loss: 0.0438 - accuracy: 0.9670
343/469 [====================>.........] - ETA: 3s - loss: 0.0436 - accuracy: 0.9671
346/469 [=====================>........] - ETA: 2s - loss: 0.0433 - accuracy: 0.9672
349/469 [=====================>........] - ETA: 2s - loss: 0.0431 - accuracy: 0.9673
352/469 [=====================>........] - ETA: 2s - loss: 0.0428 - accuracy: 0.9674
355/469 [=====================>........] - ETA: 2s - loss: 0.0426 - accuracy: 0.9676
358/469 [=====================>........] - ETA: 2s - loss: 0.0424 - accuracy: 0.9677
361/469 [======================>.......] - ETA: 2s - loss: 0.0421 - accuracy: 0.9679
364/469 [======================>.......] - ETA: 2s - loss: 0.0419 - accuracy: 0.9681
367/469 [======================>.......] - ETA: 2s - loss: 0.0416 - accuracy: 0.9683
370/469 [======================>.......] - ETA: 2s - loss: 0.0413 - accuracy: 0.9685
373/469 [======================>.......] - ETA: 2s - loss: 0.0411 - accuracy: 0.9686
376/469 [=======================>......] - ETA: 2s - loss: 0.0409 - accuracy: 0.9686
379/469 [=======================>......] - ETA: 2s - loss: 0.0407 - accuracy: 0.9688
382/469 [=======================>......] - ETA: 2s - loss: 0.0405 - accuracy: 0.9690
385/469 [=======================>......] - ETA: 2s - loss: 0.0403 - accuracy: 0.9691
388/469 [=======================>......] - ETA: 1s - loss: 0.0400 - accuracy: 0.9692
391/469 [========================>.....] - ETA: 1s - loss: 0.0399 - accuracy: 0.9693
394/469 [========================>.....] - ETA: 1s - loss: 0.0397 - accuracy: 0.9694
397/469 [========================>.....] - ETA: 1s - loss: 0.0395 - accuracy: 0.9696
400/469 [========================>.....] - ETA: 1s - loss: 0.0393 - accuracy: 0.9697
403/469 [========================>.....] - ETA: 1s - loss: 0.0392 - accuracy: 0.9697
406/469 [========================>.....] - ETA: 1s - loss: 0.0389 - accuracy: 0.9699
409/469 [=========================>....] - ETA: 1s - loss: 0.0388 - accuracy: 0.9700
412/469 [=========================>....] - ETA: 1s - loss: 0.0386 - accuracy: 0.9701
415/469 [=========================>....] - ETA: 1s - loss: 0.0384 - accuracy: 0.9702
418/469 [=========================>....] - ETA: 1s - loss: 0.0382 - accuracy: 0.9702
421/469 [=========================>....] - ETA: 1s - loss: 0.0381 - accuracy: 0.9703
424/469 [==========================>...] - ETA: 1s - loss: 0.0380 - accuracy: 0.9704
427/469 [==========================>...] - ETA: 1s - loss: 0.0378 - accuracy: 0.9705
430/469 [==========================>...] - ETA: 0s - loss: 0.0376 - accuracy: 0.9706
433/469 [==========================>...] - ETA: 0s - loss: 0.0374 - accuracy: 0.9707
436/469 [==========================>...] - ETA: 0s - loss: 0.0372 - accuracy: 0.9708
439/469 [===========================>..] - ETA: 0s - loss: 0.0371 - accuracy: 0.9710
442/469 [===========================>..] - ETA: 0s - loss: 0.0369 - accuracy: 0.9711
445/469 [===========================>..] - ETA: 0s - loss: 0.0367 - accuracy: 0.9713
448/469 [===========================>..] - ETA: 0s - loss: 0.0365 - accuracy: 0.9713
451/469 [===========================>..] - ETA: 0s - loss: 0.0364 - accuracy: 0.9715
454/469 [============================>.] - ETA: 0s - loss: 0.0362 - accuracy: 0.9716
457/469 [============================>.] - ETA: 0s - loss: 0.0361 - accuracy: 0.9716
460/469 [============================>.] - ETA: 0s - loss: 0.0359 - accuracy: 0.9717
463/469 [============================>.] - ETA: 0s - loss: 0.0358 - accuracy: 0.9718
466/469 [============================>.] - ETA: 0s - loss: 0.0356 - accuracy: 0.9719
469/469 [==============================] - ETA: 0s - loss: 0.0355 - accuracy: 0.9720
469/469 [==============================] - 12s 25ms/step - loss: 0.0355 - accuracy: 0.9720 - val_loss: 0.0345 - val_accuracy: 0.9602 - lr: 0.0010
Epoch 2/5

  1/469 [..............................] - ETA: 0s - loss: 0.0042 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 0.0035 - accuracy: 1.0000
  7/469 [..............................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9967
 10/469 [..............................] - ETA: 9s - loss: 0.0067 - accuracy: 0.9922
 13/469 [..............................] - ETA: 10s - loss: 0.0067 - accuracy: 0.9934
 16/469 [>.............................] - ETA: 10s - loss: 0.0065 - accuracy: 0.9932
 19/469 [>.............................] - ETA: 10s - loss: 0.0062 - accuracy: 0.9938
 22/469 [>.............................] - ETA: 10s - loss: 0.0062 - accuracy: 0.9940
 25/469 [>.............................] - ETA: 10s - loss: 0.0064 - accuracy: 0.9934
 28/469 [>.............................] - ETA: 10s - loss: 0.0061 - accuracy: 0.9941
 31/469 [>.............................] - ETA: 10s - loss: 0.0063 - accuracy: 0.9942
 34/469 [=>............................] - ETA: 10s - loss: 0.0062 - accuracy: 0.9943
 37/469 [=>............................] - ETA: 10s - loss: 0.0063 - accuracy: 0.9941
 40/469 [=>............................] - ETA: 10s - loss: 0.0062 - accuracy: 0.9941
 43/469 [=>............................] - ETA: 10s - loss: 0.0064 - accuracy: 0.9938
 46/469 [=>............................] - ETA: 9s - loss: 0.0062 - accuracy: 0.9941 
 49/469 [==>...........................] - ETA: 9s - loss: 0.0063 - accuracy: 0.9941
 52/469 [==>...........................] - ETA: 9s - loss: 0.0063 - accuracy: 0.9940
 55/469 [==>...........................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9939
 58/469 [==>...........................] - ETA: 9s - loss: 0.0067 - accuracy: 0.9937
 61/469 [==>...........................] - ETA: 9s - loss: 0.0067 - accuracy: 0.9935
 64/469 [===>..........................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9937
 67/469 [===>..........................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9934
 70/469 [===>..........................] - ETA: 9s - loss: 0.0068 - accuracy: 0.9931
 73/469 [===>..........................] - ETA: 9s - loss: 0.0068 - accuracy: 0.9932
 76/469 [===>..........................] - ETA: 9s - loss: 0.0067 - accuracy: 0.9932
 79/469 [====>.........................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9935
 82/469 [====>.........................] - ETA: 9s - loss: 0.0066 - accuracy: 0.9936
 85/469 [====>.........................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9937
 88/469 [====>.........................] - ETA: 9s - loss: 0.0065 - accuracy: 0.9937
 91/469 [====>.........................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9938
 94/469 [=====>........................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9938
 97/469 [=====>........................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9940
100/469 [=====>........................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9940
103/469 [=====>........................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9939
106/469 [=====>........................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9939
109/469 [=====>........................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9938
112/469 [======>.......................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9939
115/469 [======>.......................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9939
118/469 [======>.......................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9936
121/469 [======>.......................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9937
124/469 [======>.......................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9938
127/469 [=======>......................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9937
130/469 [=======>......................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9937
133/469 [=======>......................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9938
136/469 [=======>......................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9938
139/469 [=======>......................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9937
142/469 [========>.....................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9938
145/469 [========>.....................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9939
148/469 [========>.....................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9939
151/469 [========>.....................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9938
154/469 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9939
157/469 [=========>....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9939
160/469 [=========>....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9937
162/469 [=========>....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9938
165/469 [=========>....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9938
168/469 [=========>....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9938
171/469 [=========>....................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9938
174/469 [==========>...................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9938
177/469 [==========>...................] - ETA: 7s - loss: 0.0062 - accuracy: 0.9937
180/469 [==========>...................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9937
183/469 [==========>...................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9938
186/469 [==========>...................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9938
189/469 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9939
192/469 [===========>..................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9939
195/469 [===========>..................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9939
198/469 [===========>..................] - ETA: 6s - loss: 0.0062 - accuracy: 0.9939
201/469 [===========>..................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9940
204/469 [============>.................] - ETA: 6s - loss: 0.0061 - accuracy: 0.9940
207/469 [============>.................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9940
210/469 [============>.................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9941
213/469 [============>.................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9941
216/469 [============>.................] - ETA: 6s - loss: 0.0060 - accuracy: 0.9941
219/469 [=============>................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9941
222/469 [=============>................] - ETA: 5s - loss: 0.0060 - accuracy: 0.9941
225/469 [=============>................] - ETA: 5s - loss: 0.0060 - accuracy: 0.9940
228/469 [=============>................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9941
231/469 [=============>................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9941
234/469 [=============>................] - ETA: 5s - loss: 0.0059 - accuracy: 0.9942
237/469 [==============>...............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9941
240/469 [==============>...............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9940
243/469 [==============>...............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9940
246/469 [==============>...............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9941
249/469 [==============>...............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9941
252/469 [===============>..............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
255/469 [===============>..............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
258/469 [===============>..............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9941
261/469 [===============>..............] - ETA: 4s - loss: 0.0059 - accuracy: 0.9940
264/469 [===============>..............] - ETA: 4s - loss: 0.0060 - accuracy: 0.9939
267/469 [================>.............] - ETA: 4s - loss: 0.0060 - accuracy: 0.9939
270/469 [================>.............] - ETA: 4s - loss: 0.0060 - accuracy: 0.9939
273/469 [================>.............] - ETA: 4s - loss: 0.0060 - accuracy: 0.9938
276/469 [================>.............] - ETA: 4s - loss: 0.0060 - accuracy: 0.9938
279/469 [================>.............] - ETA: 4s - loss: 0.0060 - accuracy: 0.9939
282/469 [=================>............] - ETA: 4s - loss: 0.0060 - accuracy: 0.9939
285/469 [=================>............] - ETA: 4s - loss: 0.0060 - accuracy: 0.9938
288/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9937
291/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9937
294/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9937
297/469 [=================>............] - ETA: 4s - loss: 0.0061 - accuracy: 0.9937
300/469 [==================>...........] - ETA: 4s - loss: 0.0061 - accuracy: 0.9937
303/469 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9937
306/469 [==================>...........] - ETA: 3s - loss: 0.0061 - accuracy: 0.9936
309/469 [==================>...........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9936
312/469 [==================>...........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9936
315/469 [===================>..........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9936
318/469 [===================>..........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9935
321/469 [===================>..........] - ETA: 3s - loss: 0.0063 - accuracy: 0.9935
324/469 [===================>..........] - ETA: 3s - loss: 0.0063 - accuracy: 0.9935
327/469 [===================>..........] - ETA: 3s - loss: 0.0063 - accuracy: 0.9935
330/469 [====================>.........] - ETA: 3s - loss: 0.0063 - accuracy: 0.9935
333/469 [====================>.........] - ETA: 3s - loss: 0.0063 - accuracy: 0.9936
336/469 [====================>.........] - ETA: 3s - loss: 0.0062 - accuracy: 0.9936
339/469 [====================>.........] - ETA: 3s - loss: 0.0063 - accuracy: 0.9935
342/469 [====================>.........] - ETA: 3s - loss: 0.0063 - accuracy: 0.9936
345/469 [=====================>........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9936
348/469 [=====================>........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9936
351/469 [=====================>........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9936
354/469 [=====================>........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9936
357/469 [=====================>........] - ETA: 2s - loss: 0.0062 - accuracy: 0.9936
360/469 [======================>.......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
363/469 [======================>.......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
366/469 [======================>.......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
369/469 [======================>.......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
371/469 [======================>.......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
374/469 [======================>.......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
377/469 [=======================>......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
380/469 [=======================>......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
383/469 [=======================>......] - ETA: 2s - loss: 0.0062 - accuracy: 0.9935
386/469 [=======================>......] - ETA: 1s - loss: 0.0062 - accuracy: 0.9935
389/469 [=======================>......] - ETA: 1s - loss: 0.0062 - accuracy: 0.9936
392/469 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
395/469 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
398/469 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9937
401/469 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9937
404/469 [========================>.....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
407/469 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
410/469 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
413/469 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
416/469 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
419/469 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
422/469 [=========================>....] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
425/469 [==========================>...] - ETA: 1s - loss: 0.0061 - accuracy: 0.9936
428/469 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9936
431/469 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9936
434/469 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9936
437/469 [==========================>...] - ETA: 0s - loss: 0.0061 - accuracy: 0.9936
440/469 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9936
443/469 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9936
446/469 [===========================>..] - ETA: 0s - loss: 0.0060 - accuracy: 0.9936
449/469 [===========================>..] - ETA: 0s - loss: 0.0061 - accuracy: 0.9936
452/469 [===========================>..] - ETA: 0s - loss: 0.0060 - accuracy: 0.9936
455/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9937
458/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9936
461/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9936
464/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9936
467/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9937
469/469 [==============================] - 12s 25ms/step - loss: 0.0060 - accuracy: 0.9937 - val_loss: 0.0075 - val_accuracy: 0.9917 - lr: 6.3096e-04
Epoch 3/5

  1/469 [..............................] - ETA: 0s - loss: 9.3197e-04 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 0.0023 - accuracy: 0.9980    
  7/469 [..............................] - ETA: 9s - loss: 0.0025 - accuracy: 0.9967
 10/469 [..............................] - ETA: 9s - loss: 0.0024 - accuracy: 0.9977
 13/469 [..............................] - ETA: 10s - loss: 0.0022 - accuracy: 0.9982
 16/469 [>.............................] - ETA: 10s - loss: 0.0024 - accuracy: 0.9976
 19/469 [>.............................] - ETA: 10s - loss: 0.0022 - accuracy: 0.9979
 22/469 [>.............................] - ETA: 10s - loss: 0.0023 - accuracy: 0.9979
 25/469 [>.............................] - ETA: 10s - loss: 0.0022 - accuracy: 0.9978
 28/469 [>.............................] - ETA: 10s - loss: 0.0022 - accuracy: 0.9980
 31/469 [>.............................] - ETA: 10s - loss: 0.0021 - accuracy: 0.9982
 34/469 [=>............................] - ETA: 10s - loss: 0.0021 - accuracy: 0.9982
 37/469 [=>............................] - ETA: 10s - loss: 0.0024 - accuracy: 0.9975
 40/469 [=>............................] - ETA: 10s - loss: 0.0024 - accuracy: 0.9977
 43/469 [=>............................] - ETA: 10s - loss: 0.0023 - accuracy: 0.9978
 46/469 [=>............................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9978 
 49/469 [==>...........................] - ETA: 9s - loss: 0.0022 - accuracy: 0.9978
 52/469 [==>...........................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9977
 55/469 [==>...........................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9979
 58/469 [==>...........................] - ETA: 9s - loss: 0.0022 - accuracy: 0.9980
 61/469 [==>...........................] - ETA: 9s - loss: 0.0022 - accuracy: 0.9980
 64/469 [===>..........................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9978
 67/469 [===>..........................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9978
 70/469 [===>..........................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9979
 73/469 [===>..........................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9980
 76/469 [===>..........................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9979
 79/469 [====>.........................] - ETA: 9s - loss: 0.0023 - accuracy: 0.9980
 82/469 [====>.........................] - ETA: 9s - loss: 0.0022 - accuracy: 0.9981
 85/469 [====>.........................] - ETA: 9s - loss: 0.0024 - accuracy: 0.9980
 88/469 [====>.........................] - ETA: 9s - loss: 0.0025 - accuracy: 0.9980
 90/469 [====>.........................] - ETA: 9s - loss: 0.0025 - accuracy: 0.9979
 93/469 [====>.........................] - ETA: 8s - loss: 0.0025 - accuracy: 0.9979
 96/469 [=====>........................] - ETA: 8s - loss: 0.0026 - accuracy: 0.9977
 99/469 [=====>........................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9976
102/469 [=====>........................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9976
105/469 [=====>........................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9977
108/469 [=====>........................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9977
111/469 [======>.......................] - ETA: 8s - loss: 0.0026 - accuracy: 0.9977
114/469 [======>.......................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9977
117/469 [======>.......................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9977
120/469 [======>.......................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9976
123/469 [======>.......................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9976
126/469 [=======>......................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9976
129/469 [=======>......................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9977
132/469 [=======>......................] - ETA: 8s - loss: 0.0027 - accuracy: 0.9978
135/469 [=======>......................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9977
138/469 [=======>......................] - ETA: 7s - loss: 0.0027 - accuracy: 0.9977
141/469 [========>.....................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9978
144/469 [========>.....................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9978
147/469 [========>.....................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9978
150/469 [========>.....................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9978
153/469 [========>.....................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9978
156/469 [========>.....................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9978
159/469 [=========>....................] - ETA: 7s - loss: 0.0025 - accuracy: 0.9979
162/469 [=========>....................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9978
165/469 [=========>....................] - ETA: 7s - loss: 0.0025 - accuracy: 0.9978
168/469 [=========>....................] - ETA: 7s - loss: 0.0025 - accuracy: 0.9978
171/469 [=========>....................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9977
174/469 [==========>...................] - ETA: 7s - loss: 0.0026 - accuracy: 0.9977
177/469 [==========>...................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9977
180/469 [==========>...................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9977
183/469 [==========>...................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9977
186/469 [==========>...................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9978
189/469 [===========>..................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9978
192/469 [===========>..................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9978
195/469 [===========>..................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9978
198/469 [===========>..................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9978
201/469 [===========>..................] - ETA: 6s - loss: 0.0025 - accuracy: 0.9979
204/469 [============>.................] - ETA: 6s - loss: 0.0025 - accuracy: 0.9979
207/469 [============>.................] - ETA: 6s - loss: 0.0025 - accuracy: 0.9979
210/469 [============>.................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9978
213/469 [============>.................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9978
216/469 [============>.................] - ETA: 6s - loss: 0.0026 - accuracy: 0.9978
219/469 [=============>................] - ETA: 5s - loss: 0.0025 - accuracy: 0.9978
222/469 [=============>................] - ETA: 5s - loss: 0.0025 - accuracy: 0.9978
225/469 [=============>................] - ETA: 5s - loss: 0.0025 - accuracy: 0.9978
228/469 [=============>................] - ETA: 5s - loss: 0.0026 - accuracy: 0.9978
231/469 [=============>................] - ETA: 5s - loss: 0.0026 - accuracy: 0.9978
234/469 [=============>................] - ETA: 5s - loss: 0.0026 - accuracy: 0.9978
237/469 [==============>...............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9977
240/469 [==============>...............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9977
243/469 [==============>...............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9977
246/469 [==============>...............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9977
249/469 [==============>...............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9976
252/469 [===============>..............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9976
255/469 [===============>..............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9976
258/469 [===============>..............] - ETA: 5s - loss: 0.0026 - accuracy: 0.9976
261/469 [===============>..............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9976
264/469 [===============>..............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9976
267/469 [================>.............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9977
270/469 [================>.............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9976
273/469 [================>.............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9977
276/469 [================>.............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9977
279/469 [================>.............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9976
282/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9976
285/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9976
288/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9977
291/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9977
294/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9977
297/469 [=================>............] - ETA: 4s - loss: 0.0026 - accuracy: 0.9976
300/469 [==================>...........] - ETA: 4s - loss: 0.0026 - accuracy: 0.9976
303/469 [==================>...........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
306/469 [==================>...........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
309/469 [==================>...........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
312/469 [==================>...........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
315/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9977
318/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
321/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
324/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9977
327/469 [===================>..........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9977
330/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9977
333/469 [====================>.........] - ETA: 3s - loss: 0.0025 - accuracy: 0.9977
336/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
339/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
342/469 [====================>.........] - ETA: 3s - loss: 0.0026 - accuracy: 0.9976
345/469 [=====================>........] - ETA: 2s - loss: 0.0026 - accuracy: 0.9976
348/469 [=====================>........] - ETA: 2s - loss: 0.0026 - accuracy: 0.9976
351/469 [=====================>........] - ETA: 2s - loss: 0.0027 - accuracy: 0.9976
354/469 [=====================>........] - ETA: 2s - loss: 0.0027 - accuracy: 0.9976
357/469 [=====================>........] - ETA: 2s - loss: 0.0026 - accuracy: 0.9976
360/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9976
363/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9976
366/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9976
369/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9976
372/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9977
375/469 [======================>.......] - ETA: 2s - loss: 0.0026 - accuracy: 0.9977
378/469 [=======================>......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9976
381/469 [=======================>......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9976
384/469 [=======================>......] - ETA: 2s - loss: 0.0027 - accuracy: 0.9976
387/469 [=======================>......] - ETA: 1s - loss: 0.0027 - accuracy: 0.9976
390/469 [=======================>......] - ETA: 1s - loss: 0.0027 - accuracy: 0.9976
393/469 [========================>.....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9976
396/469 [========================>.....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9975
399/469 [========================>.....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9976
402/469 [========================>.....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9976
405/469 [========================>.....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9975
408/469 [=========================>....] - ETA: 1s - loss: 0.0026 - accuracy: 0.9975
411/469 [=========================>....] - ETA: 1s - loss: 0.0026 - accuracy: 0.9975
414/469 [=========================>....] - ETA: 1s - loss: 0.0026 - accuracy: 0.9975
417/469 [=========================>....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9975
420/469 [=========================>....] - ETA: 1s - loss: 0.0027 - accuracy: 0.9975
423/469 [==========================>...] - ETA: 1s - loss: 0.0026 - accuracy: 0.9975
426/469 [==========================>...] - ETA: 1s - loss: 0.0026 - accuracy: 0.9975
429/469 [==========================>...] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
432/469 [==========================>...] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
435/469 [==========================>...] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
438/469 [===========================>..] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
441/469 [===========================>..] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
444/469 [===========================>..] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
447/469 [===========================>..] - ETA: 0s - loss: 0.0027 - accuracy: 0.9975
450/469 [===========================>..] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
453/469 [===========================>..] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
456/469 [============================>.] - ETA: 0s - loss: 0.0026 - accuracy: 0.9975
459/469 [============================>.] - ETA: 0s - loss: 0.0027 - accuracy: 0.9975
462/469 [============================>.] - ETA: 0s - loss: 0.0027 - accuracy: 0.9974
465/469 [============================>.] - ETA: 0s - loss: 0.0027 - accuracy: 0.9975
468/469 [============================>.] - ETA: 0s - loss: 0.0027 - accuracy: 0.9974
469/469 [==============================] - 12s 25ms/step - loss: 0.0027 - accuracy: 0.9974 - val_loss: 0.0063 - val_accuracy: 0.9925 - lr: 3.9811e-04
Epoch 4/5

  1/469 [..............................] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 0.0011 - accuracy: 1.0000
  7/469 [..............................] - ETA: 9s - loss: 0.0018 - accuracy: 0.9989
 10/469 [..............................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9984
 13/469 [..............................] - ETA: 10s - loss: 0.0017 - accuracy: 0.9982
 16/469 [>.............................] - ETA: 10s - loss: 0.0016 - accuracy: 0.9985
 18/469 [>.............................] - ETA: 10s - loss: 0.0017 - accuracy: 0.9983
 21/469 [>.............................] - ETA: 10s - loss: 0.0016 - accuracy: 0.9985
 24/469 [>.............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9987
 27/469 [>.............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9988
 30/469 [>.............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9987
 33/469 [=>............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9988
 36/469 [=>............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9989
 39/469 [=>............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9990
 42/469 [=>............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9991
 45/469 [=>............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9991
 48/469 [==>...........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9990 
 51/469 [==>...........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9991
 54/469 [==>...........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9990
 57/469 [==>...........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 60/469 [==>...........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9990
 63/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 66/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9988
 69/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 72/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 75/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9990
 78/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 81/469 [====>.........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 84/469 [====>.........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9990
 87/469 [====>.........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 90/469 [====>.........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 93/469 [====>.........................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9989
 96/469 [=====>........................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9989
 99/469 [=====>........................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9989
102/469 [=====>........................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9989
105/469 [=====>........................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9990
108/469 [=====>........................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9990
111/469 [======>.......................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9990
114/469 [======>.......................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9990
117/469 [======>.......................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9990
120/469 [======>.......................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9990
123/469 [======>.......................] - ETA: 8s - loss: 0.0013 - accuracy: 0.9990
126/469 [=======>......................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9989
129/469 [=======>......................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9990
132/469 [=======>......................] - ETA: 8s - loss: 0.0014 - accuracy: 0.9990
135/469 [=======>......................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9990
138/469 [=======>......................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9990
141/469 [========>.....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9991
144/469 [========>.....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9991
147/469 [========>.....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9991
150/469 [========>.....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9991
153/469 [========>.....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9991
156/469 [========>.....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9991
159/469 [=========>....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9992
162/469 [=========>....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9992
165/469 [=========>....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9992
168/469 [=========>....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9992
171/469 [=========>....................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9992
174/469 [==========>...................] - ETA: 7s - loss: 0.0013 - accuracy: 0.9992
177/469 [==========>...................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9992
180/469 [==========>...................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9992
183/469 [==========>...................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
186/469 [==========>...................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9992
189/469 [===========>..................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9992
192/469 [===========>..................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
195/469 [===========>..................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
198/469 [===========>..................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
201/469 [===========>..................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
204/469 [============>.................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
207/469 [============>.................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
210/469 [============>.................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
213/469 [============>.................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
216/469 [============>.................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9991
219/469 [=============>................] - ETA: 5s - loss: 0.0012 - accuracy: 0.9991
222/469 [=============>................] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
225/469 [=============>................] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
228/469 [=============>................] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
231/469 [=============>................] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
234/469 [=============>................] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
237/469 [==============>...............] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
240/469 [==============>...............] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
243/469 [==============>...............] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
246/469 [==============>...............] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
249/469 [==============>...............] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
252/469 [===============>..............] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
255/469 [===============>..............] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
258/469 [===============>..............] - ETA: 5s - loss: 0.0012 - accuracy: 0.9992
261/469 [===============>..............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
264/469 [===============>..............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
267/469 [================>.............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
270/469 [================>.............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
273/469 [================>.............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
276/469 [================>.............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
279/469 [================>.............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
282/469 [=================>............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
285/469 [=================>............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
288/469 [=================>............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
291/469 [=================>............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
294/469 [=================>............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
297/469 [=================>............] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
300/469 [==================>...........] - ETA: 4s - loss: 0.0012 - accuracy: 0.9992
303/469 [==================>...........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
306/469 [==================>...........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
309/469 [==================>...........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
312/469 [==================>...........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
315/469 [===================>..........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
318/469 [===================>..........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
321/469 [===================>..........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
324/469 [===================>..........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
327/469 [===================>..........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
330/469 [====================>.........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
333/469 [====================>.........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
336/469 [====================>.........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
339/469 [====================>.........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
342/469 [====================>.........] - ETA: 3s - loss: 0.0012 - accuracy: 0.9992
345/469 [=====================>........] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
348/469 [=====================>........] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
351/469 [=====================>........] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
354/469 [=====================>........] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
357/469 [=====================>........] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
360/469 [======================>.......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
363/469 [======================>.......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
366/469 [======================>.......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
369/469 [======================>.......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
372/469 [======================>.......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
375/469 [======================>.......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
378/469 [=======================>......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9993
381/469 [=======================>......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9993
384/469 [=======================>......] - ETA: 2s - loss: 0.0012 - accuracy: 0.9992
387/469 [=======================>......] - ETA: 1s - loss: 0.0012 - accuracy: 0.9993
390/469 [=======================>......] - ETA: 1s - loss: 0.0012 - accuracy: 0.9993
393/469 [========================>.....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9993
396/469 [========================>.....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9993
399/469 [========================>.....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9993
402/469 [========================>.....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9993
405/469 [========================>.....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9992
408/469 [=========================>....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9993
411/469 [=========================>....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9993
414/469 [=========================>....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9992
417/469 [=========================>....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9992
420/469 [=========================>....] - ETA: 1s - loss: 0.0012 - accuracy: 0.9992
423/469 [==========================>...] - ETA: 1s - loss: 0.0012 - accuracy: 0.9992
426/469 [==========================>...] - ETA: 1s - loss: 0.0012 - accuracy: 0.9992
429/469 [==========================>...] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
432/469 [==========================>...] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
434/469 [==========================>...] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
437/469 [==========================>...] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
440/469 [===========================>..] - ETA: 0s - loss: 0.0012 - accuracy: 0.9993
443/469 [===========================>..] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
446/469 [===========================>..] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
448/469 [===========================>..] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
451/469 [===========================>..] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
454/469 [============================>.] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
457/469 [============================>.] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
460/469 [============================>.] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
463/469 [============================>.] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
466/469 [============================>.] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
469/469 [==============================] - ETA: 0s - loss: 0.0012 - accuracy: 0.9992
469/469 [==============================] - 12s 25ms/step - loss: 0.0012 - accuracy: 0.9992 - val_loss: 0.0055 - val_accuracy: 0.9933 - lr: 2.5119e-04
Epoch 5/5

  1/469 [..............................] - ETA: 0s - loss: 8.5665e-04 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 5.8853e-04 - accuracy: 1.0000
  7/469 [..............................] - ETA: 9s - loss: 5.0218e-04 - accuracy: 1.0000
 10/469 [..............................] - ETA: 9s - loss: 5.2756e-04 - accuracy: 1.0000
 13/469 [..............................] - ETA: 10s - loss: 5.7809e-04 - accuracy: 1.0000
 16/469 [>.............................] - ETA: 10s - loss: 5.9889e-04 - accuracy: 1.0000
 19/469 [>.............................] - ETA: 10s - loss: 7.1809e-04 - accuracy: 0.9996
 22/469 [>.............................] - ETA: 10s - loss: 7.3261e-04 - accuracy: 0.9996
 25/469 [>.............................] - ETA: 10s - loss: 7.3864e-04 - accuracy: 0.9994
 28/469 [>.............................] - ETA: 10s - loss: 7.1678e-04 - accuracy: 0.9994
 31/469 [>.............................] - ETA: 10s - loss: 6.9659e-04 - accuracy: 0.9995
 34/469 [=>............................] - ETA: 10s - loss: 7.0908e-04 - accuracy: 0.9995
 37/469 [=>............................] - ETA: 10s - loss: 7.0724e-04 - accuracy: 0.9996
 40/469 [=>............................] - ETA: 10s - loss: 7.2123e-04 - accuracy: 0.9996
 43/469 [=>............................] - ETA: 10s - loss: 6.8638e-04 - accuracy: 0.9996
 46/469 [=>............................] - ETA: 9s - loss: 6.8977e-04 - accuracy: 0.9997 
 49/469 [==>...........................] - ETA: 9s - loss: 6.7019e-04 - accuracy: 0.9997
 52/469 [==>...........................] - ETA: 9s - loss: 6.6063e-04 - accuracy: 0.9997
 55/469 [==>...........................] - ETA: 9s - loss: 6.6024e-04 - accuracy: 0.9997
 58/469 [==>...........................] - ETA: 9s - loss: 6.6414e-04 - accuracy: 0.9997
 61/469 [==>...........................] - ETA: 9s - loss: 6.4772e-04 - accuracy: 0.9997
 64/469 [===>..........................] - ETA: 9s - loss: 6.7277e-04 - accuracy: 0.9998
 67/469 [===>..........................] - ETA: 9s - loss: 6.6187e-04 - accuracy: 0.9998
 70/469 [===>..........................] - ETA: 9s - loss: 6.4115e-04 - accuracy: 0.9998
 73/469 [===>..........................] - ETA: 9s - loss: 6.3994e-04 - accuracy: 0.9998
 76/469 [===>..........................] - ETA: 9s - loss: 6.2980e-04 - accuracy: 0.9998
 78/469 [===>..........................] - ETA: 9s - loss: 6.2793e-04 - accuracy: 0.9998
 81/469 [====>.........................] - ETA: 9s - loss: 6.2472e-04 - accuracy: 0.9998
 84/469 [====>.........................] - ETA: 9s - loss: 6.4643e-04 - accuracy: 0.9998
 87/469 [====>.........................] - ETA: 9s - loss: 6.4102e-04 - accuracy: 0.9998
 90/469 [====>.........................] - ETA: 9s - loss: 6.3338e-04 - accuracy: 0.9998
 93/469 [====>.........................] - ETA: 8s - loss: 6.2001e-04 - accuracy: 0.9998
 96/469 [=====>........................] - ETA: 8s - loss: 6.1775e-04 - accuracy: 0.9998
 99/469 [=====>........................] - ETA: 8s - loss: 6.2887e-04 - accuracy: 0.9998
102/469 [=====>........................] - ETA: 8s - loss: 6.3246e-04 - accuracy: 0.9998
105/469 [=====>........................] - ETA: 8s - loss: 6.4432e-04 - accuracy: 0.9999
108/469 [=====>........................] - ETA: 8s - loss: 6.5759e-04 - accuracy: 0.9999
111/469 [======>.......................] - ETA: 8s - loss: 6.6741e-04 - accuracy: 0.9998
114/469 [======>.......................] - ETA: 8s - loss: 6.5363e-04 - accuracy: 0.9998
117/469 [======>.......................] - ETA: 8s - loss: 6.8952e-04 - accuracy: 0.9997
120/469 [======>.......................] - ETA: 8s - loss: 6.8814e-04 - accuracy: 0.9997
123/469 [======>.......................] - ETA: 8s - loss: 6.7721e-04 - accuracy: 0.9997
126/469 [=======>......................] - ETA: 8s - loss: 6.6847e-04 - accuracy: 0.9998
129/469 [=======>......................] - ETA: 8s - loss: 6.6198e-04 - accuracy: 0.9998
132/469 [=======>......................] - ETA: 8s - loss: 6.7393e-04 - accuracy: 0.9998
135/469 [=======>......................] - ETA: 7s - loss: 6.7623e-04 - accuracy: 0.9998
138/469 [=======>......................] - ETA: 7s - loss: 6.6784e-04 - accuracy: 0.9998
141/469 [========>.....................] - ETA: 7s - loss: 6.6170e-04 - accuracy: 0.9998
144/469 [========>.....................] - ETA: 7s - loss: 6.5082e-04 - accuracy: 0.9998
147/469 [========>.....................] - ETA: 7s - loss: 6.4500e-04 - accuracy: 0.9998
150/469 [========>.....................] - ETA: 7s - loss: 6.6429e-04 - accuracy: 0.9997
152/469 [========>.....................] - ETA: 7s - loss: 6.6248e-04 - accuracy: 0.9997
155/469 [========>.....................] - ETA: 7s - loss: 6.7025e-04 - accuracy: 0.9997
158/469 [=========>....................] - ETA: 7s - loss: 6.6520e-04 - accuracy: 0.9998
161/469 [=========>....................] - ETA: 7s - loss: 6.5964e-04 - accuracy: 0.9998
164/469 [=========>....................] - ETA: 7s - loss: 6.5903e-04 - accuracy: 0.9998
167/469 [=========>....................] - ETA: 7s - loss: 6.5335e-04 - accuracy: 0.9998
170/469 [=========>....................] - ETA: 7s - loss: 6.5198e-04 - accuracy: 0.9998
173/469 [==========>...................] - ETA: 7s - loss: 6.4961e-04 - accuracy: 0.9998
176/469 [==========>...................] - ETA: 7s - loss: 6.4890e-04 - accuracy: 0.9998
179/469 [==========>...................] - ETA: 6s - loss: 6.4479e-04 - accuracy: 0.9998
182/469 [==========>...................] - ETA: 6s - loss: 6.3752e-04 - accuracy: 0.9998
185/469 [==========>...................] - ETA: 6s - loss: 6.4537e-04 - accuracy: 0.9997
188/469 [===========>..................] - ETA: 6s - loss: 6.4473e-04 - accuracy: 0.9998
191/469 [===========>..................] - ETA: 6s - loss: 6.3845e-04 - accuracy: 0.9998
194/469 [===========>..................] - ETA: 6s - loss: 6.5003e-04 - accuracy: 0.9997
197/469 [===========>..................] - ETA: 6s - loss: 6.5322e-04 - accuracy: 0.9997
200/469 [===========>..................] - ETA: 6s - loss: 6.5934e-04 - accuracy: 0.9997
203/469 [===========>..................] - ETA: 6s - loss: 6.5539e-04 - accuracy: 0.9997
206/469 [============>.................] - ETA: 6s - loss: 6.5083e-04 - accuracy: 0.9997
209/469 [============>.................] - ETA: 6s - loss: 6.4670e-04 - accuracy: 0.9997
212/469 [============>.................] - ETA: 6s - loss: 6.4293e-04 - accuracy: 0.9997
215/469 [============>.................] - ETA: 6s - loss: 6.3986e-04 - accuracy: 0.9997
218/469 [============>.................] - ETA: 6s - loss: 6.3744e-04 - accuracy: 0.9997
221/469 [=============>................] - ETA: 5s - loss: 6.3266e-04 - accuracy: 0.9998
224/469 [=============>................] - ETA: 5s - loss: 6.3179e-04 - accuracy: 0.9998
227/469 [=============>................] - ETA: 5s - loss: 6.3251e-04 - accuracy: 0.9998
230/469 [=============>................] - ETA: 5s - loss: 6.3085e-04 - accuracy: 0.9998
233/469 [=============>................] - ETA: 5s - loss: 6.3208e-04 - accuracy: 0.9998
236/469 [==============>...............] - ETA: 5s - loss: 6.3085e-04 - accuracy: 0.9998
239/469 [==============>...............] - ETA: 5s - loss: 6.2807e-04 - accuracy: 0.9998
242/469 [==============>...............] - ETA: 5s - loss: 6.2752e-04 - accuracy: 0.9998
245/469 [==============>...............] - ETA: 5s - loss: 6.2563e-04 - accuracy: 0.9998
248/469 [==============>...............] - ETA: 5s - loss: 6.2411e-04 - accuracy: 0.9998
251/469 [===============>..............] - ETA: 5s - loss: 6.2942e-04 - accuracy: 0.9998
254/469 [===============>..............] - ETA: 5s - loss: 6.2509e-04 - accuracy: 0.9998
257/469 [===============>..............] - ETA: 5s - loss: 6.2958e-04 - accuracy: 0.9998
260/469 [===============>..............] - ETA: 5s - loss: 6.2696e-04 - accuracy: 0.9998
263/469 [===============>..............] - ETA: 4s - loss: 6.2136e-04 - accuracy: 0.9998
266/469 [================>.............] - ETA: 4s - loss: 6.1716e-04 - accuracy: 0.9998
269/469 [================>.............] - ETA: 4s - loss: 6.2339e-04 - accuracy: 0.9998
272/469 [================>.............] - ETA: 4s - loss: 6.4178e-04 - accuracy: 0.9998
275/469 [================>.............] - ETA: 4s - loss: 6.4503e-04 - accuracy: 0.9998
278/469 [================>.............] - ETA: 4s - loss: 6.4005e-04 - accuracy: 0.9998
281/469 [================>.............] - ETA: 4s - loss: 6.3472e-04 - accuracy: 0.9998
284/469 [=================>............] - ETA: 4s - loss: 6.3119e-04 - accuracy: 0.9998
287/469 [=================>............] - ETA: 4s - loss: 6.2821e-04 - accuracy: 0.9998
290/469 [=================>............] - ETA: 4s - loss: 6.2380e-04 - accuracy: 0.9998
293/469 [=================>............] - ETA: 4s - loss: 6.2172e-04 - accuracy: 0.9998
296/469 [=================>............] - ETA: 4s - loss: 6.2375e-04 - accuracy: 0.9998
299/469 [==================>...........] - ETA: 4s - loss: 6.1933e-04 - accuracy: 0.9998
302/469 [==================>...........] - ETA: 4s - loss: 6.1695e-04 - accuracy: 0.9998
305/469 [==================>...........] - ETA: 3s - loss: 6.1573e-04 - accuracy: 0.9998
308/469 [==================>...........] - ETA: 3s - loss: 6.1529e-04 - accuracy: 0.9998
311/469 [==================>...........] - ETA: 3s - loss: 6.1801e-04 - accuracy: 0.9998
314/469 [===================>..........] - ETA: 3s - loss: 6.1707e-04 - accuracy: 0.9998
317/469 [===================>..........] - ETA: 3s - loss: 6.2890e-04 - accuracy: 0.9998
320/469 [===================>..........] - ETA: 3s - loss: 6.3409e-04 - accuracy: 0.9998
323/469 [===================>..........] - ETA: 3s - loss: 6.3364e-04 - accuracy: 0.9998
326/469 [===================>..........] - ETA: 3s - loss: 6.3179e-04 - accuracy: 0.9998
329/469 [====================>.........] - ETA: 3s - loss: 6.2802e-04 - accuracy: 0.9998
332/469 [====================>.........] - ETA: 3s - loss: 6.3496e-04 - accuracy: 0.9998
335/469 [====================>.........] - ETA: 3s - loss: 6.3858e-04 - accuracy: 0.9998
338/469 [====================>.........] - ETA: 3s - loss: 6.3991e-04 - accuracy: 0.9998
341/469 [====================>.........] - ETA: 3s - loss: 6.3920e-04 - accuracy: 0.9998
344/469 [=====================>........] - ETA: 3s - loss: 6.3895e-04 - accuracy: 0.9998
347/469 [=====================>........] - ETA: 2s - loss: 6.3901e-04 - accuracy: 0.9998
350/469 [=====================>........] - ETA: 2s - loss: 6.4901e-04 - accuracy: 0.9998
353/469 [=====================>........] - ETA: 2s - loss: 6.4923e-04 - accuracy: 0.9998
356/469 [=====================>........] - ETA: 2s - loss: 6.4699e-04 - accuracy: 0.9998
359/469 [=====================>........] - ETA: 2s - loss: 6.4290e-04 - accuracy: 0.9998
361/469 [======================>.......] - ETA: 2s - loss: 6.4513e-04 - accuracy: 0.9998
364/469 [======================>.......] - ETA: 2s - loss: 6.4028e-04 - accuracy: 0.9998
367/469 [======================>.......] - ETA: 2s - loss: 6.3929e-04 - accuracy: 0.9998
370/469 [======================>.......] - ETA: 2s - loss: 6.3860e-04 - accuracy: 0.9998
373/469 [======================>.......] - ETA: 2s - loss: 6.3617e-04 - accuracy: 0.9998
376/469 [=======================>......] - ETA: 2s - loss: 6.3584e-04 - accuracy: 0.9998
379/469 [=======================>......] - ETA: 2s - loss: 6.3268e-04 - accuracy: 0.9998
382/469 [=======================>......] - ETA: 2s - loss: 6.3009e-04 - accuracy: 0.9998
385/469 [=======================>......] - ETA: 2s - loss: 6.2842e-04 - accuracy: 0.9998
388/469 [=======================>......] - ETA: 1s - loss: 6.3071e-04 - accuracy: 0.9998
391/469 [========================>.....] - ETA: 1s - loss: 6.2794e-04 - accuracy: 0.9998
394/469 [========================>.....] - ETA: 1s - loss: 6.3000e-04 - accuracy: 0.9998
397/469 [========================>.....] - ETA: 1s - loss: 6.2966e-04 - accuracy: 0.9998
400/469 [========================>.....] - ETA: 1s - loss: 6.3367e-04 - accuracy: 0.9998
403/469 [========================>.....] - ETA: 1s - loss: 6.3358e-04 - accuracy: 0.9998
406/469 [========================>.....] - ETA: 1s - loss: 6.3039e-04 - accuracy: 0.9998
409/469 [=========================>....] - ETA: 1s - loss: 6.2941e-04 - accuracy: 0.9998
412/469 [=========================>....] - ETA: 1s - loss: 6.2924e-04 - accuracy: 0.9998
415/469 [=========================>....] - ETA: 1s - loss: 6.3293e-04 - accuracy: 0.9998
418/469 [=========================>....] - ETA: 1s - loss: 6.3334e-04 - accuracy: 0.9998
421/469 [=========================>....] - ETA: 1s - loss: 6.3048e-04 - accuracy: 0.9998
424/469 [==========================>...] - ETA: 1s - loss: 6.3255e-04 - accuracy: 0.9998
427/469 [==========================>...] - ETA: 1s - loss: 6.4242e-04 - accuracy: 0.9998
430/469 [==========================>...] - ETA: 0s - loss: 6.4520e-04 - accuracy: 0.9998
433/469 [==========================>...] - ETA: 0s - loss: 6.4227e-04 - accuracy: 0.9998
436/469 [==========================>...] - ETA: 0s - loss: 6.3976e-04 - accuracy: 0.9998
439/469 [===========================>..] - ETA: 0s - loss: 6.3989e-04 - accuracy: 0.9998
442/469 [===========================>..] - ETA: 0s - loss: 6.4550e-04 - accuracy: 0.9998
445/469 [===========================>..] - ETA: 0s - loss: 6.4469e-04 - accuracy: 0.9998
448/469 [===========================>..] - ETA: 0s - loss: 6.4321e-04 - accuracy: 0.9998
451/469 [===========================>..] - ETA: 0s - loss: 6.4406e-04 - accuracy: 0.9998
454/469 [============================>.] - ETA: 0s - loss: 6.4872e-04 - accuracy: 0.9998
457/469 [============================>.] - ETA: 0s - loss: 6.4983e-04 - accuracy: 0.9998
460/469 [============================>.] - ETA: 0s - loss: 6.5971e-04 - accuracy: 0.9997
463/469 [============================>.] - ETA: 0s - loss: 6.5930e-04 - accuracy: 0.9997
466/469 [============================>.] - ETA: 0s - loss: 6.5539e-04 - accuracy: 0.9997
469/469 [==============================] - ETA: 0s - loss: 6.5384e-04 - accuracy: 0.9998
469/469 [==============================] - 12s 25ms/step - loss: 6.5384e-04 - accuracy: 0.9998 - val_loss: 0.0054 - val_accuracy: 0.9934 - lr: 1.5849e-04
Test score: 0.005374752916395664
Test accuracy: 0.993399977684021

3. Model Akida-compatibility check and changes

3.1 Compatibility check

The first step is to ensure that the model as defined doesn’t include any layers or operations that aren’t Akida-compatible (please refer to the CNN2SNN toolkit documentation for full details):

  • Standard Conv2D and Dense layers are supported (note that there is currently no support for skip, recursive and parallel layers).

  • Each of these trainable core layers except for the last one must be followed by an Activation layer.

  • All blocks can optionally include a BatchNormalization layer.

  • Convolutional blocks can optionally include a MaxPooling type layer.

Note

This configuration of layers (Conv/Dense + BatchNormalization + Activation) constitutes the basic building block of Akida-compatible models and is widely used in deep learning.

If the model defined is not fully compatible with the Akida NSoC, substitutes will be needed for the relevant layers/operations (guidelines included in the documentation).

3.2 Model adaptation

As noted above, the basic building blocks of Akida compatible models actually comprise a trio of layers: Conv/Dense + BatchNormalization + Activation (with, optionally, pooling). The CNN2SNN tool provides a set of functions that simplify using these building blocks, and subsequently enable easy application of Brainchip’s custom quantization functions.

from akida_models.quantization_blocks import conv_block, dense_block

The following code illustrates how to express the MNIST model defined above using the functions provided by Brainchip. A couple of points to avoid confusion when you look through it:

  • The weight_quantization in each block isn’t used here, but will be used later to apply a quantization method to the model weights.

  • The block_id is just used for naming the layers, and will be good practice in enabling reloading of partially trained models in more advanced training cases.

  • Note that in the final block, we set the nonlinearity activ_quantization to None. In that case, the block has no Activation layer, and the output is simply the output from the BatchNormalization layer.

# Removes all the nodes left over from the previous model and free memory
K.clear_session()

# Define the model.
# The commented code shows the sets of layers in the original definition
# that are being replaced by the provided conv_block and dense_blocks here

img_input = Input(shape=(28, 28, 1))

# x = Conv2D(filters=32,
#            kernel_size=(5, 5),
#            padding='same',
#            use_bias=False,
#            data_format='channels_last')(img_input)
# x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(x)
# x = BatchNormalization()(x)
# x = ReLU(6.)(x)
x = conv_block(img_input,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_0',
               pooling='max',
               add_batchnorm=True)

# x = Conv2D(filters=32,
#            kernel_size=(5, 5),
#            padding='same',
#            use_bias=False)(x)
# x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(x)
# x = BatchNormalization()(x)
# x = ReLU(6.)(x)
x = conv_block(x,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_1',
               pooling='max',
               add_batchnorm=True)

# x = Conv2D(filters=32,
#            kernel_size=(5, 5),
#            padding='same',
#            use_bias=False)(x)
# x = BatchNormalization()(x)
# x = ReLU(6.)(x)
x = conv_block(x,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_2',
               add_batchnorm=True)

x = Flatten()(x)

# x = Dense(512,
#           use_bias=False)(x)
# x = BatchNormalization()(x)
# x = ReLU(6.)(x)
x = dense_block(x,
                units=512,
                use_bias=False,
                name='dense_2',
                add_batchnorm=True)

# x = Dense(10,
#           use_bias=False)(x)
# x = BatchNormalization()(x)
x = dense_block(x,
                units=10,
                use_bias=False,
                name='dense_3',
                activ_quantization=None)

model_keras = Model(img_input, x, name='mnistnet')

opt = Adam(lr=lr_start)
model_keras.compile(loss='squared_hinge', optimizer=opt, metrics=['accuracy'])
model_keras.summary()

Out:

Model: "mnistnet"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(None, 28, 28, 1)]       0
_________________________________________________________________
conv_0 (QuantizedConv2D)     (None, 28, 28, 32)        800
_________________________________________________________________
conv_0_maxpool (MaxPooling2D (None, 14, 14, 32)        0
_________________________________________________________________
conv_0_BN (BatchNormalizatio (None, 14, 14, 32)        128
_________________________________________________________________
conv_0_relu (ReLU)           (None, 14, 14, 32)        0
_________________________________________________________________
conv_1 (QuantizedConv2D)     (None, 14, 14, 32)        25600
_________________________________________________________________
conv_1_maxpool (MaxPooling2D (None, 7, 7, 32)          0
_________________________________________________________________
conv_1_BN (BatchNormalizatio (None, 7, 7, 32)          128
_________________________________________________________________
conv_1_relu (ReLU)           (None, 7, 7, 32)          0
_________________________________________________________________
conv_2 (QuantizedConv2D)     (None, 7, 7, 32)          25600
_________________________________________________________________
conv_2_BN (BatchNormalizatio (None, 7, 7, 32)          128
_________________________________________________________________
conv_2_relu (ReLU)           (None, 7, 7, 32)          0
_________________________________________________________________
flatten (Flatten)            (None, 1568)              0
_________________________________________________________________
dense_2 (QuantizedDense)     (None, 512)               802816
_________________________________________________________________
dense_2_BN (BatchNormalizati (None, 512)               2048
_________________________________________________________________
dense_2_relu (ReLU)          (None, 512)               0
_________________________________________________________________
dense_3 (QuantizedDense)     (None, 10)                5120
=================================================================
Total params: 862,368
Trainable params: 861,152
Non-trainable params: 1,216
_________________________________________________________________

3.3 Performance check

Check modifed model performance:

callbacks = []
lr_scheduler = LearningRateScheduler(lambda e: lr_start * lr_decay**e)
callbacks.append(lr_scheduler)
history = model_keras.fit(x_train,
                          y_train,
                          batch_size=batch_size,
                          epochs=epochs,
                          verbose=1,
                          validation_data=(x_test, y_test),
                          callbacks=callbacks)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/5

  1/469 [..............................] - ETA: 0s - loss: 2.0339 - accuracy: 0.1016
  4/469 [..............................] - ETA: 8s - loss: 1.1835 - accuracy: 0.5059
  7/469 [..............................] - ETA: 9s - loss: 0.8182 - accuracy: 0.6283
 10/469 [..............................] - ETA: 9s - loss: 0.6154 - accuracy: 0.6992
 13/469 [..............................] - ETA: 10s - loss: 0.4946 - accuracy: 0.7476
 16/469 [>.............................] - ETA: 10s - loss: 0.4133 - accuracy: 0.7822
 19/469 [>.............................] - ETA: 10s - loss: 0.3597 - accuracy: 0.8035
 21/469 [>.............................] - ETA: 10s - loss: 0.3323 - accuracy: 0.8147
 24/469 [>.............................] - ETA: 10s - loss: 0.2974 - accuracy: 0.8304
 27/469 [>.............................] - ETA: 10s - loss: 0.2685 - accuracy: 0.8446
 30/469 [>.............................] - ETA: 10s - loss: 0.2465 - accuracy: 0.8557
 33/469 [=>............................] - ETA: 10s - loss: 0.2276 - accuracy: 0.8653
 36/469 [=>............................] - ETA: 10s - loss: 0.2115 - accuracy: 0.8737
 39/469 [=>............................] - ETA: 10s - loss: 0.1987 - accuracy: 0.8798
 42/469 [=>............................] - ETA: 10s - loss: 0.1865 - accuracy: 0.8862
 45/469 [=>............................] - ETA: 10s - loss: 0.1773 - accuracy: 0.8901
 48/469 [==>...........................] - ETA: 9s - loss: 0.1683 - accuracy: 0.8947 
 51/469 [==>...........................] - ETA: 9s - loss: 0.1599 - accuracy: 0.8994
 54/469 [==>...........................] - ETA: 9s - loss: 0.1527 - accuracy: 0.9035
 57/469 [==>...........................] - ETA: 9s - loss: 0.1460 - accuracy: 0.9072
 60/469 [==>...........................] - ETA: 9s - loss: 0.1408 - accuracy: 0.9091
 62/469 [==>...........................] - ETA: 9s - loss: 0.1379 - accuracy: 0.9109
 65/469 [===>..........................] - ETA: 9s - loss: 0.1329 - accuracy: 0.9139
 68/469 [===>..........................] - ETA: 9s - loss: 0.1283 - accuracy: 0.9159
 71/469 [===>..........................] - ETA: 9s - loss: 0.1240 - accuracy: 0.9181
 74/469 [===>..........................] - ETA: 9s - loss: 0.1203 - accuracy: 0.9196
 77/469 [===>..........................] - ETA: 9s - loss: 0.1167 - accuracy: 0.9218
 80/469 [====>.........................] - ETA: 9s - loss: 0.1138 - accuracy: 0.9231
 83/469 [====>.........................] - ETA: 9s - loss: 0.1106 - accuracy: 0.9247
 86/469 [====>.........................] - ETA: 9s - loss: 0.1075 - accuracy: 0.9264
 89/469 [====>.........................] - ETA: 9s - loss: 0.1051 - accuracy: 0.9278
 92/469 [====>.........................] - ETA: 9s - loss: 0.1024 - accuracy: 0.9294
 95/469 [=====>........................] - ETA: 8s - loss: 0.1001 - accuracy: 0.9306
 98/469 [=====>........................] - ETA: 8s - loss: 0.0977 - accuracy: 0.9322
101/469 [=====>........................] - ETA: 8s - loss: 0.0956 - accuracy: 0.9332
104/469 [=====>........................] - ETA: 8s - loss: 0.0939 - accuracy: 0.9340
107/469 [=====>........................] - ETA: 8s - loss: 0.0922 - accuracy: 0.9347
110/469 [======>.......................] - ETA: 8s - loss: 0.0903 - accuracy: 0.9358
113/469 [======>.......................] - ETA: 8s - loss: 0.0884 - accuracy: 0.9370
116/469 [======>.......................] - ETA: 8s - loss: 0.0867 - accuracy: 0.9380
119/469 [======>.......................] - ETA: 8s - loss: 0.0851 - accuracy: 0.9389
122/469 [======>.......................] - ETA: 8s - loss: 0.0837 - accuracy: 0.9399
125/469 [======>.......................] - ETA: 8s - loss: 0.0824 - accuracy: 0.9404
128/469 [=======>......................] - ETA: 8s - loss: 0.0809 - accuracy: 0.9414
131/469 [=======>......................] - ETA: 8s - loss: 0.0796 - accuracy: 0.9421
134/469 [=======>......................] - ETA: 8s - loss: 0.0781 - accuracy: 0.9430
137/469 [=======>......................] - ETA: 7s - loss: 0.0768 - accuracy: 0.9438
140/469 [=======>......................] - ETA: 7s - loss: 0.0756 - accuracy: 0.9446
143/469 [========>.....................] - ETA: 7s - loss: 0.0748 - accuracy: 0.9450
146/469 [========>.....................] - ETA: 7s - loss: 0.0735 - accuracy: 0.9458
149/469 [========>.....................] - ETA: 7s - loss: 0.0724 - accuracy: 0.9465
152/469 [========>.....................] - ETA: 7s - loss: 0.0713 - accuracy: 0.9472
155/469 [========>.....................] - ETA: 7s - loss: 0.0703 - accuracy: 0.9477
158/469 [=========>....................] - ETA: 7s - loss: 0.0693 - accuracy: 0.9483
161/469 [=========>....................] - ETA: 7s - loss: 0.0682 - accuracy: 0.9491
164/469 [=========>....................] - ETA: 7s - loss: 0.0672 - accuracy: 0.9496
167/469 [=========>....................] - ETA: 7s - loss: 0.0664 - accuracy: 0.9502
170/469 [=========>....................] - ETA: 7s - loss: 0.0657 - accuracy: 0.9506
173/469 [==========>...................] - ETA: 7s - loss: 0.0649 - accuracy: 0.9512
176/469 [==========>...................] - ETA: 7s - loss: 0.0642 - accuracy: 0.9516
179/469 [==========>...................] - ETA: 6s - loss: 0.0635 - accuracy: 0.9521
182/469 [==========>...................] - ETA: 6s - loss: 0.0628 - accuracy: 0.9523
185/469 [==========>...................] - ETA: 6s - loss: 0.0622 - accuracy: 0.9528
188/469 [===========>..................] - ETA: 6s - loss: 0.0615 - accuracy: 0.9532
191/469 [===========>..................] - ETA: 6s - loss: 0.0609 - accuracy: 0.9536
194/469 [===========>..................] - ETA: 6s - loss: 0.0602 - accuracy: 0.9541
197/469 [===========>..................] - ETA: 6s - loss: 0.0595 - accuracy: 0.9545
200/469 [===========>..................] - ETA: 6s - loss: 0.0589 - accuracy: 0.9549
203/469 [===========>..................] - ETA: 6s - loss: 0.0583 - accuracy: 0.9553
206/469 [============>.................] - ETA: 6s - loss: 0.0577 - accuracy: 0.9556
209/469 [============>.................] - ETA: 6s - loss: 0.0571 - accuracy: 0.9560
212/469 [============>.................] - ETA: 6s - loss: 0.0566 - accuracy: 0.9563
215/469 [============>.................] - ETA: 6s - loss: 0.0560 - accuracy: 0.9568
218/469 [============>.................] - ETA: 6s - loss: 0.0554 - accuracy: 0.9572
221/469 [=============>................] - ETA: 5s - loss: 0.0549 - accuracy: 0.9576
224/469 [=============>................] - ETA: 5s - loss: 0.0543 - accuracy: 0.9581
227/469 [=============>................] - ETA: 5s - loss: 0.0537 - accuracy: 0.9585
229/469 [=============>................] - ETA: 5s - loss: 0.0533 - accuracy: 0.9588
232/469 [=============>................] - ETA: 5s - loss: 0.0530 - accuracy: 0.9588
235/469 [==============>...............] - ETA: 5s - loss: 0.0526 - accuracy: 0.9591
238/469 [==============>...............] - ETA: 5s - loss: 0.0521 - accuracy: 0.9593
241/469 [==============>...............] - ETA: 5s - loss: 0.0517 - accuracy: 0.9596
244/469 [==============>...............] - ETA: 5s - loss: 0.0513 - accuracy: 0.9598
247/469 [==============>...............] - ETA: 5s - loss: 0.0508 - accuracy: 0.9601
250/469 [==============>...............] - ETA: 5s - loss: 0.0503 - accuracy: 0.9604
253/469 [===============>..............] - ETA: 5s - loss: 0.0499 - accuracy: 0.9606
256/469 [===============>..............] - ETA: 5s - loss: 0.0495 - accuracy: 0.9609
259/469 [===============>..............] - ETA: 5s - loss: 0.0491 - accuracy: 0.9612
262/469 [===============>..............] - ETA: 4s - loss: 0.0487 - accuracy: 0.9615
265/469 [===============>..............] - ETA: 4s - loss: 0.0483 - accuracy: 0.9618
268/469 [================>.............] - ETA: 4s - loss: 0.0479 - accuracy: 0.9621
271/469 [================>.............] - ETA: 4s - loss: 0.0475 - accuracy: 0.9624
274/469 [================>.............] - ETA: 4s - loss: 0.0471 - accuracy: 0.9626
277/469 [================>.............] - ETA: 4s - loss: 0.0468 - accuracy: 0.9628
280/469 [================>.............] - ETA: 4s - loss: 0.0464 - accuracy: 0.9629
283/469 [=================>............] - ETA: 4s - loss: 0.0461 - accuracy: 0.9630
286/469 [=================>............] - ETA: 4s - loss: 0.0458 - accuracy: 0.9632
289/469 [=================>............] - ETA: 4s - loss: 0.0454 - accuracy: 0.9635
292/469 [=================>............] - ETA: 4s - loss: 0.0451 - accuracy: 0.9638
295/469 [=================>............] - ETA: 4s - loss: 0.0447 - accuracy: 0.9641
298/469 [==================>...........] - ETA: 4s - loss: 0.0444 - accuracy: 0.9644
301/469 [==================>...........] - ETA: 4s - loss: 0.0440 - accuracy: 0.9646
304/469 [==================>...........] - ETA: 3s - loss: 0.0438 - accuracy: 0.9646
307/469 [==================>...........] - ETA: 3s - loss: 0.0435 - accuracy: 0.9649
310/469 [==================>...........] - ETA: 3s - loss: 0.0432 - accuracy: 0.9650
313/469 [===================>..........] - ETA: 3s - loss: 0.0430 - accuracy: 0.9651
316/469 [===================>..........] - ETA: 3s - loss: 0.0427 - accuracy: 0.9653
319/469 [===================>..........] - ETA: 3s - loss: 0.0425 - accuracy: 0.9654
322/469 [===================>..........] - ETA: 3s - loss: 0.0422 - accuracy: 0.9656
325/469 [===================>..........] - ETA: 3s - loss: 0.0419 - accuracy: 0.9658
328/469 [===================>..........] - ETA: 3s - loss: 0.0416 - accuracy: 0.9660
331/469 [====================>.........] - ETA: 3s - loss: 0.0414 - accuracy: 0.9661
334/469 [====================>.........] - ETA: 3s - loss: 0.0411 - accuracy: 0.9663
337/469 [====================>.........] - ETA: 3s - loss: 0.0408 - accuracy: 0.9665
340/469 [====================>.........] - ETA: 3s - loss: 0.0406 - accuracy: 0.9668
343/469 [====================>.........] - ETA: 3s - loss: 0.0403 - accuracy: 0.9670
346/469 [=====================>........] - ETA: 2s - loss: 0.0400 - accuracy: 0.9673
349/469 [=====================>........] - ETA: 2s - loss: 0.0398 - accuracy: 0.9675
352/469 [=====================>........] - ETA: 2s - loss: 0.0396 - accuracy: 0.9676
355/469 [=====================>........] - ETA: 2s - loss: 0.0393 - accuracy: 0.9678
358/469 [=====================>........] - ETA: 2s - loss: 0.0390 - accuracy: 0.9679
361/469 [======================>.......] - ETA: 2s - loss: 0.0388 - accuracy: 0.9681
364/469 [======================>.......] - ETA: 2s - loss: 0.0386 - accuracy: 0.9681
367/469 [======================>.......] - ETA: 2s - loss: 0.0384 - accuracy: 0.9683
370/469 [======================>.......] - ETA: 2s - loss: 0.0382 - accuracy: 0.9685
373/469 [======================>.......] - ETA: 2s - loss: 0.0380 - accuracy: 0.9686
376/469 [=======================>......] - ETA: 2s - loss: 0.0377 - accuracy: 0.9688
379/469 [=======================>......] - ETA: 2s - loss: 0.0375 - accuracy: 0.9690
382/469 [=======================>......] - ETA: 2s - loss: 0.0373 - accuracy: 0.9692
385/469 [=======================>......] - ETA: 2s - loss: 0.0371 - accuracy: 0.9692
388/469 [=======================>......] - ETA: 1s - loss: 0.0370 - accuracy: 0.9694
391/469 [========================>.....] - ETA: 1s - loss: 0.0368 - accuracy: 0.9695
394/469 [========================>.....] - ETA: 1s - loss: 0.0366 - accuracy: 0.9697
397/469 [========================>.....] - ETA: 1s - loss: 0.0364 - accuracy: 0.9698
400/469 [========================>.....] - ETA: 1s - loss: 0.0363 - accuracy: 0.9698
403/469 [========================>.....] - ETA: 1s - loss: 0.0361 - accuracy: 0.9700
406/469 [========================>.....] - ETA: 1s - loss: 0.0359 - accuracy: 0.9702
409/469 [=========================>....] - ETA: 1s - loss: 0.0358 - accuracy: 0.9703
412/469 [=========================>....] - ETA: 1s - loss: 0.0357 - accuracy: 0.9703
415/469 [=========================>....] - ETA: 1s - loss: 0.0355 - accuracy: 0.9704
418/469 [=========================>....] - ETA: 1s - loss: 0.0353 - accuracy: 0.9706
421/469 [=========================>....] - ETA: 1s - loss: 0.0351 - accuracy: 0.9708
424/469 [==========================>...] - ETA: 1s - loss: 0.0351 - accuracy: 0.9710
427/469 [==========================>...] - ETA: 1s - loss: 0.0349 - accuracy: 0.9711
430/469 [==========================>...] - ETA: 0s - loss: 0.0347 - accuracy: 0.9712
433/469 [==========================>...] - ETA: 0s - loss: 0.0346 - accuracy: 0.9713
436/469 [==========================>...] - ETA: 0s - loss: 0.0344 - accuracy: 0.9715
438/469 [===========================>..] - ETA: 0s - loss: 0.0343 - accuracy: 0.9715
441/469 [===========================>..] - ETA: 0s - loss: 0.0342 - accuracy: 0.9716
444/469 [===========================>..] - ETA: 0s - loss: 0.0340 - accuracy: 0.9717
447/469 [===========================>..] - ETA: 0s - loss: 0.0338 - accuracy: 0.9718
450/469 [===========================>..] - ETA: 0s - loss: 0.0337 - accuracy: 0.9719
453/469 [===========================>..] - ETA: 0s - loss: 0.0335 - accuracy: 0.9720
456/469 [============================>.] - ETA: 0s - loss: 0.0334 - accuracy: 0.9721
459/469 [============================>.] - ETA: 0s - loss: 0.0332 - accuracy: 0.9722
462/469 [============================>.] - ETA: 0s - loss: 0.0331 - accuracy: 0.9723
465/469 [============================>.] - ETA: 0s - loss: 0.0330 - accuracy: 0.9724
468/469 [============================>.] - ETA: 0s - loss: 0.0328 - accuracy: 0.9725
469/469 [==============================] - 12s 25ms/step - loss: 0.0328 - accuracy: 0.9725 - val_loss: 0.0674 - val_accuracy: 0.9164 - lr: 0.0010
Epoch 2/5

  1/469 [..............................] - ETA: 0s - loss: 0.0052 - accuracy: 0.9922
  4/469 [..............................] - ETA: 8s - loss: 0.0070 - accuracy: 0.9922
  7/469 [..............................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9944
 10/469 [..............................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9953
 13/469 [..............................] - ETA: 10s - loss: 0.0052 - accuracy: 0.9952
 16/469 [>.............................] - ETA: 10s - loss: 0.0056 - accuracy: 0.9951
 19/469 [>.............................] - ETA: 10s - loss: 0.0062 - accuracy: 0.9947
 22/469 [>.............................] - ETA: 10s - loss: 0.0063 - accuracy: 0.9940
 25/469 [>.............................] - ETA: 10s - loss: 0.0064 - accuracy: 0.9937
 28/469 [>.............................] - ETA: 10s - loss: 0.0061 - accuracy: 0.9944
 31/469 [>.............................] - ETA: 10s - loss: 0.0060 - accuracy: 0.9945
 34/469 [=>............................] - ETA: 10s - loss: 0.0057 - accuracy: 0.9949
 37/469 [=>............................] - ETA: 10s - loss: 0.0055 - accuracy: 0.9951
 40/469 [=>............................] - ETA: 10s - loss: 0.0055 - accuracy: 0.9951
 43/469 [=>............................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9951
 46/469 [=>............................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9952 
 49/469 [==>...........................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9951
 52/469 [==>...........................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9950
 55/469 [==>...........................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9949
 58/469 [==>...........................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9950
 61/469 [==>...........................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9950
 64/469 [===>..........................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9948
 67/469 [===>..........................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9949
 70/469 [===>..........................] - ETA: 9s - loss: 0.0055 - accuracy: 0.9949
 73/469 [===>..........................] - ETA: 9s - loss: 0.0061 - accuracy: 0.9946
 76/469 [===>..........................] - ETA: 9s - loss: 0.0060 - accuracy: 0.9947
 79/469 [====>.........................] - ETA: 9s - loss: 0.0062 - accuracy: 0.9943
 82/469 [====>.........................] - ETA: 9s - loss: 0.0062 - accuracy: 0.9943
 85/469 [====>.........................] - ETA: 9s - loss: 0.0063 - accuracy: 0.9941
 88/469 [====>.........................] - ETA: 9s - loss: 0.0063 - accuracy: 0.9941
 91/469 [====>.........................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9942
 94/469 [=====>........................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9939
 97/469 [=====>........................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9940
100/469 [=====>........................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9940
103/469 [=====>........................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9939
106/469 [=====>........................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9939
109/469 [=====>........................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9937
112/469 [======>.......................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9937
115/469 [======>.......................] - ETA: 8s - loss: 0.0064 - accuracy: 0.9936
118/469 [======>.......................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9937
121/469 [======>.......................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9937
124/469 [======>.......................] - ETA: 8s - loss: 0.0063 - accuracy: 0.9937
127/469 [=======>......................] - ETA: 8s - loss: 0.0062 - accuracy: 0.9938
130/469 [=======>......................] - ETA: 8s - loss: 0.0061 - accuracy: 0.9939
133/469 [=======>......................] - ETA: 8s - loss: 0.0061 - accuracy: 0.9938
136/469 [=======>......................] - ETA: 7s - loss: 0.0061 - accuracy: 0.9938
139/469 [=======>......................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9939
142/469 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9940
145/469 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9941
148/469 [========>.....................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9941
151/469 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9941
153/469 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9941
156/469 [========>.....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9940
159/469 [=========>....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9939
162/469 [=========>....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9940
165/469 [=========>....................] - ETA: 7s - loss: 0.0060 - accuracy: 0.9940
168/469 [=========>....................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9941
171/469 [=========>....................] - ETA: 7s - loss: 0.0059 - accuracy: 0.9940
174/469 [==========>...................] - ETA: 7s - loss: 0.0058 - accuracy: 0.9941
177/469 [==========>...................] - ETA: 6s - loss: 0.0058 - accuracy: 0.9942
180/469 [==========>...................] - ETA: 6s - loss: 0.0059 - accuracy: 0.9940
183/469 [==========>...................] - ETA: 6s - loss: 0.0058 - accuracy: 0.9940
186/469 [==========>...................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9941
189/469 [===========>..................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9942
192/469 [===========>..................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9942
195/469 [===========>..................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9943
198/469 [===========>..................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9942
201/469 [===========>..................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9943
204/469 [============>.................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9943
207/469 [============>.................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9943
210/469 [============>.................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9943
213/469 [============>.................] - ETA: 6s - loss: 0.0057 - accuracy: 0.9943
216/469 [============>.................] - ETA: 6s - loss: 0.0056 - accuracy: 0.9943
219/469 [=============>................] - ETA: 5s - loss: 0.0056 - accuracy: 0.9944
222/469 [=============>................] - ETA: 5s - loss: 0.0056 - accuracy: 0.9944
225/469 [=============>................] - ETA: 5s - loss: 0.0056 - accuracy: 0.9943
228/469 [=============>................] - ETA: 5s - loss: 0.0056 - accuracy: 0.9943
231/469 [=============>................] - ETA: 5s - loss: 0.0056 - accuracy: 0.9943
234/469 [=============>................] - ETA: 5s - loss: 0.0058 - accuracy: 0.9942
237/469 [==============>...............] - ETA: 5s - loss: 0.0059 - accuracy: 0.9941
240/469 [==============>...............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
243/469 [==============>...............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
246/469 [==============>...............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
249/469 [==============>...............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
252/469 [===============>..............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
255/469 [===============>..............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
258/469 [===============>..............] - ETA: 5s - loss: 0.0058 - accuracy: 0.9941
261/469 [===============>..............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9941
264/469 [===============>..............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9941
267/469 [================>.............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9941
270/469 [================>.............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9942
273/469 [================>.............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9942
276/469 [================>.............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9941
279/469 [================>.............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9940
282/469 [=================>............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9941
285/469 [=================>............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9940
288/469 [=================>............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9940
291/469 [=================>............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9940
294/469 [=================>............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9940
297/469 [=================>............] - ETA: 4s - loss: 0.0058 - accuracy: 0.9939
300/469 [==================>...........] - ETA: 4s - loss: 0.0058 - accuracy: 0.9939
303/469 [==================>...........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9939
306/469 [==================>...........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9939
309/469 [==================>...........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9939
312/469 [==================>...........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9938
315/469 [===================>..........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
318/469 [===================>..........] - ETA: 3s - loss: 0.0059 - accuracy: 0.9937
321/469 [===================>..........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9937
324/469 [===================>..........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9937
327/469 [===================>..........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9937
330/469 [====================>.........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9937
333/469 [====================>.........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9937
336/469 [====================>.........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9937
339/469 [====================>.........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9938
342/469 [====================>.........] - ETA: 3s - loss: 0.0058 - accuracy: 0.9938
345/469 [=====================>........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9938
348/469 [=====================>........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9939
351/469 [=====================>........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9939
354/469 [=====================>........] - ETA: 2s - loss: 0.0058 - accuracy: 0.9939
357/469 [=====================>........] - ETA: 2s - loss: 0.0057 - accuracy: 0.9939
360/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9939
363/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9939
366/469 [======================>.......] - ETA: 2s - loss: 0.0058 - accuracy: 0.9939
369/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9939
372/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9939
375/469 [======================>.......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9939
378/469 [=======================>......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9939
381/469 [=======================>......] - ETA: 2s - loss: 0.0058 - accuracy: 0.9938
384/469 [=======================>......] - ETA: 2s - loss: 0.0057 - accuracy: 0.9938
387/469 [=======================>......] - ETA: 1s - loss: 0.0058 - accuracy: 0.9938
390/469 [=======================>......] - ETA: 1s - loss: 0.0058 - accuracy: 0.9938
393/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9938
396/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9937
399/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9937
402/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9937
405/469 [========================>.....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9937
408/469 [=========================>....] - ETA: 1s - loss: 0.0058 - accuracy: 0.9937
411/469 [=========================>....] - ETA: 1s - loss: 0.0059 - accuracy: 0.9937
414/469 [=========================>....] - ETA: 1s - loss: 0.0059 - accuracy: 0.9936
417/469 [=========================>....] - ETA: 1s - loss: 0.0059 - accuracy: 0.9936
420/469 [=========================>....] - ETA: 1s - loss: 0.0059 - accuracy: 0.9936
423/469 [==========================>...] - ETA: 1s - loss: 0.0059 - accuracy: 0.9936
426/469 [==========================>...] - ETA: 1s - loss: 0.0058 - accuracy: 0.9937
429/469 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9937
432/469 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9937
435/469 [==========================>...] - ETA: 0s - loss: 0.0058 - accuracy: 0.9936
438/469 [===========================>..] - ETA: 0s - loss: 0.0058 - accuracy: 0.9936
441/469 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9935
444/469 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9935
447/469 [===========================>..] - ETA: 0s - loss: 0.0059 - accuracy: 0.9935
450/469 [===========================>..] - ETA: 0s - loss: 0.0060 - accuracy: 0.9935
453/469 [===========================>..] - ETA: 0s - loss: 0.0060 - accuracy: 0.9934
456/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9934
459/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9934
462/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9934
465/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9934
468/469 [============================>.] - ETA: 0s - loss: 0.0060 - accuracy: 0.9934
469/469 [==============================] - 12s 25ms/step - loss: 0.0060 - accuracy: 0.9934 - val_loss: 0.0079 - val_accuracy: 0.9907 - lr: 6.3096e-04
Epoch 3/5

  1/469 [..............................] - ETA: 0s - loss: 0.0065 - accuracy: 0.9922
  4/469 [..............................] - ETA: 8s - loss: 0.0067 - accuracy: 0.9922
  7/469 [..............................] - ETA: 9s - loss: 0.0046 - accuracy: 0.9955
 10/469 [..............................] - ETA: 10s - loss: 0.0039 - accuracy: 0.9969
 13/469 [..............................] - ETA: 10s - loss: 0.0038 - accuracy: 0.9970
 16/469 [>.............................] - ETA: 10s - loss: 0.0042 - accuracy: 0.9966
 19/469 [>.............................] - ETA: 10s - loss: 0.0046 - accuracy: 0.9955
 22/469 [>.............................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9954
 25/469 [>.............................] - ETA: 10s - loss: 0.0042 - accuracy: 0.9959
 28/469 [>.............................] - ETA: 10s - loss: 0.0041 - accuracy: 0.9961
 31/469 [>.............................] - ETA: 10s - loss: 0.0042 - accuracy: 0.9960
 34/469 [=>............................] - ETA: 10s - loss: 0.0040 - accuracy: 0.9961
 37/469 [=>............................] - ETA: 10s - loss: 0.0039 - accuracy: 0.9962
 40/469 [=>............................] - ETA: 10s - loss: 0.0038 - accuracy: 0.9963
 43/469 [=>............................] - ETA: 10s - loss: 0.0036 - accuracy: 0.9964
 46/469 [=>............................] - ETA: 9s - loss: 0.0036 - accuracy: 0.9963 
 49/469 [==>...........................] - ETA: 9s - loss: 0.0035 - accuracy: 0.9965
 52/469 [==>...........................] - ETA: 9s - loss: 0.0034 - accuracy: 0.9964
 55/469 [==>...........................] - ETA: 9s - loss: 0.0033 - accuracy: 0.9966
 58/469 [==>...........................] - ETA: 9s - loss: 0.0032 - accuracy: 0.9968
 61/469 [==>...........................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9969
 64/469 [===>..........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9969
 67/469 [===>..........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9970
 70/469 [===>..........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9970
 73/469 [===>..........................] - ETA: 9s - loss: 0.0029 - accuracy: 0.9970
 76/469 [===>..........................] - ETA: 9s - loss: 0.0029 - accuracy: 0.9969
 79/469 [====>.........................] - ETA: 9s - loss: 0.0030 - accuracy: 0.9967
 82/469 [====>.........................] - ETA: 9s - loss: 0.0031 - accuracy: 0.9966
 85/469 [====>.........................] - ETA: 9s - loss: 0.0032 - accuracy: 0.9965
 88/469 [====>.........................] - ETA: 9s - loss: 0.0032 - accuracy: 0.9965
 91/469 [====>.........................] - ETA: 9s - loss: 0.0032 - accuracy: 0.9966
 94/469 [=====>........................] - ETA: 8s - loss: 0.0032 - accuracy: 0.9966
 97/469 [=====>........................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9966
100/469 [=====>........................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9967
103/469 [=====>........................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9967
106/469 [=====>........................] - ETA: 8s - loss: 0.0032 - accuracy: 0.9966
109/469 [=====>........................] - ETA: 8s - loss: 0.0032 - accuracy: 0.9966
112/469 [======>.......................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9967
115/469 [======>.......................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9967
118/469 [======>.......................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9968
121/469 [======>.......................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9968
124/469 [======>.......................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9967
127/469 [=======>......................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9968
130/469 [=======>......................] - ETA: 8s - loss: 0.0031 - accuracy: 0.9969
133/469 [=======>......................] - ETA: 8s - loss: 0.0030 - accuracy: 0.9969
136/469 [=======>......................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9968
139/469 [=======>......................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9966
142/469 [========>.....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
145/469 [========>.....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
148/469 [========>.....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
151/469 [========>.....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
154/469 [========>.....................] - ETA: 7s - loss: 0.0032 - accuracy: 0.9967
157/469 [=========>....................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9967
160/469 [=========>....................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9967
163/469 [=========>....................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9967
166/469 [=========>....................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9967
169/469 [=========>....................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9968
172/469 [==========>...................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9967
175/469 [==========>...................] - ETA: 7s - loss: 0.0031 - accuracy: 0.9967
178/469 [==========>...................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9967
181/469 [==========>...................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9966
184/469 [==========>...................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9966
187/469 [==========>...................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9966
190/469 [===========>..................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9967
193/469 [===========>..................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9966
196/469 [===========>..................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9966
199/469 [===========>..................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9967
202/469 [===========>..................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9967
205/469 [============>.................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9967
208/469 [============>.................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9967
211/469 [============>.................] - ETA: 6s - loss: 0.0032 - accuracy: 0.9967
214/469 [============>.................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9968
217/469 [============>.................] - ETA: 6s - loss: 0.0031 - accuracy: 0.9968
220/469 [=============>................] - ETA: 5s - loss: 0.0031 - accuracy: 0.9967
223/469 [=============>................] - ETA: 5s - loss: 0.0031 - accuracy: 0.9967
226/469 [=============>................] - ETA: 5s - loss: 0.0031 - accuracy: 0.9968
229/469 [=============>................] - ETA: 5s - loss: 0.0031 - accuracy: 0.9967
232/469 [=============>................] - ETA: 5s - loss: 0.0031 - accuracy: 0.9967
235/469 [==============>...............] - ETA: 5s - loss: 0.0031 - accuracy: 0.9968
238/469 [==============>...............] - ETA: 5s - loss: 0.0031 - accuracy: 0.9968
241/469 [==============>...............] - ETA: 5s - loss: 0.0031 - accuracy: 0.9968
244/469 [==============>...............] - ETA: 5s - loss: 0.0031 - accuracy: 0.9968
247/469 [==============>...............] - ETA: 5s - loss: 0.0031 - accuracy: 0.9969
250/469 [==============>...............] - ETA: 5s - loss: 0.0030 - accuracy: 0.9969
253/469 [===============>..............] - ETA: 5s - loss: 0.0030 - accuracy: 0.9969
256/469 [===============>..............] - ETA: 5s - loss: 0.0030 - accuracy: 0.9969
259/469 [===============>..............] - ETA: 5s - loss: 0.0030 - accuracy: 0.9969
262/469 [===============>..............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
265/469 [===============>..............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
268/469 [================>.............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
271/469 [================>.............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
274/469 [================>.............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9970
277/469 [================>.............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
280/469 [================>.............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
283/469 [=================>............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
286/469 [=================>............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
288/469 [=================>............] - ETA: 4s - loss: 0.0030 - accuracy: 0.9970
291/469 [=================>............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9970
294/469 [=================>............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
297/469 [=================>............] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
300/469 [==================>...........] - ETA: 4s - loss: 0.0029 - accuracy: 0.9971
303/469 [==================>...........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
306/469 [==================>...........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
309/469 [==================>...........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
312/469 [==================>...........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
315/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
318/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9971
321/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9972
324/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9972
327/469 [===================>..........] - ETA: 3s - loss: 0.0029 - accuracy: 0.9972
330/469 [====================>.........] - ETA: 3s - loss: 0.0028 - accuracy: 0.9972
333/469 [====================>.........] - ETA: 3s - loss: 0.0028 - accuracy: 0.9972
336/469 [====================>.........] - ETA: 3s - loss: 0.0028 - accuracy: 0.9972
339/469 [====================>.........] - ETA: 3s - loss: 0.0028 - accuracy: 0.9973
342/469 [====================>.........] - ETA: 3s - loss: 0.0028 - accuracy: 0.9972
345/469 [=====================>........] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
348/469 [=====================>........] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
351/469 [=====================>........] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
354/469 [=====================>........] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
357/469 [=====================>........] - ETA: 2s - loss: 0.0028 - accuracy: 0.9973
360/469 [======================>.......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
363/469 [======================>.......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9973
366/469 [======================>.......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9973
369/469 [======================>.......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9973
372/469 [======================>.......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
375/469 [======================>.......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
378/469 [=======================>......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9973
381/469 [=======================>......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
384/469 [=======================>......] - ETA: 2s - loss: 0.0028 - accuracy: 0.9972
387/469 [=======================>......] - ETA: 1s - loss: 0.0028 - accuracy: 0.9972
390/469 [=======================>......] - ETA: 1s - loss: 0.0028 - accuracy: 0.9972
393/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9972
396/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
399/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
402/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
405/469 [========================>.....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
408/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
411/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
414/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
417/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
420/469 [=========================>....] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
423/469 [==========================>...] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
426/469 [==========================>...] - ETA: 1s - loss: 0.0028 - accuracy: 0.9973
429/469 [==========================>...] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
432/469 [==========================>...] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
435/469 [==========================>...] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
438/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
441/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
444/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9972
447/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
450/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
453/469 [===========================>..] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
456/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
459/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
462/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
465/469 [============================>.] - ETA: 0s - loss: 0.0027 - accuracy: 0.9973
468/469 [============================>.] - ETA: 0s - loss: 0.0028 - accuracy: 0.9973
469/469 [==============================] - 12s 25ms/step - loss: 0.0027 - accuracy: 0.9973 - val_loss: 0.0060 - val_accuracy: 0.9926 - lr: 3.9811e-04
Epoch 4/5

  1/469 [..............................] - ETA: 0s - loss: 5.9717e-04 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 0.0017 - accuracy: 0.9980    
  6/469 [..............................] - ETA: 9s - loss: 0.0017 - accuracy: 0.9987
  9/469 [..............................] - ETA: 9s - loss: 0.0019 - accuracy: 0.9983
 12/469 [..............................] - ETA: 10s - loss: 0.0015 - accuracy: 0.9987
 15/469 [..............................] - ETA: 10s - loss: 0.0015 - accuracy: 0.9984
 18/469 [>.............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9987
 21/469 [>.............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9985
 24/469 [>.............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9987
 27/469 [>.............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9988
 30/469 [>.............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9987
 33/469 [=>............................] - ETA: 10s - loss: 0.0014 - accuracy: 0.9988
 36/469 [=>............................] - ETA: 10s - loss: 0.0015 - accuracy: 0.9985
 39/469 [=>............................] - ETA: 10s - loss: 0.0015 - accuracy: 0.9982
 42/469 [=>............................] - ETA: 10s - loss: 0.0017 - accuracy: 0.9981
 45/469 [=>............................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9983 
 48/469 [==>...........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9984
 51/469 [==>...........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9985
 54/469 [==>...........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9986
 57/469 [==>...........................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9986
 60/469 [==>...........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9984
 63/469 [===>..........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9985
 66/469 [===>..........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9986
 69/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9986
 72/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9987
 75/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9987
 78/469 [===>..........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9988
 81/469 [====>.........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9988
 84/469 [====>.........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9988
 87/469 [====>.........................] - ETA: 9s - loss: 0.0016 - accuracy: 0.9988
 90/469 [====>.........................] - ETA: 9s - loss: 0.0015 - accuracy: 0.9989
 93/469 [====>.........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9987
 96/469 [=====>........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9987
 99/469 [=====>........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9987
102/469 [=====>........................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9988
105/469 [=====>........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9987
108/469 [=====>........................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9988
111/469 [======>.......................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9988
114/469 [======>.......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9987
117/469 [======>.......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9987
120/469 [======>.......................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9988
123/469 [======>.......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9987
126/469 [=======>......................] - ETA: 8s - loss: 0.0016 - accuracy: 0.9988
129/469 [=======>......................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9988
132/469 [=======>......................] - ETA: 8s - loss: 0.0015 - accuracy: 0.9988
135/469 [=======>......................] - ETA: 7s - loss: 0.0016 - accuracy: 0.9986
138/469 [=======>......................] - ETA: 7s - loss: 0.0016 - accuracy: 0.9986
140/469 [=======>......................] - ETA: 7s - loss: 0.0016 - accuracy: 0.9986
143/469 [========>.....................] - ETA: 7s - loss: 0.0016 - accuracy: 0.9986
146/469 [========>.....................] - ETA: 7s - loss: 0.0016 - accuracy: 0.9986
149/469 [========>.....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
152/469 [========>.....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9987
155/469 [========>.....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
158/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
161/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9986
164/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9987
167/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9987
170/469 [=========>....................] - ETA: 7s - loss: 0.0015 - accuracy: 0.9987
173/469 [==========>...................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9987
176/469 [==========>...................] - ETA: 7s - loss: 0.0014 - accuracy: 0.9988
179/469 [==========>...................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9988
182/469 [==========>...................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9988
185/469 [==========>...................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9988
188/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9988
191/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
194/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
197/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
200/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
203/469 [===========>..................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
206/469 [============>.................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
209/469 [============>.................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
212/469 [============>.................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
215/469 [============>.................] - ETA: 6s - loss: 0.0014 - accuracy: 0.9989
218/469 [============>.................] - ETA: 6s - loss: 0.0013 - accuracy: 0.9989
221/469 [=============>................] - ETA: 5s - loss: 0.0013 - accuracy: 0.9989
224/469 [=============>................] - ETA: 5s - loss: 0.0013 - accuracy: 0.9989
227/469 [=============>................] - ETA: 5s - loss: 0.0014 - accuracy: 0.9989
230/469 [=============>................] - ETA: 5s - loss: 0.0013 - accuracy: 0.9988
233/469 [=============>................] - ETA: 5s - loss: 0.0014 - accuracy: 0.9988
236/469 [==============>...............] - ETA: 5s - loss: 0.0014 - accuracy: 0.9988
239/469 [==============>...............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9988
242/469 [==============>...............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9988
245/469 [==============>...............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9989
248/469 [==============>...............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9988
251/469 [===============>..............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9988
254/469 [===============>..............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9988
257/469 [===============>..............] - ETA: 5s - loss: 0.0014 - accuracy: 0.9988
260/469 [===============>..............] - ETA: 5s - loss: 0.0013 - accuracy: 0.9988
263/469 [===============>..............] - ETA: 4s - loss: 0.0014 - accuracy: 0.9988
266/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9988
269/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9988
272/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9989
275/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9989
278/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9988
281/469 [================>.............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9989
284/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9989
287/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9989
290/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9988
293/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9989
296/469 [=================>............] - ETA: 4s - loss: 0.0013 - accuracy: 0.9988
299/469 [==================>...........] - ETA: 4s - loss: 0.0013 - accuracy: 0.9989
302/469 [==================>...........] - ETA: 4s - loss: 0.0013 - accuracy: 0.9988
305/469 [==================>...........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9988
308/469 [==================>...........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9988
311/469 [==================>...........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9988
314/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9989
317/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9988
320/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9989
323/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9989
326/469 [===================>..........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9988
329/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9989
332/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9989
335/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9989
338/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9989
341/469 [====================>.........] - ETA: 3s - loss: 0.0013 - accuracy: 0.9989
344/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
347/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
350/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
353/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
356/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
359/469 [=====================>........] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
362/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
365/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
368/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
371/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
374/469 [======================>.......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
377/469 [=======================>......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9988
380/469 [=======================>......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9988
383/469 [=======================>......] - ETA: 2s - loss: 0.0013 - accuracy: 0.9989
386/469 [=======================>......] - ETA: 1s - loss: 0.0013 - accuracy: 0.9989
389/469 [=======================>......] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
392/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
395/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
398/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
401/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
404/469 [========================>.....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
407/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
410/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
413/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
416/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
419/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
422/469 [=========================>....] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
424/469 [==========================>...] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
427/469 [==========================>...] - ETA: 1s - loss: 0.0013 - accuracy: 0.9988
430/469 [==========================>...] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
433/469 [==========================>...] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
436/469 [==========================>...] - ETA: 0s - loss: 0.0013 - accuracy: 0.9989
439/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9989
442/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9989
445/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9989
448/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
451/469 [===========================>..] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
454/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
457/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
460/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
463/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
466/469 [============================>.] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
469/469 [==============================] - ETA: 0s - loss: 0.0013 - accuracy: 0.9988
469/469 [==============================] - 12s 25ms/step - loss: 0.0013 - accuracy: 0.9988 - val_loss: 0.0051 - val_accuracy: 0.9936 - lr: 2.5119e-04
Epoch 5/5

  1/469 [..............................] - ETA: 0s - loss: 5.2989e-05 - accuracy: 1.0000
  4/469 [..............................] - ETA: 8s - loss: 0.0022 - accuracy: 0.9980    
  7/469 [..............................] - ETA: 9s - loss: 0.0014 - accuracy: 0.9989
 10/469 [..............................] - ETA: 9s - loss: 0.0013 - accuracy: 0.9984
 13/469 [..............................] - ETA: 10s - loss: 0.0013 - accuracy: 0.9982
 16/469 [>.............................] - ETA: 10s - loss: 0.0012 - accuracy: 0.9985
 19/469 [>.............................] - ETA: 10s - loss: 0.0011 - accuracy: 0.9988
 22/469 [>.............................] - ETA: 10s - loss: 0.0010 - accuracy: 0.9989
 25/469 [>.............................] - ETA: 10s - loss: 9.7454e-04 - accuracy: 0.9991
 28/469 [>.............................] - ETA: 10s - loss: 9.2347e-04 - accuracy: 0.9992
 31/469 [>.............................] - ETA: 10s - loss: 9.3390e-04 - accuracy: 0.9992
 34/469 [=>............................] - ETA: 10s - loss: 8.7113e-04 - accuracy: 0.9993
 37/469 [=>............................] - ETA: 10s - loss: 8.9078e-04 - accuracy: 0.9992
 40/469 [=>............................] - ETA: 10s - loss: 8.7307e-04 - accuracy: 0.9992
 43/469 [=>............................] - ETA: 10s - loss: 8.1421e-04 - accuracy: 0.9993
 46/469 [=>............................] - ETA: 9s - loss: 8.4203e-04 - accuracy: 0.9992 
 49/469 [==>...........................] - ETA: 9s - loss: 8.4998e-04 - accuracy: 0.9992
 52/469 [==>...........................] - ETA: 9s - loss: 8.1326e-04 - accuracy: 0.9992
 55/469 [==>...........................] - ETA: 9s - loss: 8.2519e-04 - accuracy: 0.9993
 58/469 [==>...........................] - ETA: 9s - loss: 8.3813e-04 - accuracy: 0.9993
 61/469 [==>...........................] - ETA: 9s - loss: 8.1860e-04 - accuracy: 0.9994
 64/469 [===>..........................] - ETA: 9s - loss: 8.3483e-04 - accuracy: 0.9994
 67/469 [===>..........................] - ETA: 9s - loss: 8.1266e-04 - accuracy: 0.9994
 70/469 [===>..........................] - ETA: 9s - loss: 8.6465e-04 - accuracy: 0.9993
 73/469 [===>..........................] - ETA: 9s - loss: 8.6936e-04 - accuracy: 0.9993
 76/469 [===>..........................] - ETA: 9s - loss: 8.8273e-04 - accuracy: 0.9992
 79/469 [====>.........................] - ETA: 9s - loss: 8.8114e-04 - accuracy: 0.9992
 82/469 [====>.........................] - ETA: 9s - loss: 8.8372e-04 - accuracy: 0.9992
 85/469 [====>.........................] - ETA: 9s - loss: 8.7609e-04 - accuracy: 0.9993
 88/469 [====>.........................] - ETA: 9s - loss: 8.7566e-04 - accuracy: 0.9993
 91/469 [====>.........................] - ETA: 8s - loss: 8.5831e-04 - accuracy: 0.9993
 94/469 [=====>........................] - ETA: 8s - loss: 8.3798e-04 - accuracy: 0.9993
 97/469 [=====>........................] - ETA: 8s - loss: 8.3574e-04 - accuracy: 0.9994
100/469 [=====>........................] - ETA: 8s - loss: 8.1971e-04 - accuracy: 0.9994
103/469 [=====>........................] - ETA: 8s - loss: 8.0813e-04 - accuracy: 0.9994
106/469 [=====>........................] - ETA: 8s - loss: 8.1613e-04 - accuracy: 0.9993
109/469 [=====>........................] - ETA: 8s - loss: 8.1596e-04 - accuracy: 0.9994
112/469 [======>.......................] - ETA: 8s - loss: 8.0225e-04 - accuracy: 0.9994
115/469 [======>.......................] - ETA: 8s - loss: 7.9793e-04 - accuracy: 0.9994
118/469 [======>.......................] - ETA: 8s - loss: 8.0857e-04 - accuracy: 0.9994
121/469 [======>.......................] - ETA: 8s - loss: 7.9757e-04 - accuracy: 0.9994
124/469 [======>.......................] - ETA: 8s - loss: 7.9989e-04 - accuracy: 0.9994
127/469 [=======>......................] - ETA: 8s - loss: 7.9810e-04 - accuracy: 0.9994
130/469 [=======>......................] - ETA: 8s - loss: 7.8527e-04 - accuracy: 0.9995
133/469 [=======>......................] - ETA: 8s - loss: 7.7393e-04 - accuracy: 0.9995
136/469 [=======>......................] - ETA: 7s - loss: 7.7404e-04 - accuracy: 0.9995
139/469 [=======>......................] - ETA: 7s - loss: 7.8162e-04 - accuracy: 0.9995
142/469 [========>.....................] - ETA: 7s - loss: 7.7592e-04 - accuracy: 0.9995
145/469 [========>.....................] - ETA: 7s - loss: 7.6176e-04 - accuracy: 0.9995
148/469 [========>.....................] - ETA: 7s - loss: 7.5092e-04 - accuracy: 0.9995
151/469 [========>.....................] - ETA: 7s - loss: 7.4431e-04 - accuracy: 0.9995
154/469 [========>.....................] - ETA: 7s - loss: 7.3987e-04 - accuracy: 0.9995
157/469 [=========>....................] - ETA: 7s - loss: 7.4009e-04 - accuracy: 0.9996
160/469 [=========>....................] - ETA: 7s - loss: 7.3701e-04 - accuracy: 0.9996
163/469 [=========>....................] - ETA: 7s - loss: 7.3292e-04 - accuracy: 0.9996
166/469 [=========>....................] - ETA: 7s - loss: 7.3133e-04 - accuracy: 0.9996
169/469 [=========>....................] - ETA: 7s - loss: 7.2527e-04 - accuracy: 0.9996
172/469 [==========>...................] - ETA: 7s - loss: 7.1815e-04 - accuracy: 0.9996
175/469 [==========>...................] - ETA: 7s - loss: 7.1240e-04 - accuracy: 0.9996
178/469 [==========>...................] - ETA: 6s - loss: 7.1161e-04 - accuracy: 0.9996
181/469 [==========>...................] - ETA: 6s - loss: 7.1724e-04 - accuracy: 0.9996
184/469 [==========>...................] - ETA: 6s - loss: 7.1445e-04 - accuracy: 0.9996
187/469 [==========>...................] - ETA: 6s - loss: 7.0764e-04 - accuracy: 0.9996
190/469 [===========>..................] - ETA: 6s - loss: 7.0042e-04 - accuracy: 0.9996
193/469 [===========>..................] - ETA: 6s - loss: 6.9467e-04 - accuracy: 0.9996
196/469 [===========>..................] - ETA: 6s - loss: 6.9161e-04 - accuracy: 0.9996
199/469 [===========>..................] - ETA: 6s - loss: 7.0068e-04 - accuracy: 0.9996
202/469 [===========>..................] - ETA: 6s - loss: 7.1611e-04 - accuracy: 0.9996
205/469 [============>.................] - ETA: 6s - loss: 7.0901e-04 - accuracy: 0.9996
208/469 [============>.................] - ETA: 6s - loss: 7.1087e-04 - accuracy: 0.9996
211/469 [============>.................] - ETA: 6s - loss: 7.0931e-04 - accuracy: 0.9996
214/469 [============>.................] - ETA: 6s - loss: 7.0129e-04 - accuracy: 0.9996
217/469 [============>.................] - ETA: 6s - loss: 6.9782e-04 - accuracy: 0.9996
220/469 [=============>................] - ETA: 5s - loss: 6.9725e-04 - accuracy: 0.9996
223/469 [=============>................] - ETA: 5s - loss: 6.9403e-04 - accuracy: 0.9996
226/469 [=============>................] - ETA: 5s - loss: 6.8840e-04 - accuracy: 0.9997
229/469 [=============>................] - ETA: 5s - loss: 6.8308e-04 - accuracy: 0.9997
232/469 [=============>................] - ETA: 5s - loss: 6.8203e-04 - accuracy: 0.9997
235/469 [==============>...............] - ETA: 5s - loss: 6.7705e-04 - accuracy: 0.9997
238/469 [==============>...............] - ETA: 5s - loss: 6.8381e-04 - accuracy: 0.9997
241/469 [==============>...............] - ETA: 5s - loss: 6.8252e-04 - accuracy: 0.9997
244/469 [==============>...............] - ETA: 5s - loss: 6.7851e-04 - accuracy: 0.9997
247/469 [==============>...............] - ETA: 5s - loss: 6.7901e-04 - accuracy: 0.9997
250/469 [==============>...............] - ETA: 5s - loss: 6.7271e-04 - accuracy: 0.9997
253/469 [===============>..............] - ETA: 5s - loss: 6.7183e-04 - accuracy: 0.9997
256/469 [===============>..............] - ETA: 5s - loss: 6.6994e-04 - accuracy: 0.9997
259/469 [===============>..............] - ETA: 5s - loss: 6.6569e-04 - accuracy: 0.9997
262/469 [===============>..............] - ETA: 4s - loss: 6.6217e-04 - accuracy: 0.9997
265/469 [===============>..............] - ETA: 4s - loss: 6.5736e-04 - accuracy: 0.9997
268/469 [================>.............] - ETA: 4s - loss: 6.5730e-04 - accuracy: 0.9997
271/469 [================>.............] - ETA: 4s - loss: 6.6590e-04 - accuracy: 0.9997
274/469 [================>.............] - ETA: 4s - loss: 6.6497e-04 - accuracy: 0.9997
277/469 [================>.............] - ETA: 4s - loss: 6.7694e-04 - accuracy: 0.9997
280/469 [================>.............] - ETA: 4s - loss: 6.7550e-04 - accuracy: 0.9997
283/469 [=================>............] - ETA: 4s - loss: 6.7817e-04 - accuracy: 0.9997
286/469 [=================>............] - ETA: 4s - loss: 6.7755e-04 - accuracy: 0.9997
289/469 [=================>............] - ETA: 4s - loss: 6.7371e-04 - accuracy: 0.9997
292/469 [=================>............] - ETA: 4s - loss: 6.8740e-04 - accuracy: 0.9997
295/469 [=================>............] - ETA: 4s - loss: 6.8856e-04 - accuracy: 0.9997
298/469 [==================>...........] - ETA: 4s - loss: 6.8703e-04 - accuracy: 0.9997
301/469 [==================>...........] - ETA: 4s - loss: 6.8644e-04 - accuracy: 0.9997
304/469 [==================>...........] - ETA: 3s - loss: 6.8240e-04 - accuracy: 0.9997
307/469 [==================>...........] - ETA: 3s - loss: 6.7838e-04 - accuracy: 0.9997
310/469 [==================>...........] - ETA: 3s - loss: 6.7786e-04 - accuracy: 0.9997
313/469 [===================>..........] - ETA: 3s - loss: 6.7424e-04 - accuracy: 0.9997
316/469 [===================>..........] - ETA: 3s - loss: 6.7570e-04 - accuracy: 0.9997
319/469 [===================>..........] - ETA: 3s - loss: 6.9537e-04 - accuracy: 0.9996
322/469 [===================>..........] - ETA: 3s - loss: 6.9433e-04 - accuracy: 0.9996
325/469 [===================>..........] - ETA: 3s - loss: 6.9140e-04 - accuracy: 0.9996
328/469 [===================>..........] - ETA: 3s - loss: 6.9032e-04 - accuracy: 0.9996
331/469 [====================>.........] - ETA: 3s - loss: 6.8706e-04 - accuracy: 0.9996
334/469 [====================>.........] - ETA: 3s - loss: 6.9282e-04 - accuracy: 0.9996
337/469 [====================>.........] - ETA: 3s - loss: 6.9719e-04 - accuracy: 0.9996
340/469 [====================>.........] - ETA: 3s - loss: 6.9496e-04 - accuracy: 0.9996
343/469 [====================>.........] - ETA: 3s - loss: 6.9013e-04 - accuracy: 0.9996
346/469 [=====================>........] - ETA: 2s - loss: 6.8751e-04 - accuracy: 0.9996
349/469 [=====================>........] - ETA: 2s - loss: 6.8533e-04 - accuracy: 0.9996
352/469 [=====================>........] - ETA: 2s - loss: 6.8610e-04 - accuracy: 0.9996
355/469 [=====================>........] - ETA: 2s - loss: 6.8825e-04 - accuracy: 0.9996
358/469 [=====================>........] - ETA: 2s - loss: 6.8450e-04 - accuracy: 0.9997
361/469 [======================>.......] - ETA: 2s - loss: 6.8704e-04 - accuracy: 0.9996
364/469 [======================>.......] - ETA: 2s - loss: 6.8300e-04 - accuracy: 0.9996
367/469 [======================>.......] - ETA: 2s - loss: 6.8070e-04 - accuracy: 0.9996
370/469 [======================>.......] - ETA: 2s - loss: 6.7874e-04 - accuracy: 0.9996
373/469 [======================>.......] - ETA: 2s - loss: 6.7667e-04 - accuracy: 0.9996
376/469 [=======================>......] - ETA: 2s - loss: 6.7708e-04 - accuracy: 0.9996
379/469 [=======================>......] - ETA: 2s - loss: 6.7319e-04 - accuracy: 0.9996
382/469 [=======================>......] - ETA: 2s - loss: 6.7604e-04 - accuracy: 0.9997
385/469 [=======================>......] - ETA: 2s - loss: 6.7386e-04 - accuracy: 0.9997
388/469 [=======================>......] - ETA: 1s - loss: 6.7674e-04 - accuracy: 0.9997
390/469 [=======================>......] - ETA: 1s - loss: 6.7638e-04 - accuracy: 0.9997
393/469 [========================>.....] - ETA: 1s - loss: 6.7582e-04 - accuracy: 0.9997
396/469 [========================>.....] - ETA: 1s - loss: 6.7381e-04 - accuracy: 0.9997
399/469 [========================>.....] - ETA: 1s - loss: 6.7396e-04 - accuracy: 0.9997
402/469 [========================>.....] - ETA: 1s - loss: 6.7314e-04 - accuracy: 0.9997
405/469 [========================>.....] - ETA: 1s - loss: 6.6874e-04 - accuracy: 0.9997
408/469 [=========================>....] - ETA: 1s - loss: 6.6504e-04 - accuracy: 0.9997
411/469 [=========================>....] - ETA: 1s - loss: 6.6658e-04 - accuracy: 0.9997
414/469 [=========================>....] - ETA: 1s - loss: 6.7606e-04 - accuracy: 0.9997
417/469 [=========================>....] - ETA: 1s - loss: 6.7163e-04 - accuracy: 0.9997
420/469 [=========================>....] - ETA: 1s - loss: 6.7364e-04 - accuracy: 0.9997
423/469 [==========================>...] - ETA: 1s - loss: 6.6944e-04 - accuracy: 0.9997
426/469 [==========================>...] - ETA: 1s - loss: 6.6823e-04 - accuracy: 0.9997
429/469 [==========================>...] - ETA: 0s - loss: 6.6402e-04 - accuracy: 0.9997
432/469 [==========================>...] - ETA: 0s - loss: 6.6309e-04 - accuracy: 0.9997
435/469 [==========================>...] - ETA: 0s - loss: 6.6369e-04 - accuracy: 0.9997
438/469 [===========================>..] - ETA: 0s - loss: 6.6868e-04 - accuracy: 0.9997
441/469 [===========================>..] - ETA: 0s - loss: 6.6582e-04 - accuracy: 0.9997
444/469 [===========================>..] - ETA: 0s - loss: 6.6443e-04 - accuracy: 0.9997
447/469 [===========================>..] - ETA: 0s - loss: 6.6282e-04 - accuracy: 0.9997
450/469 [===========================>..] - ETA: 0s - loss: 6.5937e-04 - accuracy: 0.9997
453/469 [===========================>..] - ETA: 0s - loss: 6.5706e-04 - accuracy: 0.9997
456/469 [============================>.] - ETA: 0s - loss: 6.6799e-04 - accuracy: 0.9997
459/469 [============================>.] - ETA: 0s - loss: 6.6934e-04 - accuracy: 0.9996
462/469 [============================>.] - ETA: 0s - loss: 6.7224e-04 - accuracy: 0.9996
465/469 [============================>.] - ETA: 0s - loss: 6.7869e-04 - accuracy: 0.9996
468/469 [============================>.] - ETA: 0s - loss: 6.7955e-04 - accuracy: 0.9996
469/469 [==============================] - 12s 25ms/step - loss: 6.7912e-04 - accuracy: 0.9996 - val_loss: 0.0052 - val_accuracy: 0.9933 - lr: 1.5849e-04
Test score: 0.005227956455200911
Test accuracy: 0.9933000206947327

Saving the model weights as mnistnet_act_fp_wgt_fp.hdf5 to reload them as init weights for the quantization step:

Note

This is not mandatory but helps with training speed.

temp_dir = TemporaryDirectory()
model_keras.save_weights(
    os.path.join(temp_dir.name, 'mnistnet_act_fp_wgt_fp.hdf5'))

4. Model quantization and training

4.1 Quantize the model

We can now turn to training a discretized version of the model, where the weights and activations are quantized so as to be suitable for implementation in the Akida NSoC.

For this, we just have to change very slightly the definition of the model used above, changing just the values of weight_quantization and activ_quantization used for the blocks (but still with no output nonlinearity for the final block). Additionally, we’ll initialise the model using the set of pre-trained weights that we just saved (not so important here, but for more complex datasets can make a huge difference both to the accuracy level ultimately achieved and to the speed of convergence).

Note that, for more challenging datasets, it may also be useful to make stepwise changes towards a fully quantized model - e.g. by first training with only the activations quantized, re-saving, and then adding the quantized weights. Additionally, the toolkit documentation describes how one can go further, optimizing the degree of sparsity in the model to reduce computational cost while maintaining accuracy. In this first example however, we’ll stick to a one step conversion (mainly because the MNIST dataset simply isn’t complex enough to see the benefit of the advanced techniques).

# Removes all the nodes left over from the previous model and free memory
K.clear_session()

img_input = Input(shape=(28, 28, 1))
x = conv_block(img_input,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_0',
               weight_quantization=2,
               activ_quantization=1,
               pooling='max',
               add_batchnorm=True)
x = conv_block(x,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_1',
               weight_quantization=2,
               activ_quantization=1,
               pooling='max',
               add_batchnorm=True)
x = conv_block(x,
               filters=32,
               kernel_size=(5, 5),
               padding='same',
               use_bias=False,
               name='conv_2',
               weight_quantization=2,
               activ_quantization=1,
               add_batchnorm=True)
x = Flatten()(x)
x = dense_block(x,
                units=512,
                use_bias=False,
                name='dense_2',
                weight_quantization=2,
                activ_quantization=1,
                add_batchnorm=True)
x = dense_block(x,
                units=10,
                use_bias=False,
                name='dense_3',
                weight_quantization=2,
                activ_quantization=None)

model_keras = Model(img_input, x, name='mnistnet_quantized')
lr_start = 1e-3
opt = Adam(lr=lr_start)
model_keras.compile(loss='squared_hinge', optimizer=opt, metrics=['accuracy'])
model_keras.summary()

# Reload previously computed weights as init weights for the quantization step
load_status = model_keras.load_weights(
    os.path.join(temp_dir.name, 'mnistnet_act_fp_wgt_fp.hdf5'))

Out:

Model: "mnistnet_quantized"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(None, 28, 28, 1)]       0
_________________________________________________________________
conv_0 (QuantizedConv2D)     (None, 28, 28, 32)        800
_________________________________________________________________
conv_0_maxpool (MaxPooling2D (None, 14, 14, 32)        0
_________________________________________________________________
conv_0_BN (BatchNormalizatio (None, 14, 14, 32)        128
_________________________________________________________________
conv_0_relu (ActivationDiscr (None, 14, 14, 32)        0
_________________________________________________________________
conv_1 (QuantizedConv2D)     (None, 14, 14, 32)        25600
_________________________________________________________________
conv_1_maxpool (MaxPooling2D (None, 7, 7, 32)          0
_________________________________________________________________
conv_1_BN (BatchNormalizatio (None, 7, 7, 32)          128
_________________________________________________________________
conv_1_relu (ActivationDiscr (None, 7, 7, 32)          0
_________________________________________________________________
conv_2 (QuantizedConv2D)     (None, 7, 7, 32)          25600
_________________________________________________________________
conv_2_BN (BatchNormalizatio (None, 7, 7, 32)          128
_________________________________________________________________
conv_2_relu (ActivationDiscr (None, 7, 7, 32)          0
_________________________________________________________________
flatten (Flatten)            (None, 1568)              0
_________________________________________________________________
dense_2 (QuantizedDense)     (None, 512)               802816
_________________________________________________________________
dense_2_BN (BatchNormalizati (None, 512)               2048
_________________________________________________________________
dense_2_relu (ActivationDisc (None, 512)               0
_________________________________________________________________
dense_3 (QuantizedDense)     (None, 10)                5120
=================================================================
Total params: 862,368
Trainable params: 861,152
Non-trainable params: 1,216
_________________________________________________________________

4.2 Performance check

Re-train and save the quantized model:

callbacks = []
lr_scheduler = LearningRateScheduler(lambda e: lr_start * lr_decay**e)
callbacks.append(lr_scheduler)
history = model_keras.fit(x_train,
                          y_train,
                          batch_size=batch_size,
                          epochs=epochs,
                          verbose=1,
                          validation_data=(x_test, y_test),
                          callbacks=callbacks)
score = model_keras.evaluate(x_test, y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Out:

Epoch 1/5

  1/469 [..............................] - ETA: 0s - loss: 0.1534 - accuracy: 0.8047
  3/469 [..............................] - ETA: 9s - loss: 0.1246 - accuracy: 0.8516
  5/469 [..............................] - ETA: 10s - loss: 0.1056 - accuracy: 0.8813
  7/469 [..............................] - ETA: 11s - loss: 0.0923 - accuracy: 0.8996
  9/469 [..............................] - ETA: 11s - loss: 0.0827 - accuracy: 0.9141
 11/469 [..............................] - ETA: 12s - loss: 0.0786 - accuracy: 0.9155
 13/469 [..............................] - ETA: 12s - loss: 0.0737 - accuracy: 0.9213
 15/469 [..............................] - ETA: 12s - loss: 0.0693 - accuracy: 0.9255
 17/469 [>.............................] - ETA: 12s - loss: 0.0658 - accuracy: 0.9292
 19/469 [>.............................] - ETA: 12s - loss: 0.0629 - accuracy: 0.9309
 21/469 [>.............................] - ETA: 12s - loss: 0.0611 - accuracy: 0.9312
 23/469 [>.............................] - ETA: 12s - loss: 0.0591 - accuracy: 0.9324
 25/469 [>.............................] - ETA: 12s - loss: 0.0571 - accuracy: 0.9350
 27/469 [>.............................] - ETA: 12s - loss: 0.0555 - accuracy: 0.9369
 29/469 [>.............................] - ETA: 12s - loss: 0.0541 - accuracy: 0.9386
 31/469 [>.............................] - ETA: 12s - loss: 0.0539 - accuracy: 0.9385
 33/469 [=>............................] - ETA: 12s - loss: 0.0526 - accuracy: 0.9396
 35/469 [=>............................] - ETA: 12s - loss: 0.0511 - accuracy: 0.9413
 37/469 [=>............................] - ETA: 12s - loss: 0.0497 - accuracy: 0.9436
 39/469 [=>............................] - ETA: 12s - loss: 0.0486 - accuracy: 0.9447
 41/469 [=>............................] - ETA: 12s - loss: 0.0473 - accuracy: 0.9463
 43/469 [=>............................] - ETA: 12s - loss: 0.0461 - accuracy: 0.9473
 45/469 [=>............................] - ETA: 12s - loss: 0.0447 - accuracy: 0.9493
 47/469 [==>...........................] - ETA: 11s - loss: 0.0441 - accuracy: 0.9501
 49/469 [==>...........................] - ETA: 11s - loss: 0.0434 - accuracy: 0.9512
 51/469 [==>...........................] - ETA: 11s - loss: 0.0423 - accuracy: 0.9522
 53/469 [==>...........................] - ETA: 11s - loss: 0.0416 - accuracy: 0.9528
 55/469 [==>...........................] - ETA: 11s - loss: 0.0406 - accuracy: 0.9540
 57/469 [==>...........................] - ETA: 11s - loss: 0.0396 - accuracy: 0.9553
 59/469 [==>...........................] - ETA: 11s - loss: 0.0389 - accuracy: 0.9560
 61/469 [==>...........................] - ETA: 11s - loss: 0.0387 - accuracy: 0.9563
 63/469 [===>..........................] - ETA: 11s - loss: 0.0383 - accuracy: 0.9565
 65/469 [===>..........................] - ETA: 11s - loss: 0.0382 - accuracy: 0.9567
 67/469 [===>..........................] - ETA: 11s - loss: 0.0378 - accuracy: 0.9571
 69/469 [===>..........................] - ETA: 11s - loss: 0.0374 - accuracy: 0.9573
 71/469 [===>..........................] - ETA: 11s - loss: 0.0369 - accuracy: 0.9575
 73/469 [===>..........................] - ETA: 11s - loss: 0.0366 - accuracy: 0.9580
 75/469 [===>..........................] - ETA: 11s - loss: 0.0360 - accuracy: 0.9586
 77/469 [===>..........................] - ETA: 11s - loss: 0.0354 - accuracy: 0.9592
 79/469 [====>.........................] - ETA: 11s - loss: 0.0349 - accuracy: 0.9598
 81/469 [====>.........................] - ETA: 11s - loss: 0.0346 - accuracy: 0.9601
 83/469 [====>.........................] - ETA: 11s - loss: 0.0344 - accuracy: 0.9606
 85/469 [====>.........................] - ETA: 11s - loss: 0.0343 - accuracy: 0.9608
 87/469 [====>.........................] - ETA: 10s - loss: 0.0342 - accuracy: 0.9610
 89/469 [====>.........................] - ETA: 10s - loss: 0.0338 - accuracy: 0.9615
 91/469 [====>.........................] - ETA: 10s - loss: 0.0333 - accuracy: 0.9621
 93/469 [====>.........................] - ETA: 10s - loss: 0.0331 - accuracy: 0.9624
 95/469 [=====>........................] - ETA: 10s - loss: 0.0329 - accuracy: 0.9627
 97/469 [=====>........................] - ETA: 10s - loss: 0.0328 - accuracy: 0.9627
 99/469 [=====>........................] - ETA: 10s - loss: 0.0325 - accuracy: 0.9631
101/469 [=====>........................] - ETA: 10s - loss: 0.0323 - accuracy: 0.9633
103/469 [=====>........................] - ETA: 10s - loss: 0.0322 - accuracy: 0.9636
105/469 [=====>........................] - ETA: 10s - loss: 0.0319 - accuracy: 0.9638
107/469 [=====>........................] - ETA: 10s - loss: 0.0316 - accuracy: 0.9643
109/469 [=====>........................] - ETA: 10s - loss: 0.0315 - accuracy: 0.9643
111/469 [======>.......................] - ETA: 10s - loss: 0.0313 - accuracy: 0.9645
113/469 [======>.......................] - ETA: 10s - loss: 0.0310 - accuracy: 0.9650
115/469 [======>.......................] - ETA: 10s - loss: 0.0308 - accuracy: 0.9652
117/469 [======>.......................] - ETA: 10s - loss: 0.0306 - accuracy: 0.9654
119/469 [======>.......................] - ETA: 10s - loss: 0.0305 - accuracy: 0.9656
121/469 [======>.......................] - ETA: 10s - loss: 0.0304 - accuracy: 0.9655
123/469 [======>.......................] - ETA: 9s - loss: 0.0301 - accuracy: 0.9658 
125/469 [======>.......................] - ETA: 9s - loss: 0.0299 - accuracy: 0.9659
127/469 [=======>......................] - ETA: 9s - loss: 0.0298 - accuracy: 0.9663
129/469 [=======>......................] - ETA: 9s - loss: 0.0295 - accuracy: 0.9664
131/469 [=======>......................] - ETA: 9s - loss: 0.0294 - accuracy: 0.9665
133/469 [=======>......................] - ETA: 9s - loss: 0.0292 - accuracy: 0.9668
135/469 [=======>......................] - ETA: 9s - loss: 0.0290 - accuracy: 0.9671
137/469 [=======>......................] - ETA: 9s - loss: 0.0288 - accuracy: 0.9673
139/469 [=======>......................] - ETA: 9s - loss: 0.0287 - accuracy: 0.9674
141/469 [========>.....................] - ETA: 9s - loss: 0.0286 - accuracy: 0.9676
143/469 [========>.....................] - ETA: 9s - loss: 0.0285 - accuracy: 0.9676
145/469 [========>.....................] - ETA: 9s - loss: 0.0285 - accuracy: 0.9677
147/469 [========>.....................] - ETA: 9s - loss: 0.0282 - accuracy: 0.9678
149/469 [========>.....................] - ETA: 9s - loss: 0.0281 - accuracy: 0.9679
151/469 [========>.....................] - ETA: 9s - loss: 0.0280 - accuracy: 0.9681
153/469 [========>.....................] - ETA: 9s - loss: 0.0279 - accuracy: 0.9680
155/469 [========>.....................] - ETA: 9s - loss: 0.0278 - accuracy: 0.9681
157/469 [=========>....................] - ETA: 9s - loss: 0.0276 - accuracy: 0.9685
159/469 [=========>....................] - ETA: 8s - loss: 0.0276 - accuracy: 0.9685
161/469 [=========>....................] - ETA: 8s - loss: 0.0275 - accuracy: 0.9686
163/469 [=========>....................] - ETA: 8s - loss: 0.0274 - accuracy: 0.9688
165/469 [=========>....................] - ETA: 8s - loss: 0.0273 - accuracy: 0.9690
167/469 [=========>....................] - ETA: 8s - loss: 0.0271 - accuracy: 0.9692
169/469 [=========>....................] - ETA: 8s - loss: 0.0269 - accuracy: 0.9693
171/469 [=========>....................] - ETA: 8s - loss: 0.0268 - accuracy: 0.9694
173/469 [==========>...................] - ETA: 8s - loss: 0.0267 - accuracy: 0.9696
175/469 [==========>...................] - ETA: 8s - loss: 0.0266 - accuracy: 0.9697
177/469 [==========>...................] - ETA: 8s - loss: 0.0266 - accuracy: 0.9698
179/469 [==========>...................] - ETA: 8s - loss: 0.0265 - accuracy: 0.9699
181/469 [==========>...................] - ETA: 8s - loss: 0.0265 - accuracy: 0.9699
183/469 [==========>...................] - ETA: 8s - loss: 0.0263 - accuracy: 0.9701
185/469 [==========>...................] - ETA: 8s - loss: 0.0262 - accuracy: 0.9702
187/469 [==========>...................] - ETA: 8s - loss: 0.0261 - accuracy: 0.9702
189/469 [===========>..................] - ETA: 8s - loss: 0.0260 - accuracy: 0.9703
191/469 [===========>..................] - ETA: 8s - loss: 0.0259 - accuracy: 0.9704
193/469 [===========>..................] - ETA: 7s - loss: 0.0258 - accuracy: 0.9705
195/469 [===========>..................] - ETA: 7s - loss: 0.0257 - accuracy: 0.9706
197/469 [===========>..................] - ETA: 7s - loss: 0.0257 - accuracy: 0.9706
199/469 [===========>..................] - ETA: 7s - loss: 0.0257 - accuracy: 0.9705
201/469 [===========>..................] - ETA: 7s - loss: 0.0256 - accuracy: 0.9707
203/469 [===========>..................] - ETA: 7s - loss: 0.0256 - accuracy: 0.9706
205/469 [============>.................] - ETA: 7s - loss: 0.0255 - accuracy: 0.9708
207/469 [============>.................] - ETA: 7s - loss: 0.0254 - accuracy: 0.9708
209/469 [============>.................] - ETA: 7s - loss: 0.0253 - accuracy: 0.9710
211/469 [============>.................] - ETA: 7s - loss: 0.0252 - accuracy: 0.9711
213/469 [============>.................] - ETA: 7s - loss: 0.0251 - accuracy: 0.9713
215/469 [============>.................] - ETA: 7s - loss: 0.0250 - accuracy: 0.9713
217/469 [============>.................] - ETA: 7s - loss: 0.0250 - accuracy: 0.9712
219/469 [=============>................] - ETA: 7s - loss: 0.0250 - accuracy: 0.9713
221/469 [=============>................] - ETA: 7s - loss: 0.0249 - accuracy: 0.9713
223/469 [=============>................] - ETA: 7s - loss: 0.0248 - accuracy: 0.9714
225/469 [=============>................] - ETA: 7s - loss: 0.0247 - accuracy: 0.9716
227/469 [=============>................] - ETA: 6s - loss: 0.0246 - accuracy: 0.9717
229/469 [=============>................] - ETA: 6s - loss: 0.0246 - accuracy: 0.9717
231/469 [=============>................] - ETA: 6s - loss: 0.0247 - accuracy: 0.9717
233/469 [=============>................] - ETA: 6s - loss: 0.0246 - accuracy: 0.9717
235/469 [==============>...............] - ETA: 6s - loss: 0.0245 - accuracy: 0.9718
237/469 [==============>...............] - ETA: 6s - loss: 0.0244 - accuracy: 0.9718
239/469 [==============>...............] - ETA: 6s - loss: 0.0243 - accuracy: 0.9719
241/469 [==============>...............] - ETA: 6s - loss: 0.0243 - accuracy: 0.9720
243/469 [==============>...............] - ETA: 6s - loss: 0.0243 - accuracy: 0.9719
245/469 [==============>...............] - ETA: 6s - loss: 0.0243 - accuracy: 0.9719
247/469 [==============>...............] - ETA: 6s - loss: 0.0242 - accuracy: 0.9720
249/469 [==============>...............] - ETA: 6s - loss: 0.0242 - accuracy: 0.9720
251/469 [===============>..............] - ETA: 6s - loss: 0.0240 - accuracy: 0.9721
253/469 [===============>..............] - ETA: 6s - loss: 0.0240 - accuracy: 0.9721
255/469 [===============>..............] - ETA: 6s - loss: 0.0240 - accuracy: 0.9721
257/469 [===============>..............] - ETA: 6s - loss: 0.0239 - accuracy: 0.9722
259/469 [===============>..............] - ETA: 6s - loss: 0.0238 - accuracy: 0.9723
261/469 [===============>..............] - ETA: 6s - loss: 0.0237 - accuracy: 0.9724
263/469 [===============>..............] - ETA: 5s - loss: 0.0238 - accuracy: 0.9723
265/469 [===============>..............] - ETA: 5s - loss: 0.0237 - accuracy: 0.9724
267/469 [================>.............] - ETA: 5s - loss: 0.0237 - accuracy: 0.9724
269/469 [================>.............] - ETA: 5s - loss: 0.0236 - accuracy: 0.9726
271/469 [================>.............] - ETA: 5s - loss: 0.0235 - accuracy: 0.9726
273/469 [================>.............] - ETA: 5s - loss: 0.0234 - accuracy: 0.9726
275/469 [================>.............] - ETA: 5s - loss: 0.0234 - accuracy: 0.9727
277/469 [================>.............] - ETA: 5s - loss: 0.0233 - accuracy: 0.9727
279/469 [================>.............] - ETA: 5s - loss: 0.0232 - accuracy: 0.9728
281/469 [================>.............] - ETA: 5s - loss: 0.0232 - accuracy: 0.9729
283/469 [=================>............] - ETA: 5s - loss: 0.0232 - accuracy: 0.9728
285/469 [=================>............] - ETA: 5s - loss: 0.0232 - accuracy: 0.9728
287/469 [=================>............] - ETA: 5s - loss: 0.0231 - accuracy: 0.9728
289/469 [=================>............] - ETA: 5s - loss: 0.0230 - accuracy: 0.9729
291/469 [=================>............] - ETA: 5s - loss: 0.0230 - accuracy: 0.9729
293/469 [=================>............] - ETA: 5s - loss: 0.0230 - accuracy: 0.9730
295/469 [=================>............] - ETA: 5s - loss: 0.0230 - accuracy: 0.9731
297/469 [=================>............] - ETA: 4s - loss: 0.0229 - accuracy: 0.9731
299/469 [==================>...........] - ETA: 4s - loss: 0.0228 - accuracy: 0.9732
301/469 [==================>...........] - ETA: 4s - loss: 0.0227 - accuracy: 0.9734
303/469 [==================>...........] - ETA: 4s - loss: 0.0226 - accuracy: 0.9735
305/469 [==================>...........] - ETA: 4s - loss: 0.0225 - accuracy: 0.9736
307/469 [==================>...........] - ETA: 4s - loss: 0.0224 - accuracy: 0.9737
309/469 [==================>...........] - ETA: 4s - loss: 0.0224 - accuracy: 0.9738
311/469 [==================>...........] - ETA: 4s - loss: 0.0223 - accuracy: 0.9739
313/469 [===================>..........] - ETA: 4s - loss: 0.0222 - accuracy: 0.9739
315/469 [===================>..........] - ETA: 4s - loss: 0.0222 - accuracy: 0.9740
317/469 [===================>..........] - ETA: 4s - loss: 0.0221 - accuracy: 0.9740
319/469 [===================>..........] - ETA: 4s - loss: 0.0220 - accuracy: 0.9741
321/469 [===================>..........] - ETA: 4s - loss: 0.0220 - accuracy: 0.9742
323/469 [===================>..........] - ETA: 4s - loss: 0.0219 - accuracy: 0.9742
325/469 [===================>..........] - ETA: 4s - loss: 0.0219 - accuracy: 0.9743
327/469 [===================>..........] - ETA: 4s - loss: 0.0218 - accuracy: 0.9744
329/469 [====================>.........] - ETA: 4s - loss: 0.0217 - accuracy: 0.9744
331/469 [====================>.........] - ETA: 3s - loss: 0.0217 - accuracy: 0.9745
333/469 [====================>.........] - ETA: 3s - loss: 0.0216 - accuracy: 0.9745
335/469 [====================>.........] - ETA: 3s - loss: 0.0216 - accuracy: 0.9745
337/469 [====================>.........] - ETA: 3s - loss: 0.0215 - accuracy: 0.9746
339/469 [====================>.........] - ETA: 3s - loss: 0.0215 - accuracy: 0.9746
341/469 [====================>.........] - ETA: 3s - loss: 0.0216 - accuracy: 0.9746
343/469 [====================>.........] - ETA: 3s - loss: 0.0215 - accuracy: 0.9747
345/469 [=====================>........] - ETA: 3s - loss: 0.0215 - accuracy: 0.9747
347/469 [=====================>........] - ETA: 3s - loss: 0.0214 - accuracy: 0.9748
349/469 [=====================>........] - ETA: 3s - loss: 0.0213 - accuracy: 0.9749
351/469 [=====================>........] - ETA: 3s - loss: 0.0213 - accuracy: 0.9748
353/469 [=====================>........] - ETA: 3s - loss: 0.0213 - accuracy: 0.9748
355/469 [=====================>........] - ETA: 3s - loss: 0.0213 - accuracy: 0.9748
357/469 [=====================>........] - ETA: 3s - loss: 0.0212 - accuracy: 0.9749
359/469 [=====================>........] - ETA: 3s - loss: 0.0212 - accuracy: 0.9749
361/469 [======================>.......] - ETA: 3s - loss: 0.0212 - accuracy: 0.9749
363/469 [======================>.......] - ETA: 3s - loss: 0.0211 - accuracy: 0.9750
365/469 [======================>.......] - ETA: 3s - loss: 0.0210 - accuracy: 0.9751
367/469 [======================>.......] - ETA: 2s - loss: 0.0211 - accuracy: 0.9751
369/469 [======================>.......] - ETA: 2s - loss: 0.0210 - accuracy: 0.9751
371/469 [======================>.......] - ETA: 2s - loss: 0.0210 - accuracy: 0.9752
373/469 [======================>.......] - ETA: 2s - loss: 0.0210 - accuracy: 0.9752
375/469 [======================>.......] - ETA: 2s - loss: 0.0209 - accuracy: 0.9752
377/469 [=======================>......] - ETA: 2s - loss: 0.0209 - accuracy: 0.9753
379/469 [=======================>......] - ETA: 2s - loss: 0.0209 - accuracy: 0.9753
381/469 [=======================>......] - ETA: 2s - loss: 0.0208 - accuracy: 0.9754
383/469 [=======================>......] - ETA: 2s - loss: 0.0207 - accuracy: 0.9754
385/469 [=======================>......] - ETA: 2s - loss: 0.0208 - accuracy: 0.9754
387/469 [=======================>......] - ETA: 2s - loss: 0.0207 - accuracy: 0.9755
389/469 [=======================>......] - ETA: 2s - loss: 0.0206 - accuracy: 0.9755
391/469 [========================>.....] - ETA: 2s - loss: 0.0206 - accuracy: 0.9756
393/469 [========================>.....] - ETA: 2s - loss: 0.0206 - accuracy: 0.9756
395/469 [========================>.....] - ETA: 2s - loss: 0.0205 - accuracy: 0.9757
397/469 [========================>.....] - ETA: 2s - loss: 0.0205 - accuracy: 0.9757
399/469 [========================>.....] - ETA: 2s - loss: 0.0204 - accuracy: 0.9758
401/469 [========================>.....] - ETA: 1s - loss: 0.0204 - accuracy: 0.9759
403/469 [========================>.....] - ETA: 1s - loss: 0.0204 - accuracy: 0.9759
405/469 [========================>.....] - ETA: 1s - loss: 0.0203 - accuracy: 0.9760
407/469 [=========================>....] - ETA: 1s - loss: 0.0203 - accuracy: 0.9760
409/469 [=========================>....] - ETA: 1s - loss: 0.0202 - accuracy: 0.9761
411/469 [=========================>....] - ETA: 1s - loss: 0.0202 - accuracy: 0.9762
413/469 [=========================>....] - ETA: 1s - loss: 0.0202 - accuracy: 0.9762
415/469 [=========================>....] - ETA: 1s - loss: 0.0201 - accuracy: 0.9762
417/469 [=========================>....] - ETA: 1s - loss: 0.0201 - accuracy: 0.9763
419/469 [=========================>....] - ETA: 1s - loss: 0.0201 - accuracy: 0.9763
421/469 [=========================>....] - ETA: 1s - loss: 0.0200 - accuracy: 0.9764
423/469 [==========================>...] - ETA: 1s - loss: 0.0200 - accuracy: 0.9765
425/469 [==========================>...] - ETA: 1s - loss: 0.0199 - accuracy: 0.9765
427/469 [==========================>...] - ETA: 1s - loss: 0.0199 - accuracy: 0.9766
429/469 [==========================>...] - ETA: 1s - loss: 0.0199 - accuracy: 0.9766
431/469 [==========================>...] - ETA: 1s - loss: 0.0199 - accuracy: 0.9766
433/469 [==========================>...] - ETA: 1s - loss: 0.0198 - accuracy: 0.9766
435/469 [==========================>...] - ETA: 0s - loss: 0.0198 - accuracy: 0.9766
437/469 [==========================>...] - ETA: 0s - loss: 0.0198 - accuracy: 0.9767
439/469 [===========================>..] - ETA: 0s - loss: 0.0197 - accuracy: 0.9768
441/469 [===========================>..] - ETA: 0s - loss: 0.0196 - accuracy: 0.9768
443/469 [===========================>..] - ETA: 0s - loss: 0.0196 - accuracy: 0.9768
445/469 [===========================>..] - ETA: 0s - loss: 0.0196 - accuracy: 0.9769
447/469 [===========================>..] - ETA: 0s - loss: 0.0195 - accuracy: 0.9769
449/469 [===========================>..] - ETA: 0s - loss: 0.0195 - accuracy: 0.9769
451/469 [===========================>..] - ETA: 0s - loss: 0.0195 - accuracy: 0.9770
453/469 [===========================>..] - ETA: 0s - loss: 0.0194 - accuracy: 0.9770
455/469 [============================>.] - ETA: 0s - loss: 0.0195 - accuracy: 0.9770
457/469 [============================>.] - ETA: 0s - loss: 0.0194 - accuracy: 0.9770
459/469 [============================>.] - ETA: 0s - loss: 0.0194 - accuracy: 0.9770
461/469 [============================>.] - ETA: 0s - loss: 0.0194 - accuracy: 0.9770
463/469 [============================>.] - ETA: 0s - loss: 0.0194 - accuracy: 0.9770
465/469 [============================>.] - ETA: 0s - loss: 0.0194 - accuracy: 0.9770
467/469 [============================>.] - ETA: 0s - loss: 0.0194 - accuracy: 0.9770
469/469 [==============================] - ETA: 0s - loss: 0.0193 - accuracy: 0.9770
469/469 [==============================] - 14s 31ms/step - loss: 0.0193 - accuracy: 0.9770 - val_loss: 0.0122 - val_accuracy: 0.9851 - lr: 0.0010
Epoch 2/5

  1/469 [..............................] - ETA: 0s - loss: 0.0160 - accuracy: 0.9766
  3/469 [..............................] - ETA: 8s - loss: 0.0106 - accuracy: 0.9818
  5/469 [..............................] - ETA: 10s - loss: 0.0093 - accuracy: 0.9875
  7/469 [..............................] - ETA: 11s - loss: 0.0106 - accuracy: 0.9844
  9/469 [..............................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9852
 11/469 [..............................] - ETA: 12s - loss: 0.0106 - accuracy: 0.9858
 13/469 [..............................] - ETA: 12s - loss: 0.0109 - accuracy: 0.9862
 15/469 [..............................] - ETA: 12s - loss: 0.0109 - accuracy: 0.9865
 17/469 [>.............................] - ETA: 12s - loss: 0.0108 - accuracy: 0.9862
 19/469 [>.............................] - ETA: 12s - loss: 0.0109 - accuracy: 0.9856
 21/469 [>.............................] - ETA: 12s - loss: 0.0104 - accuracy: 0.9866
 23/469 [>.............................] - ETA: 12s - loss: 0.0103 - accuracy: 0.9871
 25/469 [>.............................] - ETA: 12s - loss: 0.0110 - accuracy: 0.9859
 27/469 [>.............................] - ETA: 12s - loss: 0.0107 - accuracy: 0.9861
 29/469 [>.............................] - ETA: 12s - loss: 0.0102 - accuracy: 0.9868
 31/469 [>.............................] - ETA: 12s - loss: 0.0104 - accuracy: 0.9866
 33/469 [=>............................] - ETA: 12s - loss: 0.0102 - accuracy: 0.9867
 35/469 [=>............................] - ETA: 12s - loss: 0.0102 - accuracy: 0.9864
 37/469 [=>............................] - ETA: 12s - loss: 0.0103 - accuracy: 0.9865
 39/469 [=>............................] - ETA: 12s - loss: 0.0105 - accuracy: 0.9864
 41/469 [=>............................] - ETA: 12s - loss: 0.0103 - accuracy: 0.9867
 43/469 [=>............................] - ETA: 12s - loss: 0.0103 - accuracy: 0.9867
 45/469 [=>............................] - ETA: 12s - loss: 0.0104 - accuracy: 0.9866
 47/469 [==>...........................] - ETA: 12s - loss: 0.0102 - accuracy: 0.9869
 49/469 [==>...........................] - ETA: 11s - loss: 0.0103 - accuracy: 0.9866
 51/469 [==>...........................] - ETA: 11s - loss: 0.0101 - accuracy: 0.9870
 53/469 [==>...........................] - ETA: 11s - loss: 0.0103 - accuracy: 0.9867
 55/469 [==>...........................] - ETA: 11s - loss: 0.0102 - accuracy: 0.9869
 57/469 [==>...........................] - ETA: 11s - loss: 0.0102 - accuracy: 0.9871
 59/469 [==>...........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9866
 61/469 [==>...........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9867
 63/469 [===>..........................] - ETA: 11s - loss: 0.0107 - accuracy: 0.9867
 65/469 [===>..........................] - ETA: 11s - loss: 0.0108 - accuracy: 0.9865
 67/469 [===>..........................] - ETA: 11s - loss: 0.0107 - accuracy: 0.9865
 69/469 [===>..........................] - ETA: 11s - loss: 0.0106 - accuracy: 0.9866
 71/469 [===>..........................] - ETA: 11s - loss: 0.0106 - accuracy: 0.9868
 73/469 [===>..........................] - ETA: 11s - loss: 0.0105 - accuracy: 0.9869
 75/469 [===>..........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9870
 77/469 [===>..........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9870
 79/469 [====>.........................] - ETA: 11s - loss: 0.0103 - accuracy: 0.9871
 81/469 [====>.........................] - ETA: 11s - loss: 0.0103 - accuracy: 0.9873
 83/469 [====>.........................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9873
 85/469 [====>.........................] - ETA: 11s - loss: 0.0107 - accuracy: 0.9872
 87/469 [====>.........................] - ETA: 10s - loss: 0.0105 - accuracy: 0.9873
 89/469 [====>.........................] - ETA: 10s - loss: 0.0105 - accuracy: 0.9874
 91/469 [====>.........................] - ETA: 10s - loss: 0.0104 - accuracy: 0.9876
 93/469 [====>.........................] - ETA: 10s - loss: 0.0104 - accuracy: 0.9876
 95/469 [=====>........................] - ETA: 10s - loss: 0.0104 - accuracy: 0.9876
 97/469 [=====>........................] - ETA: 10s - loss: 0.0105 - accuracy: 0.9875
 99/469 [=====>........................] - ETA: 10s - loss: 0.0105 - accuracy: 0.9876
101/469 [=====>........................] - ETA: 10s - loss: 0.0106 - accuracy: 0.9875
103/469 [=====>........................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9873
105/469 [=====>........................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9874
107/469 [=====>........................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9874
109/469 [=====>........................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9874
111/469 [======>.......................] - ETA: 10s - loss: 0.0106 - accuracy: 0.9875
113/469 [======>.......................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9873
115/469 [======>.......................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9871
117/469 [======>.......................] - ETA: 10s - loss: 0.0107 - accuracy: 0.9872
119/469 [======>.......................] - ETA: 10s - loss: 0.0106 - accuracy: 0.9873
121/469 [======>.......................] - ETA: 10s - loss: 0.0105 - accuracy: 0.9875
123/469 [======>.......................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9877 
125/469 [======>.......................] - ETA: 9s - loss: 0.0104 - accuracy: 0.9876
127/469 [=======>......................] - ETA: 9s - loss: 0.0104 - accuracy: 0.9876
129/469 [=======>......................] - ETA: 9s - loss: 0.0103 - accuracy: 0.9877
131/469 [=======>......................] - ETA: 9s - loss: 0.0103 - accuracy: 0.9877
133/469 [=======>......................] - ETA: 9s - loss: 0.0102 - accuracy: 0.9879
135/469 [=======>......................] - ETA: 9s - loss: 0.0103 - accuracy: 0.9878
137/469 [=======>......................] - ETA: 9s - loss: 0.0102 - accuracy: 0.9879
139/469 [=======>......................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9877
141/469 [========>.....................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9876
143/469 [========>.....................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9875
145/469 [========>.....................] - ETA: 9s - loss: 0.0106 - accuracy: 0.9874
147/469 [========>.....................] - ETA: 9s - loss: 0.0106 - accuracy: 0.9876
149/469 [========>.....................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9876
151/469 [========>.....................] - ETA: 9s - loss: 0.0105 - accuracy: 0.9876
153/469 [========>.....................] - ETA: 9s - loss: 0.0106 - accuracy: 0.9876
155/469 [========>.....................] - ETA: 9s - loss: 0.0106 - accuracy: 0.9876
157/469 [=========>....................] - ETA: 8s - loss: 0.0107 - accuracy: 0.9873
159/469 [=========>....................] - ETA: 8s - loss: 0.0106 - accuracy: 0.9874
161/469 [=========>....................] - ETA: 8s - loss: 0.0107 - accuracy: 0.9874
163/469 [=========>....................] - ETA: 8s - loss: 0.0107 - accuracy: 0.9873
165/469 [=========>....................] - ETA: 8s - loss: 0.0107 - accuracy: 0.9875
167/469 [=========>....................] - ETA: 8s - loss: 0.0106 - accuracy: 0.9876
169/469 [=========>....................] - ETA: 8s - loss: 0.0106 - accuracy: 0.9876
171/469 [=========>....................] - ETA: 8s - loss: 0.0105 - accuracy: 0.9876
173/469 [==========>...................] - ETA: 8s - loss: 0.0106 - accuracy: 0.9875
175/469 [==========>...................] - ETA: 8s - loss: 0.0105 - accuracy: 0.9876
177/469 [==========>...................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9877
179/469 [==========>...................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9876
181/469 [==========>...................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9877
183/469 [==========>...................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9877
185/469 [==========>...................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9878
187/469 [==========>...................] - ETA: 8s - loss: 0.0103 - accuracy: 0.9878
189/469 [===========>..................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9878
191/469 [===========>..................] - ETA: 8s - loss: 0.0104 - accuracy: 0.9879
193/469 [===========>..................] - ETA: 7s - loss: 0.0104 - accuracy: 0.9878
195/469 [===========>..................] - ETA: 7s - loss: 0.0104 - accuracy: 0.9877
197/469 [===========>..................] - ETA: 7s - loss: 0.0103 - accuracy: 0.9879
199/469 [===========>..................] - ETA: 7s - loss: 0.0103 - accuracy: 0.9879
201/469 [===========>..................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9880
203/469 [===========>..................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9880
205/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9880
207/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9879
209/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9880
211/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9879
213/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9880
215/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9880
217/469 [============>.................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9880
219/469 [=============>................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9879
221/469 [=============>................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9879
223/469 [=============>................] - ETA: 7s - loss: 0.0102 - accuracy: 0.9878
225/469 [=============>................] - ETA: 7s - loss: 0.0101 - accuracy: 0.9879
227/469 [=============>................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9879
229/469 [=============>................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9879
231/469 [=============>................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9879
233/469 [=============>................] - ETA: 6s - loss: 0.0101 - accuracy: 0.9879
235/469 [==============>...............] - ETA: 6s - loss: 0.0101 - accuracy: 0.9879
237/469 [==============>...............] - ETA: 6s - loss: 0.0100 - accuracy: 0.9880
239/469 [==============>...............] - ETA: 6s - loss: 0.0101 - accuracy: 0.9879
241/469 [==============>...............] - ETA: 6s - loss: 0.0100 - accuracy: 0.9880
243/469 [==============>...............] - ETA: 6s - loss: 0.0100 - accuracy: 0.9880
245/469 [==============>...............] - ETA: 6s - loss: 0.0099 - accuracy: 0.9881
247/469 [==============>...............] - ETA: 6s - loss: 0.0099 - accuracy: 0.9881
249/469 [==============>...............] - ETA: 6s - loss: 0.0100 - accuracy: 0.9880
251/469 [===============>..............] - ETA: 6s - loss: 0.0100 - accuracy: 0.9881
253/469 [===============>..............] - ETA: 6s - loss: 0.0099 - accuracy: 0.9882
255/469 [===============>..............] - ETA: 6s - loss: 0.0099 - accuracy: 0.9882
257/469 [===============>..............] - ETA: 6s - loss: 0.0098 - accuracy: 0.9883
259/469 [===============>..............] - ETA: 6s - loss: 0.0099 - accuracy: 0.9883
261/469 [===============>..............] - ETA: 6s - loss: 0.0098 - accuracy: 0.9884
263/469 [===============>..............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
265/469 [===============>..............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
267/469 [================>.............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
269/469 [================>.............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
271/469 [================>.............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
273/469 [================>.............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9883
275/469 [================>.............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
277/469 [================>.............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9885
279/469 [================>.............] - ETA: 5s - loss: 0.0097 - accuracy: 0.9885
281/469 [================>.............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9885
283/469 [=================>............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
285/469 [=================>............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
287/469 [=================>............] - ETA: 5s - loss: 0.0099 - accuracy: 0.9884
289/469 [=================>............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
291/469 [=================>............] - ETA: 5s - loss: 0.0099 - accuracy: 0.9884
293/469 [=================>............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
295/469 [=================>............] - ETA: 5s - loss: 0.0098 - accuracy: 0.9884
297/469 [=================>............] - ETA: 4s - loss: 0.0098 - accuracy: 0.9884
299/469 [==================>...........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9884
301/469 [==================>...........] - ETA: 4s - loss: 0.0099 - accuracy: 0.9883
303/469 [==================>...........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9884
305/469 [==================>...........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9884
307/469 [==================>...........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9885
309/469 [==================>...........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9885
311/469 [==================>...........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9885
313/469 [===================>..........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9885
315/469 [===================>..........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9885
317/469 [===================>..........] - ETA: 4s - loss: 0.0098 - accuracy: 0.9885
319/469 [===================>..........] - ETA: 4s - loss: 0.0097 - accuracy: 0.9886
321/469 [===================>..........] - ETA: 4s - loss: 0.0097 - accuracy: 0.9886
323/469 [===================>..........] - ETA: 4s - loss: 0.0097 - accuracy: 0.9886
325/469 [===================>..........] - ETA: 4s - loss: 0.0097 - accuracy: 0.9886
327/469 [===================>..........] - ETA: 4s - loss: 0.0097 - accuracy: 0.9886
329/469 [====================>.........] - ETA: 4s - loss: 0.0097 - accuracy: 0.9886
331/469 [====================>.........] - ETA: 3s - loss: 0.0097 - accuracy: 0.9886
333/469 [====================>.........] - ETA: 3s - loss: 0.0097 - accuracy: 0.9886
335/469 [====================>.........] - ETA: 3s - loss: 0.0097 - accuracy: 0.9886
337/469 [====================>.........] - ETA: 3s - loss: 0.0097 - accuracy: 0.9886
339/469 [====================>.........] - ETA: 3s - loss: 0.0097 - accuracy: 0.9887
341/469 [====================>.........] - ETA: 3s - loss: 0.0096 - accuracy: 0.9887
343/469 [====================>.........] - ETA: 3s - loss: 0.0096 - accuracy: 0.9887
345/469 [=====================>........] - ETA: 3s - loss: 0.0096 - accuracy: 0.9887
347/469 [=====================>........] - ETA: 3s - loss: 0.0096 - accuracy: 0.9887
349/469 [=====================>........] - ETA: 3s - loss: 0.0096 - accuracy: 0.9887
351/469 [=====================>........] - ETA: 3s - loss: 0.0096 - accuracy: 0.9888
353/469 [=====================>........] - ETA: 3s - loss: 0.0095 - accuracy: 0.9888
355/469 [=====================>........] - ETA: 3s - loss: 0.0096 - accuracy: 0.9888
357/469 [=====================>........] - ETA: 3s - loss: 0.0095 - accuracy: 0.9888
359/469 [=====================>........] - ETA: 3s - loss: 0.0095 - accuracy: 0.9889
361/469 [======================>.......] - ETA: 3s - loss: 0.0095 - accuracy: 0.9889
363/469 [======================>.......] - ETA: 3s - loss: 0.0095 - accuracy: 0.9889
365/469 [======================>.......] - ETA: 3s - loss: 0.0095 - accuracy: 0.9889
367/469 [======================>.......] - ETA: 2s - loss: 0.0095 - accuracy: 0.9888
369/469 [======================>.......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9889
371/469 [======================>.......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9888
373/469 [======================>.......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9888
375/469 [======================>.......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9888
377/469 [=======================>......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9888
379/469 [=======================>......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9888
381/469 [=======================>......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9887
383/469 [=======================>......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9887
385/469 [=======================>......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9887
387/469 [=======================>......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9887
389/469 [=======================>......] - ETA: 2s - loss: 0.0096 - accuracy: 0.9887
391/469 [========================>.....] - ETA: 2s - loss: 0.0097 - accuracy: 0.9886
393/469 [========================>.....] - ETA: 2s - loss: 0.0097 - accuracy: 0.9885
395/469 [========================>.....] - ETA: 2s - loss: 0.0096 - accuracy: 0.9886
397/469 [========================>.....] - ETA: 2s - loss: 0.0096 - accuracy: 0.9886
399/469 [========================>.....] - ETA: 2s - loss: 0.0097 - accuracy: 0.9886
401/469 [========================>.....] - ETA: 1s - loss: 0.0097 - accuracy: 0.9886
403/469 [========================>.....] - ETA: 1s - loss: 0.0097 - accuracy: 0.9886
405/469 [========================>.....] - ETA: 1s - loss: 0.0097 - accuracy: 0.9886
407/469 [=========================>....] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
409/469 [=========================>....] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
411/469 [=========================>....] - ETA: 1s - loss: 0.0096 - accuracy: 0.9887
413/469 [=========================>....] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
415/469 [=========================>....] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
417/469 [=========================>....] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
419/469 [=========================>....] - ETA: 1s - loss: 0.0096 - accuracy: 0.9887
421/469 [=========================>....] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
423/469 [==========================>...] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
425/469 [==========================>...] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
427/469 [==========================>...] - ETA: 1s - loss: 0.0096 - accuracy: 0.9885
429/469 [==========================>...] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
431/469 [==========================>...] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
433/469 [==========================>...] - ETA: 1s - loss: 0.0096 - accuracy: 0.9886
435/469 [==========================>...] - ETA: 0s - loss: 0.0096 - accuracy: 0.9886
437/469 [==========================>...] - ETA: 0s - loss: 0.0096 - accuracy: 0.9886
439/469 [===========================>..] - ETA: 0s - loss: 0.0097 - accuracy: 0.9885
441/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9885
443/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9885
445/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9885
447/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9885
449/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9885
451/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9885
453/469 [===========================>..] - ETA: 0s - loss: 0.0096 - accuracy: 0.9885
455/469 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9885
457/469 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9886
459/469 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9886
461/469 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9886
463/469 [============================>.] - ETA: 0s - loss: 0.0096 - accuracy: 0.9886
465/469 [============================>.] - ETA: 0s - loss: 0.0095 - accuracy: 0.9886
467/469 [============================>.] - ETA: 0s - loss: 0.0095 - accuracy: 0.9886
469/469 [==============================] - ETA: 0s - loss: 0.0095 - accuracy: 0.9886
469/469 [==============================] - 14s 30ms/step - loss: 0.0095 - accuracy: 0.9886 - val_loss: 0.0098 - val_accuracy: 0.9879 - lr: 6.3096e-04
Epoch 3/5

  1/469 [..............................] - ETA: 0s - loss: 0.0115 - accuracy: 0.9922
  3/469 [..............................] - ETA: 8s - loss: 0.0100 - accuracy: 0.9870
  5/469 [..............................] - ETA: 10s - loss: 0.0122 - accuracy: 0.9828
  7/469 [..............................] - ETA: 11s - loss: 0.0104 - accuracy: 0.9855
  9/469 [..............................] - ETA: 11s - loss: 0.0095 - accuracy: 0.9870
 11/469 [..............................] - ETA: 12s - loss: 0.0084 - accuracy: 0.9886
 13/469 [..............................] - ETA: 12s - loss: 0.0081 - accuracy: 0.9898
 15/469 [..............................] - ETA: 12s - loss: 0.0083 - accuracy: 0.9896
 17/469 [>.............................] - ETA: 12s - loss: 0.0085 - accuracy: 0.9899
 19/469 [>.............................] - ETA: 12s - loss: 0.0081 - accuracy: 0.9905
 21/469 [>.............................] - ETA: 12s - loss: 0.0083 - accuracy: 0.9907
 23/469 [>.............................] - ETA: 12s - loss: 0.0086 - accuracy: 0.9905
 25/469 [>.............................] - ETA: 12s - loss: 0.0081 - accuracy: 0.9909
 27/469 [>.............................] - ETA: 12s - loss: 0.0081 - accuracy: 0.9905
 29/469 [>.............................] - ETA: 12s - loss: 0.0082 - accuracy: 0.9900
 31/469 [>.............................] - ETA: 12s - loss: 0.0079 - accuracy: 0.9907
 33/469 [=>............................] - ETA: 12s - loss: 0.0079 - accuracy: 0.9905
 35/469 [=>............................] - ETA: 12s - loss: 0.0080 - accuracy: 0.9902
 37/469 [=>............................] - ETA: 12s - loss: 0.0082 - accuracy: 0.9894
 39/469 [=>............................] - ETA: 12s - loss: 0.0082 - accuracy: 0.9896
 41/469 [=>............................] - ETA: 12s - loss: 0.0081 - accuracy: 0.9899
 43/469 [=>............................] - ETA: 12s - loss: 0.0081 - accuracy: 0.9902
 45/469 [=>............................] - ETA: 12s - loss: 0.0081 - accuracy: 0.9903
 47/469 [==>...........................] - ETA: 11s - loss: 0.0084 - accuracy: 0.9900
 49/469 [==>...........................] - ETA: 11s - loss: 0.0083 - accuracy: 0.9903
 51/469 [==>...........................] - ETA: 11s - loss: 0.0082 - accuracy: 0.9905
 53/469 [==>...........................] - ETA: 11s - loss: 0.0081 - accuracy: 0.9904
 55/469 [==>...........................] - ETA: 11s - loss: 0.0083 - accuracy: 0.9903
 57/469 [==>...........................] - ETA: 11s - loss: 0.0082 - accuracy: 0.9903
 59/469 [==>...........................] - ETA: 11s - loss: 0.0081 - accuracy: 0.9902
 61/469 [==>...........................] - ETA: 11s - loss: 0.0079 - accuracy: 0.9905
 63/469 [===>..........................] - ETA: 11s - loss: 0.0079 - accuracy: 0.9905
 65/469 [===>..........................] - ETA: 11s - loss: 0.0078 - accuracy: 0.9906
 67/469 [===>..........................] - ETA: 11s - loss: 0.0077 - accuracy: 0.9909
 69/469 [===>..........................] - ETA: 11s - loss: 0.0077 - accuracy: 0.9907
 71/469 [===>..........................] - ETA: 11s - loss: 0.0077 - accuracy: 0.9906
 73/469 [===>..........................] - ETA: 11s - loss: 0.0079 - accuracy: 0.9903
 75/469 [===>..........................] - ETA: 11s - loss: 0.0078 - accuracy: 0.9903
 77/469 [===>..........................] - ETA: 11s - loss: 0.0078 - accuracy: 0.9904
 79/469 [====>.........................] - ETA: 11s - loss: 0.0077 - accuracy: 0.9905
 81/469 [====>.........................] - ETA: 11s - loss: 0.0076 - accuracy: 0.9905
 83/469 [====>.........................] - ETA: 11s - loss: 0.0076 - accuracy: 0.9906
 85/469 [====>.........................] - ETA: 11s - loss: 0.0076 - accuracy: 0.9906
 87/469 [====>.........................] - ETA: 10s - loss: 0.0075 - accuracy: 0.9908
 89/469 [====>.........................] - ETA: 10s - loss: 0.0076 - accuracy: 0.9906
 91/469 [====>.........................] - ETA: 10s - loss: 0.0076 - accuracy: 0.9908
 93/469 [====>.........................] - ETA: 10s - loss: 0.0076 - accuracy: 0.9908
 95/469 [=====>........................] - ETA: 10s - loss: 0.0076 - accuracy: 0.9909
 97/469 [=====>........................] - ETA: 10s - loss: 0.0077 - accuracy: 0.9907
 99/469 [=====>........................] - ETA: 10s - loss: 0.0076 - accuracy: 0.9908
101/469 [=====>........................] - ETA: 10s - loss: 0.0075 - accuracy: 0.9909
103/469 [=====>........................] - ETA: 10s - loss: 0.0075 - accuracy: 0.9910
105/469 [=====>........................] - ETA: 10s - loss: 0.0074 - accuracy: 0.9911
107/469 [=====>........................] - ETA: 10s - loss: 0.0074 - accuracy: 0.9912
109/469 [=====>........................] - ETA: 10s - loss: 0.0075 - accuracy: 0.9911
111/469 [======>.......................] - ETA: 10s - loss: 0.0076 - accuracy: 0.9911
113/469 [======>.......................] - ETA: 10s - loss: 0.0075 - accuracy: 0.9912
115/469 [======>.......................] - ETA: 10s - loss: 0.0075 - accuracy: 0.9912
117/469 [======>.......................] - ETA: 10s - loss: 0.0075 - accuracy: 0.9912
119/469 [======>.......................] - ETA: 10s - loss: 0.0075 - accuracy: 0.9911
121/469 [======>.......................] - ETA: 10s - loss: 0.0076 - accuracy: 0.9910
123/469 [======>.......................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9910 
125/469 [======>.......................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9909
127/469 [=======>......................] - ETA: 9s - loss: 0.0077 - accuracy: 0.9909
129/469 [=======>......................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9910
131/469 [=======>......................] - ETA: 9s - loss: 0.0075 - accuracy: 0.9911
133/469 [=======>......................] - ETA: 9s - loss: 0.0075 - accuracy: 0.9911
135/469 [=======>......................] - ETA: 9s - loss: 0.0075 - accuracy: 0.9912
137/469 [=======>......................] - ETA: 9s - loss: 0.0075 - accuracy: 0.9912
139/469 [=======>......................] - ETA: 9s - loss: 0.0075 - accuracy: 0.9912
141/469 [========>.....................] - ETA: 9s - loss: 0.0074 - accuracy: 0.9913
143/469 [========>.....................] - ETA: 9s - loss: 0.0075 - accuracy: 0.9913
145/469 [========>.....................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9912
147/469 [========>.....................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9911
149/469 [========>.....................] - ETA: 9s - loss: 0.0077 - accuracy: 0.9910
151/469 [========>.....................] - ETA: 9s - loss: 0.0077 - accuracy: 0.9910
153/469 [========>.....................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9911
155/469 [========>.....................] - ETA: 9s - loss: 0.0076 - accuracy: 0.9911
157/469 [=========>....................] - ETA: 8s - loss: 0.0075 - accuracy: 0.9912
159/469 [=========>....................] - ETA: 8s - loss: 0.0075 - accuracy: 0.9912
161/469 [=========>....................] - ETA: 8s - loss: 0.0075 - accuracy: 0.9912
163/469 [=========>....................] - ETA: 8s - loss: 0.0075 - accuracy: 0.9912
165/469 [=========>....................] - ETA: 8s - loss: 0.0075 - accuracy: 0.9912
167/469 [=========>....................] - ETA: 8s - loss: 0.0074 - accuracy: 0.9913
169/469 [=========>....................] - ETA: 8s - loss: 0.0074 - accuracy: 0.9912
171/469 [=========>....................] - ETA: 8s - loss: 0.0074 - accuracy: 0.9912
173/469 [==========>...................] - ETA: 8s - loss: 0.0074 - accuracy: 0.9912
175/469 [==========>...................] - ETA: 8s - loss: 0.0074 - accuracy: 0.9912
177/469 [==========>...................] - ETA: 8s - loss: 0.0073 - accuracy: 0.9913
179/469 [==========>...................] - ETA: 8s - loss: 0.0073 - accuracy: 0.9914
181/469 [==========>...................] - ETA: 8s - loss: 0.0073 - accuracy: 0.9915
183/469 [==========>...................] - ETA: 8s - loss: 0.0072 - accuracy: 0.9915
185/469 [==========>...................] - ETA: 8s - loss: 0.0073 - accuracy: 0.9916
187/469 [==========>...................] - ETA: 8s - loss: 0.0073 - accuracy: 0.9915
189/469 [===========>..................] - ETA: 8s - loss: 0.0073 - accuracy: 0.9915
191/469 [===========>..................] - ETA: 8s - loss: 0.0072 - accuracy: 0.9916
193/469 [===========>..................] - ETA: 7s - loss: 0.0072 - accuracy: 0.9917
195/469 [===========>..................] - ETA: 7s - loss: 0.0073 - accuracy: 0.9915
197/469 [===========>..................] - ETA: 7s - loss: 0.0073 - accuracy: 0.9915
199/469 [===========>..................] - ETA: 7s - loss: 0.0073 - accuracy: 0.9915
201/469 [===========>..................] - ETA: 7s - loss: 0.0073 - accuracy: 0.9914
203/469 [===========>..................] - ETA: 7s - loss: 0.0074 - accuracy: 0.9914
205/469 [============>.................] - ETA: 7s - loss: 0.0074 - accuracy: 0.9913
207/469 [============>.................] - ETA: 7s - loss: 0.0074 - accuracy: 0.9913
209/469 [============>.................] - ETA: 7s - loss: 0.0074 - accuracy: 0.9912
211/469 [============>.................] - ETA: 7s - loss: 0.0074 - accuracy: 0.9912
213/469 [============>.................] - ETA: 7s - loss: 0.0074 - accuracy: 0.9913
215/469 [============>.................] - ETA: 7s - loss: 0.0074 - accuracy: 0.9913
217/469 [============>.................] - ETA: 7s - loss: 0.0073 - accuracy: 0.9914
219/469 [=============>................] - ETA: 7s - loss: 0.0075 - accuracy: 0.9912
221/469 [=============>................] - ETA: 7s - loss: 0.0075 - accuracy: 0.9913
223/469 [=============>................] - ETA: 7s - loss: 0.0075 - accuracy: 0.9912
225/469 [=============>................] - ETA: 7s - loss: 0.0075 - accuracy: 0.9912
227/469 [=============>................] - ETA: 6s - loss: 0.0075 - accuracy: 0.9913
229/469 [=============>................] - ETA: 6s - loss: 0.0075 - accuracy: 0.9913
231/469 [=============>................] - ETA: 6s - loss: 0.0075 - accuracy: 0.9912
233/469 [=============>................] - ETA: 6s - loss: 0.0075 - accuracy: 0.9913
235/469 [==============>...............] - ETA: 6s - loss: 0.0075 - accuracy: 0.9912
237/469 [==============>...............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9912
239/469 [==============>...............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9911
241/469 [==============>...............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9911
243/469 [==============>...............] - ETA: 6s - loss: 0.0077 - accuracy: 0.9910
245/469 [==============>...............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9910
247/469 [==============>...............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9911
249/469 [==============>...............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9911
251/469 [===============>..............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9911
253/469 [===============>..............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9911
255/469 [===============>..............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9910
257/469 [===============>..............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9910
259/469 [===============>..............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9910
261/469 [===============>..............] - ETA: 6s - loss: 0.0076 - accuracy: 0.9910
263/469 [===============>..............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
265/469 [===============>..............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
267/469 [================>.............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
269/469 [================>.............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9909
271/469 [================>.............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9909
273/469 [================>.............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9909
275/469 [================>.............] - ETA: 5s - loss: 0.0077 - accuracy: 0.9909
277/469 [================>.............] - ETA: 5s - loss: 0.0077 - accuracy: 0.9909
279/469 [================>.............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9909
281/469 [================>.............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
283/469 [=================>............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
285/469 [=================>............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9911
287/469 [=================>............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
289/469 [=================>............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
291/469 [=================>............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
293/469 [=================>............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
295/469 [=================>............] - ETA: 5s - loss: 0.0076 - accuracy: 0.9910
297/469 [=================>............] - ETA: 4s - loss: 0.0076 - accuracy: 0.9910
299/469 [==================>...........] - ETA: 4s - loss: 0.0076 - accuracy: 0.9910
301/469 [==================>...........] - ETA: 4s - loss: 0.0076 - accuracy: 0.9910
303/469 [==================>...........] - ETA: 4s - loss: 0.0076 - accuracy: 0.9910
305/469 [==================>...........] - ETA: 4s - loss: 0.0075 - accuracy: 0.9911
307/469 [==================>...........] - ETA: 4s - loss: 0.0075 - accuracy: 0.9911
309/469 [==================>...........] - ETA: 4s - loss: 0.0075 - accuracy: 0.9912
311/469 [==================>...........] - ETA: 4s - loss: 0.0075 - accuracy: 0.9912
313/469 [===================>..........] - ETA: 4s - loss: 0.0075 - accuracy: 0.9912
315/469 [===================>..........] - ETA: 4s - loss: 0.0074 - accuracy: 0.9912
317/469 [===================>..........] - ETA: 4s - loss: 0.0074 - accuracy: 0.9913
319/469 [===================>..........] - ETA: 4s - loss: 0.0074 - accuracy: 0.9913
321/469 [===================>..........] - ETA: 4s - loss: 0.0074 - accuracy: 0.9913
323/469 [===================>..........] - ETA: 4s - loss: 0.0073 - accuracy: 0.9913
325/469 [===================>..........] - ETA: 4s - loss: 0.0073 - accuracy: 0.9914
327/469 [===================>..........] - ETA: 4s - loss: 0.0073 - accuracy: 0.9914
329/469 [====================>.........] - ETA: 4s - loss: 0.0073 - accuracy: 0.9914
331/469 [====================>.........] - ETA: 3s - loss: 0.0072 - accuracy: 0.9914
333/469 [====================>.........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9914
335/469 [====================>.........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9914
337/469 [====================>.........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9914
339/469 [====================>.........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9914
341/469 [====================>.........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9913
343/469 [====================>.........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9913
345/469 [=====================>........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9914
347/469 [=====================>........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9914
349/469 [=====================>........] - ETA: 3s - loss: 0.0073 - accuracy: 0.9914
351/469 [=====================>........] - ETA: 3s - loss: 0.0072 - accuracy: 0.9915
353/469 [=====================>........] - ETA: 3s - loss: 0.0072 - accuracy: 0.9915
355/469 [=====================>........] - ETA: 3s - loss: 0.0072 - accuracy: 0.9915
357/469 [=====================>........] - ETA: 3s - loss: 0.0072 - accuracy: 0.9915
359/469 [=====================>........] - ETA: 3s - loss: 0.0072 - accuracy: 0.9915
361/469 [======================>.......] - ETA: 3s - loss: 0.0072 - accuracy: 0.9915
363/469 [======================>.......] - ETA: 3s - loss: 0.0072 - accuracy: 0.9914
365/469 [======================>.......] - ETA: 3s - loss: 0.0072 - accuracy: 0.9915
367/469 [======================>.......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9915
369/469 [======================>.......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9915
371/469 [======================>.......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9915
373/469 [======================>.......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9915
375/469 [======================>.......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9915
377/469 [=======================>......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9915
379/469 [=======================>......] - ETA: 2s - loss: 0.0071 - accuracy: 0.9915
381/469 [=======================>......] - ETA: 2s - loss: 0.0071 - accuracy: 0.9916
383/469 [=======================>......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9916
385/469 [=======================>......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9916
387/469 [=======================>......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9916
389/469 [=======================>......] - ETA: 2s - loss: 0.0072 - accuracy: 0.9916
391/469 [========================>.....] - ETA: 2s - loss: 0.0072 - accuracy: 0.9915
393/469 [========================>.....] - ETA: 2s - loss: 0.0072 - accuracy: 0.9915
395/469 [========================>.....] - ETA: 2s - loss: 0.0072 - accuracy: 0.9916
397/469 [========================>.....] - ETA: 2s - loss: 0.0071 - accuracy: 0.9916
399/469 [========================>.....] - ETA: 2s - loss: 0.0072 - accuracy: 0.9916
401/469 [========================>.....] - ETA: 1s - loss: 0.0072 - accuracy: 0.9916
403/469 [========================>.....] - ETA: 1s - loss: 0.0072 - accuracy: 0.9915
405/469 [========================>.....] - ETA: 1s - loss: 0.0072 - accuracy: 0.9916
407/469 [=========================>....] - ETA: 1s - loss: 0.0072 - accuracy: 0.9916
409/469 [=========================>....] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
411/469 [=========================>....] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
413/469 [=========================>....] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
415/469 [=========================>....] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
417/469 [=========================>....] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
419/469 [=========================>....] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
421/469 [=========================>....] - ETA: 1s - loss: 0.0071 - accuracy: 0.9917
423/469 [==========================>...] - ETA: 1s - loss: 0.0071 - accuracy: 0.9917
425/469 [==========================>...] - ETA: 1s - loss: 0.0071 - accuracy: 0.9917
427/469 [==========================>...] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
429/469 [==========================>...] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
431/469 [==========================>...] - ETA: 1s - loss: 0.0071 - accuracy: 0.9916
433/469 [==========================>...] - ETA: 1s - loss: 0.0071 - accuracy: 0.9917
435/469 [==========================>...] - ETA: 0s - loss: 0.0071 - accuracy: 0.9917
437/469 [==========================>...] - ETA: 0s - loss: 0.0071 - accuracy: 0.9917
439/469 [===========================>..] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
441/469 [===========================>..] - ETA: 0s - loss: 0.0071 - accuracy: 0.9917
443/469 [===========================>..] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
445/469 [===========================>..] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
447/469 [===========================>..] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
449/469 [===========================>..] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
451/469 [===========================>..] - ETA: 0s - loss: 0.0071 - accuracy: 0.9917
453/469 [===========================>..] - ETA: 0s - loss: 0.0071 - accuracy: 0.9917
455/469 [============================>.] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
457/469 [============================>.] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
459/469 [============================>.] - ETA: 0s - loss: 0.0070 - accuracy: 0.9918
461/469 [============================>.] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
463/469 [============================>.] - ETA: 0s - loss: 0.0070 - accuracy: 0.9917
465/469 [============================>.] - ETA: 0s - loss: 0.0071 - accuracy: 0.9917
467/469 [============================>.] - ETA: 0s - loss: 0.0071 - accuracy: 0.9917
469/469 [==============================] - ETA: 0s - loss: 0.0071 - accuracy: 0.9917
469/469 [==============================] - 14s 30ms/step - loss: 0.0071 - accuracy: 0.9917 - val_loss: 0.0084 - val_accuracy: 0.9893 - lr: 3.9811e-04
Epoch 4/5

  1/469 [..............................] - ETA: 0s - loss: 0.0054 - accuracy: 0.9922
  3/469 [..............................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9948
  5/469 [..............................] - ETA: 10s - loss: 0.0049 - accuracy: 0.9937
  7/469 [..............................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9911
  9/469 [..............................] - ETA: 11s - loss: 0.0048 - accuracy: 0.9931
 11/469 [..............................] - ETA: 12s - loss: 0.0049 - accuracy: 0.9936
 13/469 [..............................] - ETA: 12s - loss: 0.0048 - accuracy: 0.9940
 15/469 [..............................] - ETA: 12s - loss: 0.0049 - accuracy: 0.9943
 17/469 [>.............................] - ETA: 12s - loss: 0.0049 - accuracy: 0.9940
 19/469 [>.............................] - ETA: 12s - loss: 0.0045 - accuracy: 0.9947
 21/469 [>.............................] - ETA: 12s - loss: 0.0050 - accuracy: 0.9937
 23/469 [>.............................] - ETA: 12s - loss: 0.0049 - accuracy: 0.9935
 25/469 [>.............................] - ETA: 12s - loss: 0.0049 - accuracy: 0.9934
 27/469 [>.............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9919
 29/469 [>.............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9919
 31/469 [>.............................] - ETA: 12s - loss: 0.0058 - accuracy: 0.9917
 33/469 [=>............................] - ETA: 12s - loss: 0.0059 - accuracy: 0.9917
 35/469 [=>............................] - ETA: 12s - loss: 0.0059 - accuracy: 0.9915
 37/469 [=>............................] - ETA: 12s - loss: 0.0057 - accuracy: 0.9918
 39/469 [=>............................] - ETA: 12s - loss: 0.0056 - accuracy: 0.9922
 41/469 [=>............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9924
 43/469 [=>............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9926
 45/469 [=>............................] - ETA: 12s - loss: 0.0055 - accuracy: 0.9925
 47/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9924
 49/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9925
 51/469 [==>...........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9928
 53/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9928
 55/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9929
 57/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9930
 59/469 [==>...........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9930
 61/469 [==>...........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9931
 63/469 [===>..........................] - ETA: 11s - loss: 0.0054 - accuracy: 0.9933
 65/469 [===>..........................] - ETA: 11s - loss: 0.0054 - accuracy: 0.9933
 67/469 [===>..........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9932
 69/469 [===>..........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9934
 71/469 [===>..........................] - ETA: 11s - loss: 0.0054 - accuracy: 0.9935
 73/469 [===>..........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9934
 75/469 [===>..........................] - ETA: 11s - loss: 0.0056 - accuracy: 0.9933
 77/469 [===>..........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9935
 79/469 [====>.........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9936
 81/469 [====>.........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9936
 83/469 [====>.........................] - ETA: 11s - loss: 0.0055 - accuracy: 0.9935
 85/469 [====>.........................] - ETA: 11s - loss: 0.0054 - accuracy: 0.9937
 87/469 [====>.........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9934
 89/469 [====>.........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9934
 91/469 [====>.........................] - ETA: 10s - loss: 0.0053 - accuracy: 0.9935
 93/469 [====>.........................] - ETA: 10s - loss: 0.0054 - accuracy: 0.9934
 95/469 [=====>........................] - ETA: 10s - loss: 0.0053 - accuracy: 0.9935
 97/469 [=====>........................] - ETA: 10s - loss: 0.0052 - accuracy: 0.9936
 99/469 [=====>........................] - ETA: 10s - loss: 0.0053 - accuracy: 0.9936
101/469 [=====>........................] - ETA: 10s - loss: 0.0052 - accuracy: 0.9937
103/469 [=====>........................] - ETA: 10s - loss: 0.0052 - accuracy: 0.9938
105/469 [=====>........................] - ETA: 10s - loss: 0.0052 - accuracy: 0.9937
107/469 [=====>........................] - ETA: 10s - loss: 0.0052 - accuracy: 0.9936
109/469 [=====>........................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9938
111/469 [======>.......................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9939
113/469 [======>.......................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9939
115/469 [======>.......................] - ETA: 10s - loss: 0.0051 - accuracy: 0.9939
117/469 [======>.......................] - ETA: 10s - loss: 0.0053 - accuracy: 0.9935
119/469 [======>.......................] - ETA: 10s - loss: 0.0053 - accuracy: 0.9935
121/469 [======>.......................] - ETA: 10s - loss: 0.0053 - accuracy: 0.9935
123/469 [======>.......................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9935 
125/469 [======>.......................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9936
127/469 [=======>......................] - ETA: 9s - loss: 0.0054 - accuracy: 0.9935
129/469 [=======>......................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9936
131/469 [=======>......................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9936
133/469 [=======>......................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9937
135/469 [=======>......................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9937
137/469 [=======>......................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9936
139/469 [=======>......................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9936
141/469 [========>.....................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9937
143/469 [========>.....................] - ETA: 9s - loss: 0.0053 - accuracy: 0.9937
145/469 [========>.....................] - ETA: 9s - loss: 0.0052 - accuracy: 0.9937
147/469 [========>.....................] - ETA: 9s - loss: 0.0052 - accuracy: 0.9937
149/469 [========>.....................] - ETA: 9s - loss: 0.0052 - accuracy: 0.9938
151/469 [========>.....................] - ETA: 9s - loss: 0.0052 - accuracy: 0.9937
153/469 [========>.....................] - ETA: 9s - loss: 0.0051 - accuracy: 0.9938
155/469 [========>.....................] - ETA: 9s - loss: 0.0051 - accuracy: 0.9938
157/469 [=========>....................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9938
159/469 [=========>....................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9938
161/469 [=========>....................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9938
163/469 [=========>....................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9938
165/469 [=========>....................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9938
167/469 [=========>....................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9938
169/469 [=========>....................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9938
171/469 [=========>....................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9939
173/469 [==========>...................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9939
175/469 [==========>...................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9939
177/469 [==========>...................] - ETA: 8s - loss: 0.0051 - accuracy: 0.9940
179/469 [==========>...................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9939
181/469 [==========>...................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9939
183/469 [==========>...................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9939
185/469 [==========>...................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9939
187/469 [==========>...................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9939
189/469 [===========>..................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9938
191/469 [===========>..................] - ETA: 8s - loss: 0.0052 - accuracy: 0.9939
193/469 [===========>..................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9939
195/469 [===========>..................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9938
197/469 [===========>..................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9939
199/469 [===========>..................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9939
201/469 [===========>..................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9939
203/469 [===========>..................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9938
205/469 [============>.................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9939
207/469 [============>.................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9938
209/469 [============>.................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9939
211/469 [============>.................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9937
213/469 [============>.................] - ETA: 7s - loss: 0.0052 - accuracy: 0.9937
215/469 [============>.................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9937
217/469 [============>.................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9936
219/469 [=============>................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9936
221/469 [=============>................] - ETA: 7s - loss: 0.0053 - accuracy: 0.9936
223/469 [=============>................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9935
225/469 [=============>................] - ETA: 7s - loss: 0.0054 - accuracy: 0.9934
227/469 [=============>................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
229/469 [=============>................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
231/469 [=============>................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
233/469 [=============>................] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
235/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
237/469 [==============>...............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9934
239/469 [==============>...............] - ETA: 6s - loss: 0.0055 - accuracy: 0.9933
241/469 [==============>...............] - ETA: 6s - loss: 0.0055 - accuracy: 0.9933
243/469 [==============>...............] - ETA: 6s - loss: 0.0055 - accuracy: 0.9933
245/469 [==============>...............] - ETA: 6s - loss: 0.0055 - accuracy: 0.9933
247/469 [==============>...............] - ETA: 6s - loss: 0.0055 - accuracy: 0.9934
249/469 [==============>...............] - ETA: 6s - loss: 0.0055 - accuracy: 0.9934
251/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
253/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
255/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
257/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
259/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9935
261/469 [===============>..............] - ETA: 6s - loss: 0.0054 - accuracy: 0.9934
263/469 [===============>..............] - ETA: 5s - loss: 0.0055 - accuracy: 0.9934
265/469 [===============>..............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
267/469 [================>.............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
269/469 [================>.............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
271/469 [================>.............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
273/469 [================>.............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
275/469 [================>.............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
277/469 [================>.............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
279/469 [================>.............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9936
281/469 [================>.............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
283/469 [=================>............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9936
285/469 [=================>............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9936
287/469 [=================>............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
289/469 [=================>............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9935
291/469 [=================>............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9936
293/469 [=================>............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9936
295/469 [=================>............] - ETA: 5s - loss: 0.0054 - accuracy: 0.9936
297/469 [=================>............] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
299/469 [==================>...........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
301/469 [==================>...........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9935
303/469 [==================>...........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
305/469 [==================>...........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
307/469 [==================>...........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
309/469 [==================>...........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
311/469 [==================>...........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
313/469 [===================>..........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9937
315/469 [===================>..........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9937
317/469 [===================>..........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
319/469 [===================>..........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9937
321/469 [===================>..........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
323/469 [===================>..........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
325/469 [===================>..........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9936
327/469 [===================>..........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9935
329/469 [====================>.........] - ETA: 4s - loss: 0.0055 - accuracy: 0.9935
331/469 [====================>.........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9935
333/469 [====================>.........] - ETA: 3s - loss: 0.0056 - accuracy: 0.9934
335/469 [====================>.........] - ETA: 3s - loss: 0.0056 - accuracy: 0.9934
337/469 [====================>.........] - ETA: 3s - loss: 0.0056 - accuracy: 0.9934
339/469 [====================>.........] - ETA: 3s - loss: 0.0056 - accuracy: 0.9934
341/469 [====================>.........] - ETA: 3s - loss: 0.0056 - accuracy: 0.9934
343/469 [====================>.........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9935
345/469 [=====================>........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9935
347/469 [=====================>........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9935
349/469 [=====================>........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9935
351/469 [=====================>........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9935
353/469 [=====================>........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9934
355/469 [=====================>........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9934
357/469 [=====================>........] - ETA: 3s - loss: 0.0055 - accuracy: 0.9934
359/469 [=====================>........] - ETA: 3s - loss: 0.0056 - accuracy: 0.9934
361/469 [======================>.......] - ETA: 3s - loss: 0.0055 - accuracy: 0.9934
363/469 [======================>.......] - ETA: 3s - loss: 0.0055 - accuracy: 0.9934
365/469 [======================>.......] - ETA: 3s - loss: 0.0055 - accuracy: 0.9935
367/469 [======================>.......] - ETA: 2s - loss: 0.0055 - accuracy: 0.9934
369/469 [======================>.......] - ETA: 2s - loss: 0.0055 - accuracy: 0.9934
371/469 [======================>.......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
373/469 [======================>.......] - ETA: 2s - loss: 0.0055 - accuracy: 0.9934
375/469 [======================>.......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
377/469 [=======================>......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9933
379/469 [=======================>......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9933
381/469 [=======================>......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
383/469 [=======================>......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
385/469 [=======================>......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
387/469 [=======================>......] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
389/469 [=======================>......] - ETA: 2s - loss: 0.0055 - accuracy: 0.9934
391/469 [========================>.....] - ETA: 2s - loss: 0.0055 - accuracy: 0.9934
393/469 [========================>.....] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
395/469 [========================>.....] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
397/469 [========================>.....] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
399/469 [========================>.....] - ETA: 2s - loss: 0.0056 - accuracy: 0.9934
401/469 [========================>.....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
403/469 [========================>.....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
405/469 [========================>.....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9933
407/469 [=========================>....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
409/469 [=========================>....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
411/469 [=========================>....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
413/469 [=========================>....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
415/469 [=========================>....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
417/469 [=========================>....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
419/469 [=========================>....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
421/469 [=========================>....] - ETA: 1s - loss: 0.0056 - accuracy: 0.9934
423/469 [==========================>...] - ETA: 1s - loss: 0.0056 - accuracy: 0.9933
425/469 [==========================>...] - ETA: 1s - loss: 0.0056 - accuracy: 0.9933
427/469 [==========================>...] - ETA: 1s - loss: 0.0056 - accuracy: 0.9933
429/469 [==========================>...] - ETA: 1s - loss: 0.0056 - accuracy: 0.9933
431/469 [==========================>...] - ETA: 1s - loss: 0.0056 - accuracy: 0.9933
433/469 [==========================>...] - ETA: 1s - loss: 0.0056 - accuracy: 0.9933
435/469 [==========================>...] - ETA: 0s - loss: 0.0056 - accuracy: 0.9932
437/469 [==========================>...] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
439/469 [===========================>..] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
441/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9932
443/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9932
445/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9932
447/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9932
449/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9932
451/469 [===========================>..] - ETA: 0s - loss: 0.0057 - accuracy: 0.9932
453/469 [===========================>..] - ETA: 0s - loss: 0.0056 - accuracy: 0.9932
455/469 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
457/469 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
459/469 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
461/469 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9932
463/469 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
465/469 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
467/469 [============================>.] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
469/469 [==============================] - ETA: 0s - loss: 0.0056 - accuracy: 0.9933
469/469 [==============================] - 14s 30ms/step - loss: 0.0056 - accuracy: 0.9933 - val_loss: 0.0101 - val_accuracy: 0.9870 - lr: 2.5119e-04
Epoch 5/5

  1/469 [..............................] - ETA: 0s - loss: 0.0024 - accuracy: 1.0000
  3/469 [..............................] - ETA: 9s - loss: 0.0021 - accuracy: 1.0000
  5/469 [..............................] - ETA: 10s - loss: 0.0024 - accuracy: 1.0000
  7/469 [..............................] - ETA: 11s - loss: 0.0028 - accuracy: 1.0000
  9/469 [..............................] - ETA: 11s - loss: 0.0030 - accuracy: 0.9983
 11/469 [..............................] - ETA: 12s - loss: 0.0034 - accuracy: 0.9979
 13/469 [..............................] - ETA: 12s - loss: 0.0037 - accuracy: 0.9970
 15/469 [..............................] - ETA: 12s - loss: 0.0033 - accuracy: 0.9974
 17/469 [>.............................] - ETA: 12s - loss: 0.0038 - accuracy: 0.9968
 19/469 [>.............................] - ETA: 12s - loss: 0.0037 - accuracy: 0.9967
 21/469 [>.............................] - ETA: 12s - loss: 0.0035 - accuracy: 0.9967
 23/469 [>.............................] - ETA: 12s - loss: 0.0036 - accuracy: 0.9966
 25/469 [>.............................] - ETA: 12s - loss: 0.0034 - accuracy: 0.9969
 27/469 [>.............................] - ETA: 12s - loss: 0.0034 - accuracy: 0.9968
 29/469 [>.............................] - ETA: 12s - loss: 0.0034 - accuracy: 0.9968
 31/469 [>.............................] - ETA: 12s - loss: 0.0035 - accuracy: 0.9967
 33/469 [=>............................] - ETA: 12s - loss: 0.0036 - accuracy: 0.9962
 35/469 [=>............................] - ETA: 12s - loss: 0.0037 - accuracy: 0.9964
 37/469 [=>............................] - ETA: 12s - loss: 0.0038 - accuracy: 0.9962
 39/469 [=>............................] - ETA: 12s - loss: 0.0041 - accuracy: 0.9960
 41/469 [=>............................] - ETA: 12s - loss: 0.0040 - accuracy: 0.9960
 43/469 [=>............................] - ETA: 12s - loss: 0.0040 - accuracy: 0.9962
 45/469 [=>............................] - ETA: 12s - loss: 0.0040 - accuracy: 0.9962
 47/469 [==>...........................] - ETA: 11s - loss: 0.0040 - accuracy: 0.9960
 49/469 [==>...........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9960
 51/469 [==>...........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9962
 53/469 [==>...........................] - ETA: 11s - loss: 0.0040 - accuracy: 0.9962
 55/469 [==>...........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9960
 57/469 [==>...........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9959
 59/469 [==>...........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9960
 61/469 [==>...........................] - ETA: 11s - loss: 0.0040 - accuracy: 0.9962
 63/469 [===>..........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9960
 65/469 [===>..........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9962
 67/469 [===>..........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9962
 69/469 [===>..........................] - ETA: 11s - loss: 0.0040 - accuracy: 0.9963
 71/469 [===>..........................] - ETA: 11s - loss: 0.0039 - accuracy: 0.9964
 73/469 [===>..........................] - ETA: 11s - loss: 0.0040 - accuracy: 0.9963
 75/469 [===>..........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9961
 77/469 [===>..........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9961
 79/469 [====>.........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9961
 81/469 [====>.........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9960
 83/469 [====>.........................] - ETA: 11s - loss: 0.0041 - accuracy: 0.9961
 85/469 [====>.........................] - ETA: 10s - loss: 0.0041 - accuracy: 0.9961
 87/469 [====>.........................] - ETA: 10s - loss: 0.0043 - accuracy: 0.9960
 89/469 [====>.........................] - ETA: 10s - loss: 0.0042 - accuracy: 0.9961
 91/469 [====>.........................] - ETA: 10s - loss: 0.0042 - accuracy: 0.9961
 93/469 [====>.........................] - ETA: 10s - loss: 0.0042 - accuracy: 0.9961
 95/469 [=====>........................] - ETA: 10s - loss: 0.0043 - accuracy: 0.9960
 97/469 [=====>........................] - ETA: 10s - loss: 0.0043 - accuracy: 0.9959
 99/469 [=====>........................] - ETA: 10s - loss: 0.0043 - accuracy: 0.9959
101/469 [=====>........................] - ETA: 10s - loss: 0.0045 - accuracy: 0.9957
103/469 [=====>........................] - ETA: 10s - loss: 0.0045 - accuracy: 0.9958
105/469 [=====>........................] - ETA: 10s - loss: 0.0045 - accuracy: 0.9956
107/469 [=====>........................] - ETA: 10s - loss: 0.0045 - accuracy: 0.9956
109/469 [=====>........................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9957
111/469 [======>.......................] - ETA: 10s - loss: 0.0044 - accuracy: 0.9956
113/469 [======>.......................] - ETA: 10s - loss: 0.0046 - accuracy: 0.9955
115/469 [======>.......................] - ETA: 10s - loss: 0.0045 - accuracy: 0.9955
117/469 [======>.......................] - ETA: 10s - loss: 0.0047 - accuracy: 0.9954
119/469 [======>.......................] - ETA: 10s - loss: 0.0046 - accuracy: 0.9954
121/469 [======>.......................] - ETA: 10s - loss: 0.0046 - accuracy: 0.9955
123/469 [======>.......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9953 
125/469 [======>.......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9952
127/469 [=======>......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9951
129/469 [=======>......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9951
131/469 [=======>......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9951
133/469 [=======>......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9951
135/469 [=======>......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9950
137/469 [=======>......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9951
139/469 [=======>......................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9951
141/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9950
143/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9949
145/469 [========>.....................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9949
147/469 [========>.....................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9949
149/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9949
151/469 [========>.....................] - ETA: 9s - loss: 0.0047 - accuracy: 0.9949
153/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9948
155/469 [========>.....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9949
157/469 [=========>....................] - ETA: 9s - loss: 0.0048 - accuracy: 0.9949
159/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9949
161/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9950
163/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9949
165/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9949
167/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9949
169/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9950
171/469 [=========>....................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9950
173/469 [==========>...................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9950
175/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9950
177/469 [==========>...................] - ETA: 8s - loss: 0.0047 - accuracy: 0.9950
179/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9951
181/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9950
183/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9950
185/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9951
187/469 [==========>...................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9951
189/469 [===========>..................] - ETA: 8s - loss: 0.0046 - accuracy: 0.9951
191/469 [===========>..................] - ETA: 8s - loss: 0.0045 - accuracy: 0.9951
193/469 [===========>..................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9951
195/469 [===========>..................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9951
197/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9951
199/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9952
201/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9951
203/469 [===========>..................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9950
205/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9951
207/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9950
209/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9950
211/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9950
213/469 [============>.................] - ETA: 7s - loss: 0.0045 - accuracy: 0.9949
215/469 [============>.................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9949
217/469 [============>.................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9949
219/469 [=============>................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9949
221/469 [=============>................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9948
223/469 [=============>................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9948
225/469 [=============>................] - ETA: 7s - loss: 0.0046 - accuracy: 0.9948
227/469 [=============>................] - ETA: 6s - loss: 0.0046 - accuracy: 0.9948
229/469 [=============>................] - ETA: 6s - loss: 0.0047 - accuracy: 0.9947
231/469 [=============>................] - ETA: 6s - loss: 0.0047 - accuracy: 0.9948
233/469 [=============>................] - ETA: 6s - loss: 0.0047 - accuracy: 0.9948
235/469 [==============>...............] - ETA: 6s - loss: 0.0047 - accuracy: 0.9948
237/469 [==============>...............] - ETA: 6s - loss: 0.0047 - accuracy: 0.9948
239/469 [==============>...............] - ETA: 6s - loss: 0.0047 - accuracy: 0.9948
241/469 [==============>...............] - ETA: 6s - loss: 0.0047 - accuracy: 0.9948
243/469 [==============>...............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9948
245/469 [==============>...............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9948
247/469 [==============>...............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9948
249/469 [==============>...............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9949
251/469 [===============>..............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9948
253/469 [===============>..............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9948
255/469 [===============>..............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9948
257/469 [===============>..............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9949
259/469 [===============>..............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9948
261/469 [===============>..............] - ETA: 6s - loss: 0.0046 - accuracy: 0.9949
263/469 [===============>..............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9949
265/469 [===============>..............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9949
267/469 [================>.............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
269/469 [================>.............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
271/469 [================>.............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
273/469 [================>.............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
275/469 [================>.............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
277/469 [================>.............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
279/469 [================>.............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
281/469 [================>.............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
283/469 [=================>............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9948
285/469 [=================>............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9947
287/469 [=================>............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9947
289/469 [=================>............] - ETA: 5s - loss: 0.0047 - accuracy: 0.9947
291/469 [=================>............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9947
293/469 [=================>............] - ETA: 5s - loss: 0.0046 - accuracy: 0.9947
295/469 [=================>............] - ETA: 5s - loss: 0.0047 - accuracy: 0.9947
297/469 [=================>............] - ETA: 4s - loss: 0.0046 - accuracy: 0.9948
299/469 [==================>...........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9948
301/469 [==================>...........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9948
303/469 [==================>...........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9948
305/469 [==================>...........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
307/469 [==================>...........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9948
309/469 [==================>...........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9948
311/469 [==================>...........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9948
313/469 [===================>..........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
315/469 [===================>..........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
317/469 [===================>..........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
319/469 [===================>..........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
321/469 [===================>..........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
323/469 [===================>..........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
325/469 [===================>..........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
327/469 [===================>..........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
329/469 [====================>.........] - ETA: 4s - loss: 0.0046 - accuracy: 0.9949
331/469 [====================>.........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9949
333/469 [====================>.........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9950
335/469 [====================>.........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9949
337/469 [====================>.........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9949
339/469 [====================>.........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9950
341/469 [====================>.........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9949
343/469 [====================>.........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9950
345/469 [=====================>........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9950
347/469 [=====================>........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9950
349/469 [=====================>........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9949
351/469 [=====================>........] - ETA: 3s - loss: 0.0045 - accuracy: 0.9949
353/469 [=====================>........] - ETA: 3s - loss: 0.0046 - accuracy: 0.9949
355/469 [=====================>........] - ETA: 3s - loss: 0.0046 - accuracy: 0.9949
357/469 [=====================>........] - ETA: 3s - loss: 0.0046 - accuracy: 0.9949
359/469 [=====================>........] - ETA: 3s - loss: 0.0046 - accuracy: 0.9949
361/469 [======================>.......] - ETA: 3s - loss: 0.0046 - accuracy: 0.9949
363/469 [======================>.......] - ETA: 3s - loss: 0.0046 - accuracy: 0.9949
365/469 [======================>.......] - ETA: 3s - loss: 0.0046 - accuracy: 0.9949
367/469 [======================>.......] - ETA: 2s - loss: 0.0046 - accuracy: 0.9949
369/469 [======================>.......] - ETA: 2s - loss: 0.0046 - accuracy: 0.9949
371/469 [======================>.......] - ETA: 2s - loss: 0.0046 - accuracy: 0.9949
373/469 [======================>.......] - ETA: 2s - loss: 0.0046 - accuracy: 0.9950
375/469 [======================>.......] - ETA: 2s - loss: 0.0046 - accuracy: 0.9949
377/469 [=======================>......] - ETA: 2s - loss: 0.0046 - accuracy: 0.9950
379/469 [=======================>......] - ETA: 2s - loss: 0.0046 - accuracy: 0.9950
381/469 [=======================>......] - ETA: 2s - loss: 0.0045 - accuracy: 0.9950
383/469 [=======================>......] - ETA: 2s - loss: 0.0045 - accuracy: 0.9950
385/469 [=======================>......] - ETA: 2s - loss: 0.0045 - accuracy: 0.9950
387/469 [=======================>......] - ETA: 2s - loss: 0.0045 - accuracy: 0.9951
389/469 [=======================>......] - ETA: 2s - loss: 0.0045 - accuracy: 0.9951
391/469 [========================>.....] - ETA: 2s - loss: 0.0045 - accuracy: 0.9951
393/469 [========================>.....] - ETA: 2s - loss: 0.0045 - accuracy: 0.9951
395/469 [========================>.....] - ETA: 2s - loss: 0.0045 - accuracy: 0.9951
397/469 [========================>.....] - ETA: 2s - loss: 0.0045 - accuracy: 0.9951
399/469 [========================>.....] - ETA: 2s - loss: 0.0045 - accuracy: 0.9951
401/469 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
403/469 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
405/469 [========================>.....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
407/469 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
409/469 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
411/469 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
413/469 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
415/469 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
417/469 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
419/469 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
421/469 [=========================>....] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
423/469 [==========================>...] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
425/469 [==========================>...] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
427/469 [==========================>...] - ETA: 1s - loss: 0.0045 - accuracy: 0.9951
429/469 [==========================>...] - ETA: 1s - loss: 0.0045 - accuracy: 0.9950
431/469 [==========================>...] - ETA: 1s - loss: 0.0045 - accuracy: 0.9950
433/469 [==========================>...] - ETA: 1s - loss: 0.0046 - accuracy: 0.9950
435/469 [==========================>...] - ETA: 0s - loss: 0.0046 - accuracy: 0.9950
437/469 [==========================>...] - ETA: 0s - loss: 0.0045 - accuracy: 0.9950
439/469 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9950
441/469 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9950
443/469 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9950
445/469 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9950
447/469 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
449/469 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9950
451/469 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
453/469 [===========================>..] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
455/469 [============================>.] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
457/469 [============================>.] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
459/469 [============================>.] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
461/469 [============================>.] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
463/469 [============================>.] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
465/469 [============================>.] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
467/469 [============================>.] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
469/469 [==============================] - ETA: 0s - loss: 0.0046 - accuracy: 0.9949
469/469 [==============================] - 14s 30ms/step - loss: 0.0046 - accuracy: 0.9949 - val_loss: 0.0072 - val_accuracy: 0.9901 - lr: 1.5849e-04
Test score: 0.00723641412332654
Test accuracy: 0.9900000095367432

5. Convert trained model for Akida and test

5.1 Final conversion

Convert the quantized model to a version suitable to be used in the Akida NSoC in inference mode:

Note

One needs to supply the coefficients used to rescale the input dataset before the training - here input_scaling.

As with Keras, the summary() method provides a textual representation of the model.

from cnn2snn import convert

model_akida = convert(model_keras, input_scaling=input_scaling)
model_akida.summary()

Out:

                        Model Summary
______________________________________________________________
Layer (type)                 Output shape  Kernel shape
==============================================================
conv_0 (InputConvolutional)  [14, 14, 32]  (5, 5, 1, 32)
______________________________________________________________
conv_1 (Convolutional)       [7, 7, 32]    (5, 5, 32, 32)
______________________________________________________________
conv_2 (Convolutional)       [7, 7, 32]    (5, 5, 32, 32)
______________________________________________________________
dense_2 (FullyConnected)     [1, 1, 512]   (1, 1, 1568, 512)
______________________________________________________________
dense_3 (FullyConnected)     [1, 1, 10]    (1, 1, 512, 10)
______________________________________________________________
Input shape: 28, 28, 1
Backend type: Software - 1.8.5

5.2 Performances check with the Akida Execution Engine

num_samples = 1000

results = model_akida.predict(raw_x_test[:num_samples])
accuracy = accuracy_score(raw_y_test[:num_samples], results[:num_samples])

print("Accuracy: " + "{0:.2f}".format(100 * accuracy) + "%")

# For non-regression purpose
assert accuracy > 0.95

# Print model statistics
print("Model statistics")
stats = model_akida.get_statistics()
model_akida.predict(raw_x_test[:20])
for _, stat in stats.items():
    print(stat)

Out:

Accuracy: 99.00%
Model statistics
Layer (type)                  output sparsity
conv_0 (InputConvolutional)   0.85
Layer (type)                  input sparsity      output sparsity     ops
conv_1 (Convolutional)        0.85                0.76                730000
Layer (type)                  input sparsity      output sparsity     ops
conv_2 (Convolutional)        0.76                0.70                295840
Layer (type)                  input sparsity      output sparsity     ops
dense_2 (FullyConnected)      0.70                0.68                244762
Layer (type)                  input sparsity      output sparsity     ops
dense_3 (FullyConnected)      0.68                0.00                1614

Depending on the number of samples you run, you should find a performance of around 99% (better results can be achieved using more epochs for training).

Note

Akida-compatible model first layer type is InputConvolutional and holds underlying data to spike conversion (please refer to Akida Execution Engine documentation for more details).

Total running time of the script: ( 3 minutes 14.736 seconds)

Gallery generated by Sphinx-Gallery