#### Note

To overcome the accuracy drop from the original when transforming the model as per\n Section 2, it is recommended to replace the original layers all at once and to fine-tune\n afterwards.

\n\nThe example below shows the transformation of ViT (tiny) into an optimized model that can run on\nthe Akida hardware.\n\nThe [akida_models](https://pypi.org/project/akida-models)_ python package provides a Command Line\nInterface (CLI) to transform [vit_ti16](../../_modules/akida_models/transformers/model_vit.html#vit_ti16)_\nand [deit_ti16](../../_modules/akida_models/transformers/model_deit.html#deit_ti16)_ model architectures\nand fine-tune them respectively.\n\n```bash\n$ akida_models create vit_ti16 -h\nusage: akida_models create vit_ti16 [-h] [-c CLASSES] [-bw BASE_WEIGHTS] [--norm {LN,GN1,BN,LMN}]\n [--last_norm {LN,BN}] [--softmax {softmax,softmax2}]\n [--act {GeLU,ReLU8,swish}] [-i {224,384}]\n\noptional arguments:\n -h, --help show this help message and exit\n -c CLASSES, --classes CLASSES\n The number of classes, by default 1000.\n -bw BASE_WEIGHTS, --base_weights BASE_WEIGHTS\n Optional keras weights to load in the model, by default None.\n --norm {LN,GN1,BN,LMN}\n Replace normalization in model with a custom function, by default LN\n --last_norm {LN,BN} Replace last normalization in model with a custom function, by default LN\n --softmax {softmax,softmax2}\n Replace softmax operation in model with custom function, by default softmax\n --act {GeLU,ReLU8,swish}\n Replace activation function in model with custom function, by default GeLU\n -i {224,384}, --image_size {224,384}\n The square input image size\n```\nThe following shows the transformation of a vit_ti16 model architecture which was trained on ImageNet. The\nsame methods can be applied for other datasets.\n\n```bash\n# download the pre-trained weights\nwget https://data.brainchip.com/models/AkidaV2/vit/vit_ti16_224.h5\n\n# transformations: replace layer normalization with mad norm layer, last layer normalization\n# with batch normalization, GeLU layer with ReLU and softmax with shiftmax layer\nakida_models create -s vit_ti16_transformed.h5 vit_ti16 --norm LMN --last_norm BN --act ReLU8 \\\n --softmax softmax2 -bw vit_ti16_224.h5\n# fine-tuning\nimagenet_train tune -m vit_ti16_transformed.h5 -e 30 --optim Adam --lr_policy cosine_decay \\\n -lr 6e-5 -s vit_ti16_transformed.h5\n```\nThe above transformation generates a ViT model that is optimized to run efficiently on Akida hardware.\nSimilar steps can also be applied to deit_ti16. The table below highlights the accuracy of the original\nand transformed models.\n\n+--------------+-------------------+----------------------+\n| Architecture | Original accuracy | Transformed accuracy |\n+==============+===================+======================+\n| ViT | 75.48% | 74.25% |\n+--------------+-------------------+----------------------+\n| DeiT-dist | 74.17% | 75.03% |\n+--------------+-------------------+----------------------+\n\n