Abstract
For deep learning (DL) performance analysis and optimization, especially on resource-constrained edge-AI and embedded devices, the role of deep learning libraries such as cuBLAS, cuDNN, and TensorRT becomes more crucial for efficient deep neural network (DNN) models. However, thorough and systematic investigations for DL libraries on real edge or embedded platforms are rare. To address this gap, we propose a framework for DL library performance analysis and optimization, organized into three phases: 1) Design and implement micro-benchmarks for convolution layers of convolutional neural networks (CNNs) and attention layers of vision transformers (ViTs) using DL libraries (cuBLAS, cuDNN, cuDNN_opt, and TensorRT) for fair performance evaluation. 2) Compare and analyze performance from the two perspectives of per-layer inference time, and machine learning (ML) model-based analysis. 3) Propose and evaluate an ML model-based optimization using simulations of popular DNN architectures. The results indicate that, for CNNs, cuBLAS performs best when the input size is small or when the input feature size is average with a high number of kernels, and that TensorRT is better in big input sizes or when the feature map is average with a low number of kernels. For ViTs, cuDNN and cuDNN_opt achieve the best performance in configurations with small hidden dimensions and large patch sizes, whereas TensorRT performs best when the hidden dimension is large. However, at the general matrix multiplication (GEMM) level, both CNN and ViT select the same optimal library in 87.5% of the configurations. Using our ML model-based simulations, we achieve 7% improvement on average for CNNs and up to 19% improvements on ViTs, compared to the other DL libraries on a real embedded platform.