TY - JOUR ID - 2496 TI - Whitened gradient descent, a new updating method for optimizers in deep neural networks JO - Journal of AI and Data Mining JA - JADM LA - en SN - 2322-5211 AU - Gholamalinejad, H. AU - Khosravi, H. AD - Department of Computer, Faculty of Engineering, Bozorgmehr University of Qaenat, Qaen, Iran. AD - Faculty of Electrical Engineering Shahrood University of Technology. Y1 - 2022 PY - 2022 VL - 10 IS - 4 SP - 467 EP - 477 KW - deep learning KW - Optimizer KW - Whitened Gradient Descent KW - Momentum DO - 10.22044/jadm.2022.11325.2291 N2 - Optimizers are vital components of deep neural networks that perform weight updates. This paper introduces a new updating method for optimizers based on gradient descent, called whitened gradient descent (WGD). This method is easy to implement and can be used in every optimizer based on the gradient descent algorithm. It does not increase the training time of the network significantly. This method smooths the training curve and improves classification metrics. To evaluate the proposed algorithm, we performed 48 different tests on two datasets, Cifar100 and Animals-10, using three network structures, including densenet121, resnet18, and resnet50. The experiments show that using the WGD method in gradient descent based optimizers, improves the classification results significantly. For example, integrating WGD in RAdam optimizer increased the accuracy of DenseNet from 87.69% to 90.02% on the Animals-10 dataset. UR - https://jad.shahroodut.ac.ir/article_2496.html L1 - https://jad.shahroodut.ac.ir/article_2496_6374b89dbfd0e0729d4d21745d97b6c7.pdf ER -