gyroscopyГироскопия и навигацияGiroskopiya i Navigatsiya0869-70352075-0927AO «Концерн «ЦНИИ «Электроприбор»LIQJHAgyroscopy-188Research ArticleСтатьиАдаптация ковариации шума в обобщенном фильтре Калмана с использованием обучения с подкреплением для более точного определения углов ориентации БПЛАAdaptation Noise Covariance in Extended Kalman Filter Using Reinforcement Learning for Improved UAV Attitude EstimationАссадА.AssadA.

Ассад Аммар, аспирант

St. Petersburg

СериковC. А.SerikovS. A.

Сериков Сергей Анатольевич, доктор технических наук, доцент

St. Petersburg

Санкт-Петербургский государственный университет аэрокосмического приборостроенияРоссияSt. Petersburg State University of Aerospace InstrumentationRussian Federation2025241020253333350Copyright © Ассад А., Сериков C.А., 20252025Ассад А., Сериков C.А.Assad A., Serikov S.A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.gyroscopy.ru/jour/article/view/188

Точное определение углов ориентации беспилотных летательных аппаратов (БПЛА) имеет решающее значение для автономной навигации, особенно при использовании лишь измерений гироскопов, акселерометров и магнитометров без привлечения данных глобальной системы позиционирования (GPS). Перспективным представляется метод обучения искусственного интеллекта с подкреплением (ОП), который позволяет повысить эффективность применяемого при определении углов ориентации обобщенного фильтра Калмана (ОФК). Предлагаемый подход предусматривает привлечение модели Q-обучения и стратегии поиска наилучшего решения для коррекции матрицы ковариации шума измерений в ОФК в автономном режиме. За счет механизма вознаграждения, стимулирующего действия, с помощью которых сводится к минимуму погрешность прогнозирования углов ориентации относительно истинных измерений, ОП дает возможность динамически оптимизировать матрицу ковариации шума измерений. Интегрированный алгоритм ОП и ОФК (далее – ОП-ОФК) был реализован и протестирован. Результаты показывают, что он значительно превосходит традиционный ОФК в части определения углов ориентации БПЛА.

Accurate attitude determination of Unmanned Aerial Vehicles (UAVs) is crucial for autonomous navigation, particularly when relying solely on gyroscope, accelerometer, and magnetometer measurements without utilizing the Global Positioning System (GPS). Reinforcement Learning (RL) has emerged as a promising artificial intelligence technique applicable across various domains. This research introduces a novel approach that leverages RL to enhance the performance of the Extended Kalman Filter (EKF) in attitude estimation. The proposed method depends of RL which uses the Q-Learning model and policy to find best solution to adjust autonomously the measurement noise covariance matrix within the EKF. By establishing a reward mechanism that incentivizes actions minimizing the prediction error relative to true measurements, RL dynamically optimizes the measurement noise covariance matrix. This innovative integration of RL and EKF, referred to as RL-EKF, has been implemented and tested. Results demonstrate that RLEKF significantly outperforms the traditional EKF, yielding marked improvements in attitude estimation accuracy, the improvement ratios showed that selected method is very effective in the field of attitude estimation.

беспилотный летательный аппаратобучение с подкреплениемобобщенный фильтр Калманаоценивание ориентацииUnmanned Aerial VehicleReinforcement LearningExtended Kalman filterAttitude estimationReferencesNaeem, M., Rizvi, S. T. H., and Coronato, A., A gentle introduction to reinforcement learning and its application in different fields, IEEE Access, 2020, vol. 8, pp. 209320–209344.Naeem, M., Rizvi, S. T. H., and Coronato, A., A gentle introduction to reinforcement learning and its application in different fields, IEEE Access, 2020, vol. 8, pp. 209320–209344.Du, W., and Ding, S., A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications, Artificial Intelligence Review, 2021, vol. 54, no. 5, pp. 3215–3238.Du, W., and Ding, S., A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications, Artificial Intelligence Review, 2021, vol. 54, no. 5, pp. 3215–3238.Goslinski, J., Giernacki, W., and Krolikowski, A., A nonlinear filter for efficient attitude estimation of unmanned aerial vehicle (UAV), Journal of Intelligent & Robotic Systems, 2019, vol. 95, pp. 1079–1095.Goslinski, J., Giernacki, W., and Krolikowski, A., A nonlinear filter for efficient attitude estimation of unmanned aerial vehicle (UAV), Journal of Intelligent & Robotic Systems, 2019, vol. 95, pp. 1079–1095.Jing, X., Cui, J., He, H., Zhang, B., Ding, D., and Yang, Y., Attitude estimation for UAV using extended Kalman filter, Proc. 2017 29th Chinese Control And Decision Conference (CCDC), May 2017, pp. 3307–3312.Jing, X., Cui, J., He, H., Zhang, B., Ding, D., and Yang, Y., Attitude estimation for UAV using extended Kalman filter, Proc. 2017 29th Chinese Control And Decision Conference (CCDC), May 2017, pp. 3307–3312.Assad, A., Khalaf, W., and Chouaib, I., Novel adaptive fuzzy extended Kalman filter for attitude estimation in GPS-denied environment, Gyroscopy and Navigation, 2019, vol. 10, no. 3, pp. 131–146.Assad, A., Khalaf, W., and Chouaib, I., Novel adaptive fuzzy extended Kalman filter for attitude estimation in GPS-denied environment, Gyroscopy and Navigation, 2019, vol. 10, no. 3, pp. 131–146.Crassidis, J. L., Spacecraft attitude determination, in Encyclopedia of Systems and Control, Cham: Springer International Publishing, 2021, pp. 2097–2104.Crassidis, J. L., Spacecraft attitude determination, in Encyclopedia of Systems and Control, Cham: Springer International Publishing, 2021, pp. 2097–2104.Xiong, K., Wei, C., and Zhang, H., Q-learning for noise covariance adaptation in extended Kalman filter, Asian Journal of Control, 2021, vol. 23, no. 4, pp. 1803–1816.Xiong, K., Wei, C., and Zhang, H., Q-learning for noise covariance adaptation in extended Kalman filter, Asian Journal of Control, 2021, vol. 23, no. 4, pp. 1803–1816.Dai, X., Nateghi, V., Fourati, H., and Prieur, C., Q-learning-based noise covariance adaptation in Kalman filter for MARG sensors attitude estimation, Proc. 2022 IEEE International Symposium on Inertial Sensors and Systems (INERTIAL), May 2022, pp. 1–6.Dai, X., Nateghi, V., Fourati, H., and Prieur, C., Q-learning-based noise covariance adaptation in Kalman filter for MARG sensors attitude estimation, Proc. 2022 IEEE International Symposium on Inertial Sensors and Systems (INERTIAL), May 2022, pp. 1–6.Pandey Y., Bhattacharyya R., Nath Singh Y. Robust Attitude Estimation with Quaternion Left-Invariant EKF and Noise Covariance Tuning, arXiv e-prints, 2024, pp. arXiv: 2409.11496.Pandey Y., Bhattacharyya R., Nath Singh Y. Robust Attitude Estimation with Quaternion Left-Invariant EKF and Noise Covariance Tuning, arXiv e-prints, 2024, pp. arXiv: 2409.11496.Odry, A., Fuller, R., Rudas, I. J., and Odry, P., Kalman filter for mobile-robot attitude estimation: Novel optimized and adaptive solutions, Mechanical Systems and Signal Processing, 2018, vol. 110, pp. 569–589.Odry, A., Fuller, R., Rudas, I. J., and Odry, P., Kalman filter for mobile-robot attitude estimation: Novel optimized and adaptive solutions, Mechanical Systems and Signal Processing, 2018, vol. 110, pp. 569–589.Odry, A., Kecskes, I., Sarcevic, P., Vizvari, Z., Toth, A., and Odry, P., A novel fuzzy-adaptive extended Kalman filter for real-time attitude estimation of mobile robots, Sensors, 2020, vol. 20, no. 3, p. 803.Odry, A., Kecskes, I., Sarcevic, P., Vizvari, Z., Toth, A., and Odry, P., A novel fuzzy-adaptive extended Kalman filter for real-time attitude estimation of mobile robots, Sensors, 2020, vol. 20, no. 3, p. 803.Hajiyev, C., and Soken, H. E., Robust adaptive unscented Kalman filter for attitude estimation of pico satellites, International Journal of Adaptive Control and Signal Processing, 2014, vol. 28, no. 2, pp. 107–120.Hajiyev, C., and Soken, H. E., Robust adaptive unscented Kalman filter for attitude estimation of pico satellites, International Journal of Adaptive Control and Signal Processing, 2014, vol. 28, no. 2, pp. 107–120.Qiu, Y., Su, Y., and He, X., An improved IMM-KF for UAV position prediction, Proc. Third International Conference on Control and Intelligent Robotics (ICCIR 2023), December 2023, vol. 12940, pp. 7–16.Qiu, Y., Su, Y., and He, X., An improved IMM-KF for UAV position prediction, Proc. Third International Conference on Control and Intelligent Robotics (ICCIR 2023), December 2023, vol. 12940, pp. 7–16.Parwana, H., and Kothari, M., Quaternions and attitude representation, arXiv preprint arXiv:1708.08680, 2017.Parwana, H., and Kothari, M., Quaternions and attitude representation, arXiv preprint arXiv:1708.08680, 2017.Ribeiro, M. I., Kalman and extended Kalman filters: Concept, derivation and properties, Institute for Systems and Robotics, 2004, vol. 43(46), pp. 3736–3741.Ribeiro, M. I., Kalman and extended Kalman filters: Concept, derivation and properties, Institute for Systems and Robotics, 2004, vol. 43(46), pp. 3736–3741.Quan, W., Li, J., Gong, X., and Fang, J., INS/CNS/GNSS Integrated Navigation Technology, Springer, 2015.Quan, W., Li, J., Gong, X., and Fang, J., INS/CNS/GNSS Integrated Navigation Technology, Springer, 2015.Chouaib, A.I., Wainakh, B.M., and Khalaf, C.W., Robust self-corrective initial alignment algorithm for strap-down INS, Proc. 10th Asian Control Conference (ASCC), 2015, pp. 1–6.Chouaib, A.I., Wainakh, B.M., and Khalaf, C.W., Robust self-corrective initial alignment algorithm for strap-down INS, Proc. 10th Asian Control Conference (ASCC), 2015, pp. 1–6.Trawny, N., Roumeliotis, S.I., Indirect Kalman filter for 3D attitude estimation, University of Minnesota, Dept. of Comp. Sci. & Eng., Tech. Rep., 2005, vol 2, p. 2005.Trawny, N., Roumeliotis, S.I., Indirect Kalman filter for 3D attitude estimation, University of Minnesota, Dept. of Comp. Sci. & Eng., Tech. Rep., 2005, vol 2, p. 2005.Center, N.G.D., NCEI geomagnetic calculators [Электронный ресурс]. URL: https://www.ngdc.noaa. gov/geomag/calculators/magcalc.shtml (дата обращения: 01.04.2025).Center, N.G.D., NCEI geomagnetic calculators [Электронный ресурс]. URL: https://www.ngdc.noaa. gov/geomag/calculators/magcalc.shtml (дата обращения: 01.04.2025).Wang, J., Liu, Y., and Li, B., Reinforcement learning with perturbed rewards, Proc. of the AAAI Conference on Artificial Intelligence, 2020, vol. 34, no. 4, pp. 6202–6209.Wang, J., Liu, Y., and Li, B., Reinforcement learning with perturbed rewards, Proc. of the AAAI Conference on Artificial Intelligence, 2020, vol. 34, no. 4, pp. 6202–6209.Icarte, R. T., Klassen, T. Q., Valenzano, R., and Mcllraith, S. A., Reward machines: Exploiting reward function structure in reinforcement learning, Journal of Artifi Intelligence Research, 2022, vol. 73, pp. 173–208.Icarte, R. T., Klassen, T. Q., Valenzano, R., and Mcllraith, S. A., Reward machines: Exploiting reward function structure in reinforcement learning, Journal of Artifi Intelligence Research, 2022, vol. 73, pp. 173–208.

The authors declare that there are no conflicts of interest present.