Adaptation Noise Covariance in Extended Kalman Filter Using Reinforcement Learning for Improved UAV Attitude Estimation
EDN: LIQJHA
Abstract
Accurate attitude determination of Unmanned Aerial Vehicles (UAVs) is crucial for autonomous navigation, particularly when relying solely on gyroscope, accelerometer, and magnetometer measurements without utilizing the Global Positioning System (GPS). Reinforcement Learning (RL) has emerged as a promising artificial intelligence technique applicable across various domains. This research introduces a novel approach that leverages RL to enhance the performance of the Extended Kalman Filter (EKF) in attitude estimation. The proposed method depends of RL which uses the Q-Learning model and policy to find best solution to adjust autonomously the measurement noise covariance matrix within the EKF. By establishing a reward mechanism that incentivizes actions minimizing the prediction error relative to true measurements, RL dynamically optimizes the measurement noise covariance matrix. This innovative integration of RL and EKF, referred to as RL-EKF, has been implemented and tested. Results demonstrate that RLEKF significantly outperforms the traditional EKF, yielding marked improvements in attitude estimation accuracy, the improvement ratios showed that selected method is very effective in the field of attitude estimation.
About the Authors
A. Assad
St. Petersburg State University of Aerospace Instrumentation
Russian Federation
Russian Federation
St. Petersburg
S. A. Serikov
St. Petersburg State University of Aerospace Instrumentation
Russian Federation
Russian Federation
St. Petersburg
References
1. Naeem, M., Rizvi, S. T. H., and Coronato, A., A gentle introduction to reinforcement learning and its application in different fields, IEEE Access, 2020, vol. 8, pp. 209320–209344.
2. Du, W., and Ding, S., A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications, Artificial Intelligence Review, 2021, vol. 54, no. 5, pp. 3215–3238.
3. Goslinski, J., Giernacki, W., and Krolikowski, A., A nonlinear filter for efficient attitude estimation of unmanned aerial vehicle (UAV), Journal of Intelligent & Robotic Systems, 2019, vol. 95, pp. 1079–1095.
4. Jing, X., Cui, J., He, H., Zhang, B., Ding, D., and Yang, Y., Attitude estimation for UAV using extended Kalman filter, Proc. 2017 29th Chinese Control And Decision Conference (CCDC), May 2017, pp. 3307–3312.
5. Assad, A., Khalaf, W., and Chouaib, I., Novel adaptive fuzzy extended Kalman filter for attitude estimation in GPS-denied environment, Gyroscopy and Navigation, 2019, vol. 10, no. 3, pp. 131–146.
6. Crassidis, J. L., Spacecraft attitude determination, in Encyclopedia of Systems and Control, Cham: Springer International Publishing, 2021, pp. 2097–2104.
7. Xiong, K., Wei, C., and Zhang, H., Q-learning for noise covariance adaptation in extended Kalman filter, Asian Journal of Control, 2021, vol. 23, no. 4, pp. 1803–1816.
8. Dai, X., Nateghi, V., Fourati, H., and Prieur, C., Q-learning-based noise covariance adaptation in Kalman filter for MARG sensors attitude estimation, Proc. 2022 IEEE International Symposium on Inertial Sensors and Systems (INERTIAL), May 2022, pp. 1–6.
9. Pandey Y., Bhattacharyya R., Nath Singh Y. Robust Attitude Estimation with Quaternion Left-Invariant EKF and Noise Covariance Tuning, arXiv e-prints, 2024, pp. arXiv: 2409.11496.
10. Odry, A., Fuller, R., Rudas, I. J., and Odry, P., Kalman filter for mobile-robot attitude estimation: Novel optimized and adaptive solutions, Mechanical Systems and Signal Processing, 2018, vol. 110, pp. 569–589.
11. Odry, A., Kecskes, I., Sarcevic, P., Vizvari, Z., Toth, A., and Odry, P., A novel fuzzy-adaptive extended Kalman filter for real-time attitude estimation of mobile robots, Sensors, 2020, vol. 20, no. 3, p. 803.
12. Hajiyev, C., and Soken, H. E., Robust adaptive unscented Kalman filter for attitude estimation of pico satellites, International Journal of Adaptive Control and Signal Processing, 2014, vol. 28, no. 2, pp. 107–120.
13. Qiu, Y., Su, Y., and He, X., An improved IMM-KF for UAV position prediction, Proc. Third International Conference on Control and Intelligent Robotics (ICCIR 2023), December 2023, vol. 12940, pp. 7–16.
14. Parwana, H., and Kothari, M., Quaternions and attitude representation, arXiv preprint arXiv:1708.08680, 2017.
15. Ribeiro, M. I., Kalman and extended Kalman filters: Concept, derivation and properties, Institute for Systems and Robotics, 2004, vol. 43(46), pp. 3736–3741.
16. Quan, W., Li, J., Gong, X., and Fang, J., INS/CNS/GNSS Integrated Navigation Technology, Springer, 2015.
17. Chouaib, A.I., Wainakh, B.M., and Khalaf, C.W., Robust self-corrective initial alignment algorithm for strap-down INS, Proc. 10th Asian Control Conference (ASCC), 2015, pp. 1–6.
18. Trawny, N., Roumeliotis, S.I., Indirect Kalman filter for 3D attitude estimation, University of Minnesota, Dept. of Comp. Sci. & Eng., Tech. Rep., 2005, vol 2, p. 2005.
19. Center, N.G.D., NCEI geomagnetic calculators [Электронный ресурс]. URL: https://www.ngdc.noaa. gov/geomag/calculators/magcalc.shtml (дата обращения: 01.04.2025).
20. Wang, J., Liu, Y., and Li, B., Reinforcement learning with perturbed rewards, Proc. of the AAAI Conference on Artificial Intelligence, 2020, vol. 34, no. 4, pp. 6202–6209.
21. Icarte, R. T., Klassen, T. Q., Valenzano, R., and Mcllraith, S. A., Reward machines: Exploiting reward function structure in reinforcement learning, Journal of Artifi Intelligence Research, 2022, vol. 73, pp. 173–208.
Review
For citations:
Assad A., Serikov S.A. Adaptation Noise Covariance in Extended Kalman Filter Using Reinforcement Learning for Improved UAV Attitude Estimation. Giroskopiya i Navigatsiya. 2025;33(3):33-50. (In Russ.) EDN: LIQJHA
Views:
46
Email this article 