gyroscopyГироскопия и навигацияGiroskopiya i Navigatsiya / Gyroscopy and Navigation0869-70352075-0927AO «Концерн «ЦНИИ «Электроприбор»RQVMYAgyroscopy-41Research ArticleСтатьиИспользование ключевых точек и линий изображения в задаче визуально-инерциальной одометрии в реальном времениReal-time Visual-Inertial Odometry based on Point-Line Feature FusionЯнГ.YangGang

Ян Ган, доктор наук, доцент

Сиань

Xi’an

МэнВ.-Д.MengWeiDa

Мэн Вэй-Да, студент магистратуры

Сиань

Xi’an

ДунХ.-Г.HouGuoDong

Дун Хоу-Го, студент магистратуры

Сиань

Xi’an

ФэнН.-Н.FengNingNing

Фэн Нин-Нин, студент магистратуры

Сиань

Xi’an

Сианьский университет связи и телекоммуникацийКитайXi’an University of Posts and TelecommunicationsChina20231805202531496117Copyright © Ян Г., Мэн В., Дун Х., Фэн Н., 20252025Ян Г., Мэн В., Дун Х., Фэн Н.Yang G., Meng W., Hou G., Feng N.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.gyroscopy.ru/jour/article/view/41

С целью повышения точности навигации подвижного объекта и надежности извлечения информации из изображений в монокулярных системах на основе методов одновременной локализации и построения карты окружающего пространства (simultaneous localization and mapping – SLAM) при слабо выраженной текстуре сцены предложен метод визуально-инерциальной одометрии с использованием ключевых точек и линий на изображениях. Преимущества метода заключаются в простоте обнаружения линий в реальном пространстве, а также в высокой точности. Комбинирование ключевых точек и линий обеспечивает точное позиционирование SLAM-системы в пространстве со слабо выраженной текстурой, а используемые при этом данные инерциального измерительного модуля (ИИМ) формируют априорную информацию и информацию о масштабе. Оптимизация определения положения производится за счет уменьшения ошибки точечно-линейной репроекции и погрешностей ИИМ методом группового выравнивания точек (bundle adjustment). Предлагается усовершенствованный алгоритм EDlines, который позволяет контролировать длину последовательности пикселей для повышения эффективности распознавания линий и уменьшения их рассогласования. Результаты экспериментов на общедоступных базах данных EuRoC и TUM RGB-D подтверждают работоспособность алгоритма в режиме реального времени, его более высокую точность локализации и робастность по сравнению с визуальными SLAM-методами на основе только ключевых точек и только линий.

To improve the localization accuracy and tracking robustness of monocular feature-based visual SLAM systems in low-texture environments, a visual-inertial odometry method combining line features and point features is proposed, taking advantage of the easy avail ability of line features in real-world environments and the high accuracy of feature-based methods. The combination of point and line features ensures accurate positioning of the SLAM system in low-texture environments, while the inclusion of IMU data provides prior information and scale information. The pose is optimized by minimizing the reprojection error of point and line features and the IMU error using bundle adjustment. An improved EDlines algorithm is introduced, which incorporates a pixel chain length suppression pro cess to enhance the effectiveness of extracted line features and reduce the rate of line fea ture misalignment. Experimental results on the public EuRoC dataset and TUM RGB-D dataset show that the proposed method meets the real-time requirements and has higher localization accuracy and robustness compared with the visual SLAM method based on single point feature or the method adding traditional line features.

метод одновременной локализации и построения карты пространства (SLAM-метод)визуально-инерциальная одометрияключевые точкилинейные элементынелинейная оптимизацияsimultaneous localization and mappingvisual-inertial odometrypoint-line featuresnonlinear optimizationReferencesLiu, Y.J., Zhang, Y.Z., Rong, L., Jiang, H., and Deng, Y., Visual odometry based on the direct method and the inertial measurement unit, Robot, 2019, vol. 41, no. 5, pp. 683–689. https://doi.org/10.13973/j.cnki.robot.180601Liu, Y.J., Zhang, Y.Z., Rong, L., Jiang, H., and Deng, Y., Visual odometry based on the direct method and the inertial measurement unit, Robot, 2019, vol. 41, no. 5, pp. 683–689. https://doi.org/10.13973/j.cnki.robot.180601Pan, L.H., Tian, F.Q., Ying, W.J., Liang, W.G., and She, B., VI-SLAM algorithm with camera IMU extrinsic automatic calibration and online estimation, Chinese Journal of Scientific Instrument, 2019, vol. 40, no. 6, pp. 56–67. https://doi.org/10.19650/j.cnki.cjsi.J1904954 3. 3. Mourikis, A.I., and Roumeliotis, S.I., A multi-state constraint Kalman filter for vision-aided iner tial navigation, Proc. IEEE International Conference on Robotics and Automation (ICRA), 2007, pp. 3565–3572. https://doi.org/10.1109/ROBOT.2007.364024Pan, L.H., Tian, F.Q., Ying, W.J., Liang, W.G., and She, B., VI-SLAM algorithm with camera IMU extrinsic automatic calibration and online estimation, Chinese Journal of Scientific Instrument, 2019, vol. 40, no. 6, pp. 56–67. https://doi.org/10.19650/j.cnki.cjsi.J1904954 3. 3. Mourikis, A.I., and Roumeliotis, S.I., A multi-state constraint Kalman filter for vision-aided iner tial navigation, Proc. IEEE International Conference on Robotics and Automation (ICRA), 2007, pp. 3565–3572. https://doi.org/10.1109/ROBOT.2007.364024Mur-Artal, R., Montiel, J.M.M., and Tardós, J.D., ORB-SLAM: A versatile and accurate monoc ular SLAM system, IEEE Transactions on Robotics, 2015, vol. 31, no. 5, pp. 1147–1163. https://doi. org/10.1109/TRO.2015.2463671Mur-Artal, R., Montiel, J.M.M., and Tardós, J.D., ORB-SLAM: A versatile and accurate monoc ular SLAM system, IEEE Transactions on Robotics, 2015, vol. 31, no. 5, pp. 1147–1163. https://doi. org/10.1109/TRO.2015.2463671Mur-Artal, R., and Tardós, J.D., Visual-inertial monocular SLAM with map reuse, IEEE Robotics and Automation Letters, 2017, vol. 2, no. 2, pp. 796–803. https://doi.org/10.1109/LRA.2017.2653359Mur-Artal, R., and Tardós, J.D., Visual-inertial monocular SLAM with map reuse, IEEE Robotics and Automation Letters, 2017, vol. 2, no. 2, pp. 796–803. https://doi.org/10.1109/LRA.2017.2653359Qin, T., Li, P., and Shen, S., VINS-Mono: A robust and versatile monocular visual-inertial state estimator, IEEE Transactions on Robotics, 2018, vol. 34, no. 4, pp. 1004–1020. https://doi.org/10.1109/TRO.2018.2853729.Qin, T., Li, P., and Shen, S., VINS-Mono: A robust and versatile monocular visual-inertial state estimator, IEEE Transactions on Robotics, 2018, vol. 34, no. 4, pp. 1004–1020. https://doi.org/10.1109/TRO.2018.2853729.Pumarola, A., Vakhitov, A., Agudo, A., Sanfeliu, A., and Moreno-Noguer, F., PL-SLAM: Real time monocular visual SLAM with points and lines, Proc. IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 4503–4508. https://doi.org/10.1109/ICRA.2017.7989522Pumarola, A., Vakhitov, A., Agudo, A., Sanfeliu, A., and Moreno-Noguer, F., PL-SLAM: Real time monocular visual SLAM with points and lines, Proc. IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 4503–4508. https://doi.org/10.1109/ICRA.2017.7989522Gioi, R., Jakubowicz, J., Morel. J.-M., and Randall, G., LSD: A fast line segment detector with a false detection control, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, vol. 32, no. 4, pp. 722–732. https://doi.org/10.1109/TPAMI.2008.300Gioi, R., Jakubowicz, J., Morel. J.-M., and Randall, G., LSD: A fast line segment detector with a false detection control, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, vol. 32, no. 4, pp. 722–732. https://doi.org/10.1109/TPAMI.2008.300Gomez-Ojeda, R., Briales, J., and Gonzalez-Jimenez, J., PL-SVO: Semi-direct monocular visual odometry by combining points and line segments, Proc. IEEE/RSJ International Conference on Intel ligent Robots and Systems (IROS), 2016, pp. 4211–4216. https://doi.org/10.1109/IROS.2016.7759620Gomez-Ojeda, R., Briales, J., and Gonzalez-Jimenez, J., PL-SVO: Semi-direct monocular visual odometry by combining points and line segments, Proc. IEEE/RSJ International Conference on Intel ligent Robots and Systems (IROS), 2016, pp. 4211–4216. https://doi.org/10.1109/IROS.2016.7759620He, Y.J., Zhao, J., Gao, Y., He, W., and Yuan, K., PL-VIO: Tightly-coupled monocular visual inertial odometry using point and line features, Sensors, 2018, vol. 18, no. 4, pp. 1159–1184. https://doi.org/10.3390/s18041159He, Y.J., Zhao, J., Gao, Y., He, W., and Yuan, K., PL-VIO: Tightly-coupled monocular visual inertial odometry using point and line features, Sensors, 2018, vol. 18, no. 4, pp. 1159–1184. https://doi.org/10.3390/s18041159Fu, Q., Wang, J.L., Yu, H.H., Islam, A., Guo, F., and Zhang, H., PL-VINS: Real-time monoc ular visual-inertial SLAM with point and line, arXiv, 2020, preprint arXiv:2009.07462. https://doi.org/10.48550/arXiv.2009.07462Fu, Q., Wang, J.L., Yu, H.H., Islam, A., Guo, F., and Zhang, H., PL-VINS: Real-time monoc ular visual-inertial SLAM with point and line, arXiv, 2020, preprint arXiv:2009.07462. https://doi.org/10.48550/arXiv.2009.07462Shu, F.W., Wang, J.X., and Pagani, A., Structure PLP-SLAM: Efficient sparse mapping and local ization using point, line, and plane for monocular, RGB-D and stereo cameras, ArXiv, 2022, preprint ArXiv 2207 06058, https://doi.org/10.48550/arXiv.2207.06058Shu, F.W., Wang, J.X., and Pagani, A., Structure PLP-SLAM: Efficient sparse mapping and local ization using point, line, and plane for monocular, RGB-D and stereo cameras, ArXiv, 2022, preprint ArXiv 2207 06058, https://doi.org/10.48550/arXiv.2207.06058Yoon, S., and Kim, A., Line as a visual sentence: Context-aware line descriptor for visual localization, IEEE Robotics and Automation Letters, 2021, vol. 6, no. 4, pp. 8726–8733. https://doi.org/10.1109/ LRA.2021.3111760.Yoon, S., and Kim, A., Line as a visual sentence: Context-aware line descriptor for visual localization, IEEE Robotics and Automation Letters, 2021, vol. 6, no. 4, pp. 8726–8733. https://doi.org/10.1109/ LRA.2021.3111760.Yunus, R., Li, Y., and Tombari, F., Manhattan SLAM: Robust planar tracking and mapping leveraging mixture of Manhattan frames, Proc. IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 6687–6693. https://doi.org/10.1109/ICRA48506.2021.9562030.Yunus, R., Li, Y., and Tombari, F., Manhattan SLAM: Robust planar tracking and mapping leveraging mixture of Manhattan frames, Proc. IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 6687–6693. https://doi.org/10.1109/ICRA48506.2021.9562030.Wu, T.-H., and Chen, K.-W., LGC Net: Feature enhancement and consistency learning based on local and global coherence network for correspondence selection, Proc. IEEE International Conference on Robot ics and Automation (ICRA), 2023, pp. 6182–6188. https://doi.org/10.1109/ICRA48891.2023.10160290Wu, T.-H., and Chen, K.-W., LGC Net: Feature enhancement and consistency learning based on local and global coherence network for correspondence selection, Proc. IEEE International Conference on Robot ics and Automation (ICRA), 2023, pp. 6182–6188. https://doi.org/10.1109/ICRA48891.2023.10160290Akinlar, C., and Topal, C., Edlines: Real-time line segment detection by edge drawing, Proc. 18th IEEE International Conference on Image Processing (ICIP), 2011, pp. 2837–2840. https://doi. org/10.1016/j.patrec.2011.06.001Akinlar, C., and Topal, C., Edlines: Real-time line segment detection by edge drawing, Proc. 18th IEEE International Conference on Image Processing (ICIP), 2011, pp. 2837–2840. https://doi. org/10.1016/j.patrec.2011.06.001Lupton, T., and Sukkarieh, S., Visual-inertial-aided navigation for high-dynamic motion in built environments without initial conditions, IEEE Transactions on Robotics, 2012, vol. 28, no. 1, pp. 61–76. https://doi.org/10.1109/TRO.2011.2170332Lupton, T., and Sukkarieh, S., Visual-inertial-aided navigation for high-dynamic motion in built environments without initial conditions, IEEE Transactions on Robotics, 2012, vol. 28, no. 1, pp. 61–76. https://doi.org/10.1109/TRO.2011.2170332Forster, C., Carlone, L., Dellaert, F., and Scaramuzza, D., On-manifold pre-integration for re al-time visual-inertial odometry, IEEE Transactions on Robotics, 2017, vol. 33, no. 1, pp. 1–21. https://doi.org/10.1109/TRO.2016.2597321Forster, C., Carlone, L., Dellaert, F., and Scaramuzza, D., On-manifold pre-integration for re al-time visual-inertial odometry, IEEE Transactions on Robotics, 2017, vol. 33, no. 1, pp. 1–21. https://doi.org/10.1109/TRO.2016.2597321Mur-Artal, R., and Tardós, J.D., ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras, IEEE Transactions on Robotics, 2017, vol. 33, no. 5, pp. 1255–1262. https://doi.org/10.1109/TRO.2017.2705103.Mur-Artal, R., and Tardós, J.D., ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras, IEEE Transactions on Robotics, 2017, vol. 33, no. 5, pp. 1255–1262. https://doi.org/10.1109/TRO.2017.2705103.Kschischang, F.R., Frey, B.J., and Loeliger, H.-A., Factor graphs and the sum-product algorithm, IEEE Transactions on Information Theory, 2001, vol. 47, no. 2, pp. 498–519. https://doi.org/10.1109/18.910572.Kschischang, F.R., Frey, B.J., and Loeliger, H.-A., Factor graphs and the sum-product algorithm, IEEE Transactions on Information Theory, 2001, vol. 47, no. 2, pp. 498–519. https://doi.org/10.1109/18.910572.Dellaert, F., and Kaess, M., Square root SAM: Simultaneous localization and mapping via square root information smoothing, International Journal of Robotics Research, 2006, vol. 25, no. 12, pp. 1181–1203. https://doi.org/10.1177/0278364906072768Dellaert, F., and Kaess, M., Square root SAM: Simultaneous localization and mapping via square root information smoothing, International Journal of Robotics Research, 2006, vol. 25, no. 12, pp. 1181–1203. https://doi.org/10.1177/0278364906072768Kümmerle, R., Grisetti, G., Strasdat, H.M., Konolige, K., and Burgard, W., G2o: A general framework for graph optimization, Proc. IEEE International Conference on Robotics and Automa tion (ICRA), 2011, pp. 3607–3613. http://doi.org/10.1109%2FICRA.2011.5979949Kümmerle, R., Grisetti, G., Strasdat, H.M., Konolige, K., and Burgard, W., G2o: A general framework for graph optimization, Proc. IEEE International Conference on Robotics and Automa tion (ICRA), 2011, pp. 3607–3613. http://doi.org/10.1109%2FICRA.2011.5979949Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R., and Furgale, P., Keyframe-based visual inertial odometry using nonlinear optimization, International Journal of Robotics Research, 2015, vol. 34, no. 3, pp. 314–334. https://doi.org/10.1177/0278364914554813Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R., and Furgale, P., Keyframe-based visual inertial odometry using nonlinear optimization, International Journal of Robotics Research, 2015, vol. 34, no. 3, pp. 314–334. https://doi.org/10.1177/0278364914554813Burri, M., Nikolic, J., Gohl, P., Schneider, T., Rehder, J., Omari, S., Achtelik, M.W., and Siegwart, R., The EuRoC micro aerial vehicle datasets, International Journal of Robotics Research, 2016, vol. 40, no. 6, pp. 1157–1163. https://doi.org/10.1177/0278364915620033Burri, M., Nikolic, J., Gohl, P., Schneider, T., Rehder, J., Omari, S., Achtelik, M.W., and Siegwart, R., The EuRoC micro aerial vehicle datasets, International Journal of Robotics Research, 2016, vol. 40, no. 6, pp. 1157–1163. https://doi.org/10.1177/0278364915620033Sturm, J., Engelhard, N., Endres, F., Burgard, W., and Cremers, D., A benchmark for the eval uation of RGB-D SLAM systems, Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012, pp. 573–580. https://doi:10.1109/IROS.2012.6385773Sturm, J., Engelhard, N., Endres, F., Burgard, W., and Cremers, D., A benchmark for the eval uation of RGB-D SLAM systems, Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012, pp. 573–580. https://doi:10.1109/IROS.2012.6385773Campos, C., Elvira, R., Rodríguez, J.J.G., Montiel, J.M.M., and Tardós, J.D., ORB-SLAM3: An accurate open-source library for visual, visual-inertial, and multimap SLAM, IEEE Transactions on Robotics, 2021, vol. 37, no. 6, pp. 1874–1890. https://doi.org/10.1109/TRO.2021.3075644Campos, C., Elvira, R., Rodríguez, J.J.G., Montiel, J.M.M., and Tardós, J.D., ORB-SLAM3: An accurate open-source library for visual, visual-inertial, and multimap SLAM, IEEE Transactions on Robotics, 2021, vol. 37, no. 6, pp. 1874–1890. https://doi.org/10.1109/TRO.2021.3075644

The authors declare that there are no conflicts of interest present.