gyroscopyГироскопия и навигацияGiroskopiya i Navigatsiya0869-70352075-0927AO «Концерн «ЦНИИ «Электроприбор»10.17285/0869-7035.2019.27.2.028-051gyroscopy-242Research ArticleСтатьиОпределение дальности на основе данных о собственном движении, полученных с помощью монокулярной камерыDepth estimation with ego-motion assisted monocular cameraМансурМ.MansourM.

Мансур Мостафа. Аспирант

ДавидсонП.DavidsonP.

Давидсон Павел. Доктор наук, старший научный сотрудник

СтепановО. А.StepanovO. A.

Степанов Олег Андреевич. Доктор технических наук, профессор, начальник научно-образовательного центра. Вице-президент общественного объединения «Академия навигации и управления движением».

РауниоЮ.-П.RaunioJ.-P.

Раунио Юкка-Пекка. Старший инженер по НИОКР

АрефМ. М.ArefM.M.

Ареф Мохаммад М. Постдокторант

ПишеР.PichéR.

Пише Роберт. Профессор

Университет Тампере (Финляндия), Университет ИТМО (С.-Петербург).РоссияITMO University, St. Petersburg, Russia; Tampere University, FinlandRussian FederationУниверситет Тампере (Финляндия).ФинляндияTampere University, FinlandFinlandАО «Концерн «ЦНИИ «Электроприбор», Университет ИТМО (С.-Петербург).РоссияConcern CSRI Elektropribor, JSC; ITMO University, St. Petersburg, RussiaRussian FederationAAC Technologies (Тампере, Финляндия).ФинляндияAAC Technologies, Tampere, FinlandFinland2019141120252722851Copyright © Мансур М., Давидсон П., Степанов О.А., Раунио Ю., Ареф М., Пише Р., 20252025Мансур М., Давидсон П., Степанов О.А., Раунио Ю., Ареф М., Пише Р.Mansour M., Davidson P., Stepanov O.A., Raunio J., Aref M., Piché R.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.gyroscopy.ru/jour/article/view/242

Предложен метод определения дальности до объектов, основанный на взаимодополняющем характере последовательных монокулярных изображений и кинематических параметров камеры. Комплексирование измерений, полученных с помощью камеры, и кинематических параметров, которые определяются посредством инерциального измерительного модуля (ИИМ) и одометра, осуществляется с помощью обобщенного фильтра Калмана (ОФК). Результаты экспериментов с использованием колесного робота подтвердили результаты моделирования в части ожидаемой оценки точности определения дальности. Показано, что на эффективность предлагаемого метода значительное влияние оказывают взаимное расположение камеры и наблюдаемого объекта, точность измерения параметров движения камеры и пройденное им расстояние. Установлено, что при благоприятных условиях погрешность оценки дальности может составлять всего1% от расстояния до ориентира. Метод может быть использован для определения дальности до объектов, находящихся в нескольких сотнях метров от камеры.

We propose a method to estimate the distance to objects based on the complementary nature of monocular image sequences and camera kinematic parameters. The fusion of camera measurements with the kinematics parameters that are measured by an IMU and an odometer is performed using an extended Kalman filter. Results of field experiments with a wheeled robot corroborated the results of the simulation study in terms of accuracy of depth estimation. The performance of the approach in depth estimation is strongly affected by the mutual observer and feature point geometry, measurement accuracy of the observer’s motion parameters and distance covered by the observer. It was found that under favorable conditions the error in distance estimation can be as small as 1% of the distance to a feature point. This approach can be used to estimate distance to objects located hundreds of meters away from the camera.

Техническое зрениеопределение дальности по параметрам движенияобобщенный фильтр Калманапоследовательныеизображенияинерциальные датчикикомплексирование измерений датчиков.Сomputer visiondepth-from-motionextended Kalman filterimage sequenceinertial sensingsensor fusion.ReferencesHartley, R. and Zisserman, A., Multiple View Geometry in Computer Vision. Cambridge University Press, 2003.Hartley, R. and Zisserman, A., Multiple View Geometry in Computer Vision. Cambridge University Press, 2003.Özyesil, O., Voroninski, V., Basri, R., and Singer, A., A survey of structure from motion, Acta Numerica, 2017, vol. 26, pp. 305–364.Özyesil, O., Voroninski, V., Basri, R., and Singer, A., A survey of structure from motion, Acta Numerica, 2017, vol. 26, pp. 305–364.Matthies, L., Szeliski, R., and Kanade, T., Incremental estimation of dense depth maps from image sequences, Proc. Computer Society Conf. on Computer Vision and Pattern Recognition, CVPR’88, IEEE, 1988, pp. 366–374.Matthies, L., Szeliski, R., and Kanade, T., Incremental estimation of dense depth maps from image sequences, Proc. Computer Society Conf. on Computer Vision and Pattern Recognition, CVPR’88, IEEE, 1988, pp. 366–374.Matthies, L., Dynamic stereo vision, PhD Thesis, Carnegie Mellon University, 1989.Matthies, L., Dynamic stereo vision, PhD Thesis, Carnegie Mellon University, 1989.Hernandez, J., Tsotsos, K., and Soatto, S., Observability, identifiability and sensitivity of vision-aided inertial navigation, Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2015, pp. 2319–2325.Hernandez, J., Tsotsos, K., and Soatto, S., Observability, identifiability and sensitivity of vision-aided inertial navigation, Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2015, pp. 2319–2325.Longuet-Higgins, H.C., and Prazdny, K., The interpretation of a moving retinal image, Proc. of the Royal Society London Series B, 1980, vol. 208, no. 1173, pp. 385–397.Longuet-Higgins, H.C., and Prazdny, K., The interpretation of a moving retinal image, Proc. of the Royal Society London Series B, 1980, vol. 208, no. 1173, pp. 385–397.Corke, P., An inertial and visual sensing system for a small autonomous helicopter, Journal of Field Robotics, 2004, vol. 21, no. 2, pp. 43–51.Corke, P., An inertial and visual sensing system for a small autonomous helicopter, Journal of Field Robotics, 2004, vol. 21, no. 2, pp. 43–51.Strelow, D. and Singh, S., Online motion estimation from image and inertial measurements, Proc. Workshop on Integration of Vision and Inertial Sensors (INERVIS), Coimbra, Portugal, June 2003.Strelow, D. and Singh, S., Online motion estimation from image and inertial measurements, Proc. Workshop on Integration of Vision and Inertial Sensors (INERVIS), Coimbra, Portugal, June 2003.Strelow, D. and Singh, S., Motion estimation from image and inertial measurements, International Journal of Robotics Research, 2004, vol. 23, no. 12, pp. 1157–1195.Strelow, D. and Singh, S., Motion estimation from image and inertial measurements, International Journal of Robotics Research, 2004, vol. 23, no. 12, pp. 1157–1195.Corke, P., Lobo, J., and Dias, J., An introduction to inertial and visual sensing, International Journal of Robotics Research, 2007, vol. 26, no. 6, pp. 519–535.Corke, P., Lobo, J., and Dias, J., An introduction to inertial and visual sensing, International Journal of Robotics Research, 2007, vol. 26, no. 6, pp. 519–535.Grabe, V., Bulthoff, H.H., and Giordano, P.R., A comparison of scale estimation schemes for a quadrotor UAV based on optical flow and IMU measurements, Proc. Int. Conf. Intelligent Robots and Systems (IROS), IEEE, 2013, pp. 5193–5200.Grabe, V., Bulthoff, H.H., and Giordano, P.R., A comparison of scale estimation schemes for a quadrotor UAV based on optical flow and IMU measurements, Proc. Int. Conf. Intelligent Robots and Systems (IROS), IEEE, 2013, pp. 5193–5200.Delmerico, J. and Scaramuzza, D., A benchmark comparison of monocular visual-inertial odometry algorithms for flying robots, Memory, 2018, vol. 10, p. 20.Delmerico, J. and Scaramuzza, D., A benchmark comparison of monocular visual-inertial odometry algorithms for flying robots, Memory, 2018, vol. 10, p. 20.Huster, A. and Rock, S.M., Relative position sensing by fusing monocular vision and inertial rate sensors, Proc. 11th Int. Conf. Advanced Robotics (ICAR’03), Coimbra, Portugal, 2003, vol. 3, pp. 1562–1567.Huster, A. and Rock, S.M., Relative position sensing by fusing monocular vision and inertial rate sensors, Proc. 11th Int. Conf. Advanced Robotics (ICAR’03), Coimbra, Portugal, 2003, vol. 3, pp. 1562–1567.Huster A., Relative position sensing by fusing monocular vision and inertial rate sensors, PhD Thesis, Stanford University, 2003.Huster A., Relative position sensing by fusing monocular vision and inertial rate sensors, PhD Thesis, Stanford University, 2003.Dani, A., Fischer, N., Kan Z., and Dixon, W., Globally exponentially stable observer for vision-based range estimation, Mechatronics, 2012, vol. 22, no. 4, pp. 381–389.Dani, A., Fischer, N., Kan Z., and Dixon, W., Globally exponentially stable observer for vision-based range estimation, Mechatronics, 2012, vol. 22, no. 4, pp. 381–389.Spica, R., Giordano, P.R., and Chaumette, F., Plane estimation by active vision from point features and image moments, Proc. Int. Conf. Robotics and Automation (ICRA), 2015, pp. 6003–6010.Spica, R., Giordano, P.R., and Chaumette, F., Plane estimation by active vision from point features and image moments, Proc. Int. Conf. Robotics and Automation (ICRA), 2015, pp. 6003–6010.Spica, R., Giordano, P.R., and Chaumette, F., Active structure from motion: Application to point, sphere, and cylinder, IEEE Transactions on Robotics, 2014, vol. 30, no. 6, pp. 1499–1513.Spica, R., Giordano, P.R., and Chaumette, F., Active structure from motion: Application to point, sphere, and cylinder, IEEE Transactions on Robotics, 2014, vol. 30, no. 6, pp. 1499–1513.Tahri, O., Boutat, D., and Mezouar, Y., Brunovsky’s linear form of incremental structure from motion, IEEE Transactions on Robotics, 2017, vol. 33, no. 6, pp. 1491–1499.Tahri, O., Boutat, D., and Mezouar, Y., Brunovsky’s linear form of incremental structure from motion, IEEE Transactions on Robotics, 2017, vol. 33, no. 6, pp. 1491–1499.Martinelli, A., Vision and IMU data fusion: Closed-form solutions for attitude, speed, absolute scale, and bias determination, IEEE Transactions on Robotics, 2012, vol. 28, no. 1, pp. 44–60.Martinelli, A., Vision and IMU data fusion: Closed-form solutions for attitude, speed, absolute scale, and bias determination, IEEE Transactions on Robotics, 2012, vol. 28, no. 1, pp. 44–60.De Luca, A., Oriolo, G., and Robuffo Giordano, P., Feature depth observation for imagebased visual servoing: Theory and experiments, International Journal of Robotics Research, 2008, vol. 27, no. 10, pp. 1093–1116.De Luca, A., Oriolo, G., and Robuffo Giordano, P., Feature depth observation for imagebased visual servoing: Theory and experiments, International Journal of Robotics Research, 2008, vol. 27, no. 10, pp. 1093–1116.Davidson, P., Raunio, J.-P., and Piché, R., Monocular vision-based range estimation supported by proprioceptive motion, Gyroscopy and Navigation, 2017, vol. 8, no. 2, pp. 150–158.Davidson, P., Raunio, J.-P., and Piché, R., Monocular vision-based range estimation supported by proprioceptive motion, Gyroscopy and Navigation, 2017, vol. 8, no. 2, pp. 150–158.Azuma, R.T., A survey of augmented reality, Presence: Teleoperators & Virtual Environments, 1997, vol. 6, no. 4, pp. 355–385.Azuma, R.T., A survey of augmented reality, Presence: Teleoperators & Virtual Environments, 1997, vol. 6, no. 4, pp. 355–385.Bloesch, M., Omari, S., Hutter, M., and Siegwart, R., Robust visual inertial odometry using a direct EKF-based approach, Proc. Int. Conf. Intelligent Robots and Systems (IROS), IEEE, 2015, pp. 298–304.Bloesch, M., Omari, S., Hutter, M., and Siegwart, R., Robust visual inertial odometry using a direct EKF-based approach, Proc. Int. Conf. Intelligent Robots and Systems (IROS), IEEE, 2015, pp. 298–304.Matthies, L., Kanade, T., and Szeliski, R., Kalman filter-based algorithms for estimating depth from image sequences, International Journal of Computer Vision, 1989, vol. 3, no.3, pp. 209–238.Matthies, L., Kanade, T., and Szeliski, R., Kalman filter-based algorithms for estimating depth from image sequences, International Journal of Computer Vision, 1989, vol. 3, no.3, pp. 209–238.Landy, M.S., Maloney, L.T., Johnston, E.B., and Young, M., Measurement and modeling of depth cue combination: In defense of weak fusion, Vision Research, 1995, vol. 35, no. 3, pp. 389–412.Landy, M.S., Maloney, L.T., Johnston, E.B., and Young, M., Measurement and modeling of depth cue combination: In defense of weak fusion, Vision Research, 1995, vol. 35, no. 3, pp. 389–412.Corke, P., Robotics, Vision and Control: Fundamental Algorithms in MATLAB, vol. 73. Springer Science & Business Media, 2011.Corke, P., Robotics, Vision and Control: Fundamental Algorithms in MATLAB, vol. 73. Springer Science & Business Media, 2011.Montiel, J.M., Civera, J., and Davison, A.J., Unified inverse depth parametrization for monocular SLAM, Proc. Robotics: Science and Systems, 2006.Montiel, J.M., Civera, J., and Davison, A.J., Unified inverse depth parametrization for monocular SLAM, Proc. Robotics: Science and Systems, 2006.Davidson, P., Raunio, J.-P., and Piché, R., Accurate depth estimation from a sequence of monocular images supported by proprioceptive sensors, Proc. 23rd Int. Conf. on Integrated Navigation Systems, St. Petersburg, Russia, 2016, pp. 249–257.Davidson, P., Raunio, J.-P., and Piché, R., Accurate depth estimation from a sequence of monocular images supported by proprioceptive sensors, Proc. 23rd Int. Conf. on Integrated Navigation Systems, St. Petersburg, Russia, 2016, pp. 249–257.Davidson, P., Mansour, M., Stepanov, O.A., and Piché, R., Depth estimation from motion parallax: Experimental evaluation, Proc. 26th Int. Conf. on Integrated Navigation Systems, St. Petersburg, Russia, 2019.Davidson, P., Mansour, M., Stepanov, O.A., and Piché, R., Depth estimation from motion parallax: Experimental evaluation, Proc. 26th Int. Conf. on Integrated Navigation Systems, St. Petersburg, Russia, 2019.Stepanov, O.A., Optimal and sub-optimal filtering in integrated navigation systems, in Aerospace Navigation Systems; Nebylov, A., Watson, J., Eds., New York: John Wiley & Sons, Inc., 2016, pp. 392–446.Stepanov, O.A., Optimal and sub-optimal filtering in integrated navigation systems, in Aerospace Navigation Systems; Nebylov, A., Watson, J., Eds., New York: John Wiley & Sons, Inc., 2016, pp. 392–446.Särkkä, S., Bayesian Filtering and Smoothing, Cambridge University Press, 2010.Särkkä, S., Bayesian Filtering and Smoothing, Cambridge University Press, 2010.Oshman, Y. and Davidson, P., Optimal observer trajectories for passive target localization using bearing-only measurements, Proc. AIAA Guidance, Navigation, and Control Conference, San-Diego, CA, 1996, p. 3740.Oshman, Y. and Davidson, P., Optimal observer trajectories for passive target localization using bearing-only measurements, Proc. AIAA Guidance, Navigation, and Control Conference, San-Diego, CA, 1996, p. 3740.Oshman, Y. and Davidson, P., Optimization of observer trajectories for bearings-only target localization, IEEE Transactions on Aerospace and Electronic Systems, 1999, vol. 35, no. 3, pp. 892–902.Oshman, Y. and Davidson, P., Optimization of observer trajectories for bearings-only target localization, IEEE Transactions on Aerospace and Electronic Systems, 1999, vol. 35, no. 3, pp. 892–902.Bouguet, J.-Y., Camera Calibration Toolbox for Matlab, California Institute of Technology, Pasadena, CA, 2006.Bouguet, J.-Y., Camera Calibration Toolbox for Matlab, California Institute of Technology, Pasadena, CA, 2006.Geiger, A., Moosmann, F., Car, O., and Schuster, B., Automatic camera and range sensor calibration using a single shot, Proc. IEEE Int. Conf. Robotics and Automation, 2012, pp. 3936–3943.Geiger, A., Moosmann, F., Car, O., and Schuster, B., Automatic camera and range sensor calibration using a single shot, Proc. IEEE Int. Conf. Robotics and Automation, 2012, pp. 3936–3943.Shi, J. et al., Good features to track, Proc. Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 1994, pp. 593–600.Shi, J. et al., Good features to track, Proc. Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 1994, pp. 593–600.Lucas, B.D. and Kanade, T., An iterative image registration technique with an application to stereo vision, Proc. of Imaging Understanding Workshop, Vancouver, BC, Canada, 1981, pp. 121–130.Lucas, B.D. and Kanade, T., An iterative image registration technique with an application to stereo vision, Proc. of Imaging Understanding Workshop, Vancouver, BC, Canada, 1981, pp. 121–130.Bouguet, J.-Y., Pyramidal implementation of the affine Lucas-Kanade feature tracker: description of the algorithm, Intel Corporation, 2001, vol. 5, no. 1-10, p. 4.Bouguet, J.-Y., Pyramidal implementation of the affine Lucas-Kanade feature tracker: description of the algorithm, Intel Corporation, 2001, vol. 5, no. 1-10, p. 4.Gao, X.-S., Hou, X.-R., Tang, J. and Cheng, H.-F., Complete solution classification for the perspective-three-point problem, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, vol. 25, pp. 930–943.Gao, X.-S., Hou, X.-R., Tang, J. and Cheng, H.-F., Complete solution classification for the perspective-three-point problem, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, vol. 25, pp. 930–943.

The authors declare that there are no conflicts of interest present.