# Deep Learning-based Inertial Navigation: A Hybrid Navigation Filter | by Barak Or | Sep, 2022

**The inertial navigation domain is considered a very classical one. It aims to provide us with navigation solutions (position, velocity, and orientation) using (low-graded) inertial sensors and accurate (low frequency) sensors, like the GPS receiver. A question has arisen about how powerful Deep Learning tools can boost this classical domain**.

Three years ago, I started asking myself if there is a value in combining Machine Learning, generally, and Deep Learning (DL), particularly, in the classical inertial navigation system (INS). After 3 years in the industry, I started my Ph.D. studies at the University of Haifa, including working for Qualcomm and Autotalks. In these stations, I have revolved around many fascinating solutions that the DL accelerated. For example, I worked on Qualcomm’s fingerprint sensors, which currently exist in many smartphone devices.

Being a part of this revolution made me think a lot about my previous education: I graduated from the Technion- The Israeli Institute of Technology with 3 degrees: M.Sc. and B.Sc. in Aerospace Engineering and B.A. in Economics and Management (Cum Laude). I used to focus mainly on optimal control and estimation for real-time navigation and tracking purposes. It seems natural to stay in this field and integrate DL as a part of my Ph.D., titled “Machine Learning-based Inertial Navigation”.

The classical INS combines accelerometer and gyroscope measurements into kinematic equations. Eventually, the navigation solution is achieved. But, these two sensors are very noisy, making the solution bad and suffering from fast divergence. Hence, it is most likely to fuse the classical INS with an accurate sensor. For example, the global positioning system (GPS) receiver. These kinds of fusion schemes are known as INS/GPS. The big question is, “How to integrate all sensors together to achieve the most accurate navigation solution?” The answer to that involves the famous signal processing algorithm, the Kalman filter (KF). The KF is the best-known algorithm for inertial navigation sensor fusion tasks. Most of the time, the extended KF (EKF) version is used as the navigation equations involve nonlinearity.

I presented a problem and a solution -so where is the DL combined? The answer is simple. The navigation filter, or generally any EKF, must have its own parameters to work optimally.

In most of the estimation books, you will probably find that the KF is an optimal filter in the sense of minimizing the square root error (MSE) between the state and the prediction of these states (the state is a vector that could be built of 15 states: position and velocity in 3 dimensions plus the body orientation angles (3 more). There are additional six states for the accelerometer and gyro biases. But we want to minimize the error between the **true state** vector and the prediction state vector. In real life, we don’t have an access to this true state unless using a very accurate device (i.e. Real Time Kinematic Device).

As the filter designer responsible for the dynamic model for the entire navigation scenario, he might be wrong with the state and its equations (transition matrix). Remember, this is just a kinematic **model**** **and not the **true **kinematic itself! To be optimal, the designer must design an accurate dynamic model (later, we will also discuss the measurement model). As no one knows the future, the great KF allows us to add some weight to our uncertainty in the modeling phase. This uncertainty is considered during the system model noise covariance matrix. For each state, we can put some values, representing white Gaussian noise variances, to help the KF handle some dynamics it will meet during the real-time navigation.

So, KF is optimal in the sense of MSE between the model state and the estimated state. The KF is not optimal in the sense of MSE between the true state and the estimated state…we need to continue tuning its parameters in a real-time setting, depending on the environment we navigate.

K

F is optimalin the sense of MSE between the model state and the estimated state.The KF is not optimalin the sense of MSE between the true state and the estimated state…we need tokeep tuning its parameters in a real-time setting.

So, here is the part where the DL gets involved. Let’s look at the diagram below: the EKF-based navigation filter is provided by the GPS and INS inputs (green and purple, respectively) and eventually provides an accurate navigation solution. The DL is used to learn the tuning parameters of the EKF from the inertial measurements (accelerometer and gyroscope). These measurements are provided in a real-time setting, so the DL model also tunes these parameters in a real time manner. For example, there is a level of uncertainty in the dynamic model of the INS. This uncertainty is due to the noise, disturbances, and additional factors in the accelerometer and gyroscope. These factors are changed during the vehicle’s navigation and lead to degraded navigation performance. To fix it, there is a need to tune the uncertainty of the dynamic model, accordingly.

One way to do that is by tuning the noise covariance matrix of the dynamic model (Q). The matrix values influence the magnitude of the white Gaussian noise we inject into the model. Hence, as the value increases, the uncertainty of the designer increases and vice versa. There are many classical adaptive approaches, considering entities of the EKF (such as the innovation property) to tune it. In this suggested **Hybrid Approach**, we rely on a trained model from a simulative dataset with many examples of noisy signals and their related variances (labels).

Once designing such a hybrid navigation filer model, one needs to consider many technical details regarding the DL model. A partial list includes the length of the signal (example), deciding to include some feature engineering or not, frequency of updating the tuning parameters. In our most recent paper, “A Hybrid Model and Learning-Based Adaptive Navigation Filter” written by Barak Or and Itzik Klein and published in IEEE Transactions on Instrumentation and Measurement (a preprint is available here), we thought about these details.

This work is the first of its kind in the hybrid navigation model, where the classical Kalman Filter is considered, aside from a deep learning model — used to tune the filter parameters. The combination yields great improvement in some of the case studies we tested. One included quadcopter navigation, where our Hybrid approach yielded a 27% improvement in position accuracy.

*If you want to read the original paper, it can be found on the IEEE **website**:*

[1] Or, Barak, and Itzik Klein. “A Hybrid Model and Learning-Based Adaptive Navigation Filter.” *IEEE Transactions on Instrumentation and Measurement* 71 (2022): 1–11.

*If you want to keep reading about the problem of identifying the noise covariance parameters you may refer to this work on the IEEE **website**:*

[2] Zhang, Lingyi, David Sidoti, Adam Bienkowski, Krishna R. Pattipati, Yaakov Bar-Shalom, and David L. Kleinman. “On the identification of noise covariances and adaptive Kalman filtering: A new look at a 50-year-old problem.” *IEEE Access* 8 (2020): 59362–59388.

*if you want to keep reading about deep learning for inertial sensors, I recommend my previous post **here on Medium**:*

[3] Barak Or. “Deep Learning for Inertial Navigation: A short review of cutting edge deep learning-based solutions for inertial navigation.” 2020.

Barak is a researcher in the fields of artificial intelligence and sensor fusion. Barak has authored several patents and articles that have been published in professional journals. He is the founder of ALMA Tech. LTD, an AI & advanced navigation company. He was with Qualcomm 2019–2020, where he mainly dealt with DSP and machine learning algorithms. Prior to that, he led the localization project at Autotalks. He received the M.Sc. (2018) and B.Sc. (2016) degrees in Aerospace Engineering, as also a B.A. in Economics and Management (2016, Cum Laude) from the Technion, Israel Institute of Technology. Barak is currently completing his Ph.D. at the University of Haifa, Israel.

**The inertial navigation domain is considered a very classical one. It aims to provide us with navigation solutions (position, velocity, and orientation) using (low-graded) inertial sensors and accurate (low frequency) sensors, like the GPS receiver. A question has arisen about how powerful Deep Learning tools can boost this classical domain**.

Three years ago, I started asking myself if there is a value in combining Machine Learning, generally, and Deep Learning (DL), particularly, in the classical inertial navigation system (INS). After 3 years in the industry, I started my Ph.D. studies at the University of Haifa, including working for Qualcomm and Autotalks. In these stations, I have revolved around many fascinating solutions that the DL accelerated. For example, I worked on Qualcomm’s fingerprint sensors, which currently exist in many smartphone devices.

Being a part of this revolution made me think a lot about my previous education: I graduated from the Technion- The Israeli Institute of Technology with 3 degrees: M.Sc. and B.Sc. in Aerospace Engineering and B.A. in Economics and Management (Cum Laude). I used to focus mainly on optimal control and estimation for real-time navigation and tracking purposes. It seems natural to stay in this field and integrate DL as a part of my Ph.D., titled “Machine Learning-based Inertial Navigation”.

The classical INS combines accelerometer and gyroscope measurements into kinematic equations. Eventually, the navigation solution is achieved. But, these two sensors are very noisy, making the solution bad and suffering from fast divergence. Hence, it is most likely to fuse the classical INS with an accurate sensor. For example, the global positioning system (GPS) receiver. These kinds of fusion schemes are known as INS/GPS. The big question is, “How to integrate all sensors together to achieve the most accurate navigation solution?” The answer to that involves the famous signal processing algorithm, the Kalman filter (KF). The KF is the best-known algorithm for inertial navigation sensor fusion tasks. Most of the time, the extended KF (EKF) version is used as the navigation equations involve nonlinearity.

I presented a problem and a solution -so where is the DL combined? The answer is simple. The navigation filter, or generally any EKF, must have its own parameters to work optimally.

In most of the estimation books, you will probably find that the KF is an optimal filter in the sense of minimizing the square root error (MSE) between the state and the prediction of these states (the state is a vector that could be built of 15 states: position and velocity in 3 dimensions plus the body orientation angles (3 more). There are additional six states for the accelerometer and gyro biases. But we want to minimize the error between the **true state** vector and the prediction state vector. In real life, we don’t have an access to this true state unless using a very accurate device (i.e. Real Time Kinematic Device).

As the filter designer responsible for the dynamic model for the entire navigation scenario, he might be wrong with the state and its equations (transition matrix). Remember, this is just a kinematic **model**** **and not the **true **kinematic itself! To be optimal, the designer must design an accurate dynamic model (later, we will also discuss the measurement model). As no one knows the future, the great KF allows us to add some weight to our uncertainty in the modeling phase. This uncertainty is considered during the system model noise covariance matrix. For each state, we can put some values, representing white Gaussian noise variances, to help the KF handle some dynamics it will meet during the real-time navigation.

So, KF is optimal in the sense of MSE between the model state and the estimated state. The KF is not optimal in the sense of MSE between the true state and the estimated state…we need to continue tuning its parameters in a real-time setting, depending on the environment we navigate.

K

F is optimalin the sense of MSE between the model state and the estimated state.The KF is not optimalin the sense of MSE between the true state and the estimated state…we need tokeep tuning its parameters in a real-time setting.

So, here is the part where the DL gets involved. Let’s look at the diagram below: the EKF-based navigation filter is provided by the GPS and INS inputs (green and purple, respectively) and eventually provides an accurate navigation solution. The DL is used to learn the tuning parameters of the EKF from the inertial measurements (accelerometer and gyroscope). These measurements are provided in a real-time setting, so the DL model also tunes these parameters in a real time manner. For example, there is a level of uncertainty in the dynamic model of the INS. This uncertainty is due to the noise, disturbances, and additional factors in the accelerometer and gyroscope. These factors are changed during the vehicle’s navigation and lead to degraded navigation performance. To fix it, there is a need to tune the uncertainty of the dynamic model, accordingly.

One way to do that is by tuning the noise covariance matrix of the dynamic model (Q). The matrix values influence the magnitude of the white Gaussian noise we inject into the model. Hence, as the value increases, the uncertainty of the designer increases and vice versa. There are many classical adaptive approaches, considering entities of the EKF (such as the innovation property) to tune it. In this suggested **Hybrid Approach**, we rely on a trained model from a simulative dataset with many examples of noisy signals and their related variances (labels).

Once designing such a hybrid navigation filer model, one needs to consider many technical details regarding the DL model. A partial list includes the length of the signal (example), deciding to include some feature engineering or not, frequency of updating the tuning parameters. In our most recent paper, “A Hybrid Model and Learning-Based Adaptive Navigation Filter” written by Barak Or and Itzik Klein and published in IEEE Transactions on Instrumentation and Measurement (a preprint is available here), we thought about these details.

This work is the first of its kind in the hybrid navigation model, where the classical Kalman Filter is considered, aside from a deep learning model — used to tune the filter parameters. The combination yields great improvement in some of the case studies we tested. One included quadcopter navigation, where our Hybrid approach yielded a 27% improvement in position accuracy.

*If you want to read the original paper, it can be found on the IEEE **website**:*

[1] Or, Barak, and Itzik Klein. “A Hybrid Model and Learning-Based Adaptive Navigation Filter.” *IEEE Transactions on Instrumentation and Measurement* 71 (2022): 1–11.

*If you want to keep reading about the problem of identifying the noise covariance parameters you may refer to this work on the IEEE **website**:*

[2] Zhang, Lingyi, David Sidoti, Adam Bienkowski, Krishna R. Pattipati, Yaakov Bar-Shalom, and David L. Kleinman. “On the identification of noise covariances and adaptive Kalman filtering: A new look at a 50-year-old problem.” *IEEE Access* 8 (2020): 59362–59388.

*if you want to keep reading about deep learning for inertial sensors, I recommend my previous post **here on Medium**:*

[3] Barak Or. “Deep Learning for Inertial Navigation: A short review of cutting edge deep learning-based solutions for inertial navigation.” 2020.

Barak is a researcher in the fields of artificial intelligence and sensor fusion. Barak has authored several patents and articles that have been published in professional journals. He is the founder of ALMA Tech. LTD, an AI & advanced navigation company. He was with Qualcomm 2019–2020, where he mainly dealt with DSP and machine learning algorithms. Prior to that, he led the localization project at Autotalks. He received the M.Sc. (2018) and B.Sc. (2016) degrees in Aerospace Engineering, as also a B.A. in Economics and Management (2016, Cum Laude) from the Technion, Israel Institute of Technology. Barak is currently completing his Ph.D. at the University of Haifa, Israel.

**Denial of responsibility!**Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.