Democratizing Geometric AI for 360° Spherical Data | by Jason McEwen | Jul, 2022
Unlocking AI for 360° spherical data
While AI is now commonplace for standard types of data, such as structured, sequential and image data, the application of AI is severly curtailed for other more complex forms of data. These more complex datasets typical exhibit non-trivial geometry.
The field of geometric AI, or geometric deep learning, has emerged to extend the remarkable benefits of AI to these more complex — geometric — datasets [1] (for a brief introduction to geometric AI see our recent article). The use of geometric AI techniques, however, is still in its infancy since building and deploying geometric AI models remains difficult.
Some excellent geometric AI libraries and frameworks exists, mostly for graph geometric AI techniques (as discussed in this article). However, these libraries are relatively low-level, requiring an expert machine learning engineer with geometric AI expertise to build and train models, before deploying in a production environment. For other forms of geometric AI, much progress is in cutting edge research where general libraries are not yet available.
With CopernicAI we’re developing a low code platform for geometric AI so that non-experts can easily pick up geometric AI approaches for their own problems. Our first step in this direction is to make geometric AI models of 360° spherical data available.
Spherical data is in fact very common, arising in a great many fields. For example, spherical data can arise when observations are made at each point on a spherical surface, such as a topographic map of the Earth. However, they also arises when observations are made over directions, such as panoramic photos and videos captured by 360° cameras, e.g. for virtual reality, surveillance, or autonomous vehicles. Other applications include the analysis of the relic light from the Big Bang in cosmology or diffusion magnetic resonance imaging in medicine, and many others beyond.
At Kagenova we’re working to unlock the remarkable success of deep learning for these problems and others involving data with complex geometry, such as the sphere. For 360° spherical data standard AI techniques aren’t effective and we need geometric AI designed specifically for the spherical geometry of the data.
A lot of progress has been made in developing spherical AI techniques to unlock these applications (as discussed in our previous articles here and here). A significant challenge is to develop techniques that are computationally efficient and able to handle high-resolution data, such as high-resolution 360° images.
In our research we have made a lot of progress in this endeavour, firstly developing general efficient components [2], before introducing approaches to support high-resolution input data [3]. While support for high-resolution input data unlocks a lot of applications, such as classification problems, in many settings high-resolution output data must be supported. We’re currently working on support for high-resolution output data and will soon be releasing a related paper, which will open up a host of new applications in 360° image understanding, such as semantic segmentation.
Now that our geometric AI techniques for spherical data are maturing, we’re planning to make our approaches and models publicly available for anyone to use.
While work on our CopernicAI platform is still ongoing, in the interim we plan to make a number of spherical AI models available throught the AWS AI Marketplace.
We have just released our first model, Inception360, for the classification of 360° images.
This is a simple model, using an inception-based architecture [4], to perform ImageNet classification, i.e. classifying 360° images using the 1,000 ImageNet classes. ImageNet may not provide the best set of class labels for 360° classification since 360° images capture more content than standard 2D planar images; nevertheless, it provides a familiar set of class labels to get started. In this model we do not leverage our latest R&D developments but rather provide a very simple model to allow anyone to get going with 360° image classification immediately.
You can find Inception360 for free on the AWS AI Marketplace here. A notebook illustrating how to use Inception360 is available here.
Below we illustrate Inception360 classification on a (out-of-sample) 360° image. Consider the following 360° image of a dock and sailing boats.
dock, boathouse, gondola, schooner, yawl. [Original image source.]Inception360 returns the following (ordered top 5) classifications for this image, which match the image very well:
dock, boathouse, gondola, schooner, yawl
While geometric AI techniques can unlock the remarkable potential of AI for complex, geometric data, they are not yet widely exploited due to the current difficulties in building, training and applying geometric AI models.
With the CopernicAI platform, we intend to democratize geometric AI for widespread application to solve a host of problems where existing AI techniques are not applicable. Our first step in this direction is the release of geometric AI models for spherical 360° data on the AWS AI Marketplace, starting with Inception360 for the classification of 360° images. Watch this space as we plan to release a number of additional models soon!
[1] Bronstein, Bruna, Cohen, Velickovic, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (2021), arXix:2104.13478
[2] Cobb, Wallis, Mavor-Parker, Marignier, Price, d’Avezac, McEwen, Efficient Generalised Spherical CNNs, ICLR (2021), arXiv:2010.11661
[3] McEwen, Wallis, Mavor-Parker, Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs, ICLR (2022), arXiv:2102.02828
[4] Szegedy et al., Going Deeper with Convolutions, IEEE CCVPR (2015) arXiv:1409.4842
Unlocking AI for 360° spherical data
While AI is now commonplace for standard types of data, such as structured, sequential and image data, the application of AI is severly curtailed for other more complex forms of data. These more complex datasets typical exhibit non-trivial geometry.
The field of geometric AI, or geometric deep learning, has emerged to extend the remarkable benefits of AI to these more complex — geometric — datasets [1] (for a brief introduction to geometric AI see our recent article). The use of geometric AI techniques, however, is still in its infancy since building and deploying geometric AI models remains difficult.
Some excellent geometric AI libraries and frameworks exists, mostly for graph geometric AI techniques (as discussed in this article). However, these libraries are relatively low-level, requiring an expert machine learning engineer with geometric AI expertise to build and train models, before deploying in a production environment. For other forms of geometric AI, much progress is in cutting edge research where general libraries are not yet available.
With CopernicAI we’re developing a low code platform for geometric AI so that non-experts can easily pick up geometric AI approaches for their own problems. Our first step in this direction is to make geometric AI models of 360° spherical data available.
Spherical data is in fact very common, arising in a great many fields. For example, spherical data can arise when observations are made at each point on a spherical surface, such as a topographic map of the Earth. However, they also arises when observations are made over directions, such as panoramic photos and videos captured by 360° cameras, e.g. for virtual reality, surveillance, or autonomous vehicles. Other applications include the analysis of the relic light from the Big Bang in cosmology or diffusion magnetic resonance imaging in medicine, and many others beyond.
At Kagenova we’re working to unlock the remarkable success of deep learning for these problems and others involving data with complex geometry, such as the sphere. For 360° spherical data standard AI techniques aren’t effective and we need geometric AI designed specifically for the spherical geometry of the data.
A lot of progress has been made in developing spherical AI techniques to unlock these applications (as discussed in our previous articles here and here). A significant challenge is to develop techniques that are computationally efficient and able to handle high-resolution data, such as high-resolution 360° images.
In our research we have made a lot of progress in this endeavour, firstly developing general efficient components [2], before introducing approaches to support high-resolution input data [3]. While support for high-resolution input data unlocks a lot of applications, such as classification problems, in many settings high-resolution output data must be supported. We’re currently working on support for high-resolution output data and will soon be releasing a related paper, which will open up a host of new applications in 360° image understanding, such as semantic segmentation.
Now that our geometric AI techniques for spherical data are maturing, we’re planning to make our approaches and models publicly available for anyone to use.
While work on our CopernicAI platform is still ongoing, in the interim we plan to make a number of spherical AI models available throught the AWS AI Marketplace.
We have just released our first model, Inception360, for the classification of 360° images.
This is a simple model, using an inception-based architecture [4], to perform ImageNet classification, i.e. classifying 360° images using the 1,000 ImageNet classes. ImageNet may not provide the best set of class labels for 360° classification since 360° images capture more content than standard 2D planar images; nevertheless, it provides a familiar set of class labels to get started. In this model we do not leverage our latest R&D developments but rather provide a very simple model to allow anyone to get going with 360° image classification immediately.
You can find Inception360 for free on the AWS AI Marketplace here. A notebook illustrating how to use Inception360 is available here.
Below we illustrate Inception360 classification on a (out-of-sample) 360° image. Consider the following 360° image of a dock and sailing boats.
dock, boathouse, gondola, schooner, yawl. [Original image source.]Inception360 returns the following (ordered top 5) classifications for this image, which match the image very well:
dock, boathouse, gondola, schooner, yawl
While geometric AI techniques can unlock the remarkable potential of AI for complex, geometric data, they are not yet widely exploited due to the current difficulties in building, training and applying geometric AI models.
With the CopernicAI platform, we intend to democratize geometric AI for widespread application to solve a host of problems where existing AI techniques are not applicable. Our first step in this direction is the release of geometric AI models for spherical 360° data on the AWS AI Marketplace, starting with Inception360 for the classification of 360° images. Watch this space as we plan to release a number of additional models soon!
[1] Bronstein, Bruna, Cohen, Velickovic, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (2021), arXix:2104.13478
[2] Cobb, Wallis, Mavor-Parker, Marignier, Price, d’Avezac, McEwen, Efficient Generalised Spherical CNNs, ICLR (2021), arXiv:2010.11661
[3] McEwen, Wallis, Mavor-Parker, Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs, ICLR (2022), arXiv:2102.02828
[4] Szegedy et al., Going Deeper with Convolutions, IEEE CCVPR (2015) arXiv:1409.4842
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.