Augmented Reality (AR) in React Native

What is Augmented Reality (AR)?

Augmented reality (AR) is an enhanced version of the real physical world that is achieved through the use of digital visual elements, sound, or other sensory stimulations and delivered via technology. Augmented Reality is defined as the technology and methods that allow overlaying of real-world objects and environments with 3D virtual objects using an AR device, and allow the virtual to interact with the real-world objects to create intended meanings.

Unlike virtual reality which tries to recreate and replace an entire real-life environment with a virtual one, augmented reality is about enriching an image of the real world with computer-generated images and digital information. It seeks to change perception by adding video, infographics, images, sound, and other details.

Inside a device that creates AR content; virtual 3D images are overlaid on real-world objects based on their geometrical relationship. The device must be able to calculate the position and orientation of objects concerning others. The combined image is projected on mobile screens, AR glasses, etc.

Augmented Reality Technology

• Recognition-based AR: It is a camera to identify markers so that an overlay is possible if there is a marker detected. The device detects and calculates the position and orientation of the marker and replaces the real world marker with its 3D version. Then it calculates the position and orientation of others. Rotating the marker rotates the entire object.
• Location-based Approach: Here the simulations or visualizations are generated from data collected by GPS, digital compasses, accelerometers, and velocity meters. It is very common in smartphones.
• Depth Tracking Technology: Depth map tracking cameras such as Microsoft Kinect generate a real-time depth map by using different technologies to calculate the real-time distance of objects in the tracking area from the camera. The technologies isolate an object from the general depth map and analyze it.
• Natural Feature Tracking Technology: It may be used to track rigid objects in a maintenance or assembly job. A multistage tracking algorithm is used to estimate the motion of an object more accurately. Marker tracking is used, as an alternative, alongside calibration techniques. The overlaying of virtual 3D objects and animations on real-world objects is based on their geometrical relationship. Extended face-tracking cameras are now available on smartphones such as iPhone XR which has TrueDepth cameras to allow better AR experiences.

Applications of Augmented Reality

• Gaming: AR allows for better gaming experiences as gaming grounds are being moved from virtual spheres to include real-life experiences where players can perform real-life activities to play.
• Retail and Advertisement: AR can improve customer experiences by presenting customers with 3D models of products and helping them make better choices by giving them virtual walkthroughs of products such as in a real estate. It can be used to lead customers to virtual stores and rooms. Customers can overlay the 3D items on their spaces such as when buying furniture to select items best suitable to match their spaces – regarding size, shape, color, and type. In advertising, ads can be included in AR content to help companies popularize their content to viewers.
• Manufacturing and Maintenance: In maintenance, repair technicians can be directed remotely by professionals to do repairs and maintenance works while on the ground using AR apps without having the professionals travel on the location. This can be useful in places where it is hard to travel to the location.
• Education: AR interactive models are used for training and learning.
• Tourism: AR, in addition to placing ads on AR content, can be used for navigation, providing data on destinations, directions, and sightseeing.
• Medicine/Healthcare: AR can help train healthcare workers remotely, help in monitoring health situations, and for diagnosing patients.

Working Libraries for React Native Framework

• react-viro :- https://github.com/virocommunity/viro
• react-native-ar-viewer :- https://github.com/riderodd/react-native-ar
• react-native-3d-model-view :- https://github.com/BonnierNews/react-native-3d-model-view
• react-native-gl-model-view :- https://github.com/rastapasta/react-native-gl-model-view
• react-native-arkit :- https://github.com/react-native-ar/react-native-arkit

3D File Formats

• obj
• png
• glb
• usdz
• model

To make AR model viewer in react native with native android and ios code:

https://medium.com/@linjunghsuan/a-simple-way-to-use-native-ar-viewer-in-react-native-18304bc6c799

AR With Android

Augmented reality (AR) describes user experiences that add 2D or 3D elements to the live view from a device’s camera in a way that makes those elements appear to inhabit the real world. ARCore combines device motion tracking, camera scene capture, advanced scene processing, and display conveniences to simplify the task of building an AR experience. You can create many kinds of AR experiences with these technologies using the front or rear camera of an Android device.

AR Core

ARCore is Google’s platform for building augmented reality experiences. Using different APIs, ARCore enables your phone to sense its environment, understand the world and interact with information. Some of the APIs are available across Android and iOS to enable shared AR experiences.

• ARCore's motion tracking technology uses the phone's camera to identify interesting points, called features, and tracks how those points move over time. With a combination of the movement of these points and readings from the phone's inertial sensors, ARCore determines both the position and orientation of the phone as it moves through space.
• ARCore's understanding of the real world lets you place objects, annotations, or other information in a way that integrates seamlessly with the real world. You can place a napping kitten on the corner of your coffee table, or annotate a painting with biographical information about the artist. Motion tracking means that you can move around and view these objects from any angle, and even if you turn around and leave the room, when you come back, the kitten or annotation will be right where you left it.
• ARCore provides SDKs for many of the most popular development environments. These SDKs provide native APIs for all of the essential AR features like motion tracking, environmental understanding, and light estimation. With these capabilities you can build entirely new AR experiences or enhance existing apps with AR features.

Related Read: Augmented Reality in Android

AR with iOS

AR Kit

ARKit is the underlying framework that handles the "heavy lifting" of Augmented Reality experiences. ARKit configures the camera, gathers the relevant sensor data, and is responsible for detecting and locating the "anchors" that will tether your 3D content to the real world, as seen through the camera.

• In a sense, Augmented Reality is all about displaying 3D content in the real world, tethering your 3D content to anchors that are tracked and followed, making the 3D content appear as though it truly is in front of your user. As a whole, ARKit does the work to find those anchors, track those anchors, and handles the computations and augmentations to keep your 3D content tethered to those anchors, making the experience seem realistic.
• Anchors can come in a variety of forms. Anchors are most commonly planes (a horizontal plane, like a floor, table top, or the ground, or a vertical plane, like a wall, window, or door), but can also be faces (a human face), an image (where you provide your app an image, and when the camera detects that image, that becomes the "anchor" for your 3D content), an object (where you provide your app a 3D object, and when the camera detects that object in the real world, that object becomes the "anchor" for your 3D content), a body (for the purposes of tracking the movement of joints and applying that movement to a 3D character), a location (using ARGeoAnchors, which anchor your 3D content to a specific set of longitude/latitude/altitude coordinates, as a CLLocation from the CoreLocation framework, if in a supported location), or a mesh (if your device has a LiDAR scanner, ARKit becomes capable of detecting more nuanced planes, such as recognizing a floor plane vs. a table-top plane, or a door plane vs. a wall plane). In all, your 3D content has to be anchored to something in the real world, and ARKit handles finding these anchors and providing them to you for your use.

AR Supported Apple Devices

AR Supported Apple devices: https://developers.google.com/ar/devices#google_play_devices