#Tutorial 6: SLAM Tracking and Ambient Lighting with ARKit & Unity 3D 1



Literature review 

Apple doesn’t want to bring Augmented Reality to a small group of people to improve over time; it wants to bring this new tech to every iOS user all at once. ARKit is a set of tools that allows developers to build AR apps and games for the iPhone and iPad, so instead of a centralized platform, developers can decide just how deep into AR they want to bring new and existing apps.

  • With ARKit you are able to turn flat surfaces in your home into AR playgrounds. 
  • You point the phone camera at a flat surface, and when the software confirms space is good for AR, it is time to play.
  • Place something in the real world, the software makes it possible for you to move around as though the virtual world were really there.
  • You can lean in with your phone to get a closer look at something specific, walk away to get a wider view of the world or walk around the world you’ve created to see every angle.

ARKit Demo: Rabbit Gangnam Style

ARKit takes advantage of the iPhone’s camera, motion sensors, and graphics processors.

Visual-Inertial Odometry: which will let ARKit track the world, and for this, it will fuse camera sensor data with CoreMotion data. 

Scene Understanding and Lighting Estimation: Analyse the scene based on the camera view, and then find horizontal planes like tables and floors, and can track and place objects on smaller feature points as well. the ARKit also relies on the camera sensor to figure out the amount of light in a room, and will then apply the correct light to the virtual objects.

 

Basic concepts inside ARKit UnityPackage:

‘Unity ARKit Plugin’ is a native plugin that enables using all the functionality of the ARKit SDK simply within your Unity projects for iOS. The plugin exposes ARKit SDK’s world tracking capabilities, rendering the camera video input, plane detection and update, point cloud extraction, light estimation, and hit testing API to Unity developers for their AR projects.

  • “/Assets/Plugins/iOS/UnityARKit/UnityARCameraManager.cs” – this component should be placed on a gameobject in the scene that references the camera that you want to control via ARKit, and it will position and rotate the camera as well as provide the correct projection matrix to it based on updates from ARKit. 
  • “/Assets/Plugins/iOS/UnityARKit/UnityARVideo.cs” – this component should be placed on the camera and grabs the textures needed for rendering the video, and sets it on the material needed for blitting to the backbuffer, and sets up the command buffer to do the actual blit
  • “/Assets/Plugins/iOS/UnityARKit/Utility/UnityARAnchorManager.cs” – this is a utility that keeps track of the anchor updates from ARKit and can create corresponding Unity gameobjects for them (see GeneratePlanes.cs component on how to use it)
  • “/Assets/Plugins/iOS/UnityARKit/Editor/UnityARBuildPostprocessor.cs” – this is an editor script that runs at build time on iOS

 

Getting Started

In this augmented reality tutorial we will use the Unity 3D and the Apple ARkit to create an augmented reality app for your iPhone or iPad. This app will allow us to place a virtual ‘Animated Rabbit’ in the real world, move it around, and make it bigger or smaller with ambient lighting. This tutorial is designed for beginners so even if you don’t have any previous knowledge of programming or Unity 3D you should still be able to follow along!

Prerequisite (Tools Needed):

First, you will need a Mac computer (MAC OS) Apple has been really strict about this. Second, you will need an Apple developer’s account to download the necessary tools to get started. Second, you will need an Apple developer’s account to download the necessary tools to get started.

  1. Unity Engine (IMP: Unity’s AR Kit requires the patch version of Unity 5.6.1p1 or later. )
  2. Xcode 9 beta
  3. Apple A9/A10 Processor device with iOS 11 Beta installed
  4. Unity ARKit Plugin Download Here
  5. Rabbit UnityPackage (3D Animation) Download Here

 

Step by Step Implementation

1. Install all the software you will need (Prerequisite):

  • Log into the Apple developers website and click on the “Downloads” button. If your developer account is active, you will see a collection of beta software options.
latest beta releases

Latest beta releases

 

2. Setup a Simple Unity Project:

Once everything is installed and ready to go, it is time to create an empty project in Unity. Let’s start by creating a new Unity3D project by the name of “Tutorial6”. (Disable Unity Analytics for now)

 

3. Import the ARKit Plugin into newly created Unity Project:

  • Import the ARToolkit Unity Plugin which we just downloaded in prerequires section.
  • This can be done by importing custom UnityPackage, here I stored both inside Vuforia_SDK folder in my machine.
  • To import a new custom package:
    1. Choose Assets > Import Package > Custom Package… to bring up File Explorer (Windows) or Finder (Mac).
    2. Select the package you want from Explorer or Finder, and the Import Unity Package dialog box displays, with all the items in the package pre-checked, ready to install. (See Fig 4: New install Import Unity Package dialog box.)
    3. Select Import and Unity puts the contents of the package into the Assets folder, which you can access from your Project View.
  • Once it’s finished, a window will appear with all the files in the package. Click on the “Import” button.

ARKit Asset import

 

4. Import custom ‘Rabbit_Scene.unitypackage’  in Unity Project:

Rabbit_Scene.unitypackage’ is special package created by me for your ease. It is on my GitHub repository and you can download it anytime.

  • Import previously downloaded  (in Prerequisite section)  Unity Package ‘Rabbit_Scene.unitypackage’ inside our Unity Project similar to step-3.
  • You can choose either way to import package “Directly double click on Package itself” OR “import custom package through unity window“.

 

5. Open Unity-Scene “RabbitScene” inside Unity Project

  • In the Project window, double-click the “Rabbit_Scene” i.e. open Rabbit_Scene.
  • You can view scene hierarchy as similar to shown in the screenshot below.

Add Rabbitscene

 

6. Final Camera & Scene Position, Scale & Rotation Setup, and Adjustment 

  • Setup ‘CameraParent’ Prefab: Maintain the Position, Rotation & Scale properties of ‘CameraParent’ prefab as following in Inspector window
    • Position:  X=o,   Y=0.92,       Z=-3.69 
    • Rotation: X=0,    Y=90,         Z=0 
    • Scale:       X=1,    Y=1,           Z=1 
  • Setup ‘Rabbit_Parent’ Prefab: Maintain the Position, Rotation & Scale properties of Rabit_Parent prefab as following in Inspector window
    • Position:  X=o,   Y=-1.07,     Z=1.37 
    • Rotation: X=0,    Y=90,         Z=0 
    • Scale:       X=1,    Y=1,           Z=1 

arkit-scene6-map

 

  • Now your Camera pointing out exactly towards the Rabbit. Do some Light change if required.
  • Now your scene will look similar to shown screenshot below.

ARKit Tutorial Final Scene

 

7. Final Deployment Setup & Build Process

We are almost done. Let’s save the scene: File >> Save Scene and move towards deployment step.

  • The last step is to build the project for iOS iPhone/iPad Platform. We need to go to “File >> Build Settings”. We need to add the current scene by selecting “Add Open Scene”. Then, we need to select a platform (iOS) and then select “Switch Platform“. Here, we will have Build Project options:
    • Build Project: This will allow us to export the current Unity project to X-Code to deploy on the iOS device.
  • Unity provides a number of settings when building for iOS devices – select from the menu (File > Build Settings… > Player Settings…) to see/modify the current settings. 
  • With the Player Settings up in the Inspector, change the ‘Build Identifier’ find the Camera Usage Description and type ‘camera use’ into it. Then back in the Build Settings window, click the “Build” button,  as per the screenshot below.

ARKit Build Process

  • Now Unity wants to know where to put your Xcode project. In the Save As box, type “Rabbit_DEMO” and click the “Save” button .

8. Compile & Run through X-Code

  • When the build process completes, a Finder window will appear. Open“Rabbit_DEMO” folder and double click on “Unity-iPhone.xcodeproj”

X-Code Build

  • We are almost done.You should now be seeing a screen similar to the one in the screenshot below.
  • Assign Team to the Project: In the section labeled ‘Signing’, click on the ‘Team‘ drop-down and select your ‘Developer account’.
  • With that out of the way, look at the upper-left corner of the window. With your iPhone or iPad connected to your computer, hit the “Play” button. It can take a little bit to compile the first time.

X-Code Setup Deployment Setup

9. Show Time 🙂

Once it is done, you will be able to look augmented Rabbit dancing in front of you in Gangnam Style 🙂

 
Augmented Rabbit ARKit

Augmented Rabbit

If you get stuck at any point or want to view the source code, you can find it on Github. Let me know if you have any questions in the comments section below!

IMP:

 

You did a great Job !!!  It’s a Guinness Time now! 🙂  

 


1+

Sanket Prabhu

About Sanket Prabhu

People find Sanket Prabhu to be an expert in the field of Augmented Reality, Unity 3D & AR/VR based mobile app development (iOS/Android). He is working as a researcher, developer & technology evangelist with focus on, real world 'Enterprise' AR/VR app development


Leave a comment

Your email address will not be published. Required fields are marked *

One thought on “#Tutorial 6: SLAM Tracking and Ambient Lighting with ARKit & Unity 3D

  • Anu

    What a great tutorial – thanks!

    I followed your steps, the one thing I’m noticing is that the rabbit model seems to “float” around and move with me when I move the camera. It’s like it doesnt find the right surface to attach itself too. Any thoughts on how to fix this?

    Thank you!
    Anu

    0