ugmented Reality (AR) seems to be on everyone's radar. For the two of you who haven't watched a YouTube video about it: Augmented Reality is the ability to overlay location data points on the live view of a mobile device's camera. In a sense, AR allows the phone to become a window into a slightly different, data-driven world.

To me, however, AR appears to be little more than a gimmick at this point. Before you fill my inbox with hate mail, let me quickly explain why:

1. The sensors involved (compass, accelerometer, and GPS) aren't nearly advanced enough to do the kind of real-time tracking required for a useable AR app.
2. No currently available data set performs best in this medium. That is to say, any data set you can show in AR today would look better in something like a Google Maps view.

 

---

---

 

However, any good programmer knows that the best time to enter a market is before a killer application makes its debut, rather than when the market is already proven. With that in mind, DevX will publish a two-article series to set you on the path to building your own Augmented Reality engine on Android. Building an AR application requires a dash of math and four technological pieces. This first article covers the first two pieces: the camera and the compass; the next article will cover the other two: the accelerometer and GPS.

If you are not an experienced Android developer, you will need some gumption to follow along and complete the application because the instructions assume some familiarity with the Android SDK.

The Custom Camera View

The first step in implementing Augmenting Reality is finding your field of view. This requires activating the camera on your device (in this example, T-Mobile's G1). In this tutorial, you will create a custom surface view on which you'll load the camera preview.

First, you need to request permission to use the camera in your application. Do this by adding the following line just inside the <application> header in your Manifest.xml file.

<application android:icon="@drawable/icon" android:label="@string/app_name" android:debuggable="true">
<uses-permission android:name="android.permission.CAMERA" />
<!--...other stuff goes here...-->
</application>

Android's camera object requires a surface view on which to draw its captured frames. To that end, you will create your own custom SurfaceView and hand it off to the camera. The following code creates the view inside the onCreate function of your activity (always called on startup and when your activity is resumed):

public void onCreate(Bundle savedInstanceState) {
try{
super.onCreate(savedInstanceState);
cv = new CustomCameraView(
this.getApplicationContext());
FrameLayout rl = new FrameLayout(
this.getApplicationContext());
setContentView(rl);
rl.addView(cv);
} catch(Exception e){}
}

Nothing tricky here: you're creating a custom view, adding it to a frame layout, and finally setting this frame layout as the main view for your activity. Before compiling it (which won't work yet anyway), you need to define the CustomCameraView class as follows:

public class CustomCameraView extends SurfaceView
{
Camera camera;
SurfaceHolder previewHolder;
public CustomCameraView(Context context)
{
super(context);
}
}

Now that you can at least compile the example thus far, it's time to add some camera code.

First, you have to define a surface listener. Because the camera object takes a surface holder, you may assume that when your activity starts up you can create a surface view and immediately stuff it into the camera. Sadly, despite having its constructor called, the surface isn't actually in a state where you can use it just yet. To get around this timing issue, you will register a surface listener and hand it off to the camera when its surfaceCreated method gets called. Here's the real base constructor for your custom view, which this tutorial covers quickly before getting into the surface listener:

public CustomCameraView(Context ctx)
{
super(ctx);
previewHolder = this.getHolder();
previewHolder.setType
(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
previewHolder.addCallback(surfaceHolderListener);
}

Next, you'll pull out your Surface holder as a class member and then register a surface listener with it. Again, don't try to compile this just yet, you'll need to define that surfaceHodlerListener object first.

Listening to the Surface Holder

Here is the inline code for the surfaceHolderListener object:

SurfaceHolder.Callback surfaceHolderListener = new SurfaceHolder.Callback() {
public void surfaceCreated(SurfaceHolder holder) {
camera=Camera.open();
try {
camera.setPreviewDisplay(previewHolder);
}
catch (Throwable ){ }
}
public void surfaceChanged(SurfaceHolder holder, int format, int width,
int height)
{
Parameters params = camera.getParameters();
params.setPreviewSize(w, h);
params.setPictureFormat(PixelFormat.JPEG);
camera.setParameters(params);
camera.startPreview();
}
public void surfaceDestroyed(SurfaceHolder arg0)
{
camera.stopPreview();
camera.release();
}
};

When the surface has been created, your callback's surfaceCreated method will be invoked. At that point it's safe to fetch an instance of the device's camera by calling Camera.open() before running setPreviewDisplay and handing off an instance of the surface holder you stashed aside in the view's constructor. Remember; until surfaceCreated is called, you don't actually have a functional surface holder. Passing the holder into the camera object before this call will cause the camera object to throw inexplicable exceptions.

Figure 1. Preview on T-Mobile myTouch: If you're using a G1, you might find the preview is rotated on its side.

When the surface changes, you'll have to tell the camera what the new preview drawing rectangle should look like. You do this in the surfaceChanged method. You'll pull the parameters out of the camera instance, change the format and size, and then pass the parameter back in. When you've done that, you'll call startPreview and away you go!

Your results may vary depending on what device and firmware you're running. If you're running the preview in an emulator and you see a 3D box floating over a checkered background, don't despair. In lieu of any actual camera data, the emulator is drawing a stock animation to show you where the camera preview would be. Also, if you're using a G1, you might find the preview is rotated on its side (see Figure 1). You will need to apply image translation or some creativity to get this rendering correct.

Congratulations! You've now got a working camera view. It's time to start pulling in the information you'll need to actually render data on top of this camera view.

Asking the Phone for Direction(s)

The final piece of the puzzle is getting directional information from the magnetic compass. Be aware that these compasses tend to freak out when near large metal objects such as...well...computers. With that caveat in mind, you can begin using the compass.

First, you need to extend or to create inline a SensorEventListener. Here's how to do it inline:

SensorEventListener listener = new SensorEventListener(){
public void onAccuracyChanged(Sensor arg0, int
{}
public void onSensorChanged(SensorEvent evt)
{
float vals[] = evt.values;
direction = vals[0];
}
};

This code simply modifies a direction integer whenever the onSensorChanged method is invoked. You can get away with this because you're planning on registering only for orientation sensor updates. If you're listening to both the compass and the GPS, this method will get called whenever either sensor changes. Be sure to check the type of the SensorEvent before modifying any values.

With the compass, the values array should contain a single float: a number representing the bearing or degrees from north at which your device is pointed. A zero value (0) would be due north, 180 would be due south.

Now that you have a listener, you need to register it with the Sensor Manager. You create and register with a SensorManager object like this:

public static SensorManager sensorMan;
sensorMan = (SensorManager)ctx.getSystemService(Context.SENSOR_SERVICE);
sensorMan.registerListener(
listener,
sensorMan.getDefaultSensor(
SensorManager.SENSOR_ORIENTATION),
SensorManager.SENSOR_DELAY_FASTEST);

Apologies for the horrid line breaking. These functions are long, so be sure to paste the code out and delete the line breaks. As you can see (when you've deciphered the code), you're simply getting the Sensor Manager from the getSystemService call on the context. This returns a sensor manager with which you can register a listener.

Compile this mess, run it on your device, and watch the camera render while the compass updates your orientation. Remember, the compass updates will not come in on the UI thread. Android veterans know that you can change views only on the UI or Main thread, so plan accordingly.

Tying on a Bow

As you can see, when you break the process of Augmented Reality down, it becomes a series of straightforward concrete steps. The next article will reveal how to get and normalize the GPS and accelerometer information. In the meantime, this article should serve as a tutorial for using both the camera preview and the compass in whatever application you wish.