Method Profiling in Android

I've recently been using the Android implementation of OpenCV for real-time computer vision on mobile devices. Computer vision is computationally expensive - especially when you're working with a camera stream in real-time. In trying to speed up my object tracking algorithm I used Android's method profiler to analyse the time spent in each function, hoping to identify potential areas for optimisation. This makes an interesting little case study and example of how to use Android's profiling tools.

How do I enable profiling?
Traceview is part of the Eclipse ADT. Whilst in the DDMS perspective, method profiling can be enabled by selecting a debuggable process and clicking the button circled below. To stop profiling, click the button again. After the profiler is stopped, a Traceview window will appear.

Interpreting Traceview output

The image above was my first method trace, capturing around seven seconds of execution and thousands of method invocations. Each row in the trace corresponds to a method (ordered by CPU usage by default). Selecting a row expands that method, showing all methods invoked from within that method. Again, these are ordered by their CPU usage.

Optimisation using profile data
Using the above example, we can see that my object tracking algorithm spends most of its time waiting for four methods to return: Imgproc.pyrDown, MainActivity.blobUpdate, Imgproc.cvtColor and VideoCapture.retrieve. The pyrDown method downsamples an image matrix whilst applying a Gaussian blur filter. The blobUpdate method is a callback I use to give updates on a tracked object. The cvtColor method converts the values in a matrix to those of another colour space. The retrieve method captures a frame from the device camera.

The latter two methods are crucial to my object tracking algorithm, as I need to call retrieve to get images from the camera and cvtColor is used to convert from RGB to HSV colour space, as it is better to perform colour thresholding this way. The former two, however, can potentially be optimised.

From this trace I've already identified a redundant yet expensive method call: pyrDown. 30% of the time spent in the processFrame method is spent waiting for pyrDown to return. I was using this function to downsample an image from the camera to 240x320, as a smaller image can be processed faster. Instead, this call can be eliminated by requesting 240x320 images from the camera.

In the blobUpdate method I send updates about the location of the tracked object and its size. I maintain a short history of these readings and use dynamic time warping to detect gesture input. By expanding the trace for this method I see that my gesture classification function is taking the most time to execute. As dynamic time warping, by design, finds alignments between sequences of different lengths, I can reduce the frequency of checking for gestures. By only checking for gestures in every second call of blobUpdate, I effectively half the amount of time spent checking for gestures. This still maintains a high recognition rate by virtue of dynamic time warping's resilience to differences in alignment length.

Conclusion
The case study in this post demonstrates how method profiling can be used to identify potential areas for optimisation; something which can be particularly beneficial in a computationally expensive application. By profiling a few seconds of execution of a computer vision algorithm I was able to capture data about thousands of method invocations. From the trace data I identified a redundant method call which accounted for 30% of my algorithm's execution time and identified an optimisation to the second most expensive method call.

Amazon Hackathon

This weekend was the Amazon Hackathon in Glasgow, the first of its kind in the UK. The idea was simple: get 50 students to show up in teams, give them 20 hours to innovate and create, and provide them with $50 of AWS credit and a stack of pizzas to power their ideas. What followed was anything but simple. A 20 hour frenzy of brainstorming and prototyping to create something which we would eventually present to our peers and the "Amazonians". Some amazing ideas were demonstrated, with one team netting $1,000 in AWS credit to help kickstart their idea of a gifting-based social network.

I entered as part of Team Giraffa Cakes, a team of four from the University of Glasgow. Disappointingly, three of our team were the only 4th year students from Glasgow to enter. Our idea was to crowd-source information to help inform product comparisons. Users would be able to search for two products on Amazon and our web service would then gather relevant content from sources such as Twitter, Blogger and Youtube to help inform decisions. We implemented a system which did just that, performing sentiment analysis on the tweets and blog posts to give an overview of how people feel about those products. At a glance users would be able to tell if opinion about each product was generally negative, neutral or positive. Information from the web service was made available to both an Android app which I implemented and a website frantically thrown together in record time by James.

Our Android app, showing the product overview.

The experience was an interesting one and was lots of fun, despite the intense desire for sleep that kicked in around 4am. To stay sane as the night went on we found ourselves increasingly taking breaks just to get away from the computer. Cold pizza and instant coffee from a kettle of questionable hygiene proved to be a welcome respite and every hour or so we went outside for a short break. As appreciated as the fresh air was, it was probably the darkness which was most welcome; a break for our eyes. I think the location of the event may have contributed to the drowsiness of everyone. The lab it was held in is notoriously hot and stuffy; in retrospect our team probably would've been more comfortable downstairs in our own, gloriously air conditioned lab. Although then we'd have to walk further for cold pizza.

Overall it was a blast. We had a lot of fun, created something we were all proud of and the icing on the cake was that our fellow hackers voted us the People's Choice. It was great to see so many ideas brought to completion in a single night - it's not uncommon to see university coursework that doesn't work after several weeks of work! Thanks for all the pizza and fresh fruit, Amazon... same time next year?

Multimodal Android Development Part 1

This post is the first of two which gives a brief introduction to creating multimodal interactions in Android applications. I'll briefly cover some of the SDK features available to you as an Android developer which you can use to create richer interactions in your apps. Example code will be quite concise because I assume you have at least a basic knowledge of Android development. Feel free to leave any comments suggesting how I can better explain these concepts, or to let me know if I've made any mistakes or omissions.

What is "multimodal" interaction?

Multimodal interaction, put simply, is interaction involving more than one modality (e.g. multiple senses). For example, an application may provide a combination of visual and haptic (touch) feedback. These types of interaction design provide a number of benefits, for example allowing those with sensory impairment to interact using other senses, or allowing interaction in contexts where one sense may be otherwise occupied.

One of the most ubiquitous examples of a multimodal interaction is the way in which mobile phones combine visual, audible and haptic feedback to inform users of a new text, phone call, etc. This combination of modalities is particularly useful when your phone is, say, in your pocket. Obviously you can't see the phone, but you will probably feel the phone vibrate or hear your ringtone as new notifications appear.

Haptic feedback in Android

Most handheld Android devices have some sort of rotation motor in them allowing simple haptic feedback. Although not common in tablets (largely due to size constraints), all modern Android phones will have tactile feedback available. You can control the phone vibrator through the Vibrator class. Note that in order to use this, your Manifest must request the following permission: android.permission.VIBRATE

/* Request the device's vibrator service. Remember to check
 * for null return value, in case this isn't available. */
Vibrator vibrator = (Vibrator) getSystemService(Context.VIBRATOR_SERVICE);

/* Two ways to control the vibrator:
 *  1. Turn on for a specific time
 *  2. Provide a vibration pattern */

/* 1. Vibrate for 200ms */
vibrator.vibrate(200);

/* 2. Vibrate for 200ms, pause for 100ms, vibrate for 300ms. */
long[] pattern = new long[] {0, 200, 100, 300};

/* Perform this pattern once only (repeat := -1). */
vibrator.vibrate(pattern, -1);

/* Vibrate for 200ms, followed by indefinite repeat of
 * 100ms pause followed by 300ms vibrate. Setting
 * repeat := 2 tells the vibrator to repeat at offset
 * 2 into the vibration pattern. */
vibrator.vibrate(pattern, 2);

Touchscreen gestures

Using touchscreen gestures to interact with applications can be fun, efficient and useful when users may be unable to select a particular action on the screen. For example, it can be difficult to select a button on-screen when running or walking. A touch gesture, however, is a lot easier and requires less precision from the user. The disadvantage with touch gestures is that if not used sparingly, there may be too much for the user to remember!

Creating a set of gestures for your application is simple: create a gesture library on an Android Virtual Device using the Gesture Builder application (available on the AVD by default) and add a GestureOverlayView to your activity layout. In your activity, you just have to load the gesture library from your resources and implement an OnGesturePerformedListener.

private GestureLibrary mLibrary;

public void onCreate(Bundle savedInstanceState) {
  ...
  /* 1. Load gesture library from the res/raw/gestures file */
  mLibrary = GestureLibraries.fromRawResource(this, R.raw.gestures);

  if (!mLibrary.load())
    /* Error: unable to load from resources! */
    ...

  /* 2. Find reference to the gesture overlay view */
  GestureOverlayView gov = (GestureOverlayView) findViewById(R.id.gestureOverlay);

  /* 3. Register callback for gesture input */
  gov.addOnGesturePerformedListener(this);
}

The callback method for gesture performance receives a Gesture as an argument. This can be used to obtain a list of predictions: which gestures in your library that Android thought the gesture was. With these predictions, you can use the prediction score (or contextual information) to determine which gesture the user was most likely to have performed. I find it useful to define a threshold for gesture acceptance, so that you can reject erroneous or inaccurate gestures. The best way to choose this threshold value is through trial and error: see what works for you and your gestures.

private static final double ACCEPTANCE_THRESHOLD = 10.0;

public void onGesturePerformed(GestureOverlayView overlay, Gesture gesture) {
  /* 1. Get list of gesture predictions */
  ArrayList predictions = mLibrary.recognize(gesture);

  if (predictions.size() > 0) {
    /* 2. Find highest scoring prediction */
    Prediction bestPrediction = predictions.get(0);

    for (int i = 1; i < predictions.size(); i++) {
      Prediction p = predictions.get(i);
      if (p.score > bestPrediction.score)
        bestPrediction = p;
    }

    /* 3. Decide if we'll accept this gesture */
    if (bestPrediction.score > ACCEPTANCE_THRESHOLD)
      gestureAccepted(bestPrediction.name);
  }
}

private void gestureAccepted(String gestureName) {
  /* Respond appropriately to the gesture name */
  ...
}

Saving map images in Android

Recently I've been working on a little Android project and wanted to save thumbnail images of a map within the application. This post is just sharing how to do exactly that. Nothing too complicated.

public class MyMapActivity extends MapActivity {
    private MapView mapView;

    ...

    private Bitmap getMapImage() {
        /* Position map for output */
        MapController mc = mapView.getController();
        mc.setCenter(SOME_POINT);
        mc.setZoom(16);

        /* Capture drawing cache as bitmap */
        mapView.setDrawingCacheEnabled(true);
        Bitmap bmp = Bitmap.createBitmap(mapView.getDrawingCache());
        mapView.setDrawingCacheEnabled(false);

        return bmp;
    }

    private void saveMapImage() {
        String filename = "foo.png";
        File f = new File(getExternalFilesDir(null), filename);
        FileOutputStream out = new FileOutputStream(f);
    
        Bitmap bmp = getMapImage();
    
        bmp.compress(Bitmap.CompressFormat.PNG, 100, out);
    
        out.close();
    }
}

In the getMapImage method, we're telling the map controller to move to a particular point (this may not matter to you, you may just want to take the image as it appears) and zooming in to show a sufficient level of detail. Then a Bitmap is created from the map view's drawing cache. The saveMapImage method is just an example of how you may want to save an image to the application's external file directory.

Android workshop and Surface

It feels unusually warm for November, which has made the past week quite pleasant for running. I've gotten 4 runs in over the last week, and I'm hoping to keep up at least 3 runs a week until the end of the semester. Dr Cutts talk last Wednesday on computing science education has caused a bit of introspection on how I use my time, and has made me realise that I don't always spend it wisely. I'm a workaholic, I get lots of work done, and I consider myself to be quite well organised. But maybe I could achieve similar things in a lot less time. Lately I've been focusing more on coursework, really trying to get as much done as early as possible. I've never had to pull an all-nighter working towards a deadline before, and I certainly don't plan to start any time soon.

I'm excited for the start of Week 12 because, other than the obvious reasons of having no more deadlines, I'm likely going to be putting on an Android development workshop in the School of Computing Science, along with another classmate. It'll be cool to give something back like that, and hopefully attendance is pretty decent. I'd certainly hope so, given that the Mobile Software Engineering degree has over 5 times as many students this year as last. The more Android projects I work on, the more I notice patterns emerging and the ability to re-use code. Things have gotten to the point now where any project I do in Android has about 50% re-used code. I think myself and James have a decent amount of experience to offer and can help teach other developers how to address problems that we've already encountered.

Week 12 is also the start of a fortnight dedicated to project work. My project at the moment is in quite a good state, I reckon. As far as implementation is concerned, I'm well ahead of schedule and the main technical concerns have been addressed. I don't know if I've mentioned it before, but I'm working with Microsoft Surface this year, and one of the technical challenges I'm approaching is how to display information when the Surface has stuff on top of it. I find this an interesting problem because it's only natural that a tabletop computer has to remain usable at the same time as being used as a table.

So far I've been iteratively developing a prototype which displays a shape in the largest unoccluded space, and have just started to animate this shape as it moves around the tabletop due to objects being placed on or removed from the Surface. There's some really cool stuff going on to make this work, and we (my project supervisor an I) are probably going to submit a work-in-progress paper to CHI2012 about our research so far. It's a new and novel area of research and it'd be the highlight of my academic "career" if that paper gets accepted.

Here's a terrible quality video I took earlier showing a prototype in action.

Side projects and running

Yesterday I started work on a new Android project, to pass some time and to create something which I, and hopefully others, will find useful. It's going to be an editor for GPS files which allows you to create, edit and share routes, using gpx (GPS exchange format) files. With the help of a friend, it's already made good progress in to something really lacking in features and polish, but still usable.

The image above is a screenshot showing what's implemented so far. Routes can be drawn by tapping the touch-screen to add a new waypoint and altered by dragging-and-dropping existing waypoints. Something I may also implement, if I can find out an elegant way to do it, is selecting points by a long press and then giving the option to, say, delete it.

Right now we're working through a to-do list and turning this into something which may eventually end up on the Android Market. Just after starting it, though, I'll be putting it on hold for a few days while I go off to stay with family on the other side of the country. Hopefully the weather stays reasonable, because I'll be taking my hiking boots, running shoes and, maybe, bike. It'll be great to just chill out, explore new places, and unwind for a few days.

I'm almost halfway there in my 100 mile challenge; I've covered 47.5 miles so far in the past 13 days. I've been running so much lately that I'm not racking up the miles as fast as I would be if I were cycling. My running is certainly improving though; both in speed and endurance. My technique is beginning to get more consistent as well, which is helping me to pace myself.

Glasgow Bus Finder

Although it's been around 3 months since we stopped work on Glasgow Bus Finder (my team project in third year), I've gotten the urge to do a bit more Android development. Aidan and myself always had greater ambitions than just a university project and I think it's time that I visit some of the features that we never had time to initially implement.

First and foremost was the ability to move the application to the SD card, as, admittedly, it has quite a high storage footprint. 3MB is not significant by any means, but I understand how limited memory can be in some devices.

Also, at the request of a friend who owns a Galaxy Tab, I should create larger graphics to ensure that the application scales up well on tablet computers.

I came across a cool game yesterday called Proun. It's a simple racing game involving brightly coloured shapes, and jazz music. That's about all a game needs, really! The cool thing? It has no fixed price; you pay what you think it deserves.