This blog has been inactive for a while. For new content and mindless rambling, check out my new blog at: http://euanfreeman.co.uk/blog/
Tuesday 26 November 2013
Friday 30 November 2012
Method Profiling in Android
I've recently been using the Android implementation of OpenCV for real-time computer vision on mobile devices. Computer vision is computationally expensive - especially when you're working with a camera stream in real-time. In trying to speed up my object tracking algorithm I used Android's method profiler to analyse the time spent in each function, hoping to identify potential areas for optimisation. This makes an interesting little case study and example of how to use Android's profiling tools.
How do I enable profiling?
Traceview is part of the Eclipse ADT. Whilst in the DDMS perspective, method profiling can be enabled by selecting a debuggable process and clicking the button circled below. To stop profiling, click the button again. After the profiler is stopped, a Traceview window will appear.
Interpreting Traceview output
The image above was my first method trace, capturing around seven seconds of execution and thousands of method invocations. Each row in the trace corresponds to a method (ordered by CPU usage by default). Selecting a row expands that method, showing all methods invoked from within that method. Again, these are ordered by their CPU usage.
Optimisation using profile data
Using the above example, we can see that my object tracking algorithm spends most of its time waiting for four methods to return: Imgproc.pyrDown, MainActivity.blobUpdate, Imgproc.cvtColor and VideoCapture.retrieve. The pyrDown method downsamples an image matrix whilst applying a Gaussian blur filter. The blobUpdate method is a callback I use to give updates on a tracked object. The cvtColor method converts the values in a matrix to those of another colour space. The retrieve method captures a frame from the device camera.
The latter two methods are crucial to my object tracking algorithm, as I need to call retrieve to get images from the camera and cvtColor is used to convert from RGB to HSV colour space, as it is better to perform colour thresholding this way. The former two, however, can potentially be optimised.
From this trace I've already identified a redundant yet expensive method call: pyrDown. 30% of the time spent in the processFrame method is spent waiting for pyrDown to return. I was using this function to downsample an image from the camera to 240x320, as a smaller image can be processed faster. Instead, this call can be eliminated by requesting 240x320 images from the camera.
In the blobUpdate method I send updates about the location of the tracked object and its size. I maintain a short history of these readings and use dynamic time warping to detect gesture input. By expanding the trace for this method I see that my gesture classification function is taking the most time to execute. As dynamic time warping, by design, finds alignments between sequences of different lengths, I can reduce the frequency of checking for gestures. By only checking for gestures in every second call of blobUpdate, I effectively half the amount of time spent checking for gestures. This still maintains a high recognition rate by virtue of dynamic time warping's resilience to differences in alignment length.
Conclusion
The case study in this post demonstrates how method profiling can be used to identify potential areas for optimisation; something which can be particularly beneficial in a computationally expensive application. By profiling a few seconds of execution of a computer vision algorithm I was able to capture data about thousands of method invocations. From the trace data I identified a redundant method call which accounted for 30% of my algorithm's execution time and identified an optimisation to the second most expensive method call.
How do I enable profiling?
Traceview is part of the Eclipse ADT. Whilst in the DDMS perspective, method profiling can be enabled by selecting a debuggable process and clicking the button circled below. To stop profiling, click the button again. After the profiler is stopped, a Traceview window will appear.
Interpreting Traceview output
The image above was my first method trace, capturing around seven seconds of execution and thousands of method invocations. Each row in the trace corresponds to a method (ordered by CPU usage by default). Selecting a row expands that method, showing all methods invoked from within that method. Again, these are ordered by their CPU usage.
Optimisation using profile data
Using the above example, we can see that my object tracking algorithm spends most of its time waiting for four methods to return: Imgproc.pyrDown, MainActivity.blobUpdate, Imgproc.cvtColor and VideoCapture.retrieve. The pyrDown method downsamples an image matrix whilst applying a Gaussian blur filter. The blobUpdate method is a callback I use to give updates on a tracked object. The cvtColor method converts the values in a matrix to those of another colour space. The retrieve method captures a frame from the device camera.
The latter two methods are crucial to my object tracking algorithm, as I need to call retrieve to get images from the camera and cvtColor is used to convert from RGB to HSV colour space, as it is better to perform colour thresholding this way. The former two, however, can potentially be optimised.
From this trace I've already identified a redundant yet expensive method call: pyrDown. 30% of the time spent in the processFrame method is spent waiting for pyrDown to return. I was using this function to downsample an image from the camera to 240x320, as a smaller image can be processed faster. Instead, this call can be eliminated by requesting 240x320 images from the camera.
In the blobUpdate method I send updates about the location of the tracked object and its size. I maintain a short history of these readings and use dynamic time warping to detect gesture input. By expanding the trace for this method I see that my gesture classification function is taking the most time to execute. As dynamic time warping, by design, finds alignments between sequences of different lengths, I can reduce the frequency of checking for gestures. By only checking for gestures in every second call of blobUpdate, I effectively half the amount of time spent checking for gestures. This still maintains a high recognition rate by virtue of dynamic time warping's resilience to differences in alignment length.
Conclusion
The case study in this post demonstrates how method profiling can be used to identify potential areas for optimisation; something which can be particularly beneficial in a computationally expensive application. By profiling a few seconds of execution of a computer vision algorithm I was able to capture data about thousands of method invocations. From the trace data I identified a redundant method call which accounted for 30% of my algorithm's execution time and identified an optimisation to the second most expensive method call.
Thursday 7 June 2012
"Feelable" touchscreens revisited
Tactus have gotten quite a lot of attention recently after demonstrating their new touchscreen technology (pictured above; image source). Their "Tactile Layer" technology raises bubbles on the touchscreen, creating, essentially, physical objects on the touchscreen. I suppose I've taken quite an interest in this since it's similar to something I wrote about 6 months ago: feelable touchscreens.
Here are two amazing and innovative technologies, each taking a different approach towards creating tactile sensations from a touchscreen. Senseg use small electric currents to stimulate the skin, creating edges and feelings of texture, while Tactus actually create something physical.
To the best of my understanding, Tactus' technology allows bubbles (I'm reluctant to call them buttons; who knows what else interaction designers could do with this!) in pre-determined locations, configured during manufacture. Different configurations are possible, apparently, but from what I've read it seems that these are decided at manufacture. Whilst this allows some fundamental improvements to the touchscreen experience (e.g. providing a configuration for a keyboard), it lacks some flexibility as manufacture determines where bubbles can be used.
Senseg's tech, however, is more flexible and appears to be truly dynamic; application developers can control the precise location where feelings can be experienced rather than this being decided during manufacture.
Having dabbled with Microsoft Surface over the past year I'm pleased to see that both of these technologies apparently scale well to larger displays. Interactive tabletops suffer from the same loss of tactile feedback that touchscreen mobile devices do although this is perhaps less apparent on a large scale device where widgets aren't crammed into such a small space.
I don't think it's fair to ask which of these technologies is better, because they can't fairly be compared. Although the flexibility of Senseg vs the physical tactility of Tactus is an interesting comparison, I feel that a better question is could these concepts be somehow combined? Imagine a touchscreen which offers complete configuration flexibility, a richer tactile experience like Senseg claim to offer (e.g. feeling texture, not just the presence of something) and the benefits of feeling something physical on the touchscreen. Now that would be awesome.
Wednesday 6 June 2012
Kinect Disconnect
If Microsoft's announcements at E3 are anything to go by then they obviously see Kinect, their hands-free motion-sensing input, as a centrepiece in the Xbox ecosystem. Kinect is also being used increasingly more outside of gaming. There's plenty of interesting examples of novel interaction design centred around Kinect and it's becoming almost ubiquitous in hands-free interaction research. Still, there's one thing that Kinect lacks: tactile response. I feel that this has more of an impact in games than other uses, so that's what I'll discuss here.
Lack of tactile feedback is obvious; if I'm providing input to a game by waving my hands about, I'm not going to feel anything in response to my actions. Actually, that's not technically true. There's some really cool research happening at the University of Tokyo combining Kinect and ultrasound so that you feel like you're manipulating a mid-air object with your hands but this technology (so far) isn't really suited to games. Limitations with their current technology means that it wouldn't scale to the typical Kinect gaming scenario: a user on the opposite side of the living room from the television, moving around a large space.
To recap: Kinect is pretty cool, but it lacks tactile response. Why does this matter in gaming? Interaction in gaming benefits from a closed feedback loop between player and game. As the player provides some input (e.g. mouse click, button press, gesture), they receive some response from the game. This is a continuous loop where the feedback provided (visual, audible, tactile) communicates what effect that action had and allows the player to adjust their actions if appropriate.
I feel that tactile feedback is an important part of this feedback loop: in action games it can confirm success of an action (e.g. you "feel" a punch connect) or inform you some event (e.g. you "feel" your character take a hit). Tactile feedback can also be rewarding. A well designed game combines the aesthetics of the game, good sound design and the feedback from the controller to make a satisfying experience.
With devices such as Kinect you lose this tactile feedback. Touchscreen devices can also suffer from this problem; tablets often lack the rotational motors that mobile phones have to provide low fidelity feedback. Not only does this remove a way of communicating in-game state, but also removes some all-important precision of control. In some games precision is key. Actually being able to hold and feel an input device allows a similar tactile feedback as discussed previously, except this time it's the physical characteristics of the controller which the player feels. Being able to feel the controller and how it responds permits a greater amount of control and precision. A few months ago I wrote briefly about "feelable" touchscreens which could be one way to enrich the mobile gaming experience for touchscreens, but that's another discussion. Back to Kinect.
Kinect is an amazing piece of technology and it, and similar devices, have a great potential in interaction design. I can't help but feel that its use in gaming, however, suffers from the loss of a modality. It potentially detracts from two key aspects of games: communication of state and input precision. Where am I going with this? I'm not really sure. Should Kinect be written off for games? Absolutely not: it does have its uses in more casual games. But what is the future of Kinect in more "serious" games (a term I detest but can't think of an alternative)? Should we strive to develop haptic technologies which make Kinect viable for these games or just continue to treat it as a bit of a gimmick?
Lack of tactile feedback is obvious; if I'm providing input to a game by waving my hands about, I'm not going to feel anything in response to my actions. Actually, that's not technically true. There's some really cool research happening at the University of Tokyo combining Kinect and ultrasound so that you feel like you're manipulating a mid-air object with your hands but this technology (so far) isn't really suited to games. Limitations with their current technology means that it wouldn't scale to the typical Kinect gaming scenario: a user on the opposite side of the living room from the television, moving around a large space.
To recap: Kinect is pretty cool, but it lacks tactile response. Why does this matter in gaming? Interaction in gaming benefits from a closed feedback loop between player and game. As the player provides some input (e.g. mouse click, button press, gesture), they receive some response from the game. This is a continuous loop where the feedback provided (visual, audible, tactile) communicates what effect that action had and allows the player to adjust their actions if appropriate.
I feel that tactile feedback is an important part of this feedback loop: in action games it can confirm success of an action (e.g. you "feel" a punch connect) or inform you some event (e.g. you "feel" your character take a hit). Tactile feedback can also be rewarding. A well designed game combines the aesthetics of the game, good sound design and the feedback from the controller to make a satisfying experience.
With devices such as Kinect you lose this tactile feedback. Touchscreen devices can also suffer from this problem; tablets often lack the rotational motors that mobile phones have to provide low fidelity feedback. Not only does this remove a way of communicating in-game state, but also removes some all-important precision of control. In some games precision is key. Actually being able to hold and feel an input device allows a similar tactile feedback as discussed previously, except this time it's the physical characteristics of the controller which the player feels. Being able to feel the controller and how it responds permits a greater amount of control and precision. A few months ago I wrote briefly about "feelable" touchscreens which could be one way to enrich the mobile gaming experience for touchscreens, but that's another discussion. Back to Kinect.
Kinect is an amazing piece of technology and it, and similar devices, have a great potential in interaction design. I can't help but feel that its use in gaming, however, suffers from the loss of a modality. It potentially detracts from two key aspects of games: communication of state and input precision. Where am I going with this? I'm not really sure. Should Kinect be written off for games? Absolutely not: it does have its uses in more casual games. But what is the future of Kinect in more "serious" games (a term I detest but can't think of an alternative)? Should we strive to develop haptic technologies which make Kinect viable for these games or just continue to treat it as a bit of a gimmick?
Wednesday 9 May 2012
Looks like I'm graduating
For the past seven weeks I've been studying intensely, preparing for my end of year exams. I've sat 6 so far and have 3 remaining. After some good news last week I now know that these will be the last exams I ever sit as an undergraduate. Having been unconditionally accepted to start as a PhD student this autumn, I'm now graduating early with a bachelors degree, rather than the masters degree I always intended to aim for. It's a big step, but I'm excited about it. I enjoy research and now I'll get to do even more; there won't even be pesky lectures and coursework assignments getting in the way of it!
After my last exam a week on Friday I'll have three weeks of a holiday before I start work for summer. I'm going to make the most of it and try to relax; something I've hardly had the chance to do because of exams and working on my dissertation. It's tough (despite what Allie says...) but it always pays off in the end. In that time I'll also be writing a paper on my final year project. After an unsuccessful CHI submission, I'm really hoping that it'll get in to ITS. Getting a publication out of my project work would be a rewarding bonus to what has been an intense, but hopefully successful, project.
After my last exam a week on Friday I'll have three weeks of a holiday before I start work for summer. I'm going to make the most of it and try to relax; something I've hardly had the chance to do because of exams and working on my dissertation. It's tough (despite what Allie says...) but it always pays off in the end. In that time I'll also be writing a paper on my final year project. After an unsuccessful CHI submission, I'm really hoping that it'll get in to ITS. Getting a publication out of my project work would be a rewarding bonus to what has been an intense, but hopefully successful, project.
Subscribe to:
Posts (Atom)