"Feelable" touchscreens revisited

Tactus have gotten quite a lot of attention recently after demonstrating their new touchscreen technology (pictured above; image source). Their "Tactile Layer" technology raises bubbles on the touchscreen, creating, essentially, physical objects on the touchscreen. I suppose I've taken quite an interest in this since it's similar to something I wrote about 6 months ago: feelable touchscreens.

Here are two amazing and innovative technologies, each taking a different approach towards creating tactile sensations from a touchscreen. Senseg use small electric currents to stimulate the skin, creating edges and feelings of texture, while Tactus actually create something physical.

To the best of my understanding, Tactus' technology allows bubbles (I'm reluctant to call them buttons; who knows what else interaction designers could do with this!) in pre-determined locations, configured during manufacture. Different configurations are possible, apparently, but from what I've read it seems that these are decided at manufacture. Whilst this allows some fundamental improvements to the touchscreen experience (e.g. providing a configuration for a keyboard), it lacks some flexibility as manufacture determines where bubbles can be used.

Senseg's tech, however, is more flexible and appears to be truly dynamic; application developers can control the precise location where feelings can be experienced rather than this being decided during manufacture.

Having dabbled with Microsoft Surface over the past year I'm pleased to see that both of these technologies apparently scale well to larger displays. Interactive tabletops suffer from the same loss of tactile feedback that touchscreen mobile devices do although this is perhaps less apparent on a large scale device where widgets aren't crammed into such a small space.

I don't think it's fair to ask which of these technologies is better, because they can't fairly be compared. Although the flexibility of Senseg vs the physical tactility of Tactus is an interesting comparison, I feel that a better question is could these concepts be somehow combined? Imagine a touchscreen which offers complete configuration flexibility, a richer tactile experience like Senseg claim to offer (e.g. feeling texture, not just the presence of something) and the benefits of feeling something physical on the touchscreen. Now that would be awesome.

Kinect Disconnect

If Microsoft's announcements at E3 are anything to go by then they obviously see Kinect, their hands-free motion-sensing input, as a centrepiece in the Xbox ecosystem. Kinect is also being used increasingly more outside of gaming. There's plenty of interesting examples of novel interaction design centred around Kinect and it's becoming almost ubiquitous in hands-free interaction research. Still, there's one thing that Kinect lacks: tactile response. I feel that this has more of an impact in games than other uses, so that's what I'll discuss here.

Lack of tactile feedback is obvious; if I'm providing input to a game by waving my hands about, I'm not going to feel anything in response to my actions. Actually, that's not technically true. There's some really cool research happening at the University of Tokyo combining Kinect and ultrasound so that you feel like you're manipulating a mid-air object with your hands but this technology (so far) isn't really suited to games. Limitations with their current technology means that it wouldn't scale to the typical Kinect gaming scenario: a user on the opposite side of the living room from the television, moving around a large space.

To recap: Kinect is pretty cool, but it lacks tactile response. Why does this matter in gaming? Interaction in gaming benefits from a closed feedback loop between player and game. As the player provides some input (e.g. mouse click, button press, gesture), they receive some response from the game. This is a continuous loop where the feedback provided (visual, audible, tactile) communicates what effect that action had and allows the player to adjust their actions if appropriate.

I feel that tactile feedback is an important part of this feedback loop: in action games it can confirm success of an action (e.g. you "feel" a punch connect) or inform you some event (e.g. you "feel" your character take a hit). Tactile feedback can also be rewarding. A well designed game combines the aesthetics of the game, good sound design and the feedback from the controller to make a satisfying experience.

With devices such as Kinect you lose this tactile feedback. Touchscreen devices can also suffer from this problem; tablets often lack the rotational motors that mobile phones have to provide low fidelity feedback. Not only does this remove a way of communicating in-game state, but also removes some all-important precision of control. In some games precision is key. Actually being able to hold and feel an input device allows a similar tactile feedback as discussed previously, except this time it's the physical characteristics of the controller which the player feels. Being able to feel the controller and how it responds permits a greater amount of control and precision. A few months ago I wrote briefly about "feelable" touchscreens which could be one way to enrich the mobile gaming experience for touchscreens, but that's another discussion. Back to Kinect.

Kinect is an amazing piece of technology and it, and similar devices, have a great potential in interaction design. I can't help but feel that its use in gaming, however, suffers from the loss of a modality. It potentially detracts from two key aspects of games: communication of state and input precision. Where am I going with this? I'm not really sure. Should Kinect be written off for games? Absolutely not: it does have its uses in more casual games. But what is the future of Kinect in more "serious" games (a term I detest but can't think of an alternative)? Should we strive to develop haptic technologies which make Kinect viable for these games or just continue to treat it as a bit of a gimmick?

Multimodal Android Development Part 1

This post is the first of two which gives a brief introduction to creating multimodal interactions in Android applications. I'll briefly cover some of the SDK features available to you as an Android developer which you can use to create richer interactions in your apps. Example code will be quite concise because I assume you have at least a basic knowledge of Android development. Feel free to leave any comments suggesting how I can better explain these concepts, or to let me know if I've made any mistakes or omissions.

What is "multimodal" interaction?

Multimodal interaction, put simply, is interaction involving more than one modality (e.g. multiple senses). For example, an application may provide a combination of visual and haptic (touch) feedback. These types of interaction design provide a number of benefits, for example allowing those with sensory impairment to interact using other senses, or allowing interaction in contexts where one sense may be otherwise occupied.

One of the most ubiquitous examples of a multimodal interaction is the way in which mobile phones combine visual, audible and haptic feedback to inform users of a new text, phone call, etc. This combination of modalities is particularly useful when your phone is, say, in your pocket. Obviously you can't see the phone, but you will probably feel the phone vibrate or hear your ringtone as new notifications appear.

Haptic feedback in Android

Most handheld Android devices have some sort of rotation motor in them allowing simple haptic feedback. Although not common in tablets (largely due to size constraints), all modern Android phones will have tactile feedback available. You can control the phone vibrator through the Vibrator class. Note that in order to use this, your Manifest must request the following permission: android.permission.VIBRATE

/* Request the device's vibrator service. Remember to check
 * for null return value, in case this isn't available. */
Vibrator vibrator = (Vibrator) getSystemService(Context.VIBRATOR_SERVICE);

/* Two ways to control the vibrator:
 *  1. Turn on for a specific time
 *  2. Provide a vibration pattern */

/* 1. Vibrate for 200ms */
vibrator.vibrate(200);

/* 2. Vibrate for 200ms, pause for 100ms, vibrate for 300ms. */
long[] pattern = new long[] {0, 200, 100, 300};

/* Perform this pattern once only (repeat := -1). */
vibrator.vibrate(pattern, -1);

/* Vibrate for 200ms, followed by indefinite repeat of
 * 100ms pause followed by 300ms vibrate. Setting
 * repeat := 2 tells the vibrator to repeat at offset
 * 2 into the vibration pattern. */
vibrator.vibrate(pattern, 2);

Touchscreen gestures

Using touchscreen gestures to interact with applications can be fun, efficient and useful when users may be unable to select a particular action on the screen. For example, it can be difficult to select a button on-screen when running or walking. A touch gesture, however, is a lot easier and requires less precision from the user. The disadvantage with touch gestures is that if not used sparingly, there may be too much for the user to remember!

Creating a set of gestures for your application is simple: create a gesture library on an Android Virtual Device using the Gesture Builder application (available on the AVD by default) and add a GestureOverlayView to your activity layout. In your activity, you just have to load the gesture library from your resources and implement an OnGesturePerformedListener.

private GestureLibrary mLibrary;

public void onCreate(Bundle savedInstanceState) {
  ...
  /* 1. Load gesture library from the res/raw/gestures file */
  mLibrary = GestureLibraries.fromRawResource(this, R.raw.gestures);

  if (!mLibrary.load())
    /* Error: unable to load from resources! */
    ...

  /* 2. Find reference to the gesture overlay view */
  GestureOverlayView gov = (GestureOverlayView) findViewById(R.id.gestureOverlay);

  /* 3. Register callback for gesture input */
  gov.addOnGesturePerformedListener(this);
}

The callback method for gesture performance receives a Gesture as an argument. This can be used to obtain a list of predictions: which gestures in your library that Android thought the gesture was. With these predictions, you can use the prediction score (or contextual information) to determine which gesture the user was most likely to have performed. I find it useful to define a threshold for gesture acceptance, so that you can reject erroneous or inaccurate gestures. The best way to choose this threshold value is through trial and error: see what works for you and your gestures.

private static final double ACCEPTANCE_THRESHOLD = 10.0;

public void onGesturePerformed(GestureOverlayView overlay, Gesture gesture) {
  /* 1. Get list of gesture predictions */
  ArrayList predictions = mLibrary.recognize(gesture);

  if (predictions.size() > 0) {
    /* 2. Find highest scoring prediction */
    Prediction bestPrediction = predictions.get(0);

    for (int i = 1; i < predictions.size(); i++) {
      Prediction p = predictions.get(i);
      if (p.score > bestPrediction.score)
        bestPrediction = p;
    }

    /* 3. Decide if we'll accept this gesture */
    if (bestPrediction.score > ACCEPTANCE_THRESHOLD)
      gestureAccepted(bestPrediction.name);
  }
}

private void gestureAccepted(String gestureName) {
  /* Respond appropriately to the gesture name */
  ...
}

Virtual keyboards and "feelable" touchscreens

Senseg made a splash recently when they revealed their touchscreen technology which allows you to actually "feel" objects on-screen. By manipulating small electric charges, users can actually feel texture as they interact with a touchscreen. It'd be too easy to dismiss this as a gimmick, however I think this type of technology has the potential to make a positive impact on mobile devices.

Touchscreens are becoming increasingly ubiquitous in mobile devices, leading to the demise of the hardware keyboard. A glance at the list of all HTC phones in their current line-up shows only two of seventeen phones with a hardware keyboard. Samsung again only offer two phones with a hardware keyboard. While touchscreens offer the ability to eliminate hardware keyboards and other unsightly buttons for the sake of sleek aesthetics, they've so far failed (in my opinion) to provide a suitable replacement for hardware keys.

Yes, touchscreen keyboards are flexible and can offer a variety of layouts, however they still don't give sufficient physical feedback to allow fast touch typing. One reason we're better at typing on physical keyboards is because we "know" where our fingers are. The edges of keys (and the raised bumps often found on some keys) provide reference to other locations on the keyboard. Without looking at the keyboard, an experienced typist can type upwards of 100 words per minute. On a touchscreen, without proper physical feedback, you can expect just a small fraction of those speeds.

One argument against that could be the screen size, however tablets suffer from the same problems. The 26 character keys on my keyboard are of comparable size to the virtual keyboard on my 10-inch tablet. A popular approach to providing feedback for a mobile devices is to vibrate upon key press, however this provides little information other than "you've pressed a key". An alternative approach to making touchscreen keyboards easier to use has been patented by IBM; a virtual keyboard that adjusts itself to how users type on-screen. Auto-correct is another feature which has risen to aid the use of virtual keyboards, yet addresses the symptoms rather than the cause.

Enter touchscreens you can "feel". Actually being able to feel (something which resembles) the edges of keys on a virtual keyboard is likely to make it much easier to type on touchscreen devices. If technology becomes available which allows effective representation of edges (which Senseg claims their technology can), touchscreen devices will be able to offer what is, in my opinion, an improvement to virtual keyboards. I think this could be of particularly great benefit on tabletop computers which, by nature, allow a more natural typing position than handheld devices. Or perhaps this is all just wishful thinking because I go from 110WPM at my desktop to around 5WPM on my phone.