BIG

Bristol Interaction and Graphics

MUSTARD

Abstract


There are many situations wherein more than one person desires to interact with or gather information about a physical object without being able to directly touch the object. Such situations are common in museum setups, shop windows etc. Usually, if there is a digital interface for displaying information about the object, people are forced to take a turn-based approach to gain access to the digital information. While wearable AR solutions can be used, they do not allow a walk-up and play approach to the object (apart from requiring a standardized content delivery model).
Multi-User See-Through Augmented Reality Display (or MUSTARD) is aimed at delivering multi-view digital content that augments the physical objects while allowing the user to interact with the system without any hardware on person. Thus the key features of MUSTARD are:

  • Multi-view content delivery based on the physical position of the user in front of the display.
  • See-through display that augments the physical objects without affecting the visibility of the object.

Best Paper Honorable Mention MUSTARD received the 'Best Paper Honorable Mention' award at CHI 2012.

Videos


Details


Basics:

Multi-User See-Through Augmented Reality Display (or MUSTARD) delivers multi-view digital content that augments the physical objects while allowing the user to interact with the system without any hardware on person. This requires two abilities:

Multi-view content delivery:

  • There are many options to deliver multi-view content. Multiview capable systems rely on parallax barriers, lenticular lens arrays, microlens arrays, switchable diffusers, holographic optical elements (HOEs) as some of the main methods for delivering multiview information using a 'single' data panel.
  • MUSTARD relies on a concept called Random Hole Displays (RHD) which is similar to parallax barriers as it also relies on revealing only certain parts of the screen to a single viewer at a time. The major advantage of RHD over parallax barriers is that it can generate multiple simultaneous views instead of the stereo-views parallax barriers are capable of.
  • The 'Random Hole' part means that any user sees a randomly distributed points on the display at any one time through a hole-mask. The randomly positioned holes and the position of the viewer determines what part of the screen is visible to the viewer.
  • While a static hole-mask RHD will only ever show a small fraction of the screen, MUSTARD uses a dynamic hole-mask which reveals different parts of the screen over time and hence provides full coverage of the display.
  • Even with RHD, more than one user can see some parts of the screen. These parts of the screen are called conflict pixels. To resolve what should be shown at these conflict pixels, MUSTARD uses a conflict resolution function which generates a color which is least different from each of the colors to be delivered to the users seeing that pixel.

See-through capability:

  • All methods capable of delivering multiview content described above modify all the light that passes through them (the only exception is HOE) and hence cannot be used for see-through purposes.
  • MUSTARD utilizes the principle of operation of liquid crystal panels (LC) to achieve see-through capability. The LC panels work by blocking polarized light to display data. By stacking a modified LC panel (back polarizer removed) in front of an unmodified LC panel, MUSTARD allows any object placed between the panels to be visible.
  • This works because any light reflected from the object, if unpolarized, will pass through the front panel without being affected. On the other hand, polarized light exiting from the rear LC panel will be affected by the front LC panel making data on the front LC panle visible. [see figure below]

LC Operation

 

 

Publications:

Papers

  • 2012-May. MUSTARD: Multi-User See-Through Augmented Reality Display. Abhijit Karnik, Walterio Mayol-Cuevas, Sriram Subramanian. In Proceedings of the 30th International Conference on Human factors in Computing Systems. ACM, New York, NY, USA. 10 pages. (View) Best Paper Honorable Mention
  • 2011-October. MUST-D: Multi-User See Through Display. Abhijit Karnik, Walterio Mayol-Cuevas, Sriram Subramanian. In Adjunct Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (View) (Cite)

Posters

  • MUST-D: Multi-User See Through Display in UIST 2011 Posters. (View)

 

Funding
MUSTARD is partially supported by Microsoft Research through the PhD Scholarship Programme & by a University of Bristol Pump Priming grant.

 

Photos:

*Do note that these images and videos are from a system that uses time-multiplexed representation of visual information. It's capabilites are therefore not captured to the level we would be happy with*
Youtube link (here)
*Click for larger resolution images

Device Image

MUSTARD showing two different views for two different positions.

View 1  View 2

MUSTARD close-ups showing see-through operation

 See-through  See-through

User study evaluating conflict resolution functions
User Study What user sees