Selections In Mobile Cross-Device Interfaces

Selections In Mobile Cross-Device Interfaces

In my Master’s thesis Exploring Selection Management and Brushing and Linking for Mobile Cross-Device Interaction, I developed a UI concept and a web prototype for combining multiple mobile devices to explore information visualizations. I mostly focused on how to create, manage, and interact with selections of data items in environments consisting of multiple mobile devices.

Tools

  • HTML/CSS
  • Typescript
  • Node.js
  • D3.js

How It Started

When I started my Master’s thesis and was deciding on a topic, my goal was to focus on visualization in combination with mobile devices. Fortunately, I got the opportunity to expand existing work on visualizations distributed across mobile devices, called VisTiles. It builds upon the idea that through combining multiple mobile devices, we can increase the screen space and show additional perspectives of the data. The task for my thesis was to explore brushing and linking, which is a common interaction technique for visualizations, in the VisTiles setting. The basic concept behind brushing and linking is that you select data items in one visualization and the same items are highlighted in another visualization.

What I Did

First, I decided to analyze existing research in the areas of cross-device interaction, visualization on mobile devices, multiple coordinated views, brushing and linking, and finally, selection management.

While exploring brushing and linking, I discovered that it often benefits from being supported by selection management techniques. By incorporating them, we can store selections that we’ve created, we can revisit them later, or combine them if we want. My procedure was to build on the foundation of brushing and linking, which is often temporary, and extend it to a selection-based approach that is more permanent.

Inspired by and based on the related works I’ve found, I developed an interaction and UI concept that addresses temporary and permanent selections for multiple devices. Furthermore, I created the concepts based on the assumption that the devices are spatially aware, i.e., they are aware of their own position and they know the position of all other devices in the workspace. Although this is currently far from reality for our everyday devices, the tracking devices in the lab made it possible to have spatially aware devices (you can find more details on the VisTiles page). 

My analysis showed that there are several degrees of freedom, i.e., parameters which I wanted to support:

  • Interacting with selections by a single person vs multiple people at a time
  • Temporary vs permanent, i.e., stored, selections, and
  • Using only a single device vs using multiple devices.

For these different scenarios, I developed my concepts and I will now talk about some exemplary techniques. 

The Result

Basic Interaction With Selections

To be able to incorporated more advanced techniques into my concept, I first had to define some basic interactions and a visual language to create and interact with selections. I developed a consistent set of interaction techniques, inspired by common designs and the analyzed literature. 

Offloading Menus To Other Devices

One important aspect of my thesis was how different devices can fulfil different roles in a scenario with multiple devices. One assumption was that larger devices, such as tablets, are more suitable for visualizations since they can present more data due to their larger screen than smartphones. Consequently, I assumed that smartphones can be especially beneficial for organizational tasks and interface elements, such as menus. Therefore, the idea was to offload menus, often hamburger menus, and other hidden interface elements to other devices. This is done by using the spatial awareness of the devices and placing an “empty” device next to a device with a menu. This shows a prompt on the edges of both devices that offers to offload the menu. In the example below, the offloaded menu is a list that contains all selections in the workspace and allows the user to manipulate them. This shows the benefit of an offloaded menu since the list with all selections is always visible and accessible to all users through a separate device.

Combining Two Selections With Each Other

I also wanted to explore the spatial arrangement of the mobile devices and it can be used for interacting with selections. The example below shows how two selections can be combined with each other using the spatial arrangement of the devices. Two devices, each representing a selection of data items, are placed next to each other which triggers a menu at the edges. This menu offers different ways of combining two selections: Merging them into one selection, creating their intersection, removing shared elements, and subtracting one selection from another. Using one device for a single  selection and spatially manipulating the devices to manipulate and combine the selections results in a more tangible sensation and potentially a better mental grasp for the selections.

Manipulating Selections Across Devices

Spatially arranging devices is not limited to placing devices next to each other. They can also be aware of the positions of devices on a table and in their vicinity. One option is to use a smaller device as a central hub for managing selections in a workspace. The example below shows multiple devices with visualizations and one smartphone that serves as a hub. It is aware of the other devices’ positions and which visualizations they show. Therefore, little icons of the visualizations are places around the edges and function as “portals”. The hub currently represents a certain selection and by tapping on the portals this selection can be shown on a chosen device. Having a central hub that is spatially aware of all other devices around might increase the awareness for the other parts of the workspace and might help in identifying single devices.

After having developed a coherent concept, I created some mockups on the target devices to get a feel for the overall feel of my components. Finally, I also implemented parts of the concept using Typescript, Node.js, and D3.js.

Lessons Learned

I mostly focussed on pure UI design and used only basic interactions. If I had the chance to do it again today, I would try to incorporate more advanced interaction techniques that make use of the devices’ mobility, their various sensor capabilities, and their form factor.

One part, that I appreciated a lot, was the really fruitful and intense theoretical discussions about visualization concepts, like What exactly is a selection? And how is it different from a filter or is it different at all? , which led to even more thoughts and let me reflect very deeply about the concepts I developed.

Writing my thesis in English was one of the best decisions I could’ve made. I think my writing improved tremendously while working on my thesis and I learned a lot about scientific writing.