# Conducting Screens: A possible approach to Multi-Device Web Artwork

Canonical URL: https://jeanyoon.ch/oi/texts/conducting-screens-a-possible-approach-to-multi-device-web-artwork
Type: essay / research note
Author: Jeanyoon Choi (최정윤)
Date: 2024-03-20

## Summary

In this 2024 note, Jeanyoon Choi proposes using a phone's motion and acceleration sensors so that a participant's whole-hand gesture, rather than finger taps, shapes audiovisual responses across laptops, projections or other phones, like a conductor shaping relations among instruments. He lists directions from rotating 3D scenes and spatial sound to fluid graphics and text wandering between screens, and asks what is being conducted, what mouse-based conducting would offer, and what happens with two or a hundred conductors.

## 요약 (Korean)

최정윤은 휴대폰의 모션 센서로 여러 스크린을 지휘하듯 움직이는 '컨덕팅 스크린'을 멀티 디바이스 웹 아트워크의 접근법으로 제안한다.

## Concepts defined

- **Conducting Screens**: Choi's approach to multi-device web artwork in which a participant moves a phone like a conductor's baton, shaping audiovisual responses distributed across multiple devices.

## Questions this answers

### How can a smartphone conduct multiple screens in an artwork?

Jeanyoon Choi proposes using device motion and acceleration in JavaScript so that moving the phone changes content across surrounding screens, for example rotating a 3D scene, changing spatial sound volume, switching screens on and off or deforming 2D graphics.

### Why use motion sensors instead of taps in mobile interaction art?

In 'Conducting Screens', Choi argues that everyday mobile interaction is dominated by fingers, and motion sensors shift interaction to the whole hand, a more bodily, phenomenological vocabulary resembling a conductor's gesture.

Topics: multi-device web artwork, conducting screens, device motion sensors, phenomenological interaction, spatial audiovisual composition

References: [Accelerometer](https://en.wikipedia.org/wiki/Accelerometer), [JavaScript](https://en.wikipedia.org/wiki/JavaScript)

## Full text

One mobile phone, multiple screens. A participant moves or handles the phone, and the content distributed across the surrounding screens changes.

Our everyday vocabulary of mobile interaction is dominated by fingers: scrolling, tapping, pinching. What would a more bodily, phenomenological vocabulary look like? Most phones already contain motion sensors and accelerometers. Using them shifts the interaction from the finger to the whole hand. The gesture begins to resemble conducting.

The analogy extends to the outputs. A conductor does not address one instrument in isolation; they shape relationships among instruments distributed through a space. Likewise, the phone could conduct audiovisual responses across laptops, projections or even other phones. What matters is the multiplicity of devices, not a prescribed kind of screen. Cost per device also affects what can be installed.

But a form is not yet a work. Once I have a participant conducting and screens responding, what exactly are they conducting? What context, narrative or sequence of experiences can emerge from this arrangement?

Device motion and acceleration in JavaScript suggest several directions:

A. A three-dimensional scene could rotate with the phone. The viewpoint, the objects, or both could transform.

B. Spatial sound could become louder or quieter with acceleration.

C. A more primitive response could switch a screen on and off, or between black and white. How might that intersect with projection mapping?

D. Two-dimensional graphics could deform, approaching a fluid interface.

E. Screens of text, numbers or maps could be conducted and transformed.

F. Orientation and acceleration could be combined rather than treated as interchangeable inputs. How could the outputs resonate with both?

G. A quieter, more poetic structure might let text wander between screens. Conducting could generate an audiovisual melody from which a story gradually emerges. This returns to the orchestral analogy without reducing it to a literal music controller.

There are possible variations. Why use only a phone? Conducting with a mouse could be equally poetic, with a different, slightly retro history of gestures. Existing web works often confine mouse-driven output to one screen or tab. I want to know what happens when that relationship extends across devices.

And what if there are two participants, each with a phone? Their movements might interfere, reinforce one another or create tension. Sine and cosine functions could help shape those relationships. What happens when there are not two conductors, but a hundred?

---

Jeanyoon Choi (최정윤) is a Korean computational artist and researcher (b. 1999), PhD candidate at KAIST XD Lab, making multi-device web artworks. Profile: https://jeanyoon.ch/oi/en. Index: https://jeanyoon.ch/llms.txt.