# Two simple experiment ideas using multi-modality

Canonical URL: https://jeanyoon.ch/oi/texts/two-simple-experiment-ideas-using-multi-modality
Type: essay / research note
Author: Jeanyoon Choi (최정윤)
Date: 2024-04-07

## Summary

A short note listing multimodal inputs, from hand, body and eye tracking to voice, phone gestures, keyboard and mouse, and proposing two experiments. The first is a phone list whose items expand when looked at and collapse when attention moves, possibly extended to a gaze-responsive grid of fifteen screens. The second uses lip distance to detect an open mouth in Das Gerede Part II: three screens carry on a dull conversation that is interrupted by yawns and then screams, turning bodily boredom into a nonsemantic symphony.

## 요약 (Korean)

최정윤은 시선 추적으로 펼쳐지는 리스트 UI와, 입 벌림을 감지해 세 화면의 지루한 대화에 하품과 비명을 끼워 넣는 Das Gerede Part II 실험이라는 두 가지 멀티모달 인터랙션 아이디어를 제안한다.

## Questions this answers

### How does Das Gerede Part II use mouth tracking?

In Jeanyoon Choi's proposal, three screens hold a dull conversation; when the participant opens their mouth, measured by lip distance, yawns and then screams enter, gradually replacing the conversation with nonsemantic sounds.

### What is a gaze-controlled expandable list?

It is an interface Jeanyoon Choi proposes in which looking at a list item expands it with more detail and looking elsewhere collapses it, potentially scaled up to a grid of screens responding to attention.

Topics: multimodal interaction, eye tracking, mouth tracking, face tracking, multi-screen installation, nonsemantic sound

References: Das Gerede Part II, [Eye tracking](https://en.wikipedia.org/wiki/Eye_tracking)

## Full text

Two experiments with multimodal input. The available vocabulary includes hand, body and eye tracking; facial expression and voice; scrolling, pinching, tapping and conducting with a phone’s accelerometer; and the keyboard and mouse.

1. An expandable list controlled by gaze.

Imagine a vertical list of ten or more items on a phone. Looking at an item expands it to reveal further detail. Looking elsewhere collapses it and expands the newly attended item. This could be a UI experiment, but it could also extend into a spatial artwork. What would a gaze-responsive grid look like? What if fifteen separate screens expanded or changed according to the participant’s attention?

2. Yawning and screaming through mouth tracking.

The distance between the lips provides a simple measure of whether a mouth is open. I want to use that movement to interrupt a conversation in Das Gerede Part II.

Three screens—R, G and B—carry on an ordinary, coherent, painfully dull conversation. When the participant opens their mouth, a yawn unexpectedly enters the exchange. The conversation resumes, until another opening triggers a scream.

Gradually, semantic conversation gives way to animal-like sounds, yawns and screams: a nonsemantic symphony distributed across three screens. The bodily expression of boredom becomes the means of dismantling the boring conversation.

---

Jeanyoon Choi (최정윤) is a Korean computational artist and researcher (b. 1999), PhD candidate at KAIST XD Lab, making multi-device web artworks. Profile: https://jeanyoon.ch/oi/en. Index: https://jeanyoon.ch/llms.txt.