| Input | Visualization task | Why this input fits it |
|---|---|---|
| Voice | Locate an economy by name; filter to a continent | Singapore and Hong Kong are a few pixels on the land map; with a mouse they have to be found and zoomed to before they can be clicked. Saying the name skips both. A continent is also a name rather than a place to point at. |
| Hand gesture | Step through the ranking; zoom in and out | “Next economy” is a coarse command repeated many times, which suits a swipe. It works away from the keyboard, for example when presenting the maps on a projector. |
| Eye gaze | Indicate the country being looked at | The eyes are already on the target. Webcam gaze is noisy, so it only indicates after a dwell. It works best on the cartogram, where economies are large. |
Nothing listens or watches until you press its button. The browser then asks for permission, a red indicator stays in the bottom-left corner while a device is on, and Stop releases it.
| Say | What happens |
|---|---|
| “Show Asia” | Zoom both maps to Asia and grey out the rest; “next” and “previous” stay inside it. Also Europe, Africa, North America, South America and Oceania |
| “Highlight Japan” | Select an economy in both maps (any of the 50 names, and short forms like “UK”, “UAE”, “South Korea”) |
| “Where is Singapore” | Select it and zoom to it |
| “Next” / “Previous” | Step to the next smaller / next larger economy by GDP rank |
| “Zoom in” / “Zoom out” | Zoom to the selection / back to the whole world |
| “Land” / “GDP” / “Play” | Morph the cartogram to land area / to GDP area / play the morph |
| “Log scale” / “Linear scale” | Switch the choropleth colour classes |
| “Select” | Select the country your gaze is indicating |
| “Clear” / “Reset” | Clear the selection / reset everything |
| Show the camera | What happens |
|---|---|
| Swipe right → | Next economy (smaller) |
| Swipe left ← | Previous economy (larger) |
| Thumb up 👍 | Zoom to the selected economy |
| Thumb down 👎 | Zoom out to the whole world |
| V sign ✌️ | Switch the cartogram between land and GDP area |
| Open palm ✋, held still | Reset everything |
Swipe across most of the camera frame in about half a second. Hold a pose still for about half a second; it fires once, so change pose to repeat it. Commands are at least one second apart.
Fallback. Everything above also works without a microphone or camera: the mouse works as in Lab 9 (hover, click, drag, scroll to zoom), the bar above the maps has buttons for every action, the keyboard has ← → previous / next, Z zoom to the selection, Esc reset and V to start listening, and voice commands can be typed into the box below.
Voice control inactive
Camera off
Eye gaze off
When a region is shown, countries outside it turn grey of the same lightness, so their GDP class stays readable but they no longer look selected.
Area represents GDP. Each of the 50 economies covers a share of the map equal to its share of their combined $109.7 trillion.
The cartogram is the better target for gaze: the United States, China, Japan and the large European economies are several times the size they are on the land map, so a noisy gaze estimate still lands inside them.
| Rank | Economy | GDP ($B) | Share of top-50 GDP | Share of land | Share of cartogram | Cartogram ÷ land |
|---|
I extended the Lab 9 GDP choropleth and cartogram with voice, hand-gesture and eye-gaze input. Voice serves locating and filtering, as such “Show Asia” scopes both maps to a continent, and “Highlight Singapore” finds a country that is a few pixels wide on the land map, which the mouse reaches only after zooming. Gestures serve sequential browsing: a swipe steps to the next or previous economy by rank, so the ranking can be walked through like slides, and only six gestures keep recognition reliable. The limits are recognition errors. Short names are easily misheard (“Chile” as “chilly”), so the parser tries the recogniser's alternative transcripts and refuses near misses that fit two countries. An open palm at the start of a swipe can look like “reset”, so a pose must be held still before it fires.
One action layer. Each input follows the lab's
pipeline (input → recognition → interpretation → command
→ D3 update → feedback), and they meet at the command
object. commands.js parses a transcript into a
command such as {type: "region", region: "Asia"}; a
gesture label maps to a fixed command.
Voice uses the Web Speech API (en-US,
one utterance per recognition, interim results shown while you speak).
Country names come from the dataset plus common short forms, and a near miss of
one or two letters is accepted only when it fits exactly one country.
Gestures use MediaPipe's Gesture Recognizer in the browser (one hand, GPU when available). The six poses come from its built-in classifier; swipes are added from the 21 hand landmarks. The camera image is not mirrored, so movement to the user's right is leftward in the image.
Gaze uses WebGazer.js with ridge regression and its Kalman filter, trained by the calibration clicks. Countries without GDP data are ignored.
Regions come from Natural Earth's continent field, written to lab10_regions.csv by build_regions.py, so Russia counts as Europe and Turkey as Asia. Each continent has a longitude–latitude view box; the choropleth zooms to the land inside it, and the cartogram to where those same vertices ended up after the GDP distortion.
Click each dot five times while looking straight at it. Each dot turns blue when it is done. Then keep your head still.