IXD415 // Emerging Tech
Project 1

Weird Mirror

A moving image that responds.

So much cool interactive work involves video, but video itself isn't very interactive. You can make a beautiful demo in After Effects or a compelling bit in CapCut, and it does exactly the same thing every single time, forever. It's functionally not interactive.

But what if you could make it interactive?

In this project we're going to make a moving image that responds to a user. Something that is different because you interact with it.

We'll be working in TouchDesigner. It's a node-based visual programming environment that is used for a lot of the interactive installation work you've seen in museums, concerts, and lobbies. The learning curve is pretty steep, but we can connect an MCP to Claude, and then you can describe what you want in English and get help building it.

Read this part twice

WHy TouchDesigner?

Because it's unfamiliar. TouchDesigner is a good practice case for an intimidating technology.

First, we find out what the thing can actually do

Not only what the marketing says it does. Read the docs, watch somebody use it, poke it, break it, build the smallest thing that works. You are mapping the edges of a box.

Then aim your existing toolkit at it

You've spent three years learning how to design interactions, and your skill set still applies here.

  • Input and affordances. What can a person do here, and how do they know they can do it?
  • Output and feedback. What happens when they do, and how do they know it happened?
  • Constraints. What does this medium refuse to do? What does the lcation, the time, the hardware, the audience know how to do or refuse to do?
  • And then the normal loop. Ideate, prototype, test, refine, and do it again.

The failure mode of emerging tech work is stopping after the first part. Plenty of people learn a new tool and then make a demo that shows off the tool. But a demo is not usually an interesting design. We want to design an iteraction, and get people using it.

That's part of why this project will be evaluated on whether a stranger understands your piece, and not on how complicated your node network is.

You wil probably run this same sequence again in the spring with different tools. The tech can change but your process should still apply.

The Setup

Your client

No client. But you do have a user, and she is very specific: she is a person who walks up to your piece having never seen it, having read nothing, with nobody standing next to her explaining it. We want your projectto intrigue, entrance, delight, unsettle, horrify, etc this person, without any help text.

Weird Mirror

This is my mental model here; your mileage may vary.

  • Mirror. The input is the viewer. They approach your piece, and something changes. Webcam, microphone, presence, motion. The piece responds to the body in front of it.
  • But your input doesn't have to be visual. It could be audio. Or a keystroke? or a mouse movement? Or? or? or?
Deliverables
  1. A project repo. This will be a public artifact in github. I want you to be keeping notes and progress-documentation here. I'll give you specifics as we go.
  2. A working .toe file. Touchdesigner runs on your machine.
  3. A live demo, twice. An in-progress one a final. Your project is running, with people walking up to it.
  4. A documentation video, 60–90 seconds. There must be a human in the frame. A screen capture of your output is not documentation of an interactive piece; it's just a video, which is the exact thing we're trying to get away from.
  5. A build log, in markdown, committed to your repo. What you tried, what broke, what you asked Claude, what finally worked. Rough notes are fine, we want these honest about your process, and not a retrospective.

    Note that touch designer files (.toe) can be pretty big, so we're NOT going to commit them to our project repo.
  6. Some in-between things. I'll have some in-between research tasks for you that we'll add ot your repo.
Constraints
  • It has to run for three minutes without crashing and without you touching it.
  • A stranger has to be able to use it without instructions.
  • Your laptop, a webcam, a mic, a keyboard, a screen, and an internet connection are the baseline, and a good project can be made with that.
  • You may bring in other hardware. Other sensors, other input devices, projectors, motors, lights, whatever the idea needs. Do it at your own risk and your own budget. I can't guarantee it'll work, and you don't want to get slowed down by logistics. But feel free to play with options! If the hardware is the whole idea, you may need a fallback.
  • You may install it somewhere other than our classroom. LIke a hallway, a stairwell, some corner of Chalmers or Marvin. Two rules: it has to be within walking distance, and it has to be accessible to all of us during class time. Tell me early if you're going this direction.
  • We'll have about four weeks.
Go look at these

Inspo

Interactive work that's mostly about a body in a room.

Getting started with the tool

Learn from your studiomates

Repos

Here's a list of everybody's project repos:

Four weeks, ish

Sessions

Session I

What is a loop?

  • Project brief: a moving image that responds
  • Discussion: what's the actual difference between a video and an interaction? A video is a recording. An interaction is a loop: input, change, feedback, and the viewer noticing the feedback.
  • Discussion: which of those four does most "interactive" work actually get wrong?
  • Weird Mirror concept
  • Demo: what a TouchDesigner network looks like. What a node is. Graph not timeline.
  • Install TouchDesigner (the non-commercial license is free)
  • Claude & CLaudecode in the app. Letting claude do stuff in a folder for you.
  • What's an MCP?
  • In class: new repo, README.md, log.md, image.
Homework

Get TouchDesigner installed and get the Claude MCP connected. Then get anything on screen. Your webcam feed with a single small effect on it is plenty. Screenshot it. Put the screenshot in your repo.

Homework

Start a new project repo. Make it public so we can see it. Add a readme.md. Start a log.md.

And homework

Something was probably broken or difficult in this process. Write down what broke, what you asked Claude, and what happened next. Add that to log.md

Session II

Field Trip!

  • We're visiting Atlas9 in Kansas City and looking at a big complicated touchdesigner project.
Homework

Same as from Wednesday. Get something running and we'll review it together!

Session III

What can this thing actually do?

  • Everybody shows their one working thing. Two minutes each.
  • Where did it break? We put all the failures on the board and look at them together. I expect overlap.
  • Survey of the parts: TOPs, CHOPs, SOPs, DATs: not a deep dive, just enough to know what to reach for
  • What inputs are actually available to us: webcam, mic, mouse, keyboard, clock, live data, MIDI, a phone
  • What outputs: a screen, a projector, sound, a window in a dark room
  • Discussion: what are the design constraints here, and what are the opportunities?
Homework

Find three works from other designers/artsts that use responsive video or a responsive image. Have links to something with the designer/credits ready. Be ready to define the inputs and outputs.

And homework

Ten concepts. Write it down: a title, hat's the input, what changes, how does the viewer know? Make a new markdown file called 'concepts.md' and put it in your repo. Add a snap/scan/image of your sketches too.

And homework

And update your log.md!

Session IV

Pitch a concept

  • Pitch us your three favorites out of the ten
  • Class critique
  • What's the affordance/action/feedback loop?
  • Pick one.
Homework

Build the smallest possible version that demonstrates thte affordance/action/feedback loop closes. It's OK if it's ugly. Update your BUILD log!

And homework

If your piece needs hardware or a different location outside this room, plan it now

Session V

Show & Tell your POC

  • Show us your proof of concept. Does it actually have an interaction loop?
  • Suggestions/critiques. Is it working? Should you continue this one?
  • Make the call today: refine or pivot
Homework

Get the whole flow working start to "finish", even if some parts suck. A complete bad version is better now than a nice fragment. Update your build log. Post a screenshot or update.

Session VI

Did you get your full flow?

  • Quick round: what's blocked? Are you stuck?
  • Live work: we're going to debug two or three of these together. Group support/therapy!
  • Work session
Homework

By Wednesday you've got the full interaction flow, with some degree of polish. Have your build log updated with screenshots included.

Session VII

What do you have now?

    You should have:
  • a mirror that reflects a person, somehow.
  • affordances that help a user know what to do.
  • Actions a user can perform
  • Feedback to help the user know what they've done
  • And some kind of journey that the user goes on.
Homework

User test this thing, with people who don't have any background on it. Find a way to guerrilla test. Document that, snap a photo of a stranger testing. Add notes to your build log.

Fix whatever the strangers didn't get. That's more important than any other polish you have remaining! Then make it beautiful.

Session VIII

Finish it

  • Work session, final push
  • Optional check in. How did your user tests go? What did you learn, what did you fix, are you blocked anywhere?
Homework

Finish. Record your documentation video — 60 to 90 seconds, human in the frame. Write your build log. Commit everything.

Session IX

Final

  • Final installs, running live, ready for a critique.
  • We demo each project with someone else, then we discuss!
  • Look at Documentation videos?
Homework

Nothing. Push your any last changes to your repo: notes, video, build log. We're done!