AI Interaction
Prototypes

I design and prototype multimodal AI interaction systems that explore how intelligent assistants perceive users, reason about context, and generate responses during live interaction.

These experiments combine computer vision, spatial interfaces, and human-centered design to visualize how future AI systems may think and respond.

Multimodal AI Interaction Prototype

A conceptual interface exploring how AI systems process perception signals, internal reasoning states, and contextual memory during real-time interaction.

The prototype visualizes camera perception, audio input, emotional inference, environmental context, and response generation inside a continuous AI reasoning loop.

The Design Question

Future AI assistants will interact through continuous perception rather than isolated text prompts.

Designers need ways to understand how AI systems transition between perception, reasoning, and response during real-time interaction.

This prototype explores how an interface might visualize that internal reasoning loop.

Design Approach

Designing interfaces for multimodal AI requires making invisible reasoning processes understandable to users and designers. Rather than treating the assistant as a simple request–response system, this prototype models AI interaction as a continuous cognitive loop.

The interface was designed around three principles:

Legible AI State
The system exposes internal states such as listening, reasoning, and responding so that users can understand how the AI is processing information.

Multimodal Perception
Signals from camera, microphone, and environmental context are represented together to illustrate how future assistants may integrate multiple sensory inputs.

Continuous Interaction Loop
Instead of isolated prompts, the prototype models AI behavior as an ongoing cycle of perception, context formation, reasoning, and response.

Interactive Demo

Click Next State to simulate how the AI transitions between perception, reasoning, and response. Interactive prototype — best experienced on desktop

System Architecture
(Animated data flow)

Interaction Model

This prototype investigates how multimodal AI systems may structure internal reasoning during real-time interaction.

The system models four layers of intelligence:

• Perception – capturing signals from camera, microphone, and environment
• Context Memory – tracking conversation history and situational context
• Reasoning State – transitioning between listening, thinking, and responding
• Response Generation – producing language and actions in response to the user

The goal is to make AI cognition legible and interactive, allowing designers and engineers to visualize the internal loop of intelligent systems.

Future Extensions

• Real-time camera vision inference
• Live microphone transcription
• Emotion detection from facial signals
• Memory persistence across sessions
• Spatial UI for AR glasses or XR interfaces
• Tool invocation (maps, weather, search APIs)

Why Multimodal AI Interfaces Matter

Most AI systems today operate through text-only interfaces. However, future intelligent systems will operate through continuous perception, integrating vision, audio, spatial context, and memory. Designing these systems requires new interaction models that make internal reasoning visible and understandable. This prototype explores how those interfaces might work.

Technical Stack

Frontend: React
Backend: Node.js
AI model: OpenAI / LLM API
Architecture: multimodal perception–reasoning loop
Deployment: Vercel

Potential Integrations

Whisper
MediaPipe
LangChain
Vision models

AI Interaction Prototypes