Topic1: Connection – Digital Mirror in Time
A website acting as a mirror as well as a temporal bridge between us. A real-time website, tentatively named ” Digital Mirror in Time”, aims to explore shared human expressions and interactions over time using advanced AI technologies like Google AI Edge API, MediaPipe Tasks, and Face Landmark Detection. The project transforms your computer’s camera into a digital mirror, capturing and storing facial data and expression points in real time.
Inspiration: Sharing Face, Visiting the installation at either location would match your expression and pose in real time with these photos of someone else who once stood in front of the installation. Thousands of people visited the work, and saw themselves reflected in the face of another person.
- Plan A: Sharing Smiling via Face Landmarker API api doc , demo reference
QA: can I get exactly the 478 landmarks results, currently I experimented and only can get face_blendshapes as output.
- Facial Data Capture:
The website uses the computer’s camera to detect a user’s face, leveraging the MediaPipe Face Landmark Detection model. This model identifies key facial landmarks (such as eyes, nose, mouth, etc.), storing this data along with the corresponding positions on the screen. - Expression Storage (potential: Firebase)
The user’s facial expressions are stored in a premade database, including information like facial positions, angles, and specific expressions (smiling, frowning, etc.). This creates a digital archive of faces and expressions over time. - Facial Expression Matching and Dynamic Interaction (potential: Next.j + Prisma)
When a new user visits the website, their live camera feed is processed in the same way, and the system searches the database for expressions that match the current facial landmarks. When a match is found, the historical expression is retrieved and displayed on the screen, overlaying in the exact position.
This creates an interactive experience where users not only see their own reflection but also discover others’ expressions from different times, creating a temporal bridge between users. The website acts as a shared space where facial expressions transcend individual moments.
- Pan B: Hand in Hand via HandLandmarker API + Sensel Morph/ Thermal Camera, api doc, demo reference
Concept: We often perceive the images of people on our screens as cold and devoid of warmth. This project explores whether we can simulate the sensation of touch between people through a screen by combining visual input and haptic feedback.
- Using a Hand Landmarker API, the system recognizes and tracks the back of the user’s hand in front of a camera.The user places their palm on a Sensel Morph (or a similar device) that captures pressure data, creating a heatmap of the touch.
- The pressure data is then stored in a database, linked to the visual representation of the hand. Implement algorithms to match the hands of future users with those previously recorded, based on hand shape and position.
- When another user places their hand in the same position on the screen, the system matches their hand’s position and visual similarity to the previous user. Display the hand pressure heatmap on the screen when a matching hand is detected, simulating the sensation of touch visually.
Topic2: Broken and Distorted Portrait
What I find interesting about this theme is the distorted portrait. I thought of combining it with sound. When adding water to a cup, the refractive index will change, and the portrait will also change. At the same time, the sound of hitting the container can be recorded.
