An interactive, gesture-controlled 3D hologram. Like it's straight out of an Iron Man movie, you can use voice commands and natural hand movements / gestures to manipulate objects on a realistic 3D projection.

I gave a talk on OpenCV Live about my technical innovation process in building Jarvis, check out the YouTube link!

The hologram is made with a Pepper's Ghost illusion, and it's built with wood and acrylic — approximately 2.5 feet wide and 1.5 feet tall. On the software side, it's made with Next.js, Cloudflare R2, Flask, OpenCV, Google Mediapipe, Socket.IO, and 3D models from Sketchfab.

Hand tracking in 3 dimensions was tricky: First, I analyzed checkerboard images at various angles and computed a distortion matrix to fix the camera's mild fisheye effect. Then, I calculated distances between hand landmarks, comparing them to real-world depth, then transformed this data into uniform motion with a linear regression.