GestureCV

GestureCV lets me control my laptop with hand gestures through the webcam. It follows my index finger to move the cursor, a pinch of my thumb and index finger clicks, and a peace sign opens YouTube. I built it because I watch YouTube while I eat and didn't want to touch my trackpad with messy hands, and because I wanted to try out computer vision tools.

When your hands are busy or messy, you still end up touching the trackpad to pause a video or open the next one. I wanted a hands-off way to point, click and get to what I was about to watch, using only the webcam already on the laptop. Longer term, a more polished version could help people who can't use a traditional mouse and keyboard.

Each webcam frame from OpenCV goes through Google's MediaPipe Hands model, which returns 21 landmarks for one hand, and the landmarks are drawn back over the video so you can see what it's tracking. The index fingertip's position is mapped from camera coordinates to screen coordinates and mirrored, and PyAutoGUI moves the cursor there. A click fires when the distance between the thumb and index fingertips drops below a small threshold, and a peace sign, detected when the index and middle fingertips sit above their lower joints, opens YouTube through Python's webbrowser module. I also tried running it in Docker, but the container couldn't reach the webcam and display, so the repo keeps a sample Dockerfile.

It works as a hands-free mouse for the basics: pointing, clicking and opening YouTube from across the table. Next would be smoother cursor movement, more gestures for things like scrolling and pausing, and tuning the click threshold so it fits different hands and cameras.