Welcome to another innovative IoT project tutorial from YaranaIoT Guru — your ultimate destination for smart electronics, IoT innovations, and ESP32-based automation systems.
In this project, we’ll create a Home Automation Gesture Control System using OpenCV, MediaPipe, and ESP32 that can recognize your face and finger gestures to control home appliances like lights, fans, and more.
This project combines computer vision (OpenCV + MediaPipe) with microcontroller-based control (ESP32 + Relay), giving you a futuristic smart home experience — all powered by Python and IoT technology.
🎯 Project Overview
The goal of this project is to:
✅ Detect and recognize faces using OpenCV + Face Recognition library
✅ Track hand and finger gestures using MediaPipe
✅ Send control commands to ESP32 to turn appliances ON/OFF
✅ Create a smooth, real-time gesture + face-based automation system
Imagine simply looking at your camera or making a hand gesture — and your room light turns ON automatically!
⚙️ Components Required
| Component | Description |
|---|---|
| ESP32 | WiFi + Bluetooth microcontroller used for automation control |
| Relay Module | To switch ON/OFF home appliances |
| Webcam / Laptop Camera | For face and gesture detection |
| PC/Laptop | Running Python (OpenCV + MediaPipe) |
| Jumper Wires | For ESP32 to Relay connection |
| Power Supply | 5V/12V depending on relay and load |
💻 Software & Libraries Required
Install the following Python libraries before running the code:
pip install opencv-python
pip install mediapipe
pip install face-recognition
pip install numpy
Also make sure to have Python 3.7+ installed.
🧩 Project Flow Diagram
Camera → OpenCV (Face & Gesture Detection) → Python → WiFi (HTTP/MQTT) → ESP32 → Relay → Home Appliances
🧱 Folder Structure
HomeAutomation_GestureControl/
│
├── known_faces/
│ ├── Abhishek.jpg
│ ├── FriendName.jpg
│
├── gesture_control.py # Part 1 – Hand/Finger Tracking
├── face_recognition_main.py # Part 2 – Face Recognition
├── test_face_recognition.py # Part 3 – Face Recognition Test
├── combined_system.py # Part 4 – Final Integration
└── esp32_relay_code.ino # ESP32 Control (optional)
🧩 PART 1 – Gesture Control Using MediaPipe (Hand Tracking)
In this part, we’ll use MediaPipe by Google to detect and track hand gestures in real-time.
These gestures can later be mapped to send commands to the ESP32 (e.g., ON/OFF).
🧠 Working Logic
- Capture video using OpenCV.
- Detect hand landmarks using MediaPipe.
- Identify finger positions (open/closed).
- Send corresponding signal to ESP32.
💻 Code – gesture_control.py
import cv2
import mediapipe as mp
import time
mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
hands = mp_hands.Hands(min_detection_confidence=0.7, min_tracking_confidence=0.7)
cap = cv2.VideoCapture(0)
prev_time = 0
while cap.isOpened():
success, image = cap.read()
if not success:
break
image = cv2.flip(image, 1)
image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
result = hands.process(image_rgb)
if result.multi_hand_landmarks:
for hand_landmarks in result.multi_hand_landmarks:
mp_drawing.draw_landmarks(image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
curr_time = time.time()
fps = 1 / (curr_time - prev_time)
prev_time = curr_time
cv2.putText(image, f'FPS: {int(fps)}', (10, 70), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2)
cv2.imshow('YaranaIoT Guru - Hand Tracking', image)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
🧾 Output:
It tracks your hands in real-time, showing landmark points. Later, these points can be used to detect gestures like:
- ✋ Palm open → Turn ON Light
- ✊ Fist closed → Turn OFF Light
🧩 PART 2 – Face Recognition System
Now we move to face detection and recognition, using the face_recognition library to identify users and display their details.
💻 Code – face_recognition_main.py
import cv2
import face_recognition
import numpy as np
import os
known_face_encodings = []
known_face_names = []
person_details = {}
known_faces_dir = "known_faces"
for file in os.listdir(known_faces_dir):
if file.endswith(".jpg") or file.endswith(".png"):
name = os.path.splitext(file)[0]
image = face_recognition.load_image_file(os.path.join(known_faces_dir, file))
encoding = face_recognition.face_encodings(image)[0]
known_face_encodings.append(encoding)
known_face_names.append(name)
if name == "Abhishek":
person_details[name] = {"age": 25, "occupation": "Student", "hobby": "Coding"}
elif name == "FriendName":
person_details[name] = {"age": 28, "occupation": "Engineer", "hobby": "Gaming"}
🧠 Working Principle
- Load known faces from the
known_faces/folder. - Extract facial encodings for comparison.
- Compare live webcam frames with known encodings.
- Display the person’s name and details in real time.
🧾 Output
When your camera sees a known face (like Abhishek), it automatically displays:
Name: Abhishek
Age: 25 | Occupation: Student | Hobby: Coding
🧩 PART 3 – Face Recognition Test
This is a simple file to check whether your face_recognition library is installed correctly before running the main program.
💻 Code – test_face_recognition.py
import face_recognition
print("face_recognition imported successfully!")
✅ Run this first using:
python test_face_recognition.py
If it prints the message, you’re good to go!
🧩 PART 4 – Final Integration (ESP32 + Face + Gesture)
Now it’s time to combine everything — Face Recognition, Hand Tracking, and ESP32 Control into one final smart automation system.
💡 Idea
- When a known face is recognized → Enable gesture control mode.
- When a specific gesture is made → Send a command to ESP32 via WiFi.
- ESP32 receives the command → Activates the relay → Controls appliances.
💻 Combined Code – combined_system.py
import cv2
import face_recognition
import mediapipe as mp
import numpy as np
import os
import requests
# ESP32 WebServer URL (change IP as per your setup)
ESP32_URL = "http://192.168.1.10/control?device="
# Face Recognition Setup
known_face_encodings = []
known_face_names = []
known_faces_dir = "known_faces"
for file in os.listdir(known_faces_dir):
if file.endswith(".jpg") or file.endswith(".png"):
name = os.path.splitext(file)[0]
image = face_recognition.load_image_file(os.path.join(known_faces_dir, file))
encoding = face_recognition.face_encodings(image)[0]
known_face_encodings.append(encoding)
known_face_names.append(name)
# Hand Tracking Setup
mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
hands = mp_hands.Hands(min_detection_confidence=0.7, min_tracking_confidence=0.7)
cap = cv2.VideoCapture(0)
authorized = False
while True:
ret, frame = cap.read()
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
face_locations = face_recognition.face_locations(rgb_frame)
face_encodings = face_recognition.face_encodings(rgb_frame, face_locations)
for (top, right, bottom, left), face_encoding in zip(face_locations, face_encodings):
matches = face_recognition.compare_faces(known_face_encodings, face_encoding)
name = "Unknown"
face_distances = face_recognition.face_distance(known_face_encodings, face_encoding)
best_match_index = np.argmin(face_distances)
if matches[best_match_index] and face_distances[best_match_index] < 0.6:
name = known_face_names[best_match_index]
authorized = True
cv2.rectangle(frame, (left, top), (right, bottom), (0, 255, 0), 2)
cv2.putText(frame, name, (left + 6, bottom - 6), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2)
if authorized:
# Hand Gesture Detection
result = hands.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
if result.multi_hand_landmarks:
for hand_landmarks in result.multi_hand_landmarks:
mp_drawing.draw_landmarks(frame, hand_landmarks, mp_hands.HAND_CONNECTIONS)
index_finger_tip = hand_landmarks.landmark[8].y
thumb_tip = hand_landmarks.landmark[4].y
if abs(index_finger_tip - thumb_tip) < 0.05:
print("Gesture: Light ON")
requests.get(ESP32_URL + "on")
elif abs(index_finger_tip - thumb_tip) > 0.1:
print("Gesture: Light OFF")
requests.get(ESP32_URL + "off")
cv2.imshow('YaranaIoT Guru - Face + Gesture Control', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
⚙️ ESP32 Code (Relay Control)
#include <WiFi.h>
#include <WebServer.h>
const char* ssid = "Your_SSID";
const char* password = "Your_PASSWORD";
WebServer server(80);
#define RELAY_PIN 5
void handleOn() {
digitalWrite(RELAY_PIN, HIGH);
server.send(200, "text/plain", "Light ON");
}
void handleOff() {
digitalWrite(RELAY_PIN, LOW);
server.send(200, "text/plain", "Light OFF");
}
void setup() {
Serial.begin(115200);
pinMode(RELAY_PIN, OUTPUT);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) delay(500);
server.on("/control?device=on", handleOn);
server.on("/control?device=off", handleOff);
server.begin();
}
void loop() {
server.handleClient();
}
🎥 Working Principle
- Face Recognition: Only registered users (like Abhishek) can control the system.
- Gesture Recognition: Uses finger and thumb distance to detect gestures.
- ESP32 Communication: Commands sent via WiFi to control the relay.
- Appliance Control: Light/Fan turns ON or OFF instantly.
🧾 Output Preview
✅ Recognized Face → Access Granted
✋ Gesture Detected → “Light ON” command sent
✊ Gesture Detected → “Light OFF” command sent
💡 ESP32 Relay toggles appliance in real time
🚀 Future Enhancements
- Add voice control using Google Assistant or Alexa.
- Store user details in a cloud database (Firebase or MySQL).
- Integrate Blynk dashboard for remote access.
- Add security features like unknown face logging.
🧠 Conclusion
With this project, YaranaIoT Guru has successfully demonstrated a smart gesture + face-controlled home automation system using OpenCV, MediaPipe, and ESP32.
This project combines AI + IoT to create a futuristic home control experience that’s fun, practical, and highly scalable.
Stay tuned for more innovative IoT tutorials and subscribe to YaranaIoT Guru on YouTube for step-by-step video guides and project updates.
📞 Contact YaranaIoT Guru Empowering IoT Innovation | ESP32 | Home Automation | Smart Solutions | 50K+ Community
We’d love to hear from you! Whether it’s IoT project queries, collaborations, tech support, custom PCB design, bulk orders, corporate training, college workshops, or freelance development — we’re just one message away.
✉️ Email (Official)
For detailed inquiries, project support, business collaboration, sponsorships, or documentation: 📩 contact@yaranaiotguru.in 📧 Alternate: support@yaranaiotguru.in ⏳ Response: Within 24 hours (Mon–Sat) 💡 Best for attachments (code, schematics, logs, etc.)
📱 Phone / WhatsApp (24×7 Support)
Instant live help, troubleshooting, project consultation, or order updates: 📞 +91 70527 22734 💬 WhatsApp: Chat Now ⏰ Call Hours: Mon–Sat, 10 AM – 7 PM IST 🚀 Emergency? WhatsApp anytime — reply within 1 hour
🌐 Official Website
Tutorials, code, PDFs, schematics, blogs, free tools & online store: 🔗 www.yaranaiotguru.in 🛒 Shop: yaranaiotguru.in/shop (ESP32 DevKits, Sensors, Relays, Custom PCBs, Project Kits)
▶️ YouTube Channel
Step-by-step IoT builds, live coding, ESP32, Blynk, Node-RED, MQTT, Home Assistant & more: 🔗 Yarana IoT Guru 📺 1,200+ Videos | 52K+ Subs | 5.5M+ Views | 4.8★ Rating 🎥 New Video Every Week — 🔔 Subscribe & Turn On Notifications
🛠 GitHub (100% Open Source)
All codes, Arduino sketches, PlatformIO projects, Node-RED flows, MQTT configs & docs: 🔗 github.com/YaranaIotGuru ⭐ 50+ Repos | 10K+ Stars & Forks
🔥 Top Projects:
- ESP32 WebSocket Real-Time Dashboard
- Smart Home with Blynk & Alexa
- IoT Irrigation System with Soil Moisture
- MQTT + Node-RED + MySQL Logging
- OLED Weather Station with API
Daily reels, quick tips, live builds, student showcases & giveaways: 🔗 @YaranaIoTGuru 📱 10K+ Followers | Reels | Stories | Live Sessions
💼 LinkedIn (Professional Network)
B2B, IoT consulting, training, hiring & partnerships: 🔗 Yarana IoT Guru
🤝 Services Offered:
- Custom IoT Product Development
- Embedded Systems Training
- College Workshops & FDPs
- PCB Design & Prototyping
🐦 Twitter / X
Real-time updates, polls, project launches & community Q&A: 🔗 @YaranaIoTGuru 📢 Follow for instant alerts
🛠 Hackster.io (Project Showcases)
In-depth write-ups, circuits, BOM, code & ratings: 🔗 hackster.io/yaranaiotguru 🏆 50+ Projects | 100K+ Views | Top 5% Creator
📝 Instructables (DIY Guides)
Beginner-friendly step-by-step guides with templates & troubleshooting: 🔗 instructables.com/member/YaranaIoTGuru
🛠 Featured Guides:
- Automatic Plant Watering System
- Voice-Controlled Home Appliances
- WiFi-enabled Temperature Monitor
📚 Medium Blog
Deep-dive articles, trends, tutorials & career tips in embedded systems: 🔗 medium.com/@yaranaiotguru ✍️ 50+ Articles | 15K+ Readers
🛒 Online Store & Kits
Ready-to-use IoT kits, custom PCBs, sensors & merch: 🔗 yaranaiotguru.in/shop 📦 Free Shipping in India above ₹999 💳 Payment: UPI, Card, Wallet, COD
🌟 Community Platforms
| Platform | Link | Purpose |
|---|---|---|
| Telegram Channel | t.me/YaranaIoTGuru | Project files, PDFs, updates |
| Telegram Group | t.me/YaranaIoTCommunity | Peer support, doubts |
| Discord Server | discord.gg/yarana-iot | Live voice help, coding rooms |
| WhatsApp Community | Join Here | Announcements & polls |
🏢 Office & Studio Address
Yarana Studio & Software (Yarana IoT Guru HQ) 📍 Near Rookh Baba Mandir, Umariya Badal Urf Gainda, Allahabad (Prayagraj), Uttar Pradesh – 212507, India ⭐ Google Rating: 5.0 ★ (100+ Reviews)
🕒 Opening Hours: Mon–Sat: 10:00 AM – 5:00 PM Sunday: Closed
🌐 Associated Website: yaranawebtech.in 🗺️ View on Google Maps: Search “Yarana Studio & Software” 📌 Walk-ins welcome | Appointment recommended for project discussions