From 871366e6432be2d622fa2507e78fd56c35cd578d Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Mon, 19 May 2025 21:46:54 -0700 Subject: [PATCH 01/13] Add files via upload --- content/tutorials/TEMPLATE.md | 522 ++++++++++++++++++++++++++++++++++ 1 file changed, 522 insertions(+) create mode 100644 content/tutorials/TEMPLATE.md diff --git a/content/tutorials/TEMPLATE.md b/content/tutorials/TEMPLATE.md new file mode 100644 index 00000000..59794152 --- /dev/null +++ b/content/tutorials/TEMPLATE.md @@ -0,0 +1,522 @@ +--- +title: A Comprehensive Guide to Connecting a Game Controller to Any ESP32 Development Board via Bluetooth +date: 11-25-2024 +authors: + - name: Purab Balani + - name: Syler Sylvester + - name: Aleksandar Jeremic +--- + +![th](https://github.com/user-attachments/assets/b54308e5-dd03-415c-a266-4db8675e2418) + + +## Introduction + +This tutorial is designed to guide you through connecting any ESP development module to a Bluetooth game controller. Specifically, we will demonstrate using a wireless Xbox controller with the ESP32-S3 dev module to control a robotic arm. The primary objective is to establish a connection between the controller and the ESP32 module, enabling you to read all inputs effectively and then control the intensity of an LED through the joystick. + +The motivation behind this tutorial is to provide a practical example of integrating Bluetooth game controllers with ESP modules, showcasing their potential in robotics and other applications. By following this tutorial, readers will gain hands-on experience in setting up and configuring the hardware and software, ultimately enhancing their understanding of Bluetooth communication and control systems. + +### Learning Objectives + +- Bluetooth Communication: Understanding how to establish a Bluetooth connection between the ESP32-S3 and a game controller. +- ESP32-S3 Configuration: Setting up the ESP32-S3 development module for Bluetooth communication. +- Controller Input Reading: Learning how to read inputs from a wireless Xbox controller. +- Firmware Development: Writing and configuring firmware to handle Bluetooth communication and input processing. +- Troubleshooting: Identifying and resolving common issues in Bluetooth communication and hardware setup. + +Any additional notes from the developers can be included here. + +### Background Information + +This tutorial demonstrates how to connect a Bluetooth game controller to an ESP32 development module, specifically using a wireless Xbox controller with the ESP32-S3 to control a robotic arm. The primary objective is to establish a connection between the controller and the ESP module, enabling the reading of all inputs effectively. The motivation behind this tutorial is to provide a practical example of integrating Bluetooth game controllers with ESP modules, showcasing their potential in robotics and other applications. By following this tutorial, readers will gain hands-on experience in setting up and configuring the hardware and software, ultimately enhancing their understanding of Bluetooth communication and control systems. + +## Getting Started +### Required Downloads and Installations +--- +Software: Arduino IDE +Description: The Arduino Integrated Development Environment (IDE) is a cross-platform application used to write and upload programs to Arduino-compatible boards. +Installation Process: + - Download the Arduino IDE from the [official website](https://www.arduino.cc/en/software). + - Follow the installation instructions for your operating system (Windows, macOS, or Linux). + - Open the Arduino IDE and go to File > Preferences. + - In the "Additional Boards Manager URLs" field, add the following URLs: + `https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json` + `https://raw.githubusercontent.com/ricardoquesada/esp32-arduino-lib-builder/master/bluepad32_files/package_esp32_bluepad32_index.json` + - Go to Tools > Board > Boards Manager, search for "ESP32", and install the ESP32 by Espressif as well as esp32_bluepad32 by Ricardo Quesada board package. + - Go to Tools > Board > esp32_bluepad32, search for your esp board module in my case it is ESP32S3 Dev Module, and choose your COM port. +--- + + +### Required Components + +| Component Name | Quantity | +|--------------------------|----------| +| ESP32-S3 Dev Module | 1 | +| Wireless Xbox Controller | 1 | +| USB Cable | 1 | +| Power Supply | 1 | + +### Required Tools and Equipment + +All you will need is a functional computer with Arduino installed and running with an USB port. + +--- +## Part 01: Connecting controller with ESP32 + +### Introduction + +In this section, you will learn how to connect a game controller to the ESP32 using the Bluepad32 library. This forms the foundation for reading and processing input values from the controller. + +### Objective + +- Understand the process of connecting a game controller to the ESP32. +- Learn to set up Bluepad32 callbacks for controller connections. +- Identify and handle multiple connected controllers. + +### Background Information + +- Programming in C/C++ for ESP32. +- Bluetooth communication principles. +- Setting up the Arduino environment for ESP32. + +### Components + +- ESP32 development board +- Game controller (Bluetooth-compatible) +- USB cable for programming ESP32 +- Arduino IDE or PlatformIO + +### Instructional + +- Install Bluepad32 Library: Ensure you have the Bluepad32 library installed in your Arduino environment. +- Set Up Callbacks: Implement onConnectedController() and onDisconnectedController() functions to handle controller connections. +- Connect Controller: Power on your controller and put it into pairing mode. +- Upload Code to ESP32: Compile and upload the code provided in the example section to your ESP32. +- pair the controller + +## Example + +### Introduction + +This example demonstrates how to detect when a game controller connects to the ESP32 and outputs basic information about the connected controller. + +### Example + +```cpp +#include +const unsigned int LED{17}; // define a constant for the LED pin + +ControllerPtr myControllers[BP32_MAX_GAMEPADS]; + +// This callback gets called any time a new gamepad is connected. +// Up to 4 gamepads can be connected at the same time. +void onConnectedController(ControllerPtr ctl) { + bool foundEmptySlot = false; + for (int i = 0; i < BP32_MAX_GAMEPADS; i++) { + if (myControllers[i] == nullptr) { + USBSerial.printf("CALLBACK: Controller is connected, index=%d\n", i); + // Additionally, you can get certain gamepad properties like: + // Model, VID, PID, BTAddr, flags, etc. + ControllerProperties properties = ctl->getProperties(); + USBSerial.printf("Controller model: %s, VID=0x%04x, PID=0x%04x\n", ctl->getModelName().c_str(), properties.vendor_id, + properties.product_id); + myControllers[i] = ctl; + foundEmptySlot = true; + break; + } + } + if (!foundEmptySlot) { + USBSerial.println("CALLBACK: Controller connected, but could not found empty slot"); + } +} + +void onDisconnectedController(ControllerPtr ctl) { + bool foundController = false; + + for (int i = 0; i < BP32_MAX_GAMEPADS; i++) { + if (myControllers[i] == ctl) { + USBSerial.printf("CALLBACK: Controller disconnected from index=%d\n", i); + myControllers[i] = nullptr; + foundController = true; + break; + } + } + + if (!foundController) { + USBSerial.println("CALLBACK: Controller disconnected, but not found in myControllers"); + } +} + +void processControllers() { + for (auto myController : myControllers) { + if (myController && myController->isConnected() && myController->hasData()) { + if (myController->isGamepad()) { + processGamepad(myController); + } + else { + USBSerial.println("Unsupported controller"); + } + } + } +} + +``` +### Analysis + +The first part of this tutorial focuses on establishing a connection between the game controller and the ESP32 module using the Bluepad32 library. The example code includes essential steps such as initializing the library, defining callback functions for handling controller connections, and implementing basic diagnostics to confirm successful pairing. + +#### Code Walkthrough + +- **Callback Functions**: `onConnectedController` and `onDisconnectedController` handle events when a controller connects or disconnects. They assign or clear the controller pointer from the `myControllers` array. +- **Controller Properties**: The code retrieves and prints controller properties like vendor ID (VID) and product ID (PID) for diagnostic purposes. +- **Processing Controllers**: The `processControllers()` function iterates through the array of connected controllers and processes inputs only if the controller is connected and has new data. + +This section ensures a robust connection between the game controller and ESP32, enabling further interaction. + +--- + +## Part 02: Retrieving Gamepad Data + +### Introduction + +In this section, you will learn how to retrieve and display real-time data from the game controller. This involves accessing inputs such as joystick positions, button states, and motion data, enabling you to monitor the controller's state in detail. + +### Objective + +- Understand how to extract gamepad data using Bluepad32. +- Display real-time gamepad data via serial output. +- Familiarize yourself with controller properties like axis, buttons, and motion sensors. + +### Background Information + +- Reading and interpreting joystick and button inputs. +- Serial communication with the ESP32 for data monitoring. +- Controller properties and their relevance to applications. + +### Components + +- ESP32 development board +- Bluetooth-compatible game controller +- USB cable for programming and monitoring ESP32 +- Arduino IDE or PlatformIO + +### Instructional + +- **Gamepad Data Function**: Implement the `dumpGamepad()` function to retrieve all relevant data from the connected game controller. +- **Access Controller Properties**: Use the gamepad API to fetch joystick positions, button states, and sensor data. +- **Monitor Serial Output**: View the retrieved data in the serial monitor for analysis. +- **Iterate Over Controllers**: Ensure data is dumped for all connected controllers using a loop. + +## Example + +### Introduction + +The example below retrieves and displays gamepad data such as joystick axis values, button states, and accelerometer/gyroscope readings. + +### Example Code + +```cpp +void dumpGamepad(ControllerPtr ctl) { + USBSerial.printf( + "idx=%d, dpad: 0x%02x, buttons: 0x%04x, axis L: %4d, %4d, axis R: %4d, %4d, brake: %4d, throttle: %4d, " + "misc: 0x%02x, gyro x:%6d y:%6d z:%6d, accel x:%6d y:%6d z:%6d\n", + ctl->index(), // Controller Index + ctl->dpad(), // D-pad + ctl->buttons(), // Bitmask of pressed buttons + ctl->axisX(), // (-511 - 512) left X Axis + ctl->axisY(), // (-511 - 512) left Y Axis + ctl->axisRX(), // (-511 - 512) right X Axis + ctl->axisRY(), // (-511 - 512) right Y Axis + ctl->brake(), // (0 - 1023): Brake button + ctl->throttle(), // (0 - 1023): Throttle button + ctl->miscButtons(), // Bitmask of "misc" buttons + ctl->gyroX(), // Gyroscope X + ctl->gyroY(), // Gyroscope Y + ctl->gyroZ(), // Gyroscope Z + ctl->accelX(), // Accelerometer X + ctl->accelY(), // Accelerometer Y + ctl->accelZ() // Accelerometer Z + ); +} +``` +### Analysis + +This part introduces retrieving real-time input data from the controller and dumping it to the serial monitor for analysis. It helps developers understand how the controller's inputs can be accessed and interpreted. + +#### Code Walkthrough + +- **Dump Function**: The `dumpGamepad()` function retrieves key data points from the controller, such as: + - **Joystick Axes (`axisX`, `axisY`, `axisRX`, `axisRY`)**: These represent the X and Y positions of the left and right joysticks. + - **D-pad and Buttons**: Represented as bitmasks for easily determining the state of each input. + - **Accelerometer and Gyroscope Data**: Provides motion-sensing capabilities for advanced control. +- **Serial Output**: Outputs the retrieved data in a structured format for debugging or analysis. + +This section provides a clear understanding of game controller input structures, preparing developers for integration with hardware. + +--- + +## Part 03: Mapping Controller Inputs to LED Brightness + +### Introduction + +In this section, you will learn how to map the input from the game controller’s joystick to control the brightness of an LED. This exercise demonstrates how controller inputs can be used for real-time hardware interaction. + +### Objective + +- Understand how to retrieve joystick input values. +- Learn to map input values to a hardware output range. +- Control LED brightness using joystick movements. +- Explore the use of PWM for LED dimming. + +### Background Information + +- **PWM (Pulse Width Modulation)**: Used to simulate analog output for controlling devices like LEDs. +- **Mapping Values**: The `map()` function scales an input range to an output range. +- **Joystick Axis Input**: Game controllers output analog values for joystick movements, which can be utilized for dynamic hardware control. + +### Components + +- ESP32 development board +- Bluetooth-compatible game controller +- USB cable for programming and monitoring ESP32 +- Arduino IDE or PlatformIO + +### Instructional + +1. **Setup LED Output**: Connect an LED to the ESP32 with an appropriate resistor. +2. **Access Joystick Data**: Use the `axisRY()` method to retrieve the joystick's Y-axis value. +3. **Map Axis Value**: Convert the joystick’s range (-511 to 512) to a brightness range (0 to 255) using the `map()` function. +4. **Constrain Brightness**: Use `constrain()` to ensure the brightness value stays within the valid range. +5. **Apply PWM to LED**: Use the `analogWrite()` function to set the LED brightness based on the mapped value. +6. **Monitor LED Behavior**: Observe how the LED’s brightness changes as the joystick is moved. + +## Example + +### Introduction + +The following example maps the right joystick’s Y-axis value to control an LED’s brightness. The joystick input is processed in real-time, and the LED responds by dimming or brightening accordingly. + +### Example Code + +```cpp +void processGamepad(ControllerPtr ctl) { + int axisValue = ctl->axisRY(); // Get the right Y-axis value (-511 to 512) + + // Map axis value to PWM range (0 to 255) + int brightness = map(axisValue, -511, 512, 255, 0); // -511 is max bright, 512 is dim + brightness = constrain(brightness, 0, 255); // Ensure valid range + + analogWrite(LED, brightness); // Apply PWM to LED pin + + // Optional: Dump gamepad data for debugging + dumpGamepad(ctl); +} +``` +### Analysis + +This part builds on the previous sections by using the joystick’s Y-axis input to control an LED’s brightness, demonstrating the practical application of game controller inputs. + +#### Code Walkthrough + +- **Retrieve Input**: The `axisRY()` method fetches the joystick’s vertical position, which ranges from -511 (up) to 512 (down). +- **Mapping and Constraining Values**: + - `map(axisValue, -511, 512, 255, 0)` converts joystick input to PWM-compatible brightness levels (0–255). + - `constrain(brightness, 0, 255)` ensures the value stays within the valid range. +- **Analog Output**: The `analogWrite(LED, brightness)` function adjusts the LED’s brightness using PWM. + +#### Implementation + +This simple yet effective demonstration bridges input data processing and real-world hardware control. It highlights the ESP32’s capability for real-time interactive applications. + +--- + +## Part 04: Setup and Loop - Putting Everything Together + +### Introduction + +In this section, we will integrate everything you've learned so far and put it into a fully working program. You will learn how to set up your ESP32 to manage controller connections, handle data retrieval, and implement the gamepad data processing in the main loop. + + + +### Objective + +- Set up the controller system to connect and disconnect gamepads. +- Process gamepad data in the main loop. +- Control external devices (like an LED) based on gamepad input. + + +### Background Information + +- **Setup Function:** This function initializes your ESP32, sets up Bluetooth, and prepares everything necessary for the main loop to function correctly. +- **Loop Function:** This function runs repeatedly and is responsible for checking gamepad inputs and performing actions like adjusting the brightness of an LED or dumping controller data. +- **Controller Callbacks:** You'll use callbacks to handle events like when a gamepad is connected or disconnected. + + + +### Components + +- ESP32 development board +- Bluetooth-compatible game controller +- USB cable for programming and monitoring ESP32 +- Arduino IDE or PlatformIO + + +### Instructional Steps + +1. **Setup the Gamepad System:** Initialize Bluetooth communication and register controller callbacks. +2. **Define the Loop:** Implement logic to update and process controller data in the loop function. +3. **Control an LED:** Use a gamepad's input, like the right joystick axis, to control an LED's brightness. + +## Example + +### Introduction + +The example code below ties everything together. It connects controllers, processes their inputs, and uses the right joystick axis to control an LED's brightness. + +### Example Code + +```cpp +#include + +const unsigned int LED = 17; // Define the LED pin + +ControllerPtr myControllers[BP32_MAX_GAMEPADS]; + +// This callback gets called when a new gamepad is connected. +void onConnectedController(ControllerPtr ctl) { + bool foundEmptySlot = false; + for (int i = 0; i < BP32_MAX_GAMEPADS; i++) { + if (myControllers[i] == nullptr) { + USBSerial.printf("CALLBACK: Controller connected, index=%d\n", i); + ControllerProperties properties = ctl->getProperties(); + USBSerial.printf("Controller model: %s, VID=0x%04x, PID=0x%04x\n", + ctl->getModelName().c_str(), properties.vendor_id, properties.product_id); + myControllers[i] = ctl; + foundEmptySlot = true; + break; + } + } + if (!foundEmptySlot) { + USBSerial.println("CALLBACK: Controller connected, but could not find empty slot."); + } +} + +// This callback gets called when a gamepad is disconnected. +void onDisconnectedController(ControllerPtr ctl) { + bool foundController = false; + for (int i = 0; i < BP32_MAX_GAMEPADS; i++) { + if (myControllers[i] == ctl) { + USBSerial.printf("CALLBACK: Controller disconnected from index=%d\n", i); + myControllers[i] = nullptr; + foundController = true; + break; + } + } + if (!foundController) { + USBSerial.println("CALLBACK: Controller disconnected, but not found in myControllers."); + } +} + +// Dump gamepad data to serial +void dumpGamepad(ControllerPtr ctl) { + USBSerial.printf( + "idx=%d, dpad: 0x%02x, buttons: 0x%04x, axis L: %4d, %4d, axis R: %4d, %4d, brake: %4d, throttle: %4d, " + "misc: 0x%02x, gyro x:%6d y:%6d z:%6d, accel x:%6d y:%6d z:%6d\n", + ctl->index(), + ctl->dpad(), + ctl->buttons(), + ctl->axisX(), + ctl->axisY(), + ctl->axisRX(), + ctl->axisRY(), + ctl->brake(), + ctl->throttle(), + ctl->miscButtons(), + ctl->gyroX(), + ctl->gyroY(), + ctl->gyroZ(), + ctl->accelX(), + ctl->accelY(), + ctl->accelZ() + ); +} + +// Process gamepad input and control LED +void processGamepad(ControllerPtr ctl) { + int axisValue = ctl->axisRY(); // Get the right Y-axis value (-511 to 512) + int brightness = map(axisValue, -511, 512, 255, 0); // Map to PWM range + brightness = constrain(brightness, 0, 255); // Ensure within valid range + analogWrite(LED, brightness); // Apply brightness to LED + dumpGamepad(ctl); // Dump controller data +} + +// Process all connected controllers +void processControllers() { + for (auto myController : myControllers) { + if (myController && myController->isConnected() && myController->hasData()) { + if (myController->isGamepad()) { + processGamepad(myController); + } else { + USBSerial.println("Unsupported controller"); + } + } + } +} + +// Arduino setup function +void setup() { + USBSerial.begin(115200); + USBSerial.printf("Firmware: %s\n", BP32.firmwareVersion()); + const uint8_t* addr = BP32.localBdAddress(); + USBSerial.printf("BD Addr: %2X:%2X:%2X:%2X:%2X:%2X\n", addr[0], addr[1], addr[2], addr[3], addr[4], addr[5]); + + BP32.setup(&onConnectedController, &onDisconnectedController); + BP32.forgetBluetoothKeys(); // Optionally forget previous Bluetooth keys + BP32.enableVirtualDevice(false); // Disable virtual mouse/touchpad support +} + +// Arduino loop function +void loop() { + bool dataUpdated = BP32.update(); + if (dataUpdated) { + processControllers(); + } + delay(150); // Yield to lower priority tasks to avoid watchdog timeout +} +``` + +https://github.com/user-attachments/assets/34143fbf-1968-48fc-ae7c-aee796cb4d17 + +## Analysis + +This part builds on the previous sections by demonstrating how real-time gamepad input can be used to control an external device, like an LED, based on the joystick's Y-axis input. This showcases the practical application of game controller data to influence physical hardware, an essential concept for interactive projects. + +--- + +### Code Walkthrough + +- **Retrieve Input**: + The `axisRY()` method retrieves the vertical position of the right joystick. The range of values for the Y-axis is from -511 (full tilt up) to 512 (full tilt down). This range is crucial for interpreting how far the joystick is pushed in either direction. + +- **Mapping and Constraining Values**: + - The `map(axisValue, -511, 512, 255, 0)` function takes the raw joystick value (which ranges from -511 to 512) and maps it to the appropriate PWM range (0 to 255) for controlling the LED brightness. This effectively converts the joystick's movement into a usable value for the LED control. + - The `constrain(brightness, 0, 255)` function ensures that the final value for brightness is kept within the allowable PWM range (0 to 255). This prevents out-of-bound values, which could cause errors or unexpected behavior. + +- **Analog Output**: + The `analogWrite(LED, brightness)` function then uses the mapped and constrained value to adjust the LED's brightness, providing a smooth and responsive interaction based on joystick input. This is a simple yet effective way to visualize controller input through hardware. + +--- + +### Implementation + +This part demonstrates how controller data, specifically joystick input, can be translated into real-world actions. By mapping the joystick's vertical movement to the brightness of an LED, we bridge the gap between user input and hardware control. This interaction highlights the ESP32's ability to handle real-time data and execute commands efficiently, laying the groundwork for more complex applications that can use game controllers to control various devices. + +This is a simple yet powerful example of interactive technology in action, where user input via a Bluetooth gamepad translates directly into physical changes in the environment. It emphasizes the versatility of the ESP32 in handling both data processing and hardware control tasks in real-time. + +## Additional Resources + +### Useful links + +- https://racheldebarros.com/esp32-projects/connect-your-game-controller-to-an-esp32/ +- https://github.com/ricardoquesada/bluepad32-arduino +- https://stackoverflow.com/questions/66278271/task-watchdog-got-triggered-the-tasks-did-not-reset-the-watchdog-in-time From b37c65213afacc473ca53e08518351dcc59c4ae3 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Mon, 19 May 2025 21:53:42 -0700 Subject: [PATCH 02/13] Update and rename TEMPLATE.md to Team9Tutorial.md --- content/tutorials/TEMPLATE.md | 522 ----------------------------- content/tutorials/Team9Tutorial.md | 198 +++++++++++ 2 files changed, 198 insertions(+), 522 deletions(-) delete mode 100644 content/tutorials/TEMPLATE.md create mode 100644 content/tutorials/Team9Tutorial.md diff --git a/content/tutorials/TEMPLATE.md b/content/tutorials/TEMPLATE.md deleted file mode 100644 index 59794152..00000000 --- a/content/tutorials/TEMPLATE.md +++ /dev/null @@ -1,522 +0,0 @@ ---- -title: A Comprehensive Guide to Connecting a Game Controller to Any ESP32 Development Board via Bluetooth -date: 11-25-2024 -authors: - - name: Purab Balani - - name: Syler Sylvester - - name: Aleksandar Jeremic ---- - -![th](https://github.com/user-attachments/assets/b54308e5-dd03-415c-a266-4db8675e2418) - - -## Introduction - -This tutorial is designed to guide you through connecting any ESP development module to a Bluetooth game controller. Specifically, we will demonstrate using a wireless Xbox controller with the ESP32-S3 dev module to control a robotic arm. The primary objective is to establish a connection between the controller and the ESP32 module, enabling you to read all inputs effectively and then control the intensity of an LED through the joystick. - -The motivation behind this tutorial is to provide a practical example of integrating Bluetooth game controllers with ESP modules, showcasing their potential in robotics and other applications. By following this tutorial, readers will gain hands-on experience in setting up and configuring the hardware and software, ultimately enhancing their understanding of Bluetooth communication and control systems. - -### Learning Objectives - -- Bluetooth Communication: Understanding how to establish a Bluetooth connection between the ESP32-S3 and a game controller. -- ESP32-S3 Configuration: Setting up the ESP32-S3 development module for Bluetooth communication. -- Controller Input Reading: Learning how to read inputs from a wireless Xbox controller. -- Firmware Development: Writing and configuring firmware to handle Bluetooth communication and input processing. -- Troubleshooting: Identifying and resolving common issues in Bluetooth communication and hardware setup. - -Any additional notes from the developers can be included here. - -### Background Information - -This tutorial demonstrates how to connect a Bluetooth game controller to an ESP32 development module, specifically using a wireless Xbox controller with the ESP32-S3 to control a robotic arm. The primary objective is to establish a connection between the controller and the ESP module, enabling the reading of all inputs effectively. The motivation behind this tutorial is to provide a practical example of integrating Bluetooth game controllers with ESP modules, showcasing their potential in robotics and other applications. By following this tutorial, readers will gain hands-on experience in setting up and configuring the hardware and software, ultimately enhancing their understanding of Bluetooth communication and control systems. - -## Getting Started -### Required Downloads and Installations ---- -Software: Arduino IDE -Description: The Arduino Integrated Development Environment (IDE) is a cross-platform application used to write and upload programs to Arduino-compatible boards. -Installation Process: - - Download the Arduino IDE from the [official website](https://www.arduino.cc/en/software). - - Follow the installation instructions for your operating system (Windows, macOS, or Linux). - - Open the Arduino IDE and go to File > Preferences. - - In the "Additional Boards Manager URLs" field, add the following URLs: - `https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json` - `https://raw.githubusercontent.com/ricardoquesada/esp32-arduino-lib-builder/master/bluepad32_files/package_esp32_bluepad32_index.json` - - Go to Tools > Board > Boards Manager, search for "ESP32", and install the ESP32 by Espressif as well as esp32_bluepad32 by Ricardo Quesada board package. - - Go to Tools > Board > esp32_bluepad32, search for your esp board module in my case it is ESP32S3 Dev Module, and choose your COM port. ---- - - -### Required Components - -| Component Name | Quantity | -|--------------------------|----------| -| ESP32-S3 Dev Module | 1 | -| Wireless Xbox Controller | 1 | -| USB Cable | 1 | -| Power Supply | 1 | - -### Required Tools and Equipment - -All you will need is a functional computer with Arduino installed and running with an USB port. - ---- -## Part 01: Connecting controller with ESP32 - -### Introduction - -In this section, you will learn how to connect a game controller to the ESP32 using the Bluepad32 library. This forms the foundation for reading and processing input values from the controller. - -### Objective - -- Understand the process of connecting a game controller to the ESP32. -- Learn to set up Bluepad32 callbacks for controller connections. -- Identify and handle multiple connected controllers. - -### Background Information - -- Programming in C/C++ for ESP32. -- Bluetooth communication principles. -- Setting up the Arduino environment for ESP32. - -### Components - -- ESP32 development board -- Game controller (Bluetooth-compatible) -- USB cable for programming ESP32 -- Arduino IDE or PlatformIO - -### Instructional - -- Install Bluepad32 Library: Ensure you have the Bluepad32 library installed in your Arduino environment. -- Set Up Callbacks: Implement onConnectedController() and onDisconnectedController() functions to handle controller connections. -- Connect Controller: Power on your controller and put it into pairing mode. -- Upload Code to ESP32: Compile and upload the code provided in the example section to your ESP32. -- pair the controller - -## Example - -### Introduction - -This example demonstrates how to detect when a game controller connects to the ESP32 and outputs basic information about the connected controller. - -### Example - -```cpp -#include -const unsigned int LED{17}; // define a constant for the LED pin - -ControllerPtr myControllers[BP32_MAX_GAMEPADS]; - -// This callback gets called any time a new gamepad is connected. -// Up to 4 gamepads can be connected at the same time. -void onConnectedController(ControllerPtr ctl) { - bool foundEmptySlot = false; - for (int i = 0; i < BP32_MAX_GAMEPADS; i++) { - if (myControllers[i] == nullptr) { - USBSerial.printf("CALLBACK: Controller is connected, index=%d\n", i); - // Additionally, you can get certain gamepad properties like: - // Model, VID, PID, BTAddr, flags, etc. - ControllerProperties properties = ctl->getProperties(); - USBSerial.printf("Controller model: %s, VID=0x%04x, PID=0x%04x\n", ctl->getModelName().c_str(), properties.vendor_id, - properties.product_id); - myControllers[i] = ctl; - foundEmptySlot = true; - break; - } - } - if (!foundEmptySlot) { - USBSerial.println("CALLBACK: Controller connected, but could not found empty slot"); - } -} - -void onDisconnectedController(ControllerPtr ctl) { - bool foundController = false; - - for (int i = 0; i < BP32_MAX_GAMEPADS; i++) { - if (myControllers[i] == ctl) { - USBSerial.printf("CALLBACK: Controller disconnected from index=%d\n", i); - myControllers[i] = nullptr; - foundController = true; - break; - } - } - - if (!foundController) { - USBSerial.println("CALLBACK: Controller disconnected, but not found in myControllers"); - } -} - -void processControllers() { - for (auto myController : myControllers) { - if (myController && myController->isConnected() && myController->hasData()) { - if (myController->isGamepad()) { - processGamepad(myController); - } - else { - USBSerial.println("Unsupported controller"); - } - } - } -} - -``` -### Analysis - -The first part of this tutorial focuses on establishing a connection between the game controller and the ESP32 module using the Bluepad32 library. The example code includes essential steps such as initializing the library, defining callback functions for handling controller connections, and implementing basic diagnostics to confirm successful pairing. - -#### Code Walkthrough - -- **Callback Functions**: `onConnectedController` and `onDisconnectedController` handle events when a controller connects or disconnects. They assign or clear the controller pointer from the `myControllers` array. -- **Controller Properties**: The code retrieves and prints controller properties like vendor ID (VID) and product ID (PID) for diagnostic purposes. -- **Processing Controllers**: The `processControllers()` function iterates through the array of connected controllers and processes inputs only if the controller is connected and has new data. - -This section ensures a robust connection between the game controller and ESP32, enabling further interaction. - ---- - -## Part 02: Retrieving Gamepad Data - -### Introduction - -In this section, you will learn how to retrieve and display real-time data from the game controller. This involves accessing inputs such as joystick positions, button states, and motion data, enabling you to monitor the controller's state in detail. - -### Objective - -- Understand how to extract gamepad data using Bluepad32. -- Display real-time gamepad data via serial output. -- Familiarize yourself with controller properties like axis, buttons, and motion sensors. - -### Background Information - -- Reading and interpreting joystick and button inputs. -- Serial communication with the ESP32 for data monitoring. -- Controller properties and their relevance to applications. - -### Components - -- ESP32 development board -- Bluetooth-compatible game controller -- USB cable for programming and monitoring ESP32 -- Arduino IDE or PlatformIO - -### Instructional - -- **Gamepad Data Function**: Implement the `dumpGamepad()` function to retrieve all relevant data from the connected game controller. -- **Access Controller Properties**: Use the gamepad API to fetch joystick positions, button states, and sensor data. -- **Monitor Serial Output**: View the retrieved data in the serial monitor for analysis. -- **Iterate Over Controllers**: Ensure data is dumped for all connected controllers using a loop. - -## Example - -### Introduction - -The example below retrieves and displays gamepad data such as joystick axis values, button states, and accelerometer/gyroscope readings. - -### Example Code - -```cpp -void dumpGamepad(ControllerPtr ctl) { - USBSerial.printf( - "idx=%d, dpad: 0x%02x, buttons: 0x%04x, axis L: %4d, %4d, axis R: %4d, %4d, brake: %4d, throttle: %4d, " - "misc: 0x%02x, gyro x:%6d y:%6d z:%6d, accel x:%6d y:%6d z:%6d\n", - ctl->index(), // Controller Index - ctl->dpad(), // D-pad - ctl->buttons(), // Bitmask of pressed buttons - ctl->axisX(), // (-511 - 512) left X Axis - ctl->axisY(), // (-511 - 512) left Y Axis - ctl->axisRX(), // (-511 - 512) right X Axis - ctl->axisRY(), // (-511 - 512) right Y Axis - ctl->brake(), // (0 - 1023): Brake button - ctl->throttle(), // (0 - 1023): Throttle button - ctl->miscButtons(), // Bitmask of "misc" buttons - ctl->gyroX(), // Gyroscope X - ctl->gyroY(), // Gyroscope Y - ctl->gyroZ(), // Gyroscope Z - ctl->accelX(), // Accelerometer X - ctl->accelY(), // Accelerometer Y - ctl->accelZ() // Accelerometer Z - ); -} -``` -### Analysis - -This part introduces retrieving real-time input data from the controller and dumping it to the serial monitor for analysis. It helps developers understand how the controller's inputs can be accessed and interpreted. - -#### Code Walkthrough - -- **Dump Function**: The `dumpGamepad()` function retrieves key data points from the controller, such as: - - **Joystick Axes (`axisX`, `axisY`, `axisRX`, `axisRY`)**: These represent the X and Y positions of the left and right joysticks. - - **D-pad and Buttons**: Represented as bitmasks for easily determining the state of each input. - - **Accelerometer and Gyroscope Data**: Provides motion-sensing capabilities for advanced control. -- **Serial Output**: Outputs the retrieved data in a structured format for debugging or analysis. - -This section provides a clear understanding of game controller input structures, preparing developers for integration with hardware. - ---- - -## Part 03: Mapping Controller Inputs to LED Brightness - -### Introduction - -In this section, you will learn how to map the input from the game controller’s joystick to control the brightness of an LED. This exercise demonstrates how controller inputs can be used for real-time hardware interaction. - -### Objective - -- Understand how to retrieve joystick input values. -- Learn to map input values to a hardware output range. -- Control LED brightness using joystick movements. -- Explore the use of PWM for LED dimming. - -### Background Information - -- **PWM (Pulse Width Modulation)**: Used to simulate analog output for controlling devices like LEDs. -- **Mapping Values**: The `map()` function scales an input range to an output range. -- **Joystick Axis Input**: Game controllers output analog values for joystick movements, which can be utilized for dynamic hardware control. - -### Components - -- ESP32 development board -- Bluetooth-compatible game controller -- USB cable for programming and monitoring ESP32 -- Arduino IDE or PlatformIO - -### Instructional - -1. **Setup LED Output**: Connect an LED to the ESP32 with an appropriate resistor. -2. **Access Joystick Data**: Use the `axisRY()` method to retrieve the joystick's Y-axis value. -3. **Map Axis Value**: Convert the joystick’s range (-511 to 512) to a brightness range (0 to 255) using the `map()` function. -4. **Constrain Brightness**: Use `constrain()` to ensure the brightness value stays within the valid range. -5. **Apply PWM to LED**: Use the `analogWrite()` function to set the LED brightness based on the mapped value. -6. **Monitor LED Behavior**: Observe how the LED’s brightness changes as the joystick is moved. - -## Example - -### Introduction - -The following example maps the right joystick’s Y-axis value to control an LED’s brightness. The joystick input is processed in real-time, and the LED responds by dimming or brightening accordingly. - -### Example Code - -```cpp -void processGamepad(ControllerPtr ctl) { - int axisValue = ctl->axisRY(); // Get the right Y-axis value (-511 to 512) - - // Map axis value to PWM range (0 to 255) - int brightness = map(axisValue, -511, 512, 255, 0); // -511 is max bright, 512 is dim - brightness = constrain(brightness, 0, 255); // Ensure valid range - - analogWrite(LED, brightness); // Apply PWM to LED pin - - // Optional: Dump gamepad data for debugging - dumpGamepad(ctl); -} -``` -### Analysis - -This part builds on the previous sections by using the joystick’s Y-axis input to control an LED’s brightness, demonstrating the practical application of game controller inputs. - -#### Code Walkthrough - -- **Retrieve Input**: The `axisRY()` method fetches the joystick’s vertical position, which ranges from -511 (up) to 512 (down). -- **Mapping and Constraining Values**: - - `map(axisValue, -511, 512, 255, 0)` converts joystick input to PWM-compatible brightness levels (0–255). - - `constrain(brightness, 0, 255)` ensures the value stays within the valid range. -- **Analog Output**: The `analogWrite(LED, brightness)` function adjusts the LED’s brightness using PWM. - -#### Implementation - -This simple yet effective demonstration bridges input data processing and real-world hardware control. It highlights the ESP32’s capability for real-time interactive applications. - ---- - -## Part 04: Setup and Loop - Putting Everything Together - -### Introduction - -In this section, we will integrate everything you've learned so far and put it into a fully working program. You will learn how to set up your ESP32 to manage controller connections, handle data retrieval, and implement the gamepad data processing in the main loop. - - - -### Objective - -- Set up the controller system to connect and disconnect gamepads. -- Process gamepad data in the main loop. -- Control external devices (like an LED) based on gamepad input. - - -### Background Information - -- **Setup Function:** This function initializes your ESP32, sets up Bluetooth, and prepares everything necessary for the main loop to function correctly. -- **Loop Function:** This function runs repeatedly and is responsible for checking gamepad inputs and performing actions like adjusting the brightness of an LED or dumping controller data. -- **Controller Callbacks:** You'll use callbacks to handle events like when a gamepad is connected or disconnected. - - - -### Components - -- ESP32 development board -- Bluetooth-compatible game controller -- USB cable for programming and monitoring ESP32 -- Arduino IDE or PlatformIO - - -### Instructional Steps - -1. **Setup the Gamepad System:** Initialize Bluetooth communication and register controller callbacks. -2. **Define the Loop:** Implement logic to update and process controller data in the loop function. -3. **Control an LED:** Use a gamepad's input, like the right joystick axis, to control an LED's brightness. - -## Example - -### Introduction - -The example code below ties everything together. It connects controllers, processes their inputs, and uses the right joystick axis to control an LED's brightness. - -### Example Code - -```cpp -#include - -const unsigned int LED = 17; // Define the LED pin - -ControllerPtr myControllers[BP32_MAX_GAMEPADS]; - -// This callback gets called when a new gamepad is connected. -void onConnectedController(ControllerPtr ctl) { - bool foundEmptySlot = false; - for (int i = 0; i < BP32_MAX_GAMEPADS; i++) { - if (myControllers[i] == nullptr) { - USBSerial.printf("CALLBACK: Controller connected, index=%d\n", i); - ControllerProperties properties = ctl->getProperties(); - USBSerial.printf("Controller model: %s, VID=0x%04x, PID=0x%04x\n", - ctl->getModelName().c_str(), properties.vendor_id, properties.product_id); - myControllers[i] = ctl; - foundEmptySlot = true; - break; - } - } - if (!foundEmptySlot) { - USBSerial.println("CALLBACK: Controller connected, but could not find empty slot."); - } -} - -// This callback gets called when a gamepad is disconnected. -void onDisconnectedController(ControllerPtr ctl) { - bool foundController = false; - for (int i = 0; i < BP32_MAX_GAMEPADS; i++) { - if (myControllers[i] == ctl) { - USBSerial.printf("CALLBACK: Controller disconnected from index=%d\n", i); - myControllers[i] = nullptr; - foundController = true; - break; - } - } - if (!foundController) { - USBSerial.println("CALLBACK: Controller disconnected, but not found in myControllers."); - } -} - -// Dump gamepad data to serial -void dumpGamepad(ControllerPtr ctl) { - USBSerial.printf( - "idx=%d, dpad: 0x%02x, buttons: 0x%04x, axis L: %4d, %4d, axis R: %4d, %4d, brake: %4d, throttle: %4d, " - "misc: 0x%02x, gyro x:%6d y:%6d z:%6d, accel x:%6d y:%6d z:%6d\n", - ctl->index(), - ctl->dpad(), - ctl->buttons(), - ctl->axisX(), - ctl->axisY(), - ctl->axisRX(), - ctl->axisRY(), - ctl->brake(), - ctl->throttle(), - ctl->miscButtons(), - ctl->gyroX(), - ctl->gyroY(), - ctl->gyroZ(), - ctl->accelX(), - ctl->accelY(), - ctl->accelZ() - ); -} - -// Process gamepad input and control LED -void processGamepad(ControllerPtr ctl) { - int axisValue = ctl->axisRY(); // Get the right Y-axis value (-511 to 512) - int brightness = map(axisValue, -511, 512, 255, 0); // Map to PWM range - brightness = constrain(brightness, 0, 255); // Ensure within valid range - analogWrite(LED, brightness); // Apply brightness to LED - dumpGamepad(ctl); // Dump controller data -} - -// Process all connected controllers -void processControllers() { - for (auto myController : myControllers) { - if (myController && myController->isConnected() && myController->hasData()) { - if (myController->isGamepad()) { - processGamepad(myController); - } else { - USBSerial.println("Unsupported controller"); - } - } - } -} - -// Arduino setup function -void setup() { - USBSerial.begin(115200); - USBSerial.printf("Firmware: %s\n", BP32.firmwareVersion()); - const uint8_t* addr = BP32.localBdAddress(); - USBSerial.printf("BD Addr: %2X:%2X:%2X:%2X:%2X:%2X\n", addr[0], addr[1], addr[2], addr[3], addr[4], addr[5]); - - BP32.setup(&onConnectedController, &onDisconnectedController); - BP32.forgetBluetoothKeys(); // Optionally forget previous Bluetooth keys - BP32.enableVirtualDevice(false); // Disable virtual mouse/touchpad support -} - -// Arduino loop function -void loop() { - bool dataUpdated = BP32.update(); - if (dataUpdated) { - processControllers(); - } - delay(150); // Yield to lower priority tasks to avoid watchdog timeout -} -``` - -https://github.com/user-attachments/assets/34143fbf-1968-48fc-ae7c-aee796cb4d17 - -## Analysis - -This part builds on the previous sections by demonstrating how real-time gamepad input can be used to control an external device, like an LED, based on the joystick's Y-axis input. This showcases the practical application of game controller data to influence physical hardware, an essential concept for interactive projects. - ---- - -### Code Walkthrough - -- **Retrieve Input**: - The `axisRY()` method retrieves the vertical position of the right joystick. The range of values for the Y-axis is from -511 (full tilt up) to 512 (full tilt down). This range is crucial for interpreting how far the joystick is pushed in either direction. - -- **Mapping and Constraining Values**: - - The `map(axisValue, -511, 512, 255, 0)` function takes the raw joystick value (which ranges from -511 to 512) and maps it to the appropriate PWM range (0 to 255) for controlling the LED brightness. This effectively converts the joystick's movement into a usable value for the LED control. - - The `constrain(brightness, 0, 255)` function ensures that the final value for brightness is kept within the allowable PWM range (0 to 255). This prevents out-of-bound values, which could cause errors or unexpected behavior. - -- **Analog Output**: - The `analogWrite(LED, brightness)` function then uses the mapped and constrained value to adjust the LED's brightness, providing a smooth and responsive interaction based on joystick input. This is a simple yet effective way to visualize controller input through hardware. - ---- - -### Implementation - -This part demonstrates how controller data, specifically joystick input, can be translated into real-world actions. By mapping the joystick's vertical movement to the brightness of an LED, we bridge the gap between user input and hardware control. This interaction highlights the ESP32's ability to handle real-time data and execute commands efficiently, laying the groundwork for more complex applications that can use game controllers to control various devices. - -This is a simple yet powerful example of interactive technology in action, where user input via a Bluetooth gamepad translates directly into physical changes in the environment. It emphasizes the versatility of the ESP32 in handling both data processing and hardware control tasks in real-time. - -## Additional Resources - -### Useful links - -- https://racheldebarros.com/esp32-projects/connect-your-game-controller-to-an-esp32/ -- https://github.com/ricardoquesada/bluepad32-arduino -- https://stackoverflow.com/questions/66278271/task-watchdog-got-triggered-the-tasks-did-not-reset-the-watchdog-in-time diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md new file mode 100644 index 00000000..02f6f11e --- /dev/null +++ b/content/tutorials/Team9Tutorial.md @@ -0,0 +1,198 @@ +--- +title: A Comprehensive Guide to Performing Computer Vision Tasks with ESP32-CAM Module +date: 11-25-2024 +authors: + - name: Purab Balani + - name: Jasmine Le + - name: Genaro Salazar Ruiz +--- + +![th](https://github.com/user-attachments/assets/b54308e5-dd03-415c-a266-4db8675e2418) + + +## Introduction + + + +### Learning Objectives + + + + + +### Background Information + + + +## Getting Started +### Required Downloads and Installations +--- +Software: +Description: +Installation Process: +--- + + +### Required Components + +| Component Name | Quantity | +|--------------------------|----------| +| ESP32-S3 CAM | 1 | +| USB Cable | 1 | +| Power Supply | 1 | + +### Required Tools and Equipment + + +--- +## Part 01: + +### Introduction + + +### Objective + + + +### Background Information + + + +### Components + + + +### Instructional + + +## Example + +### Introduction + + + +### Example + + +### Analysis + + +#### Code Walkthrough + + + +--- + +## Part 02: + +### Introduction + +### Objective + + +### Background Information + + +### Components + +### Instructional + +## Example + +### Introduction + + +### Example Code + +### Analysis + +#### Code Walkthrough + + + +--- + +## Part 03: + +### Introduction + + +### Objective + + +### Background Information + + +### Components + + + +### Instructional + + + +## Example + +### Introduction + + +### Example Code + + +### Analysis + + + +#### Code Walkthrough + + +#### Implementation + + +--- + +## Part 04: Putting Everything Together + +### Introduction + + + +### Objective + + + +### Background Information + + +### Components + + +### Instructional Steps + + +## Example + +### Introduction + + +### Example Code + + + +https://github.com/user-attachments/assets/34143fbf-1968-48fc-ae7c-aee796cb4d17 + +## Analysis + +--- + +### Code Walkthrough + + +--- + +### Implementation + + +## Additional Resources + +### Useful links From c4ce3dca09683ef84ce9bae5935a977d10007008 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Mon, 19 May 2025 21:54:02 -0700 Subject: [PATCH 03/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index 02f6f11e..c58bb565 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -1,6 +1,6 @@ --- title: A Comprehensive Guide to Performing Computer Vision Tasks with ESP32-CAM Module -date: 11-25-2024 +date: 05-19-2025 authors: - name: Purab Balani - name: Jasmine Le From 7c1c8c95878be01e7dd5923d2422c721264b9ea5 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Mon, 19 May 2025 21:54:24 -0700 Subject: [PATCH 04/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index c58bb565..252b8edb 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -177,10 +177,6 @@ Installation Process: ### Example Code - - -https://github.com/user-attachments/assets/34143fbf-1968-48fc-ae7c-aee796cb4d17 - ## Analysis --- From 716b3bd3aac98b8d7ed8a8fdc1043ac227f516b2 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 01:34:10 -0700 Subject: [PATCH 05/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 306 ++++++++++++++++------------- 1 file changed, 166 insertions(+), 140 deletions(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index 252b8edb..fc6a29e0 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -1,194 +1,220 @@ ---- -title: A Comprehensive Guide to Performing Computer Vision Tasks with ESP32-CAM Module -date: 05-19-2025 -authors: - - name: Purab Balani - - name: Jasmine Le - - name: Genaro Salazar Ruiz ---- - -![th](https://github.com/user-attachments/assets/b54308e5-dd03-415c-a266-4db8675e2418) +--- +title: A Comprehensive Guide to Performing Computer Vision Tasks with ESP32-CAM Module +date: 05-19-2025 +authors: + - name: Purab Balani +tags: [ESP32-CAM, Computer Vision, IoT, YOLO, Object Detection, Python, OpenCV] +--- +![ESP32-CAM](https://github.com/user-attachments/assets/b54308e5-dd03-415c-a266-4db8675e2418) ## Introduction - +The ESP32-CAM is a powerful yet compact module combining a microcontroller, Wi-Fi, and a camera, making it ideal for IoT and computer vision tasks. In this tutorial, we will stream video from the ESP32-CAM to a host computer and run object detection using a YOLO model. ### Learning Objectives - - - +- Configure the ESP32-CAM module +- Stream video over Wi-Fi to a host machine +- Use OpenCV in Python to capture and display frames +- Run object detection with a YOLO model using Roboflow Inference +- Trigger audio alerts with text-to-speech (TTS) ### Background Information - +Computer vision enables machines to understand images and video. The ESP32-CAM cannot run large models directly, but it can stream footage that a host device processes using powerful libraries like OpenCV and YOLO. ## Getting Started + ### Required Downloads and Installations ---- -Software: -Description: -Installation Process: ---- +| Software | Description | Installation | +|-------------------|--------------------------------------|------------------------------------------------------------------------------| +| Arduino IDE | Upload firmware to ESP32-CAM | [Download](https://www.arduino.cc/en/software) | +| ESP32 Board Support | Adds ESP32 support to Arduino IDE | [Guide](https://randomnerdtutorials.com/installing-the-esp32-board-in-arduino-ide-windows-instructions/) | +| Python 3.x | Required for running detection script | [Download](https://www.python.org/) | +| OpenCV | Image capture & processing | `pip install opencv-python` | +| pyttsx3 | Text-to-speech engine | `pip install pyttsx3` | +| inference | Roboflow's inference SDK | `pip install inference` | ### Required Components -| Component Name | Quantity | -|--------------------------|----------| -| ESP32-S3 CAM | 1 | -| USB Cable | 1 | -| Power Supply | 1 | +| Component Name | Quantity | +|--------------------------|----------| +| ESP32-S3 CAM | 1 | +| USB Cable (for flashing) | 1 | +| Power Supply | 1 | ### Required Tools and Equipment +- Host PC (Linux/Windows/Mac) +- Wi-Fi network +- Breadboard (optional for peripherals) --- -## Part 01: - -### Introduction - - -### Objective - - - -### Background Information - - - -### Components - +## Part 01: Streaming Video from ESP32-CAM -### Instructional +### Objective +Flash ESP32-CAM and stream live video via Wi-Fi. +### Instructional Steps -## Example - -### Introduction - - - -### Example - - -### Analysis - +1. Open Arduino IDE. +2. Install the ESP32 board support and select `XIAO_ESP32S3`. +3. Use the provided ESP32-CAM code to flash the board. +4. Connect to the printed IP address and confirm video stream. -#### Code Walkthrough +### ESP32-CAM Firmware Code +
+Click to view +```cpp +// Code omitted for brevity; use full firmware from project files +``` +
--- -## Part 02: - -### Introduction - -### Objective - +## Part 02: Capturing and Displaying Frames on Host -### Background Information - - -### Components - -### Instructional - -## Example - -### Introduction - - -### Example Code - -### Analysis - -#### Code Walkthrough +### Objective +Connect to ESP32 stream and show real-time video using OpenCV. +### Instructional Steps +1. Use the ESP32 IP (e.g., `http://:81/stream`) in your Python code. +2. Use `cv2.VideoCapture()` to connect and read frames. --- -## Part 03: +## Part 03: Performing Object Detection -### Introduction +### Objective +Use Roboflow’s YOLO model to detect hazards in real-time. +### Code Walkthrough -### Objective - - -### Background Information - - -### Components - - - -### Instructional - - - -## Example - -### Introduction - - -### Example Code - - -### Analysis - - - -#### Code Walkthrough - - -#### Implementation - +- **Model Setup**: Using `get_model()` from Roboflow SDK +- **Detection**: `model.infer(frame)` returns bounding boxes and classes +- **Annotation**: Use `supervision` library to draw boxes and labels +- **TTS**: pyttsx3 announces hazards with cooldown logic + +### Full Detection Script +
+Click to view Python code + +```python +import cv2 +from inference import get_model +import supervision as sv +import pyttsx3 +import time + +STREAM_URL = "http://100.117.8.43:81/stream" + +def grab_latest_frame(cap, flush_count=4): + frame = None + for _ in range(flush_count): + ret, f = cap.read() + if not ret: + break + frame = f + return frame + +def open_stream(): + cap = cv2.VideoCapture(STREAM_URL) + if not cap.isOpened(): + print("Failed to open stream!") + return None + return cap + +cap = open_stream() +if cap is None: + raise RuntimeError("Could not open ESP32 stream.") + +model = get_model(model_id="196v2/1") +box_annotator = sv.BoxAnnotator() +label_annotator = sv.LabelAnnotator() +class_names = model.class_names +engine = pyttsx3.init() +engine.setProperty('rate', 175) + +hazard_classes = { + "wet_floor_sign": "wet floor sign", + "Safety-cone": "safety cone", + "Safety-bollard": "safety bollard", + "barrier tape": "barrier tape" +} + +last_spoken_time = 0 +SPEECH_COOLDOWN = 5 +fail_count = 0 +MAX_FAILS = 10 + +while True: + frame = grab_latest_frame(cap, flush_count=4) + if frame is None: + fail_count += 1 + cap.release() + time.sleep(1) + cap = open_stream() + if cap is None or fail_count > MAX_FAILS: + break + continue + fail_count = 0 + results = model.infer(frame)[0] + detections = sv.Detections.from_inference(results) + labels = [class_names[cid] for cid in detections.class_id] + current_time = time.time() + detected_hazards = [hazard_classes[label] for label in labels if label in hazard_classes] + detected_hazards = list(dict.fromkeys(detected_hazards)) + if detected_hazards and (current_time - last_spoken_time >= SPEECH_COOLDOWN): + if len(detected_hazards) == 1: + message = f"Warning: {detected_hazards[0]} detected ahead." + else: + combined = ", ".join(detected_hazards[:-1]) + " and " + detected_hazards[-1] + message = f"Warning: {combined} detected ahead." + engine.say(message) + engine.runAndWait() + last_spoken_time = current_time + annotated = box_annotator.annotate(scene=frame, detections=detections) + annotated = label_annotator.annotate(scene=annotated, detections=detections) + cv2.imshow("Live Detection", annotated) + if cv2.waitKey(1) & 0xFF == 27: + break +cap.release() +cv2.destroyAllWindows() +``` +
--- -## Part 04: Putting Everything Together - -### Introduction - +## Part 04: Putting It All Together +### Objective +Create a low-cost, vision-based alert system using ESP32 and YOLO. -### Objective +### Final Integration Steps +- Power up ESP32-CAM +- Run Python detection script on host +- Observe detection overlay + spoken alerts - - -### Background Information - - -### Components - - -### Instructional Steps - - -## Example - -### Introduction - - -### Example Code - -## Analysis +### Possible Use Cases +- Indoor navigation aid for visually impaired +- Smart home obstacle detection +- Security monitoring --- -### Code Walkthrough +## Additional Resources +- [ESP32-CAM Documentation](https://randomnerdtutorials.com/projects-esp32-cam/) +- [Roboflow Inference Docs](https://docs.roboflow.com/inference) +- [OpenCV Tutorials](https://docs.opencv.org/4.x/d6/d00/tutorial_py_root.html) +- [Supervision Library](https://github.com/roboflow/supervision) --- -### Implementation - - -## Additional Resources - -### Useful links +*Created by Purab Balani* From b1ed53626a5a5dd6da7a4f710def6c1b955a9d8f Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 01:35:33 -0700 Subject: [PATCH 06/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 384 ++++++++++++++++++++++++++++- 1 file changed, 383 insertions(+), 1 deletion(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index fc6a29e0..3edc8b28 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -70,7 +70,389 @@ Flash ESP32-CAM and stream live video via Wi-Fi. Click to view ```cpp -// Code omitted for brevity; use full firmware from project files +// #include "esp_camera.h" +// #include +// #include "esp_system.h" +// #include "esp_chip_info.h" + +// #define CAMERA_MODEL_XIAO_ESP32S3 // Has PSRAM +// #include "camera_pins.h" + +// const char *ssid = "RESNET-GUEST-DEVICE"; +// const char *password = "ResnetConnect"; + +// void startCameraServer(); +// void setupLedFlash(int pin); + +// void setup() { +// Serial.begin(115200); +// Serial.setDebugOutput(true); +// delay(2000); // Give time for serial monitor to connect + +// Serial.println("\n\n=== ESP32 Camera WiFi Debug ==="); + +// // Basic chip info using ESP32 Arduino methods +// Serial.printf("Chip Model: ESP32 variant\n"); +// Serial.printf("Chip Revision: %d\n", ESP.getChipRevision()); +// Serial.printf("CPU Frequency: %d MHz\n", ESP.getCpuFreqMHz()); +// Serial.printf("Flash Size: %d bytes\n", ESP.getFlashChipSize()); +// Serial.printf("Free Heap: %d bytes\n", ESP.getFreeHeap()); + +// // Check PSRAM +// if (psramFound()) { +// Serial.printf("PSRAM: Found, size: %d bytes\n", ESP.getPsramSize()); +// } else { +// Serial.println("PSRAM: Not found"); +// } + +// // Get MAC using WiFi library method +// Serial.println("\nInitializing WiFi..."); +// WiFi.mode(WIFI_STA); +// delay(100); + +// Serial.print("WiFi MAC address: "); +// Serial.println(WiFi.macAddress()); + +// // If MAC is still zeros, WiFi hardware has issues +// if (WiFi.macAddress() == "00:00:00:00:00:00") { +// Serial.println("CRITICAL ERROR: WiFi hardware not responding!"); +// Serial.println("Check:"); +// Serial.println("1. Board selection in Arduino IDE"); +// Serial.println("2. Flash settings"); +// Serial.println("3. Hardware may be defective"); +// Serial.println("Stopping here - fix WiFi hardware first"); +// while(1) delay(1000); // Stop execution +// } + +// // Camera initialization +// Serial.println("\nInitializing camera..."); +// camera_config_t config; +// config.ledc_channel = LEDC_CHANNEL_0; +// config.ledc_timer = LEDC_TIMER_0; +// config.pin_d0 = Y2_GPIO_NUM; +// config.pin_d1 = Y3_GPIO_NUM; +// config.pin_d2 = Y4_GPIO_NUM; +// config.pin_d3 = Y5_GPIO_NUM; +// config.pin_d4 = Y6_GPIO_NUM; +// config.pin_d5 = Y7_GPIO_NUM; +// config.pin_d6 = Y8_GPIO_NUM; +// config.pin_d7 = Y9_GPIO_NUM; +// config.pin_xclk = XCLK_GPIO_NUM; +// config.pin_pclk = PCLK_GPIO_NUM; +// config.pin_vsync = VSYNC_GPIO_NUM; +// config.pin_href = HREF_GPIO_NUM; +// config.pin_sccb_sda = SIOD_GPIO_NUM; +// config.pin_sccb_scl = SIOC_GPIO_NUM; +// config.pin_pwdn = PWDN_GPIO_NUM; +// config.pin_reset = RESET_GPIO_NUM; +// config.xclk_freq_hz = 20000000; +// config.frame_size = FRAMESIZE_UXGA; +// config.pixel_format = PIXFORMAT_JPEG; +// config.grab_mode = CAMERA_GRAB_WHEN_EMPTY; +// config.fb_location = CAMERA_FB_IN_PSRAM; +// config.jpeg_quality = 12; +// config.fb_count = 1; + +// if (config.pixel_format == PIXFORMAT_JPEG) { +// if (psramFound()) { +// config.jpeg_quality = 10; +// config.fb_count = 2; +// config.grab_mode = CAMERA_GRAB_LATEST; +// } else { +// config.frame_size = FRAMESIZE_SVGA; +// config.fb_location = CAMERA_FB_IN_DRAM; +// } +// } else { +// config.frame_size = FRAMESIZE_240X240; +// #if CONFIG_IDF_TARGET_ESP32S3 +// config.fb_count = 2; +// #endif +// } + +// esp_err_t err = esp_camera_init(&config); +// if (err != ESP_OK) { +// Serial.printf("Camera init failed with error 0x%x\n", err); +// return; +// } +// Serial.println("Camera initialized successfully!"); + +// sensor_t *s = esp_camera_sensor_get(); +// if (s->id.PID == OV3660_PID) { +// s->set_vflip(s, 1); +// s->set_brightness(s, 1); +// s->set_saturation(s, -2); +// } +// if (config.pixel_format == PIXFORMAT_JPEG) { +// s->set_framesize(s, FRAMESIZE_QVGA); +// } + +// #if defined(CAMERA_MODEL_ESP32S3_EYE) +// s->set_vflip(s, 1); +// #endif + +// #if defined(LED_GPIO_NUM) +// setupLedFlash(LED_GPIO_NUM); +// #endif + +// // Network scanning and connection +// Serial.println("\n=== WiFi Connection Process ==="); +// WiFi.disconnect(); +// delay(100); + +// Serial.println("Scanning for networks..."); +// int n = WiFi.scanNetworks(); +// Serial.printf("Found %d networks:\n", n); + +// bool targetFound = false; +// for (int i = 0; i < n; ++i) { +// Serial.printf("%d: %s (%d dBm) %s\n", +// i + 1, +// WiFi.SSID(i).c_str(), +// WiFi.RSSI(i), +// (WiFi.encryptionType(i) == WIFI_AUTH_OPEN) ? "[OPEN]" : "[SECURED]"); +// if (WiFi.SSID(i) == ssid) { +// targetFound = true; +// Serial.printf("*** Target network '%s' found with signal %d dBm\n", ssid, WiFi.RSSI(i)); +// } +// } + +// if (!targetFound) { +// Serial.printf("ERROR: Target network '%s' not found in scan!\n", ssid); +// Serial.println("Available networks listed above. Check network name exactly."); +// return; +// } + +// Serial.printf("Connecting to '%s'...\n", ssid); +// WiFi.begin(ssid, password); + +// int attempts = 0; +// while (WiFi.status() != WL_CONNECTED && attempts < 30) { +// delay(1000); +// attempts++; + +// wl_status_t status = WiFi.status(); +// Serial.printf("Attempt %d: ", attempts); + +// switch(status) { +// case WL_IDLE_STATUS: Serial.println("IDLE"); break; +// case WL_NO_SSID_AVAIL: Serial.println("NO_SSID - Network not found"); break; +// case WL_SCAN_COMPLETED: Serial.println("SCAN_COMPLETED"); break; +// case WL_CONNECTED: Serial.println("CONNECTED!"); break; +// case WL_CONNECT_FAILED: Serial.println("CONNECT_FAILED - Wrong password?"); break; +// case WL_CONNECTION_LOST: Serial.println("CONNECTION_LOST"); break; +// case WL_DISCONNECTED: Serial.println("DISCONNECTED"); break; +// default: Serial.printf("UNKNOWN_STATUS_%d\n", status); break; +// } + +// // Try reconnecting every 10 attempts +// if (attempts % 10 == 0 && status != WL_CONNECTED) { +// Serial.println("Retrying connection..."); +// WiFi.disconnect(); +// delay(1000); +// WiFi.begin(ssid, password); +// } +// } + +// if (WiFi.status() != WL_CONNECTED) { +// Serial.println("\n*** FAILED TO CONNECT ***"); +// Serial.println("Possible issues:"); +// Serial.println("1. Wrong password"); +// Serial.println("2. Network requires additional authentication"); +// Serial.println("3. MAC address filtering"); +// Serial.println("4. Network is 5GHz only"); +// return; +// } + +// Serial.println("\n*** WiFi Connected Successfully! ***"); +// Serial.print("IP address: "); +// Serial.println(WiFi.localIP()); +// Serial.print("Gateway: "); +// Serial.println(WiFi.gatewayIP()); +// Serial.print("DNS: "); +// Serial.println(WiFi.dnsIP()); +// Serial.print("Signal strength: "); +// Serial.print(WiFi.RSSI()); +// Serial.println(" dBm"); + +// startCameraServer(); + +// Serial.print("\nCamera server ready! Go to: http://"); +// Serial.println(WiFi.localIP()); +// } + +// void loop() { +// // Print connection status every 30 seconds +// static unsigned long lastCheck = 0; +// if (millis() - lastCheck > 30000) { +// lastCheck = millis(); +// if (WiFi.status() == WL_CONNECTED) { +// Serial.printf("WiFi OK - IP: %s, Signal: %d dBm\n", +// WiFi.localIP().toString().c_str(), WiFi.RSSI()); +// } else { +// Serial.println("WiFi connection lost!"); +// } +// } +// delay(1000); +// } + + + + + + + + + + + + + + + + +#include "esp_camera.h" +#include + +#define CAMERA_MODEL_XIAO_ESP32S3 +#include "camera_pins.h" + +const char *ssid = "RESNET-GUEST-DEVICE"; +const char *password = "ResnetConnect"; + +void startCameraServer(); +void setupLedFlash(int pin); + +void setup() { + Serial.begin(115200); + Serial.setDebugOutput(false); // Disable debug output for better performance + + // Optimize CPU frequency + setCpuFrequencyMhz(240); // Max frequency for ESP32S3 + + Serial.println("Starting optimized camera setup..."); + + camera_config_t config; + config.ledc_channel = LEDC_CHANNEL_0; + config.ledc_timer = LEDC_TIMER_0; + config.pin_d0 = Y2_GPIO_NUM; + config.pin_d1 = Y3_GPIO_NUM; + config.pin_d2 = Y4_GPIO_NUM; + config.pin_d3 = Y5_GPIO_NUM; + config.pin_d4 = Y6_GPIO_NUM; + config.pin_d5 = Y7_GPIO_NUM; + config.pin_d6 = Y8_GPIO_NUM; + config.pin_d7 = Y9_GPIO_NUM; + config.pin_xclk = XCLK_GPIO_NUM; + config.pin_pclk = PCLK_GPIO_NUM; + config.pin_vsync = VSYNC_GPIO_NUM; + config.pin_href = HREF_GPIO_NUM; + config.pin_sccb_sda = SIOD_GPIO_NUM; + config.pin_sccb_scl = SIOC_GPIO_NUM; + config.pin_pwdn = PWDN_GPIO_NUM; + config.pin_reset = RESET_GPIO_NUM; + + // Optimized camera settings for performance + config.xclk_freq_hz = 20000000; + config.pixel_format = PIXFORMAT_JPEG; + config.grab_mode = CAMERA_GRAB_LATEST; // Always get latest frame + config.fb_location = CAMERA_FB_IN_PSRAM; + + // Performance optimized settings + if (psramFound()) { + Serial.println("PSRAM found - using optimized settings"); + config.frame_size = FRAMESIZE_QQVGA; // 800x600 - good balance + config.jpeg_quality = 12; // Lower quality = faster + config.fb_count = 2; // Double buffering + } else { + Serial.println("No PSRAM - using conservative settings"); + config.frame_size = FRAMESIZE_QQVGA; // 640x480 + config.jpeg_quality = 15; + config.fb_count = 1; + config.fb_location = CAMERA_FB_IN_DRAM; + } + + // Initialize camera + esp_err_t err = esp_camera_init(&config); + if (err != ESP_OK) { + Serial.printf("Camera init failed with error 0x%x", err); + return; + } + + // Get camera sensor for optimization + sensor_t *s = esp_camera_sensor_get(); + + // Optimize sensor settings for speed + s->set_framesize(s, FRAMESIZE_QQVGA); // Start with VGA for speed + s->set_quality(s, 12); // JPEG quality (lower = faster) + + // Image enhancement settings + s->set_brightness(s, 0); // -2 to 2 + s->set_contrast(s, 0); // -2 to 2 + s->set_saturation(s, 0); // -2 to 2 + s->set_special_effect(s, 0); // 0 to 6 (0=No Effect) + s->set_whitebal(s, 1); // 0 = disable , 1 = enable + s->set_awb_gain(s, 1); // 0 = disable , 1 = enable + s->set_wb_mode(s, 0); // 0 to 4 - if awb_gain enabled + s->set_exposure_ctrl(s, 1); // 0 = disable , 1 = enable + s->set_aec2(s, 0); // 0 = disable , 1 = enable + s->set_ae_level(s, 0); // -2 to 2 + s->set_aec_value(s, 300); // 0 to 1200 + s->set_gain_ctrl(s, 1); // 0 = disable , 1 = enable + s->set_agc_gain(s, 0); // 0 to 30 + s->set_gainceiling(s, (gainceiling_t)0); // 0 to 6 + s->set_bpc(s, 0); // 0 = disable , 1 = enable + s->set_wpc(s, 1); // 0 = disable , 1 = enable + s->set_raw_gma(s, 1); // 0 = disable , 1 = enable + s->set_lenc(s, 1); // 0 = disable , 1 = enable + s->set_hmirror(s, 0); // 0 = disable , 1 = enable + s->set_vflip(s, 0); // 0 = disable , 1 = enable + s->set_dcw(s, 1); // 0 = disable , 1 = enable + s->set_colorbar(s, 0); // 0 = disable , 1 = enable + + // Camera model specific optimizations +#if defined(CAMERA_MODEL_XIAO_ESP32S3) + // No specific flips needed for XIAO ESP32S3 +#endif + +#if defined(LED_GPIO_NUM) + setupLedFlash(LED_GPIO_NUM); +#endif + + // WiFi setup with optimizations + WiFi.mode(WIFI_STA); + WiFi.setSleep(false); // Disable WiFi sleep for consistent performance + WiFi.setTxPower(WIFI_POWER_19_5dBm); // Max WiFi power + + Serial.printf("Connecting to %s", ssid); + WiFi.begin(ssid, password); + + while (WiFi.status() != WL_CONNECTED) { + delay(500); + Serial.print("."); + } + Serial.println(""); + Serial.println("WiFi connected"); + + startCameraServer(); + + Serial.print("Camera Ready! Use 'http://"); + Serial.print(WiFi.localIP()); + Serial.println("' to connect"); + + // Print optimization info + Serial.println("\nOptimization Settings Applied:"); + Serial.printf("CPU Frequency: %d MHz\n", getCpuFrequencyMhz()); + Serial.printf("PSRAM Available: %s\n", psramFound() ? "Yes" : "No"); + Serial.printf("Frame Size: %s\n", psramFound() ? "QQVGA" : "QQVGA"); + Serial.printf("JPEG Quality: %d\n", psramFound() ? 12 : 15); + Serial.printf("Frame Buffers: %d\n", psramFound() ? 2 : 1); +} + +void loop() { + // Keep loop minimal for best performance + delay(1); +} + ``` From 840983d05e88a72da8ea55c4a0d4a23064a91810 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 01:36:07 -0700 Subject: [PATCH 07/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 240 ----------------------------- 1 file changed, 240 deletions(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index 3edc8b28..864e6a6a 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -70,246 +70,6 @@ Flash ESP32-CAM and stream live video via Wi-Fi. Click to view ```cpp -// #include "esp_camera.h" -// #include -// #include "esp_system.h" -// #include "esp_chip_info.h" - -// #define CAMERA_MODEL_XIAO_ESP32S3 // Has PSRAM -// #include "camera_pins.h" - -// const char *ssid = "RESNET-GUEST-DEVICE"; -// const char *password = "ResnetConnect"; - -// void startCameraServer(); -// void setupLedFlash(int pin); - -// void setup() { -// Serial.begin(115200); -// Serial.setDebugOutput(true); -// delay(2000); // Give time for serial monitor to connect - -// Serial.println("\n\n=== ESP32 Camera WiFi Debug ==="); - -// // Basic chip info using ESP32 Arduino methods -// Serial.printf("Chip Model: ESP32 variant\n"); -// Serial.printf("Chip Revision: %d\n", ESP.getChipRevision()); -// Serial.printf("CPU Frequency: %d MHz\n", ESP.getCpuFreqMHz()); -// Serial.printf("Flash Size: %d bytes\n", ESP.getFlashChipSize()); -// Serial.printf("Free Heap: %d bytes\n", ESP.getFreeHeap()); - -// // Check PSRAM -// if (psramFound()) { -// Serial.printf("PSRAM: Found, size: %d bytes\n", ESP.getPsramSize()); -// } else { -// Serial.println("PSRAM: Not found"); -// } - -// // Get MAC using WiFi library method -// Serial.println("\nInitializing WiFi..."); -// WiFi.mode(WIFI_STA); -// delay(100); - -// Serial.print("WiFi MAC address: "); -// Serial.println(WiFi.macAddress()); - -// // If MAC is still zeros, WiFi hardware has issues -// if (WiFi.macAddress() == "00:00:00:00:00:00") { -// Serial.println("CRITICAL ERROR: WiFi hardware not responding!"); -// Serial.println("Check:"); -// Serial.println("1. Board selection in Arduino IDE"); -// Serial.println("2. Flash settings"); -// Serial.println("3. Hardware may be defective"); -// Serial.println("Stopping here - fix WiFi hardware first"); -// while(1) delay(1000); // Stop execution -// } - -// // Camera initialization -// Serial.println("\nInitializing camera..."); -// camera_config_t config; -// config.ledc_channel = LEDC_CHANNEL_0; -// config.ledc_timer = LEDC_TIMER_0; -// config.pin_d0 = Y2_GPIO_NUM; -// config.pin_d1 = Y3_GPIO_NUM; -// config.pin_d2 = Y4_GPIO_NUM; -// config.pin_d3 = Y5_GPIO_NUM; -// config.pin_d4 = Y6_GPIO_NUM; -// config.pin_d5 = Y7_GPIO_NUM; -// config.pin_d6 = Y8_GPIO_NUM; -// config.pin_d7 = Y9_GPIO_NUM; -// config.pin_xclk = XCLK_GPIO_NUM; -// config.pin_pclk = PCLK_GPIO_NUM; -// config.pin_vsync = VSYNC_GPIO_NUM; -// config.pin_href = HREF_GPIO_NUM; -// config.pin_sccb_sda = SIOD_GPIO_NUM; -// config.pin_sccb_scl = SIOC_GPIO_NUM; -// config.pin_pwdn = PWDN_GPIO_NUM; -// config.pin_reset = RESET_GPIO_NUM; -// config.xclk_freq_hz = 20000000; -// config.frame_size = FRAMESIZE_UXGA; -// config.pixel_format = PIXFORMAT_JPEG; -// config.grab_mode = CAMERA_GRAB_WHEN_EMPTY; -// config.fb_location = CAMERA_FB_IN_PSRAM; -// config.jpeg_quality = 12; -// config.fb_count = 1; - -// if (config.pixel_format == PIXFORMAT_JPEG) { -// if (psramFound()) { -// config.jpeg_quality = 10; -// config.fb_count = 2; -// config.grab_mode = CAMERA_GRAB_LATEST; -// } else { -// config.frame_size = FRAMESIZE_SVGA; -// config.fb_location = CAMERA_FB_IN_DRAM; -// } -// } else { -// config.frame_size = FRAMESIZE_240X240; -// #if CONFIG_IDF_TARGET_ESP32S3 -// config.fb_count = 2; -// #endif -// } - -// esp_err_t err = esp_camera_init(&config); -// if (err != ESP_OK) { -// Serial.printf("Camera init failed with error 0x%x\n", err); -// return; -// } -// Serial.println("Camera initialized successfully!"); - -// sensor_t *s = esp_camera_sensor_get(); -// if (s->id.PID == OV3660_PID) { -// s->set_vflip(s, 1); -// s->set_brightness(s, 1); -// s->set_saturation(s, -2); -// } -// if (config.pixel_format == PIXFORMAT_JPEG) { -// s->set_framesize(s, FRAMESIZE_QVGA); -// } - -// #if defined(CAMERA_MODEL_ESP32S3_EYE) -// s->set_vflip(s, 1); -// #endif - -// #if defined(LED_GPIO_NUM) -// setupLedFlash(LED_GPIO_NUM); -// #endif - -// // Network scanning and connection -// Serial.println("\n=== WiFi Connection Process ==="); -// WiFi.disconnect(); -// delay(100); - -// Serial.println("Scanning for networks..."); -// int n = WiFi.scanNetworks(); -// Serial.printf("Found %d networks:\n", n); - -// bool targetFound = false; -// for (int i = 0; i < n; ++i) { -// Serial.printf("%d: %s (%d dBm) %s\n", -// i + 1, -// WiFi.SSID(i).c_str(), -// WiFi.RSSI(i), -// (WiFi.encryptionType(i) == WIFI_AUTH_OPEN) ? "[OPEN]" : "[SECURED]"); -// if (WiFi.SSID(i) == ssid) { -// targetFound = true; -// Serial.printf("*** Target network '%s' found with signal %d dBm\n", ssid, WiFi.RSSI(i)); -// } -// } - -// if (!targetFound) { -// Serial.printf("ERROR: Target network '%s' not found in scan!\n", ssid); -// Serial.println("Available networks listed above. Check network name exactly."); -// return; -// } - -// Serial.printf("Connecting to '%s'...\n", ssid); -// WiFi.begin(ssid, password); - -// int attempts = 0; -// while (WiFi.status() != WL_CONNECTED && attempts < 30) { -// delay(1000); -// attempts++; - -// wl_status_t status = WiFi.status(); -// Serial.printf("Attempt %d: ", attempts); - -// switch(status) { -// case WL_IDLE_STATUS: Serial.println("IDLE"); break; -// case WL_NO_SSID_AVAIL: Serial.println("NO_SSID - Network not found"); break; -// case WL_SCAN_COMPLETED: Serial.println("SCAN_COMPLETED"); break; -// case WL_CONNECTED: Serial.println("CONNECTED!"); break; -// case WL_CONNECT_FAILED: Serial.println("CONNECT_FAILED - Wrong password?"); break; -// case WL_CONNECTION_LOST: Serial.println("CONNECTION_LOST"); break; -// case WL_DISCONNECTED: Serial.println("DISCONNECTED"); break; -// default: Serial.printf("UNKNOWN_STATUS_%d\n", status); break; -// } - -// // Try reconnecting every 10 attempts -// if (attempts % 10 == 0 && status != WL_CONNECTED) { -// Serial.println("Retrying connection..."); -// WiFi.disconnect(); -// delay(1000); -// WiFi.begin(ssid, password); -// } -// } - -// if (WiFi.status() != WL_CONNECTED) { -// Serial.println("\n*** FAILED TO CONNECT ***"); -// Serial.println("Possible issues:"); -// Serial.println("1. Wrong password"); -// Serial.println("2. Network requires additional authentication"); -// Serial.println("3. MAC address filtering"); -// Serial.println("4. Network is 5GHz only"); -// return; -// } - -// Serial.println("\n*** WiFi Connected Successfully! ***"); -// Serial.print("IP address: "); -// Serial.println(WiFi.localIP()); -// Serial.print("Gateway: "); -// Serial.println(WiFi.gatewayIP()); -// Serial.print("DNS: "); -// Serial.println(WiFi.dnsIP()); -// Serial.print("Signal strength: "); -// Serial.print(WiFi.RSSI()); -// Serial.println(" dBm"); - -// startCameraServer(); - -// Serial.print("\nCamera server ready! Go to: http://"); -// Serial.println(WiFi.localIP()); -// } - -// void loop() { -// // Print connection status every 30 seconds -// static unsigned long lastCheck = 0; -// if (millis() - lastCheck > 30000) { -// lastCheck = millis(); -// if (WiFi.status() == WL_CONNECTED) { -// Serial.printf("WiFi OK - IP: %s, Signal: %d dBm\n", -// WiFi.localIP().toString().c_str(), WiFi.RSSI()); -// } else { -// Serial.println("WiFi connection lost!"); -// } -// } -// delay(1000); -// } - - - - - - - - - - - - - - - - #include "esp_camera.h" #include From 542ffe67507143b82510ed9cc4d0bbaf431af31f Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 01:40:39 -0700 Subject: [PATCH 08/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 86 +++++++++--------------------- 1 file changed, 24 insertions(+), 62 deletions(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index 864e6a6a..9d1aba62 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -251,81 +251,43 @@ Use Roboflow’s YOLO model to detect hazards in real-time. import cv2 from inference import get_model import supervision as sv -import pyttsx3 -import time - -STREAM_URL = "http://100.117.8.43:81/stream" - -def grab_latest_frame(cap, flush_count=4): - frame = None - for _ in range(flush_count): - ret, f = cap.read() - if not ret: - break - frame = f - return frame - -def open_stream(): - cap = cv2.VideoCapture(STREAM_URL) - if not cap.isOpened(): - print("Failed to open stream!") - return None - return cap - -cap = open_stream() -if cap is None: + +# Replace with your ESP32-CAM stream URL +STREAM_URL = "http://:81/stream" #change this + +# Open video stream +cap = cv2.VideoCapture(STREAM_URL) +if not cap.isOpened(): raise RuntimeError("Could not open ESP32 stream.") -model = get_model(model_id="196v2/1") +# Load Roboflow model +model = get_model(model_id="YOUR MODEL ID") #change this box_annotator = sv.BoxAnnotator() label_annotator = sv.LabelAnnotator() class_names = model.class_names -engine = pyttsx3.init() -engine.setProperty('rate', 175) - -hazard_classes = { - "wet_floor_sign": "wet floor sign", - "Safety-cone": "safety cone", - "Safety-bollard": "safety bollard", - "barrier tape": "barrier tape" -} - -last_spoken_time = 0 -SPEECH_COOLDOWN = 5 -fail_count = 0 -MAX_FAILS = 10 +# Main loop while True: - frame = grab_latest_frame(cap, flush_count=4) - if frame is None: - fail_count += 1 - cap.release() - time.sleep(1) - cap = open_stream() - if cap is None or fail_count > MAX_FAILS: - break - continue - fail_count = 0 + ret, frame = cap.read() + if not ret: + print("Failed to read frame.") + break + + # Run inference results = model.infer(frame)[0] detections = sv.Detections.from_inference(results) - labels = [class_names[cid] for cid in detections.class_id] - current_time = time.time() - detected_hazards = [hazard_classes[label] for label in labels if label in hazard_classes] - detected_hazards = list(dict.fromkeys(detected_hazards)) - if detected_hazards and (current_time - last_spoken_time >= SPEECH_COOLDOWN): - if len(detected_hazards) == 1: - message = f"Warning: {detected_hazards[0]} detected ahead." - else: - combined = ", ".join(detected_hazards[:-1]) + " and " + detected_hazards[-1] - message = f"Warning: {combined} detected ahead." - engine.say(message) - engine.runAndWait() - last_spoken_time = current_time + + # Annotate frame annotated = box_annotator.annotate(scene=frame, detections=detections) annotated = label_annotator.annotate(scene=annotated, detections=detections) - cv2.imshow("Live Detection", annotated) + + # Show result + cv2.imshow("YOLO Detection", annotated) + + # Press ESC to quit if cv2.waitKey(1) & 0xFF == 27: break + cap.release() cv2.destroyAllWindows() ``` From a066d9ec4fa3cd7008d36239684dfabd0b3cabd8 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 01:41:50 -0700 Subject: [PATCH 09/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index 9d1aba62..7453f26b 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -1,5 +1,5 @@ --- -title: A Comprehensive Guide to Performing Computer Vision Tasks with ESP32-CAM Module +title: A Basic Guide to Performing Computer Vision Tasks with ESP32-CAM Module date: 05-19-2025 authors: - name: Purab Balani From 30eb5d541d177c0c2b99463ef84cc8d64b5c1f44 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 01:43:17 -0700 Subject: [PATCH 10/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index 7453f26b..109ea674 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -6,7 +6,8 @@ authors: tags: [ESP32-CAM, Computer Vision, IoT, YOLO, Object Detection, Python, OpenCV] --- -![ESP32-CAM](https://github.com/user-attachments/assets/b54308e5-dd03-415c-a266-4db8675e2418) +![image](https://github.com/user-attachments/assets/3ea1b21b-82df-472a-acaa-73c45b6e4162) + ## Introduction From 64b2c5ae3e2e400eceb0ad1a98200985ac8c7221 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 01:43:58 -0700 Subject: [PATCH 11/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index 109ea674..9123e92a 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -35,7 +35,6 @@ Computer vision enables machines to understand images and video. The ESP32-CAM c | ESP32 Board Support | Adds ESP32 support to Arduino IDE | [Guide](https://randomnerdtutorials.com/installing-the-esp32-board-in-arduino-ide-windows-instructions/) | | Python 3.x | Required for running detection script | [Download](https://www.python.org/) | | OpenCV | Image capture & processing | `pip install opencv-python` | -| pyttsx3 | Text-to-speech engine | `pip install pyttsx3` | | inference | Roboflow's inference SDK | `pip install inference` | ### Required Components From 2e20566e9a2faf2b2047d045087bc327988f8a4d Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 01:48:41 -0700 Subject: [PATCH 12/13] Update Team9Tutorial.md --- content/tutorials/Team9Tutorial.md | 78 +++++++++++++++++++++++++++++- 1 file changed, 76 insertions(+), 2 deletions(-) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/Team9Tutorial.md index 9123e92a..ea042acb 100644 --- a/content/tutorials/Team9Tutorial.md +++ b/content/tutorials/Team9Tutorial.md @@ -231,7 +231,81 @@ Connect to ESP32 stream and show real-time video using OpenCV. --- -## Part 03: Performing Object Detection +## Part 03: Choosing and Training Your Model with Roboflow + +### Objective +Learn how to choose the best model type and train your own dataset using Roboflow. + +### Choosing a Model Type +Roboflow supports many model architectures. For use with an ESP32-CAM that streams to a host machine (which does the heavy computation), you’ll want to select a lightweight model optimized for speed and acceptable accuracy. Good options include: + +- **YOLOv5n** – “Nano” version of YOLOv5, very fast but less accurate. +- **YOLOv8n** – Latest nano version of YOLOv8, offering better trade-offs. +- **YOLOv8s** – Slightly larger, better accuracy, still usable on laptops. +- **MobileNet-SSD** – Great for low-latency mobile applications. + +> ✅ **Tip:** Start with YOLOv8n and scale up if needed. + +--- + +### Training a Model in Roboflow + +1. Go to [https://roboflow.com](https://roboflow.com) and create a free account. +2. Click **"Create Project"** and set your object detection parameters. +3. Upload your images and annotate them using Roboflow’s labeling interface. +4. After labeling, click **"Generate Dataset"** to resize and augment your images. +5. Click **"Train Model"** and choose a suitable model type (e.g. YOLOv8n). +6. When training is done, you'll receive a `model_id` for use with the Inference SDK. + +--- + +### Using Your Trained Model + +To use your trained model with the Roboflow Inference SDK: + +```python +from inference import get_model + +model = get_model(model_id="your_model_id") +``` + +If you're deploying with a `.pt` file locally (instead of the Roboflow-hosted model): + +- Download the YOLO weights (`weights.pt`) from Roboflow. +- Use [Ultralytics YOLOv8](https://github.com/ultralytics/ultralytics) locally: + +```bash +pip install ultralytics +``` + +Then you can run: + +```python +from ultralytics import YOLO + +model = YOLO("your_model.pt") +results = model.predict("image.jpg") +``` + +--- + +### Summary + +| Scenario | Recommended Model | +|----------|--------------------| +| Streaming from ESP32-CAM to PC | YOLOv8n or YOLOv5n | +| Deployment on Jetson Nano or similar | YOLOv8s | +| Mobile deployment (e.g., Android app) | MobileNet-SSD | +| Need best accuracy | YOLOv8m or YOLOv5m | + +--- + +```markdown +💡 Consider uploading diverse real-world images for best results during training. +``` +--- + +## Part 04: Performing Object Detection ### Objective Use Roboflow’s YOLO model to detect hazards in real-time. @@ -295,7 +369,7 @@ cv2.destroyAllWindows() --- -## Part 04: Putting It All Together +## Part 05: Putting It All Together ### Objective Create a low-cost, vision-based alert system using ESP32 and YOLO. From 3845155e453daf6a3d797fe4b4cefb28c4716986 Mon Sep 17 00:00:00 2001 From: PurabB <117234132+PurabB@users.noreply.github.com> Date: Fri, 6 Jun 2025 15:02:44 -0700 Subject: [PATCH 13/13] Rename Team9Tutorial.md to CV_ESP32-CAM_Tutorial.md --- content/tutorials/{Team9Tutorial.md => CV_ESP32-CAM_Tutorial.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename content/tutorials/{Team9Tutorial.md => CV_ESP32-CAM_Tutorial.md} (100%) diff --git a/content/tutorials/Team9Tutorial.md b/content/tutorials/CV_ESP32-CAM_Tutorial.md similarity index 100% rename from content/tutorials/Team9Tutorial.md rename to content/tutorials/CV_ESP32-CAM_Tutorial.md