Watcher Quick Start Series 5# : Training a model for Watcher

[gmacario]

Hello,

I read in the "SenseCAP Watcher Wiki Center" https://wiki.seeedstudio.com/watcher/

that a few articles are marked "Planning" or "Coming Soon".

My team is particularly interested in the article highlighted above, in order to progress on the "FREISA meets SenseCAP Watcher" project we have recently posted on Hackster:

https://www.hackster.io/b-arol-o/freisa-meets-sensecap-watcher-89596b

I understand you are quite busy at the moment with the launch of the Kickstarter campaign, however it would be quite appreciated if we can get a rough timeline of when the article will be published - or at least a PR created.

Our team volunteers to review it!

[github-actions]

👋 @gmacario

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.

[djairjr]

I thought your work at Hackster was incredible! I'm also waiting for the possibility of training a custom model for Watcher, in the same way as what happens with Grove Vision AI V2. From what I understand, it seems that a lot of things will be done using the Yolo-World paradigm, that is, it will integrate LLM with object identification. I still don't quite understand how this will be done. Will we still have to go through the steps of collecting images, tagging, etc...? Or will this integration of models from different media somehow solve the training bottleneck? I'll keep following along here and if I can help in any way, I will.

[gmacario]

Hello @djairjr, thank you very much for your kind words about our "FREISA meets SenseCAP Watcher" project.

I'm also waiting for the possibility of training a custom model for Watcher, in the same way as what happens with Grove Vision AI V2.

We were also thinking about this line and considering to start with the Grove Vision AI Module V2 first.

Do you have any pointers to suggest us? More specifically, how to convert an ONNX model to a format suitable for either the Grove Vision AI V2 and/or the SenseCAP Watcher.

Concerning your offer for help

I'll keep following along here and if I can help in any way, I will.

Davide is the person who has been working on the custom training for FREISA, he will definitely get back to you soon!

[djairjr]

Did you read @Mjrovai tutorials?
I am using roboflow and sscma notebooks to train.

[djairjr]

With the Grove and Esp32-s3 I could Control an external bluetooth service. And also get Homeassistant and esphome connection too. For now, with Watcher, we can only use serial rw or i2c read.

[MatthewJeffson]

Hi! @gmacario and @djairjr ! Thanks for interest in SenseCAP Watcher!
I will ask R&D department regarding the custom models training.:)

[djairjr]

@gmacario, did you see this: https://docs.edgeimpulse.com/docs/edge-ai-hardware/mcu-+-ai-accelerators/himax-seeed-grove-vision-ai-module-v2-wise-eye-2 ?

I have a device I'm working on that uses an esp32-s3 connected to a Grove Vision AI V2 and another one that uses an esp32-c3 connected to a motor driver. I've managed to create a state machine on the esp32-s3 that sends commands via bluetooth to the esp32-c3.

The esp32-c3 is programmed with the esphome firmware. But the esp32-s3, which is connected to the Grove Vision AI V2, was programmed in C++ with the Arduino IDE.

I saw that you respected my project on Hackster. But I don't know if you noticed that there I also shared the code for the game with Bluetooth activation, from the reward device. I have the impression that this can help with your project.

/**
  Simple Paper, Rock, Scissor Game
  2024 - By Djair Guilherme @nicolaudosbrinquedos
  Using Grove Vision AI V2 - with Seeed XIAO ESP32-S3
  and Gesture Detection Model

  Access Bluetooth Device to give a prize to winner

  https://github.com/Seeed-Studio/sscma-model-zoo/blob/main/docs/en/Gesture_Detection_Swift-YOLO_192.md

*/

// Periferico Bluetooth
#include "BLEDevice.h"
// Get an UUID at https://www.uuidgenerator.net/

static BLEUUID serviceUUID("https://www.uuidgenerator.net/");
static BLEUUID activateUUID("https://www.uuidgenerator.net/");
static BLERemoteCharacteristic* bluetoothPrize;

// Communication with Grove Vision AI V2

#include <Seeed_Arduino_SSCMA.h>
SSCMA GrooveAI;

// The inference returns Index of Object, not String

#define  PAPER_TARGET   0
#define  ROCK_TARGET   1
#define  SCISSOR_TARGET   2

#ifdef ESP32
#include <HardwareSerial.h>
HardwareSerial atSerial(0);

#else
#define atSerial Serial1
#endif

bool is_paper = false;
bool is_rock = false;
bool is_scissor = false;

int computerGuess = -1;
int score = 0;
int found = -1;
int lastGuess = -1;
unsigned long lastGuessTime = 0;

void setup()
{
  randomSeed(analogRead(0));
  GrooveAI.begin(&atSerial, D3);
  Serial.begin(115200);
  delay(2000);
  Serial.println("ROCK PAPER SCISSOR BLUETOOTH PRIZE");


  BLEDevice::init("");
  BLEClient*  pClient = BLEDevice::createClient();

// Get the Mac Address of Prize Dispenser

  if (pClient->connect(BLEAddress("mac_address_prize"))) {
    Serial.println("Connected to Prize Dispenser");
  } else Serial.println("Cant connect to Prize Dispenser");

  BLERemoteService* pRemoteService = pClient->getService(serviceUUID);
  if (pRemoteService == nullptr) {
    Serial.println("Cant find Service");
    return;
  } else Serial.println("Service Found");

  bluetoothPrize = pRemoteService->getCharacteristic(activateUUID);
  if (bluetoothPrize == nullptr) {
    Serial.println("Cant found Characteristic");
    return;
  } else Serial.println("Characteristic Found");
}


void loop() {

  if (!GrooveAI.invoke(1, false, true)) {
    bool found = false;
    int playerGuess = -1;

    // Last Guess Time
    unsigned long currentTime = millis();
    unsigned long elapsedTime = currentTime - lastGuessTime;

    // For each Box, detect if is PAPER ROCK or SCISSOR
    for (int i = 0; i < GrooveAI.boxes().size(); i++) {

      if (GrooveAI.boxes()[i].target == ROCK_TARGET) {
        is_rock = true;
        playerGuess = ROCK_TARGET;
      } else if (GrooveAI.boxes()[i].target == PAPER_TARGET) {
        is_paper = true;
        playerGuess = PAPER_TARGET;
      } else if (GrooveAI.boxes()[i].target == SCISSOR_TARGET) {
        is_scissor = true;
        playerGuess = SCISSOR_TARGET;
      } else {
        is_rock = false;
        is_paper = false;
        is_scissor = false;
        found = false;
      }
    }

    // Update 'found' only if playerGuess is different than before or if 3s of last guess
    if ((playerGuess != lastGuess || elapsedTime > 3000)  && playerGuess != -1) {
      found = true;
      lastGuess = playerGuess;
    }

    if (found) {
      // Create Computer Guess after inference
      computerGuess = random(0, 3);
      Serial.println("+++++++++++++++");
      String yourGame = (playerGuess == 0) ? "PAPER" :
                       (playerGuess == 1) ? "ROCK" : "SCISSOR";
      String myGame = (computerGuess == 0) ? "PAPER" :
                       (computerGuess == 1) ? "ROCK" : "SCISSOR";

      int result = determine_winner(playerGuess, computerGuess);

      if (result == 2) {
        Serial.println("DRAW");
      } else if (result == 1) {
        Serial.println("Human WIN");
        winnerPrize();
      } else {
        Serial.println("AI WIN");
      }

      Serial.print("Your guess is ");
      Serial.print(yourGame);
      Serial.print(" and mine is ");
      Serial.println(myGame);
      Serial.println("+++++++++++++++");
      Serial.println();
      
      // Reset conditions
      is_rock = false;
      is_paper = false;
      is_scissor = false;
      found = false;    
    }
  }
  delay(1000);
}

// WINNING CONDITIONS

int determine_winner(int playerGuess, int computerGuess) {
  if (playerGuess == computerGuess) {
    return 2; // Empate
  } else if ((playerGuess == 0 && computerGuess == 1) || // Papel contra Pedra
             (playerGuess == 1 && computerGuess == 2) || // Pedra contra Tesoura
             (playerGuess == 2 && computerGuess == 0)) { // Tesoura contra Papel
    return 1; // playerGuess vence
  } else {
    return 0; // computerGuess vence
  }
}


void winnerPrize() {
  bluetoothPrize->writeValue((byte)0x01);
  delay(1000);
  bluetoothPrize->writeValue((byte)0x00);
}