The VisionAI App Backend is a machine learning-powered image recognition system that uses TensorFlow.js for feature extraction and Pinecone as a vector database for similarity search. The application allows users to train the model with new images and later detect objects by querying against the trained dataset.
- Train Mode: Capture an image, extract features, and store embeddings in Pinecone.
- Detect Mode: Capture an image, extract features, and search Pinecone for the most similar object.
- MobileNet Model: Uses either:
- ✅ A pre-trained MobileNet v2 model loaded dynamically from TensorFlow.js.
- ✅ A custom-trained MobileNet v2 model stored locally.
- Pinecone Integration: Stores and retrieves high-dimensional embeddings for efficient similarity search.
- Node.js & TensorFlow.js: Eliminates the need for Python dependencies by using TensorFlow.js in a Node.js environment.
cd vision-ai
This project uses pnpm
as the package manager instead of npm
. If you don’t have pnpm
installed, run:
npm install -g pnpm
Then install dependencies:
pnpm install
Create a .env
file in both server and client directories.
PINECONE_API_KEY=your-pinecone-api-key
PINECONE_INDEX_NAME=image-recognition
PINECONE_DIMENSIONS=768
USE_CUSTOM_MODEL=false # Set to 'true' to use the custom model in /models/mobilenet/mobilenet_tfjs/
VITE_API_HOST=http://localhost:3000
cd server
pnpm dev
cd client
pnpm dev
By default, the frontend runs on http://localhost:5173
The backend runs on http://localhost:3000
brew install ngrok
- Sign up for a free ngrok account
https://dashboard.ngrok.com/signup
- Login to your account
https://dashboard.ngrok.com/login
- In the ngrok Dashboard:
- Getting Started
- Your Authtoken
- Getting Started
cd
into~/Library/Application Support/ngrok
- run
nano ngrok.yml
, (add your ngrok auth token) paste the following and save the file:
version: "3"
agent:
authtoken: your-ngrok-auth-token
tunnels:
frontend:
addr: 5173
proto: http
backend:
addr: 3000
proto: http
cd
intovision-ai/server
and run the following command:- run
ngrok start --all
- Copy the URL for the backend tunnel
- Update the
VITE_API_HOST
variable in the client.env
file
cd server
pnpm dev
cd client
pnpm dev
- copy the ngrok URL for the frontend tunnel
- open the URL in your mobile device browser
For detailed information, check out the individual README files:
- 📜 Backend Docs → server/README.md
- 📜 Frontend Docs → client/README.md
With this guide, you should be able to:
- Start both the server and client applications.
- Capture images, train the AI, and detect objects.
- Debug issues using built-in logging.
- Use a pre-trained or custom MobileNet model.
🚀 Happy Coding! 🎉