Clara is your personal local picture search and insight assistant with data privacy:
- Ever want to dig up your ancienct receipt photos? long-lost memes? your beloved cat pictures?
- ...But don't wanna upload your photos online 'cause it's too dangerous?
- Use Clara to find all pictures relevant to a keyword by a simple search!
- No internet connection needed after downloading - all image processing happens offline, and your data is safe!
- Made with love, friendly for blind users and screen readers : )
- Ever wanted to dig up your ancienct receipt photos? long-lost memes? your beloved cat pictures? Yes, we've all been there - we spend hours scrolling through the entire photo library to find it, and this only takes longer if we have poor sight or are blind but wanna share images with our sighted friends.
- Our solution to this problem is: What if we can search our photos with a keyword, just like Google Image Search?
- So we created Clara. We made it accessible for those with vision disabilites as well, hoping that they'll also have the freedom to easily navigate through their photo albums.
Clara is your personal local picture search and insight assistant. The name borrows from the latin word clarus meaning clear, bright, famous, as this app aspires to make pictures searchable and clearly annotated for users.
Users will type a keyword of what they want to search for, for example "cat".
Users will select from our three different types of searches:
Image Description: our algorithm will return all the images containing a cat, for example: "a picture of a cat sitting on grass land" or "a cat standing on a hill". In the future, we will also return the corresponding image description texts, which existing alt-text readers could then read aloud for blind users.Object Detection: This will return all the images containing that keyword, for example all images containing a cat.OCR (Optical Character Recognition): This will return all images containing the actual keyword inputted, for example all images containing the word "cat".
Clara streamlines the image search process using image-to-text algorithms, making images more accessible for everyone.
- PC (as a web app running on localhost)
- mobile coming up soon
- We deployed open-source, pretrained machine learning models such as multimodal transformer for image captioning, ResNet-LSTM-autoencoder based OCR model, and YOLOv3 supported object recognition model to achieve our core functions:
Image Description,Object Detection, andOCR (Optical Character Recognition)to process all images in a given folder. - Then we displayed all search result images on front end, a website running on localhost only. No internet connection needed after downloading - all image processing happens offline, and user data is super safe!
- For details, see project roadmap.
- Improve UI to make it more easily read
- Display an image description text for each image in the search result (the current version has this information in backend, just need to pull it up on frontend), so a blind user can use their alt-text reader to read that aloud
- Integrate a text reader into the front end, so user's without a pre-installed reader will still be able to make the most of our application
- Pre-compile the app so it's easily installed