myGPTReader

myGPTReader is a slack bot that reads web pages and summarizes them with chatGPT. You can use it to read news and summarize them in your slack channel.

For now it is in development, but you can try it out by join this channel.

The exciting part is that the development of this project is also paired with chatGPT. I document the development process in this CDDR file.

Features

• Integrated with slack bot
  • Bot replies messages in the same thread
• Support web page reading with chatGPT
  • Consider to use cloudflare worker to scrape the html content
    • Self-hosting Web Scraper
    • Restrict to access the web scraper, only allow the API server to access it by Cloudflare Access
  • Consider to use a headless browser to scrape the web page content like twitter thread
    • https://www.browserless.io/
    • https://phantomjscloud.com/
    • https://scrapfly.io/
  • Consider to use OCR to scrape the web page content (Web crawler to get the screenshot, then OCR to get the text)
    • Azure OCR
    • Google Vision
    • may use GPT4
• Support RSS reading with chatGPT
  • RSS is a bunch of links, so it is equivalent to reading a web page to get the content.
• Support newsletter reading with chatGPT
  • Most newsletters are public and can be accessed online, so we can just give the url to the slack bot.
• Prompt fine-tue
  • Support for custom prompt
  • Show prompt templates by bot slash command 🚩
  • Auto collect the good prompt from the slack channel by emoji reactions
• Cost saving
  • by caching the web page llama index
    • Consider to use sqlite-vss to store and search the text embeddings
    • Use chromadb to store and search the text embeddings
  • Consider to use sentence-transformers or txtai to generate embeddings (vectors)
• Bot can read historical messages from the same thread, thus providing context to chatGPT
• Index fine-tune
  • Use the GPTListIndex to summarize multiple URLs
  • Use the GPTTreeIndex with summarize mode to summarize a single web page
• Bot regularly summarizes news in the slack channel 🚩
  • Refer to this approach
• Support file reading and analysis
  • PDF
  • epub
  • image
    • may use GPT4
  • docx
  • txt
• Support voice reading with self-hosting whisper
  • (whisper -> chatGPT -> azure text2speech) to play language speaking practices 💥
• Integrated with Azure OpenAI Service
• Support discord bot ❓
• Rewrite the code in Typescript ❓
• Upgrade chat model (gpt-3.5-turbo) to GPT4 (gpt-4-0314) 💥
• Documentation
• Publish bot to make it can be used in other workspaces
  • Slack marketplace