支持流式审核

HuihuangZhang · Mar 27, 2023 · 91643b5 · 91643b5
1 parent 1069c49
commit 91643b5
Show file tree

Hide file tree

Showing 6 changed files with 301 additions and 39 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1 +1,2 @@
-node_modules
+node_modules
+RoboFile.php
diff --git a/README.CN.md b/README.CN.md
@@ -1,5 +1,11 @@
 > ⚠️ 这是代理的服务器端，不是客户端。需要部署到可以联通 openai api 的网络环境后访问。
 
+## 特色功能
+
+1. 支持SSE流式输出
+1. 内置文本安全审核（需要配置腾讯云KEY）
+1. 💪 SSE流式输出支持文本安全审核，就是这么强悍
+
 ## NodeJS部署
 
 你可以把 ./app.js 部署到所有支持 nodejs 14+ 的环境，比如云函数和边缘计算平台。
@@ -20,12 +26,17 @@ Proxy地址为 http://${IP}:9000
 
 1. PORT: 服务端口
 1. PROXY_KEY: 代理访问KEY，用于限制访问
-1. TIMEOUT：请求超时时间，默认5秒
+1. TIMEOUT：请求超时时间，默认30秒
+1. TENCENT_CLOUD_SID：腾讯云secret_id
+1. TENCENT_CLOUD_SKEY：腾讯云secret_key
+1. TENCENT_CLOUD_AP：腾讯云区域（如：ap-singapore 新加坡）
 
 ## 接口使用方法
 
 1. 将原来项目中 openai 的请求地址（ 比如 https://api.openai.com ）中的域名变更为本 proxy 的域名/IP（注意带上端口号）
 1. 如果设置了PROXY_KEY，在 openai 的 key 后加上 `:<PROXY_KEY>`，如果没有设置，则不需修改
+1. moderation：true 开启审核，false 关闭审核
+1. moderation_level：high 中断所有审核结果不为 Pass 的句子，low 只中断审核结果为 Block 的句子
 
 ## 说明 
 

diff --git a/README.md b/README.md
@@ -3,15 +3,23 @@
 可以部署到docker和云函数的OpenAI API代理
 Simple proxy for OpenAi api via a one-line docker command
 
-🎉 已经支持SSE，可以实时返回内容
+🎉 已经支持SSE，可以实时返回内容 💪 支持流式内容文本安全
 
 - [腾讯云函数部署教程](FUNC.md)
 - [简体中文使用说明](README.CN.md)
 - [《如何快速开发一个OpenAI/GPT应用：国内开发者笔记》](https://github.com/easychen/openai-gpt-dev-notes-for-cn-developer)
 
 以下英文由GPT翻译。The following English was translated by GPT.
 
- ## NodeJS Deployment
+  ⚠️ This is the server-side of the proxy, not the client-side. It needs to be deployed to a network environment that can access the openai api.
+
+## Features
+
+1. Supports SSE streaming output
+2. Built-in text moderation (requires Tencent Cloud KEY configuration)
+3. 💪 SSE streaming output supports text moderation, that's how powerful it is.
+
+## NodeJS Deployment
 
 You can deploy ./app.js to any environment that supports nodejs 14+, such as cloud functions and edge computing platforms.
 
@@ -25,37 +33,41 @@ You can deploy ./app.js to any environment that supports nodejs 14+, such as clo
 docker run -p 9000:9000 easychen/ai.level06.com:latest
 ```
 
-Proxy address is http://${IP}:9000
+The proxy address is http://${IP}:9000
 
 ### Available Environment Variables
 
 1. PORT: Service port
 2. PROXY_KEY: Proxy access key, used to restrict access
-3. TIMEOUT: Request timeout, default 5 seconds
+3. TIMEOUT: Request timeout, default 30 seconds
+4. TENCENT_CLOUD_SID: Tencent Cloud secret_id
+5. TENCENT_CLOUD_SKEY: Tencent Cloud secret_key
+6. TENCENT_CLOUD_AP: Tencent Cloud region (e.g. ap-singapore Singapore)
 
-## Interface Usage
+## API Usage
 
-1. Change the domain name/IP (with port number) of the original openai request address (such as https://api.openai.com) to the domain name/IP of this proxy
+1. Change the domain/IP (with port number) of the openai request address in the original project (e.g. https://api.openai.com) to the domain/IP of this proxy.
 2. If PROXY_KEY is set, add `:<PROXY_KEY>` after the openai key. If not set, no modification is required.
+3. moderation: true enables moderation, false disables moderation
+4. moderation_level: high interrupts all sentences whose moderation result is not Pass, low only interrupts sentences whose moderation result is Block.
 
 ## Notes
 
-1. Only GET and POST method interfaces are supported, and file-related interfaces are not supported.
-2. ~~SSE is not currently supported, so the stream-related options need to be turned off~~ Supported now.
+1. Only supports GET and POST methods, not file-related interfaces.
+2. ~~SSE is not currently supported, so stream-related options need to be turned off~~ Now supported.
 
-## Client Usage Example
+## Client-side Usage Example
 
-Taking `https://www.npmjs.com/package/chatgpt` as an example:
+Using `https://www.npmjs.com/package/chatgpt` as an example:
 
 ```js
 chatApi= new gpt.ChatGPTAPI({
     apiKey: 'sk.....:<proxy_key_here>',
-    apiBaseUrl: "http://localhost:9001/v1", // Replace with proxy domain name/IP
+    apiBaseUrl: "http://localhost:9001/v1", // Replace with proxy domain/IP
 });
 
 ```
 
-## Acknowledgments
+## Acknowledgements
 
 1. SSE reference to [chatgpt-api project related code](https://github.com/transitive-bullshit/chatgpt-api/blob/main/src/fetch-sse.ts)
-
diff --git a/app.js b/app.js
@@ -1,15 +1,34 @@
 const express = require('express')
-const path = require('path')
 const fetch = require('cross-fetch')
 const app = express()
 var multer = require('multer');
 var forms = multer({limits: { fieldSize: 10*1024*1024 }});
 app.use(forms.array()); 
+const cors = require('cors');
+app.use(cors());
 
 const bodyParser = require('body-parser')
 app.use(bodyParser.json({limit : '50mb' }));  
 app.use(bodyParser.urlencoded({ extended: true }));
 
+const tencentcloud = require("tencentcloud-sdk-nodejs");
+const TmsClient = tencentcloud.tms.v20201229.Client;
+const clientConfig = {
+  credential: {
+    secretId: process.env.TENCENT_CLOUD_SID,
+    secretKey: process.env.TENCENT_CLOUD_SKEY,
+  },
+  region: process.env.TENCENT_CLOUD_AP||"ap-singapore",
+  profile: {
+    httpProfile: {
+      endpoint: "tms.tencentcloudapi.com",
+    },
+  },
+};
+const mdClient = process.env.TENCENT_CLOUD_SID && process.env.TENCENT_CLOUD_SKEY ? new TmsClient(clientConfig) : false;
+
+const controller = new AbortController();
+
 app.all(`*`, async (req, res) => {
   const url = `https://api.openai.com${req.url}`;
   // 从 header 中取得 Authorization': 'Bearer 后的 token
@@ -23,33 +42,145 @@ app.all(`*`, async (req, res) => {
   if( process.env.PROXY_KEY && proxy_key !== process.env.PROXY_KEY ) 
     return res.status(403).send('Forbidden');
 
-    //console.log( req );
+  // console.log( req );
+  const { moderation, moderation_level, ...restBody } = req.body;
+  let sentence = "";
+  // 建立一个句子缓冲区
+  let sentence_buffer = [];
+  let processing = false;
+  let processing_stop = false;
+
+  async function process_buffer(res)
+  {
+    if( processing_stop )
+    {
+      console.log("processing_stop",processing_stop);
+      return false;
+    }
+
+    console.log("句子缓冲区" + new Date(), sentence_buffer);
+
+    // 处理句子缓冲区
+    if( processing )
+    {
+      // 有正在处理的，1秒钟后重试
+      console.log("有正在处理的，1秒钟后重试");
+      setTimeout( () => process_buffer(res), 1000 );
+      return false;
+    }
+
+    processing = true;
+    const sentence = sentence_buffer.shift();
+    console.log("取出句子", sentence);
+    if( sentence )
+    {
+      if( sentence === '[DONE]' )
+      {
+        console.log("[DONE]", "结束输出");
+        res.write("data: "+sentence+"\n\n" );
+        processing = false;
+        res.end();
+        return true;
+      }else
+      {
+        // 开始对句子进行审核
+        let data_array = JSON.parse(sentence);
+        console.log("解析句子数据为array",data_array);
+
+        const sentence_content = data_array.choices[0]?.delta?.content;
+        console.log("sentence_content", sentence_content);
+        if( sentence_content )
+        {
+          const params = {"Content": Buffer.from(sentence_content).toString('base64')};  
+          const md_result = await mdClient.TextModeration(params);
+          // console.log("审核结果", md_result);
+          let md_check = moderation_level == 'high' ? md_result.Suggestion != 'Pass' : md_result.Suggestion == 'Block';
+          if( md_check )
+          {
+            // 终止输出
+            console.log("审核不通过", sentence_content, md_result);
+            let forbidden_array = data_array;
+            forbidden_array.choices[0].delta.content = "这个话题不适合讨论，换个话题吧。";
+            res.write("data: "+JSON.stringify(forbidden_array)+"\n\n" );
+            res.write("data: [DONE]\n\n" );
+            res.end();
+            controller.abort();
+            processing = false;
+            processing_stop = true;
+            return false;
+          }else
+          {
+            console.log("审核通过", sentence_content);
+            res.write("data: "+sentence+"\n\n" );
+            processing = false;
+            console.log("processing",processing);
+            return true;
+          }
+        }
+
+      }
+    }else
+    {
+      // console.log("句子缓冲区为空");
+    }
+
+    processing = false;
+  }
 
 
   const options = {
       method: req.method,
       timeout: process.env.TIMEOUT||30000,
+      signal: controller.signal,
       headers: {
         'Content-Type': 'application/json; charset=utf-8',
         'Authorization': 'Bearer '+ openai_key,
       },
-      onMessage: (data) => {
+      onMessage: async (data) => {
         // console.log(data);
-        res.write("data: "+data+"\n\n" );
         if( data === '[DONE]' )
         {
-          res.end();
-        } 
+          sentence_buffer.push(data);
+          await process_buffer(res);  
+        }else
+        {
+          if( moderation && mdClient )
+          {
+            try {
+              let data_array = JSON.parse(data);
+              const char = data_array.choices[0]?.delta?.content;
+              if( char ) sentence += char;
+              // console.log("sentence",sentence );
+              if( char == '。' || char == '？' || char == '！' || char == "\n" )
+              {
+                // 将 sentence 送审
+                console.log("遇到句号，将句子放入缓冲区", sentence);
+                data_array.choices[0].delta.content = sentence;
+                sentence = "";
+                sentence_buffer.push(JSON.stringify(data_array));
+                await process_buffer(res);
+              }
+            } catch (error) {
+              // 因为开头已经处理的了 [DONE] 的情况，这里应该不会出现无法解析json的情况 
+              console.log( "error", error );
+            }   
+          }else
+          {
+            // 如果没有文本审核参数或者设置，直接输出
+            res.write("data: "+data+"\n\n" );  
+          }
+        }
       }
   };
 
-  if( req.method.toLocaleLowerCase() === 'post' && req.body ) options.body = JSON.stringify(req.body);
+  if( req.method.toLocaleLowerCase() === 'post' && req.body ) options.body = JSON.stringify(restBody);
   // console.log({url, options});
 
   try {
 
     // 如果是 chat completion 和 text completion，使用 SSE
     if( (req.url.startsWith('/v1/completions') || req.url.startsWith('/v1/chat/completions')) && req.body.stream ) {
+      console.log("使用 SSE");
       const response = await myFetch(url, options);
       if( response.ok )
       {
@@ -61,6 +192,7 @@ app.all(`*`, async (req, res) => {
         });
         const  { createParser } = await import("eventsource-parser");
         const parser = createParser((event) => {
+          // console.log(event);    
           if (event.type === "event") {
             options.onMessage(event.data);
           }
@@ -73,6 +205,7 @@ app.all(`*`, async (req, res) => {
           body.on("readable", () => {
             let chunk;
             while (null !== (chunk = body.read())) {
+              // console.log(chunk.toString());
               parser.feed(chunk.toString());
             }
           });
@@ -86,10 +219,28 @@ app.all(`*`, async (req, res) => {
 
     }else
     {
+      console.log("使用 fetch");
       const response = await myFetch(url, options);
-      console.log(response);
+      // console.log(response);
       const data = await response.json();
-      console.log( data );
+      // 审核结果
+      if( moderation && mdClient )
+      {
+        const params = {"Content": Buffer.from(data.choices[0].message.content).toString('base64')};  
+        const md_result = await mdClient.TextModeration(params);
+        // console.log("审核结果", md_result);
+        let md_check = moderation_level == 'high' ? md_result.Suggestion != 'Pass' : md_result.Suggestion == 'Block';
+        if( md_check )
+        {
+          // 终止输出
+          console.log("审核不通过", data.choices[0].message.content, md_result);
+          data.choices[0].message.content = "这个话题不适合讨论，换个话题吧。";
+        }else
+        {
+          console.log("审核通过", data.choices[0].message.content);
+        }
+      }
+
       res.json(data);
     }
 
@@ -124,11 +275,6 @@ async function myFetch(url, options) {
   return res;
 }
 
-
-
-
-
-
 // Error handler
 app.use(function(err, req, res, next) {
   console.error(err)
@@ -138,4 +284,4 @@ app.use(function(err, req, res, next) {
 const port = process.env.PORT||9000;
 app.listen(port, () => {
   console.log(`Server start on http://localhost:${port}`);
-})
+})
diff --git a/package.json b/package.json
@@ -1,9 +1,11 @@
 {
   "dependencies": {
     "body-parser": "^1.20.2",
+    "cors": "^2.8.5",
     "cross-fetch": "^3.1.5",
     "eventsource-parser": "^0.1.0",
     "express": "^4.18.2",
-    "multer": "^1.4.5-lts.1"
+    "multer": "^1.4.5-lts.1",
+    "tencentcloud-sdk-nodejs": "^4.0.567"
   }
 }