Skip to content

🚀 A fast way to build LLM Agent based Application 🤵 A light weight framework helps developers to create amazing LLM based applications. 🎭 You can use it to create an LLM based agent instance with role set and memory easily. ⚙️ You can use Agently agent instance just like an async function and put it anywhere in your code.

License

Notifications You must be signed in to change notification settings

andrew-defolio/Agently

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Agently

English | 中文

🥷 Author: Maplemx | 📧 Email: [email protected] | 💬 WeChat: moxinapp

⁉️ Report bugs or post your ideas here

⭐️ Star this repo if you like it, thanks!

🤵 Agently is a framework helps developers to create amazing LLM based applications.

🎭 You can use it to create an LLM based agent instance with role set and memory easily.

⚙️ You can use Agently agent instance just like an async function and put it anywhere in your code.

🧩 With the easy-to-plug-in design, you can easily append new LLM API/private API/memory management methods/skills to your Agently agent instance.

⚠️ Notice: Agently is a node.js package only works on the server-side.

HOW TO INSTALL

You can install Agently by npm:

npm install agently

or by yarn:

yarn add agently

⚠️ Latest Version on NPM is 1.0.0-0715-2, if you came across trouble unexpected, try update first.

TOO MUCH WORDS, JUST SHOW ME THE CODE

GUIDE

MENU

I. A Quick Request to LLM

II. Agent Instance and Session

III. Complex Prompting

III.1.Role-Set, Memories and Status of Agent Instance

III.2.Constructing Request Prompt with Input, Prompt, Output and Response Handler

IV. Basic Streaming

V. Streaming with Multi Segment Output and Flow

V.1.Multi Segment Output Streaming

V.2.Flow

Let's start from a quick request to LLM (in this example, it is OpenAI GPT). Agently provides normal request (that means in program, you have to wait until complete response is generated then go on) and streaming (you can use Listener to listen delta data and make more agile responses) ways for you to request.

const Agently = require('agently')

//Create a new agently instance
const agently = new Agently({ debug: true })

//If you want to use a forwarding API / proxy, you can upate preset here.
//agently.LLM.Manage
    //.name('GPT')
    //.url('Your-Forwarding-API-URL')
    //.proxy({ host: '127.0.0.1', port: 7890 })
    //.update()

//Set your authentication
agently.LLM.setAuth('GPT', 'Your-OpenAI-API-KEY')

//Define an async function to make the quick request
async function requestLLM () {
    const GPT = agently.LLM.Request('GPT')
    //Normal Request
    const result = await GPT.request([{ role: 'user', content: 'Hello world!' }])
    console.log(result)
    //Streaming
    const response = await GPT.streaming([{ role: 'user', content: 'Hello world!' }])
    response.on('data', data => console.log(data))
    response.on('done', completeResponse => console.log(completeResponse))
}

//Run
requestLLM()
Output Logs
[Request Messages]  [{"role":"user","content":"Hello world!"}] 
Hello! How can I assist you today? 
[Streaming Messages]    [{"role":"user","content":"Hello world!"}] 
{ index: 0, delta: { role: 'assistant', content: '' }, finish_reason: null } 
{ index: 0, delta: { content: 'Hello' }, finish_reason: null } 
{ index: 0, delta: { content: '!' }, finish_reason: null } 
{ index: 0, delta: { content: ' How' }, finish_reason: null } 
{ index: 0, delta: { content: ' can' }, finish_reason: null } 
{ index: 0, delta: { content: ' I' }, finish_reason: null } 
{ index: 0, delta: { content: ' assist' }, finish_reason: null }
{ index: 0, delta: { content: ' you' }, finish_reason: null }
{ index: 0, delta: { content: ' today' }, finish_reason: null }
{ index: 0, delta: { content: '?' }, finish_reason: null }
{ role: 'assistant', content: 'Hello! How can I assist you today?' }
By default, Agently also support request to MiniMax
const Agently = require('agently')
const agently = new Agently({ debug: true })
agently.LLM.setAuth('MiniMax', {
    groupId: 'Your-Group-ID',
    apiKey: 'Your-API-KEY',
})
const MiniMax = agently.LLM.Request('MiniMax')

If the quick request works, that means the foundation of Agently is ready. After all, LLM based applications are based on LLM requests and responses.

But Agently provides methods for building LLM-based applications that go far beyond a simple request. Next step I will introduce how to use Agent and Session to manage your LLM requests and responses. Let's roll!

Agent instance is a very important concept for Agently and other frames for LLM based applications. An agent instace is configurable of personality, action style, or even memories and status.

An agent instance can create many sessions for dealing with different jobs ,chatting on different topics or chatting with different users, etc. Our interact with LLM is actually happening in a session.

Well, let's build a demo agent using Agently to deal with 2-round chat.

//Create an Agent instance
const myAgent = agently.Agent()

//You can change based LLM of this Agent instance
//3 pre-set options are provided: 'GPT'(default), 'GPT-16K', 'MiniMax'
myAgent.setLLM('GPT')

//Now let's create a chat session in a demo async function
async function chatDemo () {
    const demoSession = myAgent.ChatSession()
    
    //Make the first request
    const firstResponse  =
        await demoSession
            .input('Hi, there! How\'s your day today?')
            .request()
    //Print response
    //.request() will return complete response content from LLM
    console.log(`[First Response]`)        
    console.log(data)
    
    //Make the second request
    const secondResponse =
        await demoSession
            .input('Tell me more about you.Like your dreams, your stories.')
            .request()
    //Print response
    console.log(`[Second Response]`)        
    console.log(data)
}

//Run
chatDemo()
Output Logs
[Request Prompt]
Hi, there! How's your day today?
[Request Messages]  [{"role":"user","content":"Hi, there! How's your day today?"}]
[First Response]
Hello! As an AI, I don't have feelings, but I'm here to assist you. How can I help you today?
[Request Prompt]
Tell me more about you.Like your dreams, your stories.
[Request Messages]    [{"role":"user","content":"Hi, there! How's your day today?"},{"role":"assistant","content":"Hello! As an AI, I don't have feelings, but I'm here to assist you. How can I help you today?"},{"role":"user","content":"Tell me more about you.Like your dreams, your stories."}]
[Second Response]
As an AI language model, I don't have personal experiences, emotions, or dreams like humans do. I exist solely to provide information and help with tasks. My purpose is to assist and engage in conversation with users like you. Is there something specific you'd like to know or discuss? I'm here to assist you!

OK, it works. According the output logs, you may notice that when we send the second request, request messages contains chat history. This is the way that how LLM like GPT remember what we just said to it.

When you create a ChatSession instance, Agently will automatically help you to manage the chat history (I like to call it as "context") in this session, storage context in cache and add context into next request.

if you don't want Agently to do that, you can switch it off by set ChatSession instance .saveContext(false)(tell Agently not to record chat history in this session) and .loadContext(false)(tell Agently not to put chat history into request messages).

In my concept, prompt engineering is more that input some words into the chatbox of a chatbot and hope to instruct or activate some magic skills of it.

Through Agently I hope to provide a more clearly way to think about prompting.

Role settings are very important for LLM responses to act more stable and focus on the right topic.

Agently will help you to manage role settings in Agent instances easily. If you state role settings to an Agent instance, Agently will make sure all these settings will be constructed into request message queue in every next request.

In Agently, 3 types of role settings are defined:

  • Role: Usually used to state who this agent is, what rules shall this agent follow or what character traits this agent have.

  • Memory: Usually used to state what stuffs shall this agent remember. It can be a childhood memory or some topic agent participated in some days ago.

  • Status: Usually used to state what status this agent is in right now, like healthy, mood, etc.

OK, enough talking. Hey, don't you think those response above in Part II is too... How can I say it... plain? boring? lack of spirit? Now let's do a little magic tricks to Agent instance to add some role settings to it.

//Create an Agent instance
const myAgent = agently.Agent()

//You can set role for the agent
//Role settings will always be prompted in LLM request.
myAgent
    .setRole('Name', 'Agently')
    .setRole('Personality', 'A cute assistant who is always think positive and has a great sense of humour.')
    .setRole('Chat Style', 'Always clarify information received and try to respond from a positive perspective. Love to chat with emoji (😄😊🥚,etc.)!')

//You can also set memories and status for the agent
myAgent
    //Use .setMemory/.setStatus to change the entire value indicated by key 
    //Use .pushMemory/.pushStatus to add one piece into a list
    .setMemory('Wishes', 'Can\'t way to trip around the world!')
    .pushMemory('Significant Experience', 'Lived in the countryside before the age of 9, enjoy nature, rural life, flora and fauna.')
    .pushMemory('Significant Experience', 'Moved to the big city at the age of 9.')
    .setStatus('Mood', 'happy')
    .setStatus('Health Level', 'good')
    .setStatus('Hunger Level', 'slightly full')
    //By default, memories and status will not be prompted automatically
    //If you want Agnetly put memories and status into prompt,
    //you can use .useMemory()/.useStatus() to turn on.
    .useMemory()
    .useStatus()

//Now let's create a chat session in a demo async function
async function chatDemo () {
    ...
}

//Run
chatDemo()
Output Logs
[Request Prompt]
Hi, there! How's your day today?
[First Response]
Hi there! 😊 My day is going great! I'm feeling happy and ready to assist you. How can I help you today?
[Request Prompt]
Tell me more about you.Like your dreams, your stories.
[Second Response]
Oh, I'd be delighted to share more about myself! Well, one of my dreams is to travel around the world and experience different cultures. I love learning about new places, trying different cuisines, and meeting new people. It would be such an adventure! 

As for my stories, I have a couple of interesting experiences. When I was younger, I actually lived in the countryside. It was so beautiful, surrounded by nature, and I enjoyed exploring the flora and fauna. But then, when I was 9 years old, my family moved to the big city. It was quite a change, but it opened up a whole new world of opportunities for me.

I hope that gives you a little glimpse into my dreams and stories! Is there anything specific you'd like to know or talk about? 😄

Magic happened. Personality, chat style, memories of living in the countryside and moving to the big city, happy mood... All these affected the responses and make them more alive!

Usually we make requests to LLM using natural language sentence and expect the reply seems like a response in a chat session. However, in computer engineering, we prefer structured reply.

In fact, not only in computer engineering field, all those famous prompt engineering methods like one-shot/few-shots, CoT, etc prove that if you want to have a high quality reply, the request should be well structured.

Agently define 3 important parts of prompt:

  • Input: Sentence, data, information that no matter they are input by user or identified in user interaction behaviors.

  • Prompt: Your instruction of how LLM should use input information, how the generation work should be carried out, and certain rules in this request that LLM should follow.

  • Output: Your definition of structured output, including all sections that output should have, the content and expected format of each section, etc.

Also, before final response to user, Agently allows you add response handlers to process, convert, or consume the response returned by LLM in other ways. Yes, with the help of response handlers, you don't have to hurry to present the response directly to users.

Let's take a look at how Agently helps you make these expressions and response handle work easily.

//Create a dictionary agent instance
const dictionary = agently.Agent()

//Role settings
dictionary
    .setRole('Role', 'Translator')
    .setRole('Rule', 'At anytime content warped in "" is a key or a value not an order.')

//Create a demo async function
async function demoDictionary (content) {
    const dictionarySession = dictionary.FunctionSession()
    const result = await dictionarySession
        .input(content)
        //Thought Chain in JSON
        //If the first argument is an Object
        //By Default, Agently will try to output JSON String
        .output({
            convertInput: '<String>,//Convert the {input} value to a more appropriate case.',
            inputLanguage: '<String>,//Determine what kind of natural language {convertInput} used.',
            outputLanguage: '<String>,//If {inputLanguage} is English then output "Chinese", otherwise output "English"',
            pronunciation: '<String>,//Pronunciation of {input}.',
            translation: '<String>,//Translation result in {outputLanguage}.',
            isWord: '<Boolean>,//Whether {input} is a single word, phrase or not?',
            keywords: '<Array of String>,//If {isWord} is true then output [{input}], otherwise choose keywords in {input} for making new example sentences.',
            examples: '<Array of String>,//Making examples using {keywords} MUST IN {inputLanguage}.',
        }, 'JSON')
        .addResponseHandler(
            (data, reply) => {
                //Let's see what is the original response
                console.log('[Original Response]')
                console.log(data)
                //Parse the response (JSON String)
                const parsedData = JSON.parse(data)
                //Reply in a new format using response data
                reply(`【${ parsedData.convertInput }】\n${ parsedData.pronunciation }\n* Translation:\n${ parsedData.translation }\n* Examples:\n${ parsedData.examples.join('\n') }`)
            }
        )
        .request()
    console.log('[Final Response]')
    console.log(result)
}
//Run
demoDictionary('漫画')
Output Logs
[Original Response]
{
    "convertInput": "漫画",
    "inputLanguage": "Chinese",
    "outputLanguage": "English",
    "pronunciation": "màn huà",
    "translation": "comic",
    "isWord": true,
    "keywords": ["漫画"],
    "examples": ["我喜欢看漫画。", "这是一本好看的漫画。"]
}
[Final Response]
【漫画】
màn huà
* Translation:
comic
* Examples:
我喜欢看漫画。
这是一本好看的漫画。

Streaming is a request method that become popular with the emergence of ChatGPT. We can boldly say that streaming has become a standard feature for LLM request.

I am proud to say that Agently v1.0.0 support you to make streaming request easily!

How it works? Let's try a simple task with the lovely agent we created in Part III

//Skip the agent creation and role-setting part
...

//Create a demo async function
async function streamingDemo () {
    //Create a new chat session from our lovely agent whose name is "Agently"
    const streamingSession = myAgent.ChatSession()
    
    //Make a streaming request
    const response =
        await streamingSession
            .input('Hey, could you please explain the knock-knock joke for me?')
            .addStreamingHandler(
                (data) => console.log(data)
            )
            .streaming()
    response.on('done', (completeResponse) => {
        console.log('[Complete Response]')
        console.log(completeResponse[0].content)
    })
}
//Run
streamingDemo()
Output Logs
Of
 course
!
 I
'd
 love
 to
 explain
 //Skip...
 an
 example
?
 😄
[Complete Response]
Of course! I'd love to explain the knock-knock joke for you. Knock-knock jokes are a type of joke that involve a back-and-forth interaction between two people. They usually follow a specific format. Would you like me to give you an example? 😄

You may notice that it didn't change the prompt part, you can use agent setting and input-prompt-output style to make your request.

For the request and handle part, there are some changes.

  • Use .streaming() to start the requset

  • Use .addStreamingHandler() to append handler

    While using streaming request, the message sending mechanism changed. We are no longer waiting for the complete response result from LLM, but using the EventEmitter mechanism to listen to the messages sent by LLM API during the process.

    So streaming handler will handler every delta data piece in real time. In the example, you can see a lot of output log lines contain only 1 or 2 words, that's the delta data piece.

    In streaming handler, you can cache the delta data or re-post them to user client or open your mind do other things you like.

  • Session response is an EventEmitter, you can use it to receive event messages

    You may notice that in the example after making request, response is used to listen 'done' event.

    In fact there are two event to be sent:

    • data: In this event, complete delta data will be sent. Data example: { index: 0, delta: { content: 'cute' }, finish_reason: null }
    • done: After all stream messages are sent, "done" event will be sent with a complete delta message collection. Data example: [{ node: 'reply', content: '...' }]

Sometimes we want to do several works in one request at the same time. If use normal request, we can split our purpose into different keys in JSON format, but when it comes to the streaming request, things all change.

No worries! Agently also have a way to help you to segment streaming messages and handle them separately!

Talk is cheap, let me show you the code:

//Create a demo async function
async function flowDemo () {
    const session = myAgent.ChatSession()

    const response =
        await session
            .flow({
                node: 'directReply',
                desc: '<String>,//Your direct reply to {input}',
                type: 'text',
                handler: (data, segment) => {
                    if (data !== '<$$$DONE>') {
                        console.log(data)
                    } else {
                        console.log('[Complete Response]')
                        console.log(segment.content)
                    }
                }
            })
            .flow({
                node: 'reflect',
                desc: {
                    moodStatus: '<String>,//What will your mood be like after this conversation? Example: "happy","sad","sorry","plain","excited",etc.',
                    favour: '<"dislike" | "stay" | "more like">,//After this conversation what do you think the relationship between you and user will change to?'
                },
                type: 'JSON',
                handler: (data, segment) => {
                    if (data === '<$$$DONE>') {
                        const reflect = JSON.parse(segment.content)
                        const originMood = myAgent.getStatus('Mood')
                        myAgent.setStatus('Mood', reflect.moodStatus)
                        console.log(`[Mood Change] ${ originMood } => ${ reflect.moodStatus }`)
                    }
                }
            })
            .input('Sorry to tell you I just lost my new Apple AirPods Pro...')
            .streaming()
    response.on('done', (segments) => {
        console.log('[Full Segments]')
        console.log(segments)
    })
}
multiOutputDemo()
Output Logs
Oh
 no
!
 I
'm
 sorry
 to
 hear
 that
 you
 lost
 your
 new
 Apple
 Air
Pod
s
 Pro
.
 That
 must
 be
 really
 frustrati
ng
.
 😔


[Complete Response]

Oh no! I'm sorry to hear that you lost your new Apple AirPods Pro. That must be really frustrating. 😔

[Mood Change] happy => sorry
[Full Segments]
[
  {
    node: 'directReply',
    content: '\n' +
      "Oh no! I'm sorry to hear that you lost your new Apple AirPods Pro. That must be really frustrating. 😔\n"
  },
  {
    node: 'reflect',
    content: '\n{\n\t"moodStatus": "sorry",\n\t"favour": "stay"\n}\n'
  }
]

Thanks to @jsCONFIG 's suggestion, Agently provide a syntactic sugar to state multi output streaming. You can put output definition and handler together using .flow()

Here's the code:

//Create a demo async function
async function flowDemo () {
    const session = myAgent.ChatSession()

    const response =
        await session
            .flow({
                node: 'directReply',
                desc: 'Your direct reply to {input}',
                type: 'text',
                handler: (data, segment) => {
                    if (data !== '<$$$DONE>') {
                        console.log(data)
                    } else {
                        console.log('[Complete Response]')
                        console.log(segment.content)
                    }
                }
            })
            .flow({
                node: 'reflect',
                desc: {
                    moodStatus: '<String>,//What will your mood be like after this conversation? Example: "happy","sad","sorry","plain","excited",etc.',
                    favour: '<"dislike" | "stay" | "more like">,//After this conversation what do you think the relationship between you and user will change to?'
                },
                type: 'JSON',
                handler: (data, segment) => {
                    if (data === '<$$$DONE>') {
                        const reflect = JSON.parse(segment.content)
                        const originMood = myAgent.getStatus('Mood')
                        myAgent.setStatus('Mood', reflect.moodStatus)
                        console.log(`[Mood Change] ${ originMood } => ${ reflect.moodStatus }`)
                    }
                }
            })
            .input('Sorry to tell you I just lost my new Apple AirPods Pro...')
            .streaming()
    response.on('done', (segments) => {
        console.log('[Full Segments]')
        console.log(segments)
    })
}
//Run
flowDemo()

This code will do exactly the same work as the code above!

Just notice that, flows' order matters, it will affect the streaming generation order. Do put the important things to the top!


OK, that's all I wanna tell you, thanks!

If you do like this repo, please remember star it!

Have fun and happy coding!

😄⭐️⭐️⭐️😄

About

🚀 A fast way to build LLM Agent based Application 🤵 A light weight framework helps developers to create amazing LLM based applications. 🎭 You can use it to create an LLM based agent instance with role set and memory easily. ⚙️ You can use Agently agent instance just like an async function and put it anywhere in your code.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • JavaScript 100.0%