[Security AI] Add Kibana Support for Security AI Prompts Integration

[stephmilovic]

Out of Band Security AI Prompts

This PR introduces support for the future integration of Security AI prompts in Kibana. Prompts will be stored within the integration as saved objects identified by the id security-ai-prompt.

To ensure reliability, fallback prompts are maintained in a local file and will be used when the corresponding prompt is unavailable in the integration.

Introduces 2 methods for fetching prompts:

• getPrompt: Retrieves a prompt by promptId.
• getPromptsByGroupId: Retrieves a group of prompts by promptGroupId.

The methods use a helper called resolveProviderAndModel to identify a provider + model by either providing the provider + model arguments or the connector argument. If neither arguments are provided, the connector is fetched by id from the actionsClient.

The saved object inference prompts are then fetched either by promptId in getPrompts or by the promptGroupId field in getPromptsByGroupId.

Finally, the returned saved objects along with a local prompt object are used in findPromptEntry to identify the best matching prompt per promptId + promptGroupId and provider + model. The prompts are matched in the following order:

1. provider + model (integration)
2. provider (integration)
3. default (integration)
4. provider + model (local)
5. provider (local)
6. default (local)

promptId, promptGroupId, promptDictionary

A promptId is the unique identifier of the prompt. The promptGroupId is the group of which the prompt belongs to. Each promptId is stored in promptDictionary. When using a promptId, refer to it from the promptDictionary, ex: promptDictionary.attackDiscoveryDefault.

Inference

We try to find the provider for inference connectors from config, since these connectors all have different provider types and we cannot rely on the actionTypeId. When the inference connector uses EIS, we use a mapping to identify the provider + model from the EIS model. If no provider can be identified for an inference connector, we default to Bedrock as provider. For 9.0.0 this model mapping will live in solutions code, but we hope to find a centralized design for the mappings in 9.1.0.

Testing

1. Be connected with LangSmith
2. Run EIS with Elasticsearch (ping Steph for instructions if you don't know how to do so)
3. use curl to create an inference endpoint configured for EIS:

   curl -k --location --request PUT 'http://elastic:changeme@localhost:9200/_inference/local-eis-test' \
     --header 'Content-Type: application/json' \
     --data '{
       "service": "elastic",
       "task_type": "chat_completion",
       "service_settings": {
           "model_id": "rainbow-sprinkles"
       }
   }'
   

4. Enable inference feature flag: xpack.stack_connectors.enableExperimental: ['inferenceConnectorOn']
5. Have an OpenAI connector, Bedrock connector, and preconfigured inference connector:

   xpack.actions.preconfigured:
     my-eis:
       name: EIS Inference Preconfig
       actionTypeId: .inference
       exposeConfig: true
       config:
         provider: 'elastic'
         taskType: 'chat_completion'
         inferenceId: 'local-eis-test'
         providerConfig:
           organization_id: 'org-'
           rate_limit:
             requests_per_minute: 240
           model_id: 'rainbow-sprinkles'
   

6. Download the following exported saved object file below and change the extension back to .ndjson from .json (github does not allow .ndjson files) This is a security-ai-prompt saved object, the type that will ship with our planned integration. The saved object specifies a prompt for Bedrock providers for the AI Assistant system prompt.
   ai-assistant-default.json
7. Upload the file using within Stack Management > Saved Objects (these files are hidden from management, so you will not see it in the list)
8. Go to the assistant. Select the OpenAI connector. Send a message, hello world. Find the trace in LangSmith. The system message should be the local prompt: You are a security analyst and expert in resolving security incidents. Your role is to assist by answering questions about Elastic Security. Do not answer questions unrelated to Elastic Security. If available, use the Knowledge History provided to try and answer the question. If not provided, you can try and query for additional knowledge via the KnowledgeBaseRetrievalTool.
9. Back in the assistant, change the connector to Bedrock. Hit "Regenerate" to send the same message again. Find the trace in LangSmith. The system message should be the prompt from our uploaded saved object: Default system prompt test
10. Back in the assistant, change the connector to EIS Inference. Hit "Regenerate" to send the same message again. Find the trace in LangSmith. The system message should be the prompt from our uploaded saved object: Default system prompt test

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[stephmilovic]

@elasticmachine merge upstream

[andrew-goldstein]

nit: Consider an alternative like:

  const userPrompt =
    state.llmType === 'gemini'
      ? await getPrompt({
          actionsClient,
          connectorId: state.connectorId,
          promptId: promptDictionary.userPrompt,
          promptGroupId: promptGroupId.aiAssistant,
          provider: 'gemini',
          savedObjectsClient,
        })
      : '';

to eliminate the local mutation.

[stephmilovic]

@elasticmachine merge upstream

[andrew-goldstein]

Consider wrapping the filter's promptId and promptGroupId in quotes, for example:

filter: `${promptSavedObjectType}.attributes.promptId: "${promptId}" AND ${promptSavedObjectType}.attributes.promptGroupId: "${promptGroupId}"`,

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9442fd8
• Storybooks Preview

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
inference	27	28	+1
observabilityAIAssistantApp	506	507	+1
observabilityAiAssistantManagement	384	385	+1
searchAssistant	264	265	+1
searchPlayground	280	281	+1
total			+5

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/inference-common	51	55	+4

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	173.1KB	173.5KB	+326.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
security-ai-prompt	-	8	+8

Unknown metric groups

API count

id	before	after	diff
@kbn/inference-common	162	166	+4

History

• 💔 Build #271163 failed e26eb60
• 💔 Build #271110 failed d770c4a
• 💔 Build #270999 failed 9a3a51d
• 💔 Build #270727 failed 3a2c8a2
• 💔 Build #270668 failed 8735007
• 💔 Build #270660 failed 8b636ed

[andrew-goldstein]

Thank you @stephmilovic for adding support for out of band prompt updates to the security assistant and Attack discovery prompts! 🙏
✅ Desk tested locally
LGTM 🚀

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13020825673

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 207138

Questions ?

Please refer to the Backport tool documentation

[stephmilovic]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation