Skip to content

Commit

Permalink
elasticsearch:opensearch: add documentation for aws_service_name opti…
Browse files Browse the repository at this point in the history 
…on (fluent#1056)

Signed-off-by: Matthew Fala <[email protected]>
  • Loading branch information
@matthewfala
matthewfala authored Mar 2, 2023
1 parent d1c0702 commit 7f940b0
Show file tree
Hide file tree
Showing 2 changed files with 21 additions and 1 deletion.
1 change: 1 addition & 0 deletions pipeline/outputs/elasticsearch.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ The **es** output plugin, allows to ingest your records into an [Elasticsearch](
| AWS\_STS\_Endpoint | Specify the custom sts endpoint to be used with STS API for Amazon OpenSearch Service | |
| AWS\_Role\_ARN | AWS IAM Role to assume to put records to your Amazon cluster | |
| AWS\_External\_ID | External ID for the AWS IAM Role specified with `aws_role_arn` | |
| AWS\_Service\_Name | Service name to be used in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See the [FAQ](opensearch.md#faq) section on Amazon OpenSearch Serverless for more information. | es |
| Cloud\_ID | If you are using Elastic's Elasticsearch Service you can specify the cloud\_id of the cluster running. The Cloud ID string has the format `<deployment_name>:<base64_info>`. Once decoded, the `base64_info` string has the format `<deployment_region>$<elasticsearch_hostname>$<kibana_hostname>`.
| |
| Cloud\_Auth | Specify the credentials to use to connect to Elastic's Elasticsearch Service running on Elastic Cloud | |
Expand Down
21 changes: 20 additions & 1 deletion pipeline/outputs/opensearch.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,11 @@ The following instructions assumes that you have a fully operational OpenSearch
| AWS\_STS\_Endpoint | Specify the custom sts endpoint to be used with STS API for Amazon OpenSearch Service | |
| AWS\_Role\_ARN | AWS IAM Role to assume to put records to your Amazon cluster | |
| AWS\_External\_ID | External ID for the AWS IAM Role specified with `aws_role_arn` | |
| AWS\_Service\_Name | Service name to be used in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See the [FAQ](opensearch.md#faq) section on Amazon OpenSearch Serverless for more information. | es |
| HTTP\_User | Optional username credential for access | |
| HTTP\_Passwd | Password for user defined in HTTP\_User | |
| Index | Index name, supports [Record Accessor syntax](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) from 2.0.5 onwards. | fluent-bit |
| Type | Type name | \_doc |
| Type | Type name. This option is ignored if `Suppress_Type_Name` is enabled. | \_doc |
| Logstash\_Format | Enable Logstash format compatibility. This option takes a boolean value: True/False, On/Off | Off |
| Logstash\_Prefix | When Logstash\_Format is enabled, the Index name is composed using a prefix and the date, e.g: If Logstash\_Prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated. | logstash |
| Logstash\_DateFormat | Time format \(based on [strftime](http://man7.org/linux/man-pages/man3/strftime.3.html)\) to generate the second part of the Index name. | %Y.%m.%d |
Expand Down Expand Up @@ -157,3 +158,21 @@ Without this you will see errors like:
```text
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
```

### Fluent-Bit + Amazon OpenSearch Serverless <a id="#aws-opensearch-serverless"></a>
Amazon OpenSearch Serverless is an offering that eliminates your need to manage OpenSearch clusters. All existing Fluent Bit OpenSearch output plugin options work with OpenSearch Serverless. For Fluent Bit, the only difference is that you must specify the service name as `aoss` (Amazon OpenSearch Serverless) when you enable `AWS_Auth`:
```
AWS_Auth On
AWS_Region <aws-region>
AWS_Service_Name aoss
```

**Data Access Permissions**

When sending logs to OpenSearch Serverless, your AWS IAM entity needs [OpenSearch Serverless Data Access permisions](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-data-access.html). Give your IAM entity the following data access permissions to your serverless collection:
```
aoss:CreateIndex
aoss:UpdateIndex
aoss:WriteDocument
```
With data access permissions, IAM policies are not needed to access the collection.

0 comments on commit 7f940b0

Please sign in to comment.