forked from yandex-cloud/docs
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Data Transfer: documented known limitations for MySQL source endpoints. * Translations updated. * Fixes and improvements.
- Loading branch information
DataUI VCS Robot
committed
Nov 10, 2022
1 parent
9c7ae8f
commit 596485a
Showing
273 changed files
with
4,718 additions
and
1,092 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
|
||
1. Log in to [management console]({{link-console-main }}). If you haven't registered yet, go to the management console and follow the instructions. | ||
1. [On the billing page]({{ link-console-billing }}) make sure that you have a [billing account] connected (../../billing/concepts/billing-account.md) and it is in the status `ACTIVE` or `TRIAL_ACTIVE`. If there is no billing account, [create one](../../billing/quickstart/index.md#create_billing_account). | ||
1. If you don't have a folder yet, [create it](../../resource-manager/operations/folder/create.md). | ||
1. [Create] (../../iam/operations/sa/create.md) a service account and [assign] (../../iam/operations/sa/assign-role-for-sa.md) to it the `editor` role for your directory. | ||
1. [Create](../../iam/operations/sa/create-access-key.md) a static access key. | ||
1. Configure the AWS CLI: | ||
1. Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) and run the command: | ||
|
||
```bash | ||
aws configure | ||
``` | ||
|
||
1. Enter sequentially: | ||
|
||
* `AWS Access Key ID [None]:`: service account [key ID](../../iam/concepts/authorization/access-key.md). | ||
* `AWS Secret Access Key [None]:`: service account [secret key](../../iam/concepts/authorization/access-key.md). | ||
* `Default region name [None]:`: availability zone `{{ region-id }}`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
1. This example examines the interaction of Debezium and PostgreSQL. Next, we will assume that Debezium will be installed on the server where PostgreSQL is running. Install Debezium server according to [instructions](https://debezium.io/documentation/reference/stable/operations/debezium-server.html ). | ||
1. Go to the `conf` directory and create the 'application.properties` file with the following content: | ||
``` | ||
debezium.sink.type=kinesis | ||
debezium.sink.kinesis.region={{ region-id }} | ||
debezium.sink.kinesis.endpoint=<YDS_STREAM_ENDPOINT> | ||
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector | ||
debezium.source.offset.storage.file.filename=data/offsets.dat | ||
debezium.source.offset.flush.interval.ms=0 | ||
debezium.source.database.hostname=localhost | ||
debezium.source.database.port=5432 | ||
debezium.source.database.user=<DATABASE_USER> | ||
debezium.source.database.password=<DATABASE_PASSWORD> | ||
debezium.source.database.dbname=<DATABASE_NAME> | ||
debezium.source.database.server.name=debezium | ||
debezium.source.plugin.name=pgoutput | ||
debezium.source.transforms=Reroute | ||
debezium.source.transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter | ||
debezium.source.transforms.Reroute.topic.regex=(.*) | ||
debezium.source.transforms.Reroute.topic.replacement=<YDS_STREAM_NAME> | ||
``` | ||
Where: | ||
* `<YDS_STREAM_ENDPOINT>`: full endpoint to the data stream {{yds-short-name}}, for example, `https://yds.serverless .yandexcloud.net /{{region-id }}/b1p89ae43m6he2ooql88r/etn01eg4rn184nemdbb`. The full endpoint is available in the UI {{yds-full-name}}. To view the full endpoint, click **Connect**. | ||
* `<YDS_STREAM_NAME>`: data stream name {{yds-short-name}}. | ||
* `<DATABASE_NAME>`: name of the PostgreSQL database. | ||
* `<DATABASE_USER>`: username for connecting to the PostgreSQL database. | ||
* `<DATABASE_PASSWORD>`: user password for connecting to the PostgreSQL database. | ||
1. Run Debezium with the following command: | ||
```shell | ||
JAVA_OPTS=-Daws.cborEnabled=false ./run.sh | ||
``` | ||
1. Make any changes to the PostgreSQL database, for example, insert data into a table. | ||
1. If configured correctly, the following messages will appear in the Debezium console: | ||
``` | ||
2022-02-11 07:31:12,850 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 1 records sent during previous 00:19:59.999, last recorded offset: {transaction_id=null, lsn_proc=23576408, lsn_commit=23576120, lsn=23576408, txId=580, ts_usec=1644564672582666} | ||
``` | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,260 @@ | ||
# Searching for {{ yandex-cloud }} events in {{ yq-full-name }} | ||
|
||
{{ at-name }} support is integrated in [{{ yq-full-name }}](../../query/). You can analyze events of {{ yandex-cloud }} resources by executing [analytical](../../query/concepts/batch-processing.md) and [streaming](../../query/concepts/stream-processing.md) {{ yql-short-name }} queries. | ||
|
||
You can execute analytical queries for logs stored [in a bucket](../operations/export-organization-bucket.md) and streaming queries for logs stored [in a {{ yds-full-name }} data stream](../operations/export-organization-bucket.md). | ||
|
||
![](../../_assets/audit-trails/tutorials/audit-trails-query.png) | ||
|
||
To connect a bucket with [audit logs](../concepts/events.md) to {{ yq-full-name }} and execute [{{ yql-short-name }}](../../query/yql-tutorials/index.md) queries: | ||
|
||
1. [Prepare the environment.](#prepare-environment) | ||
1. [Create a connection between a trail and {{ yq-short-name }}.](#trail-yq) | ||
1. [Execute a query to logs in {{ objstorage-name }}.](#perform-request) | ||
|
||
If you no longer need these resources, [delete them](#clear-out). | ||
|
||
|
||
## Before you begin {#before-begin} | ||
|
||
{% include [before-you-begin](../../_tutorials/_tutorials_includes/before-you-begin.md) %} | ||
|
||
{% include [cli-install](../../_includes/cli-install.md) %} | ||
|
||
|
||
|
||
## Required paid resources {#paid-resources} | ||
|
||
The cost of infrastructure support includes a fee for a bucket (see [Pricing for {{ objstorage-name }}](../../storage/pricing.md)). | ||
|
||
|
||
|
||
## Prepare the environment {#prepare-environment} | ||
|
||
|
||
### Create a bucket for audit logs {#create-backet} | ||
|
||
{% list tabs %} | ||
|
||
- Management console | ||
|
||
1. In the [management console]({{ link-console-main }}), select the folder where you wish to create a [bucket](../../storage/concepts/bucket.md), for example, `example-folder`. | ||
1. Select **{{ objstorage-name }}**. | ||
1. Click **Create bucket**. | ||
1. On the bucket creation page: | ||
* Enter the **Name**: `bucket-yq`. | ||
* For the other parameters, leave the default settings. | ||
1. Click **Create bucket**. | ||
|
||
{% endlist %} | ||
|
||
|
||
### Create service accounts {#create-sa} | ||
|
||
Create a service account named `trail-sa`: | ||
|
||
{% list tabs %} | ||
|
||
- Management console | ||
|
||
1. In the [management console]({{ link-console-main }}), go to the `example-folder` folder. | ||
1. Go to the **Service accounts** tab. | ||
1. Click **Create service account**. | ||
1. Enter the **Name**: `trail-sa`. | ||
1. Click **Create**. | ||
|
||
{% endlist %} | ||
|
||
Use the same method to create a service account named `bucket-yq-sa`. | ||
|
||
|
||
### Assign rights to service accounts {#grant-roles} | ||
|
||
Assign the `audit-trails.viewer` and `storage.uploader` roles to the `trail-sa` service account: | ||
|
||
{% list tabs %} | ||
|
||
- CLI | ||
|
||
1. The `audit-trails.viewer` role to the organization: | ||
|
||
```bash | ||
yc organization-manager organization add-access-binding \ | ||
--role audit-trails.viewer \ | ||
--id <organization_ID> \ | ||
--service-account-id <trail-sa_service_account_ID> | ||
``` | ||
|
||
Result: | ||
|
||
```text | ||
done (1s) | ||
``` | ||
|
||
For more information about the `yc resource-manager organization add-access-binding` command, see the [CLI reference](../../cli/cli-ref/managed-services/organization-manager/organization/add-access-binding.md). | ||
|
||
1. The `storage.uploader` role to `example-folder`: | ||
|
||
```bash | ||
yc resource-manager folder add-access-binding example-folder \ | ||
--role storage.uploader \ | ||
--subject serviceAccount:<trail-sa_service_account_ID> | ||
``` | ||
|
||
Result: | ||
|
||
```text | ||
done (1s) | ||
``` | ||
|
||
For more information about the `yc resource-manager folder add-access-binding` command, see the [CLI reference](../../cli/cli-ref/managed-services/resource-manager/folder/add-access-binding.md). | ||
|
||
{% endlist %} | ||
|
||
Assign the `bucket-yq-sa` service account the `storage.viewer` role to `example-folder`: | ||
|
||
{% list tabs %} | ||
|
||
- CLI | ||
|
||
```bash | ||
yc resource-manager folder add-access-binding example-folder \ | ||
--role storage.viewer \ | ||
--subject serviceAccount:<bucket-yq-sa_service_account_ID> | ||
``` | ||
|
||
Result: | ||
|
||
```text | ||
done (1s) | ||
``` | ||
|
||
For more information about the `yc resource-manager folder add-access-binding` command, see the [CLI reference](../../cli/cli-ref/managed-services/resource-manager/folder/add-access-binding.md). | ||
|
||
{% endlist %} | ||
|
||
|
||
## Create a trail {#create-trail} | ||
|
||
{% list tabs %} | ||
|
||
- Management console | ||
|
||
1. In the [management console]({{ link-console-main }}), select the `example-folder` folder. | ||
1. Select **{{ at-name }}**. | ||
1. Click **Create trail** and specify: | ||
* **Name**: `logsyq`. | ||
* **Resource**: Select `Organization`. | ||
* **Organization**: An automatically populated field containing the name of the current organization. | ||
* **Destination**: `{{ objstorage-name }}`. | ||
* **Bucket**: `bucket-yq`. | ||
* **Service account**: `trail-sa`. | ||
* For the other parameters, leave the default settings. | ||
1. Click **Create**. | ||
|
||
{% endlist %} | ||
|
||
|
||
## Create a connection between a trail and {{ yq-short-name }} {#trail-yq} | ||
|
||
A connection must be created only the first time a trail is connected to {{ yq-short-name }}. | ||
|
||
{% list tabs %} | ||
|
||
- Management console | ||
|
||
1. In the [management console]({{ link-console-main }}), select the `example-folder` folder. | ||
1. Select **{{ at-name }}**. | ||
1. Select the `logsyq` trail. | ||
1. Click **Process in {{ yq-short-name }}**. | ||
1. Create a connection. | ||
* Select the `bucket-yq-sa` **service account**. | ||
* For the other parameters, leave the default settings. | ||
1. Click **Create**. | ||
1. In the window with data binding options, click **Create**. | ||
|
||
You will go to the page for creating a query to trail logs. | ||
|
||
{% endlist %} | ||
|
||
|
||
## Execute a query to logs in {{ objstorage-name }} {#perform-request} | ||
|
||
Open the page to create an analytical query to {{ at-name }} logs: | ||
|
||
{% list tabs %} | ||
|
||
- Management console | ||
|
||
1. In the [management console]({{ link-console-main }}), select a folder with a trail. | ||
1. In the list of services, select **{{ at-name }}**. | ||
1. Select the trail for which a [connection to {{ yq-short-name }}](#trail-yq) is configured. | ||
1. Click **Process in {{ yq-short-name }}** to go to the analytical query execution page. | ||
|
||
{% endlist %} | ||
|
||
Execute event queries to bind `audit-trails-logsyq-object_storage`: | ||
|
||
1. Deleting a folder: | ||
|
||
1. From the list, select query **1. Find out who deleted a folder**. | ||
1. Edit the query by specifying the folder ID: | ||
|
||
```SQL | ||
SELECT * FROM bindings.`audit-trails-logsyq-object_storage` | ||
WHERE | ||
JSON_VALUE(data, "$.event_type") = 'yandex.cloud.audit.resourcemanager.DeleteFolder' | ||
and JSON_VALUE(data, "$.details.folder_name") = '<folder_ID>' | ||
LIMIT 100; | ||
``` | ||
|
||
1. Click **Execute**. | ||
|
||
1. Enabling access via the serial console: | ||
|
||
1. From the list, select query **6. Changing a VM: Adding access to the serial console**. | ||
1. Edit the query by specifying the number of displayed records: | ||
|
||
```SQL | ||
SELECT * FROM bindings.`<audit-trails-logsyq-object_storage>` | ||
WHERE | ||
JSON_VALUE(data, "$.event_type") = 'yandex.cloud.audit.compute.UpdateInstance' | ||
and JSON_VALUE(data, "$.details.metadata_serial_port_enable") = '1' | ||
LIMIT <number_of_records>; | ||
``` | ||
|
||
1. Click **Execute**. | ||
|
||
1. Changing access rights to an {{ objstorage-name }} bucket: | ||
|
||
1. From the list, select query **11. Suspicious activities with the {{ at-name }} log repository ({{ objstorage-name }} Bucket)**. | ||
1. Edit the query by specifying the number of displayed records: | ||
|
||
```SQL | ||
SELECT * FROM bindings.`audit-trails-logsyq-object_storage` | ||
WHERE | ||
(JSON_VALUE(data, "$.event_type") = 'yandex.cloud.audit.storage.BucketAclUpdate' | ||
or JSON_VALUE(data, "$.event_type") = 'yandex.cloud.audit.storage.BucketPolicyUpdate') | ||
LIMIT <number_of_records>; | ||
``` | ||
|
||
1. Click **Execute**. | ||
|
||
1. Assigning administrator rights: | ||
|
||
1. From the list, select query **20. Assigning admin rights (for resources: folder, cloud)**. | ||
1. Edit the query by specifying the number of displayed records: | ||
|
||
```SQL | ||
SELECT * FROM bindings.`audit-trails-logsyq-object_storage` | ||
WHERE | ||
JSON_VALUE(data, "$.details.access_binding_deltas.access_binding.role_id") = 'admin' | ||
LIMIT <number_of_records>; | ||
``` | ||
|
||
1. Click **Execute**. | ||
|
||
|
||
## How to delete created resources {#clear-out} | ||
|
||
If you created a separate bucket to follow these instructions, you can [delete it](../../storage/operations/buckets/delete.md) to stop paying for [bucket use](../../storage/pricing.md). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# {{ yq-full-name }} integration | ||
|
||
{% include [query-billing-integration](../../query/_includes/billing.md) %} |
Oops, something went wrong.