Skip to content

Commit

Permalink
Release 10.11.2022
Browse files Browse the repository at this point in the history
* Data Transfer: documented known limitations for MySQL source endpoints.
* Translations updated.
* Fixes and improvements.
  • Loading branch information
DataUI VCS Robot committed Nov 10, 2022
1 parent 9c7ae8f commit 596485a
Show file tree
Hide file tree
Showing 273 changed files with 4,718 additions and 1,092 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/check.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/delete.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/info.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/launch.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/option.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/pensil.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/run.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/service-roles-hierarchy.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/stop.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions en/_assets/query/study.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions en/_includes/data-streams/create-environment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@

1. Log in to [management console]({{link-console-main }}). If you haven't registered yet, go to the management console and follow the instructions.
1. [On the billing page]({{ link-console-billing }}) make sure that you have a [billing account] connected (../../billing/concepts/billing-account.md) and it is in the status `ACTIVE` or `TRIAL_ACTIVE`. If there is no billing account, [create one](../../billing/quickstart/index.md#create_billing_account).
1. If you don't have a folder yet, [create it](../../resource-manager/operations/folder/create.md).
1. [Create] (../../iam/operations/sa/create.md) a service account and [assign] (../../iam/operations/sa/assign-role-for-sa.md) to it the `editor` role for your directory.
1. [Create](../../iam/operations/sa/create-access-key.md) a static access key.
1. Configure the AWS CLI:
1. Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) and run the command:

```bash
aws configure
```

1. Enter sequentially:

* `AWS Access Key ID [None]:`: service account [key ID](../../iam/concepts/authorization/access-key.md).
* `AWS Secret Access Key [None]:`: service account [secret key](../../iam/concepts/authorization/access-key.md).
* `Default region name [None]:`: availability zone `{{ region-id }}`.
40 changes: 40 additions & 0 deletions en/_includes/data-streams/debezium-setup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
1. This example examines the interaction of Debezium and PostgreSQL. Next, we will assume that Debezium will be installed on the server where PostgreSQL is running. Install Debezium server according to [instructions](https://debezium.io/documentation/reference/stable/operations/debezium-server.html ).
1. Go to the `conf` directory and create the 'application.properties` file with the following content:
```
debezium.sink.type=kinesis
debezium.sink.kinesis.region={{ region-id }}
debezium.sink.kinesis.endpoint=<YDS_STREAM_ENDPOINT>
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.database.hostname=localhost
debezium.source.database.port=5432
debezium.source.database.user=<DATABASE_USER>
debezium.source.database.password=<DATABASE_PASSWORD>
debezium.source.database.dbname=<DATABASE_NAME>
debezium.source.database.server.name=debezium
debezium.source.plugin.name=pgoutput
debezium.source.transforms=Reroute
debezium.source.transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
debezium.source.transforms.Reroute.topic.regex=(.*)
debezium.source.transforms.Reroute.topic.replacement=<YDS_STREAM_NAME>
```
Where:
* `<YDS_STREAM_ENDPOINT>`: full endpoint to the data stream {{yds-short-name}}, for example, `https://yds.serverless .yandexcloud.net /{{region-id }}/b1p89ae43m6he2ooql88r/etn01eg4rn184nemdbb`. The full endpoint is available in the UI {{yds-full-name}}. To view the full endpoint, click **Connect**.
* `<YDS_STREAM_NAME>`: data stream name {{yds-short-name}}.
* `<DATABASE_NAME>`: name of the PostgreSQL database.
* `<DATABASE_USER>`: username for connecting to the PostgreSQL database.
* `<DATABASE_PASSWORD>`: user password for connecting to the PostgreSQL database.
1. Run Debezium with the following command:
```shell
JAVA_OPTS=-Daws.cborEnabled=false ./run.sh
```
1. Make any changes to the PostgreSQL database, for example, insert data into a table.
1. If configured correctly, the following messages will appear in the Debezium console:
```
2022-02-11 07:31:12,850 INFO [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 1 records sent during previous 00:19:59.999, last recorded offset: {transaction_id=null, lsn_proc=23576408, lsn_commit=23576120, lsn=23576408, txId=580, ts_usec=1644564672582666}
```
8 changes: 4 additions & 4 deletions en/api-design-guide/index.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
title: Yandex Cloud API concepts
description:
- >-
This document describes the underlying concepts of the Yandex Cloud API
architecture.
Here you can find out about the underlying concepts of the Yandex Cloud API
architecture.
It will be useful for developers who want to use Yandex Cloud APIs to
automate repeatable tasks, build plugins, and other applications.
This will be useful if you are looking to use the Yandex Cloud API to build
automation systems, plugins, and other apps.
meta:
title: Yandex Cloud API concepts
keywords:
Expand Down
2 changes: 2 additions & 0 deletions en/audit-trails/toc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ items:
href: operations/export-folder-bucket.md
- name: Tutorials
items:
- name: Searching for {{ yandex-cloud }} events in {{ yq-full-name }}
href: tutorials/query.md
- name: Searching for {{ yandex-cloud }} events in {{ objstorage-name }}
href: tutorials/search-bucket.md
- name: Searching for {{ yandex-cloud }} events in {{ cloud-logging-name }}
Expand Down
260 changes: 260 additions & 0 deletions en/audit-trails/tutorials/query.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,260 @@
# Searching for {{ yandex-cloud }} events in {{ yq-full-name }}

{{ at-name }} support is integrated in [{{ yq-full-name }}](../../query/). You can analyze events of {{ yandex-cloud }} resources by executing [analytical](../../query/concepts/batch-processing.md) and [streaming](../../query/concepts/stream-processing.md) {{ yql-short-name }} queries.

You can execute analytical queries for logs stored [in a bucket](../operations/export-organization-bucket.md) and streaming queries for logs stored [in a {{ yds-full-name }} data stream](../operations/export-organization-bucket.md).

![](../../_assets/audit-trails/tutorials/audit-trails-query.png)

To connect a bucket with [audit logs](../concepts/events.md) to {{ yq-full-name }} and execute [{{ yql-short-name }}](../../query/yql-tutorials/index.md) queries:

1. [Prepare the environment.](#prepare-environment)
1. [Create a connection between a trail and {{ yq-short-name }}.](#trail-yq)
1. [Execute a query to logs in {{ objstorage-name }}.](#perform-request)

If you no longer need these resources, [delete them](#clear-out).


## Before you begin {#before-begin}

{% include [before-you-begin](../../_tutorials/_tutorials_includes/before-you-begin.md) %}

{% include [cli-install](../../_includes/cli-install.md) %}



## Required paid resources {#paid-resources}

The cost of infrastructure support includes a fee for a bucket (see [Pricing for {{ objstorage-name }}](../../storage/pricing.md)).



## Prepare the environment {#prepare-environment}


### Create a bucket for audit logs {#create-backet}

{% list tabs %}

- Management console

1. In the [management console]({{ link-console-main }}), select the folder where you wish to create a [bucket](../../storage/concepts/bucket.md), for example, `example-folder`.
1. Select **{{ objstorage-name }}**.
1. Click **Create bucket**.
1. On the bucket creation page:
* Enter the **Name**: `bucket-yq`.
* For the other parameters, leave the default settings.
1. Click **Create bucket**.

{% endlist %}


### Create service accounts {#create-sa}

Create a service account named `trail-sa`:

{% list tabs %}

- Management console

1. In the [management console]({{ link-console-main }}), go to the `example-folder` folder.
1. Go to the **Service accounts** tab.
1. Click **Create service account**.
1. Enter the **Name**: `trail-sa`.
1. Click **Create**.

{% endlist %}

Use the same method to create a service account named `bucket-yq-sa`.


### Assign rights to service accounts {#grant-roles}

Assign the `audit-trails.viewer` and `storage.uploader` roles to the `trail-sa` service account:

{% list tabs %}

- CLI

1. The `audit-trails.viewer` role to the organization:

```bash
yc organization-manager organization add-access-binding \
--role audit-trails.viewer \
--id <organization_ID> \
--service-account-id <trail-sa_service_account_ID>
```

Result:

```text
done (1s)
```

For more information about the `yc resource-manager organization add-access-binding` command, see the [CLI reference](../../cli/cli-ref/managed-services/organization-manager/organization/add-access-binding.md).

1. The `storage.uploader` role to `example-folder`:

```bash
yc resource-manager folder add-access-binding example-folder \
--role storage.uploader \
--subject serviceAccount:<trail-sa_service_account_ID>
```

Result:

```text
done (1s)
```

For more information about the `yc resource-manager folder add-access-binding` command, see the [CLI reference](../../cli/cli-ref/managed-services/resource-manager/folder/add-access-binding.md).

{% endlist %}

Assign the `bucket-yq-sa` service account the `storage.viewer` role to `example-folder`:

{% list tabs %}

- CLI

```bash
yc resource-manager folder add-access-binding example-folder \
--role storage.viewer \
--subject serviceAccount:<bucket-yq-sa_service_account_ID>
```

Result:

```text
done (1s)
```

For more information about the `yc resource-manager folder add-access-binding` command, see the [CLI reference](../../cli/cli-ref/managed-services/resource-manager/folder/add-access-binding.md).

{% endlist %}


## Create a trail {#create-trail}

{% list tabs %}

- Management console

1. In the [management console]({{ link-console-main }}), select the `example-folder` folder.
1. Select **{{ at-name }}**.
1. Click **Create trail** and specify:
* **Name**: `logsyq`.
* **Resource**: Select `Organization`.
* **Organization**: An automatically populated field containing the name of the current organization.
* **Destination**: `{{ objstorage-name }}`.
* **Bucket**: `bucket-yq`.
* **Service account**: `trail-sa`.
* For the other parameters, leave the default settings.
1. Click **Create**.

{% endlist %}


## Create a connection between a trail and {{ yq-short-name }} {#trail-yq}

A connection must be created only the first time a trail is connected to {{ yq-short-name }}.

{% list tabs %}

- Management console

1. In the [management console]({{ link-console-main }}), select the `example-folder` folder.
1. Select **{{ at-name }}**.
1. Select the `logsyq` trail.
1. Click **Process in {{ yq-short-name }}**.
1. Create a connection.
* Select the `bucket-yq-sa` **service account**.
* For the other parameters, leave the default settings.
1. Click **Create**.
1. In the window with data binding options, click **Create**.

You will go to the page for creating a query to trail logs.

{% endlist %}


## Execute a query to logs in {{ objstorage-name }} {#perform-request}

Open the page to create an analytical query to {{ at-name }} logs:

{% list tabs %}

- Management console

1. In the [management console]({{ link-console-main }}), select a folder with a trail.
1. In the list of services, select **{{ at-name }}**.
1. Select the trail for which a [connection to {{ yq-short-name }}](#trail-yq) is configured.
1. Click **Process in {{ yq-short-name }}** to go to the analytical query execution page.

{% endlist %}

Execute event queries to bind `audit-trails-logsyq-object_storage`:

1. Deleting a folder:

1. From the list, select query **1. Find out who deleted a folder**.
1. Edit the query by specifying the folder ID:

```SQL
SELECT * FROM bindings.`audit-trails-logsyq-object_storage`
WHERE
JSON_VALUE(data, "$.event_type") = 'yandex.cloud.audit.resourcemanager.DeleteFolder'
and JSON_VALUE(data, "$.details.folder_name") = '<folder_ID>'
LIMIT 100;
```

1. Click **Execute**.

1. Enabling access via the serial console:

1. From the list, select query **6. Changing a VM: Adding access to the serial console**.
1. Edit the query by specifying the number of displayed records:

```SQL
SELECT * FROM bindings.`<audit-trails-logsyq-object_storage>`
WHERE
JSON_VALUE(data, "$.event_type") = 'yandex.cloud.audit.compute.UpdateInstance'
and JSON_VALUE(data, "$.details.metadata_serial_port_enable") = '1'
LIMIT <number_of_records>;
```

1. Click **Execute**.

1. Changing access rights to an {{ objstorage-name }} bucket:

1. From the list, select query **11. Suspicious activities with the {{ at-name }} log repository ({{ objstorage-name }} Bucket)**.
1. Edit the query by specifying the number of displayed records:

```SQL
SELECT * FROM bindings.`audit-trails-logsyq-object_storage`
WHERE
(JSON_VALUE(data, "$.event_type") = 'yandex.cloud.audit.storage.BucketAclUpdate'
or JSON_VALUE(data, "$.event_type") = 'yandex.cloud.audit.storage.BucketPolicyUpdate')
LIMIT <number_of_records>;
```

1. Click **Execute**.

1. Assigning administrator rights:

1. From the list, select query **20. Assigning admin rights (for resources: folder, cloud)**.
1. Edit the query by specifying the number of displayed records:

```SQL
SELECT * FROM bindings.`audit-trails-logsyq-object_storage`
WHERE
JSON_VALUE(data, "$.details.access_binding_deltas.access_binding.role_id") = 'admin'
LIMIT <number_of_records>;
```

1. Click **Execute**.


## How to delete created resources {#clear-out}

If you created a separate bucket to follow these instructions, you can [delete it](../../storage/operations/buckets/delete.md) to stop paying for [bucket use](../../storage/pricing.md).
3 changes: 3 additions & 0 deletions en/billing/operations/query-integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# {{ yq-full-name }} integration

{% include [query-billing-integration](../../query/_includes/billing.md) %}
Loading

0 comments on commit 596485a

Please sign in to comment.