Skip to content

Commit

Permalink
IO reorg
Browse files Browse the repository at this point in the history
  • Loading branch information
Lauren committed Sep 3, 2020
1 parent 3782674 commit 50694a2
Show file tree
Hide file tree
Showing 39 changed files with 2,290 additions and 330 deletions.
1,850 changes: 1,850 additions & 0 deletions _includes/sidebar-data-v20.2.json

Large diffs are not rendered by default.

16 changes: 8 additions & 8 deletions _includes/v20.2/backups/advanced-examples-list.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
For examples of advanced `BACKUP` and `RESTORE` use cases, see [Back up and Restore Data - Advanced Options](backup-and-restore-advanced-options.html). Advanced examples include:
For examples of advanced `BACKUP` and `RESTORE` use cases, see:

- [Incremental backups with a specified destination](backup-and-restore-advanced-options.html#incremental-backups-with-explicitly-specified-destinations)
- [Backup with revision history and point-in-time restore](backup-and-restore-advanced-options.html#backup-with-revision-history-and-point-in-time-restore)
- [Locality-aware backup and restore](backup-and-restore-advanced-options.html#locality-aware-backup-and-restore)
- [Encrypted backup and restore](backup-and-restore-advanced-options.html#encrypted-backup-and-restore)
- [Restore into a different database](backup-and-restore-advanced-options.html#restore-into-a-different-database)
- [Remove the foreign key before restore](backup-and-restore-advanced-options.html#remove-the-foreign-key-before-restore)
- [Restoring users from `system.users` backup](backup-and-restore-advanced-options.html#restoring-users-from-system-users-backup)
- [Incremental backups with a specified destination](take-full-and-incremental-backups.html#incremental-backups-with-explicitly-specified-destinations)
- [Backup with revision history and point-in-time restore](take-backups-with-revision-history-and-restore-from-a-point-in-time.html)
- [Locality-aware backup and restore](take-and-restore-locality-aware-backups.html)
- [Encrypted backup and restore](take-and-restore-encrypted-backups.html)
- [Restore into a different database](restore.html#restore-into-a-different-database)
- [Remove the foreign key before restore](restore.html#remove-the-foreign-key-before-restore)
- [Restoring users from `system.users` backup](restore.html#restoring-users-from-system-users-backup)
4 changes: 2 additions & 2 deletions _includes/v20.2/backups/encrypted-backup-description.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
You can encrypt full or incremental backups by using the [`encryption_passphrase` option](backup.html#with-encryption-passphrase). Files written by the backup (including `BACKUP` manifests and data files) are encrypted using the specified passphrase to derive a key. To restore the encrypted backup, the same `encryption_passphrase` option (with the same passphrase) must included in the [`RESTORE`](restore.html) statement.

When used with [incremental backups](backup.html#incremental-backups), the `encryption_passphrase` option is applied to all the [backup file URLs](backup.html#backup-file-urls), which means the same passphrase must be used when appending another incremental backup to an existing backup. Similarly, when used with [locality-aware backups](backup-and-restore-advanced-options.html#locality-aware-backup-and-restore), the passphrase provided is applied to files in all localities.
When used with [incremental backups](backup.html#incremental-backups), the `encryption_passphrase` option is applied to all the [backup file URLs](backup.html#backup-file-urls), which means the same passphrase must be used when appending another incremental backup to an existing backup. Similarly, when used with [locality-aware backups](take-and-restore-locality-aware-backups.html), the passphrase provided is applied to files in all localities.

Encryption is done using [AES-256-GCM](https://en.wikipedia.org/wiki/Galois/Counter_Mode), and GCM is used to both encrypt and authenticate the files. A random [salt](https://en.wikipedia.org/wiki/Salt_(cryptography)) is used to derive a once-per-backup [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) key from the specified passphrase, and then a random [initialization vector](https://en.wikipedia.org/wiki/Initialization_vector) is used per-file. CockroachDB uses [PBKDF2](https://en.wikipedia.org/wiki/PBKDF2) with 64,000 iterations for the key derivation.

{{site.data.alerts.callout_info}}
`BACKUP` and `RESTORE` will use more memory when using encryption, as both the plain-text and cipher-text of a given file are held in memory during encryption and decryption.
{{site.data.alerts.end}}

For an example of an encrypted backup, see [Create an encrypted backup](backup-and-restore-advanced-options.html#create-an-encrypted-backup).
For an example of an encrypted backup, see [Create an encrypted backup](take-and-restore-encrypted-backups.html).
2 changes: 1 addition & 1 deletion _includes/v20.2/known-limitations/cdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
- Changefeeds do not share internal buffers, so each running changefeed will increase total memory usage. To watch multiple tables, we recommend creating a changefeed with a comma-separated list of tables.
- Many DDL queries (including [`TRUNCATE`](truncate.html) and [`DROP TABLE`](drop-table.html)) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed](create-changefeed.html#start-a-new-changefeed-where-another-ended).
- Changefeeds cannot be [backed up](backup.html) or [restored](restore.html).
- Partial or intermittent sink unavailability may impact changefeed stability; however, [ordering guarantees](change-data-capture.html#ordering-guarantees) will still hold for as long as a changefeed [remains active](change-data-capture.html#monitor-a-changefeed).
- Partial or intermittent sink unavailability may impact changefeed stability; however, [ordering guarantees](stream-data-out-of-cockroachdb-using-changefeeds.html#ordering-guarantees) will still hold for as long as a changefeed [remains active](stream-data-out-of-cockroachdb-using-changefeeds.html#monitor-a-changefeed).
- Changefeeds cannot be altered. To alter, cancel the changefeed and [create a new one with updated settings from where it left off](create-changefeed.html#start-a-new-changefeed-where-another-ended).
- Additional target options will be added, including partitions and ranges of primary key rows.
- Changefeeds do not pick up data ingested with the [`IMPORT INTO`](import-into.html) statement.
Expand Down
2 changes: 1 addition & 1 deletion _includes/v20.2/misc/enterprise-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Feature | Description
[Follower Reads](follower-reads.html) | This feature reduces read latency in multi-region deployments by using the closest replica at the expense of reading slightly historical data.
[`BACKUP`](backup.html) | This feature creates full or incremental backups of your cluster's schema and data that are consistent as of a given timestamp, stored on a service such as AWS S3, Google Cloud Storage, NFS, or HTTP storage.<br><br>Backups can be locality-aware such that each node writes files only to the backup destination that matches the node's [locality](cockroach-start.html#locality). This is useful for reducing cloud storage data transfer costs by keeping data within cloud regions and complying with data domiciling requirements.
[`RESTORE`](restore.html) | This feature restores your cluster's schemas and data from an enterprise `BACKUP`.
[Change Data Capture](change-data-capture.html) (CDC) | This feature provides efficient, distributed, row-level [change feeds into Apache Kafka](create-changefeed.html) for downstream processing such as reporting, caching, or full-text indexing.
[Change Data Capture](stream-data-out-of-cockroachdb-using-changefeeds.html) (CDC) | This feature provides efficient, distributed, row-level [change feeds into Apache Kafka](create-changefeed.html) for downstream processing such as reporting, caching, or full-text indexing.
[Node Map](enable-node-map.html) | This feature visualizes the geographical configuration of a cluster by plotting node localities on a world map.
[Locality-Aware Index Selection](cost-based-optimizer.html#preferring-the-nearest-index) | Given [multiple identical indexes](topology-duplicate-indexes.html) that have different locality constraints using [replication zones](configure-replication-zones.html), the cost-based optimizer will prefer the index that is closest to the gateway node that is planning the query. In multi-region deployments, this can lead to performance improvements due to improved data locality and reduced network traffic.
[Encryption at Rest](encryption.html#encryption-at-rest-enterprise) | Supplementing CockroachDB's encryption in flight capabilities, this feature provides transparent encryption of a node's data on the local disk. It allows encryption of all files on disk using AES in counter mode, with all key sizes allowed.
Expand Down
2 changes: 1 addition & 1 deletion releases/v20.2.0-alpha.2.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ $ docker pull cockroachdb/cockroach-unstable:v20.2.0-alpha.2

### Enterprise edition changes

- The default flush interval for [changefeeds](../v20.2/change-data-capture.html) that do not specify a `RESOLVED` option is now 5s instead of 200ms to more gracefully handle higher-latency sinks. [#49770][#49770] {% comment %}doc{% endcomment %}
- The default flush interval for [changefeeds](../v20.2/stream-data-out-of-cockroachdb-using-changefeeds.html) that do not specify a `RESOLVED` option is now 5s instead of 200ms to more gracefully handle higher-latency sinks. [#49770][#49770] {% comment %}doc{% endcomment %}
- [Full cluster restore](../v20.2/restore.html#full-cluster) is now more resilient to transient transaction retry errors while it restores. [#49950][#49950]
- [`RESTORE`](../v20.2/restore.html) now remaps user-defined types within the backup to existing types in the cluster. [#50349][#50349] {% comment %}doc{% endcomment %}
- `BACKUP` can now run in a new `detached` mode, which means the Job ID is now returned immediately without waiting for the `BACKUP` job to finish. [#50775][#50775] {% comment %}doc{% endcomment %}
Expand Down
10 changes: 5 additions & 5 deletions v20.2/admin-ui-cdc-dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ summary: The Changefeeds dashboard lets you monitor the changefeeds created acro
toc: true
---

The **Changefeeds** dashboard in the CockroachDB Admin UI lets you monitor the [changefeeds](change-data-capture.html) created across your cluster. To view this dashboard, [access the Admin UI](admin-ui-overview.html#admin-ui-access), click **Metrics** on the left-hand navigation bar, and then select **Dashboard** > **Changefeeds**.
The **Changefeeds** dashboard in the CockroachDB Admin UI lets you monitor the [changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) created across your cluster. To view this dashboard, [access the Admin UI](admin-ui-overview.html#admin-ui-access), click **Metrics** on the left-hand navigation bar, and then select **Dashboard** > **Changefeeds**.

{% include {{ page.version.version }}/admin-ui/admin-ui-metrics-navigation.md %}

Expand All @@ -17,7 +17,7 @@ This graph shows the maximum latency for resolved timestamps of any running chan
<img src="{{ 'images/v20.2/admin_ui_max_changefeed.png' | relative_url }}" alt="CockroachDB Admin UI Max Changefeed Latency graph" style="border:1px solid #eee;max-width:100%" />

{{site.data.alerts.callout_info}}
The maximum latency for resolved timestamps is distinct from and slower than the commit-to-emit latency for individual change messages. For more information about resolved timestamps, see [Ordering guarantees](change-data-capture.html#ordering-guarantees).
The maximum latency for resolved timestamps is distinct from and slower than the commit-to-emit latency for individual change messages. For more information about resolved timestamps, see [Ordering guarantees](stream-data-out-of-cockroachdb-using-changefeeds.html#ordering-guarantees).
{{site.data.alerts.end}}

## Sink Byte Traffic
Expand All @@ -32,7 +32,7 @@ Metric | Description

## Sink Counts

This graph shows:
This graph shows:

- The number of messages that CockroachDB sent to the sink.
- The number of flushes that the sink performed for changefeeds.
Expand All @@ -46,7 +46,7 @@ Metric | Description

## Sink Timings

This graph shows:
This graph shows:

- The time in milliseconds per second required by CockroachDB to send messages to the sink.
- The time CockroachDB spent waiting for the sink to flush the messages for changefeeds.
Expand All @@ -68,7 +68,7 @@ This graph displays the number of times changefeeds restarted due to retryable e

## See also

- [Change Data Capture](change-data-capture.html)
- [Stream Data Out of CockroachDB Using Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html)
- [Troubleshooting Overview](troubleshooting-overview.html)
- [Support Resources](support-resources.html)
- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints)
12 changes: 6 additions & 6 deletions v20.2/admin-ui-jobs-page.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ The **Jobs** page of the Admin UI provides details about long-running tasks perf
- Enterprise [`BACKUP`](backup.html) and [`RESTORE`](restore.html).
- [User-created table statistics](create-statistics.html) created for use by the [cost-based optimizer](cost-based-optimizer.html).
- [Automatic table statistics](cost-based-optimizer.html#table-statistics).
- [Changefeeds](change-data-capture.html).
- [Changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html).

{{site.data.alerts.callout_success}}
All users can see their own jobs, and `admin` users can view all jobs performed across all nodes in the cluster.
All users can see their own jobs, and `admin` users can view all jobs performed across all nodes in the cluster.
{{site.data.alerts.end}}

To view these details, [access the Admin UI](admin-ui-overview.html#admin-ui-access) and click **Jobs** in the left-hand navigation.
Expand All @@ -27,16 +27,16 @@ Use the **Type** menu to filter jobs by type.
You can toggle between showing the latest 50 jobs or all jobs on the cluster.

{{site.data.alerts.callout_info}}
Jobs are deleted every 14 days. This interval can be changed via the `jobs.retention_time` [cluster setting](cluster-settings.html).
Jobs are deleted every 14 days. This interval can be changed via the `jobs.retention_time` [cluster setting](cluster-settings.html).

The Jobs list is designed for you to manage pending work. It is not intended to display the canonical record of all jobs that have run. If you need a historical record of all jobs you have run, you should log this information externally.
The Jobs list is designed for you to manage pending work. It is not intended to display the canonical record of all jobs that have run. If you need a historical record of all jobs you have run, you should log this information externally.
{{site.data.alerts.end}}

## Jobs list

Use the **Jobs** list to see your recently created and completed jobs.

- For changefeeds, the table displays a [high-water timestamp that advances as the changefeed progresses](change-data-capture.html#monitor-a-changefeed). This is a guarantee that all changes before or at the timestamp have been emitted. Hover over the high-water timestamp to view the [system time](as-of-system-time.html).
- For changefeeds, the table displays a [high-water timestamp that advances as the changefeed progresses](stream-data-out-of-cockroachdb-using-changefeeds.html#monitor-a-changefeed). This is a guarantee that all changes before or at the timestamp have been emitted. Hover over the high-water timestamp to view the [system time](as-of-system-time.html).

- [Automatic table statistics](cost-based-optimizer.html#table-statistics) jobs are not displayed even when the **Type** menu is set to **All**. To view these jobs, set **Type** to **Automatic-Statistics Creation** as described [above](#filter-jobs).

Expand Down Expand Up @@ -77,4 +77,4 @@ The job ID, creation time, users, and status are also shown.
- [`SHOW JOBS`](show-jobs.html)
- [Troubleshooting Overview](troubleshooting-overview.html)
- [Support Resources](support-resources.html)
- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints)
- [Raw Status Endpoints](monitoring-and-alerting.html#raw-status-endpoints)
8 changes: 4 additions & 4 deletions v20.2/admin-ui-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Admin UI Overview
summary: Use the Admin UI to monitor and optimize cluster performance.
toc: true
redirect_from:
redirect_from:
- explore-the-admin-ui.html
- admin-ui-access-and-navigate.html
key: explore-the-admin-ui.html
Expand All @@ -22,7 +22,7 @@ Area | Description
[SQL Dashboard](admin-ui-sql-dashboard.html) | Metrics about SQL connections, byte traffic, queries, transactions, and service latency.
[Storage Dashboard](admin-ui-storage-dashboard.html) | Metrics about storage capacity and file descriptors.
[Replication Dashboard](admin-ui-replication-dashboard.html) | Metrics about how data is replicated across the cluster, e.g., range status, replicas per store, and replica quiescence.
[Changefeeds Dashboard](admin-ui-cdc-dashboard.html) | Metrics about the [changefeeds](change-data-capture.html) created across your cluster.
[Changefeeds Dashboard](admin-ui-cdc-dashboard.html) | Metrics about the [changefeeds](stream-data-out-of-cockroachdb-using-changefeeds.html) created across your cluster.
[Databases](admin-ui-databases-page.html) | Details about the system and user databases in the cluster.
[Statements](admin-ui-statements-page.html) | Frequently executed and high latency [SQL statements](sql-statements.html), with the option to collect statement diagnostics.
[Network Latency](admin-ui-network-latency-page.html) | Latencies and lost connections between all nodes in your cluster.
Expand All @@ -48,14 +48,14 @@ On insecure clusters, all areas of the Admin UI are accessible to all users.
On secure clusters, for each user who should have access to the Admin UI, you must [create a user with a password](create-user.html#create-a-user-with-a-password) and optionally [grant the user membership to the `admin` role](grant-roles.html).

{{site.data.alerts.callout_info}}
The default `root` user is a member of the `admin` role. Use the following command to [grant users membership to the `admin` role](grant-roles.html):
The default `root` user is a member of the `admin` role. Use the following command to [grant users membership to the `admin` role](grant-roles.html):

<code style="white-space:pre-wrap">GRANT admin TO \<sql_user\>;</code>
{{site.data.alerts.end}}

For security reasons, non-admin users access only the data over which they have privileges (e.g., their tables, jobs, and list of sessions), and data that does not require privileges (e.g., cluster health, node status, metrics).

The following areas of the Admin UI can only be accessed by [`admin` users](authorization.html#admin-role). These areas display information from privileged HTTP endpoints that operate with `admin` privilege.
The following areas of the Admin UI can only be accessed by [`admin` users](authorization.html#admin-role). These areas display information from privileged HTTP endpoints that operate with `admin` privilege.

Secure area | Privileged information
-----|-----
Expand Down
4 changes: 2 additions & 2 deletions v20.2/architecture/storage-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,11 +61,11 @@ CockroachDB regularly garbage collects MVCC values to reduce the size of data st

#### Protected timestamps

Garbage collection can only run on MVCC values which are not covered by a *protected timestamp*. The protected timestamp subsystem exists to ensure the safety of operations that rely on historical data, such as:
Garbage collection can only run on MVCC values which are not covered by a *protected timestamp*. The protected timestamp subsystem exists to ensure the safety of operations that rely on historical data, such as:

- [Imports](../import.html), including [`IMPORT INTO`](../import-into.html)
- [Backups](../backup.html)
- [Change data capture (CDC)](../change-data-capture.html) (a.k.a. [changefeeds](../create-changefeed.html))
- [Streaming data out of CockroachDB using changefeeds](../stream-data-out-of-cockroachdb-using-changefeeds.html)
- [Online schema changes](../online-schema-changes.html)

Protected timestamps ensure the safety of historical data while also enabling shorter [GC TTLs](../configure-replication-zones.html#gc-ttlseconds). A shorter GC TTL means that fewer previous MVCC values are kept around. This can help lower query execution costs for workloads which update rows frequently throughout the day, since [the SQL layer](sql-layer.html) has to scan over previous MVCC values to find the current value of a row.
Expand Down
2 changes: 1 addition & 1 deletion v20.2/authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ File name | File usage

## Authentication for cloud storage

See [Backup file URLs](backup.html#backup-file-urls)
See [Use Cloud Storage for Bulk Operations](use-cloud-storage-for-bulk-operations.html).

## Authentication best practice

Expand Down
Loading

0 comments on commit 50694a2

Please sign in to comment.