Skip to content

Commit

Permalink
Azure blob storage: Correct scheme prefix to azure-blob:// (cockroach…
Browse files Browse the repository at this point in the history
  • Loading branch information
benbardin authored Jan 12, 2024
1 parent c6206db commit 1b340cd
Show file tree
Hide file tree
Showing 34 changed files with 119 additions and 105 deletions.
2 changes: 1 addition & 1 deletion src/current/_includes/cockroachcloud/cdc/external-urls.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ If your environment requires an HTTP or HTTPS proxy server for outgoing connecti
Location | Example
-------------+----------------------------------------------------------------------------------
Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456`
Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co`
Azure | `azure-blob://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co`
Google Cloud | `gs://acme-co`
HTTP | `http://localhost:8080/employees`
NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees`&nbsp;[<sup>5</sup>](#considerations)
1 change: 0 additions & 1 deletion src/current/_includes/releases/v22.2/v22.2.15.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ Release Date: October 23, 2023
- When upgrading from previous releases, it was possible that ambiguous references to [sequences](https://www.cockroachlabs.com/docs/v22.2/create-sequence) would fail to be resolved by ID if they were in the same schema. Additionally, the logging is improved for ambiguous cases where the reference cannot be resolved. [#110721][#110721]
- Fixed a bug where [`RESET `](https://www.cockroachlabs.com/docs/v22.2/row-level-ttl#reset-a-storage-parameter-to-its-default-value)[(`ttl_expire_after`)](https://www.cockroachlabs.com/docs/v22.2/row-level-ttl#using-ttl_expire_after) could incorrectly remove [`ttl_expiration_expression`](https://www.cockroachlabs.com/docs/v22.2/row-level-ttl#using-ttl_expiration_expression). [#110750][#110750]
- Fixed a bug where [`format_type`](https://www.cockroachlabs.com/docs/v22.2/functions-and-operators#compatibility-functions) builtin did not honor `typemod` information for [array](https://www.cockroachlabs.com/docs/v22.2/array) types, leading to incorrect output. [#110939][#110939]
- External connection URLs now accept the scheme `azure-blob` for connections to [Azure](https://www.cockroachlabs.com/docs/v22.2/use-cloud-storage#url-format) Blob Storage. For backward compatibility, schemes `azure` and `azure-storage` schemes continue to work for connections to Azure Blob Storage. [#111286][#111286]
- Fixed a bug where `indoption` inside [`pg_index`](https://www.cockroachlabs.com/docs/v22.2/pg-catalog#data-exposed-by-pg_catalog) was not properly encoded, causing it to be unable to be decoded as `int2vector`. [#111956][#111956]
- Fixed a bug where the [optimizer](https://www.cockroachlabs.com/docs/v22.2/cost-based-optimizer) failed to honor the `statement_timeout` session setting when generating constrained index scans for queries with large `IN` lists or `= ANY` predicates on multiple index key columns, which may lead to an [out-of-memory crash (OOM)](https://www.cockroachlabs.com/docs/v22.2/cluster-setup-troubleshooting#out-of-memory-oom-crash) on the node. [#112075][#112075]
- Fixed a bug that caused internal errors during query [optimization](https://www.cockroachlabs.com/docs/v22.2/cost-based-optimizer) in rare cases. The bug has been present since version 2.1.11, but it is more likely to occur in version 21.2.0 and later, though it is still rare. The bug only presents when a query contains `min` and `max` [aggregate functions](https://www.cockroachlabs.com/docs/v22.2/functions-and-operators#aggregate-functions). [#112299][#112299]
Expand Down
12 changes: 6 additions & 6 deletions src/current/_includes/v22.2/cdc/external-urls.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Location | Scheme | Host | Parameters |
|-------------------------------------------------------------+-------------+--------------------------------------------------+----------------------------------------------------------------------------
Amazon | `s3` | Bucket name | `AUTH`&nbsp;[<sup>1</sup>](#considerations) (optional; can be `implicit` or `specified`), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`
Amazon | `s3` | Bucket name | `AUTH`&nbsp;[<sup>1</sup>](#considerations) (optional; can be `implicit` or `specified`), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`
Azure | `azure` | N/A (see [Example file URLs](#example-file-urls) | `AZURE_ACCOUNT_KEY`, `AZURE_ACCOUNT_NAME`
Google Cloud&nbsp;[<sup>2</sup>](#considerations) | `gs` | Bucket name | `AUTH` (optional; can be `default`, `implicit`, or `specified`), `CREDENTIALS`
HTTP&nbsp;[<sup>3</sup>](#considerations) | `http` | Remote host | N/A
Expand Down Expand Up @@ -39,10 +39,10 @@ If your environment requires an HTTP or HTTPS proxy server for outgoing connecti

#### Example file URLs

Location | Example
Location | Example
-------------+----------------------------------------------------------------------------------
Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456`
Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co`
Google Cloud | `gs://acme-co`
HTTP | `http://localhost:8080/employees`
Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456`
Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co`
Google Cloud | `gs://acme-co`
HTTP | `http://localhost:8080/employees`
NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees`&nbsp;[<sup>5</sup>](#considerations)
2 changes: 1 addition & 1 deletion src/current/_includes/v23.1/cdc/external-urls.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ If your environment requires an HTTP or HTTPS proxy server for outgoing connecti
Location | Example
-------------+----------------------------------------------------------------------------------
Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456`
Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co`
Azure | `azure-blob://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co`
Google Cloud | `gs://acme-co`
HTTP | `http://localhost:8080/employees`
NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees`&nbsp;[<sup>5</sup>](#considerations)
3 changes: 3 additions & 0 deletions src/current/_includes/v23.1/misc/azure-blob.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{{site.data.alerts.callout_info}}
For backward compatibility, schemes `azure://` and `azure-storage://` are also supported, though `azure-blob://` is recommended.
{{site.data.alerts.end}}
2 changes: 1 addition & 1 deletion src/current/_includes/v23.2/cdc/external-urls.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ If your environment requires an HTTP or HTTPS proxy server for outgoing connecti
Location | Example
-------------+----------------------------------------------------------------------------------
Amazon S3 | `s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456`
Azure | `azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co`
Azure | `azure-blob://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co`
Google Cloud | `gs://acme-co`
HTTP | `http://localhost:8080/employees`
NFS/Local | `nodelocal://1/path/employees`, `nodelocal://self/nfsmount/backups/employees`&nbsp;[<sup>5</sup>](#considerations)
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Location | Example
-------------+----------------------------------------------------------------------------------
Amazon S3 | `'s3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}'`
Azure Blob Storage | `'azure://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'`
Azure Blob Storage | `'azure-blob://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'`
Google Cloud | `'gs://{BUCKET NAME}/{PATH}?AUTH=specified&CREDENTIALS={ENCODED KEY}'`
HTTP | `'file-http(s)://localhost:8080/{PATH}'` or `'http(s)://localhost:8080/{PATH}'`<br><br>**Note:** Using `http(s)` without the `file-` prefix is deprecated as a [changefeed sink]({% link {{ page.version.version }}/changefeed-sinks.md %}) scheme. There is continued support for `http(s)`, but it will be removed in a future release. We recommend implementing the `file-http(s)` scheme for changefeed messages.
3 changes: 3 additions & 0 deletions src/current/_includes/v23.2/misc/azure-blob.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{{site.data.alerts.callout_info}}
For backward compatibility, schemes `azure://` and `azure-storage://` are also supported, though `azure-blob://` is recommended.
{{site.data.alerts.end}}
10 changes: 5 additions & 5 deletions src/current/v22.2/cancel-job.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ Parameter | Description
> SHOW JOBS;
~~~
~~~
+----------------+---------+-------------------------------------------+...
| id | type | description |...
+----------------+---------+-------------------------------------------+...
| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
+----------------+---------+-------------------------------------------+...
+----------------+---------+------------------------------------------------+...
| id | type | description |...
+----------------+---------+------------------------------------------------+...
| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
+----------------+---------+------------------------------------------------+...
~~~
~~~ sql
> CANCEL JOB 27536791415282;
Expand Down
6 changes: 3 additions & 3 deletions src/current/v22.2/pause-job.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ Parameter | Description
~~~

~~~
job_id | job_type | description |...
-----------------+-----------+-------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
job_id | job_type | description |...
-----------------+-----------+------------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
~~~

{% include_cached copy-clipboard.html %}
Expand Down
6 changes: 3 additions & 3 deletions src/current/v22.2/resume-job.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,9 @@ Parameter | Description
~~~

~~~
job_id | job_type | description |...
-----------------+-----------+-------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
job_id | job_type | description |...
-----------------+-----------+------------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
~~~

{% include_cached copy-clipboard.html %}
Expand Down
16 changes: 8 additions & 8 deletions src/current/v22.2/show-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,9 @@ Status | Description
~~~

~~~
job_id | job_type | description |...
+---------------+-----------+-------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
job_id | job_type | description |...
+---------------+-----------+------------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
~~~

### Filter jobs
Expand All @@ -122,9 +122,9 @@ You can filter jobs by using `SHOW JOBS` as the data source for a [`SELECT`](sel
~~~

~~~
job_id | job_type | description |...
+---------------+-----------+-------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
job_id | job_type | description |...
+---------------+-----------+------------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
~~~

Expand Down Expand Up @@ -235,8 +235,8 @@ To block `SHOW JOB` until the provided job ID reaches a terminal state, use `SHO
> SHOW JOB WHEN COMPLETE 27536791415282;
~~~
~~~
job_id | job_type | description |...
+----------------+-----------+-------------------------------------------+...
job_id | job_type | description |...
+----------------+-----------+------------------------------------------------+...
27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
~~~

Expand Down
4 changes: 2 additions & 2 deletions src/current/v23.1/alter-backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ To add a new KMS key to the most recent backup:

{% include_cached copy-clipboard.html %}
~~~ sql
ALTER BACKUP LATEST IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}'
ALTER BACKUP LATEST IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}'
ADD NEW_KMS = 'azure-kms:///{new key}/{new key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'
WITH OLD_KMS = 'azure-kms:///{old key}/{old key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}';
~~~
Expand All @@ -88,7 +88,7 @@ To add a new KMS key to a specific backup, issue an `ALTER BACKUP` statement tha

{% include_cached copy-clipboard.html %}
~~~ sql
ALTER BACKUP '2023/03/14-203808.29' IN 'azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}'
ALTER BACKUP '2023/03/14-203808.29' IN 'azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}'
ADD NEW_KMS = 'azure-kms:///{new key}/{new key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}'
WITH OLD_KMS = 'azure-kms:///{old key}/{old key version}?AZURE_TENANT_ID={tenant ID}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_VAULT_NAME={key vault name}';
~~~
Expand Down
10 changes: 5 additions & 5 deletions src/current/v23.1/cancel-job.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ Parameter | Description
> SHOW JOBS;
~~~
~~~
+----------------+---------+-------------------------------------------+...
| id | type | description |...
+----------------+---------+-------------------------------------------+...
| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure://backup/db/tbl' |...
+----------------+---------+-------------------------------------------+...
+----------------+---------+------------------------------------------------+...
| id | type | description |...
+----------------+---------+------------------------------------------------+...
| 27536791415282 | RESTORE | RESTORE db.* FROM 'azure-blob://backup/db/tbl' |...
+----------------+---------+------------------------------------------------+...
~~~
~~~ sql
> CANCEL JOB 27536791415282;
Expand Down
2 changes: 1 addition & 1 deletion src/current/v23.1/changefeed-sinks.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,7 @@ Examples of supported cloud storage sink URIs:
### Azure Blob Storage

~~~
'azure://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'
'azure-blob://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'
~~~

### Google Cloud Storage
Expand Down
10 changes: 7 additions & 3 deletions src/current/v23.1/cloud-storage-authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -686,7 +686,7 @@ You can authenticate to Azure with explicit credentials in the following ways:
- `AZURE_TENANT_ID`: Directory (tenant) ID for your App Registration.

~~~
azure://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}
azure-blob://{container name}?AUTH=specified&AZURE_ACCOUNT_NAME={account name}&AZURE_CLIENT_ID={client ID}&AZURE_CLIENT_SECRET={client secret}&AZURE_TENANT_ID={tenant ID}
~~~

You can authenticate to Azure Storage and Azure Key Vault with this URI format.
Expand All @@ -699,9 +699,11 @@ You can authenticate to Azure with explicit credentials in the following ways:
It is necessary to [url encode](https://wikipedia.org/wiki/Percent-encoding) the account key since it is base64-encoded and may contain `+`, `/`, `=` characters.

~~~
azure://{container name}?AZURE_ACCOUNT_NAME={account name}&AZURE_ACCOUNT_KEY={url-encoded key}&AZURE_ENVIRONMENT=AZUREUSGOVERNMENTCLOUD
azure-blob://{container name}?AZURE_ACCOUNT_NAME={account name}&AZURE_ACCOUNT_KEY={url-encoded key}&AZURE_ENVIRONMENT=AZUREUSGOVERNMENTCLOUD
~~~

{% include {{ page.version.version }}/misc/azure-blob.md %}

## Azure Blob Storage implicit authentication

{{site.data.alerts.callout_info}}
Expand Down Expand Up @@ -782,9 +784,11 @@ To set up `implicit` authentication to Azure Blob Storage (or a KMS resource), y
{% include_cached copy-clipboard.html %}
~~~sql
BACKUP DATABASE {database} INTO 'azure://{container name}?AUTH=implicit&AZURE_ACCOUNT_NAME={account name}';
BACKUP DATABASE {database} INTO 'azure-blob://{container name}?AUTH=implicit&AZURE_ACCOUNT_NAME={account name}';
~~~
{% include {{ page.version.version }}/misc/azure-blob.md %}
</section>
<section class="filter-content" markdown="1" data-scope="http">
Expand Down
2 changes: 1 addition & 1 deletion src/current/v23.1/create-changefeed.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ The following are example file URLs for each of the cloud storage schemes:
Location | Example
-------------+----------------------------------------------------------------------------------
Amazon S3 | `'s3://{BUCKET NAME}/{PATH}?AWS_ACCESS_KEY_ID={KEY ID}&AWS_SECRET_ACCESS_KEY={SECRET ACCESS KEY}'`
Azure Blob Storage | `'azure://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'`
Azure Blob Storage | `'azure-blob://{CONTAINER NAME}/{PATH}?AZURE_ACCOUNT_NAME={ACCOUNT NAME}&AZURE_ACCOUNT_KEY={URL-ENCODED KEY}'`
Google Cloud | `'gs://{BUCKET NAME}/{PATH}?AUTH=specified&CREDENTIALS={ENCODED KEY'`
HTTP | `'http://localhost:8080/{PATH}'`

Expand Down
1 change: 1 addition & 0 deletions src/current/v23.1/create-external-connection.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ Storage or sink | Operation support
[Amazon S3]({% link {{ page.version.version }}/use-cloud-storage.md %}) | Backups, restores, imports, exports, changefeeds
[Amazon S3 KMS]({% link {{ page.version.version }}/take-and-restore-encrypted-backups.md %}#aws-kms-uri-format) | Encrypted backups
[Azure Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) | Backups, restores, imports, exports, changefeeds
[Azure Key Vault]({% link {{ page.version.version }}/take-and-restore-encrypted-backups.md %}#azure-key-vault-uri-format) | Encrypted backups
[Confluent Schema Registry]({% link {{ page.version.version }}/create-changefeed.md %}#confluent-registry) | Changefeeds
[Google Cloud Pub/Sub]({% link {{ page.version.version }}/changefeed-sinks.md %}#google-cloud-pub-sub) | Changefeeds
[Google Cloud Storage]({% link {{ page.version.version }}/use-cloud-storage.md %}) | Backups, restores, imports, exports, changefeeds
Expand Down
Loading

0 comments on commit 1b340cd

Please sign in to comment.