Skip to content

Commit

Permalink
[SPARK-34044][DOCS] Add spark.sql.hive.metastore.jars.path to sql-dat…
Browse files Browse the repository at this point in the history
…a-sources-hive-tables.md

### What changes were proposed in this pull request?

This PR adds new configuration to `sql-data-sources-hive-tables`.

### Why are the changes needed?

SPARK-32852 added a new configuration, `spark.sql.hive.metastore.jars.path`.

### Does this PR introduce _any_ user-facing change?

Yes, but a document only.

### How was this patch tested?

**BEFORE**
![Screen Shot 2021-01-07 at 2 57 57 PM](https://user-images.githubusercontent.com/9700541/103954318-cc9ec200-50f8-11eb-86d3-cd89b07fcd21.png)

**AFTER**
![Screen Shot 2021-01-07 at 2 56 34 PM](https://user-images.githubusercontent.com/9700541/103954221-9d885080-50f8-11eb-8938-fb91394a33cb.png)

Closes apache#31085 from dongjoon-hyun/SPARK-34044.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
  • Loading branch information
dongjoon-hyun authored and HyukjinKwon committed Jan 8, 2021
1 parent aa388cf commit 5b16d70
Showing 1 changed file with 26 additions and 1 deletion.
27 changes: 26 additions & 1 deletion docs/sql-data-sources-hive-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ The following options can be used to configure the version of Hive that is used
<td><code>builtin</code></td>
<td>
Location of the jars that should be used to instantiate the HiveMetastoreClient. This
property can be one of three options:
property can be one of four options:
<ol>
<li><code>builtin</code></li>
Use Hive 2.3.7, which is bundled with the Spark assembly when <code>-Phive</code> is
Expand All @@ -148,6 +148,9 @@ The following options can be used to configure the version of Hive that is used
<li><code>maven</code></li>
Use Hive jars of specified version downloaded from Maven repositories. This configuration
is not generally recommended for production deployments.
<li><code>path</code></li>
Use Hive jars configured by <code>spark.sql.hive.metastore.jars.path</code>
in comma separated format. Support both local or remote paths.
<li>A classpath in the standard format for the JVM. This classpath must include all of Hive
and its dependencies, including the correct version of Hadoop. These jars only need to be
present on the driver, but if you are running in yarn cluster mode then you must ensure
Expand All @@ -156,6 +159,28 @@ The following options can be used to configure the version of Hive that is used
</td>
<td>1.4.0</td>
</tr>
<tr>
<td><code>spark.sql.hive.metastore.jars.path</code></td>
<td><code>(empty)</code></td>
<td>
Comma-separated paths of the jars that used to instantiate the HiveMetastoreClient.
This configuration is useful only when <code>spark.sql.hive.metastore.jars</code> is set as <code>path</code>.
<br/>
The paths can be any of the following format:
<ol>
<li><code>file://path/to/jar/foo.jar</code></li>
<li><code>hdfs://nameservice/path/to/jar/foo.jar</code></li>
<li><code>/path/to/jar/</code>(path without URI scheme follow conf <code>fs.defaultFS</code>'s URI schema)</li>
<li><code>[http/https/ftp]://path/to/jar/foo.jar</code></li>
</ol>
Note that 1, 2, and 3 support wildcard. For example:
<ol>
<li><code>file://path/to/jar/*,file://path2/to/jar/*/*.jar</code></li>
<li><code>hdfs://nameservice/path/to/jar/*,hdfs://nameservice2/path/to/jar/*/*.jar</code></li>
</ol>
</td>
<td>3.1.0</td>
</tr>
<tr>
<td><code>spark.sql.hive.metastore.sharedPrefixes</code></td>
<td><code>com.mysql.jdbc,<br/>org.postgresql,<br/>com.microsoft.sqlserver,<br/>oracle.jdbc</code></td>
Expand Down

0 comments on commit 5b16d70

Please sign in to comment.