Skip to content

Latest commit

 

History

History
241 lines (195 loc) · 15.7 KB

connector-file-system.md

File metadata and controls

241 lines (195 loc) · 15.7 KB
title description services documentationcenter author manager ms.reviewer ms.service ms.workload ms.tgt_pltfrm ms.devlang ms.topic ms.date ms.author
Copy data from/to a file system by using Azure Data Factory | Microsoft Docs
Learn how to copy data from file system to supported sink data stores (or) from supported source data stores to file system by using Azure Data Factory.
data-factory
linda33wj
craigg
douglasl
data-factory
data-services
na
na
conceptual
11/15/2018
jingwang

Copy data to or from a file system by using Azure Data Factory

[!div class="op_single_selector" title1="Select the version of Data Factory service you are using:"]

This article outlines how to use the Copy Activity in Azure Data Factory to copy data from and to file system. It builds on the copy activity overview article that presents a general overview of copy activity.

Supported capabilities

You can copy data from file system to any supported sink data store, or copy data from any supported source data store to file system. For a list of data stores that are supported as sources/sinks by the copy activity, see the Supported data stores table.

Specifically, this file system connector supports:

  • Copying files from/to local machine or network file share. To use a Linux file share, install Samba on your Linux server.
  • Copying files using Windows authentication.
  • Copying files as-is or parsing/generating files with the supported file formats and compression codecs.

Prerequisites

To copy data from/to a file system that is not publicly accessible, you need to set up a Self-hosted Integration Runtime. See Self-hosted Integration Runtime article for details.

Getting started

[!INCLUDE data-factory-v2-connector-get-started]

The following sections provide details about properties that are used to define Data Factory entities specific to file system.

Linked service properties

The following properties are supported for file system linked service:

Property Description Required
type The type property must be set to: FileServer. Yes
host Specifies the root path of the folder that you want to copy. Use the escape character "" for special characters in the string. See Sample linked service and dataset definitions for examples. Yes
userid Specify the ID of the user who has access to the server. Yes
password Specify the password for the user (userid). Mark this field as a SecureString to store it securely in Data Factory, or reference a secret stored in Azure Key Vault. Yes
connectVia The Integration Runtime to be used to connect to the data store. You can use Self-hosted Integration Runtime or Azure Integration Runtime (if your data store is publicly accessible). If not specified, it uses the default Azure Integration Runtime. No

Sample linked service and dataset definitions

Scenario "host" in linked service definition "folderPath" in dataset definition
Local folder on Integration Runtime machine:

Examples: D:\* or D:\folder\subfolder\*		 In JSON: D:\\
On UI: D:\		 In JSON: .\\ or folder\\subfolder
On UI: .\ or folder\subfolder
Remote shared folder:

Examples: \\myserver\share\* or \\myserver\share\folder\subfolder\*		 In JSON: \\\\myserver\\share
On UI: \\myserver\share		 In JSON: .\\ or folder\\subfolder
On UI: .\ or folder\subfolder

Note

When authoring via UI, you don't need to input double backslash (\\) to escape like you do via JSON, specify single backslash.

Example:

{
    "name": "FileLinkedService",
    "properties": {
        "type": "FileServer",
        "typeProperties": {
            "host": "<host>",
            "userid": "<domain>\\<user>",
            "password": {
                "type": "SecureString",
                "value": "<password>"
            }
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

Dataset properties

For a full list of sections and properties available for defining datasets, see the datasets article. This section provides a list of properties supported by file system dataset.

To copy data from/to file system, set the type property of the dataset to FileShare. The following properties are supported:

Property Description Required
type The type property of the dataset must be set to: FileShare Yes
folderPath Path to the folder. Wildcard filter is not supported. See Sample linked service and dataset definitions for examples. No
fileName Name or wildcard filter for the file(s) under the specified "folderPath". If you don't specify a value for this property, the dataset points to all files in the folder.

For filter, allowed wildcards are: * (matches zero or more characters) and ? (matches zero or single character).
- Example 1: "fileName": "*.csv"
- Example 2: "fileName": "???20180427.txt"
Use ^ to escape if your actual file name has wildcard or this escape char inside.

When fileName isn't specified for an output dataset and preserveHierarchy isn't specified in the activity sink, the copy activity automatically generates the file name with the following format: "Data.[activity run id GUID].[GUID if FlattenHierarchy].[format if configured].[compression if configured]". An example is "Data.0a405f8a-93ff-4c6f-b3be-f69616f1df7a.txt.gz".		 No
format If you want to copy files as-is between file-based stores (binary copy), skip the format section in both input and output dataset definitions.

If you want to parse or generate files with a specific format, the following file format types are supported: TextFormat, JsonFormat, AvroFormat, OrcFormat, ParquetFormat. Set the type property under format to one of these values. For more information, see Text Format, Json Format, Avro Format, Orc Format, and Parquet Format sections.		 No (only for binary copy scenario)
compression Specify the type and level of compression for the data. For more information, see Supported file formats and compression codecs.
Supported types are: GZip, Deflate, BZip2, and ZipDeflate.
Supported levels are: Optimal and Fastest.		 No

Tip

To copy all files under a folder, specify folderPath only.
To copy a single file with a given name, specify folderPath with folder part and fileName with file name.
To copy a subset of files under a folder, specify folderPath with folder part and fileName with wildcard filter.

Note

If you were using "fileFilter" property for file filter, it is still supported as-is, while you are suggested to use the new filter capability added to "fileName" going forward.

Example:

{
    "name": "FileSystemDataset",
    "properties": {
        "type": "FileShare",
        "linkedServiceName":{
            "referenceName": "<file system linked service name>",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "folderPath": "folder/subfolder/",
            "fileName": "myfile.csv.gz",
            "format": {
                "type": "TextFormat",
                "columnDelimiter": ",",
                "rowDelimiter": "\n"
            },
            "compression": {
                "type": "GZip",
                "level": "Optimal"
            }
        }
    }
}

Copy activity properties

For a full list of sections and properties available for defining activities, see the Pipelines article. This section provides a list of properties supported by file system source and sink.

File system as source

To copy data from file system, set the source type in the copy activity to FileSystemSource. The following properties are supported in the copy activity source section:

Property Description Required
type The type property of the copy activity source must be set to: FileSystemSource Yes
recursive Indicates whether the data is read recursively from the sub-folders or only from the specified folder. Note when recursive is set to true and sink is file-based store, empty folder/sub-folder will not be copied/created at sink.
Allowed values are: true (default), false		 No

Example:

"activities":[
    {
        "name": "CopyFromFileSystem",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<file system input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "FileSystemSource",
                "recursive": true
            },
            "sink": {
                "type": "<sink type>"
            }
        }
    }
]

File system as sink

To copy data to file system, set the sink type in the copy activity to FileSystemSink. The following properties are supported in the sink section:

Property Description Required
type The type property of the copy activity sink must be set to: FileSystemSink Yes
copyBehavior Defines the copy behavior when the source is files from file-based data store.

Allowed values are:
- PreserveHierarchy (default): preserves the file hierarchy in the target folder. The relative path of source file to source folder is identical to the relative path of target file to target folder.
- FlattenHierarchy: all files from the source folder are in the first level of target folder. The target files have auto generated name.
- MergeFiles: merges all files from the source folder to one file. If the File/Blob Name is specified, the merged file name would be the specified name; otherwise, would be auto-generated file name.		 No

Example:

"activities":[
    {
        "name": "CopyToFileSystem",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<file system output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "<source type>"
            },
            "sink": {
                "type": "FileSystemSink",
                "copyBehavior": "PreserveHierarchy"
            }
        }
    }
]

recursive and copyBehavior examples

This section describes the resulting behavior of the Copy operation for different combinations of recursive and copyBehavior values.

recursive copyBehavior Source folder structure Resulting target
true preserveHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5		 The target folder Folder1 is created with the same structure as the source:

Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5.
true flattenHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5		 The target Folder1 is created with the following structure:

Folder1
    auto-generated name for File1
    auto-generated name for File2
    auto-generated name for File3
    auto-generated name for File4
    auto-generated name for File5
true mergeFiles Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5		 The target Folder1 is created with the following structure:

Folder1
    File1 + File2 + File3 + File4 + File 5 contents are merged into one file with auto-generated file name
false preserveHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5		 The target folder Folder1 is created with the following structure

Folder1
    File1
    File2

Subfolder1 with File3, File4, and File5 are not picked up.
false flattenHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5		 The target folder Folder1 is created with the following structure

Folder1
    auto-generated name for File1
    auto-generated name for File2

Subfolder1 with File3, File4, and File5 are not picked up.
false mergeFiles Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5		 The target folder Folder1 is created with the following structure

Folder1
    File1 + File2 contents are merged into one file with auto-generated file name. auto-generated name for File1

Subfolder1 with File3, File4, and File5 are not picked up.

Next steps

For a list of data stores supported as sources and sinks by the copy activity in Azure Data Factory, see supported data stores.