The pyEGA3 download client is a python-based tool for viewing and downloading files from authorized EGA datasets. pyEGA3 uses the EGA Data API and has several key features:
- Files are transferred over secure https connections and received unencrypted, so no need for decryption after download.
- Downloads resume from where they left off in the event that the connection is interrupted.
- pyEGA3 supports file segmenting and parallelized download of segments, improving overall performance.
- After download completes, file integrity is verified using checksums.
- pyEGA3 implements the GA4GH-compliant htsget protocol for download of genomic ranges for data files with accompanying index files.
A video tutorial demonstrating the usage of pyEGA3 from installation through file download is available here.
- Python 3.6 or newer. (download instructions)
pyEGA3 makes https calls to the EGA AAI (https://ega.ebi.ac.uk:8443) and the EGA Data API (https://ega.ebi.ac.uk:8052). Ports 8443 and 8052 must both be reachable from the location where pyEGA3 is executed to avoid timeouts.
For Linux/Mac users, check if ports 8443 and 8052 are open by running the following commands:
openssl s_client -connect ega.ebi.ac.uk:8443
openssl s_client -connect ega.ebi.ac.uk:8052If the ports are open, the commands should print CONNECTED to the terminal.
For Windows users, check if ports 8443 and 8052 are open by going to the following URLs:
- https://ega.ebi.ac.uk:8443/ega-openid-connect-server/
- https://ega.ebi.ac.uk:8052/elixir/central/stats/load
If the ports are open, both of the sites should load with no timeouts.
-
Install pyEGA3 using pip3.
sudo pip3 install pyega3
-
Update pyEGA3, if needed, using pip3.
pip3 install pyega3 --upgrade
-
Test your pip3 installation by running pyEGA3.
pyega3 --help
-
Install pyEGA3 using conda.
conda config --add channels bioconda conda config --add channels conda-forge conda install pyega3
-
Update pyEGA3, if needed, using conda.
conda update pyega3
-
Test your conda installation by running pyEGA3.
pyega3 --help
-
Clone the ega-download-client GitHub repository.
-
Navigate to the directory where the repository was cloned.
cd path/to/ega-download-client -
Three scripts are provided to install the required Python environment depending on the host operating system.
- Linux (Red Hat): red_hat_dependency_install.sh
- Linux: debian_dependency_install.sh
- macOS: osx_dependency_install.sh
-
Execute the script corresponding to the host operating system. For example, if using Red Hat Linux, run:
sh red_hat_dependency_install.sh
-
Test your GitHub installation by running pyEGA3.
python -m pyega3.pyega3 --help
There are Docker images built by Bioconda: https://bioconda.github.io/recipes/pyega3/README.html An example of running pyEGA3 in a Docker container:
docker run --rm -v /tmp:/app -w /app quay.io/biocontainers/pyega3:3.4.0--py_0 pyega3 -d -t fetch EGAF00001775036This example command mounts your /tmp folder into the Docker container as /app, starts the 3.4.0 version of pyEGA3 and downloads a test file. The test file will be downloaded into your /tmp folder. You can find other, possibly newer, versions ("tags") of the pyEGA3 Docker image on the above-mentioned Bioconda page.
usage: pyega3.py [-h] [-d] [-cf CONFIG_FILE] [-sf SERVER_FILE] [-c CONNECTIONS] [-t] [-ms MAX_SLICE_SIZE] {datasets,files,fetch} ...
Download from EMBL EBI's EGA (European Genome-phenome Archive)
positional arguments:
{datasets,files,fetch}
subcommands
datasets List authorized datasets
files List files in a specified dataset
fetch Fetch a dataset or file
optional arguments:
-h, --help show this help message and exit
-d, --debug Extra debugging messages
-cf CONFIG_FILE, --config-file CONFIG_FILE
JSON file containing credentials/config e.g.{"username":"user1","password":"toor"}
-sf SERVER_FILE, --server-file SERVER_FILE
JSON file containing server config e.g.{"url_auth":"aai url","url_api":"api url", "url_api_ticket":"htsget url", "client_secret":"client secret"}
-c CONNECTIONS, --connections CONNECTIONS
Download using specified number of connections (default: 1 connection)
-t, --test Test user activated
-ms MAX_SLICE_SIZE, --max-slice-size MAX_SLICE_SIZE
Set maximum size for each slice in bytes (default: 100 MB)
We recommend that all fresh installations of pyEGA3 be tested. A test account has been created which can be used (-t) to test the following pyEGA3 actions:
pyega3 -d -t datasetspyega3 -d -t files EGAD00001003338pyega3 -d -t fetch EGAF00001775036The test dataset (EGAD00001003338) is large (almost 1TB), so please be mindful if deciding to test downloading the entire dataset. The test account does not require an EGA username and password because it contains publicaly accessible files from the 1000 Genomes Project. The files in the test dataset can be used for troubleshooting and training purposes.
To view and download files for which you have been granted access, pyEGA3 requires your EGA username (email address) and password saved to a credentials file.
Create a file called CREDENTIALS_FILE and place it in the directory where pyEGA3 will run. The credentials file must be in JSON format and must contain your registered EGA username (email address) and password provided by EGA Helpdesk.
An example CREDENTIALS_FILE is available here.
Replace <these values> with values relevant for your datasets.
pyega3 -cf </Path/To/CREDENTIALS_FILE> datasetspyega3 -cf </Path/To/CREDENTIALS_FILE> files EGAD<NUM>pyega3 -cf </Path/To/CREDENTIALS_FILE> fetch EGAD<NUM> --output-dir </Path/To/OutputDirectory>pyega3 -cf </Path/To/CREDENTIALS_FILE> fetch EGAF<NUM> --output-dir </Path/To/OutputDirectory>pyega3 -cf </Path/To/CREDENTIALS_FILE> files EGAD<NUM>nohup pyega3 -cf </Path/To/CREDENTIALS_FILE> files EGAD<NUM> </Path/To/File/md5sums.txt>pyega3 -c 5 -cf </Path/To/CREDENTIALS_FILE> fetch EGAD<NUM> --output-dir </Path/To/OutputDirectory>usage: pyega3 fetch [-h] [--reference-name REFERENCE_NAME]
[--reference-md5 REFERENCE_MD5] [--start START]
[--end END] [--format {BAM,CRAM}]
[--max-retries MAX_RETRIES] [--retry-wait RETRY_WAIT]
[--output-dir OUTPUT_DIR] [--delete-temp-files]
identifier
positional arguments:
identifier Id for dataset (e.g. EGAD00000000001) or file (e.g.
EGAF12345678901)
optional arguments:
-h, --help show this help message and exit
--reference-name REFERENCE_NAME, -r REFERENCE_NAME
The reference sequence name, for example 'chr1', '1',
or 'chrX'. If unspecified, all data is returned.
--reference-md5 REFERENCE_MD5, -m REFERENCE_MD5
The MD5 checksum uniquely representing the requested
reference sequence as a lower-case hexadecimal string,
calculated as the MD5 of the upper-case sequence
excluding all whitespace characters.
--start START, -s START
The start position of the range on the reference,
0-based, inclusive. If specified, reference-name or
reference-md5 must also be specified.
--end END, -e END The end position of the range on the reference,
0-based exclusive. If specified, reference-name or
reference-md5 must also be specified.
--format {BAM,CRAM}, -f {BAM,CRAM}
The format of data to request.
--max-retries MAX_RETRIES, -M MAX_RETRIES
The maximum number of times to retry a failed
transfer. Any negative number means infinite number of
retries.
--retry-wait RETRY_WAIT, -W RETRY_WAIT
The number of seconds to wait before retrying a failed
transfer.
--output-dir OUTPUT_DIR
Output directory. The files will be saved into this directory. Must exist. Default: the current working directory.
--delete-temp-files Do not keep those temporary, partial files which were
left on the disk after a failed transfer.Replace <these values> with values relevant for your datasets. Please note that htsget can only be used with files that have corresponding index files in EGA.
pyega3 fetch -cf </Path/To/CREDENTIALS_FILE> --reference-name 1 --format BAM --output-dir </Path/To/OutputDirectory> EGAF<NUM>pyega3 fetch -cf </Path/To/CREDENTIALS_FILE> --start 0 --end 1000000 --reference-name 1 --format BAM --output-dir </Path/To/OutputDirectory> EGAF<NUM>First, please ensure you are using the most recent version of pyEGA3 by following instructions in the "Installation and update" section for updating pyEGA3.
Please ensure that your credentials are formatted correctly. Email addresses (usernames) are case-sensitive. If you have an EGA submission account, these credentials are different from your data access credentials. Please ensure you are using your data access credentials with pyEGA3.
Download speed can be optimized using the --connections parameter which will parallelize download at the file level. If the --connections parameter is provided, all files >100Mb will be downloaded using the specified number of parallel connections.
Using a very high number of connections will introduce overhead that can slow the download of the file. It is important to note that files are still downloaded sequentially, so using multiple connections does not mean downloading multiple files in parallel. We recommend trying with 30 connections initially and adjusting from there to get maximum throughput.
Please note that when a file is being saved, it goes through two processes. First, the downloaded file "chunks" are pieced back together to reconstruct the original file. Second, pyEGA3 calculates the checksum of the file to confirm the file downloaded successfully. Larger files will take more time to reconstruct and validate the checksum.
If, after troubleshooting an issue, you are still experiencing difficulties, please email EGA Helpdesk ([email protected]) with the following information:
- Attach the log file (pyega3_output.log) located in the directory where pyEGA3 is running
- Indicate the compute environment you are running pyEGA3 in: compute cluster, single machine, other (please describe).
Parts of pyEGA3 are derived from pyEGA developed by James Blachly.