Skip to content

eleflow/pyspark-connectors

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

46 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PySpark Conectors - by: Eleflow BigData

Build and Publish on PyPI

This library provides many connections and integrations with another data sources.

Index

Installing

pip install pyspark-connectors

Development enviroment

For develop you must guarantee that you have the Python (3.8 or higher) and Spark (3.1.2 or higher) installed, if you have ready the minimum environment for development in Python language, proceed with these steps:

# Clonning the project
$ git clone [email protected]:eleflow/pyspark-connectors.git

# Inside of the project root folder
$ python -m venv .env

# If Windows
$ .\.env\Script\Activate.ps1 
# If linux dist
$ .\.env\Scripts\activate

# Installing requirements libraries
(.env) $ pip install -r .\requirements.txt

Packaging project in a .whl lib

# Installing wheel package
(.env) $ pip install wheel

# Installing wheel contents
(.env) $ pip install check-wheel-contents

# Build and packaging project to .whl
(.env) $ python setup.py bdist_wheel

Library development status

Connectors

  • Google Sheets
  • Rest API
  • SQL Database
  • CosmosDB
  • Elasticsearch
  • PipeDrive
  • ActiveCampaign
  • ReclameAqui
  • Jira

Helpers and Utils

  • AWS Secrets Manager

Version history

Version Date Changes Notes Approved by
0.0.1a2 2022-05-08 Initial development release N/A @caiodearaujo
0.1.0 2022-06-01 Initial release N/A @caiodearaujo
0.2.0 2022-07-28 New release with connectors stable N/A @caiodearaujo
0.3.0 2024-06-10 Fix - Pipedrive Person Service N/A @cehira