Skip to content

Latest commit

 

History

History
65 lines (42 loc) · 1.89 KB

data.md

File metadata and controls

65 lines (42 loc) · 1.89 KB

Data

Big Data

See big-data.md

Data Validation

Start by validating data formats for correctness.

Scripts for this can be found in both the DevOps-Python-tools and DevOps-Bash-tools repos.

Then proceed to more advanced content validation.

Data Integration

  • DBT - open-source data pipeline workflow tool
  • Informatica - proprietary legacy now available via SaaS, with self-hosted agents on VMs or Kubernetes
  • Airbyte - open source self-hosted or SaaS proprietary with 300+ connectors
  • Apache Camel - open source with 100+ connectors
  • Spring Integration - XML config, only use for Spring heavy shops
  • Mulesoft - XML config, only use for proprietary connectors

Mulesoft

  • lightweight enterprise service bus + integration framework
  • proprietary connectors
  • Anypoint Studio (Eclipse-based IDE)
  • Anypoint Enterprise Security - security features, transactions

Spring Integration

TODO

Data Visualization

See the Diagrams and Visualization docs.

Diagrams

Top 9 Systems Integrations

Encoding vs Encryption vs Tokenization

Encoding vs Encryption vs Tokenization

Ported from private Knowledge Base pages 2016+