This is a public repository to go over all the LLM-driven data engineering concepts.

Possibly the fastest DataFrame-agnostic quality check library in town.

Dynamic RAG for enterprise. Ready to run with Docker,⚡in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.

🤖 𝗟𝗲𝗮𝗿𝗻 for 𝗳𝗿𝗲𝗲 how to 𝗯𝘂𝗶𝗹𝗱 an end-to-end 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗿𝗲𝗮𝗱𝘆 𝗟𝗟𝗠 & 𝗥𝗔𝗚 𝘀𝘆𝘀𝘁𝗲𝗺 using 𝗟𝗟𝗠𝗢𝗽𝘀 best practices: ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 11 𝘩𝘢𝘯𝘥𝘴-𝘰𝘯 𝘭𝘦𝘴𝘴𝘰𝘯𝘴

Delta Lake examples

This repository hosts the code that's demos Apache Phoenix connector for Apache Spark.

Free MLOps course from DataTalks.Club

Code samples, etc. for Databricks

Quix Streams - A library for data streaming and Python Stream Processing

A list of all my posts and personal projects

A resource to help you learn system design.

A minimal docker compose setup for experimenting with cloud agnostic Lakehouse Architectures Apache Spark with Hive Metastore + Delta Lake + MinIO

Playing with different packages of the Apache Spark

Designed an efficient banking database using PostgreSQL and pgAdmin. Focused on optimizing transaction management included key aspects like data modeling, database architecture, security, and perfo…

Pyspark Dataplatform

Examples of using Terraform to deploy Databricks resources

Repository to practise Infrastructure-As-Code (IAC) with Github Actions, AWS and Terraform

A dashboard web app template built in Python using Streamlit.

A list of Terraform modules to build your Azure Data IaC templates.

Multi-container environment with Hadoop, Spark and Hive

Sample code and notebooks for Generative AI on Google Cloud

A Docker Compose template that builds a interactive development environment for PySpark with Jupyter Lab, MinIO as object storage, Hive Metastore, Trino and Kafka

DataAnonymizer is an open-source personal data anonymization tool designed for GDPR compliancy