data-pipelines

Apache DolphinScheduler is the modern data workflow orchestration platform with powerful user interface, dedicated to solving complex task dependencies in the data pipeline and providing various types of jobs available `out of the box`

An orchestration platform for the development, production, and observation of data assets.

Build data pipelines, the easy way 🛠️

🧙 A modern replacement for Airflow.

MLeap: Deploy ML Pipelines to Production

Open-source data observability for analytics engineers.

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

First open-source data discovery and observability platform. ODD Platform is based on ODD Specification.

Your CLI for ELT+. It's open source, flexible, scales to your needs. Confidently move, transform, and test your data using tools you know with a data engineering workflow you’ll love.

The best place to learn data engineering. Built and maintained by the data engineering community.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Dataplane is an Airflow inspired data platform with additional data mesh capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

Pipebird is open source infrastructure for securely sharing data with customers.

Relational data pipelines for the science lab

A lightweight CLI tool for versioning data alongside source code and building data pipelines.

An Open Source PHP Reporting Framework that helps you to write perfect data reports or to construct awesome dashboards in PHP. Working great with all PHP versions from 5.6 to latest 8.0. Fully compatible with all kinds of MVC frameworks like Laravel, CodeIgniter, Symfony.

Cloud-native, data onboarding architecture for Google Cloud Datasets

Data anomalies monitoring as dbt tests and dbt artifacts uploader.

Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3