data-pipeline

Here are 380 public repositories matching this topic...

snowplow / snowplow

The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP

data analytics snowplow data-collection data-pipeline product-analytics marketing-analytics snowplow-pipeline snowplow-events

Updated Sep 2, 2022
Scala

apache / incubator-seatunnel

Star

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

real-time offline high-performance apache data-integration sql-engine data-pipeline etl-framework seatunnel

Updated Sep 5, 2022
Java

kestra-io / kestra

Star

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.

java kubernetes yaml workflow data kafka scale etl workflow-engine scheduler orchestration data-engineering elt data-pipeline low-code data-orchestration kestra data-orchestrator

Updated Sep 2, 2022
Java

adilkhash / Data-Engineering-HowTo

Star

A list of useful resources to learn Data Engineering from scratch

distributed-systems scala cloud-providers data-engineering data-pipeline

Updated Jun 28, 2022

whylabs / whylogs

Star

The open standard for data logging

python data-science machine-learning analytics logging constraints dataset dataops data-pipeline data-quality calculate-statistics data-constraints mlops model-performance ml-pipelines ai-pipelines approximate-statistics statistical-properties

Updated Sep 2, 2022
Jupyter Notebook

pydoit / doit

Star

task management & automation tool

python workflow data-science build-automation task-runner build-tool build-system workflow-management data-pipeline workflow-automation

Updated Aug 21, 2022
Python

GoogleCloudPlatform / data-science-on-gcp

Star

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

data-science machine-learning data-visualization data-engineering cloud-computing data-analysis data-processing data-pipeline

Updated May 24, 2022
Jupyter Notebook

reugn / go-streams

Star

A lightweight stream processing library for Go

Updated Sep 4, 2022
Go

spotify / klio

Star

Smarter data pipelines for audio.

signal-processing data-pipeline audio-processing media-processing

Updated Mar 23, 2022
Python

elementary-data / elementary

Star

Open-source data observability for analytics engineers

bigquery snowflake data-warehouse dataops data-analysis redshift dbt data-pipelines data-pipeline lineage data-governance data-lineage analytics-engineer dbt-packages data-observability analaytics-engineering data-reliability dbt-artifacts

Updated Sep 4, 2022
HTML

damklis / DataEngineeringProject

Star

Example end to end data engineering project.

python redis elasticsearch airflow kafka big-data mongodb scraping django-rest-framework s3 data-engineering minio kafka-connect hacktoberfest data-pipeline debezium

Updated Jul 6, 2022
Python

infoslack / awesome-kafka

Star

A list about Apache Kafka

infrastructure kafka apache-spark stream-processing apache-kafka kafka-streams data-processing data-pipeline streaming-data

Updated Dec 21, 2021

streamlet-dev / tributary

Star

Streaming reactive and dataflow graphs in Python

python streaming kafka stream asynchronous websockets python3 lazy-evaluation data-pipeline reactive-data-streams python-data-streams

Updated Jul 5, 2022
Python

msamogh / nonechucks

Star

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

machine-learning torch pytorch data-preprocessing preprocessing data-processing data-cleaning data-pipeline

Updated Jun 9, 2021
Python

cuebook / cuelake

Star

Use SQL to build ELT pipelines on a data lakehouse.

sql apache-spark etl pipelines data-engineering data-lake data-transfer delta data-integration upsert elt data-pipeline datalake data-ingestion spark-sql zeppelin-notebook apache-iceberg lakehouse incremental-updates

Updated May 25, 2022
JavaScript

memphisdev / memphis-broker

Star

A powerful messaging platform for modern developers

kubernetes golang data data-engineering data-pipeline data-streaming data-stream-processing messaging-queue

Updated Sep 4, 2022
Go

ConduitIO / conduit

Star

Data Integration for Production Data Stores.

go kafka etl data-stream data-engineering conduit data-integration data-pipeline

Updated Sep 3, 2022
Go

ubisoft / mobydq

Star

🐳 Tool to automate data quality checks on data pipelines

big-data data-warehouse data-quality-checks data-pipeline data-quality data-quality-monitoring

Updated May 10, 2022
Vue

olirice / flupy

Star

Fluent data pipelines for python and your shell

python collections fluent data-pipeline

Updated Feb 18, 2022
Python

unnati-xyz / scalable-data-science-platform

Star

Content for architecting a data science platform for products using Luigi, Spark & Flask.

data-science machine-learning spark rest-api luigi data-pipeline data-engineer

Updated Jan 27, 2020
Jupyter Notebook

Improve this page

Add a description, image, and links to the data-pipeline topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the data-pipeline topic, visit your repo's landing page and select "manage topics."

Learn more