A Python library that implements software engineering best-practice for data and ML pipelines.

A list of useful resources to learn Data Engineering from scratch

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

Go stream processing library

A list about Apache Kafka

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

Python Data Streams

Content for architecting a data science platform for products using Luigi, Spark & Flask.

Data transformations for the ML era

🐳 Tool to automate data quality checks on data pipelines

🔍 Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.

Tools for ASR Corpus Generation from Online Video

A cross-platform tool for data pipelines.

Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena

Outbrain's data pipeline framework

一个基于浏览器环境的数据采集SDK

Model complex data transformation pipelines easily

(deprecated) A fast and memory-efficient Python data engineering framework for machine learning.

Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

A Machine Learning and Real-Time Data Analytics Workshop

Framework which helps you to make parallel/distributed calculations using data pipelines

OONI data processing pipeline

Copy Pandas DataFrames and HDF5 files to PostgreSQL database

A web service for disambiguating and canonically storing entities.

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

Network traffic data pipeline for real-time predictions and building datasets for deep neural networks

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Blog post on ETL pipelines with Airflow