#

data-processing

Here are 447 public repositories matching this topic...

lorien / awesome-web-scraping

Star

List of libraries, tools and APIs for web scraping and data processing.

Updated Dec 3, 2020
Makefile

NVIDIA / DALI

Star

A library containing both highly optimized building blocks and an execution engine for data pre-processing in deep learning applications

python machine-learning deep-learning neural-network gpu image-processing gpu-tensorflow data-processing data-augmentation image-augmentation fast-data-pipeline

Updated Dec 7, 2020
C++

miller

johnkerl / miller

Star

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Updated Dec 6, 2020
C

asyml / texar

Star

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

python machine-learning natural-language-processing deep-learning tensorflow machine-translation text-generation data-processing bert text-data dialog-systems gpt-2 texar xlnet casl-project

Updated Sep 17, 2020
Python

python-bonobo / bonobo

Star

Extract Transform Load for Python 3.5+

automation parallelization python3 data-processing bonobo extract-transform-load

Updated Nov 18, 2020
Python

Bash-Oneliner

onceupon / Bash-Oneliner

Star

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

linux shell bash terminal system hardware grep data-processing variables xargs xwindow one-liners linux-administration oneliner-commands shell-oneliner

Updated Sep 8, 2020

dashbitco / broadway

Star

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir concurrent data-processing genstage data-ingestion

Updated Nov 29, 2020
Elixir

microsoft / DialoGPT

Star

Large-scale pretraining for dialogue

machine-learning dialogue text-generation pytorch transformer data-processing text-data gpt-2 dialogpt

Updated Dec 4, 2020
Python

GoogleCloudPlatform / DataflowJavaSDK

Star

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

data-science data-mining big-data data-analysis google-cloud-dataflow data-processing

Updated Nov 25, 2020

GoogleCloudPlatform / data-science-on-gcp

Star

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

data-science machine-learning data-visualization data-engineering cloud-computing data-analysis data-processing data-pipeline

Updated Oct 14, 2020
Jupyter Notebook

asyml / texar-pytorch

Star

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/

python machine-learning natural-language-processing deep-learning machine-translation text-generation pytorch data-processing bert text-data dialog-systems roberta gpt-2 texar xlnet casl-project texar-pytorch

Updated Nov 23, 2020
Python

pandera-dev / pandera

Star

Open

str and repr of MultiIndex schema component should be unique

cosmicBboy commented Dec 3, 2020

Is your feature request related to a problem? Please describe.

Currently, the MultiIndex schema component str representation is the same as
the DataFrameSchema representation

Describe the solution you'd like

the MultiIndex schema component should implement its own str representation
so that it doesn't render columns and instead shoulds the indexes.

Read more

enhancement good first issue

Open

Replace instances of pa.PandasDtype enum with python, numpy, pandas types

Open

str and repr method of DataFrameSchema should display all attributes

Find more good first issues →

infoslack / awesome-kafka

Star

A list about Apache Kafka

infrastructure kafka apache-spark stream-processing apache-kafka kafka-streams data-processing data-pipeline streaming-data

Updated Nov 17, 2020

benibela / xidel

Star

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

Updated Nov 25, 2020
Pascal

eternal

kousun12 / eternal

Sponsor Star

👾~ music, eternal ~ 👾

art music webgl creative-coding midi glsl webaudio data-processing 3d-graphics node-based

Updated Nov 26, 2020
JavaScript

msamogh / nonechucks

Star

Open

Write tests

msamogh commented Mar 18, 2019

Write unit test coverage for SafeDataset and SafeDataLoader, along with the functions in utils.py.

Read more

good first issue

Open

Route next batch through step_to_index_fn in _SafeDataLoaderIter

alttch / rapidtables

Star

Super fast list of dicts to pre-formatted tables conversion library for Python 2/3

python library text-formatting python3 data-processing dictionary-data

Updated Nov 17, 2019
Python

Yord / pxi

Star

🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.

tsv parser json csv serializer ssv data-processing dsv marshaller deserializer pixie pxi

Updated Aug 24, 2020
JavaScript

pysparkling

svenkreiss / pysparkling

Star

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

python data-science apache-spark data-processing

Updated Nov 2, 2020
Python

constellation-rs / amadeus

Star

Harmonious distributed data analysis in Rust.

rust parallel-computing distributed-computing stream-processing data-analysis data-processing

Updated Nov 25, 2020
Rust

maykulkarni / Machine-Learning-Notebooks

Star

Machine Learning notebooks for refreshing concepts.

python machine-learning natural-language-processing reinforcement-learning deep-learning machine-learning-algorithms neural-networks deep-learning-algorithms dimensionality-reduction python-machine-learning data-processing regression-models deep-learning-tutorial data-science-notebook model-evaluation classification-trees clustering-methods machine-learning-tutorials

Updated Oct 31, 2018
Jupyter Notebook

PytLab / VASPy

Star

Manipulating VASP files with Python.

chemistry data-visualization vasp computational-chemistry data-processing catalysis-informatics

Updated Jul 29, 2020
Python

matousc89 / padasip

Star

Python Adaptive Signal Processing

machine-learning signal-processing neural-networks data-analysis data-processing adaptive-filtering

Updated Mar 26, 2018
Python

SebKrantz / collapse

Star

Advanced and Fast Data Transformation in R

data-science cran r statistics time-series high-performance data-transformation scientific-computing econometrics rstats data-analysis data-manipulation data-processing weights panel-data weighted data-aggregation

Updated Nov 25, 2020
R

distributed-dataset

utdemir / distributed-dataset

Star

Open

Concurrent executors

utdemir commented Apr 14, 2019

Run read and write ends of the conduits concurrently.
When reading a Partition created by <> operations, consume the smaller partitions in parallel.

Read more

good first issue performance

tollwerk / data-processing-agreements

Star

Collection of Data Processing Agreement (DPA) and GDPR compliance resources

data privacy compliance data-processing gdpr dpa ethics adv dsgvo auftragsdatenverarbeitung

Updated Mar 29, 2019
CSS

thu-coai / cotk

Star

Conversational Toolkit. An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation

python machine-learning natural-language-processing deep-learning metrics data-processing natural-language-generation text-data cotk

Updated Aug 31, 2020
Python

streamnative / pulsar-flink

Star

Elastic data processing with Apache Pulsar and Apache Flink

schema sql catalog schema-registry stream-processing apache-flink flink data-processing batch-processing flink-stream-processing apache-pulsar flink-connector

Updated Dec 6, 2020
Java

ColasGael / Machine-Learning-for-Solar-Energy-Prediction

Star

Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning

python machine-learning neural-network tensorflow matlab data-processing

Updated Nov 7, 2019
Python

classtag / ijcai18-mama-ads-competition

Star

IJCAI-18 阿里妈妈搜索广告转化预测初赛方案

admin lightgbm data-processing ctr tianchi catboost ijcai-18

Updated Apr 22, 2018
Jupyter Notebook

Improve this page

Add a description, image, and links to the data-processing topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the data-processing topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.