#

data-processing

Here are 425 public repositories matching this topic...

lorien / awesome-web-scraping

Star

List of libraries, tools and APIs for web scraping and data processing.

Updated Oct 14, 2020
Makefile

NVIDIA / DALI

Star

A library containing both highly optimized building blocks and an execution engine for data pre-processing in deep learning applications

python machine-learning deep-learning neural-network gpu image-processing gpu-tensorflow data-processing data-augmentation image-augmentation fast-data-pipeline

Updated Oct 15, 2020
C++

miller

johnkerl / miller

Star

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Updated Oct 15, 2020
C

asyml / texar

Star

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow

python machine-learning natural-language-processing deep-learning tensorflow machine-translation text-generation data-processing bert text-data dialog-systems gpt-2 texar xlnet

Updated Sep 17, 2020
Python

python-bonobo / bonobo

Star

Extract Transform Load for Python 3.5+

automation parallelization python3 data-processing bonobo extract-transform-load

Updated May 1, 2020
Python

Bash-Oneliner

onceupon / Bash-Oneliner

Star

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

linux shell bash terminal system hardware grep data-processing variables xargs xwindow one-liners linux-administration oneliner-commands shell-oneliner

Updated Sep 8, 2020

dashbitco / broadway

Star

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir concurrent data-processing genstage data-ingestion

Updated Oct 5, 2020
Elixir

microsoft / DialoGPT

Star

Large-scale pretraining for dialogue

machine-learning dialogue text-generation pytorch transformer data-processing text-data gpt-2 dialogpt

Updated Sep 20, 2020
Python

GoogleCloudPlatform / DataflowJavaSDK

Star

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

data-science data-mining big-data data-analysis google-cloud-dataflow data-processing

Updated Dec 19, 2018

GoogleCloudPlatform / data-science-on-gcp

Star

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

data-science machine-learning data-visualization data-engineering cloud-computing data-analysis data-processing data-pipeline

Updated Oct 14, 2020
Jupyter Notebook

asyml / texar-pytorch

Star

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation

python machine-learning natural-language-processing deep-learning machine-translation text-generation pytorch data-processing bert text-data dialog-systems roberta gpt-2 texar xlnet

Updated Sep 29, 2020
Python

pandera-dev / pandera

Star

Open

Add support for "str" alias in PandasDtype.from_str_alias

cosmicBboy commented Sep 30, 2020

Is your feature request related to a problem? Please describe.

Relating to #278, the problem is that yaml file schemas don't accept the value str for pandas_dtype argument.

Describe the solution you'd like
Add the "str" string alias to PandasDtype.from_str_alias, which should map onto the `Obje

Read more

enhancement good first issue

Open

replace '%s' string formatting for f-string formatting

2

Open

Add support for complex number data types

3

Find more good first issues →

infoslack / awesome-kafka

Star

A list about Apache Kafka

infrastructure kafka apache-spark stream-processing apache-kafka kafka-streams data-processing data-pipeline streaming-data

Updated Sep 16, 2020

benibela / xidel

Star

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

Updated Sep 30, 2020
Pascal

alttch / rapidtables

Star

Super fast list of dicts to pre-formatted tables conversion library for Python 2/3

python library text-formatting python3 data-processing dictionary-data

Updated Nov 17, 2019
Python

msamogh / nonechucks

Star

Open

Write tests

msamogh commented Mar 18, 2019

Write unit test coverage for SafeDataset and SafeDataLoader, along with the functions in utils.py.

Read more

good first issue

Open

Route next batch through step_to_index_fn in _SafeDataLoaderIter

pysparkling

pysparkling / pysparkling

Star

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

python data-science apache-spark data-processing

Updated Oct 14, 2020
Python

Yord / pxi

Star

🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.

tsv parser json csv serializer ssv data-processing dsv marshaller deserializer pixie pxi

Updated Aug 24, 2020
JavaScript

constellation-rs / amadeus

Star

Harmonious distributed data analysis in Rust.

rust parallel-computing distributed-computing stream-processing data-analysis data-processing

Updated Oct 14, 2020
Rust

maykulkarni / Machine-Learning-Notebooks

Star

Machine Learning notebooks for refreshing concepts.

python machine-learning natural-language-processing reinforcement-learning deep-learning machine-learning-algorithms neural-networks deep-learning-algorithms dimensionality-reduction python-machine-learning data-processing regression-models deep-learning-tutorial data-science-notebook model-evaluation classification-trees clustering-methods machine-learning-tutorials

Updated Oct 31, 2018
Jupyter Notebook

PytLab / VASPy

Star

Manipulating VASP files with Python.

chemistry data-visualization vasp computational-chemistry data-processing catalysis-informatics

Updated Jul 29, 2020
Python

matousc89 / padasip

Star

Python Adaptive Signal Processing

machine-learning signal-processing neural-networks data-analysis data-processing adaptive-filtering

Updated Mar 26, 2018
Python

distributed-dataset

utdemir / distributed-dataset

Star

Open

Concurrent executors

utdemir commented Apr 14, 2019

Run read and write ends of the conduits concurrently.
When reading a Partition created by <> operations, consume the smaller partitions in parallel.

Read more

good first issue performance

tollwerk / data-processing-agreements

Star

Collection of Data Processing Agreement (DPA) and GDPR compliance resources

data privacy compliance data-processing gdpr dpa ethics adv dsgvo auftragsdatenverarbeitung

Updated Mar 29, 2019
CSS

thu-coai / cotk

Star

Conversational Toolkit. An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation

python machine-learning natural-language-processing deep-learning metrics data-processing natural-language-generation text-data cotk

Updated Aug 31, 2020
Python

SebKrantz / collapse

Star

Advanced and Fast Data Transformation in R

data-science cran r statistics time-series high-performance data-transformation scientific-computing econometrics rstats data-analysis data-manipulation data-processing weights panel-data weighted data-aggregation

Updated Oct 2, 2020
R

streamnative / pulsar-flink

Star

Elastic data processing with Apache Pulsar and Apache Flink

schema sql catalog schema-registry stream-processing apache-flink flink data-processing batch-processing flink-stream-processing apache-pulsar flink-connector

Updated Oct 12, 2020
Java

ColasGael / Machine-Learning-for-Solar-Energy-Prediction

Star

Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning

python machine-learning neural-network tensorflow matlab data-processing

Updated Nov 7, 2019
Python

classtag / ijcai18-mama-ads-competition

Star

IJCAI-18 阿里妈妈搜索广告转化预测初赛方案

admin lightgbm data-processing ctr tianchi catboost ijcai-18

Updated Apr 22, 2018
Jupyter Notebook

asyml / forte

Star

Forte is a flexible and powerful NLP builder FOR TExt.

python machine-learning natural-language-processing information-retrieval deep-learning data-processing text-data

Updated Oct 15, 2020
Python

Improve this page

Add a description, image, and links to the data-processing topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the data-processing topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.