text-processing

⚡ From finding text to search and replace, from sorting to beautifying text and more 🎨

Diff Match Patch is a high-performance library in multiple languages that manipulates plain text.

Intuitive find & replace CLI (sed alternative)

fastNLP: A Modularized and Extensible NLP Framework. Currently still in incubation.

Text Classification Algorithms: A Survey

Python library for creating PEG parsers

Program to convert lines of text into a tree structure.

Natural language detection library for Go

A sharp cut(1) clone.

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

A fast implementation of Aho-Corasick in Rust.

A simple Python module for parsing human names into their individual components

Open Korean Text Processor - An Open-source Korean Text Processor

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

🗣️ Tool to generate adversarial text examples and test machine learning models against them

👄 The most accurate natural language detection library in the Go ecosystem, suitable for long and short text alike

Textpipe: clean and extract metadata from text

A low level regular expression library that uses deterministic finite automata.

Automatic Korean word spacing with Python

pyarabic

Pure-Python Japanese character interconverter for Hiragana, Katakana, Hankaku, and Zenkaku

Text vectorization tool to outperform TFIDF for classification tasks

Tool which allow you to detect and translate text.

Python library for Natural Language Preprocessing (NLPre)

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.

Util collection for Japanese text processing. Hiraganize, Katakanize, and Romanize.

Extract indicators of compromise from text, including "escaped" ones.