#

tokenization

Here are 263 public repositories matching this topic...

spaCy

explosion / spaCy

Star

💫 Industrial-strength Natural Language Processing (NLP) in Python

python nlp data-science machine-learning natural-language-processing ai deep-learning neural-network text-classification cython artificial-intelligence spacy named-entity-recognition neural-networks nlp-library tokenization entity-linking

Updated Feb 3, 2021
Python

VKCOM / YouTokenToMe

Star

Unsupervised text tokenizer focused on computational efficiency

nlp natural-language-processing word-segmentation tokenization bpe

Updated Jan 28, 2021
C++

RavenProject / Ravencoin

Star

Ravencoin Core integration/staging tree

bitcoin raven blockchain token asset tokenization ravencoin

Updated Jan 27, 2021
C

cbaziotis / ekphrasis

Star

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization

Updated Aug 13, 2020
Python

yooper / php-text-analysis

Star

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

nlp php text-analysis php-language tokenization php-text-analysis

Updated Dec 20, 2020
PHP

adobe / NLP-Cube

Star

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

parse machine-translation embeddings information-extraction dependency-parser universal-dependencies part-of-speech-tagger dependency-parsing tokenization lemmatization sentence-splitting nlp-cube language-pipeline

Updated Feb 2, 2021
Python

explosion / spacy-streamlit

Star

👑 spaCy building blocks and visualizers for Streamlit apps

nlp machine-learning natural-language-processing text-classification spacy visualizer named-entity-recognition ner dependency-parsing tokenization word-vectors visualizers streamlit part-of-speech-tagging

Updated Jan 27, 2021
Python

macmade / ClangKit

Star

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

c syntax-highlighting c-plus-plus parsing objective-c code llvm static-analysis clang source diagnostics tokenization

Updated May 9, 2017
C

nlp-uoregon / trankit

Star

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

multilingual nlp machine-learning natural-language-processing pytorch artificial-intelligence adapters deeplearning language-model universal-dependencies dependency-parsing tokenization lemmatization sentence-segmentation morphological-tagging part-of-speech-tagging xlm-roberta

Updated Jan 27, 2021
Python

CodeChain-io / codechain

Star

Open

Show the existing account if account import command fails with `AlreadyExists` error

joojis opened Oct 8, 2018

account good first issue

Open

Check --notify-work values are URLs on start

6

Open

Need a subcommand that cleans db of the given chain

3

Find more good first issues →

AmoDinho / datacamp-python-data-science-track

Star

Open

Unzip Slides

AmoDinho commented Feb 28, 2018

The slides for most of the courses need to be unzipped.

Read more

enhancement good first issue

Open

Format Code

Open

Format Instructions

razdel

natasha / razdel

Star

Rule-based token, sentence segmentation for Russian language

python nlp russian tokenization sentence-segmentation sentence-boundary-detection

Updated Nov 27, 2020
Python

OpenNMT / Tokenizer

Star

Fast and customizable text tokenization library with BPE and SentencePiece support

python unicode natural-language-processing cpp icu tokenizer machine-translation tokenization bpe sentencepiece

Updated Jan 27, 2021
C++

databunker

securitybunker / databunker

Star

Secure storage for personal records built to comply with GDPR

Updated Jan 20, 2021
Go

rth / vtext

Star

Simple NLP in Rust with Python bindings

nlp information-retrieval bag-of-words tf-idf tokenization

Updated Jul 22, 2020
Rust

TokenScript / TokenScript

Star

Open

tickets layout in MagicLink landing page

zhangzhongnan928 commented Dec 30, 2019

https://xdai.aw.app/AgAAAABeCrHfD0OSNmeEO8yv0SpcABp4OPpfyKsyMDIxMDcwNjIxMDAwMCswMzAwBQNVU0EAQVUBAQAAAORN4V9AtxeqBL0pNP6c2B22unDDzQRPt_xttOTsHSBcMGWrmZfwDpSlyRr3MQ_Hz45Rho5RFbFmP_H6a1-O9hEb

https://xdai.aw.app/AgAAAABeCrKV9rjdi6

Read more

bug good first issue

adamshamsudeen / Vaaku2Vec

Star

Language Modeling and Text Classification in Malayalam Language using ULMFiT

text-classification word2vec malayalam language-model fastai tokenization ulmfit

Updated Feb 2, 2021
Jupyter Notebook

sorami / sudachi.rs

Star

An unofficial Sudachi clone in Rust (incomplete) 🦀

segmentation pos-tagging morphological-analysis tokenization sudachi nlp-libary

Updated Aug 27, 2020
Rust

wongnai / wongnai-corpus

Star

Collection of Wongnai's datasets

nlp datasets nlp-machine-learning tokenization

Updated Aug 26, 2019

JuliaText / WordTokenizers.jl

Star

High performance tokenizers for natural language processing and other related tasks

nlp information-retrieval data-mining lexer tokenization

Updated Jan 17, 2021
Julia

neelkamath / spacy-server

Star

🦜 Containerized HTTP API for industrial-strength NLP via spaCy and sense2vec

python nlp docker spacy named-entity-recognition sense2vec part-of-speech-tagger tokenization sentence-segmentation

Updated Oct 1, 2020
Python

GoogleCloudPlatform / dlp-dataflow-deidentification

Star

Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP

bigquery data beam dataflow tokenization dlp pii

Updated Jan 5, 2021
Java

liuzl / ling

Star

Natural Language Processing Toolkit in Golang

nlp spacy opencc tokenization normalization lemmatization corenlp

Updated May 9, 2020
Go

PyThaiNLP / attacut

Star

A Fast and Accurate Neural Thai Word Segmenter

nlp cnn tokenization

Updated Jan 18, 2021
Python

vaulty-co / vaulty

Star

Tokenize, encrypt/decrypt, mask your data on the fly with Vaulty proxy

security encryption proxy tokenization

Updated Jan 24, 2021
Go

winkjs / wink-tokenizer

Star

Multilingual tokenizer that automatically tags each token with its type

multilingual german tokenizer tagging latin french hindi wink devanagari marathi tokenization konkani

Updated Nov 24, 2020
JavaScript

bastienbot / nlp-js-tools-french

Star

POS Tagger, lemmatizer and stemmer for french language in javascript

nlp tokenizer postgresql stemmer lemmatizer tokenization stemming lemmatization postagging

Updated Sep 13, 2017
JavaScript

alm

anyks / alm

Star

Smart Language Model

cpp tokenizer language-models arpa alm tokenization vocab-pruning

Updated Sep 20, 2020
C++

aatimofeev / spacy_russian_tokenizer

Star

Custom Russian tokenizer for spaCy

natural-language-processing russian tokenization spacy-pipeline

Updated May 14, 2019
Python

rosette-api / python

Star

Rosette API Client Library for Python

python nlp machine-learning natural-language-processing text-mining sentiment-analysis text morphology text-analysis language-detection fuzzy-matching name-generation tokenization categorization lemmatization relation-extraction entity-extraction language-identification name-translation name-similarity

Updated Jun 16, 2020
Python

Improve this page

Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."