#

tokenization

Here are 490 public repositories matching this topic...

explosion / spaCy

💫 Industrial-strength Natural Language Processing (NLP) in Python

python nlp data-science machine-learning natural-language-processing ai deep-learning neural-network text-classification cython artificial-intelligence spacy named-entity-recognition neural-networks nlp-library tokenization entity-linking

Updated Oct 20, 2022
Python

lunasec-io / lunasec

Sponsor

LunaSec - Open Source AppSec platform that automatically notifies you the next time vulnerabilities like Log4Shell or node-ipc happen. Track your dependencies and builds in a centralized service. Get started in one-click via our GitHub App or host it yourself. https://github.com/apps/lunatrace-by-lunasec/

Updated Oct 20, 2022
TypeScript

databunker

securitybunker / databunker

Secure SDK/vault for personal records/PII built to comply with GDPR

Updated Oct 5, 2022
Go

RavenProject / Ravencoin

Ravencoin Core integration/staging tree

bitcoin raven blockchain token asset tokenization ravencoin

Updated Sep 10, 2022
C

VKCOM / YouTokenToMe

Unsupervised text tokenizer focused on computational efficiency

nlp natural-language-processing word-segmentation tokenization bpe

Updated Oct 19, 2022
C++

nlp-uoregon / trankit

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

multilingual nlp machine-learning natural-language-processing pytorch artificial-intelligence adapters deeplearning language-model universal-dependencies dependency-parsing tokenization lemmatization sentence-segmentation morphological-tagging part-of-speech-tagging xlm-roberta

Updated Jul 30, 2022
Python

explosion / spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

nlp machine-learning natural-language-processing text-classification spacy visualizer named-entity-recognition ner dependency-parsing tokenization word-vectors visualizers streamlit part-of-speech-tagging

Updated Aug 23, 2022
Python

cbaziotis / ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization

Updated Sep 29, 2022
Python

AmoDinho / datacamp-python-data-science-track

All the slides, accompanying code and exercises all stored in this repo. 🎈

python nlp data-science natural-language-processing neural-network scikit-learn pandas datascience neural-networks bokeh machinelearning tokenization datacamp-course datacamp datacamp-exercises datacamp-projects datacamp-solutions-python datacamp-python datacamp-machine-learning

Updated Apr 19, 2022
Python

adobe / NLP-Cube

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

parse machine-translation embeddings information-extraction dependency-parser universal-dependencies part-of-speech-tagger dependency-parsing tokenization lemmatization sentence-splitting nlp-cube language-pipeline

Updated Feb 10, 2022
Python

yooper / php-text-analysis

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

nlp php text-analysis php-language tokenization php-text-analysis

Updated Aug 1, 2022
PHP

macmade / ClangKit

Sponsor

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

c syntax-highlighting c-plus-plus parsing objective-c code llvm static-analysis clang source diagnostics tokenization

Updated Aug 2, 2021
C

CodeChain-io / codechain

CodeChain's official implementation in Rust.

rust blockchain asset tokenization digital-securities

Updated Jun 24, 2022
Rust

razdel

natasha / razdel

Rule-based token, sentence segmentation for Russian language

python nlp russian tokenization sentence-segmentation sentence-boundary-detection

Updated Feb 18, 2021
Python

daac-tools / vibrato

🎤 vibrato: Viterbi-based accelerated tokenizer

nlp rust japanese tokenizer segmentation morphological-analysis tokenization

Updated Oct 20, 2022
Rust

OpenNMT / Tokenizer

Fast and customizable text tokenization library with BPE and SentencePiece support

python unicode natural-language-processing cpp icu tokenizer machine-translation tokenization bpe sentencepiece

Updated Sep 13, 2022
C++

TokenScript / TokenScript

TokenScript schema, specs and paper

security cryptography mobile xml blockchain tokens web3 tokenization tokenisation

Updated Jun 26, 2022
JavaScript

daac-tools / vaporetto

🛥 Vaporetto: Very Accelerated POintwise pREdicTion based TOkenizer

nlp rust japanese tokenizer analyzer segmentation morphological-analysis tokenization

Updated Oct 11, 2022
Rust

WorksApplications / sudachi.rs

Sponsor

Sudachi in Rust 🦀 and new generation of SudachiPy

python rust segmentation pos-tagging morphological-analysis tokenization sudachi nlp-libary

Updated Oct 12, 2022
Rust

milaan9 / Python_Natural_Language_Processing

This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.

nlp ipython-notebook named-entity-recognition bag-of-words tf-idf stopwords tokenization stemming lemmatization sentence-segmentation termfrequency partofspeech-tagger vocabulary-matching python4everybody python4datascience tutor-milaan9 inversedocumentfrequency

Updated Jul 4, 2022
Jupyter Notebook

Improve this page

Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."