#

tokenization

Here are 365 public repositories matching this topic...

explosion / spaCy

Star

💫 Industrial-strength Natural Language Processing (NLP) in Python

python nlp data-science machine-learning natural-language-processing ai deep-learning neural-network text-classification cython artificial-intelligence spacy named-entity-recognition neural-networks nlp-library tokenization entity-linking

Updated Oct 25, 2021
Python

databunker

securitybunker / databunker

Star

Secure SDK/vault for personal records/PII built to comply with GDPR

Updated Aug 24, 2021
Go

RavenProject / Ravencoin

Star

Ravencoin Core integration/staging tree

bitcoin raven blockchain token asset tokenization ravencoin

Updated Oct 16, 2021
C

VKCOM / YouTokenToMe

Star

Unsupervised text tokenizer focused on computational efficiency

nlp natural-language-processing word-segmentation tokenization bpe

Updated Jan 28, 2021
C++

nlp-uoregon / trankit

Star

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

multilingual nlp machine-learning natural-language-processing pytorch artificial-intelligence adapters deeplearning language-model universal-dependencies dependency-parsing tokenization lemmatization sentence-segmentation morphological-tagging part-of-speech-tagging xlm-roberta

Updated Oct 15, 2021
Python

cbaziotis / ekphrasis

Star

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization

Updated Feb 8, 2021
Python

explosion / spacy-streamlit

Star

👑 spaCy building blocks and visualizers for Streamlit apps

nlp machine-learning natural-language-processing text-classification spacy visualizer named-entity-recognition ner dependency-parsing tokenization word-vectors visualizers streamlit part-of-speech-tagging

Updated Oct 23, 2021
Python

adobe / NLP-Cube

Star

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

parse machine-translation embeddings information-extraction dependency-parser universal-dependencies part-of-speech-tagger dependency-parsing tokenization lemmatization sentence-splitting nlp-cube language-pipeline

Updated Oct 18, 2021
Python

yooper / php-text-analysis

Star

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

nlp php text-analysis php-language tokenization php-text-analysis

Updated May 17, 2021
PHP

AmoDinho / datacamp-python-data-science-track

Star

Open

Unzip Slides

AmoDinho commented Feb 28, 2018

The slides for most of the courses need to be unzipped.

Read more

enhancement good first issue

Open

Format Code

Open

Format Instructions

macmade / ClangKit

Sponsor Star

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

c syntax-highlighting c-plus-plus parsing objective-c code llvm static-analysis clang source diagnostics tokenization

Updated Aug 2, 2021
C

CodeChain-io / codechain

Star

Open

Show the existing account if account import command fails with `AlreadyExists` error

joojis opened Oct 8, 2018

good first issue account

Open

Check --notify-work values are URLs on start

6

Open

Need a subcommand that cleans db of the given chain

3

Find more good first issues

razdel

natasha / razdel

Star

Rule-based token, sentence segmentation for Russian language

python nlp russian tokenization sentence-segmentation sentence-boundary-detection

Updated Feb 18, 2021
Python

OpenNMT / Tokenizer

Star

Fast and customizable text tokenization library with BPE and SentencePiece support

python unicode natural-language-processing cpp icu tokenizer machine-translation tokenization bpe sentencepiece

Updated Oct 8, 2021
C++

rth / vtext

Star

Simple NLP in Rust with Python bindings

nlp information-retrieval bag-of-words tf-idf tokenization

Updated Jun 15, 2021
Rust

TokenScript / TokenScript

Star

TokenScript schema, specs and paper

security cryptography mobile xml blockchain tokens web3 tokenization tokenisation

Updated Apr 9, 2021
HTML

frankiefab100 / Blockchain-Development-Resources

Star

The contents of this repository will help you launch a career in Blockchain development. How to deploy Smart contracts on Ethereum, build DApps, DeFi, DAO, NFT and Token protocol.

bitcoin ethereum blockchain truffle cryptocurrency tokens dao solidity tokenization metamask smartcontracts nft ethers blockchain-website cryptoeconomics decentralized-applications dapps hardhat defi

Updated Oct 25, 2021

WorksApplications / sudachi.rs

Star

An official Sudachi clone in Rust (incomplete) 🦀

segmentation pos-tagging morphological-analysis tokenization sudachi nlp-libary

Updated Oct 26, 2021
Rust

JuliaText / WordTokenizers.jl

Star

High performance tokenizers for natural language processing and other related tasks

nlp information-retrieval data-mining lexer tokenization

Updated Apr 10, 2021
Julia

adamshamsudeen / Vaaku2Vec

Star

Language Modeling and Text Classification in Malayalam Language using ULMFiT

text-classification word2vec malayalam language-model fastai tokenization ulmfit

Updated Sep 8, 2021
Jupyter Notebook

lucidrains / charformer-pytorch

Star

Implementation of the GBST block from the Charformer paper, in Pytorch

deep-learning artificial-intelligence transformer tokenization

Updated Jul 15, 2021
Python

wongnai / wongnai-corpus

Star

Collection of Wongnai's datasets

nlp datasets nlp-machine-learning tokenization

Updated Aug 26, 2019

neelkamath / spacy-server

Star

Open

Migrate from GitLab CI to GitHub Actions

5

neelkamath commented Oct 2, 2021

Create a GitHub Actions workflow based on .gitlab-ci.yml. Name the workflow .github/workflows/workflow.yml. The workflow should run on pushes to the master branch, and on pull requests. Here's an example snippet:
```
on:
  push:
    branches: [ master ]
  pull_request:
```
Dele

Read more

enhancement good first issue hacktoberfest

Open

Update the FastAPI dependency

3

GoogleCloudPlatform / dlp-dataflow-deidentification

Star

Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP

bigquery data beam dataflow tokenization dlp pii

Updated Oct 22, 2021
Java

PyThaiNLP / attacut

Star

A Fast and Accurate Neural Thai Word Segmenter

nlp cnn tokenization

Updated Oct 21, 2021
Python

liuzl / ling

Star

Natural Language Processing Toolkit in Golang

nlp spacy opencc tokenization normalization lemmatization corenlp

Updated May 9, 2020
Go

GlitchedPolygons / l8w8jwt

Star

Open

Suggested extension / improvement

1

firthmj commented Jan 21, 2021

How easy would it be to change the library to have versions of the encode and decode functions where the payload JSON was provided / returned just as the JSON text?

There are other good JSON generation / parsing libraries available, and some people may wish to use them to generate or process the payload, rather than the built in claim processing.

Read more

enhancement good first issue

vaulty-co / vaulty

Star

Tokenize, encrypt/decrypt, mask your data on the fly with Vaulty proxy

security encryption proxy tokenization

Updated Sep 24, 2021
Go

winkjs / wink-tokenizer

Star

Multilingual tokenizer that automatically tags each token with its type

multilingual german tokenizer tagging latin french hindi wink devanagari marathi tokenization konkani

Updated Aug 11, 2021
JavaScript

aatimofeev / spacy_russian_tokenizer

Star

Custom Russian tokenizer for spaCy

natural-language-processing russian tokenization spacy-pipeline

Updated May 14, 2019
Python

Improve this page

Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."