Skip to content
#

tokenization

Star

Here are 319 public repositories matching this topic...

Language: All
Filter by language
All 319 Jupyter Notebook 94 Python 76 JavaScript 25 Java 20 Go 15 HTML 12 C 8 C++ 8 PHP 6 R 6
Sort: Best match
Sort options
Best match Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

cbaziotis / ekphrasis

Star

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization
  • Updated Feb 8, 2021
  • Python

GlitchedPolygons / l8w8jwt

Star
Open

Suggested extension / improvement

1
firthmj
firthmj commented Jan 21, 2021

How easy would it be to change the library to have versions of the encode and decode functions where the payload JSON was provided / returned just as the JSON text?

There are other good JSON generation / parsing libraries available, and some people may wish to use them to generate or process the payload, rather than the built in claim processing.

Read more
enhancement good first issue

Improve this page

Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."

Learn more