natural-language-processing

Related to #5142, AlbertTokenizer (which uses SentencePiece) doesn't decode special tokens (like [CLS], [MASK]) properly. This issue was discovered when adding the Nystromformer model (#14659), which uses this tokenizer.

To reproduce (Transformers v4.15 or below):

!pip install -q transformers sentencepiece

from transformers import AlbertTokenizer

tokenizer = AlbertTokenizer.from

Change tensor.data to tensor.detach() due to
pytorch/pytorch#6990 (comment)
tensor.detach() is more robust than tensor.data.

In gensim/models/fasttext.py:

    model = FastText(
        vector_size=m.dim,
        vector_size=m.dim,
        window=m.ws,
        window=m.ws,
        epochs=m.epoch,
        epochs=m.epoch,
        negative=m.neg,
        negative=m.neg,
        # FIXME: these next 2 lines read in unsupported FB FT modes (loss=3 softmax or loss=4 onevsall,
        # or model=3 supervi

Although the results look nice and ideal in all TensorFlow plots and are consistent across all frameworks, there is a small difference (more of a consistency issue). The result training loss/accuracy plots look like they are sampling on a lesser number of points. It looks more straight and smooth and less wiggly as compared to PyTorch or MXNet.

It can be clearly seen in chapter 6([CNN Lenet](ht

Is your feature request related to a problem? Please describe.

I am uploading our dataset and models for the "Constructing interval measures" method we've developed, which uses item response theory to convert multiple discrete labels into a continuous spectrum for hate speech. Once we have this outcome our NLP models conduct regression rather than classification, so binary metrics are not r

Is your feature request related to a problem? Please describe.
I typically used compressed datasets (e.g. gzipped) to save disk space. This works fine with AllenNLP during training because I can write my dataset reader to load the compressed data. However, the predict command opens the file and reads lines for the Predictor. This fails when it tries to load data from my compressed files.

Rather than simply caching nltk_data until the cache expires and it's forced to re-download the entire nltk_data, we should perform a check on the index.xml which refreshes the cache if it differs from some previous cache.

I would advise doing this in the same way that it's done for requirements.txt:
https://github.com/nltk/nltk/blob/59aa3fb88c04d6151f2409b31dcfe0f332b0c9ca/.github/wor

Hey Hackers of this spoopy month! 👻
Welcome to the Ciphey repo(s)!
This issue requires you to add a decoder.

This wiki section walks you through EVERYTHING you need to know, and we've added some more links at the bottom of this issue to detail more about the decoder.
https://github.com/Ciphey/Ciphey/wiki#adding-your-own-crackers--decoders

https://www.dcode.fr/t9-cipher
https://en.wikipe