I have a document term matrix, "mydtm" that I have created in R, using the 'tm' package. I am attempting to depict the similarities between each of the 557 documents contained within the dtm/corpus. I have been attempting to use a cosine similarity matrix using: mydtm_cosine <- dist(mydtm_matrix, method = "cosine", diag = F, upper = F) However the output matrix I get is huge with many missing values. Any help/suggestions would be much appreciated. Output Matrix
2
-
Have you tried removing stopwords, wordstemming to reduce the number of parts of speech? Are the documents very short? If they don't co-occur it will not be possible to define similarity.– CSJCampbellCommented Jun 2, 2021 at 19:59
-
@CSJCampbell yes I have conducted stemming, removal of numbers, white space etc.– Luke HansenCommented Jun 2, 2021 at 22:06
Add a comment
|
1 Answer
Likely you have few words which occur between your documents. You may wish to reduce the words in your term document matrix.
text <- c("term-document matrix is a mathematical matrix",
"we now have a tidy three-column",
"cast into a Term-Document Matrix",
"where the rows represent the text responses, or documents")
corpus <- VCorpus(VectorSource(text))
tdm <- TermDocumentMatrix(corpus,
control = list(wordLengths = c(1, Inf)))
occurrence <- apply(X = tdm,
MARGIN = 1,
FUN = function(x) sum(x > 0) / ncol(tdm))
occurrence
# a cast documents have
# 0.75 0.25 0.25 0.25
# into is mathematical matrix
# 0.25 0.25 0.25 0.50
# now or represent responses,
# 0.25 0.25 0.25 0.25
# rows term-document text the
# 0.25 0.50 0.25 0.25
# three-column tidy we where
# 0.25 0.25 0.25 0.25
quantile(occurrence, probs = c(0.5, 0.9, 0.99))
# 50% 90% 99%
# 0.2500 0.5000 0.7025
tdm_mat <- as.matrix(tdm[names(occurrence)[occurrence >= 0.5], ])
tdm_mat
# Docs
# Terms 1 2 3 4
# a 1 1 1 0
# matrix 2 0 1 0
# term-document 1 0 1 0
You can then calculate cosine similarity.
library(proxy)
dist(tdm_mat, method = "cosine", upper = TRUE)
# a matrix term-document
# a 0.2254033 0.1835034
# matrix 0.2254033 0.0513167
# term-document 0.1835034 0.0513167
