Topic Modelling in Semantic Embedding Spaces

Description

ETM is a generative topic model combining traditional topic models (LDA) with word embeddings (word2vec).

• It models each word with a categorical distribution whose natural parameter is the inner product between a word embedding and an embedding of its assigned topic.

• The model is fitted using an amortized variational inference algorithm on top of libtorch.

Usage

ETM(
  k = 20,
  embeddings,
  dim = 800,
  activation = c("relu", "tanh", "softplus", "rrelu", "leakyrelu", "elu", "selu",
    "glu"),
  dropout = 0.5,
  vocab = rownames(embeddings)
)

Arguments

Value

an object of class ETM which is a torch nn_module containing o.a.

• num_topics: the number of topics

• vocab: character vector with the terminology used in the model

• vocab_size: the number of words in vocab

• rho: The word embeddings

• alphas: The topic embeddings

Methods

fit(data, optimizer, epoch, batch_size, normalize = TRUE, clip = 0, lr_anneal_factor = 4, lr_anneal_nonmono = 10)

Fit the model on a document term matrix by splitting the data in 70/30 training/test set and updating the model weights.

Arguments

data

bag of words document term matrix in dgCMatrix format

optimizer

object of class torch_Optimizer

epoch

integer with the number of iterations to train

batch_size

integer with the size of the batch

normalize

logical indicating to normalize the bag of words data

clip

number between 0 and 1 indicating to do gradient clipping - passed on to nn_utils_clip_grad_norm_

lr_anneal_factor

divide the learning rate by this factor when the loss on the test set is monotonic for at least lr_anneal_nonmono training iterations

lr_anneal_nonmono

number of iterations after which learning rate annealing is executed if the loss does not decreases

References

https://arxiv.org/pdf/1907.04907.pdf

Examples

library(torch)
library(topicmodels.etm)
library(word2vec)
library(udpipe)
data(brussels_reviews_anno, package = "udpipe")
##
## Toy example with pretrained embeddings
##

## a. build word2vec model
x          <- subset(brussels_reviews_anno, language %in% "nl")
x          <- paste.data.frame(x, term = "lemma", group = "doc_id") 
set.seed(4321)
w2v        <- word2vec(x = x$lemma, dim = 15, iter = 20, type = "cbow", min_count = 5)
embeddings <- as.matrix(w2v)

## b. build document term matrix on nouns + adjectives, align with the embedding terms
dtm <- subset(brussels_reviews_anno, language %in% "nl" & upos %in% c("NOUN", "ADJ"))
dtm <- document_term_frequencies(dtm, document = "doc_id", term = "lemma")
dtm <- document_term_matrix(dtm)
dtm <- dtm_conform(dtm, columns = rownames(embeddings))
dtm <- dtm[dtm_rowsums(dtm) > 0, ]

## create and fit an embedding topic model - 8 topics, theta 100-dimensional
if (torch::torch_is_installed()) {

set.seed(4321)
torch_manual_seed(4321)
model       <- ETM(k = 8, dim = 100, embeddings = embeddings, dropout = 0.5)
optimizer   <- optim_adam(params = model$parameters, lr = 0.005, weight_decay = 0.0000012)
overview    <- model$fit(data = dtm, optimizer = optimizer, epoch = 40, batch_size = 1000)
scores      <- predict(model, dtm, type = "topics")

lastbatch   <- subset(overview$loss, overview$loss$batch_is_last == TRUE)
plot(lastbatch$epoch, lastbatch$loss)
plot(overview$loss_test)

## show top words in each topic
terminology <- predict(model, type = "terms", top_n = 7)
terminology

##
## Toy example without pretrained word embeddings
##
set.seed(4321)
torch_manual_seed(4321)
model       <- ETM(k = 8, dim = 100, embeddings = 15, dropout = 0.5, vocab = colnames(dtm))
optimizer   <- optim_adam(params = model$parameters, lr = 0.005, weight_decay = 0.0000012)
overview    <- model$fit(data = dtm, optimizer = optimizer, epoch = 40, batch_size = 1000)
terminology <- predict(model, type = "terms", top_n = 7)
terminology





}

Get matrices out of an ETM object

Description

Convenience function to extract

• embeddings of the topic centers

• embeddings of the words used in the model

• words emmitted by each topic (beta), which is the softmax-transformed inner product of word embedding and topic embeddings

Usage

## S3 method for class 'ETM'
as.matrix(x, type = c("embedding", "beta"), which = c("topics", "words"), ...)

Arguments

Value

a numeric matrix containing, depending on the value supplied in type either the embeddings of the topic centers, the embeddings of the words or the words emitted by each topic

See Also

ETM

Examples

library(torch)
library(topicmodels.etm)
path  <- system.file(package = "topicmodels.etm", "example", "example_etm.ckpt")
model <- torch_load(path)

topic.centers     <- as.matrix(model, type = "embedding", which = "topics")
word.embeddings   <- as.matrix(model, type = "embedding", which = "words")
topic.terminology <- as.matrix(model, type = "beta")

Bag of words sample of the 20 newsgroups dataset

Description

Data available at https://github.com/bnosac-dev/ETM/tree/master/data/20ng

Examples

data(ng20)
str(ng20$vocab)
str(ng20$bow_tr$tokens)
str(ng20$bow_tr$counts)

Plot functionality for an ETM object

Description

Convenience function allowing to plot

• the evolution of the loss on the training / test set in order to inspect training convergence

• the ETM model in 2D dimensional space using a umap projection. This plot uses function textplot_embedding_2d from the textplot R package and plots the top_n most emitted words of each topic and the topic centers in 2 dimensions

Usage

## S3 method for class 'ETM'
plot(
  x,
  type = c("loss", "topics"),
  which,
  top_n = 4,
  title = "ETM topics",
  subtitle = "",
  encircle = FALSE,
  points = FALSE,
  ...
)

Arguments

Value

In case type is set to 'topics', maps the topic centers and most emitted words for each topic to 2D using summary.ETM and returns a ggplot object by calling textplot_embedding_2d.
For type 'loss', makes a base graphics plot and returns invisibly nothing.

See Also

ETM, summary.ETM, textplot_embedding_2d

Examples

library(torch)
library(topicmodels.etm)
path  <- system.file(package = "topicmodels.etm", "example", "example_etm.ckpt")
model <- torch_load(path)
plot(model, type = "loss")



library(torch)
library(topicmodels.etm)
library(textplot)
library(uwot)
library(ggrepel)
library(ggalt)
path  <- system.file(package = "topicmodels.etm", "example", "example_etm.ckpt")
model <- torch_load(path)
plt   <- plot(model, type = "topics", top_n = 7, which = c(1, 2, 14, 16, 18, 19),
              metric = "cosine", n_neighbors = 15, 
              fast_sgd = FALSE, n_threads = 2, verbose = TRUE,
              title = "ETM Topics example")
plt

Predict functionality for an ETM object.

Description

Predict to which ETM topic a text belongs or extract which words are emitted for each topic.

Usage

## S3 method for class 'ETM'
predict(
  object,
  newdata,
  type = c("topics", "terms"),
  batch_size = nrow(newdata),
  normalize = TRUE,
  top_n = 10,
  ...
)

Arguments

Value

Returns for

• type 'topics': a matrix with topic probabilities of dimension nrow(newdata) x the number of topics

• type 'terms': a list of data.frame's where each data.frame has columns term, beta and rank indicating the top_n most emitted terms for that topic. List element 1 corresponds to the top terms emitted by topic 1, element 2 to topic 2 ...

See Also

ETM

Examples

library(torch)
library(topicmodels.etm)
path  <- system.file(package = "topicmodels.etm", "example", "example_etm.ckpt")
model <- torch_load(path)

# Get most emitted words for each topic
terminology  <- predict(model, type = "terms", top_n = 5)
terminology

# Get topics probabilities for each document
path   <- system.file(package = "topicmodels.etm", "example", "example_dtm.rds")
dtm    <- readRDS(path)
dtm    <- head(dtm, n = 5)
scores <- predict(model, newdata = dtm, type = "topics")
scores

Project ETM embeddings using UMAP

Description

Uses the uwot package to map the word embeddings and the center of the topic embeddings to a 2-dimensional space

Usage

## S3 method for class 'ETM'
summary(object, type = c("umap"), n_components = 2, top_n = 20, ...)

Arguments

Value

a list with elements

• center: a matrix with the embeddings of the topic centers

• words: a matrix with the embeddings of the words

• embed_2d: a data.frame which contains a lower dimensional presentation in 2D of the topics and the top_n words associated with the topic, containing columns type, term, cluster (the topic number), rank, beta, x, y, weight; where type is either 'words' or 'centers', x/y contain the lower dimensional positions in 2D of the word and weight is the emitted beta scaled to the highest beta within a topic where the topic center always gets weight 0.8

See Also

umap, ETM

Examples

library(torch)
library(topicmodels.etm)
library(uwot)
path     <- system.file(package = "topicmodels.etm", "example", "example_etm.ckpt")
model    <- torch_load(path)
overview <- summary(model, 
                    metric = "cosine", n_neighbors = 15, 
                    fast_sgd = FALSE, n_threads = 1, verbose = TRUE) 
overview$center
overview$embed_2d