Methodological Rigour in Algorithm Application: An Illustration of Topic Modelling Algorithm

Malmi Amadoru

REVIEW 3 major objections 5 minor 38 references

Methodological Rigour in Algorithm Application: An Illustration of Topic Modelling Algorithm

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that topic modelling studies become trustworthy when researchers follow eight guidelines spanning data preparation, model building, and pattern evaluation, and it demonstrates these guidelines on a corpus of blockchain…

desk verdict A readable, honest methods tutorial whose model-selection step doesn't live up to its own rigour message. read the letter →

arxiv 2507.00547 v1 pith:URFXRY6Y submitted 2025-07-01 cs.CL

Malmi Amadoru This is my paper · ORCID

classification cs.CL

keywords topicmodellingmethodologicalrigourstructuralmodel(STM)computationallyintensivetheoryconstructioninterpretabilitytransparencyhumanevaluationguidelines

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that methodological rigour in topic modelling is something a researcher can deliberately do, not merely aspire to. It offers eight practical guidelines organised around three stages—data preparation, model building, and pattern evaluation—and demonstrates them on a structural topic model built from 1,520 blockchain article abstracts. The author argues that algorithmic opacity and unexamined defaults undermine trust in research, and that explicit justification of every decision restores that trust. A sympathetic reader would care because the guidelines give novice researchers, editors, and reviewers a concrete checklist for judging whether topic-modelling patterns are plausible enough to support theory. The paper's contribution is guidance, not a new empirical discovery.

What carries the argument

The load-bearing object is the three-stage iterative process shown in Figure 1—data preparation, model building, and pattern evaluation—together with the eight-guideline checklist in Table 3. The machinery that carries the argument includes STM's diagnostic metrics (held-out log-likelihood, residuals, semantic coherence, and exclusivity) used to shortlist candidate topic counts, and the word- and topic-intrusion evaluation tasks that inject an intruder word or intruder topic for human coders to detect. The worked illustration on 1,520 blockchain abstracts is what turns the abstract idea of rigour into concrete, reviewable steps.

What would settle it

A replication with more coders and reported inter-coder reliability that found the 20-topic model no more interpretable than the 35-topic model (which had better topic log odds), or found the intrusion-task scores unstable, would weaken the paper's claim that its demonstration establishes rigour, even though the guidelines themselves could still stand.

Watch

Extended reading notes

Core claim

The central claim is that rigour in topic modelling comes from systematically justifying decisions at every stage of an iterative three-stage process, rather than from following standard procedures as fixed recipes. The paper illustrates this by applying the structural topic modelling (STM) algorithm to blockchain abstracts: it justifies data-preparation choices, compares twelve candidate models through diagnostic metrics such as held-out likelihood, residuals, semantic coherence, and exclusivity, and then uses word- and topic-intrusion tasks plus a label-quality evaluation by two coders to settle on a 20-topic model. The final contribution is a set of eight guidelines, six tied to the three stages and two about reporting computational procedures, which the author argues can generalise to other machine learning algorithms with context-specific adjustments.

Load-bearing premise

The load-bearing premise is that the human evaluation of interpretability—two coders on ten topic-intrusion cases per model, with the author among the coders and no reported inter-coder reliability—is sound enough to choose the final model and to demonstrate rigour across the whole process.

Editorial extensions

If this is right

If the guidelines are right, researchers can structure a topic-modelling study as a series of justified choices, and reviewers can demand those justifications.
The reporting guidelines imply that a topic-modelling paper is not complete until its computational procedure is transparent enough for replication.
The selection of the 20-topic model shows that combining statistical diagnostics with human evaluation can make the number-of-topics decision defensible.
Because the guidelines are not tied to one algorithm, they can be adapted to other machine learning algorithms with context-specific adjustments.
Novice researchers gain a starting template that replaces copying prior studies' defaults with context-aware decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The eight guidelines could be converted into a structured reporting checklist or preregistration template for topic-modelling studies, making adherence easier to verify.
The human-evaluation step would be stronger with more coders and reported inter-coder reliability; without those, the demonstration shows the method but not its stability.
The same three-stage logic could be adapted to supervised learning or large-language-model research by substituting evaluation tasks tailored to those outputs, such as calibration checks for predictions.
A testable extension would be to compare studies that follow these guidelines against studies that do not, measuring whether readers or reviewers judge the former as more trustworthy.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This is a methods/teaching paper, not a research paper. The actual contribution is a worked STM application with eight guidelines for rigorous topic modelling. The guidelines aren't new—justify preprocessing, understand hyperparameters, balance accuracy and interpretability, use human evaluation, report transparently—all appear in the cited tutorials and in Chang et al. But it does a real service in making those explicit for IS researchers and showing them in one coherent workflow. The data preparation narrative is honest: the author discusses why stemming wasn't used, how collocations were handled, and how searchK diagnostics were interpreted. That kind of concrete transparency has pedagogical value.

The soft spot is exactly where the stress-test hits: the selection of K=20 over K=35 rests on a human evaluation of 10 topic-intrusion cases per model, two coders, one being the author, and no inter-coder reliability. The topic log-odds are actually slightly worse for K=20 (-1.09 vs -1.01 for K=35), and with 20 observations the gap is noise. The model precision difference (0.68 vs 0.50) is also within binomial uncertainty. The paper doesn't report uncertainty or sensitivity. Because the whole illustration is meant to demonstrate rigour, a selection step that isn't demonstrably robust undermines the central claim. Also, no code or data are provided, despite the "report to help build the community" guideline—an irony the author doesn't address.

Minor: the generalisation to other ML algorithms is asserted, not argued. And the novelty claim about filling a gap in "explicit methodological rigour" is weak; the cited tutorials already cover much of this. The paper is honest about its scope limitations, though it doesn't flag the evaluation issue.

Verdict: solid teaching material, not a substantive methodological advance. A serious editor should send it to review before publishing—it needs a major revision on the evaluation step, or a re-framing of the contribution as a tutorial rather than a new rigour framework. But it deserves referee time; it's not a desk reject.

Referee Report

3 major / 5 minor

Summary. This paper proposes that methodological rigour in topic modelling can be established through a three-stage iterative process—data preparation, model building, and pattern evaluation—and presents eight guidelines (Table 3) for achieving that rigour. To illustrate the approach, the author applies the Structural Topic Model (STM) to 1520 academic abstracts about blockchain, describing decisions at each stage: preprocessing choices, hyperparameter selection via diagnostic metrics (held-out likelihood, residuals, semantic coherence, exclusivity), human evaluation using word and topic intrusion tasks, and topic labelling. The paper further claims that the guidelines can generalize to other machine learning algorithms with context-specific adjustments. The intended contribution is methodological guidance for novice researchers, editors, and reviewers, rather than an empirical finding about blockchain or topic models.

Significance. If the illustration is accepted as rigorous, the paper offers a useful pedagogical checklist and a worked example that consolidates scattered practices from existing tutorials (e.g., Debortoli et al. 2016; Schmiedel et al. 2019) and from the STM literature. Its stage-linked guidelines and emphasis on transparency are well aligned with current discussions of computational rigour in IS research. However, the load-bearing demonstration of rigour rests on a model-selection evaluation that is currently too thin: two coders (one the author), ten topic-intrusion cases per model, no inter-coder reliability, and no uncertainty quantification. The chosen model (K=20) is not clearly superior on the reported metrics. The generalizability claim to other ML algorithms is asserted rather than argued. These weaknesses do not invalidate the guideline content, but they prevent the paper from fully making its central claim that the illustration itself exemplifies rigour.

major comments (3)

[Section 3.4, Table 1] The model selection decision is the crucial step in the illustration, yet it rests on a small and potentially unreliable human evaluation. Two coders (one being the author), ten topic-intrusion cases per model, no inter-coder reliability statistic, and no confidence intervals are reported. The selected model, K=20, has the highest model precision (0.68), but K=35 has better topic log odds (-1.01 vs -1.09). With roughly 20 topic-intrusion judgments per model, the precision gap between 0.68 and 0.50 is plausibly within binomial sampling error, and the 0.08 log-odds difference is negligible. As reported, the preference for K=20 is not robust. To support the paper's claim that the illustration demonstrates methodological rigour, please report inter-coder agreement (e.g., Cohen's kappa), quantify uncertainty (e.g., confidence intervals or a binomial test), increase the number of intrusion cases or justify the sample size, and/or pre-specify a decision rule for selecting K. Without this, the central illustrative example does not convincingly establish the rigour it advocates.
[Abstract; Section 4; Section 5] The claim that the guidelines 'can be applied to other algorithms with context-specific adjustments' is not substantiated. The paper provides a single illustration using STM on one corpus of blockchain abstracts, and the Discussion and Conclusion acknowledge that future work is needed to develop guidelines for other algorithm families. While it is plausible that some guidelines (e.g., 'report computational procedures') are algorithm-agnostic, the transfer of the data-preparation and model-building guidelines to supervised ML or deep learning requires argument, not just assertion. Either soften the generalizability claim to a hypothesis or add a short rationale explaining which principles are algorithm-independent and why.
[Section 3.3] The shortlisting of candidate models is described qualitatively: the text states that models 10, 15, and 20 have the highest held-out likelihood, that residuals are lower from 35 to 60, and that models 35 and 45 were included because they have 'relatively higher' held-out likelihood and semantic coherence. No explicit decision rule or table of all twelve diagnostic values is provided. For a paper whose central theme is rigour, the model-selection criteria should be transparent and reproducible: for example, a table of all models with their held-out likelihood, residual, semantic coherence, and exclusivity values, and a clear statement of the threshold or ranking procedure used to discard K=5 and to retain K=35 and K=45.

minor comments (5)

[Section 3.4] The sentence 'For the topic labelling task, an external coder with the five most probable words and the ten most probable article abstracts corresponding to each of the top 10 topics' lacks a main verb; it should read 'was provided with' or 'was presented with'.
[Table 3 and throughout] There is inconsistent spelling between 'Rigor' (Table 3 header) and 'Rigour' (title and text). Choose one spelling convention and apply it consistently.
[Figure 2 and Figure 3] In the version under review, Figures 2 and 3 are not rendered properly; the axis labels and tick values appear as raw text interspersed with plotting instructions, making it impossible to verify the claims about held-out likelihood, semantic coherence, and exclusivity. Please ensure the figures are embedded as high-resolution images.
[Section 3.3] The phrase 'I did not use the lower bound metric' should briefly define what the lower bound is (the variational lower bound) and why it is relevant to model accuracy, since many readers will not be familiar with STM diagnostics.
[Section 4.4.2] The sentence 'There exists a plethora of algorithms to date' is awkward; consider revising to 'There is a plethora of algorithms available today' or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper is a methodological illustration with no equation-level reduction, fit-then-predict step, or load-bearing self-citation.

full rationale

This paper makes no formal derivation or predictive claim; it proposes practical guidelines for applying topic modelling and illustrates them on a blockchain abstract corpus. The model-selection step (Section 3.4) uses human intrusion tasks to choose K=20, but this is a reported application choice, not a prediction claimed to follow from an input fit. The guidelines in Table 3 are prescriptions assembled from prior literature and the author's own process; none is shown to be equivalent to the illustration's output by construction. Citations to Roberts et al. (2013, 2014, 2019), Chang et al. (2009), and others are external sources, not self-citations, and no uniqueness theorem is invoked to force a choice. The author's participation as one of the two coders is a validity limitation for the illustrated evaluation, not a circularity of argument, since the paper's central contribution does not reduce to that evaluation. The generalisation to other ML algorithms is asserted rather than proven, but an unsupported scope claim is not circularity. Since none of the enumerated circularity patterns is present, the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its central contribution, eight rigour guidelines, rests on assumptions about the validity of topic model diagnostics, the reliability of a small two-coder human evaluation, and the generalisability of a single worked example. The number of topics K and the top-10 topic threshold are hand-chosen quantities in the illustration.

free parameters (2)

Number of topics K = 20
Final model chosen after searchK diagnostics (held-out likelihood, residuals, semantic coherence, exclusivity) and two-coder intrusion tasks (Section 3.3-3.4). The choice is a model-selection decision, not a derived quantity.
Top 10 topics threshold = 10
Only the ten highest-probability topics were retained for labelling in Stage III; a hand-chosen inclusion criterion for the illustration (Section 3.4).

assumptions (3)

domain assumption Diagnostic metrics (held-out likelihood, residuals, semantic coherence, exclusivity) are valid proxies for topic model accuracy and interpretability.
The model shortlisting in Section 3.3 treats these metrics as sufficient indicators, without calibrating them to external ground truth.
domain assumption Word and topic intrusion tasks with two coders, including the author, provide reliable interpretability evidence.
Section 3.4 relies on this evaluation to choose K=20; no inter-coder reliability or blinding is reported.
ad hoc to paper A single illustrative corpus (blockchain abstracts) can support general-purpose guidelines for topic modelling and other ML algorithms.
The guidelines are generalised in Section 4 and the Conclusion without a multi-case or comparative validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Methodological Rigour in Algorithm Application: An Illustration of Topic Modelling Algorithm." pith.science (2026). https://pith.science/paper/URFXRY6Y

@misc{pith2026250700547,
  author       = {Pith},
  title        = {Pith review of: Methodological Rigour in Algorithm Application: An Illustration of Topic Modelling Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/URFXRY6Y}},
  note         = {Machine review of arXiv:2507.00547}
}

read the original abstract

The rise of advanced computational algorithms has opened new avenues for computationally intensive research approaches to theory development. However, the opacity of these algorithms and lack of transparency and rigour in their application pose methodological challenges, potentially undermining trust in research. The discourse on methodological rigour in this new genre of research is still emerging. Against this backdrop, I attempt to offer guidance on methodological rigour, particularly in the context of topic modelling algorithms. By illustrating the application of the structural topic modelling algorithm and presenting a set of guidelines, I discuss how to ensure rigour in topic modelling studies. Although the guidelines are for the application of topic modelling algorithms, they can be applied to other algorithms with context-specific adjustments. The guidelines are helpful, especially for novice researchers applying topic modelling, and editors and reviewers handling topic modelling manuscripts. I contribute to the literature on topic modelling and join the emerging dialogue on methodological rigour in computationally intensive theory construction research.

Figures

Figures reproduced from arXiv: 2507.00547 by the authors.

**Figure 1.** Three-Stage Iterative Process While data selection choices are important for the complete research project, for the scope of this paper, I only focus on three stages: data preparation, model building, and pattern evaluation. These three stages are more important to establish methodological rigour. Using an academic article abstract dataset, I illustrate the most important points where a researcher needs to establish… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 36 canonical work pages

[1]

Methodological Rigour in Algorithm Application: An Illustration of Topic Modelling Algorithm

Association for Information Systems Association for Information Systems AIS Electronic Library (AISeL) AIS Electronic Library (AISeL) ACIS 2024 Proceedings Australasian (ACIS) 12-10-2024 Methodological Rigour in Algorithm Application: An Illustration of Methodological Rigour in Algorithm Application: An Illustration of Topic Modelling Algorithm Topic Mode...

work page 2024
[2]

Top 10 Topics with Corresponding Top Five Words and Topic Labels 4 Discussion In this section, I offer a set of guidelines on how researchers can establish rigour when they apply algorithms. Although these guidelines are based on topic modelling algorithms, they can be applied to other algorithms (e.g., supervised machine learning algorithms) with context...

work page 2024
[3]

whether the association between a document and a topic makes sense

, the topic intrusion task assesses “ whether the association between a document and a topic makes sense” (Chang et al. 2009, p. 3). The goal of the human coder is to identify an intruder word inserted into a given topic and an intruder topic of a given article abstract which associates least with the abstract. The results of these two tasks are then eval...

work page 2009
[8]

Moreover, in their recent literature review of topic modelling, Günther and Joshi (2020) highlight the need for methodological rigour in topic modelling-based studies

However, none of these explicitly discuss the methodological rigour. Moreover, in their recent literature review of topic modelling, Günther and Joshi (2020) highlight the need for methodological rigour in topic modelling-based studies. Overall, there is little conversation about how to achieve methodological rigour with topic modelling. Therefore, the ob...

work page 2020
[9]

to a corpus of blockchain academic article abstracts, I discuss how to establish methodological rigour in each step of the process. STM was selected from a range of topic modelling algorithms because it is well suited for extracting prominent topics from a collection of short text such as article abstracts (Roberts et al. 2014). I propose a set of guideli...

work page 2014
[10]

It has been accepted for inclusion in ACIS 2024 Proceedings by an authorized administrator of AIS Electronic Library (AISeL)

https://aisel.aisnet.org/acis2024/10 This material is brought to you by the Australasian (ACIS) at AIS Electronic Library (AISeL). It has been accepted for inclusion in ACIS 2024 Proceedings by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contact elibrary@aisnet.org. Australasian Conference on Information Sys...

work page 2024
[11]

In practice, topics are described using the most probable words and assigning meaningful labels

Each topic is represented through a set of words, where each word has a probability of belonging to a topic. In practice, topics are described using the most probable words and assigning meaningful labels. Topic modelling algorithms provide these useful probabilistic measures with substantial predictive power (e.g. word distributions of topics) for us to ...

work page 2003
[12]

STM can incorporate document -specific metadata (e.g

is a combination and extension of multiple topic models including LDA. STM can incorporate document -specific metadata (e.g. publication date of each document) into the model to provide strong topical inference. It allows us to understand relationships between topics and document-level metadata. Dirichlet-multinomial regression topic model (Mimno and McCa...

work page 2008

Show all 38 references

[13]

Each of these topic modelling algorithms has its strengths and weaknesses

These topic modelling algorithms convert documents into word embedding representations before clustering them into topics. Each of these topic modelling algorithms has its strengths and weaknesses. Therefore, the choice of specific algorithms is context-dependent. Scholars hav...

2022
[14]

In this paper, I focus on the rigorous application of algorithms which is a crucial part of CITC research as this research relies on the patterns surfaced through algorithms to develop theory. The application of algorithms in a research setting is an iterative process from pre...

2011
[17]

To find the optimal number of topics (denoted by K) that is appropriate for our dataset, I used the ‘searchK’ functionality in STM

Therefore, a researcher needs to find a balance between the model’s predictive accuracy and interpretability that suits the research goal when it comes to evaluating this decision. To find the optimal number of topics (denoted by K) that is appropriate for our dataset, I used ...

2012
[18]

2019; Roberts et al

Semantic coherence is based on the idea that high -probability words tend to co - occur together frequently, thus it assesses whether a topic is internally consistent (Roberts et al. 2019; Roberts et al

2019
[19]

First, I performed several experimental rounds of topic models to refine the lists of collocation of words and custom stop words

Therefore, it is recommended to use both semantic coherence and exclusivity for evaluating the interpretability of models. First, I performed several experimental rounds of topic models to refine the lists of collocation of words and custom stop words. This is often a necessar...

2016
[24]

Researchers must critically assess the suitability of standard practices or procedures followed by prior studies against their research context before applying them

Guidelines for Achieving Methodological Rigor 4.1.2 Assess the Suitability of Existing Standard Procedures Günther and Joshi (2020) find that some topic modelling studies have followed prior practices without giving sound justifications. Researchers must critically assess the ...

2020
[25]

It is important to provide sound justifications for each of the decisions researchers make along the way. 4.2 Model Building 4.2.1 Understand How Hyperparameters Influence Model Accuracy and Interpretability One needs to understand hyperparameters (i.e., input parameters like ...

2024
[26]

Therefore, the balance between model accuracy and interpretability should be guided by the research goal. 4.3 Pattern Evaluation 4.3.1 Explore Existing Evaluation Tasks Researchers can borrow existing evaluation tasks to assess whether the patterns are meaningful and relevant ...

2020
[28]

In general, research transparency refers to the practice of explicitly reporting how the research study is undertaken, specifically the design and execution-related methodological details. Transparency is becoming ever more important for assessing the rigour in algorithmic app...

2021
[31]

The proposed guid elines are useful for novice scholars applying topic modelling and reviewers of topic modelling manuscripts

to a corpus of blockchain academic article abstracts, I discuss how to establish methodological rigour in each step of the process. The proposed guid elines are useful for novice scholars applying topic modelling and reviewers of topic modelling manuscripts. They can be applie...

2020
[32]

By focusing on the rigorous application of algorithms , I also join the emerging dialogue on methodological rigour in CITC research (e.g., Günther and Joshi 2020; Miranda et al. 2022a). The guidelines are helpful, especially for novice scholars applying topic modelling and for...

2020
[34]

Bertopic: Neural Topic Modeling with a Class -Based Tf-Idf Procedure,

"Bertopic: Neural Topic Modeling with a Class -Based Tf-Idf Procedure," arXiv preprint arXiv:2203.05794). Günther, W., and Joshi, M. P

arXiv
[35]

For the topic intrusion tasks, I configured 10 cases per model

The word intrusion tasks were performed per topic for all the selected models. For the topic intrusion tasks, I configured 10 cases per model. That is each coder performed 10 topic intrusion tasks per model. Figures 4 and 5 depict snippets of word intrusion and topic intrusion...

2024
[36]

How Information Contributed after an Idea Shapes New High-Quality Ideas in Online Ideation Contests,

"How Information Contributed after an Idea Shapes New High-Quality Ideas in Online Ideation Contests," MIS Quarterly (46:2), pp. 1195-1208. Australasian Conference on Information Systems Amadoru 2024, Canberra Methodological Rigour in Topic Modelling 13 Mimno, D., and McCallum, A

2024
[37]

Mixing Dirichlet Topic Models and Word Embeddings to Make Lda2vec,

"Mixing Dirichlet Topic Models and Word Embeddings to Make Lda2vec," arXiv preprint arXiv:1605.02019). Nakamoto, S

arXiv
[38]

Evaluating and Developing Theories in the Information Systems Discipline,

"Evaluating and Developing Theories in the Information Systems Discipline," Journal of the Association for Information Systems (13:1), pp. 1-30. Australasian Conference on Information Systems Amadoru 2024, Canberra Methodological Rigour in Topic Modelling 14 Yang, L., and Shami, A

2024
[39]

TM-OKC: An Unsupervised Topic Model for Text in Online Knowledge Communities,

"TM-OKC: An Unsupervised Topic Model for Text in Online Knowledge Communities," MIS Quarterly (48:3), pp. 931-978. Appendix Figure A1 presents the complete list of topics that surfaced from the blockchain academic article abstracts through the application of STM. Figure A1: Li...

2024
[60]

whether a topic has human - identifiable semantic coherence

1 Refer to Roberts et al. (2019) for detailed discussion. Australasian Conference on Information Systems Amadoru 2024, Canberra Methodological Rigour in Topic Modelling 6 Considering the models with high held -out likelihood and semantic coherence I chose 10, 15, and 20 models...

2019
[2009]

The encoding of know -how knowledge can increase the repertoire of skills needed for scholars who do not have a strong computer science background or programming skills. Therefore, sharing procedures and programming codes helps others to reuse the procedure with less effort an...

2023
[2011]

As the text mining community is constantly working in this area, we can expect new developments such as fully labelled interpretable topic models in the future

Therefore, exploring the available evaluation tasks that are up -to-date, especially in the computer science and social science community will be beneficial. As the text mining community is constantly working in this area, we can expect new developments such as fully labelled ...

2023
[2012]

They are being used widely by information systems and management scholars for theory -development purposes (Günther and Joshi 2020; Hannigan et al

2020
[2013]

Against this backdrop, I attempt to provide a set of guidelines for establishing methodological rigour for scholars who wish to use algorithms in research settings

, directions for using algorithms in theory development are still emerging (e.g., simulation research guidelines by Dong 2022). Against this backdrop, I attempt to provide a set of guidelines for establishing methodological rigour for scholars who wish to use algorithms in res...

2022
[2014]

From a technical point of view, STM can be best understood as running a regression on to topic proportion over document -level metadata. STM is not only capable of detecting topics and topic change over user -defined covariates, but also inferring topic correlations and decidi...

2019
[2016]

Australasian Conference on Information Systems Amadoru 2024, Canberra Methodological Rigour in Topic Modelling 11 4.4.2 Report to Help Build the Community There are emerging initiatives within scholarly communities (e.g., Management Science) to provide programming codes and ot...

2024
[2019]

2022b) and social movement s such as the Women’s March (Syed and Silva 2023)

For example, topic modelling has been used to understand IT innovation discourse (Miranda et al. 2022b) and social movement s such as the Women’s March (Syed and Silva 2023). Moreover, topic modelling algorithms are frequently used in CITC research (e.g., Jiang et al. 2021; Mi...

2023
[2020]

Spectral

I also added a few custom stop words through our exploration of the abstracts. Next, I removed special characters and numbers. As extra whitespaces may be retained after this step, I removed all the white spaces including the leading and trailing blank spaces. Then I performed...

2012
[2021]

I set the start year to 2008 which is the birth year of Bitcoin because the concept of blockchain originated through bitcoin (Nakamoto 2008)

ABI/INFORM Global includes scholarly publications covering a wide variety of business-related disciplines including prominent IS journals such as MIS Quarterly. I set the start year to 2008 which is the birth year of Bitcoin because the concept of blockchain originated through...

2008
[2022]

it is unlikely the quality of the whole will be high if the quality of the parts is not high

Often, researchers possess a limited understanding of an algorithm’s inner workings, and this obscurity can make errors go unnoticed. In addition, algorithmic output is influenced by the data quality and hyperparameters specified by the researcher (Shrestha et al. 2021; Yang a...

2021
[2023]

Due to the increasing interest, a few scholars have also offered topic modelling tutorials to guide novice researchers (Debortoli et al

Therefore, topic modelling algorithms offer a timely and relevant opportunity to demonstrate how researchers can achieve methodological rigour when using algorithms in the research process. Due to the increasing interest, a few scholars have also offered topic modelling tutori...

2016
[2024]

Moreover, data preparation decisions such as handling missing data and standardization can be important for the performance of the model (Choudhury et al

can be more suitable than using LDA or STM. Moreover, data preparation decisions such as handling missing data and standardization can be important for the performance of the model (Choudhury et al. 2021; Shmueli and Koppius 2011). Therefore, future studies can extend this wor...

2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Methodological Rigour in Algorithm Application: An Illustration of Topic Modelling Algorithm

Association for Information Systems Association for Information Systems AIS Electronic Library (AISeL) AIS Electronic Library (AISeL) ACIS 2024 Proceedings Australasian (ACIS) 12-10-2024 Methodological Rigour in Algorithm Application: An Illustration of Methodological Rigour in Algorithm Application: An Illustration of Topic Modelling Algorithm Topic Mode...

work page 2024

[2] [2]

Top 10 Topics with Corresponding Top Five Words and Topic Labels 4 Discussion In this section, I offer a set of guidelines on how researchers can establish rigour when they apply algorithms. Although these guidelines are based on topic modelling algorithms, they can be applied to other algorithms (e.g., supervised machine learning algorithms) with context...

work page 2024

[3] [3]

whether the association between a document and a topic makes sense

, the topic intrusion task assesses “ whether the association between a document and a topic makes sense” (Chang et al. 2009, p. 3). The goal of the human coder is to identify an intruder word inserted into a given topic and an intruder topic of a given article abstract which associates least with the abstract. The results of these two tasks are then eval...

work page 2009

[4] [8]

Moreover, in their recent literature review of topic modelling, Günther and Joshi (2020) highlight the need for methodological rigour in topic modelling-based studies

However, none of these explicitly discuss the methodological rigour. Moreover, in their recent literature review of topic modelling, Günther and Joshi (2020) highlight the need for methodological rigour in topic modelling-based studies. Overall, there is little conversation about how to achieve methodological rigour with topic modelling. Therefore, the ob...

work page 2020

[5] [9]

to a corpus of blockchain academic article abstracts, I discuss how to establish methodological rigour in each step of the process. STM was selected from a range of topic modelling algorithms because it is well suited for extracting prominent topics from a collection of short text such as article abstracts (Roberts et al. 2014). I propose a set of guideli...

work page 2014

[6] [10]

It has been accepted for inclusion in ACIS 2024 Proceedings by an authorized administrator of AIS Electronic Library (AISeL)

https://aisel.aisnet.org/acis2024/10 This material is brought to you by the Australasian (ACIS) at AIS Electronic Library (AISeL). It has been accepted for inclusion in ACIS 2024 Proceedings by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contact elibrary@aisnet.org. Australasian Conference on Information Sys...

work page 2024

[7] [11]

In practice, topics are described using the most probable words and assigning meaningful labels

Each topic is represented through a set of words, where each word has a probability of belonging to a topic. In practice, topics are described using the most probable words and assigning meaningful labels. Topic modelling algorithms provide these useful probabilistic measures with substantial predictive power (e.g. word distributions of topics) for us to ...

work page 2003

[8] [12]

STM can incorporate document -specific metadata (e.g

is a combination and extension of multiple topic models including LDA. STM can incorporate document -specific metadata (e.g. publication date of each document) into the model to provide strong topical inference. It allows us to understand relationships between topics and document-level metadata. Dirichlet-multinomial regression topic model (Mimno and McCa...

work page 2008