Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry

Hannah Deters; Jakob Droste; Jannik Fischbach; Kurt Schneider; Marc Herrmann; Martin Obaidi; Nicolas Vo{\ss}

REVIEW 4 major objections 5 minor 1 cited by

Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Team ranking lifts app-review routing to 79.2% correct

desk verdict Industrial deployment of the explanation-need taxonomy with a useful routing process, but the headline 79.2% team-assignment figure is resubstitution, so read the paper for the process and not for the accuracy claim. read the letter →

arxiv 2501.08087 v2 pith:3AU2NRMG submitted 2025-01-14 cs.SE

Martin Obaidi , Nicolas Vo{ss} , Hannah Deters , Jakob Droste , Marc Herrmann , Jannik Fischbach , Kurt Schneider This is my paper · ORCID

classification cs.SE

keywords explainabilityappreviewsexplanationneedstaxonomyteamassignmentrequirementsengineeringsurveystudynavigation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a semi-automated pipeline that turns app-store reviews containing explanation needs into a taxonomy category, a ranked list of internal teams that could answer, and a source that already contains the answer. Working with a navigation-app company and 2,366 scraped reviews, the authors find that ranking several candidate teams instead of picking one raises the chance of a correct assignment from 52.5% to 79.2%. The company was able to address 139 of 158 explanation needs, and only 15 required drafting a new response. If the results hold, companies can make explainability a managed requirement rather than a repetitive manual chore.

What carries the argument

The central machinery is an extended taxonomy of explanation needs plus a hierarchical team-assignment rule. The taxonomy, taken from Droste et al., sorts reviews into categories such as System Behavior, Interaction, Privacy & Security, Domain Knowledge, and User Interface, and is extended here with Business, Meta Information, and Feature Questions to capture needs that otherwise fall between categories. A word-and-phrase filter built from Obaidi's 245 phrases and Droste's trigger words flags candidate reviews, and two requirements engineers verify the labels as ground truth. For each taxonomy category, the system lists internal teams that received at least 25% of the support staff's assignments, ranked by frequency; this ranked list is the 'reference point.' A separate script compares the review text against support articles using Python's difflib SequenceMatcher and against past Google Play responses to identify a 'source.'

What would settle it

Take a fresh batch of explanation-needy reviews from the same company, have the support teams actually work the tickets, and record which team resolves each request; if the hierarchy's top-ranked team matches the resolving team no more than 52.5% of the time, the 79.2% figure overstates routing quality.

Watch

Extended reading notes

Core claim

The central claim is that a taxonomy-to-team mapping, built from the judgments of four support-staff members, lets a company route explanation-needy reviews to a ranked list of internal teams with 79.2% cumulative accuracy. The pipeline first flags candidate reviews with a word-and-phrase filter, then human requirements engineers verify the labels to produce ground truth. For each taxonomy category, any team that received at least 25% of the support staff's assignments is placed in a hierarchy, ranked by frequency, and the top three options count as a correct assignment. Using only the first-ranked team yields 52.5% accuracy, so the ranking adds 26.7 percentage points. A separate script finds a source—a support article, a past Google Play response, or a newly drafted answer—and the company reports it can answer 88% of the identified explanation needs.

Load-bearing premise

The load-bearing premise is that the assignments made by four support employees in interviews and surveys define the 'correct team' and can serve as both the training data and the ground truth for the evaluation; the paper never checks whether the assigned team actually resolves the user's request.

Editorial extensions

If this is right

If the 79.2% accuracy reproduces, companies can auto-route reviews to a short ranked list and have a human only confirm the top choice, cutting repetitive work.
The 88% addressability figure suggests most user explanation needs are recurring and can be served from existing support articles or past responses, not fresh writing.
The 25% inclusion threshold is a simple data-driven rule for turning a small set of expert labels into a team-assignment hierarchy, no machine learning required.
The taxonomy extensions (Business, Meta Information, Feature Questions) provide a template for other companies whose reviews ask about the provider or span multiple categories.
Apple App Store reviews are harder to source because past review responses are not available, so the pipeline works best where a reply history exists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 79.2% accuracy is measured against the support staff's own assignments, not against whether the assigned team actually resolves the request; the paper itself flags this, so a pilot deployment with ticket outcomes is the natural next test.
The 25% threshold likely needs recalibration as team sizes and responsibilities change, but the ranked-list idea should transfer to any organization that has a ticket taxonomy and a support team.
The pipeline's practical value depends on the cost of the human verification step, which the paper does not quantify; a cost-benefit estimate would show whether the automation pays for itself.
Feeding the ranked team list into a lightweight classifier or language model could test whether the manual verification step can be removed once more labeled data accumulates.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Two things to know. First, this is the first real exercise of the Droste et al. explanation-need taxonomy inside a live company, and it adds the practical machinery of routing reviews to internal teams and to reusable sources. Second, the headline 79.2% team-assignment accuracy is a same-data resubstitution figure: the same four support employees' assignments were used both to build the team hierarchy and to serve as ground truth for the evaluation. The paper itself concedes in Section V.C that the accuracy of these team assignments in resolving the explanation needs has not been evaluated. So treat 79.2% as internal consistency, not predicted routing quality.

What the paper does well: it is a transparent single-company case study. The taxonomy extension (Business, Meta Information, Feature Questions) is a genuine practical addition. The reporting of word-filter precision/recall trade-offs is honest, and the observation that 13 of 15 newly written responses were for Apple App Store reviews gives practitioners a concrete platform asymmetry to work with. The limitations section covers the small sample, the single-company scope, and the lack of longitudinal data.

Where the soft spots land: the core accuracy claim is circular. A leave-one-out evaluation would have tested whether the hierarchy generalizes; they did not run one. Interrater agreement for team assignment is low (Cohen's kappa 0.146–0.558), so the 'correct team' is not a stable target even among the four raters. No code or data is shipped, which limits independent verification, though that is a moderate issue for a case study. If the claims were reframed as 'our process reproduces the support team's majority assignments on the training data,' the paper would be solid.

Who should read it: researchers working on app review analysis, requirements engineers interested in explainability in practice, and people designing support tooling. I would send it to a serious referee, not desk reject. The referee should demand that the evaluation claims be scaled back to internal consistency or supported by held-out validation.

My bottom line: worth reading and worth citing for the taxonomy extension and the industrial process, but do not quote the 79.2% as an accuracy result.

Referee Report

4 major / 5 minor

Summary. The paper reports on a semi-automated system for managing explanation needs in app reviews at Graphmasters GmbH. The pipeline scrapes 2,366 Google Play and Apple App Store reviews, uses a word/phrase filter to detect explicit and implicit explanation needs, assigns reviews to taxonomy categories, and then maps categories to internal teams via a ranked hierarchy built from interviews and surveys with four support employees. External sources such as support articles and prior review responses are retrieved to answer the needs. The central reported results are that hierarchical team assignment achieves 79.2% validity (versus 52.5% for single-team assignment) and that 139 of 158 explanation needs were addressed by the company. The paper also reports low interrater agreement in taxonomy and team assignments, and several validity threats are discussed.

Significance. If the central figure were a true out-of-sample accuracy estimate, the paper would provide a useful industrial case study: it demonstrates a concrete taxonomy-to-team routing workflow, integrates real support articles and past review responses, and is honest about the challenges of interrater agreement. The authors also make their data collection process transparent, including interview guidelines and the use of a publicly documented taxonomy. However, the headline 79.2% team-assignment validity is computed on the same interview and survey data used to construct the team hierarchy, with a 25% inclusion threshold fit to those assignments, and the paper itself concedes that resolution accuracy was not evaluated. As presented, the quantitative claim does not measure routing quality for unseen reviews; it measures reproduction of the raters' majority tendencies on training data. The qualitative findings about taxonomy ambiguity and the practical difficulties of team assignment remain valuable, but the main accuracy claim needs a proper validation procedure before it can be accepted.

major comments (4)

[IV.E and V.A] The 79.2% team-assignment validity is computed against the same interview and survey responses that were used to build the team hierarchy in Section III.E.1. This is a resubstitution metric: the hierarchy is fit to the raters' assignments and then evaluated on those same assignments, while counting second- and third-ranked teams as correct. It therefore does not estimate how well the hierarchy would route unseen reviews. A leave-one-out cross-validation over the four raters, or an independently held-out set of reviews, is required before the claim that 'the correct team could be assigned in 79.2% of cases' can be supported.
[III.E.1 and Table VII] The ground truth for 'correct team' is unstable across the four support employees: Table VII reports Fleiss' κ = 0.307 for reviews 1-25, Cohen's κ = 0.558 for reviews 26-50, and Cohen's κ = 0.146 for reviews 51-75. Because the 25% inclusion threshold in Section III.E.1 is itself fit to these same assignments, the resulting hierarchy and the validity estimate are jointly determined by the same noisy data. The paper should either justify the threshold with independent data or report a sensitivity analysis over the threshold and over subsets of raters.
[V.C] The manuscript explicitly concedes that 'the accuracy of these team assignments in resolving the explanation needs has not been evaluated.' This directly limits the central claim: the 79.2% figure measures agreement with the raters' majority assignments, not whether the assigned team actually resolves the user's request. The abstract and RQ1 currently present the figure as routing accuracy, which overstates what the evaluation supports. Either the evaluation must be extended to resolution outcomes, or the claims should be reworded to describe rater-agreement, not correctness.
[III.C.3 and III.C.4] The dataset size is reported inconsistently: the text states 2,366 reviews were scraped, while Section III.C.4 later mentions 2,376 reviews in the manual labeling, and Table I sums to 2,365 reviews. The paper also reports 158 explanation needs from a total that changes by ten because of multi-need reviews. These numbers need to be reconciled, since the recall and precision figures in Section IV.A depend on a clearly defined denominator.

minor comments (5)

[III.I] There is a typo in Section III.I: 'Cohne's kappa' should be 'Cohen's kappa'.
[Table I] The row for 'nunav truck' appears to have missing or misaligned entries; the table should be reformatted so the counts for explicit, implicit, potential, and none are clear.
[IV.C] The phrase 'requriement engineer' in Section IV.C contains a typo; it should be 'requirements engineer'.
[II.A and Table VIII] The taxonomy description in Section II.A lists five main categories plus Meta Information and Timing, but Section V.B.1 adds Business and Feature Questions, and Table VIII contains additional categories such as Operation, Tutorial, and Consequences. The relationship between these category sets should be clarified, since the reader cannot tell which taxonomy was used for the final team assignments.
[References] References [23] and [24] both attribute the same Landis and Koch work to different venues and years; one of them appears to be a duplicate with incorrect bibliographic details.

Circularity Check

2 steps flagged · score 7.0 of 10

The 79.2% team-assignment accuracy is a resubstitution result: the hierarchy and its ground truth come from the same four support employees' assignments, and the paper concedes resolution accuracy was not evaluated.

fitted input called prediction [Section III.E.1 (Reference Point), Section IV.E, and Section V.C]
"The data for this assignment was gathered from the interviews with the Graphmasters support team. ... Any team associated with at least 25% of a taxonomy category is included in this hierarchy. ... By applying the 25% threshold for team inclusion in the taxonomy category assignment, an average validity of 79,2% was achieved. This calculation considers cases where a team was ranked as the second or third option but was still considered a correct match. ... the accuracy of these team assignments in resolving the explanation needs has not been evaluated."

The team hierarchy is computed from the same interview/survey assignments made by the four Graphmasters support employees that are then used as the 'correct match' ground truth for the 79.2% figure. A team enters the hierarchy if it received at least 25% of the assignments for a taxonomy category; the validity calculation then checks whether those same assignments fall within the ranked team list, counting second and third options as correct. This is an in-sample resubstitution estimate, not a prediction on unseen reviews or on actual resolution outcomes. The paper's V.C statement confirms that whether the assigned team actually resolves the need was never evaluated, so 79.2% measures the hierarchy's agreement with its own training data rather than routing quality.
fitted input called prediction [Section IV.B (Detecting Taxonomy Categories Using a Word Filter)]
"Both the fine and broad word filters were developed based on an evaluation of reviews that had already been categorized with explanation needs. These filters are not suitable for use with new apps and can only be applied after the taxonomy categories for other reviews of an already evaluated app have been established."

The precision, recall, and F1 values reported in Table IV are computed on the same already-categorized reviews that were used to develop the fine and broad filters (the text mentions 'after adjustments'). Since the filters are explicitly acknowledged to be unsuitable for new apps and were not evaluated on held-out reviews, the reported metrics are resubstitution performance rather than estimates of classification accuracy for unseen app reviews.

full rationale

The paper's central quantitative claim, that hierarchical team assignment achieves 79.2% accuracy versus 52.5% for single-team assignment, reduces to an in-sample fit. The hierarchy is built from the same support-team interview/survey assignments that define the 'correct team,' and the 79.2% figure is obtained by checking whether those same assignments fall within the ranked team list. The paper itself concedes in Section V.C that the accuracy of these team assignments in resolving the explanation needs has not been evaluated. This is a resubstitution result, not an independent estimate of routing quality; the low interrater reliability (Kappa 0.146-0.558) further shows that the 'correct team' is not a stable external target. A second, less central in-sample evaluation affects the word-filter metrics in Section IV.B, where filters were developed on the already-categorized reviews and then evaluated on them. The study does contain independent components: the requirements engineers' taxonomy labeling, the external support-article retrieval, and the 88% addressability figure based on company self-report. But because the headline 79.2% versus 52.5% comparison is an agreement of the fitted hierarchy with its own training data, the circularity burden is high, though not total.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a published taxonomy of explanation needs, on the assumption that support staff assignments are a valid proxy for the correct team, and on transferable word-filter datasets. The explicit numeric parameter is the 25% team-inclusion threshold, and the word filter vocabularies are tuned on the labeled reviews. No invented entities such as new forces, new particles, or new conserved quantities are introduced.

free parameters (3)

Team inclusion threshold = 25% of assignments to a taxonomy category
Section III.E: teams with at least 25% of participant assignments for a category enter the hierarchy. The threshold is chosen without sensitivity analysis and directly affects the 79.2% validity.
Fine and broad word filter vocabularies
Section IV.B: filters were developed on already-labeled reviews and are app-specific; the paper says they are not portable to new apps, so they act as fitted features rather than fixed inputs.
F-score beta weighting = 0.2
Section IV.B: beta is chosen to prioritize precision when comparing fine and broad filters; changing it changes which filter is preferred, though not the central routing result.

assumptions (5)

domain assumption The Droste et al. taxonomy categories are an adequate and complete way to represent explanation needs.
The entire classification and team-routing pipeline depends on this taxonomy; the paper extends it with Business, Meta Information, and Feature Questions, indicating incompleteness.
domain assumption Support team members' assignments of reviews to internal teams are a valid proxy for the correct team.
Used as both training and ground truth for the 79.2% accuracy; no independent confirmation exists that the assigned team can resolve the request.
domain assumption Word and phrase filter datasets from prior work transfer to this review corpus after translation.
Section III.C.4: filter lists from Obaidi and Droste are applied without reduction; no per-app validation is done before use.
domain assumption Difflib SequenceMatcher similarity reliably identifies relevant support articles.
Section III.E.2: the source-finding step relies on string similarity between user queries and articles; the validity of the matching is not measured.
domain assumption Reviews scraped from Google Play and Apple App Store constitute the relevant universe of user explanation needs.
The study excludes other support channels and analyzes a static snapshot, as acknowledged in the threats to validity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry." pith.science (2026). https://pith.science/paper/3AU2NRMG

@misc{pith2026250108087,
  author       = {Pith},
  title        = {Pith review of: Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3AU2NRMG}},
  note         = {Machine review of arXiv:2501.08087}
}

read the original abstract

Providing explanations in response to user reviews is a time-consuming and repetitive task for companies, as many reviews present similar issues requiring nearly identical responses. To improve efficiency, this paper proposes a semi-automated approach to managing explanation needs in user reviews. The approach leverages taxonomy categories to classify reviews and assign them to relevant internal teams or sources for responses. 2,366 app reviews from the Google Play Store and Apple App Store were scraped and analyzed using a word and phrase filtering system to detect explanation needs. The detected needs were categorized and assigned to specific internal teams at the company Graphmasters GmbH, using a hierarchical assignment strategy that prioritizes the most relevant teams. Additionally, external sources, such as existing support articles and past review responses, were integrated to provide comprehensive explanations. The system was evaluated through interviews and surveys with the Graphmasters support team, which consists of four employees. The results showed that the hierarchical assignment method improved the accuracy of team assignments, with correct teams being identified in 79.2% of cases. However, challenges in interrater agreement and the need for new responses in certain cases, particularly for Apple App Store reviews, were noted. Future work will focus on refining the taxonomy and enhancing the automation process to reduce manual intervention further.

Figures

Figures reproduced from arXiv: 2501.08087 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Identifying Explanation Needs: Towards a Catalog of User-based Indicators
cs.SE 2025-06 conditional novelty 5.0 of 10

A survey of 66 users produces a preliminary catalog of behavior, system event, emotion, and physical reaction indicators that could signal a need for explanation.

Reference graph

Works this paper leans on

51 extracted references · 49 canonical work pages · cited by 1 Pith paper

[1]

Based in Hanover, Germany, the company develops apps and systems like NUNA V 2, which cater to various use cases such as personal navigation, logis- tics, and trucking

Graphmasters GmbH: Graphmasters GmbH 1 is a com- pany specializing in intelligent navigation solutions, focusing on optimizing trafﬁc ﬂow and reducing congestion through real-time route guidance. Based in Hanover, Germany, the company develops apps and systems like NUNA V 2, which cater to various use cases such as personal navigation, logis- tics, and tr...

work page
[2]

Reviews Scraping: A custom software tool was devel- oped to scrape reviews from the Google Play Store and Apple App Store . The scraped data included metadata such as star ratings, dates, responses to reviews, review titles, and the text 1https://www.graphmasters.net/ 2https://www.graphmasters.net/nunav content of the reviews. This data was consolidated i...

work page
[3]

While the primary focus was on the Nunav Navigation App , three additional apps were included to expand the dataset due to their structural similarity

Dataset Creation: The dataset for this study was con- structed using user reviews from several apps 3 developed by Graphmasters GmbH, including the Nunav Navigation App , Nunav Courier App , Nunav Trucks App , and Nunav Bus App , sourced from both the Google Play Store and Apple App Store. While the primary focus was on the Nunav Navigation App , three ad...

work page
[4]

An approach with large language models was not possible, because of the limited training data

Dataset Analysis: Explicit and implicit explanation needs in the reviews were identiﬁed using a word and phrase ﬁltering method. An approach with large language models was not possible, because of the limited training data. The guidelines, description of implicit and explicit explanation need by Droste et al. [3] are online available [19]. Implicit explan...

work page
[5]

The speciﬁc segments of the reviews contain- ing explanation needs were manually marked and categorized as either explicit or implicit explanation needs

During this 3https://www.graphmasters.net/nunav 4https://www.deepl.com/ 5https://www.maxqda.com/ TABLE I IDENTIFIED NEED FOR EXPLANA TION ACCORDING TO WORD AND PHRASE FILTER OF THE INDIVIDUAL APPS Explanation Need App Explicit Implicit Potential None nunav navigation 24 28 425 1659 nunav truck 00 0 7 nunav logistics 2 1 49 135 nunav bus 0 1 10 25 Total 26...

work page
[6]

Each taxonomy category is mapped to one or more reference points within the company

Reference Point: To automatically assign a reference point for addressing the explanation needs identiﬁed in the reviews, taxonomy categories are employed. Each taxonomy category is mapped to one or more reference points within the company. The data for this assignment was gathered from the interviews with the Graphmasters support team. If the mapping of ...

work page
[7]

Source: The source containing the answer to the expla- nation need is determined using an API from the Graphmasters support website. The script, which uses the API, compares the similarity between support articles on the Graphmasters support page and the user’s query using the SequenceMatcher from Python’s difﬂib module

work page
[8]

I ﬁnd the upcoming motorcycle version exciting - what will make it different?

The support articles on the Graphmasters support page were created manually by Graph- masters support members for general explanatory purposes. This comparison helps identify whether a speciﬁc support article addresses the question at hand. If no relevant support article is found, the script retrieves all the answer from responses provided to the review i...

work page 2024

Show all 51 references

[9]

Mobile 43%

Mobile 28% Operation 1. Mobile 43%
[10]

Support 54%

Support 41% Tutorial 1. Support 54%
[11]

Support 35%

Mobile 30% Navigation 1. Support 35%
[12]

Routing 42%

Mobile 35% Algorithms 1. Routing 42%
[13]

Mobile 42%

Support 28% Consequences Mobile 88% Unexpected system behavior 1. Mobile 42%
[14]

UI/UX 55%

Routing 33% Bugs & Crashes Mobile 57% User Interface 1. UI/UX 55%
[15]

Mobile 75%

Mobile 36% Privacy 1. Mobile 75%
[16]

Meta 25% Security n.a. n.a. Meta information 1. Mobile 36%
[17]

Support 36% Terminology n.a. n.a. System-speciﬁc elements 1. Support 75%
[18]

Addressable Results Of the 158 identiﬁed explanation needs, 139 (88%) were successfully resolved

Mobile 25% F . Addressable Results Of the 158 identiﬁed explanation needs, 139 (88%) were successfully resolved. The remaining needs could not be addressed by the company. Common reasons for unresolved explanation needs include cases that require further inquiries, or issues u...
[19]

[3], as was also discussed in their original publication

Taxonomy Customization: During the assignment of taxonomy categories by the two requirements engineers, it became apparent that not every explanation need could be categorized using the taxonomy proposed by Droste et al. [3], as was also discussed in their original publication...
[20]

Categorization into a Taxonomy: The validity of the taxonomy category assignments (see V), based on the partici- pants’ evaluations, indicates challenges in assigning categories. These challenges arose either from a lack of complete under- standing or from subjective decision-...
[21]

To address this, a probability-based ranking system for reference point assignment was developed

Assignment to a Reference Point: The assignment of reference points among the participants was inconsistent, with Kappa values ranging from 0.146 to 0.558, indicating that clear team assignments could not be made for many explanation needs. To address this, a probability-based...
[22]

Relevance of Addressing Explanation Needs in Industry: The ﬁndings of this study highlight the practical relevance of explanation needs in industry settings. Our results show that Graphmasters was able and willing to address 88% of the identiﬁed explanation needs, underscoring...
[23]

Misunder- standing the taxonomy can result in incorrect categorizations, which may affect the outcomes of the study

Construct V alidity: When attempting to classify reviews using an unfamiliar taxonomy, the complexity of the taxonomy can lead to misinterpretations and improper usage. Misunder- standing the taxonomy can result in incorrect categorizations, which may affect the outcomes of th...
[24]

For instance, if interviews are held during par- ticipants’ personal time, they may rush through the process, negatively impacting data quality

Internal V alidity: The timing and context of the in- terviews could introduce biases that affect the participants’ responses. For instance, if interviews are held during par- ticipants’ personal time, they may rush through the process, negatively impacting data quality. To ad...
[25]

This small sample may limit the statistical signiﬁcance of the data, leading to potentially inaccurate conclusions

Conclusion V alidity: The limited sample size of four participants poses a potential threat to conclusion validity. This small sample may limit the statistical signiﬁcance of the data, leading to potentially inaccurate conclusions. To address this, participants were selected b...
[26]

External V alidity: Ensuring external validity was chal- lenging due to the limited sample size and the focus on a single company. The results of this study may not generalize to other organizations or software systems, as the study was conducted within the speciﬁc context of ...

2022
[27]

Explainability as a non-functional requirement,

M. K ¨ohl, K. Baum, M. Langer, D. Oster, T. Speith, and D. Bohlender, “Explainability as a non-functional requirement,” 27th International Requirements Engineering Conference , pp. 363–368, 2019

2019
[28]

Explainability as a non-functional requirement: challenges and recommendations,

L. Chazette and K. Schneider, “Explainability as a non-functional requirement: challenges and recommendations,” Requirements Engineer- ing, vol. 25, pp. 493–514, 2020

2020
[29]

Explanations in everyday software systems: Towards a taxonomy for explainability needs,

J. Droste, H. Deters, M. Obaidi, and K. Schneider, “Explanations in everyday software systems: Towards a taxonomy for explainability needs,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE) , 2024, pp. 55–66

2024
[30]

Explanation needs in app reviews: Taxonomy and automated detection,

M. Unterbusch, M. Sadeghi, J. Fischbach, M. Obaidi, and A. V ogelsang, “Explanation needs in app reviews: Taxonomy and automated detection,” in 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW). IEEE, 2023, pp. 102–111

2023
[31]

L. Ceci. (2024) Number of apps available in leading app stores as of august 2024. [Online]. Available: https://www.statista.com/statistics/ 276623/number-of-apps-available-in-leading-app-stores/

2024
[32]

The impact of automation on employment: Just the usual structural change?

B. V ermeulen, J. Kesselhut, A. Pyak, and P . Saviotti, “The impact of automation on employment: Just the usual structural change?” Sustain- ability, p. 27, 2018

2018
[33]

Deep learning: A critical appraisal,

G. Marcus, “Deep learning: A critical appraisal,” Computing Research Repository, p. 27, 2018

2018
[34]

Accurate and prompt answering framework based on customer reviews and question-answer pairs,

E. Kim, H. Y oon, J. Lee, and M. Kim, “Accurate and prompt answering framework based on customer reviews and question-answer pairs,” Expert Systems with Applications , vol. 203, p. 12, 2022

2022
[35]

How can i improve my app? classifying user reviews for software maintenance and evolution,

S. Panichella, A. Sorbo, E. Guzman, C. Visaggio, G. Canfora, and H. Gall, “How can i improve my app? classifying user reviews for software maintenance and evolution,” 21st International Conference on Software Maintenance and Evolution , pp. 281 – 290, 2015

2015
[36]

A taxonomy of explanations to support explainability-by-design,

N. Tsakalakis, S. Stalla-Bourdillon, T. Huynh, and L. Moreau, “A taxonomy of explanations to support explainability-by-design,” 4 2022

2022
[37]

Exploring explainability: a deﬁnition, a model, and a knowledge catalogue,

L. Chazette, W. Brunotto, and T. Speith, “Exploring explainability: a deﬁnition, a model, and a knowledge catalogue,” 29th international requirements engineering conference , vol. 29, pp. 197–208, 2021

2021
[38]

The x factor: On the relationship between user experience and explainability,

H. Deters, J. Droste, A. Hess, V . Kl ¨os, K. Schneider, T. Speith, and A. V ogelsang, “The x factor: On the relationship between user experience and explainability,” in Proceedings of the 13th Nordic Conference on Human-Computer Interaction , ser. NordiCHI ’24. New Y ork, NY ...

2024
[39]

Is chatgpt a good recommender? a preliminary study,

J. Liu, C. Liu, P . Zhou, R. Lv, K. Zhou, and Y . Zhang, “Is chatgpt a good recommender? a preliminary study,” 32nd ACM International Conference on Information and Knowledge Management , 2023

2023
[40]

Explanation needs in app reviews: Taxonomy and automated detection,

M. Unterbusch, M. Sadeghi, J. Fischbach, M. Obaidi, and A. V ogelsang, “Explanation needs in app reviews: Taxonomy and automated detection,” 31st International Requirements Engineering Conference Workshops , vol. 5, pp. 102 – 111, 2023

2023
[41]

Kann sich k ¨unstliche intelligenz selbst erkl ¨aren?

A.Horstmann, N. Kr ¨amer, C. Geminn, T. Bile, C.Weber, A. Manzeschke, L. Mavrina, S. Kopp, and B. H. A. Artelt, “Kann sich k ¨unstliche intelligenz selbst erkl ¨aren?” IMPACT, 2023

2023
[42]

User feedback in the appstore: An empirical study,

D. Pagano and W. Maalej, “User feedback in the appstore: An empirical study,” 21st International Requirements Engineering Conference , pp. 125 – 134, 2013

2013
[43]

Using ﬂow to improve com- munication of requirements in globally distributed software projects,

K. Stapel, E. Knauss, and K. Schneider, “Using ﬂow to improve com- munication of requirements in globally distributed software projects,” in 2009 Collaboration and Intercultural Issues on Requirements: Commu- nication, Understanding and Softskills , 2009, pp. 5–14

2009
[44]

Wohlin, P

C. Wohlin, P . Runeson, M. H ¨ost, M. C. Ohlsson, B. Regnell, and A. Wessl´en, Experimentation in software engineering . Berlin: Springer, 2012

2012
[45]

Supplementary Material -

J. Droste, H. Deters, M. Obaidi, and K. Schneider, “Supplementary Material - ”Explanations in Everyday Software Systems: Towards a Taxonomy for Explainability Needs” (RE’24),” Mar. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.10871086

2024 doi
[46]

Dataset: Gold standard dataset for explainability need detection in app reviews

M. Obaidi, “Dataset: Gold standard dataset for explainability need detection in app reviews.” Sep. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.11522828

2024 doi
[47]

Measuring nominal scale agreement among many raters

J. L. Fleiss, “Measuring nominal scale agreement among many raters.” Psychological Bulletin , vol. 76, no. 5, pp. 378–382, 1971

1971
[48]

A coefﬁcient of agreement for nominal scales,

J. Cohen, “A coefﬁcient of agreement for nominal scales,” Educational and Psychological Measurement , vol. 20, no. 1, pp. 37–46, 1960

1960
[49]

The measurement of observer agreement for categorical data

J. Landis and G. Koch, “The measurement of observer agreement for categorical data.” Biometrics, vol. 33 1, pp. 159–74, 1977

1977
[50]

A coefﬁcient of agreement for nominal scales,

——, “A coefﬁcient of agreement for nominal scales,” Biometrics, pp. 159–174, 1977

1977
[51]

Wohlin, P

C. Wohlin, P . Runeson, M. H ¨ost, M. Ohlsson, B. Regnell, and A. Wessl´en, Experimentation in Software Engineering . Springer, 2012

2012

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Based in Hanover, Germany, the company develops apps and systems like NUNA V 2, which cater to various use cases such as personal navigation, logis- tics, and trucking

Graphmasters GmbH: Graphmasters GmbH 1 is a com- pany specializing in intelligent navigation solutions, focusing on optimizing trafﬁc ﬂow and reducing congestion through real-time route guidance. Based in Hanover, Germany, the company develops apps and systems like NUNA V 2, which cater to various use cases such as personal navigation, logis- tics, and tr...

work page

[2] [2]

Reviews Scraping: A custom software tool was devel- oped to scrape reviews from the Google Play Store and Apple App Store . The scraped data included metadata such as star ratings, dates, responses to reviews, review titles, and the text 1https://www.graphmasters.net/ 2https://www.graphmasters.net/nunav content of the reviews. This data was consolidated i...

work page

[3] [3]

While the primary focus was on the Nunav Navigation App , three additional apps were included to expand the dataset due to their structural similarity

Dataset Creation: The dataset for this study was con- structed using user reviews from several apps 3 developed by Graphmasters GmbH, including the Nunav Navigation App , Nunav Courier App , Nunav Trucks App , and Nunav Bus App , sourced from both the Google Play Store and Apple App Store. While the primary focus was on the Nunav Navigation App , three ad...

work page

[4] [4]

An approach with large language models was not possible, because of the limited training data

Dataset Analysis: Explicit and implicit explanation needs in the reviews were identiﬁed using a word and phrase ﬁltering method. An approach with large language models was not possible, because of the limited training data. The guidelines, description of implicit and explicit explanation need by Droste et al. [3] are online available [19]. Implicit explan...

work page

[5] [5]

The speciﬁc segments of the reviews contain- ing explanation needs were manually marked and categorized as either explicit or implicit explanation needs

During this 3https://www.graphmasters.net/nunav 4https://www.deepl.com/ 5https://www.maxqda.com/ TABLE I IDENTIFIED NEED FOR EXPLANA TION ACCORDING TO WORD AND PHRASE FILTER OF THE INDIVIDUAL APPS Explanation Need App Explicit Implicit Potential None nunav navigation 24 28 425 1659 nunav truck 00 0 7 nunav logistics 2 1 49 135 nunav bus 0 1 10 25 Total 26...

work page

[6] [6]

Each taxonomy category is mapped to one or more reference points within the company

Reference Point: To automatically assign a reference point for addressing the explanation needs identiﬁed in the reviews, taxonomy categories are employed. Each taxonomy category is mapped to one or more reference points within the company. The data for this assignment was gathered from the interviews with the Graphmasters support team. If the mapping of ...

work page

[7] [7]

Source: The source containing the answer to the expla- nation need is determined using an API from the Graphmasters support website. The script, which uses the API, compares the similarity between support articles on the Graphmasters support page and the user’s query using the SequenceMatcher from Python’s difﬂib module

work page

[8] [8]

I ﬁnd the upcoming motorcycle version exciting - what will make it different?

The support articles on the Graphmasters support page were created manually by Graph- masters support members for general explanatory purposes. This comparison helps identify whether a speciﬁc support article addresses the question at hand. If no relevant support article is found, the script retrieves all the answer from responses provided to the review i...

work page 2024