Pith. sign in

REVIEW 4 major objections 6 minor 22 references

Improving IT Support by Enhancing Incident Management Process with Multi-modal Analysis

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Adding screenshot-derived text to IT support tickets improves automated routing and resolution beyond text-only analysis, with routing coverage rising from 89.3% to 96.5%.

desk verdict A credible industrial multimodal ticket-routing system whose 'resolution' gains are actually category-labeling gains; the paper says so itself, so read the headline number with care. read the letter →

arxiv 1908.01351 v1 pith:IJ3AZULV submitted 2019-08-04 cs.IR cs.CV

classification cs.IRcs.CV
keywords incidentmanagementmultimodalanalysisITsupportticketsscreenshotunderstandingticketroutingresolutionOCRenrichment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IT support tickets often arrive with screenshots that contain the information needed to solve them, but text-only automation ignores that content. The paper's claim is that a pipeline which detects application windows in screenshots, extracts the text and entities inside them, and merges that information into the ticket text improves both automated routing and resolution. On 25,000 curated real tickets, assignment accuracy and coverage rise from 86.1% and 89.3% with text only to 88.6% and 96.5% with multi-modal enrichment, and problem-category accuracy rises from 74.7% to 82.4%. The paper estimates this translates into about 194,000 man-hours saved per year for an account receiving 100,000 tickets a month. The payoff matters because a large share of support tickets contain attachments, and a system that reads them can automate a bigger fraction of the workload.

What carries the argument

Application-window detection and ticket enrichment form the core pipeline. A hybrid object detector combines contour detection and Canny edge detection with Hough lines, filters candidates by size, then uses a ResNet50-based binary classifier to keep genuine application windows and classify window type and OS. An OCR engine extracts text from the detected windows, dictionary and language-model post-processing repair occluded or misspelled text, and a CRF-based named-entity recognition system extracts entities such as OS, application, version, and error code, which are inserted into slot templates in the ticket text. The enriched text is classified by an ensemble of linear SVM and MLP that predicts resolver group and problem category; high-confidence predictions trigger direct resolution lookup, while low-confidence ones fall back to federated information retrieval and web search re-ranked with the CORI result-merging algorithm.

What would settle it

Use the same 25,000 tickets and check whether tickets whose predicted problem category matches the historical label are actually closed or resolved by the recommended action at a higher rate than tickets with a mismatched prediction; if the rates are close, the paper's resolution improvement is a classification gain, not a resolution gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that correlating text with screenshot content—extracting application-window type, OS, error text, and other entities from images and inserting them into the ticket text—improves both automated routing and resolution in IT incident management. Concretely, on 25,000 curated real tickets, assignment accuracy rises from 86.1% to 88.6% and coverage from 89.3% to 96.5%, while problem-category prediction accuracy, the paper's proxy for resolution accuracy, rises from 74.7% to 82.4%. The authors attribute the improvement to the enrichment step: screenshots supply context that users omit from text, so classifiers either gain confidence on tickets they already handled or become able to route tickets that text alone could not handle. They further estimate the operational impact at roughly 194,000 saved man-hours per year for a 100,000-tickets-per-month account.

Load-bearing premise

The resolution claim uses correct problem-category prediction as a stand-in for actually resolving the ticket, so if matching a category does not mean the recommended fix works, the reported 82.4% resolution accuracy is not a true resolution rate.

Editorial extensions

If this is right

  • Routing automation coverage rises from 89.3% to 96.5%, meaning roughly 7 more tickets in every 100 can be assigned automatically at the same confidence standard.
  • Problem-category accuracy rises from 74.7% to 82.4%, and because the category is composite, requiring all three sub-fields to match, the gain is not just a coarser label match.
  • The paper reports that some long-tail problem categories improve by more than 50%, so the benefit is concentrated where resolutions are scarce rather than only on frequent categories.
  • Estimated savings of about 194,000 man-hours per year at 100,000 monthly tickets follow directly from the coverage gains together with assumed assignment and resolution times of 3 and 10 minutes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to apply the same enrichment concept to non-screenshot attachments such as invoices, tables, and charts, where layout-aware document understanding would be needed; the authors only mention this as future work.
  • The man-hour estimate assumes fixed per-ticket handle times, so accounts with different ticket complexity would scale the savings linearly without changing the reported coverage gains.
  • Because the curated 25,000 tickets come from the 10 resolver groups with the most screenshot attachments and 33 frequent problem categories, the gains may be upper bounds; a random sample of the full 712,230-ticket corpus would show how the pipeline behaves on less screenshot-heavy traffic.
  • A direct outcome study of whether recommended resolutions actually close tickets would test whether problem-category accuracy is a faithful proxy for resolution success; the paper does not report such a study.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper describes an end-to-end multimodal system for IT support ticket routing and resolution. The system detects and classifies application windows in screenshot attachments using a hybrid of contour/Canny edge detection plus CNN filtering, extracts text from detected windows with OCR and language-model post-processing, enriches ticket text with extracted entities via CRF-based slot filling, and then predicts resolver group and problem category with an ensemble classifier. Resolution is recommended either by a direct lookup for short-head categories or by federated search over a knowledge corpus and the web for long-tail cases. The evaluation uses about 25,000 real multimodal tickets selected from a larger corpus of 712,230 tickets. The paper reports that multimodal analysis improves assignment accuracy/coverage from 86.1%/89.3% to 88.6%/96.5% and resolution accuracy from 74.7% to 82.4%, and estimates roughly 194,000 man-hours saved per year for a 100,000-tickets-per-month account.

Significance. If the claims were fully supported, this would be a useful industrial contribution: it addresses a real operational problem, combines several non-trivial components (window detection, OCR, NER-based enrichment, hybrid retrieval) in one pipeline, and evaluates on real tickets against a text-only baseline. The modular architecture and the explicit description of the resolution algorithm are strengths. The image-understanding accuracies reported in Tables 2 and 3 suggest that the window detection and classification components work reasonably on the curated data. However, the headline claim of improved resolution is not established by the evaluation metric: as the paper itself states, resolution accuracy is estimated by problem-category prediction accuracy, not by whether a suggested resolution actually resolves the incident. In addition, the curated evaluation subset and the absence of statistical rigor (no train/test split detail, no error bars, no significance tests) make the reported improvements difficult to interpret.

major comments (4)
  1. [Sec. 5, Table 4] The central claim of improved resolution is not supported by the reported metric. Section 5 states 'we estimate the accuracy of resolution with the accuracy achieved in predicting the problem category,' and Table 4's 'Resolution' row is therefore a problem-category classification accuracy, not a measure of whether the recommended resolution actually resolves the ticket. Algorithm 1 may return a resolution string from a lookup or from federated search, but the paper does not evaluate the correctness or usefulness of that returned resolution on any outcome variable (e.g., ticket closure, reopen rate, agent feedback, or expert judgment). A correct problem category does not guarantee a correct or actionable resolution, and an incorrect category may still lead to a useful result through fallback search. The abstract's statement 'significant improvements in both routing and resolution' thus overstates what the experiments show. This is load-bearing and must be fixed by either adding outcome-based evaluation or explicitly re-framing all 'resolution' claims as 'problem-category prediction' claims.
  2. [Sec. 4.1, Table 1] The evaluation dataset is a curated subset that does not represent the full ticket population. From 712,230 tickets spanning 428 resolver groups and 3,728 problem categories, the authors selected only 25,000 multimodal tickets from 10 resolver groups with the most screenshot attachments and 33 frequent problem categories for the short head, with the remaining 219 categories as long tail. Accuracy and coverage figures on this subset may not generalize to the full population, and the reported 7% improvement in automation coverage could partly reflect the selective inclusion of high-attachment, comparatively easy groups. The paper should either report results on a more representative sample (including lower-attachment groups and a random selection of categories) or clearly state the restricted scope as a limitation in the conclusions.
  3. [Sec. 4.1 and Sec. 5] The experimental evaluation lacks a documented train/test split, error bars, confidence intervals, and statistical significance tests. The reported improvements (e.g., assignment accuracy 86.1% to 88.6%, a 2.5 percentage-point change) may be within sampling variability, especially since the test set is a non-random curated subset. The paper also does not specify how the text-only baseline was configured—whether it is the same ensemble classifier with only ticket text features and no enrichment—nor how the enriched features were generated for the multimodal condition in a way that avoids label leakage. Without these details, the magnitude of the claimed improvement cannot be assessed reliably.
  4. [Sec. 5.1, Eqs. (4)-(5)] The man-hour saving estimate is based on unvalidated assumptions and on the conflated resolution metric. Equations (4) and (5) use Tcov = 90% and Rcov = 80%, plus assumptions of 3 minutes per assignment and 10 minutes per resolution, but Rcov is defined in terms of automated routing/resolution coverage without specifying whether it is category-prediction coverage or actual resolution coverage. The resulting 'about 194,000 man hours' figure in Section 5.1 also differs from the '200000 man hours' estimate in the introduction. The savings claim should be either removed or presented as an illustrative calculation with clearly stated, sensitivity-tested assumptions.
minor comments (6)
  1. [Abstract and Sec. 5.1] The number of saved man-hours is inconsistent: the abstract and introduction state about 200,000, while Section 5.1 calculates about 194,000. Please harmonize these numbers.
  2. [Algorithm 1 / Fig. 3] The pseudocode for Algorithm 1 appears to be duplicated in the typeset text (two identical columns). This is a formatting error that should be corrected.
  3. [Table 2] Table 2 reports precision/recall pairs for '1-Window', '2-windows', and '3-windows' cases, but the text does not specify whether these are per-image or per-window aggregates, nor how the numbers were averaged across test images. Please clarify the evaluation protocol for window detection.
  4. [Sec. 4.2, Annotation] The annotation process for bounding boxes uses shallow object detection as a first pass with manual correction only for hard cases; this could introduce systematic label noise into the training data for the CNN filters. A brief discussion of annotation quality control would strengthen the paper.
  5. [Sec. 3.2, Eq. (3)] In Equation (3), the variables d and c are defined in the text, but it would be clearer to state explicitly which score is the 'normalized score given by the search engine' and which is the 'relevance score computed by the language model' directly below the equation, rather than only in the prose.
  6. [Sec. 5, Text extraction] The OCR evaluation is reported primarily on synthetic images (with manual corroboration on 'a few images'); it would be helpful to report the real-image accuracy as well, since the downstream enrichment quality depends on real-world OCR performance.

Circularity Check

2 steps flagged · score 6.0 of 10

Resolution gain is a relabeled category-classification gain; window-detection accuracy is measured against boxes the detector itself produced.

  1. self definitional [Section 5, 'Routing/Resolution' paragraph; Table 4]
    "Also, the most important step in obtaining the resolution strategy is to understand the correct problem category of the ticket as in most cases, the problem category has a one-to-one mapping with the resolution strategy. Even if that is not the case, identifying the correct problem category is a key step in automated resolution as it narrows down the scope of the search. As such we estimate the accuracy of resolution with the accuracy achieved in predicting the problem category in both the short head and long tail cases."

    The Resolution row in Table 4 is defined, by this sentence, to be the problem-category classification accuracy. Thus the claimed improvement from 74.7% (text-only) to 82.4% (multimodal) is exactly the improvement of the category classifier, not a separate measurement of whether recommended resolutions resolve tickets. The 'resolution' result is therefore the same quantity as the classification result by construction; the abstract's 'improvements in routing and resolution' bundles an independent routing measurement with a re-labeled classification metric.

  2. other [Section 4.2, 'Annotation' paragraph; Table 2]
    "For bounding box annotation we used shallow object detection technique described in 3.1. This method of annotation works on most images. However whenever images contain windows with high degree of overlap and confusing images in the background the annotation may not be entirely correct. In these cases we do a manual inspection and annotation."

    The accuracy of contour/canny/ensemble window detection in Table 2 is evaluated against bounding-box annotations that were generated by the same shallow object detection pipeline, with manual correction only for difficult overlapping cases. For the majority of images the 'ground truth' is the output of the method under test, so the reported precision/recall numbers are partly self-confirming. This loop affects the intermediate image-understanding stage rather than the end-to-end routing/resolution classification, but it is still a circular evaluation step.

full rationale

The paper is an empirical pipeline rather than a formal derivation, and much of it is self-contained: the resolver-group and problem-category classifiers are evaluated on historical labels, the multimodal features come from OCR/window detection rather than from the labels being predicted, and there is no fitted parameter that is later renamed as a prediction. The self-citations to the authors' prior text-only system ([13]/[14]) supply the baseline and threshold-selection method, but the multimodal numbers are measured in this paper, so that citation is not load-bearing in a circular way. The circularity is partial and definitional. Section 5 explicitly equates resolution accuracy with problem-category prediction accuracy, so the 'resolution' half of the headline claim reduces to the category-classification result by construction. In addition, the bounding-box ground truth for the window-detection evaluation is generated by the same shallow detection technique that is being scored, making that intermediate metric partially self-confirming. The routing results and the image-classification results retain independent content, which is why the score is 6 rather than higher.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on system design choices rather than on a derivation. Most free parameters are tuning thresholds whose values are not reported, and the key assumptions concern the validity of labels and of problem-category prediction as a resolution metric.

free parameters (5)
  • Confidence cutoffs (CONF_RESOLV_CUTOFF, CONF_PROB_CUTOFF, CONF_SUBFIELD_CUTOFF)
    Used to decide the short-head versus long-tail path in Algorithm 1; selected to keep classifiers at human-level efficiency per [13], but values and tuning set are not disclosed.
  • Short-head frequency threshold for selecting PH = 33 problem categories selected
    The boundary between short head and long tail is set by a configurable histogram threshold; the paper does not report the value or how it was validated.
  • Window size filter threshold
    Removes detected boxes smaller than a threshold; the value is not reported.
  • CORI result-merging coefficient = 0.4
    Constant in Equation 3, taken from the CORI algorithm; used without re-tuning for this domain.
  • Savings assumptions Tcov and Rcov = Tcov=90%, Rcov=80%
    Used in Equations 4 and 5 to compute man-hour savings; these are assumed coverage rates, not measured values from the evaluation.
assumptions (4)
  • domain assumption Historical ticket labels (resolver group and problem category) are correct ground truth.
    Sections 3.2 and 4.1 train and evaluate classification against these labels; label errors would distort accuracy.
  • domain assumption Correctly predicting the composite problem category implies a successful resolution.
    Section 5 states that resolution accuracy is estimated by problem-category prediction accuracy; this equates classification accuracy with resolution outcome.
  • domain assumption Human agents take about 3 minutes to assign and 10 minutes to resolve a ticket.
    Section 5.1 uses these times to convert coverage into man-hour savings; they are assumed, not measured in this deployment.
  • domain assumption Synthetic and web-scraped screenshots are representative of real ticket attachments.
    Section 4.2 trains image classifiers on synthesized and web images; generalization to real screenshots is asserted but not quantified in detail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving IT Support by Enhancing Incident Management Process with Multi-modal Analysis." pith.science (2026). https://pith.science/paper/IJ3AZULV

@misc{pith2026190801351,
  author       = {Pith},
  title        = {Pith review of: Improving IT Support by Enhancing Incident Management Process with Multi-modal Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJ3AZULV}},
  note         = {Machine review of arXiv:1908.01351}
}
read the original abstract

IT support services industry is going through a major transformation with AI becoming commonplace. There has been a lot of effort in the direction of automation at every human touchpoint in the IT support processes. Incident management is one such process which has been a beacon process for AI based automation. The vision is to automate the process from the time an incident/ticket arrives till it is resolved and closed. While text is the primary mode of communicating the incidents, there has been a growing trend of using alternate modalities like image to communicate the problem. A large fraction of IT support tickets today contain attached image data in the form of screenshots, log messages, invoices and so on. These attachments help in better explanation of the problem which aids in faster resolution. Anybody who aspires to provide AI based IT support, it is essential to build systems which can handle multi-modal content. In this paper we present how incident management in IT support domain can be made much more effective using multi-modal analysis. The information extracted from different modalities are correlated to enrich the information in the ticket and used for better ticket routing and resolution. We evaluate our system using about 25000 real tickets containing attachments from selected problem areas. Our results demonstrate significant improvements in both routing and resolution with the use of multi-modal ticket analysis compared to only text based analysis.

Figures

Figures reproduced from arXiv: 1908.01351 by the authors.

Figure 1
Figure 1. (a) Canny-edge detects spurious boxes (b) Contour detection detects objects in background (i) A high-precision hybrid object detection engine which uses a combination of traditional image processing algorithms as well as deep learning based image classification. The main purpose of the detection engine is to identify if an application window (e.g. error message box, terminal, explorer window etc.) is present and if … view at source ↗
Figure 2
Figure 2. System architecture resolution and problem category, it is stored in the system and the agents can leverage it for speedy resolution. We now explain how the ticket enrichment is done using multi-modal analysis, that is, combined analysis of text and image present in the ticket. We also explain the proposed multi-step process for resolution recommendation which can choose the source of resolution based on the confide… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 20 canonical work pages

  1. [1]

    IBM Journal of Research and Development 61(1), 4:41–4:52 (Jan 2017)

    Agarwal, S., Aggarwal, V., Akula, A.R., Dasgupta, G.B., Sridhara, G.: Automatic problem extraction and analysis from unstructured text in it tickets. IBM Journal of Research and Development 61(1), 4:41–4:52 (Jan 2017)

  2. [2]

    In: 18th ACM SIGKDD 2012

    Agarwal, S., Sindhgatta, R., Sengupta, B.: Smartdispatch: Enabling efficient ticket dispatch in an it service environment. In: 18th ACM SIGKDD 2012

  3. [3]

    In: IEEE International Conference on Services Computing, SCC 2016 (2016) Title Suppressed Due to Excessive Length 15

    Aggarwal, V., Agarwal, S., Dasgupta, G.B., Sridhara, G., E, V.: React: A system for recommending actions for rapid resolution of IT service incidents. In: IEEE International Conference on Services Computing, SCC 2016 (2016) Title Suppressed Due to Excessive Length 15

  4. [4]

    In: 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    Botezatu, M.M., Bogojeska, J., Giurgiu, I., Voelzer, H., Wiesmann, D.: Multi-view incident ticket clustering for optimal ticket dispatching. In: 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 1711–

  5. [5]

    IEEE Trans

    Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (Jun 1986)

  6. [6]

    In: ICSOC 2014, Paris, France, November 3-6, 2014

    Dasgupta, G., Nayak, T.K., Akula, A.R., Agarwal, S., Nadgowda, S.J.: Towards auto-remediation in services delivery: Context-based classification of noisy and unstructured tickets. In: ICSOC 2014, Paris, France, November 3-6, 2014. Pro- ceedings. pp. 478–485 (2014)

  7. [7]

    In: CVPR09 (2009)

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large- Scale Hierarchical Image Database. In: CVPR09 (2009)

  8. [8]

    Duda, R.O., Hart, P.E.: Use of the hough transformation to detect lines and curves in pictures. Commun. ACM 15(1), 11–15 (1972)

Show all 22 references
  1. [9]

    In: COLING

    Gupta, A., Ray, A., Dasgupta, G., Singh, G., Aggarwal, P., Mohapatra, P.: Seman- tic parsing for technical support questions. In: COLING. Santa Fe, New Mexico, USA (Aug 2018)

  2. [10]

    2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recogni- tion. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  3. [11]

    In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014. pp. 740–755. Springer International Publish...

  4. [12]

    Maire, M.R.: Contour Detection and Image Segmentation. Ph.D. thesis (2009)

  5. [13]

    ArXiv e-prints (Aug 2018)

    Mandal, A., Malhotra, N., Agarwal, S., Ray, A., Sridhara, G.: Cognitive system to achieve human-level accuracy in automated assignment of helpdesk email tickets. ArXiv e-prints (Aug 2018)

  6. [14]

    In: 16th International Conference, ICSOC (2018)

    Mandal, A., Malhotra, N., Agarwal, S., Ray, A., Sridhara, G.: Cognitive system to achieve human-level accuracy in automated assignment of helpdesk email tickets. In: 16th International Conference, ICSOC (2018)

  7. [15]

    CoRR abs/1711.02012 (2017), http://arxiv.org/abs/1711.02012

    Mani, S., Gantayat, N., Aralikatte, R., Gupta, M., Dechu, S., Sankaran, A., Khare, S., Mitchell, B., Subramanian, H., Venkatarangan, H.: Hi, how can I help you?: Automating enterprise IT support help desks. CoRR abs/1711.02012 (2017), http://arxiv.org/abs/1711.02012

  8. [16]

    John Wiley & Sons, Inc., New York, NY, USA (1999)

    Mori, S., Nishida, H., Yamada, H.: Optical Character Recognition. John Wiley & Sons, Inc., New York, NY, USA (1999)

  9. [17]

    http://cs231n

    Sampat, A., Haskell, A.: Cnn for task classification using computer screenshots for integration into dynamic calendar/task management systems. http://cs231n. stanford.edu/reports/2015/pdfs/anand_avery_final.pdf

  10. [18]

    In: International Conference on Learning Representations (2015)

    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale im- age recognition. In: International Conference on Learning Representations (2015)

  11. [19]

    Smith, L.: Cyclical learning rates for training neural networks. pp. 464–472 (03 2017)

  12. [20]

    In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Devel- opment in Information Retrieval

    Xu, J., Callan, J.: Effective retrieval with distributed collections. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Devel- opment in Information Retrieval. pp. 112–120. SIGIR ’98, ACM (1998)

  13. [21]

    Grabarnik, G.: Resolution recommendation for event tickets in service management

    Zhou, W., Tang, L., Zeng, C., Li, T., Shwartz, L., Ya. Grabarnik, G.: Resolution recommendation for event tickets in service management. IEEE Transactions on Network and Service Management 13(4), 954–967 (Dec 2016) 16 A. Mandal et al

  14. [22]

    Grabarnik, G.: Star: A system for ticket analysis and resolution

    Zhou, W., Xue, W., Baral, R., Wang, Q., Zeng, C., Li, T., Xu, J., Liu, Z., Shwartz, L., Ya. Grabarnik, G.: Star: A system for ticket analysis and resolution. In: Proceed- ings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 2181–2190...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.