{"id":"6ae918f3-1a4c-41e2-a5c5-691444ca8754","arxiv_id":"2507.03811","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An LLM agent can reconstruct full descriptions of data tables by conversing with simulated employees, even when the only full-knowledge expert is never contacted.","lead":"This paper uses an AI agent powered by a large language model to interview simulated employees and piece together a complete description of a data table. The authors report a 94.9% success rate in reconstructing all columns, even without ever talking to the one employee who holds all the knowledge.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94.9% recall metric is nearly predetermined: Section III.A gives the agent the table columns at k0, and Section V's full-knowledge recall only requires mentioning those columns.","rationale":"The reader correctly flagged the column-mention criterion as lenient, but the strongest concern is sharper: the paper's own state definition makes the metric nearly tautological. This is the single most load-bearing issue because the abstract and conclusion rest on the 94.9% figure. The simulation-quality concerns (LLM-generated ground truth, unrealistically helpful simulated employees) are secondary; they would matter if the metric were informative. The proposed check is cheap and decisive: if k0 already achieves the headline recall, the central empirical claim collapses. As written, the paper should be rejected, though a revised version with a stricter reconstruction metric and explicit baselines could be re-evaluated.","tokens_in":11803,"tokens_out":4432,"duration_ms":49975,"concrete_test":"Run the identical pipeline but compute full-knowledge recall on the initial description k0 before any employee interaction, and add a zero-interaction baseline that simply echoes the provided table name and column list. If k0 recall or baseline recall matches or exceeds 94.9%, the headline metric is predetermined by the prompt and the claim that conversations recover knowledge is unsupported. Additionally, re-run with a stricter metric that requires each column's type, meaning, and at least one example value to be present, scored against k*; if that recall drops materially, the current metric overstates reconstruction.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim of 94.9% full-knowledge recall (Section V) does not actually test knowledge reconstruction, because the agent's initial state already contains the table's column names. Section III.A states: \"The agent begins with a basic understanding k0 of the table or organizational knowledge, knowing only the table's name and columns.\" The recall metric (Section V) defines success as a final description that \"included a mention of every column originally disseminated.\" Since k0 already includes every column name, this metric measures whether the final report preserves a list supplied in the initial prompt, not whether the agent recovered column semantics, data types, example values, or variable interactions — the content that the epidemic model actually disseminates and the content relevant to tacit knowledge. The same concern transfers to the \"without patient zero\" claim: if recall is just column-list persistence, the near-zero correlation between cGEvalFaith and % p0 (-0.06) tells us little, because listing columns does not require contacting anyone. This also explains why the final-score correlations in Table II are only moderate (0.26–0.50) despite 94.9% recall: the strongest reported metric is the trivial one. This is not a disagreement with external consensus; it is grounded in the paper's own definitions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based agent system that reconstructs documentation of a data table by conversing with simulated employees in synthetic organizational networks. Knowledge dissemination is modeled as an SI epidemic with waning infectivity over combined formal and informal networks, and the agent iteratively asks questions, updates a description, and self-critiques. The authors run 864 simulations with varying organizational parameters and report a 94.9% full-knowledge recall rate, correlations between the agent's self-critical scores and external reference-based metrics, and evidence that high-quality descriptions can be obtained without contacting the single domain specialist (patient zero). The paper also releases code, data, and prompts.","tokens_in":1955,"tokens_out":2051,"duration_ms":42822,"significance":"If the claims held, the paper would offer a scalable simulation-based methodology for studying tacit-knowledge elicitation in organizations and a useful benchmark for LLM-agent interaction. The work has concrete strengths: a large simulation corpus (over 300,000 interactions), a clearly specified epidemic-inspired knowledge-dissemination model, a reproducible experimental design with a public code/data deposit, and a multi-metric evaluation that includes both a non-LLM metric (METEOR) and LLM-as-judge scores. The analysis of how organizational parameters (hierarchy depth, informal connections, decay, alpha) affect retrieval is a genuinely useful contribution. However, the central quantitative claims are weakened by a near-trivial recall criterion, missing baselines, and a modest correlation between self-critique scores and external metrics, so the paper's main conclusions need substantial revision before they can be considered established.","major_comments":[{"comment":"The headline result of 94.9% full-knowledge recall does not test knowledge reconstruction. Section III.A states that the agent begins with knowledge k0 that includes the table's name and columns, and Section V.a defines full-knowledge recall as a final description that included a mention of every column originally disseminated. Since k0 already contains every column name, this metric only measures whether the agent preserves a list that was given to it in the initial prompt; it does not require recovery of column semantics, data types, example values, variable interactions, or any other content that the epidemic model actually disseminates. The claim in Section V that the agent can recover information without contacting patient zero is therefore not supported by the 94.9% number. The authors should either redefine full-knowledge recall to require recovery of non-trivial content (types, meanings, examples, interactions), or explicitly evaluate the column-list baseline and report the recall of substantive facts separately.","section":"Section III.A and Section V (Full-knowledge recall)"},{"comment":"The abstract claims that self-critical feedback scores strongly correlate with external literature critic scores, but Table II shows Spearman correlations of only 0.26 (cMETEOR vs. SCS) and 0.35 (cGEvalCoh vs. SCS), with the strongest correlation being 0.50 (cGEvalFaith vs. SCS). These are weak-to-moderate correlations, not strong ones. Only the self-critical score with context (SCS vs. SCS+C, 0.73) is strong, and that comparison is between two variants of the same LLM-based critic rather than an external evaluation. The wording should be corrected, and the interpretation of Table II should be revised to acknowledge that the agent's self-critique has limited ability to rank final descriptions according to external quality.","section":"Section V (Table II) and Abstract"},{"comment":"The evaluation lacks baselines and ablations that would establish the contribution of the proposed agent architecture. For example, there is no comparison against: (a) an agent that simply writes a report from its k0 column list without any conversation, (b) an agent that randomly selects employees instead of using self-critique-guided questions, or (c) a non-agentic prompting baseline that reads the same employee responses without the prompt-chaining/self-critique loop. Without such controls, the 94.9% recall and the correlation findings cannot be attributed to conversational knowledge retrieval or to the SI-based network traversal. The authors should add at least one baseline and one ablation, or clearly state what the incremental contribution of each component is.","section":"Section IV and Section V (Evaluation design)"},{"comment":"The validation loop is largely internal to the same LLM family: the ground-truth table descriptions are generated by GPT-4o mini, the simulated employees are GPT-4o mini, the agent is GPT-4o mini, and the G-Eval judge is an LLM (presumably also GPT-4o or similar). Only METEOR provides a non-LLM anchor. This creates a risk that the reported success partly reflects self-consistency of LLM-generated content rather than recovery of an independent ground truth. The paper should add a limitations paragraph discussing this circularity, and ideally include a small human-evaluated subsample or at least an analysis of which claims survive under the purely lexical METEOR metric (e.g., report the METEOR scores by semantic content type rather than an average of 0.17).","section":"Section IV (Simulation) and Section V (G-Eval)"},{"comment":"The conclusion that the agent's effectiveness does not depend on reaching patient zero is based on the correlation between cGEvalFaith and % p0 (-0.06) and on the UMAP visualization. However, cGEvalFaith is an LLM-judge score on semantic faithfulness, and the correlation is computed across configurations with only three repetitions each. More importantly, since the recall metric is trivially satisfied, the evidence that the agent reconstructs knowledge without patient zero is weak. The authors should report the distribution of METEOR or cGEvalFaith conditional on whether patient zero was reached, with confidence intervals, and should avoid claiming independence from patient zero when the strongest metric (94.9% recall) does not measure substantive knowledge recovery.","section":"Section V (Patient-zero analysis and Figure 4)"}],"minor_comments":[{"comment":"Equation (2) is garbled: 'bd ≈ hierarchy depth p number of employees' should presumably be the depth-th root of the number of employees, i.e., bd ≈ (number of employees)^(1/hierarchy depth). Please fix the notation.","section":"Section III.B, Eq. (2)"},{"comment":"The sentence 'For each repetition, a table subject was randomly selected from a predefined list, which was then used to generate the simulated complete knowledge s∗ through an LLM' introduces the symbol s∗, but the text elsewhere calls this knowledge k∗. Please use a single consistent notation.","section":"Section IV"},{"comment":"In the caption and text, the paper refers to 'average and mean self-critical score' for patient zero, which is redundant; please clarify which column corresponds to which statistic.","section":"Section V (Table I)"},{"comment":"The introduction says the method 'facilitates the crowdsourcing of this process using LLM agents', but the conclusion describes it as a method for 'tacit knowledge retrieval'. Please reconcile the terminology, since the simulation only measures reproduction of given facts, not the acquisition of genuinely tacit (unarticulated) knowledge.","section":"Section I and Conclusion"},{"comment":"There are some typographical issues (e.g., 'elected a range of possible values' in Section IV, 'company structures' in the conclusion). A careful proofreading pass would be helpful.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal, and the data/code release is commendable. The main risk is that the headline metric is too weak to support the paper's central claim. I would suggest the authors consider reframing the contribution as a simulation environment and an exploratory analysis of organizational parameters, rather than as demonstrated tacit-knowledge discovery. The lack of baselines is the most serious fixable issue; even a simple baseline that starts from k0 and writes a final report without conversations would help quantify the added value of the conversational loop. I would not reject the paper, because the framework and dataset are potentially reusable, but the current claims overstate what the experiments show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I agree with the stress-test note: the 94.9% full-knowledge recall is not the result it appears. The agent's initial state (Section III.A) already includes the table's name and columns, and recall only requires mentioning every column. So it measures whether the final description retains a list given at the start, not whether the agent recovered semantics, types, examples, or interactions. That is a load-bearing flaw for the main claim.\n\nWhat is new: the specific combination of an SI epidemic model for knowledge dissemination and an LLM agent that reconstructs table descriptions through conversations with simulated employees. I don't see that in the cited prior work. They also ship code, prompts, and simulation data, which is reproducible.\n\nWhat is good: the simulation design is fairly careful along some dimensions. 864 runs, parameter sweep over alpha, decay, informal connections, depth, employees, and columns. The qualitative results — path length drops with higher alpha or NIC, patient zero reached less often with more connections — are consistent with the model. The paper is clearly written, and the MDP framing is a reasonable way to organize the agent loop.\n\nSoft spots, in proportion:\n\n- The recall metric is the big one. The remaining metrics tell a weaker story: METEOR 0.17, G-Eval coherence 2.65, faithfulness 4.37. The correlations in Table II are weak to moderate (0.26–0.50), not \"strong\" as the abstract says. The only strong correlation is between the self-critic score and the self-critic with context (0.73), which compares LLM judgments to each other.\n- No baselines or ablations. The agent's design choices (stack ordering, self-critique prompt, etc.) are not tested. The \"without patient zero\" claim rests on the same weak recall metric, so the −0.06 correlation with cGEvalFaith is not meaningful.\n- The evaluation is self-referential: ground truth is LLM-generated, employees are LLMs, the agent is an LLM, and the G-Eval judge is an LLM. METEOR anchors it a little, but the score is low.\n\nWho it is for: researchers working on LLM agents in organizational settings. The simulation infrastructure and dataset are potentially reusable, even if the reported results are overstated.\n\nRecommendation: this deserves a serious referee, but with the expectation of heavy revision. The metrics need to be redone — recall should require semantic content beyond column names — and there need to be baselines. I would not cite the 94.9% number as evidence of tacit knowledge recovery.","headline":"The paper is a sensible LLM-agent simulation study, but its headline 94.9% recall metric is far weaker than it looks because the agent already knows the column names at the start.","tokens_in":12622,"tokens_out":2137,"would_cite":false,"duration_ms":22301,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM-driven agent interviewing simulated employees reconstructs full table knowledge in 94.9% of 864 runs, without ever reaching the sole domain expert.","keywords":["tacit knowledge","large language models","agent-based simulation","organizational knowledge","knowledge discovery","self-critical prompting","epidemic model","knowledge retrieval"],"falsifier":"Run the same agent against simulated employees whose backgrounds do not tell them what their colleagues know, and who answer only with facts they personally hold, then compare full-knowledge recall; if it falls well below 94.9%, the original figure was carried by leaked metadata in the simulation rather than by conversational reconstruction.","tokens_in":11622,"feed_emoji":"🤖","tokens_out":7911,"duration_ms":88377,"temperature":0.7,"pith_summary":"An LLM-powered agent can piece together fragmented organizational knowledge by holding one conversation at a time with employees, updating its description of a dataset in response to their answers. The paper claims that this method, tested across 864 simulated organizations, reaches 94.9% full-knowledge recall, meaning the final report mentioned every column of the original table in 820 runs. It also reports that this success does not depend on ever talking to the single employee who originally possessed all the facts, since faithfulness correlates at -0.06 with contacting that employee. The practical stake is that tacit knowledge, what people know but have not written down, could be documented automatically through ordinary dialogue, removing the need to first identify the specialist.","feed_headline":"LLM agent recovers 94.9% of hidden company knowledge","feed_subtitle":"Conversational interviews with simulated staff reconstruct tables even when the sole expert is never contacted.","key_machinery":"The load-bearing mechanism is a prompt-chained LLM agent whose knowledge state is a running table description. After each employee answer, the agent scores the description, writes a critique, lists gaps, and revises before choosing the next question or next employee from a stack ordered by hierarchy and by names mentioned in the conversation. On the simulation side, knowledge dissemination follows a Susceptible-Infectious model with waning infectivity, $\\beta(t) = \\beta_0 e^{-\\gamma t}$, treating each table column as a separate fact that spreads from a patient zero through formal hierarchy and informal relationship networks. The self-critical loop is the part that does the work: it turns vague replies into concrete next questions and keeps the search focused, which is how the agent recovers facts without ever reaching the sole expert.","core_discovery":"The central claim is that autonomous conversational agents can serve as a knowledge-discovery method inside organizations, reconstructing a complete dataset description from scattered partial knowledge. In the paper's simulation, each column of a table is a fact that spreads through the company like an infection, and the agent enters after the knowledge has already fragmented. Across 864 runs the agent achieved 94.9% full-knowledge recall, and in the 44 failures the final description still retained about 77% of the columns. The low correlation (-0.06) between faithfulness and patient-zero contact is the key evidence: high-quality reconstructions occur even where the original domain specialist was never reached, so the method is not just find-the-expert. The authors present this as a step toward capturing tacit knowledge that formal documentation misses.","pith_inferences":["Beyond the paper: the simulation's employee backgrounds include awareness of what their colleagues know, which real employees may lack; a deeper test would disable that information and measure how much recall drops.","Beyond the paper: the same self-critical loop could be pointed at other fragmented artifacts, such as codebases, design decisions, and incident postmortems, where the table descriptions are replaced by whatever document needs reconstruction.","Beyond the paper: the framework implies that organizational network structure is a first-class variable in LLM-agent evaluation, so future benchmarks should report conversation graphs, not just final scores."],"forward_implications":["If the 94.9% figure holds outside simulation, organizations can turn documentation into a set of short conversational interviews instead of a search for the right expert.","Because self-critical scores track external reference metrics, a deployed agent could monitor its own documentation quality without ground truth.","Parameter effects give practical guidance: more informal connections shorten the conversation path and reduce reliance on hubs, so network structure, not just prompt quality, determines retrieval cost.","The near-zero dependence on patient zero implies that knowledge can be reconstructed from partial holders, which matters when the original expert has left the company or is overloaded."],"supporting_citations":[{"why":"Supplies the generative-agent design, with LLM employees who hold memory and social relationships and can answer the questioning agent.","marker":"[16]"},{"why":"Supplies chain-of-thought prompting, which the agent's self-critical loop uses to reason about gaps in the description.","marker":"[21]"},{"why":"Supplies the self-check and self-critique mechanism that lets the agent score and revise its own description.","marker":"[11]"},{"why":"Supplies the base epidemic model that the simulations use to spread facts through the employee network.","marker":"[22]"},{"why":"Supplies the waning-immunity variant that makes each employee's ability to share a fact decay over time.","marker":"[23]"},{"why":"Supplies METEOR, an external metric used to compare reconstructed column descriptions with the original table.","marker":"[29]"},{"why":"Supplies G-Eval, the LLM-as-judge framework used to measure coherence and faithfulness against the original knowledge.","marker":"[30]"}],"fun_headline_variants":["LLM agent recovers 95% of hidden company know-how","Agent mines tacit knowledge without reaching the expert","Chatbot interviews reconstruct scattered organizational facts","94.9% recall: AI agent finds knowledge via staff chats","AI agent rebuilds lost knowledge from employee conversations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM-simulated employees behave about as helpfully and as knowledgeably as real employees would, and that the generated ground-truth descriptions are representative of real organizational knowledge; if those two hold, the 94.9% recall is a genuine property of the conversational method rather than an artifact of the simulation.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent recovers 95% of hidden company know-how","Agent mines tacit knowledge without reaching the expert","Chatbot interviews reconstruct scattered organizational facts","94.9% recall: AI agent finds knowledge via staff chats","AI agent rebuilds lost knowledge from employee conversations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1196,"prompt_tokens":866,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":254}},"tokens_in":482,"tokens_out":330,"duration_ms":4535,"temperature":1.0,"reasoning_tokens":254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:01:22.943059+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same agent against simulated employees whose backgrounds do not tell them what their colleagues know, and who answer only with facts they personally hold, then compare full-knowledge recall; if it falls well below 94.9%, the original figure was carried by leaked metadata in the simulation rather than by conversational reconstruction.","supporting_citations":[{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies chain-of-thought prompting, which the agent's self-critical loop uses to reason about gaps in the description."},{"cited_title":"A contribution to the mathe- matical theory of epidemics,","cited_arxiv_id":null,"evidence_quote":"Supplies the base epidemic model that the simulations use to spread facts through the employee network."},{"cited_title":"Sir-based mathematical modeling of infectious diseases with vaccination and waning immunity,","cited_arxiv_id":null,"evidence_quote":"Supplies the waning-immunity variant that makes each employee's ability to share a fact decay over time."},{"cited_title":"Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments,","cited_arxiv_id":null,"evidence_quote":"Supplies METEOR, an external metric used to compare reconstructed column descriptions with the original table."},{"cited_title":"G-eval: Nlg evaluation using gpt-4 with better human alignment,","cited_arxiv_id":null,"evidence_quote":"Supplies G-Eval, the LLM-as-judge framework used to measure coherence and faithfulness against the original knowledge."}],"review_version":1}