{"id":"5d0d9abf-b260-48a8-a8f5-169e4b9930c8","arxiv_id":"1909.00031","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PUMICE lets end users program smartphone automation by combining spoken instructions with demonstrations, resolving unclear concepts through conversation and app GUI references.","lead":"This paper describes PUMICE, a smartphone agent that lets people teach it new tasks by talking and by demonstrating actions in existing apps. It combines natural language commands with on-screen demonstrations so users can define vague concepts like 'hot' or 'heavy traffic' step by step.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"","rationale":"","tokens_in":18888,"tokens_out":4914,"duration_ms":45925,"concrete_test":"","verdict_should_be":"UNCHANGED","load_bearing_attack":"","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PUMICE, an end-user programmable agent that combines natural language instructions with GUI demonstrations to let users specify conditional automation rules and teach new concepts on Android smartphones. The authors first report a formative Mechanical Turk study (n=58) that motivates the design, then describe the PUMICE architecture, including a SEMPRE-based semantic parser with typed resolve() functions, a SUGILITE-based demonstration recorder, and a persistent knowledge graph for learned procedures and concepts. The evaluation is a lab usability study with 10 participants who completed four author-designed tasks; all participants completed all tasks, and post-survey Likert ratings were positive. The paper is an empirical systems paper with no formal or mathematical derivation. The main claims are that the multi-modal, top-down recursive clarification approach is usable by non-programmers and that the approach is domain-independent by virtue of grounding concepts in third-party app GUI contents.","tokens_in":18759,"tokens_out":4040,"duration_ms":39047,"significance":"If the result holds, the contribution is valuable: the paper provides concrete evidence that end users without programming expertise can define abstract conditions such as 'hot' or 'heavy traffic' through a mix of conversational clarification and in-app demonstration, and the open-sourced implementation supports replication and extension. The formative study is a useful design contribution, and the candor of the Limitations section is a strength. The significance is moderated by three factors: the usability claim rests on a small, no-control study; the correctness of the produced scripts is not reported despite being checked; and the 'domain-independent' claim is narrower than stated, as the authors themselves acknowledge in the Limitations and Future Work section. The paper is best read as a feasibility demonstration of a specific interaction design rather than a comparative validation of the approach against alternatives.","major_comments":[{"comment":"The central usability claim, stated in the Abstract as 'A lab study with 10 users showed its usability,' is supported only by a 10-participant study with no control condition, in which the tasks were designed by the authors to exercise PUMICE's features and participants received a 5-minute tutorial with an example that pre-taught the target workflow (e.g., 'late' defined as 'current time is after 8pm'). Because there is no comparison against a baseline such as programming by demonstration alone, natural language alone, or an existing tool, the data cannot distinguish the contribution of the multi-modal conversational approach from the effect of the tutorial, the pre-selected tasks, or the pre-stored locations and concepts. I recommend either adding a comparison condition, or explicitly re-scoping the claim from 'usability' to 'feasibility' with the current evidence.","section":"User Study (Tasks and Results)"},{"comment":"The paper states that the experimenter 'checked the scripts at the end of each study session to evaluate their correctness' (User Study, Procedure), but no correctness results are reported anywhere in the Results section. The statement that 'all participants were able to complete all 4 tasks' indicates self-reported or observed completion, not that the generated scripts correctly executed the intended automation. Since the authors also note that participants encountered speech recognition errors and parsing errors and used the undo feature to recover, completion alone does not establish that the final programs were correct. Please report the number of scripts that passed the correctness check and the number that required corrections, so that the reader can assess whether the interaction design actually led to correct end-user programs.","section":"User Study (Results)"},{"comment":"The claim that PUMICE is 'domain-independent' is load-bearing for the paper's contribution statement, but the paper's own Limitations section states that the system has 'no semantic understanding of information involved in tasks,' cannot handle implicit parameters, relations between concepts, references, synonyms, antonyms, or implicit conjunctions/disjunctions, and that the DSL only supports single basic Boolean operations. In addition, all concepts must be grounded in information visible in third-party app GUIs, which the stress-test note correctly identifies as a boundary of the approach. As written, the contribution is more accurately described as a domain-independent interaction mechanism (top-down conversational decomposition plus GUI demonstration) over app-specific grounding, rather than domain independence of the learned representations or of the parser. Please either soften the 'domain-independent' claim throughout the Abstract and contributions, or provide evidence that the mechanism transfers across unseen task domains without per-domain engineering beyond the four study tasks.","section":"PUMICE and Limitations and Future Work"}],"minor_comments":[{"comment":"The comparison between 33% and 9% is reported as statistically significant with p < 0.05, but the test statistic, test type, and effect size are not given; please report these details for reproducibility.","section":"Formative Study"},{"comment":"The semantic parser is described as trained on n=905 examples, but no parser accuracy, per-utterance success rate, or analysis of failure modes is reported; a small evaluation table would strengthen the reproducibility of the system claims.","section":"System Implementation"},{"comment":"The caption says the red overlays 'highlight all durations' that PUMICE identified on the Google Maps GUI; the accompanying text in the PUMICE section describes highlighting 'possible items' and 'all values visible on the screen.' Please make the figure caption and body text consistent.","section":"Figure 2"},{"comment":"The phrases 'showed its usability' and 'supported by our summative lab usability study' overstate the evidence given the small no-control design; consider replacing them with 'showed feasibility' or adding a sentence that acknowledges the lack of a control condition.","section":"Abstract and Conclusion"},{"comment":"The paper cites [Li et al. 2019] as the earlier UIST version of PUMICE; the present version appears to be the workshop version of the same work. Please clarify the delta between this submission and the UIST paper to help readers understand the incremental contribution.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable systems/usability contribution, but the novelty relative to the authors' own UIST 2019 paper (referenced as [Li et al. 2019]) should be clarified for the editor. The main risk is overclaiming: the user study is small, uncontrolled, and task-correctness data are missing. The limitations section is candid and lists several load-bearing scope restrictions, so the fix is mostly a matter of re-scoping claims and adding missing reporting rather than invalidating the core idea. I recommend major revision, not rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If you've seen the UIST 2019 version, there's nothing new here — the authors say so themselves and cite it. But the PUMICE design is genuinely useful: let users state a task at a high level, then recursively resolve vague concepts through conversation, grounding them in content visible in third-party app GUIs. The formative study result that showing app screenshots cuts vague-concept usage from 33% to 9% is a real, usable data point. The system is open source and built on SUGILITE, so the engineering is reproducible.\n\nThe usability study is honest but limited. Ten users, all completed four author-designed tasks, after a tutorial. Positive Likert scores. No control condition, no baseline against SUGILITE alone or a simpler conversational agent. The authors give a fair reason — no comparable tool exists for non-programmers — but it means the stronger claims about naturalness and low learning barrier rest on subjective ratings and a small lab sample.\n\nThe soft spots are mostly acknowledged in the paper. The parser is brittle: pronouns, synonyms, and implicit parameters are out of scope. The \"domain-independent\" claim outruns the implementation — everything must ground in visible GUI content, so user mood, implicit context, or anything not on screen is unhandled. That's a real limitation, not a minor one, and any citation should name it. The generalization features are demonstrated anecdotally, not measured.\n\nCitation pattern is fine. The paper builds on the authors' own SUGILITE and APPINITE work and cites the prior UIST paper. No red flags.\n\nThis is for someone working on end-user development, conversational agents, or programming-by-demonstration. It's a useful system description with a candid evaluation. As a fresh submission it would deserve a serious referee, though originality would be a problem given the UIST version. As a workshop recap it's fine.\n\nRecommendation: if this crosses your desk as a submission, send it to review — the system is real and the write-up is honest. But if you're deciding what to read, read the UIST paper instead; it's the same content.","headline":"Solid HCI systems paper with a real open-source implementation and honest usability data, but the arXiv version is a recap of the UIST 2019 paper and the domain-independence claim outruns what the evaluation supports.","tokens_in":19339,"tokens_out":2454,"would_cite":false,"duration_ms":22271,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pairing conversational natural-language instructions with GUI demonstrations enables non-programmers to author conditional task automations, because ambiguous concepts can be recursively grounded in values visible in…","keywords":["end-user development","natural language programming","programming by demonstration","task automation","concept learning","conditional statements","conversational agents","mobile app GUIs"],"falsifier":"Give the system a task whose condition depends on information not displayed in any app, such as \"if I am feeling tired\" or \"if the house is empty\"; the recursive clarification has no GUI content to point to, so it cannot produce a grounded Boolean concept.","tokens_in":18662,"feed_emoji":"🤖","tokens_out":7006,"duration_ms":60066,"temperature":0.7,"pith_summary":"The paper tries to show that non-programmers can author conditional task automations by speaking naturally and demonstrating inside existing mobile apps. Its key claim is that instead of forcing users to define everything up front, the agent can parse a high-level instruction, spot unknown concepts, and recursively ask for explanations or demonstrations until every condition is grounded in concrete GUI content. A formative study found that natural instructions often use vague concepts like \"hot\" or \"heavy traffic,\" and that showing app screens reduced such vagueness; the paper argues the agent's design addresses this. The paper reports a lab study with 10 users, all of whom completed four automation tasks, as evidence the approach is usable by people with little or no programming experience.","feed_headline":"Plain talk plus screen demos teaches agents conditional tasks","feed_subtitle":"PUMICE asks users to demonstrate fuzzy concepts like \"hot\" or \"heavy traffic\" inside existing mobile apps.","key_machinery":"The central mechanism is the parser's typed resolve() function, a placeholder inserted into the semantic parse wherever an unknown or unclear concept appears; at runtime the agent executes these placeholders depth-first, asking the user to explain or demonstrate each one, and replaces them with definitions. The system also maintains a UI snapshot graph of GUI objects visible in third-party apps, so a user can ground a value concept by pointing at a screen element, and a persistent knowledge graph of procedures and concepts for reuse.","core_discovery":"On its own terms, the paper establishes a workflow: a user says something like \"if it's hot, order iced coffee,\" the agent parses the conditional structure, flags \"hot\" and \"order iced coffee\" as unresolved typed components, and then asks the user how to determine \"hot\" and how to perform the order. The user can answer with more language or with a demonstration inside an app; each new unknown recursively triggers another question until everything bottoms out in values visible on app screens. The paper further claims that the resulting knowledge is stored in a persistent graph that supports generalization and reuse across contexts, and it reports that in a lab study all 10 participants completed all 4 automation tasks, with subjective ratings around 6 on the 7-point scales for ease of use and naturalness.","pith_inferences":["A consequence the paper leaves implicit is that the same recursive grounding loop could work on desktop or web interfaces whenever GUI content can be programmatically read, so the approach is not inherently limited to mobile apps.","The persistent knowledge graph points toward cross-user sharing of taught concepts, but the paper's own limitation section notes that personal preferences and generalizable knowledge are not yet separated, so sharing would require a privacy boundary first.","A testable extension would be to support logical combinations like \"hot and raining\" or arithmetic like \"at least $10 more expensive,\" which the paper lists as future work; usability with these richer expressions is unmeasured."],"forward_implications":["Non-programmers can author task automations with conditionals without writing code, by explaining vague conditions in terms of concrete values and demonstrating where those values appear.","A concept taught once, such as \"hot,\" can be reused in a new task domain with the same comparison but a different threshold or a different data source.","The agent's habit of asking about the \"else\" case addresses the frequent omission of alternative branches in natural instructions.","Because all grounding is done through existing third-party app interfaces, the approach works across task domains as long as a relevant app exists.","Learned concepts and procedures are stored persistently, so later instructions can refer to them and trigger the agent to ask whether the old definition still applies."],"supporting_citations":[{"why":"Supplies the underlying demonstration recording and replaying capability on third-party Android apps, the foundation for procedure learning.","marker":"[Li, Azaria, and Myers 2017]"},{"why":"Provides the UI snapshot graph representation that lets PUMICE interpret user references to app GUI contents and generate value queries.","marker":"[Li et al. 2018a]"},{"why":"Provides the semantic parsing framework on which PUMICE's parser is built, extended with typed resolve() placeholders.","marker":"[Berant et al. 2013]"},{"why":"Contributes the grammar-based Floating Parser architecture that lets the parser work without hand-engineered lexicalized rules.","marker":"[Pasupat and Liang 2015]"},{"why":"Demonstrates the conversational clarification strategy for ambiguous commands that PUMICE extends toward domain-independence.","marker":"[Fast et al. 2018]"},{"why":"Supplies the Natural Programming Elicitation method used in the formative study to collect unbiased language from users.","marker":"[Myers et al. 2016]"},{"why":"Adds the interaction proxy overlay that records all visible values on screen during value-concept demonstrations.","marker":"[Zhang et al. 2017]"},{"why":"Represents the instructable personal agent paradigm against which PUMICE positions its GUI-grounding, domain-independent approach.","marker":"[Azaria, Krishnamurthy, and Mitchell 2016]"}],"fun_headline_variants":["Teach agents with plain talk plus screen demos","Agent learns fuzzy concepts via chat and demos","Natural language plus demos lets users program agents","PUMICE turns vague phrases into concrete app actions","Ask, then show: teaching agents conditional tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that every concept a user needs can be grounded in information visible on the screen of an existing third-party mobile app, and that the user is familiar enough with that app to demonstrate where to find it.","fun_headline_variants_meta":{"raw":{"variants":["Teach agents with plain talk plus screen demos","Agent learns fuzzy concepts via chat and demos","Natural language plus demos lets users program agents","PUMICE turns vague phrases into concrete app actions","Ask, then show: teaching agents conditional tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1207,"prompt_tokens":856,"completion_tokens":351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":279}},"tokens_in":472,"tokens_out":351,"duration_ms":3773,"temperature":1.0,"reasoning_tokens":279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:03:24.500560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the system a task whose condition depends on information not displayed in any app, such as \"if I am feeling tired\" or \"if the house is empty\"; the recursive clarification has no GUI content to point to, so it cannot produce a grounded Boolean concept.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the semantic parsing framework on which PUMICE's parser is built, extended with typed resolve() placeholders."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates the conversational clarification strategy for ambiguous commands that PUMICE extends toward domain-independence."},{"cited_title":"A.; Ko, A","cited_arxiv_id":null,"evidence_quote":"Supplies the Natural Programming Elicitation method used in the formative study to collect unbiased language from users."},{"cited_title":"S.; Caspi, A.; Fogarty, J.; and Wobbrock, J","cited_arxiv_id":null,"evidence_quote":"Adds the interaction proxy overlay that records all visible values on screen during value-concept demonstrations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the instructable personal agent paradigm against which PUMICE positions its GUI-grounding, domain-independent approach."}],"review_version":1}