TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus

David Semedo; Diogo Silva; Diogo Tavares; In\^es Sim\~oes; Jo\~ao Bordalo; Jo\~ao Magalh\~aes; Rafael Ferreira; Rodrigo Val\'erio; Vasco Ramos

RECORD

TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus

Rafael Ferreira, Diogo Tavares, Diogo Silva, Rodrigo Valério, João Bordalo, Inês Simões, Vasco Ramos, David Semedo

show 1 more author

João Magalhães

Computation and Language · Submitted 3 Oct 2023

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Given ratings and feedback provided by users, we observed that TWIZ bot is an effective and robust system, capable of guiding users through tasks while providing several multimodal stimuli.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2310.02118 (version 2)

Pith ID6O2X6FIY, Pith's permanent record of this paper and its review

CitationDavid Semedo, Diogo Silva, Diogo Tavares, In\^es Sim\~oes, Jo\~ao Bordalo, Jo\~ao Magalh\~aes, Rafael Ferreira, Rodrigo Val\'erio, Vasco Ramos. "TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus." arXiv:2310.02118 (2023). https://pith.science/paper/6O2X6FIY

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 4,514 input tokens.

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.14 · human participants 0.75 · controlled comparison 0.03 · randomized assignment 0.01 · states quantitative result 0.05 · compares with prior results 0.05 · mentions code or data availability 0.04.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

In this report, we describe the vision, challenges, and scientific contributions of the Task Wizard team, TWIZ, in the Alexa Prize TaskBot Challenge 2022. Our vision, is to build TWIZ bot as an helpful, multimodal, knowledgeable, and engaging assistant that can guide users towards the successful completion of complex manual tasks. To achieve this, we focus our efforts on three main research questions: (1) Humanly-Shaped Conversations, by providing information in a knowledgeable way; (2) Multimodal Stimulus, making use of various modalities including voice, images, and videos; and (3) Zero-shot Conversational Flows, to improve the robustness of the interaction to unseen scenarios. TWIZ is an assistant capable of supporting a wide range of tasks, with several innovative features such as creative cooking, video navigation through voice, and the robust TWIZ-LLM, a Large Language Model trained for dialoguing about complex manual tasks. Given ratings and feedback provided by users, we observed that TWIZ bot is an effective and robust system, capable of guiding users through tasks while providing several multimodal stimuli.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	In this report, we describe the vision, challenges, and scientific contributions of the Task Wizard team, TWIZ, in the Alexa Prize TaskBot Challenge 2022.	contribution 0.90	0.56	0.04	asserted 0.81
s1	Our vision, is to build TWIZ bot as an helpful, multimodal, knowledgeable, and engaging assistant that can guide users towards the successful completion of complex manual tasks.	contribution 0.80	0.17	0.06	asserted 0.96
s2	To achieve this, we focus our efforts on three main research questions: (1) Humanly-Shaped Conversations, by providing information in a knowledgeable way; (2) Multimodal Stimulus, making use of various modalities including voice, images, and videos; and (3) Zero-shot Conversational Flows, to improve the robustness of the interaction to unseen scenarios.	contribution 0.64	0.40	0.05	asserted 0.87
s3	TWIZ is an assistant capable of supporting a wide range of tasks, with several innovative features such as creative cooking, video navigation through voice, and the robust TWIZ-LLM, a Large Language Model trained for dialoguing about complex manual tasks.	contribution 0.88	0.70	0.08	asserted 1.00
s4	Given ratings and feedback provided by users, we observed that TWIZ bot is an effective and robust system, capable of guiding users through tasks while providing several multimodal stimuli.	result 0.93	0.92	0.80	asserted 1.00

What kind of paper the classifier reports this is: method or tool 0.92 · evidence from observation or experiment 0.90.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

In this report, we describe the vision, challenges, and scientific contributions of the Task Wizard team, TWIZ, in the Alexa Prize TaskBot Challenge 2022. Our vision, is to build TWIZ bot as an helpful, multimodal, knowledgeable, and engaging assistant that can guide users towards the successful completion of complex manual tasks. To achieve this, we focus our efforts on three main research questions: (1) Humanly-Shaped Conversations, by providing information in a knowledgeable way; (2) Multimodal Stimulus, making use of various modalities including voice, images, and videos; and (3) Zero-shot Conversational Flows, to improve the robustness of the interaction to unseen scenarios. TWIZ is an assistant capable of supporting a wide range of tasks, with several innovative features such as creative cooking, video navigation through voice, and the robust TWIZ-LLM, a Large Language Model trained for dialoguing about complex manual tasks. Given ratings and feedback provided by users, we observed that TWIZ bot is an effective and robust system, capable of guiding users through tasks while providing several multimodal stimuli.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 1 sentence without a confident label

Uses the method or tool of 4 papers

We propose a new method [ref] , which uses an LLM to convert the task into a visual prompt with high consistency that can then be used in text-to-image models.

Transferring Visual Attributes from Natural Language to Verified Image Generation · Generative Task Illustration · 0.85
We generate multiple images for each task using the Stable Diffusion XL model [ref] . * The image score is given by an alignment score.

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis · Generative Task Illustration · 1.00
For each keyframe, we generated a caption, using InstructBLIP [ref] .

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning · Video Moment Retrieval · 1.00

Compares with 2 papers

The results of Table [ref] show that our method achieves better results than the TIFA [ref] baseline. * Lastly, we rank the generated images, as well as the original image, according to their score and choose the one with the highest score to illustrate the task [ref] .

TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering · Generative Task Illustration · 0.97
The results of Table [ref] show that our method achieves better results than the TIFA [ref] baseline. * Lastly, we rank the generated images, as well as the original image, according to their score and choose the one with the highest score to illustrate the task [ref] .

Transferring Visual Attributes from Natural Language to Verified Image Generation · Generative Task Illustration · 0.97

Cites as background 1 paper

Given the recent advancements in using generative methods for both text [ref] and image generation [ref] , a recent and very important research direction is how to guarantee trustworthy and consistent generations [ref] .

Transferring Visual Attributes from Natural Language to Verified Image Generation · Trustworthy TaskBot Generative Vision and Language · 0.92

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.