Pith. sign in

REVIEW 4 major objections 6 minor 25 references

Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current multimodal LLMs cannot reliably generate working code from flowcharts; the new Flow2Code benchmark measures this gap across 15 languages and shows fine-tuning helps.

desk verdict A genuinely useful benchmark whose quantitative claims are undercut by an unexplained train/test split inconsistency. read the letter →

arxiv 2506.02073 v1 pith:O6AZVFVZ submitted 2025-06-02 cs.SE cs.AI

Mengliang He , Jiayi Zeng , Yankai Jiang , Wei Zhang , Zeming Liu , Xiaoming Shi , Aimin Zhou This is my paper · ORCID
classification cs.SEcs.AI
keywords flowchart-basedcodegenerationmultimodallargelanguagemodelsbenchmarkUMLflowchartpseudoPass@kevaluationsupervisedfine-tuningmultilingual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents Flow2Code, a benchmark of 5,622 code segments paired with 16,866 flowcharts in three styles—code, UML, and pseudocode—across 15 programming languages, and uses it to test 13 multimodal LLMs on generating code from flowchart images. The central claim is that current LLMs cannot reliably perform flowchart-based code generation: the average Pass@1 across models and languages on the easiest (code) flowcharts stays below 60%, and scores fall further on UML and pseudocode flowcharts. The paper also shows that supervised fine-tuning of a 7B model on Flow2Code substantially narrows the gap, suggesting the benchmark captures a learnable skill rather than an impossible task. A sympathetic reader would care because flowcharts are a common human way to specify program logic, and no prior benchmark measured whether multimodal models can follow them.

What carries the argument

The carrying mechanism is the Flow2Code benchmark itself: a dataset plus an evaluation protocol. Source code from four existing multilingual benchmarks is converted into DOT graph descriptions by Visustin, rendered into code and UML flowcharts via Graphviz, and a GPT-4o step rewrites node labels into plain-English pseudocode, with Gemini-2.0 checking the rewrite and human reviewers verifying a sample with reported inter-annotator agreement of Krippendorff's alpha 0.88. Evaluation uses execution-based Pass@1/3/5 scores, with generated code run against test cases, and the paper also fine-tunes Qwen2-VL-7B with LoRA on the training split to test whether the capability can be improved.

What would settle it

Check whether any Flow2Code test instance's reference solution appears in the released training split or in the source datasets' own test sets, for example by exact-match or near-duplicate search on normalized code; if even a few percent of the 925 test code segments duplicate fine-tuning or pretraining data, the reported Pass@1 levels for the fine-tuned model are inflated and the capability-gap conclusion needs re-measuring on a clean split.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that flowchart-to-code generation is an open capability gap for multimodal LLMs. Across 13 models, the best performer, Gemini-2.0, clearly outpaces the rest, while many models land far lower, and the average Pass@1 across all models and languages on code flowcharts is below 60%; UML and pseudocode flowcharts are harder still, with pseudocode the most challenging. The paper further finds that fine-tuning Qwen2-VL-7B on the Flow2Code training split substantially raises its average Pass@1, bringing it near the level of much larger closed models, and that model scale helps within a family. These results support the paper's stated conclusion that current LLMs lack sufficient flowchart-based code generation capability, particularly on UML and pseudocode flowcharts.

Load-bearing premise

The load-bearing premise is that Flow2Code's test split is truly unseen—the 9:1 split claimed in Section 3.3.1 is hard to reconcile with the 925 test instances in Table 3—and that each flowchart carries exactly the logic needed to reconstruct the reference code, so scores measure flowchart understanding rather than test-set leakage or missing text.

Editorial extensions

If this is right

  • If Flow2Code is accepted, flowchart-based code generation becomes a measurable benchmark dimension alongside text-to-code and plot-to-code, giving model developers a new target to optimize.
  • The finding that supervised fine-tuning on Flow2Code lifts a 7B model close to much larger closed models implies that a substantial part of the observed gap is trainable rather than a hard architectural limit.
  • Because pseudocode flowcharts are the hardest input, the benchmark isolates natural-language understanding inside a visual structure as a distinct weakness of current multimodal LLMs.
  • The large language-dependent variance, with near-perfect PHP scores sitting next to weak Fortran and HTML scores, implies that flowchart-to-code performance is partly an artifact of training-data prevalence, not purely visual reasoning.
  • Pass@5 results are consistently higher than Pass@1, implying that models often know the right logic but fail to produce it on the first attempt, making sampling-based evaluation and decoding strategy relevant levers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: add a text-only baseline that feeds the same reference code as pseudocode text without the flowchart image; if the image adds no difficulty, the visual-flowchart gap would collapse into plain code generation.
  • The 9:1 split described in Section 3.3.1 implies about 562 test samples for 5,622 code segments, yet Table 3 reports 925 test instances; if the released test set is instead the full source-dataset test partition, some test flowcharts may overlap with the fine-tuning data, which would inflate the reported fine-tuning gains.
  • Flowchart construction itself passes through LLMs, with GPT-4o generating pseudocode labels and Gemini-2.0 checking them, so the pseudocode flowchart inputs carry an encoding bias from the same families of models being evaluated; a dataset authored entirely by humans would be needed to separate flowchart difficulty from LLM-generated-label artifacts.
  • A natural next step, flagged by the authors in their limitations section, is to extend Flow2Code from one-shot generation to debugging and refactoring; one concrete version is to give models a flowchart plus a buggy implementation and measure whether the diagram helps localize the fault.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Flow2Code, a benchmark for evaluating multimodal large language models on code generation from flowchart images. The benchmark is constructed from four existing code-generation datasets (HumanEval-X, MBXP, McEval, ClassEval), yielding 5,622 code segments and three flowchart types per segment: code flowcharts and UML flowcharts generated by Visustin, and pseudocode flowcharts produced by a GPT-4o transformation of DOT representations with a Gemini-2.0 verification step. A two-step human evaluation is reported to ensure quality. The authors evaluate 13 multimodal LLMs in zero-shot settings and one LoRA fine-tuned model (Qwen2-VL-7B-FT), reporting Pass@1, Pass@3, and Pass@5. The central claims are that current LLMs lack sufficient flowchart-based code generation capability, particularly for UML and pseudocode flowcharts, and that supervised fine-tuning substantially improves performance.

Significance. If the empirical claims are supported, Flow2Code would be a useful multilingual, multimodal benchmark for a previously understudied task: translating visual flowcharts into executable code. The paper ships a public dataset and code, covers 15 programming languages and three flowchart types, and reports inter-annotator agreement (Krippendorff's alpha = 0.88) for the human verification. The comparison across 13 models, including both open and closed systems, is a plausible starting point for future work on flowchart-based code generation. However, the benchmark's ability to measure generalization from flowcharts to code, and the attribution of the performance gap to the flowchart modality, depend on resolving several load-bearing methodological issues: the train/test split discrepancy, the absence of a text-only baseline, the role of GPT-4o and Gemini-2.0 in constructing the pseudocode data they are then evaluated on, and the inconsistent description of the human review process.

major comments (4)
  1. [Section 3.3.1; Tables 3, 5, 6] The stated 9:1 train/test split within each source dataset would produce roughly 562 test instances (10% of 5,622 code segments), but the test columns in Tables 3, 5, and 6 sum to 925 instances (ClassEval 100 + HumanEval-X 164 + MBXP 611 + McEval 50 = 925, about 16.5% of the data). This inconsistency is never explained. If the actual split is not 9:1, then either some test items were also used for fine-tuning, which would directly inflate the Qwen2-VL-7B-FT results (Table 3, row 13), or the evaluation protocol differs from the written description. In either case the reported Pass@k numbers cannot currently be read as a measure of held-out flowchart-to-code generalization. Please clarify the exact split protocol and report the correct training/test sizes.
  2. [Section 4.4; Tables 3–12] The central claim that 'current LLMs lack sufficient flowchart-based code generation capability' requires a control condition in which the same tasks are presented as the original textual problem statements (or an equivalent natural-language description) without the flowchart image. Without such a text-only baseline, the low Pass@1 values could simply reflect the difficulty of the underlying HumanEval-X, MBXP, McEval, and ClassEval tasks rather than a specific deficiency in reading flowcharts. Please add text-only evaluations for at least a representative subset of the 13 models so that the effect of the flowchart modality can be isolated.
  3. [Section 3.2.2] The pseudocode flowcharts are generated by GPT-4o (via the DOT transformation prompt) and verified by Gemini-2.0, and both of these models are among the 13 evaluated in Tables 3, 5, and 6. This creates a circularity for the pseudocode condition: the task format is shaped by two of the models being tested, which can artificially inflate their relative performance and makes the pseudocode condition not an independent benchmark. The paper acknowledges the risk of using GPT-4o to validate its own output and mitigates it with Gemini-2.0, but the same concern applies to evaluation. Please report the number of instances that were regenerated or excluded by the Gemini check, and assess whether GPT-4o's performance on pseudocode flowcharts is stable when the pseudocode labels are regenerated by a model outside the evaluated set.
  4. [Section 3.2.1 vs. Section 3.3.2] The descriptions of the human review are contradictory. Section 3.2.1 states that 'All instances are independently double-blind reviewed by two evaluators' for code and UML flowcharts, and Section 3.2.2 says the same for pseudocode flowcharts, with Krippendorff's alpha = 0.88 reported in both places. Section 3.3.2, however, states that 'We randomly sampled 100 instances from each flowchart type for evaluation' with an average score of 0.94. These statements cannot both be true as written. Please clarify whether the human review covered the full dataset or only a random sample, and report the exact number of reviewed instances, the adjudication protocol, and the correct inter-annotator reliability estimate.
minor comments (6)
  1. [Table 3 caption] The caption contains the typo 'Flow2chart'; it should read 'Flow2Code'.
  2. [Section 3.2.2] There is a typo in 'If Gemin-2.0 identifies errors in an instance'; it should be 'Gemini-2.0'.
  3. [Table 3 header] The table header for HumanEval-X appears to list only four language columns (Python, CPP, Java, JS) while the text and other tables imply the full set of supported languages; please clarify which languages are actually included and ensure the header is unambiguous.
  4. [Throughout] The dataset name is spelled inconsistently as 'McEval', 'MCEval', and 'MCEVAL'; please standardize to one spelling.
  5. [Figure 1] The text within the flowchart construction diagram is very small and likely illegible in print; a higher-resolution version or a larger font would improve readability.
  6. [Section 3.3.1] The exact split procedure is ambiguous: the phrase 'within each language subset of the original source datasets' could mean a per-language split inside each source dataset or a per-source-dataset split; please specify precisely how the split was stratified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Flow2Code's evaluation is not definitionally tied to its construction, aside from a non-load-bearing self-citation and a separate split-consistency concern.

full rationale

Flow2Code is built from external code datasets (HumanEval-X, MBXP, MCEval, ClassEval) by converting solution code into flowcharts with Visustin, paraphrasing labels with GPT-4o, checking with Gemini-2.0, and subjecting everything to human review. The benchmark evaluation then feeds flowchart images to models and scores generated code by execution against the source datasets' test cases. None of the reported Pass@k values is entailed by these construction choices: a model can fail or pass regardless of how the flowchart was made. GPT-4o's role as the pseudocode-label generator is a possible fairness/validity concern for the pseudocode condition, but it is not a circular reduction, since the target code remains the original reference solution and the flowchart is independently human-checked. The only self-citation is Liu et al. (2020), which shares co-author Zeming Liu, but it is used only for a generic two-step human evaluation workflow; the actual quality evidence is the reported Krippendorff's alpha of 0.88 and the detailed human review criteria, so the self-citation is not load-bearing. The apparent mismatch between the stated 9:1 train/test split and the test-set sizes in Tables 3, 5, and 6 is a correctness risk and could indicate contamination or misreporting, but an internal inconsistency is not a circular derivation. No load-bearing step reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's claims rest on assumptions about the fidelity of automated flowchart conversion, the reliability of the pseudocode transformation filter, and the correctness of the data split. There are no fitted free parameters in the traditional sense; the fine-tuning hyperparameters are experimental settings, not derived quantities. No new entities (forces, particles, dimensions) are introduced.

assumptions (4)
  • domain assumption The automated Visustin conversion and the human review process together yield flowcharts that are semantically equivalent to the reference code.
    Section 3.2.1 describes the conversion and human review, but the reliability statistics (Krippendorff's Alpha 0.88) are reported for a sample; full-dataset fidelity is assumed.
  • ad hoc to paper The Gemini-2.0 check for pseudocode DOT transformations is accurate, and the five-attempt exclusion rule removes only genuinely erroneous instances.
    Section 3.2.2 describes the check and up to five regeneration attempts, excluding instances that still fail. This selection mechanism is specific to this paper and can systematically remove hard-to-verbalize control flows.
  • domain assumption The 9:1 train/test split is correctly implemented, so fine-tuned model gains reflect generalization and not overlap between training and evaluation instances.
    Section 3.3.1 describes the split; Table 3 reports test counts (100, 164, 611, 50) that sum to 925, inconsistent with 10% of 5,622, so this assumption is not verified and may be false.
  • domain assumption Execution-based Pass@k with the original benchmark test cases is a valid measure of functional correctness for this task.
    Section 4.2 adopts Pass@k from Chen et al. and Athiwaratkun et al.; this is standard for code generation benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability." pith.science (2026). https://pith.science/paper/O6AZVFVZ

@misc{pith2026250602073,
  author       = {Pith},
  title        = {Pith review of: Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O6AZVFVZ}},
  note         = {Machine review of arXiv:2506.02073}
}
read the original abstract

While large language models (LLMs) show promise in code generation, existing benchmarks neglect the flowchart-based code generation. To promote further research on flowchart-based code generation, this work presents Flow2Code, a novel benchmark for flowchart-based code generation evaluation. The evaluation dataset spans 15 programming languages and includes 5,622 code segments paired with 16,866 flowcharts of three types: code, UML, and pseudocode. Extensive experiments with 13 multimodal LLMs reveal that current LLMs can not generate code based on flowcharts perfectly. Besides, experiment results show that the supervised fine-tuning technique contributes greatly to the models' performance. We publicly release our code and datasets at https://github.com/hml-github/Flow2Code.

Figures

Figures reproduced from arXiv: 2506.02073 by the authors.

Figure 1
Figure 1. Overview of the flowchart generation process. The source code is initially converted into DOT code and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The number and proportion of each program [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Stacked diagram of Pass@1, 3, and 5 of all the evaluation models on the benchmark. The suffixes of the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The average Pass@1 rate of the three difficulty levels of the samples in the Flow2Code’s McEval part on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Average Pass@1 results for Qwen2-VL￾7B and the fine-tuned Qwen2-VL-7B-FT (on the Flow2Code dataset) on code flowchart tasks. The dataset labels in the bottom-right corner of the image show which subclass corresponds to each dimension of the radar chart. ficulty and Pas…
Figure 6
Figure 6. Figure 6: The code segment example [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: The code flowchart based on Figure [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The UML flowchart based on Figure [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: The pseudocode flowchart based on Figure [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The prompt used by Gemini-2.0 to check the DOT code generated by GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: The prompt used by GPT-4o to convert DOT code into pseudocode DOT code. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: The prompt used to evaluate LLMs on flowchart-based code generation tasks using the Flow2Code [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: The LLM message code template used for evaluating large models on flowchart-based code generation [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages

  1. [1]

    A new DOT representation of the control flow for a piece of code

    work page

  2. [2]

    import math

    An old DOT representation of the control flow for the same piece of code. Your task is to evaluate if the transformation from the old DOT code to the new DOT code adheres to the following rules: --- **Example:** **Old DOT code:** ```dot digraph FlowChart { 1 [shape=ellipse, label="import math"]; 2 [shape=ellipse, label="def calculate_area(radius)"]; 3 [sh...

    work page

  3. [3]

    Import module XYZ

    If the `label` represents an import statement, describe it as “Import module XYZ”

    work page

  4. [4]

    Define function `function_name` with parameters `param1`, `param2` of type `type1`

    For function definitions, describe the function name and its input parameters in plain English, e.g., “Define function `function_name` with parameters `param1`, `param2` of type `type1`”. If a parameter has a default value, mention it explicitly, e.g., “Parameter `param1` has a default value of `10`.”

    work page

  5. [5]

    ", "") # Initialize variables to store current group and stack to track parentheses current_group =

    Bigcodebench: Benchmarking code genera- tion with diverse function calls and complex instruc- tions.CoRR. A Flowchart Example Due to page limitations in the main text, it is not feasible to include relatively complex flowcharts. Therefore, a set of representative flowchart exam- ples is provided in the appendix. These examples illustrate three different f...

    work page

  6. [6]

    Check if `distance` is less than `threshold`

    For conditionals, describe them in natural language, e.g., “Check if `distance` is less than `threshold`.”

    work page

  7. [7]

    Iterate through the list `numbers`

    For loops, describe them as “Iterate through the list `numbers`.”

    work page

  8. [8]

    Your task is to generate a natural language-oriented flowchart in `DOT` syntax

    A `DOT` representation of the control flow for a piece of code. Your task is to generate a natural language-oriented flowchart in `DOT` syntax. This flowchart must: - Clearly describe the logic in plain English. - Follow `DOT` syntax accurately. - Include key operations, conditions, and outputs. - Be mindful of the specific characteristics of the programm...

    work page

Show all 25 references
  1. [9]

    Only modify the `label` content of the nodes to describe the code's logic in natural language

    **Do not change the structure of the `DOT` code**. Only modify the `label` content of the nodes to describe the code's logic in natural language

  2. [10]

    In particular, the definitions of various data types and function types should be clarified

    For the "label" fields, describe the code inside them in clear and concise natural language, while considering the characteristics of the given programming language. In particular, the definitions of various data types and function types should be clarified

  3. [13]

    Assign the value of `radius * radius * pi` to `area`

    For operations like assignments or calculations, describe them in natural language, e.g., “Assign the value of `radius * radius * pi` to `area`.”

  4. [16]

    Return `result`

    If a function has a return statement, describe the returned value, e.g., “Return `result`.”

  5. [17]

    **Do not modify the rest of the `DOT` syntax** like nodes, edges, or graph attributes; only modify the labels as instructed

  6. [18]

    ```”). --- **Example:** **Given DOT Input:** ```dot digraph FlowChart { 1 [shape=ellipse, label=

    **Only output the `DOT` code inside code blocks** (i.e., between the triple backticks “```”). --- **Example:** **Given DOT Input:** ```dot digraph FlowChart { 1 [shape=ellipse, label="import math"]; 2 [shape=ellipse, label="def calculate_area(radius)"]; 3 [shape=box, label="ar...

  7. [19]

    Ensure that the program is functional and adheres to the flowchart's steps

    Write the **complete code** based on the flowchart's structure. Ensure that the program is functional and adheres to the flowchart's steps

  8. [20]

    - Implement loops, conditionals, and assignments as indicated by the flowchart

    The generated code must match the flowchart's described logic: - Define any necessary functions. - Implement loops, conditionals, and assignments as indicated by the flowchart. - Follow the flow and operations of the program as described by the nodes and edges

  9. [21]

    The code should **match the intended programming language**: `<<insert_language_name_here>>`

  10. [22]

    You must **only output the complete code** surrounded by **code blocks** (i.e., between triple backticks ```)

  11. [23]

    Do not include any other explanations or markdown outside the code block

  12. [24]

    The code should be solely composed of the functional elements from the flowchart, without any descriptive or explanatory annotations

    Do not include any comments or explanations within the generated code. The code should be solely composed of the functional elements from the flowchart, without any descriptive or explanatory annotations

  13. [25]

    role": "user

    Do not add any additional code that is not represented in the flowchart. For example, if there is no main function in the flowchart, do not add a main function without authorization. --- Figure 12: The prompt used to evaluate LLMs on flowchart-based code generation tasks using...

    2024

  14. [2019]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu

    Spoc: Search-based pseudocode to code.Ad- vances in Neural Information Processing Systems, 32. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark fo...

    2023

  15. [2022]

    Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu

    Competition-Level Code Generation with Al- phaCode.Science, 378(6624):1092–1097. Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020. Towards conver- sational recommendation over multi-type dialogs. In Proceedings of the 58th Annual Meeting of the ...

    2020 arXiv

  16. [2023]

    Multi-lingual evaluation of code generation models. InICLR. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, H. Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. 2021. Program Synthesis with Large Lan- guage Models.A...

    2021 arXiv

  17. [2024]

    https://blog.google/technology/google- deepmind/google-gemini-ai-update-december- 2024/

    Introducing Gemini 2.0: Our new AI model for the agentic era. https://blog.google/technology/google- deepmind/google-gemini-ai-update-december- 2024/. Qihang Ai, Jiafan Li, Jincheng Dai, Jianwu Zhou, Lemao Liu, Haiyun Jiang, and Shuming Shi. 2024. Advancement in graph understa...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.