REVIEW 1 cited by
WebApp1K: A Practical Code-Generation Benchmark for Web App Development
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce WebApp1K, a practical code-generation benchmark to measure LLM ability to develop web apps. This benchmark aims to calibrate LLM output and aid the models to progressively improve code correctness and functionality. The benchmark is lightweight and easy to run. We present the initial version of WebApp1K, and share our findings of running the benchmark against the latest frontier LLMs. First, open source LLMs deliver impressive performance, closely trailing behind GPT-4o and Claude 3.5. Second, model size has strong correlation with code correctness. Third, no prompting techniques have been found to lift performance either universally to all models, or significantly to a single model.
Forward citations
Cited by 1 Pith paper
-
CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation
CrossPL, a 1,982-task benchmark built from GitHub repositories, shows that LLMs achieve at most 79.74% pass@1 on cross-language IPC code generation and struggle with low-level protocols like Pipe.
Discussion (0). Sign in to comment.