We present a benchmark dataset for evaluating method-level code generation task. The benchmark contains a dataset of 175 samples for automated evaluation and a dataset of 161 samples for manual evaluation. We also present a new metric for automatically evaluating the correctness of the generated code, and a set of criteria to manually evaluating the overall quality of the generated code.
AixBench: A Code Generation Benchmark Dataset
A benchmark dataset and evaluation metrics for method-level code generation are provided, including automated and manual evaluation criteria.
- Year
- 2022
- Venue
- arXiv 2022
- Authors
- 8
- Hosting
- Abstract onlyARXIV-DEFAULT
Cite
Notes
Only stored in your browser.
Attribution
- Abstract & full text
- arxiv.org/abs/2206.13179ARXIV-DEFAULT
- TL;DR
- Semantic Scholar