Google AI and UC Berkeley Researchers Propose AI Approach to Teach LLMs to Debug Predicted Programs with Few-shot Demonstrations

Generating correct code in one go for many programming jobs can be difficult. Writing code has long been a problem for several applications, such as code synthesis from natural language, programming by example, and code transformation. In particular, recent large-scale language models are vastly improved over previous deep neural networks. In one study, a re-ranking method was developed to select the best candidates from multiple samples, which typically required dozens of samples. These techniques were inspired by the observation that correct code is much more likely to be predicted when different programs are sampled from the model.

Intuitively, programmers’ first code is usually inaccurate. Humans often examine code, check execution results, and then make adjustments to fix implementation flaws rather than rejecting flawed code outright. Previous studies have proposed deep learning algorithms that fix the expected code and have shown significant performance gains in a variety of coding jobs. Nevertheless, these methods require additional training of code repair models.

Previous research has suggested that large language models cannot fix code in the absence of external feedback such as unit tests or human direction, but recent research suggests that these models generate feedback messages. It has been shown that it is possible to critique and improve the output by For some natural language and reasoning domains. In this study, Google Research and UCB researchers provide self-debugging. It uses a handful of prompts to educate a huge language model on debugging your own projected code. SELFDEBUGGING instructs the model to run code and creates feedback messages based on the code and execution results without requiring additional model training.

SELF-DEBUGGING trains the model to detect implementation issues by explaining the code. This contrasts with previous work that used human feedback for code repair, where feedback messages describe errors in the code and how to fix them. This debugging procedure is similar to the rubber his duck debugging technique used by human programmers. In normal language he rubberducks the code line by line, which improves the efficiency of debugging without expert help. The entire SELF-DEBUGGING technique is shown in Figure 1. They evaluate GPT-3 model family code DaVinci-002 for his SELF-DEBUGGING.

For a variety of code generation tasks such as text-to-SQL generation, code conversion, and text-to-Python generation, SELFDEBUGGING offers state-of-the-art performance. With code descriptions in the issue description and no unit tests, the Spider benchmarks for text-to-SQL generation show a steady 2-3% increase in baseline with self-debugging for varying numbers of initial programs, We show that prediction accuracy improves for the most complex programs. 9% reduction in SQL queries.

Up to 12% improvement in accuracy when using unit tests combined with code descriptions in TransCoder for transcoding and MBPP for text-to-Python generation. In contrast, just explaining the code without debugging routinely improves transcoding performance by 2-3%. Self-debugging improves sample efficiency and can perform as well or better than a baseline model sampling over 10 predictions. According to their research, teaching large language models to self-debug without human supervision is another way to increase coding ability and lower the sampling cost required to complete difficult tasks. A promising method. This is in addition to improving our ability to generate code from scratch.

check out paper. All credit for this research goes to the researchers of this project.Also, don’t forget to participate Our 18k+ ML SubReddit, cacophony channeland email newsletterWe share the latest AI research news, cool AI projects, and more.

🚀 Check out 100 AI Tools in the AI ​​Tools Club