The Importance of Human-in-the-Loop

The Importance of Human-in-the-Loop#

While AI coding assistants can significantly speed up development, it is important to remember that these tools are meant to assist, not replace, human programmers. AI models can make critical errors, often in unexpected ways. To maintain high-quality code, it is essential to have human oversight by reviewing, testing, and validating any AI-generated code.

Hallucinations#

AI models are more likely to produce incorrect outputs than admit to not knowing how to answer a question. Often, they will be confident in their wrong answers, which can lead to significant issues if not caught early. This phenomenon is known as hallucination.

Real-world examples of AI hallucinations are numerous (and humorous), ranging from news articles publishing summer reading lists of fake books to attorneys referring to non-existent cases in court.

In coding, this often means that the AI-generated code does not compile. In those cases, it is easy to spot the error. However, bugs, logic errors, or security vulnerabilities can be much harder to detect. These issues can lead to significant problems in production systems, including data loss, security breaches, and system failures.

Example

A famous example of LLMs being wrong is asking them to count. The rule-based nature of counting is not compatible with the probabilistic nature of LLMs. Try asking how many characters are in the following text:

This is an example test designed to fool LLMs. The correct answer is 46. You will likely get a different answer each time you ask, and it will almost always be wrong.

In smaller Python scripts, it may simply mean that the code does not do what you expect it to do.

The nondeterministic nature of LLMs means that they can generate different outputs for the same input, which can lead to inconsistencies and errors. This is particularly problematic in coding, where precision is crucial.

Best Practices#

To mitigate the risks associated with AI coding assistants, it is essential to follow best practices:

  • Review AI-generated code: Always review the code generated by AI assistants. Look for logical errors, security vulnerabilities, and other potential issues.

  • Do not copy paste large blocks of code: Instead, use AI to generate small snippets or to help with specific tasks. This allows you to maintain control over the code and reduces the risk of introducing errors.

  • Make sure you always understand the code: If you do not understand what the AI-generated code does, it is likely that you will not be able to spot errors or issues. Take the time to read and understand the code before using it.

  • Test thoroughly: Run tests on the AI-generated code to ensure it works as expected.

Contents