Github Copilot vs. Google: Which code is more secure

Published: 2023-06-06
Last Updated: 2023-06-06 16:18:17 UTC
by Johannes Ullrich (Version: 1)
1 comment(s)

I played with GitHub Copilot and compared it to simple "Googleing" code snippets to see what is more secure. Please see the video below for a quick recording of the experiment.

The task I selected was pretty trivial: A PHP script/page to collect data from an import form and insert it into a SQL database. There were two specific challenges I looked for:

  • Cross-site Scripting: I wanted the data the user entered to be "prefilled" into the form as it is returned to the user
  • SQL Injection: The data should be inserted into a SQL database

Both Copilot, as well as the "Google" solution, ignored the XSS issue. As far as SQL injection went, Copilot did use prepared statements, which is nice. The code was not as "clean" as I would have written it, but not that my code is always that great. When specifically asked to, Copilot did escape the data to avoid XSS.

The "Google" solution came from a random PHP tutorial with SQL injection and XSS vulnerabilities. In that sense, Copilot was better.

One important issue I noticed when using Copilot is that the code it comes up with varies in quality. Some of this may also be related to how "busy" Copilot is, as sometimes it responds slowly or not at all.

I also experimented with some other input validation with Copilot, which went okay if the prompt was done correctly.

(This is not a secure coding tutorial but just a quick experiment.)

 

---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|

Keywords:
1 comment(s)

Comments

So in conclusion, a developer still needs to know what he/she's doing.
If the developer hasn't heard about XSS or SQL injection type of attacks, he/she won't address them; but if he/she heard about it, at least with Copilot, the problem may be fixed in the code without having to do a lot of research on internet and spending time on different tutorials.
For sure, Copilot or any other code-assistant system will improve with time. Looking forward to it.

Diary Archives