􀀂􀀟􀀍􀀆 􀀂􀀗􀀌􀀉 (Type) Checking the LLM

Some environments are better suited for Copilot than others

I just had a real modern programming experience.

I copiloted1 an error into my JavaScript code, the editor didn’t catch it, but the code happened to work in spite of my error. When I converted the same code to TypeScript later, the editor surfaced a TS error indicating the error, but it still worked fine when I ignored it. I couldn’t figure out what TS wanted from either web searches or messages with ChatGPT – to my amusement arguing with ChatGPT about this for a few minutes – and I didn’t realize until I walked away and came back what had happened.

I’m working on updating a favorite project of mine. Today it shows a single layout (mine) for a single keyboard (the ErgoDox) on a single website (https://keymap.click). I want to allow anyone to use it to display their own layouts for an expandable list of keyboards on their own websites. I’m migrating from React and Tailwind to Web Components and vanilla CSS to make it work nicely in more environments, and to make it more maintainable with fewer dependencies, and as part of that I am converting it from JavaScript to TypeScript so that I can have more confidence in my changes. I’m ok at JavaScript development, but I haven’t used Web Components before at all, so both GitHub Copilot and ChatGPT have been really helpful.

Web Components are pretty cool. If you haven’t used them before, they are basically custom HTML elements you can define in JavaScript. There are two kinds of custom elements that you can write:

Autonomous Custom Elements Customized Built-In Elements
Inherit from HTMLElement, like class KeyGrid extends HTMLElement {...}. Inherit from the built-in element you want to customize such as <button>/HTMLButtonElement, like class KeyboardKey extends HTMLButtonElement {...}
Defined as a custom element name, like customElements.define("key-grid", KeyGrid); Defined as an extension to the customized element, like customElements.define("keyboard-key", KeyboardKey, {is: "button"});
Created in HTML by the custom name, such as <key-grid id="...">...</key-grid> Created in HTML by the name of the original element with an is attribute such as <button is="keyboard-key" id="...">...</button>
Selectable in CSS by the custom name, such as key-grid { color: black; } Selectable in CSS by the name of the original element with an is selector such as button[is="keyboard-key"] { color: black; }

This API has tripped me up a lot already, because I have Web Components of both types, and I often forget when I’m working with the type that requires referencing the built-in type and selecting my customization with is. (The great thing about abstractions is that they never leak!)

Anyway, I need to find all of a certain Web Component that is the child of some other element. Easy, that’s querySelectorAll(). I start typing into my editor:

parent.querySelectorAll

and Copilot gives me something I don’t expect:

parent.querySelectorAll("button", { is: "keyboard-key" });

Oh, that’s right, the element I’m working with is a customization of <button>, good catch there Copilot. To be safe, I run the code, and of course it works fine – I get an array of all the <button is="keyboard-key"> elements I’m expecting. I finish a pretty large change converting the main UI to Web Components from React, which relies on this working. I’m really excited when see the Web Components version mostly working at the end of my session.

24 hours pass.

I decide to convert the file I was working on the previous day to TypeScript. While I was working on it before, I wished several times I had already done the conversion for more assurance when making the change… TypeScript is really overdue. When I do that, it shows an error on the querySelectorAll() line:

Expected 1 arguments, but got 2.ts(2554)

Huh. I verify again that the code is working. I try other permutations, like:

parent.querySelectorAll("keyboard-key");

This doesn’t work at all – the query returns zero elements. I change it back to the old way, and I see a TypeScript error in the editor but the code works fine.

I search the web about this, and find nothing. I ask ChatGPT, which tells me that my working code is impossible, as querySelectorAll() does not accept more than one argument. I argue with it. “But when I do it that way it works.” It insists – maybe it’s some third party library functionality? I look it up. The computer is correct: querySelectorAll() only accepts a single argument.

I have to walk away for a minute before I realize what happened:

  1. JavaScript ignores extra function arguments, and therefore
  2. querySelectorAll("button", { is: "keyboard-key" }) is the same as querySelectorAll("button"), and
  3. it was returning all <button> descendents, and my parent happened to only have <button>s that were keyboard-key Web Components as descendents.

The editor tooling that I had after converting the file to TypeScript caught the bug, while the editor tooling for vanilla JavaScript didn’t.

I’m not suggesting that TypeScript, or strong types, are required to catch this kind of bug, though I do happen to like TypeScript more than JavaScript. Sometimes when I see people arguing about strong typing vs dynamic languages, proponents of dynamism say that sufficient tests and tooling should make satisfying a type checker wasted effort. This may be true, but at least the JavaScript support that ships with VS Code is not advanced enough for that yet.

What I want to note is that checking the LLM-generated code with something found this bug. Good correctness checks will speed up LLM-assisted development.

(Also, self, don’t argue with the computer in the future, it’s unbecoming to both you and the machine.)


For anyone curious, the way to get only the keyboard-key elements I wanted is to find all <button> elements and then filter them, like

const keyboardKeys = Array.from(keyboard.querySelectorAll('button'))
  .filter(b => b.getAttribute("is") === "keyboard-key");

  1. I like this way of indicating use of LLM assistants; it conveys what happened (bad code generation) and who is responsible (me). ↩︎

Responses

Comments are hosted on this site and powered by Remark42 (thanks!).

Webmentions are hosted on remote sites and syndicated via Webmention.io (thanks!).