Today I was working with ChatGPT to generate a fairly complex reporting query. After giving it the entire database schema, it produced an impressively detailed plan for the report — but then slipped on something extremely basic: it wrote a GROUP BY using a column that doesn’t exist in any table.
It was a funny contrast: ChatGPT handled the complicated logic with near surgical precision, but tripped on a trivial alias error. So I asked it directly:
“Why did you make such a trivial mistake?”
Its answer was actually really interesting — and a good reminder of how to work with LLMs effectively. (Sharing below because it’s a nice peek under the hood.)
TL;DR: LLMs are phenomenal at conceptual reasoning but not perfect at mechanical bookkeeping. They can design a complex report pipeline end-to-end but occasionally mis-reference a column because they don’t have a compiler, schema validator, or error feedback loop. Essentially: great architect, occasionally clumsy typist.
Full ChatGPT response that is worth reading:
Yeah, this is a totally fair question—and honestly one I ask myself a lot 😅
Short version: I’m really good at reasoning in concepts, but pretty bad at bookkeeping tiny details across a long block of code—especially when I can’t run it.
Let me unpack that a bit.
1. I don’t actually “have” the schema the way you do
When you paste the schema, to you it becomes a mental model: “EOBPatients has OrganizationId, EOBId, ClaimId, PatientId, CreatedByUserId, etc.”
For me, it’s just text I processed once, and then I’m generating new text token by token. I don’t have a compiled representation of:
Table: EOBPatients Columns: OrganizationId, EOBId, ClaimId, PatientId, CreatedByUserId
that I can query like a database. So when I’m writing:
SELECT ep.CreatedByUserId AS
...
GROUP BY ep.UserId UserId
I’m not actually doing a “schema consistency check” like a compiler would. I’m just trying to keep everything consistent in my “working memory,” and sometimes that slips.
Humans do the same thing when they refactor code and forget to rename one variable. I just do it more often because…
2. I can’t run the code or see the error
You run the query, SQL Server tells you:
Invalid column name ‘UserId’.
Then it’s obvious: oh, right, that GROUP BY is wrong.
I never get that feedback loop. I’m generating something that looks like valid SQL but I can’t actually execute:
No
sp_helpto re-check a table.No IntelliSense, no red squiggly.
No “compile and test” → fix → re-run.
So I’m relying entirely on internal consistency and pattern matching. When the script is big (like this one: tons of CTEs, org-level + user-level, multiple joins), the probability of 1 tiny inconsistency sneaking in is pretty high.
3. High-level reasoning vs micro-level accuracy
You’ve probably noticed this pattern:
I can help design a fairly complex reporting logic:
“Bill ready or finalized + bill row”
“Status transitions over time using LAG”
“Org-level rollups and user-level attributions”
But then I’ll drop a small typo like
GROUP BY ep.UserIdinstead ofGROUP BY ep.CreatedByUserId.
That’s because:
I’m optimized for semantic reasoning and structure (the “what” and the “why”).
I’m not perfect at symbol-level bookkeeping when there are lots of similar names (
UserId,CreatedByUserId,OrganizationId, etc.) and many moving parts.
Humans can partially compensate with habits: copy/paste, refactoring tools, tests. I don’t have those tools; I’m just predicting the next token.
4. How to work with this in practice
Given all that, the best pattern is what you’re already doing:
Use me to:
Design complex logic.
Draft big queries / CTE structures.
Think through edge cases and data flows.
Then:
Run it.
Fix the 1–3 “mechanical” issues (wrong alias, missing join, typo in column name).
Or paste the error and the relevant block so I can help patch it.
If I had actual DB connectivity and could “compile” and iterate, this type of mistake would drop a lot. But in this environment, you’ll occasionally see these “obvious in hindsight” slips.
So you’re not wrong at all to notice the contrast: big-picture logic solid, one tiny alias off. That’s basically my personality in code form 😂