I agree with you generally, just an observation on coding specifically.
Have the models improved since Opus 4.x? I find the newer models are not better in my day job, maybe in one shotting mvp's and other tasks.
Not trying to argue your point, just intrested in the coding aspect, if the models were improving as fast as benchmarks I would expect capability improvements to be obvious, but talking to people and reading forums, it seems everyone has a different opinion.
Yes, benchmarks are gamed and only loosely indicative of real world performance.
Also yes, Fable is massively better than Opus. It requires significantly less instruction and specs and produces more directly mergeable code.
The improvement is obvious as soon as my Fable allotment runs out and I try to do something with Opus. Have you given the same (larger) task to Opus and Fable?
What you say is true, the comparison indeed doesn't hold.
But is it relevant? does it matter from a product perspective if LLMs are non-deterministic. You don't need to one shot the correct result, english is ambiguous and LLMs non-deterministic, but you can iterate.
If it's possible to iterate fast and cheap enough, even ambiguous language can produce the results you want, given enough iterations.
There are a lot of ifs and buts here, just a thought on the compiler argument.
I think it matters, because nowadays we don't look at assembly any more. I mean, I don't recall the last time I was tracking down a compiler bug but it's definitely been more than 15 years ago.
We do have to look at the LLMs' output, though, and, as you already pointed out, iterate to get the correct results. What this means is that the output must still be readable, must be analyzed by someone and I don't see it going away any time soon.
The problem is that the analysis is not cheap. Sometimes, with boilerplate, it is easy, but many times it is not and that's where we get only slight gains by using LLMs.
But from a broad market and product perspective, for most things you don't need to look at the code. If the product kinda does what it's supposed to.
For example, in my game projects I don't look at the CMakeLists anymore, or python scripts that move assets here and there, I can run my game and just see that it did what I expect it to do (renders assets etc).
Similar with frontend, I don't care that much what the code looks like anymore, mostly that the site looks and feels as I expect it, and the correct network calls are happening.
TLDR; I'm thinking there are levels to this, in some projects it matters, in others it doesn't, it's kinda two different things. Programming wasn't replaced, LLMs just brought a new paradigm of doing things on the side.
I'm just rambling at this point, my thoughts on this are not super clear, sorry for that :D
The trouble, as I see it, is that we still have to look at the code before we run it. Even if it is a one-shot script. And it's just because you never know if there isn't some "rm -rf ${undefined_variable}/" lurking somewhere in there. And if I have to check it then I would very much like it to be readable.
I mean, I do get your point, sometimes it does not matter. Sometimes we could just YOLO it. But... if that then causes a big problem, even if at only 1% of the time, then I don't want to risk it. But that may just be me. YMMV
Similar for me, I don't like the development for many reasons, but that's another discussion. I also can't deny the capabilities.
I use the tools with this "risk analysis":
- If performance doesn't improve I can just always switch back to whatever I've done for the past 10 years, so it's not really a risk to start exploring.
- If performance does improve, then I'm already familiar with it.
classic, now I can have AIs make my games for me while I still have to cook and wash the dishes! Maybe AI can soon start playing the games for me, can't wait.
Genuinely interested, which ones do you think have relevance?
If I read forums and talk to people IRL most have differing opinions what model is best. Yes, for me it's pretty clear Opus is better than earlier models, but it's at least not obvious to me that the later are significant improvements.
It can be subjective, at this stage of product availability.
Personally, I hate to be frustrated by gross intellectual faults, so I did some research in the past about the best benchmarks to assess pure (simulated, apparent) intelligence. (The quality of the found benchmarks may not reflect what the models seem to do in practice, so one's experience should be compared to the raw numbers out of the benchmarks.) Good ideas emerge in the field: it was proposed and discussed on these very pages that the LLM should be able to solve "murder mysteries", for example (alongside the pattern recognition problems in which IQ tests consist, etc.).
Moreover, the LLM shall not delirate. It is an intrinsic issue with the current architectures (they do not mirror the "Foundational theory of Knowledge", which requires confidence values and relations of foundation between notions), but it is a problem with more or less presence per model. Artificial Analysis has introduced a metric for that.
Moreover again, I want an output style that works well for the purpose - must not be a clashing style like "youngspeak" ("like, awsome") or "paternalistspeak" ("when a planet likes another very much they are attracted...") or "sycophantspeak" ("your question is so deep and interesting") or "wetspeak" ("you can do it, feel this not that")... So, for example, I very much preferred Kimi k2 to gpt-oss-120b. I doubt there are benchmarks for this - "seriousspeak", "maturespeak" - but there should be.
In game dev (weirdly enough) I don't think this is true.
I was thinking about this yesterday and realized that I don't even want to play only my own games (made by LLMs or me), I explicitly want to experience what other people have come up with, what stories they have invented.
And yes, might be it's all made by LLMs in the future, but that's another discussion. In "art" I think it still matters, I care about how it's made, not only that it exists.
EDIT: "And if you don't have a serious go to market plan with a latent user base you're SOL" ths is probably true though :D
Have the models improved since Opus 4.x? I find the newer models are not better in my day job, maybe in one shotting mvp's and other tasks.
Not trying to argue your point, just intrested in the coding aspect, if the models were improving as fast as benchmarks I would expect capability improvements to be obvious, but talking to people and reading forums, it seems everyone has a different opinion.
reply