While I'm here - I recently optimized our PR review skill to make better use of the KV cache.
Previously, it loaded up the diff, persona and prompt, and wrote those into the first message for each of 6 sub-agent reviewers. The prompt was templated with the persona name, so was slightly different for each reviewer.
My optimized version had a common first message with diff + prompt, and a script to run to atomically claim a persona. It also runs the first reviewer before the rest to warm the KV cache, and the agent doesn't launch the rest of the subagents until the first persona has been claimed (which means that the LLM is running, and therefore the KV cache is warm). The agent has a script it runs which does the waiting for it.
Agents 2-6 only run once the KV cache is warm, and because the intro message including the diff is shared, it's there in the cache. Only the persona file is different.
In my testing this brought down the cost of a review by ~half, though of course this depends on how big the diff is, how many agent are launched, and what model is used.
The layout, as well as all of the paragraphs are written by me manually, although I generally draft a completely broken English one very rapidly and ask it to paraphrase, I do apologize if the editorial pass made it harder to read, but it really isn't an LLM's opinion :)
I thought KV cache not being as important was obvious, as a big part of the benchmrks were focused on pricing, but in retrospect I can see how I could've made that more clear.
In the end cost is what matters most and as seen from the results the KV cache miss on the cheap model doesn't really matter as much.
What is there to talk about the KV Cache, they’re handing off to a different model, I thought that you can’t reuse KV cache between entirely different models?
Is it on topic to complain about the various claude-isms in this article? I don't know any actual humans that write titles like "Two floors the rate card hides".
I find my brain disengages once I suspect something of being written by an LLM. If the author didn't put much effort into writing it, should I expect them to have put much effort into fact-checking it?
Edit: this specific title has been deleted from the article. That was not my point! Please put in more effort into writing things that you want others to read! Rather than putting in low effort but being better at hiding it.
Yeah this is an extremely poorly written article. They didn’t even bother to add a “rewrite this article to make it sound less than AI”.
It’s also using a bazillion words to make a point that could be summed up in a single paragraph: there’s a huge variance in the number of tokens required to encode the same content, with code leading the charts.
To be fair, most of this was already known, and Anthropic communicated very clearly about the different tokenizer they started using.
Their compute is also mostly 1:1 correlated to the number of tokens, so I don’t believe in the conspiracy that this is just to inflate prices.
Yes it's worth commenting about. Not everyone, but many want to know that signal just like you do. It not only provides a useful heuristic about the article, but about whatever product or service they're advertising/selling.
A problem is AI by default is not very good at anything. It’s pretty mediocre. With a good harness and a lot of prompting/context - you can get it to spit something out that’s pretty good. Coders have been learning and fighting this fight for a couple of years now.
The issue is that it’s not just code - they suck at writing. Really bad. Unreadable, incoherent, messy.
Humans are also bad at judging the quality of things they themselves aren’t very good at. So a senior swe sees what claude spits out and says “This is trash.” And spends x amount of time getting it to not be trash. And Jr dev thinks “this is magic!” And pushes it to a PR.
So my theory is the people “writing” this AI slop think its great! But actually just aren’t very good at writing copy and don’t have the skill to recognize it and prompt their way out of it.
Or they don’t care. That’s an option as well.
PS for anyone reading, next time AI does something that you aren’t super familiar with that looks pretty good… maybe find an expert to review it.
Well, criticizing is, of course, great. But the reality is that English is not my native language and I dictated most of it with my voice, then processed it with the help of AI, translated, added, corrected, and converted.
It is actually a big result of work, a lot of research and attempts. And to just say that "oh, this is AI-slop," I consider unfair, but that is your choice.
There is a difference:
- There are people who do,
- And there are those who criticize.
Instead of getting offended by a fair criticism you should learn from it. In your articles consider adding a disclaimer that says exactly what you just said here in your comment here that you post-processed your voice and thoughts through LLM.
LLM speak is like the new corporate speak. Enterprise writing is fulll of fluff and nothings and they all read the same. That sameness is what most readers here are sick of.
(Your comment here that I replied to is also written by AI which is even more sad :| )
Bro, this was the only reason I opened the comments, and I'm so happy to see someone else noticed. Even if they removed that, the utter sloppiness of the prose is unbelievable. Offensive, even.
Thank you for the criticism. I heard you. I added a TLDR. I cleaned up many AI constructions. By the way, I tweaked it a bit, compressed it.
One way or another, I want to note that yes, this text was made in collaboration with AI. My English is non-native. It helps me translate, helps me structure better. Yes, there is a downside, it can bloat the text with unnecessary words. But that, unfortunately, is the price.
But the key thing is that I tried very hard to share my many years of experience, or rather a part of it, which I acquired, with all of you. And I am very glad that this information turned out to be useful to you.
The key here is:
* The information that is written in the article.
* Not how it is written, but what I was trying to convey to you.
A family member stayed in a hotel in Dubai recently and on returning said how incredible the staff were and willing they were to help them, and my response was "no shit"
I went once. It feels like Disney Land but for the wealthy. I'll never go back, and I actually regret going to be honest. It's disgusting and it's right out in the open.
Has it? This is from that repo, "PostHog FOSS is a read-only mirror of PostHog, with all proprietary code removed." Seems they have some proprietary code in the non-open-source version.
Nothing has changed, just OP found the mirror of the main repo with the ee/ folder removed. That repo has been up for most of the lifetime of the company.
That's valid criticism, I kinda hand-waved the "months" part. I read everything I could about parsers while building this (I have a CS background but hadn't thought about parsers in a long time) and came across this blog post https://lakesail.com/blog/sql-parser-in-one-week/ which talked about building a toy parser in a week, so I scaled that up to months for a production one.
Yeah, one of the interesting parts to me while working on this is that the breakpoint for when it's worth writing your own parser vs accepting ANTLR's slowness has shifted massively. Previously it would have been someone's full-time job to maintain. Now with this approach you can get the best of both worlds.
In what way? This was a geometric mean of the improvements from a small test corpus. In production, where it only parses longer SQL that didn't hit the parser cache, the mean parse time went down by 454x, across millions of parses.
Ha I did consider that! But 70x is plenty fast enough (we still have to query an actual database!) and the parser runs in a shared process on untrusted input, so it wasn't worth the security risk
If we don't care about inspecting the output, why don't we just replace the Rust compiler with something that generates assembly from Rust source, and then cross check the output with a few million random Rust programs?
I'm begging people to take the time to write things themselves rather than getting Claude to write for them.
If you want human effort from readers, please put in human effort while writing.