The TechEmpower Benchmarks measures the maximal number of requests per second you can churn through a server. They don't care for the number of connections, but simply pick the best result from trying different connection counts.
Also, they don't really measure latency correct I think, because they make the mistake of "coordinated omission" (Look it up, Gil Tene, of Zing/Azul fame is its main discoverer).
I was very careful in my wording. You need 10.000 connections, each running 3 req/s for 30.000 req/s. So we are not trying to maximize the system. We are trying to saturate it over a long period with a constant load. And you need to use wrk2 (also from Gil Tene) because otherwise the load generator and the System-under-test ends up coordinating among themselves, hiding the fact that some of the 10.000 connections are starved to get more req/s through on a subset.
The Java frameworks are very very good at handling 512 connections and throwing as many requests through those as possible. What they fail at, is 10.000 connections. What usually happens in bad load generators is that they allow the Java code to starve, say 9500 of those connections while getting all the good numbers on the last 500. wrk2 doesn't allow that to happen, and starvation is counted against the SUT.
As for the source, you would have to wait. I have the raw data, but I have not yet written the blog post. I think I've given you all the information you need to carry out the test if you are so inclined here.
I deleted my comment because I realize the benchmarks don't cover a 5 minute test. It's unproven whether the Java implementations need to in fact GC over that time. I'm skeptical though -- they probably reuse all their objects, and I think it would be interesting to prove that.
Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.[1]
I bet if you published a test case that proves Java GC impacting the max latency over 5 minutes, that would get the attention of TechEmpower and make a case for extending the test past 15 seconds.
> It's unproven whether the Java implementations need to in fact GC over that time. I'm skeptical though -- they probably reuse all their objects
Are you saying that it's common for java web-frameworks to automatically reuse objects and that they do not cause any GC?
> Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.
Read jlouis comment once more. It's not that they don't try many connections at once, it's that they take the best result that they get. Big difference.
EDIT:
Looking at the code for the Java Netty/JSON test shows that it does not reuse objects. The biggest takeaway from these tests is that it's hard to do performance testing...
