Every analysis of the problem I've seen has concluded that the main problem is that rustc generates a lot of input to LLVM. Efforts to reduce compilation times are currently focused on doing more to the IR before converting it to LLVM IR.
Rust as a language is even more reliant on monomorphization and inlining than C++, due to core language traits such as Deref, AsRef, From/Into, and so on.
That said, in practice my personal experience has been that Rust compares pretty favorably on compilation speed, even to some high-level languages like C#. On many developer machines, the actual slow part is linking.
This analysis is great, and I don't mean to knock it, but one of the things that it doesn't capture is, what choices does the frontend make that impact the amount of work that the backend does. That is, some of the work attributed to the backend could be improved without touching the backend.
That is of course true, but if anything that just proves more what I said. Also that too is part of the "common wisdom" (that as I said, is not always true).
Over at Feldera, we focus on IVM for SQL, but incremental computing problems show up far and wide: UIs, spreadsheets, control planes, compilers and more.
When I last used it for such use cases, it was better to decompose the problem into something incremental (so fixed placements become constants). Most of the latencies we saw were spent in the presolve phase which scaled with overall input size.
There are some solid ideas here and would definitely apply to the IVM engine we're building. I'm curious if some of these effects could play a role in faster rust compilation times (e.g. nopanic..)?
Some folks pointed out that no one should design a SQL schema like this and I agree. We deal with large enterprise customers, and don't control the schemas that come our way. Trust me, we often ask customers if they have any leeway with changing their SQL and their hands are often tied. We're a query engine, so have to be able to ingest data from existing data sources (warehouse, lakehouse, kafka, etc.), so we have to be able to work with existing schemas.
So what then follows is a big part of the value we add: which is, take your hideous SQL schema and queries, warts and all, run it on Feldera, and you'll get fully incremental execution at low latency and low cost.
700 isn't even the worst number that's come our way. A hyperscale prospect asked about supporting 4000 column schemas. I don't know what's in that table either. :)
This site is underweighted on OLAP. Columnstores were invented for precisely this use case; nobody in the field wants to normalize everything.
Which brings me to the question, why a rowstore? Are Z-sets hard to manage otherwise?
Another aspect of wide tables is that they tend to have a lot of dependencies, ie different columns come from different aggregations, and the whole table gets held up if one of them is late. IVM seems like a good solution for that problem.
Feldera tries to be row- and column-oriented in the respective parts that matter. E.g. our LSM trees only store the set of columns that are needed, and we need to be able to pick up individual rows from within those columns for the different operators.
I don't think we've converged on the best design yet here though. We're constantly experimenting with different layouts to see what performs best based on customer workloads.
