> My point was not that algorithms are irrelevant but they are table stakes
You're wrong. A previous team built a simple data processing pipeline for a big customer. The throughput is single digit per second. The customer was furious as they would have wait for days to get their data in. The reason is the team was not aware that their graph building and traversal algorithms are O(n^2), and they didn't understand how to distribute their processing logic. My team came in, fixed the algorithms with merely dozens of lines, and improved the throughput by more than 200 times in a matter of days. If you build large backend for data intensive systems, which pretty much every tech company needs, you will need algorithms and data structures. Probabilistic data structures? Check. Manually optimizing popular machine learning algorithms? Check. Rolling out our own implementation of graphs, trees, vector similarity search algorithms and indices, parallelization algorithms, cache oblivious data structures? Check, check, and check. You don't have to use those techniques, but your value to your projects will increase exponentially if you push the boundary.
I said algorithms are table stakes (a poker term for something you need to know just to play at the table). To continue the analogy, sounds like whoever designed that system didn't have enough to buy into the table.
> If you build large backend for data intensive systems
You're wrong. A previous team built a simple data processing pipeline for a big customer. The throughput is single digit per second. The customer was furious as they would have wait for days to get their data in. The reason is the team was not aware that their graph building and traversal algorithms are O(n^2), and they didn't understand how to distribute their processing logic. My team came in, fixed the algorithms with merely dozens of lines, and improved the throughput by more than 200 times in a matter of days. If you build large backend for data intensive systems, which pretty much every tech company needs, you will need algorithms and data structures. Probabilistic data structures? Check. Manually optimizing popular machine learning algorithms? Check. Rolling out our own implementation of graphs, trees, vector similarity search algorithms and indices, parallelization algorithms, cache oblivious data structures? Check, check, and check. You don't have to use those techniques, but your value to your projects will increase exponentially if you push the boundary.