Both arguments might be correct. If you are dealing with a language that makes a strong distinction between the object and the memory layout of that object, you might be better off handling the serialization yourself. But Varnish is in C, and data on disk is mapped to memory then used directly as a struct.
Thus there are no small objects scattered around--- each 'object' is contiguous, and most likely on a single page. Prefetches happen automatically --- in Linux at least, disk caching and the VM are essentially synonymous. The benefit of using the VM directly with mmap() is that you have more control over the details and less overhead.
> in Linux at least, disk caching and the VM are essentially synonymous
Ah that could explain it. The people I know seem to be working on fs-level caches that operate at least in part on file granularity, so maybe they assume Linux/FBSD do fs-level stuff as well. They seem to try to do things like deciding whether to cache or not based in part on how big the file being read is, and what its historical usage patterns are.
Thus there are no small objects scattered around--- each 'object' is contiguous, and most likely on a single page. Prefetches happen automatically --- in Linux at least, disk caching and the VM are essentially synonymous. The benefit of using the VM directly with mmap() is that you have more control over the details and less overhead.