Interesting test, but I find some of these benchmarks kind of miss the point. Even Grafana's.
The appeal of Thanos/Cortex/Mimir is the long term object storage. The value isn't that it is simpler or cheaper to run. The value is that I can compare data to months if not years ago. It can cost much more than the price of the instances to store good metrics over time even when the data is rolled up.
Scaling the read/write path separately has a lot of benefits as well too, but I would guess that doesn't come up often for most folks.
How much telemetry you can get in/out of your system over a day is important, but how much you can get in/out of it over years is overlooked.
Both VictoriaMetrics and Grafana Mimir perfectly fit for long-term storage for Prometheus data. The difference is in the used data storage types - VictoriaMetrics stores data to persistent disks (aka block storage), while Grafana Mimir stores data to S3-like object storage. Both storage types - block storage and object storage - can be used for long-term storage. They have the following differences in the context of major cloud providers (AWS, GCP, Azure):
- Object storage space usually costs 2x-8x less than block storage space.
- Object storage has up to 100x highest latency for data access than block storage (hundreds of milliseconds for object storage vs milliseconds for block storage).
- Block storage usually has much lower network-related error rate comparing to object storagr. For example, it is quite common practice to retry reading data from object storage on network errors, while block storage-based filesystems are much more reliable for this aspect in major cloud providers.
- Cloud providers tend to charge every read operation for object storage, while reading from block storage is free. This point is usually overlooked when estimating costs for block storage vs object storage.
Given these differences, block storage usually provides better performance than object storage. Block storage also can cost less than object storage when the stored data is read frequently.
VictoriaMetrics is optimized for HDD-based block storage, so there is no need to use more expensive SSD-based block storage in most cases. Additionally, VictoriaMetrics compresses production metrics 2x-10x better than Prometheus-like solutions, which store data to object storage (Thanos, Cortex, Grafana Mimir). This also reduces long-term storage costs.
On top of this, enterprise version of VictoriaMetrics can be configured to downsample historical data, so it will take less disk space [1].
To be fair, the benefits of "object store" it is scalable and bottomless while you have to play with EBS volume expansion etc. Some folks find managing fleet of EBS volumes not a big deal others find it problematic.
I think having "long term storage" on S3 compatible location is a way to go but you need ability to use local storage as cache to queries on recent data or just date range you're working with can be fast.
Agreed with this. That's why we at VictoriaMetrics are investigating a hybrid storage scheme - to store recently added data at block storage, while gradually moving older data from block storage to object storage in background. On the query side, the requested data should be transparently queried from both object storage and block storage.
> The value is that I can compare data to months if not years ago.
I'd say it is quite specific case. The most important data is recent data. Monitoring system should help you to identify current issues. Be reliable and performant, so you don't spend minutes waiting for response while your production is on fire.
Second in importance is data for last N days. The period when you analyze recent changes (updates, releases) or incidents. You want this data to be easy to get and pivot, changing queries ad-hoc and get results immediately. So root cause analysis won't take days of work.
Data older than month is rarely accessed. It is usually used for capacity planning, retrospective analysis - things which you do once in 3 months, or even once a year. Here, you can afford long, slow queries.
Both, VictoriaMetrics and Mimir, do a lot to provide fast access to the recent data: to get it stored and to get it ready for queries.
In my experience, the business use of retaining years-old metrics is overvalued, because changes in the system tend to prevent any kind of long-term apples-to-apples comparison. You’ll have changed your metrics engine, or your collector, or your tagging strategy, or your hosting strategy, or your containerization, or your deployments, or etc. etc. Even if you can find the like metrics from last year, you can’t trust that they meant the same thing then that they do now.
I wouldn't go so far as to say there wasn't much purpose for the ground troops in the context of the goal of eliminating Iraqi military capabilities. It involved some of the largest tank battles in US history such as the Battle of Norfolk and the Battle of the 73 Easting. While we look back at some of these as a steam roll in hindsight, the truth is the engagements were larger than most of the armor battles in WW2 involving the US.
The marine advancement in to Kuwait that was part of a larger multi-national push that much of the Iraqi forces were running from. The push by 1st Inf Div, 2nd ACR, and others on the west was to cut off that retreat.
Combat theaters are large and it is never a good idea to take a narrow view of them and generalize it to the whole. The father of my childhood friend used to tell us how his deployment in Vietnam as an MP was some of the best time in his life. He spent his entire time on a post relatively far from the fighting largely killing his time trying to surf and eat BBQ.
Casualties in warfare is typically used to measure the lose of the fighting force. Not only does it include killed and wounded, it typically also includes troops lost to disease and starvation.
It is pretty common for people to misinterpret casualties as deaths. Especially when people who are not familiar with the terms try to repeat the information to others.
There is also the common practice of not including discounts in revenue. There is a reason companies talk about revenue instead of net sales or net income.
I've never seen AWS give discounts comparable to what Azure provided. Especially to larger companies they want the brand recognition from.
Hundreds of developers were moved from Azure teams to the GitHub org about a year or so ago. Several new features they have added are effectively rebranding/built on top of other Azure projects.
Let us be straight: they effectively made the Azure DevOps (the leftover from the once might Team Foundation Server) a weak product to further foster GitHub. So when this team brought some tech over, that just means, they are now working for GitHub primarily and no longer on Azure DevOps.
Now I'm REALLY worried. Bringing in a large number of people like that at MS means they brought in A TON of managers, leads, and program managers. At MS these categories of folks get ahead primarily through political warfare, "networking", and stabbing each other in the back, and given large enough critical mass, that's what they'll continue to do, unless organizationally isolated from the rest of GH. Or at least that's how it was a little over a decade ago.
If you think the game was a disappointment, doesn't that imply you already gave them the money? It is not a service. If you bought it 5 years ago, you don't have to pay more to play it.
But you can avoid having some information brought to light specifically because of their abuse from turning you into a DAU. If their userbase goes up as a result of this scandal it's the market responding in a manner that encourages further behavior of this sort.
CNN and 24hr cable news has not been good for journalism. Opinion pundits responding to events before the facts can even be collected along with daily shows interweaving narratives pre-dates Fox News. Fox News played their game more blatantly ruthless than the others and beat them at their own game.
It seems to me that CNN accidentally stumbled on to this problem, while when Fox News came along they cynically embraced it as a revenue generator. For CNN it began as a bug, Fox made it a feature.
Burke Libbey has a few talks that are available on YouTube including a presentation at NixCon about the work he is doing at Shopify. I would highly recommend checking them out if you are interested in Nix.
Can you provide more information about setting a build infrastructure in an organization. Particularly how to set up a shared cache so if one user builds something it is available to others?
Yes, we’re hiring! You may have seen the news today that we’re all in on remote now. Send an email to burke.libbey@shopify.com, I’m as good a point of contact as any for Nix stuff.
It is pretty easy to be confused. Especially since it is also a language yet all the other replies at the time have writing have failed to mention that.
While I like the project as a whole, their naming is horrible. Just wait until you get into `nix-env` vs `nix-shell` etc...
> It is pretty easy to be confused. Especially since it is also a language yet all the other replies at the time have writing have failed to mention that.
The OP mentioned it. It helps to read the OP if one wants to avoid confusion.
Thanks, yeah. Conceptually I'm having a hard time wrapping my head around it since package managers are so tightly tied to an OS... I feel like surely I can't just use it in place of (say) apt or pacman without causing problems? And I haven't even gotten to the language/shell/etc. as you mention...
You're right that using 2 traditional package managers in the same environment will cause problems almost immediately, since both will want to control the same resources.
This is less so with Nix, as it mostly operates on its own folder called nix-store, the binaries in which are then symbolically linked to. It is actually designed to be able to be used this way. I know people are using it on Mac OS and I intend to install more and more packages through it to later possibly transition to NixOS.
You use the language to define the configuration, packages, and etc. NixOS isn't so much an OS configured by a package manager as much as it is an OS + package manager that you can configure with the Nix language.
Then there is also lot of tooling to hid all of the symlink magic. It is very deep rabbit hole. Most of the quick touch examples will basically look like magic.
Package managers are not tied to the OS. Essentially your fundamental misunderstanding here, from reading your comments, is that you are expecting a package manager on Ubuntu to manage "Ubuntu's packages", when the packages belong to a repository as accessed by a package manager, and "Ubuntu" doesn't really exist: "Ubuntu bionic" is merely "a bunch of deb packages that you can install from a particular repository using apt", while "CentOS 7" is merely "a bunch of rpm packages that you can install from a particular repository using yum".
You can add Ubuntu eoan or even Debian stretch repositories to your APT configuration on Ubuntu bionic, and install those packages... now, the naming and compatibility of dependencies might be different as you get further and further from what you were "supposed" to be using, but they will download and install (at worst an installation script will be missing a dependency and the package will be left "half installed", which is a technical APT term).
You can also install rpm on Ubuntu: it is even in the Ubuntu bionic repository. Now, what happens if you install an rpm file using it? It might very well overwrite something from a package you installed by apt, and neither might notice that happened. But you can do it. You can install yum (this will require manual install as I don't think yum is available from any common Ubuntu repositories) and then give it a CentOS 7 repository and install all the packages and what you will be left with will be approximately CentOS 7 (with some detritus from Ubuntu).
So on any system apt is managing deb packages and yum is managing rpm packages and where those packages come from is just some configuration, so you can install apt on CentOS 7 and use it to install packages from Debian stretch. You can't use apt on CentOS to manage CentOS packages as apt doesn't do that (though maybe you could teach it an "rpm method" or something but rpm's version numbering and dependency management aren't really the same so this is now just a confusing digression to avoid "technically you could" responses: yes, but then you are coding that yourself).
The only real exception to all of this is the kernel, but not even always as the kernel is kind of a file, but the way it is configured in the master boot record could be different... but like, you might have some distribution local kernel patches that their tools rely on for some reason, and upgrading libc on a running system is fraught with peril even when you aren't doing something insane this, so like: don't do this, but know it works.
I routinely thereby create a little folder on a Debian/Ubuntu box in which I can chroot to install a little world of an entirely different set of packages from a different package manager maintained by a different vendor as provided for some random specific version of their distribution. (Alternatively, rpm supports an argument to set the system root.) In that world there are no conflicts, as you made a folder for it.
When you install packages from apt or yum, usually they assume they could go in /. That is going to maximize conflicts unless you do the chroot. Nix has chosen to package their stuff not like that, and so you can install their packages on any other environment without it conflicting or overwriting.
So no, you don't "use it in place of apt" to manage Ubuntu packages, but maybe Nic and Ubuntu have chosen to both package the same thing you want (this is hopefully common as both are hopefully quite complete) and so you might "use it in place of apt" because you don't want to install an Ubuntu package, you want to install a Nix package instead. And as Nix packages go to special folders automatically, it doesn't damage your Ubuntu install.
I like how it pretty much just glosses over decades of familiarity. If there is anything the last 20 years prove, it is that the majority of developers will stick with what they know over what might be a good fit for the job. It goes even deeper in the SQL world down to the specific database flavor.
From the ops side I actually find RDBMS more difficult to deal with cause the power of relationships is easy to abuse and they are not anti-fragile. Instead of smartly reasoning about the data, it is all to easy to just "JOIN ALL THE THINGS WITH MEGA TEMP TABLES!". I've taken more database outages from bad queries then anything else.
There are bad implementations on both sides. There are reasons to pick both sides over the other given a set a circumstances. At the end of the day, the technical facts don't matter to most people's decision making though.
> Instead of smartly reasoning about the data, it is all to easy to just "JOIN ALL THE THINGS WITH MEGA TEMP TABLES!".
This is so common, I'd love for the popular databases to add table flags that prevent it by accident. Letting me configure "this table most not full scan or file sort implicitly" would get rid of half the incident callouts I've been involved in. You could always override it in the query where needed.
At best (if your users aren’t allowed to write SQL), that would change your “it is slow” calls to “it doesn’t work”.
At worst, I fear your users would learn to override it by default, as just one other part of the magic incantation needed to please the SQL gods.
I think it would be better to have the planner send out emails “this query has to use a full scan” or, “this query is on the brink of changing strategy compared to earlier runs”
> that would change your “it is slow” calls to “it doesn’t work”.
But that's exactly what I'm asking for. "It's slow" means that it will work until the breaking point and then wake me up. "It doesn't work" with the right reporting allows me to teach someone about indexing during office hours.
I found it not very useful. Sometimes you will query without indexes. Sometimes it just doesn't matter. Sometimes it's a table with 2 rows and you don't want an index.
In practice I don't want to know when it happens. I want an error to be raised instead so the database doesn't suddenly die.
> it is that the majority of developers will stick with what they know over what might be a good fit for the job
Anecdotally, I have seen this shift quite a bit in the last decade. There has been a big move toward "try whatever is new and shiny" and I think a lot of that drove the NoSQL craze.
And don't get me wrong, there are some great NoSQL options if you select the right tool for the right job.
But a lot of people who gravitated to NoSQL did so because they designed poor queries in RDBMS. Guilty myself. Often the quick reaction was "wow, this is faster" instead of "well of course it's faster, I'm not getting X, Y and Z features of a relational database. Do I need those? Did I abuse the RDBMS?" etc.
The appeal of Thanos/Cortex/Mimir is the long term object storage. The value isn't that it is simpler or cheaper to run. The value is that I can compare data to months if not years ago. It can cost much more than the price of the instances to store good metrics over time even when the data is rolled up.
Scaling the read/write path separately has a lot of benefits as well too, but I would guess that doesn't come up often for most folks.
How much telemetry you can get in/out of your system over a day is important, but how much you can get in/out of it over years is overlooked.