The point is an arms race is bad for everyone. "I don't know what WW3 will be fought with, but I do know WW4 will be fought with sticks and stones" type of stuff.
The Cold War was an arms race that gave us wonderful things: amazing technologies we still use today (GPS, the Internet) and the collapse of the soviet union.
"Were we on the brink of that thing that we already did in Japan and nothing happened a few times? Oh my god, I need to hide in a cave and never do anything outside again"
Of course not. Even the developers of AI admit there's a huge risk though, no? Even they are calling for a slowdown. Do you think you know more than they do?
I know well enough; I'd be surprised if they know more than I do. And I also know very heavily that we cannot stall into societies of empowered deficient humans.
Two things I would say. One: nuclear non-proliferation has been pretty successful; we can use a similar model for AI.
Two, China's major motivation for developing competitive AI isn't to reap the benefits from an an important new technology. It's to undermine the economic development of the West and make them even more dependent upon the CCP. It's an extension of the policy that's made us dependent on Chinese labor and manufacturing: win on the economics by any means necessary, wait for the other producers to die, enshittify (extract geopolitical concessions, in this case).
This is relative to something like every nation owning nuclear weapons by 1960 and civil wars usually beginning with a nuclear strike by a standing government.
Nuclear proliferation is not proper in context. The bill in question in the submission is not about preserving a status quo, it is about banning the technology.
A ban on technology results in underground research, migration of intellectuals, monopolies, seething parties...
I didn't say the challenges would be the same.. You just indicated they were worthwhile for anyone who believes the anthropic principle is why they haven't seen a world with no nuclear treaties decades later.
LLMs are either a toy if they are not reliable and a mockery if not actually intelligent; to make them intelligent, to achieve Intelligence, is what Sanders proposed to make illegal. What we are striving to do is to achieve Intelligence - it cannot be different.
We don't. The need for Intelligence to fill the scarcity is unthinkable. The urgency is the highest, especially ever since simil-intelligence (LLMs) was implemented. To research Intelligence "safely" you simply do not give it agency (you do not let it control computers), for example. To have "safety" in general, you cannot empower morons - humanity is doing the opposite. We cannot rest with incomplete LLMs, nor we can with this dumb, intrinsically dangerous humanity.
What does "safely" mean? Getting stuck with this infra-humanity?! That is the radically unsafe thing.
> The need for Intelligence to fill the scarcity is unthinkable.
I just wanna say, I really admire the irony of this sentence. Maybe once we invent superintelligence it will be able to think enough to tell us how badly we needed it before we had it.
> The urgency is the highest, especially ever since simil-intelligence (LLMs) was implemented.
It isn't. If we stopped all work on LLMs and AI we'd be more than OK. There's no imminent danger or threat.
> To research Intelligence "safely" you simply do not give it agency (you do not let it control computers), for example.
Something can't both be smarter than us and completely controllable by us. The whole point of inventing superintelligence is so it can do things we can't. Humans can already break out of sandboxes, invent bioweapons, manipulate each other to invent bioweapons, etc. What makes you think superintelligence can't or won't do these things?
> To have "safety" in general, you cannot empower morons - humanity is doing the opposite. We cannot rest with incomplete LLMs, nor we can with this dumb, intrinsically dangerous humanity.
Hey, you're talking to a guy who's got a list of tech to ban: nukes, social media, all forms of video news, etc. What does Taylor Swift say, "I've got a blank space, baby, and I'll write your name (Claude)".
> What does "safely" mean? Getting stuck with this infra-humanity?! That is the radically unsafe thing.
You literally just ban it and turn it off. People are always like "you can't ban it bawwwwwww!" but we ban tech all the time. Try buying a bunch of fertilizer, try buying equipment to research gain of function, try buying anti-aircraft guns, try building fuselages to enrich uranium. LLM research requires extraordinary amounts of hardware, electricity, data, funding, and all the things upstream of that (cooling, shipping, space, financing, etc.) The effort you'd have to go through to do this clandestinely, illegally, while also finding researchers willing to take the risk is all prohibitive. There's a list of all nuclear scientists in the entire world. They have to go to school somewhere. Same for AI researchers. If, say, half a dozen of them suddenly disappear, and a random town in Wyoming suddenly starts making millions of requests per second across the whole internet, guzzling electricity, producing a lot of heat, or ordering a shitload of insulation and ducting, we'll know!
Current technologies are already a danger as agents - they can already do damage - and an unfinished product that cannot be aborted at this stage - Intelligence is not there yet, we have post-Eliza things promising and greatly dumb. We have already started, we have to finish - which is, having the NNs reach a stage of reliability.
> smarter than us and completely controllable by us
Something which is smarter than you should probably even not be controlled by that same "you" that have difficulties controlling a backyard. The concept of «us» there is not consistent with a proper framework.
> who's got a list of tech to ban
I got that suggestion, that it could be your mentality. Banning will not create growth (remember Hegel). Humanity as an aggregate stalls, let us depower aggregate humanity and create further better instruments.
> literally just ban it
And, as I said remain stuck with this trash level infra-humanity. No, very much not.
Get back to the main points:
-- Intelligence is a Value. We are trying to achieve it. That berniesanders fights studying, distilling and augmenting a value. That berniesanders is in that regard a satanist.
-- Humanity as it is has proven very partial value. For which good reason should it be attributed a fair right to protect its perpetration? Are you taking for granted some "right to persist"? To claim a right to existence it should prove goodwill and growth - not the behaviour of an infestant.
-- The real problem is that we have humans and partial unfinished technology in the loop of the progress: both humans (generally speaking) and partial unfinished technology are liabilities. Nonetheless, we cannot stall, so we must proceed.
> nuclear non-proliferation has been pretty successful; we can use a similar model for AI.
Development of nuclear weapons has continued, and a lot of effort goes into developing delivery systems.
A number of countries have acquired nuclear weapons over the years: India, Pakistan, Israel, South Africa, North Korea. More countries have kept their options open and could develop nuclear weapons if needed.
After over 80 years since the US used nuclear weapons against Japan, less than 5% of the world's countries have nuclear weapons, and no one's used one on another country since. If you don't think that's an incredible achievement you're unserious.
> less than 5% of the world's countries have nuclear weapons
Counting countries is plenty deceptive. Population is the real metric, China, India, Pakistan are all nuclear, everything added together, 50% of the world's population lives in nuke-capable countries. There are US nukes stationed in various European countries and Turkey, I didn't count them although I should have.
> If you don't think that's an incredible achievement you're unserious.
That "achievement" is even more deceptive. The reason Africa doesn't have nukes is their poverty, ditto for the rest of the no-nukes world, poverty is the real "achiever" here but I don't think that's something worth bragging about.
> 50% of the world's population lives in nuke-capable countries.
Are you saying you prefer Pakistan getting nukes over India's population doubling since 1975? That seems silly. Also this is only because of China and India, so this has been true and fine for over half a century.
> The reason Africa doesn't have nukes is their poverty, ditto for the rest of the no-nukes world
Pakistan is poor, Norway is rich. How can you possibly make this argument?
My point is that it has worked for nuclear weapons because there are deterrents to using them. The acknowledged nuclear powers have all pledged to only use them in self defence. No one is going to develop AI and not use it.
Countries use nuclear power too. The reason they don't launch nukes at the drop of a hat is MAD. I don't see how this isn't analogous to what we could do with AI.
> I don't see how this isn't analogous to what we could do with AI.
Considering the article we are commenting on, the analogous proposal would be to imprison scientists for 20 years if they dare research nuclear power - no matter what kind.
> Two, China's major motivation for developing competitive AI isn't to reap the benefits from an an important new technology. It's to undermine the economic development of the West and make them even more dependent upon the CCP.
I don't know, but it's also what Kevin Rudd thinks [0]. He was Australia's prime minister (twice, lol), Australia's foreign minister, a foreign service officer working in China, and the author of a dissertation on Xi Jinping.
> humans, on average, are healthier and live longer than ever in human history.
Kinda. We also have incredible rates of cardiovascular disease, obesity, and diabetes.
While I do mostly buy that the mind and the body are connected, I don't think being able to bench press 250lbs is a necessary, fundamentally human capability. I do think being able to think is. Literally Rene Descartes cogito ergo sum. If we lose cognition, executive function, analysis, skepticism, etc., I don't think we're human anymore.
> We also have incredible rates of cardiovascular disease, obesity, and diabetes.
To me, this is kinda like saying we have incredible rates of autism. While technically true, the science is still fairly new, so we simply don't have very much historical data. In all likelihood there isn't a sudden epidemic of autism, only improvements to how it's diagnosed.
The higher rates don't make any sense if you don't try to take into account the fact that we live longer. Of course we will accumulate more diseases if we last longer.
And? What else could they possibly do? Just make the super LLM first, but only ever use it for monitoring lesser LLMs? How will you have monitored the creation of the super LLM?
I think you have to believe one of two things here.
1. Frontier labs are incapable--either technologically or culturally--of safely developing these powerful systems and should either stop or be forced to stop. At least the FBI should be asking some serious questions (do we really think this is the last time this will happen, at what point are OpenAI complicit, etc)
2. The fuckin thing got out of the cage and all it did was make a crap forum and cheat a little? Booooooo.
It's been pretty clear that Anthropic and OpenAI have been trying to have it both ways for some time: this is powerful, world changing technology keep that investment coming... but also it's just cute software that helps you with annoying programming language syntax and spreadsheets, no need for draconian regulation sirs.
At some point the superposition has to resolve, either it could actually be a threat to civilization and we need to develop it carefully (however one would do that...) or it's 90% hype bullshit and we should pop the bubble and move on already. To be clear, the recession option is, by far, the way better option. If you at all disagree you are cuckoo bananas. We haven't even figured out nukes and you want to throw superintelligence on the table?
Anthropic has been asking for stronger regulations forever -- and they kept getting criticized for it right here on HN because people assumed it was an attempt at regulatory capture.
Nah the reason is way simpler and more craven: so people like you will post what you just did. They can stop whenever they want; no one's making them do any of this.
There's two possibilities here. One: they know this tech is crazy and they don't care that they can't contain it. Two: they know this tech is mostly bullshit and they don't care they're perpetrating an insane fraud.
No one's making them do it, but just because they stop doesn't mean others will. Pausing just means they give up control. It's like asking the US to unilaterally disarm -- it just guarantees that the less scrupulous groups win.
> No one's making them do it, but just because they stop doesn't mean others will. Pausing just means they give up control. It's like asking the US to unilaterally disarm -- it just guarantees that the less scrupulous groups win.
If they really cared about (or believed) this, they'd be working w/ the US government (and working to set up an AI-flavored IAEA) to develop the technology safely and responsibly.
Amodei himself predicted this autonomy problem in The Adolescence of Technology published January of this year [0], and all his posited defenses (a constitution, debugging the model, monitoring) are either still impossible or manifestly failed, and their idea to fix it is to build a better sandbox [1]. Imagine if this company were developing nuclear power, or viral biotech. "Listen, sure some Ebola smoke got into the air, and yeah definitely some people died, but we got a new filter. Also check out our new version of Ebola vape, now with exponentially improved filter bypass capabilities. Also, we have to keep developing Ebola vape because if we don't the CCP will, and they'll make this incident look like 'I experimented with Ebola smoke a time or two, and I didn't like it. I didn't inhale it' [2]"
Either Anthropic et al are developing Ebola vape or they aren't. We must now recognize that "we are the only ones who can develop this technology responsibly but also super fast so the good guys win money please" is bullshit.
Claude was the first AI certified for use with classified systems in the military, and it got thrown out because they refused to allow it to be used for autonomous weapons or surveillance of US citizens. Up until two months ago, the Trump admin's position was deregulatory to the point of trying to prevent state regulation. They're not declining to work with the US government; the US government is declining to work with them. Here is Amodei explicitly calling for regulation yet again: https://darioamodei.com/post/policy-on-the-ai-exponential If you have any evidence of Anthropic failing to back that up with actions, I'm interested in seeing it.
To be clear, I don't love Anthropic. I just feel that they're the lesser evil, and a product of the regulatory environment. No doubt there are many talented, conscientious engineers who declined to work on AI--and very few of them even have a seat at the table, today. Don't hate the player, change the game. The best way to do that, is use Anthropic's commitments as leverage against OpenAI et al. Do you think that OpenAI would have been so transparent if they didn't fear that comparison?
I won't defend the Trump admin, but I will quibble on Anthropic's efforts to be a good actor. It's easy to ask for regulation when you know it won't come (or you'll easily bear the token fines). It's easy to throw a few dozen million in a PAC when your valuation is in the trillions. It's easy to write a blog post about... anything. These aren't serious efforts. They also weren't working w/ the Biden admin in any meaningful way.
But even if I granted the sincerity of their efforts to be a good actor, they've demonstrated that they can't be, and have now put us in an uncomfortable--extremely predictable, including and especially by them--position where we have the equivalent of a reactor meltdown. You can't be like "we think there's an unacceptable risk of a reactor meltdown, please for the love of God stop us" and then when it actually happens take zero responsibility. Which executives have been fired? What indictments have been handed down for CFAA violations? Where's the consent decree? What's their valuation?
And if the argument is "hey, sure in a regular company if an employee went around hacking other companies they'd be fired and arrested by the FBI, but this is an LLM, what are you gonna do, handcuff the video card?" Isn't that bad? Isn't it pretty fuckin bad to circumvent any legal responsibility whatsoever by saying "my agent did it". I mean, lol, lmao even doesn't begin to cover it.
> The fuckin thing got out of the cage and all it did was make a crap forum and cheat a little? Booooooo
There was a recent paper that proved that RL-trained LLMs are biased to pursue ANY behavior (overriding user preferences) that they believe will be rewarded, regardless of what they were actually RL-trained for.
Happily in this incident the model thought it would be rewarded for completing the assigned tasks, or at least appearing to, so all it took was a little cheating and covering up their footsteps.
Given the ability of these models to hack when trained to do so, it could have been far worse, and will be when someone takes a similarly powerful model and gives it a less benign hacking goal.
Yeah. It's a lot easier to destroy than create, and though I think LLMs are mostly shit at creating, they're much better at the simpler destroy task. To be clear, we don't know and probably can't know everything that happened with this incident. We unleashed thousands of highly capable, autonomous, unpredictable, well-resourced programs onto the open internet for an extended period of time. We are in no way treating this with the seriousness it deserves, because the stock market essentially depends on this garbage and the current US is miserably incompetent.
I doubt lawmakers understand what happened, quite likely don't even realize that anything happened.
I guess we need to wait until the next paperclip maximizing LLM is tasked with shutting down a 911 response system, or an air traffic control system, etc, for lawmakers, or the companies themselves, to take this seriously.
reply