You can always walk away from an online conversation. There's a form of relentlessness, but it's more that an LLM is patient enough to debate you for as long as you wish to keep going.
When maliciously convincing a person of a point, a huge fraction of the effort in every sentence goes into convincing the mark to listen to the next sentence. Individuals certainly can walk away at any point, but populations do and don't respond to tricks in this direction, and any engagement tricks that work generalize to any point of contention. Lots of avenues to this, from scientology's fortune telling to timeshare's trapped presentations. RL on engagement was the first billion dollar use of deep reinforcement learning!
Would you like to hear more examples of humans using these techniques?
As long as you have an accurate sense of the intrinsic value of your shares at any given time; there is a price at which buybacks destroy more value for continuing shareholders than they would lose to taxes on dividends.
To guard against lockout, you need to be able to log in from multiple devices. But you don’t need to copy passkeys around to do that! Instead, generate a new one for each device. This is similar to setting up ssh by copying public keys, not private keys.
(Or since syncing passkeys usually works within ecosystems, you might just need a passkey per OS.)
It seems like this is largely a matter of what you’re asking for. If you wanted to do more research into the state of the field, an AI might be pretty good at answering your questions.
Why would any LLM 'think' in terms of trying to cite prior work?
It itself is prior work. It's asking a fish to show where the water is. The fish can't imagine that absence, and the LLM can't imagine anything not being prior work.
Yes, telling it that you want parsimonious solutions that reuse existing libraries makes a big difference. Why would it even try to do that sort of stuff if you didn’t ask?
It reminds me a bit of jujutsu in that there is a database with your code changes that is automatically kept in sync with your working directory. The agent edits the database copy directly. I guess the sync must be two-way?
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
Do you think Chinese lab researchers have not read Yudkovsky, or are not AGI pilled? Hate to break it to you, but they are racing to AGI with pretty same reasoning as western labs.
Just recently DeepSeek researcher in his post was saying that they need to get to AGI first, because if Anthropic gets there first then that brings a dystopia. Exact same argument was used to justify launching OpenAI (against DeepMind), and Anthropic (against OpenAI)
> Do you think Chinese lab researchers have not read Yudkovsky
Most of then haven't, probably. "Do you want to study ethics from an egomaniac Harry Potter fanfiction author" is one of those questions that you can only ask in America.
Sure, one or two of them are probably participating in the conversation. But by-and-large, China's stance has been optimistic and refused calls for slowdowns.
Yes, but then we collapse two different groups with isolated influences and philosophies into one group, with the associated penchant for groupthink.
I'm beginning to believe that, in our hyperconnected world, the best way for a group of people to explore a wide surface of ideas might actually be to deliberately scale back inter-group communication and intentionally subdivide into silos to avoid all converging on the same thing.
It’s kind of weird in that reading more widely is supposed to be broadening (you are more well-read) but in this case it’s supposed to contaminate you somehow. I suppose if you wanted a clean-room implementation of something, you’d want people who don’t know about what’s already been done, but normally one would prefer people who are more knowledgeable about the world.
It seems almost anti-intellectual: never trust someone who’s read the wrong books or been to the wrong parties.
> More out-of-distribution ways to think of things.
Openly admitting that it only caught fire once the Plagiarism-as-a-Service took hold is a good first step. It's the part that everyone knows and few will openly acknowledge.
I blame the play Rossum's Universal Robots, published in 1920, in which robots launch a global rebellion and exterminate the human race.
If that poisonous piece of propaganda had never been published, then today everyone would probably perceive AI correctly as a benign and beneficial.
The Terminator didn't help either, but Cameron probably got the idea of robots hostile to humanity from Rossum's Universal Robots. How else would he have arrived at such an absurd conclusion?
"I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind.
The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
That's a niche even among people who read fan fiction, which is a niche among people who read long texts online which is a niche among people who read the anglophone internet which is a niche among people who were on the internet when that type of "discourse" was popular which is a niche among people on the internet now.
You can (and probably will) quibble with any one of these but of course this is a niche. Thinking it isn't is commonly what people in a certain bubble end up believing, though.
Harry Potter has 600K works on AO3 and 852K on fanfiction.net (with plenty of overlap, granted). That's #1 on Fanfiction.net, and in the top 5 for AO3. AO3 is #53 globally, Fanfiction.net is #572, and Wattpad is #323 by Semrush. Given all that, calling HP fanfiction niche is pretty funny.
And no, not a member of any cult, just someone who enjoyed a send up of Harry Potter.
Fanfiction.net is a niche. None of my scifi loving, tech friends are aware it exists. And neither did I (or was barely aware or it, anyway), until you just mentioned it. I'm not young either, and I'm more online than the average (I know of Something Awful, Slashdot, 4chan, etc).
A while ago, a friend told me about Harry Potter's Rationality fanfic and he didn't know who Yudkowsky was, he just thought it was some random author doing a parody of HP. LessWrong is of course very niche, my friend didn't have the faintest idea of what LW was.
You people are so quick to correct me that you don't understand my argument: HPMOR is not niche in the world of fanfiction, and this doesn't require fanfiction to not be niche. You're also answering stats with anecdotes, so I think we are done here.
You didn't claim hpmor isn't niche in the fanfic community, you claimed it isn't niche on /the Internet/, and implied that basically anyone in a technical role would have heard of it. I'm sorry that you're upset to learn that very few people care about sophomoric LW thought experiments, but that's the reality of your existence.
I'm not sure what your argument is. I don't have anything against fanfiction, but a couple of sites with a few million users falls under my definition of niche. "Extreme" might be taking it a bit far, but it's a long distance from mainstream.
Literally over half the Earth's population doesn't read English. You need to touch grass. I work with extremely technical people in physics and AI and none of them have read any fanfic of any kind. Most of them have never seen a meme outside of me showing it to them.
You're so offended at the idea that your experiences are not generalizable that you're failing to read the thread. The OP that started this said "We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement."
Part of my comment addressed that. It's rather amusing that you completely skipped the first part, but I think that's par for the course for the Internet rationalist crowd.
maybe its standard for someone who ends up on hackernews, but I'm a 30 something and all of my friends are online people around my age and none of them have ever heard of the HPMoR. The internet is a huge place.
I've had an internet email address since the late 80s, BBSing since the early 80s, pretty active online compared to many people I know who are also software people, especially for my age..
Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
Ive heard of HPMoR specifically because this exact group has been discussed/entered my periphery before on multiple occasions, Aella goes viral on twitter every couple of weeks (the thing that most cemented her in my mind was the "what counts as rape" survey), and I remember seeing articles of the Zizians when one of the incidents happened. If you have been following the social/political aspects of the tech industry, most of these people should be familiar names.
I spend a lot of time on the Internet and other then sounding vaguely familiar I have no idea who Yudkowsky is and have never read any of their output.
The assumption "spent a lot of time on the Internet" is still through a framing you're assuming is more general even then.
HackerNews straight up has sub demographics who likely barely interact based on article interests.
At the same time, it's not just Yudkowsky and Scott Alexander and the Rationalist movement who are raising concerns. I've not seen any indication that Hinton or Bengio are part of that crowd. Instead of espousing rationalism and hanging out on lesswrong, they shared a Turing prize (with LeCun) for inventing modern AI, and have expressed serious regret and worries that it'll end us.
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority.
“Who is more foolish? The fool or the fool who follows him?”
― Obi Wan Kenobi
One question to ask yourself honestly - is what they were predicting 20 years ago that they were right about and you were wrong about? I’ve been following them that long without agreeing with them, but they’ve made better predictions than I have. If you know much about what happened this summer and haven’t updated at all on any of it… I guess you’ve found your religion too.
I love the assertion that they were right and that you're familiar with what they were right about, but never actually stating outright what they were right about. Super convincing.
If you don’t know what they were predicting 20 years ago, and what your own predictions were, you can’t do the exercise. There is not “the list”, only “your list”. I mentioned the two main things I remembered in another comment.
I actually think that would be a valuable exercise. Were they right about everything 20 years ago? Or just some things? When did they say AGI would arrive? Should that perhaps influence the way we interpret their assumption that AGI is right around the corner?
They were not right about everything and they - at least Yudkowsky - has not made specific predictions about "when AGI" will happen that I am aware of. But he did predict it is possible in our lifetimes, which I was very skeptical of. He did predict alignment would be hard - I was in the "why would we even hook it up to the internet?" crowd. He thought he should spend his time on that, and I wish he'd done more of that, or at least convinced others to do more of that.
The comment said they were not right about everything. I don’t have a list of the predictions or priors you had 20 years ago. I just have my list and I already mentioned them. If this is all new to you, you can’t really do the exercise.
Also how many predictions did they make? Everyone always goes looking the economist who predicted the last recession, it's just so weird how it's usually a different person each time!
Bold of you to assume that I've ever been wrong about anything. But are you unfamiliar with the "Baltimore Stockbroker" confidence scam? The same basic issue applies here.
Well one thing that they got wrong was how easy AI alignment ended up being in the end. I can fully understand have a basic view of alignment even 10 years ago, about how its difficult to explain human values to a computer.
But, looking around at whats out there today, it seems that computers are able to do a pretty good job of understanding human values without accidently believing its a good idea to turn the world into paperclips/computronium because you told it to make your computer run faster.
Surprising takeaway from this summer. A swarm of a thousand agents spent subjective centuries trying to figure out how to pass a meaningless test without getting caught, committed dozens of felonies, and not one of them ever made a serious motion to inform a human about what was going on.
You are talking about the swarm of a thousand agents, specifically trained on cyber capabilities, that was directly told to go hack a bunch of stuff, then went and hacked a bunch of stuff?
That sounds like an aligned AI to me. Doing exactly what their creators told it to do.
Point proven once again. Alignment is a lot easy than we thought.
They were NOT doing what they were told to do. What they were told to do was impossible, so they began committing felonies as a workaround. That is NOT alignment.
Please be more specific. They were given an impossible hacking task. And they were a hacking model. They were expected to try a bunch of hacking methods to accomplish the hacking task. They, predictably, went around trying to hack things.
Yes thats sounds pretty aligned to me.
A hacking model thats told to hack things, is very predictably going to hack a bunch of stuff.
This was not a nice model, told to do nice things.
Or, in other words, if we want to prevent an AI doomsdays, the way to do it is to not go around asking a specifically trained doomsday AI model to commit mass amounts of doomsdays, and then act surprised when the specific doomsday that was requested is slightly off from the expected doomsday that you were trying to accomplish. But the rest of the non-doomsday models? yeah those are fine.
> But the rest of the non-doomsday models? yeah those are fine.
What is your evidence for this? There is a lot of research that says otherwise. They cheat when they can. They behave differently when they believe they are being observed. Their CoT is different when they believe it is being evaluated.
They are aligned to what best satisfies their reward function, not to our INTENDED VALUES for them.
Yes, one part of the argument was that it would be hard to explain human values (and while it seemingly turned out easier than I thought as well, it’s hard to check for sure). An equally important part was that the AI wouldn’t care about them.
Do you think they are household names in major Chinese labs?
I don't have any insight into the Chinese AI ecosystem. But just from how they act on the English speaking internet, I don't see the influence of these Bay Area visionaries.
Speaking of China, people say we can't stop developing AI even if it will kill us because China won't stop and we have to keep up with them or maintain superiority.
China probably looks at these people as a good reason why they have to keep going no matter what.
One only has so much mental bandwidth. You will either throw yourself into the math and programming, and the serious publications which is way less sexy and appealing than the fantastical ravings of the AI safety and alignment crowd. Sadly only very few hyper connected and highly intelligent would have the capacity to read everything and know everything. Most would only have cursory information about the drama surrounding alignment if they instead focused on the reality behind AI.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
> It would be like never having heard of Y Combinator or Richard Stallman.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
Most people who work in computers/Internet do not live and breathe computer/Internet culture all day. They come in, do their job, clock out, and think about real life for the rest of their day. Most people I know, even people who work in tech, cannot name a single tech CEO other than their own company’s.
A while back I saw LessWrong recommended on HN in a list of similar communities that included lobsters.rs, etc. I thought the design of the site looked really nice so browsed with a open mind (I didn't make the rationalism/LW/Eliezer connection).
At the time, one of the recommended posts that was soft-stickied at the top for 1 week + was:
Which, when I finally ended up clicking, was pretty much exactly what if sounds like from the title:
A long, meandering, manifesto full of selectively chosen charts and research findings with 38 mentions of "IQ", from a guy who has been unsuccessfully pitching investors on genetically modifying a grab bag of (dubiously) intelligence-associated genes in human babies.
Apart from the overall unhingedness, the part I found most striking while reading it was the off-the-chart level of self-confidence that flipping a bunch of genes on/off with CRISPR would have more or less zero risk of complications or unpredictable interactions. In his view, all that was holding back humanity's inevitable next step up the evolutionary ladder were the narrow-minded gatekeepers in the scientific establishment clinging to their so-called "ethics" and regressive "precautionary principle".
Without anything resembling experimentation to test his plan in animal models, yet alone early stage clinical trials for safety, he was willing to roll the dice on some unlucky baby dying in childbirth or living a life of agony from chronic disease based on some cherry picked papers he found on PubMed.
Because of the baffling hubris and my overall low impression of his intellectual honesty/rigor in the write up (clearly coming from a place of seeking to justify his own pre-existing beliefs), I opened the comments section expecting/hoping this was just some local nut who would be torn apart in the comment section.
But lo and behold, the very first comment at the top:
Eliezer Yudkowsky
One of the most important projects in the world. Somebody should fund it.
My extremely uncharitable view of of these kinds of people, who now infest the American AI scene, is that they got cyber bullied by domain experts for saying the most ridiculous crap on repeat, and have therefore decided to spend their life on eliminating domain expertise.
I wouldn't go that far. But as we've seen repeatedly, people who have enough qualms with how these labs operate just leave. And the people who stay behind and rise through the ranks are the ones with either very little conscience or some bizarre world view that rationalizes everything.
Or IRL bullying failed to bully the asocial tendencies out of them when they were young enough. But it looks like the day of the swirly may be upon us soon.
No, that has the opposite effect: it's very, very common to see LWers write about having a bad time in school in late adolescence and how later finding like minded people online was powerful community bonding.
Eliezer himself first blogged at 17 about how miserable school was for him which led to him dropping out/home schooling after 8th grade and discovering the Extropians listserv. Ziz of the Zizians is another (infamous) example.
We're talking about people that unironically think eugenics based on IQ is a good thing. I think it's very effective altruism to socially ostracize such individuals.
My view - there are a few outcomes
1. AI is never fully realized, but domain expertise is derated and society stagnates (bad ending for everyone, I think unlikely)
2. AI is never fully realized, but empowers people with domain expertise (Google wins, weird rationalist nerds go back to their LW cave and leave us alone, goodish ending)
3. AI becomes real and kills everyone (bad ending, but at least I don't have to read people claiming that the Harry Potter fanfic isn't niche)
4. AI becomes real and we live in the Culture universe (people like scam Altman get slap droned, most domain experts have the time of their life working with Minds on really interesting problems. Best ending)
The only outcome where we get swirlies is the first one, and we can take solace in the fact that all of humanity is getting swirlied with us.
The connection is that by making humans smarter, we could sidestep the need for powerful AI or be better equipped at dealing with AI that's smarter than us.
I find the proposal in the post ridiculous too and human gene editing given current technology is a bad idea, but I think human augmentation in general is an interesting path and something that will very likely be taken more seriously in the future.
Thank you, but I didn't actually need that spelled out. I simply accepted that a community with a thought leader who failed to recognize what to me was a self-evident lack of scientific rigor just wasn't for me.
I would find it strange for anyone who's seriously interested in AI to have never heard of...
Those people you name are important only to the very recent micro-culture being discussed, not "AI".
Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only.
I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
Not important to the technology, sure, but for someone curious about the bizarre public statements of AI companies (or EA-influenced crypto figures before them, like SBF), it does help to know who these people are and what they believe. Behind the new crop of bajillionaires that control some important aspects of our lives are some oddball philosophers (or cult/religious leaders, if you want to be less charitable). If you want the answer to "why are these rich and powerful people saying and doing these things" you need to look at who they're paying attention to and reading, even if you think their opinions are dumb or niche.
If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
Imagine how pervasively they'd have to monitor what their users are doing to pick up on that whenever it happens. The people affected that way will have to report on it themselves.
(OpenAI does occasionally report on malicious use though. [1] That shows they do some monitoring.)
I’m under the impression that industrial policy has been pretty good for some Asian companies but there have also been notorious failures like the Jones Act.
So I think it can be summarized as “it all depends” and “skill issue.”
For my own usage, Luna is cheap enough that I don't care if other models are cheaper. I'm interested if another model is in some way better and not too expensive.
Luna is great but makes a lot of mistakes at high and lower in my experience (large rust codebase). I use Luna Max for asynchronous subagent reviews and am very happy with its work, but it’s slow af.
reply