the same way every legal rule is enforced / checked: people look and if it seems iffy they examine/complain, and if a problem is found people get fined/jail.
Yeah could get interesting. Things like legal, programming, math, etc. are going to be relatively cheap / free. Meanwhile something like natural resources, building things, yardwork, etc. may go up in value. Or perhaps we end up with a relatively "spiky" economy since 90% of people are just doing physical jobs (the only human jobs with value), and the AI owner class of course gains all the value from intellectual products (with value diminishing towards cost of electricity). Thinking as human value may be roughly as valuable as it ever was (which is to say nearly non-existant other than the ability to solve an immediate physical problem). Might also be that physical strength and stamina are the highest value mating traits, since they will be able to protect and provide. Physically weak intellectual types will have little value since they can't out-think the machines, and they also can't provide for a family by doing valuable physical tasks. IQ overall begins to drop as mating favors intelligence less and less, as it can actually be an impedance to being productive.
For a short time in history, the ability to obsessively focus on intellectually interesting abstract concepts was highly lucrative, but just as quickly we returned to the laws of nature: those who can lift and move succeed, those who can only think are automated out of existance.
We go from a species increasingly seeing themselves as "brains with bodies" to "amazing bodies with weak brains". The limitation of robots and AI in the physical realm, power hungry, mobility limited are contrasted with human values: energy efficient, highly mobile and dexterous, extremely good strength/speed/size ratios. The brain on the other hand, while energy efficient, is completely outclassed and seen as we see our swimming/jumping abilities: a novelty for sports, but nothing we seriously consider a defining human trait.
The smartest humans can fill weekends with novelty pursuits like building circuits, games, programs, etc. But they are about as useful as whittling and hobby woodcraft, something to pass time, but ultimately of no economic value.
actually this is likely just as performant as the "Ugly but fast" code from the famous talk. After all, this is just branching on GetType() == typeof(Dog) which is presumably boiling down to an integer comparison. This roughly the same as the following C code:
void speak_generic(void* animal, int type_id) {
if (type_id == DOG) {
dog_speak((Dog*)animal);
} else {
dispatch_speak_vtable(animal);
}
}
Advantage 1:
You don't have to maintain this logic (its automatic), so you won't get weird cases if you forget to update all your switches everywhere, and/or you get weird fallthrough logic and footgun yourself in C.
Advantage 2:
You still get the flexibility of the vtable if you need it (for the case the type is chosen at runtime at not known). But for 90% of cases, its just as fast as the ugly C code.
Disadvantage 1: Losing a smug sense of superiority because you eschew abstractions and prefer writing verbose error-prone switch statements over clean easy to understand code.
Disadvantage 2: Writing performant code can no longer be gate kept behind archaic practices, now everyone can just use `var animal = new Dog()` and be done with it.
its because prior to that (2000 Bush era), congress and president had a plan to payoff debt and had a balanced budget plan in place to avoid over spending.
Surprisingly this is actually rather fitting in terms of time scale. When you consider it took ~10,000 agents 88 hours, or 880,000 hours to solve. That's 14.5 years in agent time of continuous 365/24/7 processing. Of course, humans solve things much more efficiently (and didn't also need the massive pre-training of every expert on the planet for 1,000,000,000 human years equivalent). But yeah, human researchers can solve a problem like this in a decade or so, while sleeping, teaching, traveling, and taking breaks, only working a few hours a day on the idea.
The entire controversy was that researchers were about to release their proof which OpenAI claimed to simultaneosululy discover. The proof is the researchers in question.
Actually, removing CoT might make models safer, because we can analyze the entire landscape of their potential outputs, rather than a point-sample (we'll never know how close we were to "kill all humans"). By inspecting intermediate vector spaces, we can actually get certainty bounds on how safely the model is behaving (or even trending).
Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw point samples like the model does and evaluate those (or use temperature zero to just sample the most likely output tokens).
No, this argument doesn't make any sense. With CoT, the model must compress hidden state to actual text and use its scratchpad as a bottlenecked representation of its past thinking. Thus, it is observable and we can tell by the pattern of a CoT what it was thinking to some extent if we do proper interpretability. Change the CoT text, and the model has literally changed the way it was thinking for the next tokens.
How would you do the same if all reasoning is happening in looped transformers? You would have to develop very sophisticated interventions that construct hidden states which you inject into the model instead while it is thinking. Much harder, and much easier for the model to use weird correlations across the hidden state to hide misaligned thought patterns.
What do you make of the fact that CoT doesn't necessarily have to be linear human intelligible language to be useful to the model? It seems as though both approaches potentially require sophisticated techniques. Since CoT seems to work well enough in practice provided it isn't adversarial wouldn't the other approach be expected to perform similarly?
I wrote this in response to people that have been downplaying the threat, or not understand how it would work without terminator-style robots in the streets. I also see people saying "we have a year or two to prepare". That may not be the case.
What? Shutting down infrastructure, destroying critical records (economic collapse), spoofing / impersonating world leaders (confusion, panic, world war, nuclear event), taking over comms generally (you get a message to evacuate your home due to impending disaster, is it real?). All these could cause sufficient chaos and fear as to instigate global collapse. Once the panic sets in after a few of these indicidents, unrest and rebellion can occur, causing authoritarian backlashes or civil wars. IoT and precision ag can be controlled to disrupt supply chain and food supply. Forging documents to cause mass evictings, bankruptcy, credit disruption, transfer of property, asset seizure, arrests / SWAT-ing. Intelligence / signal gathering devices can be corrupted to spread misinfo and cause massive disruption or conflict. This is all possible today with enough hacking ability and malicious intent.
Anyone's life can be suddenly ruined: falsified criminal record, banks drained, credit cards maxed, phone numbers repossessed, internet, water, power shutoff. Friends and family sent your fabricated suicide note, warrants for your arrest issued to local and feds for kidnapping and threat to the president, etc. Injecting tons of copyrighted or illegal material into your hard drives, falsiying messages from your account and sending them publicly containing damning material. Sending messages to employees / employers to destroy business / work relationships. Harnessing your social media to destroy your life or cause panic. Getting you put onto pharmacy watch list so you can't access medications. Its very possible to ruin lives and destabilize the world without money or a physical body.
Now imagine this at scale and how it would affect the world at large if this was covertly done to a substantial portion of society and/or world leaders. Hell just shutting off all the "smart refrigerators" for a few days would cause a massive disruption to the first world.
So when AI is also better at coming up with the tasks and already solving the frontier problems? 1M AI agents running full time thinking at 10M times faster than humans, with 100 times the intelligence in each agent? Any human ideas regarding "frontier problems" or "what the AI should focus on" are irrelevant. We have automated our own thinking and human cognition will be about as important as being "the best at abacus". Its an interesting party trick, but in no way useful.
Before AI, did you find that the existence of human expertise in the world far surpassing yours 'automated your thinking and cognition' or limited your intellectual scope to inventing party-tricks? I've always found greater intelligences than mine are like runways for curiosity. It's hard to believe anyone in their presence would fall into a stupor instead of being highly stimulated by them - and I can't see why it would be any different with ASI. Actually, this "AI undoes your cerebral strapping" argument seems exactly like the kind of slovenly thinking it wants to blame on AI.
yes to a degree. When I was 16, my idea about "what cutting edge thing to work on" was pointless, I couldn't make a meaningful contribution. Most developer's work is mundane and wrote. Things they think are "cutting edge" are actually just ideas they haven't considered, or markets they don't realize are saturated with really good players already. It takes time, effort, and knowledge to make a meaningful contribution. But if those fields are saturated by players that are beyond human ability, then its a bit like trying to become a competitive chess player when the bar for entry is stockfish -- no human will ever again pass this bar. At a certain point, the bar for being paid for intellectual contribution will surpass human abilities. The things humans can understand and ideas they contribute will not be worth much.
To make an unrelated anology, the presence of Lebron James definitely dampens my ability to contribute meaningfully to basketball. Even with all my practice, I cannot achieve his level. But if he didn't exist, and the best 90% of basketball players were eliminated, then I might be able to meaningfully contribute. Of course, I can still play basketball as a hobby, but you were talking about making meaningful contributions on hard novel problems at the frontier of knowledge and ability. That frontier is moving away from humans and will soon be the domain of electronic experts.
I think the main point is the ubiquity of AI intelligence. If there is a human far surpassing me, but unavailable to me, then I have to work through it myself. If the answer is a prompt away, then why bother.
reply