I don't understand how he could contribute to the field of AGI research from home, by himself, and maybe with his son. It's the kind of problem that requires incredible amounts of data, hardware, and theory to make any progress.
Wouldn't it make more sense for him to join a cutting-edge team, like DeepMind or OpenAI?
He's openly talked about his work ethic in a bunch of places. He's the type of guy who after a life time of coding calculated he's 100% efficient up until 13 hour work days and then he drops off[0]. Although he did mention working those long hours is often best working on multiple things instead of 1 topic but maybe with AGI there's a bunch of different avenues to explore.
Uuh, AI research has nothing to do with coding all nighters. This is a common misconception among software engineers. It is more a science, and less an engineering problem. It is more about running experiments than it is writing fancy algorithms.
You are bound by the amount of data and computational resources you have at your disposal. Neither are tied to man hours. You can stay up all night for days waiting for your model to train, and it will do you no good.
Everything I've ever read about Carmack suggests he'll do his best on his own at home. Much of this work can be done on reasonable hardware, and he's always been really good at getting a lot out of reasonable hardware. Further, if he needs enormous compute resources, he can get it at any of several cloud providers.
> if he needs enormous compute resources, he can get it at any of several cloud providers
This is exactly the experience of most teams I've spoken with, be they students or businesses, for all the pre-production phase. You simply can't and shouldn't spend on costly AI infrastructure before you've nailed your solution; in fact any kind of infra not just AI.
What you do is rent some cloud to power quickly through your tests — better have 10x worth of big Nvidia GPUs over 2 weeks than buy 1 or 2 max yourself and wait 5-10x more time — not even factoring that setting up clusters of GPU and running such workflows consistently over days, weeks requires pretty deep sysadmin/hardware knowledge and experience; it took me two years to really master that non-problem part on my home server (but now it's a skill I have so that was worth it, but certainly set my research and learning back by as much time).
Besides, there's a time when the familiarity, safety and general comfort of home simply can't be beat. Notwithstanding pool tables and free soda, lol.
Carmack built rockets and id bought $100Ks of NeXT machines to make Doom so I wouldn't put it past him to have incredible amounts of hardware... even at home. Considering his position at Facebook and that he is industry famous he probably has access to data and cloud resources that a researcher outside of OpenAI, Nvidia, Google, etc. normally wouldn't have access to. He could also raise money relatively easily to pursue more intense research.
What would be awesome is if he just said one day "I need 100 million dollars for my AGI project to buy hardware, anyone who wants to share in a 20% cut of the business just send funds to bitcoin address ### or ethereum address ###". He would be fully funded within an hour, probably.
Unfortunately that could never happen because of the SEC.
I think many people expect that a lot of the missing "special sauce" for AGI (if anyone can figure it out at all) is going to be something for which massive GPU power isn't a key factor.
Maybe there is no secret. Just like image recognition is just a bunch of well connected matrices running a dumb algorithm, but at a great speed by GPUs, intelligence is just 100 billions dumb nano-computers with the logic of a fairly simple finite state automata, but with 10 thousand network connections per node. How does nematoda transfer intelligence to its copies? By encoding the FSA properties in the DNA. If this is the case, we'll see the next chapter of AI once a typical smartphone runs a million dumb programmable nanocomputers with a very sense network topology: people will just run the same dumb algorithms on this devices and discover that it exhibits the basic properties of nematoda-level AI. And thus AI would be a dumb engineering problem.
Which will be the cognitive part. The machine learning is more like perception. But perception needs to be tied into an understanding of the world where inferences can be made and one can adjust quickly to a changing environment, while learning new domains or even creating new combinations. This also includes the social-emotional world of humans and language (and not just translation), of course.
My view is that you want many people working independent from each other towards the same goal, and that everyone working in one group could hinder creativity/lead to groupthink
> It's the kind of problem that requires incredible amounts of data, hardware, and theory to make any progress.
I wouldn't be surprised if the opposite was true, at least with the theory part. AI didn't really go anywhere for decades, because people focused too much on theory.
Otherwise, there's a lot of data and hardware at your disposal, even from the comfort of your home.
> Wouldn't it make more sense for him to join a cutting-edge team, like DeepMind or OpenAI?
You mean they guys that are training with videogames that people like John developed?
It might make sense for him to join a team like DeepMind, but we could guess that the "working from home by himself" bit was a lifestyle change he wouldn't compromise on.
> It's the kind of problem that requires incredible amounts of data, hardware, and theory to make any progress.
So I point out a recent example of progress (of which the sequence of fundamental insights is nearly always incremental), where the theoreetical insight was a theoretical derivation, which could be and probably was derived on paper / blackboard, as a direct counterexample: it does not require "incredible amounts of [...] theory".
Why would you compare this with performing manual backpropagation?
>Everything is easy in hindsight
Every breakthrough is non-trivial, else it would not have been a breakthrough, and yet the breakthrough itself can be a relatively simple calculation...
The concept of "AGI" is undefined and virtually worthless to me. There is only non-trivial insights, i.e. theorem and proof.
Deepmind and openai are probably not on a reasonable track to AGI. IMO if we ever make an AGI it wont actually be especially good at things. An AGI, like humans, would probably be pretty bad at math naturally. You could get one to be great at math, but first getting great at math and backing into general intelligence is probably impossible.
Wouldn't it make more sense for him to join a cutting-edge team, like DeepMind or OpenAI?