So do humans more or less? The only counterpoints I've heard are religious or unmeasured quantum brain something.

I think there are quite fundamental differences to the LLM model equivalent to an adult human mind and current LLMs (even accepting an LLM can model a human brain, which I don't think is a given - but I also don't think it's impossible).

The details matter (training set, full context). I doubt our current LLMs knows what smelling cut grass on a wet morning feels like, or how the stomach flutters on a first kiss - even if they have "read" tens to hundreds of attempts at describing such things.

You're essentially suggesting we play Turing's Guessing Game; if your LLM can guess my response to any prompt - then it can be said to have modeled how I would reply to those prompts. Expand the prompts far enough, and you could reasonably argue the LLM models me.

>The details matter (training set, full context).

But you said:

>Training shifts the probability - but doesn't change the fact that the output is a sampling based on input and the model?

So which is it?

>You're essentially suggesting we play Turing's Guessing Game; if your LLM can guess my response to any prompt - then it can be said to have modeled how I would reply to those prompts. Expand the prompts far enough, and you could reasonably argue the LLM models me.

I'm merely noting that your argument that "training+input=>output is the problem" applies to humans. There's other arguments for and against LLMs however I'm simply pointing out that your argument isn't a good one.

> I'm merely noting that your argument that "training+input=>output is the problem" applies to humans.

I'm not sure I ever said that it was a problem.

I also never meant to say that sampling across the model (formed by training) based on input couldn't produce novel token sequences.

As for human vs LLM - my argument would be that while trained on a large corpus, the models are poor in experience and "sensory input" - their training so different from growing up - that even if we could model humans as LLMs - the gap between current LLMs and "human" LLMs remain a wide one.

More to the point - I believe the current generation of LLMs are way too sensitive to input (prompt, system prompt, context/RAG) - precisely because they're still too narrowly trained, and also aligned to weigh input heavily.

What we have today are models that work well with language (generate, transform, translate) - not so great on complex tasks - because they're still generating next probable token.

No. Humans actually understand things and can reason from one fact to another. We do not just spit out the next most likely thing without any intelligence the way an LLM does.

The combination of the most probable tokens doesn't necessarily have to be in the training set, thus creating something completely novel.

If you shuffle a deck of cards, you likely have a novel sequence of cards. I'm not sure how that is relevant?

I was responding to this "merely produces syntactically likely combinations of the training data".

That is called a hallucination.

Yes, like the Jacobian counterexample.

That LLM was attached to a Python and MATLAB evaluator. It generated plausible things to feed into a real computer.

Weird take as a lot of things humans invent and test are done in similar ways.

Only if it's incorrect.

It’s not sampling anything though. At each inference step the fundamental output of the process can be interpreted as a probability for each possible token how likely it is. Sampling would imply otherwise but the probabilities are the output.