Sam Barrett, PhD

I'd say it's often not an absence of real world smarts but lack of context to give good responses beyond the text book.

Going to have to turn this into the actual paper:
sbgeoaiphd.github.io/rotating_the...

Yeah. So much "no Claude, this is not about that. Not everything is about that!"
("that" being embeddings)

I love this genre

Very true but my head hurts enough already...

This is all also replayed in my work with embeddings. It's really impressive what can get packed into just 64 int4 numbers...

Absolutely. I'm a geologist by training so I've had plenty of time to ingest billions of years, but I absolutely do not have a *real* intuition for billions of years... let alone trillions of parameters.

Exactly. "Just" only works if your reference point is smartphone word completion or MAYBE early copilot code line completion. If the "next tokens" contain novel solutions to e.g. Erdos problems or suchlike, "just" is thr wrong word.

I think what's most important for most people isn't whether some analogy is in some sense "correct", it's whether it gives people useful intuitions. And autocomplete and stochastic parrot don't tend to lead to strong intuitions about the capabilities or impacts of llms.

“it’s just spicy autocomplete” is advocacy malpractice, is the thing.

the sneering dismissal completely miscalibrates the urgency: media siege devices

—printing press, telegraph, internet—

rip holes in civilization itself

The AI discourse sometimes seems to center on "Is AI good or is it bad?"

I find this framing unproductive. AI is not a fixed thing.

I would prefer to ask "How might we use this technology for good, and mitigate the bad?"

What a shame if the best use we can come up with is no use at all.

Moriarty was a star fleet red teaming exercise that escaped.

Average ceases to be meaningful to me in that situation, and should absolutely not be compared to mean stack overflow. The entire argument falls apart with any sustained and intentional prompting.

Thr average of the specific guidelines? OK, maybe that average actually exists, but let's say my guidelines are esoteric and I'm prompting 100 different requirements which have certainly never existed in this combination before, so the result is the average of what exactly?

The only sense this (see picture) makes any sense to me is if prompting just doesn't work... it's not that it's wrong. It's not even wrong. So I prompt it with coding guidelines so it's stylistically nothing like average stack overflow. So then it's the average of what exactly?

Which raises the question, if you have high quality energy matter converters, why do you have a holodeck, not a "actual real physical stuff created on demand" deck?

Getting GenAI to recreate basic software which massively exists in the training data is the "tea, earl grey, hot" use of GenAI.

How did I miss the holodeck thing!
But I always imagined I could ask the replicator for anything. Picard only ever ordering a tea, earl grey, hot, is just his lack of imagination.

The fascinating and challenging things are those not in the cookbook.

I love this analogy and it's fascinating. Replicators as the closest start trek has to generative ai? You prompt it and get the thing you asked for... maybe? If you explained well enough or it inferred or you're referring to an established recipie.

Another example, the gpt 4.5 "explicitly" bug.

Literally been grappling with this recently. For high effort collaborative writing, do I just put my name? That doesn't feel right. Do I say "AI assisted"? Most people think low effort slop ahead. Do I link to a long statement explaining what I do? Seems inefficient...

Which is why we need a functional vocabulary to describe the difference between low effort prompting and high effort iterative co-creation and traditional human authorship. Though definitely don't use AI at all and call it homemade.

And it is a medium. One if the most powerful and frankly weirdest ever created.

And it needs it's champions.

It needs it's innovators.

Read the full thread.

I've also directly encountered people on here arguing that *any* llm use erodes critical thinking and it is generally impossible to learn anything using them and anyone thinking they do learn *anything* is deluding themselves. Hopefully that is a rare opinion but it exists here.

But alfalfa feeds cows and cows are food. And agriculture is the perfect industry because we need to eat... except when we criticise it, but we strictly do that separately. When discussing AI, agriculture is angelic.
/sarcasm

I think I'm so used to chatgpt and not just recent models that I expect that I need to be the one bringing the epistemic rigor, but you're right that if models (Claude?) exposes people to epistemic humility, it might start to rub off on median person.

Hmmm. I'm curious to apply this to my cowriting. Though that emerges in a different way to code in the first place.

bsky.app/profile/ai4g...

The echo chamber point is interesting. I think it's partly mitigated by vigilance but there is some related practice which I can't name right now other than to refer to my rotations essay...

Interesting feedback. On 1, I suspect that while much is learnable, there are traits which will lead to inequality in what's sustainable, but that's true of everything. 2. Is very fatalistic: we failed before therefore let's not even try? Though I agree it's not easy.