Embodiment doesn't exist, even in regular humans. We do not have direct access to reality; we have sensory inputs that are much lower bandwidth than one might expect but correlate with external events, and our brain uses them to form rich world models that are usefully predictive. Most of the world we experience is just in our head.
I get what you're saying (mostly based on low bandwidth from sensory organs as opposed to direct access to reality), but your conclusion that we therefore don't have embodiment goes very far. It would be like saying planes don't really fly, since they fly by wire and only have limited inputs and outputs, rather than direct access to reality itself. Well, yeah, they fly using sensors rather than knowing reality itself, but they're still flying. Humans still obviously have embodiment.
Yes, but also for humans who learn of distant lands by reading (books or news, just so long as it's reading).
And also these models have been associating with real-world phenomena from the first moment their training data did, and also those text vectors are (to varying degrees) associated with corresponding image vectors in multimodal models.
Of course, the Plato's cave critique would still be valid.
> I will concede that using LLMs in software development is a different category from fiction, insofar as the languages and APIs are much smaller and more tightly constrained and in software you actively don't want long-range dependencies between elements (something you often do want in fiction, i.e. put a gun on the mantlepiece in part 1 of the book, pull the trigger in part 3).
It can definitely plan such things. Because agentic systems don't have to output a book all at once. The mainstream still lives 1-2 years in the past.
Agents can plan out a narrative draft, write it out section by section, maybe out of order, then tweak it, plant clues if it wants to, then trigger it in another chapter file etc. Then read the whole thing, reconsider, etc. It can iterate. I'm not saying it will be great literature, but the problem isn't the inability to create long range references.
My bigger point would be this: arguing from such mistaken technical ideas, when not having any idea about the tools is a sign that people's real problem is none of those technical limitations, but Western culture has lost the vocabulary to express their true sentiment. They are grasping for words but can't reach for "soul" and other religious terms because that would sound ridiculous in a secular modern world. But they clearly want to express that and we'd be ahead if we could discuss in such actual terms. Would you want the machine to write literature if, hypothetically it could write indistinguishably from any great writer? I guess not. Because there is something mysterious and important about the human spirit. (the same vocabulary-inaccessibility is plaguing other culture war issues too). People will rather talk about fake water consumption problems than touch the topic of the human soul or spirit.
I tackle these ideas/questions head-on using Claude to help book-shape the result.
https://spine.zice.app/jeremy/the-smaller-infinity
In my view, the accumulating density of useful language/token patterns is the real story, not who/what is processing them. There is no "why" other than the universe's physics permit such accumulation, but that doesn't make the "how" any less interesting.
You can click the header and copy the whole thing to clipboard, ask your favored agent for summary/conversation.
> but Western culture has lost the vocabulary to express their true sentiment. They are grasping for words but can't reach for "soul" and other religious terms because that would sound ridiculous in a secular modern world
Ironic to implicitly accuse Stross of this; as I recall, he's one of the voices describing The Singularity as Christian Eschatology/Book of Revelation/the rapture for atheist nerds.
> Would you want the machine to write literature if, hypothetically it could write indistinguishably from any great writer? I guess not.
One of his own paragraphs explains his non-financial perspective and reason:
I do not want or need a large language model to write my fiction for me. I write fiction compulsively—before I was published I wrote for many years as a hobbyist—so why on earth would I pay someone else to take my fun away?
I may not have quite the level of obsessive desire to write as he has, but I have some, and it is different from the desire to simply consume someone (or some*thing*) else's work.
Without criticism, he is also clearly upset about the economic impact this will have on him (he says specifically he is "one of the parties to the settlement in the class action lawsuit against Anthropic AI for pirating ebooks to train their LLMs" and "train an LLM that is intended to compete for revenue with the authors of the works they stole"). Writing is not a high-earning job (with the exception of a few superstars who may as well be lottery winners), and he has written in support of very long copyright durations that can be inherited.
> People will rather talk about fake water consumption problems than touch the topic of the human soul or spirit.
For a living, he writes about Cthulhu and their ilk doing mind control; I want to say I don't think this would put him off, but if it did he'd hardly be the only professional author who can write a fictional story about something without being able to apply such ideas to rhetoric. (He'll hate being compared to Rowling here, but consider how Rowling hates trans people while her fiction has a variety of shape-shifting spells).
You need better arguments than "you don't like to use it yourself and prefer writing yourself" or some metaphysical voodoo-like theory (analogous to tribes that refuse to be captured on a photo) whereby feeding your words to a training algorithm somehow corrupts your pure Platonic words or whatnot. Your utterances are not yours to grab on to. Copyright is about expression and derivatives. You could never copyright your style, or else any music band that copies the general style of another band could have been sued and then the development of music would also have gone quite differently. No, ideas, styles, claims, knowledge are by default part of the commons. The narrow carveouts are for incentivizing more creative work and are targeted and specific, time limited. You can't generally, by default, restrict what others can do with ideas that you've put out. Patents are a narrow exception. And copyright for non-transformative derivative works or copies. And trademarks to protect consumers from mis-identifying the creator of something.
-----
> clearly upset about the economic impact this will have on him
So readers will read AI generated books because they are better than his writing? People can't invest ten bucks or however much into reading a good book? How about piracy? It exists. I bet I could download all his books in less than 5 minutes. If someone insists on not paying for that book, they can easily get it.
> he has written in support of very long copyright durations that can be inherited
This whole copyright and intellectual "property" idea is extremely new in human history. More people should know this fact. This has not been how people thought about culture before the ~1800s, or, well before the printing press. Copyright is a narrow exception to the general rule that people are free to listen to others and use the ideas and spread the ideas. There are exceptions like trade secrets and various dangerous information about nuclear topics and whatnot, but generally preventing others from talking about what they heard is tyrannical. If anything, copyright should be reduced to something like the creation of the work + 20 years. Not lifetime +X. That's plenty enough to be the sole distributor and to block others from building on top of it. After that it's time to let people use culture as it's always been. Changing, mixing, remixing, retelling, combining, etc.
> Would you want the machine to write literature if, hypothetically it could write indistinguishably from any great writer? I guess not.
I sure would. If you want a religious tone to it… It would be a signal suggesting that the human spirit(tm) had finally transcended.
If humans can someday build a machine that can truly carry a (trans)human mind, and let some future souls live unconstrained by all the quirks and limitations of our preexisting biology, exercising more control of their own lives - that’d be wonderful.
I obviously have no idea wherever language models can lead to such dream, of would be a dead end.
I think attitudes like this are traitorous to the human species. You’re not talking about a transcendence, or at least, not one that I recognise. Our biology is an innate part of us - to improve, to fix, sure, but not to cast off. Who is directing this “transcendence”, and who benefits? I absolutely do not want a religious tone to any of this. Awful vibes.
I find this "traitorous to the human species" interesting. You could not form these thoughts without an accumulated knowledge stored/accessed outside human biology. No other species forms such explicit thoughts because they lack access to that accumulated pattern of useful sequences. We've already "transcended" our biology in that sense. Who is directing... that becomes interesting in new ways once outside-a-body sequences start to accumulate. My guess is that it's that accumulation that sets the trajectory, just as DNA's "accumulation" of useful sequences produced forms fit for the local environment(s).
> Our biology is an innate part of us - to improve, to fix, sure, but not to cast off.
Right now, necessarily so.
The more mysteries that science solves, the more we unweave the rainbow*, the worse the tension between rationality and our sense of what we are. Am I the atoms, or the pattern betwixt? Am I the present moment alone, my memories mere shadows of past patterns and my current form doomed to be a shadow to a future pattern of these self-same atoms?
What are you talking about… You can use the word soul without believing in a religious concept of soul. It’s done all the time in secular societies. It’s even a very common term used to describe the problem with LLM generated content, that it lacks soul or human spirit, it lacks the human meaning, intent
It's rare. It's mostly, "but it will never be able to do X" (when it can already do X), or it uses too much water (while having no idea or reference point to what amount is much), or other climate things, or non-progressive bias, or copyright (when training is fair use) etc.
Just be upfront. 1) It threatens your livelihood that people can get the thing done faster and cheaper than what they used to pay you, 2) our relationship to an AI shaped entity is spiritually risky and a esthetically off putting, and we have to talk about golems and the tower of babel and stuff. If you think I'm being incoherent, you're in for a ride. Society will have to come to terms with major major issues, not just job replacement and bad slop art.
I think it is true that people who obviously dislike AI for ideological reasons often lead with ecological concerns or doubt about the capabilities, leading to an unproductive discussion.
Have you considered engaging on those ecological concerns? There is quite a lot to dig into and can be pretty interesting. For example, try to project how much electricity will be needed to power all the datacenters in construction, then compare that to the existing grid and infra, then figure out the cooling capacity. If done well you will see that we have a pretty massive resource problem here
I had a look now, and apparently we are projected to increase by 2.3x datacenter power consumption until 2030 from 415 TWh to 945 TWh.
That is going from 1.5% of global electricity use to 3%.
It is less than I expected.
Maybe if AI does work out better than expected for knowledge work or becomes applicable to robotics and fields like construction, we are going to see a larger buildout.
> That is going from 1.5% of global electricity use to 3%.
> It is less than I expected.
I suspect this is because of the word "global". Are not most of these currently planned specifically for the USA? Sure, the USA has a disproportionate use of electricity, but IIRC it's like 16% (1/6th) of global?
3% of global electricity is roughly the same as 18% of US electricity; a bit less because not all the new DCs will be in the US, but that's something that I consider with regards to the economics and why I'm bearish on this topic, as I think it will be a case of Dutch disease in the USA: https://en.wikipedia.org/wiki/Dutch_disease
Okay nice, after looking at the macroview look at the immediate impact into local communities and how much data these companies refuse to, rightfully I might add, share with the public.
Weird how all these concerns can simply go away if they just simply shared the information the public wants (water usage, sound generation, power usage) rather than refuse to do so.
I don't think it ever gets old to restate that the singularity and all of the gradiose promises of a glorious 21st century have last mile problems.
But this post seems to mostly channel Harlan Ellison. And while he was amazing in his prime, with some absolutely legendary rants, he didn't age well in the end.
I do like his challenge to make LLMs useful to himself and to other authors though. Someone needs to make that happen and nearly exactly how he described it.
I'd quite like a tool (running entirely locally on my own hardware, with no cloud service and no copyright-thieving grifters making bank on it via subscription fees) that digests a manuscript and derives a scene-by-scene timeline, that I could then query interactively and use to plan my next round of edits. Being able to map out where and when each protagonist and minor character shows up, and see a frequency distribution heat map of names in the manuscript, would be useful.
I'm sure I've read about Hollywood scriptwriters having tools like this.
(Or is this just the Gell-Mann amnesia effect striking again?)
This is very interesting coming from an author in whose writing autonomous, super-powerful AIs have been a common theme.
Consider his book Accelerando (which I'll take the opportunity to plug again, especially as he's made available for free here: http://www.accelerando.org/fiction/accelerando/accelerando.h...) Not only did I find it quite engaging and thought-provoking, it is also proving rather prescient, and even helpful in decoding some of the things that are happening today.
For instance, in the very first chapter the protagonist spawns agents to go research something in the background and report back to him. And then a year ago, I randomly became curious about a rather involved topic (how fast could we feasibly replace all human labor with robotics), but I did not want to spend time researching so I outsourced it to Google Deep Research which churned away for almost half an hour and came back with a 30 page report with 49 citations via "actual internet searches for verifiable sources." (If you're curious about the conclusion: not for a very long time partly due to critical supply chain constraints.)
After I went through the report, it suddenly struck me: my agent may have executed in a GPU cloud instead of a cybernetic brain, but holy crap I had literally just lived a SciFi scene! A scene that I did not expect to experience in my lifetime!
Which is why I found TFA a bit unexpected. TFA says a few things that I would disagree with. Like, no, AI is not a "stochastic parrot" and I'd assume he'd be primed to realize it. And AI is not competing with authors -- other authors with AI are, and using AI trained on real text to aid writing has been a thing since the days of red squiggly lines in word processors, which TFA even acknowledges.
If you see his last few comments (https://news.ycombinator.com/user?id=cstross) it's clear he's pretty negative on the tech industry and Capitalism, which I tend to agree with. I can see how that could color his thinking.
That may also mean he's avoided LLMs to the extent that he is not aware what the frontier models have become. I would encourage him to put aside his distate and re-engage with them deeply; maybe he'd be at least a little bit excited to see some of his writing turn out to be prophetic in some good ways besides the bad.
Not being able to see the forest for the trees I suppose is a prerequisite for success in Silicon Valley, because it one had even a fraction of Stross's perspective on the matter they'd be in a different line of work.
For some value of "work out" anywhere between "human utopia that spreads across the stars" to "we destroy everything in a ball of nuclear fire and only microbes survive", yes, it will. The universe will keep on going either way.
I’m more optimistic about humanity but I doubt the west will work out in its current trajectory.
If you haven’t read The Collapse of Complex Societies by Joseph Tainter it’s worth a read. It’s not a conspiratorial or even outright pessimistic theory.
Ill check the book out as I am extremely interested in these kind of topics. Although I am extremely pessimistic.
You wrote "the west", what system will (in theory) work out? And how much does it value the individual?
I know capitalism for example, can in theory, never succeed as its base premise is built on endless growth (which simply wont work out with finite resources). It's a short term system that we need to improve on. All the other systems we had, like feudalism, I'd rather die than live in...
Personally I'd highly prefer socialism, but that also wont work out as too many humans highly reject its concept and its also not safe from corruption.
Tainter suggests that societies collapse because the costs of maintaining complexity eventually outweigh the benefits. “Collapse” in this context is more like a simplification than a catastrophe. And he points out that for a lot of people, the collapse actually made life better for them, as it becomes more local and less totalitarian.
I mentioned the west because I’m a westerner with little experience of other systems, but I don’t think Tainter’s conception of a collapse would be limited to the west.
Interestingly since you edited your comment to talk about capitalism, I think under the lense of Tainter’s theory it’s the most resilient to collapse since it’s simpler, at least in theory. In practice it’s heavily regulated - in fact I’d say no country really is capitalist, at most you could say it’s “state capitalism” which is very different from free market capitalism. But you’re also not wrong in that it has issues of its own.
> Tainter suggests that societies collapse because the costs of maintaining complexity eventually outweigh the benefits. “Collapse” in this context is more like a simplification than a catastrophe. And he points out that for a lot of people, the collapse actually made life better for them, as it becomes more local and less totalitarian.
This premise sounds very intriguing. And yea I named capitalism just because I don't think we can go on like we do currently, and I'd agree that in theory it's not complex at all. In fact it makes a lot of sense (in the short term, and as long as people are willing to get exploited to be able to finance life).
Nonetheless maybe heavily regulated capitalism works, allthough, regulators are prone to corruption and act way too slow. And lobbyists preventing measurements to combat climate change as an example, I currently see as direct threat to my personal wellbeing. Hard to believe in a system that relies on exploitation, the big "if you work hard you can make it" lie and actively threatening ones or ones children's wellbeing by being ignorant to obvious and well researched topics for short term monetary enrichment...
Roughly a third of the market is digging a $1T-a-year-deep-hole with <5yr amortization schedule on all of it and passing around the same $100B like it's in all of their bank accounts simultaneously. Yeah, shit's going to work out for sure, in all of the same ways that flying a plane into the side of a mountain has a clear ending.
> It would be foolish to deny the effectiveness of image recognizers based on generalized adversarial networks (GANs), the key neural network technology underlying LLMs
I could be misreading this, but I hope the author doesn't think GANs are used in LLMs. They are cool, though.
It’s unfortunate - he’s an author, it would have been fine to stick with an authors perspective that LLMs can’t write (which is true), as well as the copyright stuff (which I don’t agree with but he certainly has standing to give an opinion on).
But he’s made the error of trying to come at it from a technical perspective, when he clearly knows nothing about that side of things, which discredits the rest.
> But he’s made the error of trying to come at it from a technical perspective
Huh? That quote appears to be the entirety of the "technical perspective" of the post and is an aside from his larger points that you have blessed as "fine". Literally nothing in the rest of the post relies on that incorrect statement.
Let me quibble with what is discredited here, given the entirety of your point is built upon an error.
Aren't RLVR signals typically based off formal systems not natural phenomena? The formal sciences are certainly useful for producing tools used in natural science, but it's not entirely clear they alone are sufficient to associate text to natural (real-world) phenomena. Honestly, the pretraining and RLHF are probably more tied to the world than RLVR, for all that RLVR might be useful (maybe even more useful) for making the model perform better in certain tasks (such as working with formally defined systems).
Yeah, I should have said RL, not RLVR. The point is RL interacts with external world, while autoregressive pretraining is limited to the passive ingestion, and RLHF relies on a model of human preferences that has no access to truth sources besides the limited training data it was built upon.
If you want a more concrete example, then LLMs are also trained on visual data these days, which means they do have access to the world in an important way. This directly contradicts the blogpost's claim that LLMs have
> no way to associate the text vectors they manipulate with real-world phenomena.
Historically, that LLMs were text-only used to be a major argument for why they "lack access to meaning", see the Stochastic Parrot paper and the Octopus paper that it references. But even the authors of those papers have (grudgingly) conceded that the argument no longer holds due to multimodality.
A lot of people will stop reading or become completely distracted when encountering something so blatantly wrong, and it also indicates the author is willing to say such things. I see some people downplaying it, that’s fine, it’s clearly not helping the author make a case. Even im this discussion, the current top comment and a long thread are solely about that error (granted that’s not too unusual here)
Things that are vaguely GAN-shaped might play some role in post training. Though no one would call them GANs or identify them as the key underlying technology.
There is RLHF with a model that's trained to emulate a human evaluator, but the evaluator is not really trained jointly with the main model to adapt to its distribution and tell it from real text. Though I'm sure there are some niche cases when this is done. But definitely not a prominent thing.
Yeah. Ironically, if he used an LLM to proofread his work, it would have told him that GAN's are generative not "generalized". That LLMs are primarily built on Transformers rather than GANs. And that they're more famous for image generation rather than image recognition.
In general probably not much of a stretch to get rid of convolutions or recurrence by replacing with attention and see if it works hence the title of the original transformers paper.
No, I simply think the person is not familiar with these concepts, which is not a crime, it's okay to be wrong, I didn't want to banish that person from here, just corrected the statement. But might have been also a brain fart. Happens. Just wanted to correct it in the interest of beginner reading this.
Once Musk said he believes in Transformers instead of diffusion (or the other way around). When actually diffusion transformers are very popular and mainstream. What he meant was autoregressive inference vs diffusion. People like to use buzzwords while not knowing them. Old issue, it was the same decades ago.
To make it even more confusing, there are also diffusion LLMs, which typically but not necessary, use transformers also.
And independently of diffusion and transformer and llm or image generator, you can optionally put a GAN discriminator adversarial loss on any of them.
sorry but I see incorrect AI responses almost every day in things I query on Google, generally because I am often querying on something I am an expert in and I just want a linking source. There will be a good link in the result, generally, but the AI almost always messes up some elemental fact, sometimes it is because the AI has "misunderstood" what I asked which is reasonable, but most of the time the result shows that it has "understood" what I said but there will be a statement of fact that is just wrong.
Crazy to assume that an individual of vision has so thoroughly dropped the plot simply because their position is at odds with tech industry orthodoxy. If anything that should be an indication to more closely examine your assumptions.
I just left this comment: https://news.ycombinator.com/item?id=49137863 -- tl;dr I suspect it's his view of the Tech industry and Capitalism that is coloring his views, rather than an objective evaluation of state-of-the-art LLM technology.
> I'd quite like a tool (running entirely locally on my own hardware, with no cloud service and no copyright-thieving grifters making bank on it via subscription fees) that digests a manuscript and derives a scene-by-scene timeline, that I could then query interactively and use to plan my next round of edits. Being able to map out where and when each protagonist and minor character shows up, and see a frequency distribution heat map of names in the manuscript, would be useful.
Interestingly enough, I do this with various books I'm reading. E.g, I'm currently rereading Accelerando and had Claude generate a wiki-like timeline of key events, characters, and salient plot points. That makes it easier to jump around when I want to re-read a section and grok a plot thread that is scattered across chapters. Ironically, it also exposes inconsistencies (or "hallucinations" as some might call them) in the text because the author didn't have an AI proofread the text.
You could build a tool like that and on a MBP with 48+ GB of RAM all of that can happen locally in terms of keeping your content locally. I am sort of an outliner and planner and I heavily use AI when writing documents for work. I don't see why fiction / prose would be any different? Indexing your content with a vector RAG and having a little 27B model locally could do everything he wants with some help from a big model to implement it all. It sounds like a weekend of coding for a functional prototype to me.
He doesn't want to use a model trained on someone else's text. So he would need to train a model from scratch. I don't know whether his own texts would provide enough data.
> Ironically, it also exposes inconsistencies (or "hallucinations" as some might call them) in the text because the author didn't have an AI proofread the text.
That's not irony, or even cute. He's literally describing a tool he'd like to use to plan edits to his writing.
Speaking as a fucker, I think the use of em-dashes has been pretentious since the 1890s. The rise of typewriters destroyed em-dashes already. The LLMs subsequently created an accidental parody of sophisticated writing.
It's like saying I am intimately involved in the printer's art. This is usually not true, unless you're involved with a very niche publisher who still has a box of sorts and inky fingers. Hyphen-minus all the way.
No it doesn't. I happened to have read "design and typography in easy steps" as a (bored) child of the 90s, but that's the extent of my "printer's art" knowledge.
I don't understand the celebration of ignorance myself. There are lots of things I don't know how to do or how to use, but I'd never dream of complaining because somebody else does.
People seem to LOVE parens, although I was taught that they are almost 100% never the correct thing to use in prose. But apparently "some fucker" will accuse me of being pretentious ¯\_(ツ)_/¯
Otherwise, I see people just use a regular dash a lot. I love the em dash myself, but of course stopped using it so I won't get accused of AI (which is sad). It's easy to type, at least on a Mac.
Parens and em-dashes do different things. A simple parenthetical is an secondary aside (it's a half step towards a footnote). An interrupting phrase is intended to (abruptly) command attention.
I mean, make the case em-dashes are overused and used lazily, I'm right there with you. I just don't see the pretense.
While Stross is currently a neo-Luddite who doesn't believe in the Singularity, his book Accelerando is probably the best work of Singularitarian fiction ever written. It was written in the 2000s, starts in the 2010s, and covers the various decades of this century. I highly recommend it for anyone who wants to know how weird this century will be. It's also available for free online. [1]
It's one of my favorite books and the main reason I'm e/acc and many other pro-AI people love it because we view it as utopian sci-fi rather than the dystopian world Stross invented. This might be the best case of the Torment Nexus meme [2] in action.
> This might be the best case of the Torment Nexus meme in action.
At least the author seems to think[1] it is one, if in not so many words:
> And don't ever let anyone tell you that Accelerando is techno-optimistic or pro-AI; by the end of the book our entire species is extinct, surviving only as simulations/memories recalled by something arguably not alive.
1) Having met Stross, and briefly spoken to him way back when, he never actually believed in the singularity as a possible future.
2) Turns out it was the sci-fi author who was wrong all along! Fascinating to see someone confidently citing a meme, while actively evidencing the fallacy it exposes.
>he never actually believed in the singularity as a possible future.
I think my favorite moment in the book is when one of the characters (at the time, a digitized consciousness traveling at near light speed in a space ship the size of a coke can to explore an anomalous signal sent by alien life) argues that the singularity is a ridiculous concept and will never happen. I always thought that was a really clever observation on the part of Stross that our perspectives are so narrow and provincial, we often can’t see what’s happening in front of our own faces. Your comment does kind of recontextualize that for me and I sort of wonder if that was the author speaking directly to the audience?
You think accelerando is utopian sci fi!? Which characters perspective are you identifying with so strongly? They all have pretty severe societal challenges that makes me surprised to see the word Utopia.
Manfred and Amber’s lives don’t seem particularly dystopian. Human society, up until the Field Circus encounters the router and later returns home, seems pretty great. Chaotic and challenging, yes, sad for Microsoft shareholders, sure, but I would certainly rather live in a world with access to the technology they have than our own. Space exploration, extended life spans and rejuvenation, new methods of relating more closely with other human beings are all things I would love to experience.
The protagonist who believes that information should be free and that scarcity is mostly artificial due to bad economic models. And that it would be possible to implement proper pro-human capitalism or communism if we had enough compute to solve standard CS problems like filling knapsacks or maximizing flow through a graph.
> In the background of what looks like a Panglossian techno-optimist novel, horrible things are happening. Most of humanity is wiped out, then arbitrarily resurrected in mutilated form by the Vile Offspring. Capitalism eats everything then the logic of competition pushes it so far that merely human entities can no longer compete; we're a fat, slow-moving, tasty resource – like the dodo. Our narrative perspective, Aineko, is not a talking cat: it's a vastly superintelligent AI, coolly calculating, that has worked out that human beings are more easily manipulated if they think they're dealing with a furry toy. The cat body is a sock puppet wielded by an abusive monster.
https://www.antipope.org/charlie/blog-static/2013/05/crib-sh...
Regardless of believing in the “singularity” or not, please explain how this sounds utopian in the slightest without borderline (if not literal) sociopathic anti-human worldviews.
I don't find it to be utopian, but in trying to imagine how, it occurred to me that maybe the idea that we even survive to be entangled in the wider universal future, maybe that is utopian? To matter.
I think there are a lot of paths where we just rock ourselves back to a regressed agency societal state, and just never have the capacity to have any affect on the wider universe. Which is maybe good for the universe, as we are certainly not a role-model for certain forms of balance. But maybe the balanced view doesn't seek to dominate and influence and become ubiquitous, after all
> Regardless of believing in the “singularity” or not, please explain how this sounds utopian in the slightest without borderline (if not literal) sociopathic anti-human worldviews.
There was a guy on here the other day trying to argue that Brave New World wasn't a dystopian novel. There are some absolutely wiiild world views out there.
> There was a guy on here the other day trying to argue that Brave New World wasn't a dystopian novel. There are some absolutely wiiild world views out there.
I think that's likely to be all the sex and drugs? Though I'm basing that on the Wikipedia plot summary. I have no interest in darkening my day by actually reading any work that can be rightly described as "dystopian".
I'm almost 50, but I absolutely could see how someone younger could view it that way -- precisely because the present is much more similar to than novel than it used to be?
A large part of that goes via arguing about consciousness.
Starts of with a quite defensible:
First: super-intelligent AI is unlikely because, if you pursue Vernor's program, you get there incrementally by way of human-equivalent AI, and human-equivalent AI is unlikely.
But the moment he hits that (parenthetical) paragraph about consciousness, he blends human capabilities with human-like consciousness.
If human-like capabilities required human-like consciousness, early models (like GPT-4) passing all those exams would suggest those models being "as conscious as" a university student. (My inner Douglas Adams wants to invert this, but the best I can do is "The Xarloxians did not consider university students to be conscious beings; their reasoning being…", which you may recognise as a riff on Babel fish and the non-existence of God).
Belief in the Singularity and belief in God strike me as extremely similar.
Both, I think, are extremely interesting ideas where very few theories, e.g. pro, anti, or anywhere in between, can be written off as ridiculous -- probably due to a lot of ambiguity and difficulty in defining proof.
For me, I find the Singularity very implausible due to the fact that we talk about "general intelligence" quite a bit, and the more we do, I'm finding that I don't think it exists. There are a set of low-level basic skills that I think -- if you don't have, you're not intelligent. And then there are higher level skills that vary wildly.
So, will we make AIs that can say and do very clever and amazing things; yes, this is happening. Will they be, like ascendant, or something? I would say -- yes, but perhaps not in the way we think? Like, humans made "The Bible." And that's obviously a big huge, impactful, maybe superhuman thing. But feels different from this idea of a "Singularity."
There's a reason the Singularty was called "the Rapture for nerds" before LLMs were even a thing.
You can find a lot of neo-religious behavior within the AI accelerationist community. Cults coming out of rationalism and fear of the wrath of an angry AI God, making the world right for AI's coming (and even killing the nonbelievers.) Evangelism. Promises of infinite plenty in a post-scarcity heaven. Even a persecution complex - "Luddites" are increasingly demonized, hated, accused of moral depravity for their doubt, and for not abandoning the world, taking up their prompt and following the AI.
How many people post about how AI seems to be draining them of meaning and purpose, that it doesn't live up to the hype, that keeping up with the rituals is exhausting, but feeling that the problem must be with them and not AI? That's a religious person having doubts about the nature of their faith, unable to accept that the God they believe in might not be real, and framing their issues as their own moral failing.
And all of this before the "Singularity" even shows up, and people upload their minds - I mean, meet God in the air.
It's religion, very specifically neo-Christian ideologically. Regardless of any arguments one might make about the practical benefits of using AI in any particular field, it is obviously also religion.
> They're word-association mechanisms with no embodiment and no way to associate the text vectors they manipulate with real-world phenomena.
Doesn't most of this also apply to a guy living in The Matrix?
Embodiment doesn't exist, even in regular humans. We do not have direct access to reality; we have sensory inputs that are much lower bandwidth than one might expect but correlate with external events, and our brain uses them to form rich world models that are usefully predictive. Most of the world we experience is just in our head.
I get what you're saying (mostly based on low bandwidth from sensory organs as opposed to direct access to reality), but your conclusion that we therefore don't have embodiment goes very far. It would be like saying planes don't really fly, since they fly by wire and only have limited inputs and outputs, rather than direct access to reality itself. Well, yeah, they fly using sensors rather than knowing reality itself, but they're still flying. Humans still obviously have embodiment.
Yes, but also for humans who learn of distant lands by reading (books or news, just so long as it's reading).
And also these models have been associating with real-world phenomena from the first moment their training data did, and also those text vectors are (to varying degrees) associated with corresponding image vectors in multimodal models.
Of course, the Plato's cave critique would still be valid.
Searle* strikes again!
* https://en.wikipedia.org/wiki/Chinese_room
From the comments:
> I will concede that using LLMs in software development is a different category from fiction, insofar as the languages and APIs are much smaller and more tightly constrained and in software you actively don't want long-range dependencies between elements (something you often do want in fiction, i.e. put a gun on the mantlepiece in part 1 of the book, pull the trigger in part 3).
It can definitely plan such things. Because agentic systems don't have to output a book all at once. The mainstream still lives 1-2 years in the past.
Agents can plan out a narrative draft, write it out section by section, maybe out of order, then tweak it, plant clues if it wants to, then trigger it in another chapter file etc. Then read the whole thing, reconsider, etc. It can iterate. I'm not saying it will be great literature, but the problem isn't the inability to create long range references.
My bigger point would be this: arguing from such mistaken technical ideas, when not having any idea about the tools is a sign that people's real problem is none of those technical limitations, but Western culture has lost the vocabulary to express their true sentiment. They are grasping for words but can't reach for "soul" and other religious terms because that would sound ridiculous in a secular modern world. But they clearly want to express that and we'd be ahead if we could discuss in such actual terms. Would you want the machine to write literature if, hypothetically it could write indistinguishably from any great writer? I guess not. Because there is something mysterious and important about the human spirit. (the same vocabulary-inaccessibility is plaguing other culture war issues too). People will rather talk about fake water consumption problems than touch the topic of the human soul or spirit.
I tackle these ideas/questions head-on using Claude to help book-shape the result. https://spine.zice.app/jeremy/the-smaller-infinity In my view, the accumulating density of useful language/token patterns is the real story, not who/what is processing them. There is no "why" other than the universe's physics permit such accumulation, but that doesn't make the "how" any less interesting. You can click the header and copy the whole thing to clipboard, ask your favored agent for summary/conversation.
> but Western culture has lost the vocabulary to express their true sentiment. They are grasping for words but can't reach for "soul" and other religious terms because that would sound ridiculous in a secular modern world
Ironic to implicitly accuse Stross of this; as I recall, he's one of the voices describing The Singularity as Christian Eschatology/Book of Revelation/the rapture for atheist nerds.
> Would you want the machine to write literature if, hypothetically it could write indistinguishably from any great writer? I guess not.
One of his own paragraphs explains his non-financial perspective and reason:
I may not have quite the level of obsessive desire to write as he has, but I have some, and it is different from the desire to simply consume someone (or some*thing*) else's work.Without criticism, he is also clearly upset about the economic impact this will have on him (he says specifically he is "one of the parties to the settlement in the class action lawsuit against Anthropic AI for pirating ebooks to train their LLMs" and "train an LLM that is intended to compete for revenue with the authors of the works they stole"). Writing is not a high-earning job (with the exception of a few superstars who may as well be lottery winners), and he has written in support of very long copyright durations that can be inherited.
> People will rather talk about fake water consumption problems than touch the topic of the human soul or spirit.
For a living, he writes about Cthulhu and their ilk doing mind control; I want to say I don't think this would put him off, but if it did he'd hardly be the only professional author who can write a fictional story about something without being able to apply such ideas to rhetoric. (He'll hate being compared to Rowling here, but consider how Rowling hates trans people while her fiction has a variety of shape-shifting spells).
You need better arguments than "you don't like to use it yourself and prefer writing yourself" or some metaphysical voodoo-like theory (analogous to tribes that refuse to be captured on a photo) whereby feeding your words to a training algorithm somehow corrupts your pure Platonic words or whatnot. Your utterances are not yours to grab on to. Copyright is about expression and derivatives. You could never copyright your style, or else any music band that copies the general style of another band could have been sued and then the development of music would also have gone quite differently. No, ideas, styles, claims, knowledge are by default part of the commons. The narrow carveouts are for incentivizing more creative work and are targeted and specific, time limited. You can't generally, by default, restrict what others can do with ideas that you've put out. Patents are a narrow exception. And copyright for non-transformative derivative works or copies. And trademarks to protect consumers from mis-identifying the creator of something.
-----
> clearly upset about the economic impact this will have on him
So readers will read AI generated books because they are better than his writing? People can't invest ten bucks or however much into reading a good book? How about piracy? It exists. I bet I could download all his books in less than 5 minutes. If someone insists on not paying for that book, they can easily get it.
> he has written in support of very long copyright durations that can be inherited
This whole copyright and intellectual "property" idea is extremely new in human history. More people should know this fact. This has not been how people thought about culture before the ~1800s, or, well before the printing press. Copyright is a narrow exception to the general rule that people are free to listen to others and use the ideas and spread the ideas. There are exceptions like trade secrets and various dangerous information about nuclear topics and whatnot, but generally preventing others from talking about what they heard is tyrannical. If anything, copyright should be reduced to something like the creation of the work + 20 years. Not lifetime +X. That's plenty enough to be the sole distributor and to block others from building on top of it. After that it's time to let people use culture as it's always been. Changing, mixing, remixing, retelling, combining, etc.
> Would you want the machine to write literature if, hypothetically it could write indistinguishably from any great writer? I guess not.
I sure would. If you want a religious tone to it… It would be a signal suggesting that the human spirit(tm) had finally transcended.
If humans can someday build a machine that can truly carry a (trans)human mind, and let some future souls live unconstrained by all the quirks and limitations of our preexisting biology, exercising more control of their own lives - that’d be wonderful.
I obviously have no idea wherever language models can lead to such dream, of would be a dead end.
I think attitudes like this are traitorous to the human species. You’re not talking about a transcendence, or at least, not one that I recognise. Our biology is an innate part of us - to improve, to fix, sure, but not to cast off. Who is directing this “transcendence”, and who benefits? I absolutely do not want a religious tone to any of this. Awful vibes.
I find this "traitorous to the human species" interesting. You could not form these thoughts without an accumulated knowledge stored/accessed outside human biology. No other species forms such explicit thoughts because they lack access to that accumulated pattern of useful sequences. We've already "transcended" our biology in that sense. Who is directing... that becomes interesting in new ways once outside-a-body sequences start to accumulate. My guess is that it's that accumulation that sets the trajectory, just as DNA's "accumulation" of useful sequences produced forms fit for the local environment(s).
Sure, we're cyborgs, but still very much of the flesh, and I think we have to draw a line somewhere.
> Our biology is an innate part of us - to improve, to fix, sure, but not to cast off.
Right now, necessarily so.
The more mysteries that science solves, the more we unweave the rainbow*, the worse the tension between rationality and our sense of what we are. Am I the atoms, or the pattern betwixt? Am I the present moment alone, my memories mere shadows of past patterns and my current form doomed to be a shadow to a future pattern of these self-same atoms?
Sometimes even the question is unbearable.
* https://en.wikipedia.org/wiki/Unweaving_the_Rainbow
What are you talking about… You can use the word soul without believing in a religious concept of soul. It’s done all the time in secular societies. It’s even a very common term used to describe the problem with LLM generated content, that it lacks soul or human spirit, it lacks the human meaning, intent
It's rare. It's mostly, "but it will never be able to do X" (when it can already do X), or it uses too much water (while having no idea or reference point to what amount is much), or other climate things, or non-progressive bias, or copyright (when training is fair use) etc.
Just be upfront. 1) It threatens your livelihood that people can get the thing done faster and cheaper than what they used to pay you, 2) our relationship to an AI shaped entity is spiritually risky and a esthetically off putting, and we have to talk about golems and the tower of babel and stuff. If you think I'm being incoherent, you're in for a ride. Society will have to come to terms with major major issues, not just job replacement and bad slop art.
I think it is true that people who obviously dislike AI for ideological reasons often lead with ecological concerns or doubt about the capabilities, leading to an unproductive discussion.
Have you considered engaging on those ecological concerns? There is quite a lot to dig into and can be pretty interesting. For example, try to project how much electricity will be needed to power all the datacenters in construction, then compare that to the existing grid and infra, then figure out the cooling capacity. If done well you will see that we have a pretty massive resource problem here
I had a look now, and apparently we are projected to increase by 2.3x datacenter power consumption until 2030 from 415 TWh to 945 TWh.
That is going from 1.5% of global electricity use to 3%.
It is less than I expected.
Maybe if AI does work out better than expected for knowledge work or becomes applicable to robotics and fields like construction, we are going to see a larger buildout.
> That is going from 1.5% of global electricity use to 3%.
> It is less than I expected.
I suspect this is because of the word "global". Are not most of these currently planned specifically for the USA? Sure, the USA has a disproportionate use of electricity, but IIRC it's like 16% (1/6th) of global?
3% of global electricity is roughly the same as 18% of US electricity; a bit less because not all the new DCs will be in the US, but that's something that I consider with regards to the economics and why I'm bearish on this topic, as I think it will be a case of Dutch disease in the USA: https://en.wikipedia.org/wiki/Dutch_disease
Okay nice, after looking at the macroview look at the immediate impact into local communities and how much data these companies refuse to, rightfully I might add, share with the public.
Weird how all these concerns can simply go away if they just simply shared the information the public wants (water usage, sound generation, power usage) rather than refuse to do so.
So electric cars but bad
I don't think it ever gets old to restate that the singularity and all of the gradiose promises of a glorious 21st century have last mile problems.
But this post seems to mostly channel Harlan Ellison. And while he was amazing in his prime, with some absolutely legendary rants, he didn't age well in the end.
I do like his challenge to make LLMs useful to himself and to other authors though. Someone needs to make that happen and nearly exactly how he described it.
(Or is this just the Gell-Mann amnesia effect striking again?)
This is very interesting coming from an author in whose writing autonomous, super-powerful AIs have been a common theme.
Consider his book Accelerando (which I'll take the opportunity to plug again, especially as he's made available for free here: http://www.accelerando.org/fiction/accelerando/accelerando.h...) Not only did I find it quite engaging and thought-provoking, it is also proving rather prescient, and even helpful in decoding some of the things that are happening today.
For instance, in the very first chapter the protagonist spawns agents to go research something in the background and report back to him. And then a year ago, I randomly became curious about a rather involved topic (how fast could we feasibly replace all human labor with robotics), but I did not want to spend time researching so I outsourced it to Google Deep Research which churned away for almost half an hour and came back with a 30 page report with 49 citations via "actual internet searches for verifiable sources." (If you're curious about the conclusion: not for a very long time partly due to critical supply chain constraints.)
After I went through the report, it suddenly struck me: my agent may have executed in a GPU cloud instead of a cybernetic brain, but holy crap I had literally just lived a SciFi scene! A scene that I did not expect to experience in my lifetime!
Which is why I found TFA a bit unexpected. TFA says a few things that I would disagree with. Like, no, AI is not a "stochastic parrot" and I'd assume he'd be primed to realize it. And AI is not competing with authors -- other authors with AI are, and using AI trained on real text to aid writing has been a thing since the days of red squiggly lines in word processors, which TFA even acknowledges.
If you see his last few comments (https://news.ycombinator.com/user?id=cstross) it's clear he's pretty negative on the tech industry and Capitalism, which I tend to agree with. I can see how that could color his thinking.
That may also mean he's avoided LLMs to the extent that he is not aware what the frontier models have become. I would encourage him to put aside his distate and re-engage with them deeply; maybe he'd be at least a little bit excited to see some of his writing turn out to be prophetic in some good ways besides the bad.
Not being able to see the forest for the trees I suppose is a prerequisite for success in Silicon Valley, because it one had even a fraction of Stross's perspective on the matter they'd be in a different line of work.
Dude's an artist. He should do what his artistic intuition tells him to. I'm a software developer. I'll follow my own experience. It'll all work out.
For some value of "work out" anywhere between "human utopia that spreads across the stars" to "we destroy everything in a ball of nuclear fire and only microbes survive", yes, it will. The universe will keep on going either way.
"Whatever it is, it's progress, and we're doing it!"
The only way out is through. Unironically
Don't I know it.
He says exactly this in the comments on his page. This article is specifically about writing/art.
It’s likely it won’t all work out.
Extremely likely. Humanity won't work out. Time is running out and we still kill each other over money and religions.
I’m more optimistic about humanity but I doubt the west will work out in its current trajectory.
If you haven’t read The Collapse of Complex Societies by Joseph Tainter it’s worth a read. It’s not a conspiratorial or even outright pessimistic theory.
Ill check the book out as I am extremely interested in these kind of topics. Although I am extremely pessimistic.
You wrote "the west", what system will (in theory) work out? And how much does it value the individual?
I know capitalism for example, can in theory, never succeed as its base premise is built on endless growth (which simply wont work out with finite resources). It's a short term system that we need to improve on. All the other systems we had, like feudalism, I'd rather die than live in... Personally I'd highly prefer socialism, but that also wont work out as too many humans highly reject its concept and its also not safe from corruption.
Tainter suggests that societies collapse because the costs of maintaining complexity eventually outweigh the benefits. “Collapse” in this context is more like a simplification than a catastrophe. And he points out that for a lot of people, the collapse actually made life better for them, as it becomes more local and less totalitarian.
I mentioned the west because I’m a westerner with little experience of other systems, but I don’t think Tainter’s conception of a collapse would be limited to the west.
Interestingly since you edited your comment to talk about capitalism, I think under the lense of Tainter’s theory it’s the most resilient to collapse since it’s simpler, at least in theory. In practice it’s heavily regulated - in fact I’d say no country really is capitalist, at most you could say it’s “state capitalism” which is very different from free market capitalism. But you’re also not wrong in that it has issues of its own.
> Tainter suggests that societies collapse because the costs of maintaining complexity eventually outweigh the benefits. “Collapse” in this context is more like a simplification than a catastrophe. And he points out that for a lot of people, the collapse actually made life better for them, as it becomes more local and less totalitarian.
This premise sounds very intriguing. And yea I named capitalism just because I don't think we can go on like we do currently, and I'd agree that in theory it's not complex at all. In fact it makes a lot of sense (in the short term, and as long as people are willing to get exploited to be able to finance life).
Nonetheless maybe heavily regulated capitalism works, allthough, regulators are prone to corruption and act way too slow. And lobbyists preventing measurements to combat climate change as an example, I currently see as direct threat to my personal wellbeing. Hard to believe in a system that relies on exploitation, the big "if you work hard you can make it" lie and actively threatening ones or ones children's wellbeing by being ignorant to obvious and well researched topics for short term monetary enrichment...
Roughly a third of the market is digging a $1T-a-year-deep-hole with <5yr amortization schedule on all of it and passing around the same $100B like it's in all of their bank accounts simultaneously. Yeah, shit's going to work out for sure, in all of the same ways that flying a plane into the side of a mountain has a clear ending.
> It would be foolish to deny the effectiveness of image recognizers based on generalized adversarial networks (GANs), the key neural network technology underlying LLMs
I could be misreading this, but I hope the author doesn't think GANs are used in LLMs. They are cool, though.
It’s unfortunate - he’s an author, it would have been fine to stick with an authors perspective that LLMs can’t write (which is true), as well as the copyright stuff (which I don’t agree with but he certainly has standing to give an opinion on).
But he’s made the error of trying to come at it from a technical perspective, when he clearly knows nothing about that side of things, which discredits the rest.
> But he’s made the error of trying to come at it from a technical perspective
Huh? That quote appears to be the entirety of the "technical perspective" of the post and is an aside from his larger points that you have blessed as "fine". Literally nothing in the rest of the post relies on that incorrect statement.
Let me quibble with what is discredited here, given the entirety of your point is built upon an error.
Here's another technical point:
> They're word-association mechanisms with no embodiment and no way to associate the text vectors they manipulate with real-world phenomena.
This is wrong too. RLVR grounds foundational models in reality.
Aren't RLVR signals typically based off formal systems not natural phenomena? The formal sciences are certainly useful for producing tools used in natural science, but it's not entirely clear they alone are sufficient to associate text to natural (real-world) phenomena. Honestly, the pretraining and RLHF are probably more tied to the world than RLVR, for all that RLVR might be useful (maybe even more useful) for making the model perform better in certain tasks (such as working with formally defined systems).
Yeah, I should have said RL, not RLVR. The point is RL interacts with external world, while autoregressive pretraining is limited to the passive ingestion, and RLHF relies on a model of human preferences that has no access to truth sources besides the limited training data it was built upon.
If you want a more concrete example, then LLMs are also trained on visual data these days, which means they do have access to the world in an important way. This directly contradicts the blogpost's claim that LLMs have
> no way to associate the text vectors they manipulate with real-world phenomena.
Historically, that LLMs were text-only used to be a major argument for why they "lack access to meaning", see the Stochastic Parrot paper and the Octopus paper that it references. But even the authors of those papers have (grudgingly) conceded that the argument no longer holds due to multimodality.
How does a tiny error, in an aside and unrelated to the greater point, made at near the end discredit the rest of it?
A lot of people will stop reading or become completely distracted when encountering something so blatantly wrong, and it also indicates the author is willing to say such things. I see some people downplaying it, that’s fine, it’s clearly not helping the author make a case. Even im this discussion, the current top comment and a long thread are solely about that error (granted that’s not too unusual here)
Things that are vaguely GAN-shaped might play some role in post training. Though no one would call them GANs or identify them as the key underlying technology.
There is RLHF with a model that's trained to emulate a human evaluator, but the evaluator is not really trained jointly with the main model to adapt to its distribution and tell it from real text. Though I'm sure there are some niche cases when this is done. But definitely not a prominent thing.
He hallucinated. This is how you verify non AI nowadays.
AI is so good that if you see an hallucination as obviously wrong like this one it’s a sign it’s written by a human.
Yeah. Ironically, if he used an LLM to proofread his work, it would have told him that GAN's are generative not "generalized". That LLMs are primarily built on Transformers rather than GANs. And that they're more famous for image generation rather than image recognition.
Transformers are an architecture and GANs are a training method for the architecture. There are GAN Transformers.
Examples for us less educated folk?
Jiang et al. (2021) TransGAN: https://dl.acm.org/doi/10.5555/3540261.3541391
In general probably not much of a stretch to get rid of convolutions or recurrence by replacing with attention and see if it works hence the title of the original transformers paper.
Analogy "it's not GAN, it's transformer!" - - "it's not a recursive implementation, it's object oriented!" if you are more familiar with CS.
Or "it's not 4-wheel-drive, it's diesel!", if familiar with cars.
Do you believe it was this edge case he was referring to in making his point?
No, I simply think the person is not familiar with these concepts, which is not a crime, it's okay to be wrong, I didn't want to banish that person from here, just corrected the statement. But might have been also a brain fart. Happens. Just wanted to correct it in the interest of beginner reading this.
Once Musk said he believes in Transformers instead of diffusion (or the other way around). When actually diffusion transformers are very popular and mainstream. What he meant was autoregressive inference vs diffusion. People like to use buzzwords while not knowing them. Old issue, it was the same decades ago.
To make it even more confusing, there are also diffusion LLMs, which typically but not necessary, use transformers also.
And independently of diffusion and transformer and llm or image generator, you can optionally put a GAN discriminator adversarial loss on any of them.
And that GANs haven't been relevant in image generation since 2021.
sorry but I see incorrect AI responses almost every day in things I query on Google, generally because I am often querying on something I am an expert in and I just want a linking source. There will be a good link in the result, generally, but the AI almost always messes up some elemental fact, sometimes it is because the AI has "misunderstood" what I asked which is reasonable, but most of the time the result shows that it has "understood" what I said but there will be a statement of fact that is just wrong.
Of course but for obvious mistakes like this one it’s more likely a human hallucinated than an LLM
Yeah, it's hard to take the rest seriously.
maybe you're looking for an excuse to ignore what's being said.
Crazy how forward thinking this guy was in 2005 vs today.
Crazy to assume that an individual of vision has so thoroughly dropped the plot simply because their position is at odds with tech industry orthodoxy. If anything that should be an indication to more closely examine your assumptions.
Anyone that is against American big tech immediately makes me sympathetic to their side. They'd have to do a lot wrong to destroy such good will.
I just left this comment: https://news.ycombinator.com/item?id=49137863 -- tl;dr I suspect it's his view of the Tech industry and Capitalism that is coloring his views, rather than an objective evaluation of state-of-the-art LLM technology.
> I'd quite like a tool (running entirely locally on my own hardware, with no cloud service and no copyright-thieving grifters making bank on it via subscription fees) that digests a manuscript and derives a scene-by-scene timeline, that I could then query interactively and use to plan my next round of edits. Being able to map out where and when each protagonist and minor character shows up, and see a frequency distribution heat map of names in the manuscript, would be useful.
Interestingly enough, I do this with various books I'm reading. E.g, I'm currently rereading Accelerando and had Claude generate a wiki-like timeline of key events, characters, and salient plot points. That makes it easier to jump around when I want to re-read a section and grok a plot thread that is scattered across chapters. Ironically, it also exposes inconsistencies (or "hallucinations" as some might call them) in the text because the author didn't have an AI proofread the text.
You could build a tool like that and on a MBP with 48+ GB of RAM all of that can happen locally in terms of keeping your content locally. I am sort of an outliner and planner and I heavily use AI when writing documents for work. I don't see why fiction / prose would be any different? Indexing your content with a vector RAG and having a little 27B model locally could do everything he wants with some help from a big model to implement it all. It sounds like a weekend of coding for a functional prototype to me.
He doesn't want to use a model trained on someone else's text. So he would need to train a model from scratch. I don't know whether his own texts would provide enough data.
Oh… definitely not enough text. LLMs don’t work like that. Hmm.
> Ironically, it also exposes inconsistencies (or "hallucinations" as some might call them) in the text because the author didn't have an AI proofread the text.
That's not irony, or even cute. He's literally describing a tool he'd like to use to plan edits to his writing.
Are you able to share? Both your process and the specific timeline for Accelerando. I would love to build one for A Fire Upon the Deep
I'm not going to share because https://www.antipope.org/charlie/blog-static/fiction/acceler... has a Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. But I've paid for my copy of the book so I can do what I want with it.
If you want to build your own, you can feed this DESIGN.md to your favorite clankie: https://pastebin.com/huRkGnba
Speaking as a fucker, I think the use of em-dashes has been pretentious since the 1890s. The rise of typewriters destroyed em-dashes already. The LLMs subsequently created an accidental parody of sophisticated writing.
What's pretentious about them? They express a distinctive rhythm in written English. What's the the punctuation you would use instead?
It's like saying I am intimately involved in the printer's art. This is usually not true, unless you're involved with a very niche publisher who still has a box of sorts and inky fingers. Hyphen-minus all the way.
No it doesn't. I happened to have read "design and typography in easy steps" as a (bored) child of the 90s, but that's the extent of my "printer's art" knowledge.
I don't understand the celebration of ignorance myself. There are lots of things I don't know how to do or how to use, but I'd never dream of complaining because somebody else does.
Read their original message again--you are debating with a self-proclaimed "fucker." ;)
I wasn't sure what his sex life had to do with typography.
How do you feel about a fleuron dinkus?
It's probably in the book, but I don't remember. This might well be my cue to read it a third time.
People seem to LOVE parens, although I was taught that they are almost 100% never the correct thing to use in prose. But apparently "some fucker" will accuse me of being pretentious ¯\_(ツ)_/¯
Otherwise, I see people just use a regular dash a lot. I love the em dash myself, but of course stopped using it so I won't get accused of AI (which is sad). It's easy to type, at least on a Mac.
Parens and em-dashes do different things. A simple parenthetical is an secondary aside (it's a half step towards a footnote). An interrupting phrase is intended to (abruptly) command attention.
I mean, make the case em-dashes are overused and used lazily, I'm right there with you. I just don't see the pretense.
I understand that, I'm saying that people mis-use them when they mean to use an em-dash. I think I ass-u-me'd a bit too much in my comment.
It kinda looks like we can only train AI on dead people's data.
I wouldn't be too upset about that. They write better anyway.
While Stross is currently a neo-Luddite who doesn't believe in the Singularity, his book Accelerando is probably the best work of Singularitarian fiction ever written. It was written in the 2000s, starts in the 2010s, and covers the various decades of this century. I highly recommend it for anyone who wants to know how weird this century will be. It's also available for free online. [1]
It's one of my favorite books and the main reason I'm e/acc and many other pro-AI people love it because we view it as utopian sci-fi rather than the dystopian world Stross invented. This might be the best case of the Torment Nexus meme [2] in action.
1. https://www.antipope.org/charlie/blog-static/fiction/acceler...
2. https://en.wikipedia.org/wiki/Torment_Nexus
> This might be the best case of the Torment Nexus meme in action.
At least the author seems to think[1] it is one, if in not so many words:
> And don't ever let anyone tell you that Accelerando is techno-optimistic or pro-AI; by the end of the book our entire species is extinct, surviving only as simulations/memories recalled by something arguably not alive.
[1] https://news.ycombinator.com/item?id=48163630
1) Having met Stross, and briefly spoken to him way back when, he never actually believed in the singularity as a possible future.
2) Turns out it was the sci-fi author who was wrong all along! Fascinating to see someone confidently citing a meme, while actively evidencing the fallacy it exposes.
>he never actually believed in the singularity as a possible future.
I think my favorite moment in the book is when one of the characters (at the time, a digitized consciousness traveling at near light speed in a space ship the size of a coke can to explore an anomalous signal sent by alien life) argues that the singularity is a ridiculous concept and will never happen. I always thought that was a really clever observation on the part of Stross that our perspectives are so narrow and provincial, we often can’t see what’s happening in front of our own faces. Your comment does kind of recontextualize that for me and I sort of wonder if that was the author speaking directly to the audience?
It could be either or both. He comments here sometimes, perhaps he will say. Or not, there's likely too many AI-bros in this particular thread.
https://news.ycombinator.com/threads?id=cstross
You think accelerando is utopian sci fi!? Which characters perspective are you identifying with so strongly? They all have pretty severe societal challenges that makes me surprised to see the word Utopia.
Manfred and Amber’s lives don’t seem particularly dystopian. Human society, up until the Field Circus encounters the router and later returns home, seems pretty great. Chaotic and challenging, yes, sad for Microsoft shareholders, sure, but I would certainly rather live in a world with access to the technology they have than our own. Space exploration, extended life spans and rejuvenation, new methods of relating more closely with other human beings are all things I would love to experience.
The protagonist who believes that information should be free and that scarcity is mostly artificial due to bad economic models. And that it would be possible to implement proper pro-human capitalism or communism if we had enough compute to solve standard CS problems like filling knapsacks or maximizing flow through a graph.
> a neo-Luddite who doesn't believe in the Singularity
You’ve lost the plot. There is no singularity, your machine-god is not coming to save you.
Calling people Luddites for not believing in the Singularity is like calling people communists for supporting affordable health care.
> In the background of what looks like a Panglossian techno-optimist novel, horrible things are happening. Most of humanity is wiped out, then arbitrarily resurrected in mutilated form by the Vile Offspring. Capitalism eats everything then the logic of competition pushes it so far that merely human entities can no longer compete; we're a fat, slow-moving, tasty resource – like the dodo. Our narrative perspective, Aineko, is not a talking cat: it's a vastly superintelligent AI, coolly calculating, that has worked out that human beings are more easily manipulated if they think they're dealing with a furry toy. The cat body is a sock puppet wielded by an abusive monster. https://www.antipope.org/charlie/blog-static/2013/05/crib-sh...
Regardless of believing in the “singularity” or not, please explain how this sounds utopian in the slightest without borderline (if not literal) sociopathic anti-human worldviews.
I don't find it to be utopian, but in trying to imagine how, it occurred to me that maybe the idea that we even survive to be entangled in the wider universal future, maybe that is utopian? To matter.
I think there are a lot of paths where we just rock ourselves back to a regressed agency societal state, and just never have the capacity to have any affect on the wider universe. Which is maybe good for the universe, as we are certainly not a role-model for certain forms of balance. But maybe the balanced view doesn't seek to dominate and influence and become ubiquitous, after all
Misreading SF is super popular in the techbro milieu.
> Regardless of believing in the “singularity” or not, please explain how this sounds utopian in the slightest without borderline (if not literal) sociopathic anti-human worldviews.
There was a guy on here the other day trying to argue that Brave New World wasn't a dystopian novel. There are some absolutely wiiild world views out there.
Pretty much everyone in Brave New World is happy.
You might not like the society they've built, but there are certainly arguments that a society where most people are happy is pretty utopian.
(It's very debatable, of course, but it's not instantly dismissable)
> There was a guy on here the other day trying to argue that Brave New World wasn't a dystopian novel. There are some absolutely wiiild world views out there.
I think that's likely to be all the sex and drugs? Though I'm basing that on the Wikipedia plot summary. I have no interest in darkening my day by actually reading any work that can be rightly described as "dystopian".
I'm almost 50, but I absolutely could see how someone younger could view it that way -- precisely because the present is much more similar to than novel than it used to be?
Like, "cautionary" but not full on "dystopian."
He had already disappointed me in 2011 with this post
https://www.antipope.org/charlie/blog-static/2011/06/reality...
Why is that disappointing? Everything he said there seems perfectly reasonable.
A large part of that goes via arguing about consciousness.
Starts of with a quite defensible:
But the moment he hits that (parenthetical) paragraph about consciousness, he blends human capabilities with human-like consciousness.If human-like capabilities required human-like consciousness, early models (like GPT-4) passing all those exams would suggest those models being "as conscious as" a university student. (My inner Douglas Adams wants to invert this, but the best I can do is "The Xarloxians did not consider university students to be conscious beings; their reasoning being…", which you may recognise as a riff on Babel fish and the non-existence of God).
Belief in the Singularity and belief in God strike me as extremely similar.
Both, I think, are extremely interesting ideas where very few theories, e.g. pro, anti, or anywhere in between, can be written off as ridiculous -- probably due to a lot of ambiguity and difficulty in defining proof.
For me, I find the Singularity very implausible due to the fact that we talk about "general intelligence" quite a bit, and the more we do, I'm finding that I don't think it exists. There are a set of low-level basic skills that I think -- if you don't have, you're not intelligent. And then there are higher level skills that vary wildly.
So, will we make AIs that can say and do very clever and amazing things; yes, this is happening. Will they be, like ascendant, or something? I would say -- yes, but perhaps not in the way we think? Like, humans made "The Bible." And that's obviously a big huge, impactful, maybe superhuman thing. But feels different from this idea of a "Singularity."
There's a reason the Singularty was called "the Rapture for nerds" before LLMs were even a thing.
You can find a lot of neo-religious behavior within the AI accelerationist community. Cults coming out of rationalism and fear of the wrath of an angry AI God, making the world right for AI's coming (and even killing the nonbelievers.) Evangelism. Promises of infinite plenty in a post-scarcity heaven. Even a persecution complex - "Luddites" are increasingly demonized, hated, accused of moral depravity for their doubt, and for not abandoning the world, taking up their prompt and following the AI.
How many people post about how AI seems to be draining them of meaning and purpose, that it doesn't live up to the hype, that keeping up with the rituals is exhausting, but feeling that the problem must be with them and not AI? That's a religious person having doubts about the nature of their faith, unable to accept that the God they believe in might not be real, and framing their issues as their own moral failing.
And all of this before the "Singularity" even shows up, and people upload their minds - I mean, meet God in the air.
It's religion, very specifically neo-Christian ideologically. Regardless of any arguments one might make about the practical benefits of using AI in any particular field, it is obviously also religion.