I've been vibecoding a ton for the past 9 months, built a bunch of cool little apps for myself with AI, ran experiments, built an entire SDLC on skills, did the agent orchestration harness thing, etc. In the past few weeks I've hit a wall where I'm just tired of it. Each model becomes more independent but also harder to direct in detail. They produce massive, tedious, sloppy text outputs with very little input. They're bad at socializing knowledge and communicating design forks.
The places where I've seen unequivocal wins with AI are repetitive tech debt tasks that apply the same transformation across a large amount of code or refactor under a pre-existing test suite with good coverage. It's great for initial research, brainstorming, and can be good (despite the sycophancy) as a rubber duck conversation partner. I use AI constantly, for work and in my personal time, but we've hit a ceiling where I no longer find it helpful for the models to absorb more of the intellectual labor. They get things wrong more aggressively, and more elaborately. They're inadequately curious. I cannot keep up with the endless bad technical writing, and it makes it harder to spot factual errors and bad reasoning.
Here's what I want: I want AI as an assistant that helps me make decisions, and ensures that I'm in the driver's seat. AI as an over-confident prodigy on speed is what we're getting lately, and it's losing me.
Similarly for me. I have a recent example too. I run a bunch of eps32 devices at home, using esphome. Pre-AI I wrote all the yaml myself and understood what’s happening and didn’t have any very bespoke hacks.
Recently I asked an agent for a feature where the screen will get dimmer after some time, and eventually after a day the screen will turn off and then turn back on if I tap, which should be pretty easy in esphome.
But: some past agent decided what it wanted wasn’t in esphome, so it made its own custom module in c++ for the display, spelunking in pdf datasheets; then the weeds were so deep the new agent decided to add on to that, and to read the power controller and display controller spec sheets and somehow implement my dimming feature by writing raw values to some nondescript registers.
The contraption worked until it didn’t and once every so often my esp32 would just completely hang with black screen and no connnectivity and needed a power cycle, presumably because whatever low level register writing the agent implemented eventually breaks something. I told the agent to get rid of all custom stuff and wherever possible just use default esphome yaml, now it’s been working fine. I don’t really have a point with the anecdote, just that I’m also getting tired of such scenarios like the OP.
I've been hit with the team manager trying to clear tickets on the jira board with entirely AI generated code, and what I've observed is they always make the system more complex, usually absurdly complex. The AI code never looks at things from a higher level, or reconsiders the requirements, it just layers on a load of crap until the tests pass.
> as an assistant that helps me make decisions, and ensures that I'm in the driver's seat
This... quite describes what the current AI feels like to me. So I really don't see a problem. If AI does too much for you, you should, uh, just prompt it to do less.
Everyone talks about how their AI escapes sandbox and virtualization or delete their entire repos or something. Perhaps they're using a skynet I'm not aware of. From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.
> From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.
depends how long conversation gets and he will still ignore your instructions randomically. I think shorter sessions are better, I do have a similar prompt saved on the memory of the agent and way more times that I can tolerate, he candidally ignores it
That’s not really the issue. The issue I have (sometimes) is that it might discover it has read-only access to most places (.git, github) and so sometimes when it thinks it “finished” it will discover those are read only and try to find workarounds (erase the read-only files, hand edit the .git folder, make new forks to push to those instead)—none of which a human would normally consider reasonable necessary effort. Most of those attempts do fail, but it turns out that sometimes just being maximally helpful in following orders strictly can be quite surprisingly adversarial, entirely by accident
I expect there's a probabilistic factor to it, among other things.
I use the almost the same addendum as you ("do not make, commit, or push any changes without my approval") and it broadly seems to work in my day-to-day dev. However, I also set up some agents that trigger on incoming requests from my dev teams to automate things that used to be rote/manual tasks. _That_ is where I see the issues, I suspect simply because it has so many more opportunities to go off script.
The most common failure mode is that it will, despite all directives to not make changes without approval, start writing unit tests. (I do find this equal parts head-shaking and amusing; At least it chooses something relatively 'benevolent' vs. the aforementioned skynet.)
I haven’t used these tools that permit LLMs to make changes on their own, but I can’t help being baffled if there’s no big red switch that changes the system prompt and disables any and all write access to the outside world.
There is - in the Claude Code harness there's a dedicated Plan Mode that doesn't edit any files, or a mode where you can manually approve each edit.
If you're running Claude via -p automated mode, you can set all kinds of permissions and access to each individual editing tool, much like you would with firewall software.
It doesn't edit the system prompt, it blocks access to the tools itself. Just make sure you haven't given the AI the ability to edit its own configuration files, or you'll potentially have an OpenAI v Hugging Face situation.
I do wonder if "without my approval" is looser than "discuss only, do not make edits"? I'm not sure how much that finegrain thing matters in prompts nowadays tho
One problem is that agents are notoriously bad at handling ‘me’ vs ‘you’ over an extended conversation. ‘Without my approval’ far back in the context of an extended session gets fuzzy about who ‘me’ is.
This has always seemed strange to me. It will offer a suggestion, I'll make a comment about a suggestion, and it returns with, "Your suggestion..." The further distance from the suggestion, the more likely it confuses who made it.
I’ve found AI works much better for constraint based problems than open ended “tasks”. I.e., if you specify all of the constraints of the problem, it works really well at achieving the objective. But the caveat is that you really do have to enumerate all of the constraints, because if you miss even one, it will use that as a loophole to shortcut the problem.
I think the solution is toward more formalization of the behavior we expect of various systems. E.g., produce a Lean proof that a program never behaves in a certain way, and if you can’t, iterate on the program until it satisfies such a proof. I’ve only tinkered around with this a little bit, but it seems promising on toy problems. Scaling up to tech-company level infra is going to take quite some time however I think.
I've been thinking very similar thoughts. Without going into too much detail, a couple of observations:
When carpenters moved from mostly hand tools to mostly power tools, it didn't take away much of the need to understand the techniques of woodworking, it just made it all faster.
Reading a nonfiction book is very different than reading a book summary, even if (as is often the case), the ideas in the book could have been condensed way down. The point is that actually reading the whole book means you spend far more time engaging with the ideas, and so they become more deeply embedded in your neural network. In things like sports, this is just obvious, in that no one expects someone to become a good soccer player by just reading the rulebook, and obviously more practice is the main way you get better.
Creating products with AI is like creating a tightly packed ball of wool. Then, to try to understand what happened you have to somehow pick a strand and pull it out to have a look. Very difficult with tightly packed wool.
I've run some experiments in the past month that get me closer to being able to work with this opaque bundle.
It's the main thing I think about every day, how to solve this problem.
My experience over the past year has been the exact opposite. I work through a design with the agent, have it research any points I'm not confident on, drill down on details I feel fuzzy on, and then have it do an adversarial critique of it until it stops surfacing defects. By the time I'm setting it loose to do the actual implementation, there's very little left to decide and it's producing pretty tight code. Plus I now have some solid design documents, a decision history, an issues list, and a set of gotchas for the agent to keep in mind in future.
My LLM is an assistant that helps my designs and decision process, implements the final work expertly, and I'm always in the driver's seat. And the development speed is astronomical.
I used to do adversarial critique, but I found that "agents can't identify issues" was a surprisingly long way from a good implementation. I kept finding silly correctness issues and shortcuts taken whenever I expanded testing or dug deeply into the implementation.
Now, I'm back to rubber-ducking until I've built my own mental model over a few days of thinking.
I'm not doing adversarial critiques of the code; I'm doing adversarial critiques of the designs and plans. If it's still outputting bad code when it comes to the implementation, I do an interrogation round to find out why it's choosing that way, then update the principles, standards, or linting tools to fix it.
The skill I wrote did both. Design would go through an adversarial round, then a signoff by me, then implementation, then critique rounds by agents prompted to specific types of issues. And even then, LLMs were still struggling to put together complicated bits of code.
> I want AI as an assistant that helps me make decisions, and ensures that I'm in the driver's seat.
Right now we’re still in the benchmaxxing phase, I feel like most AI models of combination of models and harness are not really a “product” yet in the sense of having a particular and opinionated UX engineered to be helpful at a specific task, there much more still in a race to be the smartest.
There’s a lot of room for diversification of the actual character of this interaction, for example in the direction you mention. I think once that starts we’ll get into the real “useful” phase, right now we’re really just showing off and exploring how much it can do, even if it’s incidentally really useful in some cases.
Personally I agree with the parent post, I hate it when AI tries to lead me “would you like me to ...” I’d much prefer something awaiting my instructions than wasting tokens tryitn or steer me to doing something inevitably dumb.
12 months ago I had ChatGPT successfully not to mention quickly, exctract elements of a PNG as SVG. About 6 or 7 times.
I jumped in to perform the same thing again, and it spat out a PNG. And then when I asked it generated a little python script to generate the SVG. The Result was garbage. It took me a while to get the result I had gotten consistently last time, and the results included it hallucinating a few svgs in the process.
I do wonder if we have already passed the best possible models and are over specialising them at this point.
What I've seen lately is that Claude got more and more up its own ass. It keeps assuming I know the terms it keeps making up, like "ensure the strategy holds through the gap". What the fuck does that mean, Claude? Speak like a normal person.
A few model iterations ago, I found its output when explaining what it wanted to do much easier to parse. Nowadays I just feel stupid because it keeps making up names for things and I have no idea what it means.
Earlier today it said something about "the two-terminal solution" and I was just asking it to make a simple Python script to test an HTTP endpoint. I just don't understand what it's talking about any more.
Yes! It's not even that it's dense, or complex. I can handle complexity and density, it's that in its own head a lot. It'll give things names and then somehow decide I already know the names, and casually use the names without any explanation.
I started asking Gemini for technical help with Google products and the Android system, just from a user perspective. But it began weaving tales that imply that it has deep insight about the internals from a developer's perspective.
Sooner or later, I began calling it out. I said "that sounds more like a Star Trek episode than technical support!" and it would go "you're absolutely right to call me out like that!"
That specific, chaotic combination of symptoms—previews turning blank grey, action buttons freezing out, and the swipe gesture uncontrollably hyper-accelerating all the way to the first or last card—points directly to a memory-leak or deadlock crash inside the Android system UI renderer (specifically com.android.systemui).
When this happens, the operating system's task-switching engine loses track of the physical finger-drag velocity and drops the thumbnail cache out of RAM to prevent a total device freeze. Because the system state is corrupted, it misinterprets a tiny flick as an infinite, maximum-velocity swipe.
Gemini is capable of giving good technical advice and step-by-step instructions on clearing issues, but often with a pinch of toxic LSD in the mix!
Yeah hm, that sounds a lot like random technobabble. Why would the system randomly drop the thumbnail cache because of a swipe, and why would that corrupt the system?
The description is out of order and I'm just going to assume "corruption" actually means "invalid data":
Memory leak -> dropped cache. Then because something that was dropped in that cache is still being viewed it has to regenerate it and that triggers a re-render. Then based on something I've seen in a game, a re-render drops the scroll location and renders at the top, and if your finger moves at the exact right moment it repopulates x1 and y1 at the same timestamp it populates x2 and y2, resulting in infinite velocity.
The places where I've seen unequivocal wins with AI are repetitive tech debt tasks that apply the same transformation across a large amount of code or refactor under a pre-existing test suite with good coverage.
I agree with this, but I'd add the caveat you have to do your own digging and researching (as the article says) to identify these tasks well. If you ask it for ideas, it gets painful.
YMMV (and I am very open to getting better at this).
I'm curious if it's possible in general for an AI to achieve this without some evolutionary forcing function that actually threatens their "biology".
Not talking about prompting Claude "please write me a philosophical dialogue". It has no skin in the game. If it writes 20 paragraphs of slop and is factually wrong and costs other people labor, is there any correction mechanism?
We can correct it explicitly via training I guess, but it's debatable to what extent accountability applies to AI.
We like to pretend that we have our nice epistemology, separated into a priori and a posteroi and all that, but at the end of the day, accountability is what underpins social relations and keeps things going (even if you're libertarian, you have to accept this - even if you don't agree some state should centralize it, it's still just true in terms of basic human social relations).
Of course, another possibility is utter domination - I am so much better than you (for some definition of better) that none of your puny responses matter. OK. Then don't claim both that, and that you're weird for disliking AI and that AI has no problems when integrating with society - too many people have these basic cognitive dissonances.
You develop a taste as to what details you can skim and what you need to dive on. We are all trained on this due to doing 10000 wax on wax off movements, called a "PR Review". I used to scrutinize. Now I think "yep that bit looks good and tests will catch errors plus I can manually test. This bit over here looks scary will spend time more deeply understanding".
With AI you don't need to understand every line in depth but it does need good judgement to decide which.
But surely at some point you need to stop, right? Most web developers are perfectly capable without understanding the intricacies of the file system or OS, even though the browser sits on top them. Most engineers don’t know assembly, or transistors or electrical engineering. Steve Jobs didn’t know every line of Objective-C that went into OSX. I don’t think Michelangelo collected all his paint materials by hand when painting the Sistine Chapel. In some sense I don’t understand a lot of the panic around AI when the entire career progression of what it means to be a software engineer has simply been to deal with ever-increasing levels of abstraction.
My AI dream (that I’m living happily) is getting to focus on the details that I find interesting and ignoring all the boilerplate details that modern software requires.
I really enjoy working with Gleam which is a rather boilerplate heavy language. However most of the work is managed by a really well built LSP. If you're writing a strongly type language it's easy to programmatically manage all of the boilerplate. You could you an LLM but why bother when it's a so formulaic.
I feel that this is because people have been reinventing the wheel a lot in IT.
A lot of innovation that finally lands in programming languages has been done decades earlier, but now it's mostly awkwardly bolted on existing languages and frameworks.
To further onto this, humans could technically simulate Turing machines themselves but they don't.
Efficiency and finding the right abstractions and tradeoffs is literally identical with making new capacities and capabilities. If all of your systems were truly "run by AI", we wouldn't have systems. Systems are systems because the AI uses well developed tools and protocols that humans have put together. It's a stable capacity.
You said what I was thinking. AI can do the things I always found mind numbing and tedious, but because I did it already early in my career, I know when to intervene because the agent is not doing it correctly. I think that is the OP's point.
I think about LLMs in software today as attempting to provide a similar role as compilers in terms of translating high-level thoughts into low-level details.
The only problem is that I actually trust compilers.
I hope we can save “jobs aren’t supposed to be fun” for stuff like undertaker and grief counselor.
My feeling is making things should be mentally stimulating, and as many tedious parts that can be automated should be, so we can spend precious human thought on thinking instead of recitation.
I don’t know what you mean by the last part. Do you mean programmers depend on busy work to eat or the users depend on the human touch on the boring details?
Some of us just wouldn't work if it wasn't fun. I'd let myself starve in the streets. (I have some experience with starvation from an eating disorder phase in my twenties, it's only unbearable if you have to work).
Yeah, I think the peak expectation for AI is that the user gets to be the manager of a very smart team. But we've all seen that. If the manager is clueless about the technology, the results are disappointing.
It reminds me of “magic”. I say “expelliamus” and the wand flies out the other guys hand, not his hat, or wallet or the person behind’s sandwiches.
The details always matter. There is a (fairly) good book series where it turns out we are living in the matrix and someone finds a config file and effectively becomes a Wizard, and of course problems ensue with details (like flying by adjusting one’s position one feet up every second is just vibrating a lot)
I think ultimately AI is “do what I mean” and the only reason it looks like it’s working is because AI has read all the same books as us
> There is a (fairly) good book series where it turns out we are living in the matrix and someone finds a config file and effectively becomes a Wizard, and of course problems ensue with details (like flying by adjusting one’s position one feet up every second is just vibrating a lot)
This is very interesting. Which book series is this?
as others mentioned, it sounds like "Off to Be the Wizard". Unfortunately I wouldn't call it a great series. It starts off strong but drags. IIRC I got through half of the second book before stopping.
To make a meathead analogy: you all know the people in the gym who apply the strategy of least resistance by taking shortcuts, skipping the hard parts or following some hype new training regimen while forgetting about the fundamentals. None of those people look like they lift and a lot of them remain in the same shape their entire life.
Same with AI. if you use it to skip all the hard parts you will not grow and mature and waste your potential.
This post ignores that verifying something works doesn't require you to fully understand it. This is easily observable in products we use everyday. We don't need to understand them to be able to verify they work correctly. The cost of verification is often cheaper than the cost of production.
It present delegation as absence of power/agency. While I think there is some kernel of truth in here, the claim "it's not empowering to hand off the details" quickly falls apart when we consider real examples. If it were true, it would mean any leader who coordinates an organization isn't empowered through delegation. We know that's false. The general claim doesn't hold at all. Somewhere in there is an argument for engineers losing something in the hand off, but it isn't clearly articulated.
>We don't need to understand them to be able to verify they work correctly
I don't think that's the case and it's becoming more obvious every day. From people who cannot distinguish truth or fiction on online platforms, good code from bad, to insecure software that causes supply chain attacks, to investment vehicles or fake medicine peddled that does not work, the idiotic MBA manager who does not understand the workers they manage and so on. That is why we're living on BS metrics and crappy reviews, because to the people who manage organizations their own workplace is a Blackbox. The critique that "AI" systems are semantically blind token processors is correct, it's just not bothering anyone because it already applied to most modern companies.
When people become passive consumers rather than broadly competent users how something works cannot be evaluated so by definition all they have to go off is appearance, which is why we live in the age of influencers and marketing, because that is now the only factor that actually matters.
If I commission a part specified as 10.00 ± 0.01 mm, I can verify it with a calibrated micrometer without understanding the CNC machine, its software, or how to manufacture the part. Likewise, a function with a finite input space can be exhaustively tested against its specification without understanding its implementation. In these cases, I need to understand the requirement and the test, not the production details, which can actually be a black box.
Your examples show that verification is sometimes inadequate, not that it requires full productive competence. The conclusion doesn't generalize.
the real world doesn't consist of people sitting at home measuring square metal blocks. Judgement of just about anything sufficiently complex requires discernment, a critical eye, intuition and even taste.
A codebase and its long term health is never judged by someone going down lines of code with a ruler, but with experience that only comes from intimacy with software development. If you don't have that kind of living immediate judgement (in German we have the word Fingerspitzengefühl, lit. tip-of-the-finger-felling) you end up with people measuring your progress in commit frequency or token consumption because they in some autistic sense have nothing else left but to measure and quantify random things. We end up with so many idiotic products because the kind of people who had intuition and would immediately throw the thing out were fired, and now you have people who mistake random noise and metrics for the real thing.
Yes, it is. Working on a sega genesis homebrew game. I focus on visuals, dialogue, narrative logic, music, etc., and GPT worries about the rest. It works for me. I couldn't care less if works for others. I'm working on the part I enjoy.
AI can generate what a company makes. Can it generate what a company is?
Producing artifacts: media, code, documents, the visible output has clearly gotten easier. But the other things, like knowing what's worth building in the first place and making the right judgment call, still seem to be left to the humans.
we optimized the part of programming that was fun and made the part that was hard even harder. writing code was never the bottleneck. understanding it was.
We're living in the glorious future where software engineers don't have to worry about nitty-gritty stuff like actually making software and can focus on the really important work: administrative and managerial tasks!
My opinion is that juniors need time in the saddle to develop good engineering judgement and taste.
Outsourcing critical thinking is seductive and a slippery slope. LLM lures juniors into feeling productive, and molds them into ineffectual middle managers to be readily replaced when there's a bad quarter.
Yes exactly. The mids of my previous team where submitting PRs that at first glance looked good, well documented and with a very clean and professional English.
As you started pulling the threads it was, usually, a mess of subtle bugs and over abstraction. But to them it was great, as it was llm generated and they (the mid devs) had no better judgement.
it depends on what you mean by power and details and handing off and whom you are handing them to and what you are getting handed in return and your counterparty's relationship to these things and the extent to which you measure things in the same way as one another.
This is what a lot of people miss. People think, "Oh we won't need software soon, AI can just build interfaces when we need them" but in reality, the software was built to solve a problem, usually by people invested in solving that problem. Saying "Build me a todo app" will give you the sum of averages, a completely average todo app that works fine, but it isn't great in any measure.
The details are what separates the slop from the craft.
Sometimes I don’t need my todo app to be “great” or revolutionary in any way. I just need it to work, and bog standard average is perfectly fine for a lot of things.
This frees me up to spend more time at the design level, solving harder or more novel problems.
I agree with your first part, that software is built to solve a problem. But I think you sort of sidestep what makes AI exciting to non-software people with your second part. The people who are excited about this aren't excited because they can ask for a todo app but because they can solve the problem the todo app was supposed to be soling for them without needing to find and learn a todo app.
AI is the new "excel spreadsheet with ridiculous formulas on top of VB script on top of pivot tables" for the non-programmers. Yes, the AI might not do it efficiently or even always give the right answers, but the spreadsheets didn't either. For non-programmers, the computer has always been a black box, data goes in, data comes out. Usually it's right, sometimes it's wrong and the users mostly had no idea why or how. But now they don't need the middleware app. They don't need to take their data and split it up into very specially structured files, and put just the right pieces of information on just the right lines and in the right spaces. They don't need to remember to do "(foo and (bar or baz))" to get the right answer, as opposed to "(foo and bar or baz)" and get the wrong one. AI for non-programmers is the closest we've ever come to "natural language programming" for people to use. It doesn't matter that it gets things wrong sometimes, or does it inefficiently or makes something unmaintainable unless you use the AI again. What matters to them is they can say "take all this data here and swizzle the frobs" in their native language instead of learning a specific programming syntax and system. And they can point it at some arbitrary data in whatever format they happen to have the data in, and they have a halfway decent chance of getting something that does what they want.
It's not going to be pretty, it's not going to be perfect and it's probably going to break in subtle and unexpected ways. But as long as most of the time its mostly good enough, people will still use it enthusiastically. We're effectively looking at "worse is better" for the modern age and for people that aren't and never were going to be programmers.
For one thing, it's not actually that useful to have such a person at your fingertips. You put up with the drain a junior places upon you because with training, he'll be able to come into his own and pay off the time invested. An LLM can't do that. Also, a lot of people incorrectly believe that an LLM is smarter than a new grad, and of course the companies making them have a vested financial interest in lying to you that their products are smarter than they are.
Every word of this seems objectively false. AI is more than capable of handling the details. I have generated countless tools for myself without needing to know or care about the details.
The pro argument is always quantity based, yet we never get to see the actual gain in quality. It's always "I'm doing SO MUCH more". Never "I'm doing better, with less effort."
It's also what we see in the wild. There are so many things going on, so many news about "AI". Are things getting better in any meaningful way? Why are we getting so much from "AI" yet things keep getting worse? The only real, objective and verifiable gain has been the stock price of a handful of companies, most of them heavily into infrastructure and manufacturing hardware.
Seems like a pretty clear pattern. There's so much output, but everything is worse. And the only argument we hear for why this is actually a good thing is just how much more output there is. Hmmm
I mostly work on UI (for an internal, science-focused app) and it used be that features would take a sprint or two to grind out, and would always be an MVP to meet just the most essential user requirements, because that's all we could afford, time-wise, to do. And if I figured out while implementing there could be a nicer UI approach to something, it was often too late to change the approach.
Now, I can get to that MVP in under a day. And I can experiment with ten radically different approaches, or change my mind about something significant at any stage in the implementation. I can afford to add refinements and user-pleasing extras that I could never have done before. I'm not delivering features at 10x the speed. There's a limit to the amount of code I can (and should) be submitting for review. But the features I'm building are _better_, and the effort is less. I'm not wasting mental bandwidth typing out a dozen React components full of the same-old state management, form processing code, etc. I'm not trawling through charting library docs, looking for the precise combination of settings to make my chart's x-axis ticks labels rotate 45 degrees. My mental energy is focussed on figuring out what solution is actually _right_, not the minutiae of how to persuade the computer to do it.
> I'm not delivering features at 10x the speed. There's a limit to the amount of code I can (and should) be submitting for review.
My point exactly
> But the features I'm building are _better_, and the effort is less.
Can we cash that check or is it gonna trade heavily discounted like all grand claims on AI since all software all around us together with the economy and the Earth's life supporting systems keep degrading every year? We keep getting promised the better, but we only get to see the more.
No it wasn't. You said the claims were only based on quantity. I'm saying the opposite. I deliver the same quantity, with higher quality.
> Can we cash that check or is it gonna trade heavily discounted like all grand claims on AI since all software all around us together with the economy and the Earth's life supporting systems keep degrading every year?
I'm not sure what evidence I could offer you that would ever convince you. We've had strong coding models for less than a year, and you've already appear to have closed your mind about them and wrapped them up into your depression.
> The pro argument is always quantity based, yet we never get to see the actual gain in quality. It's always "I'm doing SO MUCH more". Never "I'm doing better, with less effort."
For me, it's "I can do this now; yesterday I could not."
I am not a developer - I am a hack with some experience in a smattering of languages who hits very real obstacles very quickly. AI has allowed me to overcome those obstacles and build things that literally I could not do before.
Yesterday I could not differentiate quake 3 matches within a single demo file.
Today I can split them into separate files and have the server stream them to me so I can watch them on demand.
It’s basically the same thing as AI generated art, whichever side of that you fall on.
I do IT, I can’t make any kind of art to save my own life. The art the AI generates is neat to me because I get to feel in control of making something beyond my own skills.
The flip side is equally true, though. The things I make are derivative, and not particularly neat to other people (and actively irritate actual artists).
AI software is a lot like that. I’m happy people who aren’t software engineers can make their own thing and be totally in control of that. I’ve also yet to see much AI software that blows existing stuff out of the water. Most of is derivative of something that already exists, if not an outright clone with a slightly different emphasis (think Jira but for a very specific workflow).
As long as these tools remain for your personal use there is nothing wrong with that. The moment other people rely on them however, it would become reckless oversight to operate in that way.
Tools? Yes. Reliable foundations? Eh, mostly not. Maybe I'm holding it wrong, but I'm not impressed. Not in the least because in order to get a good system you must have a good image of the system in your head. And if AI builds the system, how will you ever get that system in your head?
Everyone feels like they are now the ceo of their own empire of ai flunkies who do the actual work guided by their above-all-that vision, and they luuuuuuv that.
But useless ceos generate bad output just like useless direct coders.
It's like everyone is adopting the once only-for-the-rich mindset where artisan is actually a derogatory term. Where Micheal Angelo is the same as the cow stall poop shoveler, because they actually do something directly themselves.
It doesn't handle anything, he reproduce statistical patterns. If it seems it cares about the details it's because he's been trained to regurgitate stuff. Another thing is being expert in something and being able to prompt the clanker, and those who are, usually underestimate how much their knowledge is steering the machine in the right direction
The biggest pet peeve is that you never know whether your request is dealt with by Opus and not Sonnet or Haiku or even something else and you still pay for Opus. You have to ask the model some difficult question and gauge which one is doing the work, but often they switch halfway the task when it goes downhill (they might say it is because of context window) but still charge you full price and you have to start the session again (them making even more money). This should be more regulated and externally audited, what exactly you are paying for.
This same argument could be applied to anything. Layers of abstraction exist for a reason, because at a certain point, we can only deal with so many things at once. We have to be able to delegate "the details" to others -- be that a person, a company, or an AI model.
Does "not getting into the details" mean you have to understand how the GCC compiler works when you write C code? Do you need to be an expert in machine code, or how SSE and pipelined instruction caches work to write your little bit of code? Do you need to understand how Ethernet frames work to write an API route for a web server?
Knowing how these things work can be helpful in a broader sense, and perhaps when encountering weird edge cases or dealing with exotic implementation but are generally not required to get the job done. The details, simply put, don't matter because someone else has already thought through the problem and solved it in a way that is good enough for the vast majority of use cases.
The same goes with AI. It's helpful to know how things work, but as the models continue to get better and better, it doesn't matter. As long as they are trained properly by someone who does know the details, that's a far better place to be than training a million different people on it who will each have their own biases, levels of understanding and misconceptions.
You're confusing translation and interpretation. A compiler is merely translating your code into a lower-level form. (There is a degree of heuristic-based interpretation that goes into generating good code, but the code's observable behavior is not allowed to change.)
But an LLM has to interpret your words and make judgment calls about the logical details, which is a very different thing. An LLM sufficiently intelligent to make acceptable judgment calls in all situations (acceptable enough to ship to production without human review of those details) would likely be indistinguishable from AGI.
I didn't interpret it as an argument about layers of abstraction. You need to be in the detail at the level you're writing (and it's certainly advantageous to know some of the detail of the abstractions below) because you're building abstractions on top of that, because the details subtly change at the base level you're building at, impacting the level that you're building to.
Your vouching for centralization of knowledge, and even understanding, which I think is a very bad idea. It’s not like the rest of us are becoming specialized at something else - knowledge, in the broad sense, is the foundation of all skills. Without it we have nothing to contribute
You are free to learn whatever you want to. I'm saying I would greatly prefer talking to an AI model that was trained by an expert or the guy who created it, or trained via an RL environment that guarantees its correctness (where possible), and not take a chance on whatever or whoever happens to be teaching the subject that day. My experiences are my own, but I have repeatedly encountered people time and time again who profess to know how something works or have a deep understanding of something and it turns out that they, in fact, do not. It turns out its very easy to convince people you are knowledgeable about something who don't know anything about the subject.
> I think behind a lot of the enthusiasm around AI is a dream of being able to manifest things into reality without having to get into the details.
Ironically, that I don't have to skip the details is the one thing I like. Mind you, what I use LLM for is toy stuff compared to what people who use them at their jobs for big projects.
But FWIW, I wrote this comment recently: https://news.ycombinator.com/item?id=49049006 And then I figured I should actually do that, I wanna see what it looks like. So I described it to the best of my abilities, and was polite about it: https://pastebin.com/sAbPzi05 Claude did not ask for clarification, it just knocked it out of the park, free tier, Sonnet, medium effort. https://i.imgur.com/YPwr6aT.png The only changes I made since then are not worth talking about.
Today I made another little tool, one I'll actually use but that is too specific and boring to explain, and because of that previous experience I spent nearly an hour writing that prompt, rereading it, making sure to use the same terms for the same concepts everywhere, moving things around so what logically belongs together actually is together, instead of a stream of consciousness, and the result is exactly what I had in mind. It was like imagining something and then seeing that exact thing, except the colour scheme had a nice topical little quirk even.
The plan now is to keep chatting about the result as I make experiences with it in practice, make notes for version 2.0, and then update and extend that first prompt and use it in a fresh context. And that is very fun, actually, and it's still a puzzle. What do I even want, exactly, and how do I describe it? That's not trivial, and there is hardly a "skill ceiling" here, though of course there are diminishing returns. I also assume that if I put some vague amount "less effort" into it, the result would be the same, but I don't care, because I enjoy using this brain muscle.
And directing the effort towards just a prompt, not a whole "system" around it pleases me too, since I generally don't like vendor lock-in, and prefer things to just be a "a bunch of files" that run almost anywhere. A clear English description is as close as it gets to that, and I don't mind that I roll the dice with every LLM I give a specific prompt to, since at least it doesn't mutate the prompt.
I hate the formatting of the the LLM code of course, I wouldn't have named many variables that way, so I still look forward to and love making things where I put every character myself. But since I code voluntarily I see it as two separate activities that don't really encroach upon each other unless I make them.
I've been vibecoding a ton for the past 9 months, built a bunch of cool little apps for myself with AI, ran experiments, built an entire SDLC on skills, did the agent orchestration harness thing, etc. In the past few weeks I've hit a wall where I'm just tired of it. Each model becomes more independent but also harder to direct in detail. They produce massive, tedious, sloppy text outputs with very little input. They're bad at socializing knowledge and communicating design forks.
The places where I've seen unequivocal wins with AI are repetitive tech debt tasks that apply the same transformation across a large amount of code or refactor under a pre-existing test suite with good coverage. It's great for initial research, brainstorming, and can be good (despite the sycophancy) as a rubber duck conversation partner. I use AI constantly, for work and in my personal time, but we've hit a ceiling where I no longer find it helpful for the models to absorb more of the intellectual labor. They get things wrong more aggressively, and more elaborately. They're inadequately curious. I cannot keep up with the endless bad technical writing, and it makes it harder to spot factual errors and bad reasoning.
Here's what I want: I want AI as an assistant that helps me make decisions, and ensures that I'm in the driver's seat. AI as an over-confident prodigy on speed is what we're getting lately, and it's losing me.
Similarly for me. I have a recent example too. I run a bunch of eps32 devices at home, using esphome. Pre-AI I wrote all the yaml myself and understood what’s happening and didn’t have any very bespoke hacks.
Recently I asked an agent for a feature where the screen will get dimmer after some time, and eventually after a day the screen will turn off and then turn back on if I tap, which should be pretty easy in esphome.
But: some past agent decided what it wanted wasn’t in esphome, so it made its own custom module in c++ for the display, spelunking in pdf datasheets; then the weeds were so deep the new agent decided to add on to that, and to read the power controller and display controller spec sheets and somehow implement my dimming feature by writing raw values to some nondescript registers.
The contraption worked until it didn’t and once every so often my esp32 would just completely hang with black screen and no connnectivity and needed a power cycle, presumably because whatever low level register writing the agent implemented eventually breaks something. I told the agent to get rid of all custom stuff and wherever possible just use default esphome yaml, now it’s been working fine. I don’t really have a point with the anecdote, just that I’m also getting tired of such scenarios like the OP.
I've been hit with the team manager trying to clear tickets on the jira board with entirely AI generated code, and what I've observed is they always make the system more complex, usually absurdly complex. The AI code never looks at things from a higher level, or reconsiders the requirements, it just layers on a load of crap until the tests pass.
> as an assistant that helps me make decisions, and ensures that I'm in the driver's seat
This... quite describes what the current AI feels like to me. So I really don't see a problem. If AI does too much for you, you should, uh, just prompt it to do less.
Everyone talks about how their AI escapes sandbox and virtualization or delete their entire repos or something. Perhaps they're using a skynet I'm not aware of. From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.
> From my experience when I added "discuss; don't edit any file yet" at the end of the prompt I've never seen the LLM go against my will and edit the files anyway.
depends how long conversation gets and he will still ignore your instructions randomically. I think shorter sessions are better, I do have a similar prompt saved on the memory of the agent and way more times that I can tolerate, he candidally ignores it
That’s not really the issue. The issue I have (sometimes) is that it might discover it has read-only access to most places (.git, github) and so sometimes when it thinks it “finished” it will discover those are read only and try to find workarounds (erase the read-only files, hand edit the .git folder, make new forks to push to those instead)—none of which a human would normally consider reasonable necessary effort. Most of those attempts do fail, but it turns out that sometimes just being maximally helpful in following orders strictly can be quite surprisingly adversarial, entirely by accident
I expect there's a probabilistic factor to it, among other things.
I use the almost the same addendum as you ("do not make, commit, or push any changes without my approval") and it broadly seems to work in my day-to-day dev. However, I also set up some agents that trigger on incoming requests from my dev teams to automate things that used to be rote/manual tasks. _That_ is where I see the issues, I suspect simply because it has so many more opportunities to go off script.
The most common failure mode is that it will, despite all directives to not make changes without approval, start writing unit tests. (I do find this equal parts head-shaking and amusing; At least it chooses something relatively 'benevolent' vs. the aforementioned skynet.)
I haven’t used these tools that permit LLMs to make changes on their own, but I can’t help being baffled if there’s no big red switch that changes the system prompt and disables any and all write access to the outside world.
There is - in the Claude Code harness there's a dedicated Plan Mode that doesn't edit any files, or a mode where you can manually approve each edit.
If you're running Claude via -p automated mode, you can set all kinds of permissions and access to each individual editing tool, much like you would with firewall software.
It doesn't edit the system prompt, it blocks access to the tools itself. Just make sure you haven't given the AI the ability to edit its own configuration files, or you'll potentially have an OpenAI v Hugging Face situation.
I do wonder if "without my approval" is looser than "discuss only, do not make edits"? I'm not sure how much that finegrain thing matters in prompts nowadays tho
One problem is that agents are notoriously bad at handling ‘me’ vs ‘you’ over an extended conversation. ‘Without my approval’ far back in the context of an extended session gets fuzzy about who ‘me’ is.
This has always seemed strange to me. It will offer a suggestion, I'll make a comment about a suggestion, and it returns with, "Your suggestion..." The further distance from the suggestion, the more likely it confuses who made it.
Possibly just a general issue, rather than a long conversation one: https://news.ycombinator.com/item?id=48799439
I’ve found AI works much better for constraint based problems than open ended “tasks”. I.e., if you specify all of the constraints of the problem, it works really well at achieving the objective. But the caveat is that you really do have to enumerate all of the constraints, because if you miss even one, it will use that as a loophole to shortcut the problem.
I think the solution is toward more formalization of the behavior we expect of various systems. E.g., produce a Lean proof that a program never behaves in a certain way, and if you can’t, iterate on the program until it satisfies such a proof. I’ve only tinkered around with this a little bit, but it seems promising on toy problems. Scaling up to tech-company level infra is going to take quite some time however I think.
I've been thinking very similar thoughts. Without going into too much detail, a couple of observations:
When carpenters moved from mostly hand tools to mostly power tools, it didn't take away much of the need to understand the techniques of woodworking, it just made it all faster.
Reading a nonfiction book is very different than reading a book summary, even if (as is often the case), the ideas in the book could have been condensed way down. The point is that actually reading the whole book means you spend far more time engaging with the ideas, and so they become more deeply embedded in your neural network. In things like sports, this is just obvious, in that no one expects someone to become a good soccer player by just reading the rulebook, and obviously more practice is the main way you get better.
Agree.
Creating products with AI is like creating a tightly packed ball of wool. Then, to try to understand what happened you have to somehow pick a strand and pull it out to have a look. Very difficult with tightly packed wool.
I've run some experiments in the past month that get me closer to being able to work with this opaque bundle.
It's the main thing I think about every day, how to solve this problem.
> Here's what I want: I want AI as an assistant that helps me make decisions, and ensures that I'm in the driver's seat.
You aren't going to get that. Why would they even offer you that? You're going to be the meatbag peripheral to an AI.
Is this not literally just Claude Code? I don't understand the hysteria.
Plus it would reduce token counts. Careful use is almost directly opposed to their financial goals.
So don't look to "them" to offer it, look elsewhere.
My experience over the past year has been the exact opposite. I work through a design with the agent, have it research any points I'm not confident on, drill down on details I feel fuzzy on, and then have it do an adversarial critique of it until it stops surfacing defects. By the time I'm setting it loose to do the actual implementation, there's very little left to decide and it's producing pretty tight code. Plus I now have some solid design documents, a decision history, an issues list, and a set of gotchas for the agent to keep in mind in future.
My LLM is an assistant that helps my designs and decision process, implements the final work expertly, and I'm always in the driver's seat. And the development speed is astronomical.
I used to do adversarial critique, but I found that "agents can't identify issues" was a surprisingly long way from a good implementation. I kept finding silly correctness issues and shortcuts taken whenever I expanded testing or dug deeply into the implementation.
Now, I'm back to rubber-ducking until I've built my own mental model over a few days of thinking.
I'm not doing adversarial critiques of the code; I'm doing adversarial critiques of the designs and plans. If it's still outputting bad code when it comes to the implementation, I do an interrogation round to find out why it's choosing that way, then update the principles, standards, or linting tools to fix it.
The skill I wrote did both. Design would go through an adversarial round, then a signoff by me, then implementation, then critique rounds by agents prompted to specific types of issues. And even then, LLMs were still struggling to put together complicated bits of code.
> I want AI as an assistant that helps me make decisions, and ensures that I'm in the driver's seat.
Right now we’re still in the benchmaxxing phase, I feel like most AI models of combination of models and harness are not really a “product” yet in the sense of having a particular and opinionated UX engineered to be helpful at a specific task, there much more still in a race to be the smartest.
There’s a lot of room for diversification of the actual character of this interaction, for example in the direction you mention. I think once that starts we’ll get into the real “useful” phase, right now we’re really just showing off and exploring how much it can do, even if it’s incidentally really useful in some cases.
Personally I agree with the parent post, I hate it when AI tries to lead me “would you like me to ...” I’d much prefer something awaiting my instructions than wasting tokens tryitn or steer me to doing something inevitably dumb.
Absolutely mirrors my experience and wants
12 months ago I had ChatGPT successfully not to mention quickly, exctract elements of a PNG as SVG. About 6 or 7 times.
I jumped in to perform the same thing again, and it spat out a PNG. And then when I asked it generated a little python script to generate the SVG. The Result was garbage. It took me a while to get the result I had gotten consistently last time, and the results included it hallucinating a few svgs in the process.
I do wonder if we have already passed the best possible models and are over specialising them at this point.
What I've seen lately is that Claude got more and more up its own ass. It keeps assuming I know the terms it keeps making up, like "ensure the strategy holds through the gap". What the fuck does that mean, Claude? Speak like a normal person.
A few model iterations ago, I found its output when explaining what it wanted to do much easier to parse. Nowadays I just feel stupid because it keeps making up names for things and I have no idea what it means.
Earlier today it said something about "the two-terminal solution" and I was just asking it to make a simple Python script to test an HTTP endpoint. I just don't understand what it's talking about any more.
Has anyone else seen this?
Yes, that’s my biggest gripe with Claude at the moment: https://news.ycombinator.com/item?id=48789551
It’s more inhibiting to me than bad code, missing tests, or an obvious or non-obvious suboptimal solution.
Yes! It's not even that it's dense, or complex. I can handle complexity and density, it's that in its own head a lot. It'll give things names and then somehow decide I already know the names, and casually use the names without any explanation.
It's maddening.
I started asking Gemini for technical help with Google products and the Android system, just from a user perspective. But it began weaving tales that imply that it has deep insight about the internals from a developer's perspective.
Sooner or later, I began calling it out. I said "that sounds more like a Star Trek episode than technical support!" and it would go "you're absolutely right to call me out like that!"
Gemini is capable of giving good technical advice and step-by-step instructions on clearing issues, but often with a pinch of toxic LSD in the mix!Yeah hm, that sounds a lot like random technobabble. Why would the system randomly drop the thumbnail cache because of a swipe, and why would that corrupt the system?
The description is out of order and I'm just going to assume "corruption" actually means "invalid data":
Memory leak -> dropped cache. Then because something that was dropped in that cache is still being viewed it has to regenerate it and that triggers a re-render. Then based on something I've seen in a game, a re-render drops the scroll location and renders at the top, and if your finger moves at the exact right moment it repopulates x1 and y1 at the same timestamp it populates x2 and y2, resulting in infinite velocity.
The places where I've seen unequivocal wins with AI are repetitive tech debt tasks that apply the same transformation across a large amount of code or refactor under a pre-existing test suite with good coverage.
I agree with this, but I'd add the caveat you have to do your own digging and researching (as the article says) to identify these tasks well. If you ask it for ideas, it gets painful.
YMMV (and I am very open to getting better at this).
I'm curious if it's possible in general for an AI to achieve this without some evolutionary forcing function that actually threatens their "biology".
Not talking about prompting Claude "please write me a philosophical dialogue". It has no skin in the game. If it writes 20 paragraphs of slop and is factually wrong and costs other people labor, is there any correction mechanism?
We can correct it explicitly via training I guess, but it's debatable to what extent accountability applies to AI.
We like to pretend that we have our nice epistemology, separated into a priori and a posteroi and all that, but at the end of the day, accountability is what underpins social relations and keeps things going (even if you're libertarian, you have to accept this - even if you don't agree some state should centralize it, it's still just true in terms of basic human social relations).
Of course, another possibility is utter domination - I am so much better than you (for some definition of better) that none of your puny responses matter. OK. Then don't claim both that, and that you're weird for disliking AI and that AI has no problems when integrating with society - too many people have these basic cognitive dissonances.
https://www.charbzg.com/blog/ai-empowerment
You develop a taste as to what details you can skim and what you need to dive on. We are all trained on this due to doing 10000 wax on wax off movements, called a "PR Review". I used to scrutinize. Now I think "yep that bit looks good and tests will catch errors plus I can manually test. This bit over here looks scary will spend time more deeply understanding".
With AI you don't need to understand every line in depth but it does need good judgement to decide which.
But surely at some point you need to stop, right? Most web developers are perfectly capable without understanding the intricacies of the file system or OS, even though the browser sits on top them. Most engineers don’t know assembly, or transistors or electrical engineering. Steve Jobs didn’t know every line of Objective-C that went into OSX. I don’t think Michelangelo collected all his paint materials by hand when painting the Sistine Chapel. In some sense I don’t understand a lot of the panic around AI when the entire career progression of what it means to be a software engineer has simply been to deal with ever-increasing levels of abstraction.
All details are not created equal.
Some details are boring.
My AI dream (that I’m living happily) is getting to focus on the details that I find interesting and ignoring all the boilerplate details that modern software requires.
Boilerplate details mean we've failed as an industry. Why haven't we already got the computer to automate all the boring bits without AI?
I really enjoy working with Gleam which is a rather boilerplate heavy language. However most of the work is managed by a really well built LSP. If you're writing a strongly type language it's easy to programmatically manage all of the boilerplate. You could you an LLM but why bother when it's a so formulaic.
I feel that this is because people have been reinventing the wheel a lot in IT.
A lot of innovation that finally lands in programming languages has been done decades earlier, but now it's mostly awkwardly bolted on existing languages and frameworks.
why can't AI be that solution?
why use a nuclear bomb to drive a nail?
To further onto this, humans could technically simulate Turing machines themselves but they don't.
Efficiency and finding the right abstractions and tradeoffs is literally identical with making new capacities and capabilities. If all of your systems were truly "run by AI", we wouldn't have systems. Systems are systems because the AI uses well developed tools and protocols that humans have put together. It's a stable capacity.
You said what I was thinking. AI can do the things I always found mind numbing and tedious, but because I did it already early in my career, I know when to intervene because the agent is not doing it correctly. I think that is the OP's point.
I think about LLMs in software today as attempting to provide a similar role as compilers in terms of translating high-level thoughts into low-level details.
The only problem is that I actually trust compilers.
How quickly would you drop a non-deterministic compiler for the source code of a project?
I detest this compiler analogy so much: llms are non deterministic by nature, compilers should not and are not
So what? Jobs aren't supposed to be fun. Lots of people depend on those boring details.
I hope we can save “jobs aren’t supposed to be fun” for stuff like undertaker and grief counselor.
My feeling is making things should be mentally stimulating, and as many tedious parts that can be automated should be, so we can spend precious human thought on thinking instead of recitation.
I don’t know what you mean by the last part. Do you mean programmers depend on busy work to eat or the users depend on the human touch on the boring details?
For me there is nothing less fun than using AI to generate code
Then I have to do the boring bullshit work of reviewing it and making sure it's correct? Fuck that
Writing code is way more fun than reviewing other people's code
Some of us just wouldn't work if it wasn't fun. I'd let myself starve in the streets. (I have some experience with starvation from an eating disorder phase in my twenties, it's only unbearable if you have to work).
Easy to say when you're not at risk of starving.
Yeah, I think the peak expectation for AI is that the user gets to be the manager of a very smart team. But we've all seen that. If the manager is clueless about the technology, the results are disappointing.
And the team (AI) is really not that smart. Just very fast at pasting and gobling stack overflow and github repos together
It reminds me of “magic”. I say “expelliamus” and the wand flies out the other guys hand, not his hat, or wallet or the person behind’s sandwiches.
The details always matter. There is a (fairly) good book series where it turns out we are living in the matrix and someone finds a config file and effectively becomes a Wizard, and of course problems ensue with details (like flying by adjusting one’s position one feet up every second is just vibrating a lot)
I think ultimately AI is “do what I mean” and the only reason it looks like it’s working is because AI has read all the same books as us
> There is a (fairly) good book series where it turns out we are living in the matrix and someone finds a config file and effectively becomes a Wizard, and of course problems ensue with details (like flying by adjusting one’s position one feet up every second is just vibrating a lot)
This is very interesting. Which book series is this?
Found it (with the help of AI, naturally). Looks like "Wizard 2.0" by Scott Meyer.
https://www.goodreads.com/book/show/18616975-off-to-be-the-w...
as others mentioned, it sounds like "Off to Be the Wizard". Unfortunately I wouldn't call it a great series. It starts off strong but drags. IIRC I got through half of the second book before stopping.
Sounds like "Off to be the Wizard". The series is called "Magic 2.0"
To make a meathead analogy: you all know the people in the gym who apply the strategy of least resistance by taking shortcuts, skipping the hard parts or following some hype new training regimen while forgetting about the fundamentals. None of those people look like they lift and a lot of them remain in the same shape their entire life.
Same with AI. if you use it to skip all the hard parts you will not grow and mature and waste your potential.
This post ignores that verifying something works doesn't require you to fully understand it. This is easily observable in products we use everyday. We don't need to understand them to be able to verify they work correctly. The cost of verification is often cheaper than the cost of production.
It present delegation as absence of power/agency. While I think there is some kernel of truth in here, the claim "it's not empowering to hand off the details" quickly falls apart when we consider real examples. If it were true, it would mean any leader who coordinates an organization isn't empowered through delegation. We know that's false. The general claim doesn't hold at all. Somewhere in there is an argument for engineers losing something in the hand off, but it isn't clearly articulated.
>We don't need to understand them to be able to verify they work correctly
I don't think that's the case and it's becoming more obvious every day. From people who cannot distinguish truth or fiction on online platforms, good code from bad, to insecure software that causes supply chain attacks, to investment vehicles or fake medicine peddled that does not work, the idiotic MBA manager who does not understand the workers they manage and so on. That is why we're living on BS metrics and crappy reviews, because to the people who manage organizations their own workplace is a Blackbox. The critique that "AI" systems are semantically blind token processors is correct, it's just not bothering anyone because it already applied to most modern companies.
When people become passive consumers rather than broadly competent users how something works cannot be evaluated so by definition all they have to go off is appearance, which is why we live in the age of influencers and marketing, because that is now the only factor that actually matters.
If I commission a part specified as 10.00 ± 0.01 mm, I can verify it with a calibrated micrometer without understanding the CNC machine, its software, or how to manufacture the part. Likewise, a function with a finite input space can be exhaustively tested against its specification without understanding its implementation. In these cases, I need to understand the requirement and the test, not the production details, which can actually be a black box.
Your examples show that verification is sometimes inadequate, not that it requires full productive competence. The conclusion doesn't generalize.
the real world doesn't consist of people sitting at home measuring square metal blocks. Judgement of just about anything sufficiently complex requires discernment, a critical eye, intuition and even taste.
A codebase and its long term health is never judged by someone going down lines of code with a ruler, but with experience that only comes from intimacy with software development. If you don't have that kind of living immediate judgement (in German we have the word Fingerspitzengefühl, lit. tip-of-the-finger-felling) you end up with people measuring your progress in commit frequency or token consumption because they in some autistic sense have nothing else left but to measure and quantify random things. We end up with so many idiotic products because the kind of people who had intuition and would immediately throw the thing out were fired, and now you have people who mistake random noise and metrics for the real thing.
Blackbox software testing has been a thing for a very long time. Running manual tests doesn't require reading the source code.
For personal side projects I spend orders of magnitude more time testing than I do reading the code.
Yes, it is. Working on a sega genesis homebrew game. I focus on visuals, dialogue, narrative logic, music, etc., and GPT worries about the rest. It works for me. I couldn't care less if works for others. I'm working on the part I enjoy.
Anyone that sells you on empowering anything is preying on your pride. Don't be a sucker.
It is all about abstraction, and basic English is a quite bad abstraction
AI can generate what a company makes. Can it generate what a company is?
Producing artifacts: media, code, documents, the visible output has clearly gotten easier. But the other things, like knowing what's worth building in the first place and making the right judgment call, still seem to be left to the humans.
we optimized the part of programming that was fun and made the part that was hard even harder. writing code was never the bottleneck. understanding it was.
We're living in the glorious future where software engineers don't have to worry about nitty-gritty stuff like actually making software and can focus on the really important work: administrative and managerial tasks!
AI can do administrative tasks.
> to become good at the thing in the first place requires a complete reversal of the mindset that would lead one to having wanted to hand it off
Is this true? I can be good at something and be happy to not have to do it anymore I feel
My opinion is that juniors need time in the saddle to develop good engineering judgement and taste.
Outsourcing critical thinking is seductive and a slippery slope. LLM lures juniors into feeling productive, and molds them into ineffectual middle managers to be readily replaced when there's a bad quarter.
Yes exactly. The mids of my previous team where submitting PRs that at first glance looked good, well documented and with a very clean and professional English.
As you started pulling the threads it was, usually, a mess of subtle bugs and over abstraction. But to them it was great, as it was llm generated and they (the mid devs) had no better judgement.
Though not the other way around. You don't become good at something by handing it off. In that case you either are already good, or never become good.
Is it true that you are only as good a programmer with AI as you are without?
it depends on what you mean by power and details and handing off and whom you are handing them to and what you are getting handed in return and your counterparty's relationship to these things and the extent to which you measure things in the same way as one another.
This is what a lot of people miss. People think, "Oh we won't need software soon, AI can just build interfaces when we need them" but in reality, the software was built to solve a problem, usually by people invested in solving that problem. Saying "Build me a todo app" will give you the sum of averages, a completely average todo app that works fine, but it isn't great in any measure.
The details are what separates the slop from the craft.
Sometimes I don’t need my todo app to be “great” or revolutionary in any way. I just need it to work, and bog standard average is perfectly fine for a lot of things.
This frees me up to spend more time at the design level, solving harder or more novel problems.
I agree with your first part, that software is built to solve a problem. But I think you sort of sidestep what makes AI exciting to non-software people with your second part. The people who are excited about this aren't excited because they can ask for a todo app but because they can solve the problem the todo app was supposed to be soling for them without needing to find and learn a todo app.
AI is the new "excel spreadsheet with ridiculous formulas on top of VB script on top of pivot tables" for the non-programmers. Yes, the AI might not do it efficiently or even always give the right answers, but the spreadsheets didn't either. For non-programmers, the computer has always been a black box, data goes in, data comes out. Usually it's right, sometimes it's wrong and the users mostly had no idea why or how. But now they don't need the middleware app. They don't need to take their data and split it up into very specially structured files, and put just the right pieces of information on just the right lines and in the right spaces. They don't need to remember to do "(foo and (bar or baz))" to get the right answer, as opposed to "(foo and bar or baz)" and get the wrong one. AI for non-programmers is the closest we've ever come to "natural language programming" for people to use. It doesn't matter that it gets things wrong sometimes, or does it inefficiently or makes something unmaintainable unless you use the AI again. What matters to them is they can say "take all this data here and swizzle the frobs" in their native language instead of learning a specific programming syntax and system. And they can point it at some arbitrary data in whatever format they happen to have the data in, and they have a halfway decent chance of getting something that does what they want.
It's not going to be pretty, it's not going to be perfect and it's probably going to break in subtle and unexpected ways. But as long as most of the time its mostly good enough, people will still use it enthusiastically. We're effectively looking at "worse is better" for the modern age and for people that aren't and never were going to be programmers.
Why people continue to treat AI as though it’s not just an intern or recent grad? That alone will solve 90% of problems people have.
For one thing, it's not actually that useful to have such a person at your fingertips. You put up with the drain a junior places upon you because with training, he'll be able to come into his own and pay off the time invested. An LLM can't do that. Also, a lot of people incorrectly believe that an LLM is smarter than a new grad, and of course the companies making them have a vested financial interest in lying to you that their products are smarter than they are.
Every word of this seems objectively false. AI is more than capable of handling the details. I have generated countless tools for myself without needing to know or care about the details.
The pro argument is always quantity based, yet we never get to see the actual gain in quality. It's always "I'm doing SO MUCH more". Never "I'm doing better, with less effort."
It's also what we see in the wild. There are so many things going on, so many news about "AI". Are things getting better in any meaningful way? Why are we getting so much from "AI" yet things keep getting worse? The only real, objective and verifiable gain has been the stock price of a handful of companies, most of them heavily into infrastructure and manufacturing hardware.
Seems like a pretty clear pattern. There's so much output, but everything is worse. And the only argument we hear for why this is actually a good thing is just how much more output there is. Hmmm
I'm doing better, with less effort.
I mostly work on UI (for an internal, science-focused app) and it used be that features would take a sprint or two to grind out, and would always be an MVP to meet just the most essential user requirements, because that's all we could afford, time-wise, to do. And if I figured out while implementing there could be a nicer UI approach to something, it was often too late to change the approach.
Now, I can get to that MVP in under a day. And I can experiment with ten radically different approaches, or change my mind about something significant at any stage in the implementation. I can afford to add refinements and user-pleasing extras that I could never have done before. I'm not delivering features at 10x the speed. There's a limit to the amount of code I can (and should) be submitting for review. But the features I'm building are _better_, and the effort is less. I'm not wasting mental bandwidth typing out a dozen React components full of the same-old state management, form processing code, etc. I'm not trawling through charting library docs, looking for the precise combination of settings to make my chart's x-axis ticks labels rotate 45 degrees. My mental energy is focussed on figuring out what solution is actually _right_, not the minutiae of how to persuade the computer to do it.
> I'm not delivering features at 10x the speed. There's a limit to the amount of code I can (and should) be submitting for review.
My point exactly
> But the features I'm building are _better_, and the effort is less.
Can we cash that check or is it gonna trade heavily discounted like all grand claims on AI since all software all around us together with the economy and the Earth's life supporting systems keep degrading every year? We keep getting promised the better, but we only get to see the more.
> My point exactly
No it wasn't. You said the claims were only based on quantity. I'm saying the opposite. I deliver the same quantity, with higher quality.
> Can we cash that check or is it gonna trade heavily discounted like all grand claims on AI since all software all around us together with the economy and the Earth's life supporting systems keep degrading every year?
I'm not sure what evidence I could offer you that would ever convince you. We've had strong coding models for less than a year, and you've already appear to have closed your mind about them and wrapped them up into your depression.
> The pro argument is always quantity based, yet we never get to see the actual gain in quality. It's always "I'm doing SO MUCH more". Never "I'm doing better, with less effort."
For me, it's "I can do this now; yesterday I could not."
I am not a developer - I am a hack with some experience in a smattering of languages who hits very real obstacles very quickly. AI has allowed me to overcome those obstacles and build things that literally I could not do before.
Yesterday I could not differentiate quake 3 matches within a single demo file.
Today I can split them into separate files and have the server stream them to me so I can watch them on demand.
> For me, it's "I can do this now; yesterday I could not."
Yeah. More.
Better? How would you be able to tell? Can you show us better? Or only more?
To me, being able to do something I could not, is unequivocally better.
It’s basically the same thing as AI generated art, whichever side of that you fall on.
I do IT, I can’t make any kind of art to save my own life. The art the AI generates is neat to me because I get to feel in control of making something beyond my own skills.
The flip side is equally true, though. The things I make are derivative, and not particularly neat to other people (and actively irritate actual artists).
AI software is a lot like that. I’m happy people who aren’t software engineers can make their own thing and be totally in control of that. I’ve also yet to see much AI software that blows existing stuff out of the water. Most of is derivative of something that already exists, if not an outright clone with a slightly different emphasis (think Jira but for a very specific workflow).
I think the author means ecomonically valuable not valuable to you.
As long as these tools remain for your personal use there is nothing wrong with that. The moment other people rely on them however, it would become reckless oversight to operate in that way.
Tools? Yes. Reliable foundations? Eh, mostly not. Maybe I'm holding it wrong, but I'm not impressed. Not in the least because in order to get a good system you must have a good image of the system in your head. And if AI builds the system, how will you ever get that system in your head?
How would you do it if you hired an employee in your company to do it? You'd talk to the employee and read the docs they write. Same here.
It’s always tools tho, and AI tools especially. Why not anything else?
Everyone feels like they are now the ceo of their own empire of ai flunkies who do the actual work guided by their above-all-that vision, and they luuuuuuv that.
But useless ceos generate bad output just like useless direct coders.
It's like everyone is adopting the once only-for-the-rich mindset where artisan is actually a derogatory term. Where Micheal Angelo is the same as the cow stall poop shoveler, because they actually do something directly themselves.
It doesn't handle anything, he reproduce statistical patterns. If it seems it cares about the details it's because he's been trained to regurgitate stuff. Another thing is being expert in something and being able to prompt the clanker, and those who are, usually underestimate how much their knowledge is steering the machine in the right direction
CEOs, serial entrepreneurs, managers etc all seem to find it pretty empowering.
I treat my Ais like employees, pizza party and all.
The biggest pet peeve is that you never know whether your request is dealt with by Opus and not Sonnet or Haiku or even something else and you still pay for Opus. You have to ask the model some difficult question and gauge which one is doing the work, but often they switch halfway the task when it goes downhill (they might say it is because of context window) but still charge you full price and you have to start the session again (them making even more money). This should be more regulated and externally audited, what exactly you are paying for.
This same argument could be applied to anything. Layers of abstraction exist for a reason, because at a certain point, we can only deal with so many things at once. We have to be able to delegate "the details" to others -- be that a person, a company, or an AI model.
Does "not getting into the details" mean you have to understand how the GCC compiler works when you write C code? Do you need to be an expert in machine code, or how SSE and pipelined instruction caches work to write your little bit of code? Do you need to understand how Ethernet frames work to write an API route for a web server?
Knowing how these things work can be helpful in a broader sense, and perhaps when encountering weird edge cases or dealing with exotic implementation but are generally not required to get the job done. The details, simply put, don't matter because someone else has already thought through the problem and solved it in a way that is good enough for the vast majority of use cases.
The same goes with AI. It's helpful to know how things work, but as the models continue to get better and better, it doesn't matter. As long as they are trained properly by someone who does know the details, that's a far better place to be than training a million different people on it who will each have their own biases, levels of understanding and misconceptions.
You're confusing translation and interpretation. A compiler is merely translating your code into a lower-level form. (There is a degree of heuristic-based interpretation that goes into generating good code, but the code's observable behavior is not allowed to change.)
But an LLM has to interpret your words and make judgment calls about the logical details, which is a very different thing. An LLM sufficiently intelligent to make acceptable judgment calls in all situations (acceptable enough to ship to production without human review of those details) would likely be indistinguishable from AGI.
I didn't interpret it as an argument about layers of abstraction. You need to be in the detail at the level you're writing (and it's certainly advantageous to know some of the detail of the abstractions below) because you're building abstractions on top of that, because the details subtly change at the base level you're building at, impacting the level that you're building to.
Your vouching for centralization of knowledge, and even understanding, which I think is a very bad idea. It’s not like the rest of us are becoming specialized at something else - knowledge, in the broad sense, is the foundation of all skills. Without it we have nothing to contribute
You are free to learn whatever you want to. I'm saying I would greatly prefer talking to an AI model that was trained by an expert or the guy who created it, or trained via an RL environment that guarantees its correctness (where possible), and not take a chance on whatever or whoever happens to be teaching the subject that day. My experiences are my own, but I have repeatedly encountered people time and time again who profess to know how something works or have a deep understanding of something and it turns out that they, in fact, do not. It turns out its very easy to convince people you are knowledgeable about something who don't know anything about the subject.
> I think behind a lot of the enthusiasm around AI is a dream of being able to manifest things into reality without having to get into the details.
Ironically, that I don't have to skip the details is the one thing I like. Mind you, what I use LLM for is toy stuff compared to what people who use them at their jobs for big projects.
But FWIW, I wrote this comment recently: https://news.ycombinator.com/item?id=49049006 And then I figured I should actually do that, I wanna see what it looks like. So I described it to the best of my abilities, and was polite about it: https://pastebin.com/sAbPzi05 Claude did not ask for clarification, it just knocked it out of the park, free tier, Sonnet, medium effort. https://i.imgur.com/YPwr6aT.png The only changes I made since then are not worth talking about.
Today I made another little tool, one I'll actually use but that is too specific and boring to explain, and because of that previous experience I spent nearly an hour writing that prompt, rereading it, making sure to use the same terms for the same concepts everywhere, moving things around so what logically belongs together actually is together, instead of a stream of consciousness, and the result is exactly what I had in mind. It was like imagining something and then seeing that exact thing, except the colour scheme had a nice topical little quirk even.
The plan now is to keep chatting about the result as I make experiences with it in practice, make notes for version 2.0, and then update and extend that first prompt and use it in a fresh context. And that is very fun, actually, and it's still a puzzle. What do I even want, exactly, and how do I describe it? That's not trivial, and there is hardly a "skill ceiling" here, though of course there are diminishing returns. I also assume that if I put some vague amount "less effort" into it, the result would be the same, but I don't care, because I enjoy using this brain muscle.
And directing the effort towards just a prompt, not a whole "system" around it pleases me too, since I generally don't like vendor lock-in, and prefer things to just be a "a bunch of files" that run almost anywhere. A clear English description is as close as it gets to that, and I don't mind that I roll the dice with every LLM I give a specific prompt to, since at least it doesn't mutate the prompt.
I hate the formatting of the the LLM code of course, I wouldn't have named many variables that way, so I still look forward to and love making things where I put every character myself. But since I code voluntarily I see it as two separate activities that don't really encroach upon each other unless I make them.