On small teams, AI feels like a big boost. On larger teams, it mostly just moves the work around.
We (12+ people on the team) see more LOC and faster first drafts, but also more review work. PRs look done early, but often hide shallow thinking or edge cases. Velocity goes up on paper. Review fatigue goes up too.
The best teams treat AI like a junior dev with infinite energy. Great for boilerplate and refactors, but you still need ownership. Otherwise you just ship bugs faster, which is not great honestly.
Not a full team adoption story, but relevant data point: I run a small engineering org (~40 engineers across teams) and we've been tracking AI coding tool adoption informally.
The split is roughly: 30% all-in (Claude Code or Cursor for everything), 50% selective users (use it for boilerplate, tests, docs but still hand-write core logic), 20% holdouts.
What I've noticed on PR velocity: it went up initially, then plateaued. The PRs got bigger, which means reviews take longer. We actually had to introduce a "max diff size" policy because AI-assisted PRs were becoming 800+ line monsters that nobody could review meaningfully.
The quality concern that keeps coming up: security. AI-generated code tends to take shortcuts on auth, input validation, error handling. We've started running dedicated security scans specifically tuned for patterns that AI likes to produce. That's been the biggest process change.
Net effect: probably 20-30% faster on feature delivery, but we're spending more time on review and security validation than before.
Ha, pretty accurate in my experience. Though I'd say it's more like 1.5x the PRs - Claude does the initial PR, then you do half a PR fixing the subtle stuff it got wrong, and then you spend the other half wondering if you missed something.
The security fixes are the worst because the code looks correct. It's not like a typo you'd catch immediately - it's an auth check that works for 95% of cases but fails on edge cases the model never considered.
I have seen the same Ai hallucinations that you mentioned: auth, input validation, error handling, non-existent dependencies, etc. It's tricky to get them all as LLM's have mastered the art of being "confidently wrong". What tools are you using to catch those issues? I feel current tooling is ill equiped for this new wave of Ai generated output.
"Confidently wrong" is the perfect description. The code compiles, the tests pass (because the AI also wrote the tests to match), and the auth flow looks reasonable at first glance.
For catching these we layer a few things:
- Standard SAST (Semgrep, CodeQL) catches the obvious stuff but misses AI-specific patterns
- npm audit / pip-audit for dependency issues, especially non-existent packages the AI hallucinates
- Custom rules tuned for patterns we keep seeing: overly permissive CORS, missing rate limiting, auth checks that look correct but have subtle logic bugs
- Manual review with a specific checklist for AI-generated code (different from our normal review checklist)
You're right that current tooling has a gap. Traditional scanners assume human-written code patterns. AI code looks structurally different - it tends to be more verbose but miss edge cases in ways humans wouldn't. We've been experimenting with scanning approaches specifically tuned for AI output.
The biggest wins have been simple: requiring all AI-generated auth and input validation code to go through a dedicated security reviewer, not just a regular code review.
Here are some real examples from our projects in 2025 at SIROC (for context: we are a 18 people venture studio; 140+ projects completed):
* A task estimated at 4 hours → solved with one well specified prompt
* A 20 hour engineering effort → executed in about 3 hours
* A 3 month project → delivered in 1 month
These are clearly best case scenarios. They are not the norm, yet. But they demonstrate what is possible.
We have also seen what happens when things go wrong. Companies, including startups, come to us with broken systems and spaghetti code and architecture caused by weak prompts, unclear requirements, and no verification.
It is important to understand that the efficiency gains we are seeing do not come from the tools alone. They come from a specific combination:
1) Engineers who have spent 20 years building everything from robotics to enterprise-scale technology. You cannot give a perfect instruction to an AI if you do not know what perfect looks like in a production environment.
2) A technical prompt should not be treated as a quick input or question. It is a detailed specification that requires experience and deliberate thinking.
3) Knowing the right combination of tools, workflows, and validation processes.
That said, some (many?) members of our team are dinosaurs in the software engineering world. They bring a ton of experience but are used to tools from 15 years ago and don't like change. We really had to push AI adoption (mostly Cursor and Claude Code) on them. It’s still an ongoing process, and probably will be for a while.
I was thinking last night about whether this is even a realistic moat or not.
Right now, Claude is getting trained by hundreds of thousands of programmers showing it how to ask the right architecture + PM questions.
They're just patterns, like anything else in our industry, and most of them are pretty standard patterns.
Like when I think back on 20 years of software architecture and BA work, I've done the same thing over and over. I must have implemented 4 PO systems, 3 different custom chat systems, SMS systems for reminders, monthly summary emails, etc.
Some people seem like fully gave in to using AI for everything, and nothing better than receiving multiple review requests per day for 2k+ diff PRs.. where most of the stuff is just "wtf" type of refactoring and added noise
Also funny to hear reactions when I'm grepping code for a function in front of someone and I hear "wow you look for code I would just ask Gemini" topkek
I was on a greenfield project late last year with a team that was very enthusiastic about coding agents. I would personally call it a failure, and the project is quietly being wound down after only a few months. It went in a few stages:
At first, it proceeded very quickly. Using agents, the team were able to generate a lot of code very fast, and so they were checking off requirements at an amazing pace. PRs were rubber stamped, and I found myself arguing with copy/pasted answers from an agent most of the time I tried to offer feedback.
As the components started to get more integrated, things started breaking. At first these were obvious things with easy fixes, like some code calling other code with wrong arguments, and the coding agents could handle those. But a lot of the code was written in the overly-defensive style agents were fond of, so there were a lot more subtle errors. Things like the agent adding code to substitute an invalid default value in instead of erroring out, far away from where that value was causing other errors.
At this point, the agents started making things strictly worse because they couldn't fit that much code in their context. Instead of actually fixing bugs, they'd catch any exceptions and substitute in more defaults. There was some manual work by some engineers to remove a lot of the defensive code, but they could not keep up with the agents. This is also about when the team discovered that most of the tests were effectively "assert true" because they mocked out so much.
We did ship the project, but it shipped in an incredibly buggy state, and also the performance was terrible. And, as I said, it's now being wound down. That's probably the right thing to do because it would be easier to restart from scratch than try to make sense of the mess we ended up with. Agents were used to write the documentation, and very little of it is comprehensible.
We did screw some things up. People were so enthusiastic about agents, and they produced so much code so fast, that code reviews were essentially non-existent. Instead of taking action on feedback in the reviews, a lot of the time there was some LLM-generated "won't do" response that sounded plausible enough that it could convince managers that the reviewers were slowing things down. We also didn't explicitly figure out things like how error-handling or logging should work ahead of time, and so what the agents did was all over the place depending on what was in their context.
Maybe the whole mess was a necessary learning as we figure out these new ways of working. Personally I'm still using the coding agents, but very selectively to "fill-in-the-blanks" on code where I know what it should look like, but don't need to write it all by hand myself.
I'd expect this kind of outcome to be common for anything complex, but there seem to be more claims of success stories than horror stories.
Various possibilities I suppose - I could just be overly skeptical, or failures might be kept on the down low, or perhaps most likely: many companies haven't actually reached the point where the hidden tech debt of using these things comes full circle.
Having been through it, what's your current impression of the success stories when you come across them?
I'd speculate we had a few factors working against us that made us hit the "limit" sooner.
Several different engineering teams from different parts of the company had to come together for this, and the overall architecture was modular, so there was a lot of complexity before we had to start integrating. We have some company-wide standards and conventions, but they don't cover everything. To work on the code, you might need to know module A does something one way and module B does it in a different way because different teams were involved. That was implicit in how human engineers worked on it, and so it wasn't explicitly explained to the coding agents.
The project was in the life sciences space, and the quality of code in the training data has to be worse than something like a B2B SaaS app. A lot of code in the domain is written by scientists, not software engineers, and only needs to work long enough to publish the paper. So any code an LLM writes is going to look like that by default unless an engineer is paying attention.
I don't know that either of those would be insurmountable if the company were willing to burn more tokens, but I'd guess it's an order of magnitude more than we spent already.
There are politics as well. There have been other changes in the company, and it seems like the current leadership wants to free up resources to work on completely different things, so there's no will to throw more tokens at untangling the mess.
I don't disbelieve the success stories, but I think most of them are either at the level of following already successful patterns instead of doing much novel, or from companies with much bigger budgets for inference. If Anthropic burns a bunch of money to make a C compiler, they can make it back from increased investor hype, but most companies are not in that position.
I'm personally using it a lot. For my workplace, the productivity gains are limited by human code review (need a minimum of two approvals) and manual testing. I can produce a ton of PRs, more than ever before, but I was already bottlenecked by review & test in the hand-coding era.
100% of the team (14 engineers) is using Claude Code for everything. Anecdotally it feels like velocity is 10%-20% faster. Objectively I have no idea if velocity is faster – because management is inept and does not actually track this. The most noticeable thing is test coverage has greatly improved. Every PR ships with lots of tests, whereas before we used AI we had little to no test coverage since it was just too time consuming.
As someone that has already had to "rescue" several of those oft-praised "AI-first" projects you hear all about, I can safely say this: "It's not going super well."
The LLM-produced code I've seen thus far (from moderate feature PRs up to and including entire services), tends twords 'too large' or 'too complex' to properly vet by their small team of creators; PRs against existing codebases are often riddled with minor, seemingly unrelated changes or worse: large and/or subtle test suite alterations that "all pass" but contain hidden assumptions in conflict with reality and your business requirements. Numerous edge cases go entirely missed, overly trivial things are validated instead, etc. Feedback loops with the system certainly improves output but is frustrating and in many cases no faster than just writing the code yourself. At this stage you still need a human in the loop, unless you're still in the earliest stages of building a product. This being HN I'm sure someone around here is employing LLMs successfully in those cases, but the story for established orgs tends to be more complex, especially in traditionally risk averse fields like healthcare, billing, credit card processing, defense, etc. All fields I've worked in.
The sheer amount of code these LLM systems produce in aggregate means that we have a deficit of cognitive spoons across our orgs to properly review and test all these changes that are being pumped out. In other words: more bugs end up getting found in the field instead of earlier because we're allowing our generators to validate themselves due to resource constraints.
Can we produce code faster than ever before? Sure, but that was never the real bottleneck to begin with, at least in the orgs that I've operated within the past 2 decades.
To me, every new line of code needs an inherent justification for its existence. Code is often as much a liability as it is an asset. Think on several axis like risk of security vulnerabilities or cognitive load limits when no one left in your org understands your vibe-coded system anymore, as it grows more unwieldy by the day. The business value of new code being introduced should outway its (not so) hidden cost of maintenance and risk of business disruption. With today's emphasis on using LLM output for "moving faster", we seem to be ignoring that critical risk/reward analysis and operating under the incorrect assumption that more code is universally a good thing.
So for the moment I'm just using LLMs for "rubber ducking" or spitballing/testing out ideas before committing to a course of action; an action that will ultimately get executed by a human in the short term at least.
That said, on a good day my AI can serve as a proxy for a semi-competent software engineer (with amnesia), and no worse than an actual rubber duck on the others.
(note that the above statements are my own personal observations and are not intended to represent or express any statement or opinion held by my employer, etc, etc.)
On small teams, AI feels like a big boost. On larger teams, it mostly just moves the work around.
We (12+ people on the team) see more LOC and faster first drafts, but also more review work. PRs look done early, but often hide shallow thinking or edge cases. Velocity goes up on paper. Review fatigue goes up too.
The best teams treat AI like a junior dev with infinite energy. Great for boilerplate and refactors, but you still need ownership. Otherwise you just ship bugs faster, which is not great honestly.
Wdyt?
Not a full team adoption story, but relevant data point: I run a small engineering org (~40 engineers across teams) and we've been tracking AI coding tool adoption informally.
The split is roughly: 30% all-in (Claude Code or Cursor for everything), 50% selective users (use it for boilerplate, tests, docs but still hand-write core logic), 20% holdouts.
What I've noticed on PR velocity: it went up initially, then plateaued. The PRs got bigger, which means reviews take longer. We actually had to introduce a "max diff size" policy because AI-assisted PRs were becoming 800+ line monsters that nobody could review meaningfully.
The quality concern that keeps coming up: security. AI-generated code tends to take shortcuts on auth, input validation, error handling. We've started running dedicated security scans specifically tuned for patterns that AI likes to produce. That's been the biggest process change.
Net effect: probably 20-30% faster on feature delivery, but we're spending more time on review and security validation than before.
The joke I hear is Claude Code will double your PRs
One PR from Claude. The next PR from you fixing Claude’s mistakes.
Ha, pretty accurate in my experience. Though I'd say it's more like 1.5x the PRs - Claude does the initial PR, then you do half a PR fixing the subtle stuff it got wrong, and then you spend the other half wondering if you missed something.
The security fixes are the worst because the code looks correct. It's not like a typo you'd catch immediately - it's an auth check that works for 95% of cases but fails on edge cases the model never considered.
I have seen the same Ai hallucinations that you mentioned: auth, input validation, error handling, non-existent dependencies, etc. It's tricky to get them all as LLM's have mastered the art of being "confidently wrong". What tools are you using to catch those issues? I feel current tooling is ill equiped for this new wave of Ai generated output.
"Confidently wrong" is the perfect description. The code compiles, the tests pass (because the AI also wrote the tests to match), and the auth flow looks reasonable at first glance.
For catching these we layer a few things:
- Standard SAST (Semgrep, CodeQL) catches the obvious stuff but misses AI-specific patterns - npm audit / pip-audit for dependency issues, especially non-existent packages the AI hallucinates - Custom rules tuned for patterns we keep seeing: overly permissive CORS, missing rate limiting, auth checks that look correct but have subtle logic bugs - Manual review with a specific checklist for AI-generated code (different from our normal review checklist)
You're right that current tooling has a gap. Traditional scanners assume human-written code patterns. AI code looks structurally different - it tends to be more verbose but miss edge cases in ways humans wouldn't. We've been experimenting with scanning approaches specifically tuned for AI output.
The biggest wins have been simple: requiring all AI-generated auth and input validation code to go through a dedicated security reviewer, not just a regular code review.
Here are some real examples from our projects in 2025 at SIROC (for context: we are a 18 people venture studio; 140+ projects completed):
* A task estimated at 4 hours → solved with one well specified prompt
* A 20 hour engineering effort → executed in about 3 hours
* A 3 month project → delivered in 1 month
These are clearly best case scenarios. They are not the norm, yet. But they demonstrate what is possible.
We have also seen what happens when things go wrong. Companies, including startups, come to us with broken systems and spaghetti code and architecture caused by weak prompts, unclear requirements, and no verification.
It is important to understand that the efficiency gains we are seeing do not come from the tools alone. They come from a specific combination:
1) Engineers who have spent 20 years building everything from robotics to enterprise-scale technology. You cannot give a perfect instruction to an AI if you do not know what perfect looks like in a production environment.
2) A technical prompt should not be treated as a quick input or question. It is a detailed specification that requires experience and deliberate thinking.
3) Knowing the right combination of tools, workflows, and validation processes.
That said, some (many?) members of our team are dinosaurs in the software engineering world. They bring a ton of experience but are used to tools from 15 years ago and don't like change. We really had to push AI adoption (mostly Cursor and Claude Code) on them. It’s still an ongoing process, and probably will be for a while.
I was thinking last night about whether this is even a realistic moat or not.
Right now, Claude is getting trained by hundreds of thousands of programmers showing it how to ask the right architecture + PM questions.
They're just patterns, like anything else in our industry, and most of them are pretty standard patterns.
Like when I think back on 20 years of software architecture and BA work, I've done the same thing over and over. I must have implemented 4 PO systems, 3 different custom chat systems, SMS systems for reminders, monthly summary emails, etc.
Some people seem like fully gave in to using AI for everything, and nothing better than receiving multiple review requests per day for 2k+ diff PRs.. where most of the stuff is just "wtf" type of refactoring and added noise
Also funny to hear reactions when I'm grepping code for a function in front of someone and I hear "wow you look for code I would just ask Gemini" topkek
I use AI for some of my issues, but it typically doesn't write the code for me. If it does, I review every line and transform it into my style, etc.
I was on a greenfield project late last year with a team that was very enthusiastic about coding agents. I would personally call it a failure, and the project is quietly being wound down after only a few months. It went in a few stages:
At first, it proceeded very quickly. Using agents, the team were able to generate a lot of code very fast, and so they were checking off requirements at an amazing pace. PRs were rubber stamped, and I found myself arguing with copy/pasted answers from an agent most of the time I tried to offer feedback.
As the components started to get more integrated, things started breaking. At first these were obvious things with easy fixes, like some code calling other code with wrong arguments, and the coding agents could handle those. But a lot of the code was written in the overly-defensive style agents were fond of, so there were a lot more subtle errors. Things like the agent adding code to substitute an invalid default value in instead of erroring out, far away from where that value was causing other errors.
At this point, the agents started making things strictly worse because they couldn't fit that much code in their context. Instead of actually fixing bugs, they'd catch any exceptions and substitute in more defaults. There was some manual work by some engineers to remove a lot of the defensive code, but they could not keep up with the agents. This is also about when the team discovered that most of the tests were effectively "assert true" because they mocked out so much.
We did ship the project, but it shipped in an incredibly buggy state, and also the performance was terrible. And, as I said, it's now being wound down. That's probably the right thing to do because it would be easier to restart from scratch than try to make sense of the mess we ended up with. Agents were used to write the documentation, and very little of it is comprehensible.
We did screw some things up. People were so enthusiastic about agents, and they produced so much code so fast, that code reviews were essentially non-existent. Instead of taking action on feedback in the reviews, a lot of the time there was some LLM-generated "won't do" response that sounded plausible enough that it could convince managers that the reviewers were slowing things down. We also didn't explicitly figure out things like how error-handling or logging should work ahead of time, and so what the agents did was all over the place depending on what was in their context.
Maybe the whole mess was a necessary learning as we figure out these new ways of working. Personally I'm still using the coding agents, but very selectively to "fill-in-the-blanks" on code where I know what it should look like, but don't need to write it all by hand myself.
I'd expect this kind of outcome to be common for anything complex, but there seem to be more claims of success stories than horror stories.
Various possibilities I suppose - I could just be overly skeptical, or failures might be kept on the down low, or perhaps most likely: many companies haven't actually reached the point where the hidden tech debt of using these things comes full circle.
Having been through it, what's your current impression of the success stories when you come across them?
I'd speculate we had a few factors working against us that made us hit the "limit" sooner.
Several different engineering teams from different parts of the company had to come together for this, and the overall architecture was modular, so there was a lot of complexity before we had to start integrating. We have some company-wide standards and conventions, but they don't cover everything. To work on the code, you might need to know module A does something one way and module B does it in a different way because different teams were involved. That was implicit in how human engineers worked on it, and so it wasn't explicitly explained to the coding agents.
The project was in the life sciences space, and the quality of code in the training data has to be worse than something like a B2B SaaS app. A lot of code in the domain is written by scientists, not software engineers, and only needs to work long enough to publish the paper. So any code an LLM writes is going to look like that by default unless an engineer is paying attention.
I don't know that either of those would be insurmountable if the company were willing to burn more tokens, but I'd guess it's an order of magnitude more than we spent already.
There are politics as well. There have been other changes in the company, and it seems like the current leadership wants to free up resources to work on completely different things, so there's no will to throw more tokens at untangling the mess.
I don't disbelieve the success stories, but I think most of them are either at the level of following already successful patterns instead of doing much novel, or from companies with much bigger budgets for inference. If Anthropic burns a bunch of money to make a C compiler, they can make it back from increased investor hype, but most companies are not in that position.
I'm personally using it a lot. For my workplace, the productivity gains are limited by human code review (need a minimum of two approvals) and manual testing. I can produce a ton of PRs, more than ever before, but I was already bottlenecked by review & test in the hand-coding era.
100% of the team (14 engineers) is using Claude Code for everything. Anecdotally it feels like velocity is 10%-20% faster. Objectively I have no idea if velocity is faster – because management is inept and does not actually track this. The most noticeable thing is test coverage has greatly improved. Every PR ships with lots of tests, whereas before we used AI we had little to no test coverage since it was just too time consuming.
They got fired by an AI exploit.
As someone that has already had to "rescue" several of those oft-praised "AI-first" projects you hear all about, I can safely say this: "It's not going super well."
The LLM-produced code I've seen thus far (from moderate feature PRs up to and including entire services), tends twords 'too large' or 'too complex' to properly vet by their small team of creators; PRs against existing codebases are often riddled with minor, seemingly unrelated changes or worse: large and/or subtle test suite alterations that "all pass" but contain hidden assumptions in conflict with reality and your business requirements. Numerous edge cases go entirely missed, overly trivial things are validated instead, etc. Feedback loops with the system certainly improves output but is frustrating and in many cases no faster than just writing the code yourself. At this stage you still need a human in the loop, unless you're still in the earliest stages of building a product. This being HN I'm sure someone around here is employing LLMs successfully in those cases, but the story for established orgs tends to be more complex, especially in traditionally risk averse fields like healthcare, billing, credit card processing, defense, etc. All fields I've worked in.
The sheer amount of code these LLM systems produce in aggregate means that we have a deficit of cognitive spoons across our orgs to properly review and test all these changes that are being pumped out. In other words: more bugs end up getting found in the field instead of earlier because we're allowing our generators to validate themselves due to resource constraints.
Can we produce code faster than ever before? Sure, but that was never the real bottleneck to begin with, at least in the orgs that I've operated within the past 2 decades.
To me, every new line of code needs an inherent justification for its existence. Code is often as much a liability as it is an asset. Think on several axis like risk of security vulnerabilities or cognitive load limits when no one left in your org understands your vibe-coded system anymore, as it grows more unwieldy by the day. The business value of new code being introduced should outway its (not so) hidden cost of maintenance and risk of business disruption. With today's emphasis on using LLM output for "moving faster", we seem to be ignoring that critical risk/reward analysis and operating under the incorrect assumption that more code is universally a good thing.
So for the moment I'm just using LLMs for "rubber ducking" or spitballing/testing out ideas before committing to a course of action; an action that will ultimately get executed by a human in the short term at least.
That said, on a good day my AI can serve as a proxy for a semi-competent software engineer (with amnesia), and no worse than an actual rubber duck on the others.
(note that the above statements are my own personal observations and are not intended to represent or express any statement or opinion held by my employer, etc, etc.)