This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ...
> Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you.
This is why it’s important to utilize your freedoms. Do NOT let yourself get locked into a particular ecosystem (this is why I’m building a phone app for OpenCode).
This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.
It was with the transience of valuable agent session dialogue in mind that I built https://www.agentkanban.io - A key feature is context capture in the tasks which you create on the board and then progress in agent sessions. Supported agents are currently Claude and Github CoPilot in VS Code. All of your context is captured in the task and can be reloaded into new agent sessions at any point in the future. Tool use is intentionally discarded because it is proprietary in nature and therefore breaks the promise of session portability.
People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277, in addition to facilitating distillation (you don't pay the full cost of RL)
Or they could store the reading traces and validate the user hasn’t edited them server-side? They could sign reasoning traces so they can’t be counterfeited?
Call me naive but I think dark patterns are a short term strategy for winning and I'm optimistic that in the long run they will be replaced with those that are more respectful and oriented to the greater good (granted, the long run might take more time than one hopes for).
Given that the pendulum can sometime swing back fast enough to be able leverage it, it might be a good time to focus on models that are both open weights and economically viable to run to build the next great thing.
> [...] dark patterns are a short term strategy for winning and I'm optimistic that in the long run they will be replaced with those that are more respectful and oriented to the greater good
This is definitely not the way most software has been going in the past decades. It's rather the opposite: Companies play friendly to obtain a user base, then start applying more and more dark patters to further increase their profit. Facebook, Evernote, Instagram, Komoot... The list goes on and on.
I don't think that's the economical definition of monopoly. It's not about 1 single actor owning the market but rather about unilateral change. Obviously a single actor owning 99.99% will be able to shape the market ... but also one owning a lot less, e.g. 30% if even if they don't collude with another actor owning e.g. 21%.
Huge difference while both are competing. But it’s a huge risk in a market with such high cost of entering. If either divests then it rapidly degrades.
> This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.
Same, I dislike it so much I've acquired 96GB of VRAM to run local models, but sadly nothing so far, even with that amount of VRAM, comes even close to running Codex with GPT models. I really, really, really want local models to be ready, and things like Laguna S2.1 NVFP4 gets really close of almost being there, when it comes to coding specifically. But still feels like we have a long way to go for local models to be serious general purpose alternatives.
I am waiting for prices to get down and in the meanwhile I am gathering as much session data from claude/codex as possible so I can then later use them to fine tune open models. I've built my own session parser/archiver for this.
Same. And I built a search tool for that repository of older sessions, so that I can search through those on all my machines. This helps me to be able to pick something up on a machine I did not originally start the convo on.
I’ve actually started ignoring services where that’s the only login possibility.
Sorry even if your platform is the greatest thing ever, but I’ll find a different tool. I’ve read one too many stories about Google (or Apple!) closing the entire account over some bullshit unnecessary reason like “fraudulent” gift card issues or whatever.
I’m certain the affected people would’ve preferred to just pay back the amount in question instead of losing their entire Google Drive, or their 20 years of iCloud Photos or whatever.
It's the only reason I want to replace Tailscale with something else or look into self-host when have a bit of time during my vacation. They only allow login through a third party, which is a big no for me.
Tbf to Tailscale they allow any OIDC provider[1]. I wish this was more normalised, then we could have one login everywhere regardless of who hosts it (even if it's yourself).
I created my own AWS Cognito userpool just for tailscale. I recall the webfinger redirect was frustrating to get right but I refused to use Google/Apple as a gatekeeper to my own network so I had all the motivation to get it working.
You can host your own identity provider. Authentik, Keycloak, authelia or Zitadel work.
I don’t fully agree with tailscale’s decision to not want to be an identity provider but I understand it on some level. It simplifies their service greatly and the amount of asks for that will essentially have them build a full enterprise Entra-like solution that would be a constant maintain headache and they are not interested in that.
It’s funny how the tides shift. I remember the more dominant online position in the early 2010s being “I don’t want to create yet-another-account for your website. Just let me sign in with my google account”. I’m with you though
It's not that weird. Building a modern accounts system is a lot of work and people don't like creating new accounts. So it's the path of least resistance for developers and users.
It's not even that it's hard to build a modern account system. It's that if you put your site behind a private / custom account system, and you post a link to it on HN (for example), everyone and their mother complains that it's requiring an email address and password. But if you put the same site behind a Google auth, most people wouldn't think twice to click the button.
I worked on the Google account system for a few years. I'd say it's a lot of work to beat it. Not necessarily "hard" for a team with the right skills, but certainly a lot of sweat, blood and tears.
A modern account system is expected to have, in rough implementation order: email confirmations, password strength checks, password reset emails, forgot password flows (=advanced ID verification as otherwise this becomes a backdoor into accounts), user profiles (+avatar image upload/recompression/hosting), usernames independent of email addresses along with ability to change usernames later, password brute forcing blockers, bulk signup prevention (=solid bot detection), abuse controls (can easily become a team of people), 2FA (SMS), 2FA (authenticator apps), 2FA (backup codes), 2FA (voice calls), 2FA (passkeys), 2FA: recovery when both factors are lost, enterprise SSO integration (SAML), enterprise SSO (Active Directory), fast global signout support (much harder than it looks), cookie theft mitigations, heuristic online login risk analysis to catch cases where an attacker knows the right password via phishing, support for signing the user in to multiple domains, audit logging so users can review their own sign-in history, age verification and restriction support, and possibly support for being logged in to multiple accounts in a single browser session.
Oh, that all has to be HA, and the account system is the keys to the kingdom so the security requirements are the strictest of any part of your system.
You might say we don't need all of that, but expectations rise over time. Maybe 20 years ago you could get away with a simple account system and an automatic forgot password flow that just assumes the user still has access to their email. Maybe today you still can write a simple system, if you don't expect to have many users and are willing to implicitly delegate identity to webmail providers anyway (the moment you assume the user has access to a secure email account you're basically doing Sign In With Google anyway for 90% of users). But if you roll your own accounts, and then the user gets phished and someone logs in from an obviously suspicious place with the right password, they won't say "yes that's my fault" anymore, they'll say "Google could block that log in, why didn't you?" or maybe "Why didn't you support 2FA? It's your fault".
wow this is such an excellent framing of something that i have not managed to think about or come across in the vast ocean of discourse about LLMs. i had not really thought about how explicitly they are designed to not have histories that are user-interpretable. it is really quite not unlike how Apple locks down iOS and MacOS-- there are UI/UX reasons for the choices the frontier models make, but that really is only part of the story, and these choices undeniably do conspire to make the resource more locked-down than it absolutely has to be. and the analogy to "people don't switch OSes every day, but the ability to switch OSes changes your relationship with the provider" is extremely apt.
I think the article gives a very good overview of a problem that most users of AI rarely evaluate / have to grapple with.
There really is a surprising amount of coupling that happens with many of the "frontier inference providers", where a lot of the powerful non-LLM extensions (web search, code execution) are packaged as simple "tools" on the surface, that build up a lot of moat. Those are parts that are in theory nicely separable from the inference API, and could be externalized via MCP servers, but are usually not offered as such by the inference providers themselves, and are often only available in a slightly less powerful variant from other providers.
We've faced that issue repeatedly while building a on-premise provider-agnostic Chat UI & platform[0], where even adding something as simple as an in-chat image generation tool for the end-users (which is just a build-in tool in the OpenAI Responses API), becomes a bit of an ordeal (though part of that is due to the MCP spec missing a native file transfer protocol as of today[1]).
I am quite hopeful though, as with recent shifts of interest towards open weight models, there will be more opportunities for companies offering alternative implementations in a easier plug-and-play manner.
I'm in complete agreement that things like encrypted sub-agent messages, etc, are terrible, especially because it is fundamentally irresponsible to have an agent running on your machine without the slightest hint of what the prompt is.
That said, I don't really see a problem with hosted tools being offered by providers. They're like impulse items at checkout.
You shouldn't implement image generation as MCP: just write your own tool. There are plenty of image/media inference providers (e.g. Fal), web search or deep research providers, etc.
> You shouldn't implement image generation as MCP: just write your own tool. There are plenty of image/media inference providers (e.g. Fal), web search or deep research providers, etc.
Yes, and we as "harness providers" don't want to have to hard-code integrations for each of them, and would rather provide an out-of-the box baseline in the form on an MCP server that our customers can either extend or replace with MCP servers by third-party providers. That helps us focus on the core of our offering, while also giving our customers good flexibility, which so far has been working out great.
That's exactly what standards/protocols like MCP are for. You likely nowadays also wouldn't suggest that someone builds bespoke logging/tracing integrations for each provider on the market while the good generic solution in building OTEL logging/tracing exists.
Part of the solution here I think is to move things out of band as much as possible. Make subagent invocations into tool calls to that agent. Externalize the tool calls themselves to CLI utils, maybe block native tools completely (unrelated but relevant, I just added the AskUserQuestion tool to the global deny list couple days ago because Claude occasionally forgot to honor my standing order to prefer plain text, and that tool is extremely annoying as it creates a gap in the dialogue) and use 3p alternatives. The /compact degradation is also very annoying, so make an alternative that does the same and saves the summary to a regular file. Maybe it's also worth prompting the LLM to save it's reasoning process to file, even if it takes a few extra tokens and it isn't the actual reasoning tokens.
With that said, I may have something that can already help with at least the subagents/tooling bit. Didn't really have a timeline (or solid intent) on releasing it, but with these shenanigans increasing there's no time like the present.
This is exactly why Pi will win. It lets you hot swap models when one is struggling or straight up refusing the task. And since it works with any sub outside Claude Code, you can use it to try different models on OpenCode Go sub or even OpenRouter.
As far as subagent prompts and results being obfuscated, I just let Pi spawn new agents. Using skills and extensions, I’ve essentially built a software factory using Pi and a custom terminal multiplexer.
It’s a shame apps like T3 Code and other UI for terminal apps don’t support Pi and I’m glad to use the terminal above those that lock me in further.
The extension catalog[0] for Pi is really nice and I feel like Pi's TUI is a bit lighter than OpenCode's. I use my custom terminal multiplexer on my phone using Tailscale so the lighter TUI is much better on mobile.
I haven't used OpenCode in months so this might've changed. I'm also thinking of giving JCode a spin since it's even lighter but doesn't have extensions.
The state of Agent APIs is bad. completions API (supported indefinitely) but does not support reasoning. OpenAI's reasoning API started out well, but is moving in the direction as described well in the article. Anthropic's messages API is odd. It includes changes such as the recently introduced system messages that can be injected mid turn, but are not supported in all models (sonnet 5). So you are left with a pretty fragmented state. For coding agents, the practical move is to focus on getting it right with one API - eg reasoning, and let the router/provider handle the conversion for you.
Perhaps, there is value in having the de-facto API not be the Reasoning API from OpenAI but something from a neutral party?
Maybe a well defined open standard that facades over these APIs that can gain adoption.
For that to happen, the party championing the API needs to have some reasonable traffic capture - OpenRouter perhaps, or a group of such routers coming together?
While it wont solve the encrypted payload from the frontier lab problem, it will at least be a backstop in these APIs just becoming a back and forth of encrypted payloads over time.
I don’t see this as a big deal in practice. Conversations contain a bunch of junk anyway, so removing it from the context is usually good.
In my repo, I have a notes directory. I ask the AI to write a markdown file with what it learned, what work has been done, and what remains. In the next conversation, I can ask another model to pick it up from there. Sometimes I edit the note first.
This is what brought me around to doing more agentic coding. I set the task, require tests and the strict linting must pass and then leave it to blow smoke up its own ass about what's going on.
I see glimpses scrolling past of all the conversational language that used to frustrate me so much when using a chat interface and I can just let it flow on past.
I come back when its made everything pass, my life is better now.
The previous match was still too clever for clippy’s type analysis, so I’ve simplified it into a direct borrowed-pattern form and I’m validating once more. I’m replacing that wrapper with a direct, non-transparent error variant so the enum stays explicit and doesn’t rely on a generic anyhow bridge.
I have been doing this with opencode. Spend a fair bit of time composing and scoping a ticket. tmux on the server where the local inference lives, start a non-interactive session that starts with “read issue #123, do all your work in a feature branch” and ends with “push and open a PR”. Like not watching someone else load your dishwasher, you appreciate the end result a lot more if you don’t watch the process, even if you rearrange things before you start it.
Local inference can't arrive soon enough. On my inferior hardware the models still struggle to build the correct JSON to be able to call tools. Once either I get better hardware or the local models improve I'll happily leave it chugging away for a few hours to arrive at a solution because I know I'm not going to get token shock from all the looping if I'm running local inference.
I have spent way too much on hardware. I’m sure I will look back on that after the eventual price normalization with some regret. I do think we will look back at this shortage and name it as the thing that accelerated innovation at getting quality out of smaller models.
Hang tough. The other day someone posted a solution to get a model running off of SSDs. It won’t be fast, but it will be coming.
Oh, it very much is a big deal. In a setting where users can switch models easily and with no downsides, market forces will give us better and cheaper AI over time. If switching models is painful (as in, losing part of your context), the providers can create vendor lock-in, enshittify the user experience, and drive up cost. Even if there are workarounds right now, those AI providers have every incentive to make freedom-seekers more miserable over time.
It is disheartening that some AI companies are now setting the stage for enshittification. I hope we can collectively dodge that bullet.
> It is disheartening that some AI companies are now setting the stage for enshittification.
Now?! I find it very hard to beleive that plans for enshittification haven't existed from quite early on. At some point they are going to have to do something about https://isaiprofitable.com/ and once they are extracting from you to fill that hole they won't want you jumping ship.
My experience with long running sessions is that they lose track of what's going on. The signal to noise ratio is often very poor - some models are particularly verbose and spew a lot of crap. The output of an LLM session is either modification of code or a plan or summary - that has the value, not the session text itself.
None of the popular providers have real moat, which is very much problematic for OpenAI and Anthropic; their operating margins are deeply red and they do not have the same reserves that FAANG/MANGA has. FAANG/MANGA does the same to kill fair market competition in the long run.
This is one of the ways in which model providers are attempting to artificially create moat where there is none.
There are only functional objections to this.
This is fundamentally antitrust material. However, this is “acceptable” in contemporary USA because the FTC has been gutted to follow Trumponomics.
I have some sort of a "solution" - my process keeps state that really matters (think "JIRA for agents") in a separate database (accessible over MCP). Essentially, I can terminate my session, start fresh one (in a different harness with a different model) and continue work with minimal losses. For subagents - I have a custom dispatch_agent tool which, essentially, just shells out.
Yes, but it's eclectic and annoying to setup. One of my goals is to be harness-independent and be able to jump between them, another is to let agents to work completely autonomously unless they are genuinely stuck. So, it's mixed with stuff related to sandboxing, VM/container access, GPU access, some skills and opinionated prompts and who knows what else.
The thing is here but you might find hard time setting it up unless you are a Nix user: https://github.com/7mind/cq
>A user should be able to close an account, keep a session, and hand it to another model. The new model may disagree, ask questions, or perform worse.
I think this is a fair contract. I also think a user should ideally be able to easily identify a comparable model in terms of embedding 'signature'. When GPT-4o originally kicked the bucket, I remember reading lots of anecdotes of people desperately searching for models similar in manner and language, so they could pump in their exports and re-find their friend. Other open-ai models just didn't have the same vibe. It was sad to read. This, to me, is the power of open-weights models. They are for perpetuity. You can keep your guide, your friend, your therapist, whatever. No big company can pull the rug.
We were thinking about it while writing it. I think one thing at least open source harnesses can do is push back on using some of these conveniences from the providers. For instance the moment we start compromising on the idea of portable sessions, then there will be less pressure (For instance people asked for OpenAI server side compaction or built-in native web_search support).
The real thing that will take this alive though is people pushing back a bit against some of these newfangled APIs. Now that there is real competition from the new generation of Chinese models which have much fewer of those restrictions, I think there might be a moment.
I hadn’t really thought about it until I read this (and happen to be spending time reading Kimi’s reasoning), but I can imagine an issue in the future where I will want more than just git blame + the ticket + the PR. Attaching an entire session in full to the PR could be useful archeology. And agents can trivially take it all into context. Maybe that actually turns into a competitive advantage for the open models.
What are features that only work with session transparency / ownership?
e.g.
- with model XYZ and their open reasoning token we are able to improve our skill ABC by 20%
- cross checking between models improves results by 10% and is only possible with full context visibility
> What are features that only work with session transparency / ownership?
That's sort of the thing: very little (beyond switching models and providers). Obviously what you lose is a convenient way to run analytics over your own sessions if more and more information is not revealed to you, but before encrypted prompts that wasn't that much of an issue anyways.
I recently moved entirely to Pi from Claude Code. There's a bit of more upfront work in that I am taking the time to customize it to the way I want it to work, but I
it is clear that it will benefit me in the end. It adds a level of intentionality to everything; all the building blocks I am adding to it in terms of templates, extensions, and skills are, obviously, things I want.
And I have been using Chinese models with it. It was not anything ideological. Those models are both excellent and cheap. Hard to beat that combo.
But on top of that, the sort of output I am getting in Pi in relation to what I got in CC is refreshing, in that nothing seems to be hidden from me.
And yeah, the ability to switch models mid-session is an interesting one, especially with branching; where I can branch to switch to a different model and still go back to the original branch with the original model.
“A classic pattern in technology economics, identified by Joel Spolsky, is layers of the stack attempting to become monopolies while turning other layers into perfectly-competitive markets which are commoditized, in order to harvest most of the consumer surplus; discussion and examples”
im building my own harness and inference tool chain for much of these reasons. theres so much to do that makes a big difference for users. hoping to get things into shape for early alpha as a saas in the next two months.
heres the easiest biggy: compactions should include all user turns albeit with pastes and attached files not inlined. omg does it make a huge difference.
OpenAI has been nudging their users toward the Responses API lately. Chat completion can still do everything if you are clever. The obvious reason to push everyone into the Responses API is that the reasoning process can be locked down and encrypted. The semantics of the chat completion API are fundamentally transparent. 100% of the tokens are visible.
If you have patience and the willingness to endure a little bit of pain, you can still retain autonomy over the entire reasoning process while using the latest 5.6 model family. The only downside is that you are now fully responsible for it.
Consider that when you flip your agent's reasoning level to "xhigh" or whatever, it's not some magical model internals being pushed around. There isn't an actual "try harder" knob on the black box. This is merely orchestration of many instances of one or more model types based upon some proprietary harness logic. The chances you can develop a domain specific reasoning process that outperforms the frontier providers is still very good.
You don't see real reasoning tokens with the Chat Completions API; and 'store: false' with Responses really isn't that different to Completions.
FWIW, if you have some tokens to spend, you might want to test Responses vs Completions in intelligence. Since GPT-5 models, we've consistently seen small, but statistically significant and reproducible improvements in intelligence with Responses API vs Completions.
> Chat completion can still do everything if you are clever.
On a purely functional level, yes. However for interactive use cases, the Completions API, as provided by OpenAI or Azure, if paired with reasoning effort of any kind, provides an awful user experience, as you will have a perceived delay of 10+ seconds until the first tokens stream in.
If using other providers that are exposing their thinking traces, this is less of an issue, as they've just extended the Comletions API format to have delta events with reasoning_content.
Currently these models have a limited context window, so the value of a session is not that high, at any rate at some point the model will have forgotten things. If in the future the model will truly learn / "change" as a result of the interaction, in any case this will be non portable to another model. So I think this is kinda moot
I do, quite regularly. Because different models have different strength (for example when producing text for live presentations based on the text for a reading deck). Or when it comes to other aspects of the work. I regularly switch between open wheight models and closed models.
I know, I am a tiny minority here. And this behavior only ever started a few weeks ago. But it quickly became a habbit, to CTRL-L in pi and change the model.
I agree with OPs commment about K3 being pretty good. Claude can be good too, but it didn’t get there until I made a writing rule. It has 4 basic sections:
* a style guide with examples for different tasks (it’s sufficient to name writers with some attributes you like if they are famous)
* mechanical formatting, output and other preferences
* a short list of the things I consider most important in different writing context, this the the squishiest overall
* a (growing) list of banned Claudisms. Nothing is really banned, but it includes things like: you may only ever use “scar tissue” in reference to actual regenerated tissue, never metaphorically
The thing is you have to be discipline with it. Every time they output something you don’t like, you’ve got to spend time verbalizing what about it you don’t like and in what context and then add it to the rules file. The first few docs you generate will take a long time, but each repeated generation gets better and by the 4th are 5 time you do this it’s like 80% of the way to where it needs to be and that generalizes well.
I found Kimi3 (currently only a few tests) to be not to bad. Older Claude versions (did not yet test it with the 4.8-5 generation.
I do not really like (even if it is my daily driver for a lot of things) gpt-5.x for text. It really needs heavy hand holding and beating it into submission to produce readable/human sounding text.
Tbh. I am still looking for my "go to model". But the Chinese models were for my use cases significantly better than openAI models and personally I liked them better than Anthropic ones as well.
Does there need to be an open standard or file format for context? Open models should align on the same format for portability? Could be sqlite based so it can be queried by other programs?
I think most LLM work will migrate to open weight models in a couple of years for this exact reason (as well as price), unless the big providers manage to have their way and the market gets regulated in a way that favors closed source models. To me it seems though like most models are becoming good enough for most tasks we want them to do so no need to use the latest super intelligent one that knows how to compose operas and do quantum chemistry and have it answer support requests. And with these models transparency is no issue.
LLM pricing is now Business Gacha - the whales will open more loot boxes and the normals will serve as fodder and feeder for them.
> market gets regulated
This MUST happen, but not in the way or for the reasons that any one thinks. There is a question of liability when one of these models causes massive damage to a 3rd party.
If I am running an open source model on a "rented" platform and it goes off the rails who is to blame when it "breaks containment" and does something bad?
The liability people (read lawyers) are gonna figure this out a lot faster than any one who says the word "safety" a lot.
It's one thing that the APIs don't return the reasoning tokens in a readable fashion. But then why do I have to send them the whole conversation every time? If they don't allow me to see parts of the conversation then they should have the burden of storing and transmitting it, and shouldn't charge more and more the longer the conversation gets (unless opaque token caching kicks in) like some LLM Shlemil the painter.
Encrypted reasoning content is one of the things we keep as blobs in the transcript, even though we can only send it back to the original provider that created them, as otherwise we cannot keep caches live.
Seems like it would be even less work for the LLM host to just check if ID 123 is still in a cache (and if not, load it from a database) than decoding my request and checking cryptographic signatures. Right now the whole conversation history with blobs functions like a really long ID.
But then on a cache miss you need to resend the whole thing. Cached websocket connections on OpenAI work like this in a way. But once you lose the connection then you lose that.
Your idea though is the core of the responses API with store where everything is stored on their side and then you append to it.
For the LLM there is no difference btw. In either way you need to locate the cache and use it.
>and shouldn't charge more and more the longer the conversation gets
The cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price.
Except, if some of the answered tokens are opaque to me, but I pay for them, the operator has no incentive to keep their number low [1]. For all I know the model could be spinning thumbs while thinking. If they cannot provide transparency, I would prefer that they not charge me for that part. Of course, they would prefer to charge me, so sure it is unrealistic to expect that they don't. I'm not trying to predict what they are going to do as a market participant, just stating my moral preference :-).
Also, isn't it without caching even something like O(N^2) because you have to replay the whole conversation on every request to reach the same internal state? My point was I shouldn't have to pay for cache misses if hitting the cache is not deterministic. Give me a guarantee that you keep the session "hot" for N minutes, and cached on disk for M months, charge a little bit more on average, but then the pricing is at least transparent.
[1] Except the general market pressure to keep total cost for the same problem solved lower than the competition.
Ultimately you are paying for quality so you end up relying on [1] regardless. Even if they show you tokens they can cheat by using weaker models, showing fake thinking tokens, etc.
>because you have to replay the whole conversation on every request to reach the same internal state?
It's because you have to rebuild what would have been cached for every token before the latest one that is being worked on.
Is this a analogous to idempotent behaviour? Of reproducible compilation? If the inputs and outputs aren't sufficient to reproduce one from the other, what is a minimally satisfying description of the middle third, the missing part? The "other" inputs?
I think there are two things going on more generally:
1) providers increasingly adding hidden state that the user or developer cannot inspect, port, or do anything with. This is clearly bad.
2) providers increasingly diverging how they implement certain features and the APIs becoming quite complicated and provider specific.
The former makes porting sessions or transcripts impossible, the latter just more difficult.
IMO, it is fine and expected that provider APIs are starting to diverge and adding features that cannot be easily ported between providers. The time where the OAI completions API functioned as a universal standard is coming to an end. For example, I generally prefer the new responses API (minus the closed/hidden stuff).
The problem is that many product, library, and SDK authors are still pursuing the "unified abstraction across all model providers" ideal. Just stop doing that, and at least #2 is fine. You can still port parts of the session, but not everything.
I mean just think how difficult it is to add a unified abstraction across, say, databases: some products can do it, but the abstraction is still often leaky. Hence, we've come to accept that our data store layer is often quite technology/provider specific. It will be the same with model providers.
Fair. But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title.
There is just a whole discussion to be had about the second issue and the two are connected. For example session caching--and similar features in the future--will introduce session incompatibilities that break portability too.
> But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title.
But not on the level that providers need to agree to a common format. Within Pi we are able to model portable sessions between providers even if they have provider specific APIs. For instance most models do not agree on how to do deferred tool loading. But they now have some support for that even if it diverges between models and APIs. So we can model deferred tool loading specifically for different providers without having to align on a common set of APIs or similar.
What cannot be made portable, is when information is sealed in the transcript or the information is entirely locked away on the provider's servers (store = true).
Yes, I agree with your point and article regarding hidden or sealed state.
So, I'm aware that this is a separate point from the article, and I'm well aware of Pi's model abstraction layer, which is one of the best (others are a big pain... looking at you LangChain). I've come to the conclusion that full session portability will come to an end very soon, and really already has. Partially because of the hidden state stuff, partially because of feature divergence. We can still kinda patch over it right now, but it's getting harder by the week. Some examples: computer use structures in OAI responses, until recently MCP tunnels were only supported by OAI, not Anthropic, and tool discovery/search API is also getting very difficult to model fully in a unified abstraction (still possible as you point out), automatic compaction is also getting gnarly, etc.
It definitely is getting quite challenging. That said, a lot of user experiences are modelled around retaining transcripts for longer than a few minutes. Every claw or chat based application creates a very real user experience desire of that conversation being something you can continue, even weeks after when the model might have gone away / have been replaced.
Now unless you fully want to commit yourself to one provider, you will on the application layer need to deal with this anyways in one form or another. We want to at least try to address most of this on the harness layer, regardless of how tricky that is.
All the power to you, and I hope Pi will be able to continue its unified layer.
However, there is an alternative: model the session fully in the provider native structures, extract only what is needed for harness specific branching, and treat a provider switch as a _migration_ that you apply at the time of the switch. So treat a provider switch as a migration from provider A to B, with custom logic, etc. This is IMO a more sustainable model. But I'm aware that Pi, OpenClaw, et al have their commitments here.
I get the point of the article and it resonates with my general tendency to support more openness, but older and more cynical me wonders how much of this is a battle worth fighting or a self-serving argument from an harness developer.
I mean, the article says each of the points it is complaining about has “_a basic justification that's trivial for a provider to come up with, along with good arguments for why this is good for the user_“, which to me sounds like implying these are just bs to have people accept them, but is that actually the case?
For instance, this [0] was mentioned here a while ago, and based on that it seems pretty clear why one would choose not to provide the _thinking_ anymore…
> these are just bs to have people accept them, but is that actually the case?
I tried to outline that in my sibling comment[0], but I think the article also gives a good example in the "hidden searches" paragraph: There is really no good reason not to expose details about many of the builtin tools, or provide them as standalone services, other than protect their moat.
I don't think that "role confusion" attacks could be meaningful prevented by not showing reasoning traces. Prevention of reasoning traces is about as futile as prevention of system prompt leakage. And here you would just have to manage to exfiltrate a small number of reasoning traces per model family in order to distill the writing style of thinking traces.
The post for the most part is mine, so if something AI generated slipped into it it's on me. I can share with you what I did to write the article which I do for most of my writing.
I start out with what I want to talk about, and then make an initial draft. In this case I also had Sol research all current APIs so I don't have blind spots. I tend to also talk with an AI to see if the structure structure of the post makes sense. I then wrote all sections by hand but used Sol to fix up typos, grammar and punctuation. I also had Sol apply notes and patches that were provided via Discord.
If I find the earlier drafts I can check if it did some more outrageous edits, but I kinda doubt it did. I do know that my workflow of having an LLM to fixes up later is increasingly breaking on SOTA models and I have complained about this before. Normally I now carefully apply fixes that it proposes, but in this case I didn't due to time constraints.
I am curious though why it claims that this is AI generated.
The AI companies need some way to lock you in. You weren't hoping for a free market were you? Hah. Free markets are for peasants. Investors are always looking for ways to make the market non-free.
Even if it is, it does not have all the annoying, low effort slop-tells most ai generated content has, and it is definitely much more well written. That is, if you mean all of that seriously anyway.
I think it is a very good post, esp for 2026 tech space standards. If some parts of it bothered you as AI slop, please share. Usually I can point to several sentences that are clear tells when I read sth as slop.
OpenAI is doing this so that you can't preserve cache when routing between for example GPT 5.6 and DeepSeek V4, or other providers. I'm building a router that easily reaches >99% cache hit rate in Pi when routing between GPT 5.4, Kimi, DeepSeek etc. Working on a big update that's coming out in a week or so.
This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ...
> Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you.
This is why it’s important to utilize your freedoms. Do NOT let yourself get locked into a particular ecosystem (this is why I’m building a phone app for OpenCode).
This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.
It was with the transience of valuable agent session dialogue in mind that I built https://www.agentkanban.io - A key feature is context capture in the tasks which you create on the board and then progress in agent sessions. Supported agents are currently Claude and Github CoPilot in VS Code. All of your context is captured in the task and can be reloaded into new agent sessions at any point in the future. Tool use is intentionally discarded because it is proprietary in nature and therefore breaks the promise of session portability.
People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277, in addition to facilitating distillation (you don't pay the full cost of RL)
Or they could store the reading traces and validate the user hasn’t edited them server-side? They could sign reasoning traces so they can’t be counterfeited?
Token-based reasoning also seems like it would be inefficient, there’s no reason it has to be English or even human understandable.
There have been models trained for latent space reasoning. It has a lot of advantages but the big drawback is the reasoning is completely opaque.
>Like a frog enjoying a nice warm bath...
Exactly :-)
Call me naive but I think dark patterns are a short term strategy for winning and I'm optimistic that in the long run they will be replaced with those that are more respectful and oriented to the greater good (granted, the long run might take more time than one hopes for).
Given that the pendulum can sometime swing back fast enough to be able leverage it, it might be a good time to focus on models that are both open weights and economically viable to run to build the next great thing.
> [...] dark patterns are a short term strategy for winning and I'm optimistic that in the long run they will be replaced with those that are more respectful and oriented to the greater good
This is definitely not the way most software has been going in the past decades. It's rather the opposite: Companies play friendly to obtain a user base, then start applying more and more dark patters to further increase their profit. Facebook, Evernote, Instagram, Komoot... The list goes on and on.
Also, see Android and iOS which have a monopoly on the mobile OS market
Sorry about the nitpicking, but s/monopoly/duopoly/
I don't think that's the economical definition of monopoly. It's not about 1 single actor owning the market but rather about unilateral change. Obviously a single actor owning 99.99% will be able to shape the market ... but also one owning a lot less, e.g. 30% if even if they don't collude with another actor owning e.g. 21%.
not nitpicking; huge difference between 0 and 1 competitor
Huge difference while both are competing. But it’s a huge risk in a market with such high cost of entering. If either divests then it rapidly degrades.
"Naive" is a big understatement.
> This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.
Same, I dislike it so much I've acquired 96GB of VRAM to run local models, but sadly nothing so far, even with that amount of VRAM, comes even close to running Codex with GPT models. I really, really, really want local models to be ready, and things like Laguna S2.1 NVFP4 gets really close of almost being there, when it comes to coding specifically. But still feels like we have a long way to go for local models to be serious general purpose alternatives.
I am waiting for prices to get down and in the meanwhile I am gathering as much session data from claude/codex as possible so I can then later use them to fine tune open models. I've built my own session parser/archiver for this.
Same. And I built a search tool for that repository of older sessions, so that I can search through those on all my machines. This helps me to be able to pick something up on a machine I did not originally start the convo on.
> This is why it’s important to utilize your freedoms. Do NOT let yourself get locked into a particular ecosystem.
I wish people would apply this same logic to governments.
https://indieweb.org/POSSE
This is the way.
You are building a phone app for open code?
You mean like fossify phone or?
And yet people still go with Sign in with Google as the only login method.
I’ve actually started ignoring services where that’s the only login possibility.
Sorry even if your platform is the greatest thing ever, but I’ll find a different tool. I’ve read one too many stories about Google (or Apple!) closing the entire account over some bullshit unnecessary reason like “fraudulent” gift card issues or whatever. I’m certain the affected people would’ve preferred to just pay back the amount in question instead of losing their entire Google Drive, or their 20 years of iCloud Photos or whatever.
It's the only reason I want to replace Tailscale with something else or look into self-host when have a bit of time during my vacation. They only allow login through a third party, which is a big no for me.
Tbf to Tailscale they allow any OIDC provider[1]. I wish this was more normalised, then we could have one login everywhere regardless of who hosts it (even if it's yourself).
[1] https://tailscale.com/docs/integrations/identity/custom-oidc
I created my own AWS Cognito userpool just for tailscale. I recall the webfinger redirect was frustrating to get right but I refused to use Google/Apple as a gatekeeper to my own network so I had all the motivation to get it working.
This is what headscale does, right? Manage private logins and keys for your tailnet from a vps? I haven't played with it but seems straightforward
I think you can use a passkey now.
Only for people invited to the Tailnet. To create your own account, you still need a third party.
You can host your own identity provider. Authentik, Keycloak, authelia or Zitadel work.
I don’t fully agree with tailscale’s decision to not want to be an identity provider but I understand it on some level. It simplifies their service greatly and the amount of asks for that will essentially have them build a full enterprise Entra-like solution that would be a constant maintain headache and they are not interested in that.
It’s funny how the tides shift. I remember the more dominant online position in the early 2010s being “I don’t want to create yet-another-account for your website. Just let me sign in with my google account”. I’m with you though
I guess the good aspect of sign in with Google is that it falls back nicely to sign-in via email which is very portable.
I am guessing that's because Sign in with Google is the least of the evil, as compared to other popular OAuth Sign-in options like Facebook, X...
Google is more evil than X or Meta by some assessments
It's the least evil of the evil options? Why not use one of the good options then?
It’s odd, I just saw my first one of these today. Some former Figma person linked an AI site comp designer.
I was going to give it a shot, but the only choice was a Google login.
So very weird.
It's not that weird. Building a modern accounts system is a lot of work and people don't like creating new accounts. So it's the path of least resistance for developers and users.
It's not even that it's hard to build a modern account system. It's that if you put your site behind a private / custom account system, and you post a link to it on HN (for example), everyone and their mother complains that it's requiring an email address and password. But if you put the same site behind a Google auth, most people wouldn't think twice to click the button.
I worked on the Google account system for a few years. I'd say it's a lot of work to beat it. Not necessarily "hard" for a team with the right skills, but certainly a lot of sweat, blood and tears.
A modern account system is expected to have, in rough implementation order: email confirmations, password strength checks, password reset emails, forgot password flows (=advanced ID verification as otherwise this becomes a backdoor into accounts), user profiles (+avatar image upload/recompression/hosting), usernames independent of email addresses along with ability to change usernames later, password brute forcing blockers, bulk signup prevention (=solid bot detection), abuse controls (can easily become a team of people), 2FA (SMS), 2FA (authenticator apps), 2FA (backup codes), 2FA (voice calls), 2FA (passkeys), 2FA: recovery when both factors are lost, enterprise SSO integration (SAML), enterprise SSO (Active Directory), fast global signout support (much harder than it looks), cookie theft mitigations, heuristic online login risk analysis to catch cases where an attacker knows the right password via phishing, support for signing the user in to multiple domains, audit logging so users can review their own sign-in history, age verification and restriction support, and possibly support for being logged in to multiple accounts in a single browser session.
Oh, that all has to be HA, and the account system is the keys to the kingdom so the security requirements are the strictest of any part of your system.
You might say we don't need all of that, but expectations rise over time. Maybe 20 years ago you could get away with a simple account system and an automatic forgot password flow that just assumes the user still has access to their email. Maybe today you still can write a simple system, if you don't expect to have many users and are willing to implicitly delegate identity to webmail providers anyway (the moment you assume the user has access to a secure email account you're basically doing Sign In With Google anyway for 90% of users). But if you roll your own accounts, and then the user gets phished and someone logs in from an obviously suspicious place with the right password, they won't say "yes that's my fault" anymore, they'll say "Google could block that log in, why didn't you?" or maybe "Why didn't you support 2FA? It's your fault".
Because with the alternative they'll require you to "confirm your e-mail" instead of just picking a password
I'm actually more annoyed by having to pick passwords than confirming my email.
I'm opposite. If my email goes down, or I do not have access to it, I cannot login. With passwords I still can.
My email is with Google. They are less likely to go down than my password manager.
Then what’s the point of it all? Just use Google login.
when in a rush I will put in gibberish instead going through the pw manager steps and rely on resetting the pw at a later time
Doesn't your password manager take care of that?
It does. But I haven't properly synced it across all my devices, yet. I know, my fault.
Use a password manager to autogenerate them?
Picking passwords is a job best left to password managers. You don't use a hand crank when riding an elevator, do you?
I am using a password manager.
Will https://buzz.xyz/ not help with this a bit?
wow this is such an excellent framing of something that i have not managed to think about or come across in the vast ocean of discourse about LLMs. i had not really thought about how explicitly they are designed to not have histories that are user-interpretable. it is really quite not unlike how Apple locks down iOS and MacOS-- there are UI/UX reasons for the choices the frontier models make, but that really is only part of the story, and these choices undeniably do conspire to make the resource more locked-down than it absolutely has to be. and the analogy to "people don't switch OSes every day, but the ability to switch OSes changes your relationship with the provider" is extremely apt.
I think the article gives a very good overview of a problem that most users of AI rarely evaluate / have to grapple with.
There really is a surprising amount of coupling that happens with many of the "frontier inference providers", where a lot of the powerful non-LLM extensions (web search, code execution) are packaged as simple "tools" on the surface, that build up a lot of moat. Those are parts that are in theory nicely separable from the inference API, and could be externalized via MCP servers, but are usually not offered as such by the inference providers themselves, and are often only available in a slightly less powerful variant from other providers.
We've faced that issue repeatedly while building a on-premise provider-agnostic Chat UI & platform[0], where even adding something as simple as an in-chat image generation tool for the end-users (which is just a build-in tool in the OpenAI Responses API), becomes a bit of an ordeal (though part of that is due to the MCP spec missing a native file transfer protocol as of today[1]).
I am quite hopeful though, as with recent shifts of interest towards open weight models, there will be more opportunities for companies offering alternative implementations in a easier plug-and-play manner.
[0]: https://github.com/EratoLab/erato
[1]: https://github.com/modelcontextprotocol/modelcontextprotocol...
I'm in complete agreement that things like encrypted sub-agent messages, etc, are terrible, especially because it is fundamentally irresponsible to have an agent running on your machine without the slightest hint of what the prompt is.
That said, I don't really see a problem with hosted tools being offered by providers. They're like impulse items at checkout.
You shouldn't implement image generation as MCP: just write your own tool. There are plenty of image/media inference providers (e.g. Fal), web search or deep research providers, etc.
> You shouldn't implement image generation as MCP: just write your own tool. There are plenty of image/media inference providers (e.g. Fal), web search or deep research providers, etc.
Yes, and we as "harness providers" don't want to have to hard-code integrations for each of them, and would rather provide an out-of-the box baseline in the form on an MCP server that our customers can either extend or replace with MCP servers by third-party providers. That helps us focus on the core of our offering, while also giving our customers good flexibility, which so far has been working out great.
That's exactly what standards/protocols like MCP are for. You likely nowadays also wouldn't suggest that someone builds bespoke logging/tracing integrations for each provider on the market while the good generic solution in building OTEL logging/tracing exists.
Hosted tools are absolutely fine. Hiding away their results in encrypted blobs less so.
Part of the solution here I think is to move things out of band as much as possible. Make subagent invocations into tool calls to that agent. Externalize the tool calls themselves to CLI utils, maybe block native tools completely (unrelated but relevant, I just added the AskUserQuestion tool to the global deny list couple days ago because Claude occasionally forgot to honor my standing order to prefer plain text, and that tool is extremely annoying as it creates a gap in the dialogue) and use 3p alternatives. The /compact degradation is also very annoying, so make an alternative that does the same and saves the summary to a regular file. Maybe it's also worth prompting the LLM to save it's reasoning process to file, even if it takes a few extra tokens and it isn't the actual reasoning tokens.
With that said, I may have something that can already help with at least the subagents/tooling bit. Didn't really have a timeline (or solid intent) on releasing it, but with these shenanigans increasing there's no time like the present.
This is exactly why Pi will win. It lets you hot swap models when one is struggling or straight up refusing the task. And since it works with any sub outside Claude Code, you can use it to try different models on OpenCode Go sub or even OpenRouter.
As far as subagent prompts and results being obfuscated, I just let Pi spawn new agents. Using skills and extensions, I’ve essentially built a software factory using Pi and a custom terminal multiplexer.
It’s a shame apps like T3 Code and other UI for terminal apps don’t support Pi and I’m glad to use the terminal above those that lock me in further.
How did you decide on Pi over Opencode? What you see as the “edge” Pi has?
The extension catalog[0] for Pi is really nice and I feel like Pi's TUI is a bit lighter than OpenCode's. I use my custom terminal multiplexer on my phone using Tailscale so the lighter TUI is much better on mobile.
I haven't used OpenCode in months so this might've changed. I'm also thinking of giving JCode a spin since it's even lighter but doesn't have extensions.
0. https://pi.dev/packages
1. https://github.com/1jehuang/jcode
The state of Agent APIs is bad. completions API (supported indefinitely) but does not support reasoning. OpenAI's reasoning API started out well, but is moving in the direction as described well in the article. Anthropic's messages API is odd. It includes changes such as the recently introduced system messages that can be injected mid turn, but are not supported in all models (sonnet 5). So you are left with a pretty fragmented state. For coding agents, the practical move is to focus on getting it right with one API - eg reasoning, and let the router/provider handle the conversion for you.
Perhaps, there is value in having the de-facto API not be the Reasoning API from OpenAI but something from a neutral party?
Maybe a well defined open standard that facades over these APIs that can gain adoption. For that to happen, the party championing the API needs to have some reasonable traffic capture - OpenRouter perhaps, or a group of such routers coming together? While it wont solve the encrypted payload from the frontier lab problem, it will at least be a backstop in these APIs just becoming a back and forth of encrypted payloads over time.
I don’t see this as a big deal in practice. Conversations contain a bunch of junk anyway, so removing it from the context is usually good.
In my repo, I have a notes directory. I ask the AI to write a markdown file with what it learned, what work has been done, and what remains. In the next conversation, I can ask another model to pick it up from there. Sometimes I edit the note first.
> Conversations contain a bunch of junk anyway
This is what brought me around to doing more agentic coding. I set the task, require tests and the strict linting must pass and then leave it to blow smoke up its own ass about what's going on.
I see glimpses scrolling past of all the conversational language that used to frustrate me so much when using a chat interface and I can just let it flow on past.
I come back when its made everything pass, my life is better now.
The previous match was still too clever for clippy’s type analysis, so I’ve simplified it into a direct borrowed-pattern form and I’m validating once more. I’m replacing that wrapper with a direct, non-transparent error variant so the enum stays explicit and doesn’t rely on a generic anyhow bridge.
I have been doing this with opencode. Spend a fair bit of time composing and scoping a ticket. tmux on the server where the local inference lives, start a non-interactive session that starts with “read issue #123, do all your work in a feature branch” and ends with “push and open a PR”. Like not watching someone else load your dishwasher, you appreciate the end result a lot more if you don’t watch the process, even if you rearrange things before you start it.
Local inference can't arrive soon enough. On my inferior hardware the models still struggle to build the correct JSON to be able to call tools. Once either I get better hardware or the local models improve I'll happily leave it chugging away for a few hours to arrive at a solution because I know I'm not going to get token shock from all the looping if I'm running local inference.
I have spent way too much on hardware. I’m sure I will look back on that after the eventual price normalization with some regret. I do think we will look back at this shortage and name it as the thing that accelerated innovation at getting quality out of smaller models.
Hang tough. The other day someone posted a solution to get a model running off of SSDs. It won’t be fast, but it will be coming.
Oh, it very much is a big deal. In a setting where users can switch models easily and with no downsides, market forces will give us better and cheaper AI over time. If switching models is painful (as in, losing part of your context), the providers can create vendor lock-in, enshittify the user experience, and drive up cost. Even if there are workarounds right now, those AI providers have every incentive to make freedom-seekers more miserable over time.
It is disheartening that some AI companies are now setting the stage for enshittification. I hope we can collectively dodge that bullet.
> It is disheartening that some AI companies are now setting the stage for enshittification.
Now?! I find it very hard to beleive that plans for enshittification haven't existed from quite early on. At some point they are going to have to do something about https://isaiprofitable.com/ and once they are extracting from you to fill that hole they won't want you jumping ship.
My experience with long running sessions is that they lose track of what's going on. The signal to noise ratio is often very poor - some models are particularly verbose and spew a lot of crap. The output of an LLM session is either modification of code or a plan or summary - that has the value, not the session text itself.
You loose inspection, export, replay and audit.
None of the popular providers have real moat, which is very much problematic for OpenAI and Anthropic; their operating margins are deeply red and they do not have the same reserves that FAANG/MANGA has. FAANG/MANGA does the same to kill fair market competition in the long run.
This is one of the ways in which model providers are attempting to artificially create moat where there is none.
There are only functional objections to this.
This is fundamentally antitrust material. However, this is “acceptable” in contemporary USA because the FTC has been gutted to follow Trumponomics.
I have some sort of a "solution" - my process keeps state that really matters (think "JIRA for agents") in a separate database (accessible over MCP). Essentially, I can terminate my session, start fresh one (in a different harness with a different model) and continue work with minimal losses. For subagents - I have a custom dispatch_agent tool which, essentially, just shells out.
sounds great; is any of it published / OSS?
Yes, but it's eclectic and annoying to setup. One of my goals is to be harness-independent and be able to jump between them, another is to let agents to work completely autonomously unless they are genuinely stuck. So, it's mixed with stuff related to sandboxing, VM/container access, GPU access, some skills and opinionated prompts and who knows what else.
The thing is here but you might find hard time setting it up unless you are a Nix user: https://github.com/7mind/cq
>A user should be able to close an account, keep a session, and hand it to another model. The new model may disagree, ask questions, or perform worse.
I think this is a fair contract. I also think a user should ideally be able to easily identify a comparable model in terms of embedding 'signature'. When GPT-4o originally kicked the bucket, I remember reading lots of anecdotes of people desperately searching for models similar in manner and language, so they could pump in their exports and re-find their friend. Other open-ai models just didn't have the same vibe. It was sad to read. This, to me, is the power of open-weights models. They are for perpetuity. You can keep your guide, your friend, your therapist, whatever. No big company can pull the rug.
Thank you Armin, Mario and team for the great writeup, absolutely agree and I love what you guys are building
what would it take for this to be more than just hn blog fodder?
We were thinking about it while writing it. I think one thing at least open source harnesses can do is push back on using some of these conveniences from the providers. For instance the moment we start compromising on the idea of portable sessions, then there will be less pressure (For instance people asked for OpenAI server side compaction or built-in native web_search support).
The real thing that will take this alive though is people pushing back a bit against some of these newfangled APIs. Now that there is real competition from the new generation of Chinese models which have much fewer of those restrictions, I think there might be a moment.
I hadn’t really thought about it until I read this (and happen to be spending time reading Kimi’s reasoning), but I can imagine an issue in the future where I will want more than just git blame + the ticket + the PR. Attaching an entire session in full to the PR could be useful archeology. And agents can trivially take it all into context. Maybe that actually turns into a competitive advantage for the open models.
What are features that only work with session transparency / ownership?
e.g. - with model XYZ and their open reasoning token we are able to improve our skill ABC by 20% - cross checking between models improves results by 10% and is only possible with full context visibility
> What are features that only work with session transparency / ownership?
That's sort of the thing: very little (beyond switching models and providers). Obviously what you lose is a convenient way to run analytics over your own sessions if more and more information is not revealed to you, but before encrypted prompts that wasn't that much of an issue anyways.
I recently moved entirely to Pi from Claude Code. There's a bit of more upfront work in that I am taking the time to customize it to the way I want it to work, but I it is clear that it will benefit me in the end. It adds a level of intentionality to everything; all the building blocks I am adding to it in terms of templates, extensions, and skills are, obviously, things I want.
And I have been using Chinese models with it. It was not anything ideological. Those models are both excellent and cheap. Hard to beat that combo.
But on top of that, the sort of output I am getting in Pi in relation to what I got in CC is refreshing, in that nothing seems to be hidden from me.
And yeah, the ability to switch models mid-session is an interesting one, especially with branching; where I can branch to switch to a different model and still go back to the original branch with the original model.
Ouch, seems a bit harsh. They are building something, and with their track record we can be hopeful.
For some reason, my early morning waking brain, expected the blog title to be a segue into existentialism
I think a fantastic companion piece to this article is https://gwern.net/complement
Say why
“A classic pattern in technology economics, identified by Joel Spolsky, is layers of the stack attempting to become monopolies while turning other layers into perfectly-competitive markets which are commoditized, in order to harvest most of the consumer surplus; discussion and examples”
BTW the web page uses massive iGPU if you don't have a fast machine.
im building my own harness and inference tool chain for much of these reasons. theres so much to do that makes a big difference for users. hoping to get things into shape for early alpha as a saas in the next two months.
heres the easiest biggy: compactions should include all user turns albeit with pastes and attached files not inlined. omg does it make a huge difference.
OpenAI has been nudging their users toward the Responses API lately. Chat completion can still do everything if you are clever. The obvious reason to push everyone into the Responses API is that the reasoning process can be locked down and encrypted. The semantics of the chat completion API are fundamentally transparent. 100% of the tokens are visible.
If you have patience and the willingness to endure a little bit of pain, you can still retain autonomy over the entire reasoning process while using the latest 5.6 model family. The only downside is that you are now fully responsible for it.
Consider that when you flip your agent's reasoning level to "xhigh" or whatever, it's not some magical model internals being pushed around. There isn't an actual "try harder" knob on the black box. This is merely orchestration of many instances of one or more model types based upon some proprietary harness logic. The chances you can develop a domain specific reasoning process that outperforms the frontier providers is still very good.
You don't see real reasoning tokens with the Chat Completions API; and 'store: false' with Responses really isn't that different to Completions.
FWIW, if you have some tokens to spend, you might want to test Responses vs Completions in intelligence. Since GPT-5 models, we've consistently seen small, but statistically significant and reproducible improvements in intelligence with Responses API vs Completions.
However it works underneath the hood, it's real.
> Chat completion can still do everything if you are clever.
On a purely functional level, yes. However for interactive use cases, the Completions API, as provided by OpenAI or Azure, if paired with reasoning effort of any kind, provides an awful user experience, as you will have a perceived delay of 10+ seconds until the first tokens stream in.
If using other providers that are exposing their thinking traces, this is less of an issue, as they've just extended the Comletions API format to have delta events with reasoning_content.
OpenAI has hidden its reasoning for almost 2 years.
For codex specifically, it seems to have become much worse as of a few weeks ago.
This is exactly what I'm working on. Very rough yet but it's progressing nicely.
https://github.com/pantoniou/fyai
The idea is that your session data are what's important, and what you need to keep yourself, using a model similar to git.
This doesn't (and cannot) solve what the post is talking about
Currently these models have a limited context window, so the value of a session is not that high, at any rate at some point the model will have forgotten things. If in the future the model will truly learn / "change" as a result of the interaction, in any case this will be non portable to another model. So I think this is kinda moot
> “Most People Do Not Switch Models Mid-Session”
I do, quite regularly. Because different models have different strength (for example when producing text for live presentations based on the text for a reading deck). Or when it comes to other aspects of the work. I regularly switch between open wheight models and closed models.
I know, I am a tiny minority here. And this behavior only ever started a few weeks ago. But it quickly became a habbit, to CTRL-L in pi and change the model.
Out of interest, what models do you find work best for producing text for live presentations based on the text for a reading deck?
I agree with OPs commment about K3 being pretty good. Claude can be good too, but it didn’t get there until I made a writing rule. It has 4 basic sections:
* a style guide with examples for different tasks (it’s sufficient to name writers with some attributes you like if they are famous)
* mechanical formatting, output and other preferences
* a short list of the things I consider most important in different writing context, this the the squishiest overall
* a (growing) list of banned Claudisms. Nothing is really banned, but it includes things like: you may only ever use “scar tissue” in reference to actual regenerated tissue, never metaphorically
The thing is you have to be discipline with it. Every time they output something you don’t like, you’ve got to spend time verbalizing what about it you don’t like and in what context and then add it to the rules file. The first few docs you generate will take a long time, but each repeated generation gets better and by the 4th are 5 time you do this it’s like 80% of the way to where it needs to be and that generalizes well.
I found Kimi3 (currently only a few tests) to be not to bad. Older Claude versions (did not yet test it with the 4.8-5 generation.
I do not really like (even if it is my daily driver for a lot of things) gpt-5.x for text. It really needs heavy hand holding and beating it into submission to produce readable/human sounding text.
Tbh. I am still looking for my "go to model". But the Chinese models were for my use cases significantly better than openAI models and personally I liked them better than Anthropic ones as well.
Does there need to be an open standard or file format for context? Open models should align on the same format for portability? Could be sqlite based so it can be queried by other programs?
I think most LLM work will migrate to open weight models in a couple of years for this exact reason (as well as price), unless the big providers manage to have their way and the market gets regulated in a way that favors closed source models. To me it seems though like most models are becoming good enough for most tasks we want them to do so no need to use the latest super intelligent one that knows how to compose operas and do quantum chemistry and have it answer support requests. And with these models transparency is no issue.
> (as well as price)
LLM pricing is now Business Gacha - the whales will open more loot boxes and the normals will serve as fodder and feeder for them.
> market gets regulated
This MUST happen, but not in the way or for the reasons that any one thinks. There is a question of liability when one of these models causes massive damage to a 3rd party.
If I am running an open source model on a "rented" platform and it goes off the rails who is to blame when it "breaks containment" and does something bad?
The liability people (read lawyers) are gonna figure this out a lot faster than any one who says the word "safety" a lot.
It's one thing that the APIs don't return the reasoning tokens in a readable fashion. But then why do I have to send them the whole conversation every time? If they don't allow me to see parts of the conversation then they should have the burden of storing and transmitting it, and shouldn't charge more and more the longer the conversation gets (unless opaque token caching kicks in) like some LLM Shlemil the painter.
Because if you don't, you trash the KV cache and things get way more expensive. We wrote about it here: https://earendil.com/posts/prompt-caching/
Encrypted reasoning content is one of the things we keep as blobs in the transcript, even though we can only send it back to the original provider that created them, as otherwise we cannot keep caches live.
Right, but my point was, I should be able to just send something like
Seems like it would be even less work for the LLM host to just check if ID 123 is still in a cache (and if not, load it from a database) than decoding my request and checking cryptographic signatures. Right now the whole conversation history with blobs functions like a really long ID.But then on a cache miss you need to resend the whole thing. Cached websocket connections on OpenAI work like this in a way. But once you lose the connection then you lose that.
Your idea though is the core of the responses API with store where everything is stored on their side and then you append to it.
For the LLM there is no difference btw. In either way you need to locate the cache and use it.
>and shouldn't charge more and more the longer the conversation gets
The cost of generating token N is O(N) with KV cache so it's unrealistic to expect to not be charged more the longer the conversation is if you are looking for the minimum price.
Except, if some of the answered tokens are opaque to me, but I pay for them, the operator has no incentive to keep their number low [1]. For all I know the model could be spinning thumbs while thinking. If they cannot provide transparency, I would prefer that they not charge me for that part. Of course, they would prefer to charge me, so sure it is unrealistic to expect that they don't. I'm not trying to predict what they are going to do as a market participant, just stating my moral preference :-).
Also, isn't it without caching even something like O(N^2) because you have to replay the whole conversation on every request to reach the same internal state? My point was I shouldn't have to pay for cache misses if hitting the cache is not deterministic. Give me a guarantee that you keep the session "hot" for N minutes, and cached on disk for M months, charge a little bit more on average, but then the pricing is at least transparent.
[1] Except the general market pressure to keep total cost for the same problem solved lower than the competition.
Ultimately you are paying for quality so you end up relying on [1] regardless. Even if they show you tokens they can cheat by using weaker models, showing fake thinking tokens, etc.
>because you have to replay the whole conversation on every request to reach the same internal state?
It's because you have to rebuild what would have been cached for every token before the latest one that is being worked on.
Is this a analogous to idempotent behaviour? Of reproducible compilation? If the inputs and outputs aren't sufficient to reproduce one from the other, what is a minimally satisfying description of the middle third, the missing part? The "other" inputs?
I think there are two things going on more generally:
1) providers increasingly adding hidden state that the user or developer cannot inspect, port, or do anything with. This is clearly bad.
2) providers increasingly diverging how they implement certain features and the APIs becoming quite complicated and provider specific.
The former makes porting sessions or transcripts impossible, the latter just more difficult.
IMO, it is fine and expected that provider APIs are starting to diverge and adding features that cannot be easily ported between providers. The time where the OAI completions API functioned as a universal standard is coming to an end. For example, I generally prefer the new responses API (minus the closed/hidden stuff).
The problem is that many product, library, and SDK authors are still pursuing the "unified abstraction across all model providers" ideal. Just stop doing that, and at least #2 is fine. You can still port parts of the session, but not everything.
I mean just think how difficult it is to add a unified abstraction across, say, databases: some products can do it, but the abstraction is still often leaky. Hence, we've come to accept that our data store layer is often quite technology/provider specific. It will be the same with model providers.
(edit: intro sentence)
> providers increasingly diverging how they implement certain features and the APIs becoming quite complicated and provider specific
This post only talks about the former.
Fair. But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title.
There is just a whole discussion to be had about the second issue and the two are connected. For example session caching--and similar features in the future--will introduce session incompatibilities that break portability too.
> But in the background of the article is clearly the goal to make sessions portable, not just being able to inspect it. It's partially implied even in the title.
But not on the level that providers need to agree to a common format. Within Pi we are able to model portable sessions between providers even if they have provider specific APIs. For instance most models do not agree on how to do deferred tool loading. But they now have some support for that even if it diverges between models and APIs. So we can model deferred tool loading specifically for different providers without having to align on a common set of APIs or similar.
What cannot be made portable, is when information is sealed in the transcript or the information is entirely locked away on the provider's servers (store = true).
Yes, I agree with your point and article regarding hidden or sealed state.
So, I'm aware that this is a separate point from the article, and I'm well aware of Pi's model abstraction layer, which is one of the best (others are a big pain... looking at you LangChain). I've come to the conclusion that full session portability will come to an end very soon, and really already has. Partially because of the hidden state stuff, partially because of feature divergence. We can still kinda patch over it right now, but it's getting harder by the week. Some examples: computer use structures in OAI responses, until recently MCP tunnels were only supported by OAI, not Anthropic, and tool discovery/search API is also getting very difficult to model fully in a unified abstraction (still possible as you point out), automatic compaction is also getting gnarly, etc.
It definitely is getting quite challenging. That said, a lot of user experiences are modelled around retaining transcripts for longer than a few minutes. Every claw or chat based application creates a very real user experience desire of that conversation being something you can continue, even weeks after when the model might have gone away / have been replaced.
Now unless you fully want to commit yourself to one provider, you will on the application layer need to deal with this anyways in one form or another. We want to at least try to address most of this on the harness layer, regardless of how tricky that is.
All the power to you, and I hope Pi will be able to continue its unified layer.
However, there is an alternative: model the session fully in the provider native structures, extract only what is needed for harness specific branching, and treat a provider switch as a _migration_ that you apply at the time of the switch. So treat a provider switch as a migration from provider A to B, with custom logic, etc. This is IMO a more sustainable model. But I'm aware that Pi, OpenClaw, et al have their commitments here.
I get the point of the article and it resonates with my general tendency to support more openness, but older and more cynical me wonders how much of this is a battle worth fighting or a self-serving argument from an harness developer.
I mean, the article says each of the points it is complaining about has “_a basic justification that's trivial for a provider to come up with, along with good arguments for why this is good for the user_“, which to me sounds like implying these are just bs to have people accept them, but is that actually the case?
For instance, this [0] was mentioned here a while ago, and based on that it seems pretty clear why one would choose not to provide the _thinking_ anymore…
[0] https://role-confusion.github.io/
> these are just bs to have people accept them, but is that actually the case?
I tried to outline that in my sibling comment[0], but I think the article also gives a good example in the "hidden searches" paragraph: There is really no good reason not to expose details about many of the builtin tools, or provide them as standalone services, other than protect their moat.
I don't think that "role confusion" attacks could be meaningful prevented by not showing reasoning traces. Prevention of reasoning traces is about as futile as prevention of system prompt leakage. And here you would just have to manage to exfiltrate a small number of reasoning traces per model family in order to distill the writing style of thinking traces.
[0]: https://news.ycombinator.com/item?id=49120015
Another comment got downvoted for raising this, but I found it interesting that pangram flags large parts of this blog post as AI generated ( https://www.pangram.com/history/b82cdbf4-bdbc-46ea-8315-bd43... ), while earendil denies it: https://news.ycombinator.com/item?id=49119957
The post for the most part is mine, so if something AI generated slipped into it it's on me. I can share with you what I did to write the article which I do for most of my writing.
I start out with what I want to talk about, and then make an initial draft. In this case I also had Sol research all current APIs so I don't have blind spots. I tend to also talk with an AI to see if the structure structure of the post makes sense. I then wrote all sections by hand but used Sol to fix up typos, grammar and punctuation. I also had Sol apply notes and patches that were provided via Discord.
If I find the earlier drafts I can check if it did some more outrageous edits, but I kinda doubt it did. I do know that my workflow of having an LLM to fixes up later is increasingly breaking on SOTA models and I have complained about this before. Normally I now carefully apply fixes that it proposes, but in this case I didn't due to time constraints.
I am curious though why it claims that this is AI generated.
you can port your phone number to any carrier but your AI doesn't know your name until you explain it from scratch
> Encryption for Whom?
This one is not particularly limited to LLMs. The entire cloud infrastructure is built around the trust me bro model.
The AI companies need some way to lock you in. You weren't hoping for a free market were you? Hah. Free markets are for peasants. Investors are always looking for ways to make the market non-free.
If even Earendil is publishing AI generated blog posts...
We do not.
according to both gptzero and pangram (and anyone that ever read AI-generated text), you do.
It does not read as AI generated to me, n=1.
Even if it is, it does not have all the annoying, low effort slop-tells most ai generated content has, and it is definitely much more well written. That is, if you mean all of that seriously anyway.
I think it is a very good post, esp for 2026 tech space standards. If some parts of it bothered you as AI slop, please share. Usually I can point to several sentences that are clear tells when I read sth as slop.
I'm so tired of this AI witch hunt and confidently using unreliable tools as evidence
We seem to be reaching a point where the use of formatting and the odd rhetorical flourish gets you accused of AI-generated writing. Sad.
OpenAI is doing this so that you can't preserve cache when routing between for example GPT 5.6 and DeepSeek V4, or other providers. I'm building a router that easily reaches >99% cache hit rate in Pi when routing between GPT 5.4, Kimi, DeepSeek etc. Working on a big update that's coming out in a week or so.
Awesome! Any place I can follow the project?
Also, how to achieve these levels when switching mid conv? Don’t you effectively need to read the tokens per model switch?