Although I’d like this to work, I have not seen evidence that AIs extract good reusable knowledge from sessions without strong guidance.
I do R&D work in comp sci and usually write software and reports or papers in parallel. I have been using Opus in a controlled human-in-the-loop fashion to (significantly) speed up the work. While I’m at times surprised how high-level input steers output the right way, no high level ideas emerge from the AIs output. Many attempts to abstract from the concrete are wrong.
As a consequence, most writing is poor on content and form. It writes about the wrong things and crosses abstraction levels all the time. It cannot keep implementation details and conceptual leaps apart.
I don’t know what this means for the aim of this project to maintain important info for agents. Perhaps there are enough low-hanging fruits. That said, I’m skeptical that the resulting wiki is good documentation for human contributors.
Understanding what is relevant to be documented vs not is definitely tricky and something we've spent a lot of time on. We found being more prescriptive about what should be documented is helpful, we incorporated guidance from diataxis for it (https://diataxis.fr/)
Also the wiki is primarily for your agents and not for people. Although we're working to improve the prose quality and structure so it is well written for humans too.
Do you mean that specifically those models perform badly without guidance or do all models need specific instructions for technical writing?
We have observed that it is a loss for teamwork that the sparring with AI is entirely lost when only the technical conclusion is recorded. I write to recover that for my team and an AI can use it to its advantage as a side effect.
It would be cool to have good tool support for this process, but I’d bet on support, not delegation in that case.
The models perform badly without guidance but we've found the guidance doesn't need a human in the loop (at least for this task). You can distill what makes a good wiki, what information should go in etc. and the models are good enough to do it reliably. That's what took the most time (researching and studying existing wikis)
When you say support do you mean something where you can collaborate with the model to maintain the wiki?
We built something similar for our team at work. A human has to run it and review/skim for correctness, who quickly gets overwhelmed. The purpose eventually became an AI wiki for AI to use to spawn agents more efficiently (token wise), to pull in only modules it needs and fills in the blanks from the md files. Like a knowledge graph. It has become useful for production support, design, feature dev, domain specific learnings. We also found it makes a huge difference which model you use to encode and which model you use to decode.
Looks very interesting! I'll be trying to set that up. One thing I'm wondering is: I have multiple persistent worktrees for my repo so that I start Claude from different locations (ie folders where this worktree is instantiated). Would it be possible for almanach to reconcile it all together?
I figure I could set it up into each folder and 'ln -S' the almanach folder from the base repo to each worktree. Would that work?
Nice!
Curious about one thing: how do you handle a decision that gets reversed later? If the wiki page just keeps getting updated in place, does it retain any signal that an approach was tried and explicitly abandoned?
Transcripts tend to have secrets pasted into them during debugging, api keys, .env values and so on. Is there anything that keeps the sync agent from writing those into almanac/ pages that get committed?
Are there any plans to sandbox the agents? I suspect it would be fairly trivial on macOS using sandbox-exec. This looks very useful but I hesitate to run yolo mode agents (even with clear instructions) on my machine.
LoCoMo is not quite our use case. It tests conversational memory, while Almanac is more for coding agents trying to find their way around a codebase.
These results show output quality at the same token budget, not token savings yet. We’re working on coding agent evals that should be more representative.
Glad you liked it! Would love to chat more about what kind of production workflows you are looking to improve. You could grab whichever time is convenient for you: https://cal.com/team/almanac/demo
Not in any way being negative about what you’ve done, but… Mac only. Is this just because of inbuilt knowledge about file system shape or some more important Apple exclusive tool/environmental quirk?
Spent some time playing on it just now on Linux. I couldn't easily get the automatic ingestion or gardening working, but everything else is working okay.
For my workflows I think I actually prefer wiki updates being manual and deliberate.
Although I’d like this to work, I have not seen evidence that AIs extract good reusable knowledge from sessions without strong guidance.
I do R&D work in comp sci and usually write software and reports or papers in parallel. I have been using Opus in a controlled human-in-the-loop fashion to (significantly) speed up the work. While I’m at times surprised how high-level input steers output the right way, no high level ideas emerge from the AIs output. Many attempts to abstract from the concrete are wrong.
As a consequence, most writing is poor on content and form. It writes about the wrong things and crosses abstraction levels all the time. It cannot keep implementation details and conceptual leaps apart.
I don’t know what this means for the aim of this project to maintain important info for agents. Perhaps there are enough low-hanging fruits. That said, I’m skeptical that the resulting wiki is good documentation for human contributors.
We've found Opus/Gpt are really bad at technical writing. We've found ways around this by having guidelines like https://www.openalmanac.org/ai-patterns-to-avoid.md, which have worked quite well for us.
Understanding what is relevant to be documented vs not is definitely tricky and something we've spent a lot of time on. We found being more prescriptive about what should be documented is helpful, we incorporated guidance from diataxis for it (https://diataxis.fr/)
Also the wiki is primarily for your agents and not for people. Although we're working to improve the prose quality and structure so it is well written for humans too.
Do you mean that specifically those models perform badly without guidance or do all models need specific instructions for technical writing?
We have observed that it is a loss for teamwork that the sparring with AI is entirely lost when only the technical conclusion is recorded. I write to recover that for my team and an AI can use it to its advantage as a side effect.
It would be cool to have good tool support for this process, but I’d bet on support, not delegation in that case.
The models perform badly without guidance but we've found the guidance doesn't need a human in the loop (at least for this task). You can distill what makes a good wiki, what information should go in etc. and the models are good enough to do it reliably. That's what took the most time (researching and studying existing wikis)
When you say support do you mean something where you can collaborate with the model to maintain the wiki?
Yes: the human provides the qualitative judgement and the agent provides the executive function.
We built something similar for our team at work. A human has to run it and review/skim for correctness, who quickly gets overwhelmed. The purpose eventually became an AI wiki for AI to use to spawn agents more efficiently (token wise), to pull in only modules it needs and fills in the blanks from the md files. Like a knowledge graph. It has become useful for production support, design, feature dev, domain specific learnings. We also found it makes a huge difference which model you use to encode and which model you use to decode.
Looks very interesting! I'll be trying to set that up. One thing I'm wondering is: I have multiple persistent worktrees for my repo so that I start Claude from different locations (ie folders where this worktree is instantiated). Would it be possible for almanach to reconcile it all together? I figure I could set it up into each folder and 'ln -S' the almanach folder from the base repo to each worktree. Would that work?
Nice! Curious about one thing: how do you handle a decision that gets reversed later? If the wiki page just keeps getting updated in place, does it retain any signal that an approach was tried and explicitly abandoned?
Transcripts tend to have secrets pasted into them during debugging, api keys, .env values and so on. Is there anything that keeps the sync agent from writing those into almanac/ pages that get committed?
Are there any plans to sandbox the agents? I suspect it would be fairly trivial on macOS using sandbox-exec. This looks very useful but I hesitate to run yolo mode agents (even with clear instructions) on my machine.
Any data on better outputs and/or token saving?
We’re working on evals now. We ran a small preliminary LoCoMo test with a 2k-token retrieval budget:
Almanac: 55.7% BM25: 51.8% Supermemory: 47.6% Mem0: 60.6%
LoCoMo is not quite our use case. It tests conversational memory, while Almanac is more for coding agents trying to find their way around a codebase.
These results show output quality at the same token budget, not token savings yet. We’re working on coding agent evals that should be more representative.
Cool thanks I will keep an eye on it!
This looks promising. Is there a version I can use to self improve my production workflows and not just the dev work?
Glad you liked it! Would love to chat more about what kind of production workflows you are looking to improve. You could grab whichever time is convenient for you: https://cal.com/team/almanac/demo
Not in any way being negative about what you’ve done, but… Mac only. Is this just because of inbuilt knowledge about file system shape or some more important Apple exclusive tool/environmental quirk?
We’re working on Linux/Windows support right now. Mac was the easiest to setup for us since that’s the machine we code on. What OS do you use?
Not OP, but I use NixOS. I will give it a try later tonight. Do you need public contribution for this?
Would love to hear your thoughts and appreciate public contribution.
Spent some time playing on it just now on Linux. I couldn't easily get the automatic ingestion or gardening working, but everything else is working okay. For my workflows I think I actually prefer wiki updates being manual and deliberate.
wow very nice brother
[dead]
[flagged]
[dead]