I'm curious about something. If this is based on historical datasets, and people build strategies using LLMs, then in theory this is deeply flawed since LLMs would contain the knowledge about some of the datasets, and certainly the prices of the biotech stocks. This approach cannot be used to figure out which strategies are good because they know the future outcome.
How do you prevent this problem? It's a classic problem in backtesting strategies where you leak future information into the model.
Yes this is a major problem I thought about. The makeshift solution here was to redact the “identifying information” on the press release. Even then, I benchmarked that GPT-5 could still match it back to the right TIKR around 53% of the time. It does not seem to be able to recall the price of the stock in my benchmark, but to be honest I’m not entirely sure how trustworthy this benchmark is and I may need to come up with a few more clever solutions to validate.
One solution could be to get experts to write similar press releases so that the text itself is out of distribution or if an actual quant firm has internal models, they can just make sure that there is a cutoff date to the pre-training data.
I'm curious, when you ran a quant fund, what was your approach?
We didn't use other LLMs. We built our own models and had a system designed to never leak future information at any given timepoint. Models can only access data the system allowed it at any time point gating future information. This means even training has to use the same system.
You have to design it from the ground up with that approach. Just to give you an idea of how hard it is, when a company releases an earnings report, they can update it in the future with corrected information, so if you pull it later you will leak future information into the past. So even basics like earnings need to be versioned by time.
But you know, most people don't really care and think they have an edge, and who knows maybe they do. Only live trading will prove it.
I used to work for a human that did this (sits mostly on the classical therapeutics side). He actually started a business where he was reviewing and auditing the submission processes outlining approvals but he had been around the game enough to know where the next submission would put them in the approvals process for a number of agencies.
Looks like he's still on top of everything given the most recent blog post is from 6/2/2026.
I believe the insights here could be useful given he has sense of when the penultimate submission has occured (but I'm not entirely sure what that is on a % basis nor as a basis for if the stock of the company reacts)
Interesting, biotech stocks have been notoriously hard to predict because their business model revolves around science, and it’s hard to know when the science is right. Depending on the situation, I think sentiment could potentially be a misleading/confounding variable here…
Sentiment is crucial - if you know sentiment is incorrectly oriented, you can capitalize on it. If you know it's correct, you can identify mispricing, and strategize accordingly.
I used to work at a private investment fund as a data engineer for building in house models to evaluate drug programs and biotech companies. We took a pretty varied approach with catalysts, investment data, people data, trial data, but also analyses on the molecule and drug itself. It was a lot of work and I really don't think we made a dent into understanding what succeeds and what doesnt. Also investors in biotech are really underwriting the biology. Its why they mostly invest in fast follow or me toos rather than new technology or new therapies. The work was a bit sad and less exciting than I thought.
That's interesting. I am curious, what kind of analyses did you work with on the molecule and drug itself? Was it like mostly reading papers/patents or did your team do anything experimental?
Why do you think that LLMs would do any better than monkeys throwing darts?
I am raining on your parade but this is another in a long succession of ways to loose money.
The publicly available information in markets is priced very efficiently, us computer types do not like that and we like to think that our pattern analysis machines can do better than a room full of traders. They cannot.
The money to be made in markets is from private information and that is a crime (insider trading), is widespread, and any system like this is fighting it and will loose.
Our initial goal with this project actually wasn't trying to get an edge in terms of better evaluating information, but rather, we wanted to see if an LLM can perform similarly to a human analyst at a lower latency. The latency for the market to react to catalysts is actually surprisingly high in biotech (at least in some cases) compared to other domains so there may be some edge there.
Appreciate the comment though! I generally agree with your sentiment!
I'm curious about something. If this is based on historical datasets, and people build strategies using LLMs, then in theory this is deeply flawed since LLMs would contain the knowledge about some of the datasets, and certainly the prices of the biotech stocks. This approach cannot be used to figure out which strategies are good because they know the future outcome.
How do you prevent this problem? It's a classic problem in backtesting strategies where you leak future information into the model.
EDIT: Some context, I ran a quant fund before.
Yes this is a major problem I thought about. The makeshift solution here was to redact the “identifying information” on the press release. Even then, I benchmarked that GPT-5 could still match it back to the right TIKR around 53% of the time. It does not seem to be able to recall the price of the stock in my benchmark, but to be honest I’m not entirely sure how trustworthy this benchmark is and I may need to come up with a few more clever solutions to validate.
One solution could be to get experts to write similar press releases so that the text itself is out of distribution or if an actual quant firm has internal models, they can just make sure that there is a cutoff date to the pre-training data.
I'm curious, when you ran a quant fund, what was your approach?
We didn't use other LLMs. We built our own models and had a system designed to never leak future information at any given timepoint. Models can only access data the system allowed it at any time point gating future information. This means even training has to use the same system.
You have to design it from the ground up with that approach. Just to give you an idea of how hard it is, when a company releases an earnings report, they can update it in the future with corrected information, so if you pull it later you will leak future information into the past. So even basics like earnings need to be versioned by time.
But you know, most people don't really care and think they have an edge, and who knows maybe they do. Only live trading will prove it.
I used to work for a human that did this (sits mostly on the classical therapeutics side). He actually started a business where he was reviewing and auditing the submission processes outlining approvals but he had been around the game enough to know where the next submission would put them in the approvals process for a number of agencies.
https://maestrodatabase.com/
Looks like he's still on top of everything given the most recent blog post is from 6/2/2026.
I believe the insights here could be useful given he has sense of when the penultimate submission has occured (but I'm not entirely sure what that is on a % basis nor as a basis for if the stock of the company reacts)
Yes we know of a few. Honestly, it was pretty hard to even find a good catalyst calendar for this space.
I'll give it a read to learn more. Thanks for the note!
Interesting, biotech stocks have been notoriously hard to predict because their business model revolves around science, and it’s hard to know when the science is right. Depending on the situation, I think sentiment could potentially be a misleading/confounding variable here…
Sentiment is crucial - if you know sentiment is incorrectly oriented, you can capitalize on it. If you know it's correct, you can identify mispricing, and strategize accordingly.
I used to work at a private investment fund as a data engineer for building in house models to evaluate drug programs and biotech companies. We took a pretty varied approach with catalysts, investment data, people data, trial data, but also analyses on the molecule and drug itself. It was a lot of work and I really don't think we made a dent into understanding what succeeds and what doesnt. Also investors in biotech are really underwriting the biology. Its why they mostly invest in fast follow or me toos rather than new technology or new therapies. The work was a bit sad and less exciting than I thought.
That's interesting. I am curious, what kind of analyses did you work with on the molecule and drug itself? Was it like mostly reading papers/patents or did your team do anything experimental?
Why do you think that LLMs would do any better than monkeys throwing darts?
I am raining on your parade but this is another in a long succession of ways to loose money.
The publicly available information in markets is priced very efficiently, us computer types do not like that and we like to think that our pattern analysis machines can do better than a room full of traders. They cannot.
The money to be made in markets is from private information and that is a crime (insider trading), is widespread, and any system like this is fighting it and will loose.
Our initial goal with this project actually wasn't trying to get an edge in terms of better evaluating information, but rather, we wanted to see if an LLM can perform similarly to a human analyst at a lower latency. The latency for the market to react to catalysts is actually surprisingly high in biotech (at least in some cases) compared to other domains so there may be some edge there.
Appreciate the comment though! I generally agree with your sentiment!
efficiency is not a given. also this is an eval set - they acknowledge the challenge themselves.
imho this is v cool
Very interesting ! Thanks for sharing, any reliable, LLM friendly source that aggregates press releases ?
Thanks for the strategy transparency; it's nice to see how the system and user prompts are written, especially the multi-step agents.