Repository navigation
feat: paper fetch - #163
andrea-gariboldi wants to merge 7 commits into
Conversation
MartinekV
left a comment
MartinekV
left a comment
There was a problem hiding this comment.
I agree with the sub-agent delegation approach
This PR is the first step of a tool that is basically question -> answer (supported by literature) for the agent. That can be decomposed into question - get paper pdfs - retrieve from paper pdfs - answer. This PR only handles the get paper pdfs part, however its helpful to consider the next part as well as they might become the same subagent. With that in mind:
I suggest switching from duckduckgo search to literature-specific search option (see comments)
We also need to make sure we retrieve the paper if its available. Some sources might be paywalled.
See other comments
This is not a command to change things - simply to explore that option and discuss it together. Duckduck go might still be a reasonable first version.
| query: A string describing one scientific topic or question for which relevant papers are to be fetched. | ||
| """ | ||
| try: | ||
| websearch_result = await duckduckgo_search_tool(max_results=config.paper_fetch_max_results).function(query=query) |
There was a problem hiding this comment.
Do we need separate credits for duckduckgo search?
AFAIK codex sub actually comes with search capability - you dont need an extra token for that, it comes natively with the models. We could put that as a requirement and only support models that offer native websearch capability. Not sure how that would work with self-hosted/local models though.
I had a little bit of a look and i really like semantic scholar's ASTA - its free, specialized on scientific literature, and has a few components, one of them - PaperFinder - could do this job, probably better than a general search. That would allow us to get rid of the search loop also which would be nice.
At some point we will also need some retrieval over these papers. ASTA has also a solution for this - ScholarQA - im not sure if its only summarizer of a query-based retrieval - we'd need to check. Alternatively PaperQA2 (https://github.com/future-house/paper-qa) could be out retrieval agent (it assumes you already have folder with PDFs)
There was a problem hiding this comment.
Partial answers:
- duckduckgo search does not need separate credits. It is based on the ddgs library: https://pypi.org/project/ddgs/ and as far as I can tell they are not directly enforcing some rate limit
- Yes, native web search comes natively for different providers and PydanticAI provides a WebSearch capability which is meant for this goal: https://pydantic.dev/docs/ai/capabilities/web-search/ . I wanted to avoid such solution as this would be Provider dependent and we would not have any control on the implementation itself. Additionally, as you mentioned, not all the providers might support such "native" web search capability.
There was a problem hiding this comment.
https://github.com/allenai/asta-paper-finder/blob/main/README.md
I was checking the implementation and README for asta-paper-finder and it seems like it requires many API keys: The needed keys are: OPENAI_API_KEY, S2_API_KEY, COHERE_API_KEY, GOOGLE_API_KEY. I will continue and check if semantic scholar has a different implementation, for now the implementation that seems to not ask for all those API keys is the Asta Scientific Corpus Tool: https://allenai.org/asta/resources/mcp
There was a problem hiding this comment.
What I found (which I think is what we need) is the search_papers_by_relevance method in Asta Scientific Corpus. It implements the search of papers given a query.
|
|
||
|
|
||
| PAPER_FETCH_SYSTEM_PROMPT = """ | ||
| Your goal is to find scientific papers informing a machine learning modeling task. |
There was a problem hiding this comment.
Do we want the search agent to be ML-specific? IMO the query should be fine if its generic
There was a problem hiding this comment.
I do agree on this. I will change the prompt to make it more generic. If we are going for fetching with no LLM agent (which seems to be the direction) this prompt will be deleted completely, and only the tool docstring will guide the main agent on how to make the query itself
There was a problem hiding this comment.
In general, regarding the prompt itself I was thinking ideally we would retrieve papers that are implementing/summarising a ML modeling decision, but as you pointed out, the query does not have to be specific to the ML domain
| from agentomics.utils.config import Config | ||
|
|
||
|
|
||
| def create_save_paper_pdf_tool(config: Config) -> Tool[Any]: |
There was a problem hiding this comment.
This could be automated in the validator imo, given a paper doi, there should be tools out there that scout for an available pdf of it.
| def create_save_paper_pdf_tool(config: Config) -> Tool[Any]: | ||
| fetch = web_fetch_tool(timeout=config.web_fetch_timeout).function | ||
|
|
||
| async def save_paper_pdf(url: str, filename: str) -> str: |
There was a problem hiding this comment.
Are we fetching the supplementary materials as well? I'm not sure how that is handled in existing solutions, but afaik sometimes these can be a separate pdf.
782f33a to
9eb5c6a
Compare
|
Implemented as discussed the "programmatic" way to fetch papers. The search happens using the ASTA scientific corpus search tool: https://allenai.org/asta/resources/mcp. Then we retrieve from semantic scholar the candidates. For each one of the candidates we have a pool of dbs and resources we try to fetch papers from. I made a small diagram to describe how the implementation works now (the diagram code can be used to show to your agent a schematic representation of the code). |
This branch has not been deployed
2 participants
This PR will:
The whole design is summarised in the figure below.
Please consider this is a first version, where many design choices have been taken on my own. I am more than happy to revise the design of this feature together.