feat: spatiotemporal agent #453
Conversation
maciejmajek
force-pushed
the
feat/spatiotemporal-agent
branch
3 times, most recently
from
586ab23 to
91bf95b
Compare
There was a problem hiding this comment.
@maciejmajek This PR looks really good! I'll report on testing in a separate comment. For now I left some comments to the code.
One general idea: I can see quite a lot of type: ignore in the code. One thing to consider would be to suppress annoying/not useful warnings for the entire file (or repo): see this link.
Just an idea, if you prefer type: ignore it's also fine, but for me slightly confusing.
docs/agents/spatiotemporal.md
Outdated
|
|
||
| The agent works with several key data structures: | ||
|
|
||
| - `SpatioTemporalData`: The main data container that includes: |
There was a problem hiding this comment.
I think it's more common to one entry in a database a "record"
| - `SpatioTemporalData`: The main data container that includes: | |
| - `SpatioTemporalRecord`: The main data container that includes: |
There was a problem hiding this comment.
Please remember to update the graph at the top as well: https://github.com/RobotecAI/rai/blob/0aefa1fa1a63ab640b7448e69e7c98ddf2c1a8fd/docs/agents/spatiotemporal.md#spatiotemporalagent
docs/agents/spatiotemporal.md
Outdated
| agent = ROS2SpatioTemporalAgent(config) | ||
| ``` | ||
|
|
||
| A complete working example can be found in `examples/agents/spatiotemporal.py`. |
There was a problem hiding this comment.
It would be make it easier to try the agent if we'd recommend an example o3de binary here. How about this one?
| ] | ||
| } | ||
| }, | ||
| limit=n_results, |
There was a problem hiding this comment.
I think it will not return n_results closest, but n_results first found. Is it intended?
|
@maciejmajek I've tested the example agent with o3de rosbot_xl binary. One thing I had to change was the
However, I've only tested the "data collection" part of this PR. For the retrieval part I could not find the code to test. Do you have a snippet to test the retrieval part or how to add it to the example agent? |
maciejmajek
force-pushed
the
feat/spatiotemporal-agent
branch
2 times, most recently
from
bab0457 to
0aefa1f
Compare
chore: removed duplicated code docs: add spatiotemporal.md
feat: enhance logging
docs: update type: ignore docs: add diagram
chore: cleanup
Co-authored-by: Bartek Boczek <[email protected]>
Co-authored-by: Bartłomiej Boczek <[email protected]>
feat: reintroduce mongodb package
maciejmajek
force-pushed
the
feat/spatiotemporal-agent
branch
from
0aefa1f to
b51196c
Compare
Purpose
Even though latest LLMs have the context capacity reaching hundreds of thousands of tokens, the reasoning capabilities tend to decrease with the context size.
Proposed Changes
This PR introduces the SpatioTemporalAgent which runs on interval data (image, position, text history) gathering pipeline which can be later used by the agent to retrieve memories from a long time ago.
Issues
Testing