-
Notifications
You must be signed in to change notification settings - Fork 234
feat(eval): add eval configs example #19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces example configuration and documentation updates for the evaluation framework, modernizes dependency management, and consolidates memory backend handling by removing legacy code.
- Added new dependencies to
pyproject.toml
for evaluation configs, environment loading, and model support. - Refactored the
get_client
and search/response scripts to unify thememos
backend (removingmemos_mos
) and added atop_k
parameter. - Updated CLI choices across scripts and streamlined the README to reference a
.env-example
andconfigs-example
directory.
Reviewed Changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
File | Description |
---|---|
pyproject.toml | Added scipy , python-dotenv , pydantic , langgraph , and langmem to project dependencies |
evaluation/scripts/locomo/locomo_search.py | Unified memos backend code, removed legacy blocks, and added top_k parameter support |
evaluation/scripts/locomo/locomo_ingestion.py | Removed legacy memos and memos_mos ingestion logic; now uses consolidated backend |
evaluation/README.md | Simplified setup instructions, removed LongMemEval references, and pointed to run_locomo_eval.sh |
evaluation/.env-example | Introduced an example .env with placeholder API keys and model settings |
Comments suppressed due to low confidence (4)
evaluation/scripts/locomo/locomo_ingestion.py:126
- The
ingest_session
function no longer includes a branch to handle thememos
frame, so data will not be ingested for that backend. Restoring or adding appropriate ingestion logic formemos
is needed.
)
evaluation/scripts/locomo/locomo_search.py:445
- The help string for the
--lib
flag lists frameworks but omitslangmem
(and optionallyopenai
), leading to a mismatch with thechoices
list. Please update the help text to reflect the actual available options.
help="Specify the memory framework (zep or memos or mem0 or mem0_graph)",
evaluation/README.md:27
- [nitpick] The instructions now refer to
run_locomo_eval.sh
but don't explain where to find or how to configure it, nor the steps for ingestion and search. Consider elaborating the evaluation workflow or including the script contents for clarity.
### LoCoMo Evaluation
pyproject.toml:54
- [nitpick] Added dependency lines appear to lack the same indentation as existing entries in the
[tool.poetry.dependencies]
section, which may affect readability or formatting. Consider aligning them with the existing entries.
scipy = "^1.10.1"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This branch is out of date with the base branch. Please merge dev or rebase to stay in sync.
该分支与主分支不同步。请合并dev或rebase以保持同步。
pyproject.toml
Outdated
bert-score = "^0.3.13" | ||
scipy = "^1.10.1" | ||
python-dotenv = "^1.1.1" | ||
pydantic = "^2.11.7" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This dependency (pydantic) can be removed since it is implicitly required by other dependencies and does not need to be explicitly specified. After deleting this line, run poetry lock
to update the poetry.lock.
该依赖项(pydantic)可以被移除,因为它已被其他依赖项隐式引入,无需显式指定。删除此行后,请运行 poetry lock
以更新 poetry.lock。
@Ki-Seki I have solved the two problems. |
feat: update installation instructions and add platform compatibility notes
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Please delete options that are not relevant.
Checklist:
Maintainer Checklist