feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程，直接发 HTTP
移植自 mac-agent-os 的 mediacrawler_adapter：不起子进程、不开页面，用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin，所以 ingest 那条链路一个字不用改。
**目前能用的（真环境实测，非推断）**：
profile/other : 200, 7075 字节 —— 博主主页指标（粉丝/获赞/作品数/昵称）
aweme/detail : 200, 45425 字节 —— 单条作品详情（含点赞/评论/收藏/分享）
**目前不能用的：作品列表 `aweme/post`。** 两个互相独立的原因：
1. 这个接口被抖音单独升级成了真校验：不带 x-tt-argus 回 403「Uifid Not Found」，
带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过，「真校验」过不了。
同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的，
挑中的恰好是「批量拉作品列表」这个最敏感的动作。
2. 改走页面截获也不行：CDP 浏览器打开博主主页会落到「验证码中间页」（当天大量探测的
代价，过几小时要重测）。
所以现在的边界是：**已知作品的指标刷新能做，自动发现新作品做不了**。
**排查中控住变量后得到的两条事实**（都写进注释了）：
· `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
UA+client hints+Cookie 时是 200 但仅 121 字节的空壳，补上这三个头变 7074 字节。
（我先前猜的 sec-ch-ua 不是关键。）
· 因此 UA 与 client hints 必须**成套地取自同一个浏览器**，所以 BrowserIdentity 一次
从 CDP 取齐 cookie + UA + hints，而不是各自写死。
「200 + 空 body 必须当场报错」也是刻意写死的：放过去它会在下游变成「这个博主没作品」，
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的，还被翻译成「账号被封」。
测试 +11：cookie 解析、请求头成套性（含 uifid 缺失/回退）、产物键名与 store 对齐、
以及 _get 的三条失败路径（空 body / 403 带网关原话 / 正常返回）。
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
# _archived/ — 已归档技能
|
||||
|
||||
以下技能已归档，当前无实现代码，仅供设计参考。
|
||||
|
||||
## 归档清单
|
||||
|
||||
| 原技能 | 归档日期 | 说明 |
|
||||
|:-------|:---------|:-----|
|
||||
| content_processor | 2026-06-21 | 统一内容采集入口设计，待实现 |
|
||||
| web_crawler | 2026-06-21 | 三引擎网页抓取设计，待实现 |
|
||||
| auto_collector | 2026-06-21 | 24h 自动采集设计，待实现 |
|
||||
| cloakbrowser_controller | 2026-06-21 | CloakBrowser 操作设计，待实现 |
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
name: auto_collector
|
||||
version: 1.0.0
|
||||
status: archived
|
||||
archived_date: 2026-06-21
|
||||
description: 24小时自动信息收集——已归档，无实现代码。设计保留供参考。
|
||||
triggers: []
|
||||
---
|
||||
|
||||
# Auto Collector Skill（已归档）
|
||||
|
||||
> ⚠️ 此技能已归档。当前无 Python 实现代码，不应在 WorkBuddy 中注册。
|
||||
|
||||
该技能原设计为 24h 多平台监控、定时采集、自动分类入库。
|
||||
具体实现待后续规划。
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
name: cloakbrowser_controller
|
||||
version: 1.0.0
|
||||
status: archived
|
||||
archived_date: 2026-06-21
|
||||
description: CloakBrowser 源码级反爬浏览器——已归档，无实现代码。设计保留供参考。
|
||||
triggers: []
|
||||
---
|
||||
|
||||
# CloakBrowser Controller Skill（已归档）
|
||||
|
||||
> ⚠️ 此技能已归档。当前无实现代码，不应在 WorkBuddy 中注册。
|
||||
|
||||
该技能原设计为源码级修改的 Chromium 分发版，49项 C++ 补丁修改指纹。
|
||||
具体实现待后续规划。
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
name: content_processor
|
||||
version: 2.0.0
|
||||
status: archived
|
||||
archived_date: 2026-06-21
|
||||
description: 统一内容采集入口——已归档，无实现代码。设计保留供参考。
|
||||
triggers: []
|
||||
---
|
||||
|
||||
# Content Processor Skill（已归档）
|
||||
|
||||
> ⚠️ 此技能已归档。当前无 Python 实现代码，不应在 WorkBuddy 中注册。
|
||||
|
||||
该技能原设计为统一的内容采集入口，按内容类型路由，区分知识类与素材类。
|
||||
具体实现待后续规划。
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
name: web_crawler
|
||||
version: 1.2.0
|
||||
status: archived
|
||||
archived_date: 2026-06-21
|
||||
description: 网页抓取+反爬技能——已归档，无实现代码。设计保留供参考。
|
||||
triggers: []
|
||||
---
|
||||
|
||||
# Web Crawler Skill（已归档）
|
||||
|
||||
> ⚠️ 此技能已归档。当前无 Python 实现代码，不应在 WorkBuddy 中注册。
|
||||
|
||||
该技能原设计使用 Scrapling + Crawl4AI + CloakBrowser 三引擎自适应抓取。
|
||||
具体实现待后续规划。
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: __SKILL_NAME__
|
||||
version: 1.0.0
|
||||
description: __SKILL_DESCRIPTION__
|
||||
triggers:
|
||||
- __TRIGGER_1__
|
||||
---
|
||||
|
||||
# __SKILL_NAME__ 技能
|
||||
|
||||
## 概述
|
||||
|
||||
__SKILL_DESCRIPTION__
|
||||
|
||||
## 核心能力
|
||||
|
||||
<!-- 在此描述技能的核心功能 -->
|
||||
|
||||
## 使用说明
|
||||
|
||||
<!-- 在此描述使用方法和示例命令 -->
|
||||
|
||||
## 文件结构
|
||||
|
||||
```
|
||||
__SKILL_NAME__/
|
||||
├── SKILL.md
|
||||
├── version.json
|
||||
├── skill.py
|
||||
├── install.sh
|
||||
└── README.md
|
||||
```
|
||||
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
__SKILL_NAME__ 技能主体脚本
|
||||
用途：__SKILL_DESCRIPTION__
|
||||
使用：python3 skill.py --root ~/workbuddy-agent-os/agent-sync
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="__SKILL_NAME__")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
print(f"[INFO] __SKILL_NAME__ 正在运行...")
|
||||
# 在此实现技能逻辑
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"name": "__SKILL_NAME__",
|
||||
"version": "1.0.0",
|
||||
"description": "__SKILL_DESCRIPTION__",
|
||||
"dependencies": [],
|
||||
"mcp_tools": []
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: auto_collector
|
||||
version: 1.0.0
|
||||
status: archived
|
||||
archived_date: 2026-06-21
|
||||
description: [已归档] 24小时自动信息收集设计，无实现代码
|
||||
triggers: []
|
||||
---
|
||||
|
||||
# Auto Collector Skill（24h 自动信息收集）
|
||||
|
||||
## 概述
|
||||
|
||||
24小时自动监控多平台信息源，发现新内容后自动抓取、分类、入库、通知。
|
||||
|
||||
## 触发词
|
||||
|
||||
| 触发词 | 动作 |
|
||||
|--------|------|
|
||||
| 开始收集 | 启动定时监控任务 |
|
||||
| 停止收集 | 暂停所有定时任务 |
|
||||
| 收集状态 | 返回当前监控列表、最近采集时间、今日采集数量 |
|
||||
| 收集报告 | 生成今日采集摘要 |
|
||||
| 添加监控 [URL/博主名] | 新增监控源 |
|
||||
| 删除监控 [名称] | 移除监控源 |
|
||||
|
||||
## 监控频率与工具
|
||||
|
||||
| 信息源 | 监控频率 | 采集工具 | 备注 |
|
||||
|--------|----------|----------|------|
|
||||
| B站/YouTube 订阅更新 | 每 30 分钟 | BiliNote API | 需 BiliNote Docker |
|
||||
| 小红书指定博主 | 每 2 小时 | TikOmni Skills | ✅ 已安装 |
|
||||
| 抖音指定账号 | 每 2 小时 | TikOmni Skills | ✅ 已安装 |
|
||||
| RSS 订阅源 | 每 15 分钟 | feedparser | ✅ 已安装 |
|
||||
| 指定网页监控 | 每 1 小时 | Crawl4AI | ✅ 已安装 |
|
||||
|
||||
## 采集流程
|
||||
|
||||
```
|
||||
定时触发
|
||||
→ 检查信息源是否有新内容
|
||||
├─ 无新内容 → 记录日志，等待下一轮
|
||||
└─ 有新内容 → 自动抓取
|
||||
├─ 视频 → bilinote 技能生成笔记
|
||||
├─ 图文 → web-clipper 技能提取正文
|
||||
├─ 社交帖子 → social-collector 技能结构化提取
|
||||
└─ 音频/RSS → 提取全文
|
||||
→ kb_manager 自动入库（写入 03_knowledge/00_inbox/）
|
||||
→ 标记标签：#自动采集 #平台名 #日期
|
||||
→ 凌晨 2:00 汇总日报 → 写入 Obsidian 01_daily/
|
||||
```
|
||||
|
||||
## 定时任务配置
|
||||
|
||||
### WorkBuddy 自动化配置
|
||||
|
||||
| 任务 | 频率 | 说明 |
|
||||
|------|------|------|
|
||||
| RSS 检查 | FREQ=HOURLY;INTERVAL=1 | 检查所有 RSS 源 |
|
||||
| 社交平台检查 | FREQ=HOURLY;INTERVAL=2 | 小红书/抖音博主 |
|
||||
| 网页变化检查 | FREQ=HOURLY;INTERVAL=1 | 指定网页监控 |
|
||||
| 采集日报 | FREQ=DAILY;BYHOUR=23 | 每日 23:00 汇总 |
|
||||
|
||||
### macOS cron 授权
|
||||
|
||||
```bash
|
||||
# 系统设置 → 隐私与安全性 → 完全磁盘访问权限 → 添加 /usr/sbin/cron
|
||||
```
|
||||
|
||||
## 配置文件
|
||||
|
||||
监控源列表存储在：`~/workbuddy-agent-os/agent-sync/02_skills/auto_collector/sources.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"rss": [
|
||||
{"name": "示例RSS", "url": "https://example.com/feed.xml", "last_checked": null}
|
||||
],
|
||||
"xiaohongshu": [
|
||||
{"name": "博主A", "user_id": "xxx", "last_checked": null}
|
||||
],
|
||||
"douyin": [
|
||||
{"name": "账号B", "user_id": "xxx", "last_checked": null}
|
||||
],
|
||||
"web_pages": [
|
||||
{"name": "目标网页", "url": "https://example.com", "last_hash": null}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## 依赖
|
||||
|
||||
| 包 | 版本 | 用途 | 安装状态 |
|
||||
|---|---|---|---|
|
||||
| feedparser | latest | RSS 解析 | ✅ 已安装 |
|
||||
| schedule | latest | 定时任务 | ✅ 已安装 |
|
||||
| @tikomni/skills | latest | 小红书/抖音采集 | ✅ 已安装 |
|
||||
| crawl4ai | latest | 网页抓取 | ✅ 已安装 |
|
||||
| chromadb | latest | 向量存储（可选） | ✅ 已安装 |
|
||||
|
||||
## 注意事项
|
||||
|
||||
- BiliNote 需要 Docker 运行，首次使用前需手动部署
|
||||
- 社交平台采集可能需要登录态，TikOmni 已处理
|
||||
- 网页变化检测基于内容哈希对比
|
||||
- 所有采集内容先入 `00_inbox/`，由 `collect_to_inbox` 每日 2:30 提取各目录内容转入收件箱，再由 `inbox_refine` 技能每日凌晨 3:00 自动提纯归档
|
||||
- 采集日报写入 `01_daily/`，由 `memory_manager` 每日凌晨 2:00 汇总
|
||||
@@ -0,0 +1,45 @@
|
||||
skill:
|
||||
name: "自动采集器"
|
||||
id: "auto_collector"
|
||||
version: "1.0.0"
|
||||
category: "core-system"
|
||||
status: "experimental"
|
||||
created: "2026-04-29"
|
||||
updated: "2026-04-29"
|
||||
|
||||
description:
|
||||
short: "定时自动抓取RSS/新闻/社交媒体，推送inbox/knowledge/feed/"
|
||||
use_cases:
|
||||
- "定时拉取配置的RSS源，转MD推送feed分区"
|
||||
- "抓取指定网页列表，转MD推送clipping分区"
|
||||
- "仅格式转换，不做AI提炼"
|
||||
tags: ["自动采集", "RSS", "定时", "feed"]
|
||||
|
||||
ownership:
|
||||
master_editor: "my-macbook_macOS14.4_16GB_master"
|
||||
readonly_on_other_devices: false
|
||||
|
||||
environment:
|
||||
os: "macos"
|
||||
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
|
||||
pip_dependencies:
|
||||
- "feedparser>=6.0.0"
|
||||
- "trafilatura>=1.6.0"
|
||||
|
||||
setup:
|
||||
install_script: ""
|
||||
post_check: "python3 -c 'import feedparser; print(\"OK\")'"
|
||||
|
||||
usage:
|
||||
entry_point: "auto_collector.py"
|
||||
scheduled: "每日 01:00（可配置）"
|
||||
manual_trigger: "自动采集"
|
||||
|
||||
target_inbox:
|
||||
feed: "03_knowledge/00_stream/inbox/knowledge/feed/"
|
||||
clipping: "03_knowledge/00_stream/inbox/knowledge/clipping/"
|
||||
|
||||
changelog:
|
||||
- version: "1.0.0"
|
||||
date: "2026-04-29"
|
||||
changes: "初始版本（experimental，待实现）"
|
||||
@@ -0,0 +1 @@
|
||||
{"name": "auto_collector", "version": "1.0.0", "created": "2026-04-25", "updated": "2026-04-25", "status": "active"}
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
name: cloakbrowser_controller
|
||||
version: 1.0.0
|
||||
status: archived
|
||||
archived_date: 2026-06-21
|
||||
description: [已归档] CloakBrowser 反爬浏览器设计，无实现代码
|
||||
triggers: []
|
||||
---
|
||||
|
||||
# CloakBrowser Controller Skill
|
||||
|
||||
## 概述
|
||||
|
||||
源码级修改的 Chromium 分发版，49项 C++ 补丁修改指纹，30/30反爬检测全部通过。
|
||||
**只替换浏览器操作层，不涉及 Matrix 养号的 Camoufox（已深度适配）。**
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
# agent-os venv（本机）
|
||||
/Users/chengzige/.workbuddy/binaries/python/envs/agent-os/bin/python3 -m pip install cloakbrowser
|
||||
|
||||
# 首次运行自动下载 ~350MB Chromium 二进制到 ~/.cloakbrowser/
|
||||
# 二进制位置: ~/.cloakbrowser/chromium-145.0.7632.109.2/Chromium.app
|
||||
```
|
||||
|
||||
## 验证
|
||||
|
||||
```python
|
||||
from cloakbrowser import launch
|
||||
b = launch(humanize=True)
|
||||
page = b.new_page()
|
||||
page.goto("https://www.baidu.com")
|
||||
print(page.title()) # → "百度一下，你就知道"
|
||||
b.close()
|
||||
```
|
||||
|
||||
### 代理环境
|
||||
|
||||
如使用代理，需在 Python 代码中设置环境变量：
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ['HTTPS_PROXY'] = 'socks5://127.0.0.1:6478' # SOCKS5
|
||||
os.environ['HTTP_PROXY'] = 'socks5://127.0.0.1:6478' # 或 http 代理
|
||||
```
|
||||
|
||||
### 预下载二进制
|
||||
|
||||
```python
|
||||
from cloakbrowser.download import ensure_binary, binary_info
|
||||
info = binary_info()
|
||||
print(info["installed"]) # 是否已下载
|
||||
ensure_binary() # 预下载 Chromium 二进制
|
||||
```
|
||||
|
||||
## API 用法（与 Playwright 完全兼容）
|
||||
|
||||
```python
|
||||
# 旧：Playwright（易被检测）
|
||||
# from playwright.sync_api import sync_playwright
|
||||
|
||||
# 新：CloakBrowser（30/30通过）
|
||||
from cloakbrowser import launch
|
||||
|
||||
browser = launch(humanize=True) # humanize 启用人类行为模拟
|
||||
page = browser.new_page()
|
||||
page.goto("https://protected-site.com")
|
||||
# ... 标准的 Playwright API
|
||||
browser.close()
|
||||
```
|
||||
|
||||
## 在系统中的应用
|
||||
|
||||
### 1. web_crawler 高级引擎
|
||||
|
||||
替换 Playwright stealth 引擎：
|
||||
|
||||
```python
|
||||
# web_crawler 引擎选择
|
||||
反爬级别 → 引擎
|
||||
低（静态）→ Scrapling（最快）
|
||||
高（Cloudflare等）→ CloakBrowser（新增，默认）
|
||||
```
|
||||
|
||||
### 2. content-inspiration 采集引擎
|
||||
|
||||
替换 agent-browser 进行抖音/网页内容采集：
|
||||
|
||||
```python
|
||||
# collect.py 采集引擎
|
||||
platform == "douyin" 且需要浏览器 → CloakBrowser（优先）
|
||||
platform == "douyin" 且 CloakBrowser 失败 → 回退 OpenCLI
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- **Matrix 养号的 Camoufox 不动** — 账号适配已完成，换内核需重写
|
||||
- CloakBrowser 首次启动会下载 ~200MB 二进制，需等待
|
||||
- 建议与 Scrapling 配合使用：静态页走 Scrapling，高反爬走 CloakBrowser
|
||||
- 与 Peekaboo 无冲突：CloakBrowser 做浏览器操作，Peekaboo 做桌面GUI/OCR
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
name: collect_to_inbox
|
||||
version: 1.0.0
|
||||
status: legacy
|
||||
description: [降级] 扫描旧目录的文件转入 00_inbox/。注：v2.0 设计（以 01_submissions/ 为主入口）尚未实现。当前实际代码扫描的是 50_resources/、20_methods/、01_daily/、40_references/ 等旧目录。
|
||||
triggers:
|
||||
- 归集
|
||||
- 归集收件箱
|
||||
- 收集到收件箱
|
||||
- 汇聚收件箱
|
||||
- collect inbox
|
||||
---
|
||||
|
||||
# Collect to Inbox —— 提交箱 → 收件箱
|
||||
|
||||
## 概述
|
||||
|
||||
将 `01_submissions/`（提交箱）中的采集内容转入 `00_inbox/`（待提纯收件箱）。
|
||||
这是 **收件箱提纯的前置步骤**，v2.0 以 `01_submissions/` 为主入口。
|
||||
|
||||
## 设计背景（v2.0）
|
||||
|
||||
根据 [内容收集全链路规范 v2.0](../99_system/pipelines/content-collection-pipeline.md)：
|
||||
|
||||
```
|
||||
所有机器统一流程
|
||||
↓
|
||||
收集内容 → 01_submissions/（提交箱）
|
||||
↓
|
||||
collect_to_inbox（归集）
|
||||
↓
|
||||
00_inbox/（待提纯）
|
||||
↓
|
||||
inbox_refine（提纯）→ 分类入库
|
||||
```
|
||||
|
||||
## 扫描目录配置（按优先级）
|
||||
|
||||
| 优先级 | 源目录 | 说明 | 提取策略 |
|
||||
|--------|--------|------|----------|
|
||||
| **1** | `01_submissions/` | **v2.0 主入口**，所有新建采集内容 | 读取 frontmatter 直接转存，无需重新提取 |
|
||||
| 2 | `50_resources/` | 旧版兼容，历史遗留内容 | 提取核心摘要 |
|
||||
| 3 | `20_methods/` | 旧版兼容 | 提取方法名 + 步骤 |
|
||||
| 4 | `40_references/` | 旧版兼容 | 提取标题 + 核心观点 |
|
||||
| 5 | `01_daily/闪念笔记/` | 旧版兼容 | 提取要点 + 标签 |
|
||||
|
||||
## 执行步骤（v2.0）
|
||||
|
||||
```
|
||||
1. 扫描 01_submissions/ 下所有 status:submitted 的 .md 文件
|
||||
↓
|
||||
2. 读取 frontmatter：
|
||||
├── collect_type + collect_subtype 决定提取策略
|
||||
├── purpose 判断知识类/素材类
|
||||
└── 素材类 → 跳过（已直存 materials/）
|
||||
↓
|
||||
3. 检查 00_inbox/ 中是否已有同名条目（按 title 去重）
|
||||
↓
|
||||
4. 无重复 → 复制到 00_inbox/{filename}，追加字段：
|
||||
├── staged_date: 归集日期
|
||||
└── status: inbox
|
||||
↓
|
||||
5. 更新源文件 status → staged
|
||||
↓
|
||||
6. 输出统计报告
|
||||
```
|
||||
|
||||
## 兼容旧目录
|
||||
|
||||
保留对 `50_resources/`、`20_methods/` 等旧目录的扫描能力，
|
||||
但优先级低于 `01_submissions/`。旧目录内容同等转入 `00_inbox/`。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
1. 扫描上述所有源目录，查找 `.md` 文件
|
||||
2. 对每个文件：
|
||||
a. 读取 Frontmatter 提取元数据（title, date, tags, source 等）
|
||||
b. 读取正文，根据提取策略生成摘要
|
||||
c. 检查 `00_inbox/` 中是否已有同名文件（去重）
|
||||
d. 如无重复，生成标准化 MD 文件写入 `00_inbox/`
|
||||
e. 在原文件 Frontmatter 中标记 `collected: true` 和 `collected_date`
|
||||
3. 输出统计报告（扫描数、提取数、跳过数、重复数）
|
||||
|
||||
## 提取后生成的 MD 格式
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: {原标题}
|
||||
source_dir: {原目录相对路径}
|
||||
source_file: {原文件名}
|
||||
date: {原始日期}
|
||||
collected_date: {收集日期}
|
||||
tags: {原始标签}
|
||||
nature: {推断的属性：concept/method/fact/reference/resource/opinion}
|
||||
domain: {推断的领域}
|
||||
status: inbox
|
||||
---
|
||||
|
||||
# {原标题}
|
||||
|
||||
> 来源：{原目录路径}
|
||||
|
||||
{提取的主要内容摘要}
|
||||
```
|
||||
|
||||
## 去重规则
|
||||
|
||||
- 同名文件跳过（`00_inbox/` 中已存在同文件名）
|
||||
- 同标题文件跳过（Frontmatter title 匹配）
|
||||
- 已标记 `collected: true` 的源文件跳过
|
||||
|
||||
## 前置依赖
|
||||
|
||||
| 依赖 | 说明 |
|
||||
|------|------|
|
||||
| Python 3 | 运行脚本 |
|
||||
| 03_knowledge/ 目录 | 知识库根目录 |
|
||||
|
||||
## 自动化
|
||||
|
||||
- 每日凌晨 2:30 自动执行（WorkBuddy automation）
|
||||
- 在 `inbox_refine`（3:00）之前运行，确保收件箱有内容可提纯
|
||||
- 也可手动触发：说"归集"或"收集到收件箱"
|
||||
|
||||
## 与其他技能的关系
|
||||
|
||||
```
|
||||
收集技能 → 各分类目录（视频笔记/阅读笔记/灵感素材/...）
|
||||
↓
|
||||
collect_to_inbox（每日 2:30）
|
||||
↓
|
||||
00_inbox/（收件箱，标准化 MD）
|
||||
↓
|
||||
inbox_refine（每日 3:00）
|
||||
↓
|
||||
按属性+领域分类归档 → 最终目录 + 更新首页
|
||||
```
|
||||
@@ -0,0 +1,54 @@
|
||||
skill:
|
||||
name: "内容采集推送器"
|
||||
id: "collect_to_inbox"
|
||||
version: "1.0.0"
|
||||
category: "core-system"
|
||||
status: "legacy"
|
||||
created: "2026-04-25"
|
||||
updated: "2026-04-29"
|
||||
|
||||
description:
|
||||
short: "扫描本机素材，格式转换（HTML/视频→MD），推送协同inbox"
|
||||
use_cases:
|
||||
- "扫描 agent-os-local/materials/refined_for_inbox/ 目录"
|
||||
- "HTML→MD 格式转换（不做AI提炼，原文推送）"
|
||||
- "按内容来源推送到对应inbox分区（clipping/feed/personal）"
|
||||
- "推送完成后标记已处理，避免重复"
|
||||
tags: ["采集", "推送", "格式转换", "inbox"]
|
||||
|
||||
ownership:
|
||||
master_editor: "my-macbook_macOS14.4_16GB_master"
|
||||
readonly_on_other_devices: false
|
||||
|
||||
environment:
|
||||
os: "macos"
|
||||
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
|
||||
pip_dependencies:
|
||||
- "trafilatura>=1.6.0"
|
||||
- "markdownify>=0.11.0"
|
||||
|
||||
setup:
|
||||
install_script: ""
|
||||
post_check: "python3 -c 'import trafilatura; print(\"OK\")'"
|
||||
|
||||
usage:
|
||||
entry_point: "collect_to_inbox.py"
|
||||
scheduled: "手动触发或每日定时"
|
||||
manual_trigger: "采集推送"
|
||||
|
||||
source_dirs:
|
||||
refined_for_inbox: "~/workbuddy-agent-os/agent-sync/agent-local/materials/refined_for_inbox/"
|
||||
|
||||
target_inbox:
|
||||
clipping: "03_knowledge/00_stream/inbox/knowledge/clipping/"
|
||||
feed: "03_knowledge/00_stream/inbox/knowledge/feed/"
|
||||
personal: "03_knowledge/00_stream/inbox/personal/"
|
||||
memory: "03_knowledge/00_stream/inbox/memory/"
|
||||
|
||||
changelog:
|
||||
- version: "1.1.0"
|
||||
date: "2026-04-29"
|
||||
changes: "重写：扫描agent-os-local/materials/refined_for_inbox/，纯格式转换推送"
|
||||
- version: "1.0.0"
|
||||
date: "2026-04-25"
|
||||
changes: "初始版本"
|
||||
@@ -0,0 +1,312 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
collect_to_inbox.py —— 扫描知识库各分类目录，提取主要内容生成标准化 MD 放入收件箱
|
||||
|
||||
用法：
|
||||
python3 collect_to_inbox.py --root ~/workbuddy-agent-os/agent-sync [--dry-run] [--force]
|
||||
|
||||
参数：
|
||||
--root agent-os 根目录（默认 ~/workbuddy-agent-os/agent-sync）
|
||||
--dry-run 只预览不写入
|
||||
--force 忽略 collected 标记，重新收集
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# ─── 角色检查：所有机器均可执行汇聚 ───
|
||||
try:
|
||||
sys.path.insert(0, str(Path.home() / "workbuddy-agent-os" / "agent-sync" / "05_tools" / "01_system"))
|
||||
from role_check import check_capability
|
||||
if not check_capability("knowledge_collection"):
|
||||
print("⛔ 能力 knowledge_collection 未启用，跳过 collect_to_inbox")
|
||||
sys.exit(0)
|
||||
except ImportError as e:
|
||||
print(f"⚠️ 角色检查模块不可用 ({e})，继续执行")
|
||||
|
||||
# 扫描的源目录配置：(相对路径, 提取策略)
|
||||
SCAN_DIRS = [
|
||||
("03_knowledge/50_resources/视频笔记", "extract_summary"),
|
||||
("03_knowledge/50_resources/字幕存档", "extract_head"),
|
||||
("03_knowledge/50_resources/阅读笔记", "extract_summary"),
|
||||
("03_knowledge/50_resources/全文存档", "extract_head"),
|
||||
("03_knowledge/50_resources/翻译存档", "extract_summary"),
|
||||
("03_knowledge/50_resources/灵感素材", "extract_summary"),
|
||||
("03_knowledge/50_resources/语音转写", "extract_key_points"),
|
||||
("03_knowledge/20_methods", "extract_method"),
|
||||
("03_knowledge/01_daily/闪念笔记", "extract_key_points"),
|
||||
("03_knowledge/40_references", "extract_summary"),
|
||||
]
|
||||
|
||||
# 不扫描的目录
|
||||
SKIP_DIRS = {"00_inbox", "99_system", "90_archive", "10_concepts", "30_facts", "60_opinions", "50_resources/资源索引"}
|
||||
|
||||
|
||||
def parse_frontmatter(content: str) -> tuple[dict, str]:
|
||||
"""解析 Frontmatter，返回 (metadata_dict, body_text)"""
|
||||
if not content.startswith("---"):
|
||||
return {}, content
|
||||
|
||||
end = content.find("---", 3)
|
||||
if end == -1:
|
||||
return {}, content
|
||||
|
||||
fm_text = content[3:end].strip()
|
||||
body = content[end + 3:].strip()
|
||||
|
||||
metadata = {}
|
||||
for line in fm_text.split("\n"):
|
||||
line = line.strip()
|
||||
if ":" in line:
|
||||
key, _, val = line.partition(":")
|
||||
key = key.strip()
|
||||
val = val.strip().strip('"').strip("'")
|
||||
if val:
|
||||
metadata[key] = val
|
||||
|
||||
return metadata, body
|
||||
|
||||
|
||||
def extract_summary(body: str, max_chars: int = 500) -> str:
|
||||
"""提取摘要：取前 max_chars 字符"""
|
||||
if len(body) <= max_chars:
|
||||
return body
|
||||
return body[:max_chars] + "\n\n...（内容已截断，完整内容见源文件）"
|
||||
|
||||
|
||||
def extract_head(body: str, max_chars: int = 500) -> str:
|
||||
"""提取头部内容"""
|
||||
return extract_summary(body, max_chars)
|
||||
|
||||
|
||||
def extract_key_points(body: str, max_chars: int = 500) -> str:
|
||||
"""提取关键要点：优先提取列表项"""
|
||||
lines = body.split("\n")
|
||||
points = []
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if stripped.startswith(("- ", "* ", "1.", "2.", "3.", "4.", "5.")):
|
||||
points.append(stripped)
|
||||
if len("\n".join(points)) > max_chars:
|
||||
break
|
||||
|
||||
if points:
|
||||
result = "\n".join(points[:20])
|
||||
if len(result) > max_chars:
|
||||
result = result[:max_chars] + "\n\n...（内容已截断）"
|
||||
return result
|
||||
|
||||
return extract_summary(body, max_chars)
|
||||
|
||||
|
||||
def extract_method(body: str, max_chars: int = 500) -> str:
|
||||
"""提取方法论：方法名 + 步骤"""
|
||||
lines = body.split("\n")
|
||||
# 提取标题和编号列表
|
||||
method_lines = []
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("#") or stripped.startswith(("-", "*", "1.", "2.", "3.", "4.", "5.")):
|
||||
method_lines.append(stripped)
|
||||
if len("\n".join(method_lines)) > max_chars:
|
||||
break
|
||||
|
||||
if method_lines:
|
||||
return "\n".join(method_lines[:30])
|
||||
|
||||
return extract_summary(body, max_chars)
|
||||
|
||||
|
||||
EXTRACT_FUNCS = {
|
||||
"extract_summary": extract_summary,
|
||||
"extract_head": extract_head,
|
||||
"extract_key_points": extract_key_points,
|
||||
"extract_method": extract_method,
|
||||
}
|
||||
|
||||
|
||||
def infer_nature(source_dir: str, metadata: dict) -> str:
|
||||
"""根据源目录和元数据推断 nature"""
|
||||
dir_lower = source_dir.lower()
|
||||
if "方法" in dir_lower or "method" in dir_lower:
|
||||
return "method"
|
||||
if "视频笔记" in dir_lower:
|
||||
return "resource"
|
||||
if "字幕" in dir_lower or "转写" in dir_lower:
|
||||
return "reference"
|
||||
if "阅读" in dir_lower or "全文" in dir_lower:
|
||||
return "reference"
|
||||
if "翻译" in dir_lower:
|
||||
return "reference"
|
||||
if "灵感" in dir_lower:
|
||||
return "opinion"
|
||||
if "闪念" in dir_lower:
|
||||
return "opinion"
|
||||
if "references" in dir_lower:
|
||||
return "reference"
|
||||
# 从 tags 推断
|
||||
tags = metadata.get("tags", "")
|
||||
if "方法论" in tags or "方法" in tags:
|
||||
return "method"
|
||||
if "概念" in tags:
|
||||
return "concept"
|
||||
return "resource"
|
||||
|
||||
|
||||
def infer_domain(metadata: dict) -> str:
|
||||
"""从标签推断领域"""
|
||||
tags = metadata.get("tags", "").lower()
|
||||
domain_keywords = {
|
||||
"技术": ["编程", "代码", "开发", "tech", "python", "javascript", "ai", "llm", "机器学习"],
|
||||
"金融": ["投资", "股票", "基金", "finance", "交易", "财经"],
|
||||
"产品": ["产品", "设计", "ux", "product"],
|
||||
"效率": ["效率", "工具", "workflow", "自动化"],
|
||||
"阅读": ["读书", "阅读", "书评", "book"],
|
||||
"生活": ["生活", "旅行", "美食", "健身"],
|
||||
}
|
||||
for domain, keywords in domain_keywords.items():
|
||||
if any(kw in tags for kw in keywords):
|
||||
return domain
|
||||
return "general"
|
||||
|
||||
|
||||
def collect_to_inbox(root_dir: str, dry_run: bool = False, force: bool = False):
|
||||
"""主函数：扫描各目录，提取内容到收件箱"""
|
||||
root = Path(root_dir).expanduser()
|
||||
inbox_dir = root / "03_knowledge" / "00_inbox"
|
||||
|
||||
stats = {"scanned": 0, "collected": 0, "skipped_collected": 0, "skipped_duplicate": 0, "errors": 0}
|
||||
|
||||
# 确保收件箱存在
|
||||
if not dry_run:
|
||||
inbox_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 获取收件箱已有文件名集合（用于去重）
|
||||
existing_inbox = set()
|
||||
if inbox_dir.exists():
|
||||
for f in inbox_dir.glob("*.md"):
|
||||
existing_inbox.add(f.name)
|
||||
|
||||
collected_date = datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
for rel_dir, extract_strategy in SCAN_DIRS:
|
||||
scan_dir = root / rel_dir
|
||||
if not scan_dir.exists():
|
||||
continue
|
||||
|
||||
# 递归扫描子目录（灵感素材下有平台子目录）
|
||||
for md_file in sorted(scan_dir.rglob("*.md")):
|
||||
stats["scanned"] += 1
|
||||
|
||||
try:
|
||||
content = md_file.read_text(encoding="utf-8")
|
||||
except Exception as e:
|
||||
print(f" ❌ 读取失败: {md_file.name} - {e}")
|
||||
stats["errors"] += 1
|
||||
continue
|
||||
|
||||
metadata, body = parse_frontmatter(content)
|
||||
|
||||
# 检查是否已收集
|
||||
if not force and metadata.get("collected") == "true":
|
||||
stats["skipped_collected"] += 1
|
||||
continue
|
||||
|
||||
# 去重检查
|
||||
dest_name = md_file.name
|
||||
if dest_name in existing_inbox:
|
||||
stats["skipped_duplicate"] += 1
|
||||
continue
|
||||
|
||||
# 提取内容
|
||||
extract_func = EXTRACT_FUNCS.get(extract_strategy, extract_summary)
|
||||
extracted = extract_func(body)
|
||||
|
||||
# 推断属性
|
||||
nature = infer_nature(rel_dir, metadata)
|
||||
domain = infer_domain(metadata)
|
||||
title = metadata.get("title", md_file.stem)
|
||||
source_date = metadata.get("date", collected_date)
|
||||
tags = metadata.get("tags", "")
|
||||
source_rel = str(md_file.relative_to(root))
|
||||
|
||||
# 生成收件箱文件内容
|
||||
inbox_content = f"""---
|
||||
title: "{title}"
|
||||
source_dir: {rel_dir}
|
||||
source_file: {md_file.name}
|
||||
date: {source_date}
|
||||
collected_date: {collected_date}
|
||||
tags: {tags}
|
||||
nature: {nature}
|
||||
domain: {domain}
|
||||
status: inbox
|
||||
---
|
||||
|
||||
# {title}
|
||||
|
||||
> 来源：{rel_dir}
|
||||
|
||||
{extracted}
|
||||
"""
|
||||
|
||||
if dry_run:
|
||||
print(f" 📥 [预览] {md_file.name} → 00_inbox/{dest_name}")
|
||||
print(f" nature={nature}, domain={domain}, 策略={extract_strategy}")
|
||||
else:
|
||||
# 写入收件箱
|
||||
dest_path = inbox_dir / dest_name
|
||||
dest_path.write_text(inbox_content, encoding="utf-8")
|
||||
print(f" 📥 {md_file.name} → 00_inbox/{dest_name}")
|
||||
|
||||
# 标记原文件为已收集
|
||||
if content.startswith("---"):
|
||||
# 在 Frontmatter 末尾添加 collected 标记
|
||||
end = content.find("---", 3)
|
||||
if end != -1:
|
||||
fm = content[3:end]
|
||||
if "collected:" not in fm:
|
||||
fm += f"\ncollected: true\ncollected_date: {collected_date}"
|
||||
new_content = "---" + fm + content[end:]
|
||||
md_file.write_text(new_content, encoding="utf-8")
|
||||
else:
|
||||
# 没有 Frontmatter，添加一个简单的
|
||||
new_content = f"---\ncollected: true\ncollected_date: {collected_date}\n---\n\n{content}"
|
||||
md_file.write_text(new_content, encoding="utf-8")
|
||||
|
||||
existing_inbox.add(dest_name)
|
||||
|
||||
stats["collected"] += 1
|
||||
|
||||
# 输出统计
|
||||
print(f"\n{'[预览] ' if dry_run else ''}收集统计：")
|
||||
print(f" 扫描文件：{stats['scanned']}")
|
||||
print(f" 提取到收件箱：{stats['collected']}")
|
||||
print(f" 跳过（已收集）：{stats['skipped_collected']}")
|
||||
print(f" 跳过（重复）：{stats['skipped_duplicate']}")
|
||||
print(f" 错误：{stats['errors']}")
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="收集各分类目录内容到收件箱")
|
||||
parser.add_argument("--root", default="~/workbuddy-agent-os/agent-sync", help="agent-os 根目录")
|
||||
parser.add_argument("--dry-run", action="store_true", help="只预览不写入")
|
||||
parser.add_argument("--force", action="store_true", help="忽略 collected 标记，重新收集")
|
||||
args = parser.parse_args()
|
||||
|
||||
print(f"=== Collect to Inbox ===")
|
||||
print(f"根目录：{args.root}")
|
||||
print(f"模式：{'预览' if args.dry_run else '执行'}")
|
||||
print()
|
||||
|
||||
collect_to_inbox(args.root, dry_run=args.dry_run, force=args.force)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
{"version": "1.0.0", "created": "2026-04-25", "description": "分类目录内容汇聚收件箱"}
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: content_processor
|
||||
version: 2.0.0
|
||||
status: archived
|
||||
archived_date: 2026-06-21
|
||||
description: [已归档] 统一内容采集入口设计，无实现代码
|
||||
triggers: []
|
||||
---
|
||||
|
||||
# Content Processor Skill（统一内容采集入口）
|
||||
|
||||
## 概述
|
||||
|
||||
统一的内容采集入口技能。**两个入口命令**：
|
||||
|
||||
| 你说 | 含义 | purpose | 去向 |
|
||||
|------|------|---------|------|
|
||||
| `收集这个...` | 知识收集 | knowledge | → 01_submissions/ → 提纯入库 |
|
||||
| `收集素材...` | 素材分析 | material | → materials/ + AI 分析结果 |
|
||||
|
||||
### 知识类路由（"收集这个..."）
|
||||
|
||||
| 你说 | collect_type | collect_subtype | 提取方式 |
|
||||
|------|-------------|----------------|----------|
|
||||
| 收集这个视频 | douyin-video | default | 标题+描述+标签 |
|
||||
| 收集这个视频的字幕 | douyin-video | subtitle | 字幕全文 |
|
||||
| 总结这个视频 | douyin-video | summary | AI 总结核心观点 |
|
||||
| 收集这个链接 | webpage | default | 标题+正文摘要 |
|
||||
| 收集这篇文章 | webpage / wechat-article | default | 标题+正文 |
|
||||
| 收集这个音频 | audio | transcription | 语音转文字 |
|
||||
|
||||
### 素材类路由（"收集素材..."）
|
||||
|
||||
| 你说 | collect_type | collect_subtype | AI 分析内容 |
|
||||
|------|-------------|----------------|-------------|
|
||||
| 收集素材 | douyin-video | default | 仅保存原视频 |
|
||||
| 收集素材，视频结构 | douyin-video | video-structure | 起承转合、节奏分段、框架 |
|
||||
| 收集素材，拍摄脚本 | douyin-video | shooting-script | 分镜描述、拍摄手法 |
|
||||
| 收集素材，镜头运镜 | douyin-video | camera-movement | 推拉摇移跟、构图方式 |
|
||||
| 收集素材，前三秒钩子 | douyin-video | opening-hook | 开头吸引点、悬念设置 |
|
||||
| 收集素材，悬疑点 | douyin-video | suspense-points | 悬疑设计、反转点 |
|
||||
| 收集素材，文案结构 | douyin-video | copy-structure | 黄金圈、痛点-方案框架 |
|
||||
| 收集素材 | audio | raw | 仅保存音频文件 |
|
||||
| 收集素材 | image | default | 保存原图 |
|
||||
|
||||
## 输出规则
|
||||
|
||||
### 知识类 → `01_submissions/`
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: "原标题"
|
||||
collect_type: "douyin-video"
|
||||
collect_subtype: "default"
|
||||
purpose: "knowledge"
|
||||
source_url: "链接"
|
||||
source_author: "作者"
|
||||
source_date: "日期"
|
||||
collected_by: "{机器名}"
|
||||
collected_date: "{日期}"
|
||||
status: submitted
|
||||
tags: []
|
||||
---
|
||||
```
|
||||
|
||||
### 素材类 → `agent-local/materials/{type}/`
|
||||
|
||||
```
|
||||
materials/
|
||||
├── audios/ ← 背景音乐、音频原文件
|
||||
├── screenshots/ ← 截图
|
||||
├── images/ ← 原图
|
||||
└── videos/ ← 视频片段
|
||||
```
|
||||
|
||||
## 即时 vs 批量
|
||||
|
||||
| 你说 | 模式 |
|
||||
|------|------|
|
||||
| `收集这个视频` | 批量 → 进提交箱等待 |
|
||||
| `收集这个视频并提纯` | 即时 → 进提交箱 → 立即提纯 → 返回 |
|
||||
| `提纯这个` | 即时 → 对刚收集的内容立即提纯 |
|
||||
|
||||
## 子技能依赖
|
||||
|
||||
| 子技能 | 说明 |
|
||||
|--------|------|
|
||||
| agent-browser | 抖音/网页内容抓取 |
|
||||
| voice-summary | 语音转文字（Whisper） |
|
||||
| web-crawler | 爬虫/反爬 |
|
||||
| @tikomni/skills | 社交平台采集 |
|
||||
@@ -0,0 +1 @@
|
||||
{"name": "content_processor", "version": "1.0.0", "created": "2026-04-25", "updated": "2026-04-25", "status": "active"}
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
name: inbox_refine
|
||||
version: 1.1.0
|
||||
description: 知识库收件箱提纯——每日将 00_inbox/ 内容分类归档到知识库，更新首页统计和变更日志
|
||||
triggers:
|
||||
- 提纯
|
||||
- 收件箱提纯
|
||||
- 归档inbox
|
||||
- inbox refine
|
||||
- 整理收件箱
|
||||
- 归档知识
|
||||
- 提纯收件箱
|
||||
---
|
||||
|
||||
# Inbox Refine 技能
|
||||
|
||||
## 概述
|
||||
|
||||
将 `00_inbox/` 中的原始内容按知识属性分类，应用模板后归档到对应目录，并更新知识库首页和变更日志。
|
||||
|
||||
## 触发条件
|
||||
|
||||
- 每日凌晨 3:00 自动执行（WorkBuddy 自动化）
|
||||
- 手动说"提纯"、"整理收件箱"、"inbox refine"
|
||||
|
||||
## 工作流
|
||||
|
||||
```
|
||||
1. 扫描 00_inbox/ 所有 .md 文件
|
||||
2. 逐个读取 → 分析内容
|
||||
3. 确定分类：
|
||||
- nature：fact/method/concept/reference/data/opinion/quote/axiom/regulation
|
||||
- domain：从 99_system/taxonomies/domains.md 选择 1-3 个
|
||||
- confidence：根据 source_type 评分
|
||||
4. 应用对应模板（99_system/templates/）
|
||||
5. 生成 ID：KB-YYYYMMDD-NNN
|
||||
6. 写入目标目录
|
||||
7. 删除 00_inbox/ 原文件
|
||||
8. 更新 README.md 统计
|
||||
9. 更新 CHANGELOG.md
|
||||
```
|
||||
|
||||
## 分类决策逻辑
|
||||
|
||||
### nature 判定
|
||||
| 内容特征 | nature | 目标目录 |
|
||||
|----------|--------|----------|
|
||||
| 已验证的客观事实 | fact | 30_facts/ |
|
||||
| 可操作的步骤/教程 | method | 20_methods/ |
|
||||
| 原子概念/基础原理 | concept | 10_concepts/ |
|
||||
| 公认基础原理 | axiom | 10_concepts/ |
|
||||
| 法律法规/制度 | regulation | 30_facts/ |
|
||||
| 论文/文档/外部资料 | reference | 40_references/ |
|
||||
| 测试数据/基准结果 | data | 30_facts/ |
|
||||
| 主观看法/推测 | opinion | 60_opinions/ |
|
||||
| 他人原话/语录 | quote | 60_opinions/ |
|
||||
|
||||
### confidence 评分
|
||||
| source_type | confidence |
|
||||
|-------------|-----------|
|
||||
| official_doc | 0.9 |
|
||||
| literature | 0.8 |
|
||||
| experiment | 0.7 |
|
||||
| personal_exp | 0.5 |
|
||||
| social_media | 0.3 |
|
||||
| unknown | 0.4 |
|
||||
|
||||
## 去重规则
|
||||
|
||||
1. 读取目标目录已有文件，提取 title 和 tags
|
||||
2. 标题相似度 > 0.8 → 合并（补充细节，不新建）
|
||||
3. 内容完全一致 → 跳过，记录日志
|
||||
4. 同一事实新旧矛盾 → 新事实覆盖，旧值写入 previous_version
|
||||
|
||||
## Frontmatter 模板
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: KB-YYYYMMDD-NNN
|
||||
title: "__标题__"
|
||||
type: __concept/fact/method/opinion/reference__
|
||||
status: published
|
||||
nature: __fact/method/concept/...__
|
||||
domain: [__领域1__, __领域2__]
|
||||
subdomain: []
|
||||
tags: [__标签1__, __标签2__]
|
||||
confidence: 0.8
|
||||
source: "__来源URL或出处__"
|
||||
source_type: __official_doc/literature/personal_exp/...__
|
||||
date_created: YYYY-MM-DD
|
||||
date_modified: YYYY-MM-DD
|
||||
version: 1
|
||||
previous_version: ""
|
||||
superseded_by: ""
|
||||
---
|
||||
```
|
||||
|
||||
## 更新首页统计
|
||||
|
||||
读取 `03_knowledge/README.md`，替换 `## 📊 知识统计` 下的表格：
|
||||
- 扫描各目录下 .md 文件数量
|
||||
- 找出每个目录最近修改的文件日期
|
||||
- 更新总计行
|
||||
|
||||
## 更新变更日志
|
||||
|
||||
在 `03_knowledge/CHANGELOG.md` 顶部追加：
|
||||
```markdown
|
||||
## YYYY-MM-DD
|
||||
|
||||
### 归档
|
||||
- [nature] title → 目标目录 (KB-YYYYMMDD-NNN)
|
||||
- ...
|
||||
```
|
||||
|
||||
## 依赖
|
||||
|
||||
| 包/工具 | 用途 | 安装状态 |
|
||||
|---------|------|----------|
|
||||
| file_read/file_write | 读写文件 | ✅ 内置 |
|
||||
| python3 | 脚本执行 | ✅ managed |
|
||||
| oMLX + Qwen3-8B-MLX-4bit | 本地 LLM 辅助分类（可选） | ✅ 已安装 |
|
||||
|
||||
## 路径约定
|
||||
|
||||
| 项目 | 路径 |
|
||||
|------|------|
|
||||
| 收件箱 | `~/workbuddy-agent-os/agent-sync/03_knowledge/00_inbox/` |
|
||||
| 知识库根 | `~/workbuddy-agent-os/agent-sync/03_knowledge/` |
|
||||
| 首页 | `~/workbuddy-agent-os/agent-sync/03_knowledge/README.md` |
|
||||
| 变更日志 | `~/workbuddy-agent-os/agent-sync/03_knowledge/CHANGELOG.md` |
|
||||
| 模板 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/templates/` |
|
||||
| 分类表 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/` |
|
||||
| 中文映射 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/folder-aliases.json` |
|
||||
|
||||
## 错误处理
|
||||
|
||||
- 无法分类的内容 → 保留在 00_inbox/，添加标签 `#待分类`，记录日志
|
||||
- 模板应用失败 → 使用最小 Frontmatter（id/title/nature/date_created）
|
||||
- 首页更新失败 → 只记录日志，不阻断归档流程
|
||||
@@ -0,0 +1,46 @@
|
||||
skill:
|
||||
name: "收件箱精炼器"
|
||||
id: "inbox_refine"
|
||||
version: "1.1.0"
|
||||
category: "core-system"
|
||||
status: "active"
|
||||
created: "2026-04-25"
|
||||
updated: "2026-04-29"
|
||||
|
||||
description:
|
||||
short: "扫描协同inbox五分区，AI分类，自动归档到知识库"
|
||||
use_cases:
|
||||
- "定期扫描inbox各分区，按knowledge/memory/personal分类处理"
|
||||
- "clipping分区：全文入库"
|
||||
- "feed分区：摘要入库，原文归档"
|
||||
- "memory分区：入长期记忆，同时建立知识卡"
|
||||
- "personal分区：入personal知识区"
|
||||
- "tools分区：转交工具审核流程"
|
||||
tags: ["inbox", "精炼", "分类", "归档"]
|
||||
|
||||
ownership:
|
||||
master_editor: "my-macbook_macOS14.4_16GB_master"
|
||||
readonly_on_other_devices: false
|
||||
|
||||
environment:
|
||||
os: "macos"
|
||||
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
|
||||
pip_dependencies:
|
||||
- "chromadb>=1.0.0"
|
||||
|
||||
setup:
|
||||
install_script: ""
|
||||
post_check: "python3 -c 'print(\"OK\")'"
|
||||
|
||||
usage:
|
||||
entry_point: "inbox_refine.py"
|
||||
scheduled: "每日 03:00（master机器执行）"
|
||||
manual_trigger: "收件箱精炼"
|
||||
|
||||
changelog:
|
||||
- version: "1.1.0"
|
||||
date: "2026-04-29"
|
||||
changes: "支持五分区差异化处理逻辑，适配新inbox路径"
|
||||
- version: "1.0.0"
|
||||
date: "2026-04-25"
|
||||
changes: "初始版本"
|
||||
@@ -0,0 +1,461 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
inbox_refine.py - 知识库收件箱提纯脚本
|
||||
|
||||
将 00_inbox/ 中的原始内容按知识属性分类归档到对应目录，
|
||||
更新知识库首页统计和变更日志。
|
||||
|
||||
用法：
|
||||
python3 inbox_refine.py --root ~/workbuddy-agent-os/agent-sync [--dry-run] [--date YYYY-MM-DD]
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import argparse
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# ─── 角色检查：仅 master 可执行收件箱提纯 ───
|
||||
try:
|
||||
sys.path.insert(0, str(Path.home() / "workbuddy-agent-os" / "agent-sync" / "05_tools" / "01_system"))
|
||||
from role_check import check_role, check_capability
|
||||
if not check_role(["master"]):
|
||||
print("⛔ 非 master 角色，跳过 inbox_refine")
|
||||
sys.exit(0)
|
||||
if not check_capability("inbox_refine"):
|
||||
print("⛔ 能力 inbox_refine 未启用，跳过")
|
||||
sys.exit(0)
|
||||
except ImportError as e:
|
||||
print(f"⚠️ 角色检查模块不可用 ({e})，继续执行")
|
||||
|
||||
# 导入 LLM 分类器
|
||||
try:
|
||||
from llm_classifier import classify_content_enhanced
|
||||
LLM_AVAILABLE = True
|
||||
except ImportError:
|
||||
print("⚠️ LLM 分类器不可用，将使用启发式规则")
|
||||
LLM_AVAILABLE = False
|
||||
|
||||
|
||||
# ── 配置 ──────────────────────────────────────────
|
||||
|
||||
NATURE_TO_DIR = {
|
||||
"fact": "30_facts",
|
||||
"method": "20_methods",
|
||||
"concept": "10_concepts",
|
||||
"axiom": "10_concepts",
|
||||
"regulation": "30_facts",
|
||||
"reference": "40_references",
|
||||
"data": "30_facts",
|
||||
"opinion": "60_opinions",
|
||||
"quote": "60_opinions",
|
||||
}
|
||||
|
||||
SOURCE_TYPE_CONFIDENCE = {
|
||||
"official_doc": 0.9,
|
||||
"literature": 0.8,
|
||||
"experiment": 0.7,
|
||||
"personal_exp": 0.5,
|
||||
"social_media": 0.3,
|
||||
"unknown": 0.4,
|
||||
}
|
||||
|
||||
DIR_CHINESE = {
|
||||
"00_inbox": "📥 收件箱",
|
||||
"01_daily": "📅 日记",
|
||||
"10_concepts": "💡 概念层",
|
||||
"20_methods": "🔧 方法层",
|
||||
"30_facts": "📋 事实层",
|
||||
"40_references": "📎 参考层",
|
||||
"50_resources": "🛠 资源层",
|
||||
"60_opinions": "💭 观点层",
|
||||
"90_archive": "🗄 归档层",
|
||||
}
|
||||
|
||||
TRACKED_DIRS = [
|
||||
"00_inbox", "01_daily", "10_concepts", "20_methods",
|
||||
"30_facts", "40_references", "50_resources", "60_opinions", "90_archive"
|
||||
]
|
||||
|
||||
|
||||
# ── 工具函数 ──────────────────────────────────────
|
||||
|
||||
def count_md_files(directory: Path) -> int:
|
||||
"""统计目录下 .md 文件数量"""
|
||||
if not directory.exists():
|
||||
return 0
|
||||
return sum(1 for f in directory.rglob("*.md") if f.is_file())
|
||||
|
||||
|
||||
def latest_modified(directory: Path) -> str:
|
||||
"""获取目录下最近修改的 .md 文件日期"""
|
||||
if not directory.exists():
|
||||
return "—"
|
||||
md_files = list(directory.rglob("*.md"))
|
||||
if not md_files:
|
||||
return "—"
|
||||
latest = max(f.stat().st_mtime for f in md_files)
|
||||
return datetime.fromtimestamp(latest).strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def generate_id(date_str: str, knowledge_root: Path) -> str:
|
||||
"""生成唯一知识ID：KB-YYYYMMDD-NNN"""
|
||||
prefix = f"KB-{date_str.replace('-', '')}"
|
||||
# 扫描已有ID确定序号
|
||||
existing = set()
|
||||
for md in knowledge_root.rglob("*.md"):
|
||||
content = md.read_text(encoding="utf-8", errors="ignore")[:500]
|
||||
m = re.search(r'^id:\s*(KB-\d{8}-\d+)', content, re.MULTILINE)
|
||||
if m:
|
||||
existing.add(m.group(1))
|
||||
n = 1
|
||||
while f"{prefix}-{n:03d}" in existing:
|
||||
n += 1
|
||||
return f"{prefix}-{n:03d}"
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> dict:
|
||||
"""解析 YAML frontmatter"""
|
||||
if not text.startswith("---"):
|
||||
return {}
|
||||
match = re.match(r'^---\s*\n(.*?)\n---', text, re.DOTALL)
|
||||
if not match:
|
||||
return {}
|
||||
fm = {}
|
||||
for line in match.group(1).split("\n"):
|
||||
if ":" in line:
|
||||
key, val = line.split(":", 1)
|
||||
fm[key.strip()] = val.strip().strip('"').strip("'")
|
||||
return fm
|
||||
|
||||
|
||||
def build_frontmatter(kb_id: str, title: str, nature: str, domain: str,
|
||||
confidence: float, source: str, source_type: str,
|
||||
date_str: str, summary: str = "", tags: list = None) -> str:
|
||||
"""构建知识卡片 frontmatter"""
|
||||
if tags is None:
|
||||
tags = ["待补充"]
|
||||
|
||||
# 确保 domain 是列表格式
|
||||
if isinstance(domain, str):
|
||||
domain_list = [domain]
|
||||
else:
|
||||
domain_list = domain if isinstance(domain, list) else ["other"]
|
||||
|
||||
# 构建 tags 字符串
|
||||
tags_str = ", ".join([f'"{tag}"' for tag in tags])
|
||||
|
||||
return f"""---
|
||||
id: {kb_id}
|
||||
title: "{title}"
|
||||
type: {nature if nature in ('concept', 'fact', 'method', 'opinion', 'reference') else 'concept'}
|
||||
status: published
|
||||
nature: {nature}
|
||||
domain: [{", ".join(domain_list)}]
|
||||
subdomain: []
|
||||
tags: [{tags_str}]
|
||||
confidence: {confidence}
|
||||
source: "{source}"
|
||||
source_type: {source_type}
|
||||
date_created: {date_str}
|
||||
date_modified: {date_str}
|
||||
version: 1
|
||||
previous_version: ""
|
||||
superseded_by: ""
|
||||
summary: "{summary}"
|
||||
---"""
|
||||
|
||||
|
||||
# ── 分类逻辑 ──────────────────────────────────────
|
||||
|
||||
def classify_content(text: str, fm: dict) -> dict:
|
||||
"""
|
||||
增强版分类函数：优先使用 LLM 智能分类，失败时降级到启发式规则。
|
||||
如果 frontmatter 已有 nature/domain，优先使用。
|
||||
"""
|
||||
# 如果 LLM 可用，直接使用增强版分类
|
||||
if LLM_AVAILABLE:
|
||||
try:
|
||||
# 直接使用 LLM 分类，分类器内部会处理降级
|
||||
result = classify_content_enhanced(text, fm, use_llm=True)
|
||||
# 适配现有结构
|
||||
return {
|
||||
"nature": result["nature"],
|
||||
"domain": result["domain"][0] if result["domain"] else "other",
|
||||
"confidence": result["confidence"],
|
||||
"source_type": result.get("source_type", "unknown"),
|
||||
"source": result.get("source", ""),
|
||||
"title": result.get("title", ""),
|
||||
"summary": result.get("summary", ""),
|
||||
"tags": result.get("tags", [])
|
||||
}
|
||||
except Exception as e:
|
||||
print(f"⚠️ LLM 分类失败，回退到启发式规则: {e}")
|
||||
|
||||
# LLM 不可用或失败，使用启发式规则
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": "other",
|
||||
"confidence": 0.4,
|
||||
"source_type": "unknown",
|
||||
"source": "",
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature") and fm["nature"] in NATURE_TO_DIR:
|
||||
result["nature"] = fm["nature"]
|
||||
if fm.get("domain"):
|
||||
result["domain"] = fm["domain"]
|
||||
if fm.get("confidence"):
|
||||
try:
|
||||
result["confidence"] = float(fm["confidence"])
|
||||
except ValueError:
|
||||
pass
|
||||
if fm.get("source"):
|
||||
result["source"] = fm["source"]
|
||||
if fm.get("source_type"):
|
||||
result["source_type"] = fm["source_type"]
|
||||
if fm.get("title"):
|
||||
result["title"] = fm["title"]
|
||||
|
||||
# 如果没有 nature，用启发式规则
|
||||
if not fm.get("nature"):
|
||||
lower = text.lower()
|
||||
# 方法类关键词
|
||||
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践", "best practice"]
|
||||
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
|
||||
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
|
||||
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
|
||||
|
||||
if any(kw in lower for kw in method_kw):
|
||||
result["nature"] = "method"
|
||||
elif any(kw in lower for kw in fact_kw):
|
||||
result["nature"] = "fact"
|
||||
elif any(kw in lower for kw in opinion_kw):
|
||||
result["nature"] = "opinion"
|
||||
elif any(kw in lower for kw in reference_kw):
|
||||
result["nature"] = "reference"
|
||||
|
||||
# confidence from source_type
|
||||
if result["source_type"] in SOURCE_TYPE_CONFIDENCE:
|
||||
result["confidence"] = SOURCE_TYPE_CONFIDENCE[result["source_type"]]
|
||||
|
||||
# 生成简单标题和摘要
|
||||
if not result["title"]:
|
||||
first_line = text.split('\n')[0].strip('# ')
|
||||
if len(first_line) < 50:
|
||||
result["title"] = first_line
|
||||
|
||||
if not result["summary"]:
|
||||
clean_text = re.sub(r'[#*\-`]', '', text)
|
||||
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ── 主流程 ─────────────────────────────────────────
|
||||
|
||||
def refine_inbox(root: str, dry_run: bool = False, date_str: str = None):
|
||||
"""执行收件箱提纯"""
|
||||
root = Path(os.path.expanduser(root))
|
||||
kb_root = root / "03_knowledge"
|
||||
inbox = kb_root / "00_inbox"
|
||||
|
||||
if not inbox.exists():
|
||||
print(f"❌ 收件箱不存在: {inbox}")
|
||||
return
|
||||
|
||||
date_str = date_str or datetime.now().strftime("%Y-%m-%d")
|
||||
md_files = sorted(inbox.glob("*.md"))
|
||||
|
||||
if not md_files:
|
||||
print(f"📭 收件箱为空，无需提纯 ({date_str})")
|
||||
# 仍然更新首页统计
|
||||
update_homepage(kb_root)
|
||||
return
|
||||
|
||||
print(f"📬 收件箱有 {len(md_files)} 个文件待提纯")
|
||||
|
||||
archived = []
|
||||
skipped = []
|
||||
errors = []
|
||||
|
||||
for md_file in md_files:
|
||||
try:
|
||||
text = md_file.read_text(encoding="utf-8", errors="ignore")
|
||||
fm = parse_frontmatter(text)
|
||||
title = fm.get("title", md_file.stem)
|
||||
|
||||
# 分类
|
||||
classification = classify_content(text, fm)
|
||||
nature = classification["nature"]
|
||||
target_dir_name = NATURE_TO_DIR.get(nature, "10_concepts")
|
||||
target_dir = kb_root / target_dir_name
|
||||
|
||||
# 去重检查
|
||||
is_dup = False
|
||||
for existing in target_dir.rglob("*.md"):
|
||||
existing_text = existing.read_text(encoding="utf-8", errors="ignore")[:200]
|
||||
existing_fm = parse_frontmatter(existing_text)
|
||||
if existing_fm.get("title") == title:
|
||||
is_dup = True
|
||||
break
|
||||
|
||||
if is_dup:
|
||||
print(f" ⏭ 重复跳过: {title}")
|
||||
skipped.append(title)
|
||||
if not dry_run:
|
||||
md_file.unlink()
|
||||
continue
|
||||
|
||||
# 生成 ID 和新文件
|
||||
kb_id = generate_id(date_str, kb_root)
|
||||
domain = classification["domain"]
|
||||
confidence = classification["confidence"]
|
||||
source = classification["source"]
|
||||
source_type = classification["source_type"]
|
||||
|
||||
# 构建新内容
|
||||
body = text
|
||||
if text.startswith("---"):
|
||||
# 去掉旧 frontmatter
|
||||
body = re.sub(r'^---\s*\n.*?\n---\s*\n', '', text, count=1, flags=re.DOTALL)
|
||||
|
||||
new_content = build_frontmatter(
|
||||
kb_id,
|
||||
classification.get("title", title),
|
||||
nature,
|
||||
domain,
|
||||
confidence,
|
||||
source,
|
||||
source_type,
|
||||
date_str,
|
||||
classification.get("summary", ""),
|
||||
classification.get("tags", [])
|
||||
) + "\n\n" + body.strip() + "\n"
|
||||
|
||||
# 确定目标文件名
|
||||
safe_title = re.sub(r'[^\w\u4e00-\u9fff-]', '_', title)[:50]
|
||||
target_file = target_dir / f"{date_str}_{safe_title}.md"
|
||||
|
||||
if dry_run:
|
||||
print(f" 🔍 [DRY] {title} → {target_dir_name}/ ({nature})")
|
||||
else:
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
target_file.write_text(new_content, encoding="utf-8")
|
||||
md_file.unlink()
|
||||
print(f" ✅ {title} → {target_dir_name}/ ({nature}) [{kb_id}]")
|
||||
|
||||
archived.append({
|
||||
"id": kb_id,
|
||||
"title": title,
|
||||
"nature": nature,
|
||||
"dir": target_dir_name,
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ 处理失败: {md_file.name} - {e}")
|
||||
errors.append(md_file.name)
|
||||
|
||||
# 更新首页
|
||||
if not dry_run:
|
||||
update_homepage(kb_root)
|
||||
update_changelog(kb_root, date_str, archived, skipped)
|
||||
|
||||
print(f"\n📊 提纯完成: 归档 {len(archived)}, 跳过 {len(skipped)}, 失败 {len(errors)}")
|
||||
|
||||
|
||||
def update_homepage(kb_root: Path):
|
||||
"""更新知识库首页统计"""
|
||||
readme = kb_root / "README.md"
|
||||
if not readme.exists():
|
||||
print("⚠️ 首页 README.md 不存在，跳过统计更新")
|
||||
return
|
||||
|
||||
# 计算各目录统计
|
||||
stats = {}
|
||||
total = 0
|
||||
for dirname in TRACKED_DIRS:
|
||||
dirpath = kb_root / dirname
|
||||
count = count_md_files(dirpath)
|
||||
latest = latest_modified(dirpath)
|
||||
stats[dirname] = (count, latest)
|
||||
total += count
|
||||
|
||||
# 构建统计表格
|
||||
table_lines = []
|
||||
for dirname in TRACKED_DIRS:
|
||||
count, latest = stats[dirname]
|
||||
emoji_name = DIR_CHINESE.get(dirname, dirname)
|
||||
table_lines.append(f"| {emoji_name} | {count} | {latest} |")
|
||||
table_lines.append(f"| **总计** | **{total}** | — |")
|
||||
|
||||
table_text = "\n".join(table_lines)
|
||||
|
||||
# 替换 README 中的统计部分
|
||||
content = readme.read_text(encoding="utf-8")
|
||||
pattern = r'(\| 📥 收件箱 \|.*?)(\| \*\*总计\*\* \| \*\*\d+\*\* \| — \|)'
|
||||
replacement = table_text
|
||||
content = re.sub(pattern, replacement, content, flags=re.DOTALL)
|
||||
|
||||
# 更新最后更新日期
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
content = re.sub(
|
||||
r'\*最后更新：.*? by inbox_refine 技能\*',
|
||||
f'*最后更新：{today} by inbox_refine 技能*',
|
||||
content
|
||||
)
|
||||
|
||||
readme.write_text(content, encoding="utf-8")
|
||||
print(f"📄 首页统计已更新 (总计 {total} 个文件)")
|
||||
|
||||
|
||||
def update_changelog(kb_root: Path, date_str: str, archived: list, skipped: list):
|
||||
"""更新变更日志"""
|
||||
changelog = kb_root / "CHANGELOG.md"
|
||||
if not changelog.exists():
|
||||
initial = "# 知识库变更日志\n"
|
||||
changelog.write_text(initial, encoding="utf-8")
|
||||
|
||||
content = changelog.read_text(encoding="utf-8")
|
||||
|
||||
# 构建新条目
|
||||
entry = f"\n## {date_str}\n\n"
|
||||
if archived:
|
||||
entry += "### 归档\n"
|
||||
for item in archived:
|
||||
entry += f"- [{item['nature']}] {item['title']} → {item['dir']}/ ({item['id']})\n"
|
||||
if skipped:
|
||||
entry += f"\n### 跳过（重复）\n"
|
||||
for title in skipped:
|
||||
entry += f"- {title}\n"
|
||||
|
||||
# 在第一个 ## 之前插入（跳过标题行）
|
||||
lines = content.split("\n")
|
||||
insert_idx = 1
|
||||
for i, line in enumerate(lines):
|
||||
if line.startswith("## "):
|
||||
insert_idx = i
|
||||
break
|
||||
|
||||
lines.insert(insert_idx, entry)
|
||||
changelog.write_text("\n".join(lines), encoding="utf-8")
|
||||
print(f"📝 变更日志已更新")
|
||||
|
||||
|
||||
# ── 入口 ──────────────────────────────────────────
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="AgentOS 知识库收件箱提纯")
|
||||
parser.add_argument("--root", default="~/workbuddy-agent-os/agent-sync", help="agent-os 根目录")
|
||||
parser.add_argument("--dry-run", action="store_true", help="试运行，不实际修改文件")
|
||||
parser.add_argument("--date", default=None, help="日期 (YYYY-MM-DD)")
|
||||
args = parser.parse_args()
|
||||
|
||||
refine_inbox(args.root, args.dry_run, args.date)
|
||||
@@ -0,0 +1,395 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LLM 分类器模块 - 使用 oMLX 本地大模型进行知识分类
|
||||
|
||||
功能：
|
||||
1. 根据内容判断 nature（fact/method/concept/opinion/reference 等）
|
||||
2. 根据内容判断 domain（技术/金融/产品/生活 等）
|
||||
3. 提取关键信息（标题、摘要、标签）
|
||||
|
||||
配置：
|
||||
- oMLX API: http://localhost:8000/v1/chat/completions
|
||||
- 模型: Qwen2.5-VL-3B-Instruct-8bit
|
||||
- API Key: omlx
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
from pathlib import Path
|
||||
|
||||
# ─── oMLX 配置 ──────────────────────────────────────────────────
|
||||
OLLAMA_API_URL = "http://localhost:8000/v1/chat/completions"
|
||||
OLLAMA_MODEL = "Qwen2.5-VL-3B-Instruct-8bit" # 使用 VLM 模型，因为 Qwen3-8B-MLX-4bit 有 500 错误
|
||||
OLLAMA_API_KEY = "omlx"
|
||||
|
||||
# 备份模型选项（如果 Qwen2.5 不可用）
|
||||
FALLBACK_MODEL = "Qwen3-8B-MLX-4bit"
|
||||
|
||||
# ─── 分类体系 ──────────────────────────────────────────────────
|
||||
NATURE_OPTIONS = [
|
||||
"fact", # 客观事实、数据、统计结果
|
||||
"method", # 方法、步骤、教程、最佳实践
|
||||
"concept", # 概念、定义、理论、原理
|
||||
"opinion", # 观点、看法、评价、推测
|
||||
"reference", # 参考文献、论文、书籍、资料
|
||||
"quote", # 引用、名言、摘录
|
||||
"axiom", # 公理、定律、定理
|
||||
"regulation", # 法规、政策、标准
|
||||
"data", # 原始数据、数据集、代码片段
|
||||
]
|
||||
|
||||
DOMAIN_OPTIONS = [
|
||||
"technology", # 技术、编程、AI、开发
|
||||
"finance", # 金融、投资、股票、经济
|
||||
"product", # 产品、设计、UX、PM
|
||||
"life", # 生活、健康、个人管理
|
||||
"academic", # 学术、研究、教育
|
||||
"business", # 商业、管理、创业
|
||||
"culture", # 文化、艺术、娱乐
|
||||
"other", # 其他
|
||||
]
|
||||
|
||||
# ─── LLM 调用函数 ──────────────────────────────────────────────
|
||||
def call_llm(prompt: str, system_prompt: str = None, temperature: float = 0.3) -> str:
|
||||
"""
|
||||
调用 oMLX 本地大模型 API
|
||||
|
||||
Args:
|
||||
prompt: 用户提示
|
||||
system_prompt: 系统提示（可选）
|
||||
temperature: 温度参数，越低越确定
|
||||
|
||||
Returns:
|
||||
LLM 回复文本
|
||||
"""
|
||||
import requests
|
||||
|
||||
messages = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
try:
|
||||
# 首先尝试 VLM 模型
|
||||
resp = requests.post(
|
||||
OLLAMA_API_URL,
|
||||
json={
|
||||
"model": OLLAMA_MODEL,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"stream": False
|
||||
},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OLLAMA_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
# 如果 VLM 模型失败（500 错误），尝试备选模型
|
||||
if resp.status_code == 500 and OLLAMA_MODEL == "Qwen2.5-VL-3B-Instruct-8bit":
|
||||
print(f" ⚠️ VLM 模型失败，尝试备选模型 {FALLBACK_MODEL}")
|
||||
resp = requests.post(
|
||||
OLLAMA_API_URL,
|
||||
json={
|
||||
"model": FALLBACK_MODEL,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"stream": False
|
||||
},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OLLAMA_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data["choices"][0]["message"]["content"]
|
||||
except Exception as e:
|
||||
print(f"[WARN] LLM 调用失败: {e}")
|
||||
return ""
|
||||
|
||||
# ─── 分类函数 ──────────────────────────────────────────────────
|
||||
def classify_with_llm(text: str) -> Dict:
|
||||
"""
|
||||
使用 LLM 对文本内容进行分类
|
||||
|
||||
Args:
|
||||
text: 要分类的文本内容（Markdown 格式）
|
||||
|
||||
Returns:
|
||||
{
|
||||
"nature": "fact/method/concept/...",
|
||||
"domain": "technology/finance/...",
|
||||
"confidence": 0.0-1.0,
|
||||
"title": "建议的标题",
|
||||
"summary": "简短摘要",
|
||||
"tags": ["标签1", "标签2", ...]
|
||||
}
|
||||
"""
|
||||
# 限制文本长度（防止 token 过长）
|
||||
preview = text[:2000] if len(text) > 2000 else text
|
||||
|
||||
system_prompt = """你是一个知识分类专家。请根据提供的文本内容进行以下分析：
|
||||
1. 判断知识类型（nature）：从以下选项中选择最合适的：fact（事实数据）、method（方法步骤）、concept（概念理论）、opinion（观点看法）、reference（参考文献）、quote（引用摘录）、axiom（公理定律）、regulation（法规政策）、data（原始数据）
|
||||
2. 判断所属领域（domain）：从以下选项中选择1-2个：technology（技术）、finance（金融）、product（产品）、life（生活）、academic（学术）、business（商业）、culture（文化）、other（其他）
|
||||
3. 提取关键信息：建议一个简洁的标题、生成50字内的摘要、提取2-5个关键词标签
|
||||
|
||||
请以 JSON 格式返回结果，严格遵循以下结构：
|
||||
{
|
||||
"nature": "选择的类型",
|
||||
"domain": ["领域1", "领域2"],
|
||||
"confidence": 0.0-1.0之间的数字（表示分类信心度），
|
||||
"title": "建议标题",
|
||||
"summary": "简短摘要",
|
||||
"tags": ["标签1", "标签2", "标签3"]
|
||||
}"""
|
||||
|
||||
user_prompt = f"""请对以下文本内容进行分类分析：
|
||||
|
||||
{preview}
|
||||
|
||||
注意：文本可能是截断的，请基于现有内容进行判断。"""
|
||||
|
||||
response = call_llm(user_prompt, system_prompt, temperature=0.1)
|
||||
|
||||
# 解析 JSON 响应
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.5,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
if not response:
|
||||
return result
|
||||
|
||||
try:
|
||||
# 查找 JSON 部分
|
||||
json_match = re.search(r'\{.*\}', response, re.DOTALL)
|
||||
if json_match:
|
||||
parsed = json.loads(json_match.group())
|
||||
if isinstance(parsed, dict):
|
||||
if "nature" in parsed and parsed["nature"] in NATURE_OPTIONS:
|
||||
result["nature"] = parsed["nature"]
|
||||
if "domain" in parsed:
|
||||
if isinstance(parsed["domain"], str):
|
||||
result["domain"] = [parsed["domain"]]
|
||||
elif isinstance(parsed["domain"], list):
|
||||
result["domain"] = [d for d in parsed["domain"] if d in DOMAIN_OPTIONS]
|
||||
if "confidence" in parsed:
|
||||
try:
|
||||
conf = float(parsed["confidence"])
|
||||
if 0 <= conf <= 1:
|
||||
result["confidence"] = conf
|
||||
except:
|
||||
pass
|
||||
if "title" in parsed:
|
||||
result["title"] = str(parsed["title"])
|
||||
if "summary" in parsed:
|
||||
result["summary"] = str(parsed["summary"])
|
||||
if "tags" in parsed:
|
||||
if isinstance(parsed["tags"], list):
|
||||
result["tags"] = [str(tag) for tag in parsed["tags"]]
|
||||
except Exception as e:
|
||||
print(f"[WARN] LLM 响应解析失败: {e}")
|
||||
|
||||
return result
|
||||
|
||||
# ─── 启发式分类备用方案 ────────────────────────────────────────
|
||||
def classify_heuristic(text: str, fm: dict) -> Dict:
|
||||
"""
|
||||
启发式分类（当 LLM 不可用时使用）
|
||||
|
||||
Args:
|
||||
text: 文本内容
|
||||
fm: 已有的 frontmatter
|
||||
|
||||
Returns:
|
||||
与 classify_with_llm 相同结构的字典
|
||||
"""
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.4,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature"):
|
||||
result["nature"] = fm["nature"]
|
||||
if fm.get("domain"):
|
||||
if isinstance(fm["domain"], str):
|
||||
result["domain"] = [fm["domain"]]
|
||||
elif isinstance(fm["domain"], list):
|
||||
result["domain"] = fm["domain"]
|
||||
|
||||
# 启发式规则
|
||||
lower = text.lower()
|
||||
|
||||
# nature 判断
|
||||
if not fm.get("nature"):
|
||||
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践"]
|
||||
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
|
||||
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
|
||||
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
|
||||
|
||||
if any(kw in lower for kw in method_kw):
|
||||
result["nature"] = "method"
|
||||
elif any(kw in lower for kw in fact_kw):
|
||||
result["nature"] = "fact"
|
||||
elif any(kw in lower for kw in opinion_kw):
|
||||
result["nature"] = "opinion"
|
||||
elif any(kw in lower for kw in reference_kw):
|
||||
result["nature"] = "reference"
|
||||
|
||||
# domain 判断
|
||||
if not fm.get("domain"):
|
||||
tech_kw = ["编程", "代码", "开发", "python", "javascript", "ai", "llm", "机器学习", "技术"]
|
||||
finance_kw = ["投资", "股票", "基金", "finance", "交易", "财经", "金融"]
|
||||
product_kw = ["产品", "设计", "ux", "product", "用户体验"]
|
||||
|
||||
if any(kw in lower for kw in tech_kw):
|
||||
result["domain"] = ["technology"]
|
||||
elif any(kw in lower for kw in finance_kw):
|
||||
result["domain"] = ["finance"]
|
||||
elif any(kw in lower for kw in product_kw):
|
||||
result["domain"] = ["product"]
|
||||
|
||||
# 生成简单标题（使用文件名或第一行）
|
||||
if fm.get("title"):
|
||||
result["title"] = fm["title"]
|
||||
else:
|
||||
first_line = text.split('\n')[0].strip('# ')
|
||||
if len(first_line) < 50:
|
||||
result["title"] = first_line
|
||||
|
||||
# 生成简单摘要（前 100 字符）
|
||||
clean_text = re.sub(r'[#*\-`]', '', text)
|
||||
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
|
||||
|
||||
return result
|
||||
|
||||
# ─── 主分类函数 ──────────────────────────────────────────────────
|
||||
def classify_content_enhanced(text: str, fm: dict = None, use_llm: bool = True) -> Dict:
|
||||
"""
|
||||
增强版分类函数：优先使用 LLM，失败时降级到启发式
|
||||
|
||||
Args:
|
||||
text: 文本内容
|
||||
fm: 已有的 frontmatter（可选）
|
||||
use_llm: 是否尝试使用 LLM（默认 True）
|
||||
|
||||
Returns:
|
||||
分类结果字典
|
||||
"""
|
||||
if fm is None:
|
||||
fm = {}
|
||||
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.5,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": [],
|
||||
"source_type": "unknown",
|
||||
"source": ""
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature"):
|
||||
result["nature"] = fm["nature"]
|
||||
result["confidence"] = max(result["confidence"], 0.7) # 已有信息提升信心
|
||||
|
||||
if fm.get("domain"):
|
||||
if isinstance(fm["domain"], str):
|
||||
result["domain"] = [fm["domain"]]
|
||||
elif isinstance(fm["domain"], list):
|
||||
result["domain"] = fm["domain"]
|
||||
|
||||
# 尝试使用 LLM 分类
|
||||
llm_result = None
|
||||
if use_llm and text.strip():
|
||||
try:
|
||||
print(" 🤖 使用 LLM 进行智能分类...")
|
||||
llm_result = classify_with_llm(text)
|
||||
|
||||
# 合并 LLM 结果（降低阈值要求，因为 VLM 模型可能返回较低的 confidence）
|
||||
confidence_threshold = 0.4 # 降低阈值，因为 VLM 模型可能返回较低的 confidence
|
||||
if llm_result.get("confidence", 0) > confidence_threshold:
|
||||
if not fm.get("nature") and llm_result.get("nature"):
|
||||
result["nature"] = llm_result["nature"]
|
||||
if not fm.get("domain") and llm_result.get("domain"):
|
||||
result["domain"] = llm_result["domain"]
|
||||
if llm_result.get("title"):
|
||||
result["title"] = llm_result["title"]
|
||||
if llm_result.get("summary"):
|
||||
result["summary"] = llm_result["summary"]
|
||||
if llm_result.get("tags"):
|
||||
result["tags"] = llm_result["tags"]
|
||||
result["confidence"] = llm_result.get("confidence", 0.6)
|
||||
print(f" ✅ LLM 分类成功 (confidence: {llm_result.get('confidence', 0.6):.2f})")
|
||||
else:
|
||||
print(f" ⚠️ LLM 分类信心度低 ({llm_result.get('confidence', 0):.2f} < {confidence_threshold})，使用启发式规则")
|
||||
llm_result = None
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ LLM 分类失败: {e}")
|
||||
llm_result = None
|
||||
|
||||
# LLM 失败或信心不足，使用启发式规则
|
||||
if llm_result is None:
|
||||
heuristic_result = classify_heuristic(text, fm)
|
||||
|
||||
# 合并启发式结果
|
||||
if not fm.get("nature") and heuristic_result.get("nature"):
|
||||
result["nature"] = heuristic_result["nature"]
|
||||
if not fm.get("domain") and heuristic_result.get("domain"):
|
||||
result["domain"] = heuristic_result["domain"]
|
||||
if not result["title"] and heuristic_result.get("title"):
|
||||
result["title"] = heuristic_result["title"]
|
||||
if not result["summary"] and heuristic_result.get("summary"):
|
||||
result["summary"] = heuristic_result["summary"]
|
||||
|
||||
# 来源信息
|
||||
if fm.get("source"):
|
||||
result["source"] = fm["source"]
|
||||
if fm.get("source_type"):
|
||||
result["source_type"] = fm["source_type"]
|
||||
|
||||
# 防止空列表
|
||||
if not result["domain"]:
|
||||
result["domain"] = ["other"]
|
||||
if not result["tags"]:
|
||||
result["tags"] = ["待补充"]
|
||||
|
||||
return result
|
||||
|
||||
# ─── 测试函数 ──────────────────────────────────────────────────
|
||||
if __name__ == "__main__":
|
||||
# 测试代码
|
||||
test_text = """Python 中的装饰器是一种强大的工具，它允许在不修改原函数代码的情况下，为函数添加额外的功能。
|
||||
|
||||
装饰器的基本语法：
|
||||
@decorator
|
||||
def my_function():
|
||||
pass
|
||||
|
||||
常见的装饰器应用场景包括：日志记录、性能测试、权限验证等。"""
|
||||
|
||||
print("测试 LLM 分类：")
|
||||
result = classify_content_enhanced(test_text, use_llm=True)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
print("\n测试启发式分类（关闭 LLM）：")
|
||||
result2 = classify_content_enhanced(test_text, use_llm=False)
|
||||
print(json.dumps(result2, ensure_ascii=False, indent=2))
|
||||
@@ -0,0 +1,394 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LLM 分类器模块 - 使用 oMLX 本地大模型进行知识分类
|
||||
|
||||
功能：
|
||||
1. 根据内容判断 nature（fact/method/concept/opinion/reference 等）
|
||||
2. 根据内容判断 domain（技术/金融/产品/生活 等）
|
||||
3. 提取关键信息（标题、摘要、标签）
|
||||
|
||||
配置：
|
||||
- oMLX API: http://localhost:8000/v1/chat/completions
|
||||
- 模型: Qwen3-8B-MLX-4bit
|
||||
- API Key: omlx
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
from pathlib import Path
|
||||
|
||||
# ─── oMLX 配置 ──────────────────────────────────────────────────
|
||||
OLLAMA_API_URL = "http://localhost:8000/v1/chat/completions"
|
||||
OLLAMA_MODEL = "Qwen2.5-VL-3B-Instruct-8bit" # 使用 VLM 模型，因为 Qwen3-8B-MLX-4bit 有 500 错误
|
||||
OLLAMA_API_KEY = "omlx"
|
||||
|
||||
# 备份模型选项（如果 Qwen2.5 不可用）
|
||||
FALLBACK_MODEL = "Qwen3-8B-MLX-4bit"
|
||||
|
||||
# ─── 分类体系 ──────────────────────────────────────────────────
|
||||
NATURE_OPTIONS = [
|
||||
"fact", # 客观事实、数据、统计结果
|
||||
"method", # 方法、步骤、教程、最佳实践
|
||||
"concept", # 概念、定义、理论、原理
|
||||
"opinion", # 观点、看法、评价、推测
|
||||
"reference", # 参考文献、论文、书籍、资料
|
||||
"quote", # 引用、名言、摘录
|
||||
"axiom", # 公理、定律、定理
|
||||
"regulation", # 法规、政策、标准
|
||||
"data", # 原始数据、数据集、代码片段
|
||||
]
|
||||
|
||||
DOMAIN_OPTIONS = [
|
||||
"technology", # 技术、编程、AI、开发
|
||||
"finance", # 金融、投资、股票、经济
|
||||
"product", # 产品、设计、UX、PM
|
||||
"life", # 生活、健康、个人管理
|
||||
"academic", # 学术、研究、教育
|
||||
"business", # 商业、管理、创业
|
||||
"culture", # 文化、艺术、娱乐
|
||||
"other", # 其他
|
||||
]
|
||||
|
||||
# ─── LLM 调用函数 ──────────────────────────────────────────────
|
||||
def call_llm(prompt: str, system_prompt: str = None, temperature: float = 0.3) -> str:
|
||||
"""
|
||||
调用 oMLX 本地大模型 API
|
||||
|
||||
Args:
|
||||
prompt: 用户提示
|
||||
system_prompt: 系统提示（可选）
|
||||
temperature: 温度参数，越低越确定
|
||||
|
||||
Returns:
|
||||
LLM 回复文本
|
||||
"""
|
||||
import requests
|
||||
|
||||
messages = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
try:
|
||||
# 首先尝试 VLM 模型
|
||||
resp = requests.post(
|
||||
OLLAMA_API_URL,
|
||||
json={
|
||||
"model": OLLAMA_MODEL,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"stream": False
|
||||
},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OLLAMA_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
# 如果 VLM 模型失败（500 错误），尝试备选模型
|
||||
if resp.status_code == 500 and OLLAMA_MODEL == "Qwen2.5-VL-3B-Instruct-8bit":
|
||||
print(f" ⚠️ VLM 模型失败，尝试备选模型 {FALLBACK_MODEL}")
|
||||
resp = requests.post(
|
||||
OLLAMA_API_URL,
|
||||
json={
|
||||
"model": FALLBACK_MODEL,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"stream": False
|
||||
},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OLLAMA_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data["choices"][0]["message"]["content"]
|
||||
except Exception as e:
|
||||
print(f"[WARN] LLM 调用失败: {e}")
|
||||
return ""
|
||||
|
||||
# ─── 分类函数 ──────────────────────────────────────────────────
|
||||
def classify_with_llm(text: str) -> Dict:
|
||||
"""
|
||||
使用 LLM 对文本内容进行分类
|
||||
|
||||
Args:
|
||||
text: 要分类的文本内容（Markdown 格式）
|
||||
|
||||
Returns:
|
||||
{
|
||||
"nature": "fact/method/concept/...",
|
||||
"domain": "technology/finance/...",
|
||||
"confidence": 0.0-1.0,
|
||||
"title": "建议的标题",
|
||||
"summary": "简短摘要",
|
||||
"tags": ["标签1", "标签2", ...]
|
||||
}
|
||||
"""
|
||||
# 限制文本长度（防止 token 过长）
|
||||
preview = text[:2000] if len(text) > 2000 else text
|
||||
|
||||
system_prompt = """你是一个知识分类专家。请根据提供的文本内容进行以下分析：
|
||||
1. 判断知识类型（nature）：从以下选项中选择最合适的：fact（事实数据）、method（方法步骤）、concept（概念理论）、opinion（观点看法）、reference（参考文献）、quote（引用摘录）、axiom（公理定律）、regulation（法规政策）、data（原始数据）
|
||||
2. 判断所属领域（domain）：从以下选项中选择1-2个：technology（技术）、finance（金融）、product（产品）、life（生活）、academic（学术）、business（商业）、culture（文化）、other（其他）
|
||||
3. 提取关键信息：建议一个简洁的标题、生成50字内的摘要、提取2-5个关键词标签
|
||||
|
||||
请以 JSON 格式返回结果，严格遵循以下结构：
|
||||
{
|
||||
"nature": "选择的类型",
|
||||
"domain": ["领域1", "领域2"],
|
||||
"confidence": 0.0-1.0之间的数字（表示分类信心度），
|
||||
"title": "建议标题",
|
||||
"summary": "简短摘要",
|
||||
"tags": ["标签1", "标签2", "标签3"]
|
||||
}"""
|
||||
|
||||
user_prompt = f"""请对以下文本内容进行分类分析：
|
||||
|
||||
{preview}
|
||||
|
||||
注意：文本可能是截断的，请基于现有内容进行判断。"""
|
||||
|
||||
response = call_llm(user_prompt, system_prompt, temperature=0.1)
|
||||
|
||||
# 解析 JSON 响应
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.5,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
if not response:
|
||||
return result
|
||||
|
||||
try:
|
||||
# 查找 JSON 部分
|
||||
json_match = re.search(r'\{.*\}', response, re.DOTALL)
|
||||
if json_match:
|
||||
parsed = json.loads(json_match.group())
|
||||
if isinstance(parsed, dict):
|
||||
if "nature" in parsed and parsed["nature"] in NATURE_OPTIONS:
|
||||
result["nature"] = parsed["nature"]
|
||||
if "domain" in parsed:
|
||||
if isinstance(parsed["domain"], str):
|
||||
result["domain"] = [parsed["domain"]]
|
||||
elif isinstance(parsed["domain"], list):
|
||||
result["domain"] = [d for d in parsed["domain"] if d in DOMAIN_OPTIONS]
|
||||
if "confidence" in parsed:
|
||||
try:
|
||||
conf = float(parsed["confidence"])
|
||||
if 0 <= conf <= 1:
|
||||
result["confidence"] = conf
|
||||
except:
|
||||
pass
|
||||
if "title" in parsed:
|
||||
result["title"] = str(parsed["title"])
|
||||
if "summary" in parsed:
|
||||
result["summary"] = str(parsed["summary"])
|
||||
if "tags" in parsed:
|
||||
if isinstance(parsed["tags"], list):
|
||||
result["tags"] = [str(tag) for tag in parsed["tags"]]
|
||||
except Exception as e:
|
||||
print(f"[WARN] LLM 响应解析失败: {e}")
|
||||
|
||||
return result
|
||||
|
||||
# ─── 启发式分类备用方案 ────────────────────────────────────────
|
||||
def classify_heuristic(text: str, fm: dict) -> Dict:
|
||||
"""
|
||||
启发式分类（当 LLM 不可用时使用）
|
||||
|
||||
Args:
|
||||
text: 文本内容
|
||||
fm: 已有的 frontmatter
|
||||
|
||||
Returns:
|
||||
与 classify_with_llm 相同结构的字典
|
||||
"""
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.4,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature"):
|
||||
result["nature"] = fm["nature"]
|
||||
if fm.get("domain"):
|
||||
if isinstance(fm["domain"], str):
|
||||
result["domain"] = [fm["domain"]]
|
||||
elif isinstance(fm["domain"], list):
|
||||
result["domain"] = fm["domain"]
|
||||
|
||||
# 启发式规则
|
||||
lower = text.lower()
|
||||
|
||||
# nature 判断
|
||||
if not fm.get("nature"):
|
||||
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践"]
|
||||
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
|
||||
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
|
||||
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
|
||||
|
||||
if any(kw in lower for kw in method_kw):
|
||||
result["nature"] = "method"
|
||||
elif any(kw in lower for kw in fact_kw):
|
||||
result["nature"] = "fact"
|
||||
elif any(kw in lower for kw in opinion_kw):
|
||||
result["nature"] = "opinion"
|
||||
elif any(kw in lower for kw in reference_kw):
|
||||
result["nature"] = "reference"
|
||||
|
||||
# domain 判断
|
||||
if not fm.get("domain"):
|
||||
tech_kw = ["编程", "代码", "开发", "python", "javascript", "ai", "llm", "机器学习", "技术"]
|
||||
finance_kw = ["投资", "股票", "基金", "finance", "交易", "财经", "金融"]
|
||||
product_kw = ["产品", "设计", "ux", "product", "用户体验"]
|
||||
|
||||
if any(kw in lower for kw in tech_kw):
|
||||
result["domain"] = ["technology"]
|
||||
elif any(kw in lower for kw in finance_kw):
|
||||
result["domain"] = ["finance"]
|
||||
elif any(kw in lower for kw in product_kw):
|
||||
result["domain"] = ["product"]
|
||||
|
||||
# 生成简单标题（使用文件名或第一行）
|
||||
if fm.get("title"):
|
||||
result["title"] = fm["title"]
|
||||
else:
|
||||
first_line = text.split('\n')[0].strip('# ')
|
||||
if len(first_line) < 50:
|
||||
result["title"] = first_line
|
||||
|
||||
# 生成简单摘要（前 100 字符）
|
||||
clean_text = re.sub(r'[#*\-`]', '', text)
|
||||
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
|
||||
|
||||
return result
|
||||
|
||||
# ─── 主分类函数 ──────────────────────────────────────────────────
|
||||
def classify_content_enhanced(text: str, fm: dict = None, use_llm: bool = True) -> Dict:
|
||||
"""
|
||||
增强版分类函数：优先使用 LLM，失败时降级到启发式
|
||||
|
||||
Args:
|
||||
text: 文本内容
|
||||
fm: 已有的 frontmatter（可选）
|
||||
use_llm: 是否尝试使用 LLM（默认 True）
|
||||
|
||||
Returns:
|
||||
分类结果字典
|
||||
"""
|
||||
if fm is None:
|
||||
fm = {}
|
||||
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.5,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": [],
|
||||
"source_type": "unknown",
|
||||
"source": ""
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature"):
|
||||
result["nature"] = fm["nature"]
|
||||
result["confidence"] = max(result["confidence"], 0.7) # 已有信息提升信心
|
||||
|
||||
if fm.get("domain"):
|
||||
if isinstance(fm["domain"], str):
|
||||
result["domain"] = [fm["domain"]]
|
||||
elif isinstance(fm["domain"], list):
|
||||
result["domain"] = fm["domain"]
|
||||
|
||||
# 尝试使用 LLM 分类
|
||||
llm_result = None
|
||||
if use_llm and text.strip():
|
||||
try:
|
||||
print(" 🤖 使用 LLM 进行智能分类...")
|
||||
llm_result = classify_with_llm(text)
|
||||
|
||||
# 合并 LLM 结果（降低阈值要求，因为 VLM 模型可能返回较低的 confidence）
|
||||
confidence_threshold = 0.4 # 降低阈值，因为 VLM 模型可能返回较低的 confidence
|
||||
if llm_result.get("confidence", 0) > confidence_threshold:
|
||||
if not fm.get("nature") and llm_result.get("nature"):
|
||||
result["nature"] = llm_result["nature"]
|
||||
if not fm.get("domain") and llm_result.get("domain"):
|
||||
result["domain"] = llm_result["domain"]
|
||||
if llm_result.get("title"):
|
||||
result["title"] = llm_result["title"]
|
||||
if llm_result.get("summary"):
|
||||
result["summary"] = llm_result["summary"]
|
||||
if llm_result.get("tags"):
|
||||
result["tags"] = llm_result["tags"]
|
||||
result["confidence"] = llm_result.get("confidence", 0.6)
|
||||
print(f" ✅ LLM 分类成功 (confidence: {llm_result.get('confidence', 0.6):.2f})")
|
||||
else:
|
||||
print(f" ⚠️ LLM 分类信心度低 ({llm_result.get('confidence', 0):.2f} < {confidence_threshold})，使用启发式规则")
|
||||
llm_result = None
|
||||
except Exception as e:
|
||||
print(f" ⚠️ LLM 分类失败: {e}")
|
||||
llm_result = None
|
||||
|
||||
# LLM 失败或信心不足，使用启发式规则
|
||||
if llm_result is None:
|
||||
heuristic_result = classify_heuristic(text, fm)
|
||||
|
||||
# 合并启发式结果
|
||||
if not fm.get("nature") and heuristic_result.get("nature"):
|
||||
result["nature"] = heuristic_result["nature"]
|
||||
if not fm.get("domain") and heuristic_result.get("domain"):
|
||||
result["domain"] = heuristic_result["domain"]
|
||||
if not result["title"] and heuristic_result.get("title"):
|
||||
result["title"] = heuristic_result["title"]
|
||||
if not result["summary"] and heuristic_result.get("summary"):
|
||||
result["summary"] = heuristic_result["summary"]
|
||||
|
||||
# 来源信息
|
||||
if fm.get("source"):
|
||||
result["source"] = fm["source"]
|
||||
if fm.get("source_type"):
|
||||
result["source_type"] = fm["source_type"]
|
||||
|
||||
# 防止空列表
|
||||
if not result["domain"]:
|
||||
result["domain"] = ["other"]
|
||||
if not result["tags"]:
|
||||
result["tags"] = ["待补充"]
|
||||
|
||||
return result
|
||||
|
||||
# ─── 测试函数 ──────────────────────────────────────────────────
|
||||
if __name__ == "__main__":
|
||||
# 测试代码
|
||||
test_text = """Python 中的装饰器是一种强大的工具，它允许在不修改原函数代码的情况下，为函数添加额外的功能。
|
||||
|
||||
装饰器的基本语法：
|
||||
@decorator
|
||||
def my_function():
|
||||
pass
|
||||
|
||||
常见的装饰器应用场景包括：日志记录、性能测试、权限验证等。"""
|
||||
|
||||
print("测试 LLM 分类：")
|
||||
result = classify_content_enhanced(test_text, use_llm=True)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
print("\n测试启发式分类（关闭 LLM）：")
|
||||
result2 = classify_content_enhanced(test_text, use_llm=False)
|
||||
print(json.dumps(result2, ensure_ascii=False, indent=2))
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "inbox_refine",
|
||||
"version": "1.1.0",
|
||||
"description": "知识库收件箱提纯——每日将 00_inbox/ 内容分类归档到知识库，更新首页统计和变更日志",
|
||||
"created": "2026-04-25",
|
||||
"last_modified": "2026-04-25",
|
||||
"dependencies": ["file_read", "file_write", "python3"],
|
||||
"schedule": "daily 03:00",
|
||||
"paths": {
|
||||
"inbox": "~/workbuddy-agent-os/agent-sync/03_knowledge/00_inbox/",
|
||||
"knowledge_root": "~/workbuddy-agent-os/agent-sync/03_knowledge/",
|
||||
"homepage": "~/workbuddy-agent-os/agent-sync/03_knowledge/README.md",
|
||||
"changelog": "~/workbuddy-agent-os/agent-sync/03_knowledge/CHANGELOG.md",
|
||||
"templates": "~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/templates/",
|
||||
"taxonomies": "~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
name: kb_manager
|
||||
version: 1.1.0
|
||||
description: 知识库管理技能——新增知识入库、清洗、标签分类、属性变更移动、搜索、备份
|
||||
triggers:
|
||||
- 知识库管理
|
||||
- 入库
|
||||
- 保存知识
|
||||
- 知识入库
|
||||
- 知识分类
|
||||
- 查知识库
|
||||
- 知识检索
|
||||
- 知识备份
|
||||
- 知识属性变更
|
||||
- 知识重分类
|
||||
- ingest
|
||||
- kb
|
||||
---
|
||||
|
||||
# KB Manager 技能
|
||||
|
||||
## 概述
|
||||
|
||||
管理 AgentOS 的 Obsidian 知识库（03_knowledge/），负责知识入库、清洗、分类、属性变更移动、搜索和备份。
|
||||
|
||||
## 核心能力
|
||||
|
||||
### 1. 知识入库（kb_ingest）
|
||||
- 输入：网页 URL 或纯文本
|
||||
- 流程：
|
||||
1. 抓取/接收原始内容
|
||||
2. 清洗（去广告、去导航、提取正文）
|
||||
3. 写入 `03_knowledge/00_inbox/` 作为待提纯内容
|
||||
4. 由 `collect_to_inbox` 每日 2:30 提取各目录内容转入收件箱，再由 `inbox_refine` 技能每日凌晨 3:00 自动提纯归档
|
||||
5. 或手动说"提纯"立即执行
|
||||
|
||||
### 2. 知识重分类（reclassify）
|
||||
- 当知识的 nature 属性变更时（如 opinion → fact）：
|
||||
1. 更新 Frontmatter 中 nature 和 status
|
||||
2. 将文件从旧目录移动到新目录
|
||||
3. 自动更新 Obsidian 内链
|
||||
4. 记录到 04_memory/logs/kb_reclassify.log
|
||||
|
||||
### 3. 知识检索（kb_query）
|
||||
- 检索流程：
|
||||
1. 先在 L2 摘要层检索（SQLite 索引）
|
||||
2. 命中后按需加载知识文件指定行号片段
|
||||
3. 不加载整个文件，节省 token
|
||||
|
||||
### 4. 知识备份（kb_backup）
|
||||
- 全量备份 03_knowledge/ 目录到指定位置
|
||||
- 支持增量备份（仅备份变更文件）
|
||||
|
||||
## 入库规范
|
||||
|
||||
### Frontmatter 必填字段
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: KB-YYYYMMDD-NNN
|
||||
title: "知识标题"
|
||||
type: concept|method|fact|reference|resource|opinion
|
||||
status: draft|review|published|archived
|
||||
nature: fact|opinion|method|regulation|reference|data|quote|axiom
|
||||
domain: [领域1, 领域2]
|
||||
confidence: 0.0-1.0
|
||||
source: "原始来源 URL"
|
||||
date_created: YYYY-MM-DD
|
||||
date_modified: YYYY-MM-DD
|
||||
version: 1
|
||||
---
|
||||
```
|
||||
|
||||
### 可选字段
|
||||
|
||||
```yaml
|
||||
subdomain: [子领域]
|
||||
tags: [标签列表]
|
||||
author: 作者
|
||||
source_type: personal_exp|literature|webpage|conversation|code|observation
|
||||
related: [KB-ID 列表]
|
||||
previous_version: KB-ID
|
||||
superseded_by: KB-ID
|
||||
aliases: [别称]
|
||||
keywords: [关键词]
|
||||
summary: 一句话摘要
|
||||
```
|
||||
|
||||
## 预定义领域列表
|
||||
|
||||
计算机科学, 人工智能, 金融, 法律, 医学, 物理, 数学, 心理学, 哲学, 历史, 工程, 设计, 商业, 个人管理, 个人洞见, 其他
|
||||
|
||||
> 当"其他"类别超过 5% 时，memory_manager 技能会提示新增领域
|
||||
|
||||
## 文件结构
|
||||
|
||||
```
|
||||
kb_manager/
|
||||
├── SKILL.md
|
||||
├── kb_ingest.py # 知识入库脚本
|
||||
├── kb_query.py # 知识检索脚本
|
||||
├── kb_backup.py # 知识备份脚本
|
||||
├── kb_reclassify.py # 知识重分类脚本
|
||||
└── version.json
|
||||
```
|
||||
|
||||
## 依赖
|
||||
|
||||
| 包 | 用途 | 安装状态 |
|
||||
|---|---|---|
|
||||
| trafilatura | 网页内容提取 | ✅ 已安装 |
|
||||
| chromadb | 向量存储（备选语义检索） | ✅ 已安装 |
|
||||
| sqlite-utils | 数据库操作 | ✅ 已安装 |
|
||||
|
||||
## 自动分类标准
|
||||
|
||||
- nature 字段：fact / opinion / method / regulation / reference / data / quote / axiom
|
||||
- domain 字段：从预定义领域列表选择 1-3 个
|
||||
- confidence 评分：官方文档 0.8+、个人博客 0.4-0.6、推测 < 0.4
|
||||
@@ -0,0 +1,56 @@
|
||||
skill:
|
||||
name: "知识库管理器"
|
||||
id: "kb_manager"
|
||||
version: "1.1.0"
|
||||
category: "core-system"
|
||||
status: "active"
|
||||
created: "2026-04-25"
|
||||
updated: "2026-04-29"
|
||||
|
||||
description:
|
||||
short: "知识入库、分类归档、frontmatter生成、双轨向量检索"
|
||||
use_cases:
|
||||
- "将inbox各分区内容分类归档到知识库"
|
||||
- "从URL/文件/文本生成标准知识卡片"
|
||||
- "双轨向量检索（本地+协同）"
|
||||
tags: ["知识库", "入库", "分类", "检索"]
|
||||
|
||||
ownership:
|
||||
master_editor: "my-macbook_macOS14.4_16GB_master"
|
||||
readonly_on_other_devices: true
|
||||
|
||||
environment:
|
||||
os: "macos"
|
||||
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
|
||||
pip_dependencies:
|
||||
- "trafilatura>=1.6.0"
|
||||
- "chromadb>=1.0.0"
|
||||
|
||||
setup:
|
||||
install_script: ""
|
||||
post_check: "python3 -c 'import trafilatura; print(\"OK\")'"
|
||||
|
||||
usage:
|
||||
entry_point: "kb_ingest.py"
|
||||
scheduled: ""
|
||||
manual_trigger: "知识入库 --url <url>"
|
||||
|
||||
scripts:
|
||||
- kb_ingest.py # 知识入库主程序
|
||||
- kb_search.py # 双轨向量检索
|
||||
|
||||
inbox_paths:
|
||||
knowledge_clipping: "03_knowledge/00_stream/inbox/knowledge/clipping/"
|
||||
knowledge_feed: "03_knowledge/00_stream/inbox/knowledge/feed/"
|
||||
memory: "03_knowledge/00_stream/inbox/memory/"
|
||||
tools: "03_knowledge/00_stream/inbox/tools/"
|
||||
personal: "03_knowledge/00_stream/inbox/personal/"
|
||||
media: "03_knowledge/00_stream/inbox/media/"
|
||||
|
||||
changelog:
|
||||
- version: "1.1.0"
|
||||
date: "2026-04-29"
|
||||
changes: "适配新inbox五分区+clipping/feed子分区，新增kb_search.py双轨检索"
|
||||
- version: "1.0.0"
|
||||
date: "2026-04-25"
|
||||
changes: "初始版本"
|
||||
@@ -0,0 +1,287 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AgentOS 知识入库脚本
|
||||
用途：将网页内容或纯文本清洗、分类、生成知识卡片并写入 Obsidian 知识库
|
||||
使用：python3 kb_ingest.py --root ~/workbuddy-agent-os/agent-sync --url https://example.com
|
||||
python3 kb_ingest.py --root ~/workbuddy-agent-os/agent-sync --file input.md
|
||||
python3 kb_ingest.py --root ~/workbuddy-agent-os/agent-sync --text "原始文本内容"
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# 预定义领域列表
|
||||
DOMAINS = [
|
||||
"计算机科学", "人工智能", "金融", "法律", "医学", "物理", "数学",
|
||||
"心理学", "哲学", "历史", "工程", "设计", "商业", "个人管理",
|
||||
"个人洞见", "其他"
|
||||
]
|
||||
|
||||
# 知识属性类型
|
||||
NATURE_TYPES = ["fact", "opinion", "method", "regulation", "reference", "data", "quote", "axiom"]
|
||||
|
||||
# nature → 目录映射
|
||||
NATURE_DIR_MAP = {
|
||||
"concept": "10_concepts",
|
||||
"method": "20_methods",
|
||||
"fact": "30_facts",
|
||||
"reference": "40_references/docs",
|
||||
"resource": "50_resources",
|
||||
"opinion": "60_opinions",
|
||||
"data": "30_facts",
|
||||
"regulation": "30_facts",
|
||||
"quote": "60_opinions",
|
||||
"axiom": "10_concepts",
|
||||
}
|
||||
|
||||
# 领域英文 → 中文映射
|
||||
DOMAIN_DIR_MAP = {
|
||||
"计算机科学": "cs",
|
||||
"人工智能": "ai",
|
||||
"金融": "finance",
|
||||
"法律": "law",
|
||||
"医学": "medicine",
|
||||
"物理": "physics",
|
||||
"数学": "math",
|
||||
"心理学": "psychology",
|
||||
"哲学": "philosophy",
|
||||
"历史": "history",
|
||||
"工程": "engineering",
|
||||
"设计": "design",
|
||||
"商业": "business",
|
||||
"个人管理": "personal-management",
|
||||
"个人洞见": "personal-insight",
|
||||
"其他": "other",
|
||||
}
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 知识入库")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
parser.add_argument("--url", default=None, help="网页 URL")
|
||||
parser.add_argument("--file", default=None, help="输入文件路径")
|
||||
parser.add_argument("--text", default=None, help="直接输入文本")
|
||||
parser.add_argument("--title", default=None, help="知识标题")
|
||||
parser.add_argument("--domain", default=None, help="领域（逗号分隔）")
|
||||
parser.add_argument("--nature", default=None, help="知识属性")
|
||||
parser.add_argument("--tags", default=None, help="标签（逗号分隔）")
|
||||
parser.add_argument("--inbox", action="store_true", help="仅放入收件箱，不分类")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def fetch_url_content(url: str) -> str:
|
||||
"""抓取网页内容"""
|
||||
try:
|
||||
import trafilatura
|
||||
downloaded = trafilatura.fetch_url(url)
|
||||
if downloaded:
|
||||
content = trafilatura.extract(downloaded)
|
||||
return content or ""
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# fallback: 用 urllib
|
||||
import urllib.request
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
html = resp.read().decode("utf-8", errors="ignore")
|
||||
|
||||
# 简单提取正文（去除 HTML 标签）
|
||||
text = re.sub(r'<[^>]+>', ' ', html)
|
||||
text = re.sub(r'\s+', ' ', text).strip()
|
||||
return text[:10000] # 限制长度
|
||||
|
||||
|
||||
def classify_content(content: str, title: str = "") -> dict:
|
||||
"""
|
||||
分类知识内容（简化版，实际应调用 LLM）
|
||||
返回: {domain, nature, confidence, tags, summary}
|
||||
"""
|
||||
# 简单规则分类
|
||||
domain = "其他"
|
||||
nature = "fact"
|
||||
confidence = 0.5
|
||||
tags = []
|
||||
summary = content[:60]
|
||||
|
||||
# 关键词匹配领域
|
||||
domain_keywords = {
|
||||
"计算机科学": ["代码", "编程", "程序", "数据库", "服务器", "API", "Python", "JavaScript", "算法"],
|
||||
"人工智能": ["AI", "模型", "训练", "深度学习", "LLM", "GPT", "embedding", "向量"],
|
||||
"金融": ["股票", "基金", "投资", "利率", "市场", "财报", "估值"],
|
||||
"个人洞见": ["我觉得", "我认为", "感悟", "心得", "人生", "经历"],
|
||||
}
|
||||
|
||||
for d, keywords in domain_keywords.items():
|
||||
for kw in keywords:
|
||||
if kw.lower() in content.lower():
|
||||
domain = d
|
||||
break
|
||||
if domain != "其他":
|
||||
break
|
||||
|
||||
# 关键词匹配 nature
|
||||
if any(kw in content for kw in ["应该", "步骤", "方法", "如何"]):
|
||||
nature = "method"
|
||||
elif any(kw in content for kw in ["我认为", "我觉得", "可能", "也许"]):
|
||||
nature = "opinion"
|
||||
elif any(kw in content for kw in ["规定", "法规", "制度", "条例"]):
|
||||
nature = "regulation"
|
||||
|
||||
return {
|
||||
"domain": domain,
|
||||
"nature": nature,
|
||||
"confidence": confidence,
|
||||
"tags": tags,
|
||||
"summary": summary
|
||||
}
|
||||
|
||||
|
||||
def generate_kb_id(date: str, index: int = 1) -> str:
|
||||
"""生成知识 ID"""
|
||||
return f"KB-{date.replace('-', '')}-{index:03d}"
|
||||
|
||||
|
||||
def generate_frontmatter(kb_id: str, title: str, classification: dict, source: str = "") -> str:
|
||||
"""生成 Frontmatter"""
|
||||
domain = classification.get("domain", "其他")
|
||||
if isinstance(domain, str):
|
||||
domain = [domain]
|
||||
|
||||
fm = f"""---
|
||||
id: {kb_id}
|
||||
title: "{title}"
|
||||
type: {classification.get('nature', 'concept')}
|
||||
status: draft
|
||||
nature: {classification.get('nature', 'fact')}
|
||||
domain: {json.dumps(domain, ensure_ascii=False)}
|
||||
confidence: {classification.get('confidence', 0.5)}
|
||||
source: "{source}"
|
||||
date_created: {datetime.now().strftime('%Y-%m-%d')}
|
||||
date_modified: {datetime.now().strftime('%Y-%m-%d')}
|
||||
version: 1
|
||||
summary: "{classification.get('summary', '')[:60]}"
|
||||
---"""
|
||||
return fm
|
||||
|
||||
|
||||
def determine_target_dir(root: str, nature: str, domain: str) -> Path:
|
||||
"""确定知识卡片的目标目录"""
|
||||
base_dir = NATURE_DIR_MAP.get(nature, "10_concepts")
|
||||
domain_dir = DOMAIN_DIR_MAP.get(domain, "other")
|
||||
|
||||
# 概念类知识按领域分子目录
|
||||
if base_dir == "10_concepts":
|
||||
return Path(root) / "03_knowledge" / base_dir / domain_dir
|
||||
else:
|
||||
return Path(root) / "03_knowledge" / base_dir
|
||||
|
||||
|
||||
def write_knowledge_card(root: str, target_dir: Path, filename: str,
|
||||
frontmatter: str, content: str):
|
||||
"""写入知识卡片文件"""
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
file_path = target_dir / filename
|
||||
|
||||
full_content = frontmatter + "\n\n" + content
|
||||
file_path.write_text(full_content, encoding="utf-8")
|
||||
return file_path
|
||||
|
||||
|
||||
def log_ingest(root: str, kb_id: str, decision: str, reason: str, confidence: float):
|
||||
"""记录入库日志"""
|
||||
log_dir = Path(root) / "04_memory" / "logs"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_file = log_dir / "kb_ingest.log"
|
||||
|
||||
timestamp = datetime.now().isoformat()
|
||||
with open(log_file, "a", encoding="utf-8") as f:
|
||||
f.write(f"{timestamp} | {kb_id} | {decision} | {reason} | {confidence}\n")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = os.path.expanduser(args.root)
|
||||
|
||||
# 获取内容
|
||||
content = ""
|
||||
source = ""
|
||||
|
||||
if args.url:
|
||||
print(f"[INFO] 抓取网页: {args.url}")
|
||||
content = fetch_url_content(args.url)
|
||||
source = args.url
|
||||
elif args.file:
|
||||
file_path = Path(args.file)
|
||||
if not file_path.exists():
|
||||
print(f"[ERROR] 文件不存在: {args.file}")
|
||||
return
|
||||
content = file_path.read_text(encoding="utf-8")
|
||||
source = str(file_path)
|
||||
elif args.text:
|
||||
content = args.text
|
||||
source = "direct_input"
|
||||
else:
|
||||
print("[ERROR] 请指定 --url, --file 或 --text")
|
||||
return
|
||||
|
||||
if not content.strip():
|
||||
print("[ERROR] 内容为空")
|
||||
return
|
||||
|
||||
title = args.title or "未命名知识"
|
||||
|
||||
# 分类
|
||||
if args.inbox:
|
||||
# 仅放入收件箱
|
||||
target_dir = Path(root) / "03_knowledge" / "00_inbox"
|
||||
classification = {
|
||||
"domain": args.domain or "其他",
|
||||
"nature": args.nature or "fact",
|
||||
"confidence": 0.5,
|
||||
"tags": [],
|
||||
"summary": content[:60]
|
||||
}
|
||||
else:
|
||||
# 自动分类
|
||||
classification = classify_content(content, title)
|
||||
# 命令行参数覆盖
|
||||
if args.domain:
|
||||
classification["domain"] = args.domain
|
||||
if args.nature:
|
||||
classification["nature"] = args.nature
|
||||
|
||||
# 生成知识卡片
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
kb_id = generate_kb_id(today)
|
||||
frontmatter = generate_frontmatter(kb_id, title, classification, source)
|
||||
|
||||
if args.inbox:
|
||||
target_dir = Path(root) / "03_knowledge" / "00_inbox"
|
||||
else:
|
||||
target_dir = determine_target_dir(root, classification["nature"], classification["domain"])
|
||||
|
||||
# 生成文件名（英文，用时间戳确保唯一）
|
||||
safe_title = re.sub(r'[^\w\u4e00-\u9fff-]', '_', title)[:50]
|
||||
filename = f"{datetime.now().strftime('%Y%m%d_%H%M%S')}_{safe_title}.md"
|
||||
|
||||
# 写入
|
||||
file_path = write_knowledge_card(root, target_dir, filename, frontmatter, content)
|
||||
print(f"[OK] 知识卡片已创建: {file_path}")
|
||||
|
||||
# 记录日志
|
||||
log_ingest(root, kb_id, "created", "自动分类入库", classification["confidence"])
|
||||
|
||||
print(f"[INFO] KB-ID: {kb_id}")
|
||||
print(f"[INFO] 领域: {classification['domain']}")
|
||||
print(f"[INFO] 属性: {classification['nature']}")
|
||||
print(f"[INFO] 置信度: {classification['confidence']}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,293 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
kb_search.py — 双轨知识库检索
|
||||
|
||||
先查本地向量库（agent-local/vector_db/local/），
|
||||
再查协同向量库（agent-local/vector_db/global/），
|
||||
结果合并展示，优先显示本地记忆。
|
||||
|
||||
向量库由 vector_db_rebuild.py 构建，基于 ChromaDB。
|
||||
|
||||
使用：
|
||||
python3 kb_search.py --query "记忆固化机制"
|
||||
python3 kb_search.py --query "Python装饰器" --top 10
|
||||
python3 kb_search.py --query "知识库" --track global # 只查协同库
|
||||
python3 kb_search.py --query "今天的会议" --track local # 只查本地库
|
||||
python3 kb_search.py --query "向量" --keyword # 纯关键词搜索（无需向量库）
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 双轨知识库检索")
|
||||
parser.add_argument("--query", required=True, help="检索查询语句")
|
||||
parser.add_argument("--local", default="~/workbuddy-agent-os/agent-local", help="~/workbuddy-agent-os/agent-local")
|
||||
parser.add_argument("--root", default="~/workbuddy-agent-os/agent-sync", help="agent-os 根目录")
|
||||
parser.add_argument("--top", type=int, default=5, help="返回结果数（默认5）")
|
||||
parser.add_argument("--track", choices=["local", "global", "both"], default="both",
|
||||
help="检索范围: local(本地) | global(协同) | both(默认)")
|
||||
parser.add_argument("--keyword", action="store_true",
|
||||
help="纯关键词搜索（不使用向量库，适合没有重建向量库的环境）")
|
||||
parser.add_argument("--domain", default=None, help="过滤领域（如 ai / finance）")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
# ─── 向量库检索 ───────────────────────────────────────────────────
|
||||
|
||||
def get_embedding_model():
|
||||
"""获取嵌入模型（优先 SentenceTransformers，fallback 到关键词）"""
|
||||
try:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
model = SentenceTransformer("all-MiniLM-L6-v2")
|
||||
return model
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
from chromadb.utils import embedding_functions
|
||||
return embedding_functions.DefaultEmbeddingFunction()
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def search_chroma(db_path: Path, query: str, top_n: int, domain_filter: str = None) -> list:
|
||||
"""在指定 ChromaDB 中搜索"""
|
||||
try:
|
||||
import chromadb
|
||||
except ImportError:
|
||||
print("[WARN] chromadb 未安装，跳过向量检索")
|
||||
return []
|
||||
|
||||
if not db_path.exists():
|
||||
return []
|
||||
|
||||
try:
|
||||
client = chromadb.PersistentClient(path=str(db_path))
|
||||
collections = client.list_collections()
|
||||
if not collections:
|
||||
return []
|
||||
|
||||
results = []
|
||||
for col in collections:
|
||||
collection = client.get_collection(col.name)
|
||||
where = {"domain": domain_filter} if domain_filter else None
|
||||
try:
|
||||
res = collection.query(
|
||||
query_texts=[query],
|
||||
n_results=min(top_n, collection.count()),
|
||||
where=where,
|
||||
include=["documents", "metadatas", "distances"]
|
||||
)
|
||||
except Exception:
|
||||
res = collection.query(
|
||||
query_texts=[query],
|
||||
n_results=min(top_n, collection.count()),
|
||||
include=["documents", "metadatas", "distances"]
|
||||
)
|
||||
|
||||
docs = res.get("documents", [[]])[0]
|
||||
metas = res.get("metadatas", [[]])[0]
|
||||
dists = res.get("distances", [[]])[0]
|
||||
|
||||
for doc, meta, dist in zip(docs, metas, dists):
|
||||
results.append({
|
||||
"content": doc[:300],
|
||||
"meta": meta,
|
||||
"score": round(1 - dist, 4), # 转换为相似度分数
|
||||
"track": "local" if "local" in str(db_path) else "global"
|
||||
})
|
||||
|
||||
return sorted(results, key=lambda x: x["score"], reverse=True)[:top_n]
|
||||
except Exception as e:
|
||||
print(f"[WARN] 向量检索出错: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# ─── 关键词检索（fallback） ───────────────────────────────────────
|
||||
|
||||
def search_keyword_index(index_path: Path, query: str, top_n: int) -> list:
|
||||
"""从 keyword_index.json 关键词索引中搜索"""
|
||||
if not index_path.exists():
|
||||
return []
|
||||
try:
|
||||
with open(index_path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
entries = data.get("entries", [])
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
query_terms = set(re.findall(r'[\u4e00-\u9fff]{2,}|[a-zA-Z]{3,}', query.lower()))
|
||||
scored = []
|
||||
for entry in entries:
|
||||
kws = {k.lower() for k in entry.get("keywords", [])}
|
||||
summary = entry.get("summary", "").lower()
|
||||
# 计算匹配分数
|
||||
kw_hits = len(query_terms & kws)
|
||||
content_hits = sum(1 for t in query_terms if t in summary)
|
||||
score = kw_hits * 2 + content_hits
|
||||
if score > 0:
|
||||
scored.append({
|
||||
"content": entry.get("summary", "")[:300],
|
||||
"meta": {
|
||||
"id": entry.get("fact_id", ""),
|
||||
"domain": entry.get("domain", ""),
|
||||
"nature": entry.get("nature", ""),
|
||||
"date": entry.get("date", "")
|
||||
},
|
||||
"score": score,
|
||||
"track": "local_index"
|
||||
})
|
||||
|
||||
return sorted(scored, key=lambda x: x["score"], reverse=True)[:top_n]
|
||||
|
||||
|
||||
def search_md_files(search_dir: Path, query: str, top_n: int, domain_filter: str = None) -> list:
|
||||
"""直接在 MD 文件中做关键词全文搜索"""
|
||||
if not search_dir.exists():
|
||||
return []
|
||||
|
||||
query_terms = re.findall(r'[\u4e00-\u9fff]{2,}|[a-zA-Z]{3,}', query.lower())
|
||||
if not query_terms:
|
||||
return []
|
||||
|
||||
scored = []
|
||||
for md_file in search_dir.rglob("*.md"):
|
||||
try:
|
||||
content = md_file.read_text(encoding="utf-8", errors="ignore")
|
||||
content_lower = content.lower()
|
||||
hits = sum(content_lower.count(term) for term in query_terms)
|
||||
if hits == 0:
|
||||
continue
|
||||
|
||||
# 解析 frontmatter 获取元数据
|
||||
meta = {}
|
||||
if content.startswith("---"):
|
||||
end = content.find("---", 3)
|
||||
if end != -1:
|
||||
for line in content[3:end].split("\n"):
|
||||
if ":" in line:
|
||||
k, _, v = line.partition(":")
|
||||
meta[k.strip()] = v.strip().strip('"')
|
||||
|
||||
if domain_filter and meta.get("domain") and domain_filter not in meta.get("domain", ""):
|
||||
continue
|
||||
|
||||
# 提取匹配片段
|
||||
snippet = ""
|
||||
for term in query_terms:
|
||||
idx = content_lower.find(term)
|
||||
if idx >= 0:
|
||||
start = max(0, idx - 60)
|
||||
end_idx = min(len(content), idx + 200)
|
||||
snippet = "..." + content[start:end_idx].replace("\n", " ") + "..."
|
||||
break
|
||||
|
||||
scored.append({
|
||||
"content": snippet or content[:300],
|
||||
"meta": {
|
||||
"id": meta.get("id", ""),
|
||||
"title": meta.get("title", md_file.stem),
|
||||
"domain": meta.get("domain", ""),
|
||||
"nature": meta.get("nature", ""),
|
||||
"file": str(md_file),
|
||||
},
|
||||
"score": hits,
|
||||
"track": "global_md"
|
||||
})
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return sorted(scored, key=lambda x: x["score"], reverse=True)[:top_n]
|
||||
|
||||
|
||||
# ─── 结果展示 ─────────────────────────────────────────────────────
|
||||
|
||||
def format_result(idx: int, result: dict) -> str:
|
||||
meta = result.get("meta", {})
|
||||
track = result.get("track", "?")
|
||||
score = result.get("score", 0)
|
||||
title = meta.get("title", meta.get("id", "未知"))
|
||||
domain = meta.get("domain", "?")
|
||||
nature = meta.get("nature", "?")
|
||||
content = result.get("content", "")[:200].replace("\n", " ")
|
||||
|
||||
track_icon = {"local": "🔒", "global": "🌐", "local_index": "📇", "global_md": "📄"}.get(track, "❓")
|
||||
|
||||
return (f"\n{idx}. {track_icon}[{track}] {title}\n"
|
||||
f" 领域:{domain} 类型:{nature} 相关度:{score}\n"
|
||||
f" {content}")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
local_root = Path(args.local).expanduser()
|
||||
root = Path(args.root).expanduser()
|
||||
query = args.query
|
||||
top_n = args.top
|
||||
|
||||
print(f"\n🔍 检索: '{query}' (范围:{args.track}, top:{top_n})")
|
||||
print("=" * 60)
|
||||
|
||||
all_results = []
|
||||
|
||||
if args.keyword:
|
||||
# 纯关键词模式
|
||||
if args.track in ("local", "both"):
|
||||
idx_path = local_root / "vector_db" / "local" / "keyword_index.json"
|
||||
results = search_keyword_index(idx_path, query, top_n)
|
||||
all_results.extend(results)
|
||||
|
||||
if args.track in ("global", "both"):
|
||||
kb_dir = root / "03_knowledge"
|
||||
results = search_md_files(kb_dir, query, top_n, args.domain)
|
||||
all_results.extend(results)
|
||||
|
||||
else:
|
||||
# 向量检索模式
|
||||
if args.track in ("local", "both"):
|
||||
local_db = local_root / "vector_db" / "local" / "chroma"
|
||||
results = search_chroma(local_db, query, top_n, args.domain)
|
||||
if not results:
|
||||
# fallback 到关键词索引
|
||||
idx_path = local_root / "vector_db" / "local" / "keyword_index.json"
|
||||
results = search_keyword_index(idx_path, query, top_n)
|
||||
all_results.extend(results)
|
||||
|
||||
if args.track in ("global", "both"):
|
||||
global_db = local_root / "vector_db" / "global" / "chroma"
|
||||
results = search_chroma(global_db, query, top_n, args.domain)
|
||||
if not results:
|
||||
# fallback 到 MD 全文搜索
|
||||
results = search_md_files(root / "03_knowledge", query, top_n, args.domain)
|
||||
all_results.extend(results)
|
||||
|
||||
# 去重 + 合并排序（local优先，同分数local排前）
|
||||
seen = set()
|
||||
deduped = []
|
||||
for r in sorted(all_results, key=lambda x: (-x["score"], 0 if "local" in x["track"] else 1)):
|
||||
key = r["meta"].get("id") or r["meta"].get("file") or r["content"][:50]
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
deduped.append(r)
|
||||
|
||||
if not deduped:
|
||||
print("\n未找到相关结果。")
|
||||
print("提示: 运行 vector_db_rebuild.py 重建向量库后检索效果更好")
|
||||
return
|
||||
|
||||
for i, result in enumerate(deduped[:top_n], 1):
|
||||
print(format_result(i, result))
|
||||
|
||||
print(f"\n共找到 {len(deduped)} 条结果（显示前{min(top_n, len(deduped))}条）")
|
||||
print("🔒=本地记忆 🌐=协同知识库 📇=关键词索引 📄=MD全文")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,292 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
vector_db_rebuild.py — 双轨向量库重建工具
|
||||
|
||||
重建两套向量库：
|
||||
- local：agent-os-local/ 下的私有内容（记忆、本机笔记）
|
||||
- global：agent-sync/03_knowledge/ 下的协同知识库
|
||||
|
||||
向量库存储在 agent-os-local/vector_db/ 下（永不同步）：
|
||||
agent-os-local/vector_db/local/chroma/ ← 私有内容
|
||||
agent-os-local/vector_db/global/chroma/ ← 协同知识（每机自行重建）
|
||||
|
||||
使用：
|
||||
python3 vector_db_rebuild.py --local ~/workbuddy-agent-os/agent-local --root ~/workbuddy-agent-os/agent-sync
|
||||
python3 vector_db_rebuild.py --local ~/workbuddy-agent-os/agent-local --root ~/workbuddy-agent-os/agent-sync --track local
|
||||
python3 vector_db_rebuild.py --local ~/workbuddy-agent-os/agent-local --root ~/workbuddy-agent-os/agent-sync --track global
|
||||
python3 vector_db_rebuild.py --local ~/workbuddy-agent-os/agent-local --root ~/workbuddy-agent-os/agent-sync --incremental
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 双轨向量库重建")
|
||||
parser.add_argument("--root", default="~/workbuddy-agent-os/agent-sync", help="agent-os 根目录")
|
||||
parser.add_argument("--local", default="~/workbuddy-agent-os/agent-local", help="~/workbuddy-agent-os/agent-local")
|
||||
parser.add_argument("--track", choices=["local", "global", "both"], default="both",
|
||||
help="重建哪套向量库 (默认 both)")
|
||||
parser.add_argument("--incremental", action="store_true",
|
||||
help="增量模式：只添加新文件，不清空已有向量库")
|
||||
parser.add_argument("--dry-run", action="store_true",
|
||||
help="预览模式：只列出待处理文件，不写入向量库")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
# ─── 文件扫描 ─────────────────────────────────────────────────────
|
||||
|
||||
def scan_md_files(dirs: list[Path], exclude_patterns: list = None) -> list[dict]:
|
||||
"""扫描 MD 文件，返回文档列表"""
|
||||
exclude_patterns = exclude_patterns or []
|
||||
docs = []
|
||||
for base_dir in dirs:
|
||||
if not base_dir.exists():
|
||||
continue
|
||||
for md_file in sorted(base_dir.rglob("*.md")):
|
||||
# 跳过隐藏文件和排除目录
|
||||
if any(p in str(md_file) for p in exclude_patterns):
|
||||
continue
|
||||
if md_file.name.startswith("."):
|
||||
continue
|
||||
try:
|
||||
content = md_file.read_text(encoding="utf-8", errors="ignore")
|
||||
if len(content.strip()) < 50:
|
||||
continue
|
||||
|
||||
meta = extract_frontmatter(content)
|
||||
body = strip_frontmatter(content)
|
||||
|
||||
docs.append({
|
||||
"id": meta.get("id") or f"FILE-{md_file.stat().st_mtime_ns}",
|
||||
"file": str(md_file),
|
||||
"title": meta.get("title", md_file.stem),
|
||||
"content": body[:2000], # 限制长度
|
||||
"metadata": {
|
||||
"id": meta.get("id", ""),
|
||||
"title": meta.get("title", md_file.stem),
|
||||
"nature": meta.get("nature", ""),
|
||||
"domain": meta.get("domain", ""),
|
||||
"confidence": str(meta.get("confidence", "")),
|
||||
"source_type": meta.get("source_type", ""),
|
||||
"date_created": meta.get("date_created", ""),
|
||||
"file_path": str(md_file),
|
||||
}
|
||||
})
|
||||
except Exception as e:
|
||||
print(f" [SKIP] {md_file.name}: {e}")
|
||||
return docs
|
||||
|
||||
|
||||
def extract_frontmatter(content: str) -> dict:
|
||||
if not content.startswith("---"):
|
||||
return {}
|
||||
end = content.find("---", 3)
|
||||
if end == -1:
|
||||
return {}
|
||||
meta = {}
|
||||
for line in content[3:end].split("\n"):
|
||||
if ":" in line:
|
||||
k, _, v = line.partition(":")
|
||||
meta[k.strip()] = v.strip().strip('"').strip("'")
|
||||
return meta
|
||||
|
||||
|
||||
def strip_frontmatter(content: str) -> str:
|
||||
if not content.startswith("---"):
|
||||
return content
|
||||
end = content.find("---", 3)
|
||||
return content[end + 3:].strip() if end != -1 else content
|
||||
|
||||
|
||||
# ─── 向量库操作 ───────────────────────────────────────────────────
|
||||
|
||||
def get_chroma_client(db_path: Path):
|
||||
"""获取 ChromaDB 客户端"""
|
||||
try:
|
||||
import chromadb
|
||||
db_path.mkdir(parents=True, exist_ok=True)
|
||||
return chromadb.PersistentClient(path=str(db_path))
|
||||
except ImportError:
|
||||
print("[ERROR] chromadb 未安装。请运行: pip install chromadb")
|
||||
return None
|
||||
except Exception as e:
|
||||
print(f"[ERROR] 初始化 ChromaDB 失败: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def upsert_documents(client, collection_name: str, docs: list, incremental: bool = False):
|
||||
"""向 ChromaDB 写入文档"""
|
||||
collection = client.get_or_create_collection(
|
||||
name=collection_name,
|
||||
metadata={"hnsw:space": "cosine"}
|
||||
)
|
||||
|
||||
if not incremental:
|
||||
# 全量重建：清空后写入
|
||||
existing = collection.get(include=[])
|
||||
if existing["ids"]:
|
||||
collection.delete(ids=existing["ids"])
|
||||
|
||||
# 增量模式：获取已有 ID
|
||||
existing_ids = set()
|
||||
if incremental:
|
||||
existing = collection.get(include=[])
|
||||
existing_ids = set(existing["ids"])
|
||||
|
||||
new_docs = [d for d in docs if d["id"] not in existing_ids]
|
||||
if not new_docs:
|
||||
print(f" [INFO] {collection_name}: 无新文档（共{len(docs)}个，全部已存在）")
|
||||
return 0
|
||||
|
||||
# 批量写入（ChromaDB 建议每批 ≤ 5000）
|
||||
batch_size = 500
|
||||
total_added = 0
|
||||
for i in range(0, len(new_docs), batch_size):
|
||||
batch = new_docs[i:i + batch_size]
|
||||
collection.add(
|
||||
ids=[d["id"] for d in batch],
|
||||
documents=[d["content"] for d in batch],
|
||||
metadatas=[d["metadata"] for d in batch]
|
||||
)
|
||||
total_added += len(batch)
|
||||
|
||||
return total_added
|
||||
|
||||
|
||||
# ─── 关键词索引（轻量fallback） ──────────────────────────────────
|
||||
|
||||
def build_keyword_index(docs: list, index_path: Path):
|
||||
"""构建关键词索引（JSON格式，无需向量库也可检索）"""
|
||||
entries = []
|
||||
for doc in docs:
|
||||
meta = doc["metadata"]
|
||||
content = doc["content"]
|
||||
keywords = list(set(re.findall(r'[\u4e00-\u9fff]{2,}|[a-zA-Z]{4,}', content)))[:20]
|
||||
entries.append({
|
||||
"fact_id": doc["id"],
|
||||
"title": meta.get("title", ""),
|
||||
"keywords": keywords,
|
||||
"domain": meta.get("domain", ""),
|
||||
"nature": meta.get("nature", ""),
|
||||
"summary": content[:150].replace("\n", " "),
|
||||
"date": meta.get("date_created", ""),
|
||||
"source": meta.get("source_type", ""),
|
||||
"file": meta.get("file_path", "")
|
||||
})
|
||||
|
||||
index_data = {
|
||||
"version": "2.0.0",
|
||||
"last_updated": datetime.now().isoformat(),
|
||||
"total": len(entries),
|
||||
"entries": entries
|
||||
}
|
||||
index_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(index_path, "w", encoding="utf-8") as f:
|
||||
json.dump(index_data, f, ensure_ascii=False, indent=2)
|
||||
print(f" [OK] 关键词索引: {len(entries)} 条 → {index_path}")
|
||||
|
||||
|
||||
# ─── 主流程 ──────────────────────────────────────────────────────
|
||||
|
||||
def rebuild_local(local_root: Path, incremental: bool, dry_run: bool):
|
||||
"""重建本地向量库（私有内容）"""
|
||||
print("\n── 重建 local 向量库 ──")
|
||||
local_db_path = local_root / "vector_db" / "local" / "chroma"
|
||||
keyword_index_path = local_root / "vector_db" / "local" / "keyword_index.json"
|
||||
|
||||
# 扫描本机私有内容
|
||||
scan_dirs = [
|
||||
local_root / "memory" / "raw",
|
||||
local_root / "memory" / "long_term",
|
||||
local_root / "materials" / "refined_for_inbox",
|
||||
]
|
||||
docs = scan_md_files(scan_dirs)
|
||||
print(f" 扫描到 {len(docs)} 个文档")
|
||||
|
||||
if dry_run:
|
||||
for d in docs[:5]:
|
||||
print(f" [预览] {d['title']} ({d['metadata'].get('nature', '?')})")
|
||||
return
|
||||
|
||||
# 构建关键词索引（始终构建，不依赖 chromadb）
|
||||
build_keyword_index(docs, keyword_index_path)
|
||||
|
||||
# 构建向量索引（可选，需 chromadb）
|
||||
client = get_chroma_client(local_db_path)
|
||||
if client:
|
||||
added = upsert_documents(client, "local_memory", docs, incremental)
|
||||
print(f" [OK] ChromaDB local: 新增 {added} 条向量")
|
||||
else:
|
||||
print(" [INFO] 跳过 ChromaDB，仅使用关键词索引")
|
||||
|
||||
|
||||
def rebuild_global(root: Path, local_root: Path, incremental: bool, dry_run: bool):
|
||||
"""重建协同向量库（知识库内容）"""
|
||||
print("\n── 重建 global 向量库 ──")
|
||||
global_db_path = local_root / "vector_db" / "global" / "chroma"
|
||||
keyword_index_path = local_root / "vector_db" / "global" / "keyword_index.json"
|
||||
|
||||
# 扫描协同知识库（排除 inbox 等临时目录）
|
||||
kb_root = root / "03_knowledge"
|
||||
scan_dirs = [
|
||||
kb_root / "10_concepts",
|
||||
kb_root / "20_methods",
|
||||
kb_root / "30_facts",
|
||||
kb_root / "40_references",
|
||||
kb_root / "50_resources",
|
||||
kb_root / "60_opinions",
|
||||
kb_root / "01_daily",
|
||||
]
|
||||
exclude = ["00_inbox", "00_stream", "90_archive", "99_system", ".obsidian"]
|
||||
docs = scan_md_files(scan_dirs, exclude_patterns=exclude)
|
||||
print(f" 扫描到 {len(docs)} 个文档")
|
||||
|
||||
if dry_run:
|
||||
for d in docs[:5]:
|
||||
print(f" [预览] {d['title']} ({d['metadata'].get('domain', '?')})")
|
||||
return
|
||||
|
||||
build_keyword_index(docs, keyword_index_path)
|
||||
|
||||
client = get_chroma_client(global_db_path)
|
||||
if client:
|
||||
added = upsert_documents(client, "global_knowledge", docs, incremental)
|
||||
print(f" [OK] ChromaDB global: 新增 {added} 条向量")
|
||||
else:
|
||||
print(" [INFO] 跳过 ChromaDB，仅使用关键词索引")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = Path(args.root).expanduser()
|
||||
local_root = Path(args.local).expanduser()
|
||||
|
||||
print("=" * 60)
|
||||
print("AgentOS 双轨向量库重建")
|
||||
print(f" agent-os: {root}")
|
||||
print(f" agent-os-local: {local_root}")
|
||||
print(f" 重建范围: {args.track}")
|
||||
if args.incremental:
|
||||
print(" [增量模式]")
|
||||
if args.dry_run:
|
||||
print(" [DRY-RUN 模式]")
|
||||
print("=" * 60)
|
||||
|
||||
if args.track in ("local", "both"):
|
||||
rebuild_local(local_root, args.incremental, args.dry_run)
|
||||
|
||||
if args.track in ("global", "both"):
|
||||
rebuild_global(root, local_root, args.incremental, args.dry_run)
|
||||
|
||||
print("\n[完成] 向量库重建完毕")
|
||||
print(" 下次检索时将自动使用新的向量库（kb_search.py）")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"name": "kb_manager",
|
||||
"version": "1.1.0",
|
||||
"description": "知识库管理技能——入库、分类、检索、备份、重分类",
|
||||
"dependencies": ["sqlite3"],
|
||||
"mcp_tools": []
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
---
|
||||
name: matrix
|
||||
version: 1.0.0
|
||||
description: 多平台社交账号矩阵养号——蓝图调度、账号切换、任务状态查询、蓝图管理
|
||||
triggers:
|
||||
- 矩阵
|
||||
- 养号
|
||||
- 抖音养号
|
||||
- 执行蓝图
|
||||
- 切换账号
|
||||
- 账号状态
|
||||
- matrix
|
||||
- 查看蓝图
|
||||
- 添加蓝图
|
||||
- 任务调度
|
||||
---
|
||||
|
||||
# Matrix 矩阵养号技能
|
||||
|
||||
## 概述
|
||||
|
||||
通过 Chrome CDP 和 Camoufox（Firefox 内核）控制浏览器，自动执行多平台社交账号日常养号任务。
|
||||
支持抖音（已完成）、小红书/知乎（框架就绪，待登录账号）。
|
||||
|
||||
**工具位置**: `~/workbuddy-agent-os/agent-sync/05_tools/07_matrix/`
|
||||
**本地数据**: `~/workbuddy-agent-os/agent-local/tools/matrix/`
|
||||
|
||||
---
|
||||
|
||||
## 触发词与对应动作
|
||||
|
||||
| 触发词 | 动作 |
|
||||
|--------|------|
|
||||
| `账号状态` / `查看账号` | 列出所有账号及登录状态 |
|
||||
| `执行蓝图 <蓝图名> <账号>` | 执行指定蓝图 |
|
||||
| `查看蓝图` | 列出所有可用蓝图及说明 |
|
||||
| `切换账号 <账号ID>` | 切换 Chrome Profile 到指定账号 |
|
||||
| `启动调度` | 启动定时任务调度器 |
|
||||
| `任务状态` | 查看最近任务执行记录 |
|
||||
| `matrix 安装` | 在新机运行 install.sh |
|
||||
|
||||
---
|
||||
|
||||
## 核心能力
|
||||
|
||||
### 1. 账号管理
|
||||
- 多账号隔离（每个账号独立 Chrome Profile）
|
||||
- 账号状态查询（登录态检查，Cookie 主方案 + DOM 备选）
|
||||
- 登录检测原子化模块：`auth_manager.py`
|
||||
- `check_login_by_cookie()` — Cookie 检测（主方案，不依赖 UI）
|
||||
- `check_login_by_dom()` — DOM 检测（备选，需移动端视口）
|
||||
- `get_login_status()` — 多维综合检测
|
||||
- `export_cookies()` / `inject_cookies()` — Cookie 导出/注入
|
||||
- `wait_for_login()` — 等待用户手动登录（轮询）
|
||||
- 双方案切换：Profile 切换（稳定）/ Cookie 注入（实验性）
|
||||
- **`switch_account.py` 切换后自动导出 Cookie 到 `data/cookies/{id}_cookies.json`**
|
||||
|
||||
### 2. 蓝图执行
|
||||
|
||||
| 蓝图 | 说明 | 步骤数 | 状态 |
|
||||
|------|------|--------|------|
|
||||
| `douyin_browse_v2` | 推荐页浏览 + 随机点赞/收藏 | 11步 | ✅ 活跃 |
|
||||
| `douyin_search_browse` | 搜索关键词 → 浏览结果 → 点赞 | 7步 | ✅ 活跃 |
|
||||
| `douyin_comment_interact` | 浏览推荐 → 打开评论区 → 发评论 | 8步 | ✅ 活跃 |
|
||||
|
||||
### 3. 原子操作库（抖音，18个）
|
||||
`goto_home` / `wait_watch` / `like` / `collect` / `follow` / `comment` /
|
||||
`search` / `open_comments` / `close_comments` / `scroll_feed` 等
|
||||
|
||||
### 4. 定时调度
|
||||
通过 `task_scheduler.py` 设定各账号的执行频率，后台持续运行。
|
||||
|
||||
---
|
||||
|
||||
## 常用命令
|
||||
|
||||
```bash
|
||||
TOOL=~/workbuddy-agent-os/agent-sync/05_tools/07_matrix
|
||||
|
||||
# 查看所有账号
|
||||
python $TOOL/scripts/switch_account.py --list
|
||||
|
||||
# 查看当前活跃账号（含 Cookie 检测）
|
||||
python $TOOL/scripts/switch_account.py --status
|
||||
|
||||
# 切换到指定账号（Profile 方式）+ 自动导出 Cookie
|
||||
python $TOOL/scripts/switch_account.py --method profile --target douyin_01 --port 9222
|
||||
|
||||
# 独立登录检测（auth_manager 原子化模块）
|
||||
python $TOOL/scripts/auth_manager.py 9222 # 检测端口 9222 的登录状态
|
||||
python $TOOL/scripts/auth_manager.py 9223 # 检测端口 9223
|
||||
|
||||
# 启动 Camoufox + Cookie 注入（正确方式）
|
||||
python $TOOL/scripts/yanghao_runner.py --account douyin_camo01 --blueprint douyin_browse_v2 --browser camoufox
|
||||
|
||||
# 执行蓝图
|
||||
python $TOOL/scripts/task_engine.py --blueprint douyin_browse_v2 --account douyin_01
|
||||
|
||||
# 后台定时调度
|
||||
python $TOOL/scripts/task_scheduler.py
|
||||
|
||||
# 手动启动 Chrome
|
||||
bash $TOOL/scripts/launch_chrome.sh douyin_01 9222
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 执行流程
|
||||
|
||||
```
|
||||
用户触发（对话 or 定时）
|
||||
↓
|
||||
task_scheduler.py（定时调度）
|
||||
↓
|
||||
switch_account.py（切换 Chrome Profile / Cookie 注入）
|
||||
↓
|
||||
task_engine.py（加载蓝图 JSON → 逐步执行）
|
||||
↓
|
||||
douyin_ops.py（原子操作：点赞/评论/浏览...）
|
||||
↓
|
||||
cdp_connector.py / camoufox_manager.py（浏览器 CDP 控制）
|
||||
↓
|
||||
写入 data/matrix.db（任务记录）+ logs/（执行日志）
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 账号配置文件
|
||||
|
||||
位置：`~/workbuddy-agent-os/agent-local/tools/matrix/config/accounts.yaml`
|
||||
|
||||
```yaml
|
||||
accounts:
|
||||
- id: douyin_01
|
||||
platform: douyin
|
||||
browser: chrome
|
||||
profile_dir: douyin_01 # 对应 profiles/ 下的目录名
|
||||
status: active
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 蓝图文件格式（供新增参考）
|
||||
|
||||
位置：`~/workbuddy-agent-os/agent-sync/05_tools/07_matrix/blueprints/`
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "蓝图唯一ID",
|
||||
"description": "蓝图说明",
|
||||
"platform": "douyin",
|
||||
"steps": [
|
||||
{"step": 1, "op": "goto_home", "args": {}},
|
||||
{"step": 2, "op": "wait_watch", "args": {"seconds": 8}},
|
||||
{"step": 3, "op": "like", "args": {"probability": 0.3}}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 标准化文档
|
||||
|
||||
| 文档 | 用途 |
|
||||
|------|------|
|
||||
| `docs/SYSTEM_ARCHITECTURE.md` | **系统架构设计** — 模块关系、链路、原子操作清单（必读） |
|
||||
| `docs/ACCOUNT_LOGIN_SOP.md` | **账号登录 SOP** — 标准化操作流程 |
|
||||
| `docs/FULL_TEST_REPORT.md` | **全面测试报告** — ✅/❌ 全部标注，含时间戳 |
|
||||
|
||||
**所有操作请先查阅架构文档和 SOP，按标准步骤执行，不得临时改代码。**
|
||||
|
||||
---
|
||||
|
||||
## 开发状态
|
||||
|
||||
| 阶段 | 内容 | 状态 |
|
||||
|------|------|------|
|
||||
| A | Chrome CDP + 抖音原子操作（18个）+ 蓝图引擎 + 定时调度 | ✅ 已完成 |
|
||||
| B | Camoufox（Firefox内核）集成 + 多浏览器内核 | ✅ 已完成（2026-04-30） |
|
||||
| C | 鼠标轨迹仿真 + 语料库完善 + Canvas/AudioContext 指纹噪声 | 📋 规划中 |
|
||||
| D | 小红书/知乎完整支持 + 多平台联动 | 📋 规划中 |
|
||||
|
||||
### 阶段B完成说明
|
||||
- Camoufox 通过 `camoufox.async_api.AsyncCamoufox` 原生 API 启动（`CDPConnector(browser_type="camoufox")`）
|
||||
- **重要**：Camoufox 不依赖 CDP 端口，通过 **Cookie 注入** 从 Chrome 导出的 Cookie 文件登录
|
||||
- 标准入口：`yanghao_runner.py --account douyin_camo01 --blueprint douyin_browse_v2 --browser camoufox`
|
||||
- **不要使用** `camoufox_manager.py --launch` 或 `camoufox_server.py --launch`（非标准入口）
|
||||
- 登录态验证通过 Cookie 检测（sessionid），无需 DOM 头像元素
|
||||
- Chrome 方案 + Camoufox 方案均已通过全量测试验证 ✅
|
||||
|
||||
---
|
||||
|
||||
## 多机同步
|
||||
|
||||
详见 `docs/SYNC_GUIDE.md` — 包含:
|
||||
|
||||
- 同步架构说明（什么是同步的 / 什么是每机独立的）
|
||||
- 主电脑修改→提交→推送的标准流程
|
||||
- 新机拉取→部署→登录的标准流程
|
||||
- 本地依赖安装清单（Python 包 / Playwright / Camoufox）
|
||||
- 版本对照表 + 快速检查清单
|
||||
|
||||
### 快速新机恢复
|
||||
|
||||
```bash
|
||||
# 1. 拉取最新代码
|
||||
cd ~/workbuddy-agent-os/agent-sync && git pull origin main
|
||||
|
||||
# 2. 一键安装（建立目录 + 依赖 + 配置）
|
||||
bash ~/workbuddy-agent-os/agent-sync/05_tools/07_matrix/install.sh
|
||||
|
||||
# 3. 填写本机账号配置
|
||||
vim ~/workbuddy-agent-os/agent-local/tools/matrix/config/accounts.yaml
|
||||
|
||||
# 4. 各账号重新扫码登录
|
||||
python scripts/switch_account.py --method profile --target douyin_01 --port 9222
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `profiles/` 含 Chrome 登录数据（~100MB），**换机后需重新登录**各平台
|
||||
- `config/accounts.yaml` 含账号信息，**不随 agent-os 同步**，存于 agent-os-local
|
||||
- Camoufox 阶段B尚未稳定，生产环境使用 Chrome CDP（阶段A）
|
||||
- 反检测措施：指纹注入 + 随机延迟 + 蓝图随机化，勿过度调低延迟
|
||||
@@ -0,0 +1,66 @@
|
||||
skill:
|
||||
name: "矩阵养号"
|
||||
id: "matrix"
|
||||
version: "1.0.0"
|
||||
category: "automation"
|
||||
status: "active"
|
||||
created: "2026-04-29"
|
||||
updated: "2026-04-29"
|
||||
|
||||
description:
|
||||
short: "多平台社交账号矩阵养号——蓝图调度、Chrome CDP 控制、账号切换"
|
||||
use_cases:
|
||||
- "自动执行抖音日常浏览/点赞/收藏/评论蓝图"
|
||||
- "多账号 Chrome Profile 隔离切换"
|
||||
- "定时调度器后台持续运营账号"
|
||||
- "管理蓝图（查看/新增/修改）"
|
||||
tags: ["自动化", "抖音", "养号", "CDP", "蓝图", "多账号"]
|
||||
|
||||
ownership:
|
||||
master_editor: "Redmi-12C_macOS26.4_M1_16GB"
|
||||
readonly_on_other_devices: false
|
||||
|
||||
tool_path: "~/workbuddy-agent-os/agent-sync/05_tools/07_matrix"
|
||||
data_path: "~/workbuddy-agent-os/agent-sync/agent-local/tools/matrix"
|
||||
|
||||
environment:
|
||||
os: "macos"
|
||||
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
|
||||
pip_dependencies:
|
||||
- "patchright>=1.40.0"
|
||||
- "camoufox>=0.4.0"
|
||||
- "PyYAML>=6.0"
|
||||
system_deps:
|
||||
- "Google Chrome（CDP 直连）"
|
||||
- "Camoufox（阶段B，Firefox内核）"
|
||||
|
||||
setup:
|
||||
install_script: "~/workbuddy-agent-os/agent-sync/05_tools/07_matrix/install.sh"
|
||||
post_check: "python3 ~/workbuddy-agent-os/agent-sync/05_tools/07_matrix/scripts/switch_account.py --list"
|
||||
|
||||
usage:
|
||||
entry_points:
|
||||
scheduler: "scripts/task_scheduler.py"
|
||||
engine: "scripts/task_engine.py --blueprint <name> --account <id>"
|
||||
accounts: "scripts/switch_account.py --list"
|
||||
blueprints_dir: "~/workbuddy-agent-os/agent-sync/05_tools/07_matrix/blueprints/"
|
||||
config_file: "~/workbuddy-agent-os/agent-sync/agent-local/tools/matrix/config/accounts.yaml"
|
||||
|
||||
platforms:
|
||||
- name: "抖音"
|
||||
status: "active"
|
||||
browser: "chrome"
|
||||
atomic_ops: 18
|
||||
- name: "小红书"
|
||||
status: "framework_ready"
|
||||
browser: "chrome"
|
||||
- name: "知乎"
|
||||
status: "framework_ready"
|
||||
browser: "chrome"
|
||||
- name: "快手"
|
||||
status: "planned"
|
||||
|
||||
changelog:
|
||||
- version: "1.0.0"
|
||||
date: "2026-04-29"
|
||||
changes: "初始版本——从 ~/Documents/matrix 迁入 agent-os，完成代码/数据分离，新增 SKILL.md/TOOL.md/install.sh"
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"name": "matrix",
|
||||
"version": "1.0.0",
|
||||
"description": "多平台社交账号矩阵养号——蓝图调度、Chrome CDP 控制、账号切换",
|
||||
"dependencies": ["PyYAML", "patchright", "camoufox"],
|
||||
"mcp_tools": [],
|
||||
"tool_path": "~/workbuddy-agent-os/agent-sync/05_tools/07_matrix"
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
name: memory_manager
|
||||
version: 1.2.0
|
||||
description: 智能体记忆管理技能——每日对话提炼、去重、冲突检测、版本管理、记忆导入导出、语义检索
|
||||
triggers:
|
||||
- 记忆管理
|
||||
- 记忆更新
|
||||
- 每日摘要
|
||||
- 每日提炼
|
||||
- 对话提炼
|
||||
- 整理记忆
|
||||
- 记忆去重
|
||||
- 冲突检测
|
||||
- 知识版本
|
||||
- 记忆备份
|
||||
- 记忆导出
|
||||
- 记忆检查
|
||||
- 查记忆
|
||||
- 语义检索
|
||||
- 记忆搜索
|
||||
- 搜索记忆
|
||||
- update memory
|
||||
---
|
||||
|
||||
# Memory Manager 技能
|
||||
|
||||
## 概述
|
||||
|
||||
管理 AgentOS 的 L0→L1→L1_vec→L2→L3 五级记忆体系统，负责每日对话提炼、去重、冲突消解、版本管理、语义检索和记忆导入导出。
|
||||
|
||||
## 核心能力
|
||||
|
||||
### 1. 每日对话提炼（daily_digest）
|
||||
- 触发方式：每日凌晨 2:00 自动执行，或手动触发
|
||||
- 输入：昨日所有对话记录
|
||||
- 处理流程：
|
||||
1. 提取关键事实（who/what/when/where/decision）
|
||||
2. 去重（对比 L2 已有事实，相似度 > 0.9 则跳过）
|
||||
3. 冲突检测（新事实与旧事实矛盾？→ 标记 OVERRIDE 或提示用户确认）
|
||||
4. 生成摘要（100-200 字/条）
|
||||
5. 更新 L1 关键词索引（仅写入 L2 成功的事实，避免孤儿索引）
|
||||
6. 更新 L1_vec 向量索引（ChromaDB + oMLX embedding，新事实自动向量化）
|
||||
7. 关键对话原文 → 压缩后写入 L3
|
||||
- 输出：
|
||||
- 更新 04_memory/daily_summaries/YYYY-MM-DD.md
|
||||
- 更新 04_memory/long_term/facts.db
|
||||
- 更新 04_memory/vector_db/keyword_index.json
|
||||
- 更新 04_memory/vector_db/chroma/（ChromaDB 向量库）
|
||||
- 发送摘要到 Obsidian 01_daily/ 日记
|
||||
|
||||
### 2. 冲突消解（memory_cleanup）
|
||||
- 冲突检测优先级：新事实 > 旧事实，用户明确修正 > 系统推断
|
||||
- 三种变化处理：
|
||||
- 硬事实改变 → 更新原卡片，旧值写入 previous_version
|
||||
- 软升级/延伸 → 保留 V1，新建 V2，通过 related 双向链接
|
||||
- 观点→事实 → 更新 nature 字段，通知 kb_manager 移动文件
|
||||
|
||||
### 3. 记忆导入导出
|
||||
- export_memories.py：将记忆打包为 JSON + Markdown 压缩包
|
||||
- import_memories.py：从压缩包恢复记忆，合并时智能去重
|
||||
|
||||
### 4. 语义检索（semantic_search）
|
||||
- BM25 关键词检索 + 向量语义检索 + RRF 加权融合
|
||||
- 向量引擎：ChromaDB（持久化）+ oMLX Qwen3-Embedding-0.6B（1024维）
|
||||
- 融合策略：Reciprocal Rank Fusion (RRF)，BM25 权重 0.4 / 向量权重 0.6
|
||||
- CLI 用法：
|
||||
- 检索：`python3 semantic_search.py search --root ~/workbuddy-agent-os/agent-sync --query "查询内容" --top-k 5`
|
||||
- 回填：`python3 semantic_search.py backfill --root ~/workbuddy-agent-os/agent-sync`
|
||||
- 重建：`python3 semantic_search.py rebuild --root ~/workbuddy-agent-os/agent-sync`
|
||||
- 测试：`python3 semantic_search.py embed --root ~/workbuddy-agent-os/agent-sync --text "测试"`
|
||||
|
||||
## 文件结构
|
||||
|
||||
```
|
||||
memory_manager/
|
||||
├── SKILL.md # 本文件
|
||||
├── daily_digest.py # 每日对话提炼脚本
|
||||
├── semantic_search.py # BM25 + 向量语义混合检索
|
||||
├── memory_cleanup.py # 冲突消解与过期记忆清理
|
||||
├── agent_memory_init.py # 首次初始化记忆体
|
||||
├── bootstrap_from_memory.py # 冷启动：历史数据全量导入
|
||||
├── export_memories.py # 记忆导出（JSON + L3 原文）
|
||||
├── import_memories.py # 记忆导入（智能去重合并）
|
||||
└── version.json
|
||||
```
|
||||
|
||||
## 使用说明
|
||||
|
||||
> **Python 路径**：所有脚本使用 agent-os 专用虚拟环境
|
||||
> `/Users/chengzige/.workbuddy/binaries/python/envs/agent-os/bin/python3`
|
||||
|
||||
1. 初始化：`python3 agent_memory_init.py --root ~/workbuddy-agent-os/agent-sync`
|
||||
2. 冷启动：`python3 bootstrap_from_memory.py --root ~/workbuddy-agent-os/agent-sync`
|
||||
3. 每日提炼：`python3 daily_digest.py --root ~/workbuddy-agent-os/agent-sync --date YYYY-MM-DD`
|
||||
4. 冲突清理：`python3 memory_cleanup.py --root ~/workbuddy-agent-os/agent-sync`
|
||||
5. 导出记忆：`python3 export_memories.py --root ~/workbuddy-agent-os/agent-sync --output ~/backup/`
|
||||
6. 导入记忆：`python3 import_memories.py --root ~/workbuddy-agent-os/agent-sync --input ~/backup/memories.zip`
|
||||
|
||||
## 数据源
|
||||
|
||||
| 数据源 | 路径 | 说明 |
|
||||
|--------|------|------|
|
||||
| Claw 工作日志 | `~/WorkBuddy/Claw/.workbuddy/memory/YYYY-MM-DD.md` | 每次对话后自动写入 |
|
||||
| WorkBuddy 系统画像 | `~/.workbuddy/memery/*.md` | 系统自动维护 |
|
||||
| 上轮摘要 | `04_memory/daily_summaries/YYYY-MM-DD.md` | 防丢失兜底 |
|
||||
|
||||
## 自动化
|
||||
|
||||
- WorkBuddy 自动化「AgentOS 每日记忆提炼」已配置：每日凌晨 2:00 执行
|
||||
- 命令：`/Users/chengzige/.workbuddy/binaries/python/envs/agent-os/bin/python3 ~/workbuddy-agent-os/agent-sync/02_skills/memory_manager/daily_digest.py --root ~/workbuddy-agent-os/agent-sync`
|
||||
|
||||
## 依赖
|
||||
|
||||
| 包 | 用途 | 安装状态 |
|
||||
|---|---|---|
|
||||
| sqlite3 | L2 事实库 | ✅ 标准库 |
|
||||
| sqlite-utils | SQLite 增强 | ✅ 已安装 |
|
||||
| rank_bm25 | BM25 关键词检索 | ✅ 已安装 |
|
||||
| chromadb | 向量存储与检索 | ✅ 1.5.8 |
|
||||
| numpy | 向量计算 | ✅ 2.4.4 |
|
||||
| requests | HTTP 客户端（embedding API） | ✅ 已安装 |
|
||||
| json/re/os/argparse | 基础 | ✅ 标准库 |
|
||||
|
||||
## 向量检索依赖
|
||||
|
||||
| 组件 | 说明 |
|
||||
|------|------|
|
||||
| oMLX API | localhost:8000/v1/embeddings |
|
||||
| 模型 | Qwen3-Embedding-0.6B（1024 维） |
|
||||
| 向量库 | ChromaDB，持久化到 04_memory/vector_db/chroma/ |
|
||||
@@ -0,0 +1,57 @@
|
||||
skill:
|
||||
name: "记忆管理器"
|
||||
id: "memory_manager"
|
||||
version: "1.2.0"
|
||||
category: "core-system"
|
||||
status: "active"
|
||||
created: "2026-04-25"
|
||||
updated: "2026-04-29"
|
||||
|
||||
description:
|
||||
short: "每日对话提炼、记忆固化检测、向量化、版本管理"
|
||||
use_cases:
|
||||
- "每日凌晨自动提炼前一天对话记忆"
|
||||
- "检测高频/重要记忆，推送到协同inbox/memory/"
|
||||
- "冷启动时从已有记忆初始化知识库"
|
||||
tags: ["记忆", "提炼", "固化", "向量化"]
|
||||
|
||||
ownership:
|
||||
master_editor: "my-macbook_macOS14.4_16GB_master"
|
||||
readonly_on_other_devices: true
|
||||
|
||||
environment:
|
||||
os: "macos"
|
||||
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
|
||||
pip_dependencies:
|
||||
- "sqlite-utils>=3.35"
|
||||
- "chromadb>=1.0.0"
|
||||
|
||||
setup:
|
||||
install_script: ""
|
||||
post_check: "python3 -c 'import sqlite3; print(\"OK\")'"
|
||||
|
||||
usage:
|
||||
entry_point: "daily_digest.py"
|
||||
scheduled: "每日 02:00（WorkBuddy 自动化）"
|
||||
manual_trigger: "记忆更新"
|
||||
|
||||
scripts:
|
||||
- daily_digest.py # 每日提炼主程序
|
||||
- memory_extractor.py # 记忆固化提取（推送inbox/memory/）
|
||||
- bootstrap_from_memory.py
|
||||
- agent_memory_init.py
|
||||
- export_memories.py
|
||||
- import_memories.py
|
||||
- memory_cleanup.py
|
||||
- semantic_search.py
|
||||
|
||||
changelog:
|
||||
- version: "1.2.0"
|
||||
date: "2026-04-29"
|
||||
changes: "新增 memory_extractor.py，支持固化条件检测+推送inbox/memory/"
|
||||
- version: "1.1.0"
|
||||
date: "2026-04-27"
|
||||
changes: "接入 oMLX Embedding，支持向量化"
|
||||
- version: "1.0.0"
|
||||
date: "2026-04-25"
|
||||
changes: "初始版本"
|
||||
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AgentOS 记忆体初始化脚本
|
||||
用途：首次运行时创建 L1 索引文件、L2 事实库、L3 原文目录
|
||||
使用：python3 agent_memory_init.py --root ~/workbuddy-agent-os/agent-sync
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 记忆体初始化")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def init_l1_index(root: str):
|
||||
"""初始化 L1 关键词索引"""
|
||||
index_path = Path(root) / "04_memory" / "vector_db" / "keyword_index.json"
|
||||
if index_path.exists():
|
||||
print(f"[INFO] L1 索引已存在: {index_path}")
|
||||
return
|
||||
|
||||
index_data = {
|
||||
"version": "1.0.0",
|
||||
"last_updated": "",
|
||||
"entries": []
|
||||
}
|
||||
index_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(index_path, "w", encoding="utf-8") as f:
|
||||
json.dump(index_data, f, ensure_ascii=False, indent=2)
|
||||
print(f"[OK] L1 关键词索引已创建: {index_path}")
|
||||
|
||||
|
||||
def init_l2_facts(root: str):
|
||||
"""初始化 L2 事实库"""
|
||||
db_path = Path(root) / "04_memory" / "long_term" / "facts.db"
|
||||
if db_path.exists():
|
||||
print(f"[INFO] L2 事实库已存在: {db_path}")
|
||||
return
|
||||
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute('''CREATE TABLE IF NOT EXISTS facts (
|
||||
id TEXT PRIMARY KEY,
|
||||
subject TEXT NOT NULL,
|
||||
predicate TEXT NOT NULL,
|
||||
object TEXT NOT NULL,
|
||||
confidence REAL DEFAULT 0.7,
|
||||
nature TEXT DEFAULT 'fact',
|
||||
domain TEXT,
|
||||
source TEXT,
|
||||
date_created TEXT,
|
||||
date_modified TEXT,
|
||||
previous_version TEXT,
|
||||
superseded_by TEXT,
|
||||
version INTEGER DEFAULT 1
|
||||
)''')
|
||||
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_subject ON facts(subject)')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_domain ON facts(domain)')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_confidence ON facts(confidence)')
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
print(f"[OK] L2 事实库已创建: {db_path}")
|
||||
|
||||
|
||||
def init_l3_raw(root: str):
|
||||
"""初始化 L3 原文存档目录"""
|
||||
raw_dir = Path(root) / "04_memory" / "long_term" / "raw"
|
||||
raw_dir.mkdir(parents=True, exist_ok=True)
|
||||
gitkeep = raw_dir / ".gitkeep"
|
||||
if not gitkeep.exists():
|
||||
gitkeep.touch()
|
||||
print(f"[OK] L3 原文目录已就绪: {raw_dir}")
|
||||
|
||||
|
||||
def init_logs(root: str):
|
||||
"""初始化日志目录"""
|
||||
log_dir = Path(root) / "04_memory" / "logs"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 创建空日志文件
|
||||
for log_name in ["kb_ingest.log", "errors.log", "conflicts.log", "kb_reclassify.log"]:
|
||||
log_file = log_dir / log_name
|
||||
if not log_file.exists():
|
||||
log_file.touch()
|
||||
|
||||
print(f"[OK] 日志目录已就绪: {log_dir}")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = os.path.expanduser(args.root)
|
||||
|
||||
print("=" * 50)
|
||||
print("AgentOS 记忆体初始化")
|
||||
print("=" * 50)
|
||||
|
||||
init_l1_index(root)
|
||||
init_l2_facts(root)
|
||||
init_l3_raw(root)
|
||||
init_logs(root)
|
||||
|
||||
print()
|
||||
print("=" * 50)
|
||||
print("[OK] 记忆体初始化完成！")
|
||||
print("=" * 50)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,388 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AgentOS 记忆体冷启动脚本
|
||||
用途：首次运行时，将已有的 MEMORY.md 和工作日志全量写入 L1/L2/L3
|
||||
这是记忆系统的「时间起点」——没有此步骤，L1/L2 将永远为空
|
||||
|
||||
使用：
|
||||
python3 bootstrap_from_memory.py --root ~/workbuddy-agent-os/agent-sync
|
||||
python3 bootstrap_from_memory.py --root ~/workbuddy-agent-os/agent-sync --dry-run # 预览不写入
|
||||
|
||||
数据来源（全量导入）：
|
||||
1. ~/WorkBuddy/Claw/.workbuddy/memory/MEMORY.md ← 长期精炼记忆
|
||||
2. ~/WorkBuddy/Claw/.workbuddy/memory/YYYY-MM-DD.md ← 所有历史日志
|
||||
3. ~/.workbuddy/memery/*.md ← 系统用户画像（V79+）
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
WORKBUDDY_MEMORY_DIR = Path.home() / "WorkBuddy" / "Claw" / ".workbuddy" / "memory"
|
||||
WORKBUDDY_SYSTEM_MEMERY_DIR = Path.home() / ".workbuddy" / "memery"
|
||||
|
||||
TODAY = datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 记忆体冷启动")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
parser.add_argument("--dry-run", action="store_true", help="仅预览，不写入")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def ensure_db(root: str):
|
||||
"""确保 L2 数据库存在并建好表"""
|
||||
db_path = Path(root) / "04_memory" / "long_term" / "facts.db"
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
cursor = conn.cursor()
|
||||
cursor.execute('''CREATE TABLE IF NOT EXISTS facts (
|
||||
id TEXT PRIMARY KEY,
|
||||
subject TEXT NOT NULL,
|
||||
predicate TEXT NOT NULL,
|
||||
object TEXT NOT NULL,
|
||||
confidence REAL DEFAULT 0.7,
|
||||
nature TEXT DEFAULT 'fact',
|
||||
domain TEXT,
|
||||
source TEXT,
|
||||
date_created TEXT,
|
||||
date_modified TEXT,
|
||||
previous_version TEXT,
|
||||
superseded_by TEXT,
|
||||
version INTEGER DEFAULT 1
|
||||
)''')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_subject ON facts(subject)')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_domain ON facts(domain)')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_confidence ON facts(confidence)')
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return db_path
|
||||
|
||||
|
||||
def parse_memory_md(content: str) -> list:
|
||||
"""
|
||||
解析 MEMORY.md 的结构化内容，提取事实列表。
|
||||
支持格式：
|
||||
- **字段名**：值
|
||||
- 普通列表项
|
||||
"""
|
||||
facts = []
|
||||
lines = content.split("\n")
|
||||
current_section = "general"
|
||||
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith(">"):
|
||||
continue
|
||||
|
||||
# 章节标题
|
||||
if stripped.startswith("## "):
|
||||
current_section = stripped[3:].strip()
|
||||
continue
|
||||
if stripped.startswith("# "):
|
||||
continue
|
||||
|
||||
# 格式 1：**Key**：Value 或 **Key**: Value
|
||||
m = re.match(r'\*\*(.+?)\*\*[：:]\s*(.+)', stripped)
|
||||
if m:
|
||||
key, value = m.group(1).strip(), m.group(2).strip()
|
||||
if len(value) > 5:
|
||||
facts.append({
|
||||
"subject": "ghai",
|
||||
"predicate": key,
|
||||
"object": value[:300],
|
||||
"confidence": 0.9,
|
||||
"nature": "fact",
|
||||
"domain": _infer_domain(current_section, value),
|
||||
"source": "MEMORY.md"
|
||||
})
|
||||
continue
|
||||
|
||||
# 格式 2：- 列表项（含有实质内容的行）
|
||||
if stripped.startswith("- ") and len(stripped) > 10:
|
||||
item = stripped[2:].strip()
|
||||
# 过滤纯说明性短句
|
||||
if len(item) < 15 or item.startswith("(") or item.startswith("（"):
|
||||
continue
|
||||
predicate = _infer_predicate(current_section, item)
|
||||
facts.append({
|
||||
"subject": "ghai",
|
||||
"predicate": predicate,
|
||||
"object": item[:300],
|
||||
"confidence": 0.8,
|
||||
"nature": "fact",
|
||||
"domain": _infer_domain(current_section, item),
|
||||
"source": "MEMORY.md"
|
||||
})
|
||||
|
||||
return facts
|
||||
|
||||
|
||||
def _infer_domain(section: str, text: str) -> str:
|
||||
"""根据章节标题和内容推断 domain"""
|
||||
section_lower = section.lower()
|
||||
if any(w in section_lower for w in ["项目", "工具", "agenos", "技能"]):
|
||||
return "engineering"
|
||||
if any(w in section_lower for w in ["偏好", "用户", "个人"]):
|
||||
return "personal-management"
|
||||
if any(w in section_lower for w in ["金融", "股票", "finance"]):
|
||||
return "finance"
|
||||
if any(w in text for w in ["代码", "脚本", "Node", "Python", "npm"]):
|
||||
return "engineering"
|
||||
return "personal-management"
|
||||
|
||||
|
||||
def _infer_predicate(section: str, text: str) -> str:
|
||||
"""根据章节和文本推断谓语"""
|
||||
if "偏好" in section or "偏好" in text:
|
||||
return "preference"
|
||||
if "工具" in section or "工具链" in section:
|
||||
return "uses_tool"
|
||||
if "项目" in section:
|
||||
return "owns_project"
|
||||
if "关注" in section:
|
||||
return "focuses_on"
|
||||
if "动态" in section:
|
||||
return "recent_activity"
|
||||
return "knows"
|
||||
|
||||
|
||||
def parse_daily_log(content: str, date: str) -> list:
|
||||
"""解析每日工作日志，提取完成事项和决策"""
|
||||
facts = []
|
||||
lines = content.split("\n")
|
||||
current_section = ""
|
||||
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
if stripped.startswith("## ") or stripped.startswith("### "):
|
||||
current_section = stripped.lstrip("#").strip()
|
||||
continue
|
||||
|
||||
# 提取列表项（已完成事项）
|
||||
if (stripped.startswith("- ") or stripped.startswith("✅") or
|
||||
stripped.startswith("1.") or re.match(r'^\d+\.', stripped)):
|
||||
item = re.sub(r'^[-\d\.\s✅]+', '', stripped).strip()
|
||||
if len(item) < 10:
|
||||
continue
|
||||
facts.append({
|
||||
"subject": "ghai",
|
||||
"predicate": "completed",
|
||||
"object": (f"[{current_section}] " if current_section else "") + item[:250],
|
||||
"confidence": 0.8,
|
||||
"nature": "fact",
|
||||
"domain": _infer_domain(current_section, item),
|
||||
"source": f"daily_log_{date}"
|
||||
})
|
||||
|
||||
return facts
|
||||
|
||||
|
||||
def insert_facts_to_l2(db_path: Path, facts: list, dry_run: bool = False) -> dict:
|
||||
"""批量插入事实到 L2，自动去重"""
|
||||
if dry_run:
|
||||
print(f" [DRY-RUN] 将写入 {len(facts)} 条事实到 L2")
|
||||
for f in facts[:5]:
|
||||
print(f" → [{f['domain']}] {f['predicate']}: {f['object'][:60]}...")
|
||||
if len(facts) > 5:
|
||||
print(f" ... 还有 {len(facts)-5} 条")
|
||||
return {"inserted": 0, "skipped": len(facts)}
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
cursor = conn.cursor()
|
||||
inserted = 0
|
||||
skipped = 0
|
||||
|
||||
# 获取全局已有总数，用于生成唯一 ID
|
||||
cursor.execute("SELECT COUNT(*) FROM facts")
|
||||
base = cursor.fetchone()[0]
|
||||
|
||||
for i, fact in enumerate(facts):
|
||||
# 精确去重：subject + predicate + object 三元组完全相同才跳过
|
||||
cursor.execute(
|
||||
"SELECT id FROM facts WHERE subject=? AND predicate=? AND object=?",
|
||||
(fact["subject"], fact["predicate"], fact["object"])
|
||||
)
|
||||
if cursor.fetchone():
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
fact_id = f"FACT-BOOTSTRAP-{base + i + 1:05d}"
|
||||
cursor.execute(
|
||||
"""INSERT INTO facts
|
||||
(id, subject, predicate, object, confidence, nature, domain, source,
|
||||
date_created, date_modified, version)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?)""",
|
||||
(fact_id, fact["subject"], fact["predicate"], fact["object"],
|
||||
fact.get("confidence", 0.7), fact.get("nature", "fact"),
|
||||
fact.get("domain", ""), fact.get("source", "bootstrap"),
|
||||
TODAY, TODAY, 1)
|
||||
)
|
||||
fact["id"] = fact_id
|
||||
inserted += 1
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return {"inserted": inserted, "skipped": skipped}
|
||||
|
||||
|
||||
def update_l1_index(root: str, facts: list, dry_run: bool = False):
|
||||
"""将事实写入 L1 关键词索引"""
|
||||
index_path = Path(root) / "04_memory" / "vector_db" / "keyword_index.json"
|
||||
index_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if index_path.exists():
|
||||
with open(index_path, "r", encoding="utf-8") as f:
|
||||
index_data = json.load(f)
|
||||
else:
|
||||
index_data = {"version": "1.0.0", "last_updated": "", "entries": []}
|
||||
|
||||
existing_ids = {e.get("fact_id") for e in index_data["entries"]}
|
||||
added = 0
|
||||
|
||||
for fact in facts:
|
||||
fid = fact.get("id", "")
|
||||
if not fid or fid in existing_ids:
|
||||
continue
|
||||
keywords = list(set(re.findall(r'[\u4e00-\u9fff]{2,}|[a-zA-Z]{3,}', fact["object"])))[:12]
|
||||
index_data["entries"].append({
|
||||
"fact_id": fid,
|
||||
"keywords": keywords,
|
||||
"domain": fact.get("domain", ""),
|
||||
"nature": fact.get("nature", "fact"),
|
||||
"summary": fact["object"][:120],
|
||||
"date": TODAY,
|
||||
"source": fact.get("source", "")
|
||||
})
|
||||
existing_ids.add(fid)
|
||||
added += 1
|
||||
|
||||
index_data["last_updated"] = datetime.now().isoformat()
|
||||
|
||||
if not dry_run:
|
||||
with open(index_path, "w", encoding="utf-8") as f:
|
||||
json.dump(index_data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
return added
|
||||
|
||||
|
||||
def archive_to_l3(root: str, source_name: str, content: str, dry_run: bool = False):
|
||||
"""归档原文到 L3"""
|
||||
raw_dir = Path(root) / "04_memory" / "long_term" / "raw"
|
||||
raw_dir.mkdir(parents=True, exist_ok=True)
|
||||
archive_file = raw_dir / f"bootstrap_{source_name}.md"
|
||||
|
||||
if dry_run:
|
||||
print(f" [DRY-RUN] 将归档到 L3: {archive_file.name}")
|
||||
return
|
||||
|
||||
archive_file.write_text(
|
||||
f"# L3 Bootstrap 存档 — {source_name}\n> 冷启动时间: {datetime.now().isoformat()}\n\n{content}",
|
||||
encoding="utf-8"
|
||||
)
|
||||
print(f" [OK] L3 归档: {archive_file.name}")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = os.path.expanduser(args.root)
|
||||
dry_run = args.dry_run
|
||||
|
||||
print("=" * 60)
|
||||
print("AgentOS 记忆体冷启动（Bootstrap）")
|
||||
print(f"时间起点: {TODAY}")
|
||||
if dry_run:
|
||||
print("[DRY-RUN 模式：仅预览，不写入任何文件]")
|
||||
print("=" * 60)
|
||||
|
||||
db_path = ensure_db(root)
|
||||
all_facts = []
|
||||
|
||||
# ── 来源 1：MEMORY.md 长期精炼记忆 ─────────────────────────────
|
||||
memory_md = WORKBUDDY_MEMORY_DIR / "MEMORY.md"
|
||||
if memory_md.exists():
|
||||
print(f"\n[来源1] MEMORY.md 长期记忆")
|
||||
content = memory_md.read_text(encoding="utf-8")
|
||||
facts = parse_memory_md(content)
|
||||
print(f" → 解析出 {len(facts)} 条事实")
|
||||
archive_to_l3(root, "MEMORY_md", content, dry_run)
|
||||
all_facts.extend(facts)
|
||||
else:
|
||||
print(f"[WARN] MEMORY.md 不存在: {memory_md}")
|
||||
|
||||
# ── 来源 2：每日工作日志（所有历史日期）──────────────────────────
|
||||
print(f"\n[来源2] 工作日志（{WORKBUDDY_MEMORY_DIR}）")
|
||||
log_total = 0
|
||||
if WORKBUDDY_MEMORY_DIR.exists():
|
||||
for log_file in sorted(WORKBUDDY_MEMORY_DIR.glob("????-??-??.md")):
|
||||
date_str = log_file.stem
|
||||
content = log_file.read_text(encoding="utf-8")
|
||||
facts = parse_daily_log(content, date_str)
|
||||
print(f" → {date_str}: {len(facts)} 条")
|
||||
archive_to_l3(root, f"daily_{date_str}", content, dry_run)
|
||||
all_facts.extend(facts)
|
||||
log_total += len(facts)
|
||||
print(f" 合计: {log_total} 条")
|
||||
|
||||
# ── 来源 3：WorkBuddy 系统用户画像 ──────────────────────────────
|
||||
print(f"\n[来源3] 系统用户画像（{WORKBUDDY_SYSTEM_MEMERY_DIR}）")
|
||||
if WORKBUDDY_SYSTEM_MEMERY_DIR.exists():
|
||||
for md_file in WORKBUDDY_SYSTEM_MEMERY_DIR.glob("*_memery.md"):
|
||||
content = md_file.read_text(encoding="utf-8")
|
||||
# 从用户画像提取结构化信息
|
||||
facts = []
|
||||
# 提取偏好段
|
||||
for pattern, pred, conf in [
|
||||
(r"用户名\s*([\w]+)，", "username", 0.95),
|
||||
(r"常居\s*([^\n。，,]+)", "city", 0.95),
|
||||
(r"(偏好.{20,200}?)[\n。]", "preference_summary", 0.85),
|
||||
(r"(期望.{20,200}?)[\n。]", "expectation", 0.85),
|
||||
]:
|
||||
m = re.search(pattern, content, re.DOTALL)
|
||||
if m:
|
||||
facts.append({
|
||||
"subject": "ghai",
|
||||
"predicate": pred,
|
||||
"object": m.group(1).strip()[:300],
|
||||
"confidence": conf,
|
||||
"nature": "fact",
|
||||
"domain": "personal-management",
|
||||
"source": f"system_memery_{md_file.stem}"
|
||||
})
|
||||
print(f" → {md_file.name}: {len(facts)} 条")
|
||||
archive_to_l3(root, f"system_{md_file.stem}", content, dry_run)
|
||||
all_facts.extend(facts)
|
||||
|
||||
# ── 写入 L2 + L1 ─────────────────────────────────────────────
|
||||
print(f"\n[总计] {len(all_facts)} 条候选事实")
|
||||
print("\n[写入 L2] 更新 facts.db...")
|
||||
stats = insert_facts_to_l2(db_path, all_facts, dry_run)
|
||||
print(f" → 新增 {stats['inserted']} 条，跳过重复 {stats['skipped']} 条")
|
||||
|
||||
print("\n[写入 L1] 更新 keyword_index.json...")
|
||||
l1_added = update_l1_index(root, all_facts, dry_run)
|
||||
print(f" → L1 索引新增 {l1_added} 条")
|
||||
|
||||
# ── 输出摘要 ─────────────────────────────────────────────────
|
||||
print("\n" + "=" * 60)
|
||||
if dry_run:
|
||||
print("✅ DRY-RUN 完成，未写入任何文件")
|
||||
else:
|
||||
print("✅ 冷启动完成！记忆系统已建立初始状态")
|
||||
print(f" L1 关键词索引: {Path(root) / '04_memory' / 'vector_db' / 'keyword_index.json'}")
|
||||
print(f" L2 事实库: {Path(root) / '04_memory' / 'long_term' / 'facts.db'}")
|
||||
print(f" L3 原文存档: {Path(root) / '04_memory' / 'long_term' / 'raw' / '*.md'}")
|
||||
print()
|
||||
print("下一步：每日自动提炼已配置，从明天起 daily_digest.py 将自动运行")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,536 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AgentOS 每日对话提炼脚本
|
||||
用途：从对话记录中提取关键事实，更新 L1 索引、L2 摘要库、L3 原文存档
|
||||
|
||||
数据源优先级（按顺序逐一尝试，全部合并）：
|
||||
1. WorkBuddy Claw 工作记忆日志 → ~/WorkBuddy/Claw/.workbuddy/memory/YYYY-MM-DD.md
|
||||
2. WorkBuddy 系统用户画像 → ~/.workbuddy/memery/*.md
|
||||
3. agent-os 每日摘要 → ~/workbuddy-agent-os/agent-sync/04_memory/daily_summaries/YYYY-MM-DD.md
|
||||
|
||||
使用：
|
||||
python3 daily_digest.py --root ~/workbuddy-agent-os/agent-sync --date 2026-04-25
|
||||
python3 daily_digest.py --root ~/workbuddy-agent-os/agent-sync # 默认处理昨天
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
# ─── 角色检查：所有机器都执行记忆提炼，但各自保存本地 ───
|
||||
try:
|
||||
sys.path.insert(0, str(Path.home() / "workbuddy-agent-os" / "agent-sync" / "05_tools" / "01_system"))
|
||||
from role_check import check_capability
|
||||
if not check_capability("memory_digestion"):
|
||||
print("⛔ 能力 memory_digestion 未启用，跳过 daily_digest")
|
||||
sys.exit(0)
|
||||
except ImportError as e:
|
||||
print(f"⚠️ 角色检查模块不可用 ({e})，继续执行")
|
||||
|
||||
|
||||
# ─── 数据源路径配置 ──────────────────────────────────────────────
|
||||
# 如果工作记忆路径变更，只改这里即可
|
||||
WORKBUDDY_MEMORY_DIR = Path.home() / "WorkBuddy" / "Claw" / ".workbuddy" / "memory"
|
||||
WORKBUDDY_SYSTEM_MEMERY_DIR = Path.home() / ".workbuddy" / "memery"
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 每日对话提炼")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
parser.add_argument("--date", default=None, help="日期 (YYYY-MM-DD)，默认昨天")
|
||||
parser.add_argument("--bootstrap", action="store_true",
|
||||
help="冷启动模式：处理所有可用日期，用于首次初始化")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def load_conversations(root: str, target_date: str) -> list:
|
||||
"""
|
||||
从以下来源加载对话内容（全部合并）：
|
||||
1. WorkBuddy Claw 工作记忆日志（每次工作后由 AI 写入）
|
||||
2. WorkBuddy 系统用户画像文件
|
||||
3. agent-os/04_memory/daily_summaries/（上一轮提炼结果，避免信息丢失）
|
||||
"""
|
||||
conversations = []
|
||||
|
||||
# ── 来源 1：Claw 工作记忆日志（最核心的数据源）──────────────────
|
||||
claw_daily = WORKBUDDY_MEMORY_DIR / f"{target_date}.md"
|
||||
if claw_daily.exists():
|
||||
content = claw_daily.read_text(encoding="utf-8")
|
||||
if content.strip():
|
||||
conversations.append({
|
||||
"source": str(claw_daily),
|
||||
"source_type": "claw_daily_log",
|
||||
"date": target_date,
|
||||
"content": content
|
||||
})
|
||||
print(f" [+] 来源1 Claw工作日志: {claw_daily} ({len(content)} 字符)")
|
||||
else:
|
||||
print(f" [-] 来源1 Claw工作日志不存在: {claw_daily}")
|
||||
|
||||
# ── 来源 2：WorkBuddy 系统用户画像 ──────────────────────────────
|
||||
if WORKBUDDY_SYSTEM_MEMERY_DIR.exists():
|
||||
for md_file in WORKBUDDY_SYSTEM_MEMERY_DIR.glob("*_memery.md"):
|
||||
content = md_file.read_text(encoding="utf-8")
|
||||
# 检查文件中是否包含目标日期（粗过滤）
|
||||
if target_date in content or "Last updated" in content:
|
||||
conversations.append({
|
||||
"source": str(md_file),
|
||||
"source_type": "workbuddy_system_memery",
|
||||
"date": target_date,
|
||||
"content": content
|
||||
})
|
||||
print(f" [+] 来源2 系统用户画像: {md_file.name} ({len(content)} 字符)")
|
||||
|
||||
# ── 来源 3：上一轮 agent-os 摘要（防丢失）──────────────────────
|
||||
daily_dir = Path(root) / "04_memory" / "daily_summaries"
|
||||
prev_summary = daily_dir / f"{target_date}.md"
|
||||
if prev_summary.exists():
|
||||
content = prev_summary.read_text(encoding="utf-8")
|
||||
if content.strip():
|
||||
conversations.append({
|
||||
"source": str(prev_summary),
|
||||
"source_type": "agent_os_summary",
|
||||
"date": target_date,
|
||||
"content": content
|
||||
})
|
||||
print(f" [+] 来源3 上轮摘要: {prev_summary.name} ({len(content)} 字符)")
|
||||
|
||||
return conversations
|
||||
|
||||
|
||||
def load_all_available_dates(root: str) -> list:
|
||||
"""冷启动模式：扫描所有可用日期（用于首次运行）"""
|
||||
dates = set()
|
||||
|
||||
# 扫描 Claw 工作记忆日志
|
||||
if WORKBUDDY_MEMORY_DIR.exists():
|
||||
for f in WORKBUDDY_MEMORY_DIR.glob("????-??-??.md"):
|
||||
dates.add(f.stem)
|
||||
|
||||
# 扫描 agent-os daily_summaries
|
||||
daily_dir = Path(root) / "04_memory" / "daily_summaries"
|
||||
if daily_dir.exists():
|
||||
for f in daily_dir.glob("????-??-??.md"):
|
||||
dates.add(f.stem)
|
||||
|
||||
return sorted(dates)
|
||||
|
||||
|
||||
def extract_facts(content: str, date: str, source_type: str = "unknown") -> list:
|
||||
"""
|
||||
从对话内容中提取关键事实。
|
||||
|
||||
提取策略：
|
||||
- Claw工作日志（claw_daily_log）：按 ## 章节和 ### 子项提取结构化决策
|
||||
- 系统用户画像（workbuddy_system_memery）：提取用户基本信息和偏好
|
||||
- 通用：包含决策/配置/部署等关键词的行
|
||||
"""
|
||||
facts = []
|
||||
|
||||
if source_type == "workbuddy_system_memery":
|
||||
# 用户画像：提取姓名、城市、偏好等结构化信息
|
||||
patterns = [
|
||||
(r"\*\*Name\*\*[：:]\s*(.+)", "user_profile", "name"),
|
||||
(r"\*\*City\*\*[：:]\s*(.+)", "user_profile", "city"),
|
||||
(r"用户名\s*(.+?)，", "user_profile", "username"),
|
||||
(r"常居\s*(.+?)。", "user_profile", "city"),
|
||||
]
|
||||
for pattern, subject, predicate in patterns:
|
||||
m = re.search(pattern, content)
|
||||
if m:
|
||||
facts.append({
|
||||
"subject": subject,
|
||||
"predicate": predicate,
|
||||
"object": m.group(1).strip()[:200],
|
||||
"confidence": 0.9,
|
||||
"nature": "fact",
|
||||
"domain": "personal-management",
|
||||
"date_created": date,
|
||||
"date_modified": date,
|
||||
"version": 1,
|
||||
"source": source_type
|
||||
})
|
||||
|
||||
# 提取偏好段落整体
|
||||
pref_match = re.search(r"(偏好.{10,200})", content, re.DOTALL)
|
||||
if pref_match:
|
||||
facts.append({
|
||||
"subject": "user",
|
||||
"predicate": "preference",
|
||||
"object": pref_match.group(1).strip()[:300],
|
||||
"confidence": 0.85,
|
||||
"nature": "opinion",
|
||||
"domain": "personal-management",
|
||||
"date_created": date,
|
||||
"date_modified": date,
|
||||
"version": 1,
|
||||
"source": source_type
|
||||
})
|
||||
return facts
|
||||
|
||||
# ── 通用提取：决策/配置/完成事项 ────────────────────────────────
|
||||
lines = content.split("\n")
|
||||
current_section = ""
|
||||
|
||||
for line in lines:
|
||||
line_stripped = line.strip()
|
||||
if not line_stripped:
|
||||
continue
|
||||
|
||||
# 记录当前章节标题（用于 domain 推断）
|
||||
if line_stripped.startswith("## ") or line_stripped.startswith("### "):
|
||||
current_section = line_stripped.lstrip("#").strip()
|
||||
continue
|
||||
|
||||
# 决策性关键词
|
||||
decision_keywords = [
|
||||
"决定", "确认", "选择", "修改", "配置", "部署", "安装", "更新",
|
||||
"修复", "完成", "创建", "初始化", "验证", "生成", "已部署", "已完成",
|
||||
"设计决策", "关键决策"
|
||||
]
|
||||
|
||||
for kw in decision_keywords:
|
||||
if kw in line_stripped and len(line_stripped) > 10:
|
||||
# 推断领域
|
||||
domain = "personal-management"
|
||||
if any(w in current_section for w in ["代码", "脚本", "技能", "工具", "初始化", "部署"]):
|
||||
domain = "engineering"
|
||||
elif any(w in current_section for w in ["知识", "记忆", "学习"]):
|
||||
domain = "personal-management"
|
||||
|
||||
facts.append({
|
||||
"subject": "ghai",
|
||||
"predicate": kw,
|
||||
"object": (f"[{current_section}] " if current_section else "") + line_stripped[:250],
|
||||
"confidence": 0.75,
|
||||
"nature": "fact",
|
||||
"domain": domain,
|
||||
"date_created": date,
|
||||
"date_modified": date,
|
||||
"version": 1,
|
||||
"source": source_type
|
||||
})
|
||||
break # 一行只取一次
|
||||
|
||||
return facts
|
||||
|
||||
|
||||
def check_duplicate(fact: dict, cursor) -> bool:
|
||||
"""检查事实是否已存在（去重）"""
|
||||
cursor.execute(
|
||||
"SELECT id FROM facts WHERE subject = ? AND predicate = ? AND object = ?",
|
||||
(fact["subject"], fact["predicate"], fact["object"])
|
||||
)
|
||||
return cursor.fetchone() is not None
|
||||
|
||||
|
||||
def check_conflict(fact: dict, cursor):
|
||||
"""检查是否存在内容不同但主谓相同的旧事实（冲突）"""
|
||||
cursor.execute(
|
||||
"SELECT id, object, confidence FROM facts WHERE subject = ? AND predicate = ? AND object != ?",
|
||||
(fact["subject"], fact["predicate"], fact["object"])
|
||||
)
|
||||
return cursor.fetchone()
|
||||
|
||||
|
||||
def generate_fact_id(date: str, index: int) -> str:
|
||||
return f"FACT-{date}-{index:04d}"
|
||||
|
||||
|
||||
def update_l1_index(root: str, facts: list):
|
||||
"""更新 L1 关键词索引"""
|
||||
index_path = Path(root) / "04_memory" / "vector_db" / "keyword_index.json"
|
||||
index_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if index_path.exists():
|
||||
with open(index_path, "r", encoding="utf-8") as f:
|
||||
index_data = json.load(f)
|
||||
else:
|
||||
index_data = {"version": "1.0.0", "last_updated": "", "entries": []}
|
||||
|
||||
# 避免重复插入：已有 fact_id 的不重复加
|
||||
existing_ids = {e.get("fact_id") for e in index_data["entries"]}
|
||||
|
||||
for fact in facts:
|
||||
fid = fact.get("id", "")
|
||||
if fid in existing_ids:
|
||||
continue
|
||||
keywords = list(set(re.findall(r'[\u4e00-\u9fff]{2,}|[a-zA-Z]{3,}', fact["object"])))[:12]
|
||||
entry = {
|
||||
"fact_id": fid,
|
||||
"keywords": keywords,
|
||||
"domain": fact.get("domain", ""),
|
||||
"nature": fact.get("nature", "fact"),
|
||||
"summary": fact["object"][:120],
|
||||
"date": fact.get("date_created", ""),
|
||||
"source": fact.get("source", "")
|
||||
}
|
||||
index_data["entries"].append(entry)
|
||||
existing_ids.add(fid)
|
||||
|
||||
index_data["last_updated"] = datetime.now().isoformat()
|
||||
|
||||
with open(index_path, "w", encoding="utf-8") as f:
|
||||
json.dump(index_data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
def update_l2_facts(root: str, facts: list, date: str):
|
||||
"""更新 L2 事实库（SQLite）"""
|
||||
db_path = Path(root) / "04_memory" / "long_term" / "facts.db"
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute('''CREATE TABLE IF NOT EXISTS facts (
|
||||
id TEXT PRIMARY KEY,
|
||||
subject TEXT NOT NULL,
|
||||
predicate TEXT NOT NULL,
|
||||
object TEXT NOT NULL,
|
||||
confidence REAL DEFAULT 0.7,
|
||||
nature TEXT DEFAULT 'fact',
|
||||
domain TEXT,
|
||||
source TEXT,
|
||||
date_created TEXT,
|
||||
date_modified TEXT,
|
||||
previous_version TEXT,
|
||||
superseded_by TEXT,
|
||||
version INTEGER DEFAULT 1
|
||||
)''')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_subject ON facts(subject)')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_domain ON facts(domain)')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_confidence ON facts(confidence)')
|
||||
|
||||
# 获取当日已有多少条，生成连续 ID
|
||||
cursor.execute("SELECT COUNT(*) FROM facts WHERE date_created = ?", (date,))
|
||||
existing_count = cursor.fetchone()[0]
|
||||
|
||||
conflicts = []
|
||||
duplicates = 0
|
||||
inserted = 0
|
||||
|
||||
for i, fact in enumerate(facts):
|
||||
fact_id = generate_fact_id(date, existing_count + i + 1)
|
||||
fact["id"] = fact_id
|
||||
|
||||
if check_duplicate(fact, cursor):
|
||||
duplicates += 1
|
||||
fact["_skipped"] = True
|
||||
continue
|
||||
|
||||
conflict = check_conflict(fact, cursor)
|
||||
if conflict:
|
||||
existing_id, existing_obj, existing_conf = conflict
|
||||
if fact["confidence"] > existing_conf:
|
||||
# 新事实置信度更高 → 覆盖，保留旧值到 previous_version
|
||||
cursor.execute(
|
||||
"""UPDATE facts SET object=?, confidence=?, date_modified=?,
|
||||
previous_version=?, version=version+1 WHERE id=?""",
|
||||
(fact["object"], fact["confidence"], date, existing_obj, existing_id)
|
||||
)
|
||||
conflicts.append({"action": "override", "fact_id": existing_id,
|
||||
"reason": f"新事实置信度更高({fact['confidence']} > {existing_conf})"})
|
||||
else:
|
||||
conflicts.append({"action": "skip", "fact_id": fact_id,
|
||||
"reason": f"旧事实置信度更高({existing_conf} >= {fact['confidence']})，需用户确认"})
|
||||
duplicates += 1
|
||||
fact["_skipped"] = True
|
||||
continue
|
||||
|
||||
cursor.execute(
|
||||
"""INSERT INTO facts
|
||||
(id, subject, predicate, object, confidence, nature, domain, source,
|
||||
date_created, date_modified, version)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?)""",
|
||||
(fact_id, fact["subject"], fact["predicate"], fact["object"],
|
||||
fact.get("confidence", 0.7), fact.get("nature", "fact"),
|
||||
fact.get("domain", ""), fact.get("source", ""),
|
||||
fact.get("date_created", date), fact.get("date_modified", date),
|
||||
fact.get("version", 1))
|
||||
)
|
||||
inserted += 1
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return {"inserted": inserted, "duplicates": duplicates, "conflicts": conflicts}
|
||||
|
||||
|
||||
def archive_to_l3(root: str, date: str, conversations: list):
|
||||
"""将原始对话内容压缩归档到 L3"""
|
||||
raw_dir = Path(root) / "04_memory" / "long_term" / "raw"
|
||||
raw_dir.mkdir(parents=True, exist_ok=True)
|
||||
archive_file = raw_dir / f"{date}.md"
|
||||
|
||||
if archive_file.exists():
|
||||
# 追加模式，不覆盖历史
|
||||
return
|
||||
|
||||
content = f"# L3 原文存档 — {date}\n\n"
|
||||
for conv in conversations:
|
||||
content += f"## 来源：{conv['source_type']} | {conv['source']}\n\n"
|
||||
content += conv["content"]
|
||||
content += "\n\n---\n\n"
|
||||
|
||||
archive_file.write_text(content, encoding="utf-8")
|
||||
print(f" [OK] L3 原文已归档: {archive_file}")
|
||||
|
||||
|
||||
def save_daily_summary(root: str, date: str, facts: list, stats: dict):
|
||||
"""保存每日提炼摘要"""
|
||||
summary_dir = Path(root) / "04_memory" / "daily_summaries"
|
||||
summary_dir.mkdir(parents=True, exist_ok=True)
|
||||
summary_file = summary_dir / f"{date}.md"
|
||||
|
||||
content = f"# {date} 记忆提炼摘要\n\n"
|
||||
content += f"> 提炼时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n"
|
||||
content += f"## 统计\n"
|
||||
content += f"- 新增事实: {stats['inserted']}\n"
|
||||
content += f"- 重复跳过: {stats['duplicates']}\n"
|
||||
content += f"- 冲突处理: {len(stats['conflicts'])}\n\n"
|
||||
|
||||
if facts:
|
||||
content += "## 提取的事实\n\n"
|
||||
for fact in facts:
|
||||
marker = "📌" if fact.get("nature") == "fact" else "💭"
|
||||
content += f"- {marker} **[{fact.get('domain', '?')}]** `{fact.get('predicate')}` → {fact['object'][:120]}\n"
|
||||
|
||||
if stats["conflicts"]:
|
||||
content += "\n## 冲突记录\n\n"
|
||||
for c in stats["conflicts"]:
|
||||
content += f"- `{c['fact_id']}`: **{c['action']}** — {c['reason']}\n"
|
||||
|
||||
summary_file.write_text(content, encoding="utf-8")
|
||||
print(f" [OK] 每日摘要已保存: 04_memory/daily_summaries/{date}.md")
|
||||
|
||||
|
||||
def write_conflict_log(root: str, conflicts: list, date: str):
|
||||
"""写入冲突日志"""
|
||||
if not conflicts:
|
||||
return
|
||||
log_dir = Path(root) / "04_memory" / "logs"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_file = log_dir / "conflicts.log"
|
||||
with open(log_file, "a", encoding="utf-8") as f:
|
||||
for c in conflicts:
|
||||
f.write(f"{datetime.now().isoformat()} | {date} | {c['fact_id']} | {c['action']} | {c['reason']}\n")
|
||||
|
||||
|
||||
def process_date(root: str, target_date: str):
|
||||
"""处理单个日期的完整流程"""
|
||||
print(f"\n{'='*55}")
|
||||
print(f"处理日期: {target_date}")
|
||||
print(f"{'='*55}")
|
||||
|
||||
# 1. 加载对话
|
||||
print("[步骤1] 加载对话数据源...")
|
||||
conversations = load_conversations(root, target_date)
|
||||
|
||||
if not conversations:
|
||||
print(f" [WARN] {target_date} 无可用数据源，跳过")
|
||||
return
|
||||
|
||||
# 2. 提取事实
|
||||
print(f"[步骤2] 提取关键事实...")
|
||||
all_facts = []
|
||||
for conv in conversations:
|
||||
facts = extract_facts(conv["content"], target_date, conv["source_type"])
|
||||
all_facts.extend(facts)
|
||||
print(f" → 从 {conv['source_type']} 提取 {len(facts)} 条候选事实")
|
||||
|
||||
if not all_facts:
|
||||
print(f" [INFO] {target_date} 无新事实可提取")
|
||||
return
|
||||
|
||||
print(f" [INFO] 共 {len(all_facts)} 条候选事实")
|
||||
|
||||
# 3. 更新 L2 事实库
|
||||
print("[步骤3] 更新 L2 事实库...")
|
||||
stats = update_l2_facts(root, all_facts, target_date)
|
||||
print(f" [OK] 新增 {stats['inserted']} 条，重复 {stats['duplicates']} 条，冲突 {len(stats['conflicts'])} 条")
|
||||
|
||||
# 4. 更新 L1 关键词索引（仅更新实际写入 L2 的事实，避免孤儿索引）
|
||||
print("[步骤4] 更新 L1 关键词索引...")
|
||||
inserted_facts = [f for f in all_facts if f.get("id") and not f.get("_skipped")]
|
||||
update_l1_index(root, inserted_facts)
|
||||
print(f" [OK] L1 索引已更新 ({len(inserted_facts)} 条)")
|
||||
|
||||
# 4.5 更新向量库（为新事实生成 embedding 写入 ChromaDB）
|
||||
if inserted_facts:
|
||||
try:
|
||||
print("[步骤4.5] 更新向量库...")
|
||||
# 动态导入同目录的 semantic_search 模块
|
||||
skill_dir = Path(__file__).parent
|
||||
if str(skill_dir) not in sys.path:
|
||||
sys.path.insert(0, str(skill_dir))
|
||||
from semantic_search import VectorStore
|
||||
vector_dir = str(Path(root) / "04_memory" / "vector_db" / "chroma")
|
||||
vs = VectorStore(vector_dir)
|
||||
items = []
|
||||
for f in inserted_facts:
|
||||
text = f"{f['subject']} {f['predicate']} {f['object']}"
|
||||
items.append({
|
||||
"fact_id": f["id"],
|
||||
"text": text,
|
||||
"metadata": {
|
||||
"summary": f["object"][:120],
|
||||
"domain": f.get("domain", ""),
|
||||
"nature": f.get("nature", "fact"),
|
||||
"source": f.get("source", ""),
|
||||
"date_created": f.get("date_created", target_date),
|
||||
"confidence": f.get("confidence", 0.7)
|
||||
}
|
||||
})
|
||||
vs.upsert_batch(items)
|
||||
print(f" [OK] 向量库已更新 ({len(items)} 条)")
|
||||
except Exception as e:
|
||||
print(f" [WARN] 向量库更新失败（不阻塞主流程）: {e}")
|
||||
|
||||
# 5. 归档原文到 L3
|
||||
print("[步骤5] 归档原文到 L3...")
|
||||
archive_to_l3(root, target_date, conversations)
|
||||
|
||||
# 6. 保存摘要
|
||||
print("[步骤6] 保存每日摘要...")
|
||||
save_daily_summary(root, target_date, all_facts, stats)
|
||||
|
||||
# 7. 记录冲突
|
||||
print("[步骤7] 记录冲突...")
|
||||
write_conflict_log(root, stats["conflicts"], target_date)
|
||||
if stats["conflicts"]:
|
||||
print(f" [WARN] {len(stats['conflicts'])} 条冲突已记录到 conflicts.log")
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = os.path.expanduser(args.root)
|
||||
|
||||
if args.bootstrap:
|
||||
# 冷启动模式：处理所有历史日期
|
||||
print("=" * 55)
|
||||
print("AgentOS 记忆体冷启动（处理所有历史记忆）")
|
||||
print("=" * 55)
|
||||
dates = load_all_available_dates(root)
|
||||
if not dates:
|
||||
print("[WARN] 未找到任何历史记忆文件")
|
||||
return
|
||||
print(f"发现 {len(dates)} 个日期: {', '.join(dates)}")
|
||||
for d in dates:
|
||||
process_date(root, d)
|
||||
print("\n[OK] 冷启动完成！")
|
||||
else:
|
||||
# 普通模式：处理指定日期（默认昨天）
|
||||
if args.date:
|
||||
target_date = args.date
|
||||
else:
|
||||
target_date = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
|
||||
process_date(root, target_date)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AgentOS 记忆导出脚本
|
||||
用途：将 L1 索引 + L2 事实库 + L3 原文目录 + 日志打包为 JSON + Markdown 归档
|
||||
使用：python3 export_memories.py --root ~/workbuddy-agent-os/agent-sync --output ~/backup/
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 记忆导出")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
parser.add_argument("--output", required=True, help="导出目录路径")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def export_l2(db_path: str, output_dir: Path):
|
||||
"""导出 L2 事实库为 JSON"""
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT * FROM facts ORDER BY date_created DESC")
|
||||
facts = [dict(row) for row in cursor.fetchall()]
|
||||
conn.close()
|
||||
|
||||
out_file = output_dir / "l2_facts.json"
|
||||
with open(out_file, "w", encoding="utf-8") as f:
|
||||
json.dump(facts, f, ensure_ascii=False, indent=2)
|
||||
print(f" [OK] L2 导出: {len(facts)} 条事实 → {out_file.name}")
|
||||
return facts
|
||||
|
||||
|
||||
def export_l1(index_path: Path, output_dir: Path):
|
||||
"""导出 L1 关键词索引"""
|
||||
with open(index_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
out_file = output_dir / "l1_keyword_index.json"
|
||||
with open(out_file, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, ensure_ascii=False, indent=2)
|
||||
print(f" [OK] L1 导出: {len(data['entries'])} 条索引 → {out_file.name}")
|
||||
|
||||
|
||||
def export_l3(raw_dir: Path, output_dir: Path):
|
||||
"""导出 L3 原文存档"""
|
||||
l3_out = output_dir / "l3_raw"
|
||||
l3_out.mkdir(parents=True, exist_ok=True)
|
||||
count = 0
|
||||
for md_file in raw_dir.glob("*.md"):
|
||||
shutil.copy2(md_file, l3_out / md_file.name)
|
||||
count += 1
|
||||
print(f" [OK] L3 导出: {count} 个文件 → l3_raw/")
|
||||
|
||||
|
||||
def export_logs(log_dir: Path, output_dir: Path):
|
||||
"""导出日志文件"""
|
||||
logs_out = output_dir / "logs"
|
||||
logs_out.mkdir(parents=True, exist_ok=True)
|
||||
count = 0
|
||||
for log_file in log_dir.glob("*.log"):
|
||||
shutil.copy2(log_file, logs_out / log_file.name)
|
||||
count += 1
|
||||
print(f" [OK] 日志导出: {count} 个文件 → logs/")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = os.path.expanduser(args.root)
|
||||
output_dir = Path(os.path.expanduser(args.output))
|
||||
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
export_dir = output_dir / f"agent-os-memory-{timestamp}"
|
||||
export_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
print("=" * 50)
|
||||
print("AgentOS 记忆导出")
|
||||
print("=" * 50)
|
||||
print(f"源目录: {root}/04_memory/")
|
||||
print(f"导出到: {export_dir}\n")
|
||||
|
||||
# 写入元数据
|
||||
meta = {
|
||||
"export_time": datetime.now().isoformat(),
|
||||
"source_root": root,
|
||||
"agent_os_version": "2.0",
|
||||
}
|
||||
with open(export_dir / "export_meta.json", "w", encoding="utf-8") as f:
|
||||
json.dump(meta, f, ensure_ascii=False, indent=2)
|
||||
|
||||
# 逐层导出
|
||||
memory_root = Path(root) / "04_memory"
|
||||
export_l1(memory_root / "vector_db" / "keyword_index.json", export_dir)
|
||||
export_l2(str(memory_root / "long_term" / "facts.db"), export_dir)
|
||||
export_l3(memory_root / "long_term" / "raw", export_dir)
|
||||
export_logs(memory_root / "logs", export_dir)
|
||||
|
||||
# 打包（可选）
|
||||
archive_path = str(export_dir) + ".zip"
|
||||
shutil.make_archive(str(export_dir), "zip", export_dir.parent, export_dir.name)
|
||||
|
||||
print(f"\n[OK] 导出完成！")
|
||||
print(f" 目录: {export_dir}")
|
||||
print(f" 压缩包: {archive_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,226 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AgentOS 记忆导入脚本
|
||||
用途：从导出归档恢复记忆，合并时智能去重
|
||||
使用：python3 import_memories.py --root ~/workbuddy-agent-os/agent-sync --input ~/backup/agent-os-memory-20260426.zip
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import zipfile
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 记忆导入")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
parser.add_argument("--input", required=True, help="导入的 zip 文件或目录路径")
|
||||
parser.add_argument("--dry-run", action="store_true", help="仅预览，不实际写入")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def extract_source(input_path: str) -> Path:
|
||||
"""从 zip 或目录获取源路径"""
|
||||
p = Path(input_path)
|
||||
if p.is_dir():
|
||||
return p
|
||||
if p.suffix == ".zip":
|
||||
extract_dir = p.parent / p.stem
|
||||
if extract_dir.exists():
|
||||
print(f" [INFO] 解压目录已存在: {extract_dir}")
|
||||
else:
|
||||
with zipfile.ZipFile(p, "r") as z:
|
||||
z.extractall(extract_dir)
|
||||
print(f" [OK] 已解压: {extract_dir}")
|
||||
return extract_dir
|
||||
raise ValueError(f"不支持的输入格式: {input_path}")
|
||||
|
||||
|
||||
def import_l2(src_dir: Path, db_path: Path, dry_run: bool):
|
||||
"""导入 L2 事实，智能去重"""
|
||||
src_file = src_dir / "l2_facts.json"
|
||||
if not src_file.exists():
|
||||
print(" [WARN] 未找到 l2_facts.json，跳过 L2 导入")
|
||||
return
|
||||
|
||||
with open(src_file, "r", encoding="utf-8") as f:
|
||||
imported_facts = json.load(f)
|
||||
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
cursor = conn.cursor()
|
||||
|
||||
# 确保表存在
|
||||
cursor.execute('''CREATE TABLE IF NOT EXISTS facts (
|
||||
id TEXT PRIMARY KEY,
|
||||
subject TEXT NOT NULL,
|
||||
predicate TEXT NOT NULL,
|
||||
object TEXT NOT NULL,
|
||||
confidence REAL DEFAULT 0.7,
|
||||
nature TEXT DEFAULT 'fact',
|
||||
domain TEXT,
|
||||
source TEXT,
|
||||
date_created TEXT,
|
||||
date_modified TEXT,
|
||||
previous_version TEXT,
|
||||
superseded_by TEXT,
|
||||
version INTEGER DEFAULT 1
|
||||
)''')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_subject ON facts(subject)')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_domain ON facts(domain)')
|
||||
cursor.execute('CREATE INDEX IF NOT EXISTS idx_facts_confidence ON facts(confidence)')
|
||||
|
||||
inserted = 0
|
||||
skipped = 0
|
||||
updated = 0
|
||||
|
||||
for fact in imported_facts:
|
||||
fid = fact.get("id", "")
|
||||
# 检查是否已存在
|
||||
cursor.execute("SELECT id, object, confidence FROM facts WHERE id = ?", (fid,))
|
||||
existing = cursor.fetchone()
|
||||
|
||||
if existing:
|
||||
# 已存在，检查是否需要更新
|
||||
if fact.get("date_modified", "") > (existing[2] or ""):
|
||||
# 导入版本更新 → 覆盖
|
||||
if not dry_run:
|
||||
cursor.execute(
|
||||
"""UPDATE facts SET object=?, confidence=?, nature=?, domain=?,
|
||||
source=?, date_modified=?, version=? WHERE id=?""",
|
||||
(fact["object"], fact["confidence"], fact["nature"],
|
||||
fact["domain"], fact["source"], fact["date_modified"],
|
||||
fact.get("version", 1), fid)
|
||||
)
|
||||
updated += 1
|
||||
else:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
# 三元组去重
|
||||
cursor.execute(
|
||||
"SELECT id FROM facts WHERE subject=? AND predicate=? AND object=?",
|
||||
(fact["subject"], fact["predicate"], fact["object"])
|
||||
)
|
||||
if cursor.fetchone():
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
# 新事实，插入
|
||||
if not dry_run:
|
||||
cursor.execute(
|
||||
"""INSERT INTO facts
|
||||
(id, subject, predicate, object, confidence, nature, domain, source,
|
||||
date_created, date_modified, version)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?)""",
|
||||
(fid, fact["subject"], fact["predicate"], fact["object"],
|
||||
fact.get("confidence", 0.7), fact.get("nature", "fact"),
|
||||
fact.get("domain", ""), fact.get("source", "import"),
|
||||
fact.get("date_created", ""), fact.get("date_modified", ""),
|
||||
fact.get("version", 1))
|
||||
)
|
||||
inserted += 1
|
||||
|
||||
if not dry_run:
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
print(f" L2 导入: 新增 {inserted} 条，更新 {updated} 条，跳过 {skipped} 条")
|
||||
|
||||
|
||||
def import_l1(src_dir: Path, index_path: Path, db_path: Path, dry_run: bool):
|
||||
"""导入 L1 关键词索引，与 L2 对齐"""
|
||||
src_file = src_dir / "l1_keyword_index.json"
|
||||
if not src_file.exists():
|
||||
print(" [WARN] 未找到 l1_keyword_index.json，跳过 L1 导入")
|
||||
return
|
||||
|
||||
with open(src_file, "r", encoding="utf-8") as f:
|
||||
imported_data = json.load(f)
|
||||
|
||||
# 获取 L2 中实际存在的 ID
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
cursor = conn.cursor()
|
||||
l2_ids = {row[0] for row in cursor.execute("SELECT id FROM facts").fetchall()}
|
||||
conn.close()
|
||||
|
||||
# 读取现有 L1
|
||||
if index_path.exists():
|
||||
with open(index_path, "r", encoding="utf-8") as f:
|
||||
l1 = json.load(f)
|
||||
else:
|
||||
l1 = {"version": "1.0.0", "last_updated": "", "entries": []}
|
||||
|
||||
existing_ids = {e["fact_id"] for e in l1["entries"]}
|
||||
added = 0
|
||||
|
||||
for entry in imported_data["entries"]:
|
||||
fid = entry["fact_id"]
|
||||
if fid in existing_ids or fid not in l2_ids:
|
||||
continue
|
||||
l1["entries"].append(entry)
|
||||
existing_ids.add(fid)
|
||||
added += 1
|
||||
|
||||
l1["last_updated"] = datetime.now().isoformat()
|
||||
|
||||
if not dry_run:
|
||||
index_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(index_path, "w", encoding="utf-8") as f:
|
||||
json.dump(l1, f, ensure_ascii=False, indent=2)
|
||||
|
||||
print(f" L1 导入: 新增 {added} 条索引")
|
||||
|
||||
|
||||
def import_l3(src_dir: Path, raw_dir: Path, dry_run: bool):
|
||||
"""导入 L3 原文存档"""
|
||||
src_l3 = src_dir / "l3_raw"
|
||||
if not src_l3.exists():
|
||||
print(" [WARN] 未找到 l3_raw/ 目录，跳过 L3 导入")
|
||||
return
|
||||
|
||||
raw_dir.mkdir(parents=True, exist_ok=True)
|
||||
count = 0
|
||||
for md_file in src_l3.glob("*.md"):
|
||||
dest = raw_dir / md_file.name
|
||||
if dest.exists():
|
||||
continue # 不覆盖已有文件
|
||||
if not dry_run:
|
||||
shutil.copy2(md_file, dest)
|
||||
count += 1
|
||||
|
||||
print(f" L3 导入: {count} 个文件")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = os.path.expanduser(args.root)
|
||||
dry_run = args.dry_run
|
||||
|
||||
print("=" * 50)
|
||||
print("AgentOS 记忆导入")
|
||||
if dry_run:
|
||||
print("[DRY-RUN 模式：仅预览，不写入任何文件]")
|
||||
print("=" * 50)
|
||||
|
||||
src_dir = extract_source(args.input)
|
||||
print(f"导入源: {src_dir}\n")
|
||||
|
||||
memory_root = Path(root) / "04_memory"
|
||||
db_path = memory_root / "long_term" / "facts.db"
|
||||
index_path = memory_root / "vector_db" / "keyword_index.json"
|
||||
raw_dir = memory_root / "long_term" / "raw"
|
||||
|
||||
import_l2(src_dir, db_path, dry_run)
|
||||
import_l1(src_dir, index_path, db_path, dry_run)
|
||||
import_l3(src_dir, raw_dir, dry_run)
|
||||
|
||||
print(f"\n{'✅ DRY-RUN 完成' if dry_run else '✅ 导入完成！'}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AgentOS 记忆清理与冲突消解脚本
|
||||
用途：清理过期记忆、消解冲突、归档冷数据
|
||||
使用：python3 memory_cleanup.py --root ~/workbuddy-agent-os/agent-sync [--dry-run]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 记忆清理")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
parser.add_argument("--dry-run", action="store_true", help="仅预览，不实际执行")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def cleanup_expired_facts(root: str, dry_run: bool = False):
|
||||
"""清理过期事实（技术类知识 180 天未更新自动标记待复核）"""
|
||||
db_path = Path(root) / "04_memory" / "long_term" / "facts.db"
|
||||
if not db_path.exists():
|
||||
print("[WARN] facts.db 不存在，跳过")
|
||||
return
|
||||
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
cursor = conn.cursor()
|
||||
|
||||
# 查找 180 天未更新的技术类事实
|
||||
cutoff = (datetime.now() - timedelta(days=180)).strftime("%Y-%m-%d")
|
||||
cursor.execute(
|
||||
"SELECT id, subject, date_modified FROM facts WHERE date_modified < ? AND nature != 'axiom' AND nature != 'opinion'",
|
||||
(cutoff,)
|
||||
)
|
||||
expired = cursor.fetchall()
|
||||
|
||||
if not expired:
|
||||
print("[INFO] 无过期事实")
|
||||
conn.close()
|
||||
return
|
||||
|
||||
print(f"[INFO] 发现 {len(expired)} 条可能过期的事实:")
|
||||
for fact in expired:
|
||||
print(f" - {fact[0]}: {fact[1]} (最后更新: {fact[2]})")
|
||||
|
||||
if not dry_run:
|
||||
for fact in expired:
|
||||
cursor.execute(
|
||||
"UPDATE facts SET confidence = confidence * 0.8 WHERE id = ?",
|
||||
(fact[0],)
|
||||
)
|
||||
conn.commit()
|
||||
print(f"[OK] 已降低 {len(expired)} 条过期事实的置信度")
|
||||
|
||||
conn.close()
|
||||
|
||||
|
||||
def archive_cold_data(root: str, dry_run: bool = False):
|
||||
"""归档冷数据（超过 180 天未被检索的数据）"""
|
||||
archive_dir = Path(root) / "03_knowledge" / "90_archive" / "deprecated"
|
||||
|
||||
# 检查 03_knowledge 下的文件修改时间
|
||||
knowledge_dir = Path(root) / "03_knowledge"
|
||||
cutoff = datetime.now() - timedelta(days=180)
|
||||
cold_files = []
|
||||
|
||||
for md_file in knowledge_dir.rglob("*.md"):
|
||||
if "99_system" in str(md_file) or "90_archive" in str(md_file):
|
||||
continue
|
||||
mtime = datetime.fromtimestamp(md_file.stat().st_mtime)
|
||||
if mtime < cutoff:
|
||||
cold_files.append(md_file)
|
||||
|
||||
if cold_files:
|
||||
print(f"[INFO] 发现 {len(cold_files)} 个冷知识文件:")
|
||||
for f in cold_files:
|
||||
print(f" - {f.relative_to(knowledge_dir)}")
|
||||
if not dry_run:
|
||||
print("[INFO] 冷数据归档需要确认，请手动处理或通过 kb_manager 技能执行")
|
||||
else:
|
||||
print("[INFO] 无冷数据需要归档")
|
||||
|
||||
|
||||
def resolve_conflicts(root: str, dry_run: bool = False):
|
||||
"""消解记忆冲突"""
|
||||
log_path = Path(root) / "04_memory" / "logs" / "conflicts.log"
|
||||
if not log_path.exists() or log_path.stat().st_size == 0:
|
||||
print("[INFO] 无待消解冲突")
|
||||
return
|
||||
|
||||
with open(log_path, "r", encoding="utf-8") as f:
|
||||
conflicts = f.readlines()
|
||||
|
||||
if conflicts:
|
||||
print(f"[INFO] 发现 {len(conflicts)} 条冲突记录:")
|
||||
for c in conflicts[:10]: # 只显示前 10 条
|
||||
print(f" - {c.strip()}")
|
||||
if len(conflicts) > 10:
|
||||
print(f" ... 还有 {len(conflicts) - 10} 条")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = os.path.expanduser(args.root)
|
||||
dry_run = args.dry_run
|
||||
|
||||
print("=" * 50)
|
||||
print("AgentOS 记忆清理")
|
||||
if dry_run:
|
||||
print("[DRY-RUN 模式：仅预览，不执行]")
|
||||
print("=" * 50)
|
||||
|
||||
cleanup_expired_facts(root, dry_run)
|
||||
archive_cold_data(root, dry_run)
|
||||
resolve_conflicts(root, dry_run)
|
||||
|
||||
print()
|
||||
print("[OK] 记忆清理完成")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,324 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
memory_extractor.py — 记忆固化提取器
|
||||
|
||||
从本机 L2 事实库（facts.db）和每日摘要中，检测满足固化条件的记忆，
|
||||
生成标准 MD 知识卡并推送到协同收件箱 inbox/memory/。
|
||||
|
||||
固化条件（满足任意一条即触发）：
|
||||
1. 用户主动标记 --solidify <fact_id>
|
||||
2. 高频引用：同一主谓对（subject+predicate）在 N 天内出现 >= threshold 次
|
||||
3. 高置信度：confidence >= 0.85 且 version >= 2（被多次确认未被推翻）
|
||||
|
||||
使用：
|
||||
python3 memory_extractor.py --root ~/workbuddy-agent-os/agent-sync --local ~/workbuddy-agent-os/agent-local
|
||||
python3 memory_extractor.py --root ~/workbuddy-agent-os/agent-sync --local ~/workbuddy-agent-os/agent-local --solidify FACT-2026-0001
|
||||
python3 memory_extractor.py --root ~/workbuddy-agent-os/agent-sync --local ~/workbuddy-agent-os/agent-local --dry-run
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# ─── 固化阈值配置 ─────────────────────────────────────────────────
|
||||
DEFAULT_FREQ_DAYS = 7 # 高频检测窗口（天）
|
||||
DEFAULT_FREQ_THRESHOLD = 3 # 窗口内出现次数阈值
|
||||
DEFAULT_CONFIDENCE_MIN = 0.85 # 高置信度阈值
|
||||
DEFAULT_VERSION_MIN = 2 # 最小版本数（被多次确认）
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 记忆固化提取器")
|
||||
parser.add_argument("--root", default="~/workbuddy-agent-os/agent-sync", help="agent-os 根目录")
|
||||
parser.add_argument("--local", default="~/workbuddy-agent-os/agent-local", help="~/workbuddy-agent-os/agent-local")
|
||||
parser.add_argument("--solidify", default=None, help="强制固化指定 fact_id")
|
||||
parser.add_argument("--dry-run", action="store_true", help="预览模式，不写文件")
|
||||
parser.add_argument("--freq-days", type=int, default=DEFAULT_FREQ_DAYS)
|
||||
parser.add_argument("--freq-threshold", type=int, default=DEFAULT_FREQ_THRESHOLD)
|
||||
parser.add_argument("--confidence-min", type=float, default=DEFAULT_CONFIDENCE_MIN)
|
||||
parser.add_argument("--version-min", type=int, default=DEFAULT_VERSION_MIN)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def open_facts_db(local_root: Path):
|
||||
"""打开本机 L2 事实库"""
|
||||
# L2 事实库路径（daily_digest.py 写入的路径）
|
||||
db_path = local_root / "memory" / "raw" / "04_memory" / "long_term" / "facts.db"
|
||||
if not db_path.exists():
|
||||
print(f"[WARN] 事实库不存在: {db_path}")
|
||||
return None, None
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn, conn.cursor()
|
||||
|
||||
|
||||
def detect_high_frequency(cursor, days: int, threshold: int) -> list:
|
||||
"""
|
||||
检测高频引用事实：
|
||||
在近 N 天内，subject+predicate 组合出现 >= threshold 次的最新事实
|
||||
"""
|
||||
since = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%d")
|
||||
cursor.execute("""
|
||||
SELECT subject, predicate, COUNT(*) as cnt,
|
||||
MAX(confidence) as max_conf,
|
||||
MAX(date_modified) as latest_date
|
||||
FROM facts
|
||||
WHERE date_created >= ? AND superseded_by = ''
|
||||
GROUP BY subject, predicate
|
||||
HAVING cnt >= ?
|
||||
ORDER BY cnt DESC
|
||||
""", (since, threshold))
|
||||
groups = cursor.fetchall()
|
||||
|
||||
results = []
|
||||
for row in groups:
|
||||
# 取该组最新/最高置信度的事实记录
|
||||
cursor.execute("""
|
||||
SELECT * FROM facts
|
||||
WHERE subject = ? AND predicate = ? AND superseded_by = ''
|
||||
ORDER BY confidence DESC, version DESC
|
||||
LIMIT 1
|
||||
""", (row["subject"], row["predicate"]))
|
||||
fact = cursor.fetchone()
|
||||
if fact:
|
||||
results.append(dict(fact) | {"solidify_reason": f"高频引用（{row['cnt']}次/{days}天）"})
|
||||
return results
|
||||
|
||||
|
||||
def detect_high_confidence(cursor, conf_min: float, version_min: int) -> list:
|
||||
"""检测高置信度且多次确认的事实"""
|
||||
cursor.execute("""
|
||||
SELECT * FROM facts
|
||||
WHERE confidence >= ? AND version >= ? AND superseded_by = ''
|
||||
ORDER BY confidence DESC, version DESC
|
||||
""", (conf_min, version_min))
|
||||
rows = cursor.fetchall()
|
||||
return [dict(r) | {"solidify_reason": f"高置信度({r['confidence']:.2f})·已确认{r['version']}次"} for r in rows]
|
||||
|
||||
|
||||
def get_fact_by_id(cursor, fact_id: str) -> dict:
|
||||
"""按ID获取指定事实"""
|
||||
cursor.execute("SELECT * FROM facts WHERE id = ?", (fact_id,))
|
||||
row = cursor.fetchone()
|
||||
return dict(row) | {"solidify_reason": "用户主动固化"} if row else None
|
||||
|
||||
|
||||
def already_solidified(inbox_memory_dir: Path, fact_id: str) -> bool:
|
||||
"""检查该事实是否已被推送到inbox（避免重复）"""
|
||||
for md in inbox_memory_dir.glob("*.md"):
|
||||
content = md.read_text(encoding="utf-8", errors="ignore")[:500]
|
||||
if fact_id in content:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def generate_memory_card(fact: dict, today: str) -> tuple[str, str]:
|
||||
"""
|
||||
生成记忆固化知识卡 MD 内容
|
||||
|
||||
返回: (filename, content)
|
||||
"""
|
||||
fact_id = fact.get("id", "")
|
||||
subject = fact.get("subject", "unknown")
|
||||
predicate = fact.get("predicate", "")
|
||||
obj = fact.get("object", "")
|
||||
domain = fact.get("domain", "personal-management")
|
||||
nature = fact.get("nature", "fact")
|
||||
confidence = fact.get("confidence", 0.7)
|
||||
source = fact.get("source", "memory_system")
|
||||
reason = fact.get("solidify_reason", "")
|
||||
version = fact.get("version", 1)
|
||||
date_created = fact.get("date_created", today)
|
||||
|
||||
# 生成知识卡 ID（MEM前缀区别于KB）
|
||||
mem_id = f"MEM-{today.replace('-', '')}-{fact_id[-4:] if len(fact_id) >= 4 else '0001'}"
|
||||
|
||||
# 标题：用 predicate + 前40字
|
||||
title = f"{predicate}: {obj[:40]}{'...' if len(obj) > 40 else ''}"
|
||||
|
||||
frontmatter = f"""---
|
||||
id: {mem_id}
|
||||
source_fact_id: {fact_id}
|
||||
title: "{title}"
|
||||
nature: {nature}
|
||||
domain: [{domain}]
|
||||
confidence: {confidence}
|
||||
source: "{source}"
|
||||
source_type: memory_solidification
|
||||
solidify_reason: "{reason}"
|
||||
subject: "{subject}"
|
||||
predicate: "{predicate}"
|
||||
version: {version}
|
||||
date_created: {date_created}
|
||||
date_solidified: {today}
|
||||
status: draft
|
||||
track: memory
|
||||
---"""
|
||||
|
||||
body = f"""# {title}
|
||||
|
||||
**来源**：本机记忆系统（自动固化）
|
||||
**固化原因**：{reason}
|
||||
**主体**：{subject}
|
||||
**谓语**：{predicate}
|
||||
**置信度**：{confidence:.2f}
|
||||
**原始事实ID**：`{fact_id}`
|
||||
|
||||
---
|
||||
|
||||
## 内容
|
||||
|
||||
{obj}
|
||||
|
||||
---
|
||||
|
||||
> *本卡片由 memory_extractor.py 自动生成，请主机审核后归档入知识库。*
|
||||
"""
|
||||
|
||||
safe_pred = re.sub(r'[^\w\u4e00-\u9fff-]', '_', predicate)[:20]
|
||||
filename = f"MEM-{today.replace('-', '')}-{safe_pred}-{fact_id[-6:]}.md"
|
||||
|
||||
return filename, frontmatter + "\n" + body
|
||||
|
||||
|
||||
def write_to_inbox(inbox_memory_dir: Path, filename: str, content: str, dry_run: bool) -> Path:
|
||||
"""写入协同 inbox/memory/ 分区"""
|
||||
inbox_memory_dir.mkdir(parents=True, exist_ok=True)
|
||||
target = inbox_memory_dir / filename
|
||||
if not dry_run:
|
||||
target.write_text(content, encoding="utf-8")
|
||||
return target
|
||||
|
||||
|
||||
def mark_solidified(cursor, conn, fact_id: str, dry_run: bool):
|
||||
"""在 facts.db 中标记该事实已固化（用 superseded_by 记录固化标记）"""
|
||||
if dry_run:
|
||||
return
|
||||
cursor.execute(
|
||||
"UPDATE facts SET superseded_by = ? WHERE id = ?",
|
||||
(f"SOLIDIFIED:{datetime.now().strftime('%Y-%m-%d')}", fact_id)
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def write_log(root: Path, log_entries: list):
|
||||
"""记录固化日志"""
|
||||
log_dir = root / "04_memory" / "logs"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_file = log_dir / "memory_solidification.log"
|
||||
with open(log_file, "a", encoding="utf-8") as f:
|
||||
for entry in log_entries:
|
||||
f.write(f"{entry}\n")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = Path(args.root).expanduser()
|
||||
local_root = Path(args.local).expanduser()
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
# 协同 inbox/memory/ 路径
|
||||
inbox_memory_dir = root / "03_knowledge" / "00_stream" / "inbox" / "memory"
|
||||
|
||||
print("=" * 60)
|
||||
print("AgentOS 记忆固化提取器")
|
||||
print(f" root: {root}")
|
||||
print(f" local: {local_root}")
|
||||
print(f" inbox: {inbox_memory_dir}")
|
||||
if args.dry_run:
|
||||
print(" [DRY-RUN 模式，不写入文件]")
|
||||
print("=" * 60)
|
||||
|
||||
conn, cursor = open_facts_db(local_root)
|
||||
if conn is None:
|
||||
print("[ERROR] 无法打开事实库，退出")
|
||||
sys.exit(1)
|
||||
|
||||
# ── 收集待固化事实 ────────────────────────────────────────────
|
||||
candidates = []
|
||||
seen_ids = set()
|
||||
|
||||
if args.solidify:
|
||||
# 模式1：用户主动指定
|
||||
fact = get_fact_by_id(cursor, args.solidify)
|
||||
if fact:
|
||||
candidates.append(fact)
|
||||
seen_ids.add(fact["id"])
|
||||
print(f"[INFO] 用户指定固化: {args.solidify}")
|
||||
else:
|
||||
print(f"[ERROR] 未找到事实: {args.solidify}")
|
||||
conn.close()
|
||||
sys.exit(1)
|
||||
else:
|
||||
# 模式2：高频检测
|
||||
freq_facts = detect_high_frequency(cursor, args.freq_days, args.freq_threshold)
|
||||
for f in freq_facts:
|
||||
if f["id"] not in seen_ids:
|
||||
candidates.append(f)
|
||||
seen_ids.add(f["id"])
|
||||
|
||||
# 模式3：高置信度检测
|
||||
conf_facts = detect_high_confidence(cursor, args.confidence_min, args.version_min)
|
||||
for f in conf_facts:
|
||||
if f["id"] not in seen_ids:
|
||||
candidates.append(f)
|
||||
seen_ids.add(f["id"])
|
||||
|
||||
print(f"[INFO] 检测到 {len(candidates)} 条候选固化事实")
|
||||
|
||||
if not candidates:
|
||||
print("[INFO] 无需固化，退出")
|
||||
conn.close()
|
||||
return
|
||||
|
||||
# ── 过滤已固化 ────────────────────────────────────────────────
|
||||
new_candidates = []
|
||||
for fact in candidates:
|
||||
if already_solidified(inbox_memory_dir, fact["id"]):
|
||||
print(f" ⏭ 已固化跳过: {fact['id']}")
|
||||
else:
|
||||
new_candidates.append(fact)
|
||||
|
||||
print(f"[INFO] 新增固化: {len(new_candidates)} 条")
|
||||
|
||||
if not new_candidates:
|
||||
conn.close()
|
||||
return
|
||||
|
||||
# ── 生成并写入知识卡 ──────────────────────────────────────────
|
||||
log_entries = []
|
||||
pushed = 0
|
||||
|
||||
for fact in new_candidates:
|
||||
filename, content = generate_memory_card(fact, today)
|
||||
target = write_to_inbox(inbox_memory_dir, filename, content, args.dry_run)
|
||||
|
||||
action = "[DRY-RUN]" if args.dry_run else "[写入]"
|
||||
print(f" {action} {filename}")
|
||||
print(f" 原因: {fact['solidify_reason']}")
|
||||
|
||||
# 标记已固化（仅非dry-run）
|
||||
mark_solidified(cursor, conn, fact["id"], args.dry_run)
|
||||
|
||||
log_entries.append(
|
||||
f"{datetime.now().isoformat()} | {fact['id']} | {fact['solidify_reason']} | {filename}"
|
||||
)
|
||||
pushed += 1
|
||||
|
||||
# ── 记录日志 ──────────────────────────────────────────────────
|
||||
if not args.dry_run and log_entries:
|
||||
write_log(root, log_entries)
|
||||
|
||||
conn.close()
|
||||
|
||||
print(f"\n[完成] 固化推送 {pushed} 条 → {inbox_memory_dir}")
|
||||
print(" 主维护机可通过 inbox_refine.py 将这些内容归档入知识库。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,664 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AgentOS 语义检索模块
|
||||
用途：BM25 + 向量语义混合检索，为记忆体提供语义搜索能力
|
||||
|
||||
架构：
|
||||
L1 BM25 检索（关键词匹配，基于 keyword_index.json）
|
||||
+ L1_vec 向量检索（语义匹配，基于 ChromaDB + oMLX embedding）
|
||||
→ Reciprocal Rank Fusion (RRF) 加权融合
|
||||
→ 返回排序后的 fact_id 列表
|
||||
|
||||
使用：
|
||||
# 检索
|
||||
python3 semantic_search.py search --root ~/workbuddy-agent-os/agent-sync --query "用户偏好什么编程语言" --top-k 5
|
||||
|
||||
# 向量化单条
|
||||
python3 semantic_search.py embed --root ~/workbuddy-agent-os/agent-sync --text "测试文本"
|
||||
|
||||
# 回填所有历史数据
|
||||
python3 semantic_search.py backfill --root ~/workbuddy-agent-os/agent-sync
|
||||
|
||||
# 清除向量库并重建
|
||||
python3 semantic_search.py rebuild --root ~/workbuddy-agent-os/agent-sync
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# ─── 默认配置 ──────────────────────────────────────────────────
|
||||
EMBEDDING_API_URL = "http://localhost:8000/v1/embeddings"
|
||||
EMBEDDING_MODEL = "Qwen3-Embedding-0.6B"
|
||||
EMBEDDING_API_KEY = "omlx"
|
||||
EMBEDDING_DIM = 1024
|
||||
CHROMA_COLLECTION = "agentos_memories"
|
||||
BM25_WEIGHT = 0.4 # BM25 在融合中的权重
|
||||
VECTOR_WEIGHT = 0.6 # 向量相似度在融合中的权重
|
||||
RRF_K = 60 # RRF 常数，越大排名差异的影响越小
|
||||
|
||||
|
||||
# ─── Embedding API 调用 ─────────────────────────────────────────
|
||||
def get_embeddings(texts, batch_size=5, retry=3, delay=1.0):
|
||||
"""
|
||||
调用 oMLX embedding API 获取文本向量。
|
||||
|
||||
Args:
|
||||
texts: 字符串或字符串列表
|
||||
batch_size: 每批请求的文本数量（oMLX 对大批量可能超时）
|
||||
retry: 失败重试次数
|
||||
delay: 批次间延迟（秒）
|
||||
|
||||
Returns:
|
||||
向量列表（与输入顺序对应）
|
||||
"""
|
||||
import requests
|
||||
|
||||
if isinstance(texts, str):
|
||||
texts = [texts]
|
||||
|
||||
all_embeddings = []
|
||||
for i in range(0, len(texts), batch_size):
|
||||
batch = texts[i:i + batch_size]
|
||||
for attempt in range(retry):
|
||||
try:
|
||||
resp = requests.post(
|
||||
EMBEDDING_API_URL,
|
||||
json={"model": EMBEDDING_MODEL, "input": batch},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {EMBEDDING_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# 按 index 排序确保顺序一致
|
||||
sorted_data = sorted(data["data"], key=lambda x: x["index"])
|
||||
batch_embeddings = [item["embedding"] for item in sorted_data]
|
||||
all_embeddings.extend(batch_embeddings)
|
||||
break
|
||||
except Exception as e:
|
||||
if attempt < retry - 1:
|
||||
time.sleep(delay * (attempt + 1))
|
||||
else:
|
||||
raise RuntimeError(f"Embedding API 调用失败（批次 {i//batch_size + 1}）: {e}")
|
||||
|
||||
if i + batch_size < len(texts):
|
||||
time.sleep(delay)
|
||||
|
||||
return all_embeddings
|
||||
|
||||
|
||||
# ─── ChromaDB 向量存储 ─────────────────────────────────────────
|
||||
class VectorStore:
|
||||
"""基于 ChromaDB 的向量存储，持久化到磁盘"""
|
||||
|
||||
def __init__(self, persist_dir: str):
|
||||
import chromadb
|
||||
self.persist_dir = Path(persist_dir)
|
||||
self.persist_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.client = chromadb.PersistentClient(path=str(self.persist_dir))
|
||||
self.collection = self.client.get_or_create_collection(
|
||||
name=CHROMA_COLLECTION,
|
||||
metadata={"hnsw:space": "cosine"}
|
||||
)
|
||||
|
||||
def upsert(self, fact_id: str, text: str, metadata: dict, embedding: list = None):
|
||||
"""添加或更新一条向量记录"""
|
||||
if embedding is None:
|
||||
embeddings = get_embeddings([text])
|
||||
embedding = embeddings[0]
|
||||
|
||||
self.collection.upsert(
|
||||
ids=[fact_id],
|
||||
documents=[text],
|
||||
embeddings=[embedding],
|
||||
metadatas=[metadata]
|
||||
)
|
||||
|
||||
def upsert_batch(self, items: list):
|
||||
"""
|
||||
批量添加/更新向量记录。
|
||||
|
||||
Args:
|
||||
items: [{"fact_id": str, "text": str, "metadata": dict}, ...]
|
||||
"""
|
||||
if not items:
|
||||
return
|
||||
|
||||
ids = [item["fact_id"] for item in items]
|
||||
texts = [item["text"] for item in items]
|
||||
metadatas = [item["metadata"] for item in items]
|
||||
|
||||
# 分批 embedding（避免 API 超时）
|
||||
batch_size = 5
|
||||
all_embeddings = []
|
||||
for i in range(0, len(texts), batch_size):
|
||||
batch_emb = get_embeddings(texts[i:i + batch_size])
|
||||
all_embeddings.extend(batch_emb)
|
||||
if i + batch_size < len(texts):
|
||||
time.sleep(0.5)
|
||||
|
||||
# 分批 upsert（ChromaDB 建议每批 < 100）
|
||||
upsert_batch_size = 50
|
||||
for i in range(0, len(ids), upsert_batch_size):
|
||||
self.collection.upsert(
|
||||
ids=ids[i:i + upsert_batch_size],
|
||||
documents=texts[i:i + upsert_batch_size],
|
||||
embeddings=all_embeddings[i:i + upsert_batch_size],
|
||||
metadatas=metadatas[i:i + upsert_batch_size]
|
||||
)
|
||||
|
||||
def search(self, query: str, top_k: int = 10, query_embedding: list = None) -> list:
|
||||
"""
|
||||
向量相似度搜索。
|
||||
|
||||
Returns:
|
||||
[{"fact_id": str, "score": float, "metadata": dict}, ...]
|
||||
"""
|
||||
if query_embedding is None:
|
||||
query_embedding = get_embeddings([query])[0]
|
||||
|
||||
results = self.collection.query(
|
||||
query_embeddings=[query_embedding],
|
||||
n_results=min(top_k, self.collection.count()),
|
||||
include=["metadatas", "documents", "distances"]
|
||||
)
|
||||
|
||||
items = []
|
||||
if results["ids"] and results["ids"][0]:
|
||||
for i, fid in enumerate(results["ids"][0]):
|
||||
distance = results["distances"][0][i] if results["distances"] else 0
|
||||
# ChromaDB cosine distance → 相似度分数（1 - distance）
|
||||
score = 1.0 - distance
|
||||
items.append({
|
||||
"fact_id": fid,
|
||||
"score": max(0, score),
|
||||
"metadata": results["metadatas"][0][i] if results["metadatas"] else {},
|
||||
"document": results["documents"][0][i] if results["documents"] else ""
|
||||
})
|
||||
return items
|
||||
|
||||
def delete(self, fact_id: str):
|
||||
"""删除一条向量记录"""
|
||||
self.collection.delete(ids=[fact_id])
|
||||
|
||||
def count(self) -> int:
|
||||
"""返回向量总数"""
|
||||
return self.collection.count()
|
||||
|
||||
def get_all_ids(self) -> list:
|
||||
"""获取所有 fact_id"""
|
||||
result = self.collection.get(include=[])
|
||||
return result["ids"] if result["ids"] else []
|
||||
|
||||
|
||||
# ─── BM25 关键词检索 ────────────────────────────────────────────
|
||||
class BM25Searcher:
|
||||
"""基于 L1 keyword_index.json 的 BM25 检索"""
|
||||
|
||||
def __init__(self, index_path: str):
|
||||
self.index_path = Path(index_path)
|
||||
self._build_index()
|
||||
|
||||
def _build_index(self):
|
||||
"""构建 BM25 索引"""
|
||||
from rank_bm25 import BM25Okapi
|
||||
|
||||
if not self.index_path.exists():
|
||||
self.entries = []
|
||||
self.bm25 = None
|
||||
return
|
||||
|
||||
with open(self.index_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
self.entries = data.get("entries", [])
|
||||
if not self.entries:
|
||||
self.bm25 = None
|
||||
return
|
||||
|
||||
# 用 keywords + summary 构建 BM25 语料库
|
||||
self.corpus = []
|
||||
for entry in self.entries:
|
||||
# 合并关键词和摘要作为 BM25 文档
|
||||
doc_words = entry.get("keywords", []) + [entry.get("summary", "")]
|
||||
# 中文分词：按 2-4 字切分 + 英文按空格分
|
||||
tokens = []
|
||||
for w in doc_words:
|
||||
if re.match(r'^[\u4e00-\u9fff]+$', w):
|
||||
# 中文：2-4 字 ngram
|
||||
for n in range(2, min(5, len(w) + 1)):
|
||||
tokens.extend([w[i:i+n] for i in range(len(w) - n + 1)])
|
||||
tokens.append(w) # 也保留完整词
|
||||
else:
|
||||
tokens.extend(w.lower().split())
|
||||
self.corpus.append(tokens)
|
||||
|
||||
self.bm25 = BM25Okapi(self.corpus)
|
||||
|
||||
def search(self, query: str, top_k: int = 10) -> list:
|
||||
"""
|
||||
BM25 关键词搜索。
|
||||
|
||||
Returns:
|
||||
[{"fact_id": str, "score": float, "keywords": list}, ...]
|
||||
"""
|
||||
if not self.bm25:
|
||||
return []
|
||||
|
||||
# 查询分词
|
||||
query_tokens = []
|
||||
for w in re.findall(r'[\u4e00-\u9fff]{2,}|[a-zA-Z]{2,}', query):
|
||||
if re.match(r'^[\u4e00-\u9fff]+$', w):
|
||||
for n in range(2, min(5, len(w) + 1)):
|
||||
query_tokens.extend([w[i:i+n] for i in range(len(w) - n + 1)])
|
||||
query_tokens.append(w)
|
||||
else:
|
||||
query_tokens.extend(w.lower().split())
|
||||
|
||||
if not query_tokens:
|
||||
return []
|
||||
|
||||
scores = self.bm25.get_scores(query_tokens)
|
||||
|
||||
# 取 top_k
|
||||
ranked = sorted(enumerate(scores), key=lambda x: x[1], reverse=True)[:top_k]
|
||||
results = []
|
||||
for idx, score in ranked:
|
||||
if score > 0:
|
||||
entry = self.entries[idx]
|
||||
results.append({
|
||||
"fact_id": entry.get("fact_id", ""),
|
||||
"score": float(score),
|
||||
"keywords": entry.get("keywords", []),
|
||||
"summary": entry.get("summary", ""),
|
||||
"domain": entry.get("domain", ""),
|
||||
"nature": entry.get("nature", "fact")
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# ─── Reciprocal Rank Fusion ─────────────────────────────────────
|
||||
def reciprocal_rank_fusion(bm25_results: list, vector_results: list,
|
||||
bm25_weight: float = BM25_WEIGHT,
|
||||
vector_weight: float = VECTOR_WEIGHT,
|
||||
rrf_k: int = RRF_K) -> list:
|
||||
"""
|
||||
使用 Reciprocal Rank Fusion 融合 BM25 和向量检索结果。
|
||||
|
||||
RRF 公式：score(d) = Σ 1/(k + rank(d))
|
||||
加权：score(d) = w_bm25 * rrf_bm25(d) + w_vec * rrf_vec(d)
|
||||
|
||||
Returns:
|
||||
[{"fact_id": str, "rrf_score": float, "bm25_rank": int, "vec_rank": int,
|
||||
"bm25_score": float, "vec_score": float}, ...]
|
||||
"""
|
||||
fact_scores = defaultdict(lambda: {
|
||||
"rrf_bm25": 0.0, "rrf_vec": 0.0,
|
||||
"bm25_rank": None, "vec_rank": None,
|
||||
"bm25_score": 0.0, "vec_score": 0.0,
|
||||
"summary": "", "domain": "", "nature": ""
|
||||
})
|
||||
|
||||
# BM25 排名贡献
|
||||
for rank, item in enumerate(bm25_results, 1):
|
||||
fid = item["fact_id"]
|
||||
fact_scores[fid]["rrf_bm25"] = 1.0 / (rrf_k + rank)
|
||||
fact_scores[fid]["bm25_rank"] = rank
|
||||
fact_scores[fid]["bm25_score"] = item["score"]
|
||||
fact_scores[fid]["summary"] = item.get("summary", "")
|
||||
fact_scores[fid]["domain"] = item.get("domain", "")
|
||||
fact_scores[fid]["nature"] = item.get("nature", "fact")
|
||||
|
||||
# 向量排名贡献
|
||||
for rank, item in enumerate(vector_results, 1):
|
||||
fid = item["fact_id"]
|
||||
fact_scores[fid]["rrf_vec"] = 1.0 / (rrf_k + rank)
|
||||
fact_scores[fid]["vec_rank"] = rank
|
||||
fact_scores[fid]["vec_score"] = item["score"]
|
||||
if item.get("metadata"):
|
||||
fact_scores[fid]["summary"] = item["metadata"].get("summary", fact_scores[fid]["summary"])
|
||||
fact_scores[fid]["domain"] = item["metadata"].get("domain", fact_scores[fid]["domain"])
|
||||
fact_scores[fid]["nature"] = item["metadata"].get("nature", fact_scores[fid]["nature"])
|
||||
if item.get("document") and not fact_scores[fid]["summary"]:
|
||||
fact_scores[fid]["summary"] = item["document"][:120]
|
||||
|
||||
# 加权融合 + 排序
|
||||
fused = []
|
||||
for fid, scores in fact_scores.items():
|
||||
rrf_score = bm25_weight * scores["rrf_bm25"] + vector_weight * scores["rrf_vec"]
|
||||
fused.append({
|
||||
"fact_id": fid,
|
||||
"rrf_score": rrf_score,
|
||||
"bm25_rank": scores["bm25_rank"],
|
||||
"vec_rank": scores["vec_rank"],
|
||||
"bm25_score": scores["bm25_score"],
|
||||
"vec_score": scores["vec_score"],
|
||||
"summary": scores["summary"],
|
||||
"domain": scores["domain"],
|
||||
"nature": scores["nature"]
|
||||
})
|
||||
|
||||
fused.sort(key=lambda x: x["rrf_score"], reverse=True)
|
||||
return fused
|
||||
|
||||
|
||||
# ─── 统一检索入口 ──────────────────────────────────────────────
|
||||
class MemorySearcher:
|
||||
"""
|
||||
AgentOS 记忆检索器 — BM25 + 向量语义混合检索
|
||||
|
||||
Usage:
|
||||
searcher = MemorySearcher("/path/to/agent-os")
|
||||
results = searcher.search("用户偏好什么编程语言", top_k=5)
|
||||
"""
|
||||
|
||||
def __init__(self, root: str):
|
||||
self.root = Path(root)
|
||||
self.vector_store = VectorStore(str(self.root / "04_memory" / "vector_db" / "chroma"))
|
||||
self.bm25_searcher = BM25Searcher(str(self.root / "04_memory" / "vector_db" / "keyword_index.json"))
|
||||
self.db_path = self.root / "04_memory" / "long_term" / "facts.db"
|
||||
|
||||
def search(self, query: str, top_k: int = 10,
|
||||
bm25_weight: float = BM25_WEIGHT,
|
||||
vector_weight: float = VECTOR_WEIGHT) -> dict:
|
||||
"""
|
||||
混合检索入口。
|
||||
|
||||
Args:
|
||||
query: 检索查询文本
|
||||
top_k: 返回结果数量
|
||||
bm25_weight: BM25 权重（0-1）
|
||||
vector_weight: 向量权重（0-1）
|
||||
|
||||
Returns:
|
||||
{
|
||||
"query": str,
|
||||
"total_vector": int, # 向量库总数
|
||||
"results": [
|
||||
{
|
||||
"fact_id": str,
|
||||
"rrf_score": float,
|
||||
"bm25_rank": int|null,
|
||||
"vec_rank": int|null,
|
||||
"confidence": float, # L2 置信度
|
||||
"subject": str,
|
||||
"predicate": str,
|
||||
"object": str, # L2 完整事实
|
||||
"summary": str, # L1 摘要
|
||||
"domain": str,
|
||||
"nature": str
|
||||
}, ...
|
||||
]
|
||||
}
|
||||
"""
|
||||
# 并行执行两种检索
|
||||
bm25_results = self.bm25_searcher.search(query, top_k=top_k * 2)
|
||||
vector_results = self.vector_store.search(query, top_k=top_k * 2)
|
||||
|
||||
# RRF 融合
|
||||
fused = reciprocal_rank_fusion(
|
||||
bm25_results, vector_results,
|
||||
bm25_weight=bm25_weight,
|
||||
vector_weight=vector_weight
|
||||
)
|
||||
|
||||
# 截取 top_k
|
||||
fused = fused[:top_k]
|
||||
|
||||
# 从 L2 获取完整事实信息
|
||||
results = self._enrich_with_l2(fused)
|
||||
|
||||
return {
|
||||
"query": query,
|
||||
"total_vector": self.vector_store.count(),
|
||||
"bm25_results": len(bm25_results),
|
||||
"vector_results": len(vector_results),
|
||||
"results": results
|
||||
}
|
||||
|
||||
def _enrich_with_l2(self, fused_results: list) -> list:
|
||||
"""从 L2 facts.db 补充完整事实信息"""
|
||||
if not self.db_path.exists():
|
||||
return fused_results
|
||||
|
||||
conn = sqlite3.connect(str(self.db_path))
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.cursor()
|
||||
|
||||
enriched = []
|
||||
for item in fused_results:
|
||||
fid = item["fact_id"]
|
||||
row = cursor.execute(
|
||||
"SELECT subject, predicate, object, confidence, nature, domain FROM facts WHERE id = ?",
|
||||
(fid,)
|
||||
).fetchone()
|
||||
|
||||
if row:
|
||||
enriched.append({
|
||||
"fact_id": fid,
|
||||
"rrf_score": round(item["rrf_score"], 6),
|
||||
"bm25_rank": item["bm25_rank"],
|
||||
"vec_rank": item["vec_rank"],
|
||||
"confidence": row["confidence"],
|
||||
"subject": row["subject"],
|
||||
"predicate": row["predicate"],
|
||||
"object": row["object"],
|
||||
"summary": item.get("summary", row["object"][:120]),
|
||||
"domain": row["domain"],
|
||||
"nature": row["nature"]
|
||||
})
|
||||
else:
|
||||
# L2 中不存在（不应该发生，但防御性处理）
|
||||
enriched.append({
|
||||
"fact_id": fid,
|
||||
"rrf_score": round(item["rrf_score"], 6),
|
||||
"bm25_rank": item["bm25_rank"],
|
||||
"vec_rank": item["vec_rank"],
|
||||
"confidence": 0,
|
||||
"subject": "",
|
||||
"predicate": "",
|
||||
"object": item.get("summary", ""),
|
||||
"summary": item.get("summary", ""),
|
||||
"domain": item.get("domain", ""),
|
||||
"nature": item.get("nature", "")
|
||||
})
|
||||
|
||||
conn.close()
|
||||
return enriched
|
||||
|
||||
def add_fact(self, fact_id: str, text: str, metadata: dict):
|
||||
"""添加单条事实到向量库"""
|
||||
self.vector_store.upsert(fact_id, text, metadata)
|
||||
|
||||
def remove_fact(self, fact_id: str):
|
||||
"""从向量库删除一条事实"""
|
||||
self.vector_store.delete(fact_id)
|
||||
|
||||
def backfill_all(self, batch_size: int = 5, verbose: bool = True):
|
||||
"""
|
||||
回填所有 L2 事实到向量库。
|
||||
从 facts.db 读取所有事实，生成 embedding 并写入 ChromaDB。
|
||||
"""
|
||||
if not self.db_path.exists():
|
||||
print("[ERROR] facts.db 不存在")
|
||||
return 0
|
||||
|
||||
conn = sqlite3.connect(str(self.db_path))
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.cursor()
|
||||
|
||||
facts = cursor.execute(
|
||||
"SELECT id, subject, predicate, object, confidence, nature, domain, source, date_created FROM facts"
|
||||
).fetchall()
|
||||
conn.close()
|
||||
|
||||
if not facts:
|
||||
print("[INFO] facts.db 为空，无需回填")
|
||||
return 0
|
||||
|
||||
# 获取向量库已有 ID
|
||||
existing_ids = set(self.vector_store.get_all_ids())
|
||||
|
||||
# 构建待回填列表
|
||||
to_fill = []
|
||||
for fact in facts:
|
||||
if fact["id"] not in existing_ids:
|
||||
text = f"{fact['subject']} {fact['predicate']} {fact['object']}"
|
||||
metadata = {
|
||||
"summary": fact["object"][:120],
|
||||
"domain": fact["domain"],
|
||||
"nature": fact["nature"],
|
||||
"source": fact["source"],
|
||||
"date_created": fact["date_created"],
|
||||
"confidence": fact["confidence"]
|
||||
}
|
||||
to_fill.append({
|
||||
"fact_id": fact["id"],
|
||||
"text": text,
|
||||
"metadata": metadata
|
||||
})
|
||||
|
||||
if not to_fill:
|
||||
print(f"[INFO] 全部 {len(facts)} 条事实已在向量库中，无需回填")
|
||||
return 0
|
||||
|
||||
if verbose:
|
||||
print(f"[INFO] 需要回填 {len(to_fill)} 条事实（总共 {len(facts)} 条）")
|
||||
|
||||
# 分批处理
|
||||
filled = 0
|
||||
for i in range(0, len(to_fill), batch_size):
|
||||
batch = to_fill[i:i + batch_size]
|
||||
try:
|
||||
self.vector_store.upsert_batch(batch)
|
||||
filled += len(batch)
|
||||
if verbose:
|
||||
pct = filled / len(to_fill) * 100
|
||||
print(f" [{pct:5.1f}%] {filled}/{len(to_fill)} 已回填")
|
||||
except Exception as e:
|
||||
print(f" [ERROR] 批次 {i//batch_size + 1} 失败: {e}")
|
||||
# 尝试逐条重试
|
||||
for item in batch:
|
||||
try:
|
||||
self.vector_store.upsert(
|
||||
item["fact_id"], item["text"],
|
||||
item["metadata"]
|
||||
)
|
||||
filled += 1
|
||||
if verbose:
|
||||
print(f" [RETRY] {item['fact_id']} OK")
|
||||
except Exception as e2:
|
||||
print(f" [FAIL] {item['fact_id']}: {e2}")
|
||||
|
||||
if verbose:
|
||||
print(f"[OK] 回填完成: {filled}/{len(to_fill)} 成功, 向量库总数: {self.vector_store.count()}")
|
||||
return filled
|
||||
|
||||
def rebuild(self):
|
||||
"""清除向量库并完全重建"""
|
||||
import chromadb
|
||||
persist_dir = str(self.root / "04_memory" / "vector_db" / "chroma")
|
||||
|
||||
# 删除旧数据库
|
||||
import shutil
|
||||
if os.path.exists(persist_dir):
|
||||
shutil.rmtree(persist_dir)
|
||||
print(f"[OK] 已清除旧向量库: {persist_dir}")
|
||||
|
||||
# 重新初始化
|
||||
self.vector_store = VectorStore(persist_dir)
|
||||
print("[OK] 向量库已重建")
|
||||
|
||||
# 回填
|
||||
return self.backfill_all()
|
||||
|
||||
|
||||
# ─── CLI 入口 ──────────────────────────────────────────────────
|
||||
def cmd_search(args):
|
||||
searcher = MemorySearcher(args.root)
|
||||
result = searcher.search(args.query, top_k=args.top_k)
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print(f"检索: {result['query']}")
|
||||
print(f"向量库总数: {result['total_vector']} | "
|
||||
f"BM25命中: {result['bm25_results']} | 向量命中: {result['vector_results']}")
|
||||
print(f"{'='*60}\n")
|
||||
|
||||
if not result["results"]:
|
||||
print("（无匹配结果）")
|
||||
return
|
||||
|
||||
for i, item in enumerate(result["results"], 1):
|
||||
bm25_r = f"BM25#{item['bm25_rank']}" if item["bm25_rank"] else "BM25#-"
|
||||
vec_r = f"VEC#{item['vec_rank']}" if item["vec_rank"] else "VEC#-"
|
||||
print(f"[{i}] {item['fact_id']} RRF={item['rrf_score']:.4f} conf={item['confidence']} "
|
||||
f"{bm25_r} {vec_r}")
|
||||
print(f" [{item['domain']}] {item['subject']} → {item['object'][:100]}")
|
||||
print()
|
||||
|
||||
|
||||
def cmd_embed(args):
|
||||
embeddings = get_embeddings([args.text])
|
||||
print(f"维度: {len(embeddings[0])}")
|
||||
print(f"前5值: {embeddings[0][:5]}")
|
||||
print(f"范数: {math.sqrt(sum(x*x for x in embeddings[0])):.4f}")
|
||||
|
||||
|
||||
def cmd_backfill(args):
|
||||
searcher = MemorySearcher(args.root)
|
||||
searcher.backfill_all()
|
||||
|
||||
|
||||
def cmd_rebuild(args):
|
||||
searcher = MemorySearcher(args.root)
|
||||
searcher.rebuild()
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 语义检索模块")
|
||||
parser.add_argument("--root", required=True, help="agent-os 根目录路径")
|
||||
subparsers = parser.add_subparsers(dest="command")
|
||||
|
||||
# search
|
||||
sp_search = subparsers.add_parser("search", help="混合检索")
|
||||
sp_search.add_argument("--query", required=True, help="检索查询")
|
||||
sp_search.add_argument("--top-k", type=int, default=5, help="返回结果数量")
|
||||
|
||||
# embed
|
||||
sp_embed = subparsers.add_parser("embed", help="测试 embedding")
|
||||
sp_embed.add_argument("--text", required=True, help="测试文本")
|
||||
|
||||
# backfill
|
||||
subparsers.add_parser("backfill", help="回填所有历史数据到向量库")
|
||||
|
||||
# rebuild
|
||||
subparsers.add_parser("rebuild", help="清除并重建向量库")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "search":
|
||||
cmd_search(args)
|
||||
elif args.command == "embed":
|
||||
cmd_embed(args)
|
||||
elif args.command == "backfill":
|
||||
cmd_backfill(args)
|
||||
elif args.command == "rebuild":
|
||||
cmd_rebuild(args)
|
||||
else:
|
||||
parser.print_help()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# 向量库增量更新脚本 (vector_backfill.sh v1.1)
|
||||
# 联邦系统定时任务 — 将 L2 新事实回填到 ChromaDB 向量索引
|
||||
# 依赖: oMLX macOS App + matrix Python venv
|
||||
# =============================================================================
|
||||
set -e
|
||||
|
||||
AGENT_SYNC="/Users/7kecheng/workbuddy-agent-os/agent-sync"
|
||||
PYTHON="/Users/7kecheng/.workbuddy/binaries/python/envs/matrix/bin/python3"
|
||||
SKILL_DIR="/Users/7kecheng/.workbuddy/skills/memory_manager"
|
||||
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
LOG_FILE="$AGENT_SYNC/04_memory/vector_db/backfill.log"
|
||||
|
||||
OMLX_APP="/Applications/oMLX.app"
|
||||
OMLX_PORT=8000
|
||||
OMLX_API_KEY="omlx"
|
||||
OMLX_CHECK_URL="http://localhost:${OMLX_PORT}/v1/models"
|
||||
OMLX_CHECK_HEADER="Authorization: Bearer ${OMLX_API_KEY}"
|
||||
OMLX_WAIT_MAX=60 # 最多等 60 秒（模型加载可能较慢）
|
||||
|
||||
echo "[${TIMESTAMP}] 🚀 向量库增量更新开始"
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# 1. 检查 oMLX 是否运行，未运行则自动启动
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
if curl -sf -H "$OMLX_CHECK_HEADER" "$OMLX_CHECK_URL" > /dev/null 2>&1; then
|
||||
echo "[${TIMESTAMP}] ✅ oMLX 运行中"
|
||||
else
|
||||
echo "[${TIMESTAMP}] 🚀 oMLX 未运行，正在启动..."
|
||||
open "$OMLX_APP"
|
||||
|
||||
# 等待就绪（最多 60s）
|
||||
for i in $(seq 1 $OMLX_WAIT_MAX); do
|
||||
sleep 1
|
||||
if curl -sf -H "$OMLX_CHECK_HEADER" "$OMLX_CHECK_URL" > /dev/null 2>&1; then
|
||||
echo "[${TIMESTAMP}] ✅ oMLX 启动完成（${i}s）"
|
||||
break
|
||||
fi
|
||||
if [ $i -eq $OMLX_WAIT_MAX ]; then
|
||||
echo "[${TIMESTAMP}] ❌ oMLX 启动超时（${OMLX_WAIT_MAX}s），放弃本次更新"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# 2. 统计更新前 chroma 文件数
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
BEFORE=0
|
||||
if [ -d "$AGENT_SYNC/04_memory/vector_db/chroma" ]; then
|
||||
BEFORE=$(find "$AGENT_SYNC/04_memory/vector_db/chroma" -name "*.chroma.sqlite3" 2>/dev/null | wc -l)
|
||||
fi
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# 3. 执行增量回填
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
echo "[${TIMESTAMP}] 🔄 执行增量回填..."
|
||||
cd "$SKILL_DIR"
|
||||
OUTPUT=$($PYTHON semantic_search.py --root "$AGENT_SYNC" backfill 2>&1)
|
||||
EXIT_CODE=$?
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# 4. 统计结果
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
SUCCESS=$(echo "$OUTPUT" | grep -c "\[OK\]" || true)
|
||||
FAIL=$(echo "$OUTPUT" | grep -c "\[FAIL\]" || true)
|
||||
echo "$OUTPUT" >> "$LOG_FILE"
|
||||
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo "[${TIMESTAMP}] ✅ 向量库更新完成 (成功: $SUCCESS, 失败: $FAIL)"
|
||||
echo "[${TIMESTAMP}] 更新前 chroma 文件数: $BEFORE"
|
||||
else
|
||||
echo "[${TIMESTAMP}] ❌ 向量库更新异常 (code=$EXIT_CODE)"
|
||||
fi
|
||||
|
||||
echo "[${TIMESTAMP}] 📝 日志已保存: $LOG_FILE"
|
||||
echo "[${TIMESTAMP}] 🏁 结束"
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"name": "memory_manager",
|
||||
"version": "1.2.0",
|
||||
"description": "智能体记忆管理技能——每日对话提炼、去重、冲突检测、版本管理",
|
||||
"dependencies": ["sqlite3"],
|
||||
"mcp_tools": []
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
name: peekaboo_controller
|
||||
version: 1.0.0
|
||||
description: Peekaboo v3 GUI 自动化——屏幕视觉识别+鼠标键盘操作
|
||||
triggers:
|
||||
- 截图
|
||||
- 屏幕
|
||||
- 点击
|
||||
- 桌面操作
|
||||
- 视觉识别
|
||||
- OCR
|
||||
- peekaboo
|
||||
- 自动化点击
|
||||
---
|
||||
|
||||
# Peekaboo Controller Skill
|
||||
|
||||
macOS 桌面 GUI 自动化。像素级视觉识别操控屏幕，不依赖 DOM/系统 API。
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
npm install -g @steipete/peekaboo # npm（推荐）
|
||||
# 验证权限
|
||||
peekaboo permissions status
|
||||
```
|
||||
|
||||
## MCP 配置（token 优化版）
|
||||
|
||||
已在 `01_core/mcp.json` 注册，带 token 节省参数：
|
||||
```json
|
||||
"peekaboo": {
|
||||
"command": "npx",
|
||||
"args": ["-y", "@steipete/peekaboo", "mcp", "--json", "--log-level", "error"]
|
||||
}
|
||||
```
|
||||
|
||||
- `--json`：输出机器可读 JSON，比默认文本小 60-80%
|
||||
- `--log-level error`：只输出错误，不输出 info/warning 日志
|
||||
|
||||
## 常用命令
|
||||
|
||||
```bash
|
||||
peekaboo image # 截图（自动保存）
|
||||
peekaboo list apps --json # JSON 格式输出（节省 token）
|
||||
peekaboo click "按钮" # 视觉点击
|
||||
peekaboo type "文字" # 输入
|
||||
```
|
||||
|
||||
## Token 节省技巧
|
||||
|
||||
- 命令后加 `--json`：输出量减少 60-80%
|
||||
- 截图时指定区域或窗口，避免全屏大图
|
||||
- 不需要时关闭 MCP：`peekaboo daemon stop`
|
||||
|
||||
## 使用策略（防滥用）
|
||||
|
||||
内置 `policy.py` 策略层，自动控制截图频率和用量：
|
||||
|
||||
| 规则 | 值 | 说明 |
|
||||
|------|----|------|
|
||||
| 截图冷却 | 5 秒 | 两次截图之间至少间隔 5 秒 |
|
||||
| 单会话上限 | 20 次 | 一次对话最多截 20 张 |
|
||||
| 缓存有效期 | 30 秒 | 同一应用 30 秒内复用缓存 |
|
||||
| 优先级顺序 | DOM → API → 文本 → **截图最后** | 截图是最后手段 |
|
||||
|
||||
```python
|
||||
# 使用策略层（推荐）
|
||||
from policy import screenshot, reset, status
|
||||
|
||||
path = screenshot("抖音") # 带冷却/上限检查
|
||||
path = screenshot("抖音", force=True) # 跳过检查（手动确认时用）
|
||||
reset() # 新对话前重置
|
||||
print(status()) # 查看用量
|
||||
```
|
||||
|
||||
> **原则**：能用 DOM 不用截图，能用 API 不用浏览器。
|
||||
> Peekaboo 是做**兜底**的，不是主路径。
|
||||
@@ -0,0 +1,100 @@
|
||||
"""
|
||||
Peekaboo 使用策略层 — 控制截图频率和用量，避免 token 浪费
|
||||
|
||||
核心原则：
|
||||
1. 能不用不用 — 优先用 DOM/API/文本方式
|
||||
2. 非必要不截图 — 截图是最后手段
|
||||
3. 截图有冷却 — 5秒内不重复截
|
||||
4. 单会话上限 — 最多截 20 次
|
||||
"""
|
||||
|
||||
import time
|
||||
import subprocess
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
PEEKABOO_BIN = os.path.expanduser("~/.workbuddy/binaries/node/versions/22.12.0/bin/peekaboo")
|
||||
if not os.path.exists(PEEKABOO_BIN):
|
||||
PEEKABOO_BIN = "peekaboo" # fallback to PATH
|
||||
|
||||
# ============ 使用策略常量 ============
|
||||
SCREENSHOT_COOLDOWN = 5 # 截图冷却（秒）
|
||||
MAX_SCREENSHOTS_PER_SESSION = 20 # 单次对话最大截图数
|
||||
CACHE_DIR = Path("/tmp/peekaboo_cache")
|
||||
CACHE_DIR.mkdir(exist_ok=True)
|
||||
|
||||
# ============ 运行时状态 ============
|
||||
_last_screenshot_time = 0.0
|
||||
_screenshot_count = 0
|
||||
_cache = {}
|
||||
|
||||
|
||||
def screenshot(app: str = "frontmost", force: bool = False) -> str:
|
||||
"""
|
||||
截图（带策略控制）
|
||||
|
||||
Args:
|
||||
app: 目标应用名或 frontmost
|
||||
force: 跳过冷却/上限检查
|
||||
|
||||
Returns:
|
||||
截图保存路径，或空字符串（被限制时）
|
||||
"""
|
||||
global _last_screenshot_time, _screenshot_count
|
||||
|
||||
now = time.time()
|
||||
|
||||
# 检查上限
|
||||
if not force and _screenshot_count >= MAX_SCREENSHOTS_PER_SESSION:
|
||||
print(" ⚠️ 截图已达上限 (%d次)，跳过" % MAX_SCREENSHOTS_PER_SESSION)
|
||||
return ""
|
||||
|
||||
# 检查冷却
|
||||
if not force and (now - _last_screenshot_time) < SCREENSHOT_COOLDOWN:
|
||||
remaining = SCREENSHOT_COOLDOWN - (now - _last_screenshot_time)
|
||||
print(" ⚠️ 截图冷却中，还剩 %.0f 秒" % remaining)
|
||||
return ""
|
||||
|
||||
# 检查缓存（同一应用 30 秒内不重复截）
|
||||
cache_key = f"screenshot:{app}"
|
||||
if cache_key in _cache:
|
||||
cached_time, cached_path = _cache[cache_key]
|
||||
if (now - cached_time) < 30:
|
||||
print(" ⏭ 使用缓存截图 (%s)" % cached_path)
|
||||
return cached_path
|
||||
|
||||
# 执行截图
|
||||
out_path = str(CACHE_DIR / f"screenshot_{int(now)}.png")
|
||||
env = {**os.environ, "NODE_OPTIONS": ""}
|
||||
for k in ["HTTP_PROXY", "HTTPS_PROXY"]:
|
||||
env.pop(k, None)
|
||||
|
||||
try:
|
||||
cmd = [PEEKABOO_BIN, "image", "--app", app, "--path", out_path, "--json", "--log-level", "error"]
|
||||
subprocess.run(cmd, capture_output=True, timeout=15, env=env)
|
||||
_last_screenshot_time = now
|
||||
_screenshot_count += 1
|
||||
_cache[cache_key] = (now, out_path)
|
||||
print(" 📸 截图已保存 (%s) [第%d次]" % (out_path, _screenshot_count))
|
||||
return out_path
|
||||
except Exception as e:
|
||||
print(" ⚠️ 截图失败: %s" % e)
|
||||
return ""
|
||||
|
||||
|
||||
def reset():
|
||||
"""重置截图计数器（新对话开始时调用）"""
|
||||
global _screenshot_count, _last_screenshot_time, _cache
|
||||
_screenshot_count = 0
|
||||
_last_screenshot_time = 0.0
|
||||
_cache = {}
|
||||
print(" 🔄 Peekaboo 策略已重置")
|
||||
|
||||
|
||||
def status() -> dict:
|
||||
"""查看当前策略状态"""
|
||||
return {
|
||||
"screenshots_taken": _screenshot_count,
|
||||
"max_screenshots": MAX_SCREENSHOTS_PER_SESSION,
|
||||
"cooldown_remaining": max(0, SCREENSHOT_COOLDOWN - (time.time() - _last_screenshot_time)),
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: sync_manager
|
||||
version: 1.1.0
|
||||
description: 同步管理技能——知识库备份、导出、迁移（同步由坚果云处理）
|
||||
triggers:
|
||||
- 备份知识库
|
||||
- 导出知识库
|
||||
- 同步状态
|
||||
- backup kb
|
||||
- 备份
|
||||
- 知识库备份
|
||||
---
|
||||
|
||||
# Sync Manager Skill（同步管理）
|
||||
|
||||
## 概述
|
||||
|
||||
管理 Obsidian 知识库的备份、导出和迁移。同步功能由坚果云处理，本技能不涉及。
|
||||
|
||||
## 核心能力
|
||||
|
||||
### 1. 备份知识库
|
||||
- 将 `03_knowledge/` 全量打包到 `04_memory/memory_backup/`
|
||||
- 命名：`kb_backup_YYYY-MM-DD_HHmm.tar.gz`
|
||||
|
||||
### 2. 导出知识库
|
||||
- 生成可分发的知识库压缩包
|
||||
- 可选：只导出特定领域/标签
|
||||
|
||||
### 3. 同步状态
|
||||
- 检查坚果云同步路径 `~/NutstoreCloudBridge/` 的状态
|
||||
- 报告最近同步时间和文件差异
|
||||
|
||||
## 坚果云同步
|
||||
|
||||
- 同步路径：`~/NutstoreCloudBridge/`
|
||||
- 同步方式：坚果云客户端自动同步，非 Git
|
||||
- 本技能不管理云端同步过程
|
||||
|
||||
## 依赖
|
||||
|
||||
| 依赖 | 说明 | 状态 |
|
||||
|---|---|---|
|
||||
| 坚果云客户端 | 知识库同步 | ✅ 已安装 |
|
||||
| tar/gzip | 备份打包 | ✅ 系统自带 |
|
||||
@@ -0,0 +1,55 @@
|
||||
skill:
|
||||
name: "同步管理器"
|
||||
id: "sync_manager"
|
||||
version: "1.1.0"
|
||||
category: "core-system"
|
||||
status: "active"
|
||||
created: "2026-04-29"
|
||||
updated: "2026-04-29"
|
||||
|
||||
description:
|
||||
short: "知识库备份、增量导出、坚果云同步状态检查"
|
||||
use_cases:
|
||||
- "全量备份 03_knowledge/ + facts.db 到 memory_backup/"
|
||||
- "按领域/标签导出可分发知识包"
|
||||
- "检查坚果云同步路径状态与最新同步时间"
|
||||
- "自动清理过期备份（保留最近N份）"
|
||||
tags: ["备份", "同步", "导出", "坚果云"]
|
||||
|
||||
ownership:
|
||||
master_editor: "my-macbook_macOS14.4_16GB_master"
|
||||
readonly_on_other_devices: false
|
||||
|
||||
environment:
|
||||
os: "macos"
|
||||
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
|
||||
pip_dependencies: []
|
||||
|
||||
setup:
|
||||
install_script: ""
|
||||
post_check: "python3 -c 'import tarfile, shutil; print(\"OK\")'"
|
||||
|
||||
usage:
|
||||
entry_point: "sync_manager.py"
|
||||
scheduled: ""
|
||||
manual_trigger: "python3 sync_manager.py --mode backup"
|
||||
|
||||
scripts:
|
||||
- sync_manager.py # 备份/导出/状态检查主程序
|
||||
|
||||
backup:
|
||||
source_dirs:
|
||||
- "03_knowledge/"
|
||||
- "04_memory/long_term/"
|
||||
output_dir: "04_memory/memory_backup/"
|
||||
filename_pattern: "kb_backup_YYYY-MM-DD_HHmm.tar.gz"
|
||||
keep_last: 10
|
||||
|
||||
nutstore:
|
||||
sync_path: "~/NutstoreCloudBridge/"
|
||||
mode: "client-managed"
|
||||
|
||||
changelog:
|
||||
- version: "1.1.0"
|
||||
date: "2026-04-29"
|
||||
changes: "初始实现：backup/export/status 三模式，自动清理旧备份"
|
||||
@@ -0,0 +1,320 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
sync_manager.py — 知识库备份、导出、同步状态检查
|
||||
|
||||
模式：
|
||||
backup — 全量备份 03_knowledge/ + 04_memory/long_term/ 到 memory_backup/
|
||||
export — 导出可分发知识包（可按领域/标签过滤）
|
||||
status — 检查坚果云同步路径状态
|
||||
|
||||
使用：
|
||||
python3 sync_manager.py --mode backup
|
||||
python3 sync_manager.py --mode backup --keep 5
|
||||
python3 sync_manager.py --mode export --domain 10_concepts 20_methods
|
||||
python3 sync_manager.py --mode export --tag AI工具
|
||||
python3 sync_manager.py --mode status
|
||||
python3 sync_manager.py --mode backup --dry-run
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import tarfile
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# ── 配置 ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
BACKUP_SOURCES = [
|
||||
"03_knowledge",
|
||||
"04_memory/long_term",
|
||||
]
|
||||
|
||||
KNOWLEDGE_DOMAINS = [
|
||||
"10_concepts", "20_methods", "30_facts",
|
||||
"40_references", "50_resources", "60_opinions",
|
||||
]
|
||||
|
||||
DEFAULT_NUTSTORE_PATH = "~/NutstoreCloudBridge"
|
||||
DEFAULT_KEEP = 10
|
||||
|
||||
|
||||
# ── 参数解析 ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="AgentOS 同步管理器")
|
||||
parser.add_argument("--root", default="~/workbuddy-agent-os/agent-sync", help="agent-os 根目录")
|
||||
parser.add_argument("--mode", choices=["backup", "export", "status"], required=True,
|
||||
help="运行模式")
|
||||
parser.add_argument("--keep", type=int, default=DEFAULT_KEEP,
|
||||
help="备份模式：保留最近N份（默认10）")
|
||||
parser.add_argument("--domain", nargs="+", default=None,
|
||||
help="导出模式：只导出指定领域目录（如 10_concepts 20_methods）")
|
||||
parser.add_argument("--tag", default=None,
|
||||
help="导出模式：只导出含有指定 tag 的文件")
|
||||
parser.add_argument("--nutstore", default=DEFAULT_NUTSTORE_PATH,
|
||||
help="坚果云同步路径（用于 status 模式）")
|
||||
parser.add_argument("--dry-run", action="store_true", help="模拟运行，不实际写入")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
# ── 工具函数 ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def ensure_dir(path: Path, dry_run: bool = False):
|
||||
if not dry_run:
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def format_size(size_bytes: int) -> str:
|
||||
for unit in ["B", "KB", "MB", "GB"]:
|
||||
if size_bytes < 1024:
|
||||
return f"{size_bytes:.1f} {unit}"
|
||||
size_bytes //= 1024
|
||||
return f"{size_bytes:.1f} TB"
|
||||
|
||||
|
||||
def get_md_tag(file_path: Path) -> list:
|
||||
"""从 frontmatter 提取 tags 列表"""
|
||||
try:
|
||||
content = file_path.read_text(encoding="utf-8", errors="ignore")
|
||||
if not content.startswith("---"):
|
||||
return []
|
||||
end = content.find("\n---", 3)
|
||||
if end == -1:
|
||||
return []
|
||||
frontmatter = content[3:end]
|
||||
# 匹配 tags: [a, b] 或多行列表
|
||||
m = re.search(r'tags\s*:\s*\[([^\]]+)\]', frontmatter)
|
||||
if m:
|
||||
return [t.strip().strip('"\'') for t in m.group(1).split(",")]
|
||||
# 多行格式
|
||||
tags = []
|
||||
in_tags = False
|
||||
for line in frontmatter.splitlines():
|
||||
if re.match(r'\s*tags\s*:', line):
|
||||
in_tags = True
|
||||
continue
|
||||
if in_tags:
|
||||
if line.strip().startswith("- "):
|
||||
tags.append(line.strip()[2:].strip())
|
||||
elif line.strip() and not line.startswith(" "):
|
||||
break
|
||||
return tags
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
# ── backup 模式 ────────────────────────────────────────────────────────────────
|
||||
|
||||
def run_backup(root: Path, keep: int, dry_run: bool):
|
||||
backup_dir = root / "04_memory" / "memory_backup"
|
||||
ensure_dir(backup_dir, dry_run)
|
||||
|
||||
now = datetime.now()
|
||||
filename = f"kb_backup_{now.strftime('%Y-%m-%d_%H%M')}.tar.gz"
|
||||
output_path = backup_dir / filename
|
||||
|
||||
sources = []
|
||||
for src in BACKUP_SOURCES:
|
||||
p = root / src
|
||||
if p.exists():
|
||||
sources.append(p)
|
||||
else:
|
||||
print(f" [跳过] {p} 不存在")
|
||||
|
||||
if not sources:
|
||||
print("[ERROR] 没有找到任何备份源")
|
||||
sys.exit(1)
|
||||
|
||||
total_files = sum(
|
||||
sum(1 for _ in src.rglob("*") if _.is_file())
|
||||
for src in sources
|
||||
)
|
||||
total_size = sum(
|
||||
sum(f.stat().st_size for f in src.rglob("*") if f.is_file())
|
||||
for src in sources
|
||||
)
|
||||
|
||||
print(f"备份目标: {', '.join(str(s.relative_to(root)) for s in sources)}")
|
||||
print(f"文件数量: {total_files} 预计大小: {format_size(total_size)}")
|
||||
print(f"输出路径: {output_path}")
|
||||
|
||||
if dry_run:
|
||||
print("[dry-run] 跳过实际打包")
|
||||
return
|
||||
|
||||
with tarfile.open(output_path, "w:gz") as tar:
|
||||
for src in sources:
|
||||
tar.add(src, arcname=src.relative_to(root))
|
||||
|
||||
actual_size = output_path.stat().st_size
|
||||
print(f"✅ 备份完成: {filename} ({format_size(actual_size)})")
|
||||
|
||||
# 清理旧备份
|
||||
_prune_backups(backup_dir, keep)
|
||||
|
||||
|
||||
def _prune_backups(backup_dir: Path, keep: int):
|
||||
"""保留最近 keep 份，删除更旧的"""
|
||||
pattern = re.compile(r"kb_backup_\d{4}-\d{2}-\d{2}_\d{4}\.tar\.gz")
|
||||
backups = sorted(
|
||||
[f for f in backup_dir.iterdir() if f.is_file() and pattern.match(f.name)],
|
||||
key=lambda f: f.stat().st_mtime,
|
||||
reverse=True,
|
||||
)
|
||||
to_delete = backups[keep:]
|
||||
if to_delete:
|
||||
print(f"清理旧备份（保留最近{keep}份）:")
|
||||
for f in to_delete:
|
||||
print(f" 删除: {f.name}")
|
||||
f.unlink()
|
||||
else:
|
||||
print(f"备份数量: {len(backups)} 份，无需清理")
|
||||
|
||||
|
||||
# ── export 模式 ────────────────────────────────────────────────────────────────
|
||||
|
||||
def run_export(root: Path, domains: list, tag: str, dry_run: bool):
|
||||
now = datetime.now()
|
||||
suffix = f"_tag-{tag}" if tag else ""
|
||||
suffix += f"_dom-{'_'.join(domains)}" if domains else ""
|
||||
filename = f"kb_export_{now.strftime('%Y-%m-%d_%H%M')}{suffix}.tar.gz"
|
||||
export_dir = root / "04_memory" / "memory_backup"
|
||||
ensure_dir(export_dir, dry_run)
|
||||
output_path = export_dir / filename
|
||||
|
||||
# 确定要导出的目录
|
||||
kb_root = root / "03_knowledge"
|
||||
if domains:
|
||||
scan_dirs = [kb_root / d for d in domains if (kb_root / d).exists()]
|
||||
missing = [d for d in domains if not (kb_root / d).exists()]
|
||||
if missing:
|
||||
print(f"[警告] 以下领域目录不存在: {', '.join(missing)}")
|
||||
else:
|
||||
scan_dirs = [kb_root / d for d in KNOWLEDGE_DOMAINS if (kb_root / d).exists()]
|
||||
|
||||
# 收集文件
|
||||
all_files = []
|
||||
for d in scan_dirs:
|
||||
for f in d.rglob("*.md"):
|
||||
if tag:
|
||||
if tag in get_md_tag(f):
|
||||
all_files.append(f)
|
||||
else:
|
||||
all_files.append(f)
|
||||
|
||||
if not all_files:
|
||||
print("[ERROR] 没有找到符合条件的文件")
|
||||
sys.exit(1)
|
||||
|
||||
total_size = sum(f.stat().st_size for f in all_files)
|
||||
filter_desc = f" 标签过滤: {tag}" if tag else ""
|
||||
domain_desc = f" 领域: {', '.join(domains)}" if domains else " 领域: 全部"
|
||||
print(f"导出范围: {domain_desc}{filter_desc}")
|
||||
print(f"文件数量: {len(all_files)} 预计大小: {format_size(total_size)}")
|
||||
print(f"输出路径: {output_path}")
|
||||
|
||||
if dry_run:
|
||||
print("[dry-run] 跳过实际打包")
|
||||
for f in all_files[:5]:
|
||||
print(f" 示例: {f.relative_to(root)}")
|
||||
if len(all_files) > 5:
|
||||
print(f" ... 共 {len(all_files)} 个文件")
|
||||
return
|
||||
|
||||
with tarfile.open(output_path, "w:gz") as tar:
|
||||
for f in all_files:
|
||||
tar.add(f, arcname=f.relative_to(root))
|
||||
|
||||
actual_size = output_path.stat().st_size
|
||||
print(f"✅ 导出完成: {filename} ({format_size(actual_size)}，{len(all_files)} 个文件)")
|
||||
|
||||
|
||||
# ── status 模式 ────────────────────────────────────────────────────────────────
|
||||
|
||||
def run_status(root: Path, nutstore_path: str):
|
||||
print("=" * 50)
|
||||
print("AgentOS 同步状态检查")
|
||||
print("=" * 50)
|
||||
|
||||
# 1. 本地知识库状态
|
||||
kb_root = root / "03_knowledge"
|
||||
facts_db = root / "04_memory" / "long_term" / "facts.db"
|
||||
backup_dir = root / "04_memory" / "memory_backup"
|
||||
|
||||
if kb_root.exists():
|
||||
md_count = sum(1 for _ in kb_root.rglob("*.md"))
|
||||
kb_size = sum(f.stat().st_size for f in kb_root.rglob("*") if f.is_file())
|
||||
print(f"\n📚 知识库: {md_count} 个 MD 文件 ({format_size(kb_size)})")
|
||||
else:
|
||||
print(f"\n📚 知识库: ❌ 目录不存在 ({kb_root})")
|
||||
|
||||
if facts_db.exists():
|
||||
facts_mtime = datetime.fromtimestamp(facts_db.stat().st_mtime)
|
||||
facts_size = facts_db.stat().st_size
|
||||
print(f"🧠 facts.db: {format_size(facts_size)} 最后修改: {facts_mtime.strftime('%Y-%m-%d %H:%M')}")
|
||||
else:
|
||||
print(f"🧠 facts.db: ❌ 不存在")
|
||||
|
||||
# 2. 备份状态
|
||||
if backup_dir.exists():
|
||||
pattern = re.compile(r"kb_backup_\d{4}-\d{2}-\d{2}_\d{4}\.tar\.gz")
|
||||
backups = sorted(
|
||||
[f for f in backup_dir.iterdir() if f.is_file() and pattern.match(f.name)],
|
||||
key=lambda f: f.stat().st_mtime,
|
||||
reverse=True,
|
||||
)
|
||||
if backups:
|
||||
latest = backups[0]
|
||||
latest_time = datetime.fromtimestamp(latest.stat().st_mtime)
|
||||
print(f"\n💾 最新备份: {latest.name}")
|
||||
print(f" 时间: {latest_time.strftime('%Y-%m-%d %H:%M')} 大小: {format_size(latest.stat().st_size)}")
|
||||
print(f" 共 {len(backups)} 份备份")
|
||||
else:
|
||||
print(f"\n💾 备份: ❌ 暂无备份文件")
|
||||
else:
|
||||
print(f"\n💾 备份目录: ❌ 不存在")
|
||||
|
||||
# 3. 坚果云同步状态
|
||||
nutstore = Path(nutstore_path).expanduser()
|
||||
print(f"\n☁️ 坚果云同步路径: {nutstore}")
|
||||
if nutstore.exists():
|
||||
all_files = [f for f in nutstore.rglob("*") if f.is_file()]
|
||||
if all_files:
|
||||
latest_sync = max(all_files, key=lambda f: f.stat().st_mtime)
|
||||
sync_time = datetime.fromtimestamp(latest_sync.stat().st_mtime)
|
||||
sync_size = sum(f.stat().st_size for f in all_files)
|
||||
print(f" 状态: ✅ 路径存在")
|
||||
print(f" 文件数: {len(all_files)} 总大小: {format_size(sync_size)}")
|
||||
print(f" 最近同步: {sync_time.strftime('%Y-%m-%d %H:%M')} ({latest_sync.name})")
|
||||
else:
|
||||
print(f" 状态: ⚠️ 路径存在但为空")
|
||||
else:
|
||||
print(f" 状态: ❌ 路径不存在（坚果云未挂载或未安装）")
|
||||
|
||||
print()
|
||||
|
||||
|
||||
# ── 入口 ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
root = Path(args.root).expanduser()
|
||||
|
||||
if not root.exists():
|
||||
print(f"[ERROR] agent-os 根目录不存在: {root}")
|
||||
sys.exit(1)
|
||||
|
||||
if args.mode == "backup":
|
||||
run_backup(root, args.keep, args.dry_run)
|
||||
elif args.mode == "export":
|
||||
run_export(root, args.domain or [], args.tag, args.dry_run)
|
||||
elif args.mode == "status":
|
||||
run_status(root, args.nutstore)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"name": "sync_manager",
|
||||
"version": "1.1.0",
|
||||
"description": "同步管理占位技能（由坚果云和Obsidian处理）",
|
||||
"dependencies": [],
|
||||
"mcp_tools": []
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: web_crawler
|
||||
version: 1.2.0
|
||||
status: archived
|
||||
archived_date: 2026-06-21
|
||||
description: [已归档] 网页抓取设计，无实现代码
|
||||
triggers: []
|
||||
---
|
||||
|
||||
# Web Crawler Skill（网页抓取+反爬）
|
||||
|
||||
## 概述
|
||||
|
||||
使用 Scrapling + Crawl4AI 进行结构化网页抓取，绕过 Cloudflare 等高级反爬。
|
||||
|
||||
## 抓取引擎选择
|
||||
|
||||
| 反爬级别 | 引擎 | 说明 |
|
||||
|----------|------|------|
|
||||
| 低（静态页面） | Scrapling 静态模式 | 最快，无需浏览器 |
|
||||
| 中（动态渲染） | Scrapling 动态模式 | Camoufox 隐身浏览器 |
|
||||
| 高（Cloudflare 等） | **CloakBrowser** | 源码级反爬，30/30检测通过（2026-05 新增） |
|
||||
|
||||
> **2026-05-14 更新**：Playwright + Stealth 已替换为 CloakBrowser。
|
||||
> CloakBrowser 是源码级修改的 Chromium，49项 C++ 补丁修改指纹，
|
||||
> 通过率远高于 JS 注入方案。安装：`pip install cloakbrowser`
|
||||
|
||||
## 执行流程
|
||||
|
||||
```
|
||||
1. 解析目标 URL，判断反爬级别
|
||||
2. 选择最优引擎
|
||||
3. 抓取内容，Crawl4AI 过滤广告/导航 → 生成干净 Markdown
|
||||
4. 输出到指定路径或直接传给 kb_manager 入库
|
||||
5. 失败自动重试（最多 3 次），引擎自动降级
|
||||
```
|
||||
|
||||
## 使用场景
|
||||
|
||||
- 网页内容抓取 → 传给 content_processor / kb_manager
|
||||
- 网页变化监控 → 配合 auto_collector 定时检查
|
||||
- 数据采集 → 结构化提取
|
||||
|
||||
## 依赖
|
||||
|
||||
| 包 | 用途 | 安装状态 |
|
||||
|---|---|---|
|
||||
| scrapling | 三引擎自适应抓取 | ✅ 已安装 |
|
||||
| crawl4ai | LLM 友好结构化输出 | ✅ 已安装 |
|
||||
| playwright | 浏览器控制 | ✅ 已安装 |
|
||||
| playwright-stealth | 反检测 | ✅ 已安装 |
|
||||
| Playwright Chromium | 浏览器引擎 | ⚠️ 需下载（网络问题待解决） |
|
||||
|
||||
## 注意事项
|
||||
|
||||
- Playwright Chromium 浏览器需下载，运行 `playwright install chromium`
|
||||
- 如果 Playwright 不可用，fallback 到 Scrapling 静态/动态模式
|
||||
- 抓取频率过高可能被目标网站封禁，建议设置合理间隔
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"name": "web_crawler",
|
||||
"version": "1.0.0",
|
||||
"description": "网页抓取占位技能（待实现）",
|
||||
"dependencies": [],
|
||||
"mcp_tools": []
|
||||
}
|
||||
Reference in New Issue
Block a user