feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程,直接发 HTTP
移植自 mac-agent-os 的 mediacrawler_adapter:不起子进程、不开页面,用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin,所以 ingest 那条链路一个字不用改。
**目前能用的(真环境实测,非推断)**:
profile/other : 200, 7075 字节 —— 博主主页指标(粉丝/获赞/作品数/昵称)
aweme/detail : 200, 45425 字节 —— 单条作品详情(含点赞/评论/收藏/分享)
**目前不能用的:作品列表 `aweme/post`。** 两个互相独立的原因:
1. 这个接口被抖音单独升级成了真校验:不带 x-tt-argus 回 403「Uifid Not Found」,
带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过,「真校验」过不了。
同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的,
挑中的恰好是「批量拉作品列表」这个最敏感的动作。
2. 改走页面截获也不行:CDP 浏览器打开博主主页会落到「验证码中间页」(当天大量探测的
代价,过几小时要重测)。
所以现在的边界是:**已知作品的指标刷新能做,自动发现新作品做不了**。
**排查中控住变量后得到的两条事实**(都写进注释了):
· `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
UA+client hints+Cookie 时是 200 但仅 121 字节的空壳,补上这三个头变 7074 字节。
(我先前猜的 sec-ch-ua 不是关键。)
· 因此 UA 与 client hints 必须**成套地取自同一个浏览器**,所以 BrowserIdentity 一次
从 CDP 取齐 cookie + UA + hints,而不是各自写死。
「200 + 空 body 必须当场报错」也是刻意写死的:放过去它会在下游变成「这个博主没作品」,
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的,还被翻译成「账号被封」。
测试 +11:cookie 解析、请求头成套性(含 uifid 缺失/回退)、产物键名与 store 对齐、
以及 _get 的三条失败路径(空 body / 403 带网关原话 / 正常返回)。
This commit is contained in:
@@ -0,0 +1,140 @@
|
||||
---
|
||||
name: inbox_refine
|
||||
version: 1.1.0
|
||||
description: 知识库收件箱提纯——每日将 00_inbox/ 内容分类归档到知识库,更新首页统计和变更日志
|
||||
triggers:
|
||||
- 提纯
|
||||
- 收件箱提纯
|
||||
- 归档inbox
|
||||
- inbox refine
|
||||
- 整理收件箱
|
||||
- 归档知识
|
||||
- 提纯收件箱
|
||||
---
|
||||
|
||||
# Inbox Refine 技能
|
||||
|
||||
## 概述
|
||||
|
||||
将 `00_inbox/` 中的原始内容按知识属性分类,应用模板后归档到对应目录,并更新知识库首页和变更日志。
|
||||
|
||||
## 触发条件
|
||||
|
||||
- 每日凌晨 3:00 自动执行(WorkBuddy 自动化)
|
||||
- 手动说"提纯"、"整理收件箱"、"inbox refine"
|
||||
|
||||
## 工作流
|
||||
|
||||
```
|
||||
1. 扫描 00_inbox/ 所有 .md 文件
|
||||
2. 逐个读取 → 分析内容
|
||||
3. 确定分类:
|
||||
- nature:fact/method/concept/reference/data/opinion/quote/axiom/regulation
|
||||
- domain:从 99_system/taxonomies/domains.md 选择 1-3 个
|
||||
- confidence:根据 source_type 评分
|
||||
4. 应用对应模板(99_system/templates/)
|
||||
5. 生成 ID:KB-YYYYMMDD-NNN
|
||||
6. 写入目标目录
|
||||
7. 删除 00_inbox/ 原文件
|
||||
8. 更新 README.md 统计
|
||||
9. 更新 CHANGELOG.md
|
||||
```
|
||||
|
||||
## 分类决策逻辑
|
||||
|
||||
### nature 判定
|
||||
| 内容特征 | nature | 目标目录 |
|
||||
|----------|--------|----------|
|
||||
| 已验证的客观事实 | fact | 30_facts/ |
|
||||
| 可操作的步骤/教程 | method | 20_methods/ |
|
||||
| 原子概念/基础原理 | concept | 10_concepts/ |
|
||||
| 公认基础原理 | axiom | 10_concepts/ |
|
||||
| 法律法规/制度 | regulation | 30_facts/ |
|
||||
| 论文/文档/外部资料 | reference | 40_references/ |
|
||||
| 测试数据/基准结果 | data | 30_facts/ |
|
||||
| 主观看法/推测 | opinion | 60_opinions/ |
|
||||
| 他人原话/语录 | quote | 60_opinions/ |
|
||||
|
||||
### confidence 评分
|
||||
| source_type | confidence |
|
||||
|-------------|-----------|
|
||||
| official_doc | 0.9 |
|
||||
| literature | 0.8 |
|
||||
| experiment | 0.7 |
|
||||
| personal_exp | 0.5 |
|
||||
| social_media | 0.3 |
|
||||
| unknown | 0.4 |
|
||||
|
||||
## 去重规则
|
||||
|
||||
1. 读取目标目录已有文件,提取 title 和 tags
|
||||
2. 标题相似度 > 0.8 → 合并(补充细节,不新建)
|
||||
3. 内容完全一致 → 跳过,记录日志
|
||||
4. 同一事实新旧矛盾 → 新事实覆盖,旧值写入 previous_version
|
||||
|
||||
## Frontmatter 模板
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: KB-YYYYMMDD-NNN
|
||||
title: "__标题__"
|
||||
type: __concept/fact/method/opinion/reference__
|
||||
status: published
|
||||
nature: __fact/method/concept/...__
|
||||
domain: [__领域1__, __领域2__]
|
||||
subdomain: []
|
||||
tags: [__标签1__, __标签2__]
|
||||
confidence: 0.8
|
||||
source: "__来源URL或出处__"
|
||||
source_type: __official_doc/literature/personal_exp/...__
|
||||
date_created: YYYY-MM-DD
|
||||
date_modified: YYYY-MM-DD
|
||||
version: 1
|
||||
previous_version: ""
|
||||
superseded_by: ""
|
||||
---
|
||||
```
|
||||
|
||||
## 更新首页统计
|
||||
|
||||
读取 `03_knowledge/README.md`,替换 `## 📊 知识统计` 下的表格:
|
||||
- 扫描各目录下 .md 文件数量
|
||||
- 找出每个目录最近修改的文件日期
|
||||
- 更新总计行
|
||||
|
||||
## 更新变更日志
|
||||
|
||||
在 `03_knowledge/CHANGELOG.md` 顶部追加:
|
||||
```markdown
|
||||
## YYYY-MM-DD
|
||||
|
||||
### 归档
|
||||
- [nature] title → 目标目录 (KB-YYYYMMDD-NNN)
|
||||
- ...
|
||||
```
|
||||
|
||||
## 依赖
|
||||
|
||||
| 包/工具 | 用途 | 安装状态 |
|
||||
|---------|------|----------|
|
||||
| file_read/file_write | 读写文件 | ✅ 内置 |
|
||||
| python3 | 脚本执行 | ✅ managed |
|
||||
| oMLX + Qwen3-8B-MLX-4bit | 本地 LLM 辅助分类(可选) | ✅ 已安装 |
|
||||
|
||||
## 路径约定
|
||||
|
||||
| 项目 | 路径 |
|
||||
|------|------|
|
||||
| 收件箱 | `~/workbuddy-agent-os/agent-sync/03_knowledge/00_inbox/` |
|
||||
| 知识库根 | `~/workbuddy-agent-os/agent-sync/03_knowledge/` |
|
||||
| 首页 | `~/workbuddy-agent-os/agent-sync/03_knowledge/README.md` |
|
||||
| 变更日志 | `~/workbuddy-agent-os/agent-sync/03_knowledge/CHANGELOG.md` |
|
||||
| 模板 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/templates/` |
|
||||
| 分类表 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/` |
|
||||
| 中文映射 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/folder-aliases.json` |
|
||||
|
||||
## 错误处理
|
||||
|
||||
- 无法分类的内容 → 保留在 00_inbox/,添加标签 `#待分类`,记录日志
|
||||
- 模板应用失败 → 使用最小 Frontmatter(id/title/nature/date_created)
|
||||
- 首页更新失败 → 只记录日志,不阻断归档流程
|
||||
@@ -0,0 +1,46 @@
|
||||
skill:
|
||||
name: "收件箱精炼器"
|
||||
id: "inbox_refine"
|
||||
version: "1.1.0"
|
||||
category: "core-system"
|
||||
status: "active"
|
||||
created: "2026-04-25"
|
||||
updated: "2026-04-29"
|
||||
|
||||
description:
|
||||
short: "扫描协同inbox五分区,AI分类,自动归档到知识库"
|
||||
use_cases:
|
||||
- "定期扫描inbox各分区,按knowledge/memory/personal分类处理"
|
||||
- "clipping分区:全文入库"
|
||||
- "feed分区:摘要入库,原文归档"
|
||||
- "memory分区:入长期记忆,同时建立知识卡"
|
||||
- "personal分区:入personal知识区"
|
||||
- "tools分区:转交工具审核流程"
|
||||
tags: ["inbox", "精炼", "分类", "归档"]
|
||||
|
||||
ownership:
|
||||
master_editor: "my-macbook_macOS14.4_16GB_master"
|
||||
readonly_on_other_devices: false
|
||||
|
||||
environment:
|
||||
os: "macos"
|
||||
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
|
||||
pip_dependencies:
|
||||
- "chromadb>=1.0.0"
|
||||
|
||||
setup:
|
||||
install_script: ""
|
||||
post_check: "python3 -c 'print(\"OK\")'"
|
||||
|
||||
usage:
|
||||
entry_point: "inbox_refine.py"
|
||||
scheduled: "每日 03:00(master机器执行)"
|
||||
manual_trigger: "收件箱精炼"
|
||||
|
||||
changelog:
|
||||
- version: "1.1.0"
|
||||
date: "2026-04-29"
|
||||
changes: "支持五分区差异化处理逻辑,适配新inbox路径"
|
||||
- version: "1.0.0"
|
||||
date: "2026-04-25"
|
||||
changes: "初始版本"
|
||||
@@ -0,0 +1,461 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
inbox_refine.py - 知识库收件箱提纯脚本
|
||||
|
||||
将 00_inbox/ 中的原始内容按知识属性分类归档到对应目录,
|
||||
更新知识库首页统计和变更日志。
|
||||
|
||||
用法:
|
||||
python3 inbox_refine.py --root ~/workbuddy-agent-os/agent-sync [--dry-run] [--date YYYY-MM-DD]
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import argparse
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# ─── 角色检查:仅 master 可执行收件箱提纯 ───
|
||||
try:
|
||||
sys.path.insert(0, str(Path.home() / "workbuddy-agent-os" / "agent-sync" / "05_tools" / "01_system"))
|
||||
from role_check import check_role, check_capability
|
||||
if not check_role(["master"]):
|
||||
print("⛔ 非 master 角色,跳过 inbox_refine")
|
||||
sys.exit(0)
|
||||
if not check_capability("inbox_refine"):
|
||||
print("⛔ 能力 inbox_refine 未启用,跳过")
|
||||
sys.exit(0)
|
||||
except ImportError as e:
|
||||
print(f"⚠️ 角色检查模块不可用 ({e}),继续执行")
|
||||
|
||||
# 导入 LLM 分类器
|
||||
try:
|
||||
from llm_classifier import classify_content_enhanced
|
||||
LLM_AVAILABLE = True
|
||||
except ImportError:
|
||||
print("⚠️ LLM 分类器不可用,将使用启发式规则")
|
||||
LLM_AVAILABLE = False
|
||||
|
||||
|
||||
# ── 配置 ──────────────────────────────────────────
|
||||
|
||||
NATURE_TO_DIR = {
|
||||
"fact": "30_facts",
|
||||
"method": "20_methods",
|
||||
"concept": "10_concepts",
|
||||
"axiom": "10_concepts",
|
||||
"regulation": "30_facts",
|
||||
"reference": "40_references",
|
||||
"data": "30_facts",
|
||||
"opinion": "60_opinions",
|
||||
"quote": "60_opinions",
|
||||
}
|
||||
|
||||
SOURCE_TYPE_CONFIDENCE = {
|
||||
"official_doc": 0.9,
|
||||
"literature": 0.8,
|
||||
"experiment": 0.7,
|
||||
"personal_exp": 0.5,
|
||||
"social_media": 0.3,
|
||||
"unknown": 0.4,
|
||||
}
|
||||
|
||||
DIR_CHINESE = {
|
||||
"00_inbox": "📥 收件箱",
|
||||
"01_daily": "📅 日记",
|
||||
"10_concepts": "💡 概念层",
|
||||
"20_methods": "🔧 方法层",
|
||||
"30_facts": "📋 事实层",
|
||||
"40_references": "📎 参考层",
|
||||
"50_resources": "🛠 资源层",
|
||||
"60_opinions": "💭 观点层",
|
||||
"90_archive": "🗄 归档层",
|
||||
}
|
||||
|
||||
TRACKED_DIRS = [
|
||||
"00_inbox", "01_daily", "10_concepts", "20_methods",
|
||||
"30_facts", "40_references", "50_resources", "60_opinions", "90_archive"
|
||||
]
|
||||
|
||||
|
||||
# ── 工具函数 ──────────────────────────────────────
|
||||
|
||||
def count_md_files(directory: Path) -> int:
|
||||
"""统计目录下 .md 文件数量"""
|
||||
if not directory.exists():
|
||||
return 0
|
||||
return sum(1 for f in directory.rglob("*.md") if f.is_file())
|
||||
|
||||
|
||||
def latest_modified(directory: Path) -> str:
|
||||
"""获取目录下最近修改的 .md 文件日期"""
|
||||
if not directory.exists():
|
||||
return "—"
|
||||
md_files = list(directory.rglob("*.md"))
|
||||
if not md_files:
|
||||
return "—"
|
||||
latest = max(f.stat().st_mtime for f in md_files)
|
||||
return datetime.fromtimestamp(latest).strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def generate_id(date_str: str, knowledge_root: Path) -> str:
|
||||
"""生成唯一知识ID:KB-YYYYMMDD-NNN"""
|
||||
prefix = f"KB-{date_str.replace('-', '')}"
|
||||
# 扫描已有ID确定序号
|
||||
existing = set()
|
||||
for md in knowledge_root.rglob("*.md"):
|
||||
content = md.read_text(encoding="utf-8", errors="ignore")[:500]
|
||||
m = re.search(r'^id:\s*(KB-\d{8}-\d+)', content, re.MULTILINE)
|
||||
if m:
|
||||
existing.add(m.group(1))
|
||||
n = 1
|
||||
while f"{prefix}-{n:03d}" in existing:
|
||||
n += 1
|
||||
return f"{prefix}-{n:03d}"
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> dict:
|
||||
"""解析 YAML frontmatter"""
|
||||
if not text.startswith("---"):
|
||||
return {}
|
||||
match = re.match(r'^---\s*\n(.*?)\n---', text, re.DOTALL)
|
||||
if not match:
|
||||
return {}
|
||||
fm = {}
|
||||
for line in match.group(1).split("\n"):
|
||||
if ":" in line:
|
||||
key, val = line.split(":", 1)
|
||||
fm[key.strip()] = val.strip().strip('"').strip("'")
|
||||
return fm
|
||||
|
||||
|
||||
def build_frontmatter(kb_id: str, title: str, nature: str, domain: str,
|
||||
confidence: float, source: str, source_type: str,
|
||||
date_str: str, summary: str = "", tags: list = None) -> str:
|
||||
"""构建知识卡片 frontmatter"""
|
||||
if tags is None:
|
||||
tags = ["待补充"]
|
||||
|
||||
# 确保 domain 是列表格式
|
||||
if isinstance(domain, str):
|
||||
domain_list = [domain]
|
||||
else:
|
||||
domain_list = domain if isinstance(domain, list) else ["other"]
|
||||
|
||||
# 构建 tags 字符串
|
||||
tags_str = ", ".join([f'"{tag}"' for tag in tags])
|
||||
|
||||
return f"""---
|
||||
id: {kb_id}
|
||||
title: "{title}"
|
||||
type: {nature if nature in ('concept', 'fact', 'method', 'opinion', 'reference') else 'concept'}
|
||||
status: published
|
||||
nature: {nature}
|
||||
domain: [{", ".join(domain_list)}]
|
||||
subdomain: []
|
||||
tags: [{tags_str}]
|
||||
confidence: {confidence}
|
||||
source: "{source}"
|
||||
source_type: {source_type}
|
||||
date_created: {date_str}
|
||||
date_modified: {date_str}
|
||||
version: 1
|
||||
previous_version: ""
|
||||
superseded_by: ""
|
||||
summary: "{summary}"
|
||||
---"""
|
||||
|
||||
|
||||
# ── 分类逻辑 ──────────────────────────────────────
|
||||
|
||||
def classify_content(text: str, fm: dict) -> dict:
|
||||
"""
|
||||
增强版分类函数:优先使用 LLM 智能分类,失败时降级到启发式规则。
|
||||
如果 frontmatter 已有 nature/domain,优先使用。
|
||||
"""
|
||||
# 如果 LLM 可用,直接使用增强版分类
|
||||
if LLM_AVAILABLE:
|
||||
try:
|
||||
# 直接使用 LLM 分类,分类器内部会处理降级
|
||||
result = classify_content_enhanced(text, fm, use_llm=True)
|
||||
# 适配现有结构
|
||||
return {
|
||||
"nature": result["nature"],
|
||||
"domain": result["domain"][0] if result["domain"] else "other",
|
||||
"confidence": result["confidence"],
|
||||
"source_type": result.get("source_type", "unknown"),
|
||||
"source": result.get("source", ""),
|
||||
"title": result.get("title", ""),
|
||||
"summary": result.get("summary", ""),
|
||||
"tags": result.get("tags", [])
|
||||
}
|
||||
except Exception as e:
|
||||
print(f"⚠️ LLM 分类失败,回退到启发式规则: {e}")
|
||||
|
||||
# LLM 不可用或失败,使用启发式规则
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": "other",
|
||||
"confidence": 0.4,
|
||||
"source_type": "unknown",
|
||||
"source": "",
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature") and fm["nature"] in NATURE_TO_DIR:
|
||||
result["nature"] = fm["nature"]
|
||||
if fm.get("domain"):
|
||||
result["domain"] = fm["domain"]
|
||||
if fm.get("confidence"):
|
||||
try:
|
||||
result["confidence"] = float(fm["confidence"])
|
||||
except ValueError:
|
||||
pass
|
||||
if fm.get("source"):
|
||||
result["source"] = fm["source"]
|
||||
if fm.get("source_type"):
|
||||
result["source_type"] = fm["source_type"]
|
||||
if fm.get("title"):
|
||||
result["title"] = fm["title"]
|
||||
|
||||
# 如果没有 nature,用启发式规则
|
||||
if not fm.get("nature"):
|
||||
lower = text.lower()
|
||||
# 方法类关键词
|
||||
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践", "best practice"]
|
||||
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
|
||||
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
|
||||
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
|
||||
|
||||
if any(kw in lower for kw in method_kw):
|
||||
result["nature"] = "method"
|
||||
elif any(kw in lower for kw in fact_kw):
|
||||
result["nature"] = "fact"
|
||||
elif any(kw in lower for kw in opinion_kw):
|
||||
result["nature"] = "opinion"
|
||||
elif any(kw in lower for kw in reference_kw):
|
||||
result["nature"] = "reference"
|
||||
|
||||
# confidence from source_type
|
||||
if result["source_type"] in SOURCE_TYPE_CONFIDENCE:
|
||||
result["confidence"] = SOURCE_TYPE_CONFIDENCE[result["source_type"]]
|
||||
|
||||
# 生成简单标题和摘要
|
||||
if not result["title"]:
|
||||
first_line = text.split('\n')[0].strip('# ')
|
||||
if len(first_line) < 50:
|
||||
result["title"] = first_line
|
||||
|
||||
if not result["summary"]:
|
||||
clean_text = re.sub(r'[#*\-`]', '', text)
|
||||
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ── 主流程 ─────────────────────────────────────────
|
||||
|
||||
def refine_inbox(root: str, dry_run: bool = False, date_str: str = None):
|
||||
"""执行收件箱提纯"""
|
||||
root = Path(os.path.expanduser(root))
|
||||
kb_root = root / "03_knowledge"
|
||||
inbox = kb_root / "00_inbox"
|
||||
|
||||
if not inbox.exists():
|
||||
print(f"❌ 收件箱不存在: {inbox}")
|
||||
return
|
||||
|
||||
date_str = date_str or datetime.now().strftime("%Y-%m-%d")
|
||||
md_files = sorted(inbox.glob("*.md"))
|
||||
|
||||
if not md_files:
|
||||
print(f"📭 收件箱为空,无需提纯 ({date_str})")
|
||||
# 仍然更新首页统计
|
||||
update_homepage(kb_root)
|
||||
return
|
||||
|
||||
print(f"📬 收件箱有 {len(md_files)} 个文件待提纯")
|
||||
|
||||
archived = []
|
||||
skipped = []
|
||||
errors = []
|
||||
|
||||
for md_file in md_files:
|
||||
try:
|
||||
text = md_file.read_text(encoding="utf-8", errors="ignore")
|
||||
fm = parse_frontmatter(text)
|
||||
title = fm.get("title", md_file.stem)
|
||||
|
||||
# 分类
|
||||
classification = classify_content(text, fm)
|
||||
nature = classification["nature"]
|
||||
target_dir_name = NATURE_TO_DIR.get(nature, "10_concepts")
|
||||
target_dir = kb_root / target_dir_name
|
||||
|
||||
# 去重检查
|
||||
is_dup = False
|
||||
for existing in target_dir.rglob("*.md"):
|
||||
existing_text = existing.read_text(encoding="utf-8", errors="ignore")[:200]
|
||||
existing_fm = parse_frontmatter(existing_text)
|
||||
if existing_fm.get("title") == title:
|
||||
is_dup = True
|
||||
break
|
||||
|
||||
if is_dup:
|
||||
print(f" ⏭ 重复跳过: {title}")
|
||||
skipped.append(title)
|
||||
if not dry_run:
|
||||
md_file.unlink()
|
||||
continue
|
||||
|
||||
# 生成 ID 和新文件
|
||||
kb_id = generate_id(date_str, kb_root)
|
||||
domain = classification["domain"]
|
||||
confidence = classification["confidence"]
|
||||
source = classification["source"]
|
||||
source_type = classification["source_type"]
|
||||
|
||||
# 构建新内容
|
||||
body = text
|
||||
if text.startswith("---"):
|
||||
# 去掉旧 frontmatter
|
||||
body = re.sub(r'^---\s*\n.*?\n---\s*\n', '', text, count=1, flags=re.DOTALL)
|
||||
|
||||
new_content = build_frontmatter(
|
||||
kb_id,
|
||||
classification.get("title", title),
|
||||
nature,
|
||||
domain,
|
||||
confidence,
|
||||
source,
|
||||
source_type,
|
||||
date_str,
|
||||
classification.get("summary", ""),
|
||||
classification.get("tags", [])
|
||||
) + "\n\n" + body.strip() + "\n"
|
||||
|
||||
# 确定目标文件名
|
||||
safe_title = re.sub(r'[^\w\u4e00-\u9fff-]', '_', title)[:50]
|
||||
target_file = target_dir / f"{date_str}_{safe_title}.md"
|
||||
|
||||
if dry_run:
|
||||
print(f" 🔍 [DRY] {title} → {target_dir_name}/ ({nature})")
|
||||
else:
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
target_file.write_text(new_content, encoding="utf-8")
|
||||
md_file.unlink()
|
||||
print(f" ✅ {title} → {target_dir_name}/ ({nature}) [{kb_id}]")
|
||||
|
||||
archived.append({
|
||||
"id": kb_id,
|
||||
"title": title,
|
||||
"nature": nature,
|
||||
"dir": target_dir_name,
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ 处理失败: {md_file.name} - {e}")
|
||||
errors.append(md_file.name)
|
||||
|
||||
# 更新首页
|
||||
if not dry_run:
|
||||
update_homepage(kb_root)
|
||||
update_changelog(kb_root, date_str, archived, skipped)
|
||||
|
||||
print(f"\n📊 提纯完成: 归档 {len(archived)}, 跳过 {len(skipped)}, 失败 {len(errors)}")
|
||||
|
||||
|
||||
def update_homepage(kb_root: Path):
|
||||
"""更新知识库首页统计"""
|
||||
readme = kb_root / "README.md"
|
||||
if not readme.exists():
|
||||
print("⚠️ 首页 README.md 不存在,跳过统计更新")
|
||||
return
|
||||
|
||||
# 计算各目录统计
|
||||
stats = {}
|
||||
total = 0
|
||||
for dirname in TRACKED_DIRS:
|
||||
dirpath = kb_root / dirname
|
||||
count = count_md_files(dirpath)
|
||||
latest = latest_modified(dirpath)
|
||||
stats[dirname] = (count, latest)
|
||||
total += count
|
||||
|
||||
# 构建统计表格
|
||||
table_lines = []
|
||||
for dirname in TRACKED_DIRS:
|
||||
count, latest = stats[dirname]
|
||||
emoji_name = DIR_CHINESE.get(dirname, dirname)
|
||||
table_lines.append(f"| {emoji_name} | {count} | {latest} |")
|
||||
table_lines.append(f"| **总计** | **{total}** | — |")
|
||||
|
||||
table_text = "\n".join(table_lines)
|
||||
|
||||
# 替换 README 中的统计部分
|
||||
content = readme.read_text(encoding="utf-8")
|
||||
pattern = r'(\| 📥 收件箱 \|.*?)(\| \*\*总计\*\* \| \*\*\d+\*\* \| — \|)'
|
||||
replacement = table_text
|
||||
content = re.sub(pattern, replacement, content, flags=re.DOTALL)
|
||||
|
||||
# 更新最后更新日期
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
content = re.sub(
|
||||
r'\*最后更新:.*? by inbox_refine 技能\*',
|
||||
f'*最后更新:{today} by inbox_refine 技能*',
|
||||
content
|
||||
)
|
||||
|
||||
readme.write_text(content, encoding="utf-8")
|
||||
print(f"📄 首页统计已更新 (总计 {total} 个文件)")
|
||||
|
||||
|
||||
def update_changelog(kb_root: Path, date_str: str, archived: list, skipped: list):
|
||||
"""更新变更日志"""
|
||||
changelog = kb_root / "CHANGELOG.md"
|
||||
if not changelog.exists():
|
||||
initial = "# 知识库变更日志\n"
|
||||
changelog.write_text(initial, encoding="utf-8")
|
||||
|
||||
content = changelog.read_text(encoding="utf-8")
|
||||
|
||||
# 构建新条目
|
||||
entry = f"\n## {date_str}\n\n"
|
||||
if archived:
|
||||
entry += "### 归档\n"
|
||||
for item in archived:
|
||||
entry += f"- [{item['nature']}] {item['title']} → {item['dir']}/ ({item['id']})\n"
|
||||
if skipped:
|
||||
entry += f"\n### 跳过(重复)\n"
|
||||
for title in skipped:
|
||||
entry += f"- {title}\n"
|
||||
|
||||
# 在第一个 ## 之前插入(跳过标题行)
|
||||
lines = content.split("\n")
|
||||
insert_idx = 1
|
||||
for i, line in enumerate(lines):
|
||||
if line.startswith("## "):
|
||||
insert_idx = i
|
||||
break
|
||||
|
||||
lines.insert(insert_idx, entry)
|
||||
changelog.write_text("\n".join(lines), encoding="utf-8")
|
||||
print(f"📝 变更日志已更新")
|
||||
|
||||
|
||||
# ── 入口 ──────────────────────────────────────────
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="AgentOS 知识库收件箱提纯")
|
||||
parser.add_argument("--root", default="~/workbuddy-agent-os/agent-sync", help="agent-os 根目录")
|
||||
parser.add_argument("--dry-run", action="store_true", help="试运行,不实际修改文件")
|
||||
parser.add_argument("--date", default=None, help="日期 (YYYY-MM-DD)")
|
||||
args = parser.parse_args()
|
||||
|
||||
refine_inbox(args.root, args.dry_run, args.date)
|
||||
@@ -0,0 +1,395 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LLM 分类器模块 - 使用 oMLX 本地大模型进行知识分类
|
||||
|
||||
功能:
|
||||
1. 根据内容判断 nature(fact/method/concept/opinion/reference 等)
|
||||
2. 根据内容判断 domain(技术/金融/产品/生活 等)
|
||||
3. 提取关键信息(标题、摘要、标签)
|
||||
|
||||
配置:
|
||||
- oMLX API: http://localhost:8000/v1/chat/completions
|
||||
- 模型: Qwen2.5-VL-3B-Instruct-8bit
|
||||
- API Key: omlx
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
from pathlib import Path
|
||||
|
||||
# ─── oMLX 配置 ──────────────────────────────────────────────────
|
||||
OLLAMA_API_URL = "http://localhost:8000/v1/chat/completions"
|
||||
OLLAMA_MODEL = "Qwen2.5-VL-3B-Instruct-8bit" # 使用 VLM 模型,因为 Qwen3-8B-MLX-4bit 有 500 错误
|
||||
OLLAMA_API_KEY = "omlx"
|
||||
|
||||
# 备份模型选项(如果 Qwen2.5 不可用)
|
||||
FALLBACK_MODEL = "Qwen3-8B-MLX-4bit"
|
||||
|
||||
# ─── 分类体系 ──────────────────────────────────────────────────
|
||||
NATURE_OPTIONS = [
|
||||
"fact", # 客观事实、数据、统计结果
|
||||
"method", # 方法、步骤、教程、最佳实践
|
||||
"concept", # 概念、定义、理论、原理
|
||||
"opinion", # 观点、看法、评价、推测
|
||||
"reference", # 参考文献、论文、书籍、资料
|
||||
"quote", # 引用、名言、摘录
|
||||
"axiom", # 公理、定律、定理
|
||||
"regulation", # 法规、政策、标准
|
||||
"data", # 原始数据、数据集、代码片段
|
||||
]
|
||||
|
||||
DOMAIN_OPTIONS = [
|
||||
"technology", # 技术、编程、AI、开发
|
||||
"finance", # 金融、投资、股票、经济
|
||||
"product", # 产品、设计、UX、PM
|
||||
"life", # 生活、健康、个人管理
|
||||
"academic", # 学术、研究、教育
|
||||
"business", # 商业、管理、创业
|
||||
"culture", # 文化、艺术、娱乐
|
||||
"other", # 其他
|
||||
]
|
||||
|
||||
# ─── LLM 调用函数 ──────────────────────────────────────────────
|
||||
def call_llm(prompt: str, system_prompt: str = None, temperature: float = 0.3) -> str:
|
||||
"""
|
||||
调用 oMLX 本地大模型 API
|
||||
|
||||
Args:
|
||||
prompt: 用户提示
|
||||
system_prompt: 系统提示(可选)
|
||||
temperature: 温度参数,越低越确定
|
||||
|
||||
Returns:
|
||||
LLM 回复文本
|
||||
"""
|
||||
import requests
|
||||
|
||||
messages = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
try:
|
||||
# 首先尝试 VLM 模型
|
||||
resp = requests.post(
|
||||
OLLAMA_API_URL,
|
||||
json={
|
||||
"model": OLLAMA_MODEL,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"stream": False
|
||||
},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OLLAMA_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
# 如果 VLM 模型失败(500 错误),尝试备选模型
|
||||
if resp.status_code == 500 and OLLAMA_MODEL == "Qwen2.5-VL-3B-Instruct-8bit":
|
||||
print(f" ⚠️ VLM 模型失败,尝试备选模型 {FALLBACK_MODEL}")
|
||||
resp = requests.post(
|
||||
OLLAMA_API_URL,
|
||||
json={
|
||||
"model": FALLBACK_MODEL,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"stream": False
|
||||
},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OLLAMA_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data["choices"][0]["message"]["content"]
|
||||
except Exception as e:
|
||||
print(f"[WARN] LLM 调用失败: {e}")
|
||||
return ""
|
||||
|
||||
# ─── 分类函数 ──────────────────────────────────────────────────
|
||||
def classify_with_llm(text: str) -> Dict:
|
||||
"""
|
||||
使用 LLM 对文本内容进行分类
|
||||
|
||||
Args:
|
||||
text: 要分类的文本内容(Markdown 格式)
|
||||
|
||||
Returns:
|
||||
{
|
||||
"nature": "fact/method/concept/...",
|
||||
"domain": "technology/finance/...",
|
||||
"confidence": 0.0-1.0,
|
||||
"title": "建议的标题",
|
||||
"summary": "简短摘要",
|
||||
"tags": ["标签1", "标签2", ...]
|
||||
}
|
||||
"""
|
||||
# 限制文本长度(防止 token 过长)
|
||||
preview = text[:2000] if len(text) > 2000 else text
|
||||
|
||||
system_prompt = """你是一个知识分类专家。请根据提供的文本内容进行以下分析:
|
||||
1. 判断知识类型(nature):从以下选项中选择最合适的:fact(事实数据)、method(方法步骤)、concept(概念理论)、opinion(观点看法)、reference(参考文献)、quote(引用摘录)、axiom(公理定律)、regulation(法规政策)、data(原始数据)
|
||||
2. 判断所属领域(domain):从以下选项中选择1-2个:technology(技术)、finance(金融)、product(产品)、life(生活)、academic(学术)、business(商业)、culture(文化)、other(其他)
|
||||
3. 提取关键信息:建议一个简洁的标题、生成50字内的摘要、提取2-5个关键词标签
|
||||
|
||||
请以 JSON 格式返回结果,严格遵循以下结构:
|
||||
{
|
||||
"nature": "选择的类型",
|
||||
"domain": ["领域1", "领域2"],
|
||||
"confidence": 0.0-1.0之间的数字(表示分类信心度),
|
||||
"title": "建议标题",
|
||||
"summary": "简短摘要",
|
||||
"tags": ["标签1", "标签2", "标签3"]
|
||||
}"""
|
||||
|
||||
user_prompt = f"""请对以下文本内容进行分类分析:
|
||||
|
||||
{preview}
|
||||
|
||||
注意:文本可能是截断的,请基于现有内容进行判断。"""
|
||||
|
||||
response = call_llm(user_prompt, system_prompt, temperature=0.1)
|
||||
|
||||
# 解析 JSON 响应
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.5,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
if not response:
|
||||
return result
|
||||
|
||||
try:
|
||||
# 查找 JSON 部分
|
||||
json_match = re.search(r'\{.*\}', response, re.DOTALL)
|
||||
if json_match:
|
||||
parsed = json.loads(json_match.group())
|
||||
if isinstance(parsed, dict):
|
||||
if "nature" in parsed and parsed["nature"] in NATURE_OPTIONS:
|
||||
result["nature"] = parsed["nature"]
|
||||
if "domain" in parsed:
|
||||
if isinstance(parsed["domain"], str):
|
||||
result["domain"] = [parsed["domain"]]
|
||||
elif isinstance(parsed["domain"], list):
|
||||
result["domain"] = [d for d in parsed["domain"] if d in DOMAIN_OPTIONS]
|
||||
if "confidence" in parsed:
|
||||
try:
|
||||
conf = float(parsed["confidence"])
|
||||
if 0 <= conf <= 1:
|
||||
result["confidence"] = conf
|
||||
except:
|
||||
pass
|
||||
if "title" in parsed:
|
||||
result["title"] = str(parsed["title"])
|
||||
if "summary" in parsed:
|
||||
result["summary"] = str(parsed["summary"])
|
||||
if "tags" in parsed:
|
||||
if isinstance(parsed["tags"], list):
|
||||
result["tags"] = [str(tag) for tag in parsed["tags"]]
|
||||
except Exception as e:
|
||||
print(f"[WARN] LLM 响应解析失败: {e}")
|
||||
|
||||
return result
|
||||
|
||||
# ─── 启发式分类备用方案 ────────────────────────────────────────
|
||||
def classify_heuristic(text: str, fm: dict) -> Dict:
|
||||
"""
|
||||
启发式分类(当 LLM 不可用时使用)
|
||||
|
||||
Args:
|
||||
text: 文本内容
|
||||
fm: 已有的 frontmatter
|
||||
|
||||
Returns:
|
||||
与 classify_with_llm 相同结构的字典
|
||||
"""
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.4,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature"):
|
||||
result["nature"] = fm["nature"]
|
||||
if fm.get("domain"):
|
||||
if isinstance(fm["domain"], str):
|
||||
result["domain"] = [fm["domain"]]
|
||||
elif isinstance(fm["domain"], list):
|
||||
result["domain"] = fm["domain"]
|
||||
|
||||
# 启发式规则
|
||||
lower = text.lower()
|
||||
|
||||
# nature 判断
|
||||
if not fm.get("nature"):
|
||||
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践"]
|
||||
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
|
||||
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
|
||||
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
|
||||
|
||||
if any(kw in lower for kw in method_kw):
|
||||
result["nature"] = "method"
|
||||
elif any(kw in lower for kw in fact_kw):
|
||||
result["nature"] = "fact"
|
||||
elif any(kw in lower for kw in opinion_kw):
|
||||
result["nature"] = "opinion"
|
||||
elif any(kw in lower for kw in reference_kw):
|
||||
result["nature"] = "reference"
|
||||
|
||||
# domain 判断
|
||||
if not fm.get("domain"):
|
||||
tech_kw = ["编程", "代码", "开发", "python", "javascript", "ai", "llm", "机器学习", "技术"]
|
||||
finance_kw = ["投资", "股票", "基金", "finance", "交易", "财经", "金融"]
|
||||
product_kw = ["产品", "设计", "ux", "product", "用户体验"]
|
||||
|
||||
if any(kw in lower for kw in tech_kw):
|
||||
result["domain"] = ["technology"]
|
||||
elif any(kw in lower for kw in finance_kw):
|
||||
result["domain"] = ["finance"]
|
||||
elif any(kw in lower for kw in product_kw):
|
||||
result["domain"] = ["product"]
|
||||
|
||||
# 生成简单标题(使用文件名或第一行)
|
||||
if fm.get("title"):
|
||||
result["title"] = fm["title"]
|
||||
else:
|
||||
first_line = text.split('\n')[0].strip('# ')
|
||||
if len(first_line) < 50:
|
||||
result["title"] = first_line
|
||||
|
||||
# 生成简单摘要(前 100 字符)
|
||||
clean_text = re.sub(r'[#*\-`]', '', text)
|
||||
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
|
||||
|
||||
return result
|
||||
|
||||
# ─── 主分类函数 ──────────────────────────────────────────────────
|
||||
def classify_content_enhanced(text: str, fm: dict = None, use_llm: bool = True) -> Dict:
|
||||
"""
|
||||
增强版分类函数:优先使用 LLM,失败时降级到启发式
|
||||
|
||||
Args:
|
||||
text: 文本内容
|
||||
fm: 已有的 frontmatter(可选)
|
||||
use_llm: 是否尝试使用 LLM(默认 True)
|
||||
|
||||
Returns:
|
||||
分类结果字典
|
||||
"""
|
||||
if fm is None:
|
||||
fm = {}
|
||||
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.5,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": [],
|
||||
"source_type": "unknown",
|
||||
"source": ""
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature"):
|
||||
result["nature"] = fm["nature"]
|
||||
result["confidence"] = max(result["confidence"], 0.7) # 已有信息提升信心
|
||||
|
||||
if fm.get("domain"):
|
||||
if isinstance(fm["domain"], str):
|
||||
result["domain"] = [fm["domain"]]
|
||||
elif isinstance(fm["domain"], list):
|
||||
result["domain"] = fm["domain"]
|
||||
|
||||
# 尝试使用 LLM 分类
|
||||
llm_result = None
|
||||
if use_llm and text.strip():
|
||||
try:
|
||||
print(" 🤖 使用 LLM 进行智能分类...")
|
||||
llm_result = classify_with_llm(text)
|
||||
|
||||
# 合并 LLM 结果(降低阈值要求,因为 VLM 模型可能返回较低的 confidence)
|
||||
confidence_threshold = 0.4 # 降低阈值,因为 VLM 模型可能返回较低的 confidence
|
||||
if llm_result.get("confidence", 0) > confidence_threshold:
|
||||
if not fm.get("nature") and llm_result.get("nature"):
|
||||
result["nature"] = llm_result["nature"]
|
||||
if not fm.get("domain") and llm_result.get("domain"):
|
||||
result["domain"] = llm_result["domain"]
|
||||
if llm_result.get("title"):
|
||||
result["title"] = llm_result["title"]
|
||||
if llm_result.get("summary"):
|
||||
result["summary"] = llm_result["summary"]
|
||||
if llm_result.get("tags"):
|
||||
result["tags"] = llm_result["tags"]
|
||||
result["confidence"] = llm_result.get("confidence", 0.6)
|
||||
print(f" ✅ LLM 分类成功 (confidence: {llm_result.get('confidence', 0.6):.2f})")
|
||||
else:
|
||||
print(f" ⚠️ LLM 分类信心度低 ({llm_result.get('confidence', 0):.2f} < {confidence_threshold}),使用启发式规则")
|
||||
llm_result = None
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ LLM 分类失败: {e}")
|
||||
llm_result = None
|
||||
|
||||
# LLM 失败或信心不足,使用启发式规则
|
||||
if llm_result is None:
|
||||
heuristic_result = classify_heuristic(text, fm)
|
||||
|
||||
# 合并启发式结果
|
||||
if not fm.get("nature") and heuristic_result.get("nature"):
|
||||
result["nature"] = heuristic_result["nature"]
|
||||
if not fm.get("domain") and heuristic_result.get("domain"):
|
||||
result["domain"] = heuristic_result["domain"]
|
||||
if not result["title"] and heuristic_result.get("title"):
|
||||
result["title"] = heuristic_result["title"]
|
||||
if not result["summary"] and heuristic_result.get("summary"):
|
||||
result["summary"] = heuristic_result["summary"]
|
||||
|
||||
# 来源信息
|
||||
if fm.get("source"):
|
||||
result["source"] = fm["source"]
|
||||
if fm.get("source_type"):
|
||||
result["source_type"] = fm["source_type"]
|
||||
|
||||
# 防止空列表
|
||||
if not result["domain"]:
|
||||
result["domain"] = ["other"]
|
||||
if not result["tags"]:
|
||||
result["tags"] = ["待补充"]
|
||||
|
||||
return result
|
||||
|
||||
# ─── 测试函数 ──────────────────────────────────────────────────
|
||||
if __name__ == "__main__":
|
||||
# 测试代码
|
||||
test_text = """Python 中的装饰器是一种强大的工具,它允许在不修改原函数代码的情况下,为函数添加额外的功能。
|
||||
|
||||
装饰器的基本语法:
|
||||
@decorator
|
||||
def my_function():
|
||||
pass
|
||||
|
||||
常见的装饰器应用场景包括:日志记录、性能测试、权限验证等。"""
|
||||
|
||||
print("测试 LLM 分类:")
|
||||
result = classify_content_enhanced(test_text, use_llm=True)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
print("\n测试启发式分类(关闭 LLM):")
|
||||
result2 = classify_content_enhanced(test_text, use_llm=False)
|
||||
print(json.dumps(result2, ensure_ascii=False, indent=2))
|
||||
@@ -0,0 +1,394 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LLM 分类器模块 - 使用 oMLX 本地大模型进行知识分类
|
||||
|
||||
功能:
|
||||
1. 根据内容判断 nature(fact/method/concept/opinion/reference 等)
|
||||
2. 根据内容判断 domain(技术/金融/产品/生活 等)
|
||||
3. 提取关键信息(标题、摘要、标签)
|
||||
|
||||
配置:
|
||||
- oMLX API: http://localhost:8000/v1/chat/completions
|
||||
- 模型: Qwen3-8B-MLX-4bit
|
||||
- API Key: omlx
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
from pathlib import Path
|
||||
|
||||
# ─── oMLX 配置 ──────────────────────────────────────────────────
|
||||
OLLAMA_API_URL = "http://localhost:8000/v1/chat/completions"
|
||||
OLLAMA_MODEL = "Qwen2.5-VL-3B-Instruct-8bit" # 使用 VLM 模型,因为 Qwen3-8B-MLX-4bit 有 500 错误
|
||||
OLLAMA_API_KEY = "omlx"
|
||||
|
||||
# 备份模型选项(如果 Qwen2.5 不可用)
|
||||
FALLBACK_MODEL = "Qwen3-8B-MLX-4bit"
|
||||
|
||||
# ─── 分类体系 ──────────────────────────────────────────────────
|
||||
NATURE_OPTIONS = [
|
||||
"fact", # 客观事实、数据、统计结果
|
||||
"method", # 方法、步骤、教程、最佳实践
|
||||
"concept", # 概念、定义、理论、原理
|
||||
"opinion", # 观点、看法、评价、推测
|
||||
"reference", # 参考文献、论文、书籍、资料
|
||||
"quote", # 引用、名言、摘录
|
||||
"axiom", # 公理、定律、定理
|
||||
"regulation", # 法规、政策、标准
|
||||
"data", # 原始数据、数据集、代码片段
|
||||
]
|
||||
|
||||
DOMAIN_OPTIONS = [
|
||||
"technology", # 技术、编程、AI、开发
|
||||
"finance", # 金融、投资、股票、经济
|
||||
"product", # 产品、设计、UX、PM
|
||||
"life", # 生活、健康、个人管理
|
||||
"academic", # 学术、研究、教育
|
||||
"business", # 商业、管理、创业
|
||||
"culture", # 文化、艺术、娱乐
|
||||
"other", # 其他
|
||||
]
|
||||
|
||||
# ─── LLM 调用函数 ──────────────────────────────────────────────
|
||||
def call_llm(prompt: str, system_prompt: str = None, temperature: float = 0.3) -> str:
|
||||
"""
|
||||
调用 oMLX 本地大模型 API
|
||||
|
||||
Args:
|
||||
prompt: 用户提示
|
||||
system_prompt: 系统提示(可选)
|
||||
temperature: 温度参数,越低越确定
|
||||
|
||||
Returns:
|
||||
LLM 回复文本
|
||||
"""
|
||||
import requests
|
||||
|
||||
messages = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
try:
|
||||
# 首先尝试 VLM 模型
|
||||
resp = requests.post(
|
||||
OLLAMA_API_URL,
|
||||
json={
|
||||
"model": OLLAMA_MODEL,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"stream": False
|
||||
},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OLLAMA_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
# 如果 VLM 模型失败(500 错误),尝试备选模型
|
||||
if resp.status_code == 500 and OLLAMA_MODEL == "Qwen2.5-VL-3B-Instruct-8bit":
|
||||
print(f" ⚠️ VLM 模型失败,尝试备选模型 {FALLBACK_MODEL}")
|
||||
resp = requests.post(
|
||||
OLLAMA_API_URL,
|
||||
json={
|
||||
"model": FALLBACK_MODEL,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"stream": False
|
||||
},
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {OLLAMA_API_KEY}"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data["choices"][0]["message"]["content"]
|
||||
except Exception as e:
|
||||
print(f"[WARN] LLM 调用失败: {e}")
|
||||
return ""
|
||||
|
||||
# ─── 分类函数 ──────────────────────────────────────────────────
|
||||
def classify_with_llm(text: str) -> Dict:
|
||||
"""
|
||||
使用 LLM 对文本内容进行分类
|
||||
|
||||
Args:
|
||||
text: 要分类的文本内容(Markdown 格式)
|
||||
|
||||
Returns:
|
||||
{
|
||||
"nature": "fact/method/concept/...",
|
||||
"domain": "technology/finance/...",
|
||||
"confidence": 0.0-1.0,
|
||||
"title": "建议的标题",
|
||||
"summary": "简短摘要",
|
||||
"tags": ["标签1", "标签2", ...]
|
||||
}
|
||||
"""
|
||||
# 限制文本长度(防止 token 过长)
|
||||
preview = text[:2000] if len(text) > 2000 else text
|
||||
|
||||
system_prompt = """你是一个知识分类专家。请根据提供的文本内容进行以下分析:
|
||||
1. 判断知识类型(nature):从以下选项中选择最合适的:fact(事实数据)、method(方法步骤)、concept(概念理论)、opinion(观点看法)、reference(参考文献)、quote(引用摘录)、axiom(公理定律)、regulation(法规政策)、data(原始数据)
|
||||
2. 判断所属领域(domain):从以下选项中选择1-2个:technology(技术)、finance(金融)、product(产品)、life(生活)、academic(学术)、business(商业)、culture(文化)、other(其他)
|
||||
3. 提取关键信息:建议一个简洁的标题、生成50字内的摘要、提取2-5个关键词标签
|
||||
|
||||
请以 JSON 格式返回结果,严格遵循以下结构:
|
||||
{
|
||||
"nature": "选择的类型",
|
||||
"domain": ["领域1", "领域2"],
|
||||
"confidence": 0.0-1.0之间的数字(表示分类信心度),
|
||||
"title": "建议标题",
|
||||
"summary": "简短摘要",
|
||||
"tags": ["标签1", "标签2", "标签3"]
|
||||
}"""
|
||||
|
||||
user_prompt = f"""请对以下文本内容进行分类分析:
|
||||
|
||||
{preview}
|
||||
|
||||
注意:文本可能是截断的,请基于现有内容进行判断。"""
|
||||
|
||||
response = call_llm(user_prompt, system_prompt, temperature=0.1)
|
||||
|
||||
# 解析 JSON 响应
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.5,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
if not response:
|
||||
return result
|
||||
|
||||
try:
|
||||
# 查找 JSON 部分
|
||||
json_match = re.search(r'\{.*\}', response, re.DOTALL)
|
||||
if json_match:
|
||||
parsed = json.loads(json_match.group())
|
||||
if isinstance(parsed, dict):
|
||||
if "nature" in parsed and parsed["nature"] in NATURE_OPTIONS:
|
||||
result["nature"] = parsed["nature"]
|
||||
if "domain" in parsed:
|
||||
if isinstance(parsed["domain"], str):
|
||||
result["domain"] = [parsed["domain"]]
|
||||
elif isinstance(parsed["domain"], list):
|
||||
result["domain"] = [d for d in parsed["domain"] if d in DOMAIN_OPTIONS]
|
||||
if "confidence" in parsed:
|
||||
try:
|
||||
conf = float(parsed["confidence"])
|
||||
if 0 <= conf <= 1:
|
||||
result["confidence"] = conf
|
||||
except:
|
||||
pass
|
||||
if "title" in parsed:
|
||||
result["title"] = str(parsed["title"])
|
||||
if "summary" in parsed:
|
||||
result["summary"] = str(parsed["summary"])
|
||||
if "tags" in parsed:
|
||||
if isinstance(parsed["tags"], list):
|
||||
result["tags"] = [str(tag) for tag in parsed["tags"]]
|
||||
except Exception as e:
|
||||
print(f"[WARN] LLM 响应解析失败: {e}")
|
||||
|
||||
return result
|
||||
|
||||
# ─── 启发式分类备用方案 ────────────────────────────────────────
|
||||
def classify_heuristic(text: str, fm: dict) -> Dict:
|
||||
"""
|
||||
启发式分类(当 LLM 不可用时使用)
|
||||
|
||||
Args:
|
||||
text: 文本内容
|
||||
fm: 已有的 frontmatter
|
||||
|
||||
Returns:
|
||||
与 classify_with_llm 相同结构的字典
|
||||
"""
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.4,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": []
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature"):
|
||||
result["nature"] = fm["nature"]
|
||||
if fm.get("domain"):
|
||||
if isinstance(fm["domain"], str):
|
||||
result["domain"] = [fm["domain"]]
|
||||
elif isinstance(fm["domain"], list):
|
||||
result["domain"] = fm["domain"]
|
||||
|
||||
# 启发式规则
|
||||
lower = text.lower()
|
||||
|
||||
# nature 判断
|
||||
if not fm.get("nature"):
|
||||
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践"]
|
||||
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
|
||||
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
|
||||
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
|
||||
|
||||
if any(kw in lower for kw in method_kw):
|
||||
result["nature"] = "method"
|
||||
elif any(kw in lower for kw in fact_kw):
|
||||
result["nature"] = "fact"
|
||||
elif any(kw in lower for kw in opinion_kw):
|
||||
result["nature"] = "opinion"
|
||||
elif any(kw in lower for kw in reference_kw):
|
||||
result["nature"] = "reference"
|
||||
|
||||
# domain 判断
|
||||
if not fm.get("domain"):
|
||||
tech_kw = ["编程", "代码", "开发", "python", "javascript", "ai", "llm", "机器学习", "技术"]
|
||||
finance_kw = ["投资", "股票", "基金", "finance", "交易", "财经", "金融"]
|
||||
product_kw = ["产品", "设计", "ux", "product", "用户体验"]
|
||||
|
||||
if any(kw in lower for kw in tech_kw):
|
||||
result["domain"] = ["technology"]
|
||||
elif any(kw in lower for kw in finance_kw):
|
||||
result["domain"] = ["finance"]
|
||||
elif any(kw in lower for kw in product_kw):
|
||||
result["domain"] = ["product"]
|
||||
|
||||
# 生成简单标题(使用文件名或第一行)
|
||||
if fm.get("title"):
|
||||
result["title"] = fm["title"]
|
||||
else:
|
||||
first_line = text.split('\n')[0].strip('# ')
|
||||
if len(first_line) < 50:
|
||||
result["title"] = first_line
|
||||
|
||||
# 生成简单摘要(前 100 字符)
|
||||
clean_text = re.sub(r'[#*\-`]', '', text)
|
||||
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
|
||||
|
||||
return result
|
||||
|
||||
# ─── 主分类函数 ──────────────────────────────────────────────────
|
||||
def classify_content_enhanced(text: str, fm: dict = None, use_llm: bool = True) -> Dict:
|
||||
"""
|
||||
增强版分类函数:优先使用 LLM,失败时降级到启发式
|
||||
|
||||
Args:
|
||||
text: 文本内容
|
||||
fm: 已有的 frontmatter(可选)
|
||||
use_llm: 是否尝试使用 LLM(默认 True)
|
||||
|
||||
Returns:
|
||||
分类结果字典
|
||||
"""
|
||||
if fm is None:
|
||||
fm = {}
|
||||
|
||||
result = {
|
||||
"nature": "concept",
|
||||
"domain": ["other"],
|
||||
"confidence": 0.5,
|
||||
"title": "",
|
||||
"summary": "",
|
||||
"tags": [],
|
||||
"source_type": "unknown",
|
||||
"source": ""
|
||||
}
|
||||
|
||||
# 优先使用已有 frontmatter
|
||||
if fm.get("nature"):
|
||||
result["nature"] = fm["nature"]
|
||||
result["confidence"] = max(result["confidence"], 0.7) # 已有信息提升信心
|
||||
|
||||
if fm.get("domain"):
|
||||
if isinstance(fm["domain"], str):
|
||||
result["domain"] = [fm["domain"]]
|
||||
elif isinstance(fm["domain"], list):
|
||||
result["domain"] = fm["domain"]
|
||||
|
||||
# 尝试使用 LLM 分类
|
||||
llm_result = None
|
||||
if use_llm and text.strip():
|
||||
try:
|
||||
print(" 🤖 使用 LLM 进行智能分类...")
|
||||
llm_result = classify_with_llm(text)
|
||||
|
||||
# 合并 LLM 结果(降低阈值要求,因为 VLM 模型可能返回较低的 confidence)
|
||||
confidence_threshold = 0.4 # 降低阈值,因为 VLM 模型可能返回较低的 confidence
|
||||
if llm_result.get("confidence", 0) > confidence_threshold:
|
||||
if not fm.get("nature") and llm_result.get("nature"):
|
||||
result["nature"] = llm_result["nature"]
|
||||
if not fm.get("domain") and llm_result.get("domain"):
|
||||
result["domain"] = llm_result["domain"]
|
||||
if llm_result.get("title"):
|
||||
result["title"] = llm_result["title"]
|
||||
if llm_result.get("summary"):
|
||||
result["summary"] = llm_result["summary"]
|
||||
if llm_result.get("tags"):
|
||||
result["tags"] = llm_result["tags"]
|
||||
result["confidence"] = llm_result.get("confidence", 0.6)
|
||||
print(f" ✅ LLM 分类成功 (confidence: {llm_result.get('confidence', 0.6):.2f})")
|
||||
else:
|
||||
print(f" ⚠️ LLM 分类信心度低 ({llm_result.get('confidence', 0):.2f} < {confidence_threshold}),使用启发式规则")
|
||||
llm_result = None
|
||||
except Exception as e:
|
||||
print(f" ⚠️ LLM 分类失败: {e}")
|
||||
llm_result = None
|
||||
|
||||
# LLM 失败或信心不足,使用启发式规则
|
||||
if llm_result is None:
|
||||
heuristic_result = classify_heuristic(text, fm)
|
||||
|
||||
# 合并启发式结果
|
||||
if not fm.get("nature") and heuristic_result.get("nature"):
|
||||
result["nature"] = heuristic_result["nature"]
|
||||
if not fm.get("domain") and heuristic_result.get("domain"):
|
||||
result["domain"] = heuristic_result["domain"]
|
||||
if not result["title"] and heuristic_result.get("title"):
|
||||
result["title"] = heuristic_result["title"]
|
||||
if not result["summary"] and heuristic_result.get("summary"):
|
||||
result["summary"] = heuristic_result["summary"]
|
||||
|
||||
# 来源信息
|
||||
if fm.get("source"):
|
||||
result["source"] = fm["source"]
|
||||
if fm.get("source_type"):
|
||||
result["source_type"] = fm["source_type"]
|
||||
|
||||
# 防止空列表
|
||||
if not result["domain"]:
|
||||
result["domain"] = ["other"]
|
||||
if not result["tags"]:
|
||||
result["tags"] = ["待补充"]
|
||||
|
||||
return result
|
||||
|
||||
# ─── 测试函数 ──────────────────────────────────────────────────
|
||||
if __name__ == "__main__":
|
||||
# 测试代码
|
||||
test_text = """Python 中的装饰器是一种强大的工具,它允许在不修改原函数代码的情况下,为函数添加额外的功能。
|
||||
|
||||
装饰器的基本语法:
|
||||
@decorator
|
||||
def my_function():
|
||||
pass
|
||||
|
||||
常见的装饰器应用场景包括:日志记录、性能测试、权限验证等。"""
|
||||
|
||||
print("测试 LLM 分类:")
|
||||
result = classify_content_enhanced(test_text, use_llm=True)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
print("\n测试启发式分类(关闭 LLM):")
|
||||
result2 = classify_content_enhanced(test_text, use_llm=False)
|
||||
print(json.dumps(result2, ensure_ascii=False, indent=2))
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "inbox_refine",
|
||||
"version": "1.1.0",
|
||||
"description": "知识库收件箱提纯——每日将 00_inbox/ 内容分类归档到知识库,更新首页统计和变更日志",
|
||||
"created": "2026-04-25",
|
||||
"last_modified": "2026-04-25",
|
||||
"dependencies": ["file_read", "file_write", "python3"],
|
||||
"schedule": "daily 03:00",
|
||||
"paths": {
|
||||
"inbox": "~/workbuddy-agent-os/agent-sync/03_knowledge/00_inbox/",
|
||||
"knowledge_root": "~/workbuddy-agent-os/agent-sync/03_knowledge/",
|
||||
"homepage": "~/workbuddy-agent-os/agent-sync/03_knowledge/README.md",
|
||||
"changelog": "~/workbuddy-agent-os/agent-sync/03_knowledge/CHANGELOG.md",
|
||||
"templates": "~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/templates/",
|
||||
"taxonomies": "~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user