feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程,直接发 HTTP
Deploy VitePress site to Pages / build (push) Canceled after 0s
Deploy VitePress site to Pages / Deploy (push) Canceled after 0s

移植自 mac-agent-os 的 mediacrawler_adapter:不起子进程、不开页面,用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin,所以 ingest 那条链路一个字不用改。

**目前能用的(真环境实测,非推断)**:

    profile/other : 200, 7075 字节  —— 博主主页指标(粉丝/获赞/作品数/昵称)
    aweme/detail  : 200, 45425 字节 —— 单条作品详情(含点赞/评论/收藏/分享)

**目前不能用的:作品列表 `aweme/post`。** 两个互相独立的原因:
  1. 这个接口被抖音单独升级成了真校验:不带 x-tt-argus 回 403「Uifid Not Found」,
     带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过,「真校验」过不了。
     同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的,
     挑中的恰好是「批量拉作品列表」这个最敏感的动作。
  2. 改走页面截获也不行:CDP 浏览器打开博主主页会落到「验证码中间页」(当天大量探测的
     代价,过几小时要重测)。

  所以现在的边界是:**已知作品的指标刷新能做,自动发现新作品做不了**。

**排查中控住变量后得到的两条事实**(都写进注释了):
  · `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
    UA+client hints+Cookie 时是 200 但仅 121 字节的空壳,补上这三个头变 7074 字节。
    (我先前猜的 sec-ch-ua 不是关键。)
  · 因此 UA 与 client hints 必须**成套地取自同一个浏览器**,所以 BrowserIdentity 一次
    从 CDP 取齐 cookie + UA + hints,而不是各自写死。

「200 + 空 body 必须当场报错」也是刻意写死的:放过去它会在下游变成「这个博主没作品」,
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的,还被翻译成「账号被封」。

测试 +11:cookie 解析、请求头成套性(含 uifid 缺失/回退)、产物键名与 store 对齐、
以及 _get 的三条失败路径(空 body / 403 带网关原话 / 正常返回)。
This commit is contained in:
2026-10-10 17:12:25 +08:00
parent 21ff01b894
commit 2112c1a870
1601 changed files with 367425 additions and 0 deletions
@@ -0,0 +1,140 @@
---
name: inbox_refine
version: 1.1.0
description: 知识库收件箱提纯——每日将 00_inbox/ 内容分类归档到知识库,更新首页统计和变更日志
triggers:
- 提纯
- 收件箱提纯
- 归档inbox
- inbox refine
- 整理收件箱
- 归档知识
- 提纯收件箱
---
# Inbox Refine 技能
## 概述
将 `00_inbox/` 中的原始内容按知识属性分类,应用模板后归档到对应目录,并更新知识库首页和变更日志。
## 触发条件
- 每日凌晨 3:00 自动执行(WorkBuddy 自动化)
- 手动说"提纯"、"整理收件箱"、"inbox refine"
## 工作流
```
1. 扫描 00_inbox/ 所有 .md 文件
2. 逐个读取 → 分析内容
3. 确定分类:
- nature:fact/method/concept/reference/data/opinion/quote/axiom/regulation
- domain:从 99_system/taxonomies/domains.md 选择 1-3 个
- confidence:根据 source_type 评分
4. 应用对应模板(99_system/templates/)
5. 生成 ID:KB-YYYYMMDD-NNN
6. 写入目标目录
7. 删除 00_inbox/ 原文件
8. 更新 README.md 统计
9. 更新 CHANGELOG.md
```
## 分类决策逻辑
### nature 判定
| 内容特征 | nature | 目标目录 |
|----------|--------|----------|
| 已验证的客观事实 | fact | 30_facts/ |
| 可操作的步骤/教程 | method | 20_methods/ |
| 原子概念/基础原理 | concept | 10_concepts/ |
| 公认基础原理 | axiom | 10_concepts/ |
| 法律法规/制度 | regulation | 30_facts/ |
| 论文/文档/外部资料 | reference | 40_references/ |
| 测试数据/基准结果 | data | 30_facts/ |
| 主观看法/推测 | opinion | 60_opinions/ |
| 他人原话/语录 | quote | 60_opinions/ |
### confidence 评分
| source_type | confidence |
|-------------|-----------|
| official_doc | 0.9 |
| literature | 0.8 |
| experiment | 0.7 |
| personal_exp | 0.5 |
| social_media | 0.3 |
| unknown | 0.4 |
## 去重规则
1. 读取目标目录已有文件,提取 title 和 tags
2. 标题相似度 > 0.8 → 合并(补充细节,不新建)
3. 内容完全一致 → 跳过,记录日志
4. 同一事实新旧矛盾 → 新事实覆盖,旧值写入 previous_version
## Frontmatter 模板
```yaml
---
id: KB-YYYYMMDD-NNN
title: "__标题__"
type: __concept/fact/method/opinion/reference__
status: published
nature: __fact/method/concept/...__
domain: [__领域1__, __领域2__]
subdomain: []
tags: [__标签1__, __标签2__]
confidence: 0.8
source: "__来源URL或出处__"
source_type: __official_doc/literature/personal_exp/...__
date_created: YYYY-MM-DD
date_modified: YYYY-MM-DD
version: 1
previous_version: ""
superseded_by: ""
---
```
## 更新首页统计
读取 `03_knowledge/README.md`,替换 `## 📊 知识统计` 下的表格:
- 扫描各目录下 .md 文件数量
- 找出每个目录最近修改的文件日期
- 更新总计行
## 更新变更日志
在 `03_knowledge/CHANGELOG.md` 顶部追加:
```markdown
## YYYY-MM-DD
### 归档
- [nature] title → 目标目录 (KB-YYYYMMDD-NNN)
- ...
```
## 依赖
| 包/工具 | 用途 | 安装状态 |
|---------|------|----------|
| file_read/file_write | 读写文件 | ✅ 内置 |
| python3 | 脚本执行 | ✅ managed |
| oMLX + Qwen3-8B-MLX-4bit | 本地 LLM 辅助分类(可选) | ✅ 已安装 |
## 路径约定
| 项目 | 路径 |
|------|------|
| 收件箱 | `~/workbuddy-agent-os/agent-sync/03_knowledge/00_inbox/` |
| 知识库根 | `~/workbuddy-agent-os/agent-sync/03_knowledge/` |
| 首页 | `~/workbuddy-agent-os/agent-sync/03_knowledge/README.md` |
| 变更日志 | `~/workbuddy-agent-os/agent-sync/03_knowledge/CHANGELOG.md` |
| 模板 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/templates/` |
| 分类表 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/` |
| 中文映射 | `~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/folder-aliases.json` |
## 错误处理
- 无法分类的内容 → 保留在 00_inbox/,添加标签 `#待分类`,记录日志
- 模板应用失败 → 使用最小 Frontmatter(id/title/nature/date_created)
- 首页更新失败 → 只记录日志,不阻断归档流程
@@ -0,0 +1,46 @@
skill:
name: "收件箱精炼器"
id: "inbox_refine"
version: "1.1.0"
category: "core-system"
status: "active"
created: "2026-04-25"
updated: "2026-04-29"
description:
short: "扫描协同inbox五分区,AI分类,自动归档到知识库"
use_cases:
- "定期扫描inbox各分区,按knowledge/memory/personal分类处理"
- "clipping分区:全文入库"
- "feed分区:摘要入库,原文归档"
- "memory分区:入长期记忆,同时建立知识卡"
- "personal分区:入personal知识区"
- "tools分区:转交工具审核流程"
tags: ["inbox", "精炼", "分类", "归档"]
ownership:
master_editor: "my-macbook_macOS14.4_16GB_master"
readonly_on_other_devices: false
environment:
os: "macos"
python_path: "~/.workbuddy/binaries/python/envs/agent-os/bin/python3"
pip_dependencies:
- "chromadb>=1.0.0"
setup:
install_script: ""
post_check: "python3 -c 'print(\"OK\")'"
usage:
entry_point: "inbox_refine.py"
scheduled: "每日 03:00(master机器执行)"
manual_trigger: "收件箱精炼"
changelog:
- version: "1.1.0"
date: "2026-04-29"
changes: "支持五分区差异化处理逻辑,适配新inbox路径"
- version: "1.0.0"
date: "2026-04-25"
changes: "初始版本"
@@ -0,0 +1,461 @@
#!/usr/bin/env python3
"""
inbox_refine.py - 知识库收件箱提纯脚本
将 00_inbox/ 中的原始内容按知识属性分类归档到对应目录,
更新知识库首页统计和变更日志。
用法:
python3 inbox_refine.py --root ~/workbuddy-agent-os/agent-sync [--dry-run] [--date YYYY-MM-DD]
"""
import os
import sys
import json
import re
import shutil
import argparse
from datetime import datetime
from pathlib import Path
# ─── 角色检查:仅 master 可执行收件箱提纯 ───
try:
sys.path.insert(0, str(Path.home() / "workbuddy-agent-os" / "agent-sync" / "05_tools" / "01_system"))
from role_check import check_role, check_capability
if not check_role(["master"]):
print("⛔ 非 master 角色,跳过 inbox_refine")
sys.exit(0)
if not check_capability("inbox_refine"):
print("⛔ 能力 inbox_refine 未启用,跳过")
sys.exit(0)
except ImportError as e:
print(f"⚠️ 角色检查模块不可用 ({e}),继续执行")
# 导入 LLM 分类器
try:
from llm_classifier import classify_content_enhanced
LLM_AVAILABLE = True
except ImportError:
print("⚠️ LLM 分类器不可用,将使用启发式规则")
LLM_AVAILABLE = False
# ── 配置 ──────────────────────────────────────────
NATURE_TO_DIR = {
"fact": "30_facts",
"method": "20_methods",
"concept": "10_concepts",
"axiom": "10_concepts",
"regulation": "30_facts",
"reference": "40_references",
"data": "30_facts",
"opinion": "60_opinions",
"quote": "60_opinions",
}
SOURCE_TYPE_CONFIDENCE = {
"official_doc": 0.9,
"literature": 0.8,
"experiment": 0.7,
"personal_exp": 0.5,
"social_media": 0.3,
"unknown": 0.4,
}
DIR_CHINESE = {
"00_inbox": "📥 收件箱",
"01_daily": "📅 日记",
"10_concepts": "💡 概念层",
"20_methods": "🔧 方法层",
"30_facts": "📋 事实层",
"40_references": "📎 参考层",
"50_resources": "🛠 资源层",
"60_opinions": "💭 观点层",
"90_archive": "🗄 归档层",
}
TRACKED_DIRS = [
"00_inbox", "01_daily", "10_concepts", "20_methods",
"30_facts", "40_references", "50_resources", "60_opinions", "90_archive"
]
# ── 工具函数 ──────────────────────────────────────
def count_md_files(directory: Path) -> int:
"""统计目录下 .md 文件数量"""
if not directory.exists():
return 0
return sum(1 for f in directory.rglob("*.md") if f.is_file())
def latest_modified(directory: Path) -> str:
"""获取目录下最近修改的 .md 文件日期"""
if not directory.exists():
return "—"
md_files = list(directory.rglob("*.md"))
if not md_files:
return "—"
latest = max(f.stat().st_mtime for f in md_files)
return datetime.fromtimestamp(latest).strftime("%Y-%m-%d")
def generate_id(date_str: str, knowledge_root: Path) -> str:
"""生成唯一知识ID:KB-YYYYMMDD-NNN"""
prefix = f"KB-{date_str.replace('-', '')}"
# 扫描已有ID确定序号
existing = set()
for md in knowledge_root.rglob("*.md"):
content = md.read_text(encoding="utf-8", errors="ignore")[:500]
m = re.search(r'^id:\s*(KB-\d{8}-\d+)', content, re.MULTILINE)
if m:
existing.add(m.group(1))
n = 1
while f"{prefix}-{n:03d}" in existing:
n += 1
return f"{prefix}-{n:03d}"
def parse_frontmatter(text: str) -> dict:
"""解析 YAML frontmatter"""
if not text.startswith("---"):
return {}
match = re.match(r'^---\s*\n(.*?)\n---', text, re.DOTALL)
if not match:
return {}
fm = {}
for line in match.group(1).split("\n"):
if ":" in line:
key, val = line.split(":", 1)
fm[key.strip()] = val.strip().strip('"').strip("'")
return fm
def build_frontmatter(kb_id: str, title: str, nature: str, domain: str,
confidence: float, source: str, source_type: str,
date_str: str, summary: str = "", tags: list = None) -> str:
"""构建知识卡片 frontmatter"""
if tags is None:
tags = ["待补充"]
# 确保 domain 是列表格式
if isinstance(domain, str):
domain_list = [domain]
else:
domain_list = domain if isinstance(domain, list) else ["other"]
# 构建 tags 字符串
tags_str = ", ".join([f'"{tag}"' for tag in tags])
return f"""---
id: {kb_id}
title: "{title}"
type: {nature if nature in ('concept', 'fact', 'method', 'opinion', 'reference') else 'concept'}
status: published
nature: {nature}
domain: [{", ".join(domain_list)}]
subdomain: []
tags: [{tags_str}]
confidence: {confidence}
source: "{source}"
source_type: {source_type}
date_created: {date_str}
date_modified: {date_str}
version: 1
previous_version: ""
superseded_by: ""
summary: "{summary}"
---"""
# ── 分类逻辑 ──────────────────────────────────────
def classify_content(text: str, fm: dict) -> dict:
"""
增强版分类函数:优先使用 LLM 智能分类,失败时降级到启发式规则。
如果 frontmatter 已有 nature/domain,优先使用。
"""
# 如果 LLM 可用,直接使用增强版分类
if LLM_AVAILABLE:
try:
# 直接使用 LLM 分类,分类器内部会处理降级
result = classify_content_enhanced(text, fm, use_llm=True)
# 适配现有结构
return {
"nature": result["nature"],
"domain": result["domain"][0] if result["domain"] else "other",
"confidence": result["confidence"],
"source_type": result.get("source_type", "unknown"),
"source": result.get("source", ""),
"title": result.get("title", ""),
"summary": result.get("summary", ""),
"tags": result.get("tags", [])
}
except Exception as e:
print(f"⚠️ LLM 分类失败,回退到启发式规则: {e}")
# LLM 不可用或失败,使用启发式规则
result = {
"nature": "concept",
"domain": "other",
"confidence": 0.4,
"source_type": "unknown",
"source": "",
"title": "",
"summary": "",
"tags": []
}
# 优先使用已有 frontmatter
if fm.get("nature") and fm["nature"] in NATURE_TO_DIR:
result["nature"] = fm["nature"]
if fm.get("domain"):
result["domain"] = fm["domain"]
if fm.get("confidence"):
try:
result["confidence"] = float(fm["confidence"])
except ValueError:
pass
if fm.get("source"):
result["source"] = fm["source"]
if fm.get("source_type"):
result["source_type"] = fm["source_type"]
if fm.get("title"):
result["title"] = fm["title"]
# 如果没有 nature,用启发式规则
if not fm.get("nature"):
lower = text.lower()
# 方法类关键词
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践", "best practice"]
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
if any(kw in lower for kw in method_kw):
result["nature"] = "method"
elif any(kw in lower for kw in fact_kw):
result["nature"] = "fact"
elif any(kw in lower for kw in opinion_kw):
result["nature"] = "opinion"
elif any(kw in lower for kw in reference_kw):
result["nature"] = "reference"
# confidence from source_type
if result["source_type"] in SOURCE_TYPE_CONFIDENCE:
result["confidence"] = SOURCE_TYPE_CONFIDENCE[result["source_type"]]
# 生成简单标题和摘要
if not result["title"]:
first_line = text.split('\n')[0].strip('# ')
if len(first_line) < 50:
result["title"] = first_line
if not result["summary"]:
clean_text = re.sub(r'[#*\-`]', '', text)
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
return result
# ── 主流程 ─────────────────────────────────────────
def refine_inbox(root: str, dry_run: bool = False, date_str: str = None):
"""执行收件箱提纯"""
root = Path(os.path.expanduser(root))
kb_root = root / "03_knowledge"
inbox = kb_root / "00_inbox"
if not inbox.exists():
print(f"❌ 收件箱不存在: {inbox}")
return
date_str = date_str or datetime.now().strftime("%Y-%m-%d")
md_files = sorted(inbox.glob("*.md"))
if not md_files:
print(f"📭 收件箱为空,无需提纯 ({date_str})")
# 仍然更新首页统计
update_homepage(kb_root)
return
print(f"📬 收件箱有 {len(md_files)} 个文件待提纯")
archived = []
skipped = []
errors = []
for md_file in md_files:
try:
text = md_file.read_text(encoding="utf-8", errors="ignore")
fm = parse_frontmatter(text)
title = fm.get("title", md_file.stem)
# 分类
classification = classify_content(text, fm)
nature = classification["nature"]
target_dir_name = NATURE_TO_DIR.get(nature, "10_concepts")
target_dir = kb_root / target_dir_name
# 去重检查
is_dup = False
for existing in target_dir.rglob("*.md"):
existing_text = existing.read_text(encoding="utf-8", errors="ignore")[:200]
existing_fm = parse_frontmatter(existing_text)
if existing_fm.get("title") == title:
is_dup = True
break
if is_dup:
print(f" ⏭ 重复跳过: {title}")
skipped.append(title)
if not dry_run:
md_file.unlink()
continue
# 生成 ID 和新文件
kb_id = generate_id(date_str, kb_root)
domain = classification["domain"]
confidence = classification["confidence"]
source = classification["source"]
source_type = classification["source_type"]
# 构建新内容
body = text
if text.startswith("---"):
# 去掉旧 frontmatter
body = re.sub(r'^---\s*\n.*?\n---\s*\n', '', text, count=1, flags=re.DOTALL)
new_content = build_frontmatter(
kb_id,
classification.get("title", title),
nature,
domain,
confidence,
source,
source_type,
date_str,
classification.get("summary", ""),
classification.get("tags", [])
) + "\n\n" + body.strip() + "\n"
# 确定目标文件名
safe_title = re.sub(r'[^\w\u4e00-\u9fff-]', '_', title)[:50]
target_file = target_dir / f"{date_str}_{safe_title}.md"
if dry_run:
print(f" 🔍 [DRY] {title} → {target_dir_name}/ ({nature})")
else:
target_dir.mkdir(parents=True, exist_ok=True)
target_file.write_text(new_content, encoding="utf-8")
md_file.unlink()
print(f" ✅ {title} → {target_dir_name}/ ({nature}) [{kb_id}]")
archived.append({
"id": kb_id,
"title": title,
"nature": nature,
"dir": target_dir_name,
})
except Exception as e:
print(f" ❌ 处理失败: {md_file.name} - {e}")
errors.append(md_file.name)
# 更新首页
if not dry_run:
update_homepage(kb_root)
update_changelog(kb_root, date_str, archived, skipped)
print(f"\n📊 提纯完成: 归档 {len(archived)}, 跳过 {len(skipped)}, 失败 {len(errors)}")
def update_homepage(kb_root: Path):
"""更新知识库首页统计"""
readme = kb_root / "README.md"
if not readme.exists():
print("⚠️ 首页 README.md 不存在,跳过统计更新")
return
# 计算各目录统计
stats = {}
total = 0
for dirname in TRACKED_DIRS:
dirpath = kb_root / dirname
count = count_md_files(dirpath)
latest = latest_modified(dirpath)
stats[dirname] = (count, latest)
total += count
# 构建统计表格
table_lines = []
for dirname in TRACKED_DIRS:
count, latest = stats[dirname]
emoji_name = DIR_CHINESE.get(dirname, dirname)
table_lines.append(f"| {emoji_name} | {count} | {latest} |")
table_lines.append(f"| **总计** | **{total}** | — |")
table_text = "\n".join(table_lines)
# 替换 README 中的统计部分
content = readme.read_text(encoding="utf-8")
pattern = r'(\| 📥 收件箱 \|.*?)(\| \*\*总计\*\* \| \*\*\d+\*\* \| — \|)'
replacement = table_text
content = re.sub(pattern, replacement, content, flags=re.DOTALL)
# 更新最后更新日期
today = datetime.now().strftime("%Y-%m-%d")
content = re.sub(
r'\*最后更新:.*? by inbox_refine 技能\*',
f'*最后更新:{today} by inbox_refine 技能*',
content
)
readme.write_text(content, encoding="utf-8")
print(f"📄 首页统计已更新 (总计 {total} 个文件)")
def update_changelog(kb_root: Path, date_str: str, archived: list, skipped: list):
"""更新变更日志"""
changelog = kb_root / "CHANGELOG.md"
if not changelog.exists():
initial = "# 知识库变更日志\n"
changelog.write_text(initial, encoding="utf-8")
content = changelog.read_text(encoding="utf-8")
# 构建新条目
entry = f"\n## {date_str}\n\n"
if archived:
entry += "### 归档\n"
for item in archived:
entry += f"- [{item['nature']}] {item['title']} → {item['dir']}/ ({item['id']})\n"
if skipped:
entry += f"\n### 跳过(重复)\n"
for title in skipped:
entry += f"- {title}\n"
# 在第一个 ## 之前插入(跳过标题行)
lines = content.split("\n")
insert_idx = 1
for i, line in enumerate(lines):
if line.startswith("## "):
insert_idx = i
break
lines.insert(insert_idx, entry)
changelog.write_text("\n".join(lines), encoding="utf-8")
print(f"📝 变更日志已更新")
# ── 入口 ──────────────────────────────────────────
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="AgentOS 知识库收件箱提纯")
parser.add_argument("--root", default="~/workbuddy-agent-os/agent-sync", help="agent-os 根目录")
parser.add_argument("--dry-run", action="store_true", help="试运行,不实际修改文件")
parser.add_argument("--date", default=None, help="日期 (YYYY-MM-DD)")
args = parser.parse_args()
refine_inbox(args.root, args.dry_run, args.date)
@@ -0,0 +1,395 @@
#!/usr/bin/env python3
"""
LLM 分类器模块 - 使用 oMLX 本地大模型进行知识分类
功能:
1. 根据内容判断 nature(fact/method/concept/opinion/reference 等)
2. 根据内容判断 domain(技术/金融/产品/生活 等)
3. 提取关键信息(标题、摘要、标签)
配置:
- oMLX API: http://localhost:8000/v1/chat/completions
- 模型: Qwen2.5-VL-3B-Instruct-8bit
- API Key: omlx
"""
import json
import re
import time
from typing import Dict, List, Optional, Tuple
from pathlib import Path
# ─── oMLX 配置 ──────────────────────────────────────────────────
OLLAMA_API_URL = "http://localhost:8000/v1/chat/completions"
OLLAMA_MODEL = "Qwen2.5-VL-3B-Instruct-8bit" # 使用 VLM 模型,因为 Qwen3-8B-MLX-4bit 有 500 错误
OLLAMA_API_KEY = "omlx"
# 备份模型选项(如果 Qwen2.5 不可用)
FALLBACK_MODEL = "Qwen3-8B-MLX-4bit"
# ─── 分类体系 ──────────────────────────────────────────────────
NATURE_OPTIONS = [
"fact", # 客观事实、数据、统计结果
"method", # 方法、步骤、教程、最佳实践
"concept", # 概念、定义、理论、原理
"opinion", # 观点、看法、评价、推测
"reference", # 参考文献、论文、书籍、资料
"quote", # 引用、名言、摘录
"axiom", # 公理、定律、定理
"regulation", # 法规、政策、标准
"data", # 原始数据、数据集、代码片段
]
DOMAIN_OPTIONS = [
"technology", # 技术、编程、AI、开发
"finance", # 金融、投资、股票、经济
"product", # 产品、设计、UX、PM
"life", # 生活、健康、个人管理
"academic", # 学术、研究、教育
"business", # 商业、管理、创业
"culture", # 文化、艺术、娱乐
"other", # 其他
]
# ─── LLM 调用函数 ──────────────────────────────────────────────
def call_llm(prompt: str, system_prompt: str = None, temperature: float = 0.3) -> str:
"""
调用 oMLX 本地大模型 API
Args:
prompt: 用户提示
system_prompt: 系统提示(可选)
temperature: 温度参数,越低越确定
Returns:
LLM 回复文本
"""
import requests
messages = []
if system_prompt:
messages.append({"role": "system", "content": system_prompt})
messages.append({"role": "user", "content": prompt})
try:
# 首先尝试 VLM 模型
resp = requests.post(
OLLAMA_API_URL,
json={
"model": OLLAMA_MODEL,
"messages": messages,
"temperature": temperature,
"stream": False
},
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {OLLAMA_API_KEY}"
},
timeout=30
)
# 如果 VLM 模型失败(500 错误),尝试备选模型
if resp.status_code == 500 and OLLAMA_MODEL == "Qwen2.5-VL-3B-Instruct-8bit":
print(f" ⚠️ VLM 模型失败,尝试备选模型 {FALLBACK_MODEL}")
resp = requests.post(
OLLAMA_API_URL,
json={
"model": FALLBACK_MODEL,
"messages": messages,
"temperature": temperature,
"stream": False
},
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {OLLAMA_API_KEY}"
},
timeout=30
)
resp.raise_for_status()
data = resp.json()
return data["choices"][0]["message"]["content"]
except Exception as e:
print(f"[WARN] LLM 调用失败: {e}")
return ""
# ─── 分类函数 ──────────────────────────────────────────────────
def classify_with_llm(text: str) -> Dict:
"""
使用 LLM 对文本内容进行分类
Args:
text: 要分类的文本内容(Markdown 格式)
Returns:
{
"nature": "fact/method/concept/...",
"domain": "technology/finance/...",
"confidence": 0.0-1.0,
"title": "建议的标题",
"summary": "简短摘要",
"tags": ["标签1", "标签2", ...]
}
"""
# 限制文本长度(防止 token 过长)
preview = text[:2000] if len(text) > 2000 else text
system_prompt = """你是一个知识分类专家。请根据提供的文本内容进行以下分析:
1. 判断知识类型(nature):从以下选项中选择最合适的:fact(事实数据)、method(方法步骤)、concept(概念理论)、opinion(观点看法)、reference(参考文献)、quote(引用摘录)、axiom(公理定律)、regulation(法规政策)、data(原始数据)
2. 判断所属领域(domain):从以下选项中选择1-2个:technology(技术)、finance(金融)、product(产品)、life(生活)、academic(学术)、business(商业)、culture(文化)、other(其他)
3. 提取关键信息:建议一个简洁的标题、生成50字内的摘要、提取2-5个关键词标签
请以 JSON 格式返回结果,严格遵循以下结构:
{
"nature": "选择的类型",
"domain": ["领域1", "领域2"],
"confidence": 0.0-1.0之间的数字(表示分类信心度),
"title": "建议标题",
"summary": "简短摘要",
"tags": ["标签1", "标签2", "标签3"]
}"""
user_prompt = f"""请对以下文本内容进行分类分析:
{preview}
注意:文本可能是截断的,请基于现有内容进行判断。"""
response = call_llm(user_prompt, system_prompt, temperature=0.1)
# 解析 JSON 响应
result = {
"nature": "concept",
"domain": ["other"],
"confidence": 0.5,
"title": "",
"summary": "",
"tags": []
}
if not response:
return result
try:
# 查找 JSON 部分
json_match = re.search(r'\{.*\}', response, re.DOTALL)
if json_match:
parsed = json.loads(json_match.group())
if isinstance(parsed, dict):
if "nature" in parsed and parsed["nature"] in NATURE_OPTIONS:
result["nature"] = parsed["nature"]
if "domain" in parsed:
if isinstance(parsed["domain"], str):
result["domain"] = [parsed["domain"]]
elif isinstance(parsed["domain"], list):
result["domain"] = [d for d in parsed["domain"] if d in DOMAIN_OPTIONS]
if "confidence" in parsed:
try:
conf = float(parsed["confidence"])
if 0 <= conf <= 1:
result["confidence"] = conf
except:
pass
if "title" in parsed:
result["title"] = str(parsed["title"])
if "summary" in parsed:
result["summary"] = str(parsed["summary"])
if "tags" in parsed:
if isinstance(parsed["tags"], list):
result["tags"] = [str(tag) for tag in parsed["tags"]]
except Exception as e:
print(f"[WARN] LLM 响应解析失败: {e}")
return result
# ─── 启发式分类备用方案 ────────────────────────────────────────
def classify_heuristic(text: str, fm: dict) -> Dict:
"""
启发式分类(当 LLM 不可用时使用)
Args:
text: 文本内容
fm: 已有的 frontmatter
Returns:
与 classify_with_llm 相同结构的字典
"""
result = {
"nature": "concept",
"domain": ["other"],
"confidence": 0.4,
"title": "",
"summary": "",
"tags": []
}
# 优先使用已有 frontmatter
if fm.get("nature"):
result["nature"] = fm["nature"]
if fm.get("domain"):
if isinstance(fm["domain"], str):
result["domain"] = [fm["domain"]]
elif isinstance(fm["domain"], list):
result["domain"] = fm["domain"]
# 启发式规则
lower = text.lower()
# nature 判断
if not fm.get("nature"):
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践"]
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
if any(kw in lower for kw in method_kw):
result["nature"] = "method"
elif any(kw in lower for kw in fact_kw):
result["nature"] = "fact"
elif any(kw in lower for kw in opinion_kw):
result["nature"] = "opinion"
elif any(kw in lower for kw in reference_kw):
result["nature"] = "reference"
# domain 判断
if not fm.get("domain"):
tech_kw = ["编程", "代码", "开发", "python", "javascript", "ai", "llm", "机器学习", "技术"]
finance_kw = ["投资", "股票", "基金", "finance", "交易", "财经", "金融"]
product_kw = ["产品", "设计", "ux", "product", "用户体验"]
if any(kw in lower for kw in tech_kw):
result["domain"] = ["technology"]
elif any(kw in lower for kw in finance_kw):
result["domain"] = ["finance"]
elif any(kw in lower for kw in product_kw):
result["domain"] = ["product"]
# 生成简单标题(使用文件名或第一行)
if fm.get("title"):
result["title"] = fm["title"]
else:
first_line = text.split('\n')[0].strip('# ')
if len(first_line) < 50:
result["title"] = first_line
# 生成简单摘要(前 100 字符)
clean_text = re.sub(r'[#*\-`]', '', text)
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
return result
# ─── 主分类函数 ──────────────────────────────────────────────────
def classify_content_enhanced(text: str, fm: dict = None, use_llm: bool = True) -> Dict:
"""
增强版分类函数:优先使用 LLM,失败时降级到启发式
Args:
text: 文本内容
fm: 已有的 frontmatter(可选)
use_llm: 是否尝试使用 LLM(默认 True)
Returns:
分类结果字典
"""
if fm is None:
fm = {}
result = {
"nature": "concept",
"domain": ["other"],
"confidence": 0.5,
"title": "",
"summary": "",
"tags": [],
"source_type": "unknown",
"source": ""
}
# 优先使用已有 frontmatter
if fm.get("nature"):
result["nature"] = fm["nature"]
result["confidence"] = max(result["confidence"], 0.7) # 已有信息提升信心
if fm.get("domain"):
if isinstance(fm["domain"], str):
result["domain"] = [fm["domain"]]
elif isinstance(fm["domain"], list):
result["domain"] = fm["domain"]
# 尝试使用 LLM 分类
llm_result = None
if use_llm and text.strip():
try:
print(" 🤖 使用 LLM 进行智能分类...")
llm_result = classify_with_llm(text)
# 合并 LLM 结果(降低阈值要求,因为 VLM 模型可能返回较低的 confidence)
confidence_threshold = 0.4 # 降低阈值,因为 VLM 模型可能返回较低的 confidence
if llm_result.get("confidence", 0) > confidence_threshold:
if not fm.get("nature") and llm_result.get("nature"):
result["nature"] = llm_result["nature"]
if not fm.get("domain") and llm_result.get("domain"):
result["domain"] = llm_result["domain"]
if llm_result.get("title"):
result["title"] = llm_result["title"]
if llm_result.get("summary"):
result["summary"] = llm_result["summary"]
if llm_result.get("tags"):
result["tags"] = llm_result["tags"]
result["confidence"] = llm_result.get("confidence", 0.6)
print(f" ✅ LLM 分类成功 (confidence: {llm_result.get('confidence', 0.6):.2f})")
else:
print(f" ⚠️ LLM 分类信心度低 ({llm_result.get('confidence', 0):.2f} < {confidence_threshold}),使用启发式规则")
llm_result = None
except Exception as e:
print(f" ⚠️ LLM 分类失败: {e}")
llm_result = None
# LLM 失败或信心不足,使用启发式规则
if llm_result is None:
heuristic_result = classify_heuristic(text, fm)
# 合并启发式结果
if not fm.get("nature") and heuristic_result.get("nature"):
result["nature"] = heuristic_result["nature"]
if not fm.get("domain") and heuristic_result.get("domain"):
result["domain"] = heuristic_result["domain"]
if not result["title"] and heuristic_result.get("title"):
result["title"] = heuristic_result["title"]
if not result["summary"] and heuristic_result.get("summary"):
result["summary"] = heuristic_result["summary"]
# 来源信息
if fm.get("source"):
result["source"] = fm["source"]
if fm.get("source_type"):
result["source_type"] = fm["source_type"]
# 防止空列表
if not result["domain"]:
result["domain"] = ["other"]
if not result["tags"]:
result["tags"] = ["待补充"]
return result
# ─── 测试函数 ──────────────────────────────────────────────────
if __name__ == "__main__":
# 测试代码
test_text = """Python 中的装饰器是一种强大的工具,它允许在不修改原函数代码的情况下,为函数添加额外的功能。
装饰器的基本语法:
@decorator
def my_function():
pass
常见的装饰器应用场景包括:日志记录、性能测试、权限验证等。"""
print("测试 LLM 分类:")
result = classify_content_enhanced(test_text, use_llm=True)
print(json.dumps(result, ensure_ascii=False, indent=2))
print("\n测试启发式分类(关闭 LLM):")
result2 = classify_content_enhanced(test_text, use_llm=False)
print(json.dumps(result2, ensure_ascii=False, indent=2))
@@ -0,0 +1,394 @@
#!/usr/bin/env python3
"""
LLM 分类器模块 - 使用 oMLX 本地大模型进行知识分类
功能:
1. 根据内容判断 nature(fact/method/concept/opinion/reference 等)
2. 根据内容判断 domain(技术/金融/产品/生活 等)
3. 提取关键信息(标题、摘要、标签)
配置:
- oMLX API: http://localhost:8000/v1/chat/completions
- 模型: Qwen3-8B-MLX-4bit
- API Key: omlx
"""
import json
import re
import time
from typing import Dict, List, Optional, Tuple
from pathlib import Path
# ─── oMLX 配置 ──────────────────────────────────────────────────
OLLAMA_API_URL = "http://localhost:8000/v1/chat/completions"
OLLAMA_MODEL = "Qwen2.5-VL-3B-Instruct-8bit" # 使用 VLM 模型,因为 Qwen3-8B-MLX-4bit 有 500 错误
OLLAMA_API_KEY = "omlx"
# 备份模型选项(如果 Qwen2.5 不可用)
FALLBACK_MODEL = "Qwen3-8B-MLX-4bit"
# ─── 分类体系 ──────────────────────────────────────────────────
NATURE_OPTIONS = [
"fact", # 客观事实、数据、统计结果
"method", # 方法、步骤、教程、最佳实践
"concept", # 概念、定义、理论、原理
"opinion", # 观点、看法、评价、推测
"reference", # 参考文献、论文、书籍、资料
"quote", # 引用、名言、摘录
"axiom", # 公理、定律、定理
"regulation", # 法规、政策、标准
"data", # 原始数据、数据集、代码片段
]
DOMAIN_OPTIONS = [
"technology", # 技术、编程、AI、开发
"finance", # 金融、投资、股票、经济
"product", # 产品、设计、UX、PM
"life", # 生活、健康、个人管理
"academic", # 学术、研究、教育
"business", # 商业、管理、创业
"culture", # 文化、艺术、娱乐
"other", # 其他
]
# ─── LLM 调用函数 ──────────────────────────────────────────────
def call_llm(prompt: str, system_prompt: str = None, temperature: float = 0.3) -> str:
"""
调用 oMLX 本地大模型 API
Args:
prompt: 用户提示
system_prompt: 系统提示(可选)
temperature: 温度参数,越低越确定
Returns:
LLM 回复文本
"""
import requests
messages = []
if system_prompt:
messages.append({"role": "system", "content": system_prompt})
messages.append({"role": "user", "content": prompt})
try:
# 首先尝试 VLM 模型
resp = requests.post(
OLLAMA_API_URL,
json={
"model": OLLAMA_MODEL,
"messages": messages,
"temperature": temperature,
"stream": False
},
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {OLLAMA_API_KEY}"
},
timeout=30
)
# 如果 VLM 模型失败(500 错误),尝试备选模型
if resp.status_code == 500 and OLLAMA_MODEL == "Qwen2.5-VL-3B-Instruct-8bit":
print(f" ⚠️ VLM 模型失败,尝试备选模型 {FALLBACK_MODEL}")
resp = requests.post(
OLLAMA_API_URL,
json={
"model": FALLBACK_MODEL,
"messages": messages,
"temperature": temperature,
"stream": False
},
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {OLLAMA_API_KEY}"
},
timeout=30
)
resp.raise_for_status()
data = resp.json()
return data["choices"][0]["message"]["content"]
except Exception as e:
print(f"[WARN] LLM 调用失败: {e}")
return ""
# ─── 分类函数 ──────────────────────────────────────────────────
def classify_with_llm(text: str) -> Dict:
"""
使用 LLM 对文本内容进行分类
Args:
text: 要分类的文本内容(Markdown 格式)
Returns:
{
"nature": "fact/method/concept/...",
"domain": "technology/finance/...",
"confidence": 0.0-1.0,
"title": "建议的标题",
"summary": "简短摘要",
"tags": ["标签1", "标签2", ...]
}
"""
# 限制文本长度(防止 token 过长)
preview = text[:2000] if len(text) > 2000 else text
system_prompt = """你是一个知识分类专家。请根据提供的文本内容进行以下分析:
1. 判断知识类型(nature):从以下选项中选择最合适的:fact(事实数据)、method(方法步骤)、concept(概念理论)、opinion(观点看法)、reference(参考文献)、quote(引用摘录)、axiom(公理定律)、regulation(法规政策)、data(原始数据)
2. 判断所属领域(domain):从以下选项中选择1-2个:technology(技术)、finance(金融)、product(产品)、life(生活)、academic(学术)、business(商业)、culture(文化)、other(其他)
3. 提取关键信息:建议一个简洁的标题、生成50字内的摘要、提取2-5个关键词标签
请以 JSON 格式返回结果,严格遵循以下结构:
{
"nature": "选择的类型",
"domain": ["领域1", "领域2"],
"confidence": 0.0-1.0之间的数字(表示分类信心度),
"title": "建议标题",
"summary": "简短摘要",
"tags": ["标签1", "标签2", "标签3"]
}"""
user_prompt = f"""请对以下文本内容进行分类分析:
{preview}
注意:文本可能是截断的,请基于现有内容进行判断。"""
response = call_llm(user_prompt, system_prompt, temperature=0.1)
# 解析 JSON 响应
result = {
"nature": "concept",
"domain": ["other"],
"confidence": 0.5,
"title": "",
"summary": "",
"tags": []
}
if not response:
return result
try:
# 查找 JSON 部分
json_match = re.search(r'\{.*\}', response, re.DOTALL)
if json_match:
parsed = json.loads(json_match.group())
if isinstance(parsed, dict):
if "nature" in parsed and parsed["nature"] in NATURE_OPTIONS:
result["nature"] = parsed["nature"]
if "domain" in parsed:
if isinstance(parsed["domain"], str):
result["domain"] = [parsed["domain"]]
elif isinstance(parsed["domain"], list):
result["domain"] = [d for d in parsed["domain"] if d in DOMAIN_OPTIONS]
if "confidence" in parsed:
try:
conf = float(parsed["confidence"])
if 0 <= conf <= 1:
result["confidence"] = conf
except:
pass
if "title" in parsed:
result["title"] = str(parsed["title"])
if "summary" in parsed:
result["summary"] = str(parsed["summary"])
if "tags" in parsed:
if isinstance(parsed["tags"], list):
result["tags"] = [str(tag) for tag in parsed["tags"]]
except Exception as e:
print(f"[WARN] LLM 响应解析失败: {e}")
return result
# ─── 启发式分类备用方案 ────────────────────────────────────────
def classify_heuristic(text: str, fm: dict) -> Dict:
"""
启发式分类(当 LLM 不可用时使用)
Args:
text: 文本内容
fm: 已有的 frontmatter
Returns:
与 classify_with_llm 相同结构的字典
"""
result = {
"nature": "concept",
"domain": ["other"],
"confidence": 0.4,
"title": "",
"summary": "",
"tags": []
}
# 优先使用已有 frontmatter
if fm.get("nature"):
result["nature"] = fm["nature"]
if fm.get("domain"):
if isinstance(fm["domain"], str):
result["domain"] = [fm["domain"]]
elif isinstance(fm["domain"], list):
result["domain"] = fm["domain"]
# 启发式规则
lower = text.lower()
# nature 判断
if not fm.get("nature"):
method_kw = ["步骤", "教程", "如何", "how to", "方法", "指南", "最佳实践"]
fact_kw = ["数据", "统计", "报告", "结果", "事实", "date", "report"]
opinion_kw = ["认为", "觉得", "观点", "推测", "假设", "opinion", "think"]
reference_kw = ["论文", "文献", "引用", "参考", "paper", "reference"]
if any(kw in lower for kw in method_kw):
result["nature"] = "method"
elif any(kw in lower for kw in fact_kw):
result["nature"] = "fact"
elif any(kw in lower for kw in opinion_kw):
result["nature"] = "opinion"
elif any(kw in lower for kw in reference_kw):
result["nature"] = "reference"
# domain 判断
if not fm.get("domain"):
tech_kw = ["编程", "代码", "开发", "python", "javascript", "ai", "llm", "机器学习", "技术"]
finance_kw = ["投资", "股票", "基金", "finance", "交易", "财经", "金融"]
product_kw = ["产品", "设计", "ux", "product", "用户体验"]
if any(kw in lower for kw in tech_kw):
result["domain"] = ["technology"]
elif any(kw in lower for kw in finance_kw):
result["domain"] = ["finance"]
elif any(kw in lower for kw in product_kw):
result["domain"] = ["product"]
# 生成简单标题(使用文件名或第一行)
if fm.get("title"):
result["title"] = fm["title"]
else:
first_line = text.split('\n')[0].strip('# ')
if len(first_line) < 50:
result["title"] = first_line
# 生成简单摘要(前 100 字符)
clean_text = re.sub(r'[#*\-`]', '', text)
result["summary"] = clean_text[:100] + ("..." if len(clean_text) > 100 else "")
return result
# ─── 主分类函数 ──────────────────────────────────────────────────
def classify_content_enhanced(text: str, fm: dict = None, use_llm: bool = True) -> Dict:
"""
增强版分类函数:优先使用 LLM,失败时降级到启发式
Args:
text: 文本内容
fm: 已有的 frontmatter(可选)
use_llm: 是否尝试使用 LLM(默认 True)
Returns:
分类结果字典
"""
if fm is None:
fm = {}
result = {
"nature": "concept",
"domain": ["other"],
"confidence": 0.5,
"title": "",
"summary": "",
"tags": [],
"source_type": "unknown",
"source": ""
}
# 优先使用已有 frontmatter
if fm.get("nature"):
result["nature"] = fm["nature"]
result["confidence"] = max(result["confidence"], 0.7) # 已有信息提升信心
if fm.get("domain"):
if isinstance(fm["domain"], str):
result["domain"] = [fm["domain"]]
elif isinstance(fm["domain"], list):
result["domain"] = fm["domain"]
# 尝试使用 LLM 分类
llm_result = None
if use_llm and text.strip():
try:
print(" 🤖 使用 LLM 进行智能分类...")
llm_result = classify_with_llm(text)
# 合并 LLM 结果(降低阈值要求,因为 VLM 模型可能返回较低的 confidence)
confidence_threshold = 0.4 # 降低阈值,因为 VLM 模型可能返回较低的 confidence
if llm_result.get("confidence", 0) > confidence_threshold:
if not fm.get("nature") and llm_result.get("nature"):
result["nature"] = llm_result["nature"]
if not fm.get("domain") and llm_result.get("domain"):
result["domain"] = llm_result["domain"]
if llm_result.get("title"):
result["title"] = llm_result["title"]
if llm_result.get("summary"):
result["summary"] = llm_result["summary"]
if llm_result.get("tags"):
result["tags"] = llm_result["tags"]
result["confidence"] = llm_result.get("confidence", 0.6)
print(f" ✅ LLM 分类成功 (confidence: {llm_result.get('confidence', 0.6):.2f})")
else:
print(f" ⚠️ LLM 分类信心度低 ({llm_result.get('confidence', 0):.2f} < {confidence_threshold}),使用启发式规则")
llm_result = None
except Exception as e:
print(f" ⚠️ LLM 分类失败: {e}")
llm_result = None
# LLM 失败或信心不足,使用启发式规则
if llm_result is None:
heuristic_result = classify_heuristic(text, fm)
# 合并启发式结果
if not fm.get("nature") and heuristic_result.get("nature"):
result["nature"] = heuristic_result["nature"]
if not fm.get("domain") and heuristic_result.get("domain"):
result["domain"] = heuristic_result["domain"]
if not result["title"] and heuristic_result.get("title"):
result["title"] = heuristic_result["title"]
if not result["summary"] and heuristic_result.get("summary"):
result["summary"] = heuristic_result["summary"]
# 来源信息
if fm.get("source"):
result["source"] = fm["source"]
if fm.get("source_type"):
result["source_type"] = fm["source_type"]
# 防止空列表
if not result["domain"]:
result["domain"] = ["other"]
if not result["tags"]:
result["tags"] = ["待补充"]
return result
# ─── 测试函数 ──────────────────────────────────────────────────
if __name__ == "__main__":
# 测试代码
test_text = """Python 中的装饰器是一种强大的工具,它允许在不修改原函数代码的情况下,为函数添加额外的功能。
装饰器的基本语法:
@decorator
def my_function():
pass
常见的装饰器应用场景包括:日志记录、性能测试、权限验证等。"""
print("测试 LLM 分类:")
result = classify_content_enhanced(test_text, use_llm=True)
print(json.dumps(result, ensure_ascii=False, indent=2))
print("\n测试启发式分类(关闭 LLM):")
result2 = classify_content_enhanced(test_text, use_llm=False)
print(json.dumps(result2, ensure_ascii=False, indent=2))
@@ -0,0 +1,17 @@
{
"name": "inbox_refine",
"version": "1.1.0",
"description": "知识库收件箱提纯——每日将 00_inbox/ 内容分类归档到知识库,更新首页统计和变更日志",
"created": "2026-04-25",
"last_modified": "2026-04-25",
"dependencies": ["file_read", "file_write", "python3"],
"schedule": "daily 03:00",
"paths": {
"inbox": "~/workbuddy-agent-os/agent-sync/03_knowledge/00_inbox/",
"knowledge_root": "~/workbuddy-agent-os/agent-sync/03_knowledge/",
"homepage": "~/workbuddy-agent-os/agent-sync/03_knowledge/README.md",
"changelog": "~/workbuddy-agent-os/agent-sync/03_knowledge/CHANGELOG.md",
"templates": "~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/templates/",
"taxonomies": "~/workbuddy-agent-os/agent-sync/03_knowledge/99_system/taxonomies/"
}
}