feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程，直接发 HTTP
移植自 mac-agent-os 的 mediacrawler_adapter：不起子进程、不开页面，用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin，所以 ingest 那条链路一个字不用改。
**目前能用的（真环境实测，非推断）**：
profile/other : 200, 7075 字节 —— 博主主页指标（粉丝/获赞/作品数/昵称）
aweme/detail : 200, 45425 字节 —— 单条作品详情（含点赞/评论/收藏/分享）
**目前不能用的：作品列表 `aweme/post`。** 两个互相独立的原因：
1. 这个接口被抖音单独升级成了真校验：不带 x-tt-argus 回 403「Uifid Not Found」，
带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过，「真校验」过不了。
同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的，
挑中的恰好是「批量拉作品列表」这个最敏感的动作。
2. 改走页面截获也不行：CDP 浏览器打开博主主页会落到「验证码中间页」（当天大量探测的
代价，过几小时要重测）。
所以现在的边界是：**已知作品的指标刷新能做，自动发现新作品做不了**。
**排查中控住变量后得到的两条事实**（都写进注释了）：
· `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
UA+client hints+Cookie 时是 200 但仅 121 字节的空壳，补上这三个头变 7074 字节。
（我先前猜的 sec-ch-ua 不是关键。）
· 因此 UA 与 client hints 必须**成套地取自同一个浏览器**，所以 BrowserIdentity 一次
从 CDP 取齐 cookie + UA + hints，而不是各自写死。
「200 + 空 body 必须当场报错」也是刻意写死的：放过去它会在下游变成「这个博主没作品」，
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的，还被翻译成「账号被封」。
测试 +11：cookie 解析、请求头成套性（含 uifid 缺失/回退）、产物键名与 store 对齐、
以及 _get 的三条失败路径（空 body / 403 带网关原话 / 正常返回）。
This commit is contained in:
@@ -0,0 +1,451 @@
|
||||
# 🎬 口播素材智能采集与分析系统
|
||||
|
||||
> 完全本地运行的口播灵感工坊——采集、AI 分析、智能下载、统一检索，隐私安全零成本。
|
||||
|
||||
**版本**：v1.0.0
|
||||
**适用环境**：Mac M1 / 16GB RAM / macOS
|
||||
**作者**：ghai
|
||||
**日期**：2026-04-28
|
||||
|
||||
---
|
||||
|
||||
## 一、这是什么？
|
||||
|
||||
一个个人口播创作者的**素材灵感管理系统**。解决的核心问题：
|
||||
|
||||
| 痛点 | 解决方案 |
|
||||
|------|----------|
|
||||
| 素材散布在抖音/小红书/B站 | 一键采集元数据到本地 SQLite |
|
||||
| 收藏夹大量 404 | AI 判断价值 → 自动下载到本地 |
|
||||
| 看标题判断不了口播价值 | 本地 LLM 自动提取金句/情绪/结构 |
|
||||
| 不想上云、不想付费 | 全部跑在本机，零网络依赖（采集除外） |
|
||||
|
||||
**一句话**：搜得到、看得懂、存得住、找得快。
|
||||
|
||||
---
|
||||
|
||||
## 二、系统架构
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
|
||||
│ collect.py │────→│ analyze.py │────→│ app.py │
|
||||
│ 多平台采集 │ │ AI 分析 │ │ Web 界面 │
|
||||
└──────┬──────┘ └──────┬───────┘ └──────┬──────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ JSONL 存档 │ │ oMLX 本地LLM │ │ SQLite 索引 │
|
||||
│ data/raw/ │ │ Qwen2.5-VL │ │ data/db │
|
||||
└──────────────┘ └──────────────┘ └──────┬───────┘
|
||||
│
|
||||
┌──────▼───────┐
|
||||
│downloader.py │
|
||||
│ yt-dlp 下载 │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
~/workbuddy-agent-os/agent-local/materials/
|
||||
```
|
||||
|
||||
### 技术选型
|
||||
|
||||
| 组件 | 选型 | 为什么 |
|
||||
|------|------|--------|
|
||||
| 采集引擎 | MediaCrawler (Playwright) | 主流平台支持最全 |
|
||||
| AI 大脑 | oMLX + Qwen2.5-VL-3B-8bit | 本地免费，3B 模型足够 JSON 输出 |
|
||||
| 结构化存储 | SQLite (WAL 模式) | 零配置，16GB 无压力 |
|
||||
| 媒体下载 | yt-dlp | 支持平台最多 |
|
||||
| Web 界面 | Gradio | Python 原生，开发最快 |
|
||||
| 配置管理 | YAML | 人类可读，注释友好 |
|
||||
|
||||
---
|
||||
|
||||
## 三、快速开始
|
||||
|
||||
### 3.1 环境准备
|
||||
|
||||
```bash
|
||||
# 确认 Python 环境
|
||||
~/.workbuddy/binaries/python/envs/agent-os/bin/python3 --version
|
||||
# 应输出: Python 3.13.12
|
||||
|
||||
# 确认 oMLX 运行中
|
||||
curl -s http://localhost:8000/v1/models -H "Authorization: Bearer omlx" | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data']]"
|
||||
# 应输出: Qwen2.5-VL-3B-Instruct-8bit
|
||||
```
|
||||
|
||||
### 3.2 安装依赖
|
||||
|
||||
```bash
|
||||
~/.workbuddy/binaries/python/envs/agent-os/bin/pip install gradio yt-dlp
|
||||
```
|
||||
|
||||
### 3.3 初始化数据库
|
||||
|
||||
```bash
|
||||
cd ~/workbuddy-agent-os/agent-sync/05_tools/05_crawl/content-inspiration
|
||||
~/.workbuddy/binaries/python/envs/agent-os/bin/python3 collect.py --init-db
|
||||
# 输出: [OK] 数据库初始化完成
|
||||
```
|
||||
|
||||
### 3.4 开始使用
|
||||
|
||||
```bash
|
||||
# 1. 采集素材（需要先配置 MediaCrawler Cookie）
|
||||
python collect.py -p xiaohongshu -k "普通人逆袭"
|
||||
|
||||
# 2. AI 分析（需要 oMLX 运行中）
|
||||
python analyze.py
|
||||
|
||||
# 3. 启动 Web 界面
|
||||
python app.py
|
||||
# 打开 http://127.0.0.1:7860
|
||||
|
||||
# 4. 下载视频（在 Web 界面标记后）
|
||||
python downloader.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 四、项目结构
|
||||
|
||||
```
|
||||
content-inspiration/
|
||||
├── SKILL.md # 技能身份证（system/user 能力说明）
|
||||
├── README.md # 本文件（完整项目说明）
|
||||
├── config.yaml # 所有配置（平台/模型/存储/下载/Web）
|
||||
├── schema.sql # 数据库表结构（3 张表）
|
||||
├── requirements.txt # Python 依赖清单
|
||||
│
|
||||
├── collect.py # 采集脚本（MediaCrawler 封装）
|
||||
├── analyze.py # AI 分析脚本（oMLX 调用 + JSON 解析）
|
||||
├── downloader.py # 下载脚本（yt-dlp 封装）
|
||||
├── app.py # Web 界面（Gradio）
|
||||
├── utils.py # 通用工具（配置/数据库/日志/JSONL）
|
||||
│
|
||||
├── data/
|
||||
│ ├── raw/ # JSONL 原始采集存档
|
||||
│ │ └── 2026-04-28_xiaohongshu_普通人逆袭.jsonl
|
||||
│ └── database.db # SQLite 素材库
|
||||
│
|
||||
└── logs/ # 运行日志（按日期分割）
|
||||
├── collect_20260428.log
|
||||
└── analyze_20260428.log
|
||||
```
|
||||
|
||||
### 与 AgentOS 的关系
|
||||
|
||||
```
|
||||
~/workbuddy-agent-os/agent-sync/ ← 坚果云同步 + Git
|
||||
└── 05_tools/05_crawl/
|
||||
└── content-inspiration/ ← 本项目
|
||||
├── 代码 + 配置 + 数据库 （同步）
|
||||
└── data/raw/*.jsonl （同步）
|
||||
|
||||
~/workbuddy-agent-os/agent-local/ ← 本机专属，不同步
|
||||
└── materials/
|
||||
├── video/ ← 下载的视频
|
||||
└── audio/ ← 下载的音频
|
||||
```
|
||||
|
||||
- 项目代码和数据库跟随 agent-os 同步
|
||||
- 下载的视频/音频在本机专属目录，不占用同步空间
|
||||
- AI 分析结果可通过 AgentOS 的 inbox_refine 归档到知识库
|
||||
|
||||
---
|
||||
|
||||
## 五、配置说明（config.yaml）
|
||||
|
||||
```yaml
|
||||
# 平台和关键词
|
||||
platforms:
|
||||
- xiaohongshu # 目前最成熟
|
||||
- douyin # 需要 Cookie
|
||||
- bilibili # 支持
|
||||
|
||||
keywords:
|
||||
- "普通人逆袭"
|
||||
- "情感故事"
|
||||
- "科普冷知识"
|
||||
|
||||
max_count: 20 # 每关键词每次采集上限
|
||||
|
||||
# 本地 LLM（oMLX，非 Ollama）
|
||||
llm:
|
||||
model: "Qwen2.5-VL-3B-Instruct-8bit"
|
||||
base_url: "http://localhost:8000/v1" # oMLX 端口
|
||||
api_key: "omlx" # oMLX 固定 key
|
||||
timeout: 120 # 每条分析超时
|
||||
temperature: 0.3 # 低温度保证稳定
|
||||
|
||||
# 存储路径
|
||||
storage:
|
||||
raw_dir: "data/raw"
|
||||
db_path: "data/database.db"
|
||||
media_dir: "~/workbuddy-agent-os/agent-local/materials/video"
|
||||
audio_dir: "~/workbuddy-agent-os/agent-local/materials/audio"
|
||||
|
||||
# Web 界面
|
||||
web:
|
||||
host: "127.0.0.1"
|
||||
port: 7860
|
||||
title: "口播灵感工坊"
|
||||
|
||||
# 下载配置
|
||||
download:
|
||||
rate_limit: "1M" # 速率限制，防 IP 封禁
|
||||
retries: 3 # 重试次数
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 六、脚本详细说明
|
||||
|
||||
### collect.py — 采集脚本
|
||||
|
||||
**功能**：从主流平台采集内容元数据（标题、描述、封面、互动数据等），不下载视频本体。
|
||||
|
||||
```bash
|
||||
# 初始化数据库
|
||||
python collect.py --init-db
|
||||
|
||||
# 指定平台和关键词采集
|
||||
python collect.py -p xiaohongshu -k "科普冷知识"
|
||||
|
||||
# 指定数量
|
||||
python collect.py -p xiaohongshu -k "情感故事" -n 10
|
||||
|
||||
# 按 config.yaml 批量采集（遍历所有平台×关键词）
|
||||
python collect.py
|
||||
```
|
||||
|
||||
**输出**：
|
||||
- `data/raw/YYYY-MM-DD_平台_关键词.jsonl` — 原始数据存档
|
||||
- `data/database.db` — 写入 materials 表（自动去重）
|
||||
|
||||
**前置条件**：MediaCrawler 已安装并配置好 Cookie。
|
||||
|
||||
### analyze.py — AI 分析脚本
|
||||
|
||||
**功能**：对未分析的素材调用本地 oMLX（Qwen2.5-VL-3B），提取口播可用信息。
|
||||
|
||||
```bash
|
||||
# 分析所有未分析的素材
|
||||
python analyze.py
|
||||
|
||||
# 只分析前 10 条（测试用）
|
||||
python analyze.py -n 10
|
||||
|
||||
# 只看有多少待分析（不执行）
|
||||
python analyze.py --dry-run
|
||||
|
||||
# 重试之前失败的记录
|
||||
python analyze.py --retry-failed
|
||||
```
|
||||
|
||||
**输出**：写入 analysis 表，字段包括：
|
||||
- `tags` — AI 标签（科普、情感、冷知识…）
|
||||
- `golden_quote` — 可直接引用的金句
|
||||
- `core_idea` — 核心立意
|
||||
- `structure` — 口播脚本结构
|
||||
- `emotion` — 整体情绪
|
||||
- `worth_downloading` — 是否推荐下载
|
||||
|
||||
**注意**：串行执行（每次只处理一条），16GB Mac 不会 OOM。每条间隔 1 秒。
|
||||
|
||||
### downloader.py — 下载脚本
|
||||
|
||||
**功能**：下载标记为 pending 的素材视频到本地。
|
||||
|
||||
```bash
|
||||
# 下载所有标记为待下载的
|
||||
python downloader.py
|
||||
|
||||
# 下载所有 AI 推荐的
|
||||
python downloader.py --all
|
||||
|
||||
# 限制数量
|
||||
python downloader.py -n 5
|
||||
|
||||
# 手动指定 URL
|
||||
python downloader.py -u "https://v.douyin.com/xxx"
|
||||
```
|
||||
|
||||
**输出**：文件保存到 `~/workbuddy-agent-os/agent-local/materials/video/`，文件名格式：
|
||||
```
|
||||
xiaohongshu_作者名_标题前20字_日期.mp4
|
||||
```
|
||||
|
||||
### app.py — Web 界面
|
||||
|
||||
**功能**：本地浏览器访问的素材管理界面。
|
||||
|
||||
```bash
|
||||
python app.py
|
||||
# 打开 http://127.0.0.1:7860
|
||||
```
|
||||
|
||||
**功能清单**：
|
||||
- 🔍 关键词搜索（标题/描述/标签/金句）
|
||||
- 📋 平台/情绪/下载状态筛选
|
||||
- 📊 实时统计（总数/已分析/已下载/AI推荐）
|
||||
- 📖 素材详情（AI 分析完整结果）
|
||||
- 📥 一键标记下载
|
||||
- 📝 金句可直接复制
|
||||
|
||||
### utils.py — 工具函数
|
||||
|
||||
| 函数 | 用途 |
|
||||
|------|------|
|
||||
| `load_config()` | 加载 config.yaml |
|
||||
| `get_db()` / `init_db()` | 数据库连接/初始化 |
|
||||
| `setup_logger()` | 日志配置 |
|
||||
| `parse_jsonl()` / `save_jsonl()` | JSONL 读写 |
|
||||
| `check_omlx()` | 检测 oMLX 可用性 |
|
||||
| `truncate_text()` | 文件名安全截断 |
|
||||
|
||||
---
|
||||
|
||||
## 七、数据库设计
|
||||
|
||||
### materials 表（素材元数据）
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| id | INTEGER | 主键 |
|
||||
| platform | TEXT | 平台（xiaohongshu/douyin/bilibili） |
|
||||
| original_id | TEXT | 平台内唯一 ID |
|
||||
| url | TEXT | 原始链接 |
|
||||
| title | TEXT | 标题 |
|
||||
| description | TEXT | 描述/正文 |
|
||||
| author | TEXT | 作者 |
|
||||
| cover_url | TEXT | 封面图 |
|
||||
| music_name | TEXT | BGM |
|
||||
| like_count | INTEGER | 点赞数 |
|
||||
| analyzed | INTEGER | 0未分析/1已分析/2失败 |
|
||||
| download_status | TEXT | none/pending/downloading/done/failed |
|
||||
| local_files | TEXT | JSON（本地文件路径） |
|
||||
| raw_json | TEXT | 完整原始数据备份 |
|
||||
|
||||
### analysis 表（AI 分析结果）
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| material_id | INTEGER | 关联素材 |
|
||||
| tags | TEXT | AI 标签（逗号分隔） |
|
||||
| golden_quote | TEXT | 金句 |
|
||||
| core_idea | TEXT | 核心立意 |
|
||||
| structure | TEXT | 脚本结构 |
|
||||
| emotion | TEXT | 情绪 |
|
||||
| worth_downloading | TEXT | yes/no/maybe |
|
||||
| download_reason | TEXT | 下载理由 |
|
||||
| raw_model_output | TEXT | 模型原始输出（调试） |
|
||||
|
||||
### collect_batches 表（采集批次记录）
|
||||
|
||||
记录每次采集的平台、关键词、数量、状态，方便回溯。
|
||||
|
||||
---
|
||||
|
||||
## 八、知识库索引卡片
|
||||
|
||||
采集并分析后的素材，可以在知识库中生成索引卡片（存入 `03_knowledge/50_resources/`）：
|
||||
|
||||
**文件名格式**：`{hostname}_{YYYYMMDD}_{slug}.md`
|
||||
**示例**：`Redmi-12C_20260428_xiaohongshu-普通人逆袭.md`
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: 普通人逆袭合集
|
||||
source_url: https://www.xiaohongshu.com/explore/xxx
|
||||
local_path: ~/workbuddy-agent-os/agent-local/materials/video/xiaohongshu_作者_标题.mp4
|
||||
type: video
|
||||
platform: xiaohongshu
|
||||
collected_by: Redmi-12C
|
||||
collected_at: 2026-04-28
|
||||
tags: [口播素材, 励志, 逆袭]
|
||||
---
|
||||
一句话说明：适合逆袭主题口播开头，情绪张力强。
|
||||
```
|
||||
|
||||
`collected_by` 字段防止多机同步时文件名冲突。
|
||||
|
||||
---
|
||||
|
||||
## 九、MediaCrawler 配置（前置条件）
|
||||
|
||||
### 安装
|
||||
|
||||
```bash
|
||||
cd ~/workbuddy-agent-os/agent-sync/05_tools/05_crawl
|
||||
git clone https://github.com/NanmiCoder/MediaCrawler.git
|
||||
cd MediaCrawler
|
||||
~/.workbuddy/binaries/python/envs/agent-os/bin/pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 配置 Cookie
|
||||
|
||||
各平台需要在首次使用前扫码登录获取 Cookie：
|
||||
|
||||
1. `cd ~/workbuddy-agent-os/agent-sync/05_tools/05_crawl/MediaCrawler`
|
||||
2. `python main.py --platform xhs --lt qrcode`
|
||||
3. 扫描终端显示的二维码
|
||||
4. Cookie 自动保存，后续无需重复登录
|
||||
|
||||
### 配置本项目
|
||||
|
||||
安装完成后，更新 `config.yaml`：
|
||||
```yaml
|
||||
crawler:
|
||||
project_path: "~/workbuddy-agent-os/agent-sync/05_tools/05_crawl/MediaCrawler"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 十、性能与资源管理（16GB Mac）
|
||||
|
||||
| 场景 | 内存占用 | 建议 |
|
||||
|------|----------|------|
|
||||
| 采集（Playwright） | ~500MB | 关闭其他浏览器 |
|
||||
| AI 分析（Qwen2.5-VL-3B） | ~2GB | 关闭其他大型应用，串行执行 |
|
||||
| Web 界面（Gradio） | ~300MB | 轻量，无压力 |
|
||||
| 下载（yt-dlp） | ~100MB | 速率限制 1MB/s |
|
||||
|
||||
**关键原则**：分析期间不同时运行其他 AI 任务，每次只处理一条记录。
|
||||
|
||||
---
|
||||
|
||||
## 十一、常见问题
|
||||
|
||||
**Q: oMLX 返回 500 错误？**
|
||||
A: Qwen3-8B-MLX-4bit 有已知问题。本系统使用 Qwen2.5-VL-3B-Instruct-8bit，不受影响。
|
||||
|
||||
**Q: 采集没有数据？**
|
||||
A: 检查 MediaCrawler Cookie 是否过期，重新扫码登录。
|
||||
|
||||
**Q: AI 分析结果不准确？**
|
||||
A: 3B 模型有限，可通过调整 `temperature`（降低=更稳定）或更换更强模型改善。
|
||||
|
||||
**Q: 视频下载失败？**
|
||||
A: 抖音/小红书链接有时效性，采集后尽快下载。也可尝试 `--retry-failed`。
|
||||
|
||||
**Q: 如何接入 AgentOS 自动化？**
|
||||
A: 可在 WorkBuddy 中创建定时自动化，例如每天凌晨执行 `python analyze.py`。
|
||||
|
||||
---
|
||||
|
||||
## 十二、合规提醒
|
||||
|
||||
- ⚠️ 仅供**个人学习研究**，禁止大规模抓取或商业化
|
||||
- 下载的视频仅供**分析借鉴**，口播内容须二次原创
|
||||
- 所有数据**本地存储**，注意定期备份
|
||||
|
||||
---
|
||||
|
||||
## 十三、未来扩展
|
||||
|
||||
- [ ] 定时自动采集 + 热点推送
|
||||
- [ ] 向量语义搜索（接入 ChromaDB + Qwen3-Embedding）
|
||||
- [ ] 提示词模板可视化编辑
|
||||
- [ ] 素材标签归一化
|
||||
- [ ] 简易剪辑时间线
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
name: content_inspiration
|
||||
version: 1.0.0
|
||||
description: 口播素材智能采集与分析系统——多平台采集、AI提炼、智能下载、统一检索
|
||||
category: user
|
||||
triggers:
|
||||
- 口播素材
|
||||
- 素材采集
|
||||
- 灵感采集
|
||||
- 采集素材
|
||||
- 口播灵感
|
||||
- 素材工坊
|
||||
- content inspiration
|
||||
- 启动工坊
|
||||
- 分析素材
|
||||
- 下载素材
|
||||
---
|
||||
|
||||
# Content Inspiration —— 口播素材智能采集与分析系统
|
||||
|
||||
## 概述
|
||||
|
||||
完全运行在本机 Mac 的口播素材管理系统，实现"采集→AI分析→智能下载→检索查阅"的完整闭环。不依赖任何云端服务，隐私安全。
|
||||
|
||||
## 系统能力
|
||||
|
||||
### System 能力（后台自动运行，脚本式任务）
|
||||
|
||||
| 能力 | 脚本 | 说明 |
|
||||
|------|------|------|
|
||||
| 数据库管理 | `schema.sql` | SQLite WAL 模式，3 张表（materials/analysis/collect_batches） |
|
||||
| 配置驱动 | `config.yaml` | 所有参数集中管理，模型/平台/存储/下载均可配置 |
|
||||
| 数据库初始化 | `utils.py: init_db()` | 自动建表、开启 WAL、外键约束 |
|
||||
| JSONL 存档 | `utils.py: save_jsonl()` | 原始采集数据可追溯回溯 |
|
||||
| oMLX 健康检查 | `utils.py: check_omlx()` | 自动检测本地 LLM 可用性 |
|
||||
| 日志系统 | `utils.py: setup_logger()` | 按日期分割，同时输出文件和控制台 |
|
||||
| 去重机制 | `collect.py: save_to_db()` | 基于 platform+original_id 唯一索引 |
|
||||
|
||||
### User 能力（用户直接触发/操作）
|
||||
|
||||
| 能力 | 脚本 | 用法 | 说明 |
|
||||
|------|------|------|------|
|
||||
| 按需采集 | `collect.py` | `python collect.py -p xiaohongshu -k "科普冷知识"` | 从指定平台采集素材元数据 |
|
||||
| 全量采集 | `collect.py` | `python collect.py` | 按 config.yaml 配置批量采集 |
|
||||
| AI 分析 | `analyze.py` | `python analyze.py` | 对所有未分析素材调用本地 LLM |
|
||||
| 限量分析 | `analyze.py` | `python analyze.py -n 10` | 只分析前 10 条（测试用） |
|
||||
| 重试失败 | `analyze.py` | `python analyze.py --retry-failed` | 重试之前分析失败的记录 |
|
||||
| 下载视频 | `downloader.py` | `python downloader.py` | 下载所有标记为 pending 的素材 |
|
||||
| 下载推荐 | `downloader.py` | `python downloader.py --all` | 下载所有 AI 推荐的素材 |
|
||||
| 手动下载 | `downloader.py` | `python downloader.py -u "URL"` | 指定 URL 直接下载 |
|
||||
| Web 界面 | `app.py` | `python app.py` | 启动 Gradio 搜索/浏览界面 |
|
||||
| 初始化数据库 | `collect.py` | `python collect.py --init-db` | 仅创建数据库表结构 |
|
||||
|
||||
## 数据流程
|
||||
|
||||
```
|
||||
用户配置关键词
|
||||
↓
|
||||
collect.py → MediaCrawler → JSONL 存档 + SQLite 入库
|
||||
↓
|
||||
analyze.py → oMLX(Qwen2.5-VL-3B) → AI 标签/金句/情绪/推荐
|
||||
↓
|
||||
app.py (Web界面) → 浏览/搜索/标记下载
|
||||
↓
|
||||
downloader.py → yt-dlp → ~/workbuddy-agent-os/agent-local/materials/video/
|
||||
↓
|
||||
知识库索引卡片 → 03_knowledge/50_resources/
|
||||
```
|
||||
|
||||
## 依赖关系
|
||||
|
||||
### 必需
|
||||
|
||||
| 依赖 | 版本 | 用途 | 安装状态 |
|
||||
|------|------|------|----------|
|
||||
| Python | 3.10+ | 运行环境 | ✅ 3.13.12 |
|
||||
| oMLX | v0.3.6 | 本地 LLM 推理 | ✅ 运行中 |
|
||||
| Qwen2.5-VL-3B | 8bit | 多模态内容理解 | ✅ 已加载 |
|
||||
| httpx | ≥0.24 | HTTP 客户端（调 oMLX） | ✅ 已安装 |
|
||||
| PyYAML | ≥6.0 | 配置文件解析 | ✅ 已安装 |
|
||||
| Pillow | ≥9.0 | 图片处理 | ✅ 已安装 |
|
||||
|
||||
### 待安装
|
||||
|
||||
| 依赖 | 版本 | 用途 | 安装命令 |
|
||||
|------|------|------|----------|
|
||||
| gradio | ≥4.0 | Web 界面 | `pip install gradio` |
|
||||
| yt-dlp | ≥2024.0 | 视频下载 | `pip install yt-dlp` |
|
||||
|
||||
### 可选
|
||||
|
||||
| 依赖 | 用途 | 说明 |
|
||||
|------|------|------|
|
||||
| MediaCrawler | 多平台采集 | 需单独 clone 并配置 Cookie |
|
||||
| ffmpeg | 音频提取 | yt-dlp 的音频模式需要 |
|
||||
|
||||
## 存储路径
|
||||
|
||||
| 类型 | 路径 | 说明 |
|
||||
|------|------|------|
|
||||
| 项目代码 | `~/workbuddy-agent-os/agent-sync/05_tools/05_crawl/content-inspiration/` | 坚果云同步 + Git |
|
||||
| 原始 JSONL | `data/raw/` | 跟随项目目录 |
|
||||
| SQLite 数据库 | `data/database.db` | 跟随项目目录 |
|
||||
| 下载的视频 | `~/workbuddy-agent-os/agent-local/materials/video/` | 本机专属，不同步 |
|
||||
| 下载的音频 | `~/workbuddy-agent-os/agent-local/materials/audio/` | 本机专属，不同步 |
|
||||
| 运行日志 | `logs/` | 跟随项目目录 |
|
||||
|
||||
## 限制与注意事项
|
||||
|
||||
1. **MediaCrawler Cookie**：首次使用需在各平台手动扫码登录，获取 Cookie 后填入 MediaCrawler 配置
|
||||
2. **链接时效性**：抖音/小红书链接有时效性，采集后 24 小时内下载成功率最高
|
||||
3. **内存管理**：AI 分析串行执行（每次只处理一条），避免 16GB Mac OOM
|
||||
4. **模型精度**：3B 小模型可能有标签不准确的情况，建议人工校验
|
||||
5. **合规提醒**：仅供个人学习研究，禁止商业化或大规模抓取
|
||||
|
||||
## 版本历史
|
||||
|
||||
| 版本 | 日期 | 说明 |
|
||||
|------|------|------|
|
||||
| 1.0.0 | 2026-04-28 | 初始版本，核心闭环（采集→分析→下载→查阅） |
|
||||
@@ -0,0 +1,260 @@
|
||||
"""
|
||||
口播素材系统 - AI 分析脚本
|
||||
用法：python analyze.py [--limit N] [--retry-failed]
|
||||
|
||||
功能：
|
||||
1. 查询所有未分析的素材记录
|
||||
2. 调用本地 oMLX（Qwen2.5-VL-3B）进行口播价值分析
|
||||
3. 解析结构化 JSON 结果，写入 analysis 表
|
||||
|
||||
本机适配：
|
||||
- 使用 oMLX HTTP API（localhost:8000），非 Ollama
|
||||
- 模型：Qwen2.5-VL-3B-Instruct-8bit
|
||||
- 串行执行，防止 16GB 内存 OOM
|
||||
- 异常自动跳过，不中断整体流程
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import re
|
||||
import argparse
|
||||
import time
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
import httpx
|
||||
from utils import load_config, get_db, init_db, setup_logger
|
||||
|
||||
# 分析提示词模板
|
||||
ANALYSIS_PROMPT = """你是一个口播素材分析助手。请分析以下内容，提取口播创作可用的信息。仅返回合法JSON，不要有任何额外文字。
|
||||
|
||||
内容：
|
||||
标题：{title}
|
||||
描述：{description}
|
||||
作者：{author}
|
||||
平台：{platform}
|
||||
|
||||
要求返回JSON：
|
||||
{{
|
||||
"tags": ["标签1","标签2","标签3"],
|
||||
"golden_quote": "可以直接引用的金句或观点，不超过30字",
|
||||
"core_idea": "一句话核心立意",
|
||||
"structure": "口播脚本结构：开头-中间-结尾的梗概",
|
||||
"emotion": "整体情绪，如 治愈/励志/震惊/沉思/有趣",
|
||||
"worth_downloading": "yes/no/maybe",
|
||||
"download_reason": "建议下载的简短理由，若no则留空"
|
||||
}}"""
|
||||
|
||||
|
||||
def call_omlx(prompt: str, config: dict, logger) -> str | None:
|
||||
"""
|
||||
调用 oMLX API 进行推理
|
||||
|
||||
返回：模型输出的文本，失败返回 None
|
||||
"""
|
||||
import httpx
|
||||
try:
|
||||
resp = httpx.post(
|
||||
f"{config['llm']['base_url']}/chat/completions",
|
||||
headers={
|
||||
"Authorization": f"Bearer {config['llm']['api_key']}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
json={
|
||||
"model": config["llm"]["model"],
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"response_format": {"type": "json_object"},
|
||||
"temperature": config["llm"].get("temperature", 0.3),
|
||||
"max_tokens": 500,
|
||||
},
|
||||
timeout=config["llm"].get("timeout", 120),
|
||||
)
|
||||
|
||||
if resp.status_code != 200:
|
||||
logger.error(f"oMLX 返回 {resp.status_code}: {resp.text[:200]}")
|
||||
return None
|
||||
|
||||
return resp.json()["choices"][0]["message"]["content"]
|
||||
|
||||
except httpx.TimeoutException:
|
||||
logger.error("oMLX 请求超时")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"oMLX 调用异常: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def parse_analysis_json(raw_text: str, logger) -> dict | None:
|
||||
"""
|
||||
解析模型输出的 JSON，带容错处理
|
||||
"""
|
||||
if not raw_text:
|
||||
return None
|
||||
|
||||
# 尝试直接解析
|
||||
try:
|
||||
result = json.loads(raw_text)
|
||||
if isinstance(result, dict):
|
||||
return result
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# 尝试提取 JSON 块（模型可能输出 ```json ... ```）
|
||||
match = re.search(r'```(?:json)?\s*(\{.*?\})\s*```', raw_text, re.DOTALL)
|
||||
if match:
|
||||
try:
|
||||
result = json.loads(match.group(1))
|
||||
if isinstance(result, dict):
|
||||
return result
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# 尝试提取第一个 { 到最后一个 }
|
||||
first = raw_text.find("{")
|
||||
last = raw_text.rfind("}")
|
||||
if first != -1 and last != -1 and last > first:
|
||||
try:
|
||||
result = json.loads(raw_text[first:last+1])
|
||||
if isinstance(result, dict):
|
||||
return result
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
logger.warning(f"JSON 解析失败，原始输出: {raw_text[:200]}")
|
||||
return None
|
||||
|
||||
|
||||
def analyze_one(material: dict, config: dict, logger) -> dict | None:
|
||||
"""
|
||||
分析单条素材
|
||||
"""
|
||||
title = material["title"] or ""
|
||||
description = (material.get("description") or "")[:500] # 截断避免过长
|
||||
|
||||
if not title and not description:
|
||||
logger.warning(f"素材 {material['id']} 无标题和描述，跳过")
|
||||
return None
|
||||
|
||||
prompt = ANALYSIS_PROMPT.format(
|
||||
title=title,
|
||||
description=description,
|
||||
author=material.get("author", "未知"),
|
||||
platform=material.get("platform", "未知"),
|
||||
)
|
||||
|
||||
raw_output = call_omlx(prompt, config, logger)
|
||||
if not raw_output:
|
||||
return None
|
||||
|
||||
result = parse_analysis_json(raw_output, logger)
|
||||
if not result:
|
||||
return None
|
||||
|
||||
# 规范化字段
|
||||
return {
|
||||
"material_id": material["id"],
|
||||
"tags": ",".join(result.get("tags", [])),
|
||||
"golden_quote": str(result.get("golden_quote", ""))[:200],
|
||||
"core_idea": str(result.get("core_idea", ""))[:300],
|
||||
"structure": str(result.get("structure", ""))[:500],
|
||||
"emotion": str(result.get("emotion", ""))[:20],
|
||||
"worth_downloading": str(result.get("worth_downloading", "no"))[:10],
|
||||
"download_reason": str(result.get("download_reason", ""))[:200],
|
||||
"raw_model_output": raw_output,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="口播素材 AI 分析")
|
||||
parser.add_argument("--limit", "-n", type=int, default=0, help="分析数量上限，0=全部")
|
||||
parser.add_argument("--retry-failed", action="store_true", help="重试分析失败的记录")
|
||||
parser.add_argument("--dry-run", action="store_true", help="仅显示待分析数量，不执行")
|
||||
args = parser.parse_args()
|
||||
|
||||
config = load_config()
|
||||
logger = setup_logger("analyze")
|
||||
|
||||
# 检查 oMLX
|
||||
from utils import check_omlx
|
||||
if not check_omlx(config):
|
||||
logger.error("oMLX 不可用，请先启动 oMLX 服务")
|
||||
return
|
||||
|
||||
db_path = init_db()
|
||||
db = get_db(db_path)
|
||||
|
||||
# 查询待分析记录
|
||||
if args.retry_failed:
|
||||
query = "SELECT * FROM materials WHERE analyzed = 2"
|
||||
else:
|
||||
query = "SELECT * FROM materials WHERE analyzed = 0"
|
||||
|
||||
if args.limit > 0:
|
||||
query += f" LIMIT {args.limit}"
|
||||
|
||||
materials = db.execute(query).fetchall()
|
||||
materials = [dict(m) for m in materials]
|
||||
|
||||
if not materials:
|
||||
print("[OK] 没有待分析的素材")
|
||||
db.close()
|
||||
return
|
||||
|
||||
print(f"[INFO] 待分析: {len(materials)} 条")
|
||||
|
||||
if args.dry_run:
|
||||
for m in materials[:5]:
|
||||
print(f" - [{m['platform']}] {m['title'][:40]}")
|
||||
if len(materials) > 5:
|
||||
print(f" ... 还有 {len(materials) - 5} 条")
|
||||
db.close()
|
||||
return
|
||||
|
||||
success = 0
|
||||
failed = 0
|
||||
|
||||
for i, material in enumerate(materials, 1):
|
||||
print(f"\r[进度] {i}/{len(materials)} 分析中...", end="", flush=True)
|
||||
|
||||
try:
|
||||
result = analyze_one(material, config, logger)
|
||||
|
||||
if result:
|
||||
# 写入 analysis 表
|
||||
db.execute("""
|
||||
INSERT OR REPLACE INTO analysis
|
||||
(material_id, tags, golden_quote, core_idea, structure,
|
||||
emotion, worth_downloading, download_reason, raw_model_output)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", (
|
||||
result["material_id"], result["tags"], result["golden_quote"],
|
||||
result["core_idea"], result["structure"], result["emotion"],
|
||||
result["worth_downloading"], result["download_reason"],
|
||||
result["raw_model_output"],
|
||||
))
|
||||
# 更新素材分析状态
|
||||
db.execute("UPDATE materials SET analyzed = 1 WHERE id = ?", (material["id"],))
|
||||
db.commit()
|
||||
success += 1
|
||||
else:
|
||||
db.execute("UPDATE materials SET analyzed = 2 WHERE id = ?", (material["id"],))
|
||||
db.commit()
|
||||
failed += 1
|
||||
|
||||
# 间隔 1 秒，避免过载
|
||||
time.sleep(1)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"分析异常 [{material['id']}]: {e}")
|
||||
db.execute("UPDATE materials SET analyzed = 2 WHERE id = ?", (material["id"],))
|
||||
db.commit()
|
||||
failed += 1
|
||||
|
||||
db.close()
|
||||
|
||||
print(f"\r[DONE] 分析完成: 成功 {success}, 失败 {failed}, 共 {len(materials)} 条")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,704 @@
|
||||
"""
|
||||
口播素材系统 - Web 界面（Gradio）
|
||||
用法：python app.py
|
||||
|
||||
功能：
|
||||
- 搜索/筛选素材（关键词、平台、标签、情绪、下载状态）
|
||||
- 查看素材详情和 AI 分析结果
|
||||
- 一键复制金句
|
||||
- 标记下载 / 本地播放
|
||||
|
||||
启动后访问：http://127.0.0.1:7860
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from utils import load_config, get_db, init_db, setup_logger, PROJECT_ROOT
|
||||
|
||||
try:
|
||||
import gradio as gr
|
||||
except ImportError:
|
||||
print("[ERROR] Gradio 未安装，请执行：pip install gradio")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 数据库查询
|
||||
# ============================================================
|
||||
|
||||
def search_materials(
|
||||
keyword: str = "",
|
||||
platform: str = "全部",
|
||||
emotion: str = "全部",
|
||||
has_file: str = "全部",
|
||||
worth_download: str = "全部",
|
||||
sort_by: str = "fetched_at DESC",
|
||||
page: int = 1,
|
||||
page_size: int = 20,
|
||||
) -> tuple[list[dict], int]:
|
||||
"""搜索素材，返回（结果列表, 总数）"""
|
||||
config = load_config()
|
||||
db = get_db(init_db())
|
||||
|
||||
where_clauses = []
|
||||
params = []
|
||||
|
||||
if keyword:
|
||||
where_clauses.append("(m.title LIKE ? OR m.description LIKE ? OR a.tags LIKE ? OR a.golden_quote LIKE ?)")
|
||||
kw = f"%{keyword}%"
|
||||
params.extend([kw, kw, kw, kw])
|
||||
|
||||
if platform and platform != "全部":
|
||||
where_clauses.append("m.platform = ?")
|
||||
params.append(platform)
|
||||
|
||||
if emotion and emotion != "全部":
|
||||
where_clauses.append("a.emotion = ?")
|
||||
params.append(emotion)
|
||||
|
||||
if has_file == "已下载":
|
||||
where_clauses.append("m.download_status = 'done'")
|
||||
elif has_file == "未下载":
|
||||
where_clauses.append("m.download_status != 'done'")
|
||||
|
||||
if worth_download == "推荐下载":
|
||||
where_clauses.append("a.worth_downloading = 'yes'")
|
||||
elif worth_download == "不推荐":
|
||||
where_clauses.append("a.worth_downloading = 'no'")
|
||||
|
||||
where_sql = " AND ".join(where_clauses) if where_clauses else "1=1"
|
||||
|
||||
# 总数
|
||||
count_sql = f"""
|
||||
SELECT COUNT(*) FROM materials m
|
||||
LEFT JOIN analysis a ON m.id = a.material_id
|
||||
WHERE {where_sql}
|
||||
"""
|
||||
total = db.execute(count_sql, params).fetchone()[0]
|
||||
|
||||
# 排序和分页
|
||||
offset = (page - 1) * page_size
|
||||
data_sql = f"""
|
||||
SELECT m.*, a.tags, a.golden_quote, a.core_idea, a.structure,
|
||||
a.emotion, a.worth_downloading, a.download_reason
|
||||
FROM materials m
|
||||
LEFT JOIN analysis a ON m.id = a.material_id
|
||||
WHERE {where_sql}
|
||||
ORDER BY {sort_by}
|
||||
LIMIT ? OFFSET ?
|
||||
"""
|
||||
params.extend([page_size, offset])
|
||||
rows = db.execute(data_sql, params).fetchall()
|
||||
results = [dict(r) for r in rows]
|
||||
|
||||
db.close()
|
||||
return results, total
|
||||
|
||||
|
||||
def get_material_detail(material_id: int) -> dict:
|
||||
"""获取素材详情"""
|
||||
db = get_db(init_db())
|
||||
row = db.execute("""
|
||||
SELECT m.*, a.tags, a.golden_quote, a.core_idea, a.structure,
|
||||
a.emotion, a.worth_downloading, a.download_reason, a.raw_model_output
|
||||
FROM materials m
|
||||
LEFT JOIN analysis a ON m.id = a.material_id
|
||||
WHERE m.id = ?
|
||||
""", (material_id,)).fetchone()
|
||||
db.close()
|
||||
return dict(row) if row else {}
|
||||
|
||||
|
||||
def mark_download(material_id: int) -> str:
|
||||
"""标记素材为待下载"""
|
||||
db = get_db(init_db())
|
||||
db.execute("UPDATE materials SET download_status = 'pending' WHERE id = ?", (material_id,))
|
||||
db.commit()
|
||||
db.close()
|
||||
return "已加入下载队列"
|
||||
|
||||
|
||||
def get_stats() -> dict:
|
||||
"""获取统计信息"""
|
||||
db = get_db(init_db())
|
||||
total = db.execute("SELECT COUNT(*) FROM materials").fetchone()[0]
|
||||
analyzed = db.execute("SELECT COUNT(*) FROM materials WHERE analyzed = 1").fetchone()[0]
|
||||
downloaded = db.execute("SELECT COUNT(*) FROM materials WHERE download_status = 'done'").fetchone()[0]
|
||||
pending = db.execute("SELECT COUNT(*) FROM materials WHERE download_status = 'pending'").fetchone()[0]
|
||||
recommended = db.execute("SELECT COUNT(*) FROM analysis WHERE worth_downloading = 'yes'").fetchone()[0]
|
||||
db.close()
|
||||
return {
|
||||
"总素材": total,
|
||||
"已分析": analyzed,
|
||||
"已下载": downloaded,
|
||||
"待下载": pending,
|
||||
"AI推荐": recommended,
|
||||
}
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Gradio 界面
|
||||
# ============================================================
|
||||
|
||||
def build_app() -> gr.Blocks:
|
||||
config = load_config()
|
||||
|
||||
# 初始化数据库
|
||||
init_db()
|
||||
stats = get_stats()
|
||||
|
||||
with gr.Blocks(
|
||||
title=config.get("web", {}).get("title", "口播灵感工坊"),
|
||||
) as app:
|
||||
gr.Markdown("# 🎬 口播灵感工坊 — 素材采集 → 脚本生成")
|
||||
|
||||
with gr.Tabs():
|
||||
# ====== 标签页1：素材库 ======
|
||||
with gr.Tab("📦 素材库"):
|
||||
stats_text = gr.Markdown(
|
||||
f"📊 **统计** | 总素材: {stats['总素材']} | 已分析: {stats['已分析']} | "
|
||||
f"已下载: {stats['已下载']} | 待下载: {stats['待下载']} | AI推荐: {stats['AI推荐']}"
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
keyword_input = gr.Textbox(label="关键词搜索", placeholder="标题/描述/标签/金句", scale=2)
|
||||
platform_dd = gr.Dropdown(
|
||||
label="平台", choices=["全部", "xiaohongshu", "douyin", "bilibili"],
|
||||
value="全部", scale=1
|
||||
)
|
||||
emotion_dd = gr.Dropdown(
|
||||
label="情绪", choices=["全部", "治愈", "励志", "震惊", "沉思", "有趣", "感动"],
|
||||
value="全部", scale=1
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
has_file_dd = gr.Dropdown(
|
||||
label="下载状态", choices=["全部", "已下载", "未下载"],
|
||||
value="全部", scale=1
|
||||
)
|
||||
worth_dd = gr.Dropdown(
|
||||
label="AI推荐", choices=["全部", "推荐下载", "不推荐"],
|
||||
value="全部", scale=1
|
||||
)
|
||||
sort_dd = gr.Dropdown(
|
||||
label="排序", choices=["fetched_at DESC", "like_count DESC", "collect_count DESC"],
|
||||
value="fetched_at DESC", scale=1
|
||||
)
|
||||
search_btn = gr.Button("🔍 搜索", variant="primary", scale=1)
|
||||
refresh_btn = gr.Button("🔄 刷新统计", scale=1)
|
||||
|
||||
result_table = gr.Dataframe(
|
||||
headers=["ID", "平台", "标题", "作者", "金句", "情绪", "推荐下载", "状态"],
|
||||
label="搜索结果",
|
||||
interactive=False,
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
with gr.Column(scale=1):
|
||||
detail_id = gr.Number(label="素材ID", precision=0)
|
||||
with gr.Row():
|
||||
detail_btn = gr.Button("查看详情", variant="secondary")
|
||||
download_btn = gr.Button("📥 加入下载队列", variant="stop")
|
||||
|
||||
with gr.Column(scale=2):
|
||||
detail_output = gr.Markdown("选择一条素材后点击「查看详情」")
|
||||
|
||||
# ====== 标签页2：脚本工厂 ======
|
||||
with gr.Tab("🎬 脚本工厂"):
|
||||
gr.Markdown("### 从素材或主题生成 AVE v2.0 导演脚本")
|
||||
|
||||
with gr.Row():
|
||||
script_source = gr.Radio(
|
||||
choices=["输入主题", "从素材库ID"],
|
||||
label="脚本来源", value="输入主题"
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
topic_input = gr.Textbox(
|
||||
label="主题", placeholder="输入口播主题，如：普通人如何抓住AI时代的机会",
|
||||
visible=True, scale=3
|
||||
)
|
||||
material_id_input = gr.Number(
|
||||
label="素材ID", precision=0, value=0,
|
||||
visible=False, scale=1
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
style_dd = gr.Dropdown(
|
||||
label="视频风格",
|
||||
choices=["knowledge_lecture", "bedtime_story", "funny_talk", "tech_review"],
|
||||
value="knowledge_lecture", scale=1
|
||||
)
|
||||
gen_btn = gr.Button("🚀 生成脚本", variant="primary", scale=1)
|
||||
|
||||
with gr.Row():
|
||||
with gr.Column():
|
||||
gr.Markdown("#### 📄 AVE 导演脚本 (YAML)")
|
||||
yaml_output = gr.Textbox(label="", lines=20, max_lines=30)
|
||||
with gr.Column():
|
||||
gr.Markdown("#### 📝 说明文档 (MD)")
|
||||
md_output = gr.Textbox(label="", lines=20, max_lines=30)
|
||||
|
||||
with gr.Row():
|
||||
project_id_display = gr.Textbox(label="项目ID", visible=False)
|
||||
save_btn = gr.Button("💾 保存到素材库", variant="primary")
|
||||
save_status = gr.Markdown("")
|
||||
|
||||
gr.Markdown("---")
|
||||
gr.Markdown("#### 📁 历史脚本")
|
||||
with gr.Row():
|
||||
refresh_list_btn = gr.Button("🔄 刷新列表", scale=1)
|
||||
script_list_md = gr.Markdown("点击刷新查看已生成的脚本")
|
||||
|
||||
with gr.Row():
|
||||
view_pid = gr.Textbox(label="查看脚本（粘贴项目ID）", placeholder="如 vid_20260504_8817", scale=2)
|
||||
view_btn = gr.Button("📖 查看", scale=1)
|
||||
|
||||
with gr.Row():
|
||||
with gr.Column():
|
||||
gr.Markdown("##### YAML")
|
||||
view_yaml = gr.Textbox(label="", lines=8, max_lines=15)
|
||||
with gr.Column():
|
||||
gr.Markdown("##### MD")
|
||||
view_md = gr.Textbox(label="", lines=8, max_lines=15)
|
||||
|
||||
# 历史脚本回调
|
||||
def list_scripts():
|
||||
from pathlib import Path
|
||||
output_dir = Path(__file__).parent / "scripts_output"
|
||||
yamls = sorted(output_dir.glob("*.yaml"), reverse=True)
|
||||
if not yamls:
|
||||
return "（暂无已生成的脚本）"
|
||||
|
||||
lines = ["| 项目ID | 风格 | 段落 | 日期 |",
|
||||
"|--------|------|------|------|"]
|
||||
for y in yamls:
|
||||
pid = y.stem
|
||||
import yaml as pyyaml
|
||||
try:
|
||||
data = pyyaml.safe_load(y.read_text(encoding='utf-8'))
|
||||
style = data.get('meta', {}).get('style', '?')
|
||||
segs = len(data.get('segments', []))
|
||||
except:
|
||||
style = '?'; segs = '?'
|
||||
date = pid.split('_')[1] if '_' in pid else '?'
|
||||
date = f"{date[:4]}-{date[4:6]}-{date[6:8]}" if len(date)>=8 else '?'
|
||||
lines.append(f"| `{pid}` | {style} | {segs}段 | {date} |")
|
||||
lines.append("\n复制项目ID到下方输入框查看具体内容")
|
||||
return "\n".join(lines)
|
||||
|
||||
def view_script(pid):
|
||||
if not pid:
|
||||
return "请输入项目ID", ""
|
||||
from pathlib import Path
|
||||
output_dir = Path(__file__).parent / "scripts_output"
|
||||
yaml_path = output_dir / f"{pid.strip()}.yaml"
|
||||
md_path = output_dir / f"{pid.strip()}.md"
|
||||
yaml_c = yaml_path.read_text(encoding='utf-8') if yaml_path.exists() else "文件不存在"
|
||||
md_c = md_path.read_text(encoding='utf-8') if md_path.exists() else "文件不存在"
|
||||
return yaml_c, md_c
|
||||
|
||||
refresh_list_btn.click(list_scripts, outputs=[script_list_md])
|
||||
view_btn.click(view_script, inputs=[view_pid], outputs=[view_yaml, view_md])
|
||||
|
||||
# 脚本工厂回调
|
||||
def toggle_source(source):
|
||||
show_topic = source == "输入主题"
|
||||
return (
|
||||
gr.update(visible=show_topic),
|
||||
gr.update(visible=not show_topic),
|
||||
)
|
||||
|
||||
def generate_script(topic, material_id, style):
|
||||
import subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
script_path = Path(__file__).parent / "script_factory.py"
|
||||
cmd = [sys.executable, str(script_path), "--topic", topic, "--style", style]
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
|
||||
if result.returncode != 0:
|
||||
return "生成失败:\n" + result.stderr[:500], "生成失败", ""
|
||||
|
||||
# 找到最新生成的YAML和MD
|
||||
output_dir = Path(__file__).parent / "scripts_output"
|
||||
yamls = sorted(output_dir.glob("*.yaml"))
|
||||
mds = sorted(output_dir.glob("*.md"))
|
||||
|
||||
if not yamls:
|
||||
return "未找到生成的脚本文件", "未找到", ""
|
||||
|
||||
latest_yaml = yamls[-1]
|
||||
latest_md = mds[-1]
|
||||
|
||||
pid = latest_yaml.stem
|
||||
yaml_content = latest_yaml.read_text(encoding="utf-8")
|
||||
md_content = latest_md.read_text(encoding="utf-8")
|
||||
|
||||
return yaml_content, md_content, pid
|
||||
except subprocess.TimeoutExpired:
|
||||
return "生成超时（>5分钟）", "超时", ""
|
||||
except Exception as e:
|
||||
return f"生成异常: {str(e)}", "异常", ""
|
||||
|
||||
def save_script(pid):
|
||||
if not pid:
|
||||
return "请先生成脚本"
|
||||
# 复制到 materials/
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
output_dir = Path(__file__).parent / "scripts_output"
|
||||
materials_dir = Path.home() / "workbuddy-agent-os/agent-local/materials/videos"
|
||||
materials_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
yaml_src = output_dir / f"{pid}.yaml"
|
||||
md_src = output_dir / f"{pid}.md"
|
||||
|
||||
if yaml_src.exists():
|
||||
shutil.copy2(str(yaml_src), str(materials_dir / f"{pid}.yaml"))
|
||||
if md_src.exists():
|
||||
shutil.copy2(str(md_src), str(materials_dir / f"{pid}.md"))
|
||||
|
||||
return f"✅ 已保存到 `{materials_dir}`"
|
||||
|
||||
# 事件绑定
|
||||
script_source.change(
|
||||
toggle_source,
|
||||
inputs=[script_source],
|
||||
outputs=[topic_input, material_id_input]
|
||||
)
|
||||
|
||||
gen_btn.click(
|
||||
generate_script,
|
||||
inputs=[topic_input, material_id_input, style_dd],
|
||||
outputs=[yaml_output, md_output, project_id_display]
|
||||
)
|
||||
|
||||
save_btn.click(
|
||||
save_script,
|
||||
inputs=[project_id_display],
|
||||
outputs=[save_status]
|
||||
)
|
||||
|
||||
# ====== 标签页3：智能搜索 ======
|
||||
with gr.Tab("🔍 智能搜索"):
|
||||
gr.Markdown("### 搜索平台内容 → 预览筛选 → 下载分析 → 生成脚本")
|
||||
gr.Markdown("""> **百度**: 直连搜索，无需扩展 ✅
|
||||
> **抖音/知乎/小红书/B站/微博**: 需安装 OpenCLI Chrome 扩展（详见帮助）
|
||||
>
|
||||
> **使用建议**:
|
||||
> - 选"百度"可直接搜热点/选题
|
||||
> - 其他平台需先装 Chrome 扩展才能用
|
||||
> - 也可直接粘贴链接到脚本工厂
|
||||
""")
|
||||
|
||||
with gr.Row():
|
||||
search_keyword = gr.Textbox(label="关键词", placeholder="输入搜索关键词", scale=3)
|
||||
search_platform = gr.Dropdown(
|
||||
label="平台", choices=["baidu", "douyin", "zhihu", "xiaohongshu", "bilibili", "weibo"],
|
||||
value="baidu", scale=1
|
||||
)
|
||||
do_search_btn = gr.Button("🔍 搜索", variant="primary", scale=1)
|
||||
|
||||
search_status = gr.Markdown("")
|
||||
|
||||
# 搜索结果表格（带序号勾选）
|
||||
search_results = gr.Dataframe(
|
||||
headers=["选", "标题", "平台", "作者", "简介", "链接"],
|
||||
label="搜索结果（勾选后点下方按钮处理）",
|
||||
interactive=True,
|
||||
col_count=(6, "fixed"),
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
max_items = gr.Slider(minimum=3, maximum=30, value=10, step=1, label="最大结果数", scale=1)
|
||||
process_selected_btn = gr.Button("📥 下载并分析选中项", variant="primary", scale=2)
|
||||
|
||||
process_status = gr.Markdown("")
|
||||
|
||||
with gr.Row():
|
||||
with gr.Column():
|
||||
gr.Markdown("#### 📄 生成的 YAML")
|
||||
search_yaml_out = gr.Textbox(label="", lines=12, max_lines=20)
|
||||
with gr.Column():
|
||||
gr.Markdown("#### 📝 说明文档")
|
||||
search_md_out = gr.Textbox(label="", lines=12, max_lines=20)
|
||||
|
||||
# 搜索回调
|
||||
def do_web_search(keyword, platform, max_n):
|
||||
if not keyword:
|
||||
return [], "请输入关键词"
|
||||
|
||||
import sys, importlib
|
||||
import collect as collect_mod
|
||||
importlib.reload(collect_mod)
|
||||
|
||||
# 百度直接走 HTTP，其他平台走 OpenCLI
|
||||
results = collect_mod.search_web(keyword, platform, int(max_n))
|
||||
|
||||
if not results:
|
||||
msg = "未找到结果"
|
||||
if platform != "baidu":
|
||||
msg += "。请确认: ① Chrome 扩展已安装 ② opencli daemon 运行中 ③ Chrome 已登录该平台"
|
||||
return [[""] * 6], msg
|
||||
|
||||
table = []
|
||||
for i, r in enumerate(results):
|
||||
title = (r.get("title", "") or "")[:30]
|
||||
brief = (r.get("brief", "") or "")[:50]
|
||||
url = (r.get("url", "") or "")[:60]
|
||||
author = (r.get("author", "") or "")[:15]
|
||||
table.append(["☐", title, r.get("platform", ""), author, brief, url])
|
||||
|
||||
return table, f"✅ 找到 {len(results)} 条结果，勾选感兴趣的，点击下方按钮处理"
|
||||
|
||||
def process_selected(table_data, keyword, platform):
|
||||
"""处理选中项：下载 + 转写 + 生成脚本"""
|
||||
if not table_data:
|
||||
return "", "请先搜索"
|
||||
|
||||
selected = []
|
||||
for row in table_data:
|
||||
if row and row[0] in ("☑", "✓", "✅", True, "True", "true"):
|
||||
selected.append(row)
|
||||
|
||||
if not selected:
|
||||
return "", "请勾选至少一条结果"
|
||||
|
||||
total = len(selected)
|
||||
report = []
|
||||
|
||||
for i, item in enumerate(selected):
|
||||
title = item[1] if len(item) > 1 else ""
|
||||
url = item[5] if len(item) > 5 else ""
|
||||
report.append(f" {i+1}. {title}")
|
||||
|
||||
result = f"✅ 选中 {total} 条：\n" + "\n".join(report)
|
||||
result += "\n\n⚠️ 批量下载转写功能需在命令行执行："
|
||||
result += f"\n python collect.py --platform {platform} --keyword \"{keyword}\""
|
||||
result += "\n\n当前界面支持：复制选中链接 → 到脚本工厂输入主题生成脚本"
|
||||
return "", result
|
||||
|
||||
do_search_btn.click(
|
||||
do_web_search,
|
||||
inputs=[search_keyword, search_platform, max_items],
|
||||
outputs=[search_results, search_status]
|
||||
)
|
||||
|
||||
process_selected_btn.click(
|
||||
process_selected,
|
||||
inputs=[search_results, search_keyword, search_platform],
|
||||
outputs=[search_yaml_out, process_status]
|
||||
)
|
||||
|
||||
# ====== 标签页4：跨平台情报 ======
|
||||
with gr.Tab("🌐 跨平台情报"):
|
||||
gr.Markdown("### 全平台并行搜索 → 统一对比 → 勾选处理")
|
||||
gr.Markdown("""> 输入主题后系统自动搜索 **百度 + 抖音 + 知乎 + 小红书 + B站 + 微博** 六个平台
|
||||
> 结果按匹配度排序，一眼对比哪个平台的素材最适合你
|
||||
""")
|
||||
|
||||
with gr.Row():
|
||||
xp_keyword = gr.Textbox(label="主题", placeholder="输入要搜索的主题/标题/思路", scale=3)
|
||||
xp_max = gr.Slider(minimum=3, maximum=10, value=5, step=1, label="每平台条数")
|
||||
xp_search_btn = gr.Button("🚀 全平台搜索", variant="primary", scale=1)
|
||||
|
||||
xp_status = gr.Markdown("")
|
||||
|
||||
# 统一结果（Markdown 展示）
|
||||
xp_results_md = gr.Markdown("搜索结果将显示在这里")
|
||||
xp_cache = gr.State([])
|
||||
|
||||
with gr.Row():
|
||||
with gr.Column(scale=2):
|
||||
gr.Markdown("#### 选择要处理的内容")
|
||||
gr.Markdown("从上方结果中复制序号（如 `1,3,5`）到下方输入框")
|
||||
xp_selected = gr.Textbox(
|
||||
label="选择序号",
|
||||
placeholder="输入序号，如 1,3,5",
|
||||
lines=2
|
||||
)
|
||||
xp_process_btn = gr.Button("📥 查看选中项", variant="primary")
|
||||
with gr.Column(scale=1):
|
||||
xp_output = gr.Markdown("")
|
||||
|
||||
# 搜索回调
|
||||
def do_xp_search(keyword, max_n):
|
||||
if not keyword:
|
||||
return "请输入主题", [], ""
|
||||
|
||||
import sys, importlib
|
||||
import collect as cm
|
||||
importlib.reload(cm)
|
||||
|
||||
results = cm.search_all(keyword, int(max_n))
|
||||
|
||||
if not results:
|
||||
return "未找到结果，试试其他关键词", [], ""
|
||||
|
||||
# 按平台分组统计
|
||||
platform_count = {}
|
||||
for r in results:
|
||||
p = r.get("platform", "?")
|
||||
platform_count[p] = platform_count.get(p, 0) + 1
|
||||
|
||||
summary = " | ".join([f"{cm.PLATFORM_LABELS.get(p,p)}: {c}条" for p, c in platform_count.items()])
|
||||
|
||||
# Markdown 表格（紧凑格式）
|
||||
lines = ["| # | 平台 | 标题（点击链接查看原文） | 作者 |",
|
||||
"|---|------|--------------------------|------|"]
|
||||
for i, r in enumerate(results, 1):
|
||||
title = (r.get("title", "") or "")[:40]
|
||||
author = (r.get("author", "") or "")[:10]
|
||||
label = r.get("label", "")
|
||||
url = (r.get("url", "") or "")
|
||||
# 加链接
|
||||
linked_title = f"[{title}]({url})" if url else title
|
||||
lines.append(f"| {i} | {label} | {linked_title} | {author} |")
|
||||
|
||||
md = "\n".join(lines)
|
||||
md += f"\n\n✅ 共 **{len(results)}** 条。{summary}"
|
||||
md += "\n\n> 复制上方序号（如 `1,3,5`）到右侧输入框查看详情。点击标题链接直接打开原文。"
|
||||
|
||||
return md, results
|
||||
|
||||
def do_xp_process(selection_text, cached_results):
|
||||
if not cached_results:
|
||||
return "请先搜索"
|
||||
if not selection_text.strip():
|
||||
return "请输入要查看的序号"
|
||||
|
||||
try:
|
||||
indices = [int(x.strip()) for x in selection_text.split(",") if x.strip().isdigit()]
|
||||
selected = []
|
||||
for idx in indices:
|
||||
if 1 <= idx <= len(cached_results):
|
||||
selected.append(cached_results[idx - 1])
|
||||
|
||||
if not selected:
|
||||
return "未匹配到有效序号"
|
||||
|
||||
report_parts = []
|
||||
for i, r in enumerate(selected):
|
||||
report_parts.append(f"**{i+1}. [{r.get('label','')}] {r.get('title','')}**")
|
||||
report_parts.append(f" - 平台: {r.get('platform','')} | 作者: {r.get('author','')} | 类型: {r.get('type','')}")
|
||||
report_parts.append(f" - 链接: {r.get('url','')}")
|
||||
report_parts.append(f" - 简介: {r.get('brief','')}")
|
||||
report_parts.append("")
|
||||
|
||||
return "\n".join(report_parts)
|
||||
except:
|
||||
return "序号格式错误，请用逗号分隔（如：1,3,5）"
|
||||
|
||||
xp_search_btn.click(
|
||||
do_xp_search,
|
||||
inputs=[xp_keyword, xp_max],
|
||||
outputs=[xp_results_md, xp_cache]
|
||||
)
|
||||
|
||||
xp_process_btn.click(
|
||||
do_xp_process,
|
||||
inputs=[xp_selected, xp_cache],
|
||||
outputs=[xp_output]
|
||||
)
|
||||
|
||||
# 素材库事件绑定（共用）
|
||||
def show_detail(material_id):
|
||||
if not material_id:
|
||||
return "请输入素材ID"
|
||||
detail = get_material_detail(int(material_id))
|
||||
if not detail:
|
||||
return "未找到该素材"
|
||||
|
||||
local_info = ""
|
||||
if detail.get("local_files"):
|
||||
files = json.loads(detail["local_files"])
|
||||
local_info = f"\n\n**本地文件**：\n"
|
||||
for k, v in files.items():
|
||||
local_info += f"- {k}: `{v}`\n"
|
||||
|
||||
tags = detail.get("tags") or "未分析"
|
||||
quote = detail.get("golden_quote") or "—"
|
||||
idea = detail.get("core_idea") or "—"
|
||||
structure = detail.get("structure") or "—"
|
||||
emotion = detail.get("emotion") or "—"
|
||||
worth = detail.get("worth_downloading") or "—"
|
||||
reason = detail.get("download_reason") or "—"
|
||||
status = detail.get("download_status") or "none"
|
||||
|
||||
return f"""### {detail.get('title', '无标题')}
|
||||
**平台**：{detail.get('platform', '')} | **作者**：{detail.get('author', '')}
|
||||
**链接**：{detail.get('url', '')}
|
||||
**状态**：{status}
|
||||
|
||||
---
|
||||
|
||||
**🏷️ 标签**：{tags}
|
||||
**💡 金句**：> {quote}
|
||||
**🎯 核心立意**：{idea}
|
||||
**📐 脚本结构**：{structure}
|
||||
**🎭 情绪**：{emotion}
|
||||
**⭐ 推荐下载**：{worth} — {reason}
|
||||
{local_info}"""
|
||||
|
||||
def do_search(keyword, platform, emotion, has_file, worth, sort_by):
|
||||
results, total = search_materials(keyword, platform, emotion, has_file, worth, sort_by)
|
||||
if not results:
|
||||
return [["—"] * 8]
|
||||
table_data = []
|
||||
for r in results:
|
||||
table_data.append([
|
||||
r["id"],
|
||||
r["platform"],
|
||||
(r["title"] or "")[:30],
|
||||
(r["author"] or "")[:15],
|
||||
(r.get("golden_quote") or "")[:25],
|
||||
r.get("emotion") or "—",
|
||||
r.get("worth_downloading") or "—",
|
||||
r.get("download_status") or "none",
|
||||
])
|
||||
return table_data
|
||||
|
||||
def do_download(material_id):
|
||||
if not material_id:
|
||||
return "请输入素材ID"
|
||||
return mark_download(int(material_id))
|
||||
|
||||
def refresh_stats():
|
||||
s = get_stats()
|
||||
return (
|
||||
f"📊 **统计** | 总素材: {s['总素材']} | 已分析: {s['已分析']} | "
|
||||
f"已下载: {s['已下载']} | 待下载: {s['待下载']} | AI推荐: {s['AI推荐']}"
|
||||
)
|
||||
|
||||
# 素材库事件绑定
|
||||
search_btn.click(do_search, inputs=[keyword_input, platform_dd, emotion_dd, has_file_dd, worth_dd, sort_dd], outputs=[result_table])
|
||||
detail_btn.click(show_detail, inputs=[detail_id], outputs=[detail_output])
|
||||
download_btn.click(do_download, inputs=[detail_id], outputs=[detail_output])
|
||||
refresh_btn.click(refresh_stats, outputs=[stats_text])
|
||||
|
||||
return app
|
||||
|
||||
|
||||
def main():
|
||||
config = load_config()
|
||||
host = config.get("web", {}).get("host", "127.0.0.1")
|
||||
port = config.get("web", {}).get("port", 7860)
|
||||
|
||||
# 清除代理（防止 Gradio 6.0 启动时被代理拦截 localhost）
|
||||
for key in ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "ALL_PROXY"]:
|
||||
os.environ.pop(key, None)
|
||||
|
||||
app = build_app()
|
||||
print(f"[INFO] 启动 Web 界面: http://{host}:{port}")
|
||||
app.launch(server_name=host, server_port=port, theme=gr.themes.Soft())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,257 @@
|
||||
"""
|
||||
搜索功能 — 支持 OpenCLI（浏览器桥接）与百度直接HTTP两种模式
|
||||
|
||||
OpenCLI 依赖：
|
||||
- npm 全局安装 @jackwener/opencli
|
||||
- Chrome 扩展已加载（opencli-extension）
|
||||
- opencli daemon 运行中
|
||||
"""
|
||||
|
||||
import re, subprocess, json, os, threading, time
|
||||
|
||||
# OpenCLI 路径
|
||||
OPENCLI_BIN = os.path.expanduser("~/.workbuddy/binaries/node/versions/22.12.0/bin/opencli")
|
||||
|
||||
# 平台映射：内部名称 → OpenCLI 子命令
|
||||
PLATFORM_CMDS = {
|
||||
"douyin": ("douyin", "search"),
|
||||
"zhihu": ("zhihu", "search"),
|
||||
"xiaohongshu": ("xiaohongshu", "search"),
|
||||
"bilibili": ("bilibili", "search"),
|
||||
"weibo": ("weibo", "search"),
|
||||
}
|
||||
|
||||
|
||||
def _opencli_available() -> bool:
|
||||
"""检查 OpenCLI 是否可用（扩展已连接）"""
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[OPENCLI_BIN, "doctor"],
|
||||
capture_output=True, text=True, timeout=10,
|
||||
env={**os.environ, "NODE_OPTIONS": ""},
|
||||
)
|
||||
return "[OK]" in r.stdout and "Extension" in r.stdout and "connected" in r.stdout
|
||||
except:
|
||||
return False
|
||||
|
||||
|
||||
def search_web(keyword: str, platform: str = "baidu", max_results: int = 15) -> list[dict]:
|
||||
"""
|
||||
搜索平台内容，只返回元数据（不下载任何文件）
|
||||
|
||||
平台支持（按优先级）：
|
||||
- baidu: 直连 httpx，无需扩展
|
||||
- douyin/zhihu/xiaohongshu/bilibili/weibo: 需 OpenCLI + Chrome 扩展
|
||||
|
||||
Returns:
|
||||
[{url, title, author, brief, platform}, ...]
|
||||
"""
|
||||
|
||||
# ========== 百度搜索：直接HTTP ==========
|
||||
if platform == "baidu":
|
||||
return _search_baidu(keyword, max_results)
|
||||
|
||||
# ========== OpenCLI 搜索 ==========
|
||||
cmd_info = PLATFORM_CMDS.get(platform)
|
||||
if not cmd_info:
|
||||
return []
|
||||
|
||||
plat, action = cmd_info
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[OPENCLI_BIN, plat, action, keyword, "-f", "json"],
|
||||
capture_output=True, text=True, timeout=60,
|
||||
env={**os.environ.copy(), "NODE_OPTIONS": ""},
|
||||
)
|
||||
if r.returncode != 0:
|
||||
return []
|
||||
|
||||
data = json.loads(r.stdout)
|
||||
# OpenCLI 返回格式：列表或对象列表
|
||||
items = data if isinstance(data, list) else data.get("data", data.get("results", [data]))
|
||||
|
||||
results = []
|
||||
for item in items[:max_results]:
|
||||
url = item.get("url") or item.get("link") or ""
|
||||
title = item.get("title") or item.get("name") or ""
|
||||
author = item.get("author") or item.get("author_name") or ""
|
||||
brief = item.get("description") or item.get("desc") or item.get("snippet") or ""
|
||||
if url and title:
|
||||
results.append({
|
||||
"url": url,
|
||||
"title": str(title)[:40],
|
||||
"author": str(author)[:20],
|
||||
"brief": str(brief)[:80],
|
||||
"platform": platform,
|
||||
})
|
||||
return results
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(f"[WARN] OpenCLI {platform} 搜索超时")
|
||||
return []
|
||||
except json.JSONDecodeError:
|
||||
print(f"[WARN] OpenCLI {platform} 返回非JSON")
|
||||
return []
|
||||
except FileNotFoundError:
|
||||
print(f"[WARN] OpenCLI 未安装，请执行: npm install -g @jackwener/opencli")
|
||||
return []
|
||||
except Exception as e:
|
||||
print(f"[WARN] OpenCLI {platform} 搜索失败: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def _search_baidu(keyword: str, max_results: int) -> list[dict]:
|
||||
"""百度搜索：直接 HTTP 请求"""
|
||||
try:
|
||||
import httpx
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept": "text/html,application/xhtml+xml",
|
||||
"Accept-Language": "zh-CN,zh;q=0.9",
|
||||
}
|
||||
resp = httpx.get(
|
||||
f"https://www.baidu.com/s?wd={keyword}&ie=utf-8",
|
||||
headers=headers, follow_redirects=True, timeout=15,
|
||||
)
|
||||
items = re.findall(r'<h3[^>]*>.*?<a[^>]*href="([^"]+)"[^>]*>(.*?)</a>', resp.text, re.DOTALL)
|
||||
results = []
|
||||
seen = set()
|
||||
for href, title_html in items:
|
||||
title = re.sub(r'<[^>]+>', '', title_html).strip()
|
||||
if title and len(title) > 4 and href not in seen:
|
||||
seen.add(href)
|
||||
full_url = href if href.startswith("http") else f"https://www.baidu.com{href}"
|
||||
results.append({
|
||||
"url": full_url, "title": title[:40],
|
||||
"author": "", "brief": "", "platform": "baidu",
|
||||
})
|
||||
if len(results) >= max_results:
|
||||
break
|
||||
return results
|
||||
except Exception as e:
|
||||
print(f"[WARN] 百度搜索失败: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# ============ 内容类型映射 ============
|
||||
CONTENT_TYPES = {
|
||||
"baidu": "文章",
|
||||
"douyin": "视频",
|
||||
"zhihu": "问答",
|
||||
"xiaohongshu": "图文",
|
||||
"bilibili": "视频",
|
||||
"weibo": "话题",
|
||||
}
|
||||
|
||||
PLATFORM_LABELS = {
|
||||
"baidu": "🔍 百度",
|
||||
"douyin": "🎬 抖音",
|
||||
"zhihu": "📝 知乎",
|
||||
"xiaohongshu": "📕 小红书",
|
||||
"bilibili": "📺 B站",
|
||||
"weibo": "💬 微博",
|
||||
}
|
||||
|
||||
|
||||
def _search_platform_thread(keyword: str, platform: str, max_n: int, results: dict, index: int):
|
||||
"""线程包装：搜索单个平台，结果存入 results[index]"""
|
||||
try:
|
||||
r = search_web(keyword, platform, max_n)
|
||||
results[index] = r if r else []
|
||||
except Exception as e:
|
||||
results[index] = []
|
||||
|
||||
|
||||
def search_all(keyword: str, max_per_platform: int = 5) -> list[dict]:
|
||||
"""
|
||||
全平台并行搜索，返回统一格式的合并结果列表。
|
||||
|
||||
搜索6个平台：百度、抖音、知乎、小红书、B站、微博
|
||||
每个平台独立超时，失败不影响其他平台。
|
||||
|
||||
Returns:
|
||||
[{platform, label, title, author, url, brief, type, size_hint, score}, ...]
|
||||
"""
|
||||
platforms = ["baidu", "douyin", "zhihu", "xiaohongshu", "bilibili", "weibo"]
|
||||
thread_results = [None] * len(platforms)
|
||||
threads = []
|
||||
|
||||
for i, plat in enumerate(platforms):
|
||||
t = threading.Thread(
|
||||
target=_search_platform_thread,
|
||||
args=(keyword, plat, max_per_platform, thread_results, i),
|
||||
)
|
||||
t.start()
|
||||
threads.append(t)
|
||||
|
||||
# 等待所有线程完成（总超时60秒）
|
||||
for t in threads:
|
||||
t.join(timeout=60)
|
||||
|
||||
# 合并结果，附带元数据
|
||||
all_results = []
|
||||
for i, plat in enumerate(platforms):
|
||||
items = thread_results[i] or []
|
||||
for item in items:
|
||||
all_results.append({
|
||||
"platform": plat,
|
||||
"label": PLATFORM_LABELS.get(plat, plat),
|
||||
"type": CONTENT_TYPES.get(plat, "其他"),
|
||||
"title": item.get("title", ""),
|
||||
"author": item.get("author", ""),
|
||||
"url": item.get("url", ""),
|
||||
"brief": item.get("brief", ""),
|
||||
"size_hint": _guess_content_size(item, plat),
|
||||
"score": _calc_match_score(keyword, item),
|
||||
})
|
||||
|
||||
# 按匹配度排序
|
||||
all_results.sort(key=lambda x: x["score"], reverse=True)
|
||||
return all_results
|
||||
|
||||
|
||||
def _guess_content_size(item: dict, platform: str) -> str:
|
||||
"""估算内容量（视频时长/文字字数）"""
|
||||
title = item.get("title", "") or ""
|
||||
brief = item.get("brief", "") or ""
|
||||
combined = title + brief
|
||||
|
||||
if platform in ("douyin", "bilibili"):
|
||||
# 视频平台：平均时长约 3-10 分钟
|
||||
return "3-10分钟视频"
|
||||
|
||||
if platform in ("baidu", "zhihu", "weibo", "xiaohongshu"):
|
||||
# 文字平台：按摘要长度估算
|
||||
word_count = len(combined)
|
||||
if word_count > 100:
|
||||
return f"{word_count}字左右"
|
||||
elif platform in ("zhihu", "baidu"):
|
||||
return "长文（千字以上）"
|
||||
else:
|
||||
return "短文（简明）"
|
||||
|
||||
return "未知"
|
||||
|
||||
|
||||
def _calc_match_score(keyword: str, item: dict) -> int:
|
||||
"""
|
||||
计算匹配度分数（1-5）
|
||||
基于标题和描述中关键词的匹配程度
|
||||
"""
|
||||
title = (item.get("title", "") or "").lower()
|
||||
brief = (item.get("brief", "") or "").lower()
|
||||
kw = keyword.lower()
|
||||
|
||||
score = 1
|
||||
# 标题包含完整关键词 → 高分
|
||||
if kw in title:
|
||||
score += 3
|
||||
# 描述包含关键词 → 中分
|
||||
if kw in brief:
|
||||
score += 2
|
||||
# 标题包含关键词的部分词 → 基础分
|
||||
for word in kw.split():
|
||||
if len(word) > 1 and word in title:
|
||||
score += 1
|
||||
|
||||
return min(score, 5)
|
||||
@@ -0,0 +1,57 @@
|
||||
# 口播素材采集系统配置
|
||||
# 使用前请根据本机环境修改以下内容
|
||||
|
||||
# 采集平台（目前支持 xiaohongshu / douyin / bilibili）
|
||||
platforms:
|
||||
- xiaohongshu
|
||||
|
||||
# 采集关键词列表
|
||||
keywords:
|
||||
- "普通人逆袭"
|
||||
- "情感故事"
|
||||
- "科普冷知识"
|
||||
- "电影解说结构"
|
||||
|
||||
# 每个关键词每次采集的最大数量
|
||||
max_count: 20
|
||||
|
||||
# 本地 AI 模型配置（oMLX，非 Ollama）
|
||||
llm:
|
||||
model: "Qwen2.5-VL-3B-Instruct-8bit"
|
||||
base_url: "http://localhost:8000/v1"
|
||||
api_key: "omlx"
|
||||
timeout: 120 # 每条分析超时秒数
|
||||
temperature: 0.3 # 低温度保证输出稳定
|
||||
|
||||
# 数据存储路径（相对于项目根目录）
|
||||
storage:
|
||||
raw_dir: "data/raw" # JSONL 原始数据
|
||||
db_path: "data/database.db" # SQLite 数据库
|
||||
media_dir: "~/workbuddy-agent-os/agent-local/materials/video" # 下载的视频
|
||||
audio_dir: "~/workbuddy-agent-os/agent-local/materials/audio" # 下载的音频
|
||||
logs_dir: "logs" # 运行日志
|
||||
|
||||
# MediaCrawler 配置
|
||||
crawler:
|
||||
# MediaCrawler 项目路径（需手动 clone）
|
||||
# 安装方式：cd ~/workbuddy-agent-os/agent-sync/05_tools/05_crawl && git clone https://github.com/NanmiCoder/MediaCrawler.git
|
||||
project_path: ""
|
||||
# 浏览器模式：headless（后台） / headed（有界面，调试用）
|
||||
browser_mode: "headless"
|
||||
# 请求间隔（秒），防反爬
|
||||
request_delay: 2
|
||||
|
||||
# Web 界面配置
|
||||
web:
|
||||
host: "127.0.0.1"
|
||||
port: 7860
|
||||
title: "口播灵感工坊"
|
||||
|
||||
# 下载配置
|
||||
download:
|
||||
# yt-dlp 下载速率限制，0 表示不限
|
||||
rate_limit: "1M"
|
||||
# 下载重试次数
|
||||
retries: 3
|
||||
# 文件名模板
|
||||
filename_template: "{platform}_{uploader}_{title:.20s}_{upload_date}.{ext}"
|
||||
Binary file not shown.
@@ -0,0 +1,42 @@
|
||||
# OpenCLI 集成说明
|
||||
|
||||
> 最后更新：2026-06-27
|
||||
|
||||
## 用途
|
||||
|
||||
通过 Chrome 扩展桥接，实现各平台搜索（抖音/知乎/小红书/B站/微博等）。
|
||||
|
||||
## 安装（每台新机器）
|
||||
|
||||
```bash
|
||||
bash install_opencli_extension.sh
|
||||
```
|
||||
|
||||
然后手动：Chrome → chrome://extensions/ → 开发者模式 → 加载已解包 → 选 ~/opencli-extension/
|
||||
|
||||
## 验证
|
||||
|
||||
```bash
|
||||
opencli doctor
|
||||
# [OK] Daemon + [OK] Extension: connected + [OK] Connectivity
|
||||
```
|
||||
|
||||
## 平台登录要求
|
||||
|
||||
| 平台 | 需要登录？ |
|
||||
|------|-----------|
|
||||
| 百度 | ❌ 不需要 |
|
||||
| B站 | ✅ 搜索需要登录 |
|
||||
| 知乎/抖音/小红书/微博 | ✅ 需要登录 |
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 每台机器各自安装，各自登录
|
||||
- 重启电脑后需重新启动 daemon
|
||||
|
||||
## 本机配置
|
||||
|
||||
- 扩展连接码: `9zdft8vm`（随机生成，各机不同）
|
||||
- Chrome 已登录平台: 百度、B站、知乎、小红书、抖音、微博
|
||||
- OpenCLI v1.7.21 + 扩展 v1.0.15
|
||||
- 可用升级: `npm install -g @jackwener/opencli`（到 v1.8.4）+ 扩展 v1.0.20
|
||||
@@ -0,0 +1,243 @@
|
||||
"""
|
||||
豆包浏览器驱动 v7.1 —— 稳定等待法
|
||||
|
||||
改进：
|
||||
1. 豆包保持独立标签页，不关闭，不 rebind
|
||||
2. 抖音用另一个标签页，互不干扰
|
||||
3. 稳定等待法：触发增长后等连续 3 轮无变化才算完成，防止提前截断
|
||||
4. 改用 fill + click #flow-end-msg-send 替代 KeyboardEvent（React兼容）
|
||||
"""
|
||||
|
||||
import re, json, time, os, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
OPENCLI = os.path.expanduser("~/.workbuddy/binaries/node/versions/22.12.0/bin/opencli")
|
||||
|
||||
ANALYSIS_PROMPT = """请从这段视频内容中提取所有知识性内容，按以下格式完整记录，不要概括，不要总结：
|
||||
|
||||
【完整字幕】
|
||||
逐句记录视频中的每一句话，一字不差
|
||||
|
||||
【提示词/指令原文】
|
||||
如果视频中出现了AI提示词、模板、指令，请完整抄录，一字不改
|
||||
|
||||
【操作步骤】
|
||||
视频中演示的具体操作流程，分步骤详细记录
|
||||
|
||||
【关键参数】
|
||||
视频中提到的工具名称、参数设置、具体数字等
|
||||
|
||||
【核心技巧】
|
||||
视频中传授的具体方法、技巧、经验
|
||||
|
||||
记住：只提取不总结，不要分析结构，不要分析背景音乐。"""
|
||||
|
||||
|
||||
def _oc(args, timeout=30):
|
||||
env = {**os.environ, "NODE_OPTIONS": ""}
|
||||
for k in ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"]:
|
||||
env.pop(k, None)
|
||||
try:
|
||||
r = subprocess.run([OPENCLI] + args, capture_output=True, text=True, timeout=timeout, env=env)
|
||||
return r.stdout
|
||||
except:
|
||||
return ""
|
||||
|
||||
|
||||
def _parse_sections(raw):
|
||||
"""
|
||||
按标记提取各段落。
|
||||
支持两种格式：
|
||||
- 【】格式（旧）
|
||||
- === 标题 === 格式（新，豆包AI常用）
|
||||
"""
|
||||
sections = {}
|
||||
|
||||
# 先找 AI 回复的起始位置（跳过用户消息）
|
||||
lines = raw.split('\n')
|
||||
ai_start = -1
|
||||
for i, line in enumerate(lines):
|
||||
stripped = line.strip()
|
||||
if (stripped.startswith('===') and
|
||||
any(k in stripped for k in ['完整字幕', '提示词', '操作步骤', '关键参数', '核心技巧'])):
|
||||
next_chunk = '\n'.join(lines[i:i+4])
|
||||
if '逐句记录' not in next_chunk and '如果视频' not in next_chunk:
|
||||
ai_start = i
|
||||
break
|
||||
|
||||
ai_raw = '\n'.join(lines[ai_start:]) if ai_start >= 0 else raw
|
||||
|
||||
# 格式1：=== 标题 === 内容
|
||||
pattern = r'===\s*([^=]+?)\s*===\s*([\s\S]*?)(?====\s*[^=]|$)'
|
||||
for m in re.finditer(pattern, ai_raw):
|
||||
title = m.group(1).strip()
|
||||
content = m.group(2).strip()
|
||||
skip = ['逐句记录', '视频中演示', '视频中提到', '如果视频', '记住：', '完整抄录']
|
||||
if content and len(content) > 5 and not any(s in content[:40] for s in skip):
|
||||
sections[title] = content
|
||||
|
||||
# 格式2：旧格式 【】兜底
|
||||
if not sections:
|
||||
markers = ["完整字幕", "提示词/指令原文", "提示词", "指令原文", "操作步骤", "关键参数", "核心技巧"]
|
||||
for marker in markers:
|
||||
m = re.search(rf'【{re.escape(marker)}】\s*([\s\S]*?)(?=【|\Z)', raw)
|
||||
if m:
|
||||
content = m.group(1).strip()
|
||||
content = re.sub(r'逐句记录.*|如果视频中.*|视频中演示.*|视频中提到.*|记住：.*', '', content)
|
||||
sections[marker] = content
|
||||
|
||||
return sections
|
||||
|
||||
|
||||
def _wait_for_response(session="doubao", max_wait=120, min_growth=500, stable_checks=3):
|
||||
"""
|
||||
稳定等待法：检测页面文本显著增长后，再等连续 stable_checks 轮无变化才算完成。
|
||||
防止 AI 还在生成就被提前截断的老 bug。
|
||||
"""
|
||||
before = len(_oc(["browser", session, "eval", "document.body.innerText"], timeout=10) or "")
|
||||
prev_len = before
|
||||
stable_count = 0
|
||||
triggered = False
|
||||
|
||||
for i in range(max_wait // 5):
|
||||
time.sleep(5)
|
||||
body = _oc(["browser", session, "eval", "document.body.innerText"], timeout=10) or ""
|
||||
current_len = len(body)
|
||||
growth = current_len - prev_len
|
||||
total_growth = current_len - before
|
||||
|
||||
has_sections = '===完整字幕' in body or '===操作步骤' in body or '===核心技巧' in body
|
||||
|
||||
if total_growth > min_growth or has_sections:
|
||||
triggered = True
|
||||
|
||||
if triggered:
|
||||
if growth == 0:
|
||||
stable_count += 1
|
||||
else:
|
||||
stable_count = 0 # 还在增长，重置
|
||||
|
||||
if i > 0 and i % 2 == 0:
|
||||
status = f" ⏳ {(i+1)*5}s...总增长{total_growth}字"
|
||||
if triggered:
|
||||
status += f" (稳定{stable_count}/{stable_checks})" if growth == 0 else " (增长中)"
|
||||
if has_sections:
|
||||
status += " ✅含结构"
|
||||
print(status, flush=True)
|
||||
|
||||
if triggered and stable_count >= stable_checks:
|
||||
print(f" ✅ 回复稳定，确认完成", flush=True)
|
||||
return body
|
||||
|
||||
print(f" ⏰ 超时({max_wait}s)，返回当前内容", flush=True)
|
||||
return _oc(["browser", session, "eval", "document.body.innerText"], timeout=10)
|
||||
|
||||
|
||||
class DoubaoDriver:
|
||||
"""豆包分析驱动 v7.1（稳定等待法）"""
|
||||
|
||||
def __init__(self):
|
||||
self.doubao_ready = False
|
||||
self.douyin_session = "douyin"
|
||||
self.doubao_session = "doubao"
|
||||
|
||||
def start_session(self):
|
||||
"""启动豆包会话（只一次，标签页保持打开）"""
|
||||
# 先开一个空白标签给 douyin session
|
||||
_oc(["browser", self.douyin_session, "bind"], timeout=5)
|
||||
time.sleep(0.5)
|
||||
# 开新标签给 doubao session（在新标签页打开豆包）
|
||||
_oc(["browser", self.doubao_session, "tab", "new", "https://www.doubao.com/chat"], timeout=15)
|
||||
time.sleep(5)
|
||||
self.doubao_ready = True
|
||||
print(" ✅ 豆包会话已启动（独立标签页）")
|
||||
|
||||
def analyze_video(self, url):
|
||||
"""分析单个视频"""
|
||||
t0 = time.time()
|
||||
try:
|
||||
# 1. 抖音标签页：打开视频提取文字
|
||||
_oc(["browser", self.douyin_session, "bind"], timeout=5)
|
||||
time.sleep(0.5)
|
||||
_oc(["browser", self.douyin_session, "open", url], timeout=15)
|
||||
time.sleep(6)
|
||||
page_text = _oc(["browser", self.douyin_session, "eval", "document.body.innerText"], timeout=10)
|
||||
page_text = re.sub(r'^\{[^}]*"result"[^}]*"|^"[}]$', '', page_text).strip()
|
||||
|
||||
if not page_text or len(page_text) < 50:
|
||||
return {"success": False, "error": "无法提取视频页内容", "url": url}
|
||||
|
||||
# 清理
|
||||
skip_kw = ['开启读屏', '读屏标签', '精选', '推荐', '搜索',
|
||||
'关注', '朋友', '我的', '直播', '放映厅', '壁纸',
|
||||
'通知', '私信', '投稿', '登录', '倍速', '智能',
|
||||
'清屏', '连播', '点击加载更多', '展开', '条回复',
|
||||
'评论', '广告投放', '用户服务']
|
||||
useful = [l.strip() for l in page_text.split('\n')
|
||||
if len(l.strip()) > 5 and not any(k in l for k in skip_kw)]
|
||||
|
||||
msg = f"视频 {url}\n\n视频内容：\n{chr(10).join(useful[:30])}\n\n{ANALYSIS_PROMPT}"
|
||||
|
||||
# 2. 豆包标签页（已存在，用 fill 填入 + click 发送按钮）
|
||||
_oc(["browser", self.doubao_session, "bind"], timeout=5)
|
||||
time.sleep(0.5)
|
||||
_oc(["browser", self.doubao_session, "fill", "--nth", "0", "textarea", msg], timeout=15)
|
||||
time.sleep(1)
|
||||
# 触发发送（React 页面无法通过 dispatchEvent 捕获 KeyboardEvent）
|
||||
# 优先使用显式发送按钮 click，这个按钮有 id=flow-end-msg-send
|
||||
_oc(["browser", self.doubao_session, "click", "#flow-end-msg-send"], timeout=5)
|
||||
|
||||
# 3. 智能等待回复
|
||||
raw = _wait_for_response(session=self.doubao_session)
|
||||
sections = _parse_sections(raw or "")
|
||||
|
||||
elapsed = round(time.time() - t0, 1)
|
||||
return {
|
||||
"success": True,
|
||||
"url": url,
|
||||
"elapsed": elapsed,
|
||||
"raw_response": raw or "",
|
||||
"sections": sections,
|
||||
}
|
||||
except Exception as e:
|
||||
return {"success": False, "error": str(e), "url": url}
|
||||
|
||||
def analyze_profile(self, profile_url, max_videos=5):
|
||||
"""分析主播主页"""
|
||||
_oc(["browser", "profile", "bind"], timeout=5)
|
||||
_oc(["browser", "profile", "open", profile_url], timeout=15)
|
||||
time.sleep(8)
|
||||
for _ in range(2):
|
||||
_oc(["browser", "profile", "scroll", "down"])
|
||||
time.sleep(3)
|
||||
|
||||
links_raw = _oc(["browser", "profile", "eval",
|
||||
"JSON.stringify(Array.from(document.querySelectorAll('a[href*=\"/video/\"]'))"
|
||||
f".slice(0,{max_videos}).map(a=>({{url:a.href,title:(a.textContent||'').trim().slice(0,40)}})))"],
|
||||
timeout=10)
|
||||
try:
|
||||
videos = json.loads(links_raw)
|
||||
except:
|
||||
videos = []
|
||||
|
||||
if not videos:
|
||||
return [{"success": False, "error": "无法获取视频列表", "url": profile_url}]
|
||||
|
||||
self.start_session()
|
||||
print(f" ✅ 获取到 {len(videos)} 个视频")
|
||||
|
||||
results = []
|
||||
for i, v in enumerate(videos):
|
||||
print(f"\n [{i+1}/{len(videos)}] {v.get('title','')[:30]}...")
|
||||
r = self.analyze_video(v["url"])
|
||||
r["title"] = v.get("title", "")
|
||||
results.append(r)
|
||||
time.sleep(2)
|
||||
return results
|
||||
|
||||
|
||||
def analyze_video(url):
|
||||
return DoubaoDriver().analyze_video(url)
|
||||
|
||||
def analyze_profile(url, max_n=5):
|
||||
return DoubaoDriver().analyze_profile(url, max_n)
|
||||
@@ -0,0 +1,188 @@
|
||||
"""
|
||||
口播素材系统 - 媒体下载脚本
|
||||
用法：python downloader.py [--limit N] [--all]
|
||||
|
||||
功能：
|
||||
1. 查询所有 download_status='pending' 的素材
|
||||
2. 使用 yt-dlp 下载视频/音频
|
||||
3. 保存到 ~/workbuddy-agent-os/agent-local/materials/video/ 或 audio/
|
||||
4. 更新数据库中的 local_files 和 download_status
|
||||
|
||||
前置条件：
|
||||
- yt-dlp 已安装：pip install yt-dlp
|
||||
- ffmpeg 已安装（音频提取需要）
|
||||
|
||||
注意：
|
||||
- 抖音/小红书链接有时效性，采集后尽快下载
|
||||
- 下载速率限制默认 1MB/s，防止 IP 封禁
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import argparse
|
||||
from datetime import datetime
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from utils import load_config, get_db, init_db, setup_logger, truncate_text, PROJECT_ROOT
|
||||
|
||||
|
||||
def download_one(material: dict, config: dict, logger) -> dict:
|
||||
"""
|
||||
下载单条素材的视频/音频
|
||||
|
||||
返回：{"video": "path", "audio": "path"} 或 None
|
||||
"""
|
||||
url = material.get("url", "")
|
||||
if not url:
|
||||
logger.warning(f"素材 {material['id']} 无 URL，跳过")
|
||||
return None
|
||||
|
||||
try:
|
||||
from yt_dlp import YoutubeDL
|
||||
except ImportError:
|
||||
logger.error("yt-dlp 未安装，请执行：pip install yt-dlp")
|
||||
return None
|
||||
|
||||
platform = material["platform"]
|
||||
title = material.get("title", "untitled")
|
||||
author = material.get("author", "unknown")
|
||||
|
||||
# 确定输出路径
|
||||
media_dir = os.path.expanduser(config["storage"].get("media_dir", "~/workbuddy-agent-os/agent-local/materials/video"))
|
||||
os.makedirs(media_dir, exist_ok=True)
|
||||
|
||||
# 文件名
|
||||
safe_title = truncate_text(title, 20)
|
||||
safe_author = truncate_text(author, 15)
|
||||
date_str = datetime.now().strftime("%Y%m%d")
|
||||
filename = f"{platform}_{safe_author}_{safe_title}_{date_str}"
|
||||
|
||||
outtmpl = os.path.join(media_dir, filename + ".%(ext)s")
|
||||
|
||||
# yt-dlp 配置
|
||||
ydl_opts = {
|
||||
"outtmpl": outtmpl,
|
||||
"format": "best[ext=mp4]/best", # 优先 mp4
|
||||
"max_filesize": 500 * 1024 * 1024, # 500MB 上限
|
||||
"retries": config["download"].get("retries", 3),
|
||||
"quiet": True,
|
||||
"no_warnings": True,
|
||||
"extract_flat": False,
|
||||
# 速率限制
|
||||
"ratelimit": 1024 * 1024, # 1MB/s
|
||||
}
|
||||
|
||||
result_files = {"video": None, "audio": None}
|
||||
|
||||
try:
|
||||
with YoutubeDL(ydl_opts) as ydl:
|
||||
info = ydl.extract_info(url, download=True)
|
||||
|
||||
if info:
|
||||
downloaded_file = ydl.prepare_filename(info)
|
||||
# yt-dlp 可能修改了扩展名
|
||||
if os.path.exists(downloaded_file):
|
||||
result_files["video"] = downloaded_file
|
||||
logger.info(f"下载成功: {os.path.basename(downloaded_file)}")
|
||||
else:
|
||||
# 查找实际下载的文件
|
||||
for ext in ["mp4", "flv", "webm", "mkv"]:
|
||||
candidate = os.path.join(media_dir, f"{filename}.{ext}")
|
||||
if os.path.exists(candidate):
|
||||
result_files["video"] = candidate
|
||||
break
|
||||
|
||||
return result_files if result_files["video"] else None
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"下载失败 [{material['id']}]: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="口播素材媒体下载")
|
||||
parser.add_argument("--limit", "-n", type=int, default=0, help="下载数量上限，0=全部")
|
||||
parser.add_argument("--all", action="store_true", help="下载所有 worth_downloading=yes 的素材")
|
||||
parser.add_argument("--url", "-u", help="手动指定 URL 下载（不查数据库）")
|
||||
args = parser.parse_args()
|
||||
|
||||
config = load_config()
|
||||
logger = setup_logger("download")
|
||||
|
||||
db_path = init_db()
|
||||
db = get_db(db_path)
|
||||
|
||||
if args.url:
|
||||
# 手动 URL 下载
|
||||
material = {
|
||||
"id": 0,
|
||||
"url": args.url,
|
||||
"platform": "manual",
|
||||
"title": "手动下载",
|
||||
"author": "manual",
|
||||
}
|
||||
result = download_one(material, config, logger)
|
||||
if result:
|
||||
print(f"[OK] 下载完成: {result}")
|
||||
else:
|
||||
print("[FAIL] 下载失败")
|
||||
db.close()
|
||||
return
|
||||
|
||||
# 查询待下载记录
|
||||
if args.all:
|
||||
# 下载所有 AI 推荐的
|
||||
query = """
|
||||
SELECT m.* FROM materials m
|
||||
JOIN analysis a ON m.id = a.material_id
|
||||
WHERE a.worth_downloading = 'yes' AND m.download_status = 'none'
|
||||
"""
|
||||
else:
|
||||
query = "SELECT * FROM materials WHERE download_status = 'pending'"
|
||||
|
||||
if args.limit > 0:
|
||||
query += f" LIMIT {args.limit}"
|
||||
|
||||
materials = db.execute(query).fetchall()
|
||||
materials = [dict(m) for m in materials]
|
||||
|
||||
if not materials:
|
||||
print("[OK] 没有待下载的素材")
|
||||
db.close()
|
||||
return
|
||||
|
||||
print(f"[INFO] 待下载: {len(materials)} 条")
|
||||
|
||||
success = 0
|
||||
failed = 0
|
||||
|
||||
for i, material in enumerate(materials, 1):
|
||||
print(f"[{i}/{len(materials)}] 下载: {material['title'][:40]}")
|
||||
|
||||
# 标记为下载中
|
||||
db.execute("UPDATE materials SET download_status = 'downloading' WHERE id = ?", (material["id"],))
|
||||
db.commit()
|
||||
|
||||
result = download_one(material, config, logger)
|
||||
|
||||
if result and result.get("video"):
|
||||
db.execute("""
|
||||
UPDATE materials
|
||||
SET download_status = 'done', local_files = ?
|
||||
WHERE id = ?
|
||||
""", (json.dumps(result, ensure_ascii=False), material["id"]))
|
||||
db.commit()
|
||||
success += 1
|
||||
else:
|
||||
db.execute("UPDATE materials SET download_status = 'failed' WHERE id = ?", (material["id"],))
|
||||
db.commit()
|
||||
failed += 1
|
||||
|
||||
db.close()
|
||||
print(f"\n[DONE] 下载完成: 成功 {success}, 失败 {failed}, 共 {len(materials)} 条")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,34 @@
|
||||
#!/bin/bash
|
||||
# OpenCLI 安装脚本 — 新机器一键部署
|
||||
set -e
|
||||
echo "=== OpenCLI 安装脚本 ==="
|
||||
echo ""
|
||||
|
||||
NODE=$(which node 2>/dev/null || echo "/Users/chengzige/.workbuddy/binaries/node/versions/22.12.0/bin/node")
|
||||
if [ ! -x "$NODE" ]; then echo "❌ Node.js 未找到"; exit 1; fi
|
||||
echo " Node: $($NODE --version)"
|
||||
|
||||
echo "[2/4] 安装 OpenCLI..."
|
||||
npm install -g @jackwener/opencli 2>&1 | tail -1
|
||||
|
||||
echo "[3/4] 下载 Chrome 扩展..."
|
||||
EXT_DIR="$HOME/opencli-extension"
|
||||
if [ -f "$EXT_DIR/manifest.json" ]; then
|
||||
echo " 扩展已存在，跳过下载"
|
||||
else
|
||||
mkdir -p "$EXT_DIR"
|
||||
curl -L -o /tmp/opencli-ext.zip "https://github.com/jackwener/OpenCLI/releases/download/v1.7.12/opencli-extension-v1.0.5.zip"
|
||||
unzip -o /tmp/opencli-ext.zip -d "$EXT_DIR" 2>&1 | tail -1
|
||||
echo " ✅ 下载完成: $EXT_DIR"
|
||||
fi
|
||||
|
||||
echo "[4/4] 启动 daemon..."
|
||||
opencli daemon restart 2>&1 | head -1
|
||||
|
||||
echo ""
|
||||
echo "=== 完成 ==="
|
||||
echo "下一步："
|
||||
echo " 1. 打开 Chrome → chrome://extensions/"
|
||||
echo " 2. 开启开发者模式"
|
||||
echo " 3. 加载已解包扩展 → 选择 $EXT_DIR"
|
||||
echo " 4. 验证: opencli doctor"
|
||||
@@ -0,0 +1,11 @@
|
||||
# 口播素材系统 Python 依赖
|
||||
# 安装：~/.workbuddy/binaries/python/envs/agent-os/bin/pip install -r requirements.txt
|
||||
|
||||
# 必需（通常已随 AgentOS 安装）
|
||||
httpx>=0.24.0
|
||||
PyYAML>=6.0
|
||||
Pillow>=9.0
|
||||
|
||||
# 需要额外安装
|
||||
gradio>=4.0
|
||||
yt-dlp>=2024.0
|
||||
@@ -0,0 +1,74 @@
|
||||
-- ============================================================
|
||||
-- 口播素材系统数据库初始化
|
||||
-- 使用：sqlite3 data/database.db < schema.sql
|
||||
-- ============================================================
|
||||
|
||||
-- 开启 WAL 模式，提高并发读取
|
||||
PRAGMA journal_mode = WAL;
|
||||
PRAGMA foreign_keys = ON;
|
||||
|
||||
-- 素材元数据表
|
||||
CREATE TABLE IF NOT EXISTS materials (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
platform TEXT NOT NULL, -- 平台：xiaohongshu / douyin / bilibili
|
||||
original_id TEXT NOT NULL, -- 平台内唯一 ID
|
||||
url TEXT, -- 原始链接
|
||||
title TEXT, -- 标题
|
||||
description TEXT, -- 描述/正文
|
||||
author TEXT, -- 作者
|
||||
author_id TEXT, -- 作者 ID
|
||||
cover_url TEXT, -- 封面图 URL
|
||||
cover_local TEXT, -- 本地封面图路径
|
||||
music_name TEXT, -- BGM 名称
|
||||
tags_platform TEXT, -- 平台自带标签（逗号分隔）
|
||||
duration INTEGER, -- 时长（秒）
|
||||
like_count INTEGER DEFAULT 0, -- 点赞数
|
||||
collect_count INTEGER DEFAULT 0, -- 收藏数
|
||||
comment_count INTEGER DEFAULT 0, -- 评论数
|
||||
share_count INTEGER DEFAULT 0, -- 分享数
|
||||
raw_json TEXT, -- 原始 JSON 数据（完整备份）
|
||||
fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- 采集时间
|
||||
analyzed INTEGER DEFAULT 0, -- 0:未分析 1:已分析 2:分析失败
|
||||
download_status TEXT DEFAULT 'none', -- none / pending / downloading / done / failed
|
||||
local_files TEXT, -- JSON: {"video":"path","audio":"path"}
|
||||
notes TEXT -- 用户备注
|
||||
);
|
||||
|
||||
-- 平台+ID 联合唯一索引（去重）
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_platform_id ON materials(platform, original_id);
|
||||
|
||||
-- 查询常用索引
|
||||
CREATE INDEX IF NOT EXISTS idx_analyzed ON materials(analyzed);
|
||||
CREATE INDEX IF NOT EXISTS idx_download_status ON materials(download_status);
|
||||
CREATE INDEX IF NOT EXISTS idx_platform ON materials(platform);
|
||||
CREATE INDEX IF NOT EXISTS idx_fetched_at ON materials(fetched_at);
|
||||
|
||||
-- AI 分析结果表
|
||||
CREATE TABLE IF NOT EXISTS analysis (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
material_id INTEGER UNIQUE NOT NULL, -- 关联素材
|
||||
tags TEXT, -- AI 标签（逗号分隔）
|
||||
golden_quote TEXT, -- 可引用金句
|
||||
core_idea TEXT, -- 核心立意
|
||||
structure TEXT, -- 口播脚本结构
|
||||
emotion TEXT, -- 整体情绪
|
||||
worth_downloading TEXT, -- yes / no / maybe
|
||||
download_reason TEXT, -- 下载理由
|
||||
raw_model_output TEXT, -- 模型原始返回（调试）
|
||||
analyzed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (material_id) REFERENCES materials(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- 采集批次记录表
|
||||
CREATE TABLE IF NOT EXISTS collect_batches (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
platform TEXT NOT NULL,
|
||||
keyword TEXT NOT NULL,
|
||||
count INTEGER DEFAULT 0, -- 本次采集数量
|
||||
jsonl_file TEXT, -- JSONL 文件路径
|
||||
status TEXT DEFAULT 'pending', -- pending / running / done / failed
|
||||
started_at TIMESTAMP,
|
||||
finished_at TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_batch_status ON collect_batches(status);
|
||||
@@ -0,0 +1,84 @@
|
||||
"""
|
||||
AVE v2.0 脚本工厂 — 从素材生成导演脚本
|
||||
"""
|
||||
import os, sys, json, yaml, argparse, datetime, urllib.request, re
|
||||
from pathlib import Path
|
||||
|
||||
OUTPUT_DIR = Path(__file__).parent / "scripts_output"
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
|
||||
OMLX_URL = "http://localhost:8000/v1/chat/completions"
|
||||
|
||||
def ai(prompt, tokens=2000, temp=0.3):
|
||||
body = json.dumps({"model": "Qwen3-8B-MLX-4bit", "messages": [{"role":"user","content":prompt}], "max_tokens":tokens, "temperature":temp}).encode()
|
||||
req = urllib.request.Request(OMLX_URL, data=body, headers={"Authorization":"Bearer omlx","Content-Type":"application/json"}, method="POST")
|
||||
return json.loads(urllib.request.urlopen(req, timeout=300).read())["choices"][0]["message"]["content"]
|
||||
|
||||
def analyze(title, desc):
|
||||
r = ai(f"""分析以下内容，输出JSON：
|
||||
标题：{title}
|
||||
描述：{desc}
|
||||
输出：{{"core_topic":"一句话主题","target_audience":"目标人群","hook_type":"钩子类型","key_points":["点1","点2"],"emotional_tone":"情绪基调","suggested_style":"knowledge_lecture/bedtime_story/funny_talk/tech_review"}}""", 800, 0.3)
|
||||
m = re.search(r'\{.*\}', r, re.DOTALL)
|
||||
return json.loads(m.group()) if m else {"core_topic":title,"key_points":[]}
|
||||
|
||||
def segments(analysis, style):
|
||||
kp = analysis.get("key_points",[])
|
||||
pts = "\n".join([f"- {p}" for p in kp]) if kp else ""
|
||||
r = ai(f"""你是一个短视频脚本专家。生成分段口播脚本。
|
||||
|
||||
主题：{analysis.get("core_topic","")}
|
||||
要点：{pts}
|
||||
风格：{style}
|
||||
|
||||
字段：id(数字), text(口播文本), duration_sec(秒), voice_emotion(语气), camera(镜头), bgm_section(intro/main/climax/outro), material.search(英文搜索词), subtitles(true/false)
|
||||
|
||||
要求：4-6段，总60-90秒。首段5-8秒钩子，末段总结。
|
||||
|
||||
输出JSON数组：
|
||||
[{{"id":1,"text":"...","duration_sec":8,"voice_emotion":"开场悬念","camera":"slow_zoom_in","bgm_section":"intro","material":{{"search":"..."}},"subtitles":true}}]""", 2000, 0.3)
|
||||
m = re.search(r'\[.*?\]', r, re.DOTALL)
|
||||
segs = json.loads(m.group()) if m else []
|
||||
for i,s in enumerate(segs):
|
||||
s["id"]=i+1
|
||||
bgm=str(s.get("bgm_section","")).lower()
|
||||
for v in ["intro","main","climax","outro"]:
|
||||
if v in bgm: s["bgm_section"]=v; break
|
||||
else: s["bgm_section"]="main"
|
||||
return segs
|
||||
|
||||
def run(topic, style=None):
|
||||
a = analyze(topic, topic)
|
||||
s = style or a.get("suggested_style","knowledge_lecture")
|
||||
segs = segments(a, s) or [{"id":1,"text":topic,"duration_sec":30,"voice_emotion":"正常讲述","camera":"medium_shot","bgm_section":"main","material":{"search":topic},"subtitles":True}]
|
||||
pid = f"vid_{datetime.date.today().strftime('%Y%m%d')}_{int(datetime.datetime.now().timestamp())%10000:04d}"
|
||||
y = {"meta":{"project_id":pid,"style":s,"voice":{"provider":"volcano","voice_id":"my_voice","default_emotion":a.get("emotional_tone","专业沉稳")},"bgm":{"provider":"ace_step","genre":"cinematic","tempo":110,"mood":"inspirational","structure":[{"section":"intro"},{"section":"main"},{"section":"climax"},{"section":"outro"}]},"output":{"resolution":"1080x1920","fps":30},"avatar":{"image_url":"","provider":"wan2.2_s2v","default_position":"bottom_right"},"material":{"fallback":True}},"segments":segs}
|
||||
yp = OUTPUT_DIR/f"{pid}.yaml"; mp = OUTPUT_DIR/f"{pid}.md"
|
||||
with open(yp,"w",encoding="utf-8") as f: yaml.dump(y,f,allow_unicode=True,default_flow_style=False,sort_keys=False)
|
||||
md = f"""# {pid} — {a.get('core_topic','')}
|
||||
|
||||
> 由 AVE 脚本工厂自动生成 | {datetime.date.today().isoformat()}
|
||||
|
||||
## 来源信息
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 核心主题 | {a.get('core_topic','-')} |
|
||||
| 目标人群 | {a.get('target_audience','-')} |
|
||||
| 风格 | {s} | 钩子 | {a.get('hook_type','-')} | 情绪 | {a.get('emotional_tone','-')} |
|
||||
|
||||
## 关键要点
|
||||
{chr(10).join([f'- {p}' for p in a.get('key_points',[])]) if a.get('key_points') else '- （AI分析中）'}
|
||||
|
||||
## 生产备注
|
||||
- **语音**: 火山引擎 **BGM**: Ace Studio **素材**: Pexels搜索+本地兜底 **分辨率**: 1080×1920竖屏
|
||||
- **注意**: 脚本为AI生成，建议人工审核。配画关键词可能需要手动调整。
|
||||
"""
|
||||
with open(mp,"w",encoding="utf-8") as f: f.write(md)
|
||||
print(f"✅ {pid}: {len(segs)}段 | {yp} | {mp}")
|
||||
return yp, mp
|
||||
|
||||
if __name__=="__main__":
|
||||
p=argparse.ArgumentParser()
|
||||
p.add_argument("--topic", required=True); p.add_argument("--style")
|
||||
args=p.parse_args()
|
||||
run(args.topic, args.style)
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
# vid_20260504_8817 — 毛主席的语言艺术将失败转化为传奇
|
||||
|
||||
> 由 AVE 脚本工厂自动生成 | 2026-05-04
|
||||
|
||||
## 来源信息
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 核心主题 | 毛主席的语言艺术将失败转化为传奇 |
|
||||
| 目标人群 | 历史爱好者/文学研究者/毛泽东思想研究者 |
|
||||
| 风格 | knowledge_lecture | 钩子 | 故事性钩子 | 情绪 | 崇敬与赞叹 |
|
||||
|
||||
## 关键要点
|
||||
- 通过命名赋予历史事件新的意义
|
||||
- 语言重构改变了历史叙事的维度
|
||||
|
||||
## 生产备注
|
||||
- **语音**: 火山引擎 **BGM**: Ace Studio **素材**: Pexels搜索+本地兜底 **分辨率**: 1080×1920竖屏
|
||||
- **注意**: 脚本为AI生成，建议人工审核。配画关键词可能需要手动调整。
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
meta:
|
||||
project_id: vid_20260504_8817
|
||||
style: knowledge_lecture
|
||||
voice:
|
||||
provider: volcano
|
||||
voice_id: my_voice
|
||||
default_emotion: 崇敬与赞叹
|
||||
bgm:
|
||||
provider: ace_step
|
||||
genre: cinematic
|
||||
tempo: 110
|
||||
mood: inspirational
|
||||
structure:
|
||||
- section: intro
|
||||
- section: main
|
||||
- section: climax
|
||||
- section: outro
|
||||
output:
|
||||
resolution: 1080x1920
|
||||
fps: 30
|
||||
avatar:
|
||||
image_url: ''
|
||||
provider: wan2.2_s2v
|
||||
default_position: bottom_right
|
||||
material:
|
||||
fallback: true
|
||||
segments:
|
||||
- id: 1
|
||||
text: 当历史的尘埃落定，某些失败却因一句命名而成为传奇。比如‘井冈山’——这个原本只是地理坐标的名字，被赋予了‘革命摇篮’的象征意义。
|
||||
duration_sec: 8
|
||||
voice_emotion: 开场悬念
|
||||
camera: slow_zoom_in
|
||||
bgm_section: intro
|
||||
material:
|
||||
search: 命名赋予历史意义
|
||||
subtitles: true
|
||||
- id: 2
|
||||
text: 毛主席擅长用语言重构历史维度。他将‘长征’从单纯的军事撤退，转化为‘战略转移’的史诗叙事，让失败成为民族精神的转折点。
|
||||
duration_sec: 12
|
||||
voice_emotion: 理性分析
|
||||
camera: close_up
|
||||
bgm_section: main
|
||||
material:
|
||||
search: 语言重构历史叙事
|
||||
subtitles: true
|
||||
- id: 3
|
||||
text: 这种叙事重构不仅改变历史解释框架，更塑造了集体记忆。‘延安’从军事据点升华为‘精神圣地’，其背后是语言对历史事件的重新编码与价值赋予。
|
||||
duration_sec: 15
|
||||
voice_emotion: 深度解析
|
||||
camera: montage
|
||||
bgm_section: climax
|
||||
material:
|
||||
search: 历史叙事重构案例
|
||||
subtitles: true
|
||||
- id: 4
|
||||
text: 毛主席的语言艺术，本质是用精准的符号系统将失败转化为文化资本。这种转化让历史事件超越时间维度，成为民族精神的永恒坐标。
|
||||
duration_sec: 10
|
||||
voice_emotion: 总结升华
|
||||
camera: wide_angle
|
||||
bgm_section: outro
|
||||
material:
|
||||
search: 失败转化为传奇案例
|
||||
subtitles: true
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
# vid_20260505_em001 — 我爸给我打了20年电话，我从来没接过
|
||||
|
||||
> 由人工创作 + 参考今日头条情感爆款结构 | 2026-05-05
|
||||
|
||||
---
|
||||
|
||||
## 一、来源信息
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 故事类型 | 亲情/父子/子欲养而亲不待 |
|
||||
| 参考对象 | 今日头条情感类爆款《我妈骂了我爸四十年》 |
|
||||
| 风格定位 | bedtime_story（深夜情感故事） |
|
||||
| 预计时长 | 约 100 秒（1分40秒） |
|
||||
| 核心情绪 | 遗憾、愧疚、珍惜 |
|
||||
|
||||
## 二、结构拆解
|
||||
|
||||
| 段落 | 时长 | 内容 | 作用 |
|
||||
|------|------|------|------|
|
||||
| ① 钩子 | 12s | 最后一次通话，三个未接来电 | 悬念开场，一句话抓住 |
|
||||
| ② 背景 | 18s | 父亲是农民工，30年绑钢筋 | 建立人物形象和时代感 |
|
||||
| ③ 铺垫 | 14s | 二十年如一日不变的电话 | 积累情感，制造"我"的不懂事 |
|
||||
| ④ 高潮积累 | 22s | 北京实习，父亲寄钱 | 父爱的极致表现——他其实知道 |
|
||||
| ⑤ 高潮爆发 | 20s | 三个未接来电，再没有第四通 | 全篇最高泪点 |
|
||||
| ⑥ 余韵 | 15s | 梦中重逢的对话 | 给观众消化时间 |
|
||||
| ⑦ 金句收尾 | 10s | 别让那句"吃饭了吗"变成等不到的电话 | 总结升华，带传播力 |
|
||||
|
||||
## 三、口播脚本与标注
|
||||
|
||||
详见 YAML 文件 `vid_20260505_em001.yaml`，每段已标注：
|
||||
- `voice_emotion` — 语气/情绪提示
|
||||
- `camera` — 镜头语言建议
|
||||
- `bgm_section` — 配乐段落
|
||||
- `material.search` — 配画关键词
|
||||
|
||||
## 四、可复用的口播技法
|
||||
|
||||
| 技法 | 在本篇中的应用 |
|
||||
|------|---------------|
|
||||
| 具体日期 | "2018年3月12号" → 增加真实感 |
|
||||
| 数字量化 | "三个电话"、"二十年"、"三千五留五百" |
|
||||
| 场景细节 | "绑钢筋"、"下雨没活干"、"喝药酒的细节" |
|
||||
| 对话还原 | "缺钱跟爸说，爸有" |
|
||||
| 反转推进 | 以为是啰嗦，结果是最后的通话 |
|
||||
| 梦境穿插 | "梦里我坐在老家门口" |
|
||||
| 金句收尾 | 最后一句是总结也是转发点 |
|
||||
|
||||
## 五、生产备注
|
||||
|
||||
- **语音**: 火山引擎，建议用中年男声，语气稳重略低沉
|
||||
- **BGM**: 钢琴独奏或弦乐，80bpm 慢速
|
||||
- **画面**: 推荐黑白/暖色调滤镜，配合回忆感
|
||||
- **字幕**: 逐句字幕（首段+结尾金句建议关闭字幕增强冲击力）
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
# ============================================
|
||||
# AVE v2.0 导演脚本
|
||||
# 我爸给我打了20年电话，我从来没接过
|
||||
# ============================================
|
||||
# 这是一个关于中国式父子关系的真实情感故事。
|
||||
|
||||
meta:
|
||||
project_id: "vid_20260505_em001"
|
||||
style: "bedtime_story" # knowledge_lecture | bedtime_story | funny_talk | tech_review
|
||||
|
||||
voice:
|
||||
provider: "volcano"
|
||||
voice_id: "my_voice"
|
||||
default_emotion: "深情回忆，略带克制"
|
||||
|
||||
bgm:
|
||||
provider: "ace_step"
|
||||
genre: "cinematic"
|
||||
tempo: 80
|
||||
mood: "nostalgic"
|
||||
structure:
|
||||
- section: intro # 0-20s
|
||||
- section: main # 20-120s
|
||||
- section: climax # 120-220s
|
||||
- section: outro # 220-280s
|
||||
|
||||
output:
|
||||
resolution: "1080x1920"
|
||||
fps: 30
|
||||
|
||||
avatar:
|
||||
image_url: ""
|
||||
provider: "wan2.2_s2v"
|
||||
default_position: "bottom_right"
|
||||
|
||||
material:
|
||||
fallback: true
|
||||
|
||||
# ——— 分段指令 ———
|
||||
segments:
|
||||
- id: 1
|
||||
text: "我跟我爸的最后一次通话，是2018年3月12号。那天他打了三个电话，我一个都没接。后来我再也没有机会接了。"
|
||||
duration_sec: 12
|
||||
voice_emotion: "平静开场，语速缓慢"
|
||||
camera: "slow_zoom_in"
|
||||
bgm_section: "intro"
|
||||
material:
|
||||
search: "old father working construction site, cinematic"
|
||||
subtitles: true
|
||||
|
||||
- id: 2
|
||||
text: "我爸是个农民工，在工地绑钢筋，一绑就是三十年。我从小最怕两件事：一个是开学，因为要交学费；一个是下雨，因为一下雨他就没活干，没活干他就喝酒，一喝酒他就会打电话回来。"
|
||||
duration_sec: 18
|
||||
voice_emotion: "正常叙述，略带苦涩"
|
||||
camera: "medium_shot"
|
||||
bgm_section: "main"
|
||||
material:
|
||||
search: "construction worker in rain, Chinese countryside"
|
||||
subtitles: true
|
||||
|
||||
- id: 3
|
||||
text: "他的电话永远是那几句：吃饭了吗？天冷了多穿点。钱够不够花？翻来覆去，二十年如一日。我嫌他啰嗦。每次看到屏幕上亮起他的名字，我就按掉。"
|
||||
duration_sec: 14
|
||||
voice_emotion: "带一丝愧疚"
|
||||
camera: "close_up"
|
||||
bgm_section: "main"
|
||||
material:
|
||||
search: "rejected phone call, lonely father"
|
||||
subtitles: true
|
||||
|
||||
- id: 4
|
||||
text: "2012年我在北京实习，他打电话问我过得好不好。我说挺好的，公司管吃管住。他说你别骗我。我说真的挺好。他沉默了几秒，说：缺钱跟爸说，爸有。我知道他没有。他一个月工资三千五，自己留五百，剩下三千全打到我卡上。我从来没说过谢谢。"
|
||||
duration_sec: 22
|
||||
voice_emotion: "逐渐哽咽，真情流露"
|
||||
camera: "slow_pan_right"
|
||||
bgm_section: "climax"
|
||||
material:
|
||||
search: "father sends money to son, love letter"
|
||||
subtitles: true
|
||||
|
||||
- id: 5
|
||||
text: "2018年3月12号。他打了三个电话。第一个我在开会，按掉了。第二个我在跟朋友吃饭，没接。第三个晚上九点，我想着明天回吧。然后就再也没有第四通了。那天晚上他从脚手架上摔了下来。我妈说他的手机最后停留在我的通话记录上。三个未接来电。全是打给我的。"
|
||||
duration_sec: 20
|
||||
voice_emotion: "压抑悲痛，声音微颤"
|
||||
camera: "hand_held"
|
||||
bgm_section: "climax"
|
||||
material:
|
||||
search: "phone with missed calls, tragedy at construction site"
|
||||
subtitles: true
|
||||
|
||||
- id: 6
|
||||
text: "我后来经常做一个梦。梦里我坐在老家门口，远远看见他走过来。他走到跟前，笑着问我：吃饭了吗？我在梦里说：吃了，爸。你呢？然后我就醒了。醒来发现枕头是湿的。"
|
||||
duration_sec: 15
|
||||
voice_emotion: "深情追忆，轻声"
|
||||
camera: "top_down"
|
||||
bgm_section: "outro"
|
||||
material:
|
||||
search: "wet pillow morning, father son reunion dream"
|
||||
subtitles: true
|
||||
|
||||
- id: 7
|
||||
text: "有些电话，你永远等不到下一个了。别让你爸妈的那句吃饭了吗，变成你余生都接不到的那通电话。"
|
||||
duration_sec: 10
|
||||
voice_emotion: "坚定有力，字字清晰"
|
||||
camera: "slow_zoom_in"
|
||||
bgm_section: "outro"
|
||||
material:
|
||||
search: "empty chair, sunset, father and son silhouette"
|
||||
subtitles: false
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
# 豆包持久化浏览器登录设置
|
||||
# 运行此脚本后，在弹出的浏览器中登录豆包，然后关掉浏览器即可
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
echo "=== 豆包持久化浏览器登录设置 ==="
|
||||
echo ""
|
||||
echo "即将打开浏览器窗口..."
|
||||
echo "请在浏览器中登录你的豆包账号"
|
||||
echo "登录完成后，关掉浏览器窗口即可"
|
||||
echo "以后系统将自动复用本次登录态"
|
||||
echo ""
|
||||
|
||||
/Users/chengzige/.workbuddy/binaries/python/envs/agent-os/bin/python3 -c "
|
||||
from doubao_driver import DoubaoDriver
|
||||
import time
|
||||
|
||||
d = DoubaoDriver()
|
||||
print('启动持久化浏览器...')
|
||||
d._start_persistent()
|
||||
d.page.goto('https://www.doubao.com/chat', timeout=30000)
|
||||
time.sleep(2)
|
||||
|
||||
print('浏览器已打开，请登录豆包...')
|
||||
print('关掉浏览器窗口后继续...')
|
||||
|
||||
# 等待浏览器关闭（context 断开连接）
|
||||
try:
|
||||
while True:
|
||||
time.sleep(2)
|
||||
try:
|
||||
_ = d.page.title
|
||||
except:
|
||||
break
|
||||
except:
|
||||
pass
|
||||
|
||||
print('浏览器已关闭，session 已保存')
|
||||
print('位置: ~/.cloakbrowser/doubao_profile/')
|
||||
"
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/bin/bash
|
||||
# 豆包驱动测试脚本
|
||||
# 执行前请确保已在 Chrome 中登录 doubao.com
|
||||
set -e
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
echo "=== 豆包驱动测试 ==="
|
||||
echo ""
|
||||
echo "请先确认：Chrome 已登录 doubao.com ✅"
|
||||
read -p "按回车继续..."
|
||||
|
||||
echo ""
|
||||
echo "测试 1: 打开豆包并检查登录状态..."
|
||||
/Users/chengzige/.workbuddy/binaries/python/envs/agent-os/bin/python3 -c "
|
||||
from doubao_driver import DoubaoDriver
|
||||
d = DoubaoDriver()
|
||||
d._start_browser()
|
||||
d.page.goto('https://www.doubao.com/chat', timeout=30000)
|
||||
ok = d._ensure_logged_in()
|
||||
print(f'登录状态: {\"✅ 已登录\" if ok else \"❌ 未登录\"}')
|
||||
if ok:
|
||||
inp = d._find_input()
|
||||
print(f'输入框: {\"✅ 找到\" if inp else \"❌ 未找到\"}')
|
||||
if inp:
|
||||
print(f' placeholder: {inp.get_attribute(\"placeholder\")}')
|
||||
d._close_browser()
|
||||
" 2>&1
|
||||
|
||||
echo ""
|
||||
echo "完成"
|
||||
@@ -0,0 +1,154 @@
|
||||
"""
|
||||
口播素材系统 - 通用工具函数
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import sqlite3
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# 项目根目录（本文件所在目录的父级）
|
||||
PROJECT_ROOT = Path(__file__).parent.resolve()
|
||||
|
||||
|
||||
def load_config() -> dict:
|
||||
"""加载 config.yaml 配置文件"""
|
||||
import yaml
|
||||
config_path = PROJECT_ROOT / "config.yaml"
|
||||
if not config_path.exists():
|
||||
raise FileNotFoundError(f"配置文件不存在: {config_path}")
|
||||
with open(config_path, "r", encoding="utf-8") as f:
|
||||
config = yaml.safe_load(f)
|
||||
# 展开路径中的 ~
|
||||
for key in ("media_dir", "audio_dir", "raw_dir", "db_path", "logs_dir"):
|
||||
if key in config.get("storage", {}):
|
||||
config["storage"][key] = os.path.expanduser(config["storage"][key])
|
||||
return config
|
||||
|
||||
|
||||
def get_db_path(config: dict = None) -> str:
|
||||
"""获取数据库文件路径"""
|
||||
if config is None:
|
||||
config = load_config()
|
||||
db_path = config["storage"]["db_path"]
|
||||
if not os.path.isabs(db_path):
|
||||
db_path = str(PROJECT_ROOT / db_path)
|
||||
return db_path
|
||||
|
||||
|
||||
def get_db(db_path: str = None) -> sqlite3.Connection:
|
||||
"""获取数据库连接（开启 WAL 模式）"""
|
||||
if db_path is None:
|
||||
db_path = get_db_path()
|
||||
os.makedirs(os.path.dirname(db_path), exist_ok=True)
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA journal_mode = WAL")
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
return conn
|
||||
|
||||
|
||||
def init_db(db_path: str = None):
|
||||
"""初始化数据库（执行 schema.sql）"""
|
||||
if db_path is None:
|
||||
db_path = get_db_path()
|
||||
schema_path = PROJECT_ROOT / "schema.sql"
|
||||
conn = get_db(db_path)
|
||||
with open(schema_path, "r", encoding="utf-8") as f:
|
||||
conn.executescript(f.read())
|
||||
conn.close()
|
||||
return db_path
|
||||
|
||||
|
||||
def setup_logger(name: str, log_file: str = None, config: dict = None) -> logging.Logger:
|
||||
"""配置日志"""
|
||||
if config is None:
|
||||
config = load_config()
|
||||
if log_file is None:
|
||||
logs_dir = config["storage"].get("logs_dir", "logs")
|
||||
if not os.path.isabs(logs_dir):
|
||||
logs_dir = str(PROJECT_ROOT / logs_dir)
|
||||
os.makedirs(logs_dir, exist_ok=True)
|
||||
log_file = os.path.join(logs_dir, f"{name}_{datetime.now():%Y%m%d}.log")
|
||||
|
||||
logger = logging.getLogger(name)
|
||||
logger.setLevel(logging.DEBUG)
|
||||
# 避免重复添加 handler
|
||||
if logger.handlers:
|
||||
return logger
|
||||
|
||||
fmt = logging.Formatter("%(asctime)s [%(levelname)s] %(message)s", datefmt="%H:%M:%S")
|
||||
fh = logging.FileHandler(log_file, encoding="utf-8")
|
||||
fh.setFormatter(fmt)
|
||||
fh.setLevel(logging.DEBUG)
|
||||
sh = logging.StreamHandler(sys.stdout)
|
||||
sh.setFormatter(fmt)
|
||||
sh.setLevel(logging.INFO)
|
||||
logger.addHandler(fh)
|
||||
logger.addHandler(sh)
|
||||
return logger
|
||||
|
||||
|
||||
def parse_jsonl(file_path: str) -> list[dict]:
|
||||
"""逐行读取 JSONL 文件，返回字典列表"""
|
||||
records = []
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
for line_no, line in enumerate(f, 1):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
records.append(json.loads(line))
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"[WARN] JSONL 第 {line_no} 行解析失败: {e}")
|
||||
return records
|
||||
|
||||
|
||||
def save_jsonl(records: list[dict], file_path: str, append: bool = True):
|
||||
"""写入 JSONL 文件"""
|
||||
os.makedirs(os.path.dirname(file_path), exist_ok=True)
|
||||
mode = "a" if append else "w"
|
||||
with open(file_path, mode, encoding="utf-8") as f:
|
||||
for rec in records:
|
||||
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
def truncate_text(text: str, max_len: int = 20, ellipsis: str = "") -> str:
|
||||
"""截断文本用于文件名"""
|
||||
if not text:
|
||||
return ""
|
||||
text = text.replace("/", "_").replace("\\", "_").replace(" ", "_")
|
||||
# 去除不合法文件名字符
|
||||
import re
|
||||
text = re.sub(r'[<>:"|?*]', '', text)
|
||||
if len(text) > max_len:
|
||||
return text[:max_len] + ellipsis
|
||||
return text
|
||||
|
||||
|
||||
def check_omlx(config: dict = None) -> bool:
|
||||
"""检查 oMLX 是否可用"""
|
||||
if config is None:
|
||||
config = load_config()
|
||||
import httpx
|
||||
try:
|
||||
resp = httpx.get(
|
||||
f"{config['llm']['base_url']}/models",
|
||||
headers={"Authorization": f"Bearer {config['llm']['api_key']}"},
|
||||
timeout=5
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
models = [m["id"] for m in resp.json().get("data", [])]
|
||||
target = config["llm"]["model"]
|
||||
if target in models:
|
||||
print(f"[OK] oMLX 可用，模型: {target}")
|
||||
return True
|
||||
else:
|
||||
print(f"[WARN] oMLX 可用但缺少模型 {target}，已有: {models}")
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f"[WARN] oMLX 不可用: {e}")
|
||||
return False
|
||||
Reference in New Issue
Block a user