feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程，直接发 HTTP
移植自 mac-agent-os 的 mediacrawler_adapter：不起子进程、不开页面，用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin，所以 ingest 那条链路一个字不用改。
**目前能用的（真环境实测，非推断）**：
profile/other : 200, 7075 字节 —— 博主主页指标（粉丝/获赞/作品数/昵称）
aweme/detail : 200, 45425 字节 —— 单条作品详情（含点赞/评论/收藏/分享）
**目前不能用的：作品列表 `aweme/post`。** 两个互相独立的原因：
1. 这个接口被抖音单独升级成了真校验：不带 x-tt-argus 回 403「Uifid Not Found」，
带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过，「真校验」过不了。
同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的，
挑中的恰好是「批量拉作品列表」这个最敏感的动作。
2. 改走页面截获也不行：CDP 浏览器打开博主主页会落到「验证码中间页」（当天大量探测的
代价，过几小时要重测）。
所以现在的边界是：**已知作品的指标刷新能做，自动发现新作品做不了**。
**排查中控住变量后得到的两条事实**（都写进注释了）：
· `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
UA+client hints+Cookie 时是 200 但仅 121 字节的空壳，补上这三个头变 7074 字节。
（我先前猜的 sec-ch-ua 不是关键。）
· 因此 UA 与 client hints 必须**成套地取自同一个浏览器**，所以 BrowserIdentity 一次
从 CDP 取齐 cookie + UA + hints，而不是各自写死。
「200 + 空 body 必须当场报错」也是刻意写死的：放过去它会在下游变成「这个博主没作品」，
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的，还被翻译成「账号被封」。
测试 +11：cookie 解析、请求头成套性（含 uifid 缺失/回退）、产物键名与 store 对齐、
以及 _get 的三条失败路径（空 body / 403 带网关原话 / 正常返回）。
This commit is contained in:
@@ -0,0 +1,47 @@
|
||||
"""
|
||||
AVE 配置加载器
|
||||
优先级: local.yaml > config.yaml > 默认值
|
||||
"""
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
SYNC_ROOT = Path(__file__).resolve().parents[4] # agent-sync/
|
||||
LOCAL_ROOT = SYNC_ROOT.parent / "agent-local" # agent-local/
|
||||
|
||||
|
||||
def load_config() -> dict[str, Any]:
|
||||
"""加载配置，local.yaml 覆盖 config.yaml 的对应字段"""
|
||||
config: dict[str, Any] = {}
|
||||
|
||||
# 1. 模板配置
|
||||
template_path = SYNC_ROOT / "05_tools" / "09_ave" / "config.yaml"
|
||||
if template_path.exists():
|
||||
with open(template_path) as f:
|
||||
config.update(yaml.safe_load(f) or {})
|
||||
|
||||
# 2. 本地配置（覆盖）
|
||||
local_path = LOCAL_ROOT / "tools" / "ave" / "config" / "local.yaml"
|
||||
if local_path.exists():
|
||||
with open(local_path) as f:
|
||||
local = yaml.safe_load(f) or {}
|
||||
_deep_merge(config, local)
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def _deep_merge(base: dict, override: dict) -> None:
|
||||
"""递归合并字典"""
|
||||
for k, v in override.items():
|
||||
if k in base and isinstance(base[k], dict) and isinstance(v, dict):
|
||||
_deep_merge(base[k], v)
|
||||
else:
|
||||
base[k] = v
|
||||
|
||||
|
||||
def get_api_config(service: str) -> dict:
|
||||
"""获取指定服务的 API 配置"""
|
||||
cfg = load_config()
|
||||
return cfg.get(service, {})
|
||||
@@ -0,0 +1,213 @@
|
||||
"""
|
||||
AVE 成本追踪器
|
||||
|
||||
记录每次 API 调用的费用估算, 防止不知不觉扣费过多。
|
||||
|
||||
用法:
|
||||
from lib.cost_tracker import CostTracker
|
||||
tracker = CostTracker()
|
||||
tracker.log("OmniHuman", duration=4.04, resolution="480P")
|
||||
tracker.summary()
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# ── 已知价格 (元/秒) ───────────────────────────────────────
|
||||
# 来源: 火山引擎即梦AI官方定价 (截至2026-05-09)
|
||||
PRICING = {
|
||||
# OmniHuman 1.5
|
||||
"OmniHuman_480P": 0.5, # ¥0.5/秒
|
||||
"OmniHuman_720P": 0.9, # ¥0.9/秒
|
||||
# Wan2.2 (备用)
|
||||
"Wan2.2_480P": 0.5,
|
||||
"Wan2.2_720P": 0.9,
|
||||
# DreamActor M1 (估算, 与OmniHuman同级)
|
||||
"DreamActor_480P": 0.5,
|
||||
"DreamActor_720P": 0.9,
|
||||
# CosyVoice TTS (阿里云百炼, 按字符计费约 ¥0.002/字符)
|
||||
"CosyVoice_per_char": 0.002,
|
||||
# Pexels (免费)
|
||||
"Pexels": 0,
|
||||
# Kling LipSync (fal.ai, $0.014/5s ≈ ¥0.10/s)
|
||||
"KlingLipSync_per_s": 0.10, # ¥0.10/秒 ≈ $0.014/5s
|
||||
}
|
||||
|
||||
# ── 日志路径 ───────────────────────────────────────────────
|
||||
LOG_DIR = Path(os.environ.get("AVE_LOCAL_DIR",
|
||||
str(Path.home() / "workbuddy-agent-os/agent-local"))) / "tools" / "ave" / "logs"
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
LOG_FILE = LOG_DIR / "cost_log.jsonl"
|
||||
|
||||
|
||||
class CostTracker:
|
||||
"""API 成本追踪器"""
|
||||
|
||||
def __init__(self):
|
||||
self._session_log: list[dict] = []
|
||||
|
||||
def log(self, service: str, duration: float = 0, resolution: str = "480P",
|
||||
chars: int = 0, status: str = "success", note: str = ""):
|
||||
"""
|
||||
记录一次 API 调用
|
||||
|
||||
参数:
|
||||
service: API 服务名 (OmniHuman, DreamActor, CosyVoice, Pexels)
|
||||
duration: 视频/音频时长(秒)
|
||||
resolution: 分辨率 (480P/720P)
|
||||
chars: CosyVoice 字符数
|
||||
status: success/failed
|
||||
note: 备注
|
||||
"""
|
||||
# 计算费用
|
||||
if service == "CosyVoice":
|
||||
unit_price = PRICING.get("CosyVoice_per_char", 0.002)
|
||||
cost = round(chars * unit_price, 4)
|
||||
elif service == "KlingLipSync":
|
||||
unit_price = PRICING.get("KlingLipSync_per_s", 0.10)
|
||||
cost = round(duration * unit_price, 4) if status == "success" else 0.0
|
||||
elif service == "Pexels":
|
||||
cost = 0.0
|
||||
else:
|
||||
key = f"{service}_{resolution}"
|
||||
unit_price = PRICING.get(key, 0.5)
|
||||
cost = round(duration * unit_price, 4) if status == "success" else 0.0
|
||||
|
||||
record = {
|
||||
"time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"service": service,
|
||||
"duration_s": duration,
|
||||
"resolution": resolution,
|
||||
"chars": chars,
|
||||
"status": status,
|
||||
"cost_yuan": cost,
|
||||
"note": note,
|
||||
}
|
||||
self._session_log.append(record)
|
||||
|
||||
# 追加到日志文件
|
||||
with open(LOG_FILE, "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
|
||||
# 同步写入 Dashboard DB (如果当前有 production)
|
||||
if _current_production_id > 0:
|
||||
try:
|
||||
from lib.dashboard import log_cost as _dash_log_cost
|
||||
_dash_log_cost(
|
||||
production_id=_current_production_id,
|
||||
service=service,
|
||||
step_name=note.split(" ")[0] if " " in note else "",
|
||||
duration_s=duration,
|
||||
resolution=resolution,
|
||||
chars=chars,
|
||||
cost_yuan=cost,
|
||||
status=status,
|
||||
note=note,
|
||||
)
|
||||
except ImportError:
|
||||
pass # dashboard 模块不可用时不报错
|
||||
|
||||
# 实时输出
|
||||
if cost > 0:
|
||||
print(f" 💰 {service}: {duration:.1f}s × ¥{unit_price}/s = ¥{cost:.2f}")
|
||||
elif status == "failed":
|
||||
print(f" 💰 {service}: 失败, 不计费")
|
||||
else:
|
||||
print(f" 💰 {service}: 免费")
|
||||
|
||||
return cost
|
||||
|
||||
def summary(self) -> dict:
|
||||
"""输出本次会话的费用汇总"""
|
||||
total = sum(r["cost_yuan"] for r in self._session_log)
|
||||
success_calls = sum(1 for r in self._session_log if r["status"] == "success")
|
||||
failed_calls = sum(1 for r in self._session_log if r["status"] == "failed")
|
||||
|
||||
print("\n" + "=" * 40)
|
||||
print("💰 本次会话费用汇总")
|
||||
print("=" * 40)
|
||||
print(f" 成功调用: {success_calls} 次")
|
||||
print(f" 失败调用: {failed_calls} 次")
|
||||
print(f" 总费用: ¥{total:.2f}")
|
||||
print(f" 账户余额: 未知 (请登录火山引擎控制台查看)")
|
||||
print("=" * 40)
|
||||
|
||||
# 分服务统计
|
||||
by_service = {}
|
||||
for r in self._session_log:
|
||||
s = r["service"]
|
||||
if s not in by_service:
|
||||
by_service[s] = {"count": 0, "total_s": 0, "cost": 0.0}
|
||||
by_service[s]["count"] += 1
|
||||
by_service[s]["total_s"] += r["duration_s"] if r["status"] == "success" else 0
|
||||
by_service[s]["cost"] += r["cost_yuan"]
|
||||
|
||||
if by_service:
|
||||
print("\n 按服务:")
|
||||
for s, info in sorted(by_service.items()):
|
||||
print(f" {s:15s} {info['count']}次, {info['total_s']:.1f}s, ¥{info['cost']:.2f}")
|
||||
|
||||
return {"total": total, "calls": success_calls + failed_calls}
|
||||
|
||||
def get_session_log(self) -> list[dict]:
|
||||
return self._session_log
|
||||
|
||||
def _total_cost(self) -> float:
|
||||
"""当前会话累计费用"""
|
||||
return sum(r["cost_yuan"] for r in self._session_log)
|
||||
|
||||
|
||||
# ── Dashboard 集成 ─────────────────────────────────────────
|
||||
# 当前 production_id, 由 video_factory / main.py 设置
|
||||
# CostTracker.log() 检测到该值 > 0 时自动写 dashboard DB
|
||||
_current_production_id: int = 0
|
||||
|
||||
|
||||
def set_current_production_id(pid: int):
|
||||
"""设置当前 production ID, 后续 cost log 自动关联"""
|
||||
global _current_production_id
|
||||
_current_production_id = pid
|
||||
|
||||
|
||||
def get_current_production_id() -> int:
|
||||
return _current_production_id
|
||||
|
||||
|
||||
# ── 全局单例 (在 main.py 中使用) ────────────────────────────
|
||||
_tracker: CostTracker | None = None
|
||||
|
||||
|
||||
def get_tracker() -> CostTracker:
|
||||
global _tracker
|
||||
if _tracker is None:
|
||||
_tracker = CostTracker()
|
||||
return _tracker
|
||||
|
||||
|
||||
def print_history(days: int = 7):
|
||||
"""打印最近 N 天的费用历史"""
|
||||
if not LOG_FILE.exists():
|
||||
print("暂无费用记录")
|
||||
return
|
||||
|
||||
from datetime import timedelta
|
||||
cutoff = datetime.now() - timedelta(days=days)
|
||||
total = 0.0
|
||||
count = 0
|
||||
with open(LOG_FILE, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
r = json.loads(line)
|
||||
t = datetime.strptime(r["time"], "%Y-%m-%d %H:%M:%S")
|
||||
if t >= cutoff:
|
||||
total += r["cost_yuan"]
|
||||
count += 1
|
||||
print(f" [{r['time']}] {r['service']:15s} {r['duration_s']:5.1f}s ¥{r['cost_yuan']:.2f} {r['note']}")
|
||||
except (json.JSONDecodeError, KeyError, ValueError):
|
||||
pass
|
||||
|
||||
print(f"\n 近 {days} 天: {count} 次调用, 总费用 ¥{total:.2f}")
|
||||
@@ -0,0 +1,498 @@
|
||||
from __future__ import annotations
|
||||
|
||||
"""
|
||||
AVE dashboard — 生产监控数据层
|
||||
|
||||
职责:
|
||||
SQLite schema 初始化 + 埋点封装
|
||||
CLI 自动写 DB, FastAPI 后端读 DB
|
||||
|
||||
设计原则:
|
||||
- 所有操作 try/except 包裹, 失败不影响 CLI 主线
|
||||
- DB 路径: agent-local/tools/ave/data/ave.db
|
||||
- 连接池: 每次调用创建新连接 (CLI 场景, 无并发问题)
|
||||
|
||||
用法:
|
||||
from lib.dashboard import init_db, log_production, log_step, log_asset
|
||||
|
||||
init_db()
|
||||
pid = log_production(strategy="oral", script_name="test", config={"key": "val"})
|
||||
log_step(pid, "tts", "completed", cost=0.3, detail="85字")
|
||||
log_step(pid, "compose", "completed", cost=0.0, detail="2 clips")
|
||||
log_asset(pid, "generated_clip", "/tmp/out.mp4", tags=["hermit", "kling"])
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
# ── DB 路径 ───────────────────────────────────────────────
|
||||
LOCAL_ROOT = Path(os.environ.get("AVE_LOCAL_DIR",
|
||||
str(Path.home() / "workbuddy-agent-os/agent-local")))
|
||||
DB_DIR = LOCAL_ROOT / "tools" / "ave" / "data"
|
||||
DB_DIR.mkdir(parents=True, exist_ok=True)
|
||||
DB_PATH = DB_DIR / "ave.db"
|
||||
|
||||
# ── Schema ────────────────────────────────────────────────
|
||||
# 从 PLANS/DASHBOARD_DESIGN.md 导出, 6 表
|
||||
SCHEMA_SQL = """
|
||||
CREATE TABLE IF NOT EXISTS productions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
strategy TEXT NOT NULL,
|
||||
status TEXT DEFAULT 'running',
|
||||
script_path TEXT,
|
||||
script_name TEXT,
|
||||
output_path TEXT,
|
||||
duration_sec REAL,
|
||||
total_cost REAL DEFAULT 0.0,
|
||||
config_json TEXT,
|
||||
error_message TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now','localtime')),
|
||||
completed_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS production_steps (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
production_id INTEGER NOT NULL REFERENCES productions(id),
|
||||
step_name TEXT NOT NULL,
|
||||
status TEXT DEFAULT 'pending',
|
||||
started_at TEXT,
|
||||
ended_at TEXT,
|
||||
duration_sec REAL,
|
||||
cost REAL DEFAULT 0.0,
|
||||
detail TEXT,
|
||||
error_message TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS assets (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
asset_type TEXT NOT NULL,
|
||||
name TEXT,
|
||||
file_path TEXT,
|
||||
file_size INTEGER,
|
||||
hash TEXT,
|
||||
source TEXT,
|
||||
meta_json TEXT,
|
||||
production_id INTEGER,
|
||||
created_at TEXT DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS asset_tags (
|
||||
asset_id INTEGER NOT NULL REFERENCES assets(id),
|
||||
tag TEXT NOT NULL,
|
||||
PRIMARY KEY (asset_id, tag)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS production_assets (
|
||||
production_id INTEGER NOT NULL REFERENCES productions(id),
|
||||
asset_id INTEGER NOT NULL REFERENCES assets(id),
|
||||
role TEXT,
|
||||
PRIMARY KEY (production_id, asset_id)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS cost_log (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
production_id INTEGER REFERENCES productions(id),
|
||||
step_name TEXT,
|
||||
service TEXT NOT NULL,
|
||||
duration_s REAL DEFAULT 0,
|
||||
resolution TEXT,
|
||||
chars INTEGER DEFAULT 0,
|
||||
cost_yuan REAL DEFAULT 0,
|
||||
status TEXT DEFAULT 'success',
|
||||
note TEXT,
|
||||
logged_at TEXT DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# 初始化
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
def init_db() -> bool:
|
||||
"""初始化 DB schema, 不存在则创建"""
|
||||
try:
|
||||
conn = sqlite3.connect(str(DB_PATH))
|
||||
conn.executescript(SCHEMA_SQL)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" [dashboard] DB 初始化失败: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def _get_conn() -> sqlite3.Connection | None:
|
||||
"""获取 DB 连接 (带 try/except)"""
|
||||
try:
|
||||
conn = sqlite3.connect(str(DB_PATH))
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
except Exception as e:
|
||||
print(f" [dashboard] DB 连接失败: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# 埋点函数
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
def log_production(
|
||||
strategy: str,
|
||||
script_path: str = "",
|
||||
script_name: str = "",
|
||||
config: dict | None = None,
|
||||
) -> int:
|
||||
"""
|
||||
记录一条 production 开始
|
||||
|
||||
返回:
|
||||
production_id (失败返回 -1)
|
||||
"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return -1
|
||||
try:
|
||||
cur = conn.execute(
|
||||
"""INSERT INTO productions (strategy, status, script_path, script_name, config_json)
|
||||
VALUES (?, 'running', ?, ?, ?)""",
|
||||
(strategy, script_path, script_name, json.dumps(config or {}, ensure_ascii=False)),
|
||||
)
|
||||
conn.commit()
|
||||
pid = cur.lastrowid
|
||||
return pid
|
||||
except Exception as e:
|
||||
print(f" [dashboard] log_production 失败: {e}")
|
||||
return -1
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def complete_production(
|
||||
production_id: int,
|
||||
status: str = "completed",
|
||||
output_path: str = "",
|
||||
duration_sec: float = 0.0,
|
||||
total_cost: float = 0.0,
|
||||
error_message: str = "",
|
||||
):
|
||||
"""标记 production 完成"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return
|
||||
try:
|
||||
conn.execute(
|
||||
"""UPDATE productions SET status=?, output_path=?, duration_sec=?,
|
||||
total_cost=?, error_message=?, completed_at=datetime('now','localtime')
|
||||
WHERE id=?""",
|
||||
(status, output_path, duration_sec, total_cost, error_message, production_id),
|
||||
)
|
||||
conn.commit()
|
||||
except Exception as e:
|
||||
print(f" [dashboard] complete_production 失败: {e}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def log_step(
|
||||
production_id: int,
|
||||
step_name: str,
|
||||
status: str = "completed",
|
||||
cost: float = 0.0,
|
||||
detail: str = "",
|
||||
error_message: str = "",
|
||||
) -> bool:
|
||||
"""
|
||||
记录一步生产步骤
|
||||
|
||||
参数:
|
||||
production_id: 所属 production ID
|
||||
step_name: 步骤名 (tts, search_material, compose, bgm, lipsync, ...)
|
||||
status: completed / failed
|
||||
cost: 该步骤费用
|
||||
detail: 文字详情
|
||||
"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return False
|
||||
try:
|
||||
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
conn.execute(
|
||||
"""INSERT INTO production_steps
|
||||
(production_id, step_name, status, started_at, ended_at, cost, detail, error_message)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(production_id, step_name, status, now, now, cost, detail, error_message),
|
||||
)
|
||||
conn.commit()
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" [dashboard] log_step 失败: {e}")
|
||||
return False
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def log_asset(
|
||||
production_id: int,
|
||||
asset_type: str,
|
||||
file_path: str = "",
|
||||
name: str = "",
|
||||
source: str = "",
|
||||
tags: list[str] | None = None,
|
||||
meta: dict | None = None,
|
||||
) -> int:
|
||||
"""
|
||||
记录一个生成/使用的资产
|
||||
|
||||
返回:
|
||||
asset_id (失败返回 -1)
|
||||
"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return -1
|
||||
try:
|
||||
file_size = os.path.getsize(file_path) if file_path and os.path.exists(file_path) else 0
|
||||
cur = conn.execute(
|
||||
"""INSERT INTO assets (asset_type, name, file_path, file_size, source, meta_json, production_id)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)""",
|
||||
(asset_type, name, file_path, file_size, source,
|
||||
json.dumps(meta or {}, ensure_ascii=False), production_id),
|
||||
)
|
||||
aid = cur.lastrowid
|
||||
|
||||
# 关联 production ↔ asset
|
||||
conn.execute(
|
||||
"INSERT OR IGNORE INTO production_assets (production_id, asset_id, role) VALUES (?, ?, ?)",
|
||||
(production_id, aid, "output" if source == "generated" else "input"),
|
||||
)
|
||||
|
||||
# 标签
|
||||
if tags:
|
||||
for tag in tags:
|
||||
conn.execute(
|
||||
"INSERT OR IGNORE INTO asset_tags (asset_id, tag) VALUES (?, ?)",
|
||||
(aid, tag.strip().lower()),
|
||||
)
|
||||
|
||||
conn.commit()
|
||||
return aid
|
||||
except Exception as e:
|
||||
print(f" [dashboard] log_asset 失败: {e}")
|
||||
return -1
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def log_cost(
|
||||
production_id: int,
|
||||
service: str,
|
||||
step_name: str = "",
|
||||
duration_s: float = 0.0,
|
||||
resolution: str = "",
|
||||
chars: int = 0,
|
||||
cost_yuan: float = 0.0,
|
||||
status: str = "success",
|
||||
note: str = "",
|
||||
):
|
||||
"""记录费用明细"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return
|
||||
try:
|
||||
conn.execute(
|
||||
"""INSERT INTO cost_log
|
||||
(production_id, step_name, service, duration_s, resolution, chars, cost_yuan, status, note)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(production_id, step_name, service, duration_s, resolution, chars, cost_yuan, status, note),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
# 更新 production 累计费用
|
||||
conn.execute(
|
||||
"UPDATE productions SET total_cost = total_cost + ? WHERE id = ?",
|
||||
(cost_yuan, production_id),
|
||||
)
|
||||
conn.commit()
|
||||
except Exception as e:
|
||||
print(f" [dashboard] log_cost 失败: {e}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# 查询 API (供 FastAPI 后端使用)
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
def get_productions(
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
strategy: str | None = None,
|
||||
status: str | None = None,
|
||||
) -> list[dict]:
|
||||
"""查询 production 列表"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return []
|
||||
try:
|
||||
where = []
|
||||
params = []
|
||||
if strategy:
|
||||
where.append("strategy = ?")
|
||||
params.append(strategy)
|
||||
if status:
|
||||
where.append("status = ?")
|
||||
params.append(status)
|
||||
where_clause = (" WHERE " + " AND ".join(where)) if where else ""
|
||||
rows = conn.execute(
|
||||
f"SELECT * FROM productions{where_clause} ORDER BY id DESC LIMIT ? OFFSET ?",
|
||||
params + [limit, offset],
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
except Exception as e:
|
||||
print(f" [dashboard] get_productions 失败: {e}")
|
||||
return []
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def get_production_detail(production_id: int) -> dict | None:
|
||||
"""查询单条 production + steps + assets + costs"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return None
|
||||
try:
|
||||
prod = conn.execute(
|
||||
"SELECT * FROM productions WHERE id = ?", (production_id,)
|
||||
).fetchone()
|
||||
if not prod:
|
||||
return None
|
||||
|
||||
steps = conn.execute(
|
||||
"SELECT * FROM production_steps WHERE production_id = ? ORDER BY id",
|
||||
(production_id,),
|
||||
).fetchall()
|
||||
|
||||
assets = conn.execute(
|
||||
"""SELECT a.*, GROUP_CONCAT(at.tag, ',') as tags
|
||||
FROM assets a
|
||||
LEFT JOIN asset_tags at ON at.asset_id = a.id
|
||||
WHERE a.production_id = ?
|
||||
GROUP BY a.id ORDER BY a.id""",
|
||||
(production_id,),
|
||||
).fetchall()
|
||||
|
||||
costs = conn.execute(
|
||||
"SELECT * FROM cost_log WHERE production_id = ? ORDER BY id",
|
||||
(production_id,),
|
||||
).fetchall()
|
||||
|
||||
return {
|
||||
"production": dict(prod),
|
||||
"steps": [dict(s) for s in steps],
|
||||
"assets": [dict(a) for a in assets],
|
||||
"costs": [dict(c) for c in costs],
|
||||
}
|
||||
except Exception as e:
|
||||
print(f" [dashboard] get_production_detail 失败: {e}")
|
||||
return None
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def get_assets(asset_type: str | None = None, tag: str | None = None,
|
||||
limit: int = 100, offset: int = 0) -> list[dict]:
|
||||
"""查询资产列表"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return []
|
||||
try:
|
||||
where = []
|
||||
params = []
|
||||
if asset_type:
|
||||
where.append("a.asset_type = ?")
|
||||
params.append(asset_type)
|
||||
if tag:
|
||||
where.append("EXISTS (SELECT 1 FROM asset_tags WHERE asset_id = a.id AND tag = ?)")
|
||||
params.append(tag.lower())
|
||||
where_clause = (" WHERE " + " AND ".join(where)) if where else ""
|
||||
|
||||
rows = conn.execute(
|
||||
f"""SELECT a.*, GROUP_CONCAT(at.tag, ',') as tags
|
||||
FROM assets a
|
||||
LEFT JOIN asset_tags at ON at.asset_id = a.id
|
||||
{where_clause}
|
||||
GROUP BY a.id
|
||||
ORDER BY a.id DESC LIMIT ? OFFSET ?""",
|
||||
params + [limit, offset],
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
except Exception as e:
|
||||
print(f" [dashboard] get_assets 失败: {e}")
|
||||
return []
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def get_summary() -> dict:
|
||||
"""获取总览统计"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return {}
|
||||
try:
|
||||
total_prods = conn.execute("SELECT COUNT(*) FROM productions").fetchone()[0]
|
||||
completed = conn.execute("SELECT COUNT(*) FROM productions WHERE status='completed'").fetchone()[0]
|
||||
failed = conn.execute("SELECT COUNT(*) FROM productions WHERE status='failed'").fetchone()[0]
|
||||
total_cost = conn.execute("SELECT COALESCE(SUM(total_cost), 0) FROM productions").fetchone()[0]
|
||||
total_assets = conn.execute("SELECT COUNT(*) FROM assets").fetchone()[0]
|
||||
# 本月
|
||||
month_start = datetime.now().strftime("%Y-%m-01")
|
||||
month_prods = conn.execute(
|
||||
"SELECT COUNT(*) FROM productions WHERE created_at >= ?", (month_start,)
|
||||
).fetchone()[0]
|
||||
month_cost = conn.execute(
|
||||
"SELECT COALESCE(SUM(total_cost), 0) FROM productions WHERE created_at >= ?",
|
||||
(month_start,),
|
||||
).fetchone()[0]
|
||||
|
||||
return {
|
||||
"total_productions": total_prods,
|
||||
"completed": completed,
|
||||
"failed": failed,
|
||||
"total_cost": round(total_cost, 2),
|
||||
"total_assets": total_assets,
|
||||
"month_productions": month_prods,
|
||||
"month_cost": round(month_cost, 2),
|
||||
}
|
||||
except Exception as e:
|
||||
print(f" [dashboard] get_summary 失败: {e}")
|
||||
return {}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def get_cost_breakdown() -> list[dict]:
|
||||
"""按策略统计费用"""
|
||||
conn = _get_conn()
|
||||
if not conn:
|
||||
return []
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"""SELECT strategy,
|
||||
COUNT(*) as count,
|
||||
COALESCE(SUM(duration_sec), 0) as total_duration,
|
||||
COALESCE(SUM(total_cost), 0) as total_cost,
|
||||
ROUND(AVG(total_cost), 2) as avg_cost
|
||||
FROM productions
|
||||
GROUP BY strategy
|
||||
ORDER BY total_cost DESC"""
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
except Exception:
|
||||
return []
|
||||
finally:
|
||||
conn.close()
|
||||
@@ -0,0 +1,131 @@
|
||||
"""
|
||||
lib.ffmpeg — FFmpeg / FFprobe 路径统一入口
|
||||
=============================================
|
||||
|
||||
本机 FFmpeg 装在 `~/.local/bin`（非 brew、不在默认 PATH），
|
||||
所有模块统一从这里取路径，避免 `subprocess.run(["ffmpeg", ...])` 找不到命令。
|
||||
|
||||
用法：
|
||||
from lib.ffmpeg import FFMPEG, FFPROBE
|
||||
subprocess.run([FFMPEG, "-y", "-i", src, dst])
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
_CANDIDATES = [
|
||||
os.path.expanduser("~/.local/bin/ffmpeg"),
|
||||
"/usr/local/bin/ffmpeg",
|
||||
"/opt/homebrew/bin/ffmpeg",
|
||||
]
|
||||
_CANDIDATES_PROBE = [
|
||||
os.path.expanduser("~/.local/bin/ffprobe-real"),
|
||||
"/usr/local/bin/ffprobe",
|
||||
"/opt/homebrew/bin/ffprobe",
|
||||
]
|
||||
|
||||
|
||||
def _find_real_ffprobe() -> str | None:
|
||||
"""查找真正可用的 ffprobe
|
||||
|
||||
⚠️ 本机 `~/.local/bin/ffprobe` 实为 `ffmpeg` 的**符号链接**（不支持 -show_entries）；
|
||||
真正的 ffprobe 在别处（如 Trae/火星代码自带 6.1.1 完整版）。
|
||||
"""
|
||||
import glob
|
||||
pats = [
|
||||
os.path.expanduser("~/.marscode/**/modules/ai-agent/ffprobe"),
|
||||
os.path.expanduser("~/.trae/**/ffprobe"),
|
||||
os.path.expanduser("~/.vscode/**/ffprobe"),
|
||||
"/Applications/*.app/Contents/**/ffprobe",
|
||||
]
|
||||
for pat in pats:
|
||||
for p in sorted(glob.glob(pat, recursive=True), reverse=True):
|
||||
try:
|
||||
if os.path.isfile(p) and os.access(p, os.X_OK):
|
||||
# 验证确实支持 -show_entries（真正的 ffprobe）
|
||||
import subprocess
|
||||
r = subprocess.run([p, "-h"], capture_output=True, text=True, timeout=10)
|
||||
if "show_entries" in ((r.stdout or "") + (r.stderr or "")):
|
||||
return p
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# 媒体信息探测（**用 `ffmpeg -i` 解析**）
|
||||
#
|
||||
# ⚠️ 本机 ffprobe 是精简构建：不支持 `-show_entries` / `-select_streams`
|
||||
# （会报 `Unrecognized option`）→ 相对地，`ffmpeg -i` 完全可用。
|
||||
# 因此统一改用 ffmpeg 输出解析，避免各处 ffprobe 调用**静默失败**。
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
def probe_info(media_path) -> dict:
|
||||
"""探测媒体信息 → {duration, bitrate_kbps, size_bytes, has_audio, has_video}"""
|
||||
import re
|
||||
import subprocess
|
||||
|
||||
p = Path(media_path)
|
||||
info = {"duration": 0.0, "bitrate_kbps": 0,
|
||||
"size_bytes": p.stat().st_size if p.exists() else 0,
|
||||
"has_audio": False, "has_video": False}
|
||||
if not p.exists():
|
||||
return info
|
||||
try:
|
||||
r = subprocess.run([FFMPEG, "-i", str(p)], capture_output=True, text=True, timeout=30)
|
||||
txt = (r.stderr or "") + (r.stdout or "")
|
||||
except Exception:
|
||||
return info
|
||||
m = re.search(r"Duration:\s*(\d+):(\d+):([\d.]+)", txt)
|
||||
if m:
|
||||
info["duration"] = int(m.group(1)) * 3600 + int(m.group(2)) * 60 + float(m.group(3))
|
||||
m = re.search(r"bitrate:\s*(\d+)\s*kb/s", txt)
|
||||
if m:
|
||||
info["bitrate_kbps"] = int(m.group(1))
|
||||
elif info["duration"] > 0:
|
||||
info["bitrate_kbps"] = int(info["size_bytes"] * 8 / info["duration"] / 1000)
|
||||
info["has_audio"] = "Audio:" in txt
|
||||
info["has_video"] = "Video:" in txt
|
||||
# 视频尺寸（从 "Video: h264 ..., 1080x1920" 解析）
|
||||
m = re.search(r"Video:.*?(\d{2,5})x(\d{2,5})", txt)
|
||||
if m:
|
||||
info["width"], info["height"] = int(m.group(1)), int(m.group(2))
|
||||
else:
|
||||
info["width"], info["height"] = 0, 0
|
||||
return info
|
||||
|
||||
|
||||
def probe_duration(media_path) -> float:
|
||||
"""媒体时长（秒）—— 兼容实现（ffmpeg -i 解析）"""
|
||||
return probe_info(media_path)["duration"]
|
||||
|
||||
|
||||
def has_audio_stream(media_path) -> bool:
|
||||
"""是否含音轨 —— 兼容实现"""
|
||||
return probe_info(media_path)["has_audio"]
|
||||
|
||||
|
||||
def _resolve(candidates: list[str], name: str) -> str:
|
||||
for c in candidates:
|
||||
if Path(c).exists():
|
||||
return c
|
||||
found = shutil.which(name)
|
||||
if found:
|
||||
return found
|
||||
# 找不到也返回名字（让调用时报清晰的错），但记录警告
|
||||
return name
|
||||
|
||||
|
||||
FFMPEG = _resolve(_CANDIDATES, "ffmpeg")
|
||||
# ⚠️ 本机 `~/.local/bin/ffprobe` 实为 ffmpeg 的**符号链接**（不支持 -show_entries）
|
||||
# → 优先查找**真正的** ffprobe；找不到就退回 FFMPEG（配合 probe_info 使用）
|
||||
FFPROBE = _find_real_ffprobe() or FFMPEG
|
||||
|
||||
__all__ = ["FFMPEG", "FFPROBE", "ffmpeg_available"]
|
||||
|
||||
|
||||
def ffmpeg_available() -> bool:
|
||||
"""ffmpeg 是否真的可用"""
|
||||
return Path(FFMPEG).exists() or bool(shutil.which("ffmpeg"))
|
||||
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
AVE 文件上传助手 — 上传文件到 ghvideo Gitee 仓库获取公网直链
|
||||
|
||||
用途:
|
||||
即梦/火山引擎 API 需要图片/视频/音频的公网 HTTP URL。
|
||||
通过 Gitee raw 服务提供直链: https://gitee.com/babycalf/ghvideo/raw/main/{filename}
|
||||
|
||||
用法:
|
||||
python ghvideo_upload.py /path/to/file.jpg
|
||||
python ghvideo_upload.py --dir /path/to/dir/ # 批量上传目录下所有文件
|
||||
|
||||
返回:
|
||||
https://gitee.com/babycalf/ghvideo/raw/main/filename.ext
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_DIR = Path.home() / "workbuddy-agent-os" / "agent-local" / "tools" / "ave" / "ghvideo"
|
||||
REPO_URL = "https://gitee.com/babycalf/ghvideo.git"
|
||||
RAW_BASE = "https://gitee.com/babycalf/ghvideo/raw/main"
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="上传文件到 ghvideo 中转仓库")
|
||||
parser.add_argument("path", nargs="*", help="文件或目录路径")
|
||||
parser.add_argument("--dir", help="批量上传目录")
|
||||
args = parser.parse_args()
|
||||
|
||||
files = []
|
||||
if args.dir:
|
||||
for f in sorted(Path(args.dir).iterdir()):
|
||||
if f.is_file():
|
||||
files.append(str(f))
|
||||
elif args.path:
|
||||
for p in args.path:
|
||||
if os.path.isdir(p):
|
||||
for f in sorted(Path(p).iterdir()):
|
||||
if f.is_file():
|
||||
files.append(str(f))
|
||||
else:
|
||||
files.append(p)
|
||||
else:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
if not files:
|
||||
print("没有文件")
|
||||
return
|
||||
|
||||
# 确保仓库存在
|
||||
if not (REPO_DIR / ".git").exists():
|
||||
print(f"初始化仓库: {REPO_DIR}")
|
||||
subprocess.run(["git", "init"], cwd=REPO_DIR, capture_output=True)
|
||||
subprocess.run(["git", "remote", "add", "origin", REPO_URL], cwd=REPO_DIR, capture_output=True)
|
||||
|
||||
# 复制文件到仓库目录
|
||||
for f in files:
|
||||
src = Path(f)
|
||||
if not src.exists():
|
||||
print(f" 跳过: {f} (不存在)")
|
||||
continue
|
||||
dest = REPO_DIR / src.name
|
||||
import shutil
|
||||
shutil.copy2(str(src), str(dest))
|
||||
print(f" 复制: {src.name}")
|
||||
|
||||
# Git 提交推送
|
||||
subprocess.run(["git", "add", "-A"], cwd=REPO_DIR, capture_output=True)
|
||||
subprocess.run(["git", "commit", "-m", f"upload: {len(files)} files"], cwd=REPO_DIR, capture_output=True)
|
||||
r = subprocess.run(["git", "push", "-u", "origin", "master"], cwd=REPO_DIR,
|
||||
capture_output=True, text=True, timeout=60)
|
||||
if r.returncode == 0:
|
||||
print(f"\n✅ 推送成功! {len(files)} 个文件")
|
||||
for f in files:
|
||||
name = Path(f).name
|
||||
url = f"{RAW_BASE}/{name}"
|
||||
print(f" {url}")
|
||||
else:
|
||||
# 可能已经有内容, 尝试 pull 再 push
|
||||
print(f" 推送失败, 尝试 pull...")
|
||||
subprocess.run(["git", "pull", "--rebase", "origin", "master"], cwd=REPO_DIR,
|
||||
capture_output=True, timeout=30)
|
||||
r2 = subprocess.run(["git", "push", "-u", "origin", "master"], cwd=REPO_DIR,
|
||||
capture_output=True, text=True, timeout=60)
|
||||
if r2.returncode == 0:
|
||||
print(f"✅ 推送成功!")
|
||||
for f in files:
|
||||
name = Path(f).name
|
||||
url = f"{RAW_BASE}/{name}"
|
||||
print(f" {url}")
|
||||
else:
|
||||
print(f"❌ 推送失败: {r2.stderr[:200]}")
|
||||
|
||||
|
||||
def upload_and_get_url(file_path: str) -> str:
|
||||
"""
|
||||
上传文件到 ghvideo 仓库并返回公网直链
|
||||
|
||||
参数:
|
||||
file_path: 本地文件路径
|
||||
|
||||
返回:
|
||||
公网直链 URL
|
||||
"""
|
||||
import shutil
|
||||
name = Path(file_path).name
|
||||
dest = REPO_DIR / name
|
||||
|
||||
# 复制文件到仓库
|
||||
shutil.copy2(file_path, str(dest))
|
||||
|
||||
# 提交推送
|
||||
subprocess.run(["git", "-C", str(REPO_DIR), "add", name], capture_output=True)
|
||||
subprocess.run(["git", "-C", str(REPO_DIR), "commit", "-m", f"upload {name}"],
|
||||
capture_output=True)
|
||||
r = subprocess.run(["git", "-C", str(REPO_DIR), "push", "origin", "master"],
|
||||
capture_output=True, text=True, timeout=60)
|
||||
if r.returncode != 0:
|
||||
# pull 再试
|
||||
subprocess.run(["git", "-C", str(REPO_DIR), "pull", "--rebase", "origin", "master"],
|
||||
capture_output=True, timeout=30)
|
||||
subprocess.run(["git", "-C", str(REPO_DIR), "push", "origin", "master"],
|
||||
capture_output=True, timeout=60)
|
||||
|
||||
url = f"{RAW_BASE}/{name}"
|
||||
print(f" 上传完成: {url}")
|
||||
return url
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,93 @@
|
||||
"""
|
||||
lib/http.py — 统一 HTTP 入口（禁用系统代理 + 自动重试）
|
||||
|
||||
## 为什么需要（2026-10-04 定位的坑）
|
||||
macOS 开启系统代理（如 127.0.0.1:7892）后，Python 的 `urllib.request.urlopen`
|
||||
会**自动读取系统代理设置**。而系统代理对国内 API（可灵 / 百炼 / 火山）经常
|
||||
时通时不通，表现为：
|
||||
|
||||
[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol
|
||||
|
||||
→ 用户看到的是「生成随机失败」「分镜生成失败」这类难以复现的问题。
|
||||
|
||||
实测：直连（禁代理）3/3 次 HTTP 200、0.2~0.3 秒；走代理则间歇性 SSL 错误。
|
||||
|
||||
## 用法
|
||||
from lib.http import urlopen, request_json
|
||||
|
||||
req = urllib.request.Request(url, headers={...})
|
||||
with urlopen(req, timeout=60) as r: # 已禁代理 + 自动重试
|
||||
data = r.read()
|
||||
|
||||
# 或一步到位：
|
||||
data = request_json(url, method="POST", body={...}, headers={...})
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from typing import Any, Optional
|
||||
|
||||
# 禁代理 opener（进程内复用）
|
||||
_NO_PROXY_OPENER = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
|
||||
DEFAULT_RETRIES = 3
|
||||
DEFAULT_BACKOFF = 1.5 # 秒；第 n 次失败后等待 backoff * n
|
||||
|
||||
# 值得重试的 HTTP 状态码：
|
||||
# · 429 限流（等一会儿可能就通）
|
||||
# · 5xx 服务端临时故障（可灵/百炼偶尔返回 502/503）
|
||||
# 不重试：其余 4xx（参数错/鉴权失败/余额不足）—— 重试只是浪费时间
|
||||
_RETRYABLE_HTTP = frozenset({429, 500, 502, 503, 504})
|
||||
|
||||
|
||||
def urlopen(req, timeout: int = 60, retries: int = DEFAULT_RETRIES):
|
||||
"""禁用代理地发起请求，失败自动重试（指数退避）
|
||||
|
||||
返回的是可作上下文管理器使用的 response 对象。
|
||||
重试覆盖：URLError / SSLError / OSError / 超时 / 连接中断。
|
||||
⚠️ HTTP 4xx/5xx 不重试（那是业务错误，重试无意义）——
|
||||
本函数只对**网络层异常**重试。
|
||||
"""
|
||||
last_exc: Optional[BaseException] = None
|
||||
for attempt in range(max(1, retries)):
|
||||
try:
|
||||
return _NO_PROXY_OPENER.open(req, timeout=timeout)
|
||||
except urllib.error.HTTPError as e:
|
||||
# 5xx（服务端临时故障）与 429（限流）值得重试；
|
||||
# 其余 4xx（参数错/鉴权失败/余额不足）重试无意义 → 直接抛
|
||||
if e.code in _RETRYABLE_HTTP:
|
||||
last_exc = e
|
||||
if attempt < retries - 1:
|
||||
time.sleep(DEFAULT_BACKOFF * (attempt + 1))
|
||||
continue
|
||||
raise
|
||||
except Exception as e:
|
||||
last_exc = e
|
||||
if attempt < retries - 1:
|
||||
time.sleep(DEFAULT_BACKOFF * (attempt + 1))
|
||||
raise last_exc # type: ignore[misc]
|
||||
|
||||
|
||||
def request_json(url: str, method: str = "GET", body: Any = None,
|
||||
headers: Optional[dict] = None, timeout: int = 60,
|
||||
retries: int = DEFAULT_RETRIES) -> dict:
|
||||
"""发起 JSON 请求并解析响应（禁用代理 + 自动重试）
|
||||
|
||||
· body 非 None 时自动 JSON 序列化并加 Content-Type
|
||||
· 返回值：解析后的 dict（解析失败返回 {}）
|
||||
"""
|
||||
hdr = dict(headers or {})
|
||||
data = None
|
||||
if body is not None:
|
||||
data = json.dumps(body).encode("utf-8")
|
||||
hdr.setdefault("Content-Type", "application/json")
|
||||
req = urllib.request.Request(url, data=data, headers=hdr, method=method)
|
||||
with urlopen(req, timeout=timeout, retries=retries) as r:
|
||||
raw = r.read()
|
||||
try:
|
||||
return json.loads(raw.decode("utf-8"))
|
||||
except Exception:
|
||||
return {}
|
||||
@@ -0,0 +1,151 @@
|
||||
"""
|
||||
lib.kling_webhook — 可灵 Webhook 回调验签（按官方规范）
|
||||
=========================================================
|
||||
|
||||
官方文档：https://klingai.com/document-api/api/get-started/callbacks
|
||||
|
||||
回调请求头（3 个）：
|
||||
webhook-id 任务/事件 ID（参与签名）
|
||||
webhook-timestamp Unix 秒（参与签名，±5 分钟防重放）
|
||||
webhook-signature 形如 "v1,<base64>" 或 "v1,<a> v1,<b>"（可能多个）
|
||||
|
||||
验签规则：
|
||||
signedContent = f"{webhook-id}.{webhook-timestamp}.{rawBody}"
|
||||
key = base64Decode(secret.removeprefix("whsec_"))
|
||||
expected = base64(HMAC-SHA256(key, signedContent))
|
||||
与每个 "v1,<sig>" 做**常量时间比较**，任一匹配即通过
|
||||
|
||||
安全要点：
|
||||
· secret 只从配置读取（agent-local，600 权限，不进 git），不写日志
|
||||
· 常量时间比较（hmac.compare_digest）防时序攻击
|
||||
· 时间窗校验防重放
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
log = logging.getLogger("kling.webhook")
|
||||
|
||||
TOLERANCE_SECONDS = 5 * 60 # 官方建议 ±5 分钟
|
||||
|
||||
|
||||
def load_webhook_secret() -> str:
|
||||
"""读取 webhook secret（配置来源：环境变量 > agent-local local.yaml → kling.webhook_secret）"""
|
||||
if os.environ.get("KLING_WEBHOOK_SECRET"):
|
||||
return os.environ["KLING_WEBHOOK_SECRET"].strip()
|
||||
try:
|
||||
import yaml
|
||||
base = Path(os.environ.get("AGENT_LOCAL")
|
||||
or (Path.home() / "workbuddy-agent-os" / "agent-local"))
|
||||
p = base / "tools" / "ave" / "config" / "local.yaml"
|
||||
if p.exists():
|
||||
d = yaml.safe_load(p.read_text(encoding="utf-8")) or {}
|
||||
return ((d.get("kling") or {}).get("webhook_secret") or "").strip()
|
||||
except Exception as e:
|
||||
log.warning("读取 webhook secret 失败: %s", e)
|
||||
return ""
|
||||
|
||||
|
||||
def _decode_key(secret: str) -> bytes:
|
||||
"""去掉 whsec_ 前缀后 Base64 解码（官方要求）"""
|
||||
raw = secret.strip()
|
||||
if raw.startswith("whsec_"):
|
||||
raw = raw[len("whsec_"):]
|
||||
# 补齐 padding（部分 secret 可能被截断/省略 =）
|
||||
pad = (-len(raw)) % 4
|
||||
return base64.b64decode(raw + "=" * pad)
|
||||
|
||||
|
||||
def sign(secret: str, webhook_id: str, timestamp: str, raw_body: bytes) -> str:
|
||||
"""按官方规范生成签名（供自测/回放用）"""
|
||||
signed = f"{webhook_id}.{timestamp}.".encode("utf-8") + raw_body
|
||||
digest = hmac.new(_decode_key(secret), signed, hashlib.sha256).digest()
|
||||
return base64.b64encode(digest).decode("utf-8")
|
||||
|
||||
|
||||
def verify(secret: str, raw_body: bytes, webhook_id: str,
|
||||
timestamp: str, signature_header: str,
|
||||
tolerance: int = TOLERANCE_SECONDS) -> tuple[bool, str]:
|
||||
"""验签（返回 (通过与否, 原因)）
|
||||
|
||||
失败原因便于排查（但**不泄露签名/密钥内容**）。
|
||||
"""
|
||||
if not secret:
|
||||
return False, "未配置 webhook_secret"
|
||||
for name, v in (("webhook-id", webhook_id), ("webhook-timestamp", timestamp),
|
||||
("webhook-signature", signature_header)):
|
||||
if not v:
|
||||
return False, f"缺少请求头 {name}"
|
||||
|
||||
# ① 时间窗（防重放）
|
||||
try:
|
||||
ts = int(float(timestamp))
|
||||
except Exception:
|
||||
return False, "webhook-timestamp 格式错误"
|
||||
drift = abs(int(time.time()) - ts)
|
||||
if tolerance and drift > tolerance:
|
||||
return False, f"时间戳偏差过大（{drift}s > {tolerance}s）"
|
||||
|
||||
# ② 期望签名
|
||||
try:
|
||||
expected = sign(secret, webhook_id, timestamp, raw_body)
|
||||
except Exception as e:
|
||||
return False, f"签名计算失败: {e}"
|
||||
|
||||
# ③ 逐个 "v1,<sig>" 常量时间比较
|
||||
parts = [p for p in (signature_header or "").split(" ") if p]
|
||||
for item in parts:
|
||||
ver, _, sig = item.partition(",")
|
||||
if ver.strip() != "v1" or not sig:
|
||||
continue
|
||||
if hmac.compare_digest(sig.strip(), expected):
|
||||
return True, "ok"
|
||||
return False, "签名不匹配"
|
||||
|
||||
|
||||
def parse_callback(body: dict) -> dict:
|
||||
"""解析回调 body → 统一结构（供业务处理）
|
||||
|
||||
官方字段：
|
||||
id / status(submitted|processing|succeeded|failed) / message /
|
||||
create_time / update_time / external_id / outputs[]
|
||||
"""
|
||||
if not isinstance(body, dict):
|
||||
return {"ok": False, "reason": "body 不是 JSON 对象"}
|
||||
outputs = body.get("outputs") or []
|
||||
files: list[dict] = []
|
||||
for o in outputs:
|
||||
if not isinstance(o, dict):
|
||||
continue
|
||||
t = o.get("type")
|
||||
url = o.get("url") or o.get("mp3_url") or o.get("wav_url") or ""
|
||||
if not url:
|
||||
continue
|
||||
files.append({
|
||||
"type": t, "url": url,
|
||||
"watermark_url": o.get("watermark_url") or "",
|
||||
"duration": o.get("duration") or o.get("mp3_duration") or "",
|
||||
"id": o.get("id") or "",
|
||||
})
|
||||
status = (body.get("status") or "").lower()
|
||||
return {
|
||||
"ok": True,
|
||||
"task_id": body.get("id") or "",
|
||||
"external_id": body.get("external_id") or "",
|
||||
"status": status, # submitted|processing|succeeded|failed
|
||||
"succeeded": status == "succeeded",
|
||||
"failed": status == "failed",
|
||||
"message": body.get("message") or "",
|
||||
"create_time": body.get("create_time"),
|
||||
"update_time": body.get("update_time"),
|
||||
"files": files,
|
||||
"raw_keys": sorted(body.keys()),
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
"""
|
||||
AVE 日志工具
|
||||
"""
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
LOG_DIR = Path.home() / "workbuddy-agent-os" / "agent-local" / "tools" / "ave" / "cache"
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def get_logger(name: str = "ave") -> logging.Logger:
|
||||
logger = logging.getLogger(name)
|
||||
if logger.handlers:
|
||||
return logger
|
||||
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
# 控制台
|
||||
console = logging.StreamHandler(sys.stdout)
|
||||
console.setFormatter(logging.Formatter("[%(levelname)s] %(message)s"))
|
||||
logger.addHandler(console)
|
||||
|
||||
# 文件
|
||||
fh = logging.FileHandler(LOG_DIR / "ave.log", encoding="utf-8")
|
||||
fh.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s"))
|
||||
logger.addHandler(fh)
|
||||
|
||||
return logger
|
||||
Reference in New Issue
Block a user