butubb/MediaCrawler
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程，直接发 HTTP
Deploy VitePress site to Pages / build (push) Canceled after 0s
Details
Deploy VitePress site to Pages / Deploy (push) Canceled after 0s
Details

Browse Source 
移植自 mac-agent-os 的 mediacrawler_adapter：不起子进程、不开页面，用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin，所以 ingest 那条链路一个字不用改。

**目前能用的（真环境实测，非推断）**：

    profile/other : 200, 7075 字节  —— 博主主页指标（粉丝/获赞/作品数/昵称）
    aweme/detail  : 200, 45425 字节 —— 单条作品详情（含点赞/评论/收藏/分享）

**目前不能用的：作品列表 `aweme/post`。** 两个互相独立的原因：
  1. 这个接口被抖音单独升级成了真校验：不带 x-tt-argus 回 403「Uifid Not Found」，
     带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过，「真校验」过不了。
     同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的，
     挑中的恰好是「批量拉作品列表」这个最敏感的动作。
  2. 改走页面截获也不行：CDP 浏览器打开博主主页会落到「验证码中间页」（当天大量探测的
     代价，过几小时要重测）。

  所以现在的边界是：**已知作品的指标刷新能做，自动发现新作品做不了**。

**排查中控住变量后得到的两条事实**（都写进注释了）：
  · `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
    UA+client hints+Cookie 时是 200 但仅 121 字节的空壳，补上这三个头变 7074 字节。
    （我先前猜的 sec-ch-ua 不是关键。）
  · 因此 UA 与 client hints 必须**成套地取自同一个浏览器**，所以 BrowserIdentity 一次
    从 CDP 取齐 cookie + UA + hints，而不是各自写死。

「200 + 空 body 必须当场报错」也是刻意写死的：放过去它会在下游变成「这个博主没作品」，
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的，还被翻译成「账号被封」。

测试 +11：cookie 解析、请求头成套性（含 uifid 缺失/回退）、产物键名与 store 对齐、
以及 _get 的三条失败路径（空 body / 403 带网关原话 / 正常返回）。
This commit is contained in:
butubb
2026-10-10 17:12:25 +08:00
parent 21ff01b894
commit 2112c1a870
1601 changed files with 367425 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mac-agent-os-main/05_tools/09_ave/scripts/script_schemas/__init__.py
+275
View File
@@ -0,0 +1,275 @@
"""
脚本格式标准 — 漫剧视频工厂
功能:
- 定义脚本 YAML 标准格式
- 提供脚本创建/验证/转换工具
- 生成 Gate 1/2 展示所需的摘要
用法:
from script_schemas import ScriptBuilder, validate_script
script = ScriptBuilder.create_from_text("灵感描述")
errors = validate_script(script)
"""
import os
import copy
from pathlib import Path
from typing import Optional
import yaml
# ── 默认路径 ──
SCRIPTS_DIR = Path(__file__).resolve().parent
SCHEMA_FILE = SCRIPTS_DIR / "script_schema.yaml"
# 镜头类型列表
VALID_CAMERAS = [
"wide_shot", "medium_shot", "close_up", "over_shoulder",
"aerial", "tracking", "pan", "dolly", "extreme_wide"
]
# 转场类型列表
VALID_TRANSITIONS = [
"cut", "fade_in", "fade_out", "dissolve",
"wipe_left", "wipe_right", "zoom_in", "zoom_out", "none"
]
# ═══════════════════════════════════════════════════════════
# 脚本构建器
# ═══════════════════════════════════════════════════════════
class ScriptBuilder:
"""脚本构建器 — 从各种输入创建标准格式脚本"""
@staticmethod
def empty(title: str = "未命名", character: str = "小漫") -> dict:
"""创建空脚本模板"""
return {
"meta": {
"title": title,
"source": "手动创建",
"source_url": "",
"duration_sec": 0,
"character": character,
"visual_style": "manhua",
"bgm_style": "治愈",
},
"segments": [],
}
@staticmethod
def from_text(text: str, title: str = "灵感脚本",
character: str = "小漫") -> dict:
"""从文字灵感创建脚本（需后续填充segments）"""
script = ScriptBuilder.empty(title, character)
script["meta"]["source"] = "文字灵感"
return script
@staticmethod
def add_segment(script: dict, seg_data: dict) -> dict:
"""向脚本添加一个分镜"""
if "segments" not in script:
script["segments"] = []
# 自动编号
seg_data["id"] = len(script["segments"]) + 1
# 补全默认值
seg_data.setdefault("visual_style", script["meta"].get("visual_style", "manhua"))
seg_data.setdefault("camera", "medium_shot")
seg_data.setdefault("transition", "cut")
seg_data.setdefault("dialogue", "")
seg_data.setdefault("emotion", "中性")
seg_data.setdefault("duration_sec", 5)
script["segments"].append(seg_data)
# 更新总时长
script["meta"]["duration_sec"] = sum(
s.get("duration_sec", 0) for s in script["segments"]
)
return script
@staticmethod
def remove_segment(script: dict, seg_id: int) -> dict:
"""删除指定分镜"""
script["segments"] = [s for s in script["segments"] if s.get("id") != seg_id]
# 重新编号
for i, seg in enumerate(script["segments"], 1):
seg["id"] = i
script["meta"]["duration_sec"] = sum(
s.get("duration_sec", 0) for s in script["segments"]
)
return script
# ═══════════════════════════════════════════════════════════
# 脚本验证
# ═══════════════════════════════════════════════════════════
def validate_script(script: dict) -> list[str]:
"""验证脚本格式，返回错误列表（空=通过）"""
errors = []
# 检查 meta
meta = script.get("meta", {})
if not meta.get("title"):
errors.append("缺少标题 (meta.title)")
if not meta.get("character"):
errors.append("缺少角色 (meta.character)")
# 检查 segments
segments = script.get("segments", [])
if not segments:
errors.append("分镜列表为空 (segments)")
for seg in segments:
sid = seg.get("id", "?")
if not seg.get("scene"):
errors.append(f"分镜 #{sid}: 缺少场景名称 (scene)")
if not seg.get("description"):
errors.append(f"分镜 #{sid}: 缺少场景描述 (description)")
if seg.get("camera") and seg["camera"] not in VALID_CAMERAS:
errors.append(f"分镜 #{sid}: 无效镜头类型 '{seg['camera']}'")
if seg.get("transition") and seg["transition"] not in VALID_TRANSITIONS:
errors.append(f"分镜 #{sid}: 无效转场效果 '{seg['transition']}'")
return errors
# ═══════════════════════════════════════════════════════════
# 脚本工具
# ═══════════════════════════════════════════════════════════
def load_script(path: str) -> dict:
"""从 YAML 文件加载脚本"""
with open(path, encoding="utf-8") as f:
data = yaml.safe_load(f)
return data.get("script", data)
def save_script(script: dict, path: str):
"""保存脚本到 YAML 文件"""
with open(path, "w", encoding="utf-8") as f:
yaml.dump({"script": script}, f, allow_unicode=True,
default_flow_style=False, sort_keys=False)
def generate_gate1_summary(script: dict) -> dict:
"""生成 Gate 1 展示用的脚本摘要"""
meta = script.get("meta", {})
segments = script.get("segments", [])
# 按情绪分组统计
emotion_map = {}
for seg in segments:
em = seg.get("emotion", "中性")
emotion_map[em] = emotion_map.get(em, 0) + seg.get("duration_sec", 0)
# 场景结构
scene_list = []
for seg in segments:
scene_list.append({
"id": seg.get("id", 0),
"scene": seg.get("scene", ""),
"duration": seg.get("duration_sec", 0),
"emotion": seg.get("emotion", "中性"),
})
return {
"title": meta.get("title", ""),
"source": meta.get("source", ""),
"style": meta.get("visual_style", ""),
"total_duration": meta.get("duration_sec", 0),
"segment_count": len(segments),
"character": meta.get("character", ""),
"structure": scene_list,
"emotion_distribution": emotion_map,
"bgm_style": meta.get("bgm_style", ""),
}
def generate_gate2_table(script: dict) -> list[dict]:
"""生成 Gate 2 展示用的分镜预览表"""
segments = script.get("segments", [])
return [
{
"id": seg.get("id", 0),
"scene": seg.get("scene", ""),
"description": seg.get("description", ""),
"duration": seg.get("duration_sec", 0),
"emotion": seg.get("emotion", "中性"),
"visual_style": seg.get("visual_style", ""),
"camera": seg.get("camera", ""),
}
for seg in segments
]
# ═══════════════════════════════════════════════════════════
# 测试
# ═══════════════════════════════════════════════════════════
if __name__ == "__main__":
# 创建一个示例脚本
script = ScriptBuilder.empty("小漫的一天", "小漫")
ScriptBuilder.add_segment(script, {
"scene": "清晨的咖啡馆",
"description": "小漫推开咖啡馆的门，阳光洒进来",
"dialogue": "（旁白）这是小漫最爱的角落。",
"emotion": "温暖平静",
"duration_sec": 8,
"camera": "wide_shot",
})
ScriptBuilder.add_segment(script, {
"scene": "靠窗座位",
"description": "她坐下来，拿出画板开始画画",
"dialogue": "小漫: 今天一定要把草稿画完！",
"emotion": "专注",
"duration_sec": 6,
"camera": "medium_shot",
})
ScriptBuilder.add_segment(script, {
"scene": "落日的天台",
"description": "她看着完成的画，露出满足的微笑",
"dialogue": "（旁白）有时候，最简单的快乐就在画笔下。",
"emotion": "满足",
"duration_sec": 10,
"camera": "wide_shot",
"transition": "dissolve",
})
# 验证
errors = validate_script(script)
if errors:
print("❌ 验证失败:")
for e in errors:
print(f" - {e}")
else:
print("✅ 脚本验证通过")
# Gate 1 摘要
summary = generate_gate1_summary(script)
print(f"\n📋 Gate 1 摘要:")
print(f" 标题: {summary['title']}")
print(f" 角色: {summary['character']}")
print(f" 总时长: {summary['total_duration']}s")
print(f" 分镜数: {summary['segment_count']}")
items = [f"#{s['id']} {s['scene']}({s['duration']}s)" for s in summary['structure']]
print(f" 结构: {items}")
# Gate 2 表格
table = generate_gate2_table(script)
print(f"\n📋 Gate 2 分镜表:")
for row in table:
print(f" #{row['id']:2d} | {row['scene']:12s} | {row['duration']:2d}s | {row['emotion']:8s} | {row['camera']:16s}")
# 保存到文件
save_path = Path(__file__).parent / "_sample_script.yaml"
save_script(script, str(save_path))
print(f"\n✅ 已保存样例脚本: {save_path}")
mac-agent-os-main/05_tools/09_ave/scripts/script_schemas/script_schema.yaml
+57
View File
@@ -0,0 +1,57 @@
# 脚本格式标准 — 漫剧视频工厂 v1.0
# 所有输入的脚本都转为此标准格式
# ── 顶层结构 ──
#
# script:
# meta: # 元数据
# segments: # 分镜列表
#
# ── 示例 ──
script:
meta:
title: "小漫的一天" # 脚本标题
source: "文字灵感" # 来源: 文字灵感/参考视频/外部剧本
source_url: "" # 来源链接（如果是参考视频）
duration_sec: 60 # 预估总时长(秒)
character: "小漫" # 使用的角色名（必须已在角色注册中心注册）
visual_style: "manhua" # 默认视觉风格: manhua / realistic / mixed
bgm_style: "治愈" # BGM 风格偏好
segments:
- id: 1 # 分镜编号
scene: "清晨的咖啡馆" # 场景名称
description: "小漫推开咖啡馆的门，阳光洒进来，她微笑着走向窗边的座位"
# 场景描述（叙事）
dialogue: "" # 对话内容
# 格式: "角色名: 对话内容" 或 "(旁白) 旁白内容"
emotion: "温暖平静" # 角色情绪/表情
duration_sec: 8 # 此分镜时长(秒)
visual_style: "manhua" # 视觉风格: manhua / realistic（覆盖meta级别）
camera: "wide_shot" # 镜头类型
# wide_shot / medium_shot / close_up /
# over_shoulder / aerial / tracking / pan
transition: "fade_in" # 转场效果
# cut / fade_in / fade_out / dissolve /
# wipe_left / zoom_in / none
- id: 2
scene: "咖啡馆靠窗座位"
description: "她坐下来，把画板放在桌上，拿出铅笔开始构思"
dialogue: "小漫: 今天一定要把草稿画完！"
emotion: "专注"
duration_sec: 6
visual_style: "manhua"
camera: "medium_shot"
transition: "cut"
- id: 3
scene: "落日的天台"
description: "傍晚的阳光洒在天台上，她低头看着刚完成的画，露出满足的微笑"
dialogue: "（旁白）有时候，最简单的快乐就在画笔下。"
emotion: "满足"
duration_sec: 10
visual_style: "manhua"
camera: "wide_shot"
transition: "dissolve"