feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程,直接发 HTTP
移植自 mac-agent-os 的 mediacrawler_adapter:不起子进程、不开页面,用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin,所以 ingest 那条链路一个字不用改。
**目前能用的(真环境实测,非推断)**:
profile/other : 200, 7075 字节 —— 博主主页指标(粉丝/获赞/作品数/昵称)
aweme/detail : 200, 45425 字节 —— 单条作品详情(含点赞/评论/收藏/分享)
**目前不能用的:作品列表 `aweme/post`。** 两个互相独立的原因:
1. 这个接口被抖音单独升级成了真校验:不带 x-tt-argus 回 403「Uifid Not Found」,
带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过,「真校验」过不了。
同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的,
挑中的恰好是「批量拉作品列表」这个最敏感的动作。
2. 改走页面截获也不行:CDP 浏览器打开博主主页会落到「验证码中间页」(当天大量探测的
代价,过几小时要重测)。
所以现在的边界是:**已知作品的指标刷新能做,自动发现新作品做不了**。
**排查中控住变量后得到的两条事实**(都写进注释了):
· `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
UA+client hints+Cookie 时是 200 但仅 121 字节的空壳,补上这三个头变 7074 字节。
(我先前猜的 sec-ch-ua 不是关键。)
· 因此 UA 与 client hints 必须**成套地取自同一个浏览器**,所以 BrowserIdentity 一次
从 CDP 取齐 cookie + UA + hints,而不是各自写死。
「200 + 空 body 必须当场报错」也是刻意写死的:放过去它会在下游变成「这个博主没作品」,
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的,还被翻译成「账号被封」。
测试 +11:cookie 解析、请求头成套性(含 uifid 缺失/回退)、产物键名与 store 对齐、
以及 _get 的三条失败路径(空 body / 403 带网关原话 / 正常返回)。
This commit is contained in:
@@ -0,0 +1,193 @@
|
||||
"""
|
||||
agentos backup / restore — 备份与还原 agent-local/
|
||||
|
||||
备份内容:
|
||||
- materials/ (采集的原始素材)
|
||||
- memory/raw/ (L3 记忆原文)
|
||||
+ 备份清单 manifest.json
|
||||
|
||||
不备份:
|
||||
- vector_db/ (可重建)
|
||||
- runtime/cache/ (临时缓存)
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import tarfile
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
from .utils import get_sync_root, get_local_root, info, ok, warn, err, banner
|
||||
|
||||
|
||||
def build_manifest(local_root: Path, backup_dirs: list) -> dict:
|
||||
"""生成备份清单"""
|
||||
manifest = {
|
||||
"created_at": datetime.now().isoformat(),
|
||||
"hostname": __import__("platform").node(),
|
||||
"agentos_version": "1.0.0",
|
||||
"items": [],
|
||||
}
|
||||
total_size = 0
|
||||
entry_count = 0
|
||||
for rel in backup_dirs:
|
||||
path = local_root / rel
|
||||
if not path.exists():
|
||||
continue
|
||||
if path.is_dir():
|
||||
for f in path.rglob("*"):
|
||||
if f.is_file() and f.name != ".DS_Store":
|
||||
stat = f.stat()
|
||||
total_size += stat.st_size
|
||||
entry_count += 1
|
||||
manifest["items"].append({
|
||||
"path": str(f.relative_to(local_root)),
|
||||
"size": stat.st_size,
|
||||
"mtime": datetime.fromtimestamp(stat.st_mtime).isoformat(),
|
||||
})
|
||||
else:
|
||||
stat = path.stat()
|
||||
total_size += stat.st_size
|
||||
entry_count += 1
|
||||
manifest["items"].append({
|
||||
"path": str(path.relative_to(local_root)),
|
||||
"size": stat.st_size,
|
||||
"mtime": datetime.fromtimestamp(stat.st_mtime).isoformat(),
|
||||
})
|
||||
|
||||
manifest["total_size"] = total_size
|
||||
manifest["entry_count"] = entry_count
|
||||
return manifest
|
||||
|
||||
|
||||
def cmd_backup(output_path: str = None, full: bool = False):
|
||||
"""执行备份"""
|
||||
banner()
|
||||
local_root = get_local_root()
|
||||
hostname = __import__("platform").node()
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
|
||||
if not local_root.exists():
|
||||
err(f"agent-local 目录不存在: {local_root}")
|
||||
sys.exit(1)
|
||||
|
||||
# 确定备份目录
|
||||
if full:
|
||||
sync_root = get_sync_root()
|
||||
backup_dirs = ["."] # 打包整个 agent-local/
|
||||
archive_name = f"agentos_full_{hostname}_{timestamp}.tar.gz"
|
||||
else:
|
||||
backup_dirs = ["materials", "memory/raw"]
|
||||
archive_name = f"agentos_local_{hostname}_{timestamp}.tar.gz"
|
||||
|
||||
# 输出路径
|
||||
if output_path:
|
||||
out_dir = Path(output_path)
|
||||
else:
|
||||
out_dir = Path.cwd()
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
archive_path = out_dir / archive_name
|
||||
|
||||
# 构建 manifest
|
||||
info("扫描备份内容...")
|
||||
manifest = build_manifest(local_root, backup_dirs)
|
||||
|
||||
if manifest["entry_count"] == 0:
|
||||
warn("没有需要备份的文件")
|
||||
return
|
||||
|
||||
print(f" 📊 待备份: {manifest['entry_count']} 个文件, {manifest['total_size'] / 1024 / 1024:.1f} MB")
|
||||
print(f" 📁 来源: {local_root}")
|
||||
|
||||
# 创建 tar.gz
|
||||
info("打包中...")
|
||||
with tarfile.open(str(archive_path), "w:gz") as tar:
|
||||
# 添加 manifest
|
||||
manifest_bytes = json.dumps(manifest, indent=2, ensure_ascii=False).encode("utf-8")
|
||||
import io
|
||||
manifest_file = io.BytesIO(manifest_bytes)
|
||||
tarinfo = tarfile.TarInfo(name="manifest.json")
|
||||
tarinfo.size = len(manifest_bytes)
|
||||
tar.addfile(tarinfo, manifest_file)
|
||||
|
||||
# 添加备份目录
|
||||
for rel in backup_dirs:
|
||||
path = local_root / rel
|
||||
if path.exists():
|
||||
tar.add(str(path), arcname=rel)
|
||||
|
||||
file_size_mb = archive_path.stat().st_size / 1024 / 1024
|
||||
ok(f"备份完成: {archive_path} ({file_size_mb:.1f} MB)")
|
||||
info(f"还原命令: agentos restore {archive_path}")
|
||||
print()
|
||||
|
||||
|
||||
def run_restore(args):
|
||||
"""执行还原"""
|
||||
banner()
|
||||
backup_path = Path(args.backup_path)
|
||||
if not backup_path.exists():
|
||||
err(f"备份文件不存在: {backup_path}")
|
||||
sys.exit(1)
|
||||
|
||||
local_root = get_local_root()
|
||||
force = args.force
|
||||
|
||||
info(f"从备份还原: {backup_path}")
|
||||
info(f"目标目录: {local_root}")
|
||||
|
||||
# 读取 manifest
|
||||
manifest = None
|
||||
with tarfile.open(str(backup_path), "r:gz") as tar:
|
||||
try:
|
||||
mf = tar.extractfile("manifest.json")
|
||||
if mf:
|
||||
manifest = json.loads(mf.read().decode("utf-8"))
|
||||
except KeyError:
|
||||
pass
|
||||
|
||||
if manifest:
|
||||
print(f" 📊 备份信息: {manifest['entry_count']} 个文件, {manifest['total_size'] / 1024 / 1024:.1f} MB")
|
||||
print(f" 🖥️ 来源机器: {manifest.get('hostname', '?')}")
|
||||
print(f" 🕐 备份时间: {manifest.get('created_at', '?')}")
|
||||
else:
|
||||
info(" 备份文件无 manifest.json(旧版本备份)")
|
||||
|
||||
# 冲突检查
|
||||
conflicts = []
|
||||
if manifest and not force:
|
||||
for item in manifest["items"]:
|
||||
dest = local_root / item["path"]
|
||||
if dest.exists():
|
||||
conflicts.append(item["path"])
|
||||
|
||||
if conflicts:
|
||||
print(f"\n⚠️ 发现 {len(conflicts)} 个已存在的文件: (使用 --force 覆盖)")
|
||||
for c in conflicts[:10]:
|
||||
print(f" {c}")
|
||||
if len(conflicts) > 10:
|
||||
print(f" ... 还有 {len(conflicts) - 10} 个")
|
||||
print()
|
||||
|
||||
# 确认
|
||||
if not force:
|
||||
reply = input("是否继续还原?(y/N): ").strip().lower()
|
||||
if reply != "y":
|
||||
info("已取消")
|
||||
return
|
||||
|
||||
# 执行还原
|
||||
info("还原中...")
|
||||
with tarfile.open(str(backup_path), "r:gz") as tar:
|
||||
tar.extractall(path=str(local_root), filter="data")
|
||||
|
||||
ok("还原完成")
|
||||
info("后续操作:")
|
||||
info(" 1. 运行 agentos check 验证状态")
|
||||
info(" 2. 如果向量库未一起还原,运行 agentos init 重建")
|
||||
print()
|
||||
|
||||
|
||||
def run(args):
|
||||
cmd_backup(output_path=args.output, full=args.full)
|
||||
Reference in New Issue
Block a user