feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程,直接发 HTTP
移植自 mac-agent-os 的 mediacrawler_adapter:不起子进程、不开页面,用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin,所以 ingest 那条链路一个字不用改。
**目前能用的(真环境实测,非推断)**:
profile/other : 200, 7075 字节 —— 博主主页指标(粉丝/获赞/作品数/昵称)
aweme/detail : 200, 45425 字节 —— 单条作品详情(含点赞/评论/收藏/分享)
**目前不能用的:作品列表 `aweme/post`。** 两个互相独立的原因:
1. 这个接口被抖音单独升级成了真校验:不带 x-tt-argus 回 403「Uifid Not Found」,
带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过,「真校验」过不了。
同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的,
挑中的恰好是「批量拉作品列表」这个最敏感的动作。
2. 改走页面截获也不行:CDP 浏览器打开博主主页会落到「验证码中间页」(当天大量探测的
代价,过几小时要重测)。
所以现在的边界是:**已知作品的指标刷新能做,自动发现新作品做不了**。
**排查中控住变量后得到的两条事实**(都写进注释了):
· `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
UA+client hints+Cookie 时是 200 但仅 121 字节的空壳,补上这三个头变 7074 字节。
(我先前猜的 sec-ch-ua 不是关键。)
· 因此 UA 与 client hints 必须**成套地取自同一个浏览器**,所以 BrowserIdentity 一次
从 CDP 取齐 cookie + UA + hints,而不是各自写死。
「200 + 空 body 必须当场报错」也是刻意写死的:放过去它会在下游变成「这个博主没作品」,
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的,还被翻译成「账号被封」。
测试 +11:cookie 解析、请求头成套性(含 uifid 缺失/回退)、产物键名与 store 对齐、
以及 _get 的三条失败路径(空 body / 403 带网关原话 / 正常返回)。
This commit is contained in:
@@ -0,0 +1,191 @@
|
||||
"""
|
||||
browser_helpers.py — 抓取系统的浏览器辅助函数 v2
|
||||
|
||||
用途:
|
||||
当 OpenCLI 抓取失败时,通过真实 Chrome 浏览器降级提取数据。
|
||||
|
||||
设计原则:
|
||||
⛔ 绝不使用 Camoufox(养号专用,Firefox 内核 + 特殊指纹)
|
||||
✅ 使用 Playwright 启动真实 Chrome(Chromium 内核,正常指纹)
|
||||
✅ 支持 headless 和 headed 两种模式
|
||||
✅ 浏览器实例复用,避免反复启动
|
||||
|
||||
调用方式:
|
||||
from services.adapters.browser_helpers import page_evaluate, page_extract
|
||||
|
||||
# 执行 JS 提取数据
|
||||
data = await page_evaluate("https://www.douyin.com/video/xxx", js_code)
|
||||
|
||||
# 按 CSS 选择器提取
|
||||
texts = await page_extract("https://www.douyin.com/video/xxx", ["h1.title", ".desc"])
|
||||
"""
|
||||
import asyncio, json, logging, os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("dashboard.scrape.browser")
|
||||
|
||||
# ── 浏览器单例(复用实例,避免反复启动)──
|
||||
|
||||
_browser = None
|
||||
_lock = asyncio.Lock()
|
||||
_USER_DATA_DIR = os.environ.get(
|
||||
"SCRAPE_CHROME_USER_DATA",
|
||||
str(Path.home() / "workbuddy-agent-os" / "agent-local" / "runtime" / "scrape_chrome_profile"),
|
||||
)
|
||||
|
||||
# Chrome 可执行路径
|
||||
_CHROME_PATHS = [
|
||||
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", # macOS 标准路径
|
||||
"/Applications/Chromium.app/Contents/MacOS/Chromium",
|
||||
]
|
||||
_CHROME_PATH = None
|
||||
for p in _CHROME_PATHS:
|
||||
if Path(p).exists():
|
||||
_CHROME_PATH = p
|
||||
break
|
||||
|
||||
|
||||
async def _get_browser(headless: bool = True):
|
||||
"""
|
||||
获取/创建浏览器实例(单例)。
|
||||
使用 Playwright 启动真实 Chrome,而非 Camoufox。
|
||||
"""
|
||||
global _browser
|
||||
async with _lock:
|
||||
if _browser is not None and _browser.is_connected():
|
||||
return _browser
|
||||
try:
|
||||
from playwright.async_api import async_playwright
|
||||
except ImportError:
|
||||
logger.error("Playwright 未安装,无法使用浏览器降级")
|
||||
# 尝试安装
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*["playwright", "install", "chromium"],
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
|
||||
)
|
||||
await proc.communicate()
|
||||
from playwright.async_api import async_playwright
|
||||
except Exception:
|
||||
raise RuntimeError("Playwright 不可用,无法启动浏览器降级")
|
||||
|
||||
p = await async_playwright().start()
|
||||
launch_kwargs = {
|
||||
"headless": headless,
|
||||
"args": [
|
||||
"--disable-blink-features=AutomationControlled",
|
||||
"--no-sandbox",
|
||||
"--disable-dev-shm-usage",
|
||||
"--disable-gpu",
|
||||
"--window-size=1280,720",
|
||||
],
|
||||
}
|
||||
if _CHROME_PATH:
|
||||
launch_kwargs["executable_path"] = _CHROME_PATH
|
||||
logger.info(f"使用系统 Chrome: {_CHROME_PATH}")
|
||||
else:
|
||||
logger.info("使用 Playwright 内置 Chromium")
|
||||
|
||||
try:
|
||||
_browser = await p.chromium.launch(**launch_kwargs)
|
||||
logger.info("✅ 浏览器降级: Chrome 启动成功")
|
||||
except Exception as e:
|
||||
logger.error(f"❌ 浏览器启动失败: {e}")
|
||||
await p.stop()
|
||||
raise
|
||||
return _browser
|
||||
|
||||
|
||||
async def page_evaluate(url: str, js_code: str,
|
||||
timeout: int = 30000, headless: bool = True) -> dict:
|
||||
"""
|
||||
打开 URL,执行 JavaScript 提取数据。
|
||||
|
||||
参数:
|
||||
url: 目标页面地址
|
||||
js_code: 要执行的 JS 代码(字符串形式的箭头函数或普通函数)
|
||||
timeout: 页面加载超时(毫秒)
|
||||
headless: 是否无头模式
|
||||
|
||||
返回:
|
||||
字典。JS 返回值的 JSON 表示。失败时返回 {}。
|
||||
"""
|
||||
browser = await _get_browser(headless=headless)
|
||||
context = None
|
||||
page = None
|
||||
try:
|
||||
context = await browser.new_context(
|
||||
user_agent=(
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/125.0.0.0 Safari/537.36"
|
||||
),
|
||||
viewport={"width": 1280, "height": 720},
|
||||
locale="zh-CN",
|
||||
)
|
||||
page = await context.new_page()
|
||||
await page.goto(url, wait_until="domcontentloaded", timeout=timeout)
|
||||
# 等待页面稳定
|
||||
await page.wait_for_load_state("networkidle", timeout=timeout)
|
||||
# 额外等待,让动态渲染完成
|
||||
await asyncio.sleep(1)
|
||||
|
||||
# 执行 JS
|
||||
result = await page.evaluate(js_code)
|
||||
if result is None:
|
||||
logger.warning(f" ⚠️ page_evaluate JS 返回 null: {url[:60]}")
|
||||
return {}
|
||||
|
||||
# 序列化确保可 JSON 返回
|
||||
if isinstance(result, (dict, list)):
|
||||
return result
|
||||
return {"_value": str(result)}
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f" ⚠️ page_evaluate 失败 [{url[:60]}]: {e}")
|
||||
return {}
|
||||
finally:
|
||||
if page:
|
||||
await page.close()
|
||||
if context:
|
||||
await context.close()
|
||||
|
||||
|
||||
async def page_extract(url: str, selectors: list,
|
||||
timeout: int = 30000, headless: bool = True) -> dict:
|
||||
"""
|
||||
打开 URL,按 CSS 选择器提取 DOM 文本。
|
||||
|
||||
参数:
|
||||
url: 目标页面
|
||||
selectors: CSS 选择器列表,如 ["h1", ".video-title", "[data-e2e=comment-count]"]
|
||||
timeout: 超时(毫秒)
|
||||
headless: 是否无头
|
||||
|
||||
返回:
|
||||
{selector: [text1, text2, ...], ...}
|
||||
"""
|
||||
js = f"""() => {{
|
||||
const results = {{}};
|
||||
const sel = {json.dumps(selectors)};
|
||||
for (const s of sel) {{
|
||||
const els = document.querySelectorAll(s);
|
||||
results[s] = Array.from(els).map(el => el.textContent.trim()).filter(t => t);
|
||||
}}
|
||||
return results;
|
||||
}}"""
|
||||
return await page_evaluate(url, js, timeout=timeout, headless=headless)
|
||||
|
||||
|
||||
async def close_browser():
|
||||
"""关闭浏览器实例(清理时调用)"""
|
||||
global _browser
|
||||
async with _lock:
|
||||
if _browser:
|
||||
try:
|
||||
await _browser.close()
|
||||
logger.info("浏览器降级: Chrome 已关闭")
|
||||
except Exception as e:
|
||||
logger.warning(f"关闭浏览器时出错: {e}")
|
||||
_browser = None
|
||||
Reference in New Issue
Block a user