feat(monitor): 抖音接入博主监控
上游爬虫本身不缺抖音能力(三模式、四项指标、二级评论都与小红书对等、指标还是同名同列),
缺的全在监控层的适配。这次把「平台之间不一样」的管子集中到一个新模块,再把散落的
xhs 硬编码接上去。
* 新增 api/monitor/adapters.py:产物目录名、jsonl 字段别名、目标链接形态与正则、
通知链接模板。不放进 platforms.py 是因为那个模块被 describe_all() 整个序列化进
/api/config/platforms 交给前端,塞进正则和目录名会让爬虫内部细节漏进 API 载荷。
代价是两个注册表可能漂移,用一条测试钉住「声明接通就必须有适配器」。
* 两个必须知道的坑,都在这版里处理掉了:
1) 抖音的平台 id 是 dy,而 store 把产物写在 douyin/ 下(store/douyin/_store_impl.py:47)。
不改就是 ingest 一个文件都读不到 —— 不报错,只是 0 条,然后被冒充成「疑似登录失效」。
2) 抖音的作品没有 note_id(叫 aweme_id)、评论也用 aweme_id 指作品。ingest 第一步是
`if not note_id: continue`,不映射就逐条全丢。
另外抖音顶层评论的 parent_comment_id 是字符串 "0",归一成空串,免得前端多出悬空的父节点。
* 顺带把「东西抓到了、只是没落在期望目录里」单独识别出来。这类故障的现象和登录失效
一模一样,按登录失效报会把人指去查完全错误的方向。
* 修两个既有 bug(今天只有小红书所以无害,加抖音就踩响):
- service.py update_task 换目标时漏传 task.platform,回落到默认小红书
- scheduler.py 取 cookie 没传 platform,抖音任务会读着小红书那份 cookie 不动
* 行为变更(已与用户确认):cookie 闸门改成「没 cookie 且没开 CDP」才跳过。
CDP 模式下登录态来自被接管的浏览器,粘不粘 cookie 由不得它决定;不放行的话,
选了「接管已有 Chrome」却没粘 cookie 的用户会看到任务永远不触发,而且不报错。
副作用是开启了 CDP 的小红书任务也不再被该闸门拦住 —— 语义上是对的。
* 目标输入框的示例链接与措辞改由能力矩阵提供(notes_label 抖音说「作品」、小红书说
「笔记」;「建议只填纯 ID」是小红书专属劝告,抖音链接不带令牌,不再显示)。
测试 +22 条(858 通过),其中最关键的是「抖音作品/评论不被静默丢弃」与「产物目录名
不等于平台 id」两条 —— 都是把最难查的失败模式钉死在回归网里。
注意:抖音这条路的**端到端尚未验证**,需要一份可用的抖音登录态(CDP 那台 Chrome 里
登录,或导出一份 cookie)。单测覆盖的是解析与入库,真实抓取还没跑过。
This commit is contained in:
+32
-7
@@ -24,6 +24,7 @@ import pytest_asyncio
|
||||
from sqlalchemy import text
|
||||
|
||||
from api.main import app
|
||||
from api.monitor import adapters
|
||||
from api.monitor import db as monitor_db
|
||||
from api.monitor import platforms
|
||||
from api.monitor.models import MonitorTask
|
||||
@@ -54,7 +55,28 @@ class TestCapabilityMatrix:
|
||||
# what stops the UI offering a platform that can never produce data.
|
||||
assert all("monitor_wired" in p for p in body["platforms"])
|
||||
assert by_value["xhs"]["monitor_wired"] is True
|
||||
assert by_value["dy"]["monitor_wired"] is False
|
||||
assert by_value["dy"]["monitor_wired"] is True
|
||||
|
||||
def test_every_wired_platform_has_an_adapter(self):
|
||||
"""能力矩阵说「接通了」,就必须真的有一套适配管子。
|
||||
|
||||
两个注册表(platforms.PLATFORM_CAPABILITIES 与 adapters.ADAPTERS)分开是有意的
|
||||
—— 前者是给前端看的能力描述,后者是爬虫的管道细节。代价是它们可能漂移,
|
||||
所以在这里钉一条:凡声明接通的,必须能找到适配器。
|
||||
"""
|
||||
for platform in platforms.all_platforms():
|
||||
if platforms.is_monitor_wired(platform):
|
||||
assert adapters.has_adapter(platform), f"{platform} 声明接通但没有适配器"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_target_hints_are_exposed_for_wired_platforms(self, client):
|
||||
"""前端的目标输入框拿它做 placeholder —— 让用户看到本平台该粘什么样的链接。"""
|
||||
body = (await client.get("/api/config/platforms")).json()
|
||||
by_value = {p["value"]: p for p in body["platforms"]}
|
||||
|
||||
assert "douyin.com/user/" in by_value["dy"]["target_hints"]["creator"]
|
||||
assert "douyin.com/video/" in by_value["dy"]["target_hints"]["note"]
|
||||
assert "xiaohongshu.com" in by_value["xhs"]["target_hints"]["creator"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_metrics_are_per_platform_and_labelled(self, client):
|
||||
@@ -80,14 +102,17 @@ class TestCapabilityMatrix:
|
||||
class TestTaskCreationGuard:
|
||||
@pytest.mark.asyncio
|
||||
async def test_unwired_platform_is_rejected_with_an_explanation(self, client):
|
||||
"""Accepting it would create a task that silently never produces data."""
|
||||
"""Accepting it would create a task that silently never produces data.
|
||||
|
||||
用 B站 而不是抖音:抖音现在接通了,不再是「已知但未接通」的例子。
|
||||
"""
|
||||
response = await client.post(
|
||||
"/api/monitor/tasks",
|
||||
json={"name": "抖音任务", "mode": "creator", "platform": "dy", "targets": ["x"]},
|
||||
json={"name": "B站任务", "mode": "creator", "platform": "bili", "targets": ["x"]},
|
||||
)
|
||||
assert response.status_code == 400
|
||||
detail = response.json()["detail"]
|
||||
assert "抖音" in detail
|
||||
assert "B站" in detail
|
||||
assert "尚未接通" in detail
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -102,7 +127,7 @@ class TestTaskCreationGuard:
|
||||
async def test_no_task_row_is_created_when_rejected(self, client):
|
||||
await client.post(
|
||||
"/api/monitor/tasks",
|
||||
json={"name": "抖音任务", "mode": "creator", "platform": "dy", "targets": ["x"]},
|
||||
json={"name": "B站任务", "mode": "creator", "platform": "bili", "targets": ["x"]},
|
||||
)
|
||||
assert (await client.get("/api/monitor/tasks")).json()["tasks"] == []
|
||||
|
||||
@@ -126,8 +151,8 @@ class TestTaskCreationGuard:
|
||||
|
||||
class TestPlatformScoping:
|
||||
async def _seed_two_platforms(self, client):
|
||||
"""One real XHS task plus a Douyin task inserted directly, since the API
|
||||
refuses to create the latter."""
|
||||
"""One XHS task created through the API, plus a Douyin task inserted
|
||||
directly so its fields can be pinned exactly."""
|
||||
await client.post(
|
||||
"/api/monitor/tasks",
|
||||
json={"name": "小红书任务", "mode": "creator", "targets": [XHS_TARGET]},
|
||||
|
||||
Reference in New Issue
Block a user