feat(monitor): 抖音接入博主监控
上游爬虫本身不缺抖音能力(三模式、四项指标、二级评论都与小红书对等、指标还是同名同列),
缺的全在监控层的适配。这次把「平台之间不一样」的管子集中到一个新模块,再把散落的
xhs 硬编码接上去。
* 新增 api/monitor/adapters.py:产物目录名、jsonl 字段别名、目标链接形态与正则、
通知链接模板。不放进 platforms.py 是因为那个模块被 describe_all() 整个序列化进
/api/config/platforms 交给前端,塞进正则和目录名会让爬虫内部细节漏进 API 载荷。
代价是两个注册表可能漂移,用一条测试钉住「声明接通就必须有适配器」。
* 两个必须知道的坑,都在这版里处理掉了:
1) 抖音的平台 id 是 dy,而 store 把产物写在 douyin/ 下(store/douyin/_store_impl.py:47)。
不改就是 ingest 一个文件都读不到 —— 不报错,只是 0 条,然后被冒充成「疑似登录失效」。
2) 抖音的作品没有 note_id(叫 aweme_id)、评论也用 aweme_id 指作品。ingest 第一步是
`if not note_id: continue`,不映射就逐条全丢。
另外抖音顶层评论的 parent_comment_id 是字符串 "0",归一成空串,免得前端多出悬空的父节点。
* 顺带把「东西抓到了、只是没落在期望目录里」单独识别出来。这类故障的现象和登录失效
一模一样,按登录失效报会把人指去查完全错误的方向。
* 修两个既有 bug(今天只有小红书所以无害,加抖音就踩响):
- service.py update_task 换目标时漏传 task.platform,回落到默认小红书
- scheduler.py 取 cookie 没传 platform,抖音任务会读着小红书那份 cookie 不动
* 行为变更(已与用户确认):cookie 闸门改成「没 cookie 且没开 CDP」才跳过。
CDP 模式下登录态来自被接管的浏览器,粘不粘 cookie 由不得它决定;不放行的话,
选了「接管已有 Chrome」却没粘 cookie 的用户会看到任务永远不触发,而且不报错。
副作用是开启了 CDP 的小红书任务也不再被该闸门拦住 —— 语义上是对的。
* 目标输入框的示例链接与措辞改由能力矩阵提供(notes_label 抖音说「作品」、小红书说
「笔记」;「建议只填纯 ID」是小红书专属劝告,抖音链接不带令牌,不再显示)。
测试 +22 条(858 通过),其中最关键的是「抖音作品/评论不被静默丢弃」与「产物目录名
不等于平台 id」两条 —— 都是把最难查的失败模式钉死在回归网里。
注意:抖音这条路的**端到端尚未验证**,需要一份可用的抖音登录态(CDP 那台 Chrome 里
登录,或导出一份 cookie)。单测覆盖的是解析与入库,真实抓取还没跑过。
This commit is contained in:
+24
-19
@@ -19,7 +19,6 @@
|
||||
"""Task CRUD and dashboard queries for the monitoring layer."""
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
@@ -28,7 +27,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from tools.time_util import get_current_timestamp
|
||||
|
||||
from . import app_settings, covers, platforms, schedule
|
||||
from . import adapters, app_settings, covers, platforms, schedule
|
||||
from .db import get_session
|
||||
from .platforms import PLATFORM_XHS
|
||||
from .models import (
|
||||
@@ -75,11 +74,8 @@ def _require_clock_fields(mode: str, hours: list, days: list) -> None:
|
||||
if mode == schedule.MODE_WEEKLY and not days:
|
||||
raise ValueError("按周调度至少要选一个星期")
|
||||
|
||||
_CREATOR_URL_RE = re.compile(r"xiaohongshu\.com/user/profile/([A-Za-z0-9_-]+)")
|
||||
_NOTE_URL_RE = re.compile(r"xiaohongshu\.com/(?:explore|discovery/item)/([A-Za-z0-9_-]+)")
|
||||
# XHS user ids and note ids are 24-char hex; allow a slightly wider range so a
|
||||
# format change degrades into "still accepted" rather than "rejected".
|
||||
_BARE_ID_RE = re.compile(r"^[A-Za-z0-9_-]{8,64}$")
|
||||
# 各平台的链接形态、id 形状、短链域名都在 adapters.py —— 那里是「平台之间不一样」
|
||||
# 的东西的唯一出处,所以这里不再留任何平台字面量。
|
||||
|
||||
|
||||
class TargetParseError(ValueError):
|
||||
@@ -95,27 +91,36 @@ def parse_target_input(
|
||||
Storing the id separately from the token is what keeps a long-running task
|
||||
alive: tokens expire, ids do not.
|
||||
|
||||
URL shapes are platform-specific. Only Xiaohongshu is wired, so anything else
|
||||
is rejected here as well as at task creation -- parsing a Douyin link as if it
|
||||
were a Xiaohongshu one would be worse than refusing it.
|
||||
URL shapes are platform-specific and come from ``adapters``. A platform with
|
||||
no adapter is rejected here as well as at task creation -- parsing a Douyin
|
||||
link as if it were a Xiaohongshu one would be worse than refusing it.
|
||||
"""
|
||||
if platform != PLATFORM_XHS:
|
||||
if not adapters.has_adapter(platform):
|
||||
raise TargetParseError(f"暂不支持解析该平台({platform})的目标链接")
|
||||
spec = adapters.adapter(platform)
|
||||
|
||||
raw = (value or "").strip()
|
||||
if not raw:
|
||||
raise TargetParseError("Empty target")
|
||||
|
||||
creator_mode = mode == MODE_CREATOR
|
||||
expected = "博主主页" if creator_mode else "笔记"
|
||||
|
||||
external_id = ""
|
||||
if raw.startswith("http") or "/" in raw:
|
||||
# xhslink.com and other short links are not resolvable without a network
|
||||
# round-trip, so only the direct profile/explore forms are supported.
|
||||
match = _CREATOR_URL_RE.search(raw) if mode == MODE_CREATOR else _NOTE_URL_RE.search(raw)
|
||||
if not match:
|
||||
expected = "博主主页" if mode == MODE_CREATOR else "笔记"
|
||||
if any(host in raw for host in spec.short_link_hosts):
|
||||
# 短链要联网跳一次才知道指向谁,而这里没有网络可跳。明确拒绝好过存一个
|
||||
# 解析不出 id 的值 —— 那会变成一个永远抓不到东西、还不报错的任务。
|
||||
raise TargetParseError(f"{expected}短链无法解析,请粘贴完整链接:{raw}")
|
||||
patterns = spec.creator_url_res if creator_mode else spec.note_url_res
|
||||
for pattern in patterns:
|
||||
match = pattern.search(raw)
|
||||
if match:
|
||||
external_id = match.group(1)
|
||||
break
|
||||
if not external_id:
|
||||
raise TargetParseError(f"无法从链接中解析出{expected} ID:{raw}")
|
||||
external_id = match.group(1)
|
||||
elif _BARE_ID_RE.match(raw):
|
||||
elif (spec.creator_bare_re if creator_mode else spec.note_bare_re).match(raw):
|
||||
external_id = raw
|
||||
else:
|
||||
raise TargetParseError(f"无法识别的目标:{raw}")
|
||||
@@ -260,7 +265,7 @@ async def update_task(session: AsyncSession, task_id: int, payload: Dict[str, An
|
||||
now = get_current_timestamp()
|
||||
seen: set[str] = set()
|
||||
for value in payload["targets"]:
|
||||
parsed = parse_target_input(value, task.mode)
|
||||
parsed = parse_target_input(value, task.mode, task.platform)
|
||||
if parsed["external_id"] in seen:
|
||||
continue
|
||||
seen.add(parsed["external_id"])
|
||||
|
||||
Reference in New Issue
Block a user