butubb
06718a1351
上游爬虫本身不缺抖音能力（三模式、四项指标、二级评论都与小红书对等、指标还是同名同列），
缺的全在监控层的适配。这次把「平台之间不一样」的管子集中到一个新模块，再把散落的
xhs 硬编码接上去。
* 新增 api/monitor/adapters.py：产物目录名、jsonl 字段别名、目标链接形态与正则、
通知链接模板。不放进 platforms.py 是因为那个模块被 describe_all() 整个序列化进
/api/config/platforms 交给前端，塞进正则和目录名会让爬虫内部细节漏进 API 载荷。
代价是两个注册表可能漂移，用一条测试钉住「声明接通就必须有适配器」。
* 两个必须知道的坑，都在这版里处理掉了：
1) 抖音的平台 id 是 dy，而 store 把产物写在 douyin/ 下（store/douyin/_store_impl.py:47）。
不改就是 ingest 一个文件都读不到 —— 不报错，只是 0 条，然后被冒充成「疑似登录失效」。
2) 抖音的作品没有 note_id（叫 aweme_id）、评论也用 aweme_id 指作品。ingest 第一步是
`if not note_id: continue`，不映射就逐条全丢。
另外抖音顶层评论的 parent_comment_id 是字符串 "0"，归一成空串，免得前端多出悬空的父节点。
* 顺带把「东西抓到了、只是没落在期望目录里」单独识别出来。这类故障的现象和登录失效
一模一样，按登录失效报会把人指去查完全错误的方向。
* 修两个既有 bug（今天只有小红书所以无害，加抖音就踩响）：
- service.py update_task 换目标时漏传 task.platform，回落到默认小红书
- scheduler.py 取 cookie 没传 platform，抖音任务会读着小红书那份 cookie 不动
* 行为变更（已与用户确认）：cookie 闸门改成「没 cookie 且没开 CDP」才跳过。
CDP 模式下登录态来自被接管的浏览器，粘不粘 cookie 由不得它决定；不放行的话，
选了「接管已有 Chrome」却没粘 cookie 的用户会看到任务永远不触发，而且不报错。
副作用是开启了 CDP 的小红书任务也不再被该闸门拦住 —— 语义上是对的。
* 目标输入框的示例链接与措辞改由能力矩阵提供（notes_label 抖音说「作品」、小红书说
「笔记」；「建议只填纯 ID」是小红书专属劝告，抖音链接不带令牌，不再显示）。
测试 +22 条（858 通过），其中最关键的是「抖音作品/评论不被静默丢弃」与「产物目录名
不等于平台 id」两条 —— 都是把最难查的失败模式钉死在回归网里。
注意：抖音这条路的**端到端尚未验证**，需要一份可用的抖音登录态（CDP 那台 Chrome 里
登录，或导出一份 cookie）。单测覆盖的是解析与入库，真实抓取还没跑过。
332 lines
11 KiB
Python
332 lines
11 KiB
Python
# -*- coding: utf-8 -*-
|
|
# Copyright (c) 2025 [email protected]
|
|
#
|
|
# This file is part of MediaCrawler project.
|
|
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/tests/test_monitor_notify.py
|
|
# GitHub: https://github.com/NanmiCoder
|
|
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
|
|
#
|
|
# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
|
|
# 1. 不得用于任何商业用途。
|
|
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
|
|
# 3. 不得进行大规模爬取或对平台造成运营干扰。
|
|
# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
|
|
# 5. 不得用于任何非法或不当的用途。
|
|
#
|
|
# 详细许可条款请参阅项目根目录下的LICENSE文件。
|
|
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
|
|
|
|
"""Tests for the WeCom notification layer.
|
|
|
|
The webhook is stubbed, so nothing here touches the network.
|
|
"""
|
|
|
|
import json
|
|
|
|
import pytest
|
|
import pytest_asyncio
|
|
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
|
from sqlalchemy.pool import StaticPool
|
|
|
|
from api.monitor import notify
|
|
from api.monitor.models import (
|
|
EVENT_AUTH_FAILURE,
|
|
EVENT_METRIC_DELTA,
|
|
EVENT_NEW_NOTE,
|
|
EVENT_NEW_COMMENT_POSTED,
|
|
MODE_CREATOR,
|
|
SETTING_WECOM_WEBHOOK,
|
|
MonitorBase,
|
|
MonitorEvent,
|
|
MonitorRun,
|
|
MonitorTask,
|
|
RUN_SUCCESS,
|
|
)
|
|
from api.monitor.settings import set_setting
|
|
|
|
WEBHOOK = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=abc123"
|
|
|
|
|
|
@pytest_asyncio.fixture
|
|
async def db():
|
|
engine = create_async_engine("sqlite+aiosqlite://", poolclass=StaticPool)
|
|
async with engine.begin() as conn:
|
|
await conn.run_sync(MonitorBase.metadata.create_all)
|
|
factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
|
async with factory() as session:
|
|
yield session
|
|
await engine.dispose()
|
|
|
|
|
|
async def _seed(db: AsyncSession, notify_enabled: bool = True):
|
|
task = MonitorTask(
|
|
name="竞品监控", platform="xhs", mode=MODE_CREATOR, enabled=True,
|
|
interval_minutes=60, max_notes_count=20, enable_comments=True,
|
|
max_comments_count=50, run_timeout_seconds=3600,
|
|
notify_enabled=notify_enabled, created_at=0, updated_at=0,
|
|
)
|
|
db.add(task)
|
|
await db.flush()
|
|
|
|
run = MonitorRun(
|
|
task_id=task.id, trigger="scheduled", status=RUN_SUCCESS, phase=MODE_CREATOR,
|
|
save_data_path="", queued_at=0, not_before=0, max_comments_count=50,
|
|
)
|
|
db.add(run)
|
|
await db.flush()
|
|
return task, run
|
|
|
|
|
|
def _add_event(db, task, run, event_type, title, payload=None, severity="info"):
|
|
db.add(
|
|
MonitorEvent(
|
|
task_id=task.id, run_id=run.id, type=event_type, severity=severity,
|
|
target_kind="note", target_id="note-1", title=title,
|
|
payload_json=json.dumps(payload or {}, ensure_ascii=False), created_at=0,
|
|
)
|
|
)
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Message building
|
|
# --------------------------------------------------------------------------
|
|
|
|
class TestBuildRunMessage:
|
|
@pytest.mark.asyncio
|
|
async def test_no_notifiable_events_means_no_message(self, db):
|
|
task, run = await _seed(db)
|
|
# Metric deltas are not something anyone wants pushed.
|
|
_add_event(db, task, run, EVENT_METRIC_DELTA, "点赞 10→20")
|
|
_add_event(db, task, run, EVENT_NEW_COMMENT_POSTED, "新评论")
|
|
await db.flush()
|
|
|
|
assert await notify.build_run_message(db, task, run) is None
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_new_notes_are_listed_with_links(self, db):
|
|
task, run = await _seed(db)
|
|
_add_event(
|
|
db, task, run, EVENT_NEW_NOTE, "新作品：标题A",
|
|
payload={"note_id": "abc123", "title": "标题A"},
|
|
)
|
|
await db.flush()
|
|
|
|
message = await notify.build_run_message(db, task, run)
|
|
|
|
assert "竞品监控" in message
|
|
assert "新增作品 **1** 篇" in message
|
|
assert "标题A" in message
|
|
assert "https://www.xiaohongshu.com/explore/abc123" in message
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_douyin_notes_link_to_douyin(self, db):
|
|
"""链接形状按平台走 —— 群里点进去该是能看的作品，不是 404。"""
|
|
task, run = await _seed(db)
|
|
task.platform = "dy"
|
|
_add_event(
|
|
db, task, run, EVENT_NEW_NOTE, "新作品：标题A",
|
|
payload={"note_id": "7525082444551310602", "title": "标题A"},
|
|
)
|
|
await db.flush()
|
|
|
|
message = await notify.build_run_message(db, task, run)
|
|
|
|
assert "https://www.douyin.com/video/7525082444551310602" in message
|
|
assert "xiaohongshu.com" not in message
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_long_note_lists_are_truncated(self, db):
|
|
"""A first run can find dozens; a wall of text is worse than a count."""
|
|
task, run = await _seed(db)
|
|
for index in range(14):
|
|
_add_event(
|
|
db, task, run, EVENT_NEW_NOTE, f"新作品：{index}",
|
|
payload={"note_id": f"n{index}", "title": f"标题{index}"},
|
|
)
|
|
await db.flush()
|
|
|
|
message = await notify.build_run_message(db, task, run)
|
|
|
|
assert "新增作品 **14** 篇" in message
|
|
assert "标题0" in message
|
|
assert "标题13" not in message
|
|
assert "等共 14 篇" in message
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_failure_is_reported_as_a_warning(self, db):
|
|
task, run = await _seed(db)
|
|
_add_event(
|
|
db, task, run, EVENT_AUTH_FAILURE,
|
|
"疑似登录态失效：本次未抓到任何作品", severity="error",
|
|
)
|
|
await db.flush()
|
|
|
|
message = await notify.build_run_message(db, task, run)
|
|
|
|
assert "异常" in message
|
|
assert "登录态失效" in message
|
|
assert notify._COLOR_WARNING in message
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_baseline_runs_say_so(self, db):
|
|
task, run = await _seed(db)
|
|
run.is_baseline = True
|
|
_add_event(db, task, run, EVENT_NEW_NOTE, "新作品", payload={"note_id": "x", "title": "t"})
|
|
await db.flush()
|
|
|
|
message = await notify.build_run_message(db, task, run)
|
|
|
|
assert "基线" in message
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# notify_run gating
|
|
# --------------------------------------------------------------------------
|
|
|
|
class TestNotifyRunGating:
|
|
@pytest.mark.asyncio
|
|
async def test_disabled_task_is_skipped(self, db, monkeypatch):
|
|
task, run = await _seed(db, notify_enabled=False)
|
|
_add_event(db, task, run, EVENT_NEW_NOTE, "新作品", payload={"note_id": "x", "title": "t"})
|
|
await set_setting(db, SETTING_WECOM_WEBHOOK, WEBHOOK)
|
|
await db.flush()
|
|
|
|
called = []
|
|
monkeypatch.setattr(notify, "send_wecom", lambda *a, **k: called.append(a) or _ok())
|
|
|
|
assert await notify.notify_run(db, task, run) is None
|
|
assert called == []
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_missing_webhook_is_skipped(self, db, monkeypatch):
|
|
task, run = await _seed(db, notify_enabled=True)
|
|
_add_event(db, task, run, EVENT_NEW_NOTE, "新作品", payload={"note_id": "x", "title": "t"})
|
|
await db.flush()
|
|
|
|
called = []
|
|
monkeypatch.setattr(notify, "send_wecom", lambda *a, **k: called.append(a) or _ok())
|
|
|
|
assert await notify.notify_run(db, task, run) is None
|
|
assert called == []
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_successful_push_records_the_timestamp(self, db, monkeypatch):
|
|
task, run = await _seed(db, notify_enabled=True)
|
|
_add_event(db, task, run, EVENT_NEW_NOTE, "新作品", payload={"note_id": "x", "title": "t"})
|
|
await set_setting(db, SETTING_WECOM_WEBHOOK, WEBHOOK)
|
|
await db.flush()
|
|
|
|
monkeypatch.setattr(notify, "send_wecom", lambda *a, **k: _ok())
|
|
|
|
message = await notify.notify_run(db, task, run)
|
|
|
|
assert message is not None
|
|
# Lets the UI answer "why did I not get a push for this run?".
|
|
assert task.last_notified_at is not None
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_push_failure_never_raises(self, db, monkeypatch):
|
|
"""A broken webhook must not take down the crawl that just succeeded."""
|
|
task, run = await _seed(db, notify_enabled=True)
|
|
_add_event(db, task, run, EVENT_NEW_NOTE, "新作品", payload={"note_id": "x", "title": "t"})
|
|
await set_setting(db, SETTING_WECOM_WEBHOOK, WEBHOOK)
|
|
await db.flush()
|
|
|
|
async def _boom(*args, **kwargs):
|
|
raise RuntimeError("network exploded")
|
|
|
|
monkeypatch.setattr(notify, "send_wecom", _boom)
|
|
|
|
assert await notify.notify_run(db, task, run) is None
|
|
|
|
|
|
async def _ok():
|
|
return True, "发送成功"
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# send_wecom
|
|
# --------------------------------------------------------------------------
|
|
|
|
class _FakeResponse:
|
|
def __init__(self, payload):
|
|
self._payload = payload
|
|
|
|
def raise_for_status(self):
|
|
return None
|
|
|
|
def json(self):
|
|
return self._payload
|
|
|
|
|
|
class _FakeClient:
|
|
"""Captures the request and replays a canned WeCom reply."""
|
|
|
|
last_payload = None
|
|
|
|
def __init__(self, reply=None, error=None):
|
|
self._reply = reply if reply is not None else {"errcode": 0, "errmsg": "ok"}
|
|
self._error = error
|
|
|
|
def __call__(self, *args, **kwargs):
|
|
return self
|
|
|
|
async def __aenter__(self):
|
|
return self
|
|
|
|
async def __aexit__(self, *exc):
|
|
return False
|
|
|
|
async def post(self, url, json=None):
|
|
if self._error:
|
|
raise self._error
|
|
type(self).last_payload = json
|
|
return _FakeResponse(self._reply)
|
|
|
|
|
|
class TestSendWecom:
|
|
@pytest.mark.asyncio
|
|
async def test_missing_url_is_reported(self):
|
|
ok, detail = await notify.send_wecom("", "hi")
|
|
assert ok is False
|
|
assert "未配置" in detail
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_success(self, monkeypatch):
|
|
monkeypatch.setattr(notify.httpx, "AsyncClient", _FakeClient())
|
|
|
|
ok, detail = await notify.send_wecom(WEBHOOK, "**标题**\n> 内容")
|
|
|
|
assert ok is True
|
|
assert detail == "发送成功"
|
|
# WeCom expects a markdown message envelope.
|
|
assert _FakeClient.last_payload["msgtype"] == "markdown"
|
|
assert _FakeClient.last_payload["markdown"]["content"] == "**标题**\n> 内容"
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_nonzero_errcode_is_a_failure(self, monkeypatch):
|
|
"""WeCom answers HTTP 200 even when it rejects the message."""
|
|
monkeypatch.setattr(
|
|
notify.httpx, "AsyncClient",
|
|
_FakeClient(reply={"errcode": 93000, "errmsg": "invalid webhook url"}),
|
|
)
|
|
|
|
ok, detail = await notify.send_wecom(WEBHOOK, "hi")
|
|
|
|
assert ok is False
|
|
assert "93000" in detail
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_network_error_is_returned_not_raised(self, monkeypatch):
|
|
import httpx
|
|
|
|
monkeypatch.setattr(
|
|
notify.httpx, "AsyncClient",
|
|
_FakeClient(error=httpx.ConnectError("boom")),
|
|
)
|
|
|
|
ok, detail = await notify.send_wecom(WEBHOOK, "hi")
|
|
|
|
assert ok is False
|
|
assert "请求失败" in detail
|