feat: 监控面板 / 登录鉴权 / 多平台切换 / MySQL
Deploy VitePress site to Pages / build (push) Canceled after 0s
Deploy VitePress site to Pages / Deploy (push) Canceled after 0s

在上游 MediaCrawler 之上新增一层:

- 监控层 api/monitor/ —— 多博主/多笔记的定时采集、指标快照差分、报表、
  企业微信通知。每轮采集写入独立目录,差分才成立。
- WebUI 登录鉴权 api/auth.py —— PBKDF2 口令 + 服务端会话,/api 全接口防护。
  WebSocket 单独加依赖:BaseHTTPMiddleware 对 ws 作用域直接放行,覆盖不到。
- 全局平台切换 + 能力矩阵 —— 如实区分「爬虫模块支持」与「监控层已接线」,
  未接通的平台直接拒绝建任务,而不是静默跑空。
- 监控库改用 MySQL 5.7(可回退 SQLite 供测试):逐表强制 utf8mb4
  (服务端与库默认都是 latin1),启动校验所连 schema 以防写错库,
  连接池 recycle + pre_ping 应对 MySQL 的 8 小时空闲断连。

修复上游缺陷:

- xhs/core.py: 主页抓取失败会跳掉整个博主,导致一条作品都抓不到,
  而那份资料只喂给一个空函数。改为尽力而为,失败不中断。
- xhs/login.py: cookie 登录只注入 web_session,冷启动签名会失败。
  新增 INJECT_ALL_COOKIES 开关(默认关闭,原有行为不变)。
- requirements.txt: 补上 websockets。它在上游 pyproject.toml 里有声明、
  这里漏了,导致 uvicorn 没有 WebSocket 能力,实时日志流从未工作。

改动过的上游文件清单及合并方式见 UPSTREAM.md。

测试:492 passed(另有 1 个既有的 Windows/gbk 上游测试失败,与本改动无关)
This commit is contained in:
2026-10-07 09:58:40 +08:00
parent 5d547f4586
commit 4e60524f37
88 changed files with 13224 additions and 436 deletions
+20 -5
View File
@@ -201,9 +201,22 @@ class XiaoHongShuCrawler(AbstractCrawler):
# Parse creator URL to get user_id and security tokens
creator_info: CreatorUrlInfo = parse_creator_info_from_url(creator_url)
utils.logger.info(f"[XiaoHongShuCrawler.get_creators_and_notes] Parse creator URL info: {creator_info}")
user_id = creator_info.user_id
except ValueError as e:
utils.logger.error(f"[XiaoHongShuCrawler.get_creators_and_notes] Failed to parse creator URL: {e}")
continue
# get creator detail info from web html content
user_id = creator_info.user_id
# Fetching the profile page is best-effort and must not abort the run.
# It only feeds save_creator(), which is a no-op in this build, while
# the notes themselves come from a completely different endpoint.
# Scraping the profile means parsing window.__INITIAL_STATE__ out of
# HTML, which fails whenever the platform serves a different page --
# a JSONDecodeError there is especially misleading because it is a
# ValueError subclass, so it used to be reported as "failed to parse
# creator URL" and then skipped the creator entirely, yielding zero
# notes for a perfectly valid target.
try:
createor_info: Dict = await self.xhs_client.get_creator_info(
user_id=user_id,
xsec_token=creator_info.xsec_token,
@@ -211,9 +224,6 @@ class XiaoHongShuCrawler(AbstractCrawler):
)
if createor_info:
await xhs_store.save_creator(user_id, creator=createor_info)
except ValueError as e:
utils.logger.error(f"[XiaoHongShuCrawler.get_creators_and_notes] Failed to parse creator URL: {e}")
continue
except (IPBlockError, PlatformAccessError) as e:
# Access restricted on the creator homepage, skip this creator instead of crashing the run.
utils.logger.error(
@@ -221,6 +231,11 @@ class XiaoHongShuCrawler(AbstractCrawler):
f"建议降低采集频率、更换 IP 或检查账号状态"
)
continue
except Exception as e:
utils.logger.warning(
f"[XiaoHongShuCrawler.get_creators_and_notes] Could not fetch profile for {user_id} "
f"({type(e).__name__}: {e}); continuing to fetch the creator's notes anyway"
)
# Use fixed crawling interval
crawl_interval = config.CRAWLER_MAX_SLEEP_SEC
+10 -1
View File
@@ -213,8 +213,12 @@ class XiaoHongShuLogin(AbstractLogin):
async def login_by_cookies(self):
"""login xiaohongshu website by cookies"""
utils.logger.info("[XiaoHongShuLogin.login_by_cookies] Begin login xiaohongshu by cookie ...")
injected = 0
for key, value in utils.convert_str_cookie_to_dict(self.cookie_str).items():
if key != "web_session": # Only set web_session cookie attribute
# Default (upstream) behaviour injects only web_session. Unattended runs
# need a1 / webId as well, otherwise signed API calls fail and the run
# exits 0 having fetched nothing -- a silent failure.
if not config.INJECT_ALL_COOKIES and key != "web_session":
continue
await self.browser_context.add_cookies([{
'name': key,
@@ -222,3 +226,8 @@ class XiaoHongShuLogin(AbstractLogin):
'domain': ".rednote.com" if config.XHS_INTERNATIONAL else ".xiaohongshu.com",
'path': "/"
}])
injected += 1
utils.logger.info(
f"[XiaoHongShuLogin.login_by_cookies] Injected {injected} cookie(s), "
f"inject_all_cookies={config.INJECT_ALL_COOKIES}"
)