feat: 监控面板 / 登录鉴权 / 多平台切换 / MySQL
Deploy VitePress site to Pages / build (push) Canceled after 0s
Deploy VitePress site to Pages / Deploy (push) Canceled after 0s

在上游 MediaCrawler 之上新增一层:

- 监控层 api/monitor/ —— 多博主/多笔记的定时采集、指标快照差分、报表、
  企业微信通知。每轮采集写入独立目录,差分才成立。
- WebUI 登录鉴权 api/auth.py —— PBKDF2 口令 + 服务端会话,/api 全接口防护。
  WebSocket 单独加依赖:BaseHTTPMiddleware 对 ws 作用域直接放行,覆盖不到。
- 全局平台切换 + 能力矩阵 —— 如实区分「爬虫模块支持」与「监控层已接线」,
  未接通的平台直接拒绝建任务,而不是静默跑空。
- 监控库改用 MySQL 5.7(可回退 SQLite 供测试):逐表强制 utf8mb4
  (服务端与库默认都是 latin1),启动校验所连 schema 以防写错库,
  连接池 recycle + pre_ping 应对 MySQL 的 8 小时空闲断连。

修复上游缺陷:

- xhs/core.py: 主页抓取失败会跳掉整个博主,导致一条作品都抓不到,
  而那份资料只喂给一个空函数。改为尽力而为,失败不中断。
- xhs/login.py: cookie 登录只注入 web_session,冷启动签名会失败。
  新增 INJECT_ALL_COOKIES 开关(默认关闭,原有行为不变)。
- requirements.txt: 补上 websockets。它在上游 pyproject.toml 里有声明、
  这里漏了,导致 uvicorn 没有 WebSocket 能力,实时日志流从未工作。

改动过的上游文件清单及合并方式见 UPSTREAM.md。

测试:492 passed(另有 1 个既有的 Windows/gbk 上游测试失败,与本改动无关)
This commit is contained in:
2026-10-07 09:58:40 +08:00
parent 5d547f4586
commit 4e60524f37
88 changed files with 13224 additions and 436 deletions
+34
View File
@@ -0,0 +1,34 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2025 [email protected]
#
# This file is part of MediaCrawler project.
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/schemas/auth.py
# GitHub: https://github.com/NanmiCoder
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
#
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
# 1. 不得用于任何商业用途。
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
# 3. 不得进行大规模爬取或对平台造成运营干扰。
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
# 5. 不得用于任何非法或不当的用途。
#
# 详细许可条款请参阅项目根目录下的LICENSE文件。
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
"""Request models for authentication endpoints."""
from pydantic import BaseModel, Field
# A floor, not a policy: this is a single-operator internal panel, so the goal is
# only to reject obviously weak input.
MIN_PASSWORD_LENGTH = 8
class LoginPayload(BaseModel):
password: str = Field(min_length=1)
class ChangePasswordPayload(BaseModel):
current: str = Field(min_length=1)
new: str = Field(min_length=MIN_PASSWORD_LENGTH, max_length=256)
+28
View File
@@ -78,6 +78,34 @@ class CrawlerStartRequest(BaseModel):
max_notes_count: Optional[int] = Field(default=None, ge=1, le=MAX_API_LIMIT_COUNT)
max_comments_count: Optional[int] = Field(default=None, ge=1, le=MAX_API_LIMIT_COUNT)
# --- Options only used by scheduled monitor runs. Each defaults to None so
# the corresponding CLI flag is omitted entirely for manual Crawl-tab runs,
# which keeps their behaviour byte-identical to before.
# Isolate this run's output in its own directory. The crawler's own file
# writer names files by date only, so same-day runs would otherwise append
# into one shared file and could not be told apart.
save_data_path: Optional[str] = None
# Unattended runs must not try to attach to the user's desktop Chrome.
enable_cdp_mode: Optional[bool] = None
# XHS cookie login only injects `web_session` by default, which is not enough
# to sign API requests from a cold browser profile.
inject_all_cookies: Optional[bool] = None
save_login_state: Optional[bool] = None
# Preferred over `cookies`: a value on the command line is visible in the
# process list.
cookies_file: Optional[str] = None
max_concurrency_num: Optional[int] = Field(default=None, ge=1, le=MAX_API_LIMIT_COUNT)
# Crawl-strategy and proxy knobs surfaced on the Settings page. Like the
# fields above, each stays None unless the caller sets it, so the CLI flag is
# omitted entirely and the config-file default applies.
crawler_max_sleep_sec: Optional[int] = Field(default=None, ge=0, le=600)
enable_ip_proxy: Optional[bool] = None
ip_proxy_pool_count: Optional[int] = Field(default=None, ge=1, le=100)
ip_proxy_provider_name: Optional[str] = None
static_proxy_url: Optional[str] = None
class CrawlerStatusResponse(BaseModel):
"""Crawler status response"""
+81
View File
@@ -0,0 +1,81 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2025 [email protected]
#
# This file is part of MediaCrawler project.
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/schemas/monitor.py
# GitHub: https://github.com/NanmiCoder
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
#
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
# 1. 不得用于任何商业用途。
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
# 3. 不得进行大规模爬取或对平台造成运营干扰。
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
# 5. 不得用于任何非法或不当的用途。
#
# 详细许可条款请参阅项目根目录下的LICENSE文件。
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
"""Request models for the monitoring API."""
from typing import List, Literal, Optional
from pydantic import BaseModel, Field
# A floor on the interval is a correctness guard, not a nicety: every run
# launches a browser and hits XHS with several requests, so a short interval
# across many creators is the pattern that triggers rate limiting.
MIN_INTERVAL_MINUTES = 30
MAX_INTERVAL_MINUTES = 7 * 24 * 60
class MonitorTaskCreate(BaseModel):
name: str = Field(min_length=1, max_length=200)
platform: str = "xhs"
# One subprocess handles exactly one crawler type, so a task is either
# creator-driven or note-driven.
mode: Literal["creator", "note"]
# None means "use the value configured on the Settings page", which is what
# makes those defaults meaningful. Bounds still apply when a value is given.
interval_minutes: Optional[int] = Field(
default=None, ge=MIN_INTERVAL_MINUTES, le=MAX_INTERVAL_MINUTES
)
max_notes_count: Optional[int] = Field(default=None, ge=1, le=500)
enable_comments: bool = True
# Raising this widens the comment window, which is the only lever available
# for noticing new comments -- the API has no time-sort.
max_comments_count: Optional[int] = Field(default=None, ge=1, le=500)
run_timeout_seconds: int = Field(default=3600, ge=60, le=86400)
enabled: bool = True
# Push a WeCom summary for runs that failed or found new works. Opt-in per
# task so a single webhook does not get flooded.
notify_enabled: bool = False
# Raw pasted values: full URLs or bare ids, in either form.
targets: List[str] = Field(min_length=1)
class MonitorTaskUpdate(BaseModel):
name: Optional[str] = Field(default=None, min_length=1, max_length=200)
enabled: Optional[bool] = None
interval_minutes: Optional[int] = Field(
default=None, ge=MIN_INTERVAL_MINUTES, le=MAX_INTERVAL_MINUTES
)
max_notes_count: Optional[int] = Field(default=None, ge=1, le=500)
enable_comments: Optional[bool] = None
max_comments_count: Optional[int] = Field(default=None, ge=1, le=500)
run_timeout_seconds: Optional[int] = Field(default=None, ge=60, le=86400)
notify_enabled: Optional[bool] = None
# When present, replaces the whole target list.
targets: Optional[List[str]] = None
class CookiePayload(BaseModel):
cookie: str = Field(min_length=1)
class WebhookPayload(BaseModel):
url: str = Field(default="", description="企业微信机器人 Webhook 地址,留空表示停用")
class WebhookTestPayload(BaseModel):
url: Optional[str] = Field(default=None, description="不传则使用已保存的地址")
+37
View File
@@ -0,0 +1,37 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2025 [email protected]
#
# This file is part of MediaCrawler project.
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/schemas/settings.py
# GitHub: https://github.com/NanmiCoder
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
#
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
# 1. 不得用于任何商业用途。
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
# 3. 不得进行大规模爬取或对平台造成运营干扰。
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
# 5. 不得用于任何非法或不当的用途。
#
# 详细许可条款请参阅项目根目录下的LICENSE文件。
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
"""Request model for the settings endpoint."""
from typing import Any, Dict
from pydantic import BaseModel, ConfigDict
class SettingsUpdatePayload(BaseModel):
"""Partial update of arbitrary setting keys.
Fields are not declared here on purpose: ``api/monitor/app_settings.py``
owns the registry (key, type, bounds, choices) and validates against it, so
adding a setting does not mean editing a matching schema.
"""
model_config = ConfigDict(extra="allow")
def values(self) -> Dict[str, Any]:
return dict(self.model_extra or {})