feat: 监控面板 / 登录鉴权 / 多平台切换 / MySQL
在上游 MediaCrawler 之上新增一层: - 监控层 api/monitor/ —— 多博主/多笔记的定时采集、指标快照差分、报表、 企业微信通知。每轮采集写入独立目录,差分才成立。 - WebUI 登录鉴权 api/auth.py —— PBKDF2 口令 + 服务端会话,/api 全接口防护。 WebSocket 单独加依赖:BaseHTTPMiddleware 对 ws 作用域直接放行,覆盖不到。 - 全局平台切换 + 能力矩阵 —— 如实区分「爬虫模块支持」与「监控层已接线」, 未接通的平台直接拒绝建任务,而不是静默跑空。 - 监控库改用 MySQL 5.7(可回退 SQLite 供测试):逐表强制 utf8mb4 (服务端与库默认都是 latin1),启动校验所连 schema 以防写错库, 连接池 recycle + pre_ping 应对 MySQL 的 8 小时空闲断连。 修复上游缺陷: - xhs/core.py: 主页抓取失败会跳掉整个博主,导致一条作品都抓不到, 而那份资料只喂给一个空函数。改为尽力而为,失败不中断。 - xhs/login.py: cookie 登录只注入 web_session,冷启动签名会失败。 新增 INJECT_ALL_COOKIES 开关(默认关闭,原有行为不变)。 - requirements.txt: 补上 websockets。它在上游 pyproject.toml 里有声明、 这里漏了,导致 uvicorn 没有 WebSocket 能力,实时日志流从未工作。 改动过的上游文件清单及合并方式见 UPSTREAM.md。 测试:492 passed(另有 1 个既有的 Windows/gbk 上游测试失败,与本改动无关)
This commit is contained in:
+25
-2
@@ -18,7 +18,9 @@
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from ..schemas import CrawlerStartRequest, CrawlerStatusResponse
|
||||
from ..monitor.db import get_session
|
||||
from ..monitor.settings import get_cookie
|
||||
from ..schemas import CrawlerStartRequest, CrawlerStatusResponse, LoginTypeEnum
|
||||
from ..services import crawler_manager
|
||||
|
||||
router = APIRouter(prefix="/crawler", tags=["crawler"])
|
||||
@@ -26,7 +28,28 @@ router = APIRouter(prefix="/crawler", tags=["crawler"])
|
||||
|
||||
@router.post("/start")
|
||||
async def start_crawler(request: CrawlerStartRequest):
|
||||
"""Start crawler task"""
|
||||
"""Start crawler task.
|
||||
|
||||
A cookie login with no cookie supplied falls back to the one stored for the
|
||||
selected platform. The manual crawl and the monitor therefore share a single
|
||||
credential; keeping a second paste field on the crawl page meant it went
|
||||
stale and could silently disagree with the monitor's.
|
||||
"""
|
||||
if (
|
||||
request.login_type == LoginTypeEnum.COOKIE
|
||||
and not request.cookies
|
||||
and not request.cookies_file
|
||||
):
|
||||
async with get_session() as session:
|
||||
stored = await get_cookie(session, request.platform.value)
|
||||
|
||||
if not stored:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="该平台尚未保存 Cookie,请到「设置 → 登录态」配置,或改用扫码登录",
|
||||
)
|
||||
request.cookies = stored
|
||||
|
||||
success = await crawler_manager.start(request)
|
||||
if not success:
|
||||
# Handle concurrent/duplicate requests: if process is already running, return 400 instead of 500
|
||||
|
||||
Reference in New Issue
Block a user