feat: 监控面板 / 登录鉴权 / 多平台切换 / MySQL
在上游 MediaCrawler 之上新增一层: - 监控层 api/monitor/ —— 多博主/多笔记的定时采集、指标快照差分、报表、 企业微信通知。每轮采集写入独立目录,差分才成立。 - WebUI 登录鉴权 api/auth.py —— PBKDF2 口令 + 服务端会话,/api 全接口防护。 WebSocket 单独加依赖:BaseHTTPMiddleware 对 ws 作用域直接放行,覆盖不到。 - 全局平台切换 + 能力矩阵 —— 如实区分「爬虫模块支持」与「监控层已接线」, 未接通的平台直接拒绝建任务,而不是静默跑空。 - 监控库改用 MySQL 5.7(可回退 SQLite 供测试):逐表强制 utf8mb4 (服务端与库默认都是 latin1),启动校验所连 schema 以防写错库, 连接池 recycle + pre_ping 应对 MySQL 的 8 小时空闲断连。 修复上游缺陷: - xhs/core.py: 主页抓取失败会跳掉整个博主,导致一条作品都抓不到, 而那份资料只喂给一个空函数。改为尽力而为,失败不中断。 - xhs/login.py: cookie 登录只注入 web_session,冷启动签名会失败。 新增 INJECT_ALL_COOKIES 开关(默认关闭,原有行为不变)。 - requirements.txt: 补上 websockets。它在上游 pyproject.toml 里有声明、 这里漏了,导致 uvicorn 没有 WebSocket 能力,实时日志流从未工作。 改动过的上游文件清单及合并方式见 UPSTREAM.md。 测试:492 passed(另有 1 个既有的 Windows/gbk 上游测试失败,与本改动无关)
This commit is contained in:
@@ -0,0 +1,70 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2025 [email protected]
|
||||
#
|
||||
# This file is part of MediaCrawler project.
|
||||
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/services/interpreter.py
|
||||
# GitHub: https://github.com/NanmiCoder
|
||||
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
|
||||
#
|
||||
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
|
||||
# 1. 不得用于任何商业用途。
|
||||
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
|
||||
# 3. 不得进行大规模爬取或对平台造成运营干扰。
|
||||
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
|
||||
# 5. 不得用于任何非法或不当的用途。
|
||||
#
|
||||
# 详细许可条款请参阅项目根目录下的LICENSE文件。
|
||||
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
|
||||
|
||||
"""Interpreter resolution for spawning crawler subprocesses.
|
||||
|
||||
Historically both the crawler manager and the environment check hardcoded
|
||||
``uv run``. ``uv`` is not guaranteed to be installed, so resolve the command
|
||||
prefix in one place: prefer ``uv`` (matching upstream docs), fall back to a
|
||||
project-local virtualenv, and finally to the interpreter running the server.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Project root: api/services/interpreter.py -> services -> api -> repo root
|
||||
PROJECT_ROOT = Path(__file__).parent.parent.parent
|
||||
|
||||
|
||||
def venv_python_path(project_root: Path | None = None) -> Path:
|
||||
"""Return the path to the project venv's Python executable."""
|
||||
root = project_root if project_root is not None else PROJECT_ROOT
|
||||
if sys.platform == "win32":
|
||||
return root / ".venv" / "Scripts" / "python.exe"
|
||||
return root / ".venv" / "bin" / "python"
|
||||
|
||||
|
||||
def resolve_python_cmd(project_root: Path | None = None) -> list[str]:
|
||||
"""Resolve the command prefix used to run ``main.py``.
|
||||
|
||||
Order of preference:
|
||||
1. ``uv`` if it is on PATH -- matches the upstream documented workflow.
|
||||
2. The project-local ``.venv`` if it exists.
|
||||
3. The interpreter currently running the API server.
|
||||
|
||||
Returns a list because the caller appends ``main.py`` and its flags.
|
||||
"""
|
||||
if shutil.which("uv"):
|
||||
return ["uv", "run", "python"]
|
||||
|
||||
venv_python = venv_python_path(project_root)
|
||||
if venv_python.exists():
|
||||
return [str(venv_python)]
|
||||
|
||||
return [sys.executable]
|
||||
|
||||
|
||||
def describe_interpreter(project_root: Path | None = None) -> str:
|
||||
"""Human-readable description of what resolve_python_cmd() picks."""
|
||||
cmd = resolve_python_cmd(project_root)
|
||||
if cmd[0] == "uv":
|
||||
return "uv run python"
|
||||
if cmd[0] == sys.executable:
|
||||
return f"current interpreter ({sys.executable})"
|
||||
return f"project virtualenv ({cmd[0]})"
|
||||
Reference in New Issue
Block a user