feat: 监控面板 / 登录鉴权 / 多平台切换 / MySQL
Deploy VitePress site to Pages / build (push) Canceled after 0s
Deploy VitePress site to Pages / Deploy (push) Canceled after 0s

在上游 MediaCrawler 之上新增一层:

- 监控层 api/monitor/ —— 多博主/多笔记的定时采集、指标快照差分、报表、
  企业微信通知。每轮采集写入独立目录,差分才成立。
- WebUI 登录鉴权 api/auth.py —— PBKDF2 口令 + 服务端会话,/api 全接口防护。
  WebSocket 单独加依赖:BaseHTTPMiddleware 对 ws 作用域直接放行,覆盖不到。
- 全局平台切换 + 能力矩阵 —— 如实区分「爬虫模块支持」与「监控层已接线」,
  未接通的平台直接拒绝建任务,而不是静默跑空。
- 监控库改用 MySQL 5.7(可回退 SQLite 供测试):逐表强制 utf8mb4
  (服务端与库默认都是 latin1),启动校验所连 schema 以防写错库,
  连接池 recycle + pre_ping 应对 MySQL 的 8 小时空闲断连。

修复上游缺陷:

- xhs/core.py: 主页抓取失败会跳掉整个博主,导致一条作品都抓不到,
  而那份资料只喂给一个空函数。改为尽力而为,失败不中断。
- xhs/login.py: cookie 登录只注入 web_session,冷启动签名会失败。
  新增 INJECT_ALL_COOKIES 开关(默认关闭,原有行为不变)。
- requirements.txt: 补上 websockets。它在上游 pyproject.toml 里有声明、
  这里漏了,导致 uvicorn 没有 WebSocket 能力,实时日志流从未工作。

改动过的上游文件清单及合并方式见 UPSTREAM.md。

测试:492 passed(另有 1 个既有的 Windows/gbk 上游测试失败,与本改动无关)
This commit is contained in:
2026-10-07 09:58:40 +08:00
parent 5d547f4586
commit 4e60524f37
88 changed files with 13224 additions and 436 deletions
+302
View File
@@ -0,0 +1,302 @@
import { useEffect, useMemo, useState } from 'react'
import { Download, FilePlus2, MessageSquarePlus, ThumbsUp, Star, TriangleAlert } from 'lucide-react'
import { Button } from '@/components/ui/button'
import { Checkbox } from '@/components/ui/checkbox'
import { useMonitorTasks, useReport } from '@/hooks/useMonitor'
import { useCurrentPlatform } from '@/hooks/usePlatform'
import { UnwiredPlatformNotice } from '@/components/layout/UnwiredPlatformNotice'
import { monitorApi } from '@/lib/api'
import { formatCount } from '@/lib/monitorFormat'
import type { MetricKey, ReportRow } from '@/types/monitor'
import { StatTile } from './StatTile'
/** Local-date ISO string; toISOString() would shift across the UTC boundary. */
function toISODate(value: Date): string {
const pad = (n: number) => String(n).padStart(2, '0')
return `${value.getFullYear()}-${pad(value.getMonth() + 1)}-${pad(value.getDate())}`
}
const METRIC_COLUMNS: Array<{ key: MetricKey; label: string }> = [
{ key: 'liked_count_delta', label: '点赞 Δ' },
{ key: 'comment_count_delta', label: '评论 Δ' },
{ key: 'collected_count_delta', label: '收藏 Δ' },
{ key: 'share_count_delta', label: '分享 Δ' },
]
/** Neutral for zero, since most rows are zero and colouring them all is noise. */
function DeltaCell({ value }: { value: number }) {
if (value === 0) return <span className="text-cyber-text-muted">—</span>
return (
<span className={value > 0 ? 'text-cyber-neon-green' : 'text-cyber-neon-pink'}>
{value > 0 ? '+' : ''}
{formatCount(value)}
</span>
)
}
export function ReportView() {
const { isWired, isLoading: platformLoading } = useCurrentPlatform()
const { data: tasks } = useMonitorTasks()
const [selected, setSelected] = useState<number[] | null>(null)
const [endDate, setEndDate] = useState(() => toISODate(new Date()))
const [startDate, setStartDate] = useState(() => {
const d = new Date()
d.setDate(d.getDate() - 6)
return toISODate(d)
})
// Default to every task once the list arrives; the user narrows from there.
useEffect(() => {
if (tasks && selected === null) {
setSelected(tasks.map((task) => task.id))
}
}, [tasks, selected])
const activeTaskIds = selected ?? []
const { data: report, isLoading } = useReport(activeTaskIds, startDate, endDate)
const allSelected = useMemo(
() => Boolean(tasks?.length) && activeTaskIds.length === tasks?.length,
[tasks, activeTaskIds],
)
const toggleTask = (id: number) => {
setSelected((current) => {
const base = current ?? []
return base.includes(id) ? base.filter((value) => value !== id) : [...base, id]
})
}
const applyPreset = (days: number) => {
const end = new Date()
const start = new Date()
start.setDate(start.getDate() - (days - 1))
setStartDate(toISODate(start))
setEndDate(toISODate(end))
}
if (!platformLoading && !isWired) {
return <UnwiredPlatformNotice area="报表" />
}
const totals = report?.totals
const partialLabels = (report?.partial_metrics ?? [])
.map((field) => report?.metric_labels?.[field] ?? field)
.join('、')
return (
<div className="flex-1 flex flex-col gap-3 overflow-hidden min-h-0 relative z-10">
{/* Filters */}
<div className="rounded-lg glass-panel float-panel p-3 flex-shrink-0 space-y-3">
<div className="flex items-center justify-between gap-3 flex-wrap">
<div className="flex items-center gap-2">
<span className="font-mono text-xs tracking-wider text-cyber-text-primary">
统计范围
</span>
<div className="flex items-center gap-1">
<input
type="date"
value={startDate}
onChange={(event) => setStartDate(event.target.value)}
className="rounded-md border border-cyber-border-DEFAULT bg-cyber-bg-tertiary px-2 py-1 font-mono text-[11px] text-cyber-text-primary outline-none focus-visible:border-cyber-neon-cyan/50"
/>
<span className="text-cyber-text-muted text-[11px] font-mono">至</span>
<input
type="date"
value={endDate}
onChange={(event) => setEndDate(event.target.value)}
className="rounded-md border border-cyber-border-DEFAULT bg-cyber-bg-tertiary px-2 py-1 font-mono text-[11px] text-cyber-text-primary outline-none focus-visible:border-cyber-neon-cyan/50"
/>
</div>
</div>
<div className="flex items-center gap-1">
{[7, 30, 90].map((days) => (
<Button key={days} variant="outline" size="sm" onClick={() => applyPreset(days)}>
近 {days} 天
</Button>
))}
<Button
variant="outline"
size="sm"
onClick={() =>
window.open(
monitorApi.getExportUrl({
kind: 'report',
format: 'csv',
taskIds: activeTaskIds,
startDate,
endDate,
}),
'_blank',
)
}
>
<Download className="w-3 h-3 mr-1" />
导出
</Button>
</div>
</div>
<div className="space-y-2">
<div className="flex items-center justify-between">
<span className="text-[11px] font-mono text-cyber-text-secondary">
参与统计的任务(已选{' '}
<span className="text-cyber-neon-cyan">{activeTaskIds.length}</span> 个)
</span>
<Button
variant="ghost"
size="sm"
onClick={() =>
setSelected(allSelected ? [] : (tasks ?? []).map((task) => task.id))
}
disabled={!tasks?.length}
>
{allSelected ? '全不选' : '全选'}
</Button>
</div>
{tasks?.length ? (
<div className="flex flex-wrap gap-2">
{tasks.map((task) => {
const checked = activeTaskIds.includes(task.id)
return (
<label
key={task.id}
className={`flex items-center gap-1.5 px-2 py-1 rounded-md border cursor-pointer transition-colors ${
checked
? 'border-cyber-neon-cyan/40 bg-cyber-neon-cyan/10'
: 'border-cyber-border-subtle bg-cyber-bg-tertiary'
}`}
>
<Checkbox
checked={checked}
onCheckedChange={() => toggleTask(task.id)}
/>
<span
className={`text-[11px] font-mono ${
checked ? 'text-cyber-neon-cyan' : 'text-cyber-text-muted'
}`}
>
{task.name}
</span>
</label>
)
})}
</div>
) : (
<p className="text-[11px] font-mono text-cyber-text-muted">
还没有监控任务,先去「监控」页建一个。
</p>
)}
</div>
</div>
{/* Headline totals */}
<div className="grid grid-cols-2 lg:grid-cols-4 gap-2 flex-shrink-0">
<StatTile icon={FilePlus2} label="新增作品" value={formatCount(totals?.new_notes ?? 0)} />
<StatTile
icon={MessageSquarePlus}
label="新增评论"
value={formatCount(totals?.new_comments ?? 0)}
/>
<StatTile
icon={ThumbsUp}
label="点赞增量"
value={formatCount(totals?.liked_count_delta ?? 0)}
hint={`覆盖 ${report?.note_count ?? 0} 篇作品`}
/>
<StatTile
icon={Star}
label="收藏增量"
value={formatCount(totals?.collected_count_delta ?? 0)}
/>
</div>
{/* Daily breakdown */}
<div className="flex-1 overflow-auto terminal-scroll rounded-lg glass-panel float-panel">
{activeTaskIds.length === 0 ? (
<p className="py-10 text-center text-[11px] font-mono text-cyber-text-muted">
请至少选择一个任务
</p>
) : isLoading ? (
<p className="py-10 text-center text-[11px] font-mono text-cyber-text-muted">统计中…</p>
) : (
<table className="w-full text-xs font-mono">
<thead className="sticky top-0 bg-cyber-bg-tertiary">
<tr className="text-cyber-text-secondary text-[10px]">
<th className="text-left font-normal py-2 px-3">日期</th>
<th className="text-right font-normal py-2 px-3">新增作品</th>
<th className="text-right font-normal py-2 px-3">新增评论</th>
{METRIC_COLUMNS.map((column) => (
<th key={column.key} className="text-right font-normal py-2 px-3">
{column.label}
</th>
))}
</tr>
</thead>
<tbody>
{report?.rows.map((row: ReportRow) => (
<tr key={row.date} className="border-t border-cyber-border-subtle">
<td className="py-2 px-3 text-cyber-text-primary">{row.date}</td>
<td className="py-2 px-3 text-right">
<span className={row.new_notes > 0 ? 'text-cyber-neon-green' : 'text-cyber-text-muted'}>
{row.new_notes || '—'}
</span>
</td>
<td className="py-2 px-3 text-right">
<span
className={row.new_comments > 0 ? 'text-cyber-neon-green' : 'text-cyber-text-muted'}
>
{row.new_comments || '—'}
</span>
</td>
{METRIC_COLUMNS.map((column) => (
<td key={column.key} className="py-2 px-3 text-right">
<DeltaCell value={row[column.key]} />
</td>
))}
</tr>
))}
{report && (
<tr className="border-t-2 border-cyber-neon-cyan/30 bg-cyber-bg-tertiary/60">
<td className="py-2 px-3 text-cyber-neon-cyan">合计</td>
<td className="py-2 px-3 text-right text-cyber-neon-cyan">
{report.totals.new_notes}
</td>
<td className="py-2 px-3 text-right text-cyber-neon-cyan">
{report.totals.new_comments}
</td>
{METRIC_COLUMNS.map((column) => (
<td key={column.key} className="py-2 px-3 text-right text-cyber-neon-cyan">
{report.totals[column.key] > 0 ? '+' : ''}
{formatCount(report.totals[column.key])}
</td>
))}
</tr>
)}
</tbody>
</table>
)}
</div>
{/* Honest caveats, stated where the numbers are read. */}
<div className="flex-shrink-0 space-y-1">
{partialLabels && (
<p className="flex items-center gap-1.5 text-[10px] font-mono text-cyber-neon-orange">
<TriangleAlert className="w-3 h-3" />
{partialLabels} 存在无法解析的数值,这些增量未计入统计,实际增幅会更高。
</p>
)}
<p className="text-[10px] font-mono text-cyber-text-muted">
「增量」= 当日末值 − 当日之前最后一次采到的值;作品首次出现的那天从 0 起算,
所以新增作品的全部点赞都计在其首次发现日。<br />
「新增评论」受接口限制只覆盖平台默认排序的前 N 条,沉底的新评论不计入。
</p>
</div>
</div>
)
}