feat(creator): 新增「运营」模块 —— 多账号扫码登录与创作者后台数据
侧边栏在「监控」右边加了「运营」:账号列表 → 点进二级详情看该账号的数据。 【为什么是独立模块而不是监控的子视图】两者形状不同:监控是公开数据(点赞/收藏/评论/分享)的每轮快照+差分;运营是创作者后台按日期给出的曝光/观看/完播率/涨粉。凭据不同、采集方式也不同 —— 那边要浏览器登录态,这边是纯请求。硬塞进同一个模型会同时污染两边。 【扫码登录的关键差异】监控的扫码把登录态写进浏览器默认 profile(爬虫要复用)。运营要的是 cookie 字符串(纯请求够用),所以每次登录开一个**临时上下文**,扫完取出 cookie 就丢弃 —— 登第二个账号不会把第一个顶掉,也不影响监控那个登录态,十个账号互不干扰。 【决策依据】tools/probe_creator_api.py 的 Phase 0 实测:签名可自造(XYW_:MD5 → base64 → AES-128-CBC,与 xhshow 内置实现常量逐字节一致);主站 cookie 即可认证创作者后台;接口与参数已与真实页面对齐。 后端: - api/creator/models.py: creator_account / creator_note_stat。**复用 MonitorBase**,这样 create_all 与上一轮改成元数据驱动的 _ensure_columns 会自动覆盖新表 - api/creator/signing.py: XYW_ 签名,带三条实测结论(url= 前缀、appId=ugc、401 与 406 的区别) - api/creator/client.py: 纯 httpx 客户端。字段名尚未亲眼验证过,所以写成多别名匹配;解析不出来存 None 而非 0 - api/creator/service.py: 账号 CRUD 与同步。cookie 绝不进入对外结构,只给 has_cookie - api/creator/login.py: 临时上下文的扫码登录 - api/routers/creator.py: 8 条路由,全部带鉴权 前端: - 侧边栏「运营」+ OperationView(账号列表 → 二级详情)+ AddAccountDialog - 权限状态显眼呈现:pending 时照抄后台原话「已为您申请数据权限,次日可查看」,并说明此时同步返回 0 条是正常的,不是采集失败 测试:tests/test_creator_client.py 新增 48 例,含「cookie 不得出现在对外结构里」这条不变量,以及权限未生效时空壳响应的处理。
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2025 [email protected]
|
||||
#
|
||||
# This file is part of MediaCrawler project.
|
||||
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/creator/__init__.py
|
||||
# GitHub: https://github.com/NanmiCoder
|
||||
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
|
||||
#
|
||||
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
|
||||
# 1. 不得用于任何商业用途。
|
||||
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
|
||||
# 3. 不得进行大规模爬取或对平台造成运营干扰。
|
||||
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
|
||||
# 5. 不得用于任何非法或不当的用途。
|
||||
#
|
||||
# 详细许可条款请参阅项目根目录下的LICENSE文件。
|
||||
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
|
||||
|
||||
"""运营模块:管理自己的小红书账号,读取创作者后台的数据。
|
||||
|
||||
与 `api.monitor` 是**并列关系**,不是它的扩展。两者数据形状不同:监控是「每轮
|
||||
快照 + 差分」的公开互动数据,这里是创作者后台按日期给出的曝光/观看/完播等运营
|
||||
指标。硬塞进同一个模型会同时污染两边。
|
||||
|
||||
路线是**纯请求**(无浏览器),依据见 tools/probe_creator_api.py 的 Phase 0 实测:
|
||||
签名可自造、主站 cookie 即可认证、接口与参数已与真实页面对齐。
|
||||
"""
|
||||
@@ -0,0 +1,345 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2025 [email protected]
|
||||
#
|
||||
# This file is part of MediaCrawler project.
|
||||
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/creator/client.py
|
||||
# GitHub: https://github.com/NanmiCoder
|
||||
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
|
||||
#
|
||||
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
|
||||
# 1. 不得用于任何商业用途。
|
||||
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
|
||||
# 3. 不得进行大规模爬取或对平台造成运营干扰。
|
||||
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
|
||||
# 5. 不得用于任何非法或不当的用途。
|
||||
#
|
||||
# 详细许可条款请参阅项目根目录下的LICENSE文件。
|
||||
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
|
||||
|
||||
"""小红书创作者后台的纯请求客户端。
|
||||
|
||||
无浏览器:签名在本地算(见 signing.py),请求走 httpx。
|
||||
|
||||
**关于字段名的谨慎**:Phase 0 抓到的响应里,列表接口因为账号权限未生效而返回空壳
|
||||
(`data.result` 只有 `{success, code, message}` 没有数据),所以**真实字段名尚未亲眼
|
||||
见过**。因此每个指标都写成**多别名匹配**,并且解析不出来时存 `None` 而不是 0 ——
|
||||
0 是真实值,None 是"不知道",两者混淆会让报表说谎。
|
||||
"""
|
||||
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
from .signing import sign_xyw, signed_api
|
||||
|
||||
CREATOR_ORIGIN = "https://creator.xiaohongshu.com"
|
||||
DATA_ANALYSIS_PAGE = f"{CREATOR_ORIGIN}/statistics/data-analysis"
|
||||
|
||||
USER_INFO_PATH = "/api/galaxy/user/info"
|
||||
PERMISSION_PATH = "/api/galaxy/creator/datacenter/permission/query"
|
||||
NOTE_LIST_PATH = "/api/galaxy/creator/datacenter/note/analyze/list"
|
||||
|
||||
USER_AGENT = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
|
||||
)
|
||||
|
||||
# 签名被网关拒绝时返回的响应体。区分它和普通业务错误很重要:406 说明签名写错了,
|
||||
# 而应用层的 code/-100 说明签名没问题、只是没有登录态。
|
||||
SIGNATURE_REJECTED_MARKERS = ("code", -1)
|
||||
|
||||
|
||||
class CreatorApiError(RuntimeError):
|
||||
"""调用创作者后台失败。``status`` 用于区分是网关拒绝还是业务错误。"""
|
||||
|
||||
def __init__(self, message: str, status: int = 0, payload: Any = None):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.payload = payload
|
||||
|
||||
|
||||
def trans_cookies(cookie_str: str) -> Dict[str, str]:
|
||||
"""把 cookie 字符串解析成字典。容忍末尾分号、换行和零散空格。"""
|
||||
jar: Dict[str, str] = {}
|
||||
for chunk in re.split(r"[;\n]", cookie_str or ""):
|
||||
chunk = chunk.strip()
|
||||
if not chunk or "=" not in chunk:
|
||||
continue
|
||||
name, value = chunk.split("=", 1)
|
||||
name = name.strip()
|
||||
if name:
|
||||
jar[name] = value.strip()
|
||||
return jar
|
||||
|
||||
|
||||
def _pick(item: Dict[str, Any], *names: str) -> Any:
|
||||
"""按别名顺序取第一个存在的键。
|
||||
|
||||
字段名来自二手资料,未亲眼验证,所以不赌单一命名。
|
||||
"""
|
||||
for name in names:
|
||||
if name in item and item[name] is not None:
|
||||
return item[name]
|
||||
return None
|
||||
|
||||
|
||||
_COUNT_UNITS = {"万": 10_000, "w": 10_000, "W": 10_000, "亿": 100_000_000, "k": 1_000, "K": 1_000}
|
||||
|
||||
|
||||
def as_int(value: Any) -> Optional[int]:
|
||||
"""解析计数。处理 "1.2万"、"1,234"、"123" 与已经是数字的情况。"""
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
|
||||
text = str(value).strip().replace(",", "").replace(" ", "")
|
||||
if not text or text in ("-", "--", "暂无"):
|
||||
return None
|
||||
|
||||
unit = 1
|
||||
suffix = text[-1]
|
||||
if suffix in _COUNT_UNITS:
|
||||
unit = _COUNT_UNITS[suffix]
|
||||
text = text[:-1]
|
||||
try:
|
||||
return int(float(text) * unit)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def as_float(value: Any) -> Optional[float]:
|
||||
"""解析比率。``"12.3%"`` -> 12.3;``"0.123"`` 原样返回数字。"""
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
|
||||
text = str(value).strip().replace("%", "")
|
||||
try:
|
||||
return float(text)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
_DURATION_RE = re.compile(r"(?:(\d+)\s*分)?\s*(?:(\d+)\s*秒)?")
|
||||
|
||||
|
||||
def as_seconds(value: Any) -> Optional[float]:
|
||||
"""解析时长。处理 ``"1分30秒"``、``"01:30"``、``"45"``(秒)。"""
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
|
||||
text = str(value).strip()
|
||||
if not text or text in ("-", "--"):
|
||||
return None
|
||||
|
||||
if ":" in text:
|
||||
parts = text.split(":")
|
||||
try:
|
||||
total = 0.0
|
||||
for part in parts:
|
||||
total = total * 60 + float(part)
|
||||
return total
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
if "分" in text or "秒" in text:
|
||||
minutes = re.search(r"(\d+)\s*分", text)
|
||||
seconds = re.search(r"(\d+)\s*秒", text)
|
||||
if not minutes and not seconds:
|
||||
return None
|
||||
return float(minutes.group(1) if minutes else 0) * 60 + float(
|
||||
seconds.group(1) if seconds else 0
|
||||
)
|
||||
|
||||
try:
|
||||
return float(text)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
# 指标 -> 候选原始字段名。别名来自公开资料,权威与否只能等真实响应来验证。
|
||||
_FIELD_ALIASES: Dict[str, tuple] = {
|
||||
"exposure": ("exposure", "exposure_count", "imp", "impression", "impression_count"),
|
||||
"views": ("views", "view", "view_count", "watch", "watch_count", "read_count"),
|
||||
"likes": ("likes", "like", "like_count", "liked_count"),
|
||||
"comments": ("comments", "comment", "comment_count", "comments_count"),
|
||||
"favorites": ("favorites", "favorite", "favorite_count", "collect", "collect_count", "collected_count"),
|
||||
"shares": ("shares", "share", "share_count", "shared_count"),
|
||||
"new_followers": ("new_followers", "fans_growth", "follower_growth", "increase_fans"),
|
||||
"danmaku": ("danmaku", "danmaku_count", "barrage"),
|
||||
"cover_ctr": ("cover_ctr", "cover_click_rate", "cover_click_ratio", "ctr"),
|
||||
"avg_watch_seconds": ("avg_watch_seconds", "avg_watch_time", "average_watch_time"),
|
||||
"two_second_exit_rate": ("two_second_exit_rate", "2s_exit_rate", "exit_rate_2s"),
|
||||
"completion_rate": ("completion_rate", "finish_rate", "complete_rate"),
|
||||
}
|
||||
|
||||
_COUNT_FIELDS = {
|
||||
"exposure", "views", "likes", "comments", "favorites", "shares", "new_followers", "danmaku",
|
||||
}
|
||||
_DURATION_FIELDS = {"avg_watch_seconds"}
|
||||
_RATE_FIELDS = {"two_second_exit_rate", "completion_rate", "cover_ctr"}
|
||||
|
||||
NOTE_ID_ALIASES = ("note_id", "noteId", "id", "content_id", "item_id")
|
||||
TITLE_ALIASES = ("title", "content", "note_title", "display_title")
|
||||
PUBLISH_TIME_ALIASES = ("publish_time", "publishTime", "post_time", "create_time", "time")
|
||||
|
||||
|
||||
def _normalize_metric(name: str, raw: Any) -> Any:
|
||||
if name in _COUNT_FIELDS:
|
||||
return as_int(raw)
|
||||
if name in _DURATION_FIELDS:
|
||||
return as_seconds(raw)
|
||||
if name in _RATE_FIELDS:
|
||||
return as_float(raw)
|
||||
return raw
|
||||
|
||||
|
||||
def normalize_note(item: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""把一条原始记录规范化成落库用的字段。"""
|
||||
note: Dict[str, Any] = {
|
||||
"note_id": str(_pick(item, *NOTE_ID_ALIASES) or ""),
|
||||
"title": str(_pick(item, *TITLE_ALIASES) or ""),
|
||||
"publish_time": as_int(_pick(item, *PUBLISH_TIME_ALIASES)),
|
||||
}
|
||||
for name, aliases in _FIELD_ALIASES.items():
|
||||
note[name] = _normalize_metric(name, _pick(item, *aliases))
|
||||
return note
|
||||
|
||||
|
||||
def find_note_list(payload: Any) -> List[Dict[str, Any]]:
|
||||
"""在响应里找出笔记数组。
|
||||
|
||||
接口的**确切结构还没亲眼见过**(权限未生效时 `data.result` 里没有数据),
|
||||
所以不写死路径:遍历 JSON,挑出"看起来像一批笔记记录"的那个列表 ——
|
||||
元素是 dict,且至少带一个指标字段。找不到就返回空列表,让上层如实报"没数据"。
|
||||
"""
|
||||
best: List[Dict[str, Any]] = []
|
||||
metric_keys = {alias for aliases in _FIELD_ALIASES.values() for alias in aliases}
|
||||
|
||||
def walk(value: Any) -> None:
|
||||
nonlocal best
|
||||
if isinstance(value, dict):
|
||||
for child in value.values():
|
||||
walk(child)
|
||||
elif isinstance(value, list):
|
||||
if value and isinstance(value[0], dict):
|
||||
keys = set(value[0].keys())
|
||||
if keys & metric_keys and len(value) > len(best):
|
||||
best = value
|
||||
for child in value:
|
||||
walk(child)
|
||||
|
||||
walk(payload)
|
||||
return best
|
||||
|
||||
|
||||
class CreatorClient:
|
||||
"""一个账号的客户端。``cookie`` 就是它的全部身份。"""
|
||||
|
||||
def __init__(self, cookie: str, timeout: float = 25.0):
|
||||
self.cookies = trans_cookies(cookie)
|
||||
self.a1 = self.cookies.get("a1", "")
|
||||
self._timeout = timeout
|
||||
|
||||
@property
|
||||
def looks_authenticated(self) -> bool:
|
||||
"""签名需要 a1;没有它连请求都签不出来。"""
|
||||
return bool(self.a1)
|
||||
|
||||
def _headers(self, api: str, body: dict | None = None) -> Dict[str, str]:
|
||||
cookie_header = "; ".join(f"{k}={v}" for k, v in self.cookies.items())
|
||||
return {
|
||||
"user-agent": USER_AGENT,
|
||||
"accept": "application/json, text/plain, */*",
|
||||
"accept-language": "zh-CN,zh;q=0.9",
|
||||
"origin": CREATOR_ORIGIN,
|
||||
"referer": DATA_ANALYSIS_PAGE,
|
||||
"cookie": cookie_header,
|
||||
**sign_xyw(api, self.a1, body=body),
|
||||
}
|
||||
|
||||
async def _get(self, path: str, params: Dict[str, Any]) -> Dict[str, Any]:
|
||||
if not self.looks_authenticated:
|
||||
raise CreatorApiError("cookie 里没有 a1,无法完成签名", status=0)
|
||||
|
||||
query = "&".join(f"{k}={v}" for k, v in params.items())
|
||||
api = signed_api(path, query)
|
||||
url = f"{CREATOR_ORIGIN}{path}?{query}" if query else f"{CREATOR_ORIGIN}{path}"
|
||||
|
||||
async with httpx.AsyncClient(timeout=self._timeout, follow_redirects=False) as client:
|
||||
response = await client.get(url, headers=self._headers(api))
|
||||
return self._unwrap(response)
|
||||
|
||||
@staticmethod
|
||||
def _unwrap(response: httpx.Response) -> Dict[str, Any]:
|
||||
if response.status_code == 406:
|
||||
raise CreatorApiError(
|
||||
"签名被网关拒绝(406)—— 待签字符串的拼法不对", status=406
|
||||
)
|
||||
try:
|
||||
payload = response.json()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise CreatorApiError(
|
||||
f"响应不是 JSON(HTTP {response.status_code})", status=response.status_code
|
||||
) from exc
|
||||
|
||||
if response.status_code == 401 or payload.get("code") == -100:
|
||||
raise CreatorApiError("登录态无效或已过期", status=401, payload=payload)
|
||||
if not payload.get("success", True):
|
||||
raise CreatorApiError(
|
||||
str(payload.get("msg") or "接口返回失败"),
|
||||
status=response.status_code,
|
||||
payload=payload,
|
||||
)
|
||||
return payload
|
||||
|
||||
async def fetch_user_info(self) -> Dict[str, Any]:
|
||||
"""当前 cookie 属于哪个账号。登录后用它取名与去重。"""
|
||||
payload = await self._get(USER_INFO_PATH, {})
|
||||
data = payload.get("data") or {}
|
||||
return {
|
||||
"user_id": str(data.get("userId") or ""),
|
||||
"nickname": str(data.get("userName") or ""),
|
||||
"avatar": str(data.get("userAvatar") or ""),
|
||||
"red_id": str(data.get("redId") or ""),
|
||||
"role": str(data.get("role") or ""),
|
||||
"permissions": list(data.get("permissions") or []),
|
||||
}
|
||||
|
||||
async def fetch_permission(self) -> Dict[str, Any]:
|
||||
"""数据权限状态。
|
||||
|
||||
``tip_msg`` 是后台原话(实测是"已为您申请数据权限,次日可查看"),照抄不改写 ——
|
||||
这条信息必须原样交给用户,它解释了"为什么没有数据"。
|
||||
"""
|
||||
payload = await self._get(PERMISSION_PATH, {})
|
||||
data = payload.get("data") or {}
|
||||
return {
|
||||
"display": data.get("display"),
|
||||
"status": data.get("status"),
|
||||
"tip": str(data.get("tip_msg") or ""),
|
||||
}
|
||||
|
||||
async def fetch_note_list(
|
||||
self, start_ms: int, end_ms: int, page_num: int = 1, page_size: int = 10, note_type: int = 0
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""按发布时间区间取笔记列表。
|
||||
|
||||
参数与顺序**照抄真实页面的请求**(见 Phase 0 抓包),不要凭感觉改。
|
||||
"""
|
||||
payload = await self._get(
|
||||
NOTE_LIST_PATH,
|
||||
{
|
||||
"post_begin_time": start_ms,
|
||||
"post_end_time": end_ms,
|
||||
"type": note_type,
|
||||
"page_size": page_size,
|
||||
"page_num": page_num,
|
||||
},
|
||||
)
|
||||
return [normalize_note(item) for item in find_note_list(payload)]
|
||||
@@ -0,0 +1,280 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2025 [email protected]
|
||||
#
|
||||
# This file is part of MediaCrawler project.
|
||||
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/creator/login.py
|
||||
# GitHub: https://github.com/NanmiCoder
|
||||
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
|
||||
#
|
||||
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
|
||||
# 1. 不得用于任何商业用途。
|
||||
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
|
||||
# 3. 不得进行大规模爬取或对平台造成运营干扰。
|
||||
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
|
||||
# 5. 不得用于任何非法或不当的用途。
|
||||
#
|
||||
# 详细许可条款请参阅项目根目录下的LICENSE文件。
|
||||
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
|
||||
|
||||
"""运营账号的扫码登录。
|
||||
|
||||
**与监控的扫码登录(`api.monitor.qrlogin`)有一处决定性差异**:那边把登录态写进
|
||||
浏览器**默认 profile**,因为爬虫要复用它;这边要的是 **cookie 字符串**,因为采集
|
||||
走纯 HTTP。所以这里每次登录都开一个**临时上下文**,扫完取出 cookie 就丢弃 ——
|
||||
|
||||
* 登第二个账号不会把第一个顶掉(默认 profile 只能装一个登录态);
|
||||
* 完全不影响监控那个登录态;
|
||||
* 十个账号互不干扰。
|
||||
|
||||
扫码入口仍是主站(`www.xiaohongshu.com`):Phase 0 实测证明**主站的 cookie 就能
|
||||
认证创作者后台**,不需要单独的创作者登录。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from playwright.async_api import async_playwright
|
||||
from tools import utils
|
||||
|
||||
from ..monitor.platforms import PLATFORM_XHS
|
||||
|
||||
QR_TTL_SECONDS = 300
|
||||
|
||||
STATUS_IDLE = "idle"
|
||||
STATUS_WAITING = "waiting"
|
||||
STATUS_SUCCESS = "success"
|
||||
STATUS_EXPIRED = "expired"
|
||||
STATUS_ERROR = "error"
|
||||
|
||||
LOGIN_URL = "https://www.xiaohongshu.com"
|
||||
QR_SELECTOR = "xpath=//img[@class='qrcode-img']"
|
||||
LOGIN_BUTTON_SELECTOR = "xpath=//*[@id='app']/div[1]/div[2]/div[1]/ul/div[1]/button"
|
||||
|
||||
# 与 api/monitor/qrlogin.py 同一套判据:主站在 window.__INITIAL_STATE__ 里报告登录态,
|
||||
# 而 user.loggedIn 是 Vue 的响应式引用,要 .value 解包才是布尔值。
|
||||
LOGIN_STATE_PROBE = """
|
||||
() => {
|
||||
try {
|
||||
const user = (window.__INITIAL_STATE__ || {}).user;
|
||||
if (!user) return JSON.stringify({ known: false });
|
||||
let loggedIn = user.loggedIn;
|
||||
if (loggedIn && typeof loggedIn === 'object' && 'value' in loggedIn) loggedIn = loggedIn.value;
|
||||
let info = null;
|
||||
try { info = user.userInfo || null; } catch (e) { info = null; }
|
||||
const text = (v) => (v === null || v === undefined ? null : String(v));
|
||||
return JSON.stringify({
|
||||
known: true,
|
||||
loggedIn: Boolean(loggedIn),
|
||||
nickname: info ? text(info.nickname) : null
|
||||
});
|
||||
} catch (e) {
|
||||
return JSON.stringify({ known: false, error: String(e) });
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
_lock = asyncio.Lock()
|
||||
_current: Optional["AccountLoginSession"] = None
|
||||
|
||||
_playwright: Any = None
|
||||
|
||||
|
||||
def _cdp_url() -> str:
|
||||
import os
|
||||
|
||||
import config
|
||||
|
||||
return os.getenv("MC_CDP_URL") or f"http://127.0.0.1:{config.CDP_DEBUG_PORT}"
|
||||
|
||||
|
||||
async def _connect():
|
||||
global _playwright
|
||||
if _playwright is None:
|
||||
_playwright = await async_playwright().start()
|
||||
return await _playwright.chromium.connect_over_cdp(_cdp_url(), timeout=15000)
|
||||
|
||||
|
||||
async def _disconnect() -> None:
|
||||
global _playwright
|
||||
if _playwright is not None:
|
||||
try:
|
||||
await _playwright.stop()
|
||||
except Exception:
|
||||
pass
|
||||
_playwright = None
|
||||
|
||||
|
||||
async def _read_qr(page: Any) -> str:
|
||||
image = await utils.find_login_qrcode(page, selector=QR_SELECTOR)
|
||||
if image:
|
||||
return image
|
||||
# 登录框不一定自己弹出来,这是爬虫自身扫码流程的同款兜底。
|
||||
await asyncio.sleep(0.5)
|
||||
try:
|
||||
await page.locator(LOGIN_BUTTON_SELECTOR).click(timeout=5000)
|
||||
except Exception:
|
||||
return ""
|
||||
return await utils.find_login_qrcode(page, selector=QR_SELECTOR)
|
||||
|
||||
|
||||
class AccountLoginSession:
|
||||
"""一次针对**临时上下文**的扫码尝试。"""
|
||||
|
||||
def __init__(self, context: Any, page: Any) -> None:
|
||||
self.status = STATUS_WAITING
|
||||
self.message = "请用手机扫描二维码"
|
||||
self.image = ""
|
||||
self.started_at = time.time()
|
||||
self.account: Optional[Dict[str, Any]] = None
|
||||
self.cookie: str = ""
|
||||
self.platform = PLATFORM_XHS
|
||||
self._context = context
|
||||
self._page = page
|
||||
|
||||
@property
|
||||
def elapsed(self) -> float:
|
||||
return time.time() - self.started_at
|
||||
|
||||
async def refresh(self) -> None:
|
||||
if self.status != STATUS_WAITING:
|
||||
return
|
||||
if self.elapsed > QR_TTL_SECONDS:
|
||||
self.status = STATUS_EXPIRED
|
||||
self.message = "二维码已超时,请重新获取"
|
||||
return
|
||||
|
||||
try:
|
||||
raw = await self._page.evaluate(LOGIN_STATE_PROBE)
|
||||
state = json.loads(raw) if isinstance(raw, str) else {}
|
||||
except Exception:
|
||||
self.status = STATUS_ERROR
|
||||
self.message = "二维码所在页面已被关闭,请重新获取"
|
||||
return
|
||||
|
||||
if not state.get("logged_in"):
|
||||
return
|
||||
|
||||
# 登录成功:从**这个临时上下文**取 cookie。取完上下文就丢弃,
|
||||
# 所以不会残留、也不会影响别的账号。
|
||||
cookies = await self._context.cookies()
|
||||
self.cookie = "; ".join(f"{c['name']}={c['value']}" for c in cookies)
|
||||
self.status = STATUS_SUCCESS
|
||||
self.message = "已获取登录态,正在识别账号…"
|
||||
|
||||
def snapshot(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"status": self.status,
|
||||
"message": self.message,
|
||||
"image": self.image,
|
||||
"elapsed": int(self.elapsed),
|
||||
"expires_in": max(0, int(QR_TTL_SECONDS - self.elapsed)),
|
||||
"account": self.account,
|
||||
}
|
||||
|
||||
async def close(self) -> None:
|
||||
"""关掉临时上下文。这是它存在的全部意义 —— 用完即弃。"""
|
||||
for closer in (self._page.close, self._context.close):
|
||||
try:
|
||||
await closer()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
async def _teardown_locked() -> None:
|
||||
global _current
|
||||
if _current is not None:
|
||||
await _current.close()
|
||||
_current = None
|
||||
|
||||
|
||||
async def start() -> Dict[str, Any]:
|
||||
"""开一个临时上下文,打开登录页,取回二维码。"""
|
||||
global _current
|
||||
|
||||
async with _lock:
|
||||
await _teardown_locked()
|
||||
|
||||
try:
|
||||
browser = await _connect()
|
||||
except Exception as exc:
|
||||
await _disconnect()
|
||||
raise RuntimeError(
|
||||
f"连接浏览器失败({_cdp_url()})。请确认服务器上的 Chrome 以 "
|
||||
f"--remote-debugging-port 启动。原始错误:{exc}"
|
||||
) from exc
|
||||
|
||||
# 临时上下文,不是 contexts[0]。这里刻意要一个干净的身份 ——
|
||||
# 借用操作者自己的登录态会让"新增账号"变成"再读一遍当前账号"。
|
||||
context = await browser.new_context()
|
||||
page = await context.new_page()
|
||||
|
||||
try:
|
||||
await page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=45000)
|
||||
image = await _read_qr(page)
|
||||
except Exception as exc:
|
||||
try:
|
||||
await page.close()
|
||||
await context.close()
|
||||
except Exception:
|
||||
pass
|
||||
raise RuntimeError(f"打开登录页失败:{exc}") from exc
|
||||
|
||||
session = AccountLoginSession(context, page)
|
||||
session.image = image
|
||||
if not image:
|
||||
session.status = STATUS_ERROR
|
||||
session.message = "页面上没找到二维码,请确认站点结构没有变化"
|
||||
|
||||
_current = session
|
||||
return session.snapshot()
|
||||
|
||||
|
||||
async def status() -> Dict[str, Any]:
|
||||
async with _lock:
|
||||
if _current is None:
|
||||
return {
|
||||
"status": STATUS_IDLE,
|
||||
"message": "",
|
||||
"image": "",
|
||||
"elapsed": 0,
|
||||
"expires_in": 0,
|
||||
"account": None,
|
||||
}
|
||||
await _current.refresh()
|
||||
return _current.snapshot()
|
||||
|
||||
|
||||
async def take_cookie() -> Optional[str]:
|
||||
"""取走已登录的 cookie 并结束会话。
|
||||
|
||||
由路由层在落库时调用。cookie 只经内存传递,**不进响应体** —— 它是凭证,
|
||||
前端没有任何理由看到它。
|
||||
"""
|
||||
global _current
|
||||
async with _lock:
|
||||
if _current is None or _current.status != STATUS_SUCCESS:
|
||||
return None
|
||||
cookie = _current.cookie
|
||||
await _teardown_locked()
|
||||
return cookie
|
||||
|
||||
|
||||
async def cancel() -> Dict[str, Any]:
|
||||
async with _lock:
|
||||
await _teardown_locked()
|
||||
return {
|
||||
"status": STATUS_IDLE,
|
||||
"message": "已取消",
|
||||
"image": "",
|
||||
"elapsed": 0,
|
||||
"expires_in": 0,
|
||||
"account": None,
|
||||
}
|
||||
|
||||
|
||||
async def shutdown() -> None:
|
||||
async with _lock:
|
||||
await _teardown_locked()
|
||||
await _disconnect()
|
||||
@@ -0,0 +1,129 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2025 [email protected]
|
||||
#
|
||||
# This file is part of MediaCrawler project.
|
||||
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/creator/models.py
|
||||
# GitHub: https://github.com/NanmiCoder
|
||||
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
|
||||
#
|
||||
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
|
||||
# 1. 不得用于任何商业用途。
|
||||
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
|
||||
# 3. 不得进行大规模爬取或对平台造成运营干扰。
|
||||
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
|
||||
# 5. 不得用于任何非法或不当的用途。
|
||||
#
|
||||
# 详细许可条款请参阅项目根目录下的LICENSE文件。
|
||||
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
|
||||
|
||||
"""运营模块的数据模型。
|
||||
|
||||
**刻意复用 `MonitorBase`**:这样 `init_db` 的 `create_all` 会顺手建出新表,而
|
||||
`_ensure_columns`(已改为按模型元数据推导)也会自动给新表补字段 —— 不必再维护一份
|
||||
建表语句。表落在同一个库里,与监控互不干扰。
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import BigInteger, Float, ForeignKey, Index, Integer, String, Text
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from ..monitor.models import MonitorBase
|
||||
|
||||
# 创作者后台的数据权限是「首次访问时自动申请、次日生效」。这个状态必须如实呈现:
|
||||
# 显示成"没数据"会让人以为采集坏了,实际是在等审批。
|
||||
PERMISSION_UNKNOWN = "unknown"
|
||||
PERMISSION_PENDING = "pending" # 已申请,未生效(提示语:"次日可查看")
|
||||
PERMISSION_ACTIVE = "active"
|
||||
PERMISSION_MISSING = "missing" # 接口明确说没有权限
|
||||
|
||||
# 账号自身的可用性。
|
||||
ACCOUNT_OK = "ok"
|
||||
ACCOUNT_EXPIRED = "expired" # cookie 失效,需要重新扫码
|
||||
ACCOUNT_ERROR = "error"
|
||||
|
||||
|
||||
class CreatorAccount(MonitorBase):
|
||||
"""一个自己的小红书账号。
|
||||
|
||||
纯请求路线下,**一个账号的全部身份就是一份 cookie** —— 没有浏览器 profile、
|
||||
没有独立目录。所以"多账号"在这里只是表里的多行,不是多套运行环境。
|
||||
|
||||
`cookie` 是凭证:与监控的 cookie 同样对待,只存库、绝不回显接口。
|
||||
"""
|
||||
|
||||
__tablename__ = "creator_account"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
# 展示名。优先用后台返回的昵称,用户可以改。
|
||||
nickname: Mapped[str] = mapped_column(String(128), nullable=False, default="")
|
||||
# 创作者后台的账号标识,由 /api/galaxy/user/info 返回,用于去重。
|
||||
user_id: Mapped[str] = mapped_column(String(64), nullable=False, default="", index=True)
|
||||
red_id: Mapped[str] = mapped_column(String(64), nullable=False, default="")
|
||||
avatar: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
|
||||
cookie: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
|
||||
status: Mapped[str] = mapped_column(String(16), nullable=False, default=ACCOUNT_OK)
|
||||
permission_status: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, default=PERMISSION_UNKNOWN
|
||||
)
|
||||
# 后台原话,例如"已为您申请数据权限,次日可查看"。照抄,不改写。
|
||||
permission_tip: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
|
||||
last_checked_at: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
last_synced_at: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
last_error: Mapped[Optional[str]] = mapped_column(Text)
|
||||
|
||||
created_at: Mapped[int] = mapped_column(BigInteger, nullable=False)
|
||||
updated_at: Mapped[int] = mapped_column(BigInteger, nullable=False)
|
||||
|
||||
notes: Mapped[list["CreatorNoteStat"]] = relationship(
|
||||
back_populates="account", cascade="all, delete-orphan"
|
||||
)
|
||||
|
||||
|
||||
class CreatorNoteStat(MonitorBase):
|
||||
"""一篇作品在某个采集时点的运营数据。
|
||||
|
||||
创作者后台给的是**累计值**(截至查询时点),所以反复采集天然形成时间序列 ——
|
||||
与监控的"快照 + 差分"是同一个思路,因此这里保留 `captured_at` 而不是覆盖写。
|
||||
"""
|
||||
|
||||
__tablename__ = "creator_note_stat"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
account_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("creator_account.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
note_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
title: Mapped[str] = mapped_column(Text, nullable=False, default="")
|
||||
# 发布时间(毫秒)。后台按发布时间筛选,这是它的主时间轴。
|
||||
publish_time: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
|
||||
# --- 运营指标 ---------------------------------------------------------
|
||||
# 计数用 BigInteger:曝光量可以很大,用 INT 迟早溢出。
|
||||
exposure: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
views: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
likes: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
comments: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
favorites: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
shares: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
new_followers: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
danmaku: Mapped[Optional[int]] = mapped_column(BigInteger)
|
||||
|
||||
# 比率与时长。后台返回的可能是 "12.3%"/"1分30秒" 这类字符串,解析不了的存 NULL
|
||||
# 而不是 0 —— 与监控层的口径一致:0 是真实值,NULL 是"不知道"。
|
||||
cover_ctr: Mapped[Optional[float]] = mapped_column(Float)
|
||||
avg_watch_seconds: Mapped[Optional[float]] = mapped_column(Float)
|
||||
two_second_exit_rate: Mapped[Optional[float]] = mapped_column(Float)
|
||||
completion_rate: Mapped[Optional[float]] = mapped_column(Float)
|
||||
|
||||
captured_at: Mapped[int] = mapped_column(BigInteger, nullable=False, index=True)
|
||||
|
||||
account: Mapped["CreatorAccount"] = relationship(back_populates="notes")
|
||||
|
||||
__table_args__ = (
|
||||
# 同一个时点同一篇只留一行,重复同步不会堆积。
|
||||
Index("ix_creator_note_stat_unique", "account_id", "note_id", "captured_at", unique=True),
|
||||
)
|
||||
@@ -0,0 +1,354 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2025 [email protected]
|
||||
#
|
||||
# This file is part of MediaCrawler project.
|
||||
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/creator/service.py
|
||||
# GitHub: https://github.com/NanmiCoder
|
||||
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
|
||||
#
|
||||
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
|
||||
# 1. 不得用于任何商业用途。
|
||||
# 2. 使用时应遵守对应平台的使用条款和robots.txt规则。
|
||||
# 3. 不得进行大规模爬取或对平台造成运营干扰。
|
||||
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
|
||||
# 5. 不得用于任何非法或不当的用途。
|
||||
#
|
||||
# 详细许可条款请参阅项目根目录下的LICENSE文件。
|
||||
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
|
||||
|
||||
"""运营账号的增删查与数据同步。
|
||||
|
||||
一条贯穿全文件的规则:**cookie 是凭证,永远不出现在返回给上层的结构里。**
|
||||
对外只给 `has_cookie` 这样的布尔量,与监控层对 cookie 的处理保持一致。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from datetime import datetime, time, timedelta
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from sqlalchemy import delete, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from tools.time_util import get_current_timestamp
|
||||
|
||||
from .client import CreatorApiError, CreatorClient
|
||||
from .models import (
|
||||
ACCOUNT_ERROR,
|
||||
ACCOUNT_EXPIRED,
|
||||
ACCOUNT_OK,
|
||||
PERMISSION_ACTIVE,
|
||||
PERMISSION_MISSING,
|
||||
PERMISSION_PENDING,
|
||||
PERMISSION_UNKNOWN,
|
||||
CreatorAccount,
|
||||
CreatorNoteStat,
|
||||
)
|
||||
|
||||
# 同步一次最多翻多少页。后台默认一页 10 条,200 页足以覆盖任何正常账号,
|
||||
# 同时防止"接口不返回 has_more"时无限翻下去。
|
||||
MAX_SYNC_PAGES = 200
|
||||
PAGE_SIZE = 10
|
||||
|
||||
|
||||
def _account_dict(account: CreatorAccount, note_count: int = 0) -> Dict[str, Any]:
|
||||
"""账号的对外表示。**刻意不含 cookie。**"""
|
||||
return {
|
||||
"id": account.id,
|
||||
"nickname": account.nickname,
|
||||
"user_id": account.user_id,
|
||||
"red_id": account.red_id,
|
||||
"avatar": account.avatar,
|
||||
"status": account.status,
|
||||
"permission_status": account.permission_status,
|
||||
# 后台原话照抄。"次日可查看"这类信息只能由它自己说,改写就失真了。
|
||||
"permission_tip": account.permission_tip,
|
||||
"last_checked_at": account.last_checked_at,
|
||||
"last_synced_at": account.last_synced_at,
|
||||
"last_error": account.last_error,
|
||||
"has_cookie": bool(account.cookie),
|
||||
"created_at": account.created_at,
|
||||
"note_count": note_count,
|
||||
}
|
||||
|
||||
|
||||
async def list_accounts(session: AsyncSession) -> List[Dict[str, Any]]:
|
||||
accounts = list(
|
||||
(await session.scalars(select(CreatorAccount).order_by(CreatorAccount.id))).all()
|
||||
)
|
||||
counts = dict(
|
||||
(
|
||||
await session.execute(
|
||||
select(CreatorNoteStat.account_id, func.count(func.distinct(CreatorNoteStat.note_id)))
|
||||
.group_by(CreatorNoteStat.account_id)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
return [_account_dict(account, counts.get(account.id, 0)) for account in accounts]
|
||||
|
||||
|
||||
async def get_account(session: AsyncSession, account_id: int) -> CreatorAccount:
|
||||
account = await session.get(CreatorAccount, account_id)
|
||||
if account is None:
|
||||
raise ValueError(f"账号 {account_id} 不存在")
|
||||
return account
|
||||
|
||||
|
||||
async def account_detail(session: AsyncSession, account_id: int) -> Dict[str, Any]:
|
||||
account = await get_account(session, account_id)
|
||||
notes = await latest_notes(session, account_id)
|
||||
return {
|
||||
"account": _account_dict(account, len(notes)),
|
||||
"notes": notes,
|
||||
"summary": _summarize(notes),
|
||||
}
|
||||
|
||||
|
||||
def _summarize(notes: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""账号级汇总。取最后一轮快照的累计值之和。"""
|
||||
totals = {
|
||||
key: 0
|
||||
for key in ("exposure", "views", "likes", "comments", "favorites", "shares", "new_followers")
|
||||
}
|
||||
for note in notes:
|
||||
for key in totals:
|
||||
value = note.get(key)
|
||||
if isinstance(value, (int, float)):
|
||||
totals[key] += int(value)
|
||||
return totals
|
||||
|
||||
|
||||
async def latest_notes(session: AsyncSession, account_id: int) -> List[Dict[str, Any]]:
|
||||
"""每个作品取**最近一次**快照。
|
||||
|
||||
表里保留全部历史(换个时点就是一条新行),但列表只该展示"现在",否则同一个
|
||||
作品会在列表里出现多次。
|
||||
"""
|
||||
newest = (
|
||||
select(
|
||||
CreatorNoteStat.note_id,
|
||||
func.max(CreatorNoteStat.captured_at).label("captured_at"),
|
||||
)
|
||||
.where(CreatorNoteStat.account_id == account_id)
|
||||
.group_by(CreatorNoteStat.note_id)
|
||||
.subquery()
|
||||
)
|
||||
rows = (
|
||||
await session.scalars(
|
||||
select(CreatorNoteStat)
|
||||
.join(
|
||||
newest,
|
||||
(CreatorNoteStat.note_id == newest.c.note_id)
|
||||
& (CreatorNoteStat.captured_at == newest.c.captured_at),
|
||||
)
|
||||
.where(CreatorNoteStat.account_id == account_id)
|
||||
.order_by(CreatorNoteStat.publish_time.desc().nullslast())
|
||||
)
|
||||
).all()
|
||||
return [_note_dict(row) for row in rows]
|
||||
|
||||
|
||||
def _note_dict(row: CreatorNoteStat) -> Dict[str, Any]:
|
||||
return {
|
||||
"note_id": row.note_id,
|
||||
"title": row.title,
|
||||
"publish_time": row.publish_time,
|
||||
"exposure": row.exposure,
|
||||
"views": row.views,
|
||||
"likes": row.likes,
|
||||
"comments": row.comments,
|
||||
"favorites": row.favorites,
|
||||
"shares": row.shares,
|
||||
"new_followers": row.new_followers,
|
||||
"danmaku": row.danmaku,
|
||||
"cover_ctr": row.cover_ctr,
|
||||
"avg_watch_seconds": row.avg_watch_seconds,
|
||||
"two_second_exit_rate": row.two_second_exit_rate,
|
||||
"completion_rate": row.completion_rate,
|
||||
"captured_at": row.captured_at,
|
||||
}
|
||||
|
||||
|
||||
async def upsert_account_from_cookie(session: AsyncSession, cookie: str) -> Dict[str, Any]:
|
||||
"""用一份 cookie 识别并保存账号。
|
||||
|
||||
识别靠 `user/info` 而不是让用户填名字 —— 填错名字只会让后面所有数据对不上号。
|
||||
已有同 `user_id` 的账号则更新它的 cookie(重新登录)。
|
||||
"""
|
||||
client = CreatorClient(cookie)
|
||||
if not client.looks_authenticated:
|
||||
raise ValueError("这份 cookie 里没有 a1,无法签名,请重新扫码")
|
||||
|
||||
try:
|
||||
info = await client.fetch_user_info()
|
||||
except CreatorApiError as exc:
|
||||
raise ValueError(f"登录态无法使用:{exc}") from exc
|
||||
|
||||
if not info.get("user_id"):
|
||||
raise ValueError("接口没有返回账号标识,可能登录态无效")
|
||||
|
||||
now = get_current_timestamp()
|
||||
account = await session.scalar(
|
||||
select(CreatorAccount).where(CreatorAccount.user_id == info["user_id"])
|
||||
)
|
||||
if account is None:
|
||||
account = CreatorAccount(created_at=now)
|
||||
session.add(account)
|
||||
|
||||
account.nickname = info.get("nickname") or account.nickname or "未命名账号"
|
||||
account.user_id = info["user_id"]
|
||||
account.red_id = info.get("red_id") or ""
|
||||
account.avatar = info.get("avatar") or ""
|
||||
account.cookie = cookie
|
||||
account.status = ACCOUNT_OK
|
||||
account.last_error = None
|
||||
account.last_checked_at = now
|
||||
account.updated_at = now
|
||||
|
||||
await session.flush()
|
||||
|
||||
# 顺手把权限状态也拉一次:新账号几乎必然处于"已申请、次日生效",
|
||||
# 当场告诉用户,比让他明天再回来问要好。
|
||||
await refresh_permission(session, account)
|
||||
|
||||
return _account_dict(account)
|
||||
|
||||
|
||||
async def refresh_permission(session: AsyncSession, account: CreatorAccount) -> None:
|
||||
"""查询并记录数据权限状态。失败不影响账号本身可用。"""
|
||||
try:
|
||||
permission = await CreatorClient(account.cookie).fetch_permission()
|
||||
except CreatorApiError as exc:
|
||||
if exc.status == 401:
|
||||
account.status = ACCOUNT_EXPIRED
|
||||
account.last_error = str(exc)
|
||||
else:
|
||||
account.last_error = str(exc)
|
||||
account.updated_at = get_current_timestamp()
|
||||
return
|
||||
|
||||
display = permission.get("display")
|
||||
status = permission.get("status")
|
||||
account.permission_tip = permission.get("tip") or ""
|
||||
|
||||
if display or status:
|
||||
account.permission_status = PERMISSION_ACTIVE
|
||||
elif account.permission_tip:
|
||||
# 有提示语但未开通 —— 实测就是"已为您申请数据权限,次日可查看"。
|
||||
account.permission_status = PERMISSION_PENDING
|
||||
else:
|
||||
account.permission_status = PERMISSION_MISSING
|
||||
|
||||
account.status = ACCOUNT_OK
|
||||
account.last_error = None
|
||||
account.last_checked_at = get_current_timestamp()
|
||||
account.updated_at = account.last_checked_at
|
||||
|
||||
|
||||
async def check_account(session: AsyncSession, account_id: int) -> Dict[str, Any]:
|
||||
"""重新检测一个账号:登录态还在不在、权限开通没有。"""
|
||||
account = await get_account(session, account_id)
|
||||
await refresh_permission(session, account)
|
||||
count = (
|
||||
await session.execute(
|
||||
select(func.count(func.distinct(CreatorNoteStat.note_id))).where(
|
||||
CreatorNoteStat.account_id == account_id
|
||||
)
|
||||
)
|
||||
).scalar() or 0
|
||||
return _account_dict(account, count)
|
||||
|
||||
|
||||
async def delete_account(session: AsyncSession, account_id: int) -> None:
|
||||
account = await get_account(session, account_id)
|
||||
await session.execute(
|
||||
delete(CreatorNoteStat).where(CreatorNoteStat.account_id == account_id)
|
||||
)
|
||||
await session.delete(account)
|
||||
|
||||
|
||||
def _day_bounds(days: int) -> tuple[int, int]:
|
||||
"""最近 N 天的起止(毫秒)。与后台的按发布时间筛选对齐。"""
|
||||
today = datetime.now()
|
||||
end = int(datetime.combine(today.date(), time(23, 59, 59)).timestamp() * 1000)
|
||||
start = int(
|
||||
datetime.combine((today - timedelta(days=days)).date(), time(0, 0, 0)).timestamp() * 1000
|
||||
)
|
||||
return start, end
|
||||
|
||||
|
||||
async def sync_account(
|
||||
session: AsyncSession, account_id: int, days: int = 90
|
||||
) -> Dict[str, Any]:
|
||||
"""拉取一个账号的作品运营数据并落库。
|
||||
|
||||
权限未生效时接口返回的是**空壳成功**(`data.result` 里没有数据),不是错误 ——
|
||||
所以"同步成功但 0 条"是正常结果,必须如实回报,不能让用户以为采集坏了。
|
||||
"""
|
||||
account = await get_account(session, account_id)
|
||||
if not account.cookie:
|
||||
raise ValueError("该账号没有可用的登录态,请重新扫码")
|
||||
|
||||
client = CreatorClient(account.cookie)
|
||||
start_ms, end_ms = _day_bounds(days)
|
||||
now = get_current_timestamp()
|
||||
|
||||
collected: List[Dict[str, Any]] = []
|
||||
for page in range(1, MAX_SYNC_PAGES + 1):
|
||||
try:
|
||||
batch = await client.fetch_note_list(start_ms, end_ms, page_num=page, page_size=PAGE_SIZE)
|
||||
except CreatorApiError as exc:
|
||||
account.last_error = str(exc)
|
||||
if exc.status == 401:
|
||||
account.status = ACCOUNT_EXPIRED
|
||||
account.updated_at = get_current_timestamp()
|
||||
raise ValueError(f"同步失败:{exc}") from exc
|
||||
|
||||
collected.extend(batch)
|
||||
if len(batch) < PAGE_SIZE:
|
||||
break
|
||||
await asyncio.sleep(0.6) # 对后台客气一点,这是自己的账号但仍是自动化访问
|
||||
|
||||
# 先删掉本时点可能存在的重复行,再写入 —— 表上有 (account, note, captured_at)
|
||||
# 唯一索引,重复同步不该报错。
|
||||
await session.execute(
|
||||
delete(CreatorNoteStat).where(
|
||||
CreatorNoteStat.account_id == account_id, CreatorNoteStat.captured_at == now
|
||||
)
|
||||
)
|
||||
for note in collected:
|
||||
if not note.get("note_id"):
|
||||
continue
|
||||
session.add(
|
||||
CreatorNoteStat(
|
||||
account_id=account_id,
|
||||
note_id=note["note_id"],
|
||||
title=note.get("title") or "",
|
||||
publish_time=note.get("publish_time"),
|
||||
exposure=note.get("exposure"),
|
||||
views=note.get("views"),
|
||||
likes=note.get("likes"),
|
||||
comments=note.get("comments"),
|
||||
favorites=note.get("favorites"),
|
||||
shares=note.get("shares"),
|
||||
new_followers=note.get("new_followers"),
|
||||
danmaku=note.get("danmaku"),
|
||||
cover_ctr=note.get("cover_ctr"),
|
||||
avg_watch_seconds=note.get("avg_watch_seconds"),
|
||||
two_second_exit_rate=note.get("two_second_exit_rate"),
|
||||
completion_rate=note.get("completion_rate"),
|
||||
captured_at=now,
|
||||
)
|
||||
)
|
||||
|
||||
account.last_synced_at = now
|
||||
account.last_error = None
|
||||
account.updated_at = now
|
||||
await refresh_permission(session, account)
|
||||
await session.flush()
|
||||
|
||||
return {
|
||||
"account_id": account_id,
|
||||
"fetched": len(collected),
|
||||
"permission_status": account.permission_status,
|
||||
"permission_tip": account.permission_tip,
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2025 [email protected]
|
||||
#
|
||||
# This file is part of MediaCrawler project.
|
||||
# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/creator/signing.py
|
||||
# GitHub: https://github.com/NanmiCoder
|
||||
# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
|
||||
#
|
||||
# 声明:本代码仅供学习和研究目的使用。使用者应遵守以下原则:
|
||||
# 1. 不得用于任何商业用途。
|
||||
# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
|
||||
# 3. 不得进行大规模爬取或对平台造成运营干扰。
|
||||
# 4. 应合理控制请求频率,避免给目标平台带来不必要的负担。
|
||||
# 5. 不得用于任何非法或不当的用途。
|
||||
#
|
||||
# 详细许可条款请参阅项目根目录下的LICENSE文件。
|
||||
# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
|
||||
|
||||
"""创作者后台的请求签名(XYW_ 方案)。
|
||||
|
||||
主站与创作者后台用的是**两套不同的签名**:主站是 VMP 的 `XYS_`,创作者后台是
|
||||
`XYW_`。后者简单得多 —— MD5 → base64 → AES-128-CBC,密钥与 IV 都是硬编码常量,
|
||||
纯 Python 可算,不需要浏览器。
|
||||
|
||||
常量与 `xhshow/config/config.py` 逐字节一致(该库也据此实现了 `sign_xyw`),
|
||||
并与独立的逆向实现 xiaohongshu-cli/creator_signing.py 互相印证。
|
||||
|
||||
**三条实测结论**(tools/probe_creator_api.py 的 Phase 0 输出):
|
||||
1. 待签字符串必须是 `url=` + 路径 + 查询串 的形式。只给路径、或去掉 `url=` 前缀,
|
||||
网关一律返回 **406**;写法正确时签名通过。
|
||||
2. `appId` 用 `ugc`(创作者平台的取值),不是主站的 `xhs-pc-web`。
|
||||
3. 不带 cookie 时返回的是应用层的 401「无登录信息」而非 406 —— 说明签名每次都过了,
|
||||
认证是独立的一层。
|
||||
"""
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
from datetime import datetime
|
||||
|
||||
XYW_AES_KEY = b"7cc4adla5ay0701v"
|
||||
XYW_AES_IV = b"4uzjr7mbsibcaldp"
|
||||
|
||||
# 与 xhshow 的 XYW_ENV_FLAGS_DEFAULT 一致。含义未知,但改了签名就不被接受。
|
||||
XYW_ENV_FLAGS = "0|0|0|1|0|0|1|0|0|0|1|0|0|0|0|1|0|0|0"
|
||||
|
||||
XYW_PREFIX = "XYW_"
|
||||
XYW_SIGN_SVN = "56"
|
||||
XYW_SIGN_TYPE = "x2"
|
||||
XYW_SIGN_VERSION = "1"
|
||||
|
||||
# 创作者平台的 appId。用主站的 xhs-pc-web 会被拒。
|
||||
CREATOR_APP_ID = "ugc"
|
||||
|
||||
|
||||
def _aes_encrypt_hex(plaintext: str) -> str:
|
||||
from Crypto.Cipher import AES
|
||||
from Crypto.Util.Padding import pad
|
||||
|
||||
cipher = AES.new(XYW_AES_KEY, AES.MODE_CBC, XYW_AES_IV)
|
||||
return cipher.encrypt(pad(plaintext.encode("utf-8"), AES.block_size)).hex()
|
||||
|
||||
|
||||
def sign_xyw(
|
||||
api: str,
|
||||
a1: str,
|
||||
app_id: str = CREATOR_APP_ID,
|
||||
body: dict | None = None,
|
||||
timestamp_ms: int | None = None,
|
||||
) -> dict[str, str]:
|
||||
"""为一次创作者后台请求生成 ``x-s`` / ``x-t`` 请求头。
|
||||
|
||||
``api`` 必须是待签的完整字符串:``url=`` 加路径,GET 请求还要带上查询串。
|
||||
POST 的 JSON body 追加在其后(紧凑分隔符、不转义非 ASCII),与参考实现一致。
|
||||
"""
|
||||
content = api
|
||||
if body is not None:
|
||||
content += json.dumps(body, separators=(",", ":"), ensure_ascii=False)
|
||||
|
||||
if timestamp_ms is None:
|
||||
timestamp_ms = int(datetime.now().timestamp() * 1000)
|
||||
|
||||
digest = hashlib.md5(content.encode("utf-8")).hexdigest()
|
||||
plaintext = f"x1={digest};x2={XYW_ENV_FLAGS};x3={a1};x4={timestamp_ms};"
|
||||
encoded = base64.b64encode(plaintext.encode("utf-8")).decode("utf-8")
|
||||
|
||||
envelope = {
|
||||
"signSvn": XYW_SIGN_SVN,
|
||||
"signType": XYW_SIGN_TYPE,
|
||||
"appId": app_id,
|
||||
"signVersion": XYW_SIGN_VERSION,
|
||||
"payload": _aes_encrypt_hex(encoded),
|
||||
}
|
||||
x_s = XYW_PREFIX + base64.b64encode(
|
||||
json.dumps(envelope, separators=(",", ":")).encode("utf-8")
|
||||
).decode("utf-8")
|
||||
|
||||
return {"x-s": x_s, "x-t": str(timestamp_ms)}
|
||||
|
||||
|
||||
def signed_api(url_path: str, query: str = "") -> str:
|
||||
"""把路径与查询串拼成待签字符串。
|
||||
|
||||
单独抽出来是因为这个格式**没有文档**,只能靠实测固定下来 —— 写错就是 406,
|
||||
而 406 的响应体 ``{"code":-1,"success":false}`` 完全看不出错在哪。
|
||||
"""
|
||||
return f"url={url_path}?{query}" if query else f"url={url_path}"
|
||||
Reference in New Issue
Block a user