feat(monitor): 抖音 Web 接口客户端 —— 绕开爬虫子进程,直接发 HTTP
移植自 mac-agent-os 的 mediacrawler_adapter:不起子进程、不开页面,用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin,所以 ingest 那条链路一个字不用改。
**目前能用的(真环境实测,非推断)**:
profile/other : 200, 7075 字节 —— 博主主页指标(粉丝/获赞/作品数/昵称)
aweme/detail : 200, 45425 字节 —— 单条作品详情(含点赞/评论/收藏/分享)
**目前不能用的:作品列表 `aweme/post`。** 两个互相独立的原因:
1. 这个接口被抖音单独升级成了真校验:不带 x-tt-argus 回 403「Uifid Not Found」,
带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过,「真校验」过不了。
同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的,
挑中的恰好是「批量拉作品列表」这个最敏感的动作。
2. 改走页面截获也不行:CDP 浏览器打开博主主页会落到「验证码中间页」(当天大量探测的
代价,过几小时要重测)。
所以现在的边界是:**已知作品的指标刷新能做,自动发现新作品做不了**。
**排查中控住变量后得到的两条事实**(都写进注释了):
· `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
UA+client hints+Cookie 时是 200 但仅 121 字节的空壳,补上这三个头变 7074 字节。
(我先前猜的 sec-ch-ua 不是关键。)
· 因此 UA 与 client hints 必须**成套地取自同一个浏览器**,所以 BrowserIdentity 一次
从 CDP 取齐 cookie + UA + hints,而不是各自写死。
「200 + 空 body 必须当场报错」也是刻意写死的:放过去它会在下游变成「这个博主没作品」,
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的,还被翻译成「账号被封」。
测试 +11:cookie 解析、请求头成套性(含 uifid 缺失/回退)、产物键名与 store 对齐、
以及 _get 的三条失败路径(空 body / 403 带网关原话 / 正常返回)。
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
"""
|
||||
comment_dom_scanner.py — 扫描评论区DOM结构
|
||||
"""
|
||||
import asyncio, json, os, sys
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'matrix_modules'))
|
||||
|
||||
async def main():
|
||||
from cdp_connector import CDPConnector
|
||||
|
||||
ID_DIR = os.path.expanduser('~/workbuddy-agent-os/agent-local/tools/matrix/identities/douyin_01_camo')
|
||||
conn = CDPConnector(browser_type='camoufox', identity_dir=ID_DIR,
|
||||
headless=False, window=(702,783), window_position=(652,0))
|
||||
print('🦊 启动...')
|
||||
await conn.connect()
|
||||
page = conn.page
|
||||
await conn.init_anti_detection()
|
||||
await page.goto('https://www.douyin.com/', timeout=15000)
|
||||
await asyncio.sleep(3)
|
||||
|
||||
print('\n📌 手动进入播放页,然后回车继续...')
|
||||
input()
|
||||
|
||||
# 按X打开评论区
|
||||
await page.keyboard.press('KeyX')
|
||||
print('\n⌨️ KeyX 打开了评论区')
|
||||
await asyncio.sleep(2)
|
||||
|
||||
# 扫描所有元素的class,找包含comment的
|
||||
classes = await page.evaluate("""() => {
|
||||
const all = document.querySelectorAll('*');
|
||||
const commentRelated = [...all].filter(el => {
|
||||
const c = (el.className || '').toLowerCase();
|
||||
const t = (el.tagName || '');
|
||||
return c.includes('comment') || c.includes('chat') || c.includes('reply') || c.includes('input');
|
||||
});
|
||||
return commentRelated.map(el => ({
|
||||
tag: el.tagName,
|
||||
cls: (el.className || '').slice(0,80),
|
||||
visible: el.offsetParent !== null,
|
||||
rect: Object.values(el.getBoundingClientRect()).map(Math.round),
|
||||
text: (el.textContent || '').slice(0,30)
|
||||
}));
|
||||
}""")
|
||||
print(f'\n📊 找到 {len(classes)} 个评论相关元素:')
|
||||
for c in classes:
|
||||
print(f' {c["tag"]:8s} cls={c["cls"][:40]:40s} vis={c["visible"]} rect={c["rect"]} \"{c["text"]}\"')
|
||||
|
||||
# 保存到文件
|
||||
out = '/tmp/comment_dom_scan.json'
|
||||
with open(out, 'w') as f:
|
||||
json.dump(classes, f, ensure_ascii=False, indent=2)
|
||||
print(f'\n💾 已保存到 {out}')
|
||||
|
||||
print('\n✅ 扫描完成,浏览器保持打开')
|
||||
print('按 Ctrl+C 退出')
|
||||
|
||||
while True:
|
||||
await asyncio.sleep(10)
|
||||
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
print('\n👋 退出')
|
||||
@@ -0,0 +1,253 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
comment_sender.py v2 — 状态感知的评论发送器
|
||||
|
||||
流程(锚点驱动):
|
||||
1. 检测评论区是否打开 → 否则 KeyX
|
||||
2. 检测输入框是否已激活 → 否则 JS focus
|
||||
3. 检测是否有内容 → 否则键盘输入
|
||||
4. Alt+Enter 发送
|
||||
5. 验证码检测 → 手动输入
|
||||
"""
|
||||
import asyncio, os, random, sys, json
|
||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, os.path.dirname(script_dir))
|
||||
from cdp_connector import CDPConnector
|
||||
|
||||
LOG_FILE = '/tmp/comment_sender.log'
|
||||
def log(m):
|
||||
with open(LOG_FILE, 'a') as f: f.write(m + '\n')
|
||||
print(m, flush=True)
|
||||
|
||||
async def check_state(page) -> dict:
|
||||
"""检测评论区相关状态"""
|
||||
return await page.evaluate("""() => {
|
||||
const ae = document.activeElement;
|
||||
// 评论区面板
|
||||
const panel = document.querySelector('[class*="comment-header"]')
|
||||
|| document.querySelector('[class*="comment"]')
|
||||
|| document.querySelector('[class*="Comment"]');
|
||||
// Draft.js 编辑器
|
||||
const editor = document.querySelector('[contenteditable="true"][role="combobox"]')
|
||||
|| document.querySelector('[class*="DraftEditor"] [contenteditable]');
|
||||
// 输入框是否激活(activeElement 是编辑器或其子元素)
|
||||
const editorActive = ae && (ae.isContentEditable
|
||||
|| ae.getAttribute('contenteditable') === 'true'
|
||||
|| ae.closest('[contenteditable]'));
|
||||
return {
|
||||
has_comment_area: !!panel,
|
||||
has_editor: !!editor,
|
||||
editor_active: editorActive,
|
||||
active_tag: ae ? ae.tagName : 'none',
|
||||
active_cls: ae ? (ae.className||'').slice(0,50) : '',
|
||||
panel_rect: panel ? Object.values(panel.getBoundingClientRect()).map(Math.round) : null
|
||||
};
|
||||
}""")
|
||||
|
||||
async def focus_editor(page) -> bool:
|
||||
"""尝试聚焦 Draft.js 编辑器,返回是否成功"""
|
||||
ok = await page.evaluate("""() => {
|
||||
// 直接找 contenteditable 且有 role 的
|
||||
let ed = document.querySelector('[contenteditable="true"][role="combobox"]');
|
||||
if (!ed) ed = document.querySelector('[class*="DraftEditor"] [contenteditable]');
|
||||
if (!ed) ed = document.querySelector('[contenteditable="true"]');
|
||||
if (ed) { ed.focus(); ed.click(); return true; }
|
||||
return false;
|
||||
}""")
|
||||
await asyncio.sleep(0.5)
|
||||
return ok
|
||||
|
||||
async def send_comment(page, comment_text: str):
|
||||
"""发送一条评论(状态感知版)"""
|
||||
# ── Step 1: 检测并确保评论区打开 ──
|
||||
state = await check_state(page)
|
||||
log(f'📊 初始状态: comment={state["has_comment_area"]} editor={state["has_editor"]} active={state["editor_active"]}')
|
||||
|
||||
if not state['has_comment_area']:
|
||||
log('⌨️ KeyX 打开评论区')
|
||||
await page.keyboard.press('KeyX')
|
||||
await asyncio.sleep(2)
|
||||
else:
|
||||
log('✅ 评论区已打开,跳过KeyX')
|
||||
|
||||
# ── Step 2: 用真实鼠标点击激活输入框 ──
|
||||
state = await check_state(page)
|
||||
if state['editor_active']:
|
||||
log('✅ 输入框已激活')
|
||||
else:
|
||||
# 找到编辑器位置 → 用 Playwright 真实鼠标点击
|
||||
rect = await page.evaluate("""() => {
|
||||
let el = document.querySelector('[contenteditable="true"][role="combobox"]');
|
||||
if (!el) el = document.querySelector('[class*="DraftEditor"] [contenteditable]');
|
||||
if (!el) el = document.querySelector('[contenteditable="true"]');
|
||||
if (!el) return null;
|
||||
const r = el.getBoundingClientRect();
|
||||
return {x: r.x, y: r.y, w: r.width, h: r.height};
|
||||
}""")
|
||||
if rect and rect['w'] > 0 and rect['h'] > 0:
|
||||
# 用 Playwright 生成真实鼠标事件链
|
||||
x = rect['x'] + rect['w'] // 2
|
||||
y = rect['y'] + rect['h'] // 2
|
||||
log(f'🖱️ 真实鼠标点击编辑器: ({x:.0f}, {y:.0f})')
|
||||
await page.mouse.click(x, y)
|
||||
await asyncio.sleep(1)
|
||||
# 再次确认
|
||||
state = await check_state(page)
|
||||
if state['editor_active']:
|
||||
log('✅ 鼠标点击后输入框已激活')
|
||||
else:
|
||||
log('⚠️ 鼠标点击后仍未激活')
|
||||
else:
|
||||
log('⚠️ 找不到编辑器位置')
|
||||
return 'failed'
|
||||
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
# ── Step 3: JS 设置内容(不用键盘,不依赖焦点)──
|
||||
log(f'💬 JS设置内容: "{comment_text}"')
|
||||
result = await page.evaluate(f'''() => {{
|
||||
// 找 Draft.js 编辑器
|
||||
let ed = document.querySelector('[contenteditable="true"][role="combobox"]');
|
||||
if (!ed) ed = document.querySelector('[class*="DraftEditor"] [contenteditable]');
|
||||
if (!ed) ed = document.querySelector('[contenteditable="true"]');
|
||||
|
||||
if (!ed) return {{ok: false, reason: 'no_editor'}};
|
||||
|
||||
// focus + 清空 + 设置内容
|
||||
ed.focus();
|
||||
ed.textContent = '';
|
||||
ed.textContent = "{comment_text}";
|
||||
ed.dispatchEvent(new Event('input', {{bubbles: true}}));
|
||||
ed.dispatchEvent(new Event('change', {{bubbles: true}}));
|
||||
|
||||
const finalText = ed.textContent;
|
||||
return {{ok: finalText.length > 0, text: finalText, tag: ed.tagName}};
|
||||
}}''')
|
||||
log(f' 结果: ok={result["ok"]} text="{result.get("text","")}" tag={result.get("tag","")}')
|
||||
|
||||
if not result.get('ok'):
|
||||
log('⚠️ JS设置内容失败')
|
||||
return 'failed'
|
||||
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
# ── Step 4: JS 触发发送(先找发送按钮,没有则 Alt+Enter)──
|
||||
log('⌨️ JS触发发送...')
|
||||
send_result = await page.evaluate('''() => {
|
||||
// 找发送按钮
|
||||
const btns = [...document.querySelectorAll('button')];
|
||||
const sendBtn = btns.find(b =>
|
||||
b.textContent.includes('发送') || b.textContent.includes('发布')
|
||||
);
|
||||
if (sendBtn) {
|
||||
sendBtn.click();
|
||||
return 'click_send_btn';
|
||||
}
|
||||
// 没按钮 → 触发 Alt+Enter
|
||||
const ed = document.querySelector('[contenteditable="true"][role="combobox"]')
|
||||
|| document.querySelector('[contenteditable="true"]');
|
||||
if (ed) {
|
||||
ed.dispatchEvent(new KeyboardEvent('keydown', {
|
||||
key: 'Enter', altKey: true, bubbles: true
|
||||
}));
|
||||
return 'alt_enter_event';
|
||||
}
|
||||
return 'failed';
|
||||
}''')
|
||||
log(f' 发送方式: {send_result}')
|
||||
await asyncio.sleep(2)
|
||||
|
||||
# ── Step 5: 验证码 ──
|
||||
has_code = await page.evaluate(
|
||||
"() => !!document.querySelector('input[placeholder*=\"\\u9a8c\\u8bc1\\u7801\"]')"
|
||||
)
|
||||
if has_code:
|
||||
log('⚠️ 验证码弹窗!')
|
||||
return 'verify_code_required'
|
||||
|
||||
# ── Step 6: 验证发送结果 ──
|
||||
found = await page.evaluate(
|
||||
f'() => document.body.innerText.includes("{comment_text}")'
|
||||
)
|
||||
if found:
|
||||
log('✅ 评论已发送')
|
||||
return 'success'
|
||||
else:
|
||||
log('❌ 未在页面找到评论')
|
||||
return 'failed'
|
||||
|
||||
|
||||
async def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--account', '-a', default='douyin_01')
|
||||
parser.add_argument('--comment', '-c', default='好内容,已三连')
|
||||
parser.add_argument('--count', '-n', type=int, default=2)
|
||||
args = parser.parse_args()
|
||||
|
||||
ID_DIR = os.path.expanduser(
|
||||
f'~/workbuddy-agent-os/agent-local/tools/matrix/identities/{args.account}_camo'
|
||||
)
|
||||
if not os.path.exists(ID_DIR):
|
||||
print(f'❌ 账号目录不存在: {ID_DIR}')
|
||||
return
|
||||
|
||||
print(f'🦊 启动: {args.account}')
|
||||
conn = CDPConnector(browser_type='camoufox', identity_dir=ID_DIR,
|
||||
headless=False, window=(702,783), window_position=(652,0))
|
||||
await conn.connect()
|
||||
page = conn.page
|
||||
await conn.init_anti_detection()
|
||||
await page.goto('https://www.douyin.com/', timeout=30000)
|
||||
await asyncio.sleep(3)
|
||||
|
||||
print('\n📌 请操作:')
|
||||
print(' 1. 进入视频播放页')
|
||||
print(' 2. 打开评论区(按 X)')
|
||||
print(' 3. 手动点击激活评论输入框')
|
||||
print(' 4. 回到终端按回车\n')
|
||||
input()
|
||||
|
||||
# 检查是否在播放页
|
||||
v = await page.evaluate('document.querySelectorAll("video").length')
|
||||
if v < 2:
|
||||
print('❌ 未检测到播放页')
|
||||
return
|
||||
print(f'✅ 播放页确认 (video={v})')
|
||||
|
||||
for i in range(args.count):
|
||||
print(f'\n——— 第{i+1}/{args.count}次评论 ———')
|
||||
result = await send_comment(page, args.comment)
|
||||
|
||||
if result == 'verify_code_required':
|
||||
code = input('\n📱 验证码(6位): ').strip()
|
||||
if code and len(code) == 6:
|
||||
await page.evaluate(f'''() => {{
|
||||
const inp = document.querySelector('input[placeholder*="验证码"]');
|
||||
if (inp) {{
|
||||
inp.value = "{code}";
|
||||
inp.dispatchEvent(new Event('input', {{bubbles: true}}));
|
||||
}}
|
||||
const btn = [...document.querySelectorAll('button')]
|
||||
.find(b => b.textContent.includes('确认') || b.textContent.includes('提交'));
|
||||
if (btn) btn.click();
|
||||
}}''')
|
||||
print('✅ 验证码已提交')
|
||||
await asyncio.sleep(2)
|
||||
|
||||
if i < args.count - 1:
|
||||
print('⏳ 评论区保持打开,5秒后下一次...')
|
||||
await asyncio.sleep(5)
|
||||
|
||||
print('\n✅ 完成,浏览器保持打开')
|
||||
print('按 Ctrl+C 退出')
|
||||
while True:
|
||||
await asyncio.sleep(10)
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
print('\n👋 退出')
|
||||
except Exception as e:
|
||||
import traceback; traceback.print_exc()
|
||||
@@ -0,0 +1,370 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
小红书登录后 DOM 重分析工具
|
||||
============================
|
||||
|
||||
用途:登录小红书后重新分析 DOM 结构,验证当前选择器是否仍然有效,
|
||||
输出结构化报告指导选择器更新。
|
||||
|
||||
用法:
|
||||
python tools/reanalyze_xhs_dom.py <identity_name>
|
||||
python tools/reanalyze_xhs_dom.py xhs_01 --full-dump # 输出完整 DOM 快照
|
||||
python tools/reanalyze_xhs_dom.py xhs_01 --explore-url https://www.xiaohongshu.com/explore # 自定义 URL
|
||||
|
||||
流程:
|
||||
1. 通过 Camoufox 连接身份目录(复用登录态)
|
||||
2. 导航到小红书 explore 首页
|
||||
3. 等待渲染稳定
|
||||
4. Dump 关键区域的 DOM 结构(保留选择器相关特征)
|
||||
5. 验证当前 selectors.py 中的每个选择器是否仍能匹配
|
||||
6. 输出结构化的重分析报告
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS_DIR = Path(__file__).parent.parent
|
||||
sys.path.insert(0, str(SCRIPTS_DIR))
|
||||
|
||||
LOCAL_ROOT = Path.home() / "workbuddy-agent-os" / "agent-local" / "tools" / "matrix"
|
||||
IDENTITIES_ROOT = LOCAL_ROOT / "identities"
|
||||
|
||||
|
||||
async def analyze(identity_name: str, full_dump: bool = False, explore_url: str = None):
|
||||
"""主分析流程"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f" 🔍 小红书 DOM 重分析工具")
|
||||
print(f" {'='*60}")
|
||||
print(f" 身份: {identity_name}")
|
||||
print(f" 模式: {'完整快照' if full_dump else '选择器验证'}")
|
||||
print()
|
||||
|
||||
# ── 1. 确定身份目录 ──
|
||||
# 优先从 accounts.yaml 读取
|
||||
import yaml
|
||||
config_path = LOCAL_ROOT / "config" / "accounts.yaml"
|
||||
try:
|
||||
with open(config_path) as f:
|
||||
data = yaml.safe_load(f)
|
||||
acct = None
|
||||
for a in data.get("accounts", []):
|
||||
if a["id"] == identity_name:
|
||||
acct = a
|
||||
break
|
||||
except:
|
||||
acct = None
|
||||
|
||||
if acct:
|
||||
identity_dir_str = acct.get("identity_dir", identity_name)
|
||||
if not identity_dir_str.startswith("/"):
|
||||
identity_dir_str = str(LOCAL_ROOT / identity_dir_str)
|
||||
else:
|
||||
identity_dir_str = str(IDENTITIES_ROOT / identity_name)
|
||||
|
||||
# 清理锁文件
|
||||
import glob
|
||||
for lock_file in glob.glob(f"{identity_dir_str}/user_data/**/.parentlock", recursive=True):
|
||||
try:
|
||||
os.remove(lock_file)
|
||||
except:
|
||||
pass
|
||||
for lock_file in glob.glob(f"{identity_dir_str}/user_data/**/lock", recursive=True):
|
||||
try:
|
||||
os.remove(lock_file)
|
||||
except:
|
||||
pass
|
||||
|
||||
# ── 2. 连接浏览器 ──
|
||||
from cdp_connector import CDPConnector
|
||||
conn = CDPConnector(
|
||||
identity_dir=identity_dir_str,
|
||||
headless=False,
|
||||
window=(702, 783),
|
||||
)
|
||||
print(f" 🚀 连接浏览器: {identity_dir_str}")
|
||||
try:
|
||||
await conn.connect()
|
||||
except Exception as e:
|
||||
print(f" ❌ 连接失败: {e}")
|
||||
sys.exit(1)
|
||||
print(f" ✅ 浏览器已连接")
|
||||
print(f" 📐 视口: 702×783")
|
||||
|
||||
# 反检测
|
||||
try:
|
||||
await conn.init_anti_detection()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ── 3. 导航到小红书 ──
|
||||
target_url = explore_url or "https://www.xiaohongshu.com/explore"
|
||||
print(f"\n 🌐 导航到: {target_url}")
|
||||
for attempt in range(1, 4):
|
||||
try:
|
||||
await conn.page.goto(target_url, timeout=20000, wait_until="domcontentloaded")
|
||||
print(f" ✅ 导航成功 (第{attempt}次)")
|
||||
break
|
||||
except Exception as e:
|
||||
print(f" ⚠️ 导航失败 (第{attempt}次): {e}")
|
||||
if attempt < 3:
|
||||
await asyncio.sleep(3)
|
||||
else:
|
||||
print(f" ❌ 导航最终失败")
|
||||
await conn.close()
|
||||
return
|
||||
|
||||
# ── 等待渲染 ──
|
||||
print(f"\n ⏳ 等待页面渲染...")
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# 关闭登录弹窗
|
||||
try:
|
||||
await conn.page.evaluate("""
|
||||
() => {
|
||||
document.querySelectorAll('[class*=modal], [class*=dialog], [class*=mask], [role=dialog]').forEach(m => {
|
||||
m.style.display = 'none';
|
||||
});
|
||||
}
|
||||
""")
|
||||
except:
|
||||
pass
|
||||
await asyncio.sleep(1)
|
||||
|
||||
# ── 4. 检测登录状态 ──
|
||||
print(f"\n 🔐 登录状态检测")
|
||||
try:
|
||||
from auth_manager import check_login_by_cookie_sync, count_platform_cookies
|
||||
cookies = await conn.context.cookies()
|
||||
logged_in = check_login_by_cookie_sync(cookies, "xiaohongshu")
|
||||
cookie_cnt = count_platform_cookies(cookies, "xiaohongshu")
|
||||
print(f" Cookie 检测: {'✅ 已登录' if logged_in else '❌ 未登录'}")
|
||||
print(f" 平台 Cookie 数: {cookie_cnt}")
|
||||
except Exception as e:
|
||||
print(f" ⚠️ 登录检测异常: {e}")
|
||||
|
||||
# ── 5. 检查页面基本结构 ──
|
||||
print(f"\n 📄 页面基本结构")
|
||||
page_info = await conn.page.evaluate("""
|
||||
() => {
|
||||
return {
|
||||
url: location.href,
|
||||
title: document.title,
|
||||
app_present: !!document.querySelector('#app'),
|
||||
body_children: document.body ? document.body.children.length : 0,
|
||||
viewport: {w: window.innerWidth, h: window.innerHeight},
|
||||
scroll: {x: window.scrollX, y: window.scrollY},
|
||||
};
|
||||
}
|
||||
""")
|
||||
print(f" URL: {page_info.get('url', '?')}")
|
||||
print(f" Title: {page_info.get('title', '?')}")
|
||||
print(f" #app: {'✅' if page_info.get('app_present') else '❌'}")
|
||||
print(f" body children: {page_info.get('body_children', '?')}")
|
||||
|
||||
# ── 6. 验证 selectors.py 中的选择器 ──
|
||||
print(f"\n{'='*60}")
|
||||
print(f" 📋 选择器验证")
|
||||
print(f" {'='*60}")
|
||||
|
||||
# 导入当前选择器
|
||||
sys.path.insert(0, str(SCRIPTS_DIR / "matrix_modules" / "ops" / "xhs"))
|
||||
from selectors import (
|
||||
NOTE_CARD, NOTE_CARD_COVER, NOTE_CARD_IMG,
|
||||
SEARCH_INPUT, SEARCH_BUTTON,
|
||||
FEED_CONTAINER, LOGIN_MODAL, LOGIN_MODAL_CLOSE,
|
||||
LIKE_BUTTON, COLLECT_BUTTON, FOLLOW_BUTTON,
|
||||
COMMENT_ENTRY, COMMENT_INPUT, COMMENT_SEND,
|
||||
ANCHORS,
|
||||
)
|
||||
|
||||
selector_tests = {
|
||||
"首页/瀑布流": [
|
||||
("瀑布流容器", FEED_CONTAINER),
|
||||
("笔记卡片", NOTE_CARD),
|
||||
("卡片封面链接", NOTE_CARD_COVER),
|
||||
("卡片图片", NOTE_CARD_IMG),
|
||||
],
|
||||
"搜索": [
|
||||
("搜索框", SEARCH_INPUT),
|
||||
("搜索按钮", SEARCH_BUTTON),
|
||||
],
|
||||
"弹窗": [
|
||||
("登录弹窗", LOGIN_MODAL),
|
||||
("关闭按钮", LOGIN_MODAL_CLOSE),
|
||||
],
|
||||
"互动(详情页)": [
|
||||
("点赞按钮", LIKE_BUTTON),
|
||||
("收藏按钮", COLLECT_BUTTON),
|
||||
("关注按钮", FOLLOW_BUTTON),
|
||||
("评论入口", COMMENT_ENTRY),
|
||||
("评论输入框", COMMENT_INPUT),
|
||||
("发送按钮", COMMENT_SEND),
|
||||
],
|
||||
}
|
||||
|
||||
all_ok = True
|
||||
for category, selectors in selector_tests.items():
|
||||
print(f"\n 【{category}】")
|
||||
js_code = "var sel = arguments[0]; return document.querySelector(sel) ? document.querySelector(sel).outerHTML.substring(0, 200) : null;"
|
||||
for name, sel in selectors:
|
||||
try:
|
||||
count = await conn.page.evaluate(f"document.querySelectorAll('{sel.replace(chr(39), chr(92) + chr(39))}').length")
|
||||
first_match = None
|
||||
if count > 0:
|
||||
first_match = await conn.page.evaluate(f"try {{ let el = document.querySelector('{sel.replace(chr(39), chr(92) + chr(39))}'); return el ? el.tagName + (el.className ? '.' + el.className.split(' ')[0] : '') : 'N/A'; }} catch(e) {{ return 'JS_ERR'; }}")
|
||||
status = "✅" if count > 0 else "⚠️"
|
||||
if count == 0:
|
||||
all_ok = False
|
||||
print(f" {status} {name}: \"{sel}\"")
|
||||
if count > 0:
|
||||
print(f" 匹配 {count} 个元素, 首元素: {first_match}")
|
||||
except Exception as e:
|
||||
print(f" ❌ {name}: JS 异常 - {e}")
|
||||
all_ok = False
|
||||
|
||||
# ── 7. 锚点验证 ──
|
||||
print(f"\n 【锚点验证】")
|
||||
for anchor_name, anchor_sel in ANCHORS.items():
|
||||
try:
|
||||
count = await conn.page.evaluate(f"document.querySelectorAll('{anchor_sel.replace(chr(39), chr(92) + chr(39))}').length")
|
||||
status = "✅" if count > 0 else "⚠️"
|
||||
if count == 0:
|
||||
all_ok = False
|
||||
print(f" {status} {anchor_name}: \"{anchor_sel}\" → {count} 个")
|
||||
except Exception as e:
|
||||
print(f" ❌ {anchor_name}: {e}")
|
||||
|
||||
# ── 8. 完整 DOM 快照(可选)──
|
||||
if full_dump:
|
||||
print(f"\n{'='*60}")
|
||||
print(f" 📸 完整 DOM 快照")
|
||||
print(f" {'='*60}")
|
||||
print(f" 正在保存到文件...")
|
||||
|
||||
# 获取关键区域的 HTML
|
||||
dom_snapshot = await conn.page.evaluate("""
|
||||
() => {
|
||||
const areas = {};
|
||||
|
||||
// #app 容器
|
||||
const app = document.querySelector('#app');
|
||||
if (app) areas.app = app.outerHTML.substring(0, 5000);
|
||||
|
||||
// 瀑布流区域
|
||||
const layout = document.querySelector('.layout, [class*=layout]');
|
||||
if (layout) areas.layout = layout.outerHTML.substring(0, 3000);
|
||||
|
||||
// 检查是否有详情页(如果有多个 section.note-item 说明在首页)
|
||||
const notes = document.querySelectorAll('section.note-item');
|
||||
if (notes.length > 0) {
|
||||
areas.note_first = notes[0].outerHTML.substring(0, 2000);
|
||||
areas.note_count = notes.length;
|
||||
}
|
||||
|
||||
// 记录当前页面上的所有 class 名(去重)
|
||||
const allClasses = new Set();
|
||||
document.querySelectorAll('*').forEach(el => {
|
||||
if (el.className && typeof el.className === 'string') {
|
||||
el.className.split(/\\s+/).forEach(c => {
|
||||
if (c && c.length > 1) allClasses.add(c);
|
||||
});
|
||||
}
|
||||
});
|
||||
areas.all_classes = [...allClasses].sort();
|
||||
|
||||
// 关键 URL 和 meta
|
||||
areas.meta = {
|
||||
url: location.href,
|
||||
title: document.title,
|
||||
links_count: document.querySelectorAll('a').length,
|
||||
buttons_count: document.querySelectorAll('button').length,
|
||||
img_count: document.querySelectorAll('img').length,
|
||||
video_count: document.querySelectorAll('video').length,
|
||||
};
|
||||
|
||||
return areas;
|
||||
}
|
||||
""")
|
||||
|
||||
timestamp = time.strftime("%Y%m%d_%H%M%S")
|
||||
dump_path = SCRIPTS_DIR / "data" / f"xhs_dom_dump_{timestamp}.json"
|
||||
dump_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 简化输出:移除过长内容
|
||||
output = {}
|
||||
for k, v in dom_snapshot.items():
|
||||
if isinstance(v, str) and len(v) > 500:
|
||||
output[k] = v[:500] + f"\n... [截断, 全文 {len(v)} 字符]"
|
||||
else:
|
||||
output[k] = v
|
||||
|
||||
with open(dump_path, "w", encoding="utf-8") as f:
|
||||
json.dump(output, f, indent=2, ensure_ascii=False)
|
||||
|
||||
print(f" 📁 已保存: {dump_path}")
|
||||
print(f" 页面元素: links={dom_snapshot.get('meta', {}).get('links_count','?')}, "
|
||||
f"buttons={dom_snapshot.get('meta', {}).get('buttons_count','?')}, "
|
||||
f"img={dom_snapshot.get('meta', {}).get('img_count','?')}")
|
||||
if dom_snapshot.get('note_count'):
|
||||
print(f" 笔记卡片数: {dom_snapshot.get('note_count')}")
|
||||
print(f" 页面 class 数: {len(dom_snapshot.get('all_classes', []))}")
|
||||
|
||||
# ── 9. 截图 ──
|
||||
print(f"\n 📸 截图")
|
||||
screenshot_path = SCRIPTS_DIR / "data" / f"xhs_reanalyze_{time.strftime('%Y%m%d_%H%M%S')}.png"
|
||||
try:
|
||||
await conn.page.screenshot(path=str(screenshot_path))
|
||||
print(f" ✅ 截图: {screenshot_path}")
|
||||
except:
|
||||
print(f" ⚠️ 截图失败")
|
||||
|
||||
# ── 10. 总结 ──
|
||||
print(f"\n{'='*60}")
|
||||
print(f" 📊 分析总结")
|
||||
print(f" {'='*60}")
|
||||
if all_ok:
|
||||
print(f" ✅ 所有选择器均匹配当前 DOM")
|
||||
print(f" → 当前 selectors.py 无需更新")
|
||||
else:
|
||||
print(f" ⚠️ 部分选择器未匹配到元素")
|
||||
print(f" → 建议更新 selectors.py 中的不匹配选择器")
|
||||
print(f" → 参考完整 DOM 快照了解最新结构")
|
||||
print(f"\n 💡 提示: 如果是登录后分析,对比登录前后的选择器差异")
|
||||
print(f" 尝试在探索页和笔记详情页分别运行此工具")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# 保持浏览器运行供检查
|
||||
print(f"\n 🔄 浏览器保持运行,关闭终端即退出")
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(30)
|
||||
except (KeyboardInterrupt, asyncio.CancelledError):
|
||||
pass
|
||||
finally:
|
||||
try:
|
||||
await conn.close()
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="小红书登录后 DOM 重分析工具"
|
||||
)
|
||||
parser.add_argument("identity_name", help="身份名称 (如 xhs_01)")
|
||||
parser.add_argument("--full-dump", "-f", action="store_true",
|
||||
help="输出完整 DOM 快照到 JSON 文件")
|
||||
parser.add_argument("--explore-url", "-u", default=None,
|
||||
help="自定义分析 URL (默认: /explore)")
|
||||
args = parser.parse_args()
|
||||
|
||||
asyncio.run(analyze(args.identity_name, args.full_dump, args.explore_url))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user