butubb
7442bc10b8
移植自 mac-agent-os 的 mediacrawler_adapter：不起子进程、不开页面，用浏览器里那份
登录态直接调抖音 Web 接口。产物键名照抄 store/douyin，所以 ingest 那条链路一个字不用改。
**目前能用的（真环境实测，非推断）**：
profile/other : 200, 7075 字节 —— 博主主页指标（粉丝/获赞/作品数/昵称）
aweme/detail : 200, 45425 字节 —— 单条作品详情（含点赞/评论/收藏/分享）
**目前不能用的：作品列表 `aweme/post`。** 两个互相独立的原因：
1. 这个接口被抖音单独升级成了真校验：不带 x-tt-argus 回 403「Uifid Not Found」，
带上 dummy 值回 200 + **空 body**。也就是说「头在不在」骗得过，「真校验」过不了。
同一套头打 profile/other 和 aweme/detail 都是通的 —— 抖音是挑着接口加保护的，
挑中的恰好是「批量拉作品列表」这个最敏感的动作。
2. 改走页面截获也不行：CDP 浏览器打开博主主页会落到「验证码中间页」（当天大量探测的
代价，过几小时要重测）。
所以现在的边界是：**已知作品的指标刷新能做，自动发现新作品做不了**。
**排查中控住变量后得到的两条事实**（都写进注释了）：
· `Accept` / `Accept-Language` / `Referer` 才是主页接口能返回真数据的原因 —— 只有
UA+client hints+Cookie 时是 200 但仅 121 字节的空壳，补上这三个头变 7074 字节。
（我先前猜的 sec-ch-ua 不是关键。）
· 因此 UA 与 client hints 必须**成套地取自同一个浏览器**，所以 BrowserIdentity 一次
从 CDP 取齐 cookie + UA + hints，而不是各自写死。
「200 + 空 body 必须当场报错」也是刻意写死的：放过去它会在下游变成「这个博主没作品」，
把一次失败伪装成一条正常结果 —— 爬虫那条路正是这么栽的，还被翻译成「账号被封」。
顺带：把参考项目目录加进 .gitignore。上一次 `git add -A` 把 mac-agent-os-main 整个
（1429 个文件）带进了提交，已从历史里清掉。
测试 +11：cookie 解析、请求头成套性（含 uifid 缺失/回退）、产物键名与 store 对齐、
以及 _get 的三条失败路径（空 body / 403 带网关原话 / 正常返回）。
191 lines
3.6 KiB
Plaintext
191 lines
3.6 KiB
Plaintext
# Byte-compiled / optimized / DLL files
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# C extensions
|
|
*.so
|
|
|
|
# Distribution / packaging
|
|
.Python
|
|
build/
|
|
develop-eggs/
|
|
dist/
|
|
downloads/
|
|
eggs/
|
|
.eggs/
|
|
/lib/
|
|
lib64/
|
|
parts/
|
|
sdist/
|
|
var/
|
|
wheels/
|
|
share/python-wheels/
|
|
*.egg-info/
|
|
.installed.cfg
|
|
*.egg
|
|
MANIFEST
|
|
|
|
# PyInstaller
|
|
# Usually these files are written by a python script from a template
|
|
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
*.manifest
|
|
*.spec
|
|
|
|
# Installer logs
|
|
pip-log.txt
|
|
pip-delete-this-directory.txt
|
|
|
|
# Unit test / coverage reports
|
|
htmlcov/
|
|
.tox/
|
|
.nox/
|
|
.coverage
|
|
.coverage.*
|
|
.cache
|
|
nosetests.xml
|
|
coverage.xml
|
|
*.cover
|
|
*.py,cover
|
|
.hypothesis/
|
|
.pytest_cache/
|
|
cover/
|
|
|
|
# Translations
|
|
*.mo
|
|
*.pot
|
|
|
|
# Django stuff:
|
|
*.log
|
|
local_settings.py
|
|
db.sqlite3
|
|
db.sqlite3-journal
|
|
|
|
# Flask stuff:
|
|
instance/
|
|
.webassets-cache
|
|
|
|
# Scrapy stuff:
|
|
.scrapy
|
|
|
|
# Sphinx documentation
|
|
docs/_build/
|
|
|
|
# PyBuilder
|
|
.pybuilder/
|
|
target/
|
|
|
|
# Jupyter Notebook
|
|
.ipynb_checkpoints
|
|
|
|
# IPython
|
|
profile_default/
|
|
ipython_config.py
|
|
|
|
# pyenv
|
|
# For a library or package, you might want to ignore these files since the code is
|
|
# intended to run in multiple environments; otherwise, check them in:
|
|
# .python-version
|
|
|
|
# pipenv
|
|
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
|
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
|
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
|
# install all needed dependencies.
|
|
#Pipfile.lock
|
|
|
|
# poetry
|
|
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
|
|
# This is especially recommended for binary packages to ensure reproducibility, and is more
|
|
# commonly ignored for libraries.
|
|
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
|
|
#poetry.lock
|
|
|
|
# pdm
|
|
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
|
|
#pdm.lock
|
|
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
|
|
# in version control.
|
|
# https://pdm.fming.dev/#use-with-ide
|
|
.pdm.toml
|
|
|
|
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
|
|
__pypackages__/
|
|
|
|
# Celery stuff
|
|
celerybeat-schedule
|
|
celerybeat.pid
|
|
|
|
# SageMath parsed files
|
|
*.sage.py
|
|
|
|
# Environments
|
|
.env
|
|
.venv
|
|
/env/
|
|
venv/
|
|
/ENV/
|
|
env.bak/
|
|
venv.bak/
|
|
|
|
# Spyder project settings
|
|
.spyderproject
|
|
.spyproject
|
|
|
|
# Rope project settings
|
|
.ropeproject
|
|
|
|
# mkdocs documentation
|
|
/site
|
|
|
|
# mypy
|
|
.mypy_cache/
|
|
.dmypy.json
|
|
dmypy.json
|
|
|
|
# Pyre type checker
|
|
.pyre/
|
|
|
|
# pytype static type analyzer
|
|
.pytype/
|
|
|
|
# Cython debug symbols
|
|
cython_debug/
|
|
|
|
# PyCharm
|
|
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
|
|
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
|
|
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
|
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
|
#.idea/
|
|
|
|
*.xml
|
|
*.iml
|
|
.idea
|
|
/temp_image/
|
|
/browser_data/
|
|
/data/
|
|
|
|
*/.DS_Store
|
|
.vscode
|
|
/node_modules
|
|
webui/node_modules
|
|
docs/.vitepress/cache
|
|
|
|
# WebUI build output (generated by `cd webui && npm run build`)
|
|
api/webui/
|
|
|
|
# other gitignore
|
|
.venv
|
|
.refer
|
|
|
|
agent_zone
|
|
debug_tools
|
|
|
|
database/*.db
|
|
.omx/
|
|
|
|
# 别人放在这儿的参考项目（mac-agent-os）。它是独立仓库、21MB，不属于本项目 ——
|
|
# 一旦被 `git add -A` 扫进来就是永久留在历史里（踩过一次：1601 个文件里 1429 个是它）。
|
|
# 要读它就直接读磁盘上的目录，别提交。
|
|
mac-agent-os-main/ |