butubb
e348de48d3
本仓库在上游 MediaCrawler 之上加了一整层（见 UPSTREAM.md），可合并流程默认 「有人知道上游动了」。而部署是 git pull --ff-only，只从自己的 Gitea 拉——上游的 提交不主动 fetch 就永远看不见。拖着不合并的代价是复利的：越久越难合。 于是把「上游动了没有」变成一条会自己跑、会推企业微信的通知： * api/monitor/upstream.py：git fetch <url> <branch> 到 FETCH_HEAD，用 rev-list --count HEAD..FETCH_HEAD 算落后数、FETCH_HEAD..HEAD 算领先数。 用 git 而非托管商 API，因为只有 git 知道共同祖先在哪——本仓库含有上游没有的 提交，直接比 tip 会得出错误结论。增量 fetch 只传几个新提交，不会遇到 UPSTREAM.md 里说的「大包必断」。 * 只 fetch 到 FETCH_HEAD：不配 remote、不写 refs/remotes、不碰索引与工作区， 所以不打断正在跑的采集，也不和 deploy.sh 的 git pull 抢锁。 * 挂在调度器 tick 上（不是采集，所以不看 is_busy、不受活跃时段限制——定时检查 放在半夜反而最合适），按 checked_at + 间隔 到期才跑；失败也写 checked_at， 于是 GitHub 不通时是每间隔重试一次，而不是每个 tick 撞一次墙。 * 同一个 tip 只推一次（记 tip 而不是「推过没」），上游真又动了会再推。 * 两个接口：GET /monitor/upstream 只读缓存；POST /monitor/upstream/check 手动 查一次且刻意不推通知——点按钮的人正看着结果。 * 默认关闭，间隔默认一天。 Dockerfile 显式装 git（python:slim 不带，而这是唯一的依赖）；deploy.sh 顺带补上 一个真 bug 的提示：Dockerfile/requirements.txt 变了只 up -d 用的还是旧镜像。
72 lines
3.5 KiB
Docker
72 lines
3.5 KiB
Docker
# Server deployment image.
|
|
#
|
|
# No browser is bundled on purpose. On this deployment the crawler attaches over
|
|
# CDP to the Chrome already running on the host (see the 接管已有 Chrome setting),
|
|
# so shipping a second copy of Chromium would only add hundreds of megabytes and
|
|
# a login state that nothing uses. The Playwright Python package is still needed
|
|
# -- that is what speaks CDP -- hence PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD.
|
|
FROM python:3.11-slim
|
|
|
|
ENV PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
|
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 \
|
|
TZ=Asia/Shanghai
|
|
|
|
# asyncmy compiles a Cython extension, so a toolchain has to exist at build time.
|
|
# It is left installed: purging it risks taking libmysqlclient with it, and a
|
|
# slightly larger image is cheaper than a runtime that fails months later.
|
|
#
|
|
# deb.debian.org is effectively unusable from this network -- it was pulling the
|
|
# 96 MB of build dependencies at roughly 13 kB/s, which puts a build at well over
|
|
# half an hour. Point apt at a domestic mirror; override APT_MIRROR when building
|
|
# from somewhere that does not need it.
|
|
#
|
|
# The libgl1/libxcb1/... group is not for a GUI: opencv-python links against X11
|
|
# at import time, and tools/utils.py reaches cv2 through slider_util, so without
|
|
# them the *application* fails to import, not just some image utility. tzdata is
|
|
# here because TZ=Asia/Shanghai is silently ignored without it, which would put
|
|
# every stored timestamp in UTC. nodejs is for PyExecJS: douyin/help.py compiles
|
|
# libs/douyin.js at *import* time, and because main.py imports every platform,
|
|
# that single platform being importable-or-not decides whether the whole app
|
|
# (and the environment self-check) comes up. npm rides along so the WebUI can be
|
|
# rebuilt on the server (see deploy.sh) instead of only on a workstation --
|
|
# corepack is present but does not cover npm, only yarn and pnpm.
|
|
#
|
|
# The pip mirror is set for the same reason as the apt one: this host's route to
|
|
# the public index is slow.
|
|
#
|
|
# git is for the 上游更新检查 (api/monitor/upstream.py): it fetches the upstream
|
|
# repository into the mounted checkout to count how far behind this fork is.
|
|
# python:slim does not ship git, and nothing else here pulls it in.
|
|
ARG APT_MIRROR=mirrors.tuna.tsinghua.edu.cn
|
|
RUN set -eux; \
|
|
for f in /etc/apt/sources.list /etc/apt/sources.list.d/debian.sources; do \
|
|
if [ -f "$f" ]; then \
|
|
sed -i "s|deb.debian.org|${APT_MIRROR}|g; s|security.debian.org|${APT_MIRROR}|g" "$f"; \
|
|
fi; \
|
|
done; \
|
|
apt-get update; \
|
|
apt-get install -y --no-install-recommends \
|
|
build-essential pkg-config default-libmysqlclient-dev git \
|
|
libgl1 libglib2.0-0 libsm6 libxext6 libxrender1 libxcb1 libgomp1 \
|
|
tzdata nodejs npm; \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
WORKDIR /app
|
|
|
|
# Requirements only -- this is the one layer that is expensive to build and
|
|
# changes rarely.
|
|
ARG PIP_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple
|
|
COPY requirements.txt ./
|
|
RUN pip install --no-cache-dir -i "$PIP_INDEX" -r requirements.txt
|
|
|
|
# The application code is deliberately NOT copied in. compose mounts it at /app,
|
|
# so a code change is "re-upload the tarball, restart the container" instead of
|
|
# an image rebuild. Treat this image as the dependency layer and nothing else;
|
|
# rebuild it when, and only when, requirements.txt or this file changes.
|
|
EXPOSE 18051
|
|
|
|
# api.main reads MC_HOST / MC_PORT from the environment; compose supplies both.
|
|
CMD ["python", "-m", "api.main"]
|