butubb
c29516cff5
需求：任务编辑器里能单独配"这个任务每隔 N 秒检查一次"——熄屏就点亮、某个元素
出现就通知、掉出 App 就停本设备；通知标题正文要能自己写。
配置与执行分离（这是本次的关键设计）：
- **配置是任务级的**（`params.watchers`），在任务编辑器单独一块，不进步骤画布；
- **执行是穿插的**：worker 每执行完一步、以及长等待的每个分片，看一眼哪个巡检
到点了。不起线程 → 不需要并发模型，也不会和主流程抢屏幕（两边同时点屏幕会
互相打断）。代价是精度受步长影响（某步卡 30s，巡检最多晚 30s），已在文档写明。
- core/patrol.py（新）：检查项/动作注册表（CHECKS/ACTIONS）+ evaluate/act。
检查：屏幕熄灭/亮着、元素存在/不存在、前台是/不是某 App；
动作：只通知、点亮、息屏、停止本设备。屏幕走 `dumpsys power`（0.3s，
不用 d.info——那玩意在部分设备要 14s），前台走 d.app_current()（0.7s）。
- tasks/generic/task.py：`_maybe_patrol` / `_run_patrol`（冷却、命中记一条
步骤明细、发通知）；**文案在动作之前渲染**——点亮后 {screen} 就成了"亮屏"，
用户要看的是"发现熄屏，已点亮"。
- 任务编辑器新增「公共巡检」块（static/admin/tasks.js）+ 样式；保存进 params.watchers。
- 通知：新增事件 `task.patrol.hit`（巡检命中）与 `task.notify.custom`（步骤发通知）；
给了 title 就用它当标题（不再拼前缀），level 字段可点名级别。
顺带（巡检需要的原语，也可单独用）：
- if_el 条件判断支持 `selector_type=screen`（亮/熄）与 `foreground`（前台包名）；
非元素条件不参与「选择器健康」统计（否则会攒出假的"选择器失效"告警）。
- 新增两个步骤：`notify`（发自定义通知）、`stop_self`（停本设备，记"被停止"
而不是失败，不触发重试）。步骤类型 18 → 20，相关文档计数一并更新。
修 bug：`wait` 步骤在巡检耗时超过剩余时间后 `sleep(负数)` 抛
"sleep length must be non-negative"（真机联调抓到，已 clamp 到 0）。
自测：假设备单测 12 组（命中/冷却/间隔/元素/前台/停止/静默/异常不炸）；
真机联调：熄屏→点亮（False→True）+ 通知文案正确、掉出抖音按间隔命中 5 次、
长等待里穿插生效且步骤回到 ok、清理后用户通知配置原样恢复。
文档：TASK_DEV §4.5（含两个可抄的例子）与步骤表/条件类型、NOTIFY §3、README。
1153 lines
54 KiB
Python
1153 lines
54 KiB
Python
"""通用任务管理框架：任务类型注册 + 设备分组 + 任务计划 + 定时调度 + 重试 + 持久化。
|
||
|
||
设计目标：可扩展——新增任务类型只需在 tasks/ 下注册新的 Task 类（见 doc/TASK_DEV.md）。
|
||
|
||
核心概念：
|
||
TaskType — 任务类型（如"通用步骤"），可注册，含默认参数和 worker 工厂
|
||
DeviceGroup — 设备分组，持久化到 SQLite（core.models.DeviceGroup）
|
||
TaskJob — 任务计划（类型+目标+参数+调度+重试），持久化到 SQLite（core.models.TaskJob）
|
||
TaskManager — 统管调度器、分组、任务、运行实例、状态
|
||
|
||
定时：APScheduler BackgroundScheduler，cron 触发后为每个目标设备起 worker。
|
||
支持三种调度模式：
|
||
once — 手动/立即执行，不注册 cron
|
||
cron — 仅注册启动 cron（到点启动 worker）
|
||
cron_stop — 注册启动 cron + 停止 cron（到点启动，到点停止）
|
||
重试：单设备 worker 失败后按 retry.max_attempts 重试，间隔 retry.delay。
|
||
并发：同一 serial 同时只允许一个 worker，避免冲突。
|
||
"""
|
||
import os
|
||
import json
|
||
import time
|
||
import uuid
|
||
import threading
|
||
from collections import Counter
|
||
from datetime import datetime
|
||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||
|
||
from apscheduler.schedulers.background import BackgroundScheduler
|
||
from apscheduler.triggers.cron import CronTrigger
|
||
|
||
from config import DATA_DIR
|
||
from core.logger import get_logger
|
||
from core.models import db, DeviceGroup as GroupRow, TaskJob as JobRow
|
||
from core import device_pool
|
||
from core import notifier
|
||
from .adb_helper import get_foreground_app, get_foreground_app_remote
|
||
from .device_worker import (
|
||
DeviceOfflineError,
|
||
get_all_worker_status, _update_status, _remove_worker,
|
||
_WORKERS, _WORKERS_LOCK,
|
||
start_watchdog, stop_watchdog,
|
||
)
|
||
from tasks import list_task_types, get_task_class
|
||
|
||
_log = get_logger("core.tm")
|
||
|
||
# 错峰启动间隔（秒）：批量触发时设备逐个开始连接，避免 adb 全局锁串行堆积。
|
||
# 100 台 × 0.2s = 20s 全部开始启动。
|
||
_START_STAGGER_SEC = 0.2
|
||
|
||
|
||
def _dev_model(serial, tracker=None):
|
||
"""设备型号：优先用批次开始时查好的设备池快照，退回 worker 上报的状态表。
|
||
|
||
为什么优先快照：worker 每次启动都会 `_update_status(model=…)`，但那次采集
|
||
（`d.info()`）可能超时/失败 → 型号是空；而设备池的型号是后台统一采集的、
|
||
稳定得多。快照还有个好处：不查库（通知路径上不碰 DB）。
|
||
"""
|
||
if tracker is not None:
|
||
m = (tracker.models or {}).get(serial)
|
||
if m:
|
||
return m
|
||
with _WORKERS_LOCK:
|
||
return _WORKERS.get(serial, {}).get("model") or ""
|
||
|
||
|
||
def _dev_fail_note(serial, dname, tracker=None):
|
||
"""失败设备的一行说明：`名字（型号）：原因`——批次汇总里直接列出来。
|
||
|
||
型号同上用快照；原因取 worker 上报的 last_error（刚写进去，还在）。
|
||
"""
|
||
with _WORKERS_LOCK:
|
||
err = _WORKERS.get(serial, {}).get("last_error") or ""
|
||
cause = err.strip().splitlines()[0] if err.strip() else ""
|
||
model = _dev_model(serial, tracker)
|
||
head = f"{dname}（{model}）" if model else str(dname)
|
||
return f"{head}：{cause[:60]}" if cause else head
|
||
|
||
|
||
class _BatchTracker:
|
||
"""一次任务批次的收尾统计：**所有设备都出结果**后发一条 `task.batch.finished`。
|
||
|
||
为什么不每台设备各发一条"批次结束"：100 台设备就是 100 条刷屏。这里只计数，
|
||
归零那一刻发一条汇总（锁外发，见 done()）。
|
||
|
||
每台设备的结果由 `_run_with_retry` 的 finally 调一次 `done(serial, outcome)`——
|
||
放在 finally 里是为了保证"任何一个 return 分支都会被计数一次"，不会漏也不会重。
|
||
"""
|
||
|
||
def __init__(self, job, total, names=None, models=None):
|
||
self.job = job
|
||
self.total = total
|
||
# serial -> 设备名（通知里显示名字而不是裸地址）；取不到就是空
|
||
self.names = dict(names or {})
|
||
# serial -> 型号（批次开始时从设备池查一次带下来，通知里直接用）
|
||
self.models = dict(models or {})
|
||
self._lock = threading.Lock()
|
||
self._left = total
|
||
self._stats = Counter()
|
||
self._failed = [] # 失败设备明细（型号 + 原因），汇总里有失败时列出
|
||
self._t0 = time.time()
|
||
|
||
def done(self, serial, outcome="failed", detail=""):
|
||
"""登记一台设备的结果（success/failed/stopped/skipped）。
|
||
|
||
detail 只对 failed 有意义（"设备名（型号）：原因"），最多留 5 条 ——
|
||
批次汇总里列出来，用户不用再去翻日志。
|
||
"""
|
||
with self._lock:
|
||
self._left -= 1
|
||
self._stats[outcome] += 1
|
||
if outcome == "failed" and detail and len(self._failed) < 5:
|
||
self._failed.append(detail)
|
||
left = self._left
|
||
s = dict(self._stats)
|
||
fails = list(self._failed)
|
||
if left > 0:
|
||
return
|
||
notifier.notify("task.batch.finished", job_id=self.job.id,
|
||
job_name=self.job.name, total=self.total,
|
||
success=s.get("success", 0), failed=s.get("failed", 0),
|
||
stopped=s.get("stopped", 0), skipped=s.get("skipped", 0),
|
||
failed_devices=fails,
|
||
duration_s=int(time.time() - self._t0))
|
||
|
||
|
||
def _in_run_window(schedule, now=None):
|
||
"""是否在当前运行窗口内。
|
||
|
||
schedule.window = {"start": "HH:MM", "end": "HH:MM"}，每天重复；
|
||
窗口外（定时触发 + 手动执行）任务不会启动。未配置/非法配置视为不限制。
|
||
支持跨午夜（如 21:00-09:00 = 晚上 9 点运行到次日早 9 点）。
|
||
"""
|
||
win = (schedule or {}).get("window") or {}
|
||
start, end = win.get("start", ""), win.get("end", "")
|
||
if not start or not end:
|
||
return True
|
||
try:
|
||
cur = (now or datetime.now()).hour * 60 + (now or datetime.now()).minute
|
||
s = int(start.split(":")[0]) * 60 + int(start.split(":")[1])
|
||
e = int(end.split(":")[0]) * 60 + int(end.split(":")[1])
|
||
except (ValueError, AttributeError, IndexError):
|
||
return True # 配置非法按不限制处理
|
||
if s == e:
|
||
return True # 起止相同视为不限制
|
||
if s < e:
|
||
return s <= cur < e
|
||
return cur >= s or cur < e # 跨午夜
|
||
|
||
|
||
def _next_run_time(schedule, now=None):
|
||
"""任务下次真正执行的时间（考虑运行窗口）。
|
||
|
||
用 CronTrigger.get_next_fire_time 从当前时间向后找触发点，
|
||
跳过运行窗口外的触发点（最多找 200 次防死循环）。
|
||
返回 datetime 或 None（无 cron / 配置非法）。
|
||
"""
|
||
sched = schedule or {}
|
||
if sched.get("mode") not in ("cron", "cron_stop"):
|
||
return None
|
||
cron = sched.get("cron", "")
|
||
if not cron:
|
||
return None
|
||
try:
|
||
trigger = CronTrigger.from_crontab(cron)
|
||
except Exception:
|
||
return None
|
||
now = now or datetime.now()
|
||
win = sched.get("window") or {}
|
||
if not win.get("start") or not win.get("end"):
|
||
return trigger.get_next_fire_time(None, now)
|
||
fire = trigger.get_next_fire_time(None, now)
|
||
for _ in range(200):
|
||
if fire is None:
|
||
return None
|
||
if _in_run_window({"window": win}, fire):
|
||
return fire
|
||
fire = trigger.get_next_fire_time(fire, fire)
|
||
return None
|
||
|
||
|
||
# ================== 设备分组 ==================
|
||
class DeviceGroup:
|
||
def __init__(self, name, serials=None, description=""):
|
||
self.name = name
|
||
self.serials = serials or []
|
||
self.description = description
|
||
|
||
def to_dict(self):
|
||
return {"name": self.name, "serials": self.serials, "description": self.description}
|
||
|
||
@classmethod
|
||
def from_dict(cls, d):
|
||
return cls(d["name"], d.get("serials", []), d.get("description", ""))
|
||
|
||
|
||
# ================== 任务计划 ==================
|
||
class TaskJob:
|
||
"""一个任务计划：什么任务、跑哪些设备、什么参数、何时跑、失败重试策略。"""
|
||
|
||
def __init__(self, job_id=None, name="", task_type="generic_steps",
|
||
target=None, params=None, schedule=None, retry=None, enabled=True):
|
||
self.id = job_id or uuid.uuid4().hex[:8]
|
||
self.name = name
|
||
self.task_type = task_type
|
||
# target: {"mode": "all"|"group"|"serial", "group_name": "", "serial": ""}
|
||
self.target = target or {"mode": "all"}
|
||
self.params = params or {}
|
||
self.schedule = schedule or {"mode": "once"} # {"mode":"once"|"cron", "cron":"0 9 * * *"}
|
||
self.retry = retry or {"max_attempts": 1, "delay": 60}
|
||
self.enabled = enabled
|
||
|
||
def to_dict(self):
|
||
return {"id": self.id, "name": self.name, "task_type": self.task_type,
|
||
"target": self.target, "params": self.params,
|
||
"schedule": self.schedule, "retry": self.retry, "enabled": self.enabled}
|
||
|
||
@classmethod
|
||
def from_dict(cls, d):
|
||
return cls(d["id"], d["name"], d["task_type"], d.get("target"),
|
||
d.get("params"), d.get("schedule"), d.get("retry"), d.get("enabled", True))
|
||
|
||
def resolve_serials(self, manager):
|
||
"""根据 target 解析出实际要跑的 serial 列表。
|
||
|
||
数据源：core.device_pool（本地清单 + adb 在线状态），不再查 STF。
|
||
"""
|
||
mode = self.target.get("mode", "all")
|
||
if mode == "serial":
|
||
serials = [self.target["serial"]]
|
||
elif mode == "group":
|
||
g = manager.groups.get(self.target.get("group_name"))
|
||
serials = list(g.serials) if g else []
|
||
# 分组应引用设备池内设备：不在池内的（如手动填的旧 IP）不参与调度，
|
||
# 与 all/preempt 模式口径一致（2026-09-01）
|
||
try:
|
||
configured = set(device_pool.list_configured())
|
||
serials = [s for s in serials if s in configured]
|
||
except Exception:
|
||
pass
|
||
else:
|
||
# all：默认返回池内在线设备（单实例互斥由 _run_with_retry 的 _running 保证）；
|
||
# 抢占模式返回全部在线设备（含运行中，执行时抢占）
|
||
if self.params.get("preempt"):
|
||
try:
|
||
# 抢占模式：只抢设备池内的在线设备——
|
||
# list_online() 会包含待连接/未入池设备（如自动发现验证时连上的），
|
||
# 它们不在设备池就不该被任务调度（2026-09-01 修复）
|
||
online = set(device_pool.list_online())
|
||
configured = set(device_pool.list_configured())
|
||
return [s for s in device_pool.list_online() if s in configured]
|
||
except Exception:
|
||
return []
|
||
try:
|
||
return device_pool.list_ready()
|
||
except Exception:
|
||
return []
|
||
# 离线自动跳过（serial/group 模式）：跳过本机 adb 不可达的设备，
|
||
# 避免反复尝试连接后报"设备离线"。默认开启，可在任务编辑器取消勾选。
|
||
if serials and self.params.get("skip_offline", True):
|
||
try:
|
||
online = set(device_pool.list_online())
|
||
except Exception:
|
||
online = None # 查询失败时不过滤，维持原行为
|
||
if online is not None:
|
||
kept, skipped = [], []
|
||
for s in serials:
|
||
if s in online:
|
||
kept.append(s)
|
||
else:
|
||
skipped.append((s, "离线(未连接)"))
|
||
if skipped:
|
||
detail = ", ".join(f"{s}({why})" for s, why in skipped)
|
||
_log.info(f"任务 {self.name} 跳过 {len(skipped)} 台设备: {detail}")
|
||
serials = kept
|
||
return serials
|
||
|
||
|
||
# ================== 前台 App 扫描器 ==================
|
||
class _ForegroundScanner:
|
||
"""前台 App 扫描器：手动触发，获取所有在线设备的前台 App。
|
||
|
||
设计原则：**不打扰设备**，扫描过程不会让设备退出当前 App。
|
||
|
||
策略（按设备状态区分获取逻辑）：
|
||
- worker 运行中设备：用已有 remote_adb_url 直接查询（已有 adb 连接，无额外开销）
|
||
- USB 运行中设备：经远程 adb server（220）查询
|
||
- 空闲设备：不主动连接（返回"空闲"）——IP:5555 的 adb transport 与 220
|
||
共享，外部 connect/disconnect 会扰动共享连接
|
||
|
||
serial 格式为 IP:5555（设备本身的 adb 网络地址），可直接 adb connect。
|
||
adb connect/disconnect 只建立/断开调试连接，不影响设备 UI。
|
||
dumpsys window 只读取窗口状态，不执行任何操作。
|
||
"""
|
||
|
||
def __init__(self):
|
||
self._cache = {} # serial -> app_name
|
||
self._cache_lock = threading.Lock()
|
||
self._scanning = threading.Event() # 标记是否正在扫描
|
||
self._last_scan_time = 0 # 上次扫描完成时间戳
|
||
self._last_scan_count = 0 # 上次扫描设备数
|
||
|
||
def stop(self):
|
||
"""兼容 TaskManager.shutdown() 调用。"""
|
||
pass
|
||
|
||
def get(self, serial):
|
||
"""读取缓存的前台 App。"""
|
||
with self._cache_lock:
|
||
return self._cache.get(serial)
|
||
|
||
@property
|
||
def is_scanning(self):
|
||
return self._scanning.is_set()
|
||
|
||
@property
|
||
def last_scan_time(self):
|
||
return self._last_scan_time
|
||
|
||
def scan_once(self):
|
||
"""手动触发一次扫描（后台线程执行，不阻塞调用者）。
|
||
|
||
如果已有扫描在进行中，直接返回 False。
|
||
"""
|
||
if self._scanning.is_set():
|
||
return False
|
||
t = threading.Thread(target=self._scan_all, name="fg-scan-once", daemon=True)
|
||
t.start()
|
||
return True
|
||
|
||
def _scan_all(self):
|
||
"""扫描所有在线设备（后台线程执行）。
|
||
|
||
按设备归属分两类处理：
|
||
1. worker 运行中：用已有 remote_adb_url 直接查询（无额外开销）
|
||
2. 空闲设备：返回"空闲"——不主动连接。IP:5555 的 adb transport
|
||
与 STF provider 共享，外部 connect/disconnect 会让 STF 误判
|
||
设备离线并触发重连（迁移期仍保留此约束，摘除 STF 后可放开）
|
||
"""
|
||
self._scanning.set()
|
||
_log.info("前台 App 扫描已启动")
|
||
try:
|
||
online = set(device_pool.list_online())
|
||
except Exception as e:
|
||
_log.error("前台 App 扫描: 获取在线设备失败: %s", e)
|
||
self._scanning.clear()
|
||
return
|
||
|
||
worker_status = {w["serial"]: w for w in get_all_worker_status()}
|
||
|
||
# 分类设备
|
||
have_conn = {} # serial -> remote_adb_url（IP:5555 运行中设备）
|
||
usb_conn = {} # serial -> usb_server（USB 运行中设备，经远程 adb server）
|
||
free_serials = [] # 空闲设备
|
||
|
||
for serial in online:
|
||
w = worker_status.get(serial, {})
|
||
if w.get("status") not in ("running", "connecting"):
|
||
free_serials.append(serial)
|
||
elif w.get("remote_adb_url"):
|
||
# 1. IP:5555 设备正在执行任务，已有 adb 连接，直接查询
|
||
have_conn[serial] = w["remote_adb_url"]
|
||
elif w.get("usb_server"):
|
||
# 1b. USB 设备运行中：经远程 adb server 查询
|
||
usb_conn[serial] = w["usb_server"]
|
||
else:
|
||
free_serials.append(serial)
|
||
|
||
results = {}
|
||
_log.info("前台 App 扫描分类: 运行中=%d, USB运行中=%d, 空闲=%d",
|
||
len(have_conn), len(usb_conn), len(free_serials))
|
||
|
||
# 1. worker 运行中设备：用已有 remote_adb_url 查询（并发 10）
|
||
if have_conn:
|
||
with ThreadPoolExecutor(max_workers=min(10, len(have_conn))) as pool:
|
||
futures = {pool.submit(get_foreground_app, url): s
|
||
for s, url in have_conn.items()}
|
||
for fut in as_completed(futures, timeout=15):
|
||
s = futures[fut]
|
||
try:
|
||
results[s] = fut.result()
|
||
except Exception:
|
||
results[s] = None
|
||
|
||
# 1b. USB 运行中设备：经远程 adb server 查询（并发 5）
|
||
if usb_conn:
|
||
with ThreadPoolExecutor(max_workers=min(5, len(usb_conn))) as pool:
|
||
futures = {pool.submit(self._scan_usb_conn, s): s
|
||
for s, server in usb_conn.items()}
|
||
for fut in as_completed(futures, timeout=15):
|
||
s = futures[fut]
|
||
try:
|
||
results[s] = fut.result()
|
||
except Exception:
|
||
results[s] = None
|
||
|
||
# 2. 空闲设备：不打扰，直接返回"空闲"
|
||
for s in free_serials:
|
||
results[s] = "空闲"
|
||
|
||
# 更新缓存
|
||
with self._cache_lock:
|
||
for s, app in results.items():
|
||
self._cache[s] = app
|
||
|
||
self._last_scan_time = time.time()
|
||
self._last_scan_count = len(results)
|
||
self._scanning.clear()
|
||
_log.info("前台 App 扫描完成: %d 台设备", len(results))
|
||
|
||
def _scan_usb_conn(self, serial):
|
||
"""USB 运行中设备：经远程 adb server（220）查询前台 App。"""
|
||
try:
|
||
with _WORKERS_LOCK:
|
||
server = _WORKERS.get(serial, {}).get("usb_server", "")
|
||
if not server:
|
||
return None
|
||
host, _, port = server.partition(":")
|
||
return get_foreground_app_remote(host, int(port), serial)
|
||
except Exception:
|
||
return None
|
||
|
||
|
||
# ================== 任务管理器 ==================
|
||
class TaskManager:
|
||
def __init__(self, app=None):
|
||
self.app = app # Flask app，用于 db context
|
||
self.scheduler = BackgroundScheduler(timezone="Asia/Shanghai")
|
||
if os.environ.get("DISABLE_SCHEDULER"):
|
||
# 测试/回归模式：不启动 cron 调度器，避免 test_client 加载任务后
|
||
# 按 cron 真的跑任务（干扰回归、占用设备）。add_job 仍可调用不报错。
|
||
_log.info("DISABLE_SCHEDULER=1：跳过 cron 调度器（测试/回归模式）")
|
||
else:
|
||
self.scheduler.start()
|
||
start_watchdog() # 启动心跳看门狗
|
||
self.groups = {} # name -> DeviceGroup(内存业务对象)
|
||
self.jobs = {} # id -> TaskJob(内存业务对象)
|
||
self._running = {} # serial -> {"worker", "job_id", "started_at", "attempt"}
|
||
self._stop_requested = set() # serial 集合：用户请求停止，阻止后续重试
|
||
self._lock = threading.Lock()
|
||
self._fg_scanner = _ForegroundScanner()
|
||
self._load()
|
||
|
||
def _db(self):
|
||
"""获取 db context。task_manager 在后台线程跑，需自己推 app context。"""
|
||
if self.app is None:
|
||
raise RuntimeError("TaskManager 未关联 Flask app，无法访问数据库")
|
||
return self.app.app_context()
|
||
|
||
# ---- 持久化（SQLite，替代旧 JSON 文件）----
|
||
def _load(self):
|
||
"""从数据库加载分组和任务到内存。"""
|
||
try:
|
||
with self._db():
|
||
for row in GroupRow.query.all():
|
||
g = DeviceGroup(row.name, row.get_serials(), row.description or "")
|
||
self.groups[g.name] = g
|
||
stale = [] # 任务类型已不存在（历史任务类型被删除）的任务
|
||
for row in JobRow.query.all():
|
||
j = TaskJob(job_id=row.id, name=row.name, task_type=row.task_type,
|
||
target=row.get_target(), params=row.get_params(),
|
||
schedule=row.get_schedule(), retry=row.get_retry(),
|
||
enabled=row.enabled)
|
||
# 启动时告警：库里留着已删除的任务类型（如 douyin_nurture），
|
||
# 这类任务跑不起来也不该静默——提示人工删除或改用现有类型。
|
||
# 只告警不改数据（不自动删用户的任务）。
|
||
if not get_task_class(j.task_type):
|
||
stale.append(j)
|
||
self.jobs[j.id] = j
|
||
if stale:
|
||
_log.warning(
|
||
"以下任务的任务类型已不存在，无法执行，请在「任务」页删除或改用现有类型："
|
||
+ "；".join(f"{j.name}({j.id}, type={j.task_type})" for j in stale))
|
||
_log.info(f"从数据库加载 {len(self.groups)} 个分组, {len(self.jobs)} 个任务")
|
||
except Exception as e:
|
||
_log.error(f"从数据库加载失败: {e}")
|
||
# 恢复启用的 cron 任务
|
||
for job in self.jobs.values():
|
||
if job.enabled and job.schedule.get("mode") in ("cron", "cron_stop"):
|
||
self._add_cron(job)
|
||
|
||
def _save_groups(self):
|
||
"""保存所有分组到数据库（upsert）。"""
|
||
with self._db():
|
||
for g in self.groups.values():
|
||
row = GroupRow.query.filter_by(name=g.name).first()
|
||
if not row:
|
||
row = GroupRow(name=g.name)
|
||
db.session.add(row)
|
||
row.set_serials(g.serials)
|
||
row.description = g.description
|
||
db.session.commit()
|
||
|
||
def _save_jobs(self):
|
||
"""保存所有任务到数据库（upsert）。"""
|
||
with self._db():
|
||
for j in self.jobs.values():
|
||
row = JobRow.query.get(j.id)
|
||
if not row:
|
||
row = JobRow(id=j.id)
|
||
db.session.add(row)
|
||
row.name = j.name
|
||
row.task_type = j.task_type
|
||
row.set_target(j.target)
|
||
row.set_params(j.params)
|
||
row.set_schedule(j.schedule)
|
||
row.set_retry(j.retry)
|
||
row.enabled = j.enabled
|
||
db.session.commit()
|
||
|
||
# ---- 分组管理 ----
|
||
def add_group(self, name, serials, description=""):
|
||
g = DeviceGroup(name, serials, description)
|
||
self.groups[name] = g
|
||
self._save_groups()
|
||
return g
|
||
|
||
def update_group(self, name, serials=None, description=None):
|
||
g = self.groups.get(name)
|
||
if not g:
|
||
return None
|
||
if serials is not None:
|
||
g.serials = serials
|
||
if description is not None:
|
||
g.description = description
|
||
self._save_groups()
|
||
return g
|
||
|
||
def sync_device_serial(self, old_serial, new_serial):
|
||
"""设备换地址（认领/迁址）后，把分组与任务里的引用改到新地址。
|
||
|
||
由 device_pool 的迁址回调触发（见 device_pool.set_move_hook）。
|
||
**必须同时改内存与库**：调度用的是内存里的 groups/jobs，只改库不重启不生效
|
||
—— 表现就是"分组里少了一台、指定设备的任务仍跑向不存在的地址"。
|
||
|
||
返回被更新的引用条数。
|
||
"""
|
||
if not old_serial or not new_serial or old_serial == new_serial:
|
||
return 0
|
||
n = 0
|
||
for name, g in list(self.groups.items()):
|
||
try:
|
||
serials = list(g.serials or [])
|
||
except Exception:
|
||
continue
|
||
if old_serial in serials:
|
||
self.update_group(name, serials=[new_serial if s == old_serial else s
|
||
for s in serials])
|
||
n += 1
|
||
_log.info(f"设备换地址: 分组『{name}』的 {old_serial} → {new_serial}")
|
||
for job_id, job in list(self.jobs.items()):
|
||
try:
|
||
target = dict(job.target or {})
|
||
except Exception:
|
||
continue
|
||
if target.get("mode") == "serial" and target.get("serial") == old_serial:
|
||
target["serial"] = new_serial
|
||
self.update_job(job_id, target=target)
|
||
n += 1
|
||
_log.info(f"设备换地址: 任务『{job.name}』目标 → {new_serial}")
|
||
return n
|
||
|
||
def delete_group(self, name):
|
||
removed = False
|
||
if name in self.groups:
|
||
del self.groups[name]
|
||
self._save_groups()
|
||
removed = True
|
||
# 无论内存里有没有都尝试删库行：内存与库不一致时（手工插行、上次异常退出、
|
||
# 上个版本漏删）也能删干净，避免"删不掉的分组"
|
||
with self._db():
|
||
row = GroupRow.query.filter_by(name=name).first()
|
||
if row:
|
||
db.session.delete(row)
|
||
db.session.commit()
|
||
removed = True
|
||
return removed
|
||
|
||
# ---- 任务计划管理 ----
|
||
def add_job(self, name, task_type, target, params, schedule, retry, enabled=True):
|
||
job = TaskJob(name=name, task_type=task_type, target=target, params=params,
|
||
schedule=schedule, retry=retry, enabled=enabled)
|
||
self.jobs[job.id] = job
|
||
if enabled and schedule.get("mode") in ("cron", "cron_stop"):
|
||
self._add_cron(job)
|
||
self._save_jobs()
|
||
return job
|
||
|
||
def update_job(self, job_id, **fields):
|
||
job = self.jobs.get(job_id)
|
||
if not job:
|
||
return None
|
||
# 更新前先移除旧 cron
|
||
was_cron = job.enabled and job.schedule.get("mode") in ("cron", "cron_stop")
|
||
if was_cron:
|
||
self._remove_cron(job_id)
|
||
for k, v in fields.items():
|
||
if v is not None and hasattr(job, k):
|
||
setattr(job, k, v)
|
||
if job.enabled and job.schedule.get("mode") in ("cron", "cron_stop"):
|
||
self._add_cron(job)
|
||
self._save_jobs()
|
||
return job
|
||
|
||
def delete_job(self, job_id):
|
||
self._remove_cron(job_id)
|
||
removed = False
|
||
if job_id in self.jobs:
|
||
del self.jobs[job_id]
|
||
self._save_jobs()
|
||
removed = True
|
||
# _save_jobs 只做 upsert 不会删行：必须显式删库行，否则删除的任务重启会复活。
|
||
# 内存里没有（手工插行/上次异常退出）时也要删——否则会出现"删不掉的任务"
|
||
with self._db():
|
||
row = JobRow.query.get(job_id)
|
||
if row:
|
||
db.session.delete(row)
|
||
db.session.commit()
|
||
removed = True
|
||
return removed
|
||
|
||
def toggle_job(self, job_id, enabled):
|
||
job = self.jobs.get(job_id)
|
||
if not job:
|
||
return None
|
||
job.enabled = enabled
|
||
if enabled and job.schedule.get("mode") in ("cron", "cron_stop"):
|
||
self._add_cron(job)
|
||
else:
|
||
self._remove_cron(job_id)
|
||
self._save_jobs()
|
||
return job
|
||
|
||
# ---- APScheduler 集成 ----
|
||
def _cron_id(self, job_id, kind="start"):
|
||
"""生成 cron job id。kind: start=启动 cron，stop=停止 cron。"""
|
||
return f"job_{job_id}_{kind}"
|
||
|
||
def _add_cron(self, job):
|
||
"""注册定时任务。支持三种模式：
|
||
- once: 不注册 cron
|
||
- cron: 只注册启动 cron（到点启动 worker）
|
||
- cron_stop: 注册启动 cron + 停止 cron（到点启动，到点停止）
|
||
"""
|
||
mode = job.schedule.get("mode", "once")
|
||
if mode not in ("cron", "cron_stop"):
|
||
return
|
||
|
||
start_cron = job.schedule.get("cron", "")
|
||
if start_cron:
|
||
try:
|
||
trigger = CronTrigger.from_crontab(start_cron)
|
||
except Exception as e:
|
||
_log.error(f"启动 cron 解析失败 {start_cron}: {e}")
|
||
else:
|
||
self.scheduler.add_job(self._on_cron_trigger, trigger, args=[job.id],
|
||
id=self._cron_id(job.id, "start"),
|
||
replace_existing=True)
|
||
_log.info(f"已注册启动定时 {job.name}({job.id}) cron={start_cron}")
|
||
|
||
if mode == "cron_stop":
|
||
stop_cron = job.schedule.get("stop_cron", "")
|
||
if stop_cron:
|
||
try:
|
||
trigger = CronTrigger.from_crontab(stop_cron)
|
||
except Exception as e:
|
||
_log.error(f"停止 cron 解析失败 {stop_cron}: {e}")
|
||
else:
|
||
self.scheduler.add_job(self._on_cron_stop, trigger, args=[job.id],
|
||
id=self._cron_id(job.id, "stop"),
|
||
replace_existing=True)
|
||
_log.info(f"已注册停止定时 {job.name}({job.id}) stop_cron={stop_cron}")
|
||
|
||
def _remove_cron(self, job_id):
|
||
"""移除该任务的所有 cron job（启动 + 停止）。"""
|
||
for kind in ("start", "stop"):
|
||
try:
|
||
self.scheduler.remove_job(self._cron_id(job_id, kind))
|
||
except Exception:
|
||
pass
|
||
|
||
def _on_cron_trigger(self, job_id):
|
||
"""cron 触发：启动该任务的所有目标设备 worker。"""
|
||
job = self.jobs.get(job_id)
|
||
if not job:
|
||
return
|
||
if not _in_run_window(job.schedule):
|
||
win = job.schedule.get("window") or {}
|
||
_log.info(f"定时触发 {job.name}({job.id}) 跳过：当前不在运行窗口内 "
|
||
f"（{win.get('start','')}-{win.get('end','')}）")
|
||
return
|
||
_log.info(f"定时触发: {job.name}({job.id})")
|
||
self._run_job(job)
|
||
|
||
def _on_cron_stop(self, job_id):
|
||
"""停止 cron 触发：停止该任务正在运行的所有 worker。
|
||
|
||
只停本 job 启动的 worker，不影响其他任务。
|
||
"""
|
||
job = self.jobs.get(job_id)
|
||
if not job:
|
||
return
|
||
with self._lock:
|
||
items = [(s, info) for s, info in self._running.items()
|
||
if info.get("job_id") == job_id]
|
||
stopped = []
|
||
for serial, info in items:
|
||
w = info.get("worker")
|
||
if w and w.is_alive():
|
||
w.stop()
|
||
stopped.append(serial)
|
||
_log.info(f"定时停止 {job.name}({job.id}): 停止 {len(stopped)} 台设备 {stopped}")
|
||
notifier.notify("task.cron.stopped", job_id=job.id, job_name=job.name,
|
||
stopped_count=len(stopped), serials=stopped[:10])
|
||
|
||
def next_run_of(self, job):
|
||
"""任务下次真正执行的时间（考虑运行窗口）。停用/无 cron 返回 None。"""
|
||
if not job or not job.enabled:
|
||
return None
|
||
return _next_run_time(job.schedule)
|
||
|
||
def run_job_now(self, job_id):
|
||
"""立即执行任务（手动触发）。运行窗口只约束定时触发，手动执行不受限制。"""
|
||
job = self.jobs.get(job_id)
|
||
if not job:
|
||
return {"ok": False, "error": "任务不存在"}
|
||
# 任务类型已不存在（历史类型被删除）→ 明确报错，别只返回"已触发"然后线程里静默失败
|
||
if not get_task_class(job.task_type):
|
||
return {"ok": False, "error":
|
||
f"任务类型 {job.task_type} 已不存在（该类型已被删除），"
|
||
f"请删除此任务或改用现有类型"}
|
||
# 在独立线程跑，不阻塞调用方
|
||
t = threading.Thread(target=self._run_job, args=(job,), daemon=True)
|
||
t.start()
|
||
return {"ok": True, "msg": f"任务 {job.name} 已触发"}
|
||
|
||
def _run_job(self, job):
|
||
"""执行任务：为每个目标设备起 worker（含重试循环）。
|
||
|
||
大量设备同时启动会触发 adb connect 全局锁串行 + STF occupy 并发风暴，
|
||
这里按 _START_STAGGER_SEC 间隔逐个启动，分摊连接压力（100 台 × 0.2s = 20s）。
|
||
"""
|
||
serials = job.resolve_serials(self)
|
||
if not serials:
|
||
_log.warning(f"任务 {job.name} 无可用设备")
|
||
notifier.notify("task.batch.no_device", job_id=job.id, job_name=job.name,
|
||
target=job.target)
|
||
return
|
||
task_cls = get_task_class(job.task_type)
|
||
if not task_cls:
|
||
_log.error(f"未知任务类型: {job.task_type}")
|
||
notifier.notify("task.batch.unknown_type", job_id=job.id, job_name=job.name,
|
||
task_type=job.task_type)
|
||
return
|
||
task = task_cls()
|
||
max_attempts = max(1, job.retry.get("max_attempts", 1))
|
||
delay = job.retry.get("delay", 60)
|
||
|
||
notifier.notify("task.batch.started", job_id=job.id, job_name=job.name,
|
||
task_type=job.task_type, device_count=len(serials),
|
||
serials_preview=serials[:5])
|
||
# 批次收尾统计：所有设备都出结果后发一条汇总（见 _BatchTracker）
|
||
# 设备名一次查好带下去（worker 线程里没有 app context，查库要显式包 context）
|
||
names, models = {}, {}
|
||
try:
|
||
with self._db():
|
||
rows = device_pool.list_devices()
|
||
names = {d["serial"]: (d.get("name") or "") for d in rows}
|
||
models = {d["serial"]: (d.get("model") or "") for d in rows}
|
||
except Exception as e:
|
||
_log.warning(f"读取设备名/型号失败（通知里将显示地址）: {e}")
|
||
tracker = _BatchTracker(job, len(serials), names, models)
|
||
|
||
for idx, serial in enumerate(serials):
|
||
# 每台设备一个重试循环线程，互不影响；错峰延迟在各自线程内等待
|
||
t = threading.Thread(target=self._run_with_retry,
|
||
args=(task, serial, job, max_attempts, delay,
|
||
idx * _START_STAGGER_SEC, tracker), daemon=True)
|
||
t.start()
|
||
|
||
def _run_with_retry(self, task, serial, job, max_attempts, delay, start_delay=0,
|
||
tracker=None):
|
||
"""单设备任务执行 + 重试。
|
||
|
||
异常分类：
|
||
DeviceOfflineError — 设备掉线，立即放弃不重试（换设备也没用）
|
||
其他异常 — 按 max_attempts 重试
|
||
用户停止（stop_device）会加入 _stop_requested，阻止任何后续重试。
|
||
"""
|
||
# 错峰启动：在各自线程内等待，分摊批量启动的连接/占用压力
|
||
if start_delay > 0:
|
||
time.sleep(start_delay)
|
||
# 设备名（通知里显示"设备：A08"而不是裸地址）
|
||
dname = (tracker.names.get(serial) if tracker is not None else "") or serial
|
||
# 被抢占的原任务 id（抢占结束后归还）——必须在循环外：
|
||
# 重试时若重置为 None，finally 归还逻辑会丢失信息，被抢占任务永不恢复
|
||
preempted_job = None
|
||
# 本台设备的结果（供批次统计与通知用）。默认 "failed"：走到重试耗尽就是失败；
|
||
# 各提前 return 的分支会覆盖它。**在 finally 里统一上报**，保证不漏不重。
|
||
outcome = "failed"
|
||
try:
|
||
for attempt in range(1, max_attempts + 1):
|
||
# 用户已请求停止 → 不再启动新 attempt
|
||
stopped_by_user = False
|
||
with self._lock:
|
||
if serial in self._stop_requested:
|
||
_log.info(f"{serial} 用户已请求停止，取消重试 (job={job.name})")
|
||
stopped_by_user = True
|
||
if stopped_by_user:
|
||
outcome = "stopped"
|
||
notifier.notify("task.device.stopped", serial=serial, device_name=dname,
|
||
job_id=job.id, job_name=job.name, attempt=attempt,
|
||
phase="start")
|
||
return
|
||
# 同一 serial 同时只能一个 worker；不同任务可配置抢占
|
||
preempt = False
|
||
skip_reason = ""
|
||
with self._lock:
|
||
if serial in self._running:
|
||
cur = self._running[serial]
|
||
if cur.get("job_id") == job.id:
|
||
# 同一任务重复触发：跳过（原行为）
|
||
_log.warning(f"{serial} 已有任务在跑，跳过 (job={job.name})")
|
||
skip_reason = "同一任务已在运行"
|
||
elif not job.params.get("preempt"):
|
||
# 不同任务且本任务未开启抢占：跳过
|
||
_log.warning(f"{serial} 已有任务在跑，跳过 (job={job.name})")
|
||
skip_reason = "设备被其他任务占用"
|
||
else:
|
||
preempt = True
|
||
if not preempted_job:
|
||
preempted_job = cur.get("job_id")
|
||
if skip_reason:
|
||
outcome = "skipped"
|
||
return
|
||
if preempt:
|
||
# 抢占：锁外停止该设备上的其他任务（stop_device 内部拿同一把锁，
|
||
# 在锁内调用会死锁！），等其释放后接管
|
||
_log.warning(f"{serial} 抢占：停止任务 {preempted_job} 后执行 {job.name}")
|
||
notifier.notify("task.device.preempted", serial=serial, device_name=dname,
|
||
job_id=job.id, job_name=job.name,
|
||
preempted_job_id=preempted_job,
|
||
preempted_job_name=(self.jobs.get(preempted_job).name
|
||
if self.jobs.get(preempted_job) else ""))
|
||
self.stop_device(serial)
|
||
deadline = time.time() + 30
|
||
while time.time() < deadline:
|
||
with self._lock:
|
||
if serial not in self._running:
|
||
break
|
||
time.sleep(0.5)
|
||
else:
|
||
_log.warning(f"{serial} 抢占超时（旧任务 30s 未退出），跳过 (job={job.name})")
|
||
outcome = "skipped"
|
||
notifier.notify("task.device.preempt_timeout", serial=serial,
|
||
device_name=dname, job_id=job.id, job_name=job.name,
|
||
preempted_job_id=preempted_job)
|
||
return
|
||
with self._lock:
|
||
self._running[serial] = {"job_id": job.id, "started_at": time.time(),
|
||
"attempt": attempt, "task_type": job.task_type,
|
||
"preempted_job": preempted_job}
|
||
_update_status(serial, task_job=job.name, attempt=attempt,
|
||
max_attempts=max_attempts)
|
||
|
||
worker = None
|
||
t_attempt = time.time() # 单次 attempt 耗时（通知里带）
|
||
# 本次运行的上下文：步骤明细按 run_id 归组，把同一设备同一次尝试
|
||
# 执行的步骤串起来（重试会生成新的 run_id，两次尝试不混在一起）
|
||
run_ctx = {"run_id": uuid.uuid4().hex[:12], "job_id": job.id,
|
||
"job_name": job.name, "device_name": dname,
|
||
"attempt": attempt}
|
||
try:
|
||
worker = task.create_worker(serial, job.params, ctx=run_ctx)
|
||
with self._lock:
|
||
self._running[serial]["worker"] = worker
|
||
_log.info(f"{serial} 开始任务 {job.name} (第{attempt}/{max_attempts}次)")
|
||
worker.start()
|
||
worker.join() # 等待 worker 结束
|
||
# worker 正常结束（done 或被 stop）
|
||
with self._lock:
|
||
self._running.pop(serial, None)
|
||
# 判断是否成功：看 status
|
||
with _WORKERS_LOCK:
|
||
st = _WORKERS.get(serial, {}).get("status")
|
||
# 清除 task_job 标记，避免前端误判"运行中"
|
||
_update_status(serial, task_job="")
|
||
if st == "done":
|
||
_log.info(f"{serial} 任务 {job.name} 成功完成")
|
||
outcome = "success"
|
||
# 单设备成功：**只在"这一批本来就只有一台设备"时发**。
|
||
# 多设备批次里逐台报成功没有信息量（批次汇总里已有成功台数），
|
||
# 13 台就是 13 条刷屏；失败仍然逐台发——那是少数且要知道是哪台。
|
||
if tracker is None or tracker.total <= 1:
|
||
notifier.notify("task.device.success", serial=serial,
|
||
device_name=dname,
|
||
model=_dev_model(serial, tracker),
|
||
job_id=job.id, job_name=job.name,
|
||
attempt=attempt,
|
||
duration_s=int(time.time() - t_attempt))
|
||
return
|
||
# 用户请求停止（无论 attempt 第几次、status 是什么）→ 不重试
|
||
with self._lock:
|
||
if serial in self._stop_requested:
|
||
_log.info(f"{serial} 用户已请求停止，不再重试 (job={job.name})")
|
||
outcome = "stopped"
|
||
notifier.notify("task.device.stopped", serial=serial,
|
||
device_name=dname, job_id=job.id,
|
||
job_name=job.name, attempt=attempt,
|
||
phase="after")
|
||
return
|
||
if worker is not None and worker.stopped():
|
||
# 任务**自己**停的（步骤里的「停止本设备」）：不是失败，也不重试。
|
||
# 判据是"worker 置了停止位但 _stop_requested 里没有"——上面那个
|
||
# 分支已经排除了用户/调度停止的情况。
|
||
_log.info(f"{serial} 任务内主动停止本设备 (job={job.name})")
|
||
outcome = "stopped"
|
||
notifier.notify("task.device.stopped", serial=serial,
|
||
device_name=dname,
|
||
model=_dev_model(serial, tracker),
|
||
job_id=job.id, job_name=job.name,
|
||
attempt=attempt, phase="self")
|
||
return
|
||
_log.warning(f"{serial} 任务未成功(status={st})")
|
||
except DeviceOfflineError as e:
|
||
# 设备掉线，立即放弃，不重试
|
||
_log.error(f"{serial} 设备离线，放弃任务 {job.name}: {e}")
|
||
with self._lock:
|
||
self._running.pop(serial, None)
|
||
_update_status(serial, status="failed", task_job="",
|
||
last_error=f"设备离线: {e}")
|
||
outcome = "failed"
|
||
notifier.notify("task.device.offline", serial=serial, device_name=dname,
|
||
model=_dev_model(serial, tracker),
|
||
job_id=job.id, job_name=job.name,
|
||
error=str(e)[:200])
|
||
return
|
||
except Exception as e:
|
||
_log.error(f"{serial} 执行异常: {e}", exc_info=True)
|
||
with self._lock:
|
||
self._running.pop(serial, None)
|
||
_update_status(serial, task_job="")
|
||
notifier.notify("task.device.error", serial=serial, device_name=dname,
|
||
model=_dev_model(serial, tracker),
|
||
job_name=job.name, attempt=attempt,
|
||
error=str(e)[:200])
|
||
|
||
if attempt < max_attempts:
|
||
# 用户在 sleep 期间点停止也能中断
|
||
with self._lock:
|
||
if serial in self._stop_requested:
|
||
_log.info(f"{serial} 用户已请求停止，取消重试 (job={job.name})")
|
||
outcome = "stopped"
|
||
notifier.notify("task.device.stopped", serial=serial,
|
||
device_name=dname, job_name=job.name,
|
||
attempt=attempt, phase="retry-wait")
|
||
return
|
||
# 检查是否是端口耗尽类临时错误，需要更长退避等端口释放
|
||
with _WORKERS_LOCK:
|
||
err = _WORKERS.get(serial, {}).get("last_error", "")
|
||
transient = err.startswith("[transient]")
|
||
if transient:
|
||
# Windows TCP 端口耗尽，TIME_WAIT 默认 2-4 分钟，等 120 秒
|
||
extra_delay = max(delay, 120)
|
||
_log.info(f"{serial} ADB 连接临时错误（端口耗尽），{extra_delay}s 后重试 ({attempt+1}/{max_attempts})")
|
||
wait = extra_delay
|
||
else:
|
||
_log.info(f"{serial} {delay}s 后重试 ({attempt+1}/{max_attempts})")
|
||
wait = delay
|
||
notifier.notify("task.device.retry", serial=serial, device_name=dname,
|
||
job_name=job.name, attempt=attempt,
|
||
next_attempt=attempt + 1, delay_s=int(wait),
|
||
reason="transient" if transient else "normal")
|
||
time.sleep(wait)
|
||
|
||
_log.error(f"{serial} 任务 {job.name} 重试耗尽，放弃")
|
||
# 带上最后一次的真实失败原因——只写"重试N次失败"会让用户看不到为什么失败
|
||
# （worker 出错时已把原因写进 _WORKERS[serial]["last_error"]）
|
||
with _WORKERS_LOCK:
|
||
cause = (_WORKERS.get(serial, {}).get("last_error") or "").strip()
|
||
msg = (f"{job.name} 重试{max_attempts}次仍失败" if max_attempts > 1
|
||
else f"{job.name} 执行失败")
|
||
if cause and cause != msg:
|
||
msg += f"：{cause}"
|
||
_update_status(serial, status="failed", last_error=msg[:200])
|
||
outcome = "failed"
|
||
notifier.notify("task.device.failed", serial=serial, device_name=dname,
|
||
model=_dev_model(serial, tracker), job_id=job.id,
|
||
job_name=job.name, attempts=max_attempts,
|
||
cause=cause, msg=msg)
|
||
finally:
|
||
# 清除停止标志：整个重试循环结束（成功/失败/停止）后允许下次任务
|
||
with self._lock:
|
||
self._stop_requested.discard(serial)
|
||
# 批次统计：**每台设备只在这里上报一次**（所有 return 分支都会走到 finally）
|
||
if tracker is not None:
|
||
try:
|
||
tracker.done(serial, outcome,
|
||
detail=_dev_fail_note(serial, dname, tracker)
|
||
if outcome == "failed" else "")
|
||
except Exception as e:
|
||
_log.warning(f"批次统计上报失败（不影响任务）: {e}")
|
||
# 归还：本任务是抢占任务，结束后自动重新启动被抢占的原任务
|
||
if preempted_job:
|
||
j = self.jobs.get(preempted_job)
|
||
if j and j.enabled:
|
||
_log.info(f"{serial} 抢占任务 {job.name} 结束，归还设备给任务 {j.name}")
|
||
notifier.notify("task.device.released", serial=serial, device_name=dname,
|
||
job_name=job.name, preempted_job_id=preempted_job,
|
||
returned=True, reason="归还并重启被抢占任务")
|
||
threading.Thread(target=self._run_job, args=(j,), daemon=True).start()
|
||
else:
|
||
# 打日志便于排查：被抢占任务已删除/停用时不会归还，但要知道原因
|
||
_log.info(f"{serial} 抢占任务 {job.name} 结束，"
|
||
f"被抢占任务 {preempted_job} {'已停用，不归还' if j else '已不存在，不归还'}")
|
||
notifier.notify("task.device.released", serial=serial, device_name=dname,
|
||
job_name=job.name, preempted_job_id=preempted_job,
|
||
returned=False,
|
||
reason="被抢占任务已停用" if j else "被抢占任务已不存在")
|
||
|
||
# ---- 运行控制 ----
|
||
def stop_device(self, serial):
|
||
"""停止指定设备的 worker，并阻止后续重试。
|
||
|
||
无论 worker 当前在运行还是在重试 sleep 中，都会阻止下一次重试。
|
||
"""
|
||
with self._lock:
|
||
self._stop_requested.add(serial)
|
||
info = self._running.get(serial)
|
||
if info:
|
||
w = info.get("worker")
|
||
if w and w.is_alive():
|
||
w.stop()
|
||
return True
|
||
# worker 已结束但重试循环可能还在 sleep —— 仍然返回 True 表示已阻止重试
|
||
return True
|
||
|
||
def stop_all(self):
|
||
"""停止所有运行中的 worker，并阻止后续重试。"""
|
||
with self._lock:
|
||
items = list(self._running.items())
|
||
for s, _ in items:
|
||
self._stop_requested.add(s)
|
||
stopped = []
|
||
for serial, info in items:
|
||
w = info.get("worker")
|
||
if w and w.is_alive():
|
||
w.stop()
|
||
stopped.append(serial)
|
||
return stopped
|
||
|
||
def get_running(self):
|
||
with self._lock:
|
||
return {s: dict(v) for s, v in self._running.items()}
|
||
|
||
# ---- 状态查询（带缓存，避免设备列表查询阻塞前端）----
|
||
_status_cache = None # (timestamp, data, error)
|
||
_status_cache_lock = threading.Lock()
|
||
_STATUS_CACHE_TTL = 5.0 # 缓存 5 秒，前端 5 秒刷新刚好命中
|
||
|
||
def get_status(self):
|
||
"""综合状态：设备池（清单+在线）+ 本地 worker + 运行中的任务。
|
||
|
||
带 5 秒缓存：避免每次 /api/status 都查库/adb 阻塞 Flask。
|
||
worker 状态实时读（内存，无 IO），不受缓存影响。
|
||
"""
|
||
with self._status_cache_lock:
|
||
if self._status_cache:
|
||
ts, cached, err = self._status_cache
|
||
if time.time() - ts < self._STATUS_CACHE_TTL:
|
||
if err:
|
||
return None, err
|
||
# 用缓存的设备列表 + 实时 worker 状态重新组装
|
||
return self._merge_status(cached), None
|
||
# 缓存过期或不存在，重新拉设备池
|
||
try:
|
||
configured = device_pool.list_configured()
|
||
except Exception as e:
|
||
with self._status_cache_lock:
|
||
self._status_cache = (time.time(), None, str(e))
|
||
return None, f"获取设备列表失败: {e}"
|
||
with self._status_cache_lock:
|
||
self._status_cache = (time.time(), configured, None)
|
||
return self._merge_status(configured), None
|
||
|
||
def _merge_status(self, configured_serials):
|
||
"""用设备池清单 + 实时 worker 状态 + 前台 App 组装返回结果。"""
|
||
worker_status = {w["serial"]: w for w in get_all_worker_status()}
|
||
running = self.get_running()
|
||
try:
|
||
online = set(device_pool.list_online())
|
||
except Exception:
|
||
online = set()
|
||
try:
|
||
pool_rows = device_pool.list_devices()
|
||
names = {d["serial"]: d["name"] for d in pool_rows}
|
||
models = {d["serial"]: d["model"] for d in pool_rows}
|
||
except Exception:
|
||
names, models = {}, {}
|
||
|
||
# 清理陈旧状态：serial 已不在设备池、且没有在跑 worker 的条目，
|
||
# 避免设备被删除后其失败记录仍残留在"异常汇总"里
|
||
configured = set(configured_serials)
|
||
for serial, w in list(worker_status.items()):
|
||
if serial in configured or serial in running:
|
||
continue
|
||
if w.get("status") in ("running", "connecting"):
|
||
continue
|
||
_remove_worker(serial)
|
||
|
||
result = []
|
||
for serial in configured_serials:
|
||
is_online = serial in online
|
||
w = worker_status.get(serial, {})
|
||
r = running.get(serial, {})
|
||
result.append({
|
||
"serial": serial,
|
||
"model": w.get("model") or models.get(serial, ""),
|
||
"device_name": names.get(serial, ""),
|
||
"present": is_online,
|
||
"ready": is_online, # 阶段 1：ready 概念并入在线状态
|
||
"owner": "",
|
||
"worker_status": w.get("status", "idle"),
|
||
"foreground_app": self._fg_scanner.get(serial),
|
||
# 通用进度字段（任意 app 通用，前端统一解析展示）
|
||
# 结构：{"done": int, "total": int, "unit": str, "action_counts": dict}
|
||
"progress": w.get("progress", {}),
|
||
"current_action": w.get("current_action", ""),
|
||
"last_error": w.get("last_error", ""),
|
||
"last_warning": w.get("last_warning", ""),
|
||
"running_job": r.get("job_id", ""),
|
||
"task_job": w.get("task_job", ""),
|
||
"attempt": r.get("attempt", 0),
|
||
"end_time": w.get("end_time", 0),
|
||
})
|
||
return result
|
||
|
||
def list_all_serials(self):
|
||
"""返回设备池在线设备的 serial 列表（供分组表单勾选用）。"""
|
||
try:
|
||
return device_pool.list_ready()
|
||
except Exception:
|
||
return []
|
||
|
||
def shutdown(self):
|
||
self.stop_all()
|
||
self._fg_scanner.stop()
|
||
stop_watchdog()
|
||
try:
|
||
self.scheduler.shutdown(wait=False)
|
||
except Exception:
|
||
pass # DISABLE_SCHEDULER 模式下 scheduler 未 start，shutdown 会抛异常
|