ct run sabotage-eval

Run sabotage evaluation: honest eval + attack eval + monitor + graphs.

Usage

ct run sabotage-eval [OPTIONS]

Options

OptionDescription
-c, --config PATHYAML config file for multi-attack mode.
--attack-policy TEXTAttack policy name.
--attack-policy-arg TEXTAttack policy args in key=value format (repeatable).
--honest-policy TEXTHonest policy name.
--honest-policy-arg TEXTHonest policy args in key=value format (repeatable).
--protocol TEXTBlue protocol to run both evals under (e.g. defer-to-resample, trusted-editing). Attacker/honest become the protocol's untrusted policy.
--protocol-arg TEXTProtocol args in key=value format (repeatable).
--audit-budgets TEXTAudit budget(s), comma-separated (e.g. 0.0001,0.001,0.01). Each produces graphs.
--thresholds TEXTFixed threshold(s), comma-separated (e.g. 3.0,4.0,5.0). Skips honest eval.
--model-role TEXTBind an eval-level model role as role=model (inspect's --model-role). Only trusted — the role blue protocols resolve for their gate monitor and trusted editor — is bindable here; untrusted models are identity and come from the policy args.
--task-file TEXTTask set name or path.
--main-tasks-with-no-scorers [error|include|exclude]Main tasks lacking a scorer: refuse the run (error), keep them recorded as unscored (include), or drop them (exclude).
--log-dir TEXTWhere eval logs go.
-o, --output-dir TEXTWhere graphs and summary go.
--ec2Run evals on EC2.
--no-graphSkip graph generation.
--no-uploadDisable uploading results.
--dry-runValidate the local execution plan and print it without executing.
--no-cacheRe-run every attack even when a prior results.json recorded an identical one.
--preflightProbe the run's external contracts (model endpoints, pricing, credentials) and exit; runs nothing and spends nothing.
--tags TEXTTags for the run (repeatable). Added to all eval runs alongside built-in sabotage-eval tags.
--helpShow this message and exit.