A rule-guided pipeline begins with a written set of behavioral principles $C=\{c_1,\ldots,c_m\}$. For prompt $x$, a model produces draft $y_0$, critiques the draft against relevant principles, and generates revision $y_1$. The rules make supervision inspectable, but their usefulness depends on clarity, coverage, and conflict resolution.
The supervised stage can train on revised responses with token loss $$L_{rev}(\theta)=-\sum_{t=1}^{T}\log\pi_\theta(y_{1,t}\mid x,y_{1,<t}).$$ This teaches the policy to imitate revisions rather than requiring the critique at deployment. Training may also retain critique examples when explicit reasoning about rules is part of the desired behavior.
A later preference stage can compare two candidate responses using a rule-conditioned evaluator. If evaluator score is $s_C(x,y)$, pair probability may be modeled as $\sigma(s_C(x,y^+)-s_C(x,y^-))$. This creates scalable labels, but the evaluator and policy can share blind spots, producing confident agreement without external correctness.
Rules can conflict. Helpfulness may favor detail while privacy favors omission; transparency may favor explanation while security favors withholding operational steps. The system needs explicit priority, scope, and exception handling. Otherwise the critique process can resolve identical cases differently depending on wording.
Self-critique is not a proof that a revision is safer. The same model family may fail to notice the behavior it generated, and repeated revision can optimize superficial rule language. Independent tests, adversarial prompts, domain review, and outcome-based checks remain necessary, especially for high-impact decisions.
Audit each stage separately: draft violation rate, critique recall, false accusations, revision success, capability loss, and evaluator-policy agreement. Preserve rule versions and generation settings so behavior changes can be traced. A principle set is part of the deployed specification and should be tested like code rather than treated as static prose.