Why it matters

KTO uses cheaper data than DPO. Binary signals + prospect theory. Growing adoption.

Advertisement

The architecture

Binary: each sample chosen or rejected.

Prospect-theory loss: asymmetric.

KTO trainingBinary labelschosen or rejectedProspect theoryasymmetric lossOptimize policyKTO gradientNo pairs needed; binary signals sufficient; cheaper data collection
KTO structure.
Advertisement

How it works end to end

Prospect theory: losses hurt more than gains help.

Loss: sigmoid-based value function.

Reference model like DPO.