This section presents an extension of the Direct Nash Optimization (DNO) framework to handle regularized preferences. The main difference between SPO and Nash-MD lies in the use of smoothed policies for the latter, which helps obtain a late convergence guarantee. The section introduces a new version of DNO, designed to converge to a Nash equilibrium using KL-regularization. The algorithm (Algorithm 3) works iteratively, adjusting the policy distribution through a partition function and a reward function, ultimately refining the policy with each iteration. This approach helps address the challenges of regularized preferences while ensuring stable convergence.





