The Heavenly Sword Sect is a noble and decent sect.
The first patriarch abandoned Li Jing, the immortal sword master, and the legendary invincible. It may be a bit reluctant to select moral models, but no one will object to saying that he is not close to women and has no secular desire.
Lu Bei, the second patriarch, abandoned his estranged successor. Even if his moral standard had declined to a certain extent, it would not have plummeted.
The king of Yuan Xuan estimated that Lu Bei probably made use of his beloved daughter's eagerness to threaten him and would not do anything out of line.
Let's put the facts in perspective. This is not the first time that Xinli Jun has been captured by Lu Bei