Provided you haven't used the addresses generated by the extended public key before, you should be able to ask your wallet to generate a new public address and that new address should match the next address Swan will withdraw to.
If you have already used some of the addresses generated by your extended public key then you can simply go back in your history to review your old addresses and confirm the next withdrawal is to an address that you own.