Is there a chance that taking advantage of the royalty clause could piss the publisher off? Showing that you don't trust them. I understand that if they aren't giving you all your royalties you wouldn't want to work with them. But what if there were no errors - couldn't that just hurt future chances of book sales? Seems like a lose-lose situation.
An auditor will look at your contract, and your royalty statements and give it "the smell test". If nothing looks amiss, they won't seek an audit. It's only when things look off that auditors go in.
And any reasonable company is glad to show they are doing things correctly. They're not too happy if they find out they owe money to a client, but that's not my problem; getting my clients paid correctly is.