There aren't two payoff matrices. If there's a random choice between the two payoff matrices, then the overall payoff matrix will just be a linear combination of the two matrices you've presented, according to the probabilities. For instance, if the probability of getting caught is 10%, then each entry in the overall matrix is .9 times the corresponding entry in the first matrix, plus .1 times the entry in the second matrix. Since the two matrices are identical but for the bottom right, only that square will depend on the probability. If the probability of getting caught is $p$, then the payoff in that square will be $4(1-2p)$. More generally, if they have probability $p$ of getting a fine $f$ instead of the regular payoff, then the overall payoff will be $4(1-p)-pf=4-4p-pf=4-(f+4)p$. As you say, a higher probability of getting caught means less of a fine is required, and vice versa.
You have the payoffs for both players being equal, but suppose the Supplier and Consumer have different expected payoffs for the bottom right, SEP and CEP respectively. For the Supplier, any positive SEP will be an incentive to choose Corrupt. But for the Consumer, they need a CEP>2 to have an incentive to choose Corrupt. So we can remove the Nash Equilibrium by punishing just the Conusmer, and the required probability/size of the punishment for the Consumer will be lower than for the Supplier.
You say that even if you disincentivize the Consumer from Corrupt, the Supplier can go to another Consumer, but if the other Consumer has the same payoff matrix, then from a game theory point of view, as soon as we get a matrix for which the Consumer will choose Honest, the Supplier shouldn't be able to find any Consumer to be Corrupt.
Keep in mind that any game theory analysis will be based on only what's in the payoff matrix, so if there's some phenomenon that you think will affect the result, then for it to be included in the analysis, the payoff matrix needs to reflect it somehow. One way to analyze a party's ability to move on to another counterparty is opportunity costs. Suppose I have a painting, and one potential buyer might pay 100k, while another might pay 90k. You might think that the payoff for getting the first buyer to buy it is 100k, but by selling it to the first buyer, I'm giving up the opportunity to sell it to the second. So making sure the first deal goes through might be worth only 10k to me. Similarly, even if a Supplier gets 2 units of utility for selling something, if they're giving up the opportunity to sell it to someone else, then it might be appropriate to consider their payoff to be less than 2.
Besides subtracting a NE we don't like, we can also look at how we can add one we do like. Currently, the Supplier's choice matter only if the Consumer chooses Corrupt; otherwise, they are indifferent. Thus, the only way to induce them to choose Honest is by changing the payoffs when the Consumer chooses Corrupt. If the Supplier's payoff in the bottom left corner were less than 2, then they would have an incentive to choose Honest when the Consumer chooses Honest. This would make (Honest, Honest) a Nash equilibrium. So if the Supplier had a chance, no matter how small, of being punished every time they choose Corrupt, regardless of whether the Consumer chooses Corrupt, then the upper left would be a NE.
They would then be in a situation where are two NE. They would prefer the bottom right one, while you prefer that they be in the top left. Such a situation is heavily history-dependent: once they're in one NE, getting into another requires coordination. So one strategy would be to first put a lot of effort towards catching and punishing the Consumer, lowering the payoff in the bottom right and eliminating it as a NE. Once they're in the top left NE, you could reduce the enforcement. This would re-introduce the bottom right as a NE, but they would now be in the upper left NE and would have to coordinate getting to the other one. You can then put in some effort into punishing the Supplier whenever they choose Corrupt but the amount of effort to keep that a NE would be less than that need to keep (Honest, Honest) a NE would be less than that needed to keep (Corrupt, Corrupt) not a NE.
An even better solution would be adding a "Snitch" option; if Honest players had the option of snitching on Corrupt ones, and were rewarded for doing so, then that would make Corrupt much less attractive.