The answers above focus on how to mitigate the 'counterparty risk' that you correctly see as inherent in derivatives. But we need to also consider how big the losses can be -- the 'tons' you refer to -- and this is where the most common misunderstanding arises. The 'notional' amounts do not constitute the credit exposures and, depending on the type of derivative, either cannot be (eg, interest rate swaps) or are in practice heavily mitigated (credit derivatives).
In an interest rate swap with a notional of, say, \$100 million, all that changes hands in the related periodic payments is a fraction of that -- to be precise the difference between a fixed amount of interest on that \$100m and 'floating' rate of interest that is periodically calculated by reference to a market benchmark such as Libor. (Hence some of the concerns about manipulating Libor.) One needs to specify a notional amount, in order to have a basis for calculating those interest payments. The swap can then be used as a bet on rates or as a hedge (say of a bond position that is itself truly \$100m in size). The 'mark-to-market' (MtM) value of the swap at any given time is the expected difference, over the remaining life of the swap (which typically runs for 5-10 years), and that MtM is the amount I would be concerned about in the event that my counterparty goes bust. If the prospective difference between fixed and floating rates is in my favour, then that is the amount that I would ask for today in collateral. The Bank for International Settlements (the BIS, in its regular semi-annual statistics on OTC derivatives) has warned against looking at the notional as though it was the exposure.
Now, in the case of credit derivatives, if I sell you \$50 million of protection on, say, Deutsche Bank, there is indeed a possibility that I will (if Deutsche actually defaults) have to pay out that notional and you will in effect be taking a view on my ability to do so, as well as on Deutsche's creditworthiness. (So, if Deutsche and I were related entities, the 'joint-default' risk to you would be higher.) However, one still needs to be a little cautious on the numbers. For one thing, defaults rarely come entirely out of the blue -- the collapse of Lehman Bros, for example, was heavily trailed -- giving time to get in collateral or hedge or even close out or the contract at its mark-to-market value (essentially, the difference between Deutsche's creditworthiness when we entered the contract and its creditworthiness today). For the exposure to be systemic, which is where your query goes, that \$50 million (less any collateral) has to be life threatening to you and by extension to those whom you owe money. With the notable exception of AIG, whose regulator tellingly was later dissolved (in 2011 -- see wiki for more details) net exposures were not large. Net exposures arise where I sell you \$50m of protection on Deutsche but buy maybe \$40m worth from someone else. Check out DTCC for stats on the ratio of net to gross sales of protection. The last time I looked, net exposures were about 10% of the gross. Of course, if the $40m hedge I buy does not perform because that counterparty goes bust, then I am still on the hook for the full \$50m. But, aside from diversifying my counterparties, so that I am not overly-reliant on any one protection seller, I as a regulated entity have ways to manage the counterparty risk (more on this below); banks have regulatory limits on how much they can rely on the creditworthiness of any one entity; and have to hold capital against such credit risks as they do incur, including on derivatives.
Don't forget also, that credit derivatives are mainly focused on the more creditworthy entities and that even AIG's book was fine except for the bit linked to US mortgage repayments --from memory, about 15% of its book contributed 90% of its losses, compounded by other bets on US property. So, you do have to have some regard for the underlying risk.
Finally, bear in mind that even if 1) I do go bust owing you the full \$50m on that credit derivative on Deutsche and 2) the contract is not centrally cleared (which would insulate either of us from the default of the other) -- even in those circumstances, I would be obliged (not just able, but obliged) to take into account the \$50m I owed you on the credit derivative in claiming from you any amounts that you might owe me on any other derivatives that you and I had entered, whether credit derivatives on other entities or derivatives on interest rates, equities, currencies, commodities or any other asset class you care to imagine. In other words, the mark-to-market on an interest rate swap between us could offset that on the credit derivative. That may not sound much of an offset, given what I said above about the quantum of exposures on interest rate swaps. But again, check the BIS statistics. They show a big difference in practice between a) gross mark-to-market values and b) net ones that take account of counterparty exposures across bilateral derivatives books as a whole. If the 'gross market value' (ie, actual, mark-to-market exposures between pairs of counterparties, in aggregate, globally) is in the order of \$15 trillion, the 'gross credit exposures' (which is what the BIS calls the numbers after netting of mark-to-market exposures, taking account of offsets such as our hypothetical credit derivative and interest rate swap) is around $3 trillion. So it is roughly 1/5 of the gross mark-to-market values and considerably less than 1/100 of the notional amounts, which are around \$500 tr. So that is \$3tr of exposures, and that NB is before taking into account any collateral, which in bilateral books is calculated on a portfolio basis.
\$3tr is still a bigger amount than you or I could ever carry around in pocket money. But it's also 1/20 of the amount of government debt outstanding today and a fraction of the size of most other markets -- corporate bonds, loans, equity markets. The \$3tr could change, as things like interest rates change (thereby affecting the mark-to-market value of individual swaps), but in practice does not fluctuate wildly (as the BIS time series shows) and that is because a huge chunk of the market consists of intermediary banks buying and selling risks from customers with opposing needs or views -- risks that cancel each other out.