Let me preface this answer with a word of caution: your question is a very good and important one but it is also one that depends tremendously on the definitions of the terms it uses. I'm going to attempt to answer it in the most unassuming and non-technical way. You could pose it in technical terms and get a more precise answer.
The wealth of the rich can be attributed to the poverty of the poor. In North Korea, one of the poorest countries in terms of per-capita GDP, there are few rich people and many poor people. It is well-known that almost all of the rich people work either in the highest levels of government or the military. The poor people who are lucky enough to make any income largely do so at the behest of the rich people, and those who refuse are imprisoned or worse. Since almost all economic activity in North Korea is centrally planned via force, it follows that the rich derive their wealth from the suffering of the poor.
However, in societies with more private ownership of production and capital - and less centralization - force is still useful in general but less of a factor in determining who is wealthy and who is poor.
Regardless of our relative lot in life, if you and I freely trade with each other, we only do so when both of us feel like we are "better off" as as a result. So, if you are a wealthy industrialist and I am a poor banana farmer, and I freely decide to trade 100 of my bananas for 100 of your dollars, you necessarily think a banana is worth at least 1 dollar to you, and I necessarily think a banana is worth at most 1 dollar to me.
Excluding outright fraud (misrepresentation either of your dollars or my bananas) we must conclude that each of us either thinks this is a fair trade or that the other guy is a dope. For instance, you may know something about the value of bananas that I don't. Maybe you've just discovered that bananas have special healing powers or that my bananas are particularly good. On the other hand, maybe I know something you don't. Maybe I know that there are about to be many more banana producers in the area or that I can buy two bananas for 1 dollar somewhere else.
Usually, however, it's just a matter of what each party in the trade is able to do with the traded goods that determines what he/she thinks they are worth.
If you get rich from selling my bananas at a higher price, you are not doing so "at my expense" as long as you are not preventing me from doing the same thing. There may be reasons you have the ability to sell bananas at a higher price that have nothing to do with force - maybe you own a distribution system or retail outlets, and I don't - but my choice isn't between selling retail bananas and selling wholesale bananas, but rather selling bananas at the highest possible price vs not selling bananas at all.