Brazil is a more complicated country than most. It has extreme variability in wealth and education. It is very ethnically diverse, has vast natural resources and inconvenient geography.
Unfortunately, there's not an easy straightforward answer to your question. A lot of what affects Brazil is political fragmentation and instability. When ranking countries by stability, The Fragile State Index ranks Brazil as the 108th most stable country, putting is firmly below average. This results in poorly centralized power which means most case studies are unique to local resources, local labor and local leadership.
The specific situation you ask about in regards to manual harvesting is a combination of insufficient capital, disregard for worker's rights, or a complete lack of organization.
Having lived in Brazil, there are an infinite number of economic and social issues that require attention, but not enough time or resources to devote to all of them. The country relies on resource extraction while trying to grow manufacturing and tech while trying to deal with massive inequality. Poor geography.
Here is a study that might give you more precise information for your particular question: