Let's work such a very simple model. We have a Robinson Crusoe island economy, an isolated individual that lives totally alone. In order to consume something Crusoe must work. Assume for even more simplicity that capital is not needed (say, fruit-gathering by hand). Crusoe does not like to work but he would rather sit relaxed and enjoy the good weather in his island. Apart from the biological necessity to eat something, Crusoe also likes the taste of fruits he gathers. Fruits cannot be stored.
So we have a, daily utility function of the form
$$U = U(c, \ell), \;\; \ell + L = T$$
$$U_c >0,\;\; U_{\ell}>0,\;\; U_{cc} <0, \;\; U_{\ell \ell} <0,\;\; U_{c\ell} =U_{\ell c} \geq 0$$
The last assumption on the cross-partial seems like the more reasonable one.
$\ell$ is leisure and $L$ is amount of work, and $T$ the maximum amount of time that Crusoe can work (say, $16$ hours). We also have a "fruit-gathering" function
$$Q = f(L) = AL^a$$
We can reasonably assume that we have decreasing marginal product of labor (due to the body getting tired going up and down all these trees), so $0<a<1$.
Since fruit cannot be stored and capital is absent, the only optimization problem that makes sense here is a static one, meaning $c=f(L)$.
Then we want to max over $L$ the function $U[f(L), T-L]$.
The f.o.c. is
$$\frac {\partial U[f(L), T-L]}{\partial L}\equiv G = U_c[f(L), T-L]\cdot f'(L) - U_{\ell}[f(L), T-L] = 0$$
and the s.o.c is
$$ \partial G/\partial L = U_{cc}\cdot f'(L) - U_{c\ell}\cdot f'(L) + U_c\cdot f''(L) - U_{\ell c}f'(L) +U_{\ell \ell} <0$$
Given our assumptions, this expression is everywhere negative, so we have our maximum.
We want to examine what happens as $A$ in the fruit-gathering function increases. Using the implicit function theorem we have, around the solution,
$$\frac {dL}{dA} = -\frac {\partial G/\partial A}{\partial G/\partial L}$$
We know that the denominator is negative, so together with the minus sign in front becomes positive. Therefore
$$\text{sign}\left\{\frac {dL}{dA}\right\} = \text{sign}\big\{\partial G/\partial A\big\}$$
$$= U_{cc}\cdot[ \partial f/\partial A ]\cdot f' +[\partial f'/\partial A]\cdot U_c - U_{\ell c}\cdot [\partial f/\partial A] $$
This expression is of ambiguous sign, which is a first indication that things may not be that straightforward after all.
To validate the conjecture of the OP we would need to obtain,
$$\text{sign}\left\{\frac {dL}{dA}\right\} < 0 \implies U_{cc}\cdot[ \partial f/\partial A ]\cdot f' +[\partial f'/\partial A]\cdot U_c - U_{\ell c}\cdot [\partial f/\partial A] < 0$$
$$\implies U_{\ell c}\cdot [\partial f/\partial A] > U_{cc}\cdot[ \partial f/\partial A ]\cdot f' +[\partial f'/\partial A]\cdot U_c$$
$$\implies U_{\ell c} > U_{cc}\cdot f' +\frac {\partial f'/\partial A}{\partial f/\partial A}\cdot U_c$$
Using the first order condition, we can substitute for $U_c = U_{\ell}/f'$, so
$$\text{sign}\left\{\frac {dL}{dA}\right\} < 0 \implies U_{\ell c} > U_{cc}\cdot f' +\frac {\partial f'/\partial A}{\partial f/\partial A}\cdot [U_{\ell}/f']$$
Now
$$\frac {\partial f'/\partial A}{\partial f/\partial A} \cdot \frac {1}{f'} = \frac {(a/L)L^a} { L^a} \cdot \frac {1}{(a/L)AL^a}$$
$$= \frac {1}{f} = \frac {1}{c}$$
So
$$\text{sign}\left\{\frac {dL}{dA}\right\} < 0 \implies U_{\ell c} > U_{cc} \cdot f'+ U_{\ell}/c$$
and re-arranging while multiplying throughout by $c/U_c$ and using the f.o.c. again, we get
$$\text{sign}\left\{\frac {dL}{dA}\right\} < 0 \implies f'\cdot (RRA_c-1) > - U_{\ell c}\cdot c/ U_c$$
where $RRA_c$ is the coefficient of relative risk aversion related to consumption.
So we see that if $RRA_c\geq 1$, which is the usual assumption, this inequality is certain to hold and so indeed higher productivity will lead to less time spent working... so why reality does not conform with our theoretical results?
But it does. Historically, the amount of time spent working has been declining . The problem with the OP's question is that the phrase "we do as much as we can" is contaminated by framework effects: if what happens around me is people working, say, $10$ hours a day, $5$ days a week, and I work $11$ hours a day, $6$ days a week, I tend to think that, since I exceed the standard around me, "I do what I can", being oblivious to the fact that people used to work $16$ hours a day, $7$ days a week, with perhaps a few days per year off at best. So no, we do not really "do as much as we can".