The following is slightly adapted from
Mas-Colell, Andreu. "Pareto optima and equilibria: the finite
dimensional case." Advances in equilibrium theory. Springer, Berlin,
Heidelberg, 1985. 25-42.
Consider an exchange economy in which everyone has the consumption set $\mathbb{R}^l_+$, the aggregate endowment is $\omega$, and free disposal is possible. By abuse of notation, we can use $\leq$ both for the coordiantewise order of vectors and the usual order on the real line.
The feasible set is
$$X^*=\{(x_1,\ldots,x_N)\in \mathbb{R}_+^{lN}\mid x_1+\cdots+x_n\leq\omega\}.$$
If every agent $i$ has a monotone, continuous, and concave utility function $u_i$ such that $u_i(0)=0$, then the utility possibility set
$$U=\Big\{\big(u_1(x_1),\ldots,u_n(x_N)\big)\mid (x_1,\ldots,x_N)\in X^*\Big\}$$
is convex. For notation, if $x=(x_1,\ldots,x_N)\in X^*$, we write $u(x)$ for $\big(u_1(x_1),\ldots,u_N(x_N)\big)$.
To see that $U$ is convex, note that first that $X^*$ is convex and for every allocation $x\in X^*$, if $0\leq x'\leq x$, then $x'\in X$. Now, let $u,u'\in U$ and $\alpha\in [0,1]$. There are $x,x'\in X^*$ such that $u(x)=u$ and $u(x')=u'$. Since all utility functions are concave, we have $u(\alpha x+(1-\alpha)x')\geq\alpha u+(1-\alpha)u'$. Since each utility function is monotone and has the value $0$ at $0$, there exist a vector $x''$ satisfying $0\leq x''\leq\alpha x+(1-\alpha)x'$ such that
$u(x'')=\alpha u+(1-\alpha)u'$ and $x''\in X^*$.