I too have pondered this question.
I found a neat definition from the book 'Existence and Optimality of Competitive Equilibria' by Charalambos D. Aliprantis, Donald J. Brown, Owen (see page 9). I quote it here:
"Mathematically, a 'convex to the origin' curve is described by saying that if $A$ and $B$ are any two points on the curve, then a ray passing through the origin $O$ and any point $X$ on the line segment $AB$ will meet the curve at most at one point $D$ between $O$ and $X$". EDIT - this should read 'at exactly one point $D$ between $0$ and $X$ as opposed to at most one; the author appears to have made an error.

Under this definition, a curve shaped like the left-half of a U-shaped parabola, but never reaching a point at which its derivative is 0, is the type of nice convex to the origin indifference curve we dream about as economists. If we were to allow the slope of the indifference curve to become positive after some point then it will be possible to find draw a line from the origin that crosses two points of the curve, in which case the curve would not be convex to the origin.
When a utility function is a function of two variables x and y, an indifference curve is convex to the origin if the derivative of the indifference curves are always negative and the second derivatives are positive. That is, the indifference curves slope downwards always but the slope of the slope increases (from a negative value to a less negative value) as we move to the right. That is, the marginal rate of substitution (the negative of the slope of the indifference curve or equivalently, the magnitude of the slope of the indifference curve) decreases as we move to the right along an indifference curve.
Best,
Dutchman
Edit in response to Giskard's comment:
Giskard is right, the definition would need to be amended to exactly one.