"Calculus at infinity" is a useful tool in Economics also, when we have managed to represent mathematically an economic situation. For example, when we model an intertemporal utility maximization problem with an "infinite" planning horizon, we only do it because it simplifies the mathematics considerably, while the approximation error is usually negligible for our purposes, since due to discounting, a finite-horizon setup would not produce any additional insight of value. We do not really argue that economic agents have "infinite" planning horizon.
The convention to talk about "infinite supply" belongs to the same category -but in this case it does not appear to help in anything.
The reason that the Economics discipline exists is the finiteness of resources. Supply of a good or service, or anything else related to Economics cannot be infinite - so whenever we use the term we usually mean "very large"...
...and the excerpt the OP quotes from the textbook does exactly that: it talks about "price elasticity approaching infinity... meaning that very small changes in price lead to very large changes in quantity supplied". This passage takes a mathematical concept, and correctly translates it in meaningful economic terms.
More formally, "any quantity" does not mean "infinite quantity" because "infinity" is not a number, so it cannot represent a quantity (and let's not confuse quantity with the concept of cardinality used in set theory). The expression $x\geq 0$ does not include "$\infty$". $x$ belongs to the real numbers, and $\infty$ is added to the real numbers to create the "extended real numbers" - it is not a real number itself.
Perhaps by "logical extension", if all non-zero real quantities are supplied at USD $4$, then what is left for higher prices is only "infinity". But this has no economic meaning -apart from the approximate one already mentioned in the passage quoted.
