The potential risk of borrowing too much now is that interest rates will be higher when the debt has to be rolled over. Even though it is completely correct to say that most government securities have fixed interest rate, most of these securities are not perpetual, meaning they expire at some point.
For example, as mentioned on the UK Debt Management Office website:
Conventional gilts also have a specific maturity date. In the case of 1½% Treasury Gilt 2047 the principal will be repaid to investors on 22 July 2047. In recent years the Government has concentrated issuance of conventional gilts around the 5-, 10- and 30-year maturity areas, but in May 2005 the DMO issued a new 50-year maturity conventional gilt. In June 2013, following market consultation the DMO issued a new 55 year maturity conventional gilt.
This carries some potential risk. For example, suppose for the sake of argument government could borrow now $£ 1000$ with coupon rate $0\%$ (coupon rate is essentially the interest paid on face value of fixed income security), for 5 years. That is great deal now but in 5 years government will face the following problem, it will either have to repay the debt (which could be difficult to do for country with high level of debt) or instead roll it over meaning the government issues another $£ 1000$, but this time if interest rates are higher and government is forced to issue it with, let us say $20\%$ coupon rate that would be a problem because suddenly the cost of servicing debt to the government would skyrocket.
This being said, it is worth noting that even though the above is potential risk that should be taken seriously, most economists believe that governments should borrow more when the interest rates are low, especially if they do so long term (e.g. 10, 30 or 50 years) or even better some argue they could start issuing perpetuities i.e. debt instruments where principal never has to be repaid (see discussion in Corsetti, Erce, Pascual 14 May 2020). This is because issuing long term debt or perpetuities locks the low interest rates on that particular debt for very long time or indefinitely meaning there would be plenty of time to consolidate public finances if interest rates were rising sharply. In addition, government has also other venues how to handle debt such as monetization of the debt although that can have some serious disadvantages as well but discussing that is beyond scope of this answer. Consequently, while it is definitely real risk as you pointed out in the question itself most economists would argue governments should take advantage of these low rates while they can.