Can the federal reserve just "add zeros" to a bank's balance?
They can do that. For example, when the Fed buys government bond it does that with money that it creates electronically. For example, if Fed wants to buy US treasury for 1000 dollars, they just electronically create 1000 dollars worth of reserves to pay for the bond.
Also when private banks want to borrow from Fed extra reserves, Fed just creates them electronically at a keystrokes of a keyboard.
However, they will not do that on other account they don’t own directly. They will create the reserves on their accounts and then transfer them. But I suppose if you would want to simplify the description you could do that.
Assuming this is possible (that the Fed can literally just "add zeros" to a bank's account), where does that "money" come from?
From government fiat (decree) hence why it’s called fiat money. Government declares there is money so there is money. It’s an equivalent of you creating 1000 dollar dad-bucks for your kids to pay them for chores. Where the money comes from? From your decree. Other then the decree there is nothing else so many economists will also say it’s out of thin air/ex nihilo.
Does it eventually get removed from the economy? How? Do they just delete a zero later?
No it gets removed when the debt is repaid. For example, when private bank returns reserves to Fed money is destroyed. Likewise when government repays the debt to itself money is destroyed (since both Fed and Treasury are gov institutions Fed buying debt just means government borrows money to itself).
Again they don’t delete or add money to other peoples accounts they do it on their own accounting once transactions are settled. The source you cite oversimplifies it.
And what does it mean that "banks trust the government to always pay back its debts" if the government (I'm assuming they mean the Fed?)
It’s really unclear what the source means by it works only when “bank trusts the government to always pay back its debt”. That’s not correct fiat system would work even with government default. There are several countries that defaulted and that kept fiat money. See some examples in Reinhart and Rogoff 2008.
are supposedly just making money out of thin air?
It’s not just “supposedly” it’s a well known fact you can read in any macro textbook. Have look at Mankiw Macroeconomics or Blanchard et al Macroeconomics European Perspective. This is not even a secret and it’s by design.
Wouldn't that greatly devalue the dollar?
Not necessarily. Value of any money be it commodity money, money under gold standard, cryptocurrency etc depends on supply and demand for the money.
Money market equilibrium in textbook simplified model is given by:
Where $M$ is money supply, $P$ price level and value of money is in essence $1/P$, $L$ is the money demand that is positive function of real output of an economy $Y$ and negative function of interest rate $i$.
Whether $P$ goes up and so value of money drops depends on the changes in $M$, $Y$ and $i$.
In addition the simple 101 textbook model is bit oversimplified as well. Some additional important factor is peoples expectations. If people expect money to be worth less next year they might become worth less due to self-fulfilling prophecy. The demand for money is also generated by some extra factors such as the fact that US government asks you to pay taxes in US dollars and also US dollars must be accepted to settle debts in US. This forces people to hold some dollars even if they would otherwise refuse to.
There are further nuances to it that are beyond scope of this answer.