You are right the two series closely follow each other for the reasons you mention. During quite some time the discount rate actually used to be a ceiling for a funds rate. This is precisely, because if the federal funds rate was below
the discount rate, most banks adjusted their reserve positions in the federal funds market and when the federal funds rate rose to the level of the discount rate it was cheaper to borrow at discount rate so banks simply do that.
However, they are not always as close as you would expect, especially the discount rate does not anymore acts as a ceiling. There are several reasons for that in literature, primarily because borrowing at both of those rates has a different strings attached to it.
For example, Meulendyke (1998) argues that:
Since the mid-1960s, the basic discount rate frequently has
been below the prevailing Federal funds rate. The Federal Reserve
has relied on administrative procedures to limit access to the window by restricting the frequency and amounts of borrowing.
... The Federal Reserve’s administrative
restrictions used to be the primary factor that discouraged borrowing, but in the last decade banks themselves have been responsible
for much of the limitation. Heavy borrowing in the 1980s by a few
banks with financial difficulties caused others to avoid the window
for fear depositors might conclude that they were also in trouble.
Reluctance to borrow contributes to a seemingly contradictory
result—namely, that increases in the amount of reserves in the banking system, when provided through the discount window, make
reserve availability more restrictive on the margin because such
increases put banks under pressure to find other sources of reserves
to repay the loans.
and she further argues that:
Until the mid-1960s, the Federal funds rate did
not trade above the discount rate. During “tight money periods,” when the
Trading Desk was fostering significant net borrowed reserve positions, funds
generally traded at the discount rate, and the funds rate was not considered a
useful indicator of money market conditions. When free reserves were high,
funds often traded below the discount rate and showed some day-to-day
variation. At such times, the funds rate received greater attention as an
indicator of reserve availability.
There was considerable surprise when the funds rate first rose above
the discount rate, briefly in October 1964 and more persistently in 1965. As
large banks became more active managers of the liability side of their balance sheets, they borrowed funds in the market in a sustained way. Banks
had introduced large negotiable certificates of deposit (CDs) in 1961. But CD
borrowings were (until 1991) subject to reserve requirements and (until
1970) to interest rate ceilings under Regulation Q. Borrowings from other
banks through the Federal funds market were free of reserve requirements
and interest rate ceilings. Furthermore, they were not subject to the restrictions on prolonged use that were applied to the Federal Reserve’s discount
window. The changes in liability management techniques meant that
individual banks could expand credit even when they did not have free
reserves if they were willing to bid aggressively for wholesale funding from
other banks.
These and possibly also some other reasons will lead the two to diverge in some situations as borrowing at those rate might be subject to different conditions and have different implications. You can see on the graph below which plots of the federal funds rate in red and the discount rate in blue (based on FRED data) that they can sometimes differ quite substantially but indeed tend to follow each other closely.
