In the neo-classical growth model there is the following transversality condition:
$$\lim_{t\rightarrow\infty}\beta^{t}u'(c_{t})k_{t+1}= 0,$$ where $k_{t+1}$ is the capital at period $t$.
My questions are:
How we derive this condition?
Why do we require this, if we want to rule out paths with no debt accumulation?
Why are the Lagrange multipliers $\beta^{t}u'(c_{t}) = \beta^{t}\lambda_{t}$ the present discounted value of the capital?
The transversality condition may be more easily understood if we start from a problem with finite horizon.
In the standard version, our objective is to $$ \max_{\{c_t,k_{t+1}\}_{t=0}^T} \sum_{t=0}^T\beta^t u(c_t) $$ subject to $$ \begin{aligned} f(k_t)-c_t-k_{t+1}&\ge0,\quad t=0,\dots,T &&\text{(resource/budget constraint)}\\ c_t,k_{t+1}&\ge0,\quad t=0,\dots,T &&\text{(non-negativity constraint)} \end{aligned} $$ with $k_0$ given. The associated Lagrangian (with multipliers $\lambda_t$, $\mu_t$, and $\omega_t$) is $$ \max_{\{c_t,k_{t+1},\lambda_t,\mu_t,\omega_t\}_{t=0}^T} \sum_{t=0}^T \beta^tu(c_t)+\lambda_t(f(k_t)-c_t-k_{t+1})+\mu_tc_t+\omega_tk_{t+1} $$ The FOCs are $$ \begin{align} c_t:&& \beta^tu'(c_t)-\lambda_t+\mu_t&=0,\quad t=0,\dots,T \\ k_{t+1}:&& -\lambda_t+\lambda_{t+1}f'(k_{t+1})+\omega_t&=0,\quad t=0,\dots,T-1 \\ k_{T+1}:&& -\lambda_T+\omega_T&=0,\quad T+1 \tag{1} \end{align} $$ with the Kuhn-Tucker complementary slackness conditions: for $t=0,\dots,T$, $$ \begin{align} \lambda_t(f(k_t)-c_t-k_{t+1})&=0 & \lambda_t&\ge0 \\ \mu_tc_t&=0 & \mu_t&\ge0\\ \omega_tk_{t+1}&=0&\omega_t&\ge0\tag{2} \end{align} $$ Since resource constraint must be binding in all periods, i.e. $\lambda_t>0$ for all $t$, it follows that at the last period $T$, $\omega_T=\lambda_T>0$, which in turn implies $k_{T+1}=0$.
Usually we assume $c_t>0$ for all $t$ (the Inada condition), and this implies $\mu_t=0$ for all $t$. So the consumption FOC becomes $$ \beta^tu'(c_t)=\lambda_t \tag{3} $$
Looking at conditions $(1)$ $(2)$ and $(3)$ in the last period $T$, we get $$\beta^Tu'(c_T)k_{T+1}=0$$ Extending this to the infinite horizon, we get the transversality condition $$\lim_{T\to\infty}\beta^Tu'(c_T)k_{T+1}=0$$
The intuition of the transversality condition is partly that "there is no savings in the last period". But as there is no "last period" in an infinite horizon environment, we take the limit as time goes to infinity.
In my opinion, the best derivation is by logic. Think about it this way: If the only thing we are telling the household is to maximize its utility, optimal behaviour would then be just making infinite debt and consume infinitely. This is no sensible solution. We therefore need another optimality condition. This should answer question 2.
In a finite horizon setting, feasibility would be achieved by debt having to be repayed by the last period. This is not possible in a infinite horizon setting. However, "ruling out debt accumulation", as you suggest, is too strict a condition (The transversality condition allows for debt!).
To answer question 3, let us look at the term $\beta^t \lambda_t k_{t+1}$. It stands for the (marginal) utility gain (in present-value utils) of shifting $k_{t+1}$ units of capital to period t and consuming them. If this utility gain were positive at infinity, we could increase overall utility by consuming more at "period infinity", hence our capital path would not be optimal.
To question 1: To derive this condition, you can either make the logical argument I just made, showing that without the transversality condition holding, the capital path is not optimal, or, for a mathematical proof, you can check out, for example, Per Krusell's Notes (although it is rather hard to grasp)
