You do not provide proper reference to the source and there are several handbooks of macroeconomics so it is hard to discern what authors meant by just reading one sentence. However, stating that:
physical capital contributes almost nothing to differences in GDP per worker across countries.
Does not mean there is no relationship. It says the relationship is very weak.
In fact your data also show very weak relationship. You do not share the details of your regression, but just eyeballing the graph I would guess that the coefficient is very low and the regression explains only very little variation in the data. Moreover, when observations are clustered like that regression will be very sensitive to outliers.
Regarding your questions:
When can we say X and Y have an association?
For any association (as opposed to causality), it is enough to show there is significant correlation between two variables. Typically most textbooks would say that correlation coefficient (in absolute value) below 0.3 indicates weak (or none in case of zero or close to it) association, between 0.3 and 0.7 moderate and after 0.7 strong association. Positive (negative) coefficient indicates positive (negative) association.
"physical capital contributes almost nothing to differences in GDP per worker across countries"? Is that his opinion?
That depends on your definition of opinion. Any data analysis is in the end an opinion, but not all opinions are created equal. As my econometrics teacher used to say data never speak for themselves, statisticians speak for the data. Multiple skilled statisticians can interpret exactly the same datasets differently. Statistical reasoning is inductive reasoning and from an epistemological perspective induction (as opposed to deduction) can't give you generally valid 'truth' and can be influenced by subjective perspectives (see discussion about it on WikiLectures).
This being said, rigorous statistical methods are created in a way that can help us inductively analyze relationships in most objective way possible. Even if induction can't really give you general truth statistics allows researchers to create general conclusions from empirical data in the most objective inductive reasoning based on probability. However, in the end it is an opinion whether there is an relationship (e.g. if the coefficient would not be significant someone could claim there is too noise in the data, or if it is significant it could be just statistical artefact). If you read the handbook and if the author supports his statement by rigorous analysis I would say that comes as close to a fact as humanly possible, but using broad definition of opinion it would still qualify as an informed opinion.
Or I should take it that the statement is only true in 2011?
This can't be answered from that one sentence. English is high context language. Depending on the context he might have been talking just about 2011 or making more general claim.
I was expecting the relationship between average $K_t/Y_t$ and average $Y_t/n_t$ to be insignificant, but it is the opposite. Can I say the author is wrong? And that capital or capital-output ratio matter for cross-country differences in $Y_t/n_t$?
That depends on how rigorously you analyzed the data and what exactly you found.
First, the author does not say that there is no relationship, as I read it he says it can explain almost nothing, but not nothing.
Second, if you just regressed $Y_t/n_t$ on $K_t/Y_t$ I do not think you would have enough evidence to say that the author is wrong. Perhaps there are important covariates you omit and you found significant relationship just because of omitted variable bias.
If you run some state of the art empirical model and found significant and economically strong relationship you would have some solid grounds to claim the author is wrong. Otherwise, I would advise using some softer language, such as his result is inconsistent with yours or something like that.
