Sample size and power

Correct answer: 1356. The effect shrinks to three fifths of what it was and the sample size more than triples - sample size scales roughly with the inverse square of the effect

Sample size scales roughly with the inverse square of the effect. Going from five percentage points down to three is a factor of three fifths, whose inverse square is close to three, and 435 scaled up a little over threefold is 1356. Under a linear rule the answer would be 721 - which happens to be the row for detecting four percentage points, so it looks like a reasonable number, and that is exactly why linear intuition is so hard to catch. 2712 is the total for the same row, and reading a total as a per-arm figure halves the apparent size of the trial. What to take away: setting the minimal clinically important difference lower does not cost proportionally more, it costs quadratically more, and that value is a clinical judgement rather than a statistical one.

Correct answer: 9346. The relative effect is the same, but events have become rare, so accumulating enough of them to tell the arms apart takes far more recruitment

What drives sample size is the number of events, not the number of people. At a control risk of 30%, 242 accumulate enough events; at 1% the same relative effect needs 9346, because each participant now contributes a thirtieth of the event probability. 1812 is the row for a control risk of 5%, and one more row sits between it and 9346, so modest rise does not survive contact with this table. 4673 is the per-arm figure for the same row: the total column is not double-counting, it is the two arms added, and reading it as double-counting halves the apparent size of the trial. This is why rare adverse events almost never reach a conclusion inside an ordinary efficacy trial, and why safety signals usually come from post-marketing database studies.

Correct answer: The Spearman correlation between the two is -1.000, a completely determined monotone relationship - post-hoc power carries no information the p value does not

A correlation of -1.000 is not high correlation but a function: given the sample size and the design, post-hoc power can be computed from the p value and the p value from it. Low power and a large p value are therefore the same sentence on two scales, and neither can stand as independent evidence about sample size. 0.195 is a product of this particular simulation setup; another set of parameters gives another number, so quoting it as a general fact says something false. 0.503 is the largest post-hoc power among the non-significant runs, and why it lands there is handled by another question on this page. What has to be refused here is the one thing more: how far the data were from detecting an effect is fixed by the p value, and converting one into the other and back adds nothing. What a non-significant study should report is the confidence interval, because it answers directly how large an effect the data rule out.

Correct answer: The upper confidence limit. With 40 per arm the median upper limit is 0.451 standard deviations, already a medium effect by Cohen's convention, so this null rules out nothing of clinical importance

Both studies say no difference was detected, but what they exclude differs by more than threefold: 40 per arm rules out only effects larger than 0.451 standard deviations, while 400 per arm reaches 0.138. The first is already at the scale Cohen's convention calls a medium effect, so nothing of clinical importance has been excluded; the second actually says something. Claiming both converge near 0.138 takes a property of the large-sample row and gives it to both, when the monotone shrinking of the upper limit down the three rows is exactly what this section exists to show. The post-hoc power route simply loops back on itself: power is a function of the p value, a non-significant study cannot have high post-hoc power by definition, and so it cannot separate these two studies at all.

Correct answer: No. When the relative effect is the same, the absolute difference is what drives sample size; the protocol with 20% control risk needs 398 in total, far fewer than the other

Halving is a relative quantity, and sample size feeds on the absolute one. Halving a control risk of 20% is a ten point absolute fall and costs 398 in total; halving a control risk of 5% is a quarter of that fall and costs 1812. Saying both need 1812 takes the second protocol's figure and applies it to both, which is the relative effect misread as independent of baseline risk. 199 is the first protocol's per-arm figure, and the total is twice that - reading a per-arm figure as a total is what produces a difference of only a few dozen, when the real gap is several fold. Whenever a protocol says halving the risk, the first question is what the baseline risk is.

Correct answer: It is 0.503, and it depends on neither the sample size nor the true effect - when p lands exactly on the significance threshold, observed power must sit near a half

0.503 is the only universal number in this section: post-hoc power is a function of the p value, and when p equals the significance threshold exactly, observed power has to sit near a half regardless of sample size or true effect. Not significant and post-hoc power is low are therefore the same fact twice, and the first guarantees the second. 0.195 is the median of the same batch of simulations, a product of the 40 per arm and 0.35 standard deviation setup, so it changes with the parameters and quoting it as a general fact says something false. 0.278 is not even in the same column: it is a median upper confidence limit at a larger per-arm size, measured in standard deviations rather than in probability. To say how informative a null is, read the confidence interval, not post-hoc power.