What statistical methods  are used in NCA?

NCA's statistical significance test is an 'appoximate permutation test' that estimates the p values by randomly drawing samples from the permutation distribution. This test can be activated in the NCA software using the argument test.rep = 10000 (or another number) in the nca_analysis function. 

A simulation of the power of the test for estimating a desirable sample size for a new NCA study can be activated in the NCA software using the nca_power function. 

How is NCA's statistical significance test done?

NCA’s significance test has the following parts

1. Calculate the necessity effect size for the observed sample.
2. Formulate the null-hypothesis that suggests that X and Y in the population are not related. Any effect size is a random effect.
3. Create a large set of random resamples (e.g., 10,000) using approximate permutation. In a permutation test the X and Y values that are observed in the sample are shuffled to create new resamples (same sample size) with ‘cases’ where X and Y are unrelated.
4. Calculate the effect size of all resamples. The set of effect sizes comprises an estimated distribution of effect size under the assumption that X and Y are not related.
5. Compare the effect size of the observed sample (see part 1) with the distribution of effect sizes of the random resamples. The fraction of random resamples for which the effect size is equal to, or greater than the observed effect size (p value) informs us about the statistical (in)compatibility of the data with the null hypothesis.

How is NCA's statistical significance used in the desision process about necessity?

NCA’s significance test (p value) is only one part of the decision process about necessity (e.g. p < 0.05). The other parts are the availability of theoretical support (e.g., a formulated and justified necessity hypothesis) which focuses the analysis on the right expected empty corner of the XY plot, and a large enough effect size that is considered practically relevant (e.g., > 0.10). If one of these reuirements is not satisfied NCA rejects necessity. NCA consideres necessity being supported (not rejected) only if all three requirements are met. Simulations show that with this approach NCA has a high True Positive Rate (sensitivity) and a high True Negative Rate (specificity), which help to minimize the risk of false positive and false negative conclusions about necessity.