Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions R/kaefa-package.r
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,12 @@
#'
#' The kaefa may inspect this issues from the MMMM or MM
#' in statistical learning theory perspectives using model
#' selection criteria like the DIC (Kang, 2008; Kang, Cohen, & Sung, 2009;
#' Jiao, Kamata, Wang, & Jin, 2012; Jiao & Zhang, 2015) with maximising
#' generalisability of the number of factor decisions in every calibration
#' (Kang, 2008; Preacher, Zhang, Kim, & Mels, 2013).
#' likelihood-based model-selection criteria such as AIC, AICc, BIC, and
#' sample-size-adjusted BIC. Posterior DIC (Spiegelhalter et al., 2002) is
#' accepted only when the fitted model actually supplies it; DIC is not
#' reconstructed from AIC. These criteria support generalisable factor-count
#' decisions in each calibration (Kang, 2008; Preacher, Zhang, Kim, & Mels,
#' 2013).
#'
#' If researcher provide of demographical information in kaefa,
#' kaefa will inspect the optimal number of factor and optimal IRT model,
Expand Down
365 changes: 300 additions & 65 deletions R/kaefa.R

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion README.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The goal of kaefa is to improve researchers' ability to identify unexplained fac
The automated exploratory factor analysis (aefa) framework implements a **greedy search algorithm** to efficiently explore the model space and find improved model configurations. The algorithm iteratively:

1. Evaluates multiple model candidates with different factor structures and item response models
2. Selects the best model based on information criteria (DIC, AIC, BIC, etc.)
2. Selects the best model using AIC by default, with AICc, BIC, and sample-size-adjusted BIC available. DIC is used only when a fitted model actually supplies posterior DIC; it is never approximated with AIC.
3. Assesses item fit and removes poorly fitting items one at a time
4. Re-estimates the model until convergence to a locally optimal solution

Expand All @@ -32,6 +32,8 @@ This greedy approach enables efficient exploration of the model space while seek

- Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research, 48(1), 28-56. https://doi.org/10.1080/00273171.2012.710386
- Jennrich, R. I., & Bentler, P. M. (2011). Exploratory bi-factor analysis. Psychometrika, 76(4), 537-549. https://doi.org/10.1007/s11336-011-9218-4
- Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series model selection in small samples. Biometrika, 76(2), 297-307. https://doi.org/10.1093/biomet/76.2.297
- Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B, 64(4), 583-639. https://doi.org/10.1111/1467-9868.00353

## Installation

Expand Down
12 changes: 10 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@ find improved model configurations. The algorithm iteratively:

1. Evaluates multiple model candidates with different factor structures
and item response models
2. Selects the best model based on information criteria (DIC, AIC, BIC,
etc.)
2. Selects the best model using AIC by default, with AICc, BIC, and
sample-size-adjusted BIC available. DIC is used only when a fitted model
actually supplies posterior DIC; it is never approximated with AIC.
3. Assesses item fit and removes poorly fitting items one at a time
4. Re-estimates the model until convergence to a locally optimal
solution
Expand All @@ -38,6 +39,13 @@ research (Preacher, Zhang, Kim, & Mels, 2013; Jennrich & Bentler, 2011).
- Jennrich, R. I., & Bentler, P. M. (2011). Exploratory bi-factor
analysis. Psychometrika, 76(4), 537-549.
<https://doi.org/10.1007/s11336-011-9218-4>
- Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series model
selection in small samples. Biometrika, 76(2), 297-307.
<https://doi.org/10.1093/biomet/76.2.297>
- Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A.
(2002). Bayesian measures of model complexity and fit. Journal of the
Royal Statistical Society: Series B, 64(4), 583-639.
<https://doi.org/10.1111/1467-9868.00353>

## Installation

Expand Down
41 changes: 37 additions & 4 deletions docs/papers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,43 @@ cited with its DOI. Open-access / preprint links are noted where available.

Both have expectation 1 under good fit.

## 5. Corrected Akaike information criterion (`AICc`)

- **Source:** Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series
model selection in small samples. *Biometrika, 76*(2), 297-307.
DOI: [10.1093/biomet/76.2.297](https://doi.org/10.1093/biomet/76.2.297)
- **Canonical equation.** For maximized log-likelihood `logLik`, parameter count
`k`, and sample size `n`:

AIC = -2 * logLik + 2 * k
AICc = AIC + 2 * k * (k + 1) / (n - k - 1)

Current `mirt` fits supply `AIC` and `logLik` but do not consistently expose
`AICc`. Kaefa therefore recovers `k = (AIC + 2 * logLik) / 2` and applies the
correction exactly. The statistic is undefined when `n <= k + 1`; kaefa
reports that reason rather than returning a fabricated finite score.

## 6. Deviance information criterion (`DIC`) boundary

- **Source:** Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde,
A. (2002). Bayesian measures of model complexity and fit. *Journal of the
Royal Statistical Society: Series B, 64*(4), 583-639.
DOI: [10.1111/1467-9868.00353](https://doi.org/10.1111/1467-9868.00353)
- **Canonical equation.** With posterior mean deviance `Dbar`, deviance at the
posterior mean parameters `D(theta_bar)`, and effective parameter count `pD`:

pD = Dbar - D(theta_bar)
DIC = Dbar + pD

DIC is a posterior-deviance criterion. The maximum-likelihood/MAP models
produced by current `mirt` versions do not expose the posterior quantities
needed to reconstruct it. Kaefa accepts DIC only when the fitted model
supplies a finite DIC value and never relabels AIC as DIC.

## Audit note

kaefa does **not** re-implement `P(theta)`, the MML-EM E-/M-step, `S-X2`, `infit`,
or `outfit`; those are delegated verbatim to `mirt` and are therefore correct by
construction (subject to `mirt`'s own validation). The only package-local numeric
formulas are the fit-based **decision rules** documented above; the `Zh` rule is
the one that had drifted out of internal consistency and has been restored.
or `outfit`; those are delegated verbatim to `mirt` and remain subject to
`mirt`'s validation. Package-local formulas and decision rules are pinned above:
the `Zh` cutoff, the exact Hurvich-Tsai AICc correction, and the explicit
posterior-information boundary that prevents DIC from being fabricated.
2 changes: 2 additions & 0 deletions inst/WORDLIST
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
18637
20
200
2002
2008
2009
2011
Expand Down Expand Up @@ -102,6 +103,7 @@ Kang
LAN
LCA
Leiman
MAP
MH
MHRM
MIRT
Expand Down
6 changes: 4 additions & 2 deletions inst/shiny-app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ launchAEFA()
2. **Configure Model**:
- Set minimum and maximum number of factors to explore
- Choose rotation method (bifactorQ recommended for most cases)
- Select model selection criteria (DIC is default)
- Select model selection criteria (AIC is default for mirt ML/MAP fits)
- Optionally enable model history saving to inspect candidate models

3. **Run Analysis**: Click the "Run Analysis" button
Expand All @@ -56,7 +56,9 @@ following inputs and pass them to `aefa()`:
- **Rotation method**: select input mapped to `rotate` (e.g., `bifactorQ`,
`geominQ`, `oblimin`).
- **Model selection criteria**: select input mapped to
`modelSelectionCriteria` (e.g., `DIC`, `AIC`, `AICc`, `BIC`, `saBIC`).
`modelSelectionCriteria` (e.g., `AIC`, `AICc`, `BIC`, `saBIC`). DIC is an
API-only option and requires a fitted model that actually supplies posterior
DIC; the application never substitutes AIC for it.
- **Model history toggle**: checkbox mapped to `saveModelHistory` (recommended
for inspecting candidate models).

Expand Down
6 changes: 3 additions & 3 deletions inst/shiny-app/app.R
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ ui <- fluidPage(
selected = "bifactorQ"),

selectInput("modelSelection", "Model Selection Criteria:",
choices = c("DIC", "AIC", "AICc", "BIC", "SABIC"),
selected = "DIC"),
choices = c("AIC", "AICc", "BIC", "SABIC"),
selected = "AIC"),

checkboxInput("saveHistory", "Save Model History", TRUE),

Expand Down Expand Up @@ -139,7 +139,7 @@ ui <- fluidPage(
tags$ul(
tags$li(strong("Minimum/Maximum Factors:"), "Set the range of factors to explore. The algorithm will test models with different numbers of factors and select the best one."),
tags$li(strong("Rotation Method:"), "Choose the rotation method for factor loadings. 'bifactorQ' is recommended for most cases."),
tags$li(strong("Model Selection Criteria:"), "Choose the criteria for selecting the best model. 'DIC' (Deviance Information Criterion) is the default.")
tags$li(strong("Model Selection Criteria:"), "Choose the criterion for selecting the best model. AIC is the default for the maximum-likelihood/MAP models fitted by mirt; AICc applies the finite-sample correction.")
),

h4("Step 3: Run the Analysis"),
Expand Down
12 changes: 9 additions & 3 deletions man/aefa.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 6 additions & 4 deletions man/kaefa.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 10 additions & 1 deletion tests/testthat/helper-test-data.R
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,16 @@ create_binary_test_data <- function(n_items = 10, n_obs = 100) {
}

.merge_test_defaults <- function(defaults, dots) {
utils::modifyList(defaults, dots)
dot_names <- names(dots)
if (is.null(dot_names)) {
dot_names <- rep("", length(dots))
}

is_named <- !is.na(dot_names) & nzchar(dot_names)
positional <- dots[!is_named]
named <- dots[is_named]

c(positional, utils::modifyList(defaults, named))
}

.skip_expensive_ci_calls <- function(function_name) {
Expand Down
21 changes: 11 additions & 10 deletions tests/testthat/test-aefa-advanced-parameters.R
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

context("AEFA Advanced Parameter Interactions and Combinations")

.ensure_kaefa_namespace()

expect_valid_aefa <- function(result) {
expect_false(inherits(result, "try-error"))
expect_s3_class(result, "aefa")
Expand All @@ -15,16 +17,15 @@ expect_valid_aefa <- function(result) {
# Test Suite 1: Model Selection Criteria Variations
# ============================================================

test_that("aefa respects different model selection criteria - DIC", {
test_data <- create_test_data(n_items = 5, n_obs = 100)

result <- try(aefa(test_data,
minExtraction = 1,
maxExtraction = 1,
modelSelectionCriteria = "DIC"),
silent = TRUE)

expect_valid_aefa(result)
test_that("DIC is not fabricated for mirt ML/MAP fits", {
expect_error(
kaefa:::.aefaFitCriterionValue(
list(AIC = 120, logLik = -50),
criterion = "DIC",
sample_size = 100
),
"posterior deviance draws.*never replaced by AIC"
)
})

test_that("aefa respects different model selection criteria - AIC", {
Expand Down
4 changes: 2 additions & 2 deletions tests/testthat/test-aefa-greedy-algorithm.R
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ test_that("aefa greedy search evaluates model candidates", {
expect_true(is.list(result$itemFitTrials))
last_model <- result$estModelTrials[[length(result$estModelTrials)]]
if (isS4(last_model) && "Fit" %in% slotNames(last_model)) {
fit_values <- c(last_model@Fit$AIC, last_model@Fit$BIC, last_model@Fit$DIC)
fit_values <- c(last_model@Fit$AIC, last_model@Fit$BIC, last_model@Fit$SABIC)
expect_true(any(is.finite(fit_values)),
info = "Selected model should include information criteria values")
}
Expand All @@ -107,7 +107,7 @@ test_that("aefa greedy search evaluates model candidates", {
test_that("aefa uses information criteria for model selection", {
test_data <- create_test_data(n_items = 6, n_obs = 100)

# The function should select best model based on DIC, AIC, BIC, etc.
# The function should select the best model using a criterion supplied by mirt.
result <- try(aefa(test_data, minExtraction = 1, maxExtraction = 2), silent = TRUE)

if (!inherits(result, "try-error") && !is.null(result)) {
Expand Down
68 changes: 68 additions & 0 deletions tests/testthat/test-model-selection-criteria.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
.ensure_kaefa_namespace()

test_that("AICc preserves the Hurvich-Tsai small-sample correction", {
fit <- list(AIC = 120, logLik = -50)
parameter_count <- 10
expected <- 120 + (2 * parameter_count * (parameter_count + 1)) /
(100 - parameter_count - 1)

expect_equal(
kaefa:::.aefaFitCriterionValue(fit, "AICc", sample_size = 100),
expected,
tolerance = 1e-12
)
})

test_that("model-supplied criteria are preserved exactly", {
fit <- list(AIC = 120, AICc = 123.5, BIC = 140, SABIC = 130, DIC = 117.25)

expect_identical(kaefa:::.aefaFitCriterionValue(fit, "AIC", 100), 120)
expect_identical(kaefa:::.aefaFitCriterionValue(fit, "AICc", 100), 123.5)
expect_identical(kaefa:::.aefaFitCriterionValue(fit, "BIC", 100), 140)
expect_identical(kaefa:::.aefaFitCriterionValue(fit, "saBIC", 100), 130)
expect_identical(kaefa:::.aefaFitCriterionValue(fit, "DIC", 100), 117.25)
})

test_that("invalid information-criterion substitutions fail visibly", {
fit <- list(AIC = 120, logLik = -50)

expect_error(
kaefa:::.aefaFitCriterionValue(fit, "DIC", 100),
"DIC is unavailable"
)
expect_error(
kaefa:::.aefaFitCriterionValue(fit, "CAIC", 100),
"not an alias for corrected AIC"
)
expect_error(
kaefa:::.aefaFitCriterionValue(fit, "AICc", 11),
"AICc is undefined"
)
})

test_that("candidate selection retains original indices after skipped fits", {
expect_identical(kaefa:::.aefaBestScoreIndex(c(Inf, 20, 10)), 3L)
expect_identical(kaefa:::.aefaBestScoreIndex(c(NA, Inf)), NA_integer_)
})

test_that("item exclusions require a new name present in the fitted model", {
expect_identical(
kaefa:::.aefaValidNewItemExclusions(
excluded = "Item1",
proposed = c("Item1", "Item2", "missing", NA_character_),
available = c("Item1", "Item2", "Item3")
),
"Item2"
)
expect_identical(
kaefa:::.aefaValidNewItemExclusions("Item1", "Item1", c("Item1", "Item2")),
character()
)
})

test_that("local worker selection survives unavailable CPU telemetry", {
expect_identical(kaefa:::.aefaParallelProcessorCount(NULL, 12), 4L)
expect_identical(kaefa:::.aefaParallelProcessorCount(c(10, 20), 12), 4L)
expect_identical(kaefa:::.aefaParallelProcessorCount(5, 12), 1L)
expect_identical(kaefa:::.aefaParallelProcessorCount(80, NA_real_), 1L)
})
2 changes: 1 addition & 1 deletion tests/testthat/test-shiny-app.R
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ test_that("Model selection criteria choices are valid", {
ui_html <- as.character(ui)

# Check for expected model selection criteria
expected_criteria <- c("DIC", "AIC", "AICc", "BIC", "SABIC")
expected_criteria <- c("AIC", "AICc", "BIC", "SABIC")

for (criterion in expected_criteria) {
expect_true(
Expand Down
14 changes: 14 additions & 0 deletions tests/testthat/test-test-bootstrap-contract.R
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,17 @@ test_that("test file lookup uses explicit base package helpers", {
perl = TRUE
))
})

test_that("test defaults preserve positional data and override named options", {
data_arg <- data.frame(item = 1:3)

merged <- .merge_test_defaults(
list(NCYCLES = 120, BURNIN = 40),
list(data_arg, BURNIN = 10)
)

expect_identical(merged[[1]], data_arg)
expect_identical(names(merged)[[1]], "")
expect_identical(merged$NCYCLES, 120)
expect_identical(merged$BURNIN, 10)
})
Loading