AutoMLR

Automated survival-analysis, binary-classification, continuous-outcome, and ordinal-outcome machine-learning combination models in R. Train and compare model variants on multi-cohort data with a single call, select models by outcome-appropriate out-of-fold metrics, and report integrated diagnostics.

Status: 1.0.0 release candidate. Survival, binary, continuous, and ordinal workflows are available with cohort-aware preparation, 18-variant default candidate pools, single-/two-model combination search, HTML reports, CSV audit tables, publication-style figures, and automated tests.

Why AutoMLR?

AutoMLR is a multi-outcome biomedical modeling framework. The survival workflow is designed to answer one question end-to-end:

Given multiple cohorts with survival outcomes, which single model (or combination) gives the most robust prognostic signature, and how confident am I?

Survival Algorithm Families

# Family Algorithm Package
1 Penalised Cox Lasso Cox glmnet
2 Penalised Cox Ridge Cox glmnet
3 Penalised Cox Elastic Net Cox glmnet
4 Classic Cox Stepwise Cox survival
5 Boosted Cox CoxBoost CoxBoost
6 Dimension-reduced plsRcox (partial least squares) plsRcox
7 Dimension-reduced SuperPC superpc
8 Tree-boosted GBM-Cox gbm
9 Kernel survival-SVM survivalsvm
10 Tree-ensemble Random Survival Forest (RSF) randomForestSRC

Installation

# install from a local source checkout
remotes::install_local("AutoMLR", dependencies = c("Depends", "Imports"))

# or load directly from a local checkout:
# pkgload::load_all("AutoMLR")

AutoMLR keeps heavyweight model engines as optional suggested dependencies so the package can be installed even on machines where some modelling packages are hard to compile. Missing backends skip only the affected model variants while the remaining algorithms and reports continue to run.

deps <- check_automlr_dependencies()
deps

# Install only the optional engines you need.
install.packages(c("glmnet", "gbm", "randomForestSRC", "timeROC"))

Progress messages are mirrored to a session log when logging is enabled:

initialize_auto_logging("automlr_logs")
# Run AutoMLR analysis here.
disable_auto_logging()

Model-evaluation and ensemble-fitting messages such as Evaluating ... and Fitting ... are then written to the log file as well as the console.

Minimal Example

library(AutoMLR)

# Use the canonical CRAN survival::lung dataset.
data("cancer", package = "survival")
lung <- na.omit(lung)
lung$cohort <- rep(c("A", "B"), length.out = nrow(lung))

prep <- prepare_cohort_input(
  data    = lung,
  cohort  = "cohort",
  time    = "time",
  status  = "status"
)
report_cohort_intersection(prep)

params <- automlr_parameters()
list_surv_algorithms()

# `prep` can be passed directly; cohort labels are then used automatically for
# stability diagnostics in the combination table.
ens <- fit_surv_ensemble(
  prep,
  params = automlr_parameters(algorithms = "lasso_cox", verbose = FALSE),
  min_models = 1,
  max_models = 1
)

export_surv_results(
  ens,
  output_dir = "results/lung_example",
  formats = c("pdf", "png")
)

This verifies: data contract, cohort intersection, parameter defaults, and the 10-algorithm registry load. list_model_variants() shows the concrete algorithm + hyperparameter candidates; count_surv_combinations() reports how many combinations will be searched. Single-model fitting, all-subsets combination ranking, and weighted ensemble fitting. By default, combinations are searched across distinct base algorithm families rather than pairing two thresholds from the same algorithm. Combination tables keep C-index as the primary rank by default while reporting queue / subsample stability, member failure rate, risk spread, and optional time-dependent AUC diagnostics. When fit_surv_ensemble() receives the automlr_input object returned by prepare_cohort_input(), the original cohort labels are wired into stability_groups automatically. The surv_svm candidate is evaluated with k-fold resampling by default because the optional survivalsvm backend can fail on single-row predictions during leave-one-out cross-validation. Users can restore the old behavior with automlr_parameters(surv_svm_resampling = "loocv") or exclude the model via the algorithms argument. export_surv_results() writes a batch result bundle with an HTML report, publication figures, diagnostic figures, CSV tables, fitted RDS objects, and session metadata. It also exports per-sample risk scores plus time-dependent AUC / ROC tables when timeROC is installed, and writes bilingual English / Chinese Markdown summaries for data preparation, base-model screening, and ensemble selection under summaries/. The current regular survival report has four step-specific interpretation files: data_summary.md, base_model_summary.md, ensemble_summary.md, and explainability_summary.md. The last one explains feature importance, SHAP-style approximations, nomogram, calibration, DCA, and model forest-plot outputs. The default publication bundle is deduplicated to a final figure set: an all-single-model cohort C-index heatmap, an IRLS-style combination-by-cohort benchmark matrix with mean C-index side bars, multi-cohort Kaplan-Meier panels when risk groups are estimable, and multi-cohort timeROC panels when timeROC is available. Publication heatmaps omit rows with no finite metric values, keep partial missing cells as explicit NA tiles, and adapt color scales to the finite metric range so small differences remain visible.

For a figure-rich walkthrough with standalone code blocks, see inst/tutorials/AutoMLR_tutorial.md after installing or unpacking the package.

Lower-level evaluation helpers

Prepared cohort objects can be converted to modeling matrices without using internal ::: calls:

xy <- automlr_input_to_surv_xy(prep)

single <- evaluate_algorithm_loocv(
  "lasso_cox",
  X = xy$X,
  y = xy$y,
  params = automlr_parameters(algorithms = "lasso_cox", verbose = FALSE)
)

The same convenience pattern is available for the other workflows: automlr_input_to_binary_xy(), automlr_input_to_continuous_xy(), and automlr_input_to_ordinal_xy().

For threshold-style ensemble selection, users may keep fixed cutoffs or ask AutoMLR to recommend a data-adaptive cutoff from the candidate metric distribution:

params_auto <- automlr_parameters(
  algorithms = c("lasso_cox", "ridge_cox"),
  auto_min_cindex = TRUE,
  auto_quantile = 0.50,
  verbose = FALSE
)

ens_auto <- fit_surv_ensemble(
  prep,
  params = params_auto,
  strategy = "threshold"
)

ens_auto$selection_threshold

auto_quantile = 0.50 uses the median candidate C-index; larger values such as 0.75 are more selective. Analogous controls are available as auto_min_auc, auto_min_r2, and auto_min_qwk for binary, continuous, and ordinal workflows.

Binary classification workflow

The binary workflow mirrors the survival workflow but uses positive-class probabilities and ROC AUC:

data(iris)
iris2 <- iris[iris$Species != "setosa", ]
iris2$cohort <- rep(c("A", "B"), length.out = nrow(iris2))
iris2$sample_id <- paste0("iris_", seq_len(nrow(iris2)))

prep_bin <- prepare_binary_cohort_input(
  iris2,
  cohort = "cohort",
  outcome = "Species",
  id = "sample_id",
  positive_class = "versicolor",
  negative_class = "virginica"
)

params_bin <- binarymlr_parameters(
  resampling = "loocv",
  verbose = FALSE
)
count_binary_combinations(params_bin, min_size = 1, max_size = 2)

bin_fit <- fit_binary_ensemble(
  prep_bin,
  params = params_bin,
  min_models = 1,
  max_models = 2,
  rank_by = "auc"
)

export_binary_results(
  bin_fit,
  output_dir = "results/iris_binary",
  formats = c("png", "pdf"),
  summary_language = "bilingual"
)

The default binary pool contains 18 model variants across penalized logistic, standard / stepwise logistic, GBM, random forest, PCA-logistic, and Gaussian naive Bayes families. The default search includes single models and two-model probability combinations while excluding same-algorithm variant pairs. export_binary_results() writes ROC/PR curves, cohort AUC heatmaps, combination benchmark matrices, calibration, DCA, confusion matrix, feature importance, a model-performance forest plot, predicted probabilities, threshold metrics, preprocessing audit tables, resampling manifests, and the same four step-specific bilingual interpretation documents used by the survival report. Multi-class outcomes are rejected by default to avoid accidental “positive vs all other” analyses; set collapse_other = TRUE only when that collapse is intentional. Exported prediction tables distinguish apparent probabilities from out-of-fold probabilities, and the default report uses the out-of-fold probabilities for model diagnostics.

Continuous outcome workflow

The continuous workflow targets numeric outcomes and ranks models by out-of-fold RMSE by default:

d <- mtcars
d$sample_id <- rownames(mtcars)
d$cohort <- rep(c("A", "B"), length.out = nrow(d))

prep_cont <- prepare_continuous_cohort_input(
  d,
  cohort = "cohort",
  outcome = "mpg",
  id = "sample_id"
)

params_cont <- continuousmlr_parameters(
  resampling = "kfold",
  k_folds = 5,
  verbose = FALSE
)

cont_fit <- fit_continuous_ensemble(
  prep_cont,
  params = params_cont,
  min_models = 1,
  max_models = 2,
  rank_by = "rmse"
)

export_continuous_results(
  cont_fit,
  output_dir = "results/mtcars_continuous",
  formats = c("png", "pdf")
)

The default pool contains 18 regression variants. Exported outputs include single-model and combination rankings, predictions, cohort-level RMSE / MAE / R-squared diagnostics, observed-vs-predicted plots, residual distributions, feature importance, preprocessing audits, fitted objects, session metadata, and an HTML report.

Ordinal outcome workflow

The ordinal workflow targets ordered categorical outcomes and ranks models by out-of-fold quadratic weighted kappa:

data(iris)
iris$cohort <- rep(c("A", "B", "C"), length.out = nrow(iris))
iris$sample_id <- paste0("iris_", seq_len(nrow(iris)))

prep_ord <- prepare_ordinal_cohort_input(
  iris,
  cohort = "cohort",
  outcome = "Species",
  id = "sample_id",
  ordered_levels = c("setosa", "versicolor", "virginica")
)

params_ord <- ordinalmlr_parameters(
  resampling = "kfold",
  k_folds = 5,
  verbose = FALSE
)

ord_fit <- fit_ordinal_ensemble(
  prep_ord,
  params = params_ord,
  min_models = 1,
  max_models = 2,
  rank_by = "qwk"
)

export_ordinal_results(
  ord_fit,
  output_dir = "results/iris_ordinal",
  formats = c("png", "pdf")
)

Ordinal exports include QWK rankings, predicted ordered classes, cohort performance, confusion matrices, observed-score plots, feature importance, preprocessing audits, fitted objects, session metadata, and an HTML report. Users should supply ordered_levels whenever the natural order is not already encoded in an ordered factor.

Extreme screening

For exploratory upper-bound analysis, extreme_surv_screen() first fits all candidate variants on the full dataset and evaluates them on the same samples to rank apparent single- or two-model combinations. It then carries only the top N combinations into a real stratified train/validation seed search.

extreme <- extreme_surv_screen(
  prep,
  top_n = 10,
  seeds = 1:500,
  train_fraction = 0.70,
  min_models = 1,
  max_models = 2,
  rank_by = "apparent_cindex"
)

extreme$combination_summary
extreme$top_combinations
extreme$best

export_extreme_screen_results(
  extreme,
  output_dir = "results/lung_extreme_screen",
  formats = c("png", "pdf"),
  summary_language = "bilingual"
)

The first stage is intentionally optimistic and is meant to estimate the dataset’s apparent ceiling, not external validation performance. The second stage reports the best random seed, model combination, train C-index, validation C-index, and cohort-level validation diagnostics when cohorts are available. export_extreme_screen_results() writes complete audit tables plus Morandi-toned PNG/PDF figures for apparent ranking, seed-search distributions, seed-by-combination heatmaps, best-by-seed trajectories, apparent-vs-validation gaps, and cohort-level top-row diagnostics. It also writes a bilingual English / Chinese summary_report.md by default, or a single-language report with summary_language = "zh" or "en", summarizing the current best apparent model, best validation seed, train / validation C-index, top seed-search rows, combination-level stability, and any failed seed-search fits.

Release Scope

AutoMLR 1.0.0 includes survival, binary, continuous, and ordinal workflows; cohort-aware input preparation; 18-variant default candidate pools; single- and two-model combination search; publication figures; HTML reports; fitted objects; CSV audit tables; bilingual summaries for survival and binary workflows; and automated tests covering the main exported interfaces.

Data contract

Single long-format data frame:

prepare_cohort_input() splits by cohort and returns the feature intersection across cohorts, along with a diagnostic report.

License

MIT (see LICENSE).