tune 2.1.0

• Model tuning has been enabled for quantile regression models. (#1125)

• When calculating resampling estimates, we can now use a weighted mean based on the number of rows in the assessment set. You can opt-in to this using the new add_resample_weights() function. See ?calculate_resample_weights (#990)

• The warning threshold when check the size of a workflow is now a parameter to the control functions and has a new default of 100MB. (#914)

• Tuning of models with submodel parameters now predicts all submodels at once (again) to speed up the tuning process (#1140).

• tune_bayes() now uses an uncertainty sample if all predictions from the Gaussian process model fail (#1167).

Breaking Changes

• The Gaussian process model package was changed from \pkg{GPfit} to \pkg{GauPro} because the former is no longer actively maintained. There are some differences:

  • Fit diagnostics are computed and reported. If the fit quality is poor, an "uncertainty sample" that is furthest away from the existing data is used as the new candidate.
  • The GP no longer uses binary indicators for qualitative predictors. Instead, a "categorical kernel" is used for those parameter columns. Fewer starting values are required with this change.
  • For numeric predictors, the Matern 3/2 kernel is always used.

Bug Fixes

• Fixed a bug where models with submodel parameters would train all calibration models on predictions from a single submodel value instead of the correct value for each submodel (#1144).

• A bug was fixed where NULL results generated during simulated annealing would cause errors when logging.

• Fixed a bug for cases where we tune a grid without a model parameter but with a postprocessing parameter (#1119)

• Fixed a bug in augment() when using last_fit() objects (#1110)

• In tune_grid(), a bug was fixed that caused inefficiency where preprocessing steps were applied to data being predicted several times (redundantly). We now execute that operation once. (#1101)

Other Changes

• The developer functions val_class_and_single() and val_class_or_null() are now deprecated (#1161).

• check_metrics() is deprecated in favor of check_metrics_arg() (#1160).

• finalize_workflow_preprocessor() is deprecated (#1157).

• Prepends a period to the name, and exports the following functions. This is to improve integration with other packages:

  • check_grid() (e.g., now named .check_grid())
  • determine_pred_types()
  • get_config_key()
  • loop_over_all_stages()
  • update_parallel_over()
  • get_data_subsets()

tune 2.0.1

• Fixed a bug where int_pctl() wouldn't work on last_fit() outcomes when future parallelism was enabled. (#1099)

tune 2.0.0

Changes to tune_grid().

• A major rewrite/refactor of the underlying code that runs tune_grid(). This was an upgrade to add postprocessing and to modernize our parallel processing infrastructure.

• The pattern of .config values has changed.

  • For grid search, it changes from Preprocessor{num}_Model{num} to pre{num}_mod{num}_post{num}. The numbers include a zero when that element was static. For example, a value of pre0_mod3_post4 means no preprocessors were tuned and the model and postprocessor(s) had at least three and four candidates, respectively.
  • For iterative search, the pattern is not iter{num} instead of Iter{num} and the numbers are now zero padded to sort better. For example, if there between 10 and 99 iterations, the first .config value is now iter01 instead of Iter1.

• The package will now log a backtrace for errors and warnings that occur during tuning. When a tuning process encounters issues, see the new trace column in the collect_notes(.Last.tune.result) output to find precisely where the error occurred (#873).

• Postprocessors can now be tuned. Currently, we support the tailor package.

Parallel Processing

• Introduced support for parallel processing with mirai in addition to the currently supported framework future. See ?parallelism to learn more (#1028).

• Sequential and parallel processing all use the same L'Ecuyer-CMRG seeds (conditional on parallel_over) (#1033).

Breaking Changes

• The foreach package is no longer supported. Instead, use the future or mirai packages.

• The parallel backend(s) and the methods of constructing seeds for workers have changed. There will be a lack of reproducibility between objects created in this version of tune and previous versions.

Other Changes

• int_pctl() now includes an option (keep_replicates) to retain the individual bootstrap estimates. It also processes the resamples more efficiently (#1000).

• A min_grid() methods was added for proportional_hazards models so that their submodels are processed appropriately.

• Post-processing: new schedule_grid() for scheduling a grid including post-processing (#988).

• Removed functions deprecated since tune version 1.6.0 (circa 2021-07-21).

tune 1.3.0

• The package will now warn when parallel processing has been enabled with foreach but not with future. See ?parallelism to learn more about transitioning your code to future (#878, #866). The next version of tune will move to a pure future implementation.

• When automatic grids are used, dials::grid_space_filling() is now used (instead of dials::grid_latin_hypercube()). Overall, the new function produces optimized designs (not depending on random numbers). When using Bayesian models, we will use a Latin Hypercube since we produce 5,000 candidates, which is too slow to do with pre-optimized designs.

tune 1.2.1

• Addressed issue in int_pctl() where the function would error when parallelized using makePSOCKcluster() (#885).

• Addressed issue where tuning functions would raise the error object 'iteration' not found with plan(multisession) and the control option parallel_over = "everything" (#888).

Note that this GitHub release is a re-release with the tag v1.2.1, corresponding to the package version that arrived on CRAN on 2024-04-18. The previous GitHub release with this tag did not match the version released to CRAN. Closes #897.

tune 1.2.0

New Features

• tune now fully supports models in the "censored regression" mode. These models can be fit, tuned, and evaluated like the regression and classification modes. tidymodels.org has more information and tutorials on how to work with survival analysis models.

• Introduced support for parallel processing using the future framework. The tune package previously supported parallelism with foreach, and users can use either framework for now. In a future release, tune will begin the deprecation cycle for parallelism with foreach, so we encourage users to begin migrating their code now. See the Parallel Processing section in the "Optimizations" article to learn more (#866).

• Added a type argument to collect_metrics() to indicate the desired output format. The default, type = "long", returns output as before, while type = "wide" pivots the output such that each metric has its own column (#839).

• Added a new function, compute_metrics(), that allows for computing new metrics after evaluating against resamples. The arguments and output formats are closely related to those from collect_metrics(), but this function requires that the input be generated with the control option save_pred = TRUE and additionally takes a metrics argument with a metric set for new metrics to compute. This allows for computing new performance metrics without requiring users to re-fit and re-predict from each model (#663).

• A method for rsample's int_pctl() function that will compute percentile confidence intervals on performance metrics for objects produced by fit_resamples(), tune_*(), and last_fit().

• The Brier score is now part of the default metric set for classification models.

Bug Fixes

• last_fit() will now error when supplied a fitted workflow (#678).

• Fixes bug where .notes entries were sorted in the wrong order in tuning results for resampling schemes with IDs that aren't already in alphabetical order (#728).

• Fixes bug where .config entries in the .extracts column in tune_bayes() output didn't align with the entries they ought to in the .metrics and .predictions columns (#715).

• Metrics from apparent resamples are no longer included when estimating performance with estimate_tune_results() (and thus with collect_metrics(..., summarize = TRUE) and compute_metrics(..., summarize = TRUE), #714).

• Handles edge cases for tune_bayes()' iter argument more soundly. For iter = 0, the output of tune_bayes() should match tune_grid(), and tune_bayes() will now error when iter < 0. tune_bayes() will now alter the state of RNG slightly differently, resulting in changed Bayesian optimization search output (#720).

• augment() methods to tune_results, resample_results, and last_fit objects now always return tibbles (#759).

Other Changes

• Improved error message when needed packages aren't installed (#727).

• augment() methods to tune_results, resample_results, and last_fit objects now always returns tibbles (#759).

• Improves documentation related to the hyperparameters associated with extracted objects that are generated from submodels. See the "Extracting with submodels" section of ?collect_extracts to learn more.

• eval_time and eval_time_target attribute was added to tune objects. There are also .get_tune_eval_times() and .get_tune_eval_time_target() functions.

• collect_predictions() now reorders the columns so that all prediction columns come first (#798).

• augment() methods to tune_results, resample_results, and last_fit objects now return prediction results in the first columns (#761).

• autoplot() will now meaningfully error if only 1 grid point is present, rather than producing a plot (#775).

• Added notes on case weight usage to several functions (#805).

• For iterative optimization routines, autoplot() will use integer breaks when type = "performance" or type = "parameters".

Breaking Changes

• Several functions gained an eval_time argument for the evaluation time of dynamic metrics for censored regression. The placement of the argument breaks passing-by-position for one or more other arguments to autoplot.tune_results() and the developer-focused check_initial() (#857).

• Ellipses (...) are now used consistently in the package to require optional arguments to be named. For functions that previously had ellipses at the end of the function signature, they have been moved to follow the last argument without a default value: this applies to augment.tune_results(), collect_predictions.tune_results(), collect_metrics.tune_results(), select_best.tune_results(), show_best.tune_results(), and the developer-focused estimate_tune_results(), load_pkgs(), and encode_set(). Several other functions that previously did not have ellipses in their signatures gained them: this applies to conf_mat_resampled() and the developer-focused check_workflow(). Optional arguments previously passed by position will now error informatively prompting them to be named. These changes don't apply in cases when the ellipses are currently in use to forward arguments to other functions (#863).

tune 1.1.2

• last_fit() now works with the 3-way validation split objects from rsample::initial_validation_split(). last_fit() and fit_best() now have a new argument add_validation_set to include or exclude the validation set in the dataset used to fit the model (#701).

• Disambiguates the verbose and verbose_iter control options to better align with documented functionality. The former controls logging for general progress updates, while the latter only does so for the Bayesian search process. (#682)

tune 1.1.1

• Fixed a bug introduced in tune 1.1.0 in collect_() functions where the
  .iter column was dropped.

tune 1.1.0

tune 1.1.0 introduces a number of new features and bug fixes, accompanied by various optimizations that substantially decrease the total evaluation time to tune hyperparameters in the tidymodels.

New features

• Introduced a new function fit_best() that provides a shorthand interface to fit a final model after parameter tuning. (#586)

• Refined machinery for logging issues during tuning. Rather than printing out warnings and errors as they appear, the package will now only print unique tuning issues, updating a dynamic summary message that maintains counts of each unique issue. This feature is only enabled for tuning sequentially and can be manually toggled with the verbose option. (#588)

• Introduced collect_extracts(), a function for collecting extracted objects from tuning results. The format of results closely mirrors collect_notes(), where the extracted objects are contained in a list-column alongside the resample ID and workflow .config. (#579)

Bug fixes

• Fixed bug in select_by_pct_loss() where the model with the greatest loss within the limit was returned rather than the most simple model whose loss was within the limit. (#543)

• Fixed bug in tune_bayes() where .Last.tune.result would return intermediate tuning results. (#613)

• Extended show_best(), select_best(), select_by_one_std_error(), select_by_pct_loss() to accommodate metrics with a target value of zero (notably, yardstick::mpe() and yardstick::msd()). (#243)

Other changes

• Implemented various optimizations in tune's backend that substantially decrease the total evaluation time to tune hyperparameters with the tidymodels. (#634, #635, #636, #637, #640, #641, #642, #648, #649, #653, #656, #657)

• Allowed users to supply list-columns in grid arguments. This change allows for manually specifying grid values that must be contained in list-columns, like functions or lists. (#625)

• Clarified error messages in select_by_* functions. Error messages now only note entries in ... that are likely candidates for failure to arrange(), and those error messages are no longer duplicated for each entry in ....

• Improved condition handling for errors that occur during extraction from workflows. While messages and warnings were appropriately handled, errors occurring due to misspecified extract() functions being supplied to control_*() functions were silently caught. As with warnings, errors are now surfaced both during execution and at print() (#575).

• Moved forward with the deprecation of parameters() methods for workflows, model_specs, and recipes. Each of these methods will now warn on every usage and will be defunct in a later release of the package. (#650)

• Various bug fixes and improvements to documentation.

tune 1.0.1

• last_fit(), fit_resamples(), tune_grid(), and tune_bayes() do not automatically error if the wrong type of control object is passed. If the passed control object is not a superset of the one that is needed, the function will still error. As an example, passing control_grid() to tune_bayes() will fail but passing control_bayes() to tune_grid() will not. (#449)

• The collect_metrics() method for racing objects was removed (and is now in the finetune package).

• Improved prompts related to parameter tuning. When tuning parameters are supplied that are not compatible with the given engine, tune_*() functions will now error. (#549)

• The control_bayes() got a new argument verbose_iter that is used to control the verbosity of the Bayesian calculations. This change means that the verbose argument is being passed to tune_grid() to control its verbosity.

• The control_last_fit() function gained an argument allow_par that defaults to FALSE. This change addresses failures after last_fit() using modeling engines that require native serialization, and we anticipate little to no increase in time-to-fit resulting from this change. (#539, tidymodels/bonsai#52)

• show_notes() does a better jobs of... showing notes. (#558)