diff --git a/docs/src/tutorials/regularization/regularization.md b/docs/src/tutorials/regularization/regularization.md index 02d3b3ba..37e42975 100644 --- a/docs/src/tutorials/regularization/regularization.md +++ b/docs/src/tutorials/regularization/regularization.md @@ -5,40 +5,23 @@ For ridge regularization, you can simply use `SemRidge` as an additional loss function (for example, a model with the loss functions `SemML` and `SemRidge` corresponds to ridge-regularized maximum likelihood estimation). -For lasso, elastic net and (far) beyond, we provide the `ProximalSEM` package. You can install it and load it alongside `StructuralEquationModels`: +For lasso, elastic net and (far) beyond, you can load the `ProximalAlgorithms.jl` and `ProximalOperators.jl` packages alongside `StructuralEquationModels`: ```@setup reg -import Pkg -Pkg.add(url = "https://github.com/StructuralEquationModels/ProximalSEM.jl") - -using StructuralEquationModels, ProximalSEM -``` - -```julia -import Pkg -Pkg.add(url = "https://github.com/StructuralEquationModels/ProximalSEM.jl") - -using StructuralEquationModels, ProximalSEM -``` - -!!! warning "ProximalSEM is still WIP" - The ProximalSEM package does not have any releases yet, and is not well tested - until the first release, use at your own risk and expect interfaces to change without prior notice. - -Additionally, you need to install and load `ProximalOperators.jl`: - -```@setup reg -using ProximalOperators +using StructuralEquationModels, ProximalAlgorithms, ProximalOperators ``` ```julia +using Pkg +Pkg.add("ProximalAlgorithms") Pkg.add("ProximalOperators") -using ProximalOperators +using StructuralEquationModels, ProximalAlgorithms, ProximalOperators ``` ## `SemOptimizerProximal` -`ProximalSEM` provides a new "building block" for the optimizer part of a model, called `SemOptimizerProximal`. +To estimate regularized models, we provide a "building block" for the optimizer part, called `SemOptimizerProximal`. It connects our package to the [`ProximalAlgorithms.jl`](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl) optimization backend, providing so-called proximal optimization algorithms. Those can handle, amongst other things, various forms of regularization. @@ -102,7 +85,9 @@ model = Sem( We labeled the covariances between the items because we want to regularize those: ```@example reg -ind = param_indices([:cov_15, :cov_24, :cov_26, :cov_37, :cov_48, :cov_68], model) +ind = getindex.( + [param_indices(model)], + [:cov_15, :cov_24, :cov_26, :cov_37, :cov_48, :cov_68]) ``` In the following, we fit the same model with lasso regularization of those covariances. @@ -127,8 +112,7 @@ optimizer_lasso = SemOptimizerProximal( model_lasso = Sem( specification = partable, - data = data, - optimizer = optimizer_lasso + data = data ) ``` @@ -136,7 +120,7 @@ Let's fit the regularized model ```@example reg -fit_lasso = sem_fit(model_lasso) +fit_lasso = sem_fit(optimizer_lasso, model_lasso) ``` and compare the solution to unregularizted estimates: @@ -151,6 +135,12 @@ update_partable!(partable, :estimate_lasso, params(fit_lasso), solution(fit_lass details(partable) ``` +Instead of explicitely defining a `SemOptimizerProximal` object, you can also pass `engine = :Proximal` and additional keyword arguments to `sem_fit`: + +```@example reg +fit = sem_fit(model; engine = :Proximal, operator_g = NormL1(λ)) +``` + ## Second example - mixed l1 and l0 regularization You can choose to penalize different parameters with different types of regularization functions. @@ -165,16 +155,14 @@ To define a sup of separable proximal operators (i.e. no parameter is penalized we can use [`SlicedSeparableSum`](https://juliafirstorder.github.io/ProximalOperators.jl/stable/calculus/#ProximalOperators.SlicedSeparableSum) from the `ProximalOperators` package: ```@example reg -prox_operator = SlicedSeparableSum((NormL1(0.02), NormL0(20.0), NormL0(0.0)), ([ind], [12:22], [vcat(1:11, 23:25)])) +prox_operator = SlicedSeparableSum((NormL0(20.0), NormL1(0.02), NormL0(0.0)), ([ind], [9:11], [vcat(1:8, 12:25)])) model_mixed = Sem( specification = partable, - data = data, - optimizer = SemOptimizerProximal, - operator_g = prox_operator + data = data, ) -fit_mixed = sem_fit(model_mixed) +fit_mixed = sem_fit(model_mixed; engine = :Proximal, operator_g = prox_operator) ``` Let's again compare the different results: diff --git a/ext/SEMNLOptExt/NLopt.jl b/ext/SEMNLOptExt/NLopt.jl index 95938029..a614c501 100644 --- a/ext/SEMNLOptExt/NLopt.jl +++ b/ext/SEMNLOptExt/NLopt.jl @@ -1,75 +1,3 @@ -############################################################################################ -### Types -############################################################################################ -""" -Connects to `NLopt.jl` as the optimization backend. - -# Constructor - - SemOptimizerNLopt(; - algorithm = :LD_LBFGS, - options = Dict{Symbol, Any}(), - local_algorithm = nothing, - local_options = Dict{Symbol, Any}(), - equality_constraints = Vector{NLoptConstraint}(), - inequality_constraints = Vector{NLoptConstraint}(), - kwargs...) - -# Arguments -- `algorithm`: optimization algorithm. -- `options::Dict{Symbol, Any}`: options for the optimization algorithm -- `local_algorithm`: local optimization algorithm -- `local_options::Dict{Symbol, Any}`: options for the local optimization algorithm -- `equality_constraints::Vector{NLoptConstraint}`: vector of equality constraints -- `inequality_constraints::Vector{NLoptConstraint}`: vector of inequality constraints - -# Example -```julia -my_optimizer = SemOptimizerNLopt() - -# constrained optimization with augmented lagrangian -my_constrained_optimizer = SemOptimizerNLopt(; - algorithm = :AUGLAG, - local_algorithm = :LD_LBFGS, - local_options = Dict(:ftol_rel => 1e-6), - inequality_constraints = NLoptConstraint(;f = my_constraint, tol = 0.0), -) -``` - -# Usage -All algorithms and options from the NLopt library are available, for more information see -the NLopt.jl package and the NLopt online documentation. -For information on how to use inequality and equality constraints, -see [Constrained optimization](@ref) in our online documentation. - -# Extended help - -## Interfaces -- `algorithm(::SemOptimizerNLopt)` -- `local_algorithm(::SemOptimizerNLopt)` -- `options(::SemOptimizerNLopt)` -- `local_options(::SemOptimizerNLopt)` -- `equality_constraints(::SemOptimizerNLopt)` -- `inequality_constraints(::SemOptimizerNLopt)` - -## Implementation - -Subtype of `SemOptimizer`. -""" -struct SemOptimizerNLopt{A, A2, B, B2, C} <: SemOptimizer{:NLopt} - algorithm::A - local_algorithm::A2 - options::B - local_options::B2 - equality_constraints::C - inequality_constraints::C -end - -Base.@kwdef struct NLoptConstraint - f::Any - tol = 0.0 -end - Base.convert( ::Type{NLoptConstraint}, tuple::NamedTuple{(:f, :tol), Tuple{F, T}}, diff --git a/ext/SEMNLOptExt/SEMNLOptExt.jl b/ext/SEMNLOptExt/SEMNLOptExt.jl index a159f6dc..c79fc2b8 100644 --- a/ext/SEMNLOptExt/SEMNLOptExt.jl +++ b/ext/SEMNLOptExt/SEMNLOptExt.jl @@ -1,11 +1,10 @@ module SEMNLOptExt using StructuralEquationModels, NLopt +using StructuralEquationModels: SemOptimizerNLopt, NLoptConstraint SEM = StructuralEquationModels -export SemOptimizerNLopt, NLoptConstraint - include("NLopt.jl") end diff --git a/ext/SEMProximalOptExt/ProximalAlgorithms.jl b/ext/SEMProximalOptExt/ProximalAlgorithms.jl index eceff0dc..2f1775e8 100644 --- a/ext/SEMProximalOptExt/ProximalAlgorithms.jl +++ b/ext/SEMProximalOptExt/ProximalAlgorithms.jl @@ -1,28 +1,3 @@ -############################################################################################ -### Types -############################################################################################ -""" -Connects to `ProximalAlgorithms.jl` as the optimization backend. - -# Constructor - - SemOptimizerProximal(; - algorithm = ProximalAlgorithms.PANOC(), - operator_g, - operator_h = nothing, - kwargs..., - -# Arguments -- `algorithm`: optimization algorithm. -- `operator_g`: gradient of the objective function -- `operator_h`: optional hessian of the objective function -""" -mutable struct SemOptimizerProximal{A, B, C} <: SemOptimizer{:Proximal} - algorithm::A - operator_g::B - operator_h::C -end - SEM.SemOptimizer{:Proximal}(args...; kwargs...) = SemOptimizerProximal(args...; kwargs...) SemOptimizerProximal(; diff --git a/ext/SEMProximalOptExt/SEMProximalOptExt.jl b/ext/SEMProximalOptExt/SEMProximalOptExt.jl index 15631136..192944fe 100644 --- a/ext/SEMProximalOptExt/SEMProximalOptExt.jl +++ b/ext/SEMProximalOptExt/SEMProximalOptExt.jl @@ -2,8 +2,7 @@ module SEMProximalOptExt using StructuralEquationModels using ProximalAlgorithms - -export SemOptimizerProximal +using StructuralEquationModels: SemOptimizerProximal, print_type_name, print_field_types SEM = StructuralEquationModels diff --git a/src/StructuralEquationModels.jl b/src/StructuralEquationModels.jl index 7c8923dc..5d6b23ef 100644 --- a/src/StructuralEquationModels.jl +++ b/src/StructuralEquationModels.jl @@ -82,6 +82,10 @@ include("frontend/fit/fitmeasures/fit_measures.jl") # standard errors include("frontend/fit/standard_errors/hessian.jl") include("frontend/fit/standard_errors/bootstrap.jl") +# extensions +include("package_extensions/SEMNLOptExt.jl") +include("package_extensions/SEMProximalOptExt.jl") + export AbstractSem, AbstractSemSingle, @@ -183,5 +187,8 @@ export AbstractSem, →, ←, ↔, - ⇔ + ⇔, + SemOptimizerNLopt, + NLoptConstraint, + SemOptimizerProximal end diff --git a/src/package_extensions/SEMNLOptExt.jl b/src/package_extensions/SEMNLOptExt.jl new file mode 100644 index 00000000..7eae2f26 --- /dev/null +++ b/src/package_extensions/SEMNLOptExt.jl @@ -0,0 +1,69 @@ +""" +Connects to `NLopt.jl` as the optimization backend. +Only usable if `NLopt.jl` is loaded in the current Julia session! + +# Constructor + + SemOptimizerNLopt(; + algorithm = :LD_LBFGS, + options = Dict{Symbol, Any}(), + local_algorithm = nothing, + local_options = Dict{Symbol, Any}(), + equality_constraints = Vector{NLoptConstraint}(), + inequality_constraints = Vector{NLoptConstraint}(), + kwargs...) + +# Arguments +- `algorithm`: optimization algorithm. +- `options::Dict{Symbol, Any}`: options for the optimization algorithm +- `local_algorithm`: local optimization algorithm +- `local_options::Dict{Symbol, Any}`: options for the local optimization algorithm +- `equality_constraints::Vector{NLoptConstraint}`: vector of equality constraints +- `inequality_constraints::Vector{NLoptConstraint}`: vector of inequality constraints + +# Example +```julia +my_optimizer = SemOptimizerNLopt() + +# constrained optimization with augmented lagrangian +my_constrained_optimizer = SemOptimizerNLopt(; + algorithm = :AUGLAG, + local_algorithm = :LD_LBFGS, + local_options = Dict(:ftol_rel => 1e-6), + inequality_constraints = NLoptConstraint(;f = my_constraint, tol = 0.0), +) +``` + +# Usage +All algorithms and options from the NLopt library are available, for more information see +the NLopt.jl package and the NLopt online documentation. +For information on how to use inequality and equality constraints, +see [Constrained optimization](@ref) in our online documentation. + +# Extended help + +## Interfaces +- `algorithm(::SemOptimizerNLopt)` +- `local_algorithm(::SemOptimizerNLopt)` +- `options(::SemOptimizerNLopt)` +- `local_options(::SemOptimizerNLopt)` +- `equality_constraints(::SemOptimizerNLopt)` +- `inequality_constraints(::SemOptimizerNLopt)` + +## Implementation + +Subtype of `SemOptimizer`. +""" +struct SemOptimizerNLopt{A, A2, B, B2, C} <: SemOptimizer{:NLopt} + algorithm::A + local_algorithm::A2 + options::B + local_options::B2 + equality_constraints::C + inequality_constraints::C +end + +Base.@kwdef struct NLoptConstraint + f::Any + tol = 0.0 +end \ No newline at end of file diff --git a/src/package_extensions/SEMProximalOptExt.jl b/src/package_extensions/SEMProximalOptExt.jl new file mode 100644 index 00000000..e8b25670 --- /dev/null +++ b/src/package_extensions/SEMProximalOptExt.jl @@ -0,0 +1,21 @@ +""" +Connects to `ProximalAlgorithms.jl` as the optimization backend. + +# Constructor + + SemOptimizerProximal(; + algorithm = ProximalAlgorithms.PANOC(), + operator_g, + operator_h = nothing, + kwargs..., + +# Arguments +- `algorithm`: optimization algorithm. +- `operator_g`: gradient of the objective function +- `operator_h`: optional hessian of the objective function +""" +mutable struct SemOptimizerProximal{A, B, C} <: SemOptimizer{:Proximal} + algorithm::A + operator_g::B + operator_h::C +end \ No newline at end of file