Fix extensions

docs/src/tutorials/regularization/regularization.md

@@ -5,40 +5,23 @@
 For ridge regularization, you can simply use `SemRidge` as an additional loss function 
 (for example, a model with the loss functions `SemML` and `SemRidge` corresponds to ridge-regularized maximum likelihood estimation).
 
-For lasso, elastic net and (far) beyond, we provide the `ProximalSEM` package. You can install it and load it alongside `StructuralEquationModels`:
+For lasso, elastic net and (far) beyond, you can load the `ProximalAlgorithms.jl` and `ProximalOperators.jl` packages alongside `StructuralEquationModels`:
 
 ```@setup reg
-import Pkg
-Pkg.add(url = "https://github.com/StructuralEquationModels/ProximalSEM.jl")
-
-using StructuralEquationModels, ProximalSEM
-```
-
-```julia
-import Pkg
-Pkg.add(url = "https://github.com/StructuralEquationModels/ProximalSEM.jl")
-
-using StructuralEquationModels, ProximalSEM
-```
-
-!!! warning "ProximalSEM is still WIP"
-    The ProximalSEM package does not have any releases yet, and is not well tested - until the first release, use at your own risk and expect interfaces to change without prior notice.
-
-Additionally, you need to install and load `ProximalOperators.jl`:
-
-```@setup reg
-using ProximalOperators
+using StructuralEquationModels, ProximalAlgorithms, ProximalOperators
 ```
 
 ```julia
+using Pkg
+Pkg.add("ProximalAlgorithms")
 Pkg.add("ProximalOperators")
 
-using ProximalOperators
+using StructuralEquationModels, ProximalAlgorithms, ProximalOperators
 ```
 
 ## `SemOptimizerProximal`
 
-`ProximalSEM` provides a new "building block" for the optimizer part of a model, called `SemOptimizerProximal`.
+To estimate regularized models, we provide a "building block" for the optimizer part, called `SemOptimizerProximal`.
 It connects our package to the [`ProximalAlgorithms.jl`](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl) optimization backend, providing so-called proximal optimization algorithms. 
 Those can handle, amongst other things, various forms of regularization.
 
@@ -102,7 +85,9 @@ model = Sem(
 We labeled the covariances between the items because we want to regularize those:
 
 ```@example reg
-ind = param_indices([:cov_15, :cov_24, :cov_26, :cov_37, :cov_48, :cov_68], model)
+ind = getindex.(
+    [param_indices(model)], 
+    [:cov_15, :cov_24, :cov_26, :cov_37, :cov_48, :cov_68])
 ```
 
 In the following, we fit the same model with lasso regularization of those covariances.
@@ -127,16 +112,15 @@ optimizer_lasso = SemOptimizerProximal(
 
 model_lasso = Sem(
     specification = partable,
-    data = data,
-    optimizer = optimizer_lasso
+    data = data
 )
 ```
 
 Let's fit the regularized model
 
 ```@example reg
 
-fit_lasso = sem_fit(model_lasso)
+fit_lasso = sem_fit(optimizer_lasso, model_lasso)
 ```
 
 and compare the solution to unregularizted estimates:
@@ -151,6 +135,12 @@ update_partable!(partable, :estimate_lasso, params(fit_lasso), solution(fit_lass
 details(partable)
 ```
 
+Instead of explicitely defining a `SemOptimizerProximal` object, you can also pass `engine = :Proximal` and additional keyword arguments to `sem_fit`:
+
+```@example reg
+fit = sem_fit(model; engine = :Proximal, operator_g = NormL1(λ))
+```
+
 ## Second example - mixed l1 and l0 regularization
 
 You can choose to penalize different parameters with different types of regularization functions.
@@ -165,16 +155,14 @@ To define a sup of separable proximal operators (i.e. no parameter is penalized
 we can use [`SlicedSeparableSum`](https://juliafirstorder.github.io/ProximalOperators.jl/stable/calculus/#ProximalOperators.SlicedSeparableSum) from the `ProximalOperators` package:
 
 ```@example reg
-prox_operator = SlicedSeparableSum((NormL1(0.02), NormL0(20.0), NormL0(0.0)), ([ind], [12:22], [vcat(1:11, 23:25)]))
+prox_operator = SlicedSeparableSum((NormL0(20.0), NormL1(0.02), NormL0(0.0)), ([ind], [9:11], [vcat(1:8, 12:25)]))
 
 model_mixed = Sem(
     specification = partable,
-    data = data,
-    optimizer = SemOptimizerProximal,
-    operator_g = prox_operator
+    data = data,    
 )
 
-fit_mixed = sem_fit(model_mixed)
+fit_mixed = sem_fit(model_mixed; engine = :Proximal, operator_g = prox_operator)
 ```
 
 Let's again compare the different results:

ext/SEMNLOptExt/NLopt.jl

@@ -1,75 +1,3 @@
-############################################################################################
-### Types
-############################################################################################
-"""
-Connects to `NLopt.jl` as the optimization backend.
-
-# Constructor
-
-    SemOptimizerNLopt(;
-        algorithm = :LD_LBFGS,
-        options = Dict{Symbol, Any}(),
-        local_algorithm = nothing,
-        local_options = Dict{Symbol, Any}(),
-        equality_constraints = Vector{NLoptConstraint}(),
-        inequality_constraints = Vector{NLoptConstraint}(),
-        kwargs...)
-
-# Arguments
-- `algorithm`: optimization algorithm.
-- `options::Dict{Symbol, Any}`: options for the optimization algorithm
-- `local_algorithm`: local optimization algorithm
-- `local_options::Dict{Symbol, Any}`: options for the local optimization algorithm
-- `equality_constraints::Vector{NLoptConstraint}`: vector of equality constraints
-- `inequality_constraints::Vector{NLoptConstraint}`: vector of inequality constraints
-
-# Example
-```julia
-my_optimizer = SemOptimizerNLopt()
-
-# constrained optimization with augmented lagrangian
-my_constrained_optimizer = SemOptimizerNLopt(;
-    algorithm = :AUGLAG,
-    local_algorithm = :LD_LBFGS,
-    local_options = Dict(:ftol_rel => 1e-6),
-    inequality_constraints = NLoptConstraint(;f = my_constraint, tol = 0.0),
-)
-```
-
-# Usage
-All algorithms and options from the NLopt library are available, for more information see
-the NLopt.jl package and the NLopt online documentation.
-For information on how to use inequality and equality constraints,
-see [Constrained optimization](@ref) in our online documentation.
-
-# Extended help
-
-## Interfaces
-- `algorithm(::SemOptimizerNLopt)`
-- `local_algorithm(::SemOptimizerNLopt)`
-- `options(::SemOptimizerNLopt)`
-- `local_options(::SemOptimizerNLopt)`
-- `equality_constraints(::SemOptimizerNLopt)`
-- `inequality_constraints(::SemOptimizerNLopt)`
-
-## Implementation
-
-Subtype of `SemOptimizer`.
-"""
-struct SemOptimizerNLopt{A, A2, B, B2, C} <: SemOptimizer{:NLopt}
-    algorithm::A
-    local_algorithm::A2
-    options::B
-    local_options::B2
-    equality_constraints::C
-    inequality_constraints::C
-end
-
-Base.@kwdef struct NLoptConstraint
-    f::Any
-    tol = 0.0
-end
-
 Base.convert(
     ::Type{NLoptConstraint},
     tuple::NamedTuple{(:f, :tol), Tuple{F, T}},

ext/SEMNLOptExt/SEMNLOptExt.jl

@@ -1,11 +1,10 @@
 module SEMNLOptExt
 
 using StructuralEquationModels, NLopt
+using StructuralEquationModels: SemOptimizerNLopt, NLoptConstraint
 
 SEM = StructuralEquationModels
 
-export SemOptimizerNLopt, NLoptConstraint
-
 include("NLopt.jl")
 
 end

ext/SEMProximalOptExt/ProximalAlgorithms.jl

@@ -1,28 +1,3 @@
-############################################################################################
-### Types
-############################################################################################
-"""
-Connects to `ProximalAlgorithms.jl` as the optimization backend.
-
-# Constructor
-
-    SemOptimizerProximal(;
-        algorithm = ProximalAlgorithms.PANOC(),
-        operator_g,
-        operator_h = nothing,
-        kwargs...,
-
-# Arguments
-- `algorithm`: optimization algorithm.
-- `operator_g`: gradient of the objective function
-- `operator_h`: optional hessian of the objective function
-"""
-mutable struct SemOptimizerProximal{A, B, C} <: SemOptimizer{:Proximal}
-    algorithm::A
-    operator_g::B
-    operator_h::C
-end
-
 SEM.SemOptimizer{:Proximal}(args...; kwargs...) = SemOptimizerProximal(args...; kwargs...)
 
 SemOptimizerProximal(;

ext/SEMProximalOptExt/SEMProximalOptExt.jl

@@ -2,8 +2,7 @@ module SEMProximalOptExt
 
 using StructuralEquationModels
 using ProximalAlgorithms
-
-export SemOptimizerProximal
+using StructuralEquationModels: SemOptimizerProximal, print_type_name, print_field_types
 
 SEM = StructuralEquationModels
 

src/StructuralEquationModels.jl

@@ -82,6 +82,10 @@ include("frontend/fit/fitmeasures/fit_measures.jl")
 # standard errors
 include("frontend/fit/standard_errors/hessian.jl")
 include("frontend/fit/standard_errors/bootstrap.jl")
+# extensions
+include("package_extensions/SEMNLOptExt.jl")
+include("package_extensions/SEMProximalOptExt.jl")
+
 
 export AbstractSem,
     AbstractSemSingle,
@@ -183,5 +187,8 @@ export AbstractSem,
     →,
     ←,
     ↔,
-    ⇔
+    ⇔,
+    SemOptimizerNLopt,
+    NLoptConstraint,
+    SemOptimizerProximal
 end

src/package_extensions/SEMNLOptExt.jl

@@ -0,0 +1,69 @@
+"""
+Connects to `NLopt.jl` as the optimization backend.
+Only usable if `NLopt.jl` is loaded in the current Julia session!
+
+# Constructor
+
+    SemOptimizerNLopt(;
+        algorithm = :LD_LBFGS,
+        options = Dict{Symbol, Any}(),
+        local_algorithm = nothing,
+        local_options = Dict{Symbol, Any}(),
+        equality_constraints = Vector{NLoptConstraint}(),
+        inequality_constraints = Vector{NLoptConstraint}(),
+        kwargs...)
+
+# Arguments
+- `algorithm`: optimization algorithm.
+- `options::Dict{Symbol, Any}`: options for the optimization algorithm
+- `local_algorithm`: local optimization algorithm
+- `local_options::Dict{Symbol, Any}`: options for the local optimization algorithm
+- `equality_constraints::Vector{NLoptConstraint}`: vector of equality constraints
+- `inequality_constraints::Vector{NLoptConstraint}`: vector of inequality constraints
+
+# Example
+```julia
+my_optimizer = SemOptimizerNLopt()
+
+# constrained optimization with augmented lagrangian
+my_constrained_optimizer = SemOptimizerNLopt(;
+    algorithm = :AUGLAG,
+    local_algorithm = :LD_LBFGS,
+    local_options = Dict(:ftol_rel => 1e-6),
+    inequality_constraints = NLoptConstraint(;f = my_constraint, tol = 0.0),
+)
+```
+
+# Usage
+All algorithms and options from the NLopt library are available, for more information see
+the NLopt.jl package and the NLopt online documentation.
+For information on how to use inequality and equality constraints,
+see [Constrained optimization](@ref) in our online documentation.
+
+# Extended help
+
+## Interfaces
+- `algorithm(::SemOptimizerNLopt)`
+- `local_algorithm(::SemOptimizerNLopt)`
+- `options(::SemOptimizerNLopt)`
+- `local_options(::SemOptimizerNLopt)`
+- `equality_constraints(::SemOptimizerNLopt)`
+- `inequality_constraints(::SemOptimizerNLopt)`
+
+## Implementation
+
+Subtype of `SemOptimizer`.
+"""
+struct SemOptimizerNLopt{A, A2, B, B2, C} <: SemOptimizer{:NLopt}
+    algorithm::A
+    local_algorithm::A2
+    options::B
+    local_options::B2
+    equality_constraints::C
+    inequality_constraints::C
+end
+
+Base.@kwdef struct NLoptConstraint
+    f::Any
+    tol = 0.0
+end

src/package_extensions/SEMProximalOptExt.jl

@@ -0,0 +1,21 @@
+"""
+Connects to `ProximalAlgorithms.jl` as the optimization backend.
+
+# Constructor
+
+    SemOptimizerProximal(;
+        algorithm = ProximalAlgorithms.PANOC(),
+        operator_g,
+        operator_h = nothing,
+        kwargs...,
+
+# Arguments
+- `algorithm`: optimization algorithm.
+- `operator_g`: gradient of the objective function
+- `operator_h`: optional hessian of the objective function
+"""
+mutable struct SemOptimizerProximal{A, B, C} <: SemOptimizer{:Proximal}
+    algorithm::A
+    operator_g::B
+    operator_h::C
+end