Skip to content

Commit bcfae6a

Browse files
Maximilian-Stefan-ErnstMaximilian-Stefan-Ernst
authored
Merge pull request #246 from StructuralEquationModels/fix_extensions
Fix extensions
2 parents 9244edf + f67d48c commit bcfae6a

File tree

8 files changed

+120
-134
lines changed

8 files changed

+120
-134
lines changed

‎docs/src/tutorials/regularization/regularization.md‎

Lines changed: 20 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -5,40 +5,23 @@
55
For ridge regularization, you can simply use `SemRidge` as an additional loss function
66
(for example, a model with the loss functions `SemML` and `SemRidge` corresponds to ridge-regularized maximum likelihood estimation).
77

8-
For lasso, elastic net and (far) beyond, we provide the `ProximalSEM` package. You can install it and load it alongside `StructuralEquationModels`:
8+
For lasso, elastic net and (far) beyond, you can load the `ProximalAlgorithms.jl` and `ProximalOperators.jl` packages alongside `StructuralEquationModels`:
99

1010
```@setup reg
11-
import Pkg
12-
Pkg.add(url = "https://github.com/StructuralEquationModels/ProximalSEM.jl")
13-
14-
using StructuralEquationModels, ProximalSEM
15-
```
16-
17-
```julia
18-
import Pkg
19-
Pkg.add(url = "https://github.com/StructuralEquationModels/ProximalSEM.jl")
20-
21-
using StructuralEquationModels, ProximalSEM
22-
```
23-
24-
!!! warning "ProximalSEM is still WIP"
25-
The ProximalSEM package does not have any releases yet, and is not well tested - until the first release, use at your own risk and expect interfaces to change without prior notice.
26-
27-
Additionally, you need to install and load `ProximalOperators.jl`:
28-
29-
```@setup reg
30-
using ProximalOperators
11+
using StructuralEquationModels, ProximalAlgorithms, ProximalOperators
3112
```
3213

3314
```julia
15+
using Pkg
16+
Pkg.add("ProximalAlgorithms")
3417
Pkg.add("ProximalOperators")
3518

36-
using ProximalOperators
19+
using StructuralEquationModels, ProximalAlgorithms, ProximalOperators
3720
```
3821

3922
## `SemOptimizerProximal`
4023

41-
`ProximalSEM` provides a new "building block" for the optimizer part of a model, called `SemOptimizerProximal`.
24+
To estimate regularized models, we provide a "building block" for the optimizer part, called `SemOptimizerProximal`.
4225
It connects our package to the [`ProximalAlgorithms.jl`](https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl) optimization backend, providing so-called proximal optimization algorithms.
4326
Those can handle, amongst other things, various forms of regularization.
4427

@@ -102,7 +85,9 @@ model = Sem(
10285
We labeled the covariances between the items because we want to regularize those:
10386

10487
```@example reg
105-
ind = param_indices([:cov_15, :cov_24, :cov_26, :cov_37, :cov_48, :cov_68], model)
88+
ind = getindex.(
89+
[param_indices(model)],
90+
[:cov_15, :cov_24, :cov_26, :cov_37, :cov_48, :cov_68])
10691
```
10792

10893
In the following, we fit the same model with lasso regularization of those covariances.
@@ -127,16 +112,15 @@ optimizer_lasso = SemOptimizerProximal(
127112
128113
model_lasso = Sem(
129114
specification = partable,
130-
data = data,
131-
optimizer = optimizer_lasso
115+
data = data
132116
)
133117
```
134118

135119
Let's fit the regularized model
136120

137121
```@example reg
138122
139-
fit_lasso = sem_fit(model_lasso)
123+
fit_lasso = sem_fit(optimizer_lasso, model_lasso)
140124
```
141125

142126
and compare the solution to unregularizted estimates:
@@ -151,6 +135,12 @@ update_partable!(partable, :estimate_lasso, params(fit_lasso), solution(fit_lass
151135
details(partable)
152136
```
153137

138+
Instead of explicitely defining a `SemOptimizerProximal` object, you can also pass `engine = :Proximal` and additional keyword arguments to `sem_fit`:
139+
140+
```@example reg
141+
fit = sem_fit(model; engine = :Proximal, operator_g = NormL1(λ))
142+
```
143+
154144
## Second example - mixed l1 and l0 regularization
155145

156146
You can choose to penalize different parameters with different types of regularization functions.
@@ -165,16 +155,14 @@ To define a sup of separable proximal operators (i.e. no parameter is penalized
165155
we can use [`SlicedSeparableSum`](https://juliafirstorder.github.io/ProximalOperators.jl/stable/calculus/#ProximalOperators.SlicedSeparableSum) from the `ProximalOperators` package:
166156

167157
```@example reg
168-
prox_operator = SlicedSeparableSum((NormL1(0.02), NormL0(20.0), NormL0(0.0)), ([ind], [12:22], [vcat(1:11, 23:25)]))
158+
prox_operator = SlicedSeparableSum((NormL0(20.0), NormL1(0.02), NormL0(0.0)), ([ind], [9:11], [vcat(1:8, 12:25)]))
169159
170160
model_mixed = Sem(
171161
specification = partable,
172-
data = data,
173-
optimizer = SemOptimizerProximal,
174-
operator_g = prox_operator
162+
data = data,
175163
)
176164
177-
fit_mixed = sem_fit(model_mixed)
165+
fit_mixed = sem_fit(model_mixed; engine = :Proximal, operator_g = prox_operator)
178166
```
179167

180168
Let's again compare the different results:

‎ext/SEMNLOptExt/NLopt.jl‎

Lines changed: 0 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -1,75 +1,3 @@
1-
############################################################################################
2-
### Types
3-
############################################################################################
4-
"""
5-
Connects to `NLopt.jl` as the optimization backend.
6-
7-
# Constructor
8-
9-
SemOptimizerNLopt(;
10-
algorithm = :LD_LBFGS,
11-
options = Dict{Symbol, Any}(),
12-
local_algorithm = nothing,
13-
local_options = Dict{Symbol, Any}(),
14-
equality_constraints = Vector{NLoptConstraint}(),
15-
inequality_constraints = Vector{NLoptConstraint}(),
16-
kwargs...)
17-
18-
# Arguments
19-
- `algorithm`: optimization algorithm.
20-
- `options::Dict{Symbol, Any}`: options for the optimization algorithm
21-
- `local_algorithm`: local optimization algorithm
22-
- `local_options::Dict{Symbol, Any}`: options for the local optimization algorithm
23-
- `equality_constraints::Vector{NLoptConstraint}`: vector of equality constraints
24-
- `inequality_constraints::Vector{NLoptConstraint}`: vector of inequality constraints
25-
26-
# Example
27-
```julia
28-
my_optimizer = SemOptimizerNLopt()
29-
30-
# constrained optimization with augmented lagrangian
31-
my_constrained_optimizer = SemOptimizerNLopt(;
32-
algorithm = :AUGLAG,
33-
local_algorithm = :LD_LBFGS,
34-
local_options = Dict(:ftol_rel => 1e-6),
35-
inequality_constraints = NLoptConstraint(;f = my_constraint, tol = 0.0),
36-
)
37-
```
38-
39-
# Usage
40-
All algorithms and options from the NLopt library are available, for more information see
41-
the NLopt.jl package and the NLopt online documentation.
42-
For information on how to use inequality and equality constraints,
43-
see [Constrained optimization](@ref) in our online documentation.
44-
45-
# Extended help
46-
47-
## Interfaces
48-
- `algorithm(::SemOptimizerNLopt)`
49-
- `local_algorithm(::SemOptimizerNLopt)`
50-
- `options(::SemOptimizerNLopt)`
51-
- `local_options(::SemOptimizerNLopt)`
52-
- `equality_constraints(::SemOptimizerNLopt)`
53-
- `inequality_constraints(::SemOptimizerNLopt)`
54-
55-
## Implementation
56-
57-
Subtype of `SemOptimizer`.
58-
"""
59-
struct SemOptimizerNLopt{A, A2, B, B2, C} <: SemOptimizer{:NLopt}
60-
algorithm::A
61-
local_algorithm::A2
62-
options::B
63-
local_options::B2
64-
equality_constraints::C
65-
inequality_constraints::C
66-
end
67-
68-
Base.@kwdef struct NLoptConstraint
69-
f::Any
70-
tol = 0.0
71-
end
72-
731
Base.convert(
742
::Type{NLoptConstraint},
753
tuple::NamedTuple{(:f, :tol), Tuple{F, T}},

‎ext/SEMNLOptExt/SEMNLOptExt.jl‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,10 @@
11
module SEMNLOptExt
22

33
using StructuralEquationModels, NLopt
4+
using StructuralEquationModels: SemOptimizerNLopt, NLoptConstraint
45

56
SEM = StructuralEquationModels
67

7-
export SemOptimizerNLopt, NLoptConstraint
8-
98
include("NLopt.jl")
109

1110
end

‎ext/SEMProximalOptExt/ProximalAlgorithms.jl‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,3 @@
1-
############################################################################################
2-
### Types
3-
############################################################################################
4-
"""
5-
Connects to `ProximalAlgorithms.jl` as the optimization backend.
6-
7-
# Constructor
8-
9-
SemOptimizerProximal(;
10-
algorithm = ProximalAlgorithms.PANOC(),
11-
operator_g,
12-
operator_h = nothing,
13-
kwargs...,
14-
15-
# Arguments
16-
- `algorithm`: optimization algorithm.
17-
- `operator_g`: gradient of the objective function
18-
- `operator_h`: optional hessian of the objective function
19-
"""
20-
mutable struct SemOptimizerProximal{A, B, C} <: SemOptimizer{:Proximal}
21-
algorithm::A
22-
operator_g::B
23-
operator_h::C
24-
end
25-
261
SEM.SemOptimizer{:Proximal}(args...; kwargs...) = SemOptimizerProximal(args...; kwargs...)
272

283
SemOptimizerProximal(;

‎ext/SEMProximalOptExt/SEMProximalOptExt.jl‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,7 @@ module SEMProximalOptExt
22

33
using StructuralEquationModels
44
using ProximalAlgorithms
5-
6-
export SemOptimizerProximal
5+
using StructuralEquationModels: SemOptimizerProximal, print_type_name, print_field_types
76

87
SEM = StructuralEquationModels
98

‎src/StructuralEquationModels.jl‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,10 @@ include("frontend/fit/fitmeasures/fit_measures.jl")
8282
# standard errors
8383
include("frontend/fit/standard_errors/hessian.jl")
8484
include("frontend/fit/standard_errors/bootstrap.jl")
85+
# extensions
86+
include("package_extensions/SEMNLOptExt.jl")
87+
include("package_extensions/SEMProximalOptExt.jl")
88+
8589

8690
export AbstractSem,
8791
AbstractSemSingle,
@@ -183,5 +187,8 @@ export AbstractSem,
183187
→,
184188
←,
185189
↔,
186-
⇔
190+
⇔,
191+
SemOptimizerNLopt,
192+
NLoptConstraint,
193+
SemOptimizerProximal
187194
end

‎src/package_extensions/SEMNLOptExt.jl‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
"""
2+
Connects to `NLopt.jl` as the optimization backend.
3+
Only usable if `NLopt.jl` is loaded in the current Julia session!
4+
5+
# Constructor
6+
7+
SemOptimizerNLopt(;
8+
algorithm = :LD_LBFGS,
9+
options = Dict{Symbol, Any}(),
10+
local_algorithm = nothing,
11+
local_options = Dict{Symbol, Any}(),
12+
equality_constraints = Vector{NLoptConstraint}(),
13+
inequality_constraints = Vector{NLoptConstraint}(),
14+
kwargs...)
15+
16+
# Arguments
17+
- `algorithm`: optimization algorithm.
18+
- `options::Dict{Symbol, Any}`: options for the optimization algorithm
19+
- `local_algorithm`: local optimization algorithm
20+
- `local_options::Dict{Symbol, Any}`: options for the local optimization algorithm
21+
- `equality_constraints::Vector{NLoptConstraint}`: vector of equality constraints
22+
- `inequality_constraints::Vector{NLoptConstraint}`: vector of inequality constraints
23+
24+
# Example
25+
```julia
26+
my_optimizer = SemOptimizerNLopt()
27+
28+
# constrained optimization with augmented lagrangian
29+
my_constrained_optimizer = SemOptimizerNLopt(;
30+
algorithm = :AUGLAG,
31+
local_algorithm = :LD_LBFGS,
32+
local_options = Dict(:ftol_rel => 1e-6),
33+
inequality_constraints = NLoptConstraint(;f = my_constraint, tol = 0.0),
34+
)
35+
```
36+
37+
# Usage
38+
All algorithms and options from the NLopt library are available, for more information see
39+
the NLopt.jl package and the NLopt online documentation.
40+
For information on how to use inequality and equality constraints,
41+
see [Constrained optimization](@ref) in our online documentation.
42+
43+
# Extended help
44+
45+
## Interfaces
46+
- `algorithm(::SemOptimizerNLopt)`
47+
- `local_algorithm(::SemOptimizerNLopt)`
48+
- `options(::SemOptimizerNLopt)`
49+
- `local_options(::SemOptimizerNLopt)`
50+
- `equality_constraints(::SemOptimizerNLopt)`
51+
- `inequality_constraints(::SemOptimizerNLopt)`
52+
53+
## Implementation
54+
55+
Subtype of `SemOptimizer`.
56+
"""
57+
struct SemOptimizerNLopt{A, A2, B, B2, C} <: SemOptimizer{:NLopt}
58+
algorithm::A
59+
local_algorithm::A2
60+
options::B
61+
local_options::B2
62+
equality_constraints::C
63+
inequality_constraints::C
64+
end
65+
66+
Base.@kwdef struct NLoptConstraint
67+
f::Any
68+
tol = 0.0
69+
end

‎src/package_extensions/SEMProximalOptExt.jl‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
"""
2+
Connects to `ProximalAlgorithms.jl` as the optimization backend.
3+
4+
# Constructor
5+
6+
SemOptimizerProximal(;
7+
algorithm = ProximalAlgorithms.PANOC(),
8+
operator_g,
9+
operator_h = nothing,
10+
kwargs...,
11+
12+
# Arguments
13+
- `algorithm`: optimization algorithm.
14+
- `operator_g`: gradient of the objective function
15+
- `operator_h`: optional hessian of the objective function
16+
"""
17+
mutable struct SemOptimizerProximal{A, B, C} <: SemOptimizer{:Proximal}
18+
algorithm::A
19+
operator_g::B
20+
operator_h::C
21+
end

0 commit comments

Comments
 (0)