Elastic Net Regression

This example demonstrates solving an Elastic Net regression problem, which combines L1 and L2 regularization:

\[\operatorname{minimize}_{\beta} \frac{1}{2}\|X\beta - y\|_2^2 + \lambda_1 \|\beta\|_1 + \lambda_2 \|\beta\|_2^2\]

Problem Setup

import torch
from rlaopt.expression import Variable, Constant
from rlaopt.atoms import ElasticNet, SumSquares
from rlaopt.solvers import ProxGrad, ProxGradConfig

# Generate synthetic data
n_samples, n_features = 100, 50
X = torch.randn(n_samples, n_features)
y = torch.randn(n_samples)

# Create optimization variable
beta = Variable((n_features,), name='beta')

# X and y are data tensors, not variables
X_const = Constant(X)
y_const = Constant(y)

Building the Objective

We can use the ElasticNet atom directly:

# Regularization parameters
lambda_l1 = 0.1
lambda_l2 = 0.01

# Build objective
residual = X_const @ beta - y_const
data_fit = SumSquares(residual)
regularization = ElasticNet(beta, l1_scaling=lambda_l1, l2_scaling=lambda_l2)
objective = data_fit + regularization

Solving

config = ProxGradConfig(eta=0.01, use_linesearch=True, max_iters=1000, tol=1e-4)
solver = ProxGrad(objective, config)
result = solver.solve()

print(f"Solution: {result.variable_values}")
print(f"Final error: {result.err}")