GPU Acceleration with DecisionRulesExa.jl

DecisionRulesExa.jl is a companion package that implements the same TS-DDR algorithm using ExaModels.jl instead of JuMP for the optimization backend. It targets GPU-accelerated training via MadNLP.jl with CUDSS-backed interior-point solves.

When to use DecisionRulesExa.jl

DecisionRules.jl (JuMP)DecisionRulesExa.jl (ExaModels)
BackendJuMP + DiffOptExaModels + MadNLP
HardwareCPUCPU or GPU (CUDA)
Training modesDE, stage-wise, multiple shootingDeterministic equivalent
Gradient sourceDiffOpt implicit diff + dualsEnvelope theorem (duals only)
Best forModerate NLPs, integer variables, stage-wise decompositionLarge NLPs (AC-OPF), GPU speedup, many samples per batch

Choose DecisionRulesExa.jl when the inner NLP is large enough that GPU acceleration matters (e.g., AC-OPF with hundreds of buses and thousands of variables per stage) and you want to run many training samples per gradient step on a single GPU.

Choose DecisionRules.jl when you need stage-wise or multiple-shooting decomposition, integer variable support, or DiffOpt-based solution sensitivities.

Installation

using Pkg
Pkg.add(url="https://github.com/LearningToOptimize/DecisionRulesExa.jl.git")

For GPU support, also install CUDA.jl and MadNLPGPU:

Pkg.add(["CUDA", "MadNLPGPU"])

Quick start: CPU

The simplest way to get started is with the built-in linear tracking problem:

using DecisionRulesExa
using ExaModels, Flux, MadNLP, Random

Random.seed!(1)

T  = 8   # horizon
nx = 1   # state dimension

# Build a parametric deterministic-equivalent NLP on CPU
prob = build_linear_tracking_problem(
    horizon       = T,
    nx            = nx,
    backend       = nothing,       # CPU
    slack_penalty = 10.0,
    u_bounds      = (-2.0, 2.0),
)

# LSTM policy: maps [w_t ; x_{t-1}] → target x̂_t at each stage
policy = StateConditionedPolicy(nx, nx, nx, [64, 64])

# Uncertainty sampler: returns a flat vector of length T * nw
sampler() = Float32.(0.1 .* randn(T * nx))

# Train with TS-DDR policy gradient (envelope theorem)
train_tsddr(
    policy,
    Float32.([1.0]),               # initial state
    prob,
    prob.p_x0,
    prob.p_target,
    prob.p_w,
    sampler;
    num_batches         = 100,
    num_train_per_batch = 4,
    optimizer           = Flux.Adam(1f-3),
    madnlp_kwargs       = (print_level = MadNLP.ERROR, tol = 1e-6),
)

Moving to GPU

To run the same problem on GPU, change the backend and add a GPU-native linear solver:

using CUDA, MadNLPGPU

prob_gpu = build_linear_tracking_problem(
    horizon       = T,
    nx            = nx,
    backend       = CUDABackend(),
    slack_penalty = 10.0,
    u_bounds      = (-2.0, 2.0),
)

train_tsddr(
    policy,
    Float32.([1.0]),
    prob_gpu,
    prob_gpu.p_x0,
    prob_gpu.p_target,
    prob_gpu.p_w,
    sampler;
    num_batches         = 100,
    num_train_per_batch = 4,
    optimizer           = Flux.Adam(1f-3),
    madnlp_kwargs       = (
        print_level   = MadNLP.ERROR,
        tol           = 1e-6,
        linear_solver = CUDSSSolver,
    ),
)

The policy (Flux model) stays on CPU; only the NLP solve runs on GPU. Parameter updates (ExaModels.set_parameter!) and multiplier extraction handle CPU↔GPU transfers automatically.

Custom problems

For domain-specific models (power systems, robotics, etc.), build the ExaModels NLP directly instead of using build_linear_tracking_problem. The key requirements are:

  1. Add target constraints last so their multipliers form a contiguous slice of result.multipliers.
  2. Parameterize the initial state (p_x0), uncertainty trajectory (p_w), and target trajectory (p_target) as ExaModels parameters.
  3. Return a struct with fields .core, .model, .horizon, and .target_con_range.

Parallel GPU solves

When training samples are independent, multiple NLP instances can be solved concurrently on the same GPU. Build a pool of independent problem copies and pass it to train_tsddr:

pool = [(prob, prob.p_x0, prob.p_target, prob.p_w)]
for _ in 2:num_workers
    p = build_my_problem(backend = CUDABackend())
    push!(pool, (p, p.p_x0, p.p_target, p.p_w))
end

train_tsddr(policy, x0, prob, prob.p_x0, prob.p_target, prob.p_w, sampler;
    problem_pool        = pool,
    num_train_per_batch = num_workers,
)

Each pool entry gets its own MadNLP solver instance. Samples are distributed round-robin across the pool and solved via Threads.@spawn.

Penalty annealing

DecisionRulesExa.jl supports penalty annealing through the adjust_hyperparameters callback. The target penalty coefficient $\rho$ is stored as an ExaModels parameter and can be updated at runtime:

adjust_hyperparameters = function(iter, opt_state, num_train)
    phase = iter < 100 ? 0.1 :
            iter < 200 ? 1.0 :
            iter < 300 ? 10.0 : 30.0
    ρ = base_penalty * phase
    penalty_vals = fill(ρ / 2, T * nx)
    ExaModels.set_parameter!(prob.core, prob.p_penalty_half, penalty_vals)
    return num_train
end

This mirrors the penalty_schedule keyword in DecisionRules.jl's train_multistage.

Rollout evaluation

RolloutEvaluation in DecisionRules.jl evaluates policies stage-by-stage under deployment semantics. DecisionRulesExa.jl provides an analogous RolloutEvaluation that solves stage subproblems sequentially:

eval = RolloutEvaluation(
    stage_problem, x0, eval_scenarios;
    horizon              = T,
    n_uncertainty        = nw,
    set_stage_parameters! = my_stage_setter!,
    realized_state       = my_realized_state,
    stride               = 25,
    policy_state         = :realized,
)

Both packages report the same metrics: operational cost excluding target-deficit penalty, and target-violation share.

Mapping between packages

DecisionRules.jlDecisionRulesExa.jlNotes
train_multistagetrain_tsddrMain training loop
state_conditioned_policyStateConditionedPolicyLSTM policy
dense_multilayer_nnMLPPolicyMLP policy
state_params_inp_x0Initial state parameter
state_params_outp_targetTarget parameter
uncertainty_samplesp_w + samplerUncertainty parameter
SampleLog / recordrecord_lossPer-iteration callback
RolloutEvaluationRolloutEvaluationStage-wise eval
penalty_scheduleadjust_hyperparametersPenalty annealing
ScoreFunctionConfig—Not yet ported to ExaModels
Stage-wise decomposition—JuMP only
Multiple shooting—JuMP only

Embedded deterministic equivalent

The standard DeterministicEquivalentProblem treats the policy's target trajectory as an external parameter: the training loop generates $\hat{x}_{1:T}$ outside the NLP and passes it in via set_targets!. This is open-loop — the policy does not see the realized states from the coupled solve.

EmbeddedDeterministicEquivalentProblem embeds the policy inside the NLP via a VectorNonlinearOracle. The NLP constraint becomes:

\[\pi_\theta(w_t,\, x_{t-1}^*) - x_t - \delta_t = 0 \quad \forall t\]

where $x_{t-1}^*$ is the solver's realized state. This is closed-loop: the policy sees realized states from the coupled solve, and the duals $\lambda_t$ reflect the joint (policy + physics) system.

prob = build_embedded_deterministic_equivalent(
    policy;
    horizon       = T,
    nx            = nx,
    nu            = nu,
    nw            = nw,
    dynamics_eq   = my_dynamics,
    stage_cost    = my_cost,
    backend       = CUDABackend(),
)

train_tsddr_embedded(
    policy, x0, prob, sampler;
    num_batches         = 500,
    num_train_per_batch = 4,
    optimizer           = Flux.Adam(1f-3),
    madnlp_kwargs       = (print_level = MadNLP.ERROR, tol = 1e-6),
)

The oracle closures capture the policy by reference — updating Flux parameters between solves automatically changes the NLP without rebuilding it. Use invalidate_policy_cache! if your oracle caches policy-dependent intermediates.

Strict reachable targets

When the policy is guaranteed to produce feasible targets (e.g., via a reachable-set mapping), the slack variables $\delta_t$ can be removed entirely. This is strict mode: target constraints are hard equalities, the duals are pure shadow prices, and there is no target penalty to tune.

There are two strict deterministic-equivalent paths.

Embedded strict DE evaluates the policy inside the NLP against realized state decision variables.

Its constraint is simply $x_t = \pi_\theta(w_t, x_{t-1}^*)$. Because the policy receives the realized previous state, a reachable-set map can guarantee that the next strict equality is dynamically feasible.

Regular strict DE keeps the policy outside the NLP but rolls out targets from the known initial state:

\[\hat{x}_0 = x_0,\qquad \hat{x}_t = \pi_\theta(w_t, \hat{x}_{t-1}).\]

If the policy returns $\hat{x}_t \in R(\hat{x}_{t-1}, w_t)$ at every stage, then the entire strict DE target trajectory is feasible by induction. The solve then enforces $x_t = \hat{x}_t$ for every stage, so the realized state path is exactly the reachable target path.

For battery storage, the charge/discharge and energy bounds give a cheap one-stage battery-dynamic interval. It is not the complete reachable set of an AC-OPF: a target can still conflict with generation, branch, voltage, or reactive-power limits. The battery-storage specification therefore requires true-ACP zero-shedding tests before strict mode becomes the production default.

Sequential rollout evaluation

train_tsddr solves the full deterministic equivalent in one shot. For deployment diagnostics, DecisionRulesExa.jl provides RolloutEvaluation, which solves a one-stage ExaModels problem sequentially over a materialized scenario:

eval = RolloutEvaluation(
    stage_problem,
    x0,
    eval_scenarios;
    horizon             = T,
    n_uncertainty        = nw,
    set_stage_parameters! = my_setter!,
    realized_state       = my_state_reader,
    policy_state         = :realized,
)

This mirrors deployment semantics: the policy can be evaluated with the realized previous state (policy_state = :realized) or with its previous target (policy_state = :target) to match regular-DE target-generation semantics.

Critic control variate

train_tsddr optionally trains a scalar critic $C(w, \hat{x})$ that provides a learned control variate for the dual gradient signal. The critic does not replace the NLP solve — dual multipliers remain the primary actor gradient. The critic reduces gradient variance by subtracting a correlated baseline.

critic = Chain(Dense(input_dim => 128, tanh), Dense(128 => 128, tanh), Dense(128 => 1))

cv = ScalarCriticControlVariate(critic;
    featurizer          = default_critic_featurizer,
    value_loss_weight   = 1.0,
    gradient_loss_weight = 0.0,
)

critic_target = RolloutCriticTarget(stage_problem;
    horizon            = T,
    n_uncertainty      = nw,
    set_stage_parameters! = my_setter!,
    realized_state     = my_state_reader,
    policy_state       = :target,
)

train_tsddr(policy, x0, prob, prob.p_x0, prob.p_target, prob.p_w, sampler;
    control_variate              = cv,
    critic_training_target       = critic_target,
    actor_gradient_mode          = :control_variate,
    critic_cv_weight             = 1.0,
    critic_optimizer             = Flux.Adam(1f-3),
)

Two actor modes are supported:

  • :control_variate — subtracts $\nabla_{\hat{x}} C$ from the dual signal and adds it back as a differentiable surrogate. Unbiased when the critic is exact; reduces variance otherwise.
  • :surrogate — blends dual and critic actor gradients via explicit weights (dual_actor_weight, critic_actor_weight). Useful when raw duals are noisy, but no longer strictly unbiased.