Cloud Environment with Re-imaging
This tutorial demonstrates the configuration and use of a BSK-RL environment considering cloud coverage and re-imaging capabilities. Two reward functions are introduced: a single-picture binary case (where targets are deemed occluded by clouds or not and no re-imaging is allowed) and a success-probability-based re-imaging case where the problem is formulated in terms of the targets’ probability of being successfully observed. Still, the satellite cannot observe the true cloud coverage of each target, only its forecast. The satellite has to image targets while keeping a positive battery level. Moreover, custom actor and critic modules are defined leveraging a token-based architecture. This example script is part of an upcoming publication.
Loading Modules
[1]:
from collections.abc import Callable
from typing import ClassVar
import numpy as np
from Basilisk.architecture import bskLogging
from Basilisk.utilities import orbitalMotion
from bsk_rl import ConstellationTasking, act, obs, sats
from bsk_rl.data.base import Data, DataStore, GlobalReward
from bsk_rl.data.unique_image_data import (
UniqueImageData,
UniqueImageReward,
UniqueImageStore,
)
from bsk_rl.scene.targets import UniformTargets
from bsk_rl.sim import fsw
bskLogging.setDefaultLogLevel(bskLogging.BSK_WARNING)
Making a Scenario with Cloud Covered Targets
To account for clouds in the simulation process, we can associate a cloud coverage value to each target that represents the percentage of cloud coverage over that area. Cloud coverage can be randomly generated or derived from real data. Here, we have an example on how to use a stochastic cloud model using UniformTargets as a base and attach the following information to each target:
cloud_cover_truerepresents the true cloud coverage. Information from external sources, such as historical cloud data, can be used here based on each target’s position.cloud_cover_forecastrepresents the cloud coverage forecast. Forecast from external sources can be plugged in here.cloud_cover_sigmarepresents the standard deviation of the cloud coverage forecast.beliefrepresents the probability that the target was successfully observed.prev_obstime at which the last picture of the target was taken.belief_update_varstores the latest belief update
[2]:
class CloudTargets(UniformTargets):
mu_data = 0.6740208166434426 # Average global cloud coverage
def __init__(
self,
n_targets: int | tuple[int, int],
priority_distribution: Callable | None = None,
radius: float = orbitalMotion.REQ_EARTH * 1e3,
sigma_levels: tuple[float, float] = (0.01, 0.05),
reward_thresholds: float | tuple[float, float] = 0.95,
belief_init: tuple[float, float] = (0.0, 0.94),
prev_obs_init: tuple[float, float] = (0.0, 5700.0),
) -> None:
super().__init__(n_targets, priority_distribution, radius)
self.reward_thresholds = reward_thresholds
self.sigma_levels = sigma_levels
self.belief_init = belief_init
self.prev_obs_init = prev_obs_init
def regenerate_targets(self) -> None:
super().regenerate_targets()
for target in self.targets:
# Initialize true cloud coverage
cloud_cover_true = np.random.uniform(
0.0, self.mu_data * 2
) # Instead, true cloud coverage can be obtained by historical data based on the target's position
cloud_cover_true = np.clip(cloud_cover_true, 0.0, 1.0)
target.cloud_cover_true = cloud_cover_true
# Initialize cloud coverage forecast
target.cloud_cover_sigma = np.random.uniform(
self.sigma_levels[0], self.sigma_levels[1]
)
cloud_cover_forecast = np.random.normal(
target.cloud_cover_true, target.cloud_cover_sigma
)
target.cloud_cover_forecast = np.clip(cloud_cover_forecast, 0.0, 1.0)
# Set reward threshold
if isinstance(self.reward_thresholds, float):
target.reward_threshold = self.reward_thresholds
else:
target.reward_threshold = np.random.uniform(
self.reward_thresholds[0], self.reward_thresholds[1]
)
# Initialize beliefs and previous observations
b_S1 = np.random.uniform(self.belief_init[0], self.belief_init[1])
b_S0 = 1 - b_S1
target.belief = np.array([b_S0, b_S1])
target.prev_obs = -np.random.uniform(
self.prev_obs_init[0], self.prev_obs_init[1]
)
target.belief_update_var = 0.0
# Define the randomization interval for the number of targets
n_targets = (1000, 10000)
scenario = CloudTargets(n_targets=n_targets)
Making a Rewarder Considering Cloud Coverage for the Single-picture Case
When considering targets potentially covered by clouds, we can use a binary reward model where the reward is proportional to the target priority if the target’s cloud coverage is below its reward_threshold (how much cloud coverage is acceptable). Therefore, we create a modified rewarder CloudImageBinaryRewarder; it has similar settings as the UniqueImageReward class, but cloud_covered and cloud_free information is added. Additionally, the
calculate_reward function is modified for the binary reward model.
For this case, the reward function is given by
\[\begin{split}R = \begin{cases}
r_i & \text{if } c_{t_i} \leq c_{\text{thr}_i} \\
0 & \text{otherwise.}
\end{cases}\end{split}\]
where \(r_i\) is priority, \(c_{t_i}\) is the true cloud coverage, and \(c_{\text{thr}_i}\) is the reward_threshold for target \(i\). For a case where the reward is linearly proportional to the cloud coverage, see Cloud Environment
[3]:
from typing import TYPE_CHECKING
if TYPE_CHECKING: # pragma: no cover
from bsk_rl.scene.targets import (
Target,
)
class CloudImageBinaryData(UniqueImageData):
"""DataType for unique images of targets."""
def __init__(
self,
imaged: set["Target"] | None = None,
duplicates: int = 0,
known: set["Target"] | None = None,
cloud_covered: set["Target"] | None = None,
cloud_free: set["Target"] | None = None,
) -> None:
"""Construct unit of data to record unique images.
Keeps track of ``imaged`` targets, a count of ``duplicates`` (i.e. images that
were not rewarded due to the target already having been imaged), and all
``known`` targets in the environment. It also keeps track of which targets are considered
``cloud_covered`` and ``cloud_free`` based on the specified threshold.
Args:
imaged: Set of targets that are known to be imaged.
duplicates: Count of target imaging duplication.
known: Set of targets that are known to exist (imaged and unimaged).
cloud_covered: Set of imaged targets that are known to be cloud covered.
cloud_free: Set of imaged targets that are known to be cloud free.
"""
super().__init__(imaged=imaged, duplicates=duplicates, known=known)
if cloud_covered is None:
cloud_covered = set()
if cloud_free is None:
cloud_free = set()
self.cloud_covered = set(cloud_covered)
self.cloud_free = set(cloud_free)
def __add__(self, other: "CloudImageBinaryData") -> "CloudImageBinaryData":
"""Combine two units of data.
Args:
other: Another unit of data to combine with this one.
Returns:
Combined unit of data.
"""
imaged = self.imaged | other.imaged
duplicates = (
self.duplicates
+ other.duplicates
+ len(self.imaged)
+ len(other.imaged)
- len(imaged)
)
known = self.known | other.known
cloud_covered = self.cloud_covered | other.cloud_covered
cloud_free = self.cloud_free | other.cloud_free
return self.__class__(
imaged=imaged,
duplicates=duplicates,
known=known,
cloud_covered=cloud_covered,
cloud_free=cloud_free,
)
class CloudImageBinaryDataStore(UniqueImageStore):
"""DataStore for unique images of targets."""
data_type = CloudImageBinaryData
def compare_log_states(
self, old_state: np.ndarray, new_state: np.ndarray
) -> CloudImageBinaryData:
"""Check for an increase in logged data to identify new images.
Args:
old_state: older storedData from satellite storage unit
new_state: newer storedData from satellite storage unit
Returns:
list: Targets imaged at new_state that were unimaged at old_state
"""
data_increase = new_state - old_state
if data_increase <= 0:
return CloudImageBinaryData()
else:
assert self.satellite.latest_target is not None
self.update_target_colors([self.satellite.latest_target])
cloud_coverage = self.satellite.latest_target.cloud_cover_true
cloud_threshold = self.satellite.latest_target.reward_threshold
if cloud_coverage > cloud_threshold:
cloud_covered = [self.satellite.latest_target]
cloud_free = []
else:
cloud_covered = []
cloud_free = [self.satellite.latest_target]
return CloudImageBinaryData(
imaged={self.satellite.latest_target},
cloud_covered=cloud_covered,
cloud_free=cloud_free,
)
class CloudImageBinaryRewarder(UniqueImageReward):
"""DataManager for rewarding unique images."""
data_store_type = CloudImageBinaryDataStore
def calculate_reward(
self, new_data_dict: dict[str, CloudImageBinaryData]
) -> dict[str, float]:
"""Reward new each unique image once using self.reward_fn().
Args:
new_data_dict: Record of new images for each satellite
Returns:
reward: Cumulative reward across satellites for one step
"""
reward = {}
imaged_counts = {}
for new_data in new_data_dict.values():
for target in new_data.imaged:
if target not in imaged_counts:
imaged_counts[target] = 0
imaged_counts[target] += 1
for sat_id, new_data in new_data_dict.items():
reward[sat_id] = 0.0
for target in new_data.cloud_free:
if target not in self.data.imaged:
reward[sat_id] += (
self.reward_fn(target.priority) / imaged_counts[target]
)
return reward
# Define the reward function as a function of the priority of the target and the cloud cover
def reward_function_binary(priority):
return priority
# Uncomment this line and comment the reward in the cell below to use the binary reward function
# rewarder = CloudImageBinaryRewarder(reward_fn=reward_function_binary)
Making a Rewarder Considering Cloud Coverage for the Success-Probability-Based Re-imaging Case
If the target is deemed occluded by clouds, it won’t be tasked again in the single-picture case. However, the problem can be formulated in terms of the probability of observing the target (\(\text{P}(S=1)\), represented by the variable belief in the code) given the number of pictures and time difference between them (\(\delta t_i\)). Thus, a new rewarder named CloudImageProbabilityRewarder is created to accommodate this new formulation, as well as a new reward function.
The reward function accounts for the desired success probability threshold for each target (\(\theta_{\text{thr}_i}\), represented by reward_threshold in the code) and has a tunable parameter \(\alpha\in[0,1]\):
\[\begin{split}R = \begin{cases}
r_i\alpha\Delta \text{P}(S=1) + r_i(1 - \alpha) & \text{ if } \text{P}_i(S=1) \geq \theta_{\text{thr}_i} \\
r_i\alpha\Delta \text{P}(S=1) & \text{ otherwise.}
\end{cases}\end{split}\]
[4]:
class CloudImageProbabilityData(Data):
"""DataType for unique images of targets."""
def __init__(
self,
imaged: list["Target"] | None = None,
imaged_complete: set["Target"] | None = None,
list_belief_update_var: list[float] | None = None,
known: set["Target"] | None = None,
) -> None:
"""Construct unit of data to record unique images.
Keeps track of ``imaged`` targets and completely imaged targets (those with a success probability
higher than the ``reward_threshold``).
Args:
imaged: List of targets that are known to be imaged.
imaged_complete: Set of targets that are known to be completely imaged (P(S=1) >= reward_threshold).
list_belief_update_var: List of belief update variations for each target after each picture.
known: List of targets that are known to exist (imaged and not imaged)
"""
if imaged is None:
imaged = []
if imaged_complete is None:
imaged_complete = set()
if list_belief_update_var is None:
list_belief_update_var = []
if known is None:
known = set()
self.known = set(known)
self.imaged = imaged
self.imaged_complete = imaged_complete
self.list_belief_update_var = list(list_belief_update_var)
def __add__(
self, other: "CloudImageProbabilityData"
) -> "CloudImageProbabilityData":
"""Combine two units of data.
Args:
other: Another unit of data to combine with this one.
Returns:
Combined unit of data.
"""
imaged = self.imaged + other.imaged
imaged_complete = self.imaged_complete | other.imaged_complete
list_belief_update_var = (
self.list_belief_update_var + other.list_belief_update_var
)
known = self.known | other.known
return self.__class__(
imaged=imaged,
imaged_complete=imaged_complete,
list_belief_update_var=list_belief_update_var,
known=known,
)
class CloudImageProbabilityDataStore(DataStore):
"""DataStore for unique images of targets."""
data_type = CloudImageProbabilityData
def __init__(self, *args, **kwargs) -> None:
"""DataStore for unique images.
Detects new images by watching for an increase in data in each target's corresponding
buffer.
"""
super().__init__(*args, **kwargs)
def get_log_state(self) -> np.ndarray:
"""Log the instantaneous storage unit state at the end of each step.
Returns:
array: storedData from satellite storage unit
"""
msg = self.satellite.dynamics.storageUnit.storageUnitDataOutMsg.read()
return msg.storedData[0]
def compare_log_states(
self, old_state: np.ndarray, new_state: np.ndarray
) -> CloudImageProbabilityData:
"""Check for an increase in logged data to identify new images.
This method also performs the belief update (new probability of success) for each target
based on the cloud coverage forecast and the time difference between the current time and
the previous observation time. It also keeps track of the variation in the belief update.
Args:
old_state: older storedData from satellite storage unit
new_state: newer storedData from satellite storage unit
Returns:
list: Targets imaged at new_state that were unimaged at old_state
"""
data_increase = new_state - old_state
if data_increase <= 0:
return CloudImageProbabilityData()
else:
assert self.satellite.latest_target is not None
# return UniqueImageData(imaged={self.satellite.latest_target})
target = self.satellite.latest_target
current_sim_time = self.satellite.simulator.sim_time
belief_update_func = self.satellite.belief_update_func
target_prev_obs = (
target.prev_obs
) # Time at which the target was previously observed
target_time_diff = (
current_sim_time - target_prev_obs
) # Time difference between the current time and the previous observation time
target_belief = (
target.belief
) # Belief of the target before the current picture
target_cloud_cover_forecast = target.cloud_cover_forecast
updated_belief = belief_update_func(
target_belief, target_cloud_cover_forecast, target_time_diff
)
target.belief = updated_belief # Update the belief of the target
target.belief_update_var = updated_belief[1] - target_belief[1]
target.prev_obs = current_sim_time # Update the previous observation time
if updated_belief[1] > target.reward_threshold:
list_imaged_complete = [target]
else:
list_imaged_complete = []
list_belief_update_var = target.belief_update_var
return CloudImageProbabilityData(
imaged=[target],
imaged_complete=set(list_imaged_complete),
list_belief_update_var=[list_belief_update_var],
)
class CloudImageProbabilityRewarder(GlobalReward):
data_store_type = CloudImageProbabilityDataStore
def __init__(
self,
reward_fn: Callable,
alpha: float = 0.5,
) -> None:
"""
Modifies the constructor to include the alpha parameter to tune the reward function and
the reward function.
Args:
reward_fn: Reward as function of priority, targets belief, and alpha.
"""
super().__init__()
self.reward_fn = reward_fn
self.alpha = alpha
def initial_data(self, satellite: "sats.Satellite") -> "CloudImageProbabilityData":
"""Furnish data to the scenario.
Currently, it is assumed that all targets are known a priori, so the initial data
given to the data store is the list of all targets.
"""
return self.data_type(known=self.scenario.targets)
def calculate_reward(
self, new_data_dict: dict[str, CloudImageProbabilityData]
) -> dict[str, float]:
"""Reward new each unique image once using self.reward_fn().
Args:
new_data_dict: Record of new images for each satellite
Returns:
reward: Cumulative reward across satellites for one step
"""
reward = {}
for sat_id, new_data in new_data_dict.items():
reward[sat_id] = 0.0
for target, belief_variation in zip(
new_data.imaged, new_data.list_belief_update_var
):
if target not in self.data.imaged_complete:
reward[sat_id] += self.reward_fn(
target.priority,
belief_variation,
self.alpha,
reach_threshold=target in new_data.imaged_complete,
)
return reward
# Define the reward function as a function of the priority of the target, the cloud cover, and the number of times the target has been imaged
def reward_function_probability(
priority: float, belief_variation: float, alpha: float, reach_threshold: bool
) -> float:
"""
Rewards based on the priority of the target, the belief variation, and the alpha parameter.
Args:
priority: Priority of the target.
belief_variation: Variation in the belief of the target after the picture.
alpha: Tuning parameter between 0 and 1.
reach_threshold: Boolean indicating whether the target has reached the reward threshold.
Returns:
float: Reward for the target.
"""
if reach_threshold:
return priority * (1 - alpha) + priority * belief_variation * alpha
else:
return priority * belief_variation * alpha
rewarder = CloudImageProbabilityRewarder(
reward_fn=reward_function_probability, alpha=1.0
)
CloudImageProbabilityDataStore requires a function belief_update_func that returns the updated success probability for target \(i\) (\(\text{P}^{(k+1)}_i(S=1)\)) given its current success probability (\(\text{P}^{(k)}_i(S=1)\)), cloud coverage forecast (\(c_{f_i}\)), and the time different between the current and previous image (\(\delta t_i\)).
The update in the success probability is given by:
\[\text{P}^{(k+1)}(S=1) = 1 - \text{P}^{(k)}(S=0)\bar{c}_{f_i}\]
To penalize two consecutive pictures without enough elapsed time (and not enough shift in clouds’ position), a new cloud-free probability variable \(g_{f_i}\) is introduced such that
\[g^{(k)}_{f_i} = (1-c^{(k)}_{f_i})\beta(\delta t_i)\]
where \(\beta\) is given by a sigmoid
\[\beta(\delta t) = \frac{1}{\eta_3+e^{-\eta_1(\frac{\delta t}{\tau}-\eta_2)}}\]
and
\[\bar{c}_{f_i} = 1 - g_{f_i}^{(k)}\]
leading to:
\[\text{P}^{(k+1)}(S=1) = \text{P}^{(k)}(S=1) + (1-\text{P}^{(k)}(S=1))(1-c^{(k)}_{f_i})\beta(\delta t_i)\]
[5]:
def time_variation(
delta_t: float,
t_const: float,
eta_1: float = 2.5,
eta_2: float = 2.5,
eta_3: float = 1.0,
) -> float:
"""
Time variation function based on sigmoid function.
Args:
delta_t (float): Time difference between the current time and the previous observation time.
t_const (float): Time constant for the sigmoid function.
eta_1 (float): Sigmoid function parameter.
eta_2 (float): Sigmoid function parameter.
eta_3 (float): Sigmoid function parameter.
Returns:
float: Time variation value.
"""
if delta_t <= 0:
return 0
else:
return 1 / (eta_3 + np.exp(-eta_1 * (delta_t / t_const - eta_2)))
def belief_update(
b: list[float], cloud_cover_forecast: float, delta_t: float, t_const: float
) -> np.array:
"""
Update the belief based on the cloud forecast and the time variation.
Args:
b (np.array): Belief array (b(S=0), b(S=1)).
cloud_forecast (float): Cloud coverage forecast.
delta_t (float): Time difference between the current time and the previous observation time.
t_const (float): Time constant for the sigmoid function.
Returns:
np.array: Updated belief array
"""
cloud_time_variation = time_variation(delta_t, t_const)
cloud_free = (1 - cloud_cover_forecast) * cloud_time_variation
cloud_cover_bar = 1 - cloud_free
b_0 = b[0] * cloud_cover_bar
b_1 = 1 - b_0
return np.array([b_0, b_1])
def belief_update_func(
b: list[float], cloud_cover_forecast: float, delta_t: float
) -> np.array:
"""
Belief update function for the satellite.
Args:
b (np.array): Belief array (b(S=0), b(S=1)).
cloud_forecast (float): Cloud coverage forecast.
delta_t (float): Time difference between the current time and the previous observation time.
Returns:
np.array: Updated belief array
"""
time_constant = (
30 * 60 / 5
) # 30 minutes for the time variation function to reach 0.998 with the predefined parameters
return belief_update(b, cloud_cover_forecast, delta_t, time_constant)
Configuring the Satellite to Have Access to Cloud Information
The satellite has observations and actions associated with it that are relevant to the decision-making process. The observation space can be modified to include information about the targets and the weather (cloud coverage forecast, reward threshold, success probability, etc) which allows better informed decision-making. Satellite class C_F is used for the single-picture case, whereas class C_F_Bayesian has additional observations, which are useful in the success-probability-based
re-imaging case.
-
SatProperties: Body angular velocity, instrument pointing direction, body position, body velocity, battery charge (properties in flight software model or dynamics model). Also, customized dynamics property in CustomDynModel below: spacecraft-to-Sun unit direction vector.
CloudCoverDensityprovides a summary of cloud cover over the upcoming targets. The equation is based on reward density discussed in the paper Learning Policies for Autonomous Earth-Observing Satellite Scheduling over Semi-Markov Decision Processes.OpportunityProperties: Target’s priority, cloud coverage forecast, standard deviation of cloud coverage forecast, probability of being successfully imaged, and last time it was imaged (upcoming 40 targets).
Eclipse: Next eclipse start and end times.
-
Charge: Enter a sun-pointing charging mode for 60 seconds.
Image: Image target from upcoming 40 targets
Dynamics model: FullFeaturedDynModel is used and a property, spacecraft-to-Sun unit direction vector, is added.
Flight software model: SteeringImagerFSWModel is used.
[6]:
def s_hat_H(sat):
r_SN_N = (
sat.simulator.world.gravFactory.spiceObject.planetStateOutMsgs[
sat.simulator.world.sun_index
]
.read()
.PositionVector
)
r_BN_N = sat.dynamics.r_BN_N
r_SB_N = np.array(r_SN_N) - np.array(r_BN_N)
r_SB_H = sat.dynamics.HN @ r_SB_N
return r_SB_H / np.linalg.norm(r_SB_H)
class CloudCoverDensity(obs.Observation):
def __init__(
self,
interval_duration=60 * 3,
intervals=10,
norm=3,
):
self.satellite: sats.ImagingSatellite
super().__init__()
self.interval_duration = interval_duration
self.intervals = intervals
self.norm = norm
def get_obs(self):
if self.intervals == 0:
return []
self.satellite.calculate_additional_windows(
self.simulator.sim_time
+ (self.intervals + 1) * self.interval_duration
- self.satellite.window_calculation_time
)
soonest = self.satellite.upcoming_opportunities_dict(types="target")
cloud_covers = np.array([target.cloud_cover_forecast for target in soonest])
times = np.array([opportunities[0][1] for opportunities in soonest.values()])
time_bins = np.floor((times - self.simulator.sim_time) / self.interval_duration)
densities = [sum(cloud_covers[time_bins == i]) for i in range(self.intervals)]
return np.array(densities) / self.norm
class C_F(sats.ImagingSatellite):
observation_spec: ClassVar[list[obs.Observation]] = [
obs.SatProperties(
dict(prop="omega_BN_B", norm=0.03),
dict(prop="c_hat_H"),
dict(prop="r_BN_P", norm=orbitalMotion.REQ_EARTH * 1e3),
dict(prop="v_BN_P", norm=7616.5),
dict(prop="battery_charge_fraction"),
dict(prop="s_hat_H", fn=s_hat_H),
),
obs.Eclipse(norm=5700),
CloudCoverDensity(intervals=20, norm=5),
obs.OpportunityProperties(
dict(prop="priority"),
dict(prop="r_LB_H", norm=orbitalMotion.REQ_EARTH * 1e3),
dict(prop="target_angle", norm=np.pi / 2),
dict(prop="target_angle_rate", norm=0.03),
dict(prop="opportunity_open", norm=300.0),
dict(prop="opportunity_close", norm=300.0),
dict(
prop="cloud_forecast",
fn=lambda sat, opp: opp["object"].cloud_cover_forecast,
),
dict(
prop="cloud_sigma",
fn=lambda sat, opp: opp["object"].cloud_cover_sigma,
norm=0.05,
),
type="target",
n_ahead_observe=40,
),
]
action_spec: ClassVar[list[act.Action]] = [
act.Charge(duration=60.0),
act.Image(n_ahead_image=40),
]
fsw_type = fsw.SteeringImagerFSWModel
def time_since_prev_obs(sat, opp):
prev_obs = opp["object"].prev_obs
if prev_obs is None:
return 0.0
else:
return sat.simulator.sim_time - prev_obs
def belief_expected(sat, opp):
belief = opp["object"].belief.copy()
cloud_cover_forecast = opp["object"].cloud_cover_forecast.copy()
delta_t = time_since_prev_obs(sat, opp)
expected = belief_update_func(belief, cloud_cover_forecast, delta_t)
return expected[1]
class C_F_Bayesian(sats.ImagingSatellite):
observation_spec: ClassVar[list[obs.Observation]] = [
obs.SatProperties(
dict(prop="omega_BN_B", norm=0.03),
dict(prop="c_hat_H"),
dict(prop="r_BN_P", norm=orbitalMotion.REQ_EARTH * 1e3),
dict(prop="v_BN_P", norm=7616.5),
dict(prop="battery_charge_fraction"),
dict(prop="s_hat_H", fn=s_hat_H),
),
obs.Eclipse(norm=5700),
CloudCoverDensity(intervals=20, norm=5),
obs.OpportunityProperties(
dict(prop="priority"),
dict(prop="r_LB_H", norm=orbitalMotion.REQ_EARTH * 1e3),
dict(prop="target_angle", norm=np.pi / 2),
dict(prop="target_angle_rate", norm=0.03),
dict(prop="opportunity_open", norm=300.0),
dict(prop="opportunity_close", norm=300.0),
dict(
prop="cloud_forecast",
fn=lambda sat, opp: opp["object"].cloud_cover_forecast,
),
dict(prop="belief", fn=lambda sat, opp: opp["object"].belief[1], norm=1.0),
dict(prop="time_since_prev_obs", fn=time_since_prev_obs, norm=5700.0),
dict(prop="belief_expected", fn=belief_expected, norm=1.0),
type="target",
n_ahead_observe=40,
),
]
action_spec: ClassVar[list[act.Action]] = [
act.Charge(duration=60.0),
act.Image(n_ahead_image=40),
]
fsw_type = fsw.SteeringImagerFSWModel
def __init__(self, *args, belief_update_func=None, **kwargs) -> None:
super().__init__(*args, **kwargs)
self.belief_update_func = belief_update_func
It is necessary to add a filter to remove targets that reached the success threshold from the targets list when re-imaging is allowed such that:
[7]:
def belief_threshold_filter(opportunity):
if opportunity["type"] == "target":
return (
opportunity["object"].belief[1] < opportunity["object"].reward_threshold
)
return True
When instantiating a satellite, these parameters can be overriden with a constant or rerandomized every time the environment is reset using the sat_args dictionary.
[8]:
dataStorageCapacity: float = 20 * 8e6 * 100
sat_args = C_F_Bayesian.default_sat_args(
imageAttErrorRequirement=0.01,
imageRateErrorRequirement=0.01,
batteryStorageCapacity=80.0 * 3600 * 2,
storedCharge_Init=lambda: np.random.uniform(0.4, 1.0) * 80.0 * 3600 * 2,
u_max=0.1,
K1=0.2,
K3=0.5,
servo_P=30,
nHat_B=np.array([0, 0, -1]),
imageTargetMinimumElevation=np.radians(45),
rwBasePower=20,
maxWheelSpeed=1500,
storageInit=lambda: np.random.randint(
0 * dataStorageCapacity,
0.01 * dataStorageCapacity,
), # Initialize storage use close to zero
wheelSpeeds=lambda: np.random.uniform(
-1, 1, 3
), # Initialize reaction wheel speeds close to zero
dataStorageCapacity=dataStorageCapacity, # Large storage to avoid filling up in three orbits
# oe=partial(
# random_circular_orbit,
# i=45.0,
# alt=500,
# ), # Optional for when using a single satellite
)
Greedy Heuristic
A greedy heuristic can be created to task the satellite in the single-picture case, where the expected reward is used to compute the reward density per target with
\[\rho_i^\star = \arg\max_{\rho_i \in \mathcal{U}_{35}} \left\{ \frac{r_i p_i}{t_{i,\text{max}}} \; : \; t_{i,\text{max}} \leq t_{i,\mathrm{close}} \right\}\]
where \(t_{i,\text{max}}\) is the maximum time between the slew time required to image the target (returned by a neural network or by a regression model as shown in function slew_time_regression) and the opening of the opportunity window for the target (\(t_{i,\text{open}}\)). Targets with a required slew time larger than the end of the opportunity window (\(t_{i,\text{close}}\)) are discarded.
The expected reward is obtained with
\[p_i = \text{P}(c_{t_i}\leq c_{\text{thr}_i}|c_{f_i}, \sigma_{i})\]
[9]:
import scipy as sp
def slew_time_regression(
tgt_angle: float,
tgt_angle_rate: float,
x_0: float = 3.504,
x_1: float = 96.8639,
x_2: float = 1325.3045,
x_3: float = -23.6751,
x_4: float = -1.681e4,
x_5: float = -1009.7423,
) -> float:
"""
Regression model to estimate the slew time based on the target angle and target angle rate.
Created by fitting a polynomial regression model to the data obtained from the simulation. A NN could be used instead for better accuracy.
"""
slew_time = (
x_0
+ x_1 * tgt_angle
+ x_2 * tgt_angle_rate
+ x_3 * tgt_angle**2
+ x_4 * tgt_angle_rate**2
+ x_5 * tgt_angle * tgt_angle_rate
)
return np.clip(slew_time, 1e-3, None)
def collect_tgt_info(sat, lookahead: int) -> tuple:
"""
Collects information about the targets in the satellite's observation space.
"""
obs = sat.get_obs()
list_slew_time = []
list_priority = []
list_cloud_forecast = []
list_window_open = []
list_window_close = []
list_cloud_sigma = []
list_belief = []
list_time_since_prev_obs = []
obs_keys = sat.observation_builder.obs_array_keys()
for tgt_i in range(lookahead):
pos_angle_i = obs_keys.index(f"target.target_{tgt_i}.target_angle_normd")
pos_angle_rate_i = obs_keys.index(
f"target.target_{tgt_i}.target_angle_rate_normd"
)
pos_window_open_i = obs_keys.index(
f"target.target_{tgt_i}.opportunity_open_normd"
)
pos_window_close_i = obs_keys.index(
f"target.target_{tgt_i}.opportunity_close_normd"
)
pos_priority_i = obs_keys.index(f"target.target_{tgt_i}.priority")
pos_cloud_forecast_i = obs_keys.index(f"target.target_{tgt_i}.cloud_forecast")
if f"target.target_{tgt_i}.cloud_sigma_normd" in obs_keys:
pos_cloud_sigma_i = obs_keys.index(
f"target.target_{tgt_i}.cloud_sigma_normd"
)
else:
pos_cloud_sigma_i = None
if f"target.target_{tgt_i}.belief" in obs_keys:
pos_belief_i = obs_keys.index(f"target.target_{tgt_i}.belief")
pos_time_since_prev_obs_i = obs_keys.index(
f"target.target_{tgt_i}.time_since_prev_obs_normd"
)
else:
pos_belief_i = None
pos_time_since_prev_obs_i = None
angle_i = obs[pos_angle_i] * np.pi / 2
angle_rate_i = obs[pos_angle_rate_i] * 0.03
window_open_i = obs[pos_window_open_i] * 300.0
window_close_i = obs[pos_window_close_i] * 300.0
priority_i = obs[pos_priority_i]
cloud_forecast = obs[pos_cloud_forecast_i]
cloud_sigma_i = (
obs[pos_cloud_sigma_i] * 0.05 if pos_cloud_sigma_i is not None else 0
)
belief_i = obs[pos_belief_i] if pos_belief_i is not None else 0
time_since_prev_obs_i = (
obs[pos_time_since_prev_obs_i] * 5700.0
if pos_time_since_prev_obs_i is not None
else 0
)
slew_time_i = slew_time_regression(angle_i, angle_rate_i)
list_slew_time.append(slew_time_i)
list_priority.append(priority_i)
list_cloud_forecast.append(cloud_forecast)
list_window_open.append(window_open_i)
list_window_close.append(window_close_i)
list_cloud_sigma.append(cloud_sigma_i)
list_belief.append(belief_i)
list_time_since_prev_obs.append(time_since_prev_obs_i)
return (
np.array(list_slew_time),
np.array(list_priority),
np.array(list_cloud_forecast),
np.array(list_window_open),
np.array(list_window_close),
np.array(list_cloud_sigma),
np.array(list_belief),
np.array(list_time_since_prev_obs),
)
def greedy_heuristic_single_picture(sat, lookahead: int = 35):
"""
Greedy heuristic for selecting the best target to image based on the expected reward per time.
Args:
sat: Satellite object.
lookahead: Number of targets to consider in the observation space.
Returns:
int: Index of the best target to image next in the action space (0 action is charge).
"""
(
list_slew_time,
list_priority,
list_cloud_forecast,
list_opening_windows,
list_closing_windows,
list_cloud_sigma,
_,
_,
) = collect_tgt_info(sat, lookahead)
time_to_tgt = np.maximum(list_slew_time, list_opening_windows)
time_to_tgt = np.clip(time_to_tgt, 1.0, None)
list_scores = np.zeros(len(time_to_tgt))
for i in range(len(time_to_tgt)):
if time_to_tgt[i] > list_closing_windows[i]:
list_scores[i] = -np.inf
else:
tgt_probability_clear_i = np.clip(
sp.stats.norm.cdf(
0.2,
loc=list_cloud_forecast[i],
scale=list_cloud_sigma[i],
),
0.0,
1.0,
)
list_scores[i] = list_priority[i] * tgt_probability_clear_i / time_to_tgt[i]
target_pos = np.argmax(list_scores)
return target_pos + 1
For the success-probability-based re-imaging case, the heuristic for \(\alpha=1.0\) selects a target from the \(N=35\) upcoming targets with
\[\rho^\star=\arg\max_{\rho_i \in \mathcal{U}_{35}} \left\{\frac{r_i \Delta\text{P}_i(S=1)}{t_{i,{\text{max}}}} \; : \; \text{P}_i(S=1) < \theta_{\text{thr}_i} \; t_{i,\text{max}} < t_{i,\text{close}}\right\}\]
assuming that the next image is taken at the next decision step. For the heuristic with \(\alpha=0.0\) the target is selected with
\[\rho^\star=\arg\max_{\rho_i \in \mathcal{U}_{35}} \left\{\frac{r_i}{t_{i,{\text{max}}}k_i}, \; : \; \text{P}_i(S=1) < \theta_{\text{thr}_i}, \; t_{i,\text{max}} < t_{i,\text{close}}\right\}\]
where \(k_i\) estimates the number of images required to reach the success threshold and assuming enough elapsed time between images.
[10]:
def greedy_heuristic_reimaging(sat, alpha, lookahead):
"""
Greedy heuristic for reimaging targets based on the expected belief update and the priority of the target.
Args:
sat: Satellite object.
alpha: Tuning parameter. Should be 0 or 1.
lookahead: Number of targets to consider in the observation space.
Returns:
int: Index of the best target to image next in the action space (0 action is charge).
"""
(
list_slew_time,
list_priority,
list_cloud_forecast,
list_opening_windows,
list_closing_windows,
_,
list_belief,
list_time_since_prev_obs,
) = collect_tgt_info(sat, lookahead)
time_to_tgt = np.maximum(list_slew_time, list_opening_windows)
time_to_tgt = np.clip(time_to_tgt, 1.0, None)
list_scores = []
for i in range(lookahead):
if time_to_tgt[i] > list_closing_windows[i]:
list_scores.append(-np.inf)
else:
tgt_cloud_forecast = list_cloud_forecast[i]
tgt_belief = list_belief[i]
priority = list_priority[i]
expected_belief = belief_update_func(
[1 - tgt_belief, tgt_belief],
tgt_cloud_forecast,
list_time_since_prev_obs[i],
)
if alpha == 1:
score = (expected_belief[1] - tgt_belief) * priority / time_to_tgt[i]
else:
if expected_belief[1] >= 0.95:
score = priority / time_to_tgt[i]
else:
if tgt_cloud_forecast <= 1.0:
new_belief_temp = expected_belief
count = 1
for _ in range(10):
new_belief_temp = belief_update_func(
new_belief_temp.copy(),
tgt_cloud_forecast,
list_time_since_prev_obs[i],
)
count += 1
if new_belief_temp[1] >= 0.95:
break
score = priority / time_to_tgt[i] / count
else:
score = 0.0
list_scores.append(score)
best_target = np.argmax(list_scores)
return best_target + 1
Initializing and Interacting with the Environment
For this example, we will be using the multi-agent ConstellationTasking environment. Along with passing the satellite that we configured, the environment takes a scenario, which defines the environment the satellite is acting in, and a rewarder, which defines how data collected from the scenario is rewarded.
[11]:
from bsk_rl.utils.orbital import walker_delta_args
sat_arg_randomizer = walker_delta_args(
altitude=500.0, n_planes=1, inc=45, clustersize=5, clusterspacing=72
)
satellites = [
C_F_Bayesian(f"EO-{i}", sat_args, belief_update_func=belief_update_func)
for i in range(5)
]
# Add filter to satellites to remove targets that have already reached the belief threshold
for sat in satellites:
sat.add_access_filter(belief_threshold_filter)
env = ConstellationTasking(
satellites=satellites,
scenario=scenario,
rewarder=rewarder,
sat_arg_randomizer=sat_arg_randomizer,
sim_rate=0.5,
max_step_duration=300.0,
time_limit=95 * 60 / 2, # half orbit
log_level="INFO",
failure_penalty=0.0,
)
First, reset the environment. It is possible to specify the seed when resetting the environment.
[12]:
observation, info = env.reset(seed=1)
2026-07-28 23:38:33,086 gym INFO Resetting environment with seed=1
2026-07-28 23:38:33,089 scene.targets INFO Generating 9597 targets
2026-07-28 23:38:33,336 sats.satellite.EO-0 INFO <0.00> EO-0: Finding opportunity windows from 0.00 to 3000.00 seconds
2026-07-28 23:38:33,658 sats.satellite.EO-1 INFO <0.00> EO-1: Finding opportunity windows from 0.00 to 3000.00 seconds
2026-07-28 23:38:34,059 sats.satellite.EO-2 INFO <0.00> EO-2: Finding opportunity windows from 0.00 to 3000.00 seconds
2026-07-28 23:38:34,398 sats.satellite.EO-3 INFO <0.00> EO-3: Finding opportunity windows from 0.00 to 3000.00 seconds
2026-07-28 23:38:34,732 sats.satellite.EO-4 INFO <0.00> EO-4: Finding opportunity windows from 0.00 to 3000.00 seconds
2026-07-28 23:38:35,082 sats.satellite.EO-0 INFO <0.00> EO-0: Finding opportunity windows from 3000.00 to 4200.00 seconds
2026-07-28 23:38:35,315 sats.satellite.EO-1 INFO <0.00> EO-1: Finding opportunity windows from 3000.00 to 4200.00 seconds
2026-07-28 23:38:35,538 sats.satellite.EO-2 INFO <0.00> EO-2: Finding opportunity windows from 3000.00 to 4200.00 seconds
2026-07-28 23:38:35,773 sats.satellite.EO-3 INFO <0.00> EO-3: Finding opportunity windows from 3000.00 to 4200.00 seconds
2026-07-28 23:38:35,994 sats.satellite.EO-4 INFO <0.00> EO-4: Finding opportunity windows from 3000.00 to 4200.00 seconds
2026-07-28 23:38:36,205 gym INFO <0.00> Environment reset
It is possible to print out the actions and observations. The composed satellite action_description returns a human-readable action map each satellite has the same action space and similar observation space.
[13]:
print("Actions:", env.satellites[0].action_description, "\n")
print("Observations:", env.unwrapped.satellites[0].observation_description, "\n")
# # Uncomment to see the observation for each satellite
# # Using the composed satellite features also provides a human-readable state:
# for satellite in env.unwrapped.satellites:
# for k, v in satellite.observation_builder.obs_dict().items():
# print(f"{k}: {v}")
Actions: ['action_charge', 'action_image_0', 'action_image_1', 'action_image_2', 'action_image_3', 'action_image_4', 'action_image_5', 'action_image_6', 'action_image_7', 'action_image_8', 'action_image_9', 'action_image_10', 'action_image_11', 'action_image_12', 'action_image_13', 'action_image_14', 'action_image_15', 'action_image_16', 'action_image_17', 'action_image_18', 'action_image_19', 'action_image_20', 'action_image_21', 'action_image_22', 'action_image_23', 'action_image_24', 'action_image_25', 'action_image_26', 'action_image_27', 'action_image_28', 'action_image_29', 'action_image_30', 'action_image_31', 'action_image_32', 'action_image_33', 'action_image_34', 'action_image_35', 'action_image_36', 'action_image_37', 'action_image_38', 'action_image_39']
Observations: [np.str_('sat_props.omega_BN_B_normd[0]'), np.str_('sat_props.omega_BN_B_normd[1]'), np.str_('sat_props.omega_BN_B_normd[2]'), np.str_('sat_props.c_hat_H[0]'), np.str_('sat_props.c_hat_H[1]'), np.str_('sat_props.c_hat_H[2]'), np.str_('sat_props.r_BN_P_normd[0]'), np.str_('sat_props.r_BN_P_normd[1]'), np.str_('sat_props.r_BN_P_normd[2]'), np.str_('sat_props.v_BN_P_normd[0]'), np.str_('sat_props.v_BN_P_normd[1]'), np.str_('sat_props.v_BN_P_normd[2]'), np.str_('sat_props.battery_charge_fraction'), np.str_('sat_props.s_hat_H[0]'), np.str_('sat_props.s_hat_H[1]'), np.str_('sat_props.s_hat_H[2]'), np.str_('eclipse[0]'), np.str_('eclipse[1]'), np.str_('obs[0]'), np.str_('obs[1]'), np.str_('obs[2]'), np.str_('obs[3]'), np.str_('obs[4]'), np.str_('obs[5]'), np.str_('obs[6]'), np.str_('obs[7]'), np.str_('obs[8]'), np.str_('obs[9]'), np.str_('obs[10]'), np.str_('obs[11]'), np.str_('obs[12]'), np.str_('obs[13]'), np.str_('obs[14]'), np.str_('obs[15]'), np.str_('obs[16]'), np.str_('obs[17]'), np.str_('obs[18]'), np.str_('obs[19]'), np.str_('target.target_0.priority'), np.str_('target.target_0.r_LB_H_normd[0]'), np.str_('target.target_0.r_LB_H_normd[1]'), np.str_('target.target_0.r_LB_H_normd[2]'), np.str_('target.target_0.target_angle_normd'), np.str_('target.target_0.target_angle_rate_normd'), np.str_('target.target_0.opportunity_open_normd'), np.str_('target.target_0.opportunity_close_normd'), np.str_('target.target_0.cloud_forecast'), np.str_('target.target_0.belief'), np.str_('target.target_0.time_since_prev_obs_normd'), np.str_('target.target_0.belief_expected'), np.str_('target.target_1.priority'), np.str_('target.target_1.r_LB_H_normd[0]'), np.str_('target.target_1.r_LB_H_normd[1]'), np.str_('target.target_1.r_LB_H_normd[2]'), np.str_('target.target_1.target_angle_normd'), np.str_('target.target_1.target_angle_rate_normd'), np.str_('target.target_1.opportunity_open_normd'), np.str_('target.target_1.opportunity_close_normd'), np.str_('target.target_1.cloud_forecast'), np.str_('target.target_1.belief'), np.str_('target.target_1.time_since_prev_obs_normd'), np.str_('target.target_1.belief_expected'), np.str_('target.target_2.priority'), np.str_('target.target_2.r_LB_H_normd[0]'), np.str_('target.target_2.r_LB_H_normd[1]'), np.str_('target.target_2.r_LB_H_normd[2]'), np.str_('target.target_2.target_angle_normd'), np.str_('target.target_2.target_angle_rate_normd'), np.str_('target.target_2.opportunity_open_normd'), np.str_('target.target_2.opportunity_close_normd'), np.str_('target.target_2.cloud_forecast'), np.str_('target.target_2.belief'), np.str_('target.target_2.time_since_prev_obs_normd'), np.str_('target.target_2.belief_expected'), np.str_('target.target_3.priority'), np.str_('target.target_3.r_LB_H_normd[0]'), np.str_('target.target_3.r_LB_H_normd[1]'), np.str_('target.target_3.r_LB_H_normd[2]'), np.str_('target.target_3.target_angle_normd'), np.str_('target.target_3.target_angle_rate_normd'), np.str_('target.target_3.opportunity_open_normd'), np.str_('target.target_3.opportunity_close_normd'), np.str_('target.target_3.cloud_forecast'), np.str_('target.target_3.belief'), np.str_('target.target_3.time_since_prev_obs_normd'), np.str_('target.target_3.belief_expected'), np.str_('target.target_4.priority'), np.str_('target.target_4.r_LB_H_normd[0]'), np.str_('target.target_4.r_LB_H_normd[1]'), np.str_('target.target_4.r_LB_H_normd[2]'), np.str_('target.target_4.target_angle_normd'), np.str_('target.target_4.target_angle_rate_normd'), np.str_('target.target_4.opportunity_open_normd'), np.str_('target.target_4.opportunity_close_normd'), np.str_('target.target_4.cloud_forecast'), np.str_('target.target_4.belief'), np.str_('target.target_4.time_since_prev_obs_normd'), np.str_('target.target_4.belief_expected'), np.str_('target.target_5.priority'), np.str_('target.target_5.r_LB_H_normd[0]'), np.str_('target.target_5.r_LB_H_normd[1]'), np.str_('target.target_5.r_LB_H_normd[2]'), np.str_('target.target_5.target_angle_normd'), np.str_('target.target_5.target_angle_rate_normd'), np.str_('target.target_5.opportunity_open_normd'), np.str_('target.target_5.opportunity_close_normd'), np.str_('target.target_5.cloud_forecast'), np.str_('target.target_5.belief'), np.str_('target.target_5.time_since_prev_obs_normd'), np.str_('target.target_5.belief_expected'), np.str_('target.target_6.priority'), np.str_('target.target_6.r_LB_H_normd[0]'), np.str_('target.target_6.r_LB_H_normd[1]'), np.str_('target.target_6.r_LB_H_normd[2]'), np.str_('target.target_6.target_angle_normd'), np.str_('target.target_6.target_angle_rate_normd'), np.str_('target.target_6.opportunity_open_normd'), np.str_('target.target_6.opportunity_close_normd'), np.str_('target.target_6.cloud_forecast'), np.str_('target.target_6.belief'), np.str_('target.target_6.time_since_prev_obs_normd'), np.str_('target.target_6.belief_expected'), np.str_('target.target_7.priority'), np.str_('target.target_7.r_LB_H_normd[0]'), np.str_('target.target_7.r_LB_H_normd[1]'), np.str_('target.target_7.r_LB_H_normd[2]'), np.str_('target.target_7.target_angle_normd'), np.str_('target.target_7.target_angle_rate_normd'), np.str_('target.target_7.opportunity_open_normd'), np.str_('target.target_7.opportunity_close_normd'), np.str_('target.target_7.cloud_forecast'), np.str_('target.target_7.belief'), np.str_('target.target_7.time_since_prev_obs_normd'), np.str_('target.target_7.belief_expected'), np.str_('target.target_8.priority'), np.str_('target.target_8.r_LB_H_normd[0]'), np.str_('target.target_8.r_LB_H_normd[1]'), np.str_('target.target_8.r_LB_H_normd[2]'), np.str_('target.target_8.target_angle_normd'), np.str_('target.target_8.target_angle_rate_normd'), np.str_('target.target_8.opportunity_open_normd'), np.str_('target.target_8.opportunity_close_normd'), np.str_('target.target_8.cloud_forecast'), np.str_('target.target_8.belief'), np.str_('target.target_8.time_since_prev_obs_normd'), np.str_('target.target_8.belief_expected'), np.str_('target.target_9.priority'), np.str_('target.target_9.r_LB_H_normd[0]'), np.str_('target.target_9.r_LB_H_normd[1]'), np.str_('target.target_9.r_LB_H_normd[2]'), np.str_('target.target_9.target_angle_normd'), np.str_('target.target_9.target_angle_rate_normd'), np.str_('target.target_9.opportunity_open_normd'), np.str_('target.target_9.opportunity_close_normd'), np.str_('target.target_9.cloud_forecast'), np.str_('target.target_9.belief'), np.str_('target.target_9.time_since_prev_obs_normd'), np.str_('target.target_9.belief_expected'), np.str_('target.target_10.priority'), np.str_('target.target_10.r_LB_H_normd[0]'), np.str_('target.target_10.r_LB_H_normd[1]'), np.str_('target.target_10.r_LB_H_normd[2]'), np.str_('target.target_10.target_angle_normd'), np.str_('target.target_10.target_angle_rate_normd'), np.str_('target.target_10.opportunity_open_normd'), np.str_('target.target_10.opportunity_close_normd'), np.str_('target.target_10.cloud_forecast'), np.str_('target.target_10.belief'), np.str_('target.target_10.time_since_prev_obs_normd'), np.str_('target.target_10.belief_expected'), np.str_('target.target_11.priority'), np.str_('target.target_11.r_LB_H_normd[0]'), np.str_('target.target_11.r_LB_H_normd[1]'), np.str_('target.target_11.r_LB_H_normd[2]'), np.str_('target.target_11.target_angle_normd'), np.str_('target.target_11.target_angle_rate_normd'), np.str_('target.target_11.opportunity_open_normd'), np.str_('target.target_11.opportunity_close_normd'), np.str_('target.target_11.cloud_forecast'), np.str_('target.target_11.belief'), np.str_('target.target_11.time_since_prev_obs_normd'), np.str_('target.target_11.belief_expected'), np.str_('target.target_12.priority'), np.str_('target.target_12.r_LB_H_normd[0]'), np.str_('target.target_12.r_LB_H_normd[1]'), np.str_('target.target_12.r_LB_H_normd[2]'), np.str_('target.target_12.target_angle_normd'), np.str_('target.target_12.target_angle_rate_normd'), np.str_('target.target_12.opportunity_open_normd'), np.str_('target.target_12.opportunity_close_normd'), np.str_('target.target_12.cloud_forecast'), np.str_('target.target_12.belief'), np.str_('target.target_12.time_since_prev_obs_normd'), np.str_('target.target_12.belief_expected'), np.str_('target.target_13.priority'), np.str_('target.target_13.r_LB_H_normd[0]'), np.str_('target.target_13.r_LB_H_normd[1]'), np.str_('target.target_13.r_LB_H_normd[2]'), np.str_('target.target_13.target_angle_normd'), np.str_('target.target_13.target_angle_rate_normd'), np.str_('target.target_13.opportunity_open_normd'), np.str_('target.target_13.opportunity_close_normd'), np.str_('target.target_13.cloud_forecast'), np.str_('target.target_13.belief'), np.str_('target.target_13.time_since_prev_obs_normd'), np.str_('target.target_13.belief_expected'), np.str_('target.target_14.priority'), np.str_('target.target_14.r_LB_H_normd[0]'), np.str_('target.target_14.r_LB_H_normd[1]'), np.str_('target.target_14.r_LB_H_normd[2]'), np.str_('target.target_14.target_angle_normd'), np.str_('target.target_14.target_angle_rate_normd'), np.str_('target.target_14.opportunity_open_normd'), np.str_('target.target_14.opportunity_close_normd'), np.str_('target.target_14.cloud_forecast'), np.str_('target.target_14.belief'), np.str_('target.target_14.time_since_prev_obs_normd'), np.str_('target.target_14.belief_expected'), np.str_('target.target_15.priority'), np.str_('target.target_15.r_LB_H_normd[0]'), np.str_('target.target_15.r_LB_H_normd[1]'), np.str_('target.target_15.r_LB_H_normd[2]'), np.str_('target.target_15.target_angle_normd'), np.str_('target.target_15.target_angle_rate_normd'), np.str_('target.target_15.opportunity_open_normd'), np.str_('target.target_15.opportunity_close_normd'), np.str_('target.target_15.cloud_forecast'), np.str_('target.target_15.belief'), np.str_('target.target_15.time_since_prev_obs_normd'), np.str_('target.target_15.belief_expected'), np.str_('target.target_16.priority'), np.str_('target.target_16.r_LB_H_normd[0]'), np.str_('target.target_16.r_LB_H_normd[1]'), np.str_('target.target_16.r_LB_H_normd[2]'), np.str_('target.target_16.target_angle_normd'), np.str_('target.target_16.target_angle_rate_normd'), np.str_('target.target_16.opportunity_open_normd'), np.str_('target.target_16.opportunity_close_normd'), np.str_('target.target_16.cloud_forecast'), np.str_('target.target_16.belief'), np.str_('target.target_16.time_since_prev_obs_normd'), np.str_('target.target_16.belief_expected'), np.str_('target.target_17.priority'), np.str_('target.target_17.r_LB_H_normd[0]'), np.str_('target.target_17.r_LB_H_normd[1]'), np.str_('target.target_17.r_LB_H_normd[2]'), np.str_('target.target_17.target_angle_normd'), np.str_('target.target_17.target_angle_rate_normd'), np.str_('target.target_17.opportunity_open_normd'), np.str_('target.target_17.opportunity_close_normd'), np.str_('target.target_17.cloud_forecast'), np.str_('target.target_17.belief'), np.str_('target.target_17.time_since_prev_obs_normd'), np.str_('target.target_17.belief_expected'), np.str_('target.target_18.priority'), np.str_('target.target_18.r_LB_H_normd[0]'), np.str_('target.target_18.r_LB_H_normd[1]'), np.str_('target.target_18.r_LB_H_normd[2]'), np.str_('target.target_18.target_angle_normd'), np.str_('target.target_18.target_angle_rate_normd'), np.str_('target.target_18.opportunity_open_normd'), np.str_('target.target_18.opportunity_close_normd'), np.str_('target.target_18.cloud_forecast'), np.str_('target.target_18.belief'), np.str_('target.target_18.time_since_prev_obs_normd'), np.str_('target.target_18.belief_expected'), np.str_('target.target_19.priority'), np.str_('target.target_19.r_LB_H_normd[0]'), np.str_('target.target_19.r_LB_H_normd[1]'), np.str_('target.target_19.r_LB_H_normd[2]'), np.str_('target.target_19.target_angle_normd'), np.str_('target.target_19.target_angle_rate_normd'), np.str_('target.target_19.opportunity_open_normd'), np.str_('target.target_19.opportunity_close_normd'), np.str_('target.target_19.cloud_forecast'), np.str_('target.target_19.belief'), np.str_('target.target_19.time_since_prev_obs_normd'), np.str_('target.target_19.belief_expected'), np.str_('target.target_20.priority'), np.str_('target.target_20.r_LB_H_normd[0]'), np.str_('target.target_20.r_LB_H_normd[1]'), np.str_('target.target_20.r_LB_H_normd[2]'), np.str_('target.target_20.target_angle_normd'), np.str_('target.target_20.target_angle_rate_normd'), np.str_('target.target_20.opportunity_open_normd'), np.str_('target.target_20.opportunity_close_normd'), np.str_('target.target_20.cloud_forecast'), np.str_('target.target_20.belief'), np.str_('target.target_20.time_since_prev_obs_normd'), np.str_('target.target_20.belief_expected'), np.str_('target.target_21.priority'), np.str_('target.target_21.r_LB_H_normd[0]'), np.str_('target.target_21.r_LB_H_normd[1]'), np.str_('target.target_21.r_LB_H_normd[2]'), np.str_('target.target_21.target_angle_normd'), np.str_('target.target_21.target_angle_rate_normd'), np.str_('target.target_21.opportunity_open_normd'), np.str_('target.target_21.opportunity_close_normd'), np.str_('target.target_21.cloud_forecast'), np.str_('target.target_21.belief'), np.str_('target.target_21.time_since_prev_obs_normd'), np.str_('target.target_21.belief_expected'), np.str_('target.target_22.priority'), np.str_('target.target_22.r_LB_H_normd[0]'), np.str_('target.target_22.r_LB_H_normd[1]'), np.str_('target.target_22.r_LB_H_normd[2]'), np.str_('target.target_22.target_angle_normd'), np.str_('target.target_22.target_angle_rate_normd'), np.str_('target.target_22.opportunity_open_normd'), np.str_('target.target_22.opportunity_close_normd'), np.str_('target.target_22.cloud_forecast'), np.str_('target.target_22.belief'), np.str_('target.target_22.time_since_prev_obs_normd'), np.str_('target.target_22.belief_expected'), np.str_('target.target_23.priority'), np.str_('target.target_23.r_LB_H_normd[0]'), np.str_('target.target_23.r_LB_H_normd[1]'), np.str_('target.target_23.r_LB_H_normd[2]'), np.str_('target.target_23.target_angle_normd'), np.str_('target.target_23.target_angle_rate_normd'), np.str_('target.target_23.opportunity_open_normd'), np.str_('target.target_23.opportunity_close_normd'), np.str_('target.target_23.cloud_forecast'), np.str_('target.target_23.belief'), np.str_('target.target_23.time_since_prev_obs_normd'), np.str_('target.target_23.belief_expected'), np.str_('target.target_24.priority'), np.str_('target.target_24.r_LB_H_normd[0]'), np.str_('target.target_24.r_LB_H_normd[1]'), np.str_('target.target_24.r_LB_H_normd[2]'), np.str_('target.target_24.target_angle_normd'), np.str_('target.target_24.target_angle_rate_normd'), np.str_('target.target_24.opportunity_open_normd'), np.str_('target.target_24.opportunity_close_normd'), np.str_('target.target_24.cloud_forecast'), np.str_('target.target_24.belief'), np.str_('target.target_24.time_since_prev_obs_normd'), np.str_('target.target_24.belief_expected'), np.str_('target.target_25.priority'), np.str_('target.target_25.r_LB_H_normd[0]'), np.str_('target.target_25.r_LB_H_normd[1]'), np.str_('target.target_25.r_LB_H_normd[2]'), np.str_('target.target_25.target_angle_normd'), np.str_('target.target_25.target_angle_rate_normd'), np.str_('target.target_25.opportunity_open_normd'), np.str_('target.target_25.opportunity_close_normd'), np.str_('target.target_25.cloud_forecast'), np.str_('target.target_25.belief'), np.str_('target.target_25.time_since_prev_obs_normd'), np.str_('target.target_25.belief_expected'), np.str_('target.target_26.priority'), np.str_('target.target_26.r_LB_H_normd[0]'), np.str_('target.target_26.r_LB_H_normd[1]'), np.str_('target.target_26.r_LB_H_normd[2]'), np.str_('target.target_26.target_angle_normd'), np.str_('target.target_26.target_angle_rate_normd'), np.str_('target.target_26.opportunity_open_normd'), np.str_('target.target_26.opportunity_close_normd'), np.str_('target.target_26.cloud_forecast'), np.str_('target.target_26.belief'), np.str_('target.target_26.time_since_prev_obs_normd'), np.str_('target.target_26.belief_expected'), np.str_('target.target_27.priority'), np.str_('target.target_27.r_LB_H_normd[0]'), np.str_('target.target_27.r_LB_H_normd[1]'), np.str_('target.target_27.r_LB_H_normd[2]'), np.str_('target.target_27.target_angle_normd'), np.str_('target.target_27.target_angle_rate_normd'), np.str_('target.target_27.opportunity_open_normd'), np.str_('target.target_27.opportunity_close_normd'), np.str_('target.target_27.cloud_forecast'), np.str_('target.target_27.belief'), np.str_('target.target_27.time_since_prev_obs_normd'), np.str_('target.target_27.belief_expected'), np.str_('target.target_28.priority'), np.str_('target.target_28.r_LB_H_normd[0]'), np.str_('target.target_28.r_LB_H_normd[1]'), np.str_('target.target_28.r_LB_H_normd[2]'), np.str_('target.target_28.target_angle_normd'), np.str_('target.target_28.target_angle_rate_normd'), np.str_('target.target_28.opportunity_open_normd'), np.str_('target.target_28.opportunity_close_normd'), np.str_('target.target_28.cloud_forecast'), np.str_('target.target_28.belief'), np.str_('target.target_28.time_since_prev_obs_normd'), np.str_('target.target_28.belief_expected'), np.str_('target.target_29.priority'), np.str_('target.target_29.r_LB_H_normd[0]'), np.str_('target.target_29.r_LB_H_normd[1]'), np.str_('target.target_29.r_LB_H_normd[2]'), np.str_('target.target_29.target_angle_normd'), np.str_('target.target_29.target_angle_rate_normd'), np.str_('target.target_29.opportunity_open_normd'), np.str_('target.target_29.opportunity_close_normd'), np.str_('target.target_29.cloud_forecast'), np.str_('target.target_29.belief'), np.str_('target.target_29.time_since_prev_obs_normd'), np.str_('target.target_29.belief_expected'), np.str_('target.target_30.priority'), np.str_('target.target_30.r_LB_H_normd[0]'), np.str_('target.target_30.r_LB_H_normd[1]'), np.str_('target.target_30.r_LB_H_normd[2]'), np.str_('target.target_30.target_angle_normd'), np.str_('target.target_30.target_angle_rate_normd'), np.str_('target.target_30.opportunity_open_normd'), np.str_('target.target_30.opportunity_close_normd'), np.str_('target.target_30.cloud_forecast'), np.str_('target.target_30.belief'), np.str_('target.target_30.time_since_prev_obs_normd'), np.str_('target.target_30.belief_expected'), np.str_('target.target_31.priority'), np.str_('target.target_31.r_LB_H_normd[0]'), np.str_('target.target_31.r_LB_H_normd[1]'), np.str_('target.target_31.r_LB_H_normd[2]'), np.str_('target.target_31.target_angle_normd'), np.str_('target.target_31.target_angle_rate_normd'), np.str_('target.target_31.opportunity_open_normd'), np.str_('target.target_31.opportunity_close_normd'), np.str_('target.target_31.cloud_forecast'), np.str_('target.target_31.belief'), np.str_('target.target_31.time_since_prev_obs_normd'), np.str_('target.target_31.belief_expected'), np.str_('target.target_32.priority'), np.str_('target.target_32.r_LB_H_normd[0]'), np.str_('target.target_32.r_LB_H_normd[1]'), np.str_('target.target_32.r_LB_H_normd[2]'), np.str_('target.target_32.target_angle_normd'), np.str_('target.target_32.target_angle_rate_normd'), np.str_('target.target_32.opportunity_open_normd'), np.str_('target.target_32.opportunity_close_normd'), np.str_('target.target_32.cloud_forecast'), np.str_('target.target_32.belief'), np.str_('target.target_32.time_since_prev_obs_normd'), np.str_('target.target_32.belief_expected'), np.str_('target.target_33.priority'), np.str_('target.target_33.r_LB_H_normd[0]'), np.str_('target.target_33.r_LB_H_normd[1]'), np.str_('target.target_33.r_LB_H_normd[2]'), np.str_('target.target_33.target_angle_normd'), np.str_('target.target_33.target_angle_rate_normd'), np.str_('target.target_33.opportunity_open_normd'), np.str_('target.target_33.opportunity_close_normd'), np.str_('target.target_33.cloud_forecast'), np.str_('target.target_33.belief'), np.str_('target.target_33.time_since_prev_obs_normd'), np.str_('target.target_33.belief_expected'), np.str_('target.target_34.priority'), np.str_('target.target_34.r_LB_H_normd[0]'), np.str_('target.target_34.r_LB_H_normd[1]'), np.str_('target.target_34.r_LB_H_normd[2]'), np.str_('target.target_34.target_angle_normd'), np.str_('target.target_34.target_angle_rate_normd'), np.str_('target.target_34.opportunity_open_normd'), np.str_('target.target_34.opportunity_close_normd'), np.str_('target.target_34.cloud_forecast'), np.str_('target.target_34.belief'), np.str_('target.target_34.time_since_prev_obs_normd'), np.str_('target.target_34.belief_expected'), np.str_('target.target_35.priority'), np.str_('target.target_35.r_LB_H_normd[0]'), np.str_('target.target_35.r_LB_H_normd[1]'), np.str_('target.target_35.r_LB_H_normd[2]'), np.str_('target.target_35.target_angle_normd'), np.str_('target.target_35.target_angle_rate_normd'), np.str_('target.target_35.opportunity_open_normd'), np.str_('target.target_35.opportunity_close_normd'), np.str_('target.target_35.cloud_forecast'), np.str_('target.target_35.belief'), np.str_('target.target_35.time_since_prev_obs_normd'), np.str_('target.target_35.belief_expected'), np.str_('target.target_36.priority'), np.str_('target.target_36.r_LB_H_normd[0]'), np.str_('target.target_36.r_LB_H_normd[1]'), np.str_('target.target_36.r_LB_H_normd[2]'), np.str_('target.target_36.target_angle_normd'), np.str_('target.target_36.target_angle_rate_normd'), np.str_('target.target_36.opportunity_open_normd'), np.str_('target.target_36.opportunity_close_normd'), np.str_('target.target_36.cloud_forecast'), np.str_('target.target_36.belief'), np.str_('target.target_36.time_since_prev_obs_normd'), np.str_('target.target_36.belief_expected'), np.str_('target.target_37.priority'), np.str_('target.target_37.r_LB_H_normd[0]'), np.str_('target.target_37.r_LB_H_normd[1]'), np.str_('target.target_37.r_LB_H_normd[2]'), np.str_('target.target_37.target_angle_normd'), np.str_('target.target_37.target_angle_rate_normd'), np.str_('target.target_37.opportunity_open_normd'), np.str_('target.target_37.opportunity_close_normd'), np.str_('target.target_37.cloud_forecast'), np.str_('target.target_37.belief'), np.str_('target.target_37.time_since_prev_obs_normd'), np.str_('target.target_37.belief_expected'), np.str_('target.target_38.priority'), np.str_('target.target_38.r_LB_H_normd[0]'), np.str_('target.target_38.r_LB_H_normd[1]'), np.str_('target.target_38.r_LB_H_normd[2]'), np.str_('target.target_38.target_angle_normd'), np.str_('target.target_38.target_angle_rate_normd'), np.str_('target.target_38.opportunity_open_normd'), np.str_('target.target_38.opportunity_close_normd'), np.str_('target.target_38.cloud_forecast'), np.str_('target.target_38.belief'), np.str_('target.target_38.time_since_prev_obs_normd'), np.str_('target.target_38.belief_expected'), np.str_('target.target_39.priority'), np.str_('target.target_39.r_LB_H_normd[0]'), np.str_('target.target_39.r_LB_H_normd[1]'), np.str_('target.target_39.r_LB_H_normd[2]'), np.str_('target.target_39.target_angle_normd'), np.str_('target.target_39.target_angle_rate_normd'), np.str_('target.target_39.opportunity_open_normd'), np.str_('target.target_39.opportunity_close_normd'), np.str_('target.target_39.cloud_forecast'), np.str_('target.target_39.belief'), np.str_('target.target_39.time_since_prev_obs_normd'), np.str_('target.target_39.belief_expected')]
Then, run the simulation until timeout or agent failure using the heuristic aforementioned.
[14]:
steps = 0
while True:
if steps == 0:
# Vector with an action for each satellite (we can pass different actions for each satellite)
# Tasking all satellites to charge (tasking None as the first action will raise a warning)
action_dict = {sat_i.name: 0 for sat_i in env.satellites}
else:
# Using the greedy heuristic to select the best target for each satellite based on the current observation
action_dict = {
sat_i.name: greedy_heuristic_reimaging(sat_i, 0.0, 35)
for sat_i in env.satellites
}
steps += 1
observation, reward, terminated, truncated, info = env.step(action_dict)
if all(terminated.values()) or all(truncated.values()) or steps >= 5:
print("Episode complete.")
break
2026-07-28 23:38:36,216 gym INFO <0.00> === STARTING STEP ===
2026-07-28 23:38:36,217 sats.satellite.EO-0 INFO <0.00> EO-0: action_charge tasked for 60.0 seconds
2026-07-28 23:38:36,217 sats.satellite.EO-0 INFO <0.00> EO-0: setting timed terminal event at 60.0
2026-07-28 23:38:36,218 sats.satellite.EO-1 INFO <0.00> EO-1: action_charge tasked for 60.0 seconds
2026-07-28 23:38:36,219 sats.satellite.EO-1 INFO <0.00> EO-1: setting timed terminal event at 60.0
2026-07-28 23:38:36,220 sats.satellite.EO-2 INFO <0.00> EO-2: action_charge tasked for 60.0 seconds
2026-07-28 23:38:36,220 sats.satellite.EO-2 INFO <0.00> EO-2: setting timed terminal event at 60.0
2026-07-28 23:38:36,221 sats.satellite.EO-3 INFO <0.00> EO-3: action_charge tasked for 60.0 seconds
2026-07-28 23:38:36,222 sats.satellite.EO-3 INFO <0.00> EO-3: setting timed terminal event at 60.0
2026-07-28 23:38:36,223 sats.satellite.EO-4 INFO <0.00> EO-4: action_charge tasked for 60.0 seconds
2026-07-28 23:38:36,223 sats.satellite.EO-4 INFO <0.00> EO-4: setting timed terminal event at 60.0
2026-07-28 23:38:36,243 sats.satellite.EO-0 INFO <60.00> EO-0: timed termination at 60.0 for action_charge
2026-07-28 23:38:36,244 sats.satellite.EO-1 INFO <60.00> EO-1: timed termination at 60.0 for action_charge
2026-07-28 23:38:36,244 sats.satellite.EO-2 INFO <60.00> EO-2: timed termination at 60.0 for action_charge
2026-07-28 23:38:36,245 sats.satellite.EO-3 INFO <60.00> EO-3: timed termination at 60.0 for action_charge
2026-07-28 23:38:36,245 sats.satellite.EO-4 INFO <60.00> EO-4: timed termination at 60.0 for action_charge
2026-07-28 23:38:36,248 data.base INFO <60.00> Total reward: {}
2026-07-28 23:38:36,249 sats.satellite.EO-0 INFO <60.00> EO-0: Satellite EO-0 requires retasking
2026-07-28 23:38:36,249 sats.satellite.EO-1 INFO <60.00> EO-1: Satellite EO-1 requires retasking
2026-07-28 23:38:36,250 sats.satellite.EO-2 INFO <60.00> EO-2: Satellite EO-2 requires retasking
2026-07-28 23:38:36,250 sats.satellite.EO-3 INFO <60.00> EO-3: Satellite EO-3 requires retasking
2026-07-28 23:38:36,251 sats.satellite.EO-4 INFO <60.00> EO-4: Satellite EO-4 requires retasking
2026-07-28 23:38:36,316 gym INFO <60.00> Step reward: {}
2026-07-28 23:38:36,335 gym INFO <60.00> === STARTING STEP ===
2026-07-28 23:38:36,336 sats.satellite.EO-0 INFO <60.00> EO-0: target index 22 tasked
2026-07-28 23:38:36,336 sats.satellite.EO-0 INFO <60.00> EO-0: Target(tgt-2996) tasked for imaging
2026-07-28 23:38:36,337 sats.satellite.EO-0 INFO <60.00> EO-0: Target(tgt-2996) window enabled: 221.3 to 334.9
2026-07-28 23:38:36,337 sats.satellite.EO-0 INFO <60.00> EO-0: setting timed terminal event at 334.9
2026-07-28 23:38:36,338 sats.satellite.EO-1 INFO <60.00> EO-1: target index 30 tasked
2026-07-28 23:38:36,339 sats.satellite.EO-1 INFO <60.00> EO-1: Target(tgt-3666) tasked for imaging
2026-07-28 23:38:36,339 sats.satellite.EO-1 INFO <60.00> EO-1: Target(tgt-3666) window enabled: 162.8 to 288.1
2026-07-28 23:38:36,340 sats.satellite.EO-1 INFO <60.00> EO-1: setting timed terminal event at 288.1
2026-07-28 23:38:36,341 sats.satellite.EO-2 INFO <60.00> EO-2: target index 14 tasked
2026-07-28 23:38:36,341 sats.satellite.EO-2 INFO <60.00> EO-2: Target(tgt-1296) tasked for imaging
2026-07-28 23:38:36,342 sats.satellite.EO-2 INFO <60.00> EO-2: Target(tgt-1296) window enabled: 106.5 to 224.1
2026-07-28 23:38:36,342 sats.satellite.EO-2 INFO <60.00> EO-2: setting timed terminal event at 224.1
2026-07-28 23:38:36,343 sats.satellite.EO-3 INFO <60.00> EO-3: target index 8 tasked
2026-07-28 23:38:36,344 sats.satellite.EO-3 INFO <60.00> EO-3: Target(tgt-4446) tasked for imaging
2026-07-28 23:38:36,344 sats.satellite.EO-3 INFO <60.00> EO-3: Target(tgt-4446) window enabled: 25.4 to 137.2
2026-07-28 23:38:36,345 sats.satellite.EO-3 INFO <60.00> EO-3: setting timed terminal event at 137.2
2026-07-28 23:38:36,345 sats.satellite.EO-4 INFO <60.00> EO-4: target index 34 tasked
2026-07-28 23:38:36,346 sats.satellite.EO-4 INFO <60.00> EO-4: Target(tgt-9448) tasked for imaging
2026-07-28 23:38:36,347 sats.satellite.EO-4 INFO <60.00> EO-4: Target(tgt-9448) window enabled: 167.3 to 280.0
2026-07-28 23:38:36,347 sats.satellite.EO-4 INFO <60.00> EO-4: setting timed terminal event at 280.0
2026-07-28 23:38:36,384 sats.satellite.EO-3 INFO <131.50> EO-3: imaged Target(tgt-4446)
2026-07-28 23:38:36,387 data.base INFO <131.50> Total reward: {'EO-3': np.float64(0.017923627535731398)}
2026-07-28 23:38:36,387 sats.satellite.EO-3 INFO <131.50> EO-3: Satellite EO-3 requires retasking
2026-07-28 23:38:36,450 gym INFO <131.50> Step reward: {'EO-3': np.float64(0.017923627535731398)}
2026-07-28 23:38:36,469 gym INFO <131.50> === STARTING STEP ===
2026-07-28 23:38:36,469 sats.satellite.EO-0 INFO <131.50> EO-0: target index 14 tasked
2026-07-28 23:38:36,470 sats.satellite.EO-0 INFO <131.50> EO-0: Target(tgt-2996) window enabled: 221.3 to 334.9
2026-07-28 23:38:36,470 sats.satellite.EO-0 INFO <131.50> EO-0: setting timed terminal event at 334.9
2026-07-28 23:38:36,471 sats.satellite.EO-1 INFO <131.50> EO-1: target index 22 tasked
2026-07-28 23:38:36,472 sats.satellite.EO-1 INFO <131.50> EO-1: Target(tgt-3666) window enabled: 162.8 to 288.1
2026-07-28 23:38:36,472 sats.satellite.EO-1 INFO <131.50> EO-1: setting timed terminal event at 288.1
2026-07-28 23:38:36,473 sats.satellite.EO-2 INFO <131.50> EO-2: target index 10 tasked
2026-07-28 23:38:36,474 sats.satellite.EO-2 INFO <131.50> EO-2: Target(tgt-1296) window enabled: 106.5 to 224.1
2026-07-28 23:38:36,474 sats.satellite.EO-2 INFO <131.50> EO-2: setting timed terminal event at 224.1
2026-07-28 23:38:36,476 sats.satellite.EO-3 INFO <131.50> EO-3: target index 0 tasked
2026-07-28 23:38:36,476 sats.satellite.EO-3 INFO <131.50> EO-3: Target(tgt-595) tasked for imaging
2026-07-28 23:38:36,477 sats.satellite.EO-3 INFO <131.50> EO-3: Target(tgt-595) window enabled: 46.8 to 151.7
2026-07-28 23:38:36,477 sats.satellite.EO-3 INFO <131.50> EO-3: setting timed terminal event at 151.7
2026-07-28 23:38:36,478 sats.satellite.EO-4 INFO <131.50> EO-4: target index 24 tasked
2026-07-28 23:38:36,479 sats.satellite.EO-4 INFO <131.50> EO-4: Target(tgt-9448) window enabled: 167.3 to 280.0
2026-07-28 23:38:36,479 sats.satellite.EO-4 INFO <131.50> EO-4: setting timed terminal event at 280.0
2026-07-28 23:38:36,491 sats.satellite.EO-3 INFO <152.00> EO-3: timed termination at 151.7 for Target(tgt-595) window
2026-07-28 23:38:36,494 data.base INFO <152.00> Total reward: {}
2026-07-28 23:38:36,494 sats.satellite.EO-3 INFO <152.00> EO-3: Satellite EO-3 requires retasking
2026-07-28 23:38:36,557 gym INFO <152.00> Step reward: {}
2026-07-28 23:38:36,576 gym INFO <152.00> === STARTING STEP ===
2026-07-28 23:38:36,576 sats.satellite.EO-0 INFO <152.00> EO-0: target index 13 tasked
2026-07-28 23:38:36,577 sats.satellite.EO-0 INFO <152.00> EO-0: Target(tgt-2996) window enabled: 221.3 to 334.9
2026-07-28 23:38:36,577 sats.satellite.EO-0 INFO <152.00> EO-0: setting timed terminal event at 334.9
2026-07-28 23:38:36,578 sats.satellite.EO-1 INFO <152.00> EO-1: target index 19 tasked
2026-07-28 23:38:36,578 sats.satellite.EO-1 INFO <152.00> EO-1: Target(tgt-3666) window enabled: 162.8 to 288.1
2026-07-28 23:38:36,579 sats.satellite.EO-1 INFO <152.00> EO-1: setting timed terminal event at 288.1
2026-07-28 23:38:36,579 sats.satellite.EO-2 INFO <152.00> EO-2: target index 6 tasked
2026-07-28 23:38:36,580 sats.satellite.EO-2 INFO <152.00> EO-2: Target(tgt-1296) window enabled: 106.5 to 224.1
2026-07-28 23:38:36,580 sats.satellite.EO-2 INFO <152.00> EO-2: setting timed terminal event at 224.1
2026-07-28 23:38:36,581 sats.satellite.EO-3 INFO <152.00> EO-3: target index 18 tasked
2026-07-28 23:38:36,581 sats.satellite.EO-3 INFO <152.00> EO-3: Target(tgt-8572) tasked for imaging
2026-07-28 23:38:36,582 sats.satellite.EO-3 INFO <152.00> EO-3: Target(tgt-8572) window enabled: 196.9 to 263.9
2026-07-28 23:38:36,582 sats.satellite.EO-3 INFO <152.00> EO-3: setting timed terminal event at 263.9
2026-07-28 23:38:36,583 sats.satellite.EO-4 INFO <152.00> EO-4: target index 23 tasked
2026-07-28 23:38:36,583 sats.satellite.EO-4 INFO <152.00> EO-4: Target(tgt-9448) window enabled: 167.3 to 280.0
2026-07-28 23:38:36,584 sats.satellite.EO-4 INFO <152.00> EO-4: setting timed terminal event at 280.0
2026-07-28 23:38:36,588 sats.satellite.EO-2 INFO <156.00> EO-2: imaged Target(tgt-1296)
2026-07-28 23:38:36,591 data.base INFO <156.00> Total reward: {'EO-2': np.float64(0.15950168707407258)}
2026-07-28 23:38:36,591 sats.satellite.EO-2 INFO <156.00> EO-2: Satellite EO-2 requires retasking
2026-07-28 23:38:36,651 gym INFO <156.00> Step reward: {'EO-2': np.float64(0.15950168707407258)}
2026-07-28 23:38:36,670 gym INFO <156.00> === STARTING STEP ===
2026-07-28 23:38:36,671 sats.satellite.EO-0 INFO <156.00> EO-0: target index 12 tasked
2026-07-28 23:38:36,671 sats.satellite.EO-0 INFO <156.00> EO-0: Target(tgt-2996) window enabled: 221.3 to 334.9
2026-07-28 23:38:36,672 sats.satellite.EO-0 INFO <156.00> EO-0: setting timed terminal event at 334.9
2026-07-28 23:38:36,673 sats.satellite.EO-1 INFO <156.00> EO-1: target index 17 tasked
2026-07-28 23:38:36,673 sats.satellite.EO-1 INFO <156.00> EO-1: Target(tgt-3666) window enabled: 162.8 to 288.1
2026-07-28 23:38:36,674 sats.satellite.EO-1 INFO <156.00> EO-1: setting timed terminal event at 288.1
2026-07-28 23:38:36,674 sats.satellite.EO-2 INFO <156.00> EO-2: target index 5 tasked
2026-07-28 23:38:36,675 sats.satellite.EO-2 INFO <156.00> EO-2: Target(tgt-5526) tasked for imaging
2026-07-28 23:38:36,676 sats.satellite.EO-2 INFO <156.00> EO-2: Target(tgt-5526) window enabled: 108.1 to 231.3
2026-07-28 23:38:36,676 sats.satellite.EO-2 INFO <156.00> EO-2: setting timed terminal event at 231.3
2026-07-28 23:38:36,677 sats.satellite.EO-3 INFO <156.00> EO-3: target index 17 tasked
2026-07-28 23:38:36,678 sats.satellite.EO-3 INFO <156.00> EO-3: Target(tgt-8572) window enabled: 196.9 to 263.9
2026-07-28 23:38:36,678 sats.satellite.EO-3 INFO <156.00> EO-3: setting timed terminal event at 263.9
2026-07-28 23:38:36,679 sats.satellite.EO-4 INFO <156.00> EO-4: target index 23 tasked
2026-07-28 23:38:36,680 sats.satellite.EO-4 INFO <156.00> EO-4: Target(tgt-9448) window enabled: 167.3 to 280.0
2026-07-28 23:38:36,680 sats.satellite.EO-4 INFO <156.00> EO-4: setting timed terminal event at 280.0
2026-07-28 23:38:36,687 sats.satellite.EO-4 INFO <168.50> EO-4: imaged Target(tgt-9448)
2026-07-28 23:38:36,690 data.base INFO <168.50> Total reward: {'EO-4': np.float64(0.1390228426733928)}
2026-07-28 23:38:36,691 sats.satellite.EO-4 INFO <168.50> EO-4: Satellite EO-4 requires retasking
2026-07-28 23:38:36,751 gym INFO <168.50> Step reward: {'EO-4': np.float64(0.1390228426733928)}
Episode complete.
After the running the simulation, we can check the reward, number of imaged targets that were covered by clouds and that were not covered by clouds (according to the threshold set in the rewarder).
[15]:
print("Total reward:", env.unwrapped.rewarder.cum_reward)
print("Number of total images taken:", len(env.unwrapped.rewarder.data.imaged))
print(
"Number of imaged targets (once or more):",
len(set(env.unwrapped.rewarder.data.imaged)),
)
print(
"Number of re-images:",
len(env.unwrapped.rewarder.data.imaged)
- len(set(env.unwrapped.rewarder.data.imaged)),
)
print(
"Number of completely imaged targets:",
len(env.unwrapped.rewarder.data.imaged_complete),
)
Total reward: {'EO-0': 0.0, 'EO-1': 0.0, 'EO-2': np.float64(0.15950168707407258), 'EO-3': np.float64(0.017923627535731398), 'EO-4': np.float64(0.1390228426733928)}
Number of total images taken: 3
Number of imaged targets (once or more): 3
Number of re-images: 0
Number of completely imaged targets: 2
Training and custom actor-critic modules
Instead of using a handcrafted heuristic, it is possible to obtain a learned-based policy using reinforcement learning algorithms such as PPO.
The observation space contains concatenated information regarding the spacecraft and individual targets, which naturally decomposes into two components: a spacecraft state vector and a set of target-specific state vectors. This representation motivates the use of shared target encoders, attention mechanisms, and permutation-invariant pooling operations instead of applying an MLP directly to the flattened observation.
Then, custom actor and critic modules are defined for training with PPO.
[16]:
from ray.rllib.algorithms.ppo.torch.ppo_torch_rl_module import PPOTorchRLModule
from ray.rllib.core import Columns
from ray.rllib.core.models.base import ACTOR, CRITIC, ENCODER_OUT
from ray.rllib.core.models.configs import RecurrentEncoderConfig
from ray.rllib.core.rl_module.torch.torch_rl_module import TorchRLModule
from ray.rllib.models.torch.torch_distributions import (
TorchCategorical,
)
from ray.rllib.utils.annotations import (
override,
)
from ray.rllib.utils.framework import try_import_torch
from ray.rllib.utils.typing import TensorType
torch, nn = try_import_torch()
class MultiHeadSelfAttention(nn.Module):
def __init__(self, d_in: int, d_model: int, n_heads: int = 4):
super().__init__()
assert d_model % n_heads == 0
self.n_heads = n_heads
self.d_head = d_model // n_heads
self.qkv = nn.Linear(d_in, 3 * d_model, bias=False)
self.out = nn.Linear(d_model, d_in, bias=False)
def forward(self, x):
B, N, _ = x.shape
qkv = self.qkv(x)
qkv = qkv.view(B, N, 3, self.n_heads, self.d_head)
qkv = qkv.permute(2, 0, 3, 1, 4)
q, k, v = qkv[0], qkv[1], qkv[2]
attn = torch.nn.functional.scaled_dot_product_attention(
q, k, v, dropout_p=0.0, is_causal=False
)
attn = attn.transpose(1, 2).contiguous().view(B, N, -1)
return self.out(attn)
class MultiHeadAttention(nn.Module):
def __init__(self, d_in_q: int, d_in_kv: int, d_model: int, n_heads: int = 4):
super().__init__()
assert d_model % n_heads == 0
self.n_heads = n_heads
self.d_head = d_model // n_heads
self.q_proj = nn.Linear(d_in_q, d_model, bias=False)
self.k_proj = nn.Linear(d_in_kv, d_model, bias=False)
self.v_proj = nn.Linear(d_in_kv, d_model, bias=False)
self.out = nn.Linear(d_model, d_in_q, bias=False)
def forward(self, x, y=None):
if y is None:
y = x
B, N_q, _ = x.shape
N_kv = y.shape[1]
q = self.q_proj(x).view(B, N_q, self.n_heads, self.d_head).transpose(1, 2)
k = self.k_proj(y).view(B, N_kv, self.n_heads, self.d_head).transpose(1, 2)
v = self.v_proj(y).view(B, N_kv, self.n_heads, self.d_head).transpose(1, 2)
attn = torch.nn.functional.scaled_dot_product_attention(
q, k, v, dropout_p=0.0, is_causal=False
)
attn = attn.transpose(1, 2).contiguous().view(B, N_q, -1)
return self.out(attn)
class Critic(nn.Module):
def __init__(
self,
inputs: int,
width_phi: int = 32,
depth_phi: int = 4,
tgt_encoded_dim: int = 16,
width_psi: int = 32,
depth_psi: int = 4,
n_tgts: int = 32,
obs_sat: int = 38,
dropout: float = 0.0,
):
super().__init__()
self.obs_sat = obs_sat
act_function = nn.ReLU
self.n_tgts = n_tgts
# Define the number of features per target based on the input size and the number of targets
self.features_per_tgt = (inputs - self.obs_sat) // n_tgts
input_size_phi = self.features_per_tgt
layers_phi = []
layers_phi.append(nn.Linear(input_size_phi, width_phi))
layers_phi.append(act_function())
if dropout > 0:
layers_phi.append(nn.Dropout(dropout))
for _ in range(depth_phi - 1):
layers_phi.append(nn.Linear(width_phi, width_phi))
layers_phi.append(act_function())
if dropout > 0:
layers_phi.append(nn.Dropout(dropout))
layers_phi.append(nn.Linear(width_phi, tgt_encoded_dim))
self.phi = nn.Sequential(*layers_phi)
input_size_psi = 2 * tgt_encoded_dim + self.obs_sat
layers_psi = []
layers_psi.append(nn.Linear(input_size_psi, width_psi))
layers_psi.append(act_function())
if dropout > 0:
layers_psi.append(nn.Dropout(dropout))
for _ in range(depth_psi - 1):
layers_psi.append(nn.Linear(width_psi, width_psi))
layers_psi.append(act_function())
if dropout > 0:
layers_psi.append(nn.Dropout(dropout))
layers_psi.append(nn.Linear(width_psi, 1))
self.psi = nn.Sequential(*layers_psi)
def forward(self, x):
if isinstance(x, dict) and "obs" in x:
x = x["obs"]
B = x.shape[0]
x_sat = x[:, : self.obs_sat]
x_tgts = x[:, self.obs_sat :]
# Allows changes in the number of targets during runtime without changing internal variables as long as the input dimension is consistent with the number of targets
n_tgts = x_tgts.shape[1] // self.features_per_tgt
x_tgts = x_tgts.view(
B, n_tgts, self.features_per_tgt
) # (B, n_tgts, features_per_tgt)
latent_tgts = self.phi(x_tgts)
latent = torch.cat(
[
x_sat,
torch.mean(latent_tgts, dim=1),
torch.max(latent_tgts, dim=1).values,
],
dim=-1,
)
critic_value = self.psi(latent).squeeze(-1) # (B,)
return critic_value
class Actor(nn.Module):
def __init__(
self,
inputs: int,
width_phi: int = 32,
depth_phi: int = 4,
tgt_encoded_dim: int = 16,
num_heads: int = 2,
attention_dim: int = 32,
width_psi_img: int = 32,
depth_psi_img: int = 2,
n_tgts: int = 32,
obs_sat: int = 38,
non_imaging_actions: int = 1,
dropout: float = 0.0,
width_psi_sat: int = 32,
depth_psi_sat: int = 2,
width_phi_sat: int = 32,
depth_phi_sat: int = 2,
sat_attention_dim: int = 32,
sat_attention_heads: int = 2,
sat_encoded_dim: int = 32,
hierarchical: bool = False,
):
super().__init__()
self.obs_sat = obs_sat
act_function = nn.ReLU
self.n_tgts = n_tgts
# Define the number of features per target based on the input size and the number of targets
self.features_per_tgt = (inputs - self.obs_sat) // n_tgts
self.non_imaging_actions = non_imaging_actions
self.hierarchical = hierarchical
self.phi = self._build_phi_model(
input_dim=self.features_per_tgt,
width=width_phi,
depth=depth_phi,
output_dim=tgt_encoded_dim,
act_function=act_function,
dropout=dropout,
)
self.self_attention = MultiHeadSelfAttention(
tgt_encoded_dim, attention_dim, num_heads
)
self.norm_self_attention = nn.LayerNorm(tgt_encoded_dim)
self.psi_img = self._build_psi_model(
3 * tgt_encoded_dim + sat_encoded_dim,
width=width_psi_img,
depth=depth_psi_img,
output_dim=tgt_encoded_dim,
act_function=act_function,
dropout=dropout,
add_norm=False,
)
self.out_layer_psi_img = nn.Linear(tgt_encoded_dim, 1)
self.phi_sat = self._build_phi_model(
input_dim=self.obs_sat,
width=width_phi_sat,
depth=depth_phi_sat,
output_dim=sat_encoded_dim,
act_function=act_function,
dropout=dropout,
)
self.cross_attention = MultiHeadAttention(
sat_encoded_dim,
tgt_encoded_dim,
sat_attention_dim,
sat_attention_heads,
)
self.norm_cross_attention = nn.LayerNorm(sat_encoded_dim)
self.psi_sat = self._build_psi_model(
2 * tgt_encoded_dim + sat_encoded_dim,
width=width_psi_sat,
depth=depth_psi_sat,
output_dim=sat_encoded_dim,
act_function=act_function,
dropout=dropout,
add_norm=False,
)
out_dim_psi_sat = 1 + non_imaging_actions
self.out_layer_psi_sat = nn.Linear(sat_encoded_dim, out_dim_psi_sat)
def _build_phi_model(
self,
input_dim: int,
width: int,
depth: int,
output_dim: int,
act_function,
dropout: float,
):
layers = []
layers.append(nn.Linear(input_dim, width))
layers.append(act_function())
if dropout > 0:
layers.append(nn.Dropout(dropout))
for _ in range(depth - 1):
layers.append(nn.Linear(width, width))
layers.append(act_function())
if dropout > 0:
layers.append(nn.Dropout(dropout))
layers.append(nn.Linear(width, output_dim))
return nn.Sequential(*layers)
def _build_psi_model(
self,
input_dim: int,
width: int,
depth: int,
output_dim: int,
act_function,
dropout: float,
add_norm: bool,
):
layers = []
layers.append(nn.Linear(input_dim, width))
layers.append(act_function())
if dropout > 0:
layers.append(nn.Dropout(dropout))
for _ in range(depth - 1):
layers.append(nn.Linear(width, width))
layers.append(act_function())
if dropout > 0:
layers.append(nn.Dropout(dropout))
layers.append(nn.Linear(width, output_dim))
layers.append(act_function())
return nn.Sequential(*layers)
def forward(self, x):
if isinstance(x, dict) and "obs" in x:
x = x["obs"]
B = x.shape[0]
x_sat = x[:, : self.obs_sat]
x_tgts = x[:, self.obs_sat :]
# Allows changes in the number of targets during runtime without changing internal variables as long as the input dimension is consistent with the number of targets
n_tgts = x_tgts.shape[1] // self.features_per_tgt
x_tgts = x_tgts.view(
B, n_tgts, self.features_per_tgt
) # (B, n_tgts, features_per_tgt)
latent_tgts = self.phi(x_tgts) # (B, n_tgts, tgt_encoded_dim)
latent_sat = self.phi_sat(x_sat) # (B, sat_encoded_dim)
attention_out_self = self.self_attention(
latent_tgts
) # (B, n_tgts, tgt_encoded_dim)
latent_tgts_self = self.norm_self_attention(latent_tgts + attention_out_self)
latent_tgts = self.psi_img(
torch.cat(
[
latent_tgts_self,
torch.mean(latent_tgts_self, dim=1)
.unsqueeze(1)
.expand(-1, n_tgts, -1),
torch.max(latent_tgts_self, dim=1)
.values.unsqueeze(1)
.expand(-1, n_tgts, -1),
latent_sat.unsqueeze(1).expand(-1, n_tgts, -1),
],
dim=-1,
)
) # (B, n_tgts, tgt_encoded_dim
# Cross attention with satellite features
sat_attention_out = self.cross_attention(
latent_sat.unsqueeze(1), latent_tgts
) # (B, 1, sat_attention_dim)
latent_sat = self.norm_cross_attention(
latent_sat + sat_attention_out.squeeze(1)
) # (B, sat_encoded_dim)
latent_sat = self.psi_sat(
torch.cat(
[
latent_sat,
torch.mean(latent_tgts, dim=1),
torch.max(latent_tgts, dim=1).values,
],
dim=-1,
)
)
logits_tgts = self.out_layer_psi_img(latent_tgts).squeeze(-1)
psi_sat_out = self.out_layer_psi_sat(latent_sat) # (B, non_imaging_actions + 1)
img_modulated = psi_sat_out[:, 0:1]
non_img_logit = psi_sat_out[:, 1:]
if self.hierarchical:
return torch.cat([non_img_logit, img_modulated, logits_tgts], dim=1)
return torch.cat([non_img_logit, logits_tgts + img_modulated], dim=1)
A custom RLModule is created, inheriting from the existing RLLib PPOTorchRLModule. This new module instantiates the new actor and critic modules and re-defines the pi head.
[17]:
class CustomModule(PPOTorchRLModule, nn.Module):
def setup(self):
catalog = self.config.get_catalog()
is_stateful = isinstance(
catalog.actor_critic_encoder_config.base_encoder_config,
RecurrentEncoderConfig,
)
if is_stateful:
self.config.inference_only = False
if self.config.inference_only and self.framework == "torch":
catalog.actor_critic_encoder_config.inference_only = True
self.encoder = lambda x: {ENCODER_OUT: {ACTOR: x, CRITIC: x}}
config_dict = self.config.model_config_dict
self.pi_head = Actor(
inputs=self.config.observation_space.shape[0],
n_tgts=config_dict["n_targets"],
obs_sat=config_dict["obs_sat"],
width_phi=config_dict["width_phi"],
depth_phi=config_dict["depth_phi"],
tgt_encoded_dim=config_dict["tgt_encoded_dim"],
num_heads=config_dict["num_heads"],
attention_dim=config_dict["attention_dim"],
width_psi_img=config_dict["width_psi_img"],
depth_psi_img=config_dict["depth_psi_img"],
dropout=config_dict.get("dropout", 0.0),
non_imaging_actions=config_dict.get("non_imaging_actions", 1),
width_psi_sat=config_dict["width_psi_sat"],
depth_psi_sat=config_dict["depth_psi_sat"],
sat_attention_dim=config_dict["sat_attention_dim"],
sat_attention_heads=config_dict["sat_attention_heads"],
width_phi_sat=config_dict["width_phi_sat"],
depth_phi_sat=config_dict["depth_phi_sat"],
sat_encoded_dim=config_dict["sat_encoded_dim"],
)
if not self.config.inference_only or self.framework != "torch":
self.vf = Critic(
inputs=self.config.observation_space.shape[0],
n_tgts=config_dict["n_targets"],
obs_sat=config_dict["obs_sat"],
width_phi=config_dict["critic_width_phi"],
depth_phi=config_dict["critic_depth_phi"],
tgt_encoded_dim=config_dict["critic_tgt_encoded_dim"],
width_psi=config_dict["critic_width_psi"],
depth_psi=config_dict["critic_depth_psi"],
dropout=config_dict.get("dropout", 0.1),
)
self._inference_only_state_dict_keys = {}
self.action_dist_cls = catalog.get_action_dist_cls(framework=self.framework)
def pi(
self, batch: dict[str, TensorType], inference: bool = False
) -> dict[str, TensorType]:
pi_outs = {}
logits = self.pi_head(batch)
discrete_action_dist = TorchCategorical.from_logits(logits)
if inference:
discrete_action = discrete_action_dist.to_deterministic().sample()
else:
discrete_action = discrete_action_dist.sample()
discrete_action_logp = discrete_action_dist.logp(discrete_action)
pi_outs[Columns.ACTION_LOGP] = discrete_action_logp
pi_outs[Columns.ACTION_DIST_INPUTS] = logits
pi_outs[Columns.ACTIONS] = discrete_action
return pi_outs
@override(TorchRLModule)
def _forward_inference(self, batch: dict[str, TensorType]) -> dict[str, TensorType]:
return self.pi(batch, inference=True)
@override(TorchRLModule)
def _forward_exploration(
self, batch: dict[str, TensorType], **kwargs
) -> dict[str, TensorType]:
return self.pi(batch, inference=False)
@override(TorchRLModule)
def _forward_train(self, batch: dict[str, TensorType]) -> dict[str, TensorType]:
outs = {}
outs.update(self.pi(batch))
vf_out = self.vf(batch)
outs[Columns.VF_PREDS] = vf_out.squeeze(-1)
return outs
Then, the training hyperparameters (obtained using a hyperparameter optimization algorithm) and modules parameters are defined, as well as the custom RLModule.
[18]:
from ray.rllib.core.rl_module.rl_module import RLModuleSpec
training_args = dict(
lr=[
[0, 0.00033003435881682255],
[40000, 0.00033003435881682255 / 16.749479444886223],
],
gamma=0.999,
train_batch_size=300 * 10 * 3,
num_sgd_iter=30,
lambda_=0.8713548569911232,
use_kl_loss=False,
clip_param=0.14701727973480344,
grad_clip=0.3104924935285628,
vf_clip_param=2.0,
entropy_coeff=[[0, 0.023694512589767867], [750_000, 0.0]],
)
rl_module_args = dict(
model_config_dict={
"n_targets": 40,
"obs_sat": 38,
"width_phi": 256,
"depth_phi": 2,
"width_psi_img": 128,
"depth_psi_img": 4,
"tgt_encoded_dim": 128,
"attention_depth": 1,
"num_heads": 2,
"attention_dim": 128,
"width_phi_sat": 256,
"depth_phi_sat": 2,
"width_psi_sat": 128,
"depth_psi_sat": 4,
"sat_attention_dim": 128,
"sat_attention_heads": 2,
"sat_encoded_dim": 128,
"act_function": "ReLU",
"critic_tgt_encoded_dim": 128,
"critic_width_phi": 256,
"critic_depth_phi": 2,
"critic_width_psi": 64,
"critic_depth_psi": 3,
"dropout": 0.1,
"critic_dropout": 0.1,
"non_imaging_actions": 1,
},
rl_module_spec=RLModuleSpec(module_class=CustomModule),
)
[19]:
from ray.rllib.algorithms.ppo import PPOConfig
from bsk_rl.utils.rllib.callbacks import WrappedEpisodeDataCallbacks
from bsk_rl.utils.rllib.discounting import TimeDiscountedGAEPPOTorchLearner
N_CPUS = 12
def example_data_callback(env):
reward = env.rewarder.cum_reward
reward = sum(reward.values())
orbits = env.simulator.sim_time / (95 * 60)
data = dict(
reward=reward,
alive=float(env.satellite.is_alive()),
battery_status_valid=float(env.satellite.dynamics.battery_valid()),
orbits_complete=orbits,
)
if orbits > 0:
data["reward_per_orbit"] = reward / orbits
if not env.satellite.is_alive():
data["orbits_complete_partial_only"] = orbits
return data
env_args_training = dict(
satellite=C_F_Bayesian("C_F", sat_args, belief_update_func=belief_update_func),
scenario=scenario,
rewarder=rewarder,
sat_arg_randomizer=sat_arg_randomizer,
sim_rate=0.5,
max_step_duration=300.0,
time_limit=95 * 60 * 3, # Three orbits
log_level="INFO",
failure_penalty=0.0,
)
ppo_config = (
PPOConfig()
.training(
**training_args,
learner_class=TimeDiscountedGAEPPOTorchLearner,
)
.env_runners(num_env_runners=N_CPUS - 1, sample_timeout_s=1000.0)
.environment(
env="SatelliteTasking-RLlib",
env_config=dict(
**env_args_training, episode_data_callback=example_data_callback
),
)
.reporting(
metrics_num_episodes_for_smoothing=1,
metrics_episode_collection_timeout_s=180,
)
.checkpointing(export_native_model_files=True)
.framework(framework="torch")
.api_stack(
enable_rl_module_and_learner=True,
enable_env_runner_and_connector_v2=True,
)
.callbacks(WrappedEpisodeDataCallbacks)
)
ppo_config.rl_module(**rl_module_args)
# Uncomment to run training
# import ray
# ray.init(
# ignore_reinit_error=True,
# num_cpus=N_CPUS,
# object_store_memory=2_000_000_000, # 2 GB
# )
# results = ray.tune.run(
# "PPO",
# config=ppo_config.to_dict(),
# stop={
# "num_env_steps_sampled_lifetime": 264
# }, # Total number of steps to train the model. Originally 774,000
# checkpoint_freq=1,
# checkpoint_at_end=True,
# )
# ray.shutdown()
[19]:
<ray.rllib.algorithms.ppo.ppo.PPOConfig at 0x7fcb08c27510>