8.2 POLICY
Several di.erent strategies may be used to formulate the MDP to be solved o.ine by DP. This section discusses three broad categories.
Joint Policy
A coordinated encounter between two aircraft with full state observability can be modeled as a multi-agent MDP (MMDP). An MMDP extends an MDP by allowing for a .nite set of agents, each with a .nite set of available actions. The joint action space is the Cartesian product of the action spaces of the individual agents. The state space is the set of all joint states, and the transition and cost functions are de.ned over these joint state and action spaces. In an MMDP, each of the agents can fully identify the true state through the use of its own observations.
An MMDP can be reduced to a single-agent joint MDP over the joint state and action spaces. For a coordinated encounter, the action space is the set of all joint aircraft actions, such as DES1500 for one aircraft and CL1500 for the other. The state space is the set of all joint aircraft states. Extending
the
MDP
introduced
in
Section
3
to
an
MMDP
requires
augmenting
the
advisory
state
variable sRA to contain all information regarding the response of both pilots to their respective advisories. The cost function should also be amended so that cost is incurred when the other aircraft alerts, strengthens, or reverses. The MDP for the coordinated encounter generally consists of more states and actions than for the uncoordinated encounter. The solution to the joint MDP is a joint policy that speci.es the actions for both aircraft from every state. So long as the size of the problem does not become too large, solving for the optimal joint policy can be done o.ine using DP.
So long as computation and storage are not an issue, there is no harm in including in the joint action space all possible action pairs, including CL1500/CL1500. During the optimization, however, such incompatible action pairs would be assigned higher expected cost compared to other actions because they do not aid in reducing separation. Consequently, those incompatible actions would never be selected. To reduce the computational and storage requirements, these incompatible actions can be eliminated from consideration in the .rst place. Restricting the set of available joint advisories results in a smaller action space as well as a smaller state space. The number of states is reduced because the advisory state variable can assume fewer values.
The set of joint actions can be restricted even further by assuming that both aircraft issue initial alerts, reversals, and strengthenings at the same time. This reduced action set will be referred to as the synchronous action set and will be used to generate the results later in this section. The synchronous action set does not contain actions for which one of the aircraft alerts while the other one does not, although it might be bene.cial to include such joint actions if it is possible to prevent NMAC reliably through the maneuvering of only one aircraft. A desirable property of the joint MDP with the synchronous action set is that it has the same number of states and actions as the MDP modeling the uncoordinated encounter when using the deterministic pilot response model.
中国航空网 www.aero.cn
航空翻译 www.aviation.cn
本文链接地址：Robust Airborne Collision Avoidance through Dynamic Programm(39)
