Reinforcement Learning Based ADP:

Computational reductions, faster learning, and more complex problems

– An Application

Space is Becoming Congested

“We will be using proliferated LEO,” Lt. Gen. David Thompson told a New America Foundation conference today in response to my question. “It is simply a matter of for what missions.”

Excerpted from “LEO and MEO broadband constellations mega source of consternation,” a 2018 SpaceNews article by Caleb Henry

https://breakingdefense.com/2019/04/thompson-afspc-will-use-mega-constellations/

https://breakingdefense.com/2019/04/thompson-afspc-will-use-mega-constellations/

Challenges

• Threats• Cyber • Physical (i.e., debris)

• Sources of space debris• Abandoned upper stages and satellites• Collisions, ASAT demos, fragmentation

• Space debris population • Baseball size & larger (≥ 10 cm) ~23K• Marble size (≥ 1 cm) ~500K• Dot size mall (≥ 1 mm) ~100M

• Population growth projected• Mega constellations significant SSA/space traffic management challenge

• OneWeb: > 600 satellites (w/ Airbus)• SpaceX Starlink: > 4,000 satellites• Samsung: > 4,000 satellites• Others

• Kessler Syndrome additional debris created from collisions with debris

Reference: http://orbitaldebris.jsc.nasa.gov

Small (mm-to-cm) sized debris dominate mission ending threat

http://orbitaldebris.jsc.nasa.gov/

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

16000

1956

1958

1960

1962

1964

1966

1968

1970

1972

1974

1976

1978

1980

1982

1984

1986

1988

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

Nu

mb

er o

f O

bje

cts

Year

Monthly Number of Objects in Earth Orbit by Object Type (US Satellite Catalog)

Total Objects

Fragmentation Debris

Spacecraft

Mission-related Debris

Rocket Bodies Iridium-Cosmos

The Debris Problem

FY-1C ASAT Test

0

100

200

300

400

500

600

700

800

200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000

Eff

ec

tive

Nu

mb

er o

f O

bje

cts

pe

r 2

0 k

m A

lt B

in

Altitude (km)

1-Apr-10 SSN catalog

1-Jan-07 SSN catalog

Chinese ASAT resulted in ~120% increase in the region below 1000 km

The Debris Problem

• Observed and predicted A/M distribution of Cosmos 2251 fragments matches well

0

0.05

0.1

0.15

0.2

0.25

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Nu

mb

er

(No

rmal

ize

d)

Log10(A/M m2/kg)

A/M Distribution of Cosmos 2251 Fragments

NASA Breakup Model Prediction

SATCAT Data (17 September 2010)

• Observed A/M distribution of Iridium 33 fragments is higher than prediction; ~3 times higher)

• Possible source of difference: lightweight materials (e.g., MLI, CFRP) used in the newer vehicle 0

20

40

60

80

100

120

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Nu

mb

er

Log10(A/M m2/kg)

A/M Distribution of Iridium 33 Fragments

NASA Breakup Model Prediction

SATCAT Data (17 September 2010)

Iridium 33(560 kg)

Cosmos 2251(900 kg)

0

20

40

60

80

100

120

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Nu

mb

er

Log10(A/M m2/kg)

A/M Distribution of Iridium 33 Fragments

NASA Breakup Model Prediction

SATCAT Data, A/M/3 (17 September 2010)

DebriSat Laboratory HVI Test

MIRO310(14886)(MirrorCorrection).avi

Results

• Preliminary outcomes of the HVI test• Developed a unique database with ~200K fragments (> 250K projected – x3?)• Imaging systems developed to characterize fragments• Models developed to compute fragments’ (i) characteristic length (ii) average cross-

sectional area, (iii) volume

➢Task: Utilize center’s modelling tools to develop models to better characterize the debris environment and facilitate future space traffic management / SSA capabilities in congested and/or contested environments (e.g., COLA maneuvers)

Current models predicted 85K fragments

Recorded Collected

0

10000

20000

30000

40000

50000

60000

70000

1950 1970 1990 2010 2030 2050 2070 2090 2110 2130 2150 2170 2190 2210

Eff

ec

tive

Nu

mb

er o

f O

bje

cts

(>

10

cm

)

Year

Non-Mitigation Projection (averages and 1-σ from 100 MC runs)

LEO (200-2000 km alt)

MEO (2000-35,586 km alt)

GEO (35,586-35,986 km alt)

The Debris Problem

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

1950 1970 1990 2010 2030 2050 2070 2090 2110 2130 2150 2170 2190 2210

Eff

ec

tive

Nu

mb

er o

f O

bje

cts

(>

10

cm

)

Year

LEO Environment Projection (averages of 100 LEGEND MC runs)

Reg Launches + 90% PMD

Reg Launches + 90% PMD + ADR2020/02

Reg Launches + 90% PMD + ADR2020/05

• PMD scenario predicts the LEO populations would increase by ~75% in 200 years

• LEO environment can be stabilized with PMD and a removal rate of ~5 objects/year

Average collisions in the next 200 years(non-mitigation scenario)

Cat. Non-cat. Total

LEO 83 95 178

MEO 0.5 1.5 2

GEO 1.5 1.5 3

ADR Approaches

Necropolis Concept (European)• A two spacecraft system• A “Hunter” that hunts, collects, and transports spent

GEO spacecraft and a “Terminus” for final storage

RemoveDEBRIS Mission• On-orbit demo by Surrey Space Centre for active

debris removal (ADR)• Successfully demonstrated (i) on-orbit debris

identification via vision-based navigation (ii) “debris” capture via tethered net and harpoon, and (iii) deorbiting via drag sail

Others:spacecraft• GLiDeR – electrostatic tug (Hanspeter Schaub)• Tether-Nets for ADR (Botta, Sharf, Misra)• ESA’s Clean Space Initiative (E.DeOrbit mission)

http://www.esa.int/Our_Activities/Space_Safety/Clean_Space/ESA_s_e.Deorbit_debris_removal_mission_reborn_as_servicing_vehicle

http://www.esa.int/Our_Activities/Space_Safety/Clean_Space/ESA_s_e.Deorbit_debris_removal_mission_reborn_as_servicing_vehicle

ADR Approaches

Active Debris Removal (ADR)

Contact Noncontact

Cooperative Non-cooperative

• Orbital Express (DARPA)

• SUMO/FREND (DARPA ARD)

• RemoveDEBRIS

• Risky (nets, harpoons) • Needs investigation➢ Game-based

(minimax, Stackelberg, Nash)

• GLiDeR (and others)• Necropolis Hunter• Herding approaches➢ Game-based

(minimax, Stackelberg, Nash)

Task: Design strategies to address interactions

(contact/noncontact) with non-cooperative RSOs

Two-Person Differential Games

Two-person differential game

• Minimax: zero sum, cooperative

• Stackelberg: non-zero sum, leader-follower

• Nash: non-zero sum, non-cooperative game

Multiple and/or no solution

(e.g., Prisoner’s Dilemma)

Summary

Debris abatement includes removal for disposal (or repair) – need to:• Minimize interactions between vehicles (space tug and

disabled RSO) • Navigate congested environment while minimizing

likelihood of collision with other RSO (herding)

Minimax, Stackelberg, Nash strategies

• The National Space Traffic Management Policy (SPD-3), includes the following:

– Space Situational Awareness (SSA) shall mean the knowledge and characterization of space objects and their operational environment to support safe, stable, and sustainable space activities.

– Space Traffic Management (STM) shall mean the planning, coordination, and on-orbit synchronization of activities to enhance the safety, stability, and sustainability of operations in the space environment.

– Orbital debris (OD), or space debris, shall mean any human-made space object orbiting Earth that no longer serves any useful purpose.

– Advance SSA and STM Science and Technology. The United States should continue to engage in and enable S&T research and development to support the practical applications of SSA and STM. These activities include improving fundamental knowledge of the space environment, such as the characterization of small debris, advancing the S&T of critical SSA inputs such as observational data, algorithms, and models necessary to improve SSA capabilities, and developing new hardware and software to support data processing and observations.

Conclusion

https://www.whitehouse.gov/presidential-actions/space-policy-directive-3-national-space-traffic-management-policy/

https://www.whitehouse.gov/presidential-actions/space-policy-directive-3-national-space-traffic-management-policy/