Skip to content

Commit

Permalink
rebrand to TRAC:
Browse files Browse the repository at this point in the history
  • Loading branch information
@Aneeshers
Aneeshers committed Jun 11, 2024
1 parent 147635e commit 68b90f8
Show file tree
Hide file tree
Showing 14 changed files with 20,092 additions and 145 deletions.
18 changes: 9 additions & 9 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,27 +1,27 @@
# PACE: Adaptive Parameter-free Optimization ⚡️
# TRAC: Adaptive Parameter-free Optimization ⚡️ 🏎️💨
[![arXiv](https://img.shields.io/badge/arXiv-2405.16642-b31b1b.svg)](https://arxiv.org/abs/2405.16642) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1c5OxMa5fiSVnl5w6J7flrjNUteUkp6BV?usp=sharing)

This repository is the official implementation of the **PACE** optimizer in ***Pick up the PACE: A Parameter-Free Optimizer for Lifelong Reinforcement Learning***.
This repository is the official implementation of the **TRAC** optimizer in ***Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning***.

How can you _quickly_ adapt to new tasks or distribution shifts? Without knowing when or how much to adapt? And without _ANY_ tuning?
🤔💭

Well, we suggest you go fast. 🏎️💨 Go ahead and pick up (the) **PACE**.
Well, we suggest you get on the fast **TRAC** 🏎️💨.

**PACE** is a parameter-free optimizer for continual environments inspired by [online convex optimization](https://arxiv.org/abs/1912.13213) and uses [discounted adaptive online prediction](https://arxiv.org/abs/2402.02720).
**TRAC** is a parameter-free optimizer for continual environments inspired by [online convex optimization](https://arxiv.org/abs/1912.13213) and uses [discounted adaptive online prediction](https://arxiv.org/abs/2402.02720).

## Implement with only one line change.
Like other [meta-tuners](https://openreview.net/pdf?id=uhKtQMn21D), PACE can work with any of your continual, fine-tuning, or lifelong experiments with just one line change.
Like other [meta-tuners](https://openreview.net/pdf?id=uhKtQMn21D), TRAC can work with any of your continual, fine-tuning, or lifelong experiments with just one line change.

```python
from pace import start_pace
from trac import start_trac
# original optimizer
optimizer = torch.optim.Adam
lr = 0.001
optimizer = start_pace(log_file='logs/pace.text', optimizer)(model.parameters(), lr=lr)
optimizer = start_trac(log_file='logs/trac.text', optimizer)(model.parameters(), lr=lr)
```

After this modification, you can continue using your optimizer methods exactly as you did before. Whether it's calling `optimizer.step()` to update your model's parameters or `optimizer.zero_grad()` to clear gradients, everything stays the same. PACE seamlessly integrates into your existing workflow without any additional overhead.
After this modification, you can continue using your optimizer methods exactly as you did before. Whether it's calling `optimizer.step()` to update your model's parameters or `optimizer.zero_grad()` to clear gradients, everything stays the same. TRAC integrates into your existing workflow without any additional overhead.

## Control Experiments

Expand All @@ -40,7 +40,7 @@ Our vision-based experiments for [Procgen](https://openai.com/index/procgen-benc
To initiate an experiment with the default configuration in the Procgen "starpilot" environment, use the command below. You can easily switch to other game environments, like Atari, by altering the `--exp_name="atari"` parameter:

```bash
python vision_exp/train.py --exp_name="procgen" --env_name="starpilot" --optimizer="pace" --warmstart_step=0
python vision_exp/train.py --exp_name="procgen" --env_name="starpilot" --optimizer="TRAC" --warmstart_step=0
```
<p align="center">
<img src="figures/games1.gif" alt="Game 1" width="30%">
Expand Down
28 changes: 28 additions & 0 deletions logs/base_reward_log_Acrobot-v1_1.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-369.9999999999999
-400.0
-310.0
-295.0
-213.5
-192.0
-270.5
-197.5
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
278 changes: 278 additions & 0 deletions logs/base_reward_log_Acrobot-v1_2024.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1999,3 +1999,281 @@
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-334.5
-223.49999999999997
-173.5
-144.99999999999997
-154.5
-100.49999999999999
-145.0
-172.0
-151.0
-187.0
-179.99999999999997
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
-400.0
Loading

0 comments on commit 68b90f8

Please sign in to comment.