Every year around this time, I wish for introducing a matching procedure for CS grad school as in medical residency. This would be a mental health improvement for everyone involved. How are doctors better at this than computer scientists? Where is our occupational pride?
Had a great time speaking at Stanford's Center for Image System Engineering and Vision seminars over the last two weeks. 👨🏫 Thank you for hosting me, Gordon Wetzstein and Wenlong Huang.
Running PPO for continuous control and frustrated by instability or long hyperparameter tuning? 🤯 Give REPPO a try. More integrations covering your favourite RL library are coming.
📄https://arxiv.org/abs/2507.11019
💻https://github.com/cvoelcker/reppo
✍️https://cvoelcker.de/blog/2025/reppo-intro/
Huge kudos to @cvoelcker.bsky.social for bringing this all together and leading a collaboration also involving @axelbrunnbauer.bsky.social, @marcelhussing.bsky.social, Michal Nauman, Pieter Abbeel, Ragu Grosu, @sologen.bsky.social, across UofT, Vector, Poly, Mila, TU Vienna, UPenn, UC Berkeley
Despite training a critic, REPPO is fast. ⚡️
In JAX, it matches PPO's wall-clock time while delivering ~33% higher returns. The sample efficiency of pathwise gradients offsets the extra per-update computation.
Maybe our favorite result: REPPO trains “reliably”. ❤️
Once performance crosses a threshold, it stays there. ~80% of REPPO runs reach high performance without ever dropping back down. PPO? About 40 percentage points fewer.
No more "it was working at 3am but crashed by morning."😱
The results? REPPO significantly outperforms PPO on DMC and ManiSkill benchmarks in both sample efficiency and final performance.
It even rivals off-policy methods (like FastTD3) while using a fraction of the memory (no massive replay buffers needed). 📉💾
Our ablations show that HL-Gauss and KL regularization are the most critical pieces.
To improve surrogate value function estimation, we incorporated
→ HL-Gauss categorical loss for Q-learning (scale-invariant gradients!)
→ Layer normalization for stable representations
→ Auxiliary self-prediction tasks for richer features
Naively using pathwise gradients on-policy leads to collapse. 📉Two ingredients keep REPPO stable:
1️⃣ Maximum entropy objective — keeps the policy exploring
2️⃣ KL-constrained updates (the "Relative Entropy" in REPPO) — prevents the policy from jumping too far from where the Q-function is accurate
REPPO's key insight: train an accurate Q-function purely from on-policy data, then use pathwise (reparameterized) gradients to update the policy. 💡
We avoid importance sampling corrections — a major source of variance in PPO. 😵💫
The "Standard" PPO uses score-based gradients. While popular, these estimators are notoriously noisy, leading to training instability.
We asked: Can we use the reparametrization trick (as in DDPG/SAC) but keep the simplicity of on-policy learning?
🚀 Excited to share REPPO, a new on-policy RL agent!
TL;DR: Replace PPO with REPPO for fewer hyperparameter headaches and more robust training.
REPPO, led by @cvoelcker.bsky.social, will be presented at ICLR 2026. How does it work? 🧵👇
Given the diverse work on world models, there is a debate on where to draw the line between world models and other dynamic systems models. I'd argue that, like LLMs for general-purpose language modelling, world models are not designed for a single specific task or phenomenon.
Given how teleop solutions for AVs work, some of the risks outlined in this article seem contrived, even without knowing Waymo's specifics. I assume one reason for having teleop teams in locations such as Philippines (beyond cost) is to account for different time zones as the vehicles operate 24/7.
2025/2026 may be remembered as the years when many grad students "stopped" and many professors "resumed" programming.
I'll be at #NeurIPS2025 in San Diego from Thu to Sat, and I am looking for PostDocs in Embodied AI, particularly in world modeling and simulator learning. Please reach out if you are interested.
Check out the Department of Computer Science website to apply. The application deadline is December 1. web.cs.toronto.edu/graduate/pro...
We are affiliated with @vectorinstitute.ai, the Schwartz Reisman Institute, @uoftdsi.bsky.social, the Acceleration Consortium, and the Robotics Institute. There is a vibrant AI community around U of T, and we have a fantastic set of collaborators. 4/n
Admittedly, Toronto winters can be cold ❄️, but we have access to multiple GPU clusters that help us stay warm 🔥. And we have a growing set of robots to keep us busy 🦾🦿🚘. 3/n
This involves exciting topics such as control techniques for generative models 🎮, editing techniques for scene representations ✂️, physical dynamics learning ⚙️, planning in learned models 🗺️, policy learning 🧠, and real-world robot deployment 🦾. 2/n
I'm looking for graduate students to join my group in fall 2026. We work at the intersection of Computer Vision, Deep Learning, and Robotics.
The goal of our work is to create and understand organic 🍀 simulation systems, i.e., controllable data-curation engines built from real-world data. 1/n
Together with our brilliant graduate researcher Ziyi Wu and our fantastic undergrad, Brayden Zhang, we had a wonderful time at #ICCV2025!
Interesting @anthropic.com work on LLM poisoning. I'd argue that the glass is half full, and this can be interpreted as a positive result: Regardless of model size, the same number of samples seems to be sufficient to learn a new "skill". www.anthropic.com/research/sma...
Excited to have been appointed as a faculty member at @vectorinstitute.ai. 🎉 This would not have been possible without the amazing work of my collaborators and the brilliant students in my group within @uoftcompsci.bsky.social. We all look forward to deepening ties with the Vector community. ❤️
Want to help train the next generation of leaders, researchers, and innovators in CS? We are looking for teaching stream faculty to join us at @uoftcompsci.bsky.social. Check out our job posting and join a team of brilliant colleagues and amazing students. 🇨🇦❤️💻 academicjobsonline.org/ajo/jobs/30410
Proud German advisor moment: Discovered a pair of Birkenstocks under a grad student's desk.
Would you be surprised to learn that many empirical implementations of value-aware model learning (VAML) algos, including MuZero, lead to incorrect model & value functions when training stochastic models 🤕? In our new @icmlconf.bsky.social 2025 paper, we show why this happens and how to fix it 🦾!
Agree with all of the above. There is also a version of this among researchers, i.e., people knowingly defending the continued existence of likely unfruitful research directions to protect their own careers. Ultimately, each insight needs to be evaluated on its own merits.
Thrilled to share the work that our lab will present at
@cvprconference.bsky.social. Check out the papers and meet @kai-he.bsky.social, @yashkant.bsky.social, @dazitu616.bsky.social, and Toshiya Yura in Nashville at their poster sessions!
