With all renewed discussion about "Sparse AutoEncoders (#SAE)" as a way of doing #MechanisticInterpretability of #LLMs, I am resharing a part of my PhD where we proved years ago about how sparsity automatically emerges in autoencoding.
arxiv.org/abs/1708.03735
Registrations close for #DRSciML by Noon (Manchester time). Do register soon to ensure you get the Zoom links to attend this exciting event on the foundations of #ScientificML 💥
Recently I gave an online talk @
India's premier institute IISc 's "Bangalore Theory Seminars" where I explained our results on size lowerbounds for neural models of solving PDEs via neural nets. #SciML #AI4SCIENCE I cover work by one of my, 1st year PhD student, Sebastien.
youtu.be/CWvnhv1nMRY?...
Today is 70th anniversary of the summer meeting at Dartmouth which officially marked the beginning of AI research 💥 Interestingly "Objective 3" in 1955 was already about having theory of neural nets. 🙂
stanford.io/2WJJJGN
I got selected for the "Early Career Highlights" of ACM IKDD International Conference on Data Science (CODS) 2025. Looking forward to the talk at IISER, Pune in December.
www.acm.org/articles/acm...
Why does noisy gradient-descent train neural nets? This fundamental question in ML remains unclear.
In our hugely revised draft my student @dkumar9.bsky.social gives the full proof that a form of noisy-GD, Langevin Monte-Carlo (#LMC), can learn arbitrary depth 2 nets.
arxiv.org/abs/2503.10428
Registrations are now open for the international workshop on foundations of #AI4Science #SciML that we are hosting with Prof. Jakob Zech. In-person seats are very limited, please do register to join online 💥
drsciml.github.io/drsciml/
Please do get in touch if you have published paper(s) on solving singularly perturbed PDEs using neural nets. #AI4Science #SciML
@aifunmcr.bsky.social
Some luck to be hosted by a Godel Prize winner, Prof. Sebastien Pokutta, and to present our work in their group 💥 Sebastien heads this "Zuse Institute Berlin (#ZIB) " which is an amazing oasis of applied mathematics bringing together experts from different institutes in Berlin.
Interested in statistics? Prof Subhashis Ghoshal will be delivering the below public lecture tomorrow:
Title: Immersion posterior: Meeting Frequentist Goals under Structural Restrictions
Time: Aug 5 16:00-17:00
Abstract: www.newton.ac.uk/seminar/45562/
Livestream: www.newton.ac.uk/news/watch-l...
Hello #FAU. Thanks for the quick plan to host me and letting me present our exciting mathematics of ML in infinite-dimensions, #operatorlearning. #sciML Their "Pattern Recognition Laboratory" is completing 50 years! @andreasmaier.bsky.social 💥
@aifunmcr.bsky.social
University of Manchester has a 1 year post-doc position that I am happy to support in our group if you are currently an #EPSRC funded PhD student - and have the required specialization for work in our group. Typicall we prefer candidates who have published in deep-learning theory or fluid theory.
@aifunmcr.bsky.social
#aiforscience
Do mark your calendars for "DRSciML" (Dr. Scientific ML 😉) on September 9 and 10 🔥
drsciml.github.io/drsciml/
- We are hosting a 2 day international workshop on understanding scientific-ML.
- We have leading experts from around the world giving talks.
- There might be ticketing. Watch this space!
@aifunmcr.bsky.social
Major ML journals that have come up in the recent years,
- dl.acm.org/journal/topml
- jds.acm.org
- link.springer.com/journal/44439
- academic.oup.com/rssdat
- jmlr.org/tmlr/
- data.mlr.press
No reason why these cant replace everything the current conferences are doing and most likely better.
Thanks. No, AutoSGD is not going as far as delta-GClip goes. It's Theorem 4.5 is where they have any global minima convergence happening - but it uses assumptions which are not known to be true for nets. Our convergence holds for *all* nets wide enough.
Do link to the paper! I can have a look and check.
So, the next time you train a deep-learning model, it's probably worthwhile to have a baseline for the only provable adaptive gradient deep-learning algorithm - our delta-GClip 🙂
Our insight is to introduce an intermediate form of gradient clipping that can leverage the PL* inequality of wide nets - something not known for standard clipping. Given our algorithm works for transformers maybe that points to some yet unkown algebraic property of them. #TMLR
Our "delta-GCLip" is the *only* known adaptive gradient algorithm that provably trains deep-nets AND is practically competitive. That's the message of our recently accepted #TMLR paper - and my 4th TMLR journal 🙂
openreview.net/pdf?id=ABT1X...
#optimization #deeplearningtheory
An updated version of our slides on necessary conditions for #SciML,
- and more specially,
"Machine Learning in Function Spaces/Infinite Dimensions".
Its all about the 2 key inequalities on slides 27 and 33.
Both come via similar proofs.
github.com/Anirbit-AI/S...
Now our research group has a logo to succinctly convey what we do - prove theorems about using ML to solve PDEs, leaning towards operator learning. Thanks to #ChatGPT4o for converting my sketches into a digital image 🔥 #AI4Science #SciML
It would be great to be able to see a compiles list of useful PDEs that #PINNs struggle to solve - and how would we measure success there.
We know of edge-cases with simple PDEs, where PINNs struggle, but then often those aren't the cutting-edge of use-cases of PDEs.
@prochetasen.bsky.social @mingfei.bsky.social @omarrivasplata.bsky.social
@aifunmcr.bsky.social 🙂
A revised version of our delta-GClip algorithm - which is probably the *only* deep-learning algorithm that provably trains deep-nets while using step-size scheduling - and competes/supersedes heuristics like Adam and even Adam+Clipping on transformers.
arxiv.org/abs/2404.08624
