Latest posts tagged with #pretraining on Bluesky
New #J2C Certification:
BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via...
Gustav Wagner Zakarias, Lars Kai Hansen, Zheng-Hua Tan
https://openreview.net/forum?id=GQAGlqOpyA
#supervised #pretraining #pretrained
Model Collapse - What Happens When AI Feeds Itself #ai #science #viral
#stockimages #Stockvideos #dataset #Training #Datasales #Datalicensing #MachineLearning #imagelicensing #transformermodels #pretraining #transferlearning #objectdetection #LoRA #Largevisionmodels #GANS
Model Collapse - What Happens When AI Feeds Itself #ai #science #viral
#stockimages #Stockvideos #dataset #Training #Datasales #Datalicensing #MachineLearning #imagelicensing #diffusionmodel #transformermodels #pretraining #transferlearning #objectdetection #LoRA #Largevisionmodels #GANS
Pre-training is back! 🚀 Forget the 'scaling laws are dead' talk. While everyone thought RL was king, top labs like OpenAI were wrong. Pre-training is set for a renaissance by 2026, driving major AI progress! #AI #Pretraining #MachineLearning
Towards Scalable Pre-training of Visual Tokenizers for Generation
Jingfeng Yao, Xinggang Wang et al.
Paper
Details
#VisualTokenizers #Pretraining #GenerativeAI
Big Computers, New Questions - Ilya Sutskever and Dwarkesh Patel
#research #pretraining
Understanding Emergent In-Context Learning from a Kernel Regression Perspective
Chi Han, Ziqi Wang, Han Zhao, Heng Ji
Action editor: Yingbin Liang
https://openreview.net/forum?id=6rD50Q6yYz
#context #attention #pretraining
AI models can acquire backdoors from surprisingly few malicious documents https://arstechni.ca #UKAISecurityInstitute #alanturinginstitute #AIvulnerabilities #backdoorattacks #machinelearning #datapoisoning #trainingdata #LLMsecurity #modelsafety #pretraining #AIresearch #AIsecurity…
Probabilistic Language-Image Pre-Training Boosts Vision-Language Models
A new probabilistic language-image pre-training approach is reported to boost performance of vision-language models. Read more: getnews.me/probabilistic-language-i... #visionlanguage #pretraining #ai
TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretraining
Ruiyi Zhang, Sai Ashish Somayajula, Pengtao Xie
Action editor: Simon Kornblith
https://openreview.net/forum?id=DCCw2CEVFS
#pretraining #tapweight #tap
How Pretraining Data Shapes In-Context Learning
A new study finds that heavier-tailed pretraining data improves accuracy on rare numerical tasks, while broader coverage cuts the demos needed for target performance. Read more: getnews.me/how-pretraining-data-sha... #incontextlearning #pretraining
Academic Pre-Training Feasible: $100K or 100 Days Trade-Offs
Academic teams can pre‑train a billion‑parameter model for about $100,000, using four GPUs over 18 days—a trade‑off from the original 64‑GPU three‑day run. Read more: getnews.me/academic-pre-training-fe... #academiacompute #pretraining #llm
CoMTIP: Contrastive Masked Pre‑training for Spatial Transcriptomics
CoMTIP, a pre‑training framework linking histology images, gene names and expression values, was submitted on 21 September 2025. It offers zero‑shot gene prediction and beats prior methods. getnews.me/comtip-contrastive-pre-t... #spatial #pretraining
Leaps, Not Just Steps - Demis Hassabis on Lex Fridman
#scurve #pretraining #aiscaling
🎯 How is ELLIOT strengthening Europe’s AI ecosystem?
Jenia Jitsev explains how his team leads #pretraining of #MultimodalAI open foundation models in the #HorizonEU project — using scaling laws to improve core building blocks for trustworthy, reusable #GeneralistAI.
🎥 Watch the video to learn more
Intro to Procedural Animation in Unity #Ai #Chatbot #Gpt #Openai #Transformer #Nlp #Deeplearning #Gpt3 #Gpt2 #Conversational #Languagemodel #Neuralnetwork #Pretraining #Finetuning
I'll also be presenting multiple papers at #CVPR2025! First up: "AdaVid: Adaptive Video-Language Pretraining".
🗓️ Thu Jun 12, 12:00-13:00PM
📍 ExHall D Poster #202
🔗 Paper: arxiv.org/abs/2504.12513
🌐 Website: chaitanya100100.github.io/AdaVid/
#VideoLanguage #Pretraining
Random Policy Enables In-Context Reinforcement Learning within Trust Horizons
Weiqin Chen, Santiago Paternain
Action editor: Pin-Yu Chen
https://openreview.net/forum?id=mAiMKnr9r5
#pretraining #trained #pretrained
MLCommons' MLPerf Training suite has a new #pretraining #benchmark based on #Meta’s Llama 3.1 405B model. We use the same dataset with a bigger model and longer context, offering a more relevant and challenging measure for today’s #AI systems. mlcommons.org/2025/05/trai...
4/15 Is pretraining + RLHF optimization surpassing scale, or are benchmarks just improving for specific tasks? 🤔 Good question raised by badmonster in the HN thread. #Pretraining #RLHF #Optimization https://news.ycombinator.com/item?id=43842683#43852749
New #Featured Certification:
Random Policy Enables In-Context Reinforcement Learning within Trust Horizons
Weiqin Chen, Santiago Paternain
https://openreview.net/forum?id=mAiMKnr9r5
#pretraining #trained #pretrained
Mixed Sparsity Training: Achieving 4$\times$ FLOP Reduction for Transformer Pretraining
Pihe Hu, Shaolong Li, Xun Wang, Longbo Huang
Action editor: Vincent Tan
https://openreview.net/forum?id=XosdLS7KVE
#sparse #pretraining #gpu
How Does Code Pretraining Affect Language Model Task Performance?
Jackson Petty, Sjoerd van Steenkiste, Tal Linzen
Action editor: John Timothy Halloran
https://openreview.net/forum?id=pxxmUKKgel
#linguistic #pretraining #pretrain
Joint work with Junhong Shen, Genghan Zhang @zhang677.bsky.social, Ning Dong, Luke Zettlemoyer, Lili Yu
#LLM #MultiModal #pretraining
Pretraining a Neural Operator in Lower Dimensions
AmirPouya Hemmasian, Amir Barati Farimani
Action editor: Xingyou Song
https://openreview.net/forum?id=ZewaRoZehI
#pdes #pretraining #pde
Why Fine-grained Labels in Pretraining Benefit Generalization?
Guan Zhe Hong, Yin Cui, Ariel Fuxman, Stanley H. Chan, Enming Luo
Action editor: Dmitry Kangin
https://openreview.net/forum?id=FojAV72owK
#pretraining #labeled #deep
Adaptive Training Distributions with Scalable Online Bilevel Optimization
David Grangier, Pierre Ablin, Awni Hannun
Action editor: Changjian Shui
https://openreview.net/forum?id=JP1GVyF5i5
#pretraining #pretrained #adaptive
Strategies for Pretraining Neural Operators
Anthony Zhou, Cooper Lorsung, AmirPouya Hemmasian, Amir Barati Farimani
Action editor: Antonio Vergari
https://openreview.net/forum?id=9vEVeX9oIv
#pretraining #models #modeling
