стабилизатор напряжения 220в для дома на 8 квт цена
Главное меню:
Интересное
стабилизатор напряжения 220в для дома на 8 квт цена, Interconnects | Nathan Lambert | Substack, Latent Space: The AI Engineer Podcast - Apple Podcasts, Techmeme: A look at Apple's technical approach to AI , The interface era of AI - by Nathan Lambert - Interconnects, How to cultivate a high-signal AI feed - by Nathan Lambert, Apple Intelligence Just Made All Other AI Look Pointless, This Week in AI: Apple won’t say how the sausage gets made.
01/11/20241HR 26MIN RLHF 201 - with Nathan Lambert of AI2 and Interconnects Latent Space: The AI Engineer Podcast Play In 2023 we did a few Fundamentals episodes covering Benchmarks 101, Datasets 101, FlashAttention, and Transformers Math, and it turns out those were some of your evergreen favorites! So we are experimenting with more educational/survey content in the mix alongside our regular founder and event coverage. Pls request more!We have a new calendar for events; join to be notified of upcoming things in 2024!Today we visit the shoggoth mask factory: how do transformer models go from trawling a deeply learned latent space for next-token prediction to a helpful, honest, harmless chat assistant? Our guest “lecturer” today is Nathan Lambert ; you might know him from his prolific online writing on Interconnects and Twitter, or from his previous work leading RLHF at HuggingFace and now at the Allen Institute for AI (AI2) which recently released the open source GPT3.5-class Tulu 2 model which was trained with DPO. He’s widely considered one of the most knowledgeable people on RLHF and RLAIF. He recently gave an “RLHF 201” lecture at Stanford, so we invited him on the show to re-record it for everyone to enjoy! You can find the full slides here, which you can use as reference through this episode. Full video with synced slidesFor audio-only listeners, this episode comes with slide presentation along our discussion. You can find it on our YouTube (like, subscribe, tell a friend, et al).Theoretical foundations of RLHFThe foundation and assumptions that go into RLHF go back all the way to Aristotle (and you can find guidance for further research in the slide below) but there are two key concepts that will be helpful in thinking through this topic and LLMs in general:* Von Neumann–Morgenstern utility theorem: you can dive into the math here, but the TLDR is that when humans make decision there’s usually a “maximum utility” function that measures what the best decision would be; the fact that this function exists, makes it possible for RLHF to model human preferences and decision making.* Bradley-Terry model: given two items A and B from a population, you can model the probability that A will be preferred to B (or vice-versa). In our world, A and B are usually two outputs from an LLM (or at the lowest level, the next token). It turns out that from this minimal set of assumptions, you can build up the mathematical foundations supporting the modern RLHF paradigm!The RLHF loopOne important point Nathan makes is that "for many tasks we want to solve, evaluation of outcomes is easier than producing the correct behavior". For example, it might be difficult for you to write a poem, but it's really easy to say if you like or dislike a poem someone else wrote. Going back to the Bradley-Terry Model we mentioned, the core idea behind RLHF is that when given two outputs from a model, you will be able to say which of the two you prefer, and we'll then re-encode that preference into the model.An important point that Nathan mentions is that when you use these preferences to change model behavior "it doesn't mean that the model believes these things. It's just trained to prioritize these things". When you have preference for a model to not return instructions on how to write a computer virus for example, you're not erasing the weights that have that knowledge, but you're simply making it hard for that information to surface by prioritizing answers that don't return it. We'll talk more about this in our future Fine Tuning 101 episode as we break down how information is stored in models and how fine-tuning affects it.At a high level, the loop looks something like this:For many RLHF use cases today, we can assume the model we're training is already instruction-tuned for chat or whatever behavior the model is looking to achieve. In the "Reward Model & Other Infrastructure" we have multiple pieces:Reward + Preference ModelThe reward model is trying to signal to the model how much it should change its behavior based on the human preference, subject to a KL constraint. The preference model itself scores the pairwise preferences from the same prompt (worked better than scalar rewards).One way to think about it is that the reward model tells the model how big of a change this new preference should make in the behavior in absolute terms, while the preference model calculates how big of a difference there is between the two outputs in relative terms. A lot of this derives from John Schulman’s work on PPO:We recommend watching him talk about it in the video above, and also Nathan’s pseudocode distillation of the process:Feedback InterfacesUnlike the "thumbs up/down" buttons in ChatGPT, data annotation from labelers is much more thorough and has many axis of judgement. At a simple level, the LLM generates two outputs, A and B, for a given human conversation. It then asks the labeler to use a Likert scale to score which one it preferred, and by how much:Through the labeling process, there are many other ways to judge a generation:We then use all of this data to train a model from the preference pairs we have. We start from the base instruction-tuned model, and then run training in which the loss of our gradient descent is the difference between the good and the bad prompt.Constitutional AI (RLAIF, model-as-judge)As these models have gotten more sophisticated, people started asking the question of whether or not humans are actually a better judge of harmfulness, bias, etc, especially at the current price of data labeling. Anthropic's work on the "Constitutional AI" paper is using models to judge models. This is part of a broader "RLAIF" space: Reinforcement Learning from AI Feedback.By using a "constitution" that the model has to follow, you are able to generate fine-tuning data for a new model that will be RLHF'd on this constitution principles. The RLHF model will then be able to judge outputs of models to make sure that they follow its principles:Emerging ResearchRLHF is still a nascent field, and there are a lot of different research directions teams are taking; some of the newest and most promising / hyped ones:* Rejection sampling / Best of N Sampling: the core idea here is that rather than just scoring pairwise generations, you are generating a lot more outputs (= more inference cost), score them all with your reward model and then pick the top N results. LLaMA2 used this approach, amongst many others.* Process reward models: in Chain of Thought generation, scoring each step in the chain and treating it like its own state rather than just scoring the full output. This is most effective in fields like math that inherently require step-by-step reasoning.* Direct Preference Optimization (DPO): We covered DPO in our NeurIPS Best Papers recap, and Nathan has a whole blog post on this; DPO isn’t technically RLHF as it doesn’t have the RL part, but it’s the “GPU Poor” version of it. Mistral-Instruct was a DPO model, as do Intel’s Neural Chat and StableLM Zephyr. Expect to see a lot more variants in 2024 given how “easy” this was.* Superalignment: OpenAI launched research on weak-to-strong generalization which we briefly discuss at the 1hr mark.Note: Nathan also followed up this post with RLHF resources from his and peers’ work:Show Notes* Full RLHF Slides* Interconnects* Episode Webpage Information Show Latent Space: The AI Engineer Podcast Frequency Updated Weekly Published January 11, 2024 at 7:22 PM UTC Length 1h 26m Rating Clean United States Select a country or region Africa, Middle East, and India See All Asia Pacific See All Europe See All Latin America and the Caribbean See All The United States and Canada See All Copyright © 2025 All rights reserved. To listen to explicit episodes, sign in. Sign In Stay up to date with this show Sign in or sign up to follow shows, save episodes, and get the latest updates. Sign In Select a country or region Africa, Middle East, and India See All Asia Pacific See All Europe See All Latin America and the Caribbean See All The United States and Canada See All , Catching up with one of the leaders of open-source AI. Join for the best insider coverage of AI research. The cutting edge of AI, from inside the frontier AI labs, minus the hype. The border between high-level and technical thinking. Read by leading engineers, researchers, and investors on Wednesday mornings., Latent Space: The AI Engineer Podcast — Practitioners talking LLMs, CodeGen, Agents, Multimodality, AI UX, GPU Infra and al: RLHF 201 - with Nathan Lambert of AI2 and Interconnects on Apple Podcasts..
Apple says it doesn't train its models on users' private data or user interactions, and instead relies on licensed materials and publicly available online data — At WWDC on Monday, Apple revealed Apple Intelligence, a suite of features bringing generative AI tools like rewriting an email draft …, Search in a few years will look like one of the many tools that our chatbots use beneath a Universal Interface (UI, thanks Ben Thompson). Slightly further out, these chatbots will be the central fixture of a new era of hardware devices we're welcoming into the world., How to cultivate a high-signal AI feed Basic tips on how to assess inbound ML content and cultivate your news feed., Apple Intelligence finally brings us the sci-fi promise of computer assistants from TV shows and movies. So far, the AI hype has been all about generative junk, Chat GPT, and Google spewing made-up nonsense cobbled together from unauthorized data scraped from across the internet., Apple Intelligence powers a whole host of features, from an upgraded Siri to AI-generated emoji to photo-editing tools that remove unwanted people and objects from photos. The company, .