Видео

"hey" a bezos fan , I liked bezos electricity analogy used here , he used it for aws on y combinator startup school and he used it for LLM

The presentation is great - ) I hope soon we will have new models from Anthropic:)

Man absolutely love Claude. Anthropic is absolutely amazing in terms of the finished product, completely stopped using ChatGPT almost by habit.

this micro agents structure was exactly what i was thinking yesterday and want to sell a saas about it

This is absolutely amazing!

Dudes squeaky style of speech really annoying. audience guy with totally dumb questions also annoying. 30 minutes in. Didn’t learn anything. Does it get better ??

All of this was already possible before... Already back in early 2023. What they did was just save you 15 minutes (otherwise you'd have to download an inference program and weights separately)

this is just fantastic.

As soon as you say you're backed by a Fortune 500 company, and they like your coding and data. probably means you're a sellout to that companies too and you're trying to take human data and sell it to the highest-rated players. Why should we believe you? Do you run crowdstrike as secure communication systems too?

If we see Agents as Microservices, why not reusing existing Microservices infrastructures proved reliable from years now? Truly curious about the reasons.

Thanks for speaking Daniel!!

the goat

layer norml helps to avoid gradient banishing or explotion. Before that, it was almost impossible to train a deep network.

Is this Roger from American Dad?

How can you ever know if these LLM providers don’t intentionally mix leaderboard data in smart ways to game the ranking.

@KevinHou22 - We are looking for AI experts interested in part-time collaboration. If you're interested in contributing to cutting-edge AI projects for a startup, let's connect!

🎯 Key points for quick navigation: 00:35 *📊 Overview of Workshop and Introductions* - Introduction to low-level technical analysis of language models by Daniel Han. - Discussing the purpose of finding and fixing bugs in various language model implementations. 01:56 *🐛 Finding Bugs in Gemma and Other Models* - Explanation of initial bug discoveries in Gemma, including issues with approximate vs exact calculations. - Highlighting the complexity and variation in different model implementations. 04:03 *🧠 Analyzing Architecture and Quirks of Large Models* - Discussing architectural quirks in large models like Nvidia's 340 billion parameter model. - Exploration of non-standard implementations such as squared activations. 05:16 *🧩 Challenges in Tokenization* - Addressing tokenization challenges and discrepancies across models. - Example issues with token variants causing different results across implementations. 06:25 *📊 Broadening Discussion Beyond Language Models* - Introduction to broader technical knowledge including SVD, PCA, and machine learning fundamentals. - Encouragement for exploration and understanding of foundational algorithms. 10:09 *💻 UNS Slof and Optimization Techniques* - Introduction to UNS Slof, optimizing fine-tuning of language models for efficiency. - Discussion on Triton kernels and CUDA programming for GPU optimization. 13:05 *⚙️ Understanding Sparsity in GPUs* - Explanation of sparsity feature in GPUs and its impact on training speed. - Clarifying benefits and challenges of enabling sparsity in language model training. 15:09 *📈 Learning Rate Schedules and Model Training* - Discussion on the impact of learning rate schedules and epochs on model training. - Evaluation of different methodologies and their influence on model performance. 18:11 *🏆 Impact of Bug Fixes on Model Performance* - Audience anecdote on the unexpected performance improvement post-bug fix in Gemma. - Speculation on the multifaceted nature of bug fixes and their cumulative effect. 20:00 *🛠️ Overview of GPU memory management and efficiency in model training* - Understanding how offloading GPU memory to system RAM can affect execution speed. - Importance of correct memory offloading to avoid performance degradation. 21:13 *🧠 Introduction to Transformer architecture and its applications* - Explanation of the Transformer's role in language models like GPT-4, GPT-3, and others. - Versatility of Transformers beyond language modeling for sequence modeling tasks. 32:34 *📊 Tokenization strategies: from simple to industry standard* - Creation and shortcomings of a basic tokenization method with combined punctuation. - Issues identified such as vocabulary inflation and lack of normalization. 43:17 *🧠 Understanding Sequence Modeling* - Sequence modeling involves predicting subsequent words iteratively. - Never use future data in training machine learning models. 44:16 *🎲 Importance of Tokenization in Language Models* - Tokenization requires each component to have the same number of numerical tokens. - The number of combinations in token assignment can be infinite in theory. 46:06 *🛠️ Initialization and Training Considerations* - Random initialization of model parameters can lead to issues like exploding gradients. - Proper initialization is crucial to prevent training instability. 47:34 *📊 Structure of Training Data for Language Models* - Training data for language models consists of sequences of tokenized text. - Each sequence of tokens can be represented as a table of numerical embeddings. 49:15 *🔄 Training Mechanism and Transformer Architecture* - Language models predict the next word in a sequence using shifted token predictions. - The Transformer architecture includes attention and MLP layers for prediction refinement. 50:13 *🌐 Components of Language Models* - Language models consist of prediction and MLP (Multi-Layer Perceptron) components. - The attention mechanism in Transformers enhances sequence modeling capabilities. 51:08 *🤔 Exploring Multi-Token Prediction in Transformers* - Transformers can predict multiple tokens at once by adjusting training objectives. - Multi-token prediction can expedite inference time in language models. 52:51 *🔑 Tokenization and Embedding Process* - Tokenizers convert text tokens into numerical IDs for embedding lookup. - Embedding dimensions determine the vector representation's complexity for each token. 54:24 *🚀 Enhancing Training Efficiency with Padding and Tokenization* - Padding tokens to a specific length can optimize GPU caching and training speed. - Tokenizers with padded vocabularies enhance data processing efficiency. 56:06 *⚠️ Handling Tokenization Errors and Untrained Tokens* - Tokenization errors can occur when using untrained tokens in fine-tuning. - Setting untrained tokens to mean embeddings mitigates model training issues. 59:25 *🌐 Complexity Reduction in Language Model Training* - Language models utilize attention mechanisms to reduce computational complexity. - Masked attention allows language models to skip predicting future tokens. 01:00:49 *🧩 Mechanisms of Attention and Masking in Transformers* - Attention mechanisms in Transformers utilize masking to skip irrelevant token interactions. - Softmax normalization in attention mechanisms ensures probabilistic token predictions. 01:06:26 *🧮 Softmax and Layer Norms* - Layer Norms normalize inputs across features, stabilizing training and improving model performance. 01:09:12 *📊 Backpropagation Challenges* - Differentiating Layer Norms during backpropagation involves complex matrix operations. - Triton's implementation complexities lie in managing gradients effectively for Layer Norms. 01:13:24 *🧩 Positional Encodings: Rope Embeddings* - Rope embeddings enhance Transformer accuracy by encoding positional information dynamically. - Absolute positional encodings are simpler but less effective compared to dynamic methods like rope embeddings. 01:21:01 *🔄 Derivatives and MLP in Transformers* - Deriving gradients for rope embeddings involves specialized matrix operations like rotation matrices. - MLP components in Transformers mix signals to enhance model expressiveness and learning flexibility. 01:27:54 *🧠 Understanding Matrix Operations in LLMs* - Matrix operations like W_up, W_gate, and W_down are crucial in attention mechanisms. - These matrices are trained to enhance model capacity and projection efficiency. 01:29:42 *📊 Managing Derivatives and Mathematical Formulas* - Deriving formulas manually for complex functions like softmax derivatives is challenging and time-consuming. - Tools like Desmos aid in visualizing and verifying mathematical derivations. 01:32:06 *🛠️ Enhancing Stability and Performance with Chunking* - Chunking techniques optimize GPU memory usage for large vocabulary sizes in models like Llama. - Techniques such as subtracting the maximum value in softmax enhance stability during training. 01:35:17 *🔍 Exploring Implementation Details of Llama Architecture* - Detailed examination of key components like Layer Norms (LNorm) and rotary embeddings in Llama models. - Insight into specific code segments for Layer Norm kernels and architectural optimizations. 01:50:03 *🧠 Low-level technical details of LLMs:* - Understanding the architecture of LLMs involves multiple layers and operations, culminating in generating logits for token prediction. - Upcasting to float32 from float16 enhances training stability by preventing NaNs due to large exponentials in softmax calculations. 01:57:40 *🔍 Analyzing JMA bugs:* - Detailed exploration of bugs in JMA models reveals issues such as missing BOS tokens and typographical errors in papers. 02:01:22 *🔄 Decisions in model implementation:* - Choosing between different model fixes (like the blue versus black line) involves balancing multiple errors and aligning with original implementations. 02:12:52 *🧮 Floating Point Formats and Performance Comparison* - Overview of floating point formats (float16, float32) and their transistor requirements. 02:15:13 *🚀 Future of GPU Precision: Float16 and Beyond* - Discussion on the potential future of GPU precision beyond float16. 02:22:12 *🔍 Analyzing Precision Issues in Machine Learning Models* - Issues and considerations when implementing different precision formats in machine learning models. 02:28:11 *🛠️ Debugging Challenges in Precision Implementation* - Challenges and methodologies for debugging precision-related issues in ML frameworks. 02:34:16 *🐍 Analyzing Implementation Differences* - Comparing implementations between Hugging Face, PyTorch, and J implementations. 02:36:26 *🐞 Issues with Sliding Window Implementation* - Discussing the sliding window bug in LLMs, specifically with a token limit of 2047 instead of 2048. 02:40:25 *🛠️ Tokenization Challenges and Solutions* - Addressing challenges in tokenizer configurations and functionality.

Thank you for inviting me! Let me know what you guys would like me to talk about next time! 😊 Also if you guys want me to clarify something, I'll try my best to reply! 🙏

Cursor plus codeium is BEAST

Thank you for having me! Always great to share what we’ve learned with the world.

Clutch

this guy is a war machine

I am dying for access to github copilot workspaces

that one guy on the right is annoying!!

This guy is a gem. Keep it up

Thanks for the tech and for AIE for making the video available with such great production quality. One thing I missed is what do you train on? Especially that generated code will have ownership and IP concerns/issues, is this custom trained on organization private codebase?

Brilliant talk. As an engineering leader this approach is way more powerful than the state of the art from the larger companies. Appreciate the deep dive and I can see all the deep thinking going into your product.

Justine Tunney was caught plagiarizing code from a user on llama.cpp and banned by its creator, Greg Gerganov

Shoutout to the AI Engineer World's Fair for having us and Kevin Hou! We're constantly improving our code retrieval system and working on tough problems at the cutting edge of AI. We're hiring across all roles!

Watching so far and like how engaging the workshop is! We need a workshop for Triton or CUDA entirely.

Good one!!

first

thank you

Devin, trained on the collective mistakes of two generations of stackoverflow users. Imagine if the control systems on the Nuclear Submarine Parked 100km off shore were written by Devin. Or more likely an ecomm website. Imagine what happens when Devin has a transitory hallucination, my clients celebrate negative prices and I go broke in minutes. The best use case is Devin creating the code that is usde as examples for teaching debugging.

Amazing 👏👏👏. Just tried it on ubuntu with arm (aarch64 ). Just cpu. works great. Also, RAM usage is quite low, which is surprising.

at 8:16 the presenter is comparing MAX against llama.cpp using CPU as inference, now the main contributor of llamafile claims that llama.cpp mainly focus on GPU stack (which sort of makes sense since CPU can be comparatively slower), so I'm not sure how big of a impact that would be.

What about quality trade-off? Did they mention about that?

The Real AI

Is there a way to get Windows to run llamafiles bigger than 4Gb? Without being able to do that, it is very limiting in the models you can run.

Awesome presentation. That was so well done 👍🏻

जावा में।

Justine is a GOAT

Tired -> wired around @9:30 😂

Llamafile now supports OpenAI API and non-AVX CPUs. Finally! Having the OpenAI API is a must.

Outside of Python AI bubble this is so old and natural that you would never call it an Invention 😂 Well that happens when some data scientists try to host their Jupyter Notebook 😂

Free candy, I mean, Free open source Ai for everyone. It’s a like a trick. Don’t fall for it. Cease Ai.

what music is that by the way?

I have zero interest in Modular neither in MAX as long as it’s not fully open source. They have the right to make it closed, but the “we are making some things open” without any clarity or guarantee that the rest of the stack will eventually become open, is equal to it just being closed source. I would consider it a rug pull, as Chris has been teasing the community and been earning positive press -as if- it’s an open source project.

OMG if there is new standard of API can communicate with LLM OMG that really change the world if they all use this standard automation in simple step! uh not really what about security.... like rouge LLM roaming around and exploiting those API wow more talks like these please. oh if it's open source LLM can communicate with those API might get more secured? maybe