# Matt Carey Hey I'm Matt, welcome to my website. AI Engineer and Community Builder based between London and Lisbon. Find me on: - [GitHub](https://github.com/mattzcarey) - [Bluesky](https://bsky.app/profile/mattzcarey.com) - [X](https://x.com/mattzcarey) - [LinkedIn](https://www.linkedin.com/in/mattzcarey/) More: [Blog](https://mattzcarey.com/blog.md) · [Projects](https://mattzcarey.com/projects.md) · [Work](https://mattzcarey.com/work.md) --- # Projects ## [You've Been a Bad Agent](https://bad-agent.transistor.fm/) (podcast) 2025 - Present — Wil and Matt discuss tech, startups, and building really cool things with AI. Sometimes joined by (actual expert) friends. ## [Model Context Protocol](https://github.com/modelcontextprotocol/typescript-sdk) (open source) 2025 - Present — maintainer of the TypeScript SDK. ## [AI Demo Days](https://lu.ma/ai-demo-days) (events) 2024 - Present — the best place to see new AI tech globally, events in London, SF, NYC, Stockholm.. ## [OpenUK AI Advisory Board](https://openuk.uk/) (open source) 2024 - 2025 — got to speak to some important people about AI. ## [Shippie](https://github.com/mattzcarey/shippie) (open source) 2022 - 2025 — extensible AI code review tool ## [ParliamentWow](https://parliamentwow.com) (hackathon) 2024 - 2024 — unpicking what actually happens in parliament. Winner of a16z Hack UK October 2024 with Sunil Pai and Thomas Ankcorn. ## [Quivr (YC W24)](https://www.quivr.app) (open source) 2023 - 2023 — founding team member --- # Work ## [Cloudflare](https://cloudflare.com) — Senior Systems Engineer 2025 - Present — Agents and MCP. Built the first versions of Artifacts and maintain a bunch of tools for agents. ## [StackOne](https://stackone.com) — Founding AI Engineer 2024 - 2025 — wrote code that generated integrations for APIs you use every day and built agents to automate our company. started AI Demo Days along the way. hired some much smarter friends, built a team and raised $20M from GV (Google Ventures). ## [aleios (part of Theodo Group)](https://www.theodo.com/en-uk) — Cloud Engineer 2022 - 2024 — Cloud migrations and greenfield builds on AWS serverless. Made lots of friends and became an AWS Community Builder. Brought Serverless London back from the dead after covid - ran regular events and generally had a great time. ## Professional Windsurfing — Athlete & Coach 2017 - 2021 — Raced for Team Malta at World and European Championships, then coached junior, youth and Olympic-level athletes wanting to make waves on the world stage. --- # Posts - 2026-07-17 — [Give your agent a computer](https://mattzcarey.com/blog/agent-computer.md) - 2025-07-18 — [Codebase Patterns for AI](https://mattzcarey.com/blog/codebase-patterns-for-ai.md) - 2024-12-11 — [Advent of ML Day 11: Test Time Compute](https://mattzcarey.com/blog/advent-of-ml-day-11.md) - 2024-12-07 — [Advent of ML Day 7: Transformers](https://mattzcarey.com/blog/advent-of-ml-day-7.md) - 2024-12-06 — [Advent of ML Day 6: Measuring Success](https://mattzcarey.com/blog/advent-of-ml-day-6.md) - 2024-12-05 — [Advent of ML Day 5: Rerankers](https://mattzcarey.com/blog/advent-of-ml-day-5.md) - 2024-12-04 — [Advent of ML Day 4: BM25 and Hybrid Search](https://mattzcarey.com/blog/advent-of-ml-day-4.md) - 2024-12-03 — [Advent of ML Day 3: Embeddings](https://mattzcarey.com/blog/advent-of-ml-day-3.md) - 2024-12-02 — [Advent of ML Day 2: Data](https://mattzcarey.com/blog/advent-of-ml-day-2.md) - 2024-12-01 — [Advent of ML Day 1: Tokenizers](https://mattzcarey.com/blog/advent-of-ml-day-1.md) - 2024-09-15 — [Speech on AI at the House of Lords](https://mattzcarey.com/blog/mini-speech-on-ai-house-of-lords.md) - 2024-07-10 — [A Letter to the Users and Contributors of Code Review GPT](https://mattzcarey.com/blog/a-letter-to-users-of-code-review-gpt.md) - 2024-07-09 — [5 Skills I've needed as an AI Engineer](https://mattzcarey.com/blog/five-skills-as-an-ai-engineer-in-2024.md) All medium posts: https://medium.com/@mattzcarey --- --- title: "Give your agent a computer" description: "Keep the agent and its files durable, then give it a computer when it needs one." pubDate: 2026-07-17 canonical: https://mattzcarey.com/blog/agent-computer --- When I start a Claude Code on my laptop, it works on files and maybe runs a dev server. When I close the lid the agent might disappear but my laptop and its disk are still there (obs). To run these agents in the cloud, my laptop becomes a VPS, sandbox, or container. The agent is free to be started and maybe stopped when done. This is a perfectly sensible way to build one agent. I have a Raspberry Pi at home running this setup and it works great. It gets awkward when you want to give every user an agent. Maybe more than one. They should wake for events, run in the background, and remember jobs that take days. Keeping a Linux machine around for each one just in case it needs shell access feels like the wrong shape. Maybe we could make the agent durable and let the computers it uses be ephemeral. ## Split the harness from the tools Katelyn Lesse described this in [You can't avoid the hard part](https://x.com/katelyn_lesse/status/2073902681668931927). An early version of Claude Managed Agents spawned a sandbox, started Claude Code inside it, and kept the whole session in that container. Claude could not start thinking until it had booted. If it died, the session died. Code also ran beside MCP credentials. Anthropic split the harness from where code runs. State has to survive when an execution environment does not. Files written by one tool need to be visible to the next. Credentials need to stay on the right side of the boundary. ## A durable agent with ephemeral hands The harness owns durable identity and state. It sleeps when idle and wakes for a message, webhook, or schedule. The tools are its hands. Reading a file might be a function over a durable store. A short script could run in an isolate. A build tool might need a container with a full Linux userspace. ```text Durable agent / harness read / write / edit -------------> durable workspace execute(command) |-- isolate -------------------> same workspace `-- Linux computer ------------> borrow files run return changes ``` The workspace belongs to the agent, not to one (or all) of its tools. If a container owns the filesystem, reads and writes files. If the container dies or becomes disconnected from the agent, we loose its progress. Big sad. Instead, the container spins up with the workspace loaded and syncs its changes back to the agent when done. The agent owns everything. ## One experiment with this pattern At Cloudflare, we are testing one implementation in [`@cloudflare/workspace`](https://github.com/cloudflare/workspace). It is a preview for experiments and its design will change. Cloudflare Workspace keeps a virtual filesystem in SQLite inside a Durable Object. A Worker backend runs [just-bash](https://github.com/vercel-labs/just-bash) in an isolate for cheap textual tools. A container backend exposes the same workspace through FUSE when the agent needs Linux. ## How much work needs a Linux box? Less than I expected. Take a normal coding task. The agent reads files, searches for a symbol, changes a few lines, checks the diff, and writes some notes. `read`, `write`, and `edit` can act directly on the durable workspace. No container. No sync. Git operations can just call APIs (eg. Cloudflare Artifacts). Parsers and bundlers can run inside an isolate. That covers a surprisingly large chunk of a coding agent's tasks. A container is still there for native binaries or running a dev server. Start it when the agent needs it, return the changes, then let it go. The agent is free to pick the backend it needs ## Where this fits For one local agent, a durable computer is often the best design. The files are already there and there is no distributed system to operate. The inversion makes more sense when agents outlive requests, wake in response to events, or run in large numbers. It can also make a useful security boundary: credentials stay with the harness (agent) while a tool receives limited files and network access. You have to sync files, recover from partial failures, and decide what happens when two tools write at once. This pattern is useful when durability, scale, or isolation pays for that extra machinery. Give your agent a computer when it needs one. Then let the computer go away. The agent should still be there. --- --- title: "Codebase Patterns for AI" pubDate: 2025-07-18 canonical: https://mattzcarey.com/blog/codebase-patterns-for-ai --- wip. - **everything is tested.** this is the big one. tests are your contract. they're your documentation that the AI has done the right thing — the developer documentation, the contract between you and the code. the AI helps you write that contract. you have to invest in this, and then remove all the rest. - **use a language that AI knows.** - **set up repo tooling.** automatic tests, automatic linting, automatic type checking. be as strict as possible, but not pedantically strict. - **have solid patterns for testing.** each new feature needs a test, and the test needs to look like *this*. document the patterns in `AGENTS.md`. have good fixtures so the agent can't (and doesn't want to) cheat — because it's easy to write good tests. - **use linters extensively.** custom lint rules for bad practices and code smells. you can do things like restricting imports — basically trying to deal with the LLM-isms of our time. - **keep docs in the repo.** keep them organised; they're the public-facing surface of your repo. when anyone wants to use it, you point them to the docs that live in the repo. - **keep examples in the repo too.** docs and examples have different jobs: docs tell you *why* to do something, examples *demonstrate how* to do something. so you have docs, you have tests, and you have examples. these are the things you work on. the code is the emergent property. i think this is really important. --- --- title: "Advent of ML Day 11: Test Time Compute" pubDate: 2024-12-11 canonical: https://mattzcarey.com/blog/advent-of-ml-day-11 --- _Quick Note: Day 11 of Advent of ML is kindly sponsored by [ElevenLabs](https://elevenlabs.io). You can now listen to this post, as well as the rest of the Advent of ML series, read aloud by AI-generated Matt._ Imagine you're taking a difficult math test. Would you perform better if you had a bigger brain, or if you had more time to think about a problem? This isn't just a weird philosophical question - it's at the heart of some interesting research in artificial intelligence: scaling of computational resources at test time. ## Models That Can Reason OpenAI unveiled their O1 models in September 2024, the first models with an inbuilt chain of thought. O1 models generate special tokens which act as a notepad for the model to write down their 'thoughts'. These models don't just rush to spit out answers. They pause. They reflect. They sometimes even go back and revise their thoughts. To do this OpenAI had to spend a lot more computation at inference/run time (often called test time) to get this bump in reasoning performance. This is a big departure from the traditional approach to AI scaling which basically stated: more training data + more GPU hours = better model. The first chain of thought (COT) paper came out in January 2022, however O1 is the first model series which has included COT in the model/api architecture without a need for additional prompting or fine tuning. A few months before the O1 release, the Large Language Monkeys paper, explored the idea of increasing the number of samples generated at test time. Using a formal verification process to validate the solutions, they found increases from 15.9% with one sample to 56% with 250 samples on the SWE-Bench-Lite benchmark (a popular programming benchmark), beating the previous state of the art of 43%. Large Language Monkeys also found that the relationship between coverage and the number of samples is log-linear and can be modelled with an exponentiated power law, suggesting the existence of a previously unexplored inference-time scaling law. Essentially, if you can build a verifier for your task, you should scale the number of samples you generate at test time to improve performance. ## The Pattern-Matching Problem In research circles there is still a lot of debate about whether models actually 'reason'. One of the more popular benchmarks used to test reasoning is the Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI or just ARC). When François Chollet created ARC in 2019, he was trying to prove a point about AI's limitations. His stance is that AI systems aren't really thinking at all. They are pattern matching machines and true intelligence isn't about pattern matching - it's about being able to derive novel solutions. The ARC-AGI presented deceptively simple visual puzzles that required discovering underlying rules from just a few examples. They are really quite easy for humans. However, for the last few years, AI progress was slow. From 2020 to early 2024, the top score only increased from 20% to 33%. Even as language models got dramatically better at other tasks, they remained stumped by ARC. The original GPT-3 scored 0% via direct prompting. ![ARC-AGI](/images/arc-agi.png) Example ARC-AGI task (0ca9ddb6) _picture credit: [ARC Prize 2024: Technical Report](https://arcprize.org/media/arc-prize-2024-technical-report.pdf)_ ## Paths to Better Reasoning The year 2024 witnessed significant breakthroughs in ARC scores, primarily driven by the concept of scaling at test time. During the 2024 ARC Prize competition, a $1 million challenge to surpass and open-source a solution to ARC-AGI, scores improved dramatically, with MindsAI achieving an impressive 55.5% on the private evaluation set. The most successful strategies can be summarized into three categories, which are explored in more detail in the ARC Prize 2024 technical report. ### Deep Learning-Guided Program Synthesis Ryan Greenblatt's groundbreaking approach showed how to use test-time compute for systematic exploration: - Generate thousands of potential Python programs for each puzzle - Use an LLM (GPT-4o) to guide the search and debug promising candidates - Keep refining solutions through multiple iterations - Apply sophisticated verification to test programs against examples The key insight was using the LLM not to solve the problem directly, but to guide a search through program space. With enough computation time, this approach achieved up to 42% accuracy. ### Test-Time Training (TTT) MindsAI pioneered a different strategy that showed the power of real-time adaptation: - Generate additional variations of the example problems - Apply stability-based selection criteria to validate solutions - Additional fine-tuning is performed on these specific transformations This "test-time training" proved that models could effectively learn from just a few examples during inference, challenging the traditional fixed-model paradigm. ### Hybrid Approaches The most successful team, ARChitects - 53.5%, recognized that different puzzles needed different strategies. They used a combination of heuristics to determine the best approach for each puzzle. Then they used a combined method of program synthesis with direct prediction (input + task description => solution) and applied these multiple solving strategies in parallel. ### Selective Information Fine-Tuning Although not applied to ARC (yet), there is a new paper from Jonas Hubotter et al. at ETH Zurich, Switzerland which shows a new approach to test-time compute scaling. They call it Selective Information Fine-Tuning (SIFT). Similarly to the MindsAI approach on ARC, SIFT allows models to learn and adapt during use by performing some training of the model on the fly. The key innovation is how it selects what to learn from - optimizing for information gain rather than just similarity. Whereas test time training (TTT) generates a new training set from each example, SIFT uses an external corpus of data from which it selects specific fine-tuning examples. This approach should be much more generally applicable to problems where you have an external corpus of data already available. ## Learnings from ARC 2024 The ARC challenge revealed a fundamental trade-off in how computation time can be used through this new scaling law. Program synthesis approaches excel at problems with precise, rule-based answers, but their cost grows exponentially with program complexity. Test-time training, on the other hand, shows particular strength in pattern recognition tasks by fine-tuning on specific examples during inference. The most successful systems learned to combine both approaches. However, important limitations remain. Even infinite compute time cannot overcome a model's basic limitations - if a model lacks the architectural capacity or training to understand certain concepts, no amount of test-time computation will help. A model's base quality strongly influences its maximum achievable performance through test-time compute, suggesting these techniques amplify existing capabilities rather than creating new ones. Real-world applications face several practical challenges. Program synthesis approaches face exponentially growing costs as program size increases. Test-time training requires significant computation for each example. Perhaps most challengingly, verification remains difficult - determining whether a solution is correct often requires either expensive computation or imperfect heuristics. These imperfect verifiers place an upper bound on achievable accuracy. ## Future of Reasoning The massive increase in performance in ARC-AGI during 2024 suggests we're at an interesting inflection point in AI development. While we still haven't achieved human-level reasoning (humans easily score 97-98% compared to the current best of 55.5%), we're seeing new approaches that look more like genuine problem-solving. Perhaps most importantly, these developments are forcing us to reconsider what we mean by "intelligence." Instead of just building bigger pattern-matching machines, we're starting to create systems that can actually explore, reason, and learn from their attempts - even if it takes a little more compute at run time. Personally, I am interested in seeing how test time scaling will be used by developers to build more intelligent application. Will the compute requirements for AI shift from the current training dominated allocation to a more balanced allocation with test time compute being another control knob to dial in? Could be interesting, especially for the Nvidia stock price :) Happy day 11! Matt --- Resources: - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903) - [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://arxiv.org/abs/2407.21787) - [Scaling LLM Test-Time Compute Optimally](https://arxiv.org/abs/2408.03314) - [Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs](https://arxiv.org/pdf/2410.08020.pdf) - [François Chollet on Pattern Recognition vs True Intelligence](https://www.youtube.com/watch?v=JTU8Ha4Jyfc) - [ARC Prize](https://www.arcprize.org) --- --- title: "Advent of ML Day 7: Transformers" pubDate: 2024-12-07 canonical: https://mattzcarey.com/blog/advent-of-ml-day-7 --- How did we get from simple neural networks to the powerful language models of today? What actually is a transformer? Today, we're having a look into the architecture that revolutionized natural language processing and laid the foundation for modern AI. The story of transformers is fascinating because it represents one of those rare paradigm shifts where a new architecture completely changes what's possible. Before 2017, the idea that a model could write coherent paragraphs, generate code, or engage in meaningful dialogue seemed like science fiction. Now, it's reality. ## Before Transformers Before the last 7 or so years, Recurrent Neural Networks (RNNs) were the go-to architecture. The idea was simple and intuitive: process text one word at a time, maintaining a "memory" of what came before. This worked reasonably well for short sequences but had major limitations: 1. **Vanishing Gradients**: As sequences got longer, the network struggled to connect information from many steps ago 2. **Sequential Processing**: Each word had to be processed one after another, making training slow 3. **Limited Context**: Practical limitations meant RNNs could only look back a few dozen words Researchers tried various solutions. LSTMs (Long Short-Term Memory) and GRUs (Gated Recurrent Units) helped with the vanishing gradient problem. Bidirectional architectures allowed models to look at both past and future context. But these were incremental improvements. ## The Transformer The 2017 paper "Attention is All You Need" introduced the Transformer architecture and changed the course of AI. Attention itself had been around for a while. It was introduced in the 2014 paper from Yoshua Bengio's lab titled "Neural Machine Translation by Jointly Learning to Align and Translate". They combined RNNs with "context vectors" (what we now call attention). Remarkably, both the original attention paper and the Transformer paper were focused on solving one specific problem: machine translation. Neither paper talked about creating a general-purpose sequence computer. They were just trying to make Google Translate better! Also interesting is that neither papers were particularly well received by the research community at the time. They were considered a niche solution to a narrow problem. The Transformer was published at NeurIPS 2017, one of the top AI conferences worldwide. Yet it didn't even get an Oral presentation, let alone awards. ## What made Transformers special? First lets have a quick primer on attention. Attention can be thought of like this: when you read a sentence, you don't process it one word at a time. You look at groups of words together, understanding how they relate to each other. This is what self-attention does. Transformer models are able to capture complex relationships in text that sequential models missed. The Transformer was revolutionary because it pushed attention to the extreme. Instead of using attention as just one component alongside RNNs, it showed that attention alone could power the entire architecture. The model could look at the entire sequence at once, using pure self-attention to weigh the importance of different words in relation to each other. This architecture unlocked unprecedented parallel processing capabilities on GPUs. Because attention can process all tokens simultaneously during training rather than waiting for sequential computations, training speed and scalability reached new heights. This was the game-changer that enabled the massive models we see today. ## What's Next? The main drawback of transformers is that they are not memory efficient with long contexts. As every token attends to every other token, the memory requirements grow quadratically with the sequence length. This has sparked several interesting research directions: ### RWKV (Receptance Weighted Key Value) RWKV combines the best aspects of RNNs and Transformers. It processes text sequentially like RNNs (making it more memory efficient) but uses mechanisms inspired by transformers to maintain performance. RWKVs can potentially handle much longer context windows with linear memory requirements. ### Alternative Attention Mechanisms - **Paged Attention**: Implemented in vLLM, this approach manages attention computation more efficiently by breaking it into smaller chunks - **Sparse Attention**: Only compute attention for the most relevant tokens, approximating the attention matrix to reduce complexity. - **Flash Attention**: Optimizes memory access patterns for faster computation. Essentially it compresses operations into a single `fused` kernel. ### State Space Models Recent work like the Mamba architecture shows promise in modeling sequences using state space models instead of attention. These models can process sequences in linear time and memory, potentially offering better efficiency for certain tasks. While there's still fierce debate about whether transformer-based models can achieve AGI (Artificial General Intelligence), recent innovations in inference and training methods continue to push the boundaries of what's possible with transformers. For now it seems the 2017 authors were right and attention is really all you need. Tomorrow we will look at attention in more detail. Happy day 7! Matt --- Resources: - [Attention Is All You Need](https://arxiv.org/abs/1706.03762) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473) - [RWKV: Reinventing RNNs for the Transformer Era](https://arxiv.org/abs/2305.13048) - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752) - [vLLM: Easy, Fast, and Cheap LLM Serving](https://docs.vllm.ai/en/latest/) - [Flash Attention: Fast and Memory-Efficient Exact Attention](https://arxiv.org/abs/2205.14135) - [ELI5: FlashAttention](https://gordicaleksa.medium.com/eli5-flash-attention-5c44017022ad) --- --- title: "Advent of ML Day 6: Measuring Success" pubDate: 2024-12-06 canonical: https://mattzcarey.com/blog/advent-of-ml-day-6 --- How do you know if your AI system is working? What metrics should you track? When should you use LLM-as-a-judge? Over the past few days, we've explored various components of AI systems - from tokenizers to embeddings to hybrid search. Today, we'll tackle the crucial but often overlooked topic of evaluation. I'm thinking about these questions a lot today. I've been building a new retrieval strategy at StackOne this week. It's been a lot of testing chunking, finding the limits of usable context windows and building datasets to try and test everything. The changes feel good - response times are slower but within threshold, and the results look better at first glance. But "feeling good" and "looking better" aren't metrics you can track or improve upon systematically. This is the challenge at the heart of building AI systems: how do you measure success in a meaningful way? ## AI Testing If you're coming from a software engineering background, the way people work in AI and ML might feel quite alien. It often seems like everyone is playing in notebooks rather than working on production systems. That's because the logic flow in these systems is normally not that complex - take some data from somewhere, store some indexes in a DB, and at runtime query and send it to an LLM. However, it's the knowledge and insight to build the right system that is hard to come by. As everyone has been telling me for the past two years, AI demos very well but production is hard. How do you handle adversarial questions? How do you handle hallucinations? How do you handle data drift? What do you do when your user uploads an 8,500-page PDF file of handwritten notes? ## Eval Strategy ### 1. Start with Simple Assertions The simplest place to start is with basic string matching and assertions. Find test cases where you know exactly what the output should look like. While testing, literally add these assertions to your running code. This helps you: - Get your prompts in the right space - Ensure your system fails gracefully - Build confidence in some basic functionality For example, if you're building a system to extract dates from text, you might assert that "December 25th, 2024" is correctly identified as a date. ### 2. Measure Retrieval Quality Retrieval (see [Day 3](./advent-of-ml-day-3.md), [Day 4](./advent-of-ml-day-4.md) and [Day 5](./advent-of-ml-day-5.md)) is normally where everything starts falling apart and can be the lowest hanging fruit for improvement. I've definitely been guilty of spending time making end to end tests when it was actually the retrieval that was the problem. - Precision@K: How many of the top K retrieved documents are relevant? - Recall@K: What fraction of relevant documents are in the top K? - Mean Reciprocal Rank (MRR): How highly ranked is the first relevant document? If you're using rerankers (from [Day 5](./advent-of-ml-day-5.md)), track the average reranker scores over time. A declining trend might indicate: - Data drift in your source documents - Changes in user query patterns - Issues with your embedding model ### 3. LLM-as-a-Judge LLM-as-a-judge isn't a silver bullet. Here are some guidelines: - Avoid using models from the same family to evaluate each other (e.g., GPT-4o evaluating GPT-4o outputs). They tend to be overly nice to their family. - Use pairwise comparisons instead of absolute scoring. - Include clear evaluation criteria in your judge prompts - Validate judge decisions against human evaluations Models have no idea what "good" is, you will have more success with solid questions like: does this answer contain the 3 points from this reference answer? Be sure to validate your judges against human evaluations and include examples of good and bad in your prompts. ## Building Your Dataset You need three things to start evaluating: 1. Representative questions/queries 2. Gold standard answers or relevance judgments 3. A process for collecting more data The questions you should be able to get from users, your boss, or a previous product. The answers can be harder to come by. ### Subject Matter Expert (SME) Approach - Find the person who knows your domain best, or buckle up cause this is going to be you. - Send them one question per day via email or Slack and try get an answer back. - Often the hardest part of this process is you have to become the SME yourself. ### Synthetic Data Generation - Use larger models to generate test cases based on some examples and domain knowledge - Validate a subset with humans - Use for stress testing and edge cases ## Creating a Data Flywheel The real power comes from creating a continuous improvement cycle: 1. Collect user interactions and feedback 2. Label and validate examples 3. Use validated examples to: - Improve retrieval - Train better judges - Generate synthetic data 4. Feed improvements back into production 5. Repeat This flywheel effect not only improves your current system but also opens up possibilities for fine-tuning models and more sophisticated strategies in future. ## Practical Tips 1. **Start Small**: Begin with a core set of test cases that represent your most important use cases. 2. **Log Everything**: You can't improve what you don't measure. Log: - User queries - Retrieved documents - Generated responses - User feedback (explicit and implicit) 3. **Build for Iteration**: Your first evaluation system won't be perfect. Design it so you can easily: - Add new test cases - Modify evaluation criteria - Update gold standard answers 4. **Make Evaluation Easy**: Remove friction from the evaluation process: - Build simple tools for annotators - Automate what you can - Make results easily accessible Remember, the goal isn't to achieve perfect scores on your metrics - it's to build a system that reliably helps your users. Sometimes a simple system with clear limitations is better than a complex one that fails in unpredictable ways. Happy day 6! Matt --- Resources: - [Your AI Product Needs Evals - Hamel Hussain](https://hamel.dev/blog/posts/evals/) - [Applied LLMs - Evaluation Monitoring](https://applied-llms.org/#evaluation-monitoring) - [LLM Judge - Hamel's Blog](https://hamel.dev/blog/posts/llm-judge/) - [Who Validates the Validators? - Shreya Shankar et al.](https://arxiv.org/abs/2404.12272) - [How Dosu Used LangSmith for Continual Learning](https://blog.langchain.dev/dosu-langsmith-no-prompt-eng/) --- --- title: "Advent of ML Day 5: Rerankers" pubDate: 2024-12-05 canonical: https://mattzcarey.com/blog/advent-of-ml-day-5 --- Cohere just released Rerank v3.5 and Pinecone just released Rerank v0. But what exactly are rerank models? How do they work? And how are they useful in a retrieval augmented generation (RAG) pipeline? Yesterday, we explored hybrid search, combining [BM25 algorithm](./advent-of-ml-day-4.md) with semantic search using embeddings. We discussed using reciprocal rank fusion (RRF) to merge results. Today, we'll dive into a more sophisticated approach: cross-encoder rerankers. ## What are Rerankers? Imagine you're searching through a massive library. First, you quickly scan the shelves for potentially relevant books (this is like BM25 or embedding search). Then, you pick up each promising book and carefully examine its contents, ordering them by relevance (this is reranking). While initial retrieval needs to be fast and work with potentially millions of documents, reranking only needs to handle the top-k results (usually 20-100 documents), allowing us to use more sophisticated models. ## From Bi-Encoders to Cross-Encoders Embedding models (like those we discussed on [Day 3](./advent-of-ml-day-3.md)) are bi-encoders: they encode queries and documents separately into vectors and then compare with a nearest neighbor search. This approach is computationally efficient but misses more subtle relationships. Cross-encoders look at the query and document together, enabling them to be more nuanced. The trade-off? They're much slower, which is why we use them only for reranking a small set of documents during any one search. Rerankers are built on large language models fine-tuned specifically for relevance ranking. They typically: 1. Take a query-document pair as input 2. Process them together through multiple transformer layers 3. Output a relevance score For Cohere's latest model, they've focused training on specific fields like finance and legal, along with including many more examples of structured data (JSON, XML etc). Recent rerankers are also multi-lingual and much better at handling non-English documents. Fun fact: you can actually trick any LLMs into ranking documents by restricting the tokens that they can output. Restricting the output to a binary `Yes` or `No` for relevance and then using the logprobs as a score is a simple way to do this. Here is the [OpenAI cookbook](https://cookbook.openai.com/examples/search_reranking_with_cross-encoders) doing just that using their `completions` API. ## Retrieval Pipeline With hybrid search and a reranker, a typical pipeline looks like this: 1. **Initial Retrieval**: Use fast methods (BM25/embeddings) to get top-k candidates 2. **Reranking**: Apply cross-encoder to re-score candidates 3. **Final Ranking**: Sort by final scores ![Reranking Pipeline](/images/rerank_workflow.png) _picture credit: Pinecone_ ## Measuring Success How do you know if your reranker is working? Common evaluation metrics for retrieval systems include: - Mean Reciprocal Rank (MRR) - Normalized Discounted Cumulative Gain (NDCG) - Precision@k You can also use an LLM as a judge (optionally finetuned) with a set of multi-shot examples. Hamel Hussain has a great [blog post](https://hamel.dev/blog/posts/llm-judge/) on this topic. Tomorrow, we'll go into some more practical examples to evaluate your new retrieval system. Happy day 5! Matt --- Resources: - [Cohere Rerank v3.5 Release](https://txt.cohere.com/rerank/) - [Pinecone Rerank v0 Announcement](https://www.pinecone.io/blog/pinecone-rerank-v0-announcement/) - [Reranking with Cross-Encoders - OpenAI Cookbook](https://cookbook.openai.com/examples/search_reranking_with_cross-encoders) - [LLM Judge - Hamel's Blog](https://hamel.dev/blog/posts/llm-judge/) --- --- title: "Advent of ML Day 4: BM25 and Hybrid Search" pubDate: 2024-12-04 canonical: https://mattzcarey.com/blog/advent-of-ml-day-4 --- How do search engines actually work? When you type "cute cat videos" into a search bar, what's happening behind the scenes? And why do modern AI companies still use search algorithms from the 1990s? [Yesterday, we explored embeddings models](./advent-of-ml-day-3.md) and how they enable semantic search. If you remember its all about finding the distance between vectors in a high-dimensional space. Today we are moving away from pure AI and looking at another older search method that still powers much of the internet today: keyword search or full-text search. The tenuous link to AI is that a lot of AI powered applications implement a retrieval system to find relevant documents. Building a good search is a vital skill for many AI applications. The idea behind keyword search is simple: look for documents that contain the exact words in your query. If you search for `python programming`, it will find documents with `python` and `programming` in them. In practise it gets a little bit more complicated than that, but not much. For example, a document mentioning `python` once isn't as useful as a Python programming tutorial which will contain multiple references to `python` and `programming`. This is where BM25 comes in: an algorithm that has been the backbone of search engines since the 1990s. ## What is BM25? BM25 (Best Match 25) is a ranking function that determines how relevant a document is to a search query based on the appearance of query terms in the document. It's the successor to TF-IDF (Term Frequency-Inverse Document Frequency) and was developed by Stephen Robertson and others at City University, London. It has a pretty elegant mathematical formula: $$ score(D,Q) = \sum_{i=1}^n IDF(q_i) \cdot \frac{f(q_i,D) \cdot (k_1 + 1)}{f(q_i,D) + k_1 \cdot (1 - b + b \cdot \frac{|D|}{avgdl})} $$ Where: - D is the document - Q is the query - f(qi,D) is the frequency of term qi in document D - |D| is the length of document D - avgdl is the average document length - k1 and b are free parameters Don't worry too much about about that – the key is that BM25 considers: 1. How often the query terms appear in a document 2. How rare those terms are across all documents 3. The length of the document (to avoid bias towards longer texts) ## Why BM25 Still Matters While semantic search is powerful, BM25 has several advantages: - Extremely fast and computationally efficient - Excellent at finding specific terms such as product codes or IDs - No training required - Completely interpretable results - Works well with technical terms and proper nouns A strange fact is that over the last 20 years, users have been conditioned to use keyword search by using Google. They are often very good at naturally picking the right search terms, making BM25 a great method to have in your search stack. Imagine you are looking for a replacement part for a washing machine or a specific computer. If you know the brand and/or model number, a keyword search will be very effective. Semantic search likely will be too general. ## Hybrid Search As with many things in software, its all about the trade-offs. Different search tasks need different approaches. Consider these queries: "What's the capital of France?" - BM25 wins: The answer is probably in a document containing "capital" and "France" "I'm looking for vacation spots with good surfing and a relaxed vibe" - Embeddings win: Understanding the semantic meaning of "relaxed vibe" is crucial The solution? Use both and combine them! A basic approach is using reciprocal rank fusion (RRF). Read more about that algorithm in the resources below. In future days we will cover more sophisticated options, such as the brand new Cohere Rerank v3.5, which adds a bit of deep learning to the mix. ## End-to-End Example Let's build a simple hybrid search system in pseudocode: ```python class HybridSearcher: def __init__(self, documents): # Initialize BM25 tokenized_docs = [doc.split() for doc in documents] self.bm25 = BM25(tokenized_docs) # Initialize embeddings self.model = EmbeddingModel() self.doc_embeddings = self.model.encode(documents) self.documents = documents def search(self, query, k=10): # Get BM25 results bm25_scores = self.bm25.get_scores(query.split()) bm25_results = [(i, score) for i, score in enumerate(bm25_scores)] bm25_results = sorted(bm25_results, key=lambda x: x[1], reverse=True)[:k] # Get embedding results query_embedding = self.model.encode([query]) similarities = cosine_similarity(query_embedding, self.doc_embeddings)[0] embedding_results = [(i, score) for i, score in enumerate(similarities)] embedding_results = sorted(embedding_results, key=lambda x: x[1], reverse=True)[:k] # Combine results final_results = reciprocal_rank_fusion(bm25_results, embedding_results) return [(self.documents[i], score) for i, score in final_results[:k]] ``` ## Takeaways The best search system is the one that helps users find what they're looking for. If you haven't tried adding BM25 to your retrieval system, you should. It is quick to implement, and often worth just trying it out. How do you know if it's working? Evals :) But that's a topic for another day. See you tomorrow! Matt --- Resources: - [Reciprocal Rank Fusion Explained in 2 mins](https://medium.com/@devalshah1619/mathematical-intuition-behind-reciprocal-rank-fusion-rrf-explained-in-2-mins-002df0cc5e2a) - [Hybrid Search Implementation](https://docs.turbopuffer.com/docs/hybrid-search) - [The Probabilistic Relevance Framework: BM25 and Beyond](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf) - [Practical BM25: Part 2 - The BM25 Algorithm and its Variables](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables) --- --- title: "Advent of ML Day 3: Embeddings" pubDate: 2024-12-03 canonical: https://mattzcarey.com/blog/advent-of-ml-day-3 --- We hear a lot about generative language models like Claude Sonnet and GPT-4, or open-source alternatives like the new QwQ models. However, there's a class of models we don't hear much about: embeddings models. This will be the start of a few-day deep dive into using machine learning to build really good search systems across various document types. You've probably heard of RAG (Retrieval Augmented Generation): Get a question, search for relevant context, include it in a prompt (hopefully containing the answer), and ask a generative model to answer based on that context. But how do you actually do the search part? ## Embeddings Embeddings models translate tokens (see Day 1) into vectors - essentially lists of numbers - that represent semantic meaning in a high-dimensional space. Think of it like giving each piece of text coordinates in a vast semantic universe: - Similar concepts end up close together - Different concepts end up far apart - Related concepts maintain meaningful distances For example, in this space: - "dog" and "puppy" might be very close - "cat" would be relatively nearby (it's still a pet) - "car" would be much further away - But "vehicle" and "car" would be close to each other Embeddings search is just a fancy name for finding the closest results to your query in the vector space. You can use cosine distance or other metrics to find the `nearest neighbors`. ## Model Architecture Embeddings models generally start out as a pre-trained language model such as BERT (hello 2018). This is a smallish (by today's standards) deep learning model already trained to understand language. Some tricks are applied to extract how the model encodes the semantic meaning of tokens. They chop off the decoder (we only want the latent representation not the output) and add a projection layer to get to the final embedding size. Models typically output vectors with 1,536, 3,072, or even bigger dimensions. Each dimension captures some aspect of meaning. The full vector represents the complete semantic content. ## Common Uses 1. **Semantic Search** - Convert documents and queries into embeddings - Find nearest neighbors in the vector space - Better than keyword matching (sometimes!) 2. **Recommendation Systems** - Embed items and user preferences - Recommend based on vector similarity - Works across different types of content 3. **Clustering and Organization** - Group similar documents automatically - Find themes in large collections - Create knowledge graphs 4. **Multimodal Applications** - CLIP-like models embed both images and text - Enable searching images with text (and vice versa) - Power multimodal AI applications ## Fine-tuning While pretrained embedding models are powerful, fine-tuning them on your specific data can dramatically improve performance on your particular use case. It allows for better handling of domain-specific terms and nuances. The training process involves: - Curate a training set of pairs of items, where some pairs are marked as “similar” or "positive" (like sentences from the same paragraph) and some are marked as “different” or "negative" (like two sentences chosen at random) - Train the model to distinguish between similar and different items That's it! This process is relatively low-cost compared to other types of model training (see day 2) and can yield significant improvements. Below you can find a great article below on how to do this with a few hundred data samples on Modal. There is also a more in depth article by Tom Aarsen on the nuts and bolts of the Sentence Transformers library. Interesting note: State-of-the-art embedding models are open source! Starting with relatively small pre-trained LLMs, advances in GPU-poor techniques and healthy competition have allowed open-source models to rival or even surpass proprietary ones. The best improvements often come from fine-tuning datasets and clever architecture tweaks—achievable without the massive budgets needed to train SOTA LLMs. ## Other Interesting Ideas Before we wrap up, here are some fun avenues to explore to improve your search embeddings: - **Late stage interaction with Vision Models**: Models like ColPali use Vision Language Models to embed document images directly, skipping OCR. - **Dual Encoder Models**: By using separate encoders for queries and documents and optimizing them jointly, these models enhance retrieval performance by capturing nuances specific to each. - **Synthetic Data Generation**: Creating artificial data that resembles potential queries can improve model robustness—a technique known as data augmentation. - **Query Expansion**: Modifying queries to include additional relevant terms helps in retrieving more pertinent results, enhancing search effectiveness. The next time you use a good search system or get a relevant recommendation, remember: there's probably an embeddings model working behind the scenes! Happy day 3! Matt --- Resources: - [An intuitive introduction to text embeddings](https://stackoverflow.blog/2023/11/09/an-intuitive-introduction-to-text-embeddings/) - [How to Build a State-of-the-art Text Embedding Model](https://medium.com/snowflake/how-to-build-a-state-of-the-art-text-embedding-model-a8cd0c86a19e) - [Fine-tuning Embeddings Efficiently - Modal Labs](https://modal.com/blog/fine-tuning-embeddings) - [ColPali: Efficient Document Retrieval with Vision Language Models - Manuel Faysse](https://huggingface.co/blog/manu/colpali) - [Training and Finetuning Embedding Models with Sentence Transformers v3 - Tom Aarsen](https://huggingface.co/blog/train-sentence-transformers) --- --- title: "Advent of ML Day 2: Data" pubDate: 2024-12-02 canonical: https://mattzcarey.com/blog/advent-of-ml-day-2 --- Why do LLMs require such massive amounts of data? What exactly are data labellers, and why are they so important? What are the different ways AI models learn from data? Large Language Models are, well, large. Llama 3.1 405B was trained on over 15 trillion tokens. To put it into perspective, the whole of Wikipedia contains about 2.24 billion tokens, 6,800x smaller in size! So if you want to use a language model to solve a task, what data do you need to collect? Well, the answer is, as with most things: it depends. It turns out that we need very different types of data for different stages of LLM development. ## Crash Course on LLM Training 1. **Base Model Pre-training** - Uses massive amounts of data from different sources including the internet - Mostly unsupervised learning = no labels on the data - Builds general language understanding - Data is generally scraped from the internet and datasets like Common Crawl, Books, Wikipedia are used 2. **Reinforcement Learning from Human Feedback (RLHF)** - Pre-training data can often have a bunch of biases and not safe content. - RLHF reduces harmful outputs and allows model companies to add personality and preferences to the model - Very useful for chatbots and other applications that require a lot of human-like interaction - Requires data labellers (human annotators) to provide manual feedback on model outputs! - Dataset sizes are smaller than the pre-training stage (approx 10 - 100k samples) 3. **Fine-Tuning** - Used to add specific capabilities to the model - Needs carefully curated, task-specific datasets - Much smaller datasets than the previous stages (approx 100 - 10k samples) 4. **In-Context Learning (Prompting)** - Also gives the model knowledge of specific tasks - Uses some retrieval system to find relevant data and add it to the prompt - The whole process is normally called RAG (Retrieval Augmented Generation) - Normally uses 1- 10 samples (limited by usable context window) ## Which Stage do I Focus On? The reality is that the first two stages of training are very expensive. Collecting huge amounts of clean data is hard. Adding consistent labels to model outputs is even harder and requires teams of trained annotators. As an LLM practitioner not working in a large lab you would mostly focus on the last two. Generating data from larger models is a very interesting area of research. Generally called `synthetic data`, it is a way to create training data for smaller task specific models. We often call this process of taking a larger model and generating data for a smaller model `model distillation`. ## Why Data Matters Understanding the role of data in LLM development helps us grasp both the capabilities and limitations of these models. When an LLM makes a mistake, it might be because: - It never saw similar examples in training - The training data was noisy or incorrect - The labeling was inconsistent - Contradictory information was supplied in the prompt - Any combination of the above Happy day 2! Matt --- Resources: - [Google LLM Crach Course - Tuning](https://developers.google.com/machine-learning/crash-course/llm/tuning) - [Introducing Llama 3.1](https://ai.meta.com/blog/meta-llama-3-1/) - [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155) --- --- title: "Advent of ML Day 1: Tokenizers" pubDate: 2024-12-01 canonical: https://mattzcarey.com/blog/advent-of-ml-day-1 --- Why do LLMs gaslight us that there are 2 Rs in Strawberry? Why is it so hard to get the right answer to the question "which is bigger, 9.8 or 9.11?" The problem, or at least part of the problem, is the fault of a component called a tokenizer. Neural networks powering LLMs like GPT-4 and Claude Sonnet, ironically given the name large language model, cannot work with sentences very well. A tokenizer's job is to break down text into smaller units that the model can understand. These units, called tokens, can represent words, parts of words, or even individual characters. The most common approach used in modern language models is using an algorithm called Byte Pair Encoding (BPE). ## How BPE Works BPE is an elegant algorithm that identifies and combines the most frequently occurring pairs of characters or subwords: 1. Start with a basic vocabulary of individual characters 2. Count all pairs of adjacent characters in the training data 3. Merge the most frequent pair and add it to the vocabulary 4. Repeat the process until reaching a desired vocabulary size For example, let's see how BPE might process the word "strawberry": - Initial split: s, t, r, a, w, b, e, r, r, y - Common pairs might be merged: st, ra, rr, ry - Final tokens might be: [st][raw][berry] This explains why models might struggle with spelling - they see words as combinations of subword pieces rather than whole units. Moreover, the word "strawberry" might be tokenized differently in different contexts. Check out [OpenAI’s handy tokenizer website](https://platform.openai.com/tokenizer) to see how your words are being split! ## Training a Tokenizer Creating a tokenizer involves several key steps: 1. **Data Collection**: Gather a large, representative corpus of text in the target language(s) 2. **Preprocessing**: Clean the text and apply basic normalization (handling spaces, special characters, etc.) 3. **Vocabulary Building**: - Start with base characters - Apply BPE or similar algorithm to find common patterns - Create a final vocabulary of desired size (typically 32,000 to 50,000 tokens) 4. **Special Token Addition**: Add special tokens like [START], [END], [PAD], etc. The resulting tokenizer is then "frozen" and used consistently for both training and inference of the language model. ## Why LLMs Can’t do Maths (yet) Remember the "9.8 vs 9.11" example? The problem arises because numbers are often split into separate tokens. For instance: - "9.8" might become ["9", ".", "8"] - "9.11" might become ["9", ".", "1", "1"] This tokenization makes it harder for the model to understand the numerical values as single units, leading to general confusion. The next time you see a language model make an strange mistake, consider if it could be the tokenizer playing tricks. Happy day 1! Matt --- Resources: - [Let's build the GPT Tokenizer - Andrej Karpathy](https://www.youtube.com/watch?v=zduSFxRajkE) - [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - Aaditya K. Singh, DJ Strouse](https://arxiv.org/abs/2402.14903) - [Tiktoken - Github](https://github.com/openai/tiktoken) - [_(NEW)_ Number Tokenization - Hugging Face](https://huggingface.co/spaces/huggingface/number-tokenization-blog) --- --- title: "Speech on AI at the House of Lords" pubDate: 2024-09-15 canonical: https://mattzcarey.com/blog/mini-speech-on-ai-house-of-lords --- On September 11, 2024, I had the privilege of attending a roundtable discussion on AI openness at the House of Lords, hosted by Lord Wei. This event was part of a series of roundtable discussions scheduled between September and December, focusing on various aspects of AI policy and development with particular focus on AI openness. The roundtable featured speakers from industry and policy organizations, including parliamentarians such as Baroness Stowell. I was invited to share my perspectives as a member of the OpenUK AI advisory board. Below is the quick speech I delivered as part of opening the discussion. It is not a polished speech and I am not a politician. I am an engineer trying to do my best to understand the issues and give my views. --- Good morning. I am Matt Carey, an AI engineer and a member of the AI community here in London. I will give my views as a practitioner, as an open source contributor, as OpenUK AI advisory board member and as a team lead at a startup building and hiring here in London. A quick note about definitions of AI openness. I find it funny, almost a quirk that open weights are considered open source. I cannot replicate an open weight model from purely the weights alone with no dataset. Therefore imo the 'source' is not really open and that model is effectively a black box. I have no interpretability on decision making and the best I can be is reactive with my observations. I have a couple of points on the topic of AI regulation in this parliament and I'd like to raise some worrying trends I have been seeing which seem misguided. Firstly, I would hugely discourage any regulation which affects individual developers directly, whether they be individual contributors or researchers. Regulations proposed such as putting arbitrary limits on the size of a model being trained (this is normally done in terms of parameters) or the number of FLOPS (operations/computation) used to train the model is a folly. These are almost meaningless metrics and can be fiddled in multiple ways. They will not tell you anything about the capabilities of the model or whether those capabilities are safeguarded against attacks or misuse. They are nice numbers, but not a good basis for policy. You can speak to me afterwards if you would like to chat more about the technicalities of these metrics. Secondly, I would urge the policy makers to put their efforts into working with universities. Funding PHD programs to work on building better methods of model testing and understanding. We should embrace closed benchmarks and make strides into building our own. Government can use these to determine model strength in the future when deciding on thresholds for further regulation. Benchmarks such as the ARC Challenge have been successful in the regard of determining general model capabilities and policy makers would do well to make use of these. Older benchmarks have become saturated as models providers work to game the system and overfit to them, skewing the results in their favour. New closed benchmarks which test for things like AI safety, robustness and fairness are needed to give a fuller picture of model capabilities. I think that we should also think about how policy makers can help support the building blocks of AI innovation in the UK. Talent is important but the best talent will achieve nothing without sufficient computational power to run their experiments. My company, like many, rent GPU's (these computers which are very good at the bulk mathematics needed for AI) from AWS and other US-based hyper-scalers. I would encourage policy makers to look to build sovereignty and security of this very limited compute resource with a creation of a UK Compute Fund. A sovereign compute cloud if you will, to be used by startups and universities to help build the UK as an AI superpower. Lastly on the talent front and I don't have any answers here. We have a huge leaky bucket problem with talent leaving to go to the US. In the UK we have some of the best AI training institutions in the world, UCL, Imperial, Oxbridge and many more. We need to do more to support this talent in building and joining technical companies here in the UK, whether that be through tax breaks for investors, funding for incubators and the startup ecosystem or through other means. Thank you for your time. --- --- title: "A Letter to the Users and Contributors of Code Review GPT" pubDate: 2024-07-10 canonical: https://mattzcarey.com/blog/a-letter-to-users-of-code-review-gpt --- _Originally posted on the Code Review GPT GitHub repository [discussions board](https://github.com/mattzcarey/code-review-gpt/discussions/338)_ Hey everyone, Thanks for all your interest and support of this package. Lots of changes have been happening in my life in the last 6 months hence the stall in progress. I wanted to share some updates on Code Review GPT and my thoughts about code review tools more widely. Looking at the current state of market of these AI-powered review tools as of July 2024, Code Review GPT is up against increasingly sophisticated alternatives such as CodeRabbit and Codium. These companies are raising huge amounts of money to automate away the review process. The question is to me, whether this is sustainable or the right time to be doing this. Obviously raising money for this is not something that I am not going to do. I believe there is another way to make a great tool especially in the code review space. Looking back over the last year I see that the majority of the gain in this field has not been from the advances in code understanding but by the increasingly performant agentic abilities of AI models such as Sonnet 3.5, and to some extent GPT 4o. This is not an advancement rate which more money or team members in these previously mentioned companies can alter. I think there is a space for a more open method which piggybacks off these model advances and those of researchers in the space more than it does to create its own IP. An easily configurable, and flexible platform for code review and understanding, using primarily state-of-the-art foundational models but leaving the door open for locally fine-tuned LLMs. If this package provides the ability to use local models then it should also provide a method to build them easily. That is where I believe that this niche will lead and I want to bet on it. Last summer I derailed this project trying to build out a SaaS tool. We decided to go with a Serverless specific hosting solution of a GitHub app. This was designed to be able to be easily deployed by us and used by you for some nominal fee. In this new phase I want to take this back to basics. I think we can get a huge amount done as a GitHub action and this will be more approachable to more people. We will not tie ourselves into another hosting provider. I will be building the package code into a container for deployment as an app if necessary but this will be secondary to the main package. This project should be about building the best open project not trying to make a quick buck. For the short term I have some UX improvements on my mind for CRGPT. These are in no particular order: - More (or less) than 3 comments on PRs. This solid limit is a hack and should be fixed. - We should have a better method of judging code which actually looks at the changes as a cohesive mass not as individual files. - A LGTM comment. - Line by line for typos and regressions. Not prioritising chat. Things I will not work on but will be very happy for others to: - Supporting more foundational LLMs other than GPT4o and Sonnet 3.5 - More platforms such as Gitlab/Azure with a better integration. Future upgrades: - Better algorithms to understand the codebase. If we have to store artifacts we will store it in the codebase itself. - Potential to have finetuned models and an ability to make them easily. - Better testing and evals. - Multiple methods of code understanding and judgement. Users will be able to pick the one they want and contribute new methods. If anyone is looking for some open source LLM hobby contributions or maintainer roles please pop me a message. This new phase is going to need more than just me. You will be working for free (at least in the beginning), will need an interest in the space and preferably another job. If that sounds interesting let me know. I have learnt a huge amount in this last year about code understanding and I am looking forward to build that back into this project. Have a great week, Matt --- --- title: "5 Skills I've needed as an AI Engineer" pubDate: 2024-07-09 canonical: https://mattzcarey.com/blog/five-skills-as-an-ai-engineer-in-2024 --- I've been an AI Engineer for about a year now based out of London,UK. Previously I was working at a consultancy, where I helped build open source tools like Quivr (now YC W24) and [Code Review GPT](https://github.com/mattzcarey/code-review-gpt]). Now I work at [StackOne](https://stackone.com), automating the creation of unified APIs for SaaS companies. AI Engineer is a super new role and its scope is still somewhat contentious. The way I see it is that I sit somewhere between a full-stack software engineer (product engineer?) and an ML Engineer. Day to day, I do R&D, some data engineering, and a lot of full-stack app development. I wrote this article to share a few of the skills I would invest in if I were starting out as an AI Engineer today. ## 1. DevOps: the ugly duckling Knowing your way round some basic devops is a lifesaver. I can ship stuff way faster if I can serve it myself. New platforms like Modal Labs make this stupid easy. But knowing how to whip up a quick AWS Lambda or Cloudflare Worker is super useful. Nobody wants to be waiting for a backend or devops engineer to deploy their app for them. I have built a bunch of apps at StackOne which wouldn't have been viable if I couldn't do the infra myself. We made an internal chatbot in a 24 hour hackathon at an offsite in Malaga. I used a bunch of Lambda functions and queues to make a Slack bot called "StackBot". Slack webhooks are a pain and so to avoid a proper dev setup (only had 24 hours) we essentially tested this out as a deployed app with different environments (dev, staging and prod). It lent itself to a serverless style architecture. Pros: scaled to zero and cost nothing to setup. Cons: had to deploy the app between each dev feedback loop. Mitigated by using hotswap deployments on AWS Lambda which deployed in under 5 seconds. Learn how to deploy basic primitives like Serverless compute functions, containers, a queue and database. The platform doesnt matter so much. Pick one that is commonly used and you can get some credits for the learning phase. ## 2. Models: size isn't everything Models come and go. They mostly get better. Use the smallest one you can - it's usually cheaper. Finetune for fun, not for prod. Unless you're swimming in compute and cash, in which case, can I have some? Countless stories of AI companies trying to build their own models before product market fit. It epitomises the tech for tech's sake mentality. If you're building a product you need customers, not a better model. If you're building a model and are not a researcher then imo you need a problem, not a better model. PMF then GPUs is how it should be not the other way round. ## 3. Community: your safety net It's super hard to learn anything state of the art in a vacuum. I don't do proper research, so I gotta keep one eye on those who do. Understanding the latest papers and techniques is a full time job so my best advice is to consume condensed versions from people who do this well. Everyone has bugs, not everyone can fix them. The flip side to the research problem is that as a software engineer I spend most of my time building (an fixing) things. I'm a decent resource for my researcher friends for all things hosting models, testing or just general swe work. Add value any why you can. Distilled advice: Make friends, join group chats, and give your help freely. It'll come back around. Check out [AI Demo Days](https://demodays.ai) and [AI Tinkerers](https://aitinkerers.org/) for great communities of AI Engineers, founders (and investors). ## 4. Data: the real mvp From data cleaning to evals, this has been the biggest part of actually getting results. Look at your data. No, really look at it. You'd be surprised how many "anomalies" are just typos, bad formatting or just plain broken retrieval pipelines. As a beginner data whisperer I invested a bunch of time in visualization tooling. I find it alot easier when dealing with data if I can use some nice UI rather than just terminal psql command. Plotting maps of your data is cool as well. In 2024 there are some great LLM specific logging and observability tools on the market now. Ones I have used are Langsmith, Langfuse and Logfire. All work to help you understand your application better and speed up the debugging process. ## 5. App Development: getting the show on the road I get ideas by contributing and generally reading open source codebases. See what works or what feels grim. I take those ideas into my day to day work. Also I have a bunch of cool article bookmarks from SWEs I think write cool code. One being this one on [writing good python](https://www.ivanleo.com/blog/good-python-code). Some of the best advice I ever got was to ask the best person you can find to review your code. The Japanese have this idea of a `gemba walk`. It says that managers should go to the shop floor and see how things are done. This helps them understand the process more intimately but also is a great learning opportunity for the worker, in this case me. Learn stuff from the people who've seen it all before. Let's try not to make the same mistakes of the past. --- There you have it. Five skills that helped me do my job in the last year working as an AI Engineer. It's such a new role by the time you read this it will probably have changed. Even so I hope that if you are considering a role in AI Engineering that this gives you a good idea of what to expect and work on to get there. Happy building, Matt 🤖 _This article is based on a [tweet I wrote](https://x.com/mattzcarey/status/1809230369943896260) which people liked. I'd appreciate any [feedback](mailto:matt@stackone.com) you have._