An opinionated perspective on how to do important research that makes a difference (and sometimes win awards).

Are the harms that LLMs have caused, and will soon cause, worth the benefits they may bring? This article (a written version of a keynote talk I gave at CoLM) tries to explore this question.

A feature-complete gate-level microcoded Intel 4004 in 4004 bytes of C, capable of emulating the original Busicom calculator ROM for which the chip was originally designed.

A small hardware description language implemented as a DSL on Python, with a small 170 LoC 32-bit RISC CPU.

Future LLMs have the potential to cause significant harm due to their ruthless effiency. I'm worried this will happen, and discuss the ways in which it might.

I have very wide error bars on the potential future of large language models, and I think you should too. It's possible LLMs basically lead to AGI, and it's also possible they platteau.

My work on privacy-preserving machine learning is often cited by lawyers arguing for or against how generative AI models violate copyright. This maybe isn't the right work to be citing.

I have decided to leave Google, and will be joining Anthropic to continue my work on machine learning security

A one-year review of people's predictions on an AI-forecasting survey I made last year. Most people were over-confident in their predictions.

I wrote a (list of) regular expressions that will play a (not very good) chess game by running a 2-ply minimax search.

I let a language model write my bio. It went about as well as you might expect.

You should forecast the future of AI in this quiz, so that you can see just how right or wrong you are.

I don't think that AI models (by which I mean: large language models) are over-hyped. In this post I will list 50 ways I've used them.

Yesterday I was forwarded a bunch of messages that Prof. Ben Zhao (a computer science professor [a] A full professor with tenure, so I feel entirely within my rights to call him out here. at the University of Chicago) wrote about me on a public Discord server with 15,000 members, including this gem:

IEEE SP 2024 (one of the top computer security conferences) has, again, accepted an adversarial example defense paper that is broken with simple attacks. It contains claims that are mathematically impossible, does not follow recommended guidance on evaluating adversarial robustness, and its own figures present all the necessary evidence that the evaluation was conducted incorrectly.

A benchmark of ~100 tests for language models, collected from actual questions I've asked of language models in the last year.

How do I pick what research problems I want to solve? I get asked this question often, most recently in December at NeurIPS, and so on my flight back I decided to describe the only piece of my incredibly rudimentary system that's at all a process. I maintain a single file called ideas.txt, where I just append a new entry every time I think of something that might be an interesting topic for a paper. When it's time to pick my next project, I skim through the list, and pick whichever I think is most interesting. or exciting. or important. or whatever I'm looking for at the moment. (Or find something new entirely if nothing looks compelling.)

So let's suppose you had a 1980s Apple ProFile Hard Drive, and you wanted to recover the data.

Building a chess bot that queries GPT-3.5-turbo-instruct to play chess at the level of a skilled human player.

TODO

This program is a dependency-free implementation of GPT-2, including

I recently got back from attending USENIX Security 2022, and someone pointed out to me that it's been five years since I wrote Towards Evaluating the Robustness of Neural Networks (with my at-the-time advisor) and they asked if I had any thoughts on this paper. I didn't respond with that great an answer, but thought it was an interesting question, so I figured I'd write one down here instead. (In fact, I got the same question at IEEE SP earlier this year too, and didn't have a great answer then either.)

A brief discussion about a tool I use to make rapid iteration in ML research possible.

A recent paper ('A Roadmap for Big Model') has copied a bunch of text from over a dozen prior papers. This is bad.

Abstract: Improving digital logic gates on Conway's game of life by allowing 8-bit logic gates instead of boolean logic gates.

This page contains a few checklists that help prevent embarrassing issues when releasing research papers online (e.g., via arXiv or a conference publication).

Abstract: An implementation of a minimal CPU on Conway's the Game of Life (an 'unlimited register machine'), and runs at ~10Hz.

Abstract: This post describes improvemnets made to my prior digital logic gate constructions (e.g., AND/OR/NOT) built on top of Conway's Game of Life, resulting in 100x faster simulations.

This year I entered in JS13K 2020, a game jam for JavaScript games in under 13KB (total size). I wrote a 3rd-person space shooter game, building on top of game engine I built last year for a doom clone.

InstaHide (a recent method that claims to give a way to train neural networks while preserving training data privacy) was just awarded the 2nd place Bell Labs Prize (an award for finding solutions to some of the greatest challenges facing the information and telecommunications industry.). This is a grave error.