2026-01-09
Lecture Friday: Smart-Pointers, RAII, ZII? Becoming an N+2 programmer
Not a fan of using "stupid" to describe any of this. No point insulting anyone since even he admits (and I'd echo) we all have to go through thinking about software architectures like this. As you get better and better at software development, you can conceptualize more and more of a system in one go. At one point, breaking a string on commas was a hard problem to me. Now I can conceptualize complex stateful distributed systems.
The problem is that breaking them down by thinking in terms of serial actors is becoming an ever more inefficient way of having the hardware do the work these days. Computers are basically only getting wider and wider. Not much faster. It started doing this around 2005.
By wider, I mean in parallel. Doing more of the same thing all in one go. Performing dozens or even hundreds of the same operation in just one execution. In this case, he's talking about memory but it applies to all shared resource contention including compute, memory, disk, and network. Doing one thing at a time just has an incredibly high overhead when you see it play out in real contexts where the code is usually tasked to operate on hundreds or thousands of that thing.
Think of it like this. You're tasked with taking groceries from your vehicle to the house. Are you going to do it one condiment at a time? Well, we often program like this to make the tasks easier to conceptualize. We design and build a system to pick out a single remaining item and carry it to the house. Then we make a system that knows where everything goes and delegates items as they arrive to systems that each know how to put an item on a specific shelf. We then connect all those systems to ship. This would be for loops to the top, conditionals to the bottom.
If you instead loaded bags as you shopped by destination shelf, you could carry many bags in together. Your dispatcher can hand a bag to each stocker as you get the next load, and each stocker can basically dump the contents on the shelf and neaten up. If your bandwidth gets wider from going to the gym, your program gets even faster until it's a single fetch and load. This would be conditionals to the top, for loops to the bottom.
You may have never thought of sorting at the store and instead sort on a counter at home. This is because you're thinking discretely about the store system and the home system. If you think about them together, a more optimal solution is possible. Instead of randomly loading your basket, you can load it by destination making it much faster later.
When he's talking about arena allocators, he's talking about the overhead of deallocating hundreds of objects one at a time versus setting next_alloc_index = 0. It's significantly faster if you have a data boundary like a rendered video frame or a backend API endpoint handler. Any time you've got a distinct work boundary you really shouldn't be cleaning as you go but instead do all the setup, all the work, and then all the teardown.
This is why it makes sense not to explicitly close files or sockets or even deallocate memory if your program is a short lived application like a command line utility. The operating system will clean up everything for you all at once when you exit much faster than you can request them each be cleaned up one at a time. In effect, your program's memory is running in an arena allocator. Your file descriptors are being managed in a pool. (That's why you usually get the same ID back if you close and open an new one.)