In Mindstorms, a short but wonderful book about how children can use computers to learn in new ways, Seymour Papert argues that the process of creating and debugging a computer program intuitively matches the way that children naturally form theories about the world. For example, he says that if you ask a young child “what makes the wind blow?” then you will rarely get an “I don’t know,” but instead get an answer like “the trees waving makes the wind blow.” In essence, children attempt to understand complex systems like the weather by searching for correlations—the trees always wave when the wind blows, so that’s a clear possible cause.
Of course, such children are too young to fully understand correlation vs. causation (or the Coriolis effect), but the key idea is that this process of theory formation comes naturally to us. Papert claims that this intuitive process gets stamped out in schools by enforcing a rigid notion of correctness. Rather than rewarding children for identifying patterns, they are told that their theories are wrong and should think otherwise. Papert spends the rest of the book explaining how his LOGO language/system, the precursor to LEGO Mindstorms (controlling robots with simple commands) helps children gain a motivated yet bottom-up understanding of formal thinking, encouraging them to use analogies between the motion of their own bodies and that of the robot’s to carefully introspect and construct algorithms.
After reading the book, I started thinking about how this methodology could be applied to other more complex pursuits. Learning programming, generally, uses a bottom-up approach. A top-down approach would be something like, “let’s start with the source code for the Linux kernel and understand how operating systems work.” Doesn’t seem very feasible. By contrast, a bottom-up approach says “lets learn one concept at a time in isolation, then slowly put them together, e.g. learn about virtual memory, process scheduling, etc. and then build an OS from them.” This is a much more successful pedagogy, but the core problem is that the sub-components tend to feel less motivated until the full system brought together. You have to slog through a virtual memory implementation until you start to write a process scheduler, and only then appreciate the idea of memory isolation.
When thinking about my own learning experiences, I realized that while my formal computer science education largely followed that mold, my unstructured education in creative media—games, websites, and music—took an entirely different approach. In each discipline, I started with some existing implementation of a game/website/song and then tweaked it, remixed it, slowly poked and prodded the black box until I got more of a feel for what was going on. And it was so much fun! I felt like the aforementioned children, learning about a system by immersing myself in it and finding patterns, not by building up a mental model from first principles. Critically, it was motivated at every step of the way. I wasn’t dealing with trivial homework problems, but instead creating a game that I actually wanted to play, or music I actually wanted to hear.
In formal education, I’ve never felt the same way I have exploring systems on my own, which is a tragedy—my unstructured education has had far more of an impact on who I am as a person and a thinker than any individual class I took. I think it’s worth asking, then: what are the characteristics of this kind of learning? Could we design new curricula around these principles? For starters, I’ll tell you my story, and we’ll see what patterns we can find across these anecdotes.
As you might notice from this blog’s design and my homepage, I’m a big fan of video games. Loved them growing up, played them on every platform, and they still bring me joy to this day. The entire reason I became a programmer is because of one game: Garry’s Mod (Gmod). Gmod is a sandbox game built on the Source Engine, the game engine powering Valve’s games like Half-Life 2, Portal, and others. Its basic premise is basically to make the underlying game engine more accessible. It’s less of a game and more of a framework that enables the creation of custom structures, game modes, maps, and more.
Like many other games, Gmod uses the Lua scripting language to provide a simple API for programmatically modifying the game. However, when I picked up the game at 14, I knew absolutely nothing about Lua or really programming in general. I had tried in vain to follow a Python tutorial before, but lost interest every time. Once I started playing Gmod, I was hooked. I had always loved building things, starting with Lincoln Logs and LEGOs. I had never before had an unfettered digital sandbox to build whatever I wanted, and Gmod has a bunch of amazing addons like Wiremod for creating all sorts of intricate structures. And that was without every touching a line of code.
However, perhaps a year into my experience, I became increasingly dissatisfied with the game. I saw other people on the servers I attended programming these amazing light displays or creating custom entities, whereas I was just stuck with the vanilla tools the game provides. When I ran my own gamemodes, I couldn’t change the default settings provided by the mode creators. So for my initial foray into Lua scripting, I decided to just tweak simple gameplay settings. I was running a roleplay server, and wanted to change the starting money provided to a particular class. Simple enough, but man, did that take forever. The tutorials were not nearly the same quality as today, and I had to figure out: where were the relevant files in my file system? How do I edit them? How do I know if my changes work? How can I debug a broken change? What is Lua? What is a function? And so on.
But, at some point, I figured it out, and it was liberating. I felt empowered that I could navigate the game engine morass and actually write a line of code. It started with more money in my roleplay game, but over time I gradually expanded the scope of my ambitions. I modified existing libraries into a custom entity, then a custom map, then a custom gamemode. I dived into the rendering internals, and later after learning about HTML/CSS, rewrote the imperative GUI system into a declarative GUI language. Although my learning was bottom-up in the sense that I started small and worked upwards, I was always modifying parts of a bigger system which kept the experience enjoyable, not rote.
A number of factors made Gmod amenable to remixing:
My earliest exposure to programming was 12 years ago, 2006, in sixth grade. Our ELP teacher provided us a dead-simple HTML template and had us pick our favorite topic, then make a website about it. Sadly, it appears that my website has disappeared, despite the best efforts of the Internet Archive (mine was the one about “Magic”). You can get a sense of the style in the remains of the Abyssinian cats website.
I don’t think I’m alone in saying that web design was my first spark in lighting the programming fire. In discussing this with my friends, in particular amongst women, sites like Neopets and MySpace with customizable HTML/CSS introduced a lot of kids in my generation to programming. There was simply no better way to show how cool you were than to bask in the glory of the Geocities aesthetic. As kids, we often had limited artistic control over our immediate environment; activities like drawing on the walls or splashing paint on the ceiling were generally off-limits. But these websites offered us the ability to have our own corner of the world that’s entirely ours to make, no matter how horrendous the color palette. It’s a little sad that such customizability is quite rare in today’s websites.
But what is it that made HTML/CSS so friendly to beginners? Consider for a moment a world where websites were created in LaTeX instead of HTML: obtuse syntax, confusing semantics, full of inscrutable syntax errors. It would be a disaster! But specifically, I think it’s the copy/pastability of HTML and CSS that make them so uniquely beginner-friendly. Even if I understand almost nothing about the languages, I can take code that someone else wrote, throw it into my website, and most of the time it will just work! There are never compiler errors, only visual tweaks to be made. There is no scary program complaining that your program is “incorrect.” This makes remixing a painless process, even if the bugs can be hard to diagnose.
The remixing style of learning isn’t just for technical fields, however! During my undergrad at CMU, I started to become more interested in all the non-technical activities I completely missed out on in high school, namely music. I knew absolutely nothing about music in theory or practice beyond my feeble attempts to sing in the shower. But practice I did, and eventually I found an a cappella group that would take me in. Over time, I started to gain an ear for the intervals, chords, melodies, and progressions underlying the songs we performed.
However, since I had no formal training in music, I was unfamiliar with the notation. Reading sheet music took a long time, and it certainly wasn’t to the point where I could hear the music in my head as I read. I couldn’t easily translate any of my intuitions about music into concrete directions on how or what to sing, yet I still wanted to be able to write music.
Thankfully, most collegiate a cappella music is cover songs, so I didn’t have to start from scratch. I decided to cover No Surprises by Radiohead. I searched only for a piano cover of the song and found the sheet music. Armed with the seed for my composition, I transcribed it into the key tool underlying my learning efforts: Noteflight (pictured above). For me, the critical feature of Noteflight was enabling me to listen to the score as I was changing it. You can either click on individual notes to hear them standalone, or play the whole score to hear it in unison. This meant that I could listen to the MIDI playback and test whether my changes made sense by ear.
For the No Surprises cover, I took the transcribed piano sheet and the guitar chords and converted them into separate a cappella parts (lead, soprano, alto, tenor, baritone, bass, beatboxer). The main challenge was ensuring that for any given part, the singers didn’t have to jump around too much in the progress for vocal consistency. Then I searched around for inspiration on how to remix the song to add our own flair, and pulled in ideas a few different covers, e.g. the composition from choir! choir! choir!. Ultimately I settled on a piece I was quite happy with for my first effort (link).
Did I get every chord progression correct or match strict musical standards? Unlikely. Would I have been able to write a piece from scratch? Nope. Is the MIDI playback the same as how it would be sung by actual people? Of course not. But Noteflight’s playback feature made it so I could vet all my changes with an “ear test” before showing it to my friends. With zero prior knowledge, I was able to use this tool to successfully remix songs (see Pokemon Johto/Main Theme, Addicted to You), and even though my formal musical knowledge is still lacking, the experience significantly improved my ability to read and write sheet music.
The examples above all share a few common features:
divis not likely to affect other
divs not nested near your current one.
It’s also important to note that all of the above activities have an inherent sensory component, either visual or auditory. For a lot of people, this provides a baseline excitement/involvment in interacting with these media. To what extent does this exist or could this exist for other less sensory pursuits? Let’s consider the example of data analysis, specifically machine learning, on each of these criteria.
Ultimately, what this means is that newcomers cannot learn how to make machine learning models in the same way they can learn how to make games, websites, and music. This limits the ability of learners to play around, remix, and theorize without a thorough bottom-up grounding in the theory and practice of machine learning, which in turn makes the field less approachable. However, I hope that by looking at how we learn through remixing in creative media, we can continue to gain insight that guide the development of tools and curriculum in other fields.