&Notepad

Remixing as Learning in Creative Media

In Mindstorms, a short but wonderful book about how children can use computers to learn in new ways, Seymour Papert argues that the process of creating and debugging a computer program intuitively matches the way that children naturally form theories about the world. For example, he says that if you ask a young child “what makes the wind blow?” then you will rarely get an “I don’t know,” but instead get an answer like “the trees waving makes the wind blow.” In essence, children attempt to understand complex systems like the weather by searching for correlations—the trees always wave when the wind blows, so that’s a clear possible cause.

Of course, such children are too young to fully understand correlation vs. causation (or the Coriolis effect), but the key idea is that this process of theory formation comes naturally to us. Papert claims that this intuitive process gets stamped out in schools by enforcing a rigid notion of correctness. Rather than rewarding children for identifying patterns, they are told that their theories are wrong and should think otherwise. Papert spends the rest of the book explaining how his LOGO language/system, the precursor to LEGO Mindstorms (controlling robots with simple commands) helps children gain a motivated yet bottom-up understanding of formal thinking, encouraging them to use analogies between the motion of their own bodies and that of the robot’s to carefully introspect and construct algorithms.

After reading the book, I started thinking about how this methodology could be applied to other more complex pursuits. Learning programming, generally, uses a bottom-up approach. A top-down approach would be something like, “let’s start with the source code for the Linux kernel and understand how operating systems work.” Doesn’t seem very feasible. By contrast, a bottom-up approach says “lets learn one concept at a time in isolation, then slowly put them together, e.g. learn about virtual memory, process scheduling, etc. and then build an OS from them.” This is a much more successful pedagogy, but the core problem is that the sub-components tend to feel less motivated until the full system brought together. You have to slog through a virtual memory implementation until you start to write a process scheduler, and only then appreciate the idea of memory isolation.

When thinking about my own learning experiences, I realized that while my formal computer science education largely followed that mold, my unstructured education in creative media—games, websites, and music—took an entirely different approach. In each discipline, I started with some existing implementation of a game/website/song and then tweaked it, remixed it, slowly poked and prodded the black box until I got more of a feel for what was going on. And it was so much fun! I felt like the aforementioned children, learning about a system by immersing myself in it and finding patterns, not by building up a mental model from first principles. Critically, it was motivated at every step of the way. I wasn’t dealing with trivial homework problems, but instead creating a game that I actually wanted to play, or music I actually wanted to hear.

In formal education, I’ve never felt the same way I have exploring systems on my own, which is a tragedy—my unstructured education has had far more of an impact on who I am as a person and a thinker than any individual class I took. I think it’s worth asking, then: what are the characteristics of this kind of learning? Could we design new curricula around these principles? For starters, I’ll tell you my story, and we’ll see what patterns we can find across these anecdotes.

Example #1: Games

As you might notice from this blog’s design and my homepage, I’m a big fan of video games. Loved them growing up, played them on every platform, and they still bring me joy to this day. The entire reason I became a programmer is because of one game: Garry’s Mod (Gmod). Gmod is a sandbox game built on the Source Engine, the game engine powering Valve’s games like Half-Life 2, Portal, and others. Its basic premise is basically to make the underlying game engine more accessible. It’s less of a game and more of a framework that enables the creation of custom structures, game modes, maps, and more.

Like many other games, Gmod uses the Lua scripting language to provide a simple API for programmatically modifying the game. However, when I picked up the game at 14, I knew absolutely nothing about Lua or really programming in general. I had tried in vain to follow a Python tutorial before, but lost interest every time. Once I started playing Gmod, I was hooked. I had always loved building things, starting with Lincoln Logs and LEGOs. I had never before had an unfettered digital sandbox to build whatever I wanted, and Gmod has a bunch of amazing addons like Wiremod for creating all sorts of intricate structures. And that was without every touching a line of code.

However, perhaps a year into my experience, I became increasingly dissatisfied with the game. I saw other people on the servers I attended programming these amazing light displays or creating custom entities, whereas I was just stuck with the vanilla tools the game provides. When I ran my own gamemodes, I couldn’t change the default settings provided by the mode creators. So for my initial foray into Lua scripting, I decided to just tweak simple gameplay settings. I was running a roleplay server, and wanted to change the starting money provided to a particular class. Simple enough, but man, did that take forever. The tutorials were not nearly the same quality as today, and I had to figure out: where were the relevant files in my file system? How do I edit them? How do I know if my changes work? How can I debug a broken change? What is Lua? What is a function? And so on.

But, at some point, I figured it out, and it was liberating. I felt empowered that I could navigate the game engine morass and actually write a line of code. It started with more money in my roleplay game, but over time I gradually expanded the scope of my ambitions. I modified existing libraries into a custom entity, then a custom map, then a custom gamemode. I dived into the rendering internals, and later after learning about HTML/CSS, rewrote the imperative GUI system into a declarative GUI language. Although my learning was bottom-up in the sense that I started small and worked upwards, I was always modifying parts of a bigger system which kept the experience enjoyable, not rote.

A number of factors made Gmod amenable to remixing:

1. Lua is a wonderful starting language. Dynamic typing means you don’t have to worry about compilation. Since it’s usually embedded in a game engine, the language itself is quite simple—everything is just a table, and once you understand that, you get 95% of the Lua mental model. By contrast, other dynamic languages like Python are complex beasts, as one needs to understand list comprehensions, classes and inheritance, “LEGB” scoping, and more to fully grok most code.
2. Gmod has a thriving community of players and developers that would help and encourage each other during development. I frequented the developer forum and often found a place to get a question answered if I was stymied.
3. Due to its scriptable nature, sharing code openly was the norm in the community, and it was easy to find everything from huge gamemodes to tiny self-contained snippets that were easy to copy/paste and modify. No one ever cared about licensing so long as proper attribution was provided.

Example #2: Websites

My earliest exposure to programming was 12 years ago, 2006, in sixth grade. Our ELP teacher provided us a dead-simple HTML template and had us pick our favorite topic, then make a website about it. Sadly, it appears that my website has disappeared, despite the best efforts of the Internet Archive (mine was the one about “Magic”). You can get a sense of the style in the remains of the Abyssinian cats website.

At the time, all we had was an eMac and the 2006 equivalent of Notepad on OS X, but that was all we needed. One of the first things I remember about programming with HTML is the sheer immediacy of it: type a letter, refresh the page, and see your change instantly. Once I started to realize the full extent of the visual magic one could work—blinks! marquees! cursor trails!—I was utterly determined to stuff every feature possible into my website. The content was ancillary, what mattered was the visuals. My crowning achievement was finding a Matrix-style text effect in JavaScript and somehow incorporating it into my website about magicians. Truly a gem lost to the sands of time.

I don’t think I’m alone in saying that web design was my first spark in lighting the programming fire. In discussing this with my friends, in particular amongst women, sites like Neopets and MySpace with customizable HTML/CSS introduced a lot of kids in my generation to programming. There was simply no better way to show how cool you were than to bask in the glory of the Geocities aesthetic. As kids, we often had limited artistic control over our immediate environment; activities like drawing on the walls or splashing paint on the ceiling were generally off-limits. But these websites offered us the ability to have our own corner of the world that’s entirely ours to make, no matter how horrendous the color palette. It’s a little sad that such customizability is quite rare in today’s websites.

But what is it that made HTML/CSS so friendly to beginners? Consider for a moment a world where websites were created in LaTeX instead of HTML: obtuse syntax, confusing semantics, full of inscrutable syntax errors. It would be a disaster! But specifically, I think it’s the copy/pastability of HTML and CSS that make them so uniquely beginner-friendly. Even if I understand almost nothing about the languages, I can take code that someone else wrote, throw it into my website, and most of the time it will just work! There are never compiler errors, only visual tweaks to be made. There is no scary program complaining that your program is “incorrect.” This makes remixing a painless process, even if the bugs can be hard to diagnose.

Example #3: Music

The remixing style of learning isn’t just for technical fields, however! During my undergrad at CMU, I started to become more interested in all the non-technical activities I completely missed out on in high school, namely music. I knew absolutely nothing about music in theory or practice beyond my feeble attempts to sing in the shower. But practice I did, and eventually I found an a cappella group that would take me in. Over time, I started to gain an ear for the intervals, chords, melodies, and progressions underlying the songs we performed.

However, since I had no formal training in music, I was unfamiliar with the notation. Reading sheet music took a long time, and it certainly wasn’t to the point where I could hear the music in my head as I read. I couldn’t easily translate any of my intuitions about music into concrete directions on how or what to sing, yet I still wanted to be able to write music.

Thankfully, most collegiate a cappella music is cover songs, so I didn’t have to start from scratch. I decided to cover No Surprises by Radiohead. I searched only for a piano cover of the song and found the sheet music. Armed with the seed for my composition, I transcribed it into the key tool underlying my learning efforts: Noteflight (pictured above). For me, the critical feature of Noteflight was enabling me to listen to the score as I was changing it. You can either click on individual notes to hear them standalone, or play the whole score to hear it in unison. This meant that I could listen to the MIDI playback and test whether my changes made sense by ear.

For the No Surprises cover, I took the transcribed piano sheet and the guitar chords and converted them into separate a cappella parts (lead, soprano, alto, tenor, baritone, bass, beatboxer). The main challenge was ensuring that for any given part, the singers didn’t have to jump around too much in the progress for vocal consistency. Then I searched around for inspiration on how to remix the song to add our own flair, and pulled in ideas a few different covers, e.g. the composition from choir! choir! choir!. Ultimately I settled on a piece I was quite happy with for my first effort (link).

Did I get every chord progression correct or match strict musical standards? Unlikely. Would I have been able to write a piece from scratch? Nope. Is the MIDI playback the same as how it would be sung by actual people? Of course not. But Noteflight’s playback feature made it so I could vet all my changes with an “ear test” before showing it to my friends. With zero prior knowledge, I was able to use this tool to successfully remix songs (see Pokemon Johto/Main Theme, Addicted to You), and even though my formal musical knowledge is still lacking, the experience significantly improved my ability to read and write sheet music.

Analysis

The examples above all share a few common features:

1. Experimentation was possible through rapid feedback. HTML and Noteflight (and to some extent Gmod) both enable creators to have an immediate connection from their program to their output (as Bret Victor would say). As a learner, when you’re dealing with a big system that you don’t fully understand whether it’s programming languages or music, the ability to try something, see if it works, and branch on that decision is absolutely critical. This is particularly true for more mature learners who are quick to pick up patterns, but need short iteration times to determine which direction to take.
2. Third-party content was easily accessible and modifiable. Noteflight and Gmod both have built-in infrastructure for distributing open-source libraries/compositions, and part of their culture is the sharing and reuse of others work. On the web, getting simple content is as easy as View Source (modulo all the JavaScript these days), and copy/pasting nearly always works.
3. Large creations have many independent parts with localizable modifications. In each medium, a large creation taken from someone else (a game mode, a website template, a full score) usually has well defined entry points, and modifications aren’t likely to affect other parts of the system.
   • Gmod scripts are structurally required to separate out client code (running on the player’s computer, controlling GUIs/rendering) from server code (running on the server, controlling game logic), and server/client only interact on well-defined boundaries so changes from one likely don’t affect the other.
   • Websites have their logical visual structure embedded in the indentation of the source code (not necessarily true for imperative GUIs). Changing text or color in one div is not likely to affect other divs not nested near your current one.
   • Musical scores often have both well-defined overarching structure (e.g. verse, prechorus, chorus, bridge) and local structure (e.g. four chord progression), and by design changing one note of a score does not literally change the others even though the overall sound may change.

It’s also important to note that all of the above activities have an inherent sensory component, either visual or auditory. For a lot of people, this provides a baseline excitement/involvment in interacting with these media. To what extent does this exist or could this exist for other less sensory pursuits? Let’s consider the example of data analysis, specifically machine learning, on each of these criteria.

• Rapid feedback: in 2018, machine learning is hardly interactive—training a model can take hours or days and cost a lot of money if you have to rent hardware.
• Third-party content: while ML frameworks like TensorFlow and PyTorch go a long way towards defining a common format for sharing models, it’s still quite hard in practice to take an arbitrary model off the shelf and run it, fine-tune it, optimize it, and so on.
• Localizable modifications: most modern models are not well understood. Changing one part of a neural network and retraining could have huge implications in training time, hyperparameter choice, and overall accuracy. In general, there is no clear way to make small changes that only affect a small part of the system.

Ultimately, what this means is that newcomers cannot learn how to make machine learning models in the same way they can learn how to make games, websites, and music. This limits the ability of learners to play around, remix, and theorize without a thorough bottom-up grounding in the theory and practice of machine learning, which in turn makes the field less approachable. However, I hope that by looking at how we learn through remixing in creative media, we can continue to gain insight that guide the development of tools and curriculum in other fields.

As always, let me know what you think. Either send me an email at wcrichto@cs.stanford.edu or comment on Hacker News thread.