(In which I talk about my feeeeeeels)
Disclaimer: This post comes in the middle of an existential crisis. I'm struggling a lot with programming as a career choice and feeling disconnected from a community of excited hackers. These feelings and opinions are my own and I think it's totally fine if you don't subscribe to them or want to write me off as an F-ing idiot.
A lot of the ideas in this post have been buzzing around in my head since I saw Jen Myers deliver her keynote at Strangeloop last year.
I've been keeping my thoughts in my head mostly because I'm already an established programmer. A lot of the motive for the talk was to be more welcoming to newcomers and minorities that struggled to be included in our communities. But I think this problem affects all of us, every last one, regardless of gender, race, or class.
The short version is that I think the tone of programming communities, especially online ones, is horrific. It's filled with religious debate over things less important than getting people excited about and interested in computing. For me, whether it's smart people posturing for social status or individuals genuinely trying to enlighten others is irrelevant.
Our first reaction to any comrade, any other person passionate about and interested in building things with computers, any human crazy and masochistic enough to try and expand the capabilities of these absurd machines, should be empathy and love.
This may seem ridiculous at first glance. It's harder than it sounds.
The Same Old Arguments
You already know the religious wars I'm talking about. They're silly little things. Are static or dynamic types better? (For some, is there even such a thing as being dynamically typed?) Is Vim or Emacs better? Should I learn programming with PHP or Haskell? Should my app use JSON, XML, or a self-describing binary format? Is programming math, art, or craft? Can code be literature?
For a host of reasons, these are questions we have a vested interest in. And I think, more often than not, our motive is to encourage more learning and exploration. But the conversation is almost always full of condescension and judgment, especially if the medium for response is limited. We simply cannot let supporting curiosity become secondary to proselytizing "the right thing".
Plain and simple, turning a prompt for exploration into a right-or-wrong religious debate is curiosity destroying. And that's precisely the opposite of our intent, the opposite of what we as a community should aspire to.
Our opinions are important, and I'm not precluding the existence of a right answer. But someone pondering a tricky subject isn't best met by bludgeoning them over the head with a conclusion. As long as the principal motive of those we interact with is the fractal question "Why?", we are together.
This connects to a lot of things. It connects to people wondering if they're good at programming, or how to know such a thing. It contributes to impostor syndrome. I've struggled to hack on hobby code for fun because I don't feel like I can be proud of it. Not smart enough, not groundbreaking enough, not important enough. And I know that's silly, because there are more important things to worry about.
So the more we can get away from emphasizing that the most important thing in programming is being right, the better that will be for newcomers, for hobbyists, and I believe, for all of us.
I'm reminded of an Alan Perlis quote in SICP:
"I think that it's extraordinarily important that we keep the fun in computing. ... We began to feel as if we really were responsible for the successful, error-free, perfect use of these machines. I don't think we are.
I think we're responsible for stretching them, setting them off in new directions, and keeping fun in the house."
I'm not perfect at this either. It is difficult to never be dismissive, let alone to always be gentle. But sometimes people are just trying to make it through the day. Not use the best tool, not come up with a groundbreaking solution, not fix the world. We need to try to meet other programmers where they are. Not move them to our habitat before empathizing, before loving.
Ironically, I know this has been a bit of a high-horse diatribe. At least let me give you a gift for coming so far and listening to me ramble so much. Here, have something I love, bits of Milosz:
To whom do we tell what happened on the earth, for whom do we place everywhere huge mirrors in the hope that they will be filled up and will stay so? I think that I am here, on this earth, To present a report on it, but to whom I don’t know. As if I were sent so that whatever takes place Has meaning because it changes into memory. To find my home in one sentence, concise, as if hammered in metal. Not to enchant anybody. Not to earn a lasting name in posterity. An unnamed need for order, for rhythm, for form, which three words are opposed to chaos and nothingness. What did I really want to tell them? That I labored to transcend my place and time, searching for the Real. And we could have been united only by what we have in common: the same nakedness in a garden beyond time, but the moments are short when it seems to me that, at odds with time, we hold each other's hands. And I drink wine and I shake my head and say: "What man feels and thinks will never be expressed."
Life is going pretty well. Norma and I have moved into a new apartment. I'm figuring out how to balance work and my other goals. My student loans will be paid off by my 28th birthday in August.
I decided to work from home today. I slept poorly, I think it's the pollen. I've been thinking a lot about what I want lately. Neither Academia nor Industry quite seem to fit. Academia only cares about work that advances the state of the art in specific subfields, Industry only cares about work that advances the bottom line.
My goals don't quite fit in with either of those things. I want to write software to help people understand how other software works. Specifically, a Nintendo Emulator and set of tools for observing and modifying old games on the fly. And so I find myself thinking about how to work fewer hours so I can find more time for my 'art projects' without sacrificing hobbies, relationships, and a social life.
I've been too drowsy today to get meaningful work done on work or hobby projects, but here's what I thought about in the shower:
- I should add a proper JIT compiler to cl-6502 that uses the "compile-to-closures" technique described by xach.
- I should fix trowel so that it can correctly compute the control flow graph of simple (NROM) NES games that aren't self-modifying. Like I meant to do last summer.
- I should implement the UNROM mapper for famiclom using neth, sprocketnes, or tenes for reference if needed. That way, I could play with Mega Man 1 and its annotated code in addition to Super Mario Bros.
- I should start on a lispy macroassembler/compiler to 6502 because there are idioms to capture in these games and neslisp isn't gonna cut it.
- Also, I thought about adding crossposting support and 'embeddables' to coleslaw briefly but that doesn't count. That's my one halfway 'practical' piece of software. People do like to blog.
Seriously, is there somewhere I can do an MFA but really just write weird artsy software for 2 years?
Oh, well. Time to take a nap and then try to write some code to find Biconnected Components in C again.
Long time, no blog.
I've been offline for a while. I burned out last July and only really started hacking on my lisp projects again in March. So what's changed in the last two months? Actually, kind of a lot.
Coleslaw 0.9.4 is hereby released. I apologize that 0.9.3 which went out in the last quicklisp release had an embarrassing escaping bug.
The most fun part of Coleslaw is trying my hand at API design. Lisp is a great tool for writing extensible software and Coleslaw has been a good proving ground for that since everyone has a slightly different set of requirements for their blogware.
I've been reading Sonya Keene's Object Oriented Programming in CL lately which led to a large refactoring around the new Document Protocol. I'm not prepared to say anything intelligent about protocols yet, but thankfully plenty of people have done so elsewhere. This blog post by sykopomp isn't a bad place to start.
In addition to the document protocol and the usual litany of bugfixes, Coleslaw now has a new theme based on bootswatch readable, user-defined routing, support for static pages, and greatly expanded docs.
The main things to tackle before 1.0 are a plugin to support incremental compilation for very large sites and a twitter/tumblr cross-posting plugin.
Additionally, someone actually found a use for my Readable CPU emulator! Dustin Long was working on a homebrew Nintendo game and wanted a way to unit test his code, so he's been using cl-6502 to get cycle counts and otherwise check behavior. Naturally, the very basic assembler got on his nerves so he sent me a nice pull request adding support for labels, compile-time expressions, and decimal, hex, and binary literals. Thanks, Dustin!
With any luck, I'll get back to work on famiclom or tools for analyzing old NES games like Super Mario Bros and Mega Man 2. It's good to be back.
I've been away for a while and I needed it. My priorities in the past 6 months have shifted pretty drastically from the last few years "hacking, blogging, hacking, blogging". It was a long overdue shift to grow in new ways, form new relationships, and rediscover old hobbies. So, here's the latest:
I Love Smash Brothers
For years, traditional competitive sports haven't clicked for me. I compare myself to others reflexively but avoid active competition. I'm trying to compare myself to others less, bad habits die hard. But I encountered a documentary series about Smash Brothers back in October and was reminded that a video game is the one place I've found competition fun, win or lose.
It's crazy how much free time I've spent since then learning about a game I'd already put a few thousand hours into in college. There is incredible depth to the game and while I grant that calling a video gamer an "athlete" sounds ridiculous on its face, I now believe eSports are deserving of a mainstream audience even if they never find it.
Personally, I'm enjoying getting better at the game and learning to be patient with my own progress. I've gained some appreciation for how people get so excited about traditional sports. I'd strongly encourage you to watch the first episode of the documentary if you have any interest in competitive gaming whatsoever, even if you think the very notion silly.
I Love Electronic Music
I've told myself for a long time that I would try to make music one day. I've played the guitar and the drums though am self-taught at both and never got serious enough to develop a real aptitude. I also have a long held love of electronic music, particularly sample-based wizardry such as Amon Tobin's Supermodified.
I'm finally pursuing this passion. I've purchased a copy of an old-school "tracker-style" software called Renoise. While I'm not composing actual songs yet, I've been greatly enjoying sampling music and video games from my youth and constructing odd melodies and instruments. I've also been learning the basics of sound synthesis from my friend, Matt Simpson. Sound design is a crazy, crazy thing.
I Love My Job
I've been working for Emcien since May and it's been a great environment to grow as a software engineer. Aside from me, there are 5 engineers and I've been able to pair with people to quickly get up to speed on the products and learn new tools and techniques. In particular, I've enjoyed doing my first serious C hacking ever on the engine that powers Emcien's data analytics products.
I've also spent a huge amount of time this year with my beautiful girlfriend, Norma, and her two dogs. I've tried to see friends in town a bit more, I've played video games and read sci-fi novels a bit more. I'm remembering what leisure feels like and I have no regrets. I still want to finish the Lisp NES emulator and other projects but I'm branching out for a little while ... and my timeframe is flexible. Good luck with your 2014.
I'm still on a break from recreational programming. I've been gaming a bit to remember why I was writing an Emulator in the first place and got the idea for this article. Here are 16 games that left a major impression on me (and 4 honorary mentions) presented in something close to Autobiographical order:
- Command & Conquer: Red Alert -- also, Starcraft
- Pokemon Blue
- The Legend of Zelda: Ocarina of Time
- Final Fantasy VIII
- Metal Gear Solid
- Tony Hawk's Pro Skater 2 -- also, Thrasher: Skate & Destroy
- Gran Turismo 2
- Freelancer -- also, Descent: Freespace
Certainly there have been other titles/series I've enjoyed putting hours into: Naughty Dog's Uncharted, Bethesda's Skyrim, Borderlands, and so on. But the above are the games which uniquely effected me. Each holds special nostalgic value, many represent discovery of a genre or fictional setting that I've come to love. Hell, the Tony Hawk's Pro Skater series got me skateboarding in real life and I'm not sure I would have owned an Acura NSX if not for Gran Turismo.
It would be fun to do a similar list for Movies (would I have become a programmer without Hackers?) or Music. I'll call this good for now. :)
My Recent Absence
You may have noticed it has been two months since I've done any serious hacking on my open source projects. That is no accident. It has been a little over 5 years since I decided that programming would be my future.
That decision worked out far better than I could have hoped. I find programming immensely rewarding and, for someone who has only been doing it for 5 years, I think I'm doing reasonably well. I have a steady job that I'm happy with and rewarding friends and hobbies.
But I'm tired. I'm stepping back and trying to take stock of how I use my time at a high level. I'm not micro-optimizing. I'm asking, for the first time in quite a while, how I want to divy up what free time I have and what my priorities are. I'm not sure when I'll come back to programming. I am sure that when I do, my focus and enjoyment of it will be improved. But there is so much I want to do besides code.
It's been a big year. I had two jobs before my present one. I briefly had a supercar. I started a new relationship for the first time since 2007. So perhaps it is no surprise that I haven't found as much time for hacking. That other forms of creation have fallen by the wayside a bit.
I miss the huge chunks of time I had in college. I want to both consume and create. To consume music, video games, MOOC courses, books I've been meaning to read for ages. To create code, mixtapes, music of my own, and who knows what else. But even though its hard to think of all that I get done versus all that I want to do and each weekend flies by like a screaming jet, I have to admit that things are going pretty well.
I can't believe Strangeloop is only a week away!
- Machine Learning for Relevance and Serendipity - Jenny Finkel (keynote)
- Fast and Dynamic - Maxime Chevalier-Boisvert
- Graph Computing at Scale - Matthias Broecheler
- The History of Women in Technology
- Software for Programming Cells - Colin Gravill
- Learnfun and Playfun - Tom Murphy VII
- Linear Logic Programming - Chris Martens
- Creative Machines - Joseph Wilk
- Making Software Development Make Sense to Everyone - Jen Myers (keynote)
- The Trouble with Types - Martin Odersky (keynote)
- Abstract Algebra Meets Analytics - Avi Bryant
- Programming a 144-computer chip to minimize power - Chuck Moore
- Web Apps in Clojure and Clojurescript with Pedestal - Brenton Ashworth
- Getting Pushy - David Pollak || Why Ruby Isn't Slow - Alex Gaynor
- Thinking DSLs for Massive Visualization - Leo Meyerovich
- Finding a Way Out - Chris Granger || Servo - Jack Moffitt
- What is a Strange Loop? - Douglas Hofstadter (keynote)
- Thrown for a Loop - David Stutz
This will be the last post about emulation that doesn't involve graphics or disassembly of old NES games, I promise. cl-6502 0.9.5 is out and, in my testing with SBCL, pretty snappy. The book has received updates and is also available on lulu. Below is the 'Lessons Learned - Common Lisp' chapter:
Structures can be preferable to classes
Structures are much more static than classes. They also enforce their slot types. When you have a solid idea of the layout of your data and really need speed, they're ideal.
CLOS is fast enough
CLOS, for single-dispatch at least, is really quite fast. When I redesigned the emulator to avoid a method call for every memory read/write, my benchmark only ran ~10% faster. I eventually chose to stick with the new scheme for several reasons, performance was only a minor factor.
Destructuring is more expensive than you think
My second big speedup came, indirectly, from changing the arguments to the
opcode lambdas. By having the opcode only take a single argument, the CPU, I
avoided the need to destructure the opcode metadata in
step-cpu. You don't
want to destructure a list in your inner loop, no matter how readable it is!
Eval-when is about data more than code
That is, the times I found myself using it always involved computing data at
compile-time that would be stored or accessed in a later phase. E.g. I used
it to ensure that the status-bit enum was created for use by
*mode-bodies* variable was bound in time for
try to go without it if possible.
Use DECLAIM (and DECLARE) wisely
DECLAIM is for global declarations and DECLARE is for local ones. Once you've
eked out as many algorithmic gains as possible and figured out your hotspots with
the profiler, recompile your code with
(declaim (optimize speed)) to see what
is keeping the compiler from generating fast code. Letting the compiler know the
FTYPE of your most called functions and inlining a few things can make a big
I haven't been doing any hacking on coleslaw or famiclom for the last month. I've been focused almost entirely on my 6502 CPU emulator. In particular, I've been optimizing it and turning it into a "readable program".
The optimizations have gone swimmingly, taking cl-6502 from 3.8 emulated mhz circa May 1st (commit eecfe7) to 29.3 emulated mhz today (commit b729e8).(0) A factor of 8 speedup feels pretty good though it would be fun to coax more speed out later.
(0): All figures obtained with SBCL 1.1.4 on Debian 64-bit on an old Thinkpad X200. See this 6502 forum post.
I feel that the readability of the program has remained, maybe even improved, through all that optimization. The same overall design is in place; most refactorings were, approximately, tweaking macros and their callsites. The readability is especially improved when the code is broken into chapters, each with an introduction for context, and typeset with LaTeX. The latter is done thanks to a simple Makefile and some very nifty code swiped from Luke Gorrie's snabbswitch. If you've been curious about how cl-6502 is implemented or just wanted to dive in, there's never been a better time. Grab the book!
NES + Lisp + Static Analysis = ?
I'm still planning to make famiclom a full NES emulator. I won't consider it done until I can play Mega Man 2 with it. Hopefully using a USB controller. It doesn't make much sense for Lisp in Summer Projects though. I've already started the project, the scope is ill-defined, and I want to work on something fresh and new to me. So I've come up with a project that I can't possibly complete instead. It'll be great!
In short, I intend to rewrite Super Mario Bros 1. ... in a not-yet-existing lisp-like macro-assembler/incredibly simple compiler. I already have a 6502 assembler/disassembler in cl-6502 and a tool to parse NES roms in romreader. There's also a very thorough annotated disassembly of Super Mario Bros floating around. I've got a good start on a static analyzer that will take the SMB binary and an entry point and try to build a CFG of the game. The current scheme won't work with memory mapped titles but it's good enough for Mario.
Once I have a graph representation of Mario Bros, I'll try both manual analysis of the annotated disassembly with a pen and pad, and automated analysis on the graph using Lisp. I'll try to find as many idioms and patterns as possible to condense and improve readability of the code. Then, somewhere in early August, I'll start trying to rewrite the source and see how far I can get.
This approach is probably completely: insane, unworkable, unviable, inadvisable, and just all around wrong. But I think I'll have fun and learn something, so it's good enough for me. And hell, who knows, maybe I'll get lucky and be able to attend ECLM next year. :)
Every. Single. Time.
Installing linux on a Mac never manages to be pleasant. I always remember, "Oh, yeah. I got this to work before." and dive in cheerily only to shortly recoil at what a fucking nightmare it is.
These days I use crunchbang, a very clean Debian-based distro. It's hard to even know who to be mad at. You'd think the installer would just detect the EFI/GPT stuff and "do the right thing" by now. Or be able to resize HFS+ partitions.
Out of Desperation
For whatever reason, I had to resort to this:
Resize OS X with Disk Utility to 50% of the current partition size.
Install rEFInd as rEFIt is now deprecated/unmaintained.
Burn a CD (!!!) since EFI doesn't want to boot from unrecognized USB drives.
Boot from the CD and install linux to the freespace.
When prompted to install the bootloader, install it on the linux partition, not the disk!
Do whatever it takes to get a linux command prompt, rescue disk, whatever. Mount the new install at /target.
mount -o bind /dev /target/dev mount -o bind /sys /target/sys mount -o bind /proc /target/proc chroot /target
Now you can actually do useful things, like fix the broken fucking bootloader! Note that you may need to
ifconfig eth0 up && dhclient eth0for network access.
Install and use gptsync on the drive with the linux partition:
apt-get update && apt-get install gptsync && gptsync /dev/sda
Reboot and pray.
Now your glorious Linux distro should boot on your "is-it-designed-not-to-work-with-a-single-fucking-Open-Source-thing?" Mac. Of course, you may need to apt-get remove the busted radeon drivers and add xforcevesa to the linux invocation in your /boot/grub/grub.cfg and remember to always gptsync after any future update-grub operations but...yeah, fuck all these goddamned computers.
This blog covers 2015, Books, Butler, C, Dad, Discrete Math, Displays, Education, Erlang, Essay, Gaming, Gapingvoid, HTDP, Hardware, IP Law, LISP, Lecture, Lessig, Linkpost, Linux, Lists, MPAA, Milosz, Music, Neruda, Open Source, Operating Systems, Personal, Pics, Poetry, Programming, Programming Languages, Project Euler, Quotes, Reddit, SICP, Self-Learning, Uncategorized, Webcomic, XKCD, Xmas, \"Real World\", adulthood, apple, careers, coleslaw, consumption, creation, fqa, games, goals, heroes, injustice, linux, lisp, math, melee, metapost, milosz, personal, poetry, programming, ragequit, recreation, rip, strangeloop, work
View content from 2015-05, 2015-03, 2015-02, 2015-01, 2014-11, 2014-09, 2014-07, 2014-05, 2014-01, 2013-10, 2013-09, 2013-07, 2013-06, 2013-05, 2013-04, 2013-03, 2013-01, 2012-12, 2012-10, 2012-09, 2012-08, 2012-06, 2012-05, 2012-04, 2012-03, 2012-01, 2011-10, 2011-09, 2011-08, 2011-07, 2011-06, 2011-05, 2011-04, 2011-02, 2011-01, 2010-11, 2010-10, 2010-09, 2010-08, 2010-07, 2010-05, 2010-04, 2010-03, 2010-02, 2010-01, 2009-12, 2009-11, 2009-10, 2009-09, 2009-08, 2009-07, 2009-06, 2009-05, 2009-04, 2009-03, 2009-02, 2009-01, 2008-12, 2008-11, 2008-10, 2008-09, 2008-08, 2008-07, 2008-06, 2008-05, 2008-04, 2008-03, 2008-02, 2008-01, 2007-12, 2007-11, 2007-10, 2007-09, 2007-08, 2007-07, 2007-06, 2007-05