Experiments in managing perfectionism

Compilers

Andrey and I talked about startups over a fun coffee chat today. We have a tendency as developers to build something cool that we think will translate well into a startup, but in doing so overlook two key skills.

Launching a startup in 2016 is akin to assembling an alt-rock band in 1996 or protesting the Vietnam War in 1971 - an act of youthful rebellion gone conformist. Since 2005, the year Y Combinator began, accelerators have sprung up everywhere to help transform startups from a skein of code into a bona-fide company.

It’s useful to focus on adding another zero to whatever you define as your success metric - money, status, impact on the world, or whatever. I am willing to take as much time as needed between projects to find my next thing. But I always want it to be a project that, if successful, will make the rest of my career look like a footnote.

Compilers

I felt a little out of depth trying to follow along more closely JavaScript optimizations from last week; I started taking a look at compilers to develop a bit more background.

Compilers was the first class I signed up for at Bradfield. I remember Oz described how fortunate we were as developers to have the ability to work at a higher level (libraries, frameworks) as well as at a lower levels (compilers). In light of that, the syllabus started with the following quote by Ras Bodik.

Take historical note of textile and steel industries: do you want to build machines and tools, or do you want to operate those machines?

Alas I discovered on Day 1 how it's more helpful to cover computer architecture first; I dropped compilers for that course instead. I went on to do all the core CS classes at Bradfield... except compilers. The class at the time had the Dragon book as the main text. This may have switched to Crafting Interpreters by Bob Nystrom (available for free online), which has an excerpt I find particularly endearing.

It’s the book I wish I had when I first started getting into languages, and it’s the book I’ve been writing in my head for nearly a decade.

If you find that you’re spending almost all your time on theory, start turning some attention to practical things; it will improve your theories. If you find that you’re spending almost all your time on practice, start turning some attention to theoretical things; it will improve your practice.

How the browser works

Running microbenchmarks led me down the path of JavaScript engines but also browsers in general. The mental model of the browser that I found helpful separates the browser UI, called the chrome, from the browser engine.

Let's split the browser engine into two parts. The first part 'talks to the internet' - starts with the URL, finds the remote server, and gets the desired HTML payload i.e. HTML/CSS/JS. The second part converts the HTML/CSS/JS into what you see and interact with in the browser.

I followed along in Python the series by Matt Brubeck on building a toy HTML/CSS rendering engine. This involves parsing the HTML/CSS text into tree-like objects that the computer understands, which are then processed and rendered visually.

I also came across a post describing how Google created V8 because Google Maps had been running really slowly, and getting Google Maps to run fast helps Google sell more ads. Perhaps. On the flip side, Google Maps is a fantastic product, V8 led to Node, and the need for speed led to network innovations like SPDY and QUIC.

Python

I've been a little distracted today, moving between browsers, V8 and JIT. Given the success of JIT, I'm a little surprised it's not more widely adopted by other languages, or perhaps, I just haven't noticed it before.

I also switched back to Python after time spent learning Go and Rust, in light of algorithm interview prep in the coming weeks. It took me a bit of time to adjust, but it felt liberating once I got started. It's like speaking a language without having to search for the words, allowing you to focus on the conversation itself.

Content: Grow the puzzle around you

I know I'm not in the Bay Area anymore when I have to explain what Y Combinator is. YC is a three-month startup school where founders receive coaching on how to refine their product and sell their ideas to investors. Its head-turning record of success has inspired hundreds of other accelerators worldwide, keen to replicate YC's investment and training philosophy (source: The Economist).

You are a jigsaw puzzle piece of a certain shape. You could change your shape to fit an existing hole in the world. That was the traditional plan. But there’s another way that can often be better for you and for the world: to grow a new puzzle around you. That’s what I did, and I was a pretty weird-shaped piece. So if I can do it, there’s more hope for you than you probably realize.

WebAssembly

Nick Fitzgerald has a post on using Rust compiled to WebAssembly to replace performance-sensitive parts of the JavaScript library source-map, achieving a 6x speedup. Vyacheslav Egorov responded with his own post on matching the speedups with JavaScript alone. The profiling and optimization process is detailed and very cool to see. In the third and final post, Fitzgerald discussed the end result of an 11x speedup of the Rust and WebAssembly implementation by incorporating ideas from both sources.

What I find illuminating is the process in which these optimizations are at obtained (as per the title). The JavaScript version resulted in hard-to-maintain code in the name of performance, whereas in Rust one was able to get the best of both worlds - clear expression of interest and optimal runtime performance.

Excessive allocation rates make any garbage collector (or malloc and free implementation) a bottleneck. Monomorphization and inlining are crucial to eking out performance in both Rust and JavaScript. Algorithms transcend programming languages. But a distinction between JavaScript and Rust+WebAssembly emerges when we consider the effort required to attain inlining and monomorphization, or to avoid allocations.

Rust lets us explicitly state our desires to the compiler, we can rely on the optimizations occurring. Rust's natural idioms guide us towards fast code, so we don’t have to be performance wizards to get fast code. In JavaScript, on the other hand, we must communicate with oblique incantations to match each JavaScript engine’s JIT’s heuristics.

Content: Custodians of the custard

Antiga Confeitaria de Belém is a cafe/bakery in the Lisbon suburb of Belém that's been making Portuguese egg tarts since 1837. The secret recipe of the filling is entrusted to three pastry chefs, the most recent one after 20 years in the kitchen! The quest for the perfect tart, and resisting the temptation to move to the mass market, makes for a charming story (enclosed below).

Content: Somewhere Inside

I love Tiësto, and I especially love how the track comes together with Julie Thompson's energetic vocals.

Content: POP/STARS

I've always been fascinated by new mediums that inspire new business models. Defense of the Ancients (more commonly-known by its acronym DotA) started as a fan-made modification of the computer game Warcraft 3. The game developer Valve then hired DotA's lead designer to develop a sequel to the highly popular 'mod'. DotA 2 was released for free, but monetized through sales of cosmetic enhancements for the game characters (referred to as 'skins').

To promote the shift to DotA 2, Valve organized a gaming competition for top teams from all over the world called The International. The International started in 2011 with a $1.6mm prize pool. In 2013, Valve started crowdfunding the prize pool through sales of an in-game battle pass. The prize pool grew to $3mm and was already the largest prize pool for any e-sports competition. By 2020 the prize pool ballooned to $34mm.

The featured video is actually from League of Legends, DotA's main competitor. League has Worlds, their own version of The International, as well as a talented team of composers, songwriters and producers to orchestrate cinematic music for the opening ceremony. The musical score for Worlds 2018 was performed by a virtual K-pop group called K/DA, created especially for the occasion. The song POP/STARS went viral, receiving 5 million views within 24 hours.

How did this start? To promote sales of skins.

Content: Startup

I read Jerry Kaplan's Startup from time to time, and always amazed how his struggles as a founder in the late 1980s are ever familiar today - raising money, hiring talent, finding product-market fit, dealing with partners and competitors (while constantly assessing which is which). I particularly enjoy the framing of the venture as a game.

It begins with an aspiring entrepreneur who is willing to step right up and be tested. As in many other games, the player starts with an artificial currency - in this case the stock of a new venture. The goal is simple: increase the value of the entrepreneurs' shares, because when the game is over, these can be cashed in for real money. The trick is to swap some of the stock for three resources - ideas, money and people - then use these resources to increase the value of the remaining stock.

I know. It's become a cliché. I'm still a believer.

It is comforting to know that the human impulse to make the world a better place is not today confined to the young or the foolish. It is alive and well among the people that live and work in Silicon Valley.

Classification of audio data

At yesterday's workshop on neural networks, we trained a convolutional neural network (CNN) on the MNIST dataset. The MNIST dataset consists of hand-drawn digits from zero to nine, 28 pixels by 28 pixels in size. A traditional ML model would treat each of the 784 pixels as simply another feature (hence agnostic to column ordering). A CNN would take 'snapshots' of the image, retaining its 2-dimensional structure and (in line with intuition) outperforms the traditional ML model.

There's a team at RC participating in Kaggle's Birdcall Recognition competition. The competition involves identifying the name of the bird in a birdcall audio file. I learned that converting the audio format into an image and then training a CNN on the image is a common technique. The visual representation of the spectrum of frequencies over time is called a spectrogram.

The question that arose was how to deal with metadata. I won't go into too much detail since the competition is still ongoing; we explored a number of possibilities with multi-label models. The basic premise is we have a set of models that perform equally well on the main performance metric, and we use the other labels to select the best (or perhaps, the most robust) among them. Excited to see how this plays out!

Content: Data engineering

A common functional split in a company is between data producers and data consumers. Data producers tend to be engineering team(s) that create and maintain ETLs to move and transform data from one source to another. Data consumers tend to be data science team(s) that use the data for analysis and/or modeling.

The problem with this setup relates to handoffs. Data consumers have more context on the data. When there are data quality issues, data consumer will raise this with data producers. There's usually a bottleneck: data consumers care about the data but are blocked pending a fix, data producers don't have as much insight into use cases but have to prioritize inbound requests.

The following Stitch Fix post proposes a different model, reducing instances where the problem is passed "over the wall". In this model, data producers create tools to make it easy for consumers to create and maintain ETLs i.e. "design new Lego blocks". Data consumers owns the process end-to-end, i.e. "assemble [the Lego blocks] in creative ways to create new data science".

https://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/

This way each team can take ownership of the things they care about, iterate autonomously and move faster together.

WebAssembly

The primary use case for WebAssembly in recent discussions is to enable languages other than JavaScript to run in the browser. The .wasm format, however, could itself become the standard in portable binaries.

Suppose we wanted to speed up Python code with Rust bindings. This requires the Rust code to be compiled to a dynamic library, i.e. .dylib file on Mac, .so on Linux, or .dll on Windows, which can then be imported in Python with a standard Python import (discussion here).

Now suppose the Rust code was compiled to a .wasm file instead. We can similarly import the .wasm file into Python with wasmtime (example here). Since the .wasm format is OS-independent, I believe we now have a portable version of the dynamic library.

Neural networks

At Square, I ran workshops on machine learning and (separately) on neural networks. I presented on the latter today at RC.

Starting with the most basic representation of a neural network as matrix operations, we build it up in stages - reducing loss systematically, capturing non-linear behavior, and introducing regularization with dropout layers. We would train a convolutional neural network by the end of the session, highlighting the improved performance when the model architecture incorporates the 2-dimensional structure of image data.

The Github repo has been updated to Python 3 and the latest Keras API. Then and now, I couldn't resist re-iterating Andrej Karpathy's advice to use model training best practices i.e. "don't be a hero".

Content: Why doctors hate their computers

At the start of my coffee chat with SengMing, I told him I was choosing between New Yorker articles to feature and had just opened a browser tab on Atul Gawande's article. SengMing then shared how much he loved the article; this made the decision easy.

It was published in 2018 but I enjoyed reading it so much more this time around. The references to pain points in the user experience were all too familiar - too many clicks, too difficult to find relevant information, too complicated to use. The issues pertaining to software development are universal except in this case, to use Sengming's words, the quality "literally affects life and death". 

https://www.newyorker.com/magazine/2018/11/12/why-doctors-hate-their-computers

Consider that, in recent years, one of the fastest-growing occupations in health care has been medical-scribe work, a field that hardly existed before electronic medical records. Medical scribes are trained assistants who work alongside physicians to take computer-related tasks off their hands. This fix is, admittedly, a little ridiculous. We replaced paper with computers because paper was inefficient. Now computers have become inefficient, so we’re hiring more humans. And it sort of works.

WebAssembly

I ran a demo on WebAssembly at Presentations last Friday. I had to gloss over a number of details given the 5-minute format, so I did an extended version today.

As per Lin Clark's post, WebAssembly is "taking code written in programming languages other than JavaScript and running that code in the browser". Today I added how a major project underway is a standalone runtime so .wasm binaries can be run independent of the browser.

Re: portability, I previously used the following image with C, C++ and Rust as starting points. These so-called system languages are particularly suitable for compiling to WebAssembly given the absence of a garbage collector and minimal language runtime. I'm often confused by the term runtime, more detailed explanation here.

Re: performance, I had mentioned how WebAssembly gets to skip some execution steps compared to JavaScript. The wrinkle is the JIT currently doesn't know how to deal with WebAssembly directly, limiting speed gains (see 'trampolining' here). In fact, the main example I've come across in practice is a Figma post focusing on speed ups from smaller binaries.

Re: security, I had skipped this entirely. I provided a high-level overview of the ArrayBuffer today.

In the demo, I compiled and ran WebAssembly. I spoke a bit more on future improvements on the roadmap (see 'post MVP features' here), in particular access to the DOM and direct loading of WebAssembly modules (hence no need for a web server).

Microbenchmarks

Reviewing the feedback on the 5-minute presentation, comparing the performance of WebAssembly against JavaScript was at the top of the list. What's amusing is I wanted to learn WebAssembly but ended up having newfound respect for the V8 JavaScript engine.

I started with prime number generation, and puzzled how WebAssembly ran slower against JavaScript. Then I realized how even the Rust implementation ran slower against JavaScript. Andrey found a post on this exact comparison. My sense was the execution involved one main loop, which provided a textbook example for JIT optimization.

Moving on, I found a post on Julia microbenchmarks and thought I can simply use the tests where JavaScript wasn't as fast. Running matrix multiply, I had LLVM issues and discovered I needed to install Fortran to use BLAS. Oh boy. Post-installation I might still have been missing some symlinks, so kept this on hold for the time being.

With quicksort, Rust completed in half the time of JavaScript but I didn't have time to set this up in the browser before the presentation. I did have the n-queens problem as an example where Rust (compiled to WebAssembly) ran in half the time of JavaScript, in the browser.

When running microbenchmarks in the past, it's always been about comparing two algorithms in the same language. Cross-language benchmarks are fascinating. Do you replicate step-by-step, or rewrite in the way the language performs best (hence needing to know best practices in both languages)? If the latter, would there be more subtlety when comparing imperative against functional languages?

The other issue I had was microbenchmarking in the browser. If you set up the start and end times in JavaScript, does this mean you can't time the parsing stage (since by that point the code is already parsed)?

Content: 15 startups in 21 months

In his essay, Paul Graham isolated the key quality needed in a founder - relentless resourceful(ness). The story that often gets told is of Airbnb, where the co-founders managed to extend their runway by selling cereal during the 2008 Presidential Elections. The story I quite like is actually this one.

https://medium.com/@mecolalu/how-we-did-15-startups-in-21-months-a7008c8f2c97

There's a happy ending; I won't spoil it more than that.

WebAssembly

Following up on the 'watch this space' declaration in last week's post, I created a Github repo on minimal examples to compile and run WebAssembly. It's designed to help consider different options - whether it be Rust or .wat as a starting point, whether to run in the browser or with an independent runtime, whether to extend with Rust-JavaScript interoperability or even go full stack Rust.

In summary, a starting point to those new to WebAssembly to construct a mental model and build on top of.

Learning WebAssembly, I've gone down the path of reading up more on how the browser works. I had actually mentioned wanting to review this in my first week of RC. This time, however, feels a lot more engaging. There's a lot more concepts to make connections with. Most importantly it feels more fun this way; it's not something looked into simply to solve a problem.

Python

Vinayak recommended a PyCon talk by Russell Keith-Magee at our coffee chat last week. I had the chance to watch this talk over the weekend. The talk was illuminating.

First, in thinking about the future of Python, one needs to think about black swan events. A black swan event is an event that is a surprise to the observer with a major effect on the world, but can be easily in explained in hindsight. This being the case, the best way to avoid being on the wrong side of a black swan event is to actively challenge one's own assumptions. The example he used was America's Cup; adhering to this principle has injected innovation into the sailing competition, breathing new life and excitement for many years to come.

Next was the framing of WebAssembly as Python's (or for that matter, all programming language's) black swan. WebAssembly started life as a research project to identify the set of primitive JavaScript operations that will execute in modern browser engines. The result of this work is the ability to run languages other than JavaScript in the browser, and do so in a fast and secure way. He's essentially throwing down the gauntlet, galvanizing the community to embrace this new paradigm instead of simply continuing to do what Python is good at. In this regard, perhaps be more like... the Rust community?

The final point that hit home was Keith-Magee's experience of burnout as a maintainer in the open source community. It's very disheartening to hear some members he had to deal with, who receive a product at no cost but somehow feel a strong sense of entitlement. I felt a renewed sense of appreciation for the open source community, and a continued desire to contribute.

It turned out the talk Vinayak recommended was the talk at PyCon AU 2019! This was also very good, and is more practical re: Python and WebAssembly. The talk covers a bit of history of WebAssembly, tools to help compile Python to WebAssembly, and the future for Python on the web. In a previous post I described how WebAssembly reminded me of Java, it's curious hearing Keith-Magee describe how "wasm has delivered on the promise Java made".

OK one last note about previous posts - I mentioned how some open source projects may need a bit of hand holding. Keith-Magee's current project is called BeeWare and has a spectacular first-time contributors page.

Content: Climbing to the top

OK. Today I'm not just overloading with previous posts but also with content. Things are slowly coming together. I'm reminded of Rainer Maria Rilke in Letters to a Young Poet.

I would like to beg you dear Sir, as well as I can, to have patience with everything unresolved in your heart and try to love the questions themselves as if they were locked rooms or books written in a very foreign language. Don’t search for the answers, which could not be given to you now, because you would not be able to live them. And the point is to live everything. Live the questions now. Perhaps then, someday far in the future, you will gradually, without even noticing it, live your way into the answer.

I'm reminded of Steve Job's Stanford commencement speech on how you can't connect the dots moving forward, you can only connect them looking backwards, so you have to trust that the dots will somehow connect in the future.

That being said, I've only just gotten started. I'm reminded of Terence Tao who, despite all his achievements, speaks modestly of the journey still to come.

https://www.latimes.com/local/lanow/la-me-ln-ucla-math-prize-20140626-story.html

You want to get to the top of the cliff. But that’s not what you focus on immediately. You focus on the next ledge just beyond your reach, because you need to do one clever thing to get up there. And then once you get there, you do it again. A lot of this is rather boring and not very glamorous. But you can’t jump cliffs in a single bound.

Kubernetes

I had a coffee chat with Sophia today, we got talking about Kubernetes, and I shared this comic that actually explains pretty well what the 'open source container orchestration system'.

I find it interesting how polished Google's more recent open source efforts are, with serious marketing budgets. Another project that had a huge launch (though not as much fanfare these days) was Tensorflow. I wonder if this was the lesson from having come up with MapReduce, but see the technology popularized as Hadoop instead?

Content: International Committee of the Red Cross

This article came to the top of the list on what next to feature. It highlights the delicate balance between maintaining confidentiality to help those in need, against exposing the morally-corrupt but risking access in future missions (enclosed below).

What I didn't pick up as prominently on the first reading was how the founder of Médecins Sans Frontières came out of ICRC, with the goal of creating an organization that combined relief with advocacy.

Content: Life in Mono

I can't remember where I first heard this. How apt given the song sets a wistful mood.

Content: Yassin Falafel

OK I'm going to gush about Square again, though it's a timely refrain of the theme of adapting to an adopted homeland (Farouk al-Kasim of a previous post).

Seriously, though, I was blown away when I watched this.

Content: The Ride of a Lifetime

The first book I listened to on Audible was Bob Iger's biography. In general I've not been a fan of this genre; biographies I've read in the past have narratives that go on and on about how this person was destined to be successful, all the stars align. In this one, I recall he worked hard and treated people with respect. I remember an honest and candid retelling of his rise to the top, where in no way was the path assured.

The Ride of a Lifetime and Shoe Dog - I'd love to find more books like these. I recorded this exact quote on my phone so I could replay it again and again.

A company’s culture is shaped by a lot of things, but this is one of the most important - you have to convey your priorities clearly and repeatedly. In my experience, it’s what separates great managers from the rest. If leaders don’t articulate their priorities clearly, then the people around them don’t know what their own priorities should be. Time and energy and capital get wasted. People in your organization suffer unnecessary anxiety because they don’t know what they should be focused on. Inefficiency sets in, frustration builds up, morale sinks.

You can do a lot for the morale of the people around you (and therefore the people around them) just by taking the guesswork out of their day-to-day life.

WebAssembly

I was preparing slides on WebAssembly today when I came across Lin Clark's cartoon intros. It's spectacular. I wished it was the first thing I read on the topic; it appears l'esprit de l'escalier is a thing for content too.

• A cartoon intro to WebAssembly
• Creating and working with WebAssembly modules
• What makes WebAssembly fast?
• Where is WebAssembly now and what's next?

The background articles also provide helpful context and an easy read.

• A crash course in just-in-time (JIT) compilers
• A crash course in assembly

It's amazing how Google, Mozilla, Apple and Microsoft actually got together and agreed on the specs. The project has now expanded beyond the browser, with wasmtime as an independent runtime and WASI as the unified interface. In addition to its own foundation, called the Bytecode Alliance.

I do wonder, more broadly, if the technology will further consolidate the dominance that large tech companies have, or will a thousand startups bloom? Will there be a lot of end users who like it but only few love? What will the killer app be (or will there actually be one)? I was late to notice iPhones and bitcoin as platforms. I'm ecstatic at being able to follow the WebAssembly life cycle from an early stage, and see where it goes from here.

Julia

I woke up in the middle of the night, had trouble going back to sleep and actually looked up why Julia is 1-indexed (or rather, why most languages are 0-indexed). I came across this post which had the following quote.

So: the technical reason we started counting arrays at zero is that in the mid-1960’s, you could shave a few cycles off of a program’s compilation time on an IBM 7094. The social reason is that we had to save every cycle we could, because if the job didn’t finish fast it might not finish at all and you never know when you’re getting bumped off the hardware...

Intuitively it sort of make sense - if you start from zero it's a no-op vs having to do an `add immediate`. Fascinating.

Content: Dev Ops

This week's feature is Increment's post on cloud migration; it fits in nicely with a friend joining Stripe this week. I enjoyed reading how Netflix introduced planned instance failures so the on-call team can to deal with it during business hours. What's particularly impressive is the decision to implement this workflow at a time of rapid growth, forcing adoption of industry best practice at the same time as battling other fires.

https://increment.com/cloud/case-studies-in-cloud-migration

A hat tip to the marketing team who coined the term 'chaos engineering' from 'chaos monkeys'.