Experiments in managing perfectionism

WebAssembly

Following up on the 'watch this space' declaration in last week's post, I created a Github repo on minimal examples to compile and run WebAssembly. It's designed to help consider different options - whether it be Rust or .wat as a starting point, whether to run in the browser or with an independent runtime, whether to extend with Rust-JavaScript interoperability or even go full stack Rust.

In summary, a starting point to those new to WebAssembly to construct a mental model and build on top of.

Learning WebAssembly, I've gone down the path of reading up more on how the browser works. I had actually mentioned wanting to review this in my first week of RC. This time, however, feels a lot more engaging. There's a lot more concepts to make connections with. Most importantly it feels more fun this way; it's not something looked into simply to solve a problem.

Python

Vinayak recommended a PyCon talk by Russell Keith-Magee at our coffee chat last week. I had the chance to watch this talk over the weekend. The talk was illuminating.

First, in thinking about the future of Python, one needs to think about black swan events. A black swan event is an event that is a surprise to the observer with a major effect on the world, but can be easily in explained in hindsight. This being the case, the best way to avoid being on the wrong side of a black swan event is to actively challenge one's own assumptions. The example he used was America's Cup; adhering to this principle has injected innovation into the sailing competition, breathing new life and excitement for many years to come.

Next was the framing of WebAssembly as Python's (or for that matter, all programming language's) black swan. WebAssembly started life as a research project to identify the set of primitive JavaScript operations that will execute in modern browser engines. The result of this work is the ability to run languages other than JavaScript in the browser, and do so in a fast and secure way. He's essentially throwing down the gauntlet, galvanizing the community to embrace this new paradigm instead of simply continuing to do what Python is good at. In this regard, perhaps be more like... the Rust community?

The final point that hit home was Keith-Magee's experience of burnout as a maintainer in the open source community. It's very disheartening to hear some members he had to deal with, who receive a product at no cost but somehow feel a strong sense of entitlement. I felt a renewed sense of appreciation for the open source community, and a continued desire to contribute.

It turned out the talk Vinayak recommended was the talk at PyCon AU 2019! This was also very good, and is more practical re: Python and WebAssembly. The talk covers a bit of history of WebAssembly, tools to help compile Python to WebAssembly, and the future for Python on the web. In a previous post I described how WebAssembly reminded me of Java, it's curious hearing Keith-Magee describe how "wasm has delivered on the promise Java made".

OK one last note about previous posts - I mentioned how some open source projects may need a bit of hand holding. Keith-Magee's current project is called BeeWare and has a spectacular first-time contributors page.

Content: Climbing to the top

OK. Today I'm not just overloading with previous posts but also with content. Things are slowly coming together. I'm reminded of Rainer Maria Rilke in Letters to a Young Poet.

I would like to beg you dear Sir, as well as I can, to have patience with everything unresolved in your heart and try to love the questions themselves as if they were locked rooms or books written in a very foreign language. Don’t search for the answers, which could not be given to you now, because you would not be able to live them. And the point is to live everything. Live the questions now. Perhaps then, someday far in the future, you will gradually, without even noticing it, live your way into the answer.

I'm reminded of Steve Job's Stanford commencement speech on how you can't connect the dots moving forward, you can only connect them looking backwards, so you have to trust that the dots will somehow connect in the future.

That being said, I've only just gotten started. I'm reminded of Terence Tao who, despite all his achievements, speaks modestly of the journey still to come.

https://www.latimes.com/local/lanow/la-me-ln-ucla-math-prize-20140626-story.html

You want to get to the top of the cliff. But that’s not what you focus on immediately. You focus on the next ledge just beyond your reach, because you need to do one clever thing to get up there. And then once you get there, you do it again. A lot of this is rather boring and not very glamorous. But you can’t jump cliffs in a single bound.

WebAssembly

I ran a demo on WebAssembly at Presentations last Friday. I had to gloss over a number of details given the 5-minute format, so I did an extended version today.

As per Lin Clark's post, WebAssembly is "taking code written in programming languages other than JavaScript and running that code in the browser". Today I added how a major project underway is a standalone runtime so .wasm binaries can be run independent of the browser.

Re: portability, I previously used the following image with C, C++ and Rust as starting points. These so-called system languages are particularly suitable for compiling to WebAssembly given the absence of a garbage collector and minimal language runtime. I'm often confused by the term runtime, more detailed explanation here.

Re: performance, I had mentioned how WebAssembly gets to skip some execution steps compared to JavaScript. The wrinkle is the JIT currently doesn't know how to deal with WebAssembly directly, limiting speed gains (see 'trampolining' here). In fact, the main example I've come across in practice is a Figma post focusing on speed ups from smaller binaries.

Re: security, I had skipped this entirely. I provided a high-level overview of the ArrayBuffer today.

In the demo, I compiled and ran WebAssembly. I spoke a bit more on future improvements on the roadmap (see 'post MVP features' here), in particular access to the DOM and direct loading of WebAssembly modules (hence no need for a web server).

Microbenchmarks

Reviewing the feedback on the 5-minute presentation, comparing the performance of WebAssembly against JavaScript was at the top of the list. What's amusing is I wanted to learn WebAssembly but ended up having newfound respect for the V8 JavaScript engine.

I started with prime number generation, and puzzled how WebAssembly ran slower against JavaScript. Then I realized how even the Rust implementation ran slower against JavaScript. Andrey found a post on this exact comparison. My sense was the execution involved one main loop, which provided a textbook example for JIT optimization.

Moving on, I found a post on Julia microbenchmarks and thought I can simply use the tests where JavaScript wasn't as fast. Running matrix multiply, I had LLVM issues and discovered I needed to install Fortran to use BLAS. Oh boy. Post-installation I might still have been missing some symlinks, so kept this on hold for the time being.

With quicksort, Rust completed in half the time of JavaScript but I didn't have time to set this up in the browser before the presentation. I did have the n-queens problem as an example where Rust (compiled to WebAssembly) ran in half the time of JavaScript, in the browser.

When running microbenchmarks in the past, it's always been about comparing two algorithms in the same language. Cross-language benchmarks are fascinating. Do you replicate step-by-step, or rewrite in the way the language performs best (hence needing to know best practices in both languages)? If the latter, would there be more subtlety when comparing imperative against functional languages?

The other issue I had was microbenchmarking in the browser. If you set up the start and end times in JavaScript, does this mean you can't time the parsing stage (since by that point the code is already parsed)?

Content: 15 startups in 21 months

In his essay, Paul Graham isolated the key quality needed in a founder - relentless resourceful(ness). The story that often gets told is of Airbnb, where the co-founders managed to extend their runway by selling cereal during the 2008 Presidential Elections. The story I quite like is actually this one.

https://medium.com/@mecolalu/how-we-did-15-startups-in-21-months-a7008c8f2c97

There's a happy ending; I won't spoil it more than that.

WebAssembly

The primary use case for WebAssembly in recent discussions is to enable languages other than JavaScript to run in the browser. The .wasm format, however, could itself become the standard in portable binaries.

Suppose we wanted to speed up Python code with Rust bindings. This requires the Rust code to be compiled to a dynamic library, i.e. .dylib file on Mac, .so on Linux, or .dll on Windows, which can then be imported in Python with a standard Python import (discussion here).

Now suppose the Rust code was compiled to a .wasm file instead. We can similarly import the .wasm file into Python with wasmtime (example here). Since the .wasm format is OS-independent, I believe we now have a portable version of the dynamic library.

Neural networks

At Square, I ran workshops on machine learning and (separately) on neural networks. I presented on the latter today at RC.

Starting with the most basic representation of a neural network as matrix operations, we build it up in stages - reducing loss systematically, capturing non-linear behavior, and introducing regularization with dropout layers. We would train a convolutional neural network by the end of the session, highlighting the improved performance when the model architecture incorporates the 2-dimensional structure of image data.

The Github repo has been updated to Python 3 and the latest Keras API. Then and now, I couldn't resist re-iterating Andrej Karpathy's advice to use model training best practices i.e. "don't be a hero".

Content: Why doctors hate their computers

At the start of my coffee chat with SengMing, I told him I was choosing between New Yorker articles to feature and had just opened a browser tab on Atul Gawande's article. SengMing then shared how much he loved the article; this made the decision easy.

It was published in 2018 but I enjoyed reading it so much more this time around. The references to pain points in the user experience were all too familiar - too many clicks, too difficult to find relevant information, too complicated to use. The issues pertaining to software development are universal except in this case, to use Sengming's words, the quality "literally affects life and death". 

https://www.newyorker.com/magazine/2018/11/12/why-doctors-hate-their-computers

Consider that, in recent years, one of the fastest-growing occupations in health care has been medical-scribe work, a field that hardly existed before electronic medical records. Medical scribes are trained assistants who work alongside physicians to take computer-related tasks off their hands. This fix is, admittedly, a little ridiculous. We replaced paper with computers because paper was inefficient. Now computers have become inefficient, so we’re hiring more humans. And it sort of works.

Classification of audio data

At yesterday's workshop on neural networks, we trained a convolutional neural network (CNN) on the MNIST dataset. The MNIST dataset consists of hand-drawn digits from zero to nine, 28 pixels by 28 pixels in size. A traditional ML model would treat each of the 784 pixels as simply another feature (hence agnostic to column ordering). A CNN would take 'snapshots' of the image, retaining its 2-dimensional structure and (in line with intuition) outperforms the traditional ML model.

There's a team at RC participating in Kaggle's Birdcall Recognition competition. The competition involves identifying the name of the bird in a birdcall audio file. I learned that converting the audio format into an image and then training a CNN on the image is a common technique. The visual representation of the spectrum of frequencies over time is called a spectrogram.

The question that arose was how to deal with metadata. I won't go into too much detail since the competition is still ongoing; we explored a number of possibilities with multi-label models. The basic premise is we have a set of models that perform equally well on the main performance metric, and we use the other labels to select the best (or perhaps, the most robust) among them. Excited to see how this plays out!

Content: Data engineering

A common functional split in a company is between data producers and data consumers. Data producers tend to be engineering team(s) that create and maintain ETLs to move and transform data from one source to another. Data consumers tend to be data science team(s) that use the data for analysis and/or modeling.

The problem with this setup relates to handoffs. Data consumers have more context on the data. When there are data quality issues, data consumer will raise this with data producers. There's usually a bottleneck: data consumers care about the data but are blocked pending a fix, data producers don't have as much insight into use cases but have to prioritize inbound requests.

The following Stitch Fix post proposes a different model, reducing instances where the problem is passed "over the wall". In this model, data producers create tools to make it easy for consumers to create and maintain ETLs i.e. "design new Lego blocks". Data consumers owns the process end-to-end, i.e. "assemble [the Lego blocks] in creative ways to create new data science".

https://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/

This way each team can take ownership of the things they care about, iterate autonomously and move faster together.

WebAssembly

Nick Fitzgerald has a post on using Rust compiled to WebAssembly to replace performance-sensitive parts of the JavaScript library source-map, achieving a 6x speedup. Vyacheslav Egorov responded with his own post on matching the speedups with JavaScript alone. The profiling and optimization process is detailed and very cool to see. In the third and final post, Fitzgerald discussed the end result of an 11x speedup of the Rust and WebAssembly implementation by incorporating ideas from both sources.

What I find illuminating is the process in which these optimizations are at obtained (as per the title). The JavaScript version resulted in hard-to-maintain code in the name of performance, whereas in Rust one was able to get the best of both worlds - clear expression of interest and optimal runtime performance.

Excessive allocation rates make any garbage collector (or malloc and free implementation) a bottleneck. Monomorphization and inlining are crucial to eking out performance in both Rust and JavaScript. Algorithms transcend programming languages. But a distinction between JavaScript and Rust+WebAssembly emerges when we consider the effort required to attain inlining and monomorphization, or to avoid allocations.

Rust lets us explicitly state our desires to the compiler, we can rely on the optimizations occurring. Rust's natural idioms guide us towards fast code, so we don’t have to be performance wizards to get fast code. In JavaScript, on the other hand, we must communicate with oblique incantations to match each JavaScript engine’s JIT’s heuristics.

Content: Custodians of the custard

Antiga Confeitaria de Belém is a cafe/bakery in the Lisbon suburb of Belém that's been making Portuguese egg tarts since 1837. The secret recipe of the filling is entrusted to three pastry chefs, the most recent one after 20 years in the kitchen! The quest for the perfect tart, and resisting the temptation to move to the mass market, makes for a charming story (enclosed below).

Content: Somewhere Inside

I love Tiësto, and I especially love how the track comes together with Julie Thompson's energetic vocals.

Content: POP/STARS

I've always been fascinated by new mediums that inspire new business models. Defense of the Ancients (more commonly-known by its acronym DotA) started as a fan-made modification of the computer game Warcraft 3. The game developer Valve then hired DotA's lead designer to develop a sequel to the highly popular 'mod'. DotA 2 was released for free, but monetized through sales of cosmetic enhancements for the game characters (referred to as 'skins').

To promote the shift to DotA 2, Valve organized a gaming competition for top teams from all over the world called The International. The International started in 2011 with a $1.6mm prize pool. In 2013, Valve started crowdfunding the prize pool through sales of an in-game battle pass. The prize pool grew to $3mm and was already the largest prize pool for any e-sports competition. By 2020 the prize pool ballooned to $34mm.

The featured video is actually from League of Legends, DotA's main competitor. League has Worlds, their own version of The International, as well as a talented team of composers, songwriters and producers to orchestrate cinematic music for the opening ceremony. The musical score for Worlds 2018 was performed by a virtual K-pop group called K/DA, created especially for the occasion. The song POP/STARS went viral, receiving 5 million views within 24 hours.

How did this start? To promote sales of skins.

Content: Startup

I read Jerry Kaplan's Startup from time to time, and always amazed how his struggles as a founder in the late 1980s are ever familiar today - raising money, hiring talent, finding product-market fit, dealing with partners and competitors (while constantly assessing which is which). I particularly enjoy the framing of the venture as a game.

It begins with an aspiring entrepreneur who is willing to step right up and be tested. As in many other games, the player starts with an artificial currency - in this case the stock of a new venture. The goal is simple: increase the value of the entrepreneurs' shares, because when the game is over, these can be cashed in for real money. The trick is to swap some of the stock for three resources - ideas, money and people - then use these resources to increase the value of the remaining stock.

I know. It's become a cliché. I'm still a believer.

It is comforting to know that the human impulse to make the world a better place is not today confined to the young or the foolish. It is alive and well among the people that live and work in Silicon Valley.

Python

I've been a little distracted today, moving between browsers, V8 and JIT. Given the success of JIT, I'm a little surprised it's not more widely adopted by other languages, or perhaps, I just haven't noticed it before.

I also switched back to Python after time spent learning Go and Rust, in light of algorithm interview prep in the coming weeks. It took me a bit of time to adjust, but it felt liberating once I got started. It's like speaking a language without having to search for the words, allowing you to focus on the conversation itself.

Content: Grow the puzzle around you

I know I'm not in the Bay Area anymore when I have to explain what Y Combinator is. YC is a three-month startup school where founders receive coaching on how to refine their product and sell their ideas to investors. Its head-turning record of success has inspired hundreds of other accelerators worldwide, keen to replicate YC's investment and training philosophy (source: The Economist).

You are a jigsaw puzzle piece of a certain shape. You could change your shape to fit an existing hole in the world. That was the traditional plan. But there’s another way that can often be better for you and for the world: to grow a new puzzle around you. That’s what I did, and I was a pretty weird-shaped piece. So if I can do it, there’s more hope for you than you probably realize.

Compilers

I felt a little out of depth trying to follow along more closely JavaScript optimizations from last week; I started taking a look at compilers to develop a bit more background.

Compilers was the first class I signed up for at Bradfield. I remember Oz described how fortunate we were as developers to have the ability to work at a higher level (libraries, frameworks) as well as at a lower levels (compilers). In light of that, the syllabus started with the following quote by Ras Bodik.

Take historical note of textile and steel industries: do you want to build machines and tools, or do you want to operate those machines?

Alas I discovered on Day 1 how it's more helpful to cover computer architecture first; I dropped compilers for that course instead. I went on to do all the core CS classes at Bradfield... except compilers. The class at the time had the Dragon book as the main text. This may have switched to Crafting Interpreters by Bob Nystrom (available for free online), which has an excerpt I find particularly endearing.

It’s the book I wish I had when I first started getting into languages, and it’s the book I’ve been writing in my head for nearly a decade.

If you find that you’re spending almost all your time on theory, start turning some attention to practical things; it will improve your theories. If you find that you’re spending almost all your time on practice, start turning some attention to theoretical things; it will improve your practice.

How the browser works

Running microbenchmarks led me down the path of JavaScript engines but also browsers in general. The mental model of the browser that I found helpful separates the browser UI, called the chrome, from the browser engine.

Let's split the browser engine into two parts. The first part 'talks to the internet' - starts with the URL, finds the remote server, and gets the desired HTML payload i.e. HTML/CSS/JS. The second part converts the HTML/CSS/JS into what you see and interact with in the browser.

I followed along in Python the series by Matt Brubeck on building a toy HTML/CSS rendering engine. This involves parsing the HTML/CSS text into tree-like objects that the computer understands, which are then processed and rendered visually.

I also came across a post describing how Google created V8 because Google Maps had been running really slowly, and getting Google Maps to run fast helps Google sell more ads. Perhaps. On the flip side, Google Maps is a fantastic product, V8 led to Node, and the need for speed led to network innovations like SPDY and QUIC.

Compilers

Andrey and I talked about startups over a fun coffee chat today. We have a tendency as developers to build something cool that we think will translate well into a startup, but in doing so overlook two key skills.

Launching a startup in 2016 is akin to assembling an alt-rock band in 1996 or protesting the Vietnam War in 1971 - an act of youthful rebellion gone conformist. Since 2005, the year Y Combinator began, accelerators have sprung up everywhere to help transform startups from a skein of code into a bona-fide company.

It’s useful to focus on adding another zero to whatever you define as your success metric - money, status, impact on the world, or whatever. I am willing to take as much time as needed between projects to find my next thing. But I always want it to be a project that, if successful, will make the rest of my career look like a footnote.

Compilers

Learning about compilers has been fun. We started with a stack-based virtual machine that executes bytecode, just completed the scanner (converts source code into tokens), and now getting into the compiler (converts tokens into bytecode).

Right now things still feel a little removed from the Fitzgerald vs Egorov post. The optimization that stood out to me was inlining, where the compilation process replaces a function call with the body of the function itself (since you don't have functions in machine code). Egorov achieved this in JavaScript by stringifying a function to get its source text; Fitzgerald noted in Rust this simply involves annotating the function with #[inline].

Stack-based VMs are simple yet can elegantly do a lot! Register-based VMs, in contrast, are more complex but can achieve better performance. I amuse myself thinking about the time I reviewed my notes on MIPS (register) thinking it could help me understand WebAssembly (stack). Bob Nystrom's all-time favourite CS paper is actually on this topic, on Lua moving from a stack-based to a register-based instruction set.

Professional services

Paul Graham's essay described how the 1980s saw a shift from large corporations being the most desirable employers, to professional services. I think of this topic fondly. My very first job was in finance, which was the default of sorts for college graduates then.

The progression in most professional services roles has junior employees executing what the client asks for, and senior employees managing client relationships. I liked the execution part, and always wondered, "What if I'm not into golf?" (cue Mad Men episode). It's curious how for some cases this model is flipped - Google ICs goes all the way up to Senior Fellow i.e. Level 11.

Content: Engineering management

We've covered data science, product management, design, dev ops and data engineering so far. I've saved this one for last - engineering management. I love Julia Evans' zine on the topic, as well as this excellent post by Raylene Yung.

https://firstround.com/review/the-engineers-guide-to-career-growth-advice-from-my-time-at-stripe-and-facebook

I wouldn't say I have a burning desire to be a manager. That said, I do feel (1) there are lots of soft skills that get honed when you're responsible for your team's success, and (2) having that experience helps you empathize with your own manager. Plus think of all of the books that now become more interesting...

Halftime

It's the end of my half batch at RC. I had extended my stay from 6 weeks to 12, so today marks the halfway point.

We had an end-of-batch ceremony, but true to RC's philosophy of Never Graduate, it merely marks the start of a new phase. I was touched by the kind words to those making the transition. The nicest words have to go to the RC faculty; I'm amazed at how well they've been keeping the program going despite the lockdown. I wished I could give all of them a hug, or given the new world we now live in, an elbow bump.

I started RC planning to develop my front end skills. I thought this would involve JavaScript and frameworks; it appears I've gone a bit of a detour. That said I'm still on the path of thinking of the browser as the platform, as initially intended.

• Week 1 - Getting started
• Week 2 - Open source
• Week 3 - Rust
• Week 4 - WebAssembly
• Week 5 - Microbenchmarks
• Week 6 - Compilers

I had the chance to answer questions from the incoming batch this week, and excited to be vicariously infused with a healthy dose of enthusiasm next week! RC faculty are fantastic at getting the most interesting and intellectually curious people to join. I mentioned to the incoming batch how I'm a little jealous at my batchmates with interests in domains with well-defined boundaries, it appears they're better able to concentrate their collective efforts and get a lot done that way.

The varied interests on my part continue irrespective. In the morning, I polished up a bit more code from my Python implementation of lox.

I'm not sure what got me started looking at algebraic data types (ADTs), but I spent some time between events translating a Rust example to Python. The PEP 484 change on forward references seems to have made all the difference here. I haven't yet been able to fully wrap my head around "Why bother?", but did discover a thoughtful post by Mark Seemann. Instead of thinking about these more sophisticated types as 'stronger' tests, the better framing is to consider how ADTs make illegal states unrepresentable.

I discovered the RC publication Code Words along the way, with a post on ADTs and one on compilers. I'm reminded of Legal Affairs, an excellent general interest magazine about the law by Yale Law School students. Like Code Words, the publication sadly only lasted a few years.

Re: frameworks, I recently found out Vue has surpassed React in the number of Github stars. I mentioned this amusingly at the Rust learning group, and Vinayak responded by sharing this cool documentary. Since I do use the blog as an accountability tool, my plan is to try out Vue in addition to reading this Evan You interview in Increment.

On writing

To become a better developer, there's no way around writing lots of code. Making the effort to write daily has definitely made me feel a lot more comfortable about sharing what I've learned. As per a previous post, the problem I feared was that people might care too much when in reality it's the opposite.

From now on I'll shift to a weekly cadence. I'll try to keep the length reasonable. *wink*

Content: E lucevan le stelle

I also use the blog to embarrass my younger self. I previously claimed a love for opera, thinking I'll emerge as a more cultured version of myself. It's not clear that has happened, but I did come out of this phase with a few favorites. Here's the aria from Tosca, performed by Luciano Pavarotti.

Content: The rise of human-computer cooperation

I know, it's a TED talk (and not Brené Brown either). When I first picked up programming, I was fascinated by the idea how humans and computers excel at different skills. The best teams are usually not the smartest humans or fastest computers working individually, but composite teams that can assess who's better where and have a 'streamlined' human-computer interface. This idea was touched on briefly in the featured article of a previous post, and it's put across really well by Shyam Sankar.

Content: Change is the only constant

When I was making the career transition to software, I constantly wondered if I was making the right move. A part of me wanted to return to professional services; it felt familiar not just to myself but also to my family and friends. It felt safe.

I feel extremely fortunate to have made the shift, not for any sector specifically, but for taking the step towards lifelong learning. If I could do it once, perhaps I can do it again. Not an unlikely proposition as the world changes ever more rapidly.

This FT Weekend article by John Lanchester captures this sentiment perfectly (enclosed below).

The moral of the story is that even if you think you understand the impact economic forces can have, they can still strike dangerously close to home. It helps to have a compass, and mine is based on two principles, both of them learnt from my banker father: anxiety is freedom, and the way you are living will have been your life.

Content: Designing Data-Intensive Applications

This last section of my Friday posts usually feature a book excerpt. Designing Data-Intensive Applications by Martin Kleppmann is not one that I've completed, or at least, not cover-to-cover (conveniently the reading group starts next week). The excerpt I'd like to feature is its excellent dedication.

Technology is a powerful force in our society. Data, software, and communication can be used for bad: to entrench unfair power structures, to undermine human rights, and to protect vested interests. But they can also be used for good: to make underrepresented people’s voices heard, to create opportunities for everyone, and to avert disasters. This book is dedicated to everyone working toward the good.