Experiments in managing perfectionism

I remember focusing on fintech startups in the previous job search. Personally I felt it's important that the business problems the company is solving are the ones that I care about. This interview cycle, I realized that I also care about data tools having had first-hand experience of the pain points (though perhaps I employed artistic license in the retelling).

The dominant theme this interview cycle, however, is AI.

AI startups getting wiped out

Last year, the thought of joining an AI startup scared me. Imagine joining a startup that wraps the OpenAI API to do pdf extraction. Further down the road OpenAI enables pdf uploads and the startup gets wiped out.

This year, a startup I interviewed with does exactly that!

It turns out there's a spectrum. At one end, you have generic legal documents that have no repeatable visual structure, where the meaning of each word is enriched by other words surrounding it. This is where LLMs shine. At the other end, you have say a Bank of America statement with the logo on the top right and each row being a date - transaction description - amount triplet; existing visual layout-based techniques do a good job here extracting the contents based on a pre-defined schema.

Thus the value-add is not just wrapping OpenAI's API, it's knowing when to use existing techniques vs running it through an LLM. Like all tools there are trade-offs, in this case extraction quality, latency and cost. In fact, LLMs are useful for converting the 'long tail' of bank statements into schemas, introducing a more efficient workflow to run existing techniques!

AI startups interviewing differently

I had take-home exercises as part of the interview process. The first two had mixed results - I moved on to the next stage for one, got passed on the other. I had used ChatGPT sparingly in my submissions, and I wondered, should I have used it more extensively?

Things got more interesting when I received the next two take-homes. The first was for an AI team so the use of AI tools is implied. For the second it was explicitly encouraged! This completely threw me off. Without ChatGPT I had a rough sense where the 'bar' was, with ChatGPT isn't the sky the limit? Surely I can make my submissions arbitrarily sophisticated.

I panicked. I meditated on the issue. I felt better after letting my thoughts settle.

I realized the more complicated your submission is, the larger the surface area of questions you'll get asked on. While you can make your submission go much further than your comfort zone, you're taking on much more risk in being less able to answer questions that might come later!

AI startups having an unfair advantage

The final thought I had was around startups vs incumbents. The refrain is startups are able to beat incumbents by being more nimble, and by taking advantage of new technology (before you say Clay Christensen, I present to you Jill Lepore). Think Uber disrupting taxis since everyone now has location-enabled smartphones.

This time the new paradigm is AI. Not only startups starting today can more easily build their product on top of AI, they can additionally curate a team with ChatGPT skills to help each other level up and streamline internal business processes.

Surely startups starting today have a huge edge, being at the crest of this new wave?

AI is a complicated tool. It's powerful, sure, but lots of startups (some supported by eye-watering amounts of fundraising) are still trying to figure it out. It goes without saying that recent startups have an advantage, but it's tempting to take this too far.

Let's say a startup has been around for a few years, has product-market fit and proprietary data. This acts as a moat from new AI startups! It's true there's a correlation between company age and rigidity of business processes, but what's more important is the company having self-awareness in realizing the need to adopt new tools and workflows.

In other words, there's a 'sweet spot' - you want to be early enough that you're able to integrate AI well, but also have maturity in knowing your customers and their problems well. This way you can focus on the challenge of learning new tools and workflows, instead of having to do that as well as learning about your customers.

I had been behind on writing blog posts, and I took notes to write out in full when I find the time. I now declare ‘writing bankruptcy’, but I’d still like to share my notes.

A theme I’ve been coming across recently is “software is eating the world, but now AI is eating software”. It reminded me of this quote.

Technological advance is an inherently iterative process. One does not simply take sand from the beach and produce a Dataprobe. We use crude tools to fashion better tools, and then our better tools to fashion more precise tools, and so on. Each minor refinement is a step in the process, and all of the steps must be taken.

I spoke to a few early-stage startup founders, and I wondered how they thought about building a team around them. Cue Hank Paulson.

There is no perfect leader. Everyone is flawed and their strengths are usually the opposite of a weakness. The key ingredient I found for any of these CEOs that was just essential to success is the team they put around them. I called it "the right people in the right seats". You needed to play to your strength but you need to put people around you who you listen to, who can compensate for your weaknesses. If you didn't these big jobs always uncover your weaknesses.

I had been at law school in the early 2010s, and one summer I met up with a lawyer friend in London. I shared with her my disappointment in not getting an internship in corporate law.

Instead of offering words of comfort, she said something along the lines of “you have a tendency to speak your mind but as lawyers we often need to err on the side of being diplomatic; I’m not sure you missing out on internships is a necessarily a bad thing.”

Now I can tell her, she’s right. Working with software is not without its quirks, but I’ve yet to see someone describe their experience in the vein of the following excerpt.

There’s a saying out here. Every man has three hearts. One in his mouth for the world to know. Another in his chest, just for his friends. And a secret heart buried deep where no one can find it.

I grew up in Kuala Lumpur. I’ve since had the chance to live in London, New York, Paris and Berlin. I’ve been in San Francisco since 2015.

While SF is the smallest city in the list, being here I’ve ironically discovered the wonder of being away from the city. I spent time in Mendocino this week. It’s amazing to get away from the day-to-day. It’s like moving out of the trenches to see the battlefield.

We transition from school to college to work, often learning skills throughout (at least, ideally). What I’ve also since discovered is the importance of learning to unlearn.

We develop mental models to simplify how things work in reality, except often we retain those models even when they don’t work so well any more. Either the reason we adopted them, or our priorities, or the underlying world itself have changed; yet the models remain. Sometimes they become our sacred cows.

What’s helpful to set the ground in revisiting those mental models is creating more space. How to create space? Different things work for different people. Meditation. Journaling. Family. Friends. Nature.

I’m grateful to have made this latest discovery. Especially at times when the world feels like it’s at an inflection point, it’s reassuring to be surrounded by timelessness.

A question I had during an interview was “what does a perfect day look like to you?”.

I responded talking about working with the perfect team, a variation on a post that I had put up on LinkedIn.

The best team I was a part of made meetings fun. I loved hearing what others are working on and which bits are blockers for me, and sharing what I’m working on to learn which bits are blockers for them. Without prompting from managers, team members would regularly share scripts, documentation and resources to help each other level up and move faster.

After the interview I thought about the question a bit more, and followed up by e-mail with a note of thanks and a quick postscript - coming up with a elegant solution after meditating on the problem for a few days (a la Hammock-Driven Development).

Having slept on it, I lean back towards the team. This is one of my favorite excerpts from Sam Altman’s advice when he turned 30.

Go out of your way to be around smart, interesting, ambitious people. Work for them and hire them (in fact, one of the most satisfying parts of work is forging deep relationships with really good people). Try to spend time with people who are either among the best in the world at what they do or extremely promising but totally unknown. It really is true that you become an average of the people you spend the most time with.

It’s tempting to think about my own path on the back of some design, but this desire to be around smart, interesting, ambitious people has been a recurring theme.

I studied math in college partly because I thought I’d get into a better school vs a more practical subject. I went to law school because my friends who did law seem to ‘know how the system works’, making me realize that knowing the grey areas in life is an effective complement to being good with numbers.

Today I work with software. In terms of unique aspects, there is a level of objectivity (it’s pretty clear when your code runs fast) as well as value capture from network effects. The aspect that I like the most can perhaps be best explained through a story.

Cash App came out of a hack week at Square, yet the buyer-facing Cash App now processes more volume than the seller-facing side of the company. There’s no amount of of top-down effort that gets people excited to build something that eventually overtakes the original idea. That happens through getting smart, interesting, ambitious people in the same room and having them believe the have the agency to make magic happen.

When I interviewing last year, I was keen to move back to fintech. I wanted better context on problems in a specific domain, to match that against new capabilities technology now ‘unlocks’. The example that came to mind was Square helping vendors at farmers market to accept credit card payments, enabled by more accurate ML-based risk models.

Now I realize that domain doesn’t have to be fintech. It’s true, most of the exposure I’ve had has been in payments and lending. That being said, what’s important is being at the interface between the business domain and technology. The same principle, of understanding domain-specific pain points and how to alleviate that with more sophisticated tools, still applies.

Perhaps Justine Musk says this best.

Choose one thing and become a master of it. Choose a second thing and become a master of that. When you become a master of two worlds (say, engineering and business), you can bring them together in a way that will a) introduce hot ideas to each other, so they can have idea sex and make idea babies that no one has seen before and b) create a competitive advantage because you can move between worlds, speak both languages, connect the tribes, mash the elements to spark fresh creative insight until you wake up with the epiphany that changes your life.

I shared her post on the first blog entry of my first time at Recurse Center. It’s a nice way to come full circle.

In a previous role, we had audits to help us to refactor our data pipelines with confidence. While it may make sense to run audits in staging, staging data can very different vs production. This means that the changes get merged based on guardrails that allow staging to run successfully, but end up breaking in production.

To get around this, you run tests in production. Never test in production you say? What’s the use of testing if it doesn’t stop you from breaking things. The following quote is from Erik Bernhardsson.

Lets let the right workflows emerge from what makes teams the most productive, and lets let data workflows stand on their own feet.

The book of best practices for data (perhaps ML too) is still being written. Plus there’s probably a startup idea there somewhere.

Spending time at Recurse Center between roles is a great opportunity to think a bit deeper about what I’d like to do next. This is usually through chat with fellow RCs in batch, and it’s always fun to hear how others are going about their search.

I didn't do RC this time around, so that ‘refinement’ process has to be done on the go with real interviews. An interesting question I was asked was “what’s special about having machine learning in your system?”.

I thought about my time at Square and what came to mind was this idea of moving from a deterministic system to a probabilistic one. Suppose a seller owes Square $100, and Square initiates a $100 ACH debit to recover those funds. It takes 3 business days before you know if the debit succeeds. Now the next day the seller needs to be credited $50. What do you do?

In the optimistic case, you assume that the debit will succeed so you let the $50 go through. In the pessimistic case, you assume that the debit will fail so you hold the $50 for two business days. Which one is the right call?

The flow with ML is to train a model based on historical data. If the seller looks closer to one where the debit would succeed, then let the $50 go through. If the seller looks closer to one where the debit would fail, then hold the $50. Hence we switch from making a binary decision, to having a threshold that determines how we would act.

Of course, there is also the question of whether our historical data is sufficient for model training, and if it is, the return on investment on extra complexity is worth it.

Andrej Karpathy takes this a step further in his post Software 2.0, jokingly describing it as follows.

Gradient descent can write better code that you. I’m sorry.

This was written in 2017, and perhaps even more pertinent now.

For my first data science project, I trained a machine learning model using Lending Club data. The hypothesis is I can use that data to ‘cherry pick’ loans in a way that outperforms the average return. More specifically, all loans within a certain grade pay the same interest rate and if I can stack rank to select the top 5% say, then I beat the average.

What I didn’t quite expect is to later work for a startup that did exactly this.

I digress. The point I’m trying to get to relates to the loan data itself. The model I built mainly used the numerical features like the borrower’s FICO score and income. There was a text blob column which is what the borrower wrote as their intention for the loan proceeds. This text blob became simple features like number of characters, number of words, number of sentences.

With text embeddings, this column can now be a vector and the whole model be trained on both the numerical features and the semantic meaning of the text. It’s one use case of many that language models now unlock.

In a post on job searching, Haseeb Qureshi advises candidates to be excited.

Be excited about the company. It’s trite, but it makes a huge difference. Be excited to interview. Be excited to learn about what your interviewer does and the prospect of getting to work with them.

I'm often fascinated how much you learn about companies through the interview process; plus it's often less motivating to do so outside of a job search. This helps me stay excited.

I've been on the job search a few times now, but I still get the emotional swings. I guess it'll never go away, so the only thing you can do is make the experience less painful.

I came across this quote by Jack Kornfield on Tim Ferriss' podcast.

Let suffering teach you compassion because in Tibet, in some of the Tibetan teachings, they actually pray for suffering. They say, "May I be granted enough suffering so that the great heart of compassion will open in me."

It's a helpful reframing, to think of the experience as another rep at the emotional gym.