thinking about time horizons
overfitting error, strategy, and the case for hope
I recently read the book The Psychology of Money by Morgan Housel. There were many things that stood out to me about the book (it’s probably one of the best books I’ve ever read and I think everyone should read the book if given the chance), but this quote stood out in particular:
Bubbles form when the momentum of short-term returns attracts enough money that the makeup of investors shifts from mostly long-term to mostly short-term.
Incredible! The idea of the formation of bubbles being tied to altered investor time horizons is so simple and so powerful, and I think it explains the phenomenon of bubbles better than I’ve ever seen it articulated.
But what stood out to me about this quote isn’t necessarily the wisdom about bubbles; it’s the concept of different people operating under different time horizons. It’s something I’ve never thought about before, and I haven’t stopped thinking about it since.
In general, I think a lot of the issues we’re facing today, both on a societal as well as a personal level, have to do with our inability to see past a short-term time horizon. For example, our current struggles to craft any meaningful legislation might have to do with our legislators (and voters) inability to see past the next news cycle (let alone the next election cycle), and our own personal struggles to find meaning and fulfillment in our careers might have to do with our inability to see past our next promotion or paycheck at the expense of long-term skill and relationship development.
But for me, the most interesting applications of the concept of time horizons are these:
Overfitting error and strategy - I think our understanding of the world is worse than it’s ever been, and I think it largely has to do with news and short-term thinking dominating our worldviews. This is making us awful forecasters and strategists.
Pessimism and the case for hope - Within a short-term frame of thinking, it is much harder to see progress and much more difficult to find reasons to believe the world is moving in a positive direction. I think this explains a great deal of the overall pessimism that’s prevalent throughout society.
I’ll spend the rest of this article expanding on these above two ideas, and hopefully I’ll leave you with the feeling that maybe things aren’t as bad as they seem.
overfitting error and strategy
I’ll start with explaining what overfitting error is, and then will explain how it relates to time horizons and strategy. This will start out as pretty abstract, but I promise it’ll all be much more practical by the end. The initial analytical aside isn’t too important in the overall point I’m trying to make, but it’ll maybe help give a little context for where I’m coming from. Anyway, let’s get to it.
I’ll start with a quick discussion on how predictive models are built. Let’s say I want to build a model that predicts how much a house will sell for based on its size and age. Here’s how that process would go:
Find a data set that contains a list of houses and their respective sizes and ages.
Split the data set into training and testing data sets.
Feed the training data set into some computer algorithm. The computer will take the data and spit out a formula that in theory will estimate the selling price of the house really well based on its size and age based on all the data contained within the training data set.
Test the computer formula on the testing data set. Basically, see how well the formula predictions of what the price of a house should be based on its size and age compare with the actual listed prices in the testing data set.
The concept of overfitting error comes in when the computer formula performs really well at predicting the prices in the training data set (a.k.a. the data set used to build the model), but really poorly at predicting the prices in the testing data set. This is due to generalizability; when data sets are tailored too closely to the training data set, they end up functioning well only in that limited environment but are unable to have any kind of generalizable utility outside of that specific environment (like, for instance, within the testing data set).
The way overfitting error relates to time horizons is that I think it’s the mechanism that explains why we’re so bad at forecasting anything right now; our minds are filled with short-term noise (news, media, etc.), and our worldview is overfitted to this noise that we’re exposed to on a daily basis. The reason I refer to news and media as noise is because as seductive as it is and as productive as it may feel, it has a built-in commercial incentive to make you feel like the world is fundamentally changing every minute; that the rules are being rewritten every day. This makes it biased. Think about it - would you rather listen to a podcast that promises to tell you something you already know, or a podcast that promises to drop a hot new piece of insight that you’ve never heard before?
And while the noise makes us feel like we’re very educated about the world and makes us feel that we have more than enough information to make really good predictions about what will happen in the future, the reality is that the inherently short-term nature of this information makes it next to useless when trying to imagine what the future is going to look like; it means any kind of strategy that’s based on this information and this worldview is probably going to fail.
To prove my point, look at the failure rates of entrepreneurs. There are so, so many companies that are founded by people that are lauded as geniuses that have raised millions and millions of dollars in funding that have failed spectacularly. In fact, that’s essentially the default outcome in entrepreneurship, with the exceptions being a few companies whose glorious successes can probably be more attributed to luck than any kind of specialized insight about the future. All these failures are companies that have done a tremendous job in convincing really smart investors that they know what they’re doing and that there’s a massive cost if they don’t immediately capture this short-term opportunity. The failure rates don’t make sense if you operate under the belief that a greater knowledge base of short-term information that stems from news and media makes you better at predicting the future; it should mean that these people shouldn’t be failing at such ridiculously high rates. The point isn’t that VC and entrepreneurs are stupid; the point is that the short-term nature of the thinking within these enterprises makes it rife for failure.
So how do we become better strategists?
I think it starts from the realization that the world hasn’t really changed. As tempting as it is to fall into the trap of thinking that the world keeps fundamentally changing, it really hasn’t. And that’s because people haven’t fundamentally changed.
The problems we’re experiencing - problems with the environment, problems with polarization, problems with health, problems with inequality, etc. - have less to do with any specific trend that has emerged recently and more to do with our human tendency towards greed and hubris, our struggle throughout our existence to find a sense of purpose, and our difficulty grappling with our own mortality. These general human tendencies have always manifested themselves throughout history in some way or form, and everything we’re dealing with now is just the way they’re manifesting themselves now. A discussion of how to resolve the environmental crisis is incomplete without mentioning greed, a discussion about how to resolve the health crisis is incomplete without mentioning purpose, and a discussion about how to resolve inequality is incomplete without mentioning hubris.
Interestingly enough, you won’t find the pathways to transcending these human tendencies within the news and media that infiltrates everything we consume these days; it’s too busy trying to convince you that the way forward is to buy this new product, try this new trend, or invest in this new company. You won’t even find them in the new genre of ‘behavioral science’ which promises to uncover deeper, never before known, insights and knowledge into how people work. The ideas from behavioral science are interesting and compelling (there’s lots of policies that are based on the Nudge theories by Richard Thayler and Cass Sunstein), except that the ideas are based upon bad science:
The replication crisis has shaken the behavioral sciences, famous studies have been retracted due to data fraud, and, more generally, the very effectiveness of nudging has been put into question.
To truly learn about how humans work, I think we’ve got to look way further back. The pathways towards understanding ourselves and transcending our tendencies are covered nowhere more deeply than in the spiritual texts and mythologies that form the basis of most religions, and whether or not you’re religious, there’s no denying that those texts and mythologies contain a greater reservoir of human knowledge than anything else we have; there’s a reason they’ve persisted throughout society for centuries and centuries.
Regardless of where you turn to to learn about human nature, I have a pretty firm level of belief that a better strategy for understanding how the world will progress and develop is first by learning deeply about humans and how we work and then contextualizing that knowledge to the modern issues that we’re currently facing instead of starting with the modern issues first and then trying to hypothesize how humans will react. This, in essence, expands our time horizon and broadens our perspective in a way that will help us escape the noise we’re constantly surrounded by. And that will make us better strategists, both in our personal as well as our professional lives.
I’ll leave the creation and consumption of news and media to all the podcasters and content creators out there; I’m much more excited to spend my time diving into Norse Mythology and Hinduism and finding interesting ways in which the wisdom present in those stories manifests in my everyday life.
pessimism and the case for hope
There’s a couple more interesting quotes from The Psychology of Money about the strange allure of pessimism:
For reasons I have never understood, people like to hear that the world is going to hell.
John Stuart Mill wrote in the 1840s: “I have observed that not the man who hopes when others despair, but the man who despairs when others hope, is admired by a large class of persons as a sage.”
It’s so true - the optimists that have the AUDACITY to believe that we’ll be okay are labelled as naive idiots, while the pessimists who write about how we’re terminally fucked and all going to die in the next 20 years repeatedly go viral.
I think this is a mistake; I think this once again has to do with thinking dominated by short-term time horizons. In the same way that a tree that has taken 100 years to grow can be chopped down by an axe in an instant, a world that has biased us towards short term thinking will naturally cause pessimistic notions to arise, because in the short term, only negative action can really be visible. Communities can’t be rebuilt in a week, but that’s all the time it takes for gun violence, natural disasters, political scandals, and more to dominate the news cycles and re-instill the belief that the world is going off the rails in a way that we can’t fix or control. We miss the fact that there is some really cool building going on because it doesn’t fit in the time horizon that we’ve been trained to think in.
I think if we expand the time horizons that we think in, there’s a lot of reasons to be really hopeful! Here are a few examples of some things that are slowly being built that make me really, really excited for the future:
Harmonization of eastern and western approaches to health - I’ve become a strong believer that while western approaches towards health and medicine have yielded dramatic progress in things like vaccine development and treatment of ambulatory conditions like broken bones, they remain painfully ignorant about how our body works as a complete system. That is an area that eastern approaches towards health have a far greater understanding on. For example, western approaches are clueless on how to treat chronic pain, while eastern approaches towards health recognize that chronic pain is typically a manifestation of unhealed mental trauma, stress, and lack of purpose. But that’s slowly changing! Books like The Mindbody Prescription by an NYU physician make me really hopeful that we’re on the right track in terms of learning how to best incorporate approaches from both the east and the west to best serve societal health. THIS IS SO EXCITING!!
Decoupling of local economies and the shift towards sustainable globalization - Globalization has been cool and all, but it’s terrifying just how interconnected everything is; a banking failure in Italy risks affecting farmers in Idaho. It’s crazy, and this networked and hyper-interdependent world we live in is not sustainable at all; it is putting all of our wellbeing at risk. There’s also good reason to believe it’s hurting our nutrition as well — the ‘specialization’ of crops within individual countries is absolutely killing the diversity and nutrient density that used to be found in local ecosystems. But we’re slowly shifting away! There’s a lot of fascinating research and work being done about how to build self-sustaining local economies that aren’t impacted by failures elsewhere in the world. I want to be a part of this work some day and I can’t wait to see how it all progresses. We’re going to learn how to keep the best aspects of globalization without uprooting local communities!!
Widespread acedia paving the way for spiritual rebirth - It feels as if people in their 20s and 30s are hitting their midlife crises decades early these days, and I think it’s because we’re quickly coming face to face with the idea that material possessions and things can’t bring meaning into our lives. It’s scary seeing the mental health issues going around, but I think the vacuum of meaning that’s created by this crisis of materialism is actually paving the way for a really dramatic spiritual rebirth. While it’s hard watching everyone struggle, I’m really hopeful that this space and vacuum of meaning will be filled with a renewed sense of purpose and intent that’s based on virtue-based living instead of the pursuit of prestige, status, and wealth. And that journey is going to be so, so gratifying to be a part of.
And there are so many things that I didn’t even list off. There’s a lot of exciting work being done! We are slowly but surely building towards a better, more sustainable future, and in the face of a lot of reasons to be pessimistic, I will always stubbornly choose to hope; I have a tremendous level of belief that the future is going to be better than we currently have the capacity to imagine.
There’s a lot of interesting ways to expand on the discussion about time horizons, but at a high level, I hope I’ve highlighted the dangers of holding onto a short-term time horizon-based view of the world, and the wonders and exciting things that you open yourself to if you just expand your perspective a little bit.
I for one couldn’t be more excited to learn from the wisdom contained in ancient texts and to actively support the incredible projects that are underway in the world.
other things that have been on my mind
It’s been a second since I’ve written, and there have been a few other things that have come across my mind in the last few weeks:
Indian patriotism
I went to India a few weeks ago for the first time in over 5 years, which is crazy considering I lived there for 6 years after I was born and pretty much my entire family is from there. It’s so hard to make any kind of real conclusions about the lifestyle in India based on my experience (I was only there for 12 days so it’s ridiculous to make any statements about India as a whole based on that dramatically limited sample size), but the one thing that stood out to me the most was the immense sense of patriotism that my family in India had.
Yes there’s a lot that’s wrong with India, and yes India is behind in a lot of ways, but everyone I met in India was so excited about the future and so determined to be a part of India’s growth and to build it up into a global superpower. And the reason I noticed that sense of patriotism and national pride is because I realized I hadn’t felt that in ages back in the States. We’ve lost it all here; our national narrative is in disarray, and I haven’t met anyone that’s felt a true sense of pride in our country in so long. It’s so tragic considering the inspirational and romantic origins of our country; a country of freedom fighters, championing democracy and equality for all. That feeling is all lost now, and we’re more interested in tearing down our past and our history than in continuing to build our country up.
Obviously the issue is a lot more nuanced than the way I described, but I dunno, it makes me sad. I miss the feeling of patriotism. I still think we as a country have a lot to be proud of, and hopefully we can find that sense of shared community and shared pride once again.
self-awareness
I’ve been really struggling to reconcile my generally good instinct in understanding people and situations with my inability to fully understand myself and be fully honest with my intentions. Like, how can I be so good (in theory) at reading situations and understanding what’s going on, but still struggle to place my own actions in a larger context? It’s maddening.
I like to think of myself as someone who’s led fully by instinct, but it gets really problematic when I start to question my own impulses. For example, if I feel an impulse to watch TV, is it genuine instinct that’s leading me to give myself some time for rest and recreation, or is it just my desire for a quick dopamine hit? Obviously this is a really trivial example, but you get the idea. It gets even more confusing because I always start presenting an iron-clad defense of my impulses to myself, and I obviously can’t argue with myself so I end up always following my impulses for better or for worse, convincing myself that it’s instinct and that everything is going to work out as it should. And that’s not to say that I regret following my impulses or that they’re coming from a bad place; I think I’m just struggling to figure out what’s instinct and what’s self-deceit.
Basically, I think I’m pretty lost inside my own head, and I want to learn how to better understand where my impulses are coming from.
I think the path forward is to learn more about the Hindu concept of maya as it relates to self-awareness and liberating yourself from self-deceit. Here’s a high-level explanation of it:
Under maya’s influence, the atman, (the soul) mistakenly identifies with the body. He accepts such thoughts as “I am white and I am a man,” or “This is my house, my country, and my religion.” Thus the illusioned soul identifies with the temporary body and everything connected to it, such as race, gender, family, nation, bank balance, and sectarian religion. Under this sense of false-ego (false-identity) the soul aspires to control and enjoy matter. However, in so doing he continuously serves lust, greed, and anger. In frustration he often redoubles his efforts and, compounding mistake upon mistake, only falls deeper into illusion.
In ignorance (tamas), he is fully convinced that right is wrong and wrong is right. In passion he is unsure, hesitant, sometimes enjoying and at others times repenting. Only in goodness does the soul begin to develop wisdom – to see things in the real light. Thus enlightenment means moving away from tamas towards sattva. By so doing, the soul gradually escapes the clutches of maya and moves towards liberation.
I think this neatly distinguishes instinct (sattva) from self-deceit (tamas), a duality I’m just starting to explore and discover within myself. I want to escape the clutches of maya and escape the illusions my mind lays out for me on a daily basis, and I’m determined to learn how.
One thing I have started to realize, though, is that I think I tend to get a little too cerebral and analytical when it comes to things like this; I think I take myself way too seriously sometimes. I have to constantly remind myself to relax, that the most rich experiences come when I adopt a more playful and experimental attitude towards my life and when I get out of my own head and just breathe. That’s when I feel most like myself and feel most alive.
noise
As a result of my crazy travel slate the last few weeks, I missed so many email newsletters that I absolutely loved reading on a daily basis during the months before. I thought I would feel a massive loss as a result, but I was shocked to realize that I didn’t. I didn’t notice any kind of dramatic impact on the quality of my experience; all I felt was a desperate desire for some time to myself to process everything and to continue building myself up, but I didn’t find myself missing the newsletters at all.
Perhaps what this means is that there was a lot more noise contained in them than I had initially realized; perhaps this is yet another signal for me to continue resisting the content cycle that buzzes nonstop every day in the favor of deeper, richer literature.
Either way, I’m excited to let go of the need to read every single thing that comes into my inbox and create some more space to invest in myself and my daily practices.
quotes
I know I didn’t read that much recently, but there were still some fire quotes I came across:
The remarkable thing to observe, in children’s work…, is that the child gives the impression of having done it with his whole being. They surrender themselves completely to what is in hand. Whereas even the biggest artist has to wage a constant fight against distraction. He is conscious not only of the future opinions of the critics, the price it will fetch (or not fetch!), the value of his tubes, the nicety of his choice of color or line, but also the temperature of the room, the stains on the floor, the bath he forgot to take, and so on.
I think we can all learn from children in how to put the totality of our being into our work. That is where we discover true, genuine expression, and that is where true art comes from.
Strangling your passions in exchange for an elite life is like being on the Titanic after the iceberg, water up to your chin, with everybody telling you that you’re so lucky to be on the greatest steamship of all time. And the Titanic is indeed so huge and wonderful that you can’t help but agree, but you’re also feeling a bit cold and wet at the moment, and you’re not sure why.
Think this describes the way a lot of people feel these days. That obviously begs the question - do people even know what their passions are? Do they know what they’re giving up in making this Faustian bargain? I’d wager that most people don’t.
The great obstacle to discovering the shape of the earth, the continents, and the oceans was not ignorance, but the illusion of knowledge.
…
To a large extent, your skill as a researcher comes down to how well you understand how dumb you are, which is always “very”. Once you realize how stupid you are, you can start to make progress. A different writer might say “humility” here rather than stupidity. But calling this virtue humility might make you feel smug and self-satisfied, which is not the right feeling at all. Instead, you should feel dumb. The virtue of stupidity is all about feeling like a tiny mote in a vast universe that you don’t understand even a little bit, and calling it humility doesn’t strike that note.
“The illusion of knowledge”. What a wonderful phrase. I love the principle of embracing stupidity. There’s a near certainty that everything we think we know about the world is going to be proven false at some point in the future. Functionally all knowledge that we have is illusory and only acts as a barrier towards any kind of discovery.
Teaching someone painting techniques without teaching them composition will lead to lifeless paintings. Giving business advice to someone who lacks civic duty will lead to parasitic companies. Teaching generals strategy without teaching them honor gets you warlords. So teaching someone the methods of science without teaching them the virtues will lead to dull, pointless projects. Virtue is the key to happy, creative, important, meaningful research.
Virtues first, then skills. We’ve got it backwards right now; we’re in such a rush to empower people and give them the tools to build incredible things that we’ve forgotten that the most important question isn’t how to build, but what to build.
Whitman was in many ways the embodiment of the spirit Emerson so fiercely celebrated against the tide of his time: a spirit animated by the central doctrine “trust thyself,” anchored in resolute resistance to the tyranny of opinion, and rooted in the belief that had gotten Emerson banned from Harvard’s campus for thirty years when he was Whitman’s age — the belief that divinity is to be found not in some outside deity, but in the human soul itself, in its fidelity to itself as a fractal of nature, a particle of the perfect totality of the universe, which Margaret Fuller — Emerson’s greatest influence — called “the All.”
:) so beautiful.
Will catch you next time with a (probably?) less lengthy post.
🤙