The limitations of ‘big data’ and user feedback
Feb 27, 2023
When founders start a company and are in the process of fleshing out an idea, they jump around a lot and follow their intuition. They have an innate sense for which ideas are worth building and expanding upon. There is no user feedback or data to look at to verify their hunches and these ideas tend to hold the soul of the future product. It is a magical feeling. As a company’s team grows, this emotionally in-touch way of building is frequently replaced with all sorts of processes that strip away our ability to channel the magic.
The common consensus is that companies should build an MVP (minimum viable product) that is as slim as possible, while still housing the core features of the product. This allows you to show the product to your customers as soon as possible and get feedback. The company then continuously reorients itself around that feedback by reprioritizing its roadmap, determining which features to build next, etc. This is the main strategy employed by companies to find product market fit, and overall it’s great. It allows you to verify your theories, and gain new insights that get you closer to having a product that people love. I do believe, however, that this mindset of data oriented prioritization has its limits and has been taken too far by most of the tech industry. Data is great for testing something out, but not as great at determining what to do next. If we necessitate that an idea has data backing it in order to build it, we will potentially lose out on a lot of magical ideas. This scenario often goes like this: if someone on a product team has an idea of what they should create, they search through the data, consciously or unconsciously, to find information that confirms their perspective. If the data supports their idea, they present it to the rest of the team to gain approval. If the evidence is lacking, they either abandon the concept or spend more time figuring out ways to obtain the "relevant" data.
This overly conscious approach to building makes it difficult to lead from our ‘gut’, and makes it harder for teams to innovate and do the unexpected. In Julian Jaynes book, The Origin of Consciousness in the Breakdown of the Bicameral Mind, he goes through the alternate modes of thinking that are frequently associated with consciousness - things like problem solving, critical thinking, logic, etc. He then goes on to show how “consciousness” is not necessary in order to perform most of these functions:
“But more complex reasoning without consciousness is continually going on. Our minds work much faster than consciousness can keep up with. We commonly make general assertions based on our past experiences in an automatic way, and only as an afterthought are we sometimes able to retrieve any of the past experiences on which an assertion is based. How often we reach sound conclusions and are quite unable to justify them!
Because reasoning is not conscious. And consider the kind of reasoning that we do about others' feelings and character, or in reasoning out the motives of others from their actions. These are clearly the result of automatic inferences by our nervous systems in which consciousness is not only unnecessary, but, as we have seen in the performance of motor skills, would probably hinder the process.”
Jaynes argues that our subconscious minds are the ones processing most of the information and making sense of our experience. It is our conscious mind, our linguistic layer, that then after the fact attempts to justify / rationalize the signals from the subconscious. In my eyes, this is very similar to the scenario I described above where somebody has an idea for the product and then goes looking for ways to verify that idea so as to convince other people that it is worth investing resources into. The core difference, of course, is that if there is no relevant data supporting the ideas, those ideas are either forgotten or are deprioritized into oblivion. I believe that this trend is what has caused originally innovative startups like instagram to resort to copying others ideas and limiting themselves to the ‘safe and proven ideas’. This phenomenon goes beyond just tech and startups though. It is the same phenomenon that influences people to label a song or movie as “commercial”. It is when the emphasis on originality is replaced with security. The result is a cultural wasteland where everything is derivative.
I believe that we will be able to build better and more innovative products if we allow ourselves to listen more deeply to our intuitions and ideas - regardless of whether or not there is the necessary data at the time to support it. We should attempt to understand the market and the target demographics as well as possible, but we must acknowledge that the data collected in the past is incomplete. Ironically, Google, one of the biggest proponents of "big data," famously gave employees 20% of their time to work on whatever they thought would be most impactful. Many great things came out of this, including Gmail. In addition to listening to our users, we should listen to ourselves and the spark that inspired us to create something new. I’m tired of boring software that all feels the same, so let’s create space in our processes for experimentation and risks, regardless of the numbers that we don’t yet have.