Without options, momentum becomes an unplayable game. It's these options that provide the playground for finding the optimum.

Random choices are the seeds from which options grow. This approach embraces the untested and potentially risky, until a solution is proven. Choices offer the freedom to enhance, like adjusting prices or interest rates to achieve welfare.

illustration

Copyright© Schmied Enterprises LLC, 2024.

However, there are scenarios where gambling is off the table. In emergency situations or when dealing with proven science, repeated stress tests are unnecessary. Randomized controlled trials are reserved for cases that remain unknown.

But let's not forget the caveats. Randomized controlled trials and A/B testing pit a hypothesis against its opposite or a neutral option. This approach is costly and typically confined to scientific experiments. In our data-rich economy and industry, finding directions is straightforward. Welcome to the era of data-driven decision making.

Randomized controlled trials come with a hefty price tag. They were easily conducted in the realm of chemistry. When it comes to individuals, it's a different story. Legal and ethical standards come into play. You can't simply harass someone under the guise of random selection. Consent and permissions are required. It's far more cost-effective to analyze existing data, particularly if the trial is invasive.

Randomized trials can be skewed. If you can't muster a few thousand participants, you're likely researching a non-issue. Political polls can easily find their audience, and randomness aids in prioritizing issues. Trials can trigger events that wouldn't otherwise occur. Transparency, public communication, and consent processes are crucial. Researchers bear the responsibility for their trials.

Credibility is key. The degree of randomness in a random selection matters. The NSA backdoor in 2015 utilized pseudo-random numbers. Such randomness can skew results and decisions. Factory data can be manipulated. It's vital that data-driven decisions are also guided by knowledge and skill to eliminate false positives. The answer isn't in the database, but in a comprehensive investigation that includes the database.

Testing can alter individual psychology. Constant trial harassment can warp the mind, leading to altered behavior in different situations. Repeated trials can be skewed, much like the Rorschach test. Rechecking results can be costly. Trials need to be limited. Are there any adverse or long-term side effects of the trial itself?

Trials themselves are not random. Different trials can distort the results of each other. Opt-in requirements can change the population. Only a few randomized controlled trials can be conducted on the same population within a certain period. This is why it was originally limited to top-tier Ivy League schools.

Documenting the methodology is also crucial. Are minorities overrepresented or underrepresented in the trial? Is the resulting treatment still valid and safe for those minorities?

Is it ethical to divide people? Is the trial truly research or an attempt and excuse to change a situation? Randomized controlled trials can sometimes resemble gambling. A wager is placed on health or happiness. Regulations abound in many places.

The author advocates for limiting such tests and favoring existing data whenever possible. There are cases, like clinical trials for new medicine, where it's a necessity. Caution is advised elsewhere.

Marketing presents a slightly different scenario. Once a product starts selling, momentum is an effective way to leverage existing know-how and data. Many companies fail due to a lack of initial data for accurate research. Gathering such data is costly. This occurs when choosing a marketing partner and needing to spend a fixed amount initially to gauge the system's receptiveness.

Marketing is akin to a chessboard with a few pieces. It's like the blue ocean strategy, finding and building on an uncertain niche market. You need to invest in a few trials to see where the pieces land. You might spend and find nothing. You might find a pawn that doesn't yield a good return. You might find a queen that allows for more spending with higher customer acquisition costs.

Look at today's industry, and you'll see the result of a random experiment. Some paid Ivy League schools to find the location of their pieces. Some companies were fortunate to find a pawn and built on that knowledge to grow and find the next one with less research. Often, these companies are a one-product show, and their second experiment is less successful. Consider the Metaverse.

Many other firms found nothing and failed. Many startups simply didn't have the right data or funds to experiment enough. There are numerous market research firms with data that can assist in scanning the market.

The most successful companies, like Amazon, Apple, Google, or Microsoft, have teams capable of launching any new product in a few months or years. They just need to step in when they see market momentum, like the case of artificial intelligence and language models. They don't gamble; they simply wait and act when the time is right.