Experimentation Matters

For many businesses, the scariest four words in the world are “Let’s try something new.” New is the unknown, the dark hallway, the monster in the closet, the thing in the shadows with the teeth and the claws. New is also what drives businesses forward, creates new opportunities, and inspires innovation in the workplace.

Every revolutionary idea started as something new — an experiment and a possibility that might it fail. In order for these paradigm shifts to arrive and take their business to the next level, companies must embrace a culture of curiosity, autonomy, and most importantly, experimentation.

Experimentation is the only way businesses can achieve major breakthroughs, or even make incremental changes that will positively affect their bottom line over time. New things must be tried consistently and on a large scale to ensure long-term improvement and growth.

A culture of experimentation means knowing that trying new things is risky, and instead of seeing failure as something to be avoided, embracing test failures as opportunities to learn, improve, and test new and better ideas the next time. It means fostering a willingness to attempt changes, and to accept the results objectively.

Willingness to experiment at scale is crucial for getting the most from tests. The simple fact is that tests will usually fail, but widespread experimentation is actually less risky than occasional testing because users see the changes as less drastic, and full-scale feedback is many times more useful than that from a tiny sample group. While learning from failure is important, by testing broadly, you are much more likely to encounter positive feedback.

Clearly, tests must be administered constantly to see significant value in return.

If, for example, tests have a 10% success rate, when five tests are conducted, you will most likely get zero positive results. However, if ten tests are conducted, at least one should prove successful. That one success with nine failures is much more valuable than five failures, and when you run thousands of tests, you get that many more successes. Remember, while you can learn from failure, you can build upon success. The only way to discover what really works is to be testing constantly.

Of course, tests can’t be created and deployed without administration. This means that companies must establish a centralized experimentation infrastructure where these trials run. This integrated space where ideas are put into action and tested on their own merits can decentralize an organization and create more autonomy for employees because, after all, good ideas can come from anyone.

Curious employees that think freely will innovate independently, which builds continual value for a company. By decentralizing, collaboration across specialities becomes simpler and well-thought out experiments are the result. This horizontal model forces executives to empower employees, not dictate to them, which in turn makes employees happier, more involved in the business, and more likely to contribute. The more independent thinkers you can empower in the workplace, the more unique ideas you can generate for the company as a whole.

Of course, enacting this mentality takes strong leadership to provide the insight, infrastructure, and an environment where employees feel that they can experiment freely. This means giving employees the freedom to fail, a difficult idea for many managers and companies to embrace. But as we’ve seen, failure is a huge aspect of experimentation, and a key aspect of the learning process. Leaders must embrace and allow failure, knowing they are the groundwork for future success.

In addition, leaders must ensure the tests are administered and evaluated objectively. We’re only human and our biases and shortcomings can inadvertently make their way into tests. It is the job of the leaders to guarantee what is being tested is ethical, verifiable, and of course, beneficial to business. In addition, leaders must take the helm and guide testing based on results. Too often people think they know how a test is going to turn out — the perceived result — even if the data doesn’t back it up. The purpose of experimentation is to learn what works and improve using that knowledge. Leaders must make their decisions based on data, not opinions, otherwise, what is the point of experimentation?

While the value of consistent wide scale testing is plain to see, this value is not an end-all-be-all in itself. Testing must be conducted properly to be of any use, but even when you check every box, sometimes testing will come up short when searching for a solution.

The main reason for this is that with wide scale testing you gather results across your platform and learn what works for the average customer. Unfortunately, the “average” customer is not a real person, it is an amalgam of your entire user base. By only attempting to appeal to this imaginary average, you can distance yourself from some of your most valuable customers and make many unhappy. Just as casinos appeal to the public at large yet also have private areas devoted to high rollers, offering what most people want is a good idea, but you might also be missing out.

In addition, testing can fool companies into thinking all short term improvements will become long term improvements. For example, when companies test new interfaces, unless they’re truly awful, a new interface will show an initial heightened engagement with users simply because it’s new. However, this engagement decays over time as the novelty wears off and could eventually drop below the engagement of the original interface. Positive or negative, the simple way to avoid this issue is to let tests run their course, allow the data to stabilize, and then make changes objectively.

Additionally, when you’re ready to make a switch, it’s a good idea to run “holdout” experiments. In “holdout” experiments you make a change for 99% of your users, but holdout changing 1% as a control for several days or weeks to ensure you’re delivering the effect you intended.

The Last Word

Experimentation is a scary notion, and it is this fear that acts as an invisible restraint for many companies. It’s crucial that leaders encourage the implementation of new ideas and acknowledge that while they will not always incur the intended results, they will always create learning opportunities. Leadership and a positive work environment are the ultimate cultivators of curious employees. Employees that think independently and generate unique, innovative ideas are the driving force behind successful experimentation in any workplace. By constantly experimenting at scale you are sure to create positive results, and allowing those results to build on each other is how the most successful companies continue to move forward.

Don’t be afraid: Let’s try something new.