A/B testing a new feature

The problem: One of our team OKRs was to increase the percentage of Babbel Live learners doing reviews in the Babbel self-study app to 10%. We hypothesized that if we created a direct connection to the review feature from within the Live tab, we could increase the percentage of learners doing reviews by 5%.

The solution: A card on the booked state of a class page that appears shortly before a learner’s class started. The card encourages learners to “warm up” before class with a short, 2-minute exercise. When learners click on the card, they’re taken to the review feature. When they finish the review, they’re returned to the page where they can join their class.

How we got there: The product designer, PM, EM, and I debated where and how to create this connection between the two features. We finally landed on the booked state of the class page, because we knew that learners were likely to open this page shortly before their class started, where they would see the warm-up card. We also knew from learner feedback that people were interested in doing lessons in the self-study app before class, but those typically lasted 10-15 minutes–time that most people don’t have right before class. A quick review, however, would take only two minutes and would be a quick way to get learners thinking in the language they’re learning.

The challenges: Our first big challenge was aligning all collaborators on the feature’s placement. Once I’d framed the connection to the reviews as a pre-class warm-up, we could agree that it made sense to have it on the page learners would visit just before joining class. 

The next challenge was technical. We wanted to ensure that learners would be able to go directly into the review exercise without having more options to click through. I argued especially strongly for this one, to make sure that learners wouldn’t be confused about where they’d ended up in the flow. Luckily, our engineering manager was able to work with the agile team who owned the review feature and create the direct connection we wanted.

The biggest win: The test was a huge success, and the percentage of Babbel Live learners doing reviews increased from 8.36% to 19.24%.