Kevin Costner is a lousy entrepreneur. Don’t sell what you can make. Make what you can sell.
In a startup, the purpose of analytics is to iterate to a product/market fit before the money runs out.
Everyone’s idea is the best right? People love this part! (but that’s not always a good thing) This is where things fall apart. No data, no learning.
The five Stages of Lean Analytics Empathy Stickiness Virality Revenue Scale Thestageyou’reat E-commerce SaaS Media Mobile app User-gen content 2-sided market The business you’re in One Metric That Matters.
Five lessons beyond the basics • Mining your existing data • A new product every day • Are you at your local maximum? • It’s different when you’re big • Build analytics into how you operate
Mining your existing data Moving beyond guesswork.
Identify a key business problem, pick the OMTM, draw a line in the sand, and get started.
Draw a new line Pivot or give up Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data: find a commonality Without data: make a good guess Find a potential improvement Draw a linePick a KPI The Lean Analytics Cycle
Do hosts withDo hosts with professionalprofessional photography getphotography get more business?more business? Do hosts withDo hosts with professionalprofessional photography getphotography get more business?more business? Airbnb experiments...
Professional photography helps Airbnb’s business Gut instinct Concierge MVP 20 photographers in the field Test results Two to three times more bookings! Back to the beginning Use additional data to keep experimenting
5,000 shoots/month in Feb. 2012
Hang on a second...
Really? Professional photography helps Airbnb’s business Gut instinct
Draw a new line Pivot or give up Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data: find a commonality Without data: make a good guess Find a potential improvement Draw a linePick a KPI Remember this?
“Gee, those houses that do well look really nice.” Maybe it’s the camera. “Computer: What do all the highly rented houses have in common?” Camera model. With data: find a commonality Without data: make a good guess
The only thing worse than bad feedback is no feedback at all. - Dave McClure, Startupfest 2012
Because it gives you nothing to analyze.
Virality stage: Circle of Moms finds an engaged market • Stage: Stickiness • Model: UGC • Launched as Circle of Friends in 2007, it was a way for small groups to interact atop Facebook’s platform; but when engagement wasn’t good enough, the founders decided to dig deeper.
The problem: Not enough engagement • Too few people were actually using the product • Less than 20% of any circles had any activity after their initial creation • A few million monthly uniques from 10M registered users, but no sustained traction
A new product every day Why cohorts matter more than you think.
Segments, cohorts, A/B, and multivariates Segment: Cross-sectional comparison of all people divided by some attribute (age, gender, etc.) ☀ ☁ Cohort: Comparison of similar groups along a timeline. A/B test: Changing one thing (i.e. color) and measuring the result (i.e. revenue.) Multivariate analysis: Changing several things at once to see which correlates with a result. ☀ ☁ ☀ ☁ JanJan FebFeb MarMar AprApr MayMay
Why use cohorts? Here’s an example. January February March April May Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50 Is this company growing or stagnating? Cohort 1 2 3 4 5 How about this one? January $5 $3 $2 $1 $0.5 February $6 $4 $2 $1 March $7 $6 $5 April $8 $7 May $9
Two views of the same company Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5 February $6 $4 $2 $1 March $7 $6 $5 April $8 $7 May $9 Averages $7 $5 $3 $1 $0.5 Look at the same data in cohorts
Are we at a local maximum? You need optimization and inspiration.
The vertebrate eye is “backwards and upside down”. Light travels through cornea, lens, aqueous fluid, blood vessels, ganglion cells, amacrine cells, horizontal cells, and bipolar cells. The cephalopod eye is constructed the “right way out”, with the nerves attached to the rear of the retina. No blind spot. Extension of the brain Invagination of the head
It’s different when you’re big Tilting at the corporate windmill.
“A startup is an organization designed to search for a sustainable, repeatable business model.” - Steve Blank Remember when I said,
“A big company is an organization designed to perpetuate a business model.” - Me
In other words, if your job is change you have your work cut out for you.
Span of control and the railroads The story of Daniel C. McCallum
Disruption may not fit neatly into an existing part of the organization, so you need to incubate it internally for a while.
Unlike a VC or startup, when the initiative fails the organization still learns.
Big organizations might not innovate at speed—but they innovate at scale.
The Lean Analytics lifecycle for an Intrapreneur Empathy Find problems; don’t test demand. Skip the business case, do analytics Entitled, aggrieved customers Stickiness Know your real minimum based on expectations, regulations Hidden “must haves”, feature creep Virality Build inherent virality in from the start; attention is the new currency Luddites who don’t understand sharing Revenue Consider the ecosystem, channels, and established agreements Channel conflict, resistance, contracts Scale Hand the baton to others gracefully Hating what happens to your baby Stage Do this Fear this Beforehand Get buy-in Political fallout
A large organization needs a portfolio of investments, judged by different criteria. Treasury bonds Venture capital
A large organization needs a portfolio of investments, judged by different criteria. Safe Predictable Evolutionary Achieve the business plan Risky Uncertain Revolutionary Discover the business model Times of peace Times of war
Acquisition or disruption? Innovate fast: nothing to lose Innovate slowly: conform to brand, legislation, approvals Found P/M fit Found P/M fit Integration complexity, founder churn, indigestion Biz case Scale Scale Biz model VC pitch
Moving from box to box • How businesses think about products or companies • Lean is about moving up and to the right
Frame the project as a learning exercise; product creation is almost accidental. Learn it’s a failure Learn it’s a success Industry knowledge Industry knowledge Possible product Business plan metrics here Market Business model metrics here Lean product Lines in the sand here
Build analytics into how you operate the business Three threes and the Problem-Solution Canvas
Three threes Three assumptions What big bets are you making? •“People will answer questions” •“Organizers are frustrated with how to run conferences” •“We’ll make money from parents” •“Amazon is reliable enough for our users.” Three actions to take What are you doing to make these assumptions happen (or identify they’re wrong and change course?) •Product enhancements •Marketing strategies Three experiments to run What are you going to learn today? •Feature tests •Continuous deployment •A/B testing •Customer survey
Three threes Monthly Weekly Daily Board, investors, founders Executive team Employees Strategy Tactics Execution Three assumptions Three actions to take Three experiments to run
Three threes Get more people Increase answer % Test better questions Change the UI Test timings Questions from peers Many people will answer questions Three assumptions Three actions to take Three experiments to run