Data-Driven Product Development at Etsy - Nellwyn Thomas Director of Analytics, Etsy @nellwyn
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Data-Driven Product Development at Etsy Tech Planet 2014 November 3rd, 2014 Nellwyn Thomas Director of Analytics, Etsy nellwyn@etsy.com @nellwyn
today ! etsy, briefly how we deploy code how we use data takeaways
today ! etsy, briefly how we deploy code how we use data takeaways
The world’s handmade and vintage marketplace.
Over 26M items from 1M shops in over 150 countries.
Information as of Nov 3rd 2014
$1.35B sales in 2013
!
Information as of Nov 3rd 2014150GB web logs daily
!
Information as of Nov 3rd 2014dozens deploys per day
!
continuous delivery
!
!
!
Information as of Nov 3rd 2014dozens of experiments running simultaneously
!
continuous experimentation
!
!
!
Information as of Nov 3rd 2014today ! etsy, briefly how we deploy code how we use data takeaways
Basic Principles ! Single source repository Automate the build Automate the testing Automate deployment Always ready to deploy quality software
Why Continuous Delivery? ! Move faster Fail faster More empowering Breeds confidence
config.php template.tpl
..................................................................
Deploy Dashboard Git Commits Princess Supergrep Supergrep Forum Watcher
Time: 1 hour Until: Now Hide Deploys Deploys hidden: Config Web / Api / Atlas Search Photos Blog Chef (prod) Schemanator
CI Status (1 hour)
Push
rbateman
on qa
6m since 7, 2012 9:54
End-User (1 hour)
Three-Armed Sweaters clip Akamai Whoopsies clip Edgecast Whoopsies clip Screwed Users clip
error pages · historical origin failures · historical origin failures · historical unique users · historical
Error Logs (All) (1 hour) All
File or Script Not Found clip Exceptions clip Fatals clip Errors clip Warnings clip
current · historical current · historical current · historical current · historical current · historical
Segmentation Faults clip PHP Allowed Memory clip PHP Maximum Execution clip Memcache Connection clip Thrift Exceptions clip
current · historical Size Exhausted Time Failures current · historical
current · historical current · historical current · historicalWhy Continuous Delivery? ! “Because you’re integrating so frequently, there is significantly less back-tracking to discover where things went wrong, so you can spend more time building features.” ! http://www.thoughtworks.com/continuous-integration
All credit: John Allspaw
All credit: John Allspaw
All credit: John Allspaw
All credit: John Allspaw
today ! etsy, briefly how we deploy code how we use data takeaways
first, the stack
desktop & mobile website apps emails
beacon beacon beacon
Etsydoop
(HDFS)
bcn.etsy.com event logs scalding skills
visit logs
Production
Production
Databases
Production
Databases
Production
Databases Vertica
Databases SQL skills
A/B SQL Dash- everyone!
Analyzer Builder boardsworking with “data” requires multiple methods ! a/b testing, cohort analysis, predictive modeling, opportunity, feature, impact analysis, usability studies, ethnographic interview, surveys !
working with “data” requires multiple methods ! a/b testing, cohort analysis, predictive modeling, opportunity, feature, impact analysis, usability studies, ethnographic interview, surveys !
step one: hypothesize
Buy Sell Registry Community Blogs Mobile Gift Cards Help
Register Sign In Search for items and shops Search Cart
Home › BayTowneLeatherUSA › IPAD Cases
$74.00 USD Only 1 available
Like this item?
Favorite
Add it to your favorites to revisit it later.
Add to Cart
Leather Ipad Sleeve/ Case/ Envelope/ Folio for IPAD 2 or 3. Full
Grain Veg Tan Leather. Handmade by BayTowneLeatherUSA. Favorite
Shop
BayTowneLeatherUSA
BayTowneLeatherUSA
38
items
Add shop to favorites
See who favorites this shop
Shop owner
Roger
Marion, Illinois, United States
Favorites
Circle
Feedback: 27, 100% pos.
This luxurious/rugged and well built IPAD case will protect your IPAD in style. Or you can use
it as a folio to carry your necessities . Luxurious enough for use at the office, and rugged Contact
enough for use on the trail.
Made of beautiful - rugged 5/6 oz. Premium veg tan leather (which will break in beautifully) ; it
will develop a wonderful natural patina over time as only full grain veg tan leather can do. Item
(Gets better with age)
Add item to favorites
***I make these cases ONE BY ONE as orders are received*** See who favorites this itemstep one: hypothesize
control treatment
Buy Sell Registry Community Blogs Mobile Gift Cards Help
Register Sign In Search for items and shops Search Cart
Home › BayTowneLeatherUSA › IPAD Cases
$74.00 USD Only 1 available
Like this item?
Favorite
Add it to your favorites to revisit it later.
Add to Cart
Leather Ipad Sleeve/ Case/ Envelope/ Folio for IPAD 2 or 3. Full
Grain Veg Tan Leather. Handmade by BayTowneLeatherUSA. Favorite
Shop
BayTowneLeatherUSA
BayTowneLeatherUSA
38
items
Add shop to favorites
See who favorites this shop
Shop owner
Roger
Marion, Illinois, United States
Favorites
Circle
Feedback: 27, 100% pos.
This luxurious/rugged and well built IPAD case will protect your IPAD in style. Or you can use
it as a folio to carry your necessities . Luxurious enough for use at the office, and rugged Contact
enough for use on the trail.
Made of beautiful - rugged 5/6 oz. Premium veg tan leather (which will break in beautifully) ; it
will develop a wonderful natural patina over time as only full grain veg tan leather can do. Item
(Gets better with age)
Add item to favorites
***I make these cases ONE BY ONE as orders are received*** See who favorites this itemstep two: launch
step three: wait Dan McKinley’s http://www.experimentcalculator.com/
step four: analyze
step five: decide on off
it’s not this simple old new
it’s not this simple
Design Develop Measureit’s not this simple
?at etsy we try to use data more often
Short. Iterative. Measurable.Buy Sell Registry Community Blogs Mobile Gift Cards Help
Register Sign In Search for items and shops Search Cart
Home › BayTowneLeatherUSA › IPAD Cases
$74.00 USD Only 1 available
Like this item?
Favorite
Add it to your favorites to revisit it later.
Add to Cart
Leather Ipad Sleeve/ Case/ Envelope/ Folio for IPAD 2 or 3. Full
Grain Veg Tan Leather. Handmade by BayTowneLeatherUSA. Favorite
Shop
BayTowneLeatherUSA
BayTowneLeatherUSA
38
items
Add shop to favorites
See who favorites this shop
Shop owner
Roger
Marion, Illinois, United States
Favorites
Circle
Feedback: 27, 100% pos.
This luxurious/rugged and well built IPAD case will protect your IPAD in style. Or you can use
it as a folio to carry your necessities . Luxurious enough for use at the office, and rugged Contact
enough for use on the trail.
Made of beautiful - rugged 5/6 oz. Premium veg tan leather (which will break in beautifully) ; it
will develop a wonderful natural patina over time as only full grain veg tan leather can do. ItemBuy Sell Registry Community Blogs Mobile Gift Cards Help
Register Sign In Search for items and shops Search Cart
Similar Items See more
Leather Ipad Sleeve/ Case/ E… Leather Ipad Sleeve/ Case/ E… Leather Document Case, Foli… Leather ipad case. Handstitc… Buckled leather iPad sleeve …
BayTowneL… $74.00 USD BayTowneL… $74.00 USD BayTowneL… $94.00 USD StudioFi $55.00 USD LUSCIOUSL… $99.00 USD
Home › BayTowneLeatherUSA › IPAD Cases
$74.00 USD Only 1 available
Like this item?
Favorite
Add it to your favorites to revisit it later.
Add to Cart
Leather Ipad Sleeve/ Case/ Envelope/ Folio for IPAD 2 or 3. Full
Grain Veg Tan Leather. Handmade by BayTowneLeatherUSA. Favorite
Shop
BayTowneLeatherUSA
BayTowneLeatherUSA
38
items
Add shop to favoritesresults ! 10% reduction in bounces ++ increase in listing pages viewed ++ conversion rate win
full redesign launched in august 2013
continuous iteration Idea Iterate Iterate Iterate Iterate Iterate Death
with data where it helps
Idea Iterate Iterate Iterate Iterate Iterate Death
Opportunity Feature A/B
Analysis Analysis Testing
User Funnel
Surveys
Research Analysisover and over again
Idea Iterate Iterate Iterate Iterate Iterate Death
Opportunity Feature A/B
Analysis Analysis Testing
User Funnel
Surveys
Research AnalysisAll credit: John Allspaw
change
direction
time
All credit: medecisions informed by data, every day
today ! etsy, briefly how we deploy code how we use data takeaways
validate early and often. ! data helps validate ideas.
good tooling
don’t confuse
iteration
leaves more
with
time
“tweaking
for real thinking.
pixels”leverage existing
workflows.
“Changing behavior works only if it
can be based on the existing ‘culture.’”
Peter Druckerexperimentation is not just for optimization
more talks on these topics Steve Mardenfeld on Experiments http://www.slideshare.net/smardenfeld ! Dan McKinley on Experiments http://mcfunley.com/design-for-continuous-experimentation John Allspaw on Continuous Delivery http://www.kitchensoap.com/category/talks/ ! Mike Brittain on Continuous Delivery www.mikebrittain.com/blog/talks/
Learn more etsy.com/careers github.com/etsy ! ! Nellwyn Thomas nellwyn@etsy.com @nellwyn
You can also read
