Email Accessibility & Reliability - Combining Dovecot and Scality to Win Dan Shain & Jim Perry - Open-Xchange
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Email Accessibility & Reliability Combining Dovecot and Scality to Win Dan Shain & Jim Perry Cloud Office R&D
Who are we?
• Dan Shain
‣ Heads Engineering, Development, and
Operations for Rackspace Applications Services
Cloud Office Division
‣ 30+ years in the IT Realm with focus on large
distributed systems
• Jim Perry
‣ Technical Manager for Software Development
‣ Responsible for all things Dovecot
‣ 25 years in the software development and IT
management
IN 2014 WE NEEDED TO
CHANGE OUR OPERATIONAL
PARADIGM
Dilemma We Faced
Open Source Dovecot with filesystem based storage
Current path would increase base product cost by 200%
Hardware Projections
4500
4000
3500
CAPEX Costs
3000
OPEX Costs
2500
2000
1500
Logistical
1000
Nightmare
500
0
2015 2016 2017 2018 2019
Status Quo
4 Footer
Dovecot Deployment that was…
4,500+ Assets across 3 Datacenters and in 5 physical zones
• Scaling • Compute and Storage could not scale independently
Limitations • Complicated support processes and scripts
• Data Protection • N+2 Protection for Geo-redundant failures was not reliable and very costly
• Environment • Hardware management was a logistical nightmare
• Management • System Rebalancing was Managed Process
• Product • On boarding limitations caused by write width
Impacts • Cost basis rising
5
Our Use Case
Wide range of uses on a multi-tenant platform
• Multi-tenant environment > 800,000 Domains
• Wide variety of “Domain” level settings
• Usage patterns, although normalized, have large variances
• Normal Does not cause problems, “Abby Normal” does… we have extremes…that break the normal
Mailbox Distribution By Average Messages Accessed Everywhere
Message Count Sent/Received by Mailbox/Day
Mailbox Count
Sent
Received
1 2 10000
10,000
< 10 < 100 < 500 < 5,000 > 3,000 > 20,000 > 200,000 > 1,000,000 > 2,000,000
6
Messages per MailboxDemands
Product Team/Customers Engineering/Operations
o Happy Customers o Stay abreast of Vulnerabilities
o Improved cost margins o Reduce Operational Costs to run and
o Value Added Customer Support Contacts maintain the business
o No Down Time o Stabilize Capital Expenditures
o Ensure Deliverability o Ease Supply Train Impacts
o Zero Data Loss o Reduce/optimize Datacenter Space
o Faster Feature Delivery o Move to virtualized deployment models
7DovecotPro – Scality Solution Things WILL Fail
Why did we chose this solution?
‣ Access Patterns
Disk
• Very High Write, High Read, Medium Delete
• Data after 10 days, stays… and stays Node
‣ Customer Activity Profile
Rack
• Business Class Usage (Extremely High Daily Active Percentage)
‣ N+2 Architecture Row
• Need multi-zone and Geo-Graphic managed solution
Network
‣ Fault Segmentation (Hierarchy of failure events)
‣ Scaling Datacenter
• Must scale compute and storage separately
‣ Maintenance & Upgrade
‣ Knowledge of Dovecot
‣ Reputation of Vendors
8Deployment Data Center Data Center
Chosen
DoveCot Pro DoveCot Pro
Backend Backend
Dovecot/Scality Multi-Geo
Load Balancer Load Balancer
(Local haproxy) (Local haproxy)
Location Deployment
Cass andra Cass andra
sproxyd
Availability Zone I
Availability Zone I
Scality Ring – Stretched over
2 Data Centers Availability Zone II
Availability Zone II
Cass andra Cass andra
Load Balancer Load Balancer
(Local haproxy) (Local haproxy)
DoveCot Pro DoveCot Pro
Backend BackendOptimized Scality
Infrastructure
• Leverage Scality to Optimize Hardware
‣ Apollo 4510 platform
‣ Tremendous scaling options
‣ Gigabyte Cost optimization
‣ Normalized and consistent environment
• Key – We worked with Scality to
create an optimized “Email
Storage Ring”
‣ Focus on the nature of email access patterns as
well as sizing
‣ Do not overload the use case (We are not
creating an S3 like environment)
10 OX SummitFootprint Reduction 1400
Hardware Repairs
• Over 2,000 Assets Decommissioned to date 1200
• Avoided ordering over 1,440 Assets and opening 1 new server room 1000
• Vacating several rooms 800
• Reduced Maintenance to less than 35% of Peak (and dropping) 600
Progress to Date
400
2500
200
0
Q1 2016 Q2 2016 Q3 2016 Q4 2016 Q1 2017 Q2 2017 Q3 2017 Q4 2017 Q1 2018
2000
1500
1000
500
0
Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Mar-18 Jun-18 Sep-18 Dec-18Storage Stability • Scality survived hardware outages; Disk and Chassis • Handled Massive Cassandra Rebuild Efforts • Rack Moves & Room Reconfigurations • Only 1 Scality Ring associated customer impacting event ‣ Root Cause related to HP Firmware defect • Only a handful of special tuning needed to run to: ‣ Re-engage disks post hardware oddness ‣ Node release of expired data • Expansion Scheduled for this Summer
CHALLENGES FACED
‣ Interplay between DovecotPro, Cassandra
& Scality
‣ Early adoption woes “Optimist: Someone
‣ Initial errors in Dovecot Director Ring
Configuration
who figures that taking
‣ Fault Segmentation a step backward
‣ Too many customized plugins on our behalf after taking a step
‣ Need to improve path to upgrade forward is not a
‣ Allowance needed for “Bake-In” disaster, it's a cha-cha.”
‣ Tuning of the “Ring” – Overhead Reduction
‣ Expanded our internal knowledge base
- Robert Brault
13THANK YOU
ONE FANATICAL PLACE | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
Copyright © 2018 Rackspace | Rackspace® Fanatical Support® and other Rackspace marks are either registered service marks or service marks of Rackspace US, Inc. in the United States and other countries. Features, benefits and pricing
presented depend on system configuration and are subject to change without notice. Rackspace disclaims any representation, warranty or other legal commitment regarding its services except for those expressly stated
in a Rackspace services agreement. All other trademarks, service marks, images, products and brands
14 remain the sole property of their respective holders and do not imply endorsement or sponsorship.You can also read
