How We Centralized Log Collection Across a Cabinet-Level Department - Jonathan Margulies, Splunk Architect, Large Federal Agency Jay Benfield ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Copyright © 2016 Splunk Inc.
How We Centralized Log Collection
Across a Cabinet-Level Department
Jonathan Margulies, Splunk Architect, Large Federal Agency
Jay Benfield – Sales Engineering Manager, Splunk
Agenda
About me and the Department my team supports
Challenges at the Department
The Solution
Turning problems into opportunities
Replacing ArcSight with Splunk
Underrated role of developers in Splunk administration
How what we did can help you
2
My Cabinet-Level Department
Oversees tens of agencies, varying from the very large to very
small
IT investment at agencies varies wildly:
Some don’t administer their own workstations
Some have their own IT teams comparable to the
department HQ
Many lie in the middle, with small pockets of IT expertise
3
Who Am I?
Author and security researcher
Three years as a Splunk Architect
Nine years as a security researcher at
Sandia National Labs
Masters in Computer Science from Cornell
University
Unskilled amateur ice hockey player
4
Challenges in the Government
• Bureaucracy discourages intelligent risk-taking, slows innovation
• Tendency to make “safe” investments in legacy technologies like ArcSight
• Demonstrating security compliance is seen as being secure
• Government will pay for compliance that doesn’t improve security
• Trouble justifying costs that improve security without helping compliance
• Crippling shortage of security and log management expertise
• Epidemic of systems (silos) that have the wrong security controls
• Very few in government do security log management well
5
How These Challenges Affect Us
• Incremental improvements are not sufficient
• Need vision, understanding, and willingness to make big bets
on game-changing solutions
• Can’t throw more people at problems
• Have to do more with less
• Disparate environment due to siloed implementations
• Lack of enterprise-wide tools
• Harder to manage and gain holistic view of department
• Need to make compromises
• Try to solve both security and compliance
6
Approach
• Splunk-as-a-Service for the whole department
• One large investment promotes efficiencies and consistency
• Leverage ‘whole is better than sum of its parts’ theory
• Economies of scale
– Share across agencies trying to solve the same problems
– Dashboards and alerts that help with FISMA audit compliance
• Single pane of glass promotes visibility and security
– Department SOC can monitor and alert on everyone’s log data
7Signs You’re Doing Splunk Wrong
Paying for two weeks of Splunk professional services to set Splunk
up while they “train my administrators”
Buying underpowered hardware (or, worse, VMs) and then
complaining that “Splunk is slow”
Telling the Cisco/FireEye/Proxy/Windows admin to add Splunk to
his plate
Running Splunk just to collect logs from the web proxy because
Splunk has a prettier user interface
8Getting the Data is the Hardest Part
I spent my first 6 months at the department just trying to
get the data
My three best friends:
Department-level services
Compliance
Personnel turnover
10Friend 1: Department-Level Services
11A Big Money-Saver
12Friend 2: Compliance
13A Compliance Win for Everyone
14Friend 3: Personnel Turnover
15Splunk as a Service
16What Splunk Offers vs ArcSight Complete programming platform with great documentation and tons of flexibility Intuitive searches from the get-go; experts can make it do backflips Splunkbase offers tons of apps that let you easily extend Splunk’s out-of-the-box functionality, usually for free Built to store and work on raw log data, which makes compliance, legal, and security happy More forgiving of parsing mistakes
Splunk is Great for Developers
Fully programmable, crazy flexible big data analytics
platform!
My team: group of developers and statisticians with
expertise in IT and security; Splunk is our Heaven
Our toolkit:
SPL, Simple XML, macros, data models, custom commands (Python),
custom endpoints (Python), custom UI widgets (JavaScript), search
rewriting (JavaScript), automated config file generation (Python)
18What a Developer Can Achieve
19What a Developer Can Achieve
20Automate Everything
Generating indexes, including automatically choosing safe max sizes
Generating user roles and assigning them to the right indexes
Generating serverclass.conf to properly configure our master deployment
server, which in turn configures the other deployment servers
Generating inputs.conf for our syslog servers, which collect syslog from over
5,000 hosts that have about 100 different sourcetypes and go to 100 different
indexes on multiple sets of indexers
Generating SSL certificates, inputs.conf, outputs.conf, deploymentclient.conf,
and serverclass.conf for new agencies when they come onboard
21Benefits
Saved $900,000 in ArcSight maintenance this year!
Investigations into email campaigns used to take hours, now they take minutes
Weekly log audits used to average 4 hours for a moderate system, but now they
take about 5 minutes
ISSOs used to print out hundreds of pages of logs and sign them
All the data in one place, so no waiting three days for IT admins
Correlating logs against host inventory from BigFix, workstation logins, threat
indicators, LDAP data, geolocation databases
Reduced false positives and negatives, with much more maintainable alerts
22Next Steps
Release all of my team’s Splunk scripts and other tools as an open
source toolkit
Make more use of Splunk Machine Learning Toolkit
Document our architecture and lessons learned publicly so that other
departments and complex companies can benefit
Evangelize to developers on the greatness of Splunk as a software
platform
Grow our service
23Lessons Learned / Insights
• Don’t build Splunk for one system
– Take a leap. Build a great Splunk environment, and build it big and beautiful,
and they’ll come
• Splunk gives you back the effort you put into it
• Use the power of Splunk—the more you correlate the better
• Invest in the right resources with the right skills
• CDM is an opportunity—don’t let it pass you by!
24Contact jonathan@margulies.me @unsaltedHash Check out our open source Splunk tools at www.gitlab.com/rationalcyber
Thank You
Announcements
.conf2017 is coming to
Washington, D.C.!
September 25-28, 2017
Walter E. Washington Convention Center
Reserve your seat for .conf2017 now through November 30th to get
the super saver discount!
Visit the
Reserve your spot today, pay later! Information
Kiosk in the
Solution
Sign Up Today: http://live.splunk.com/LP=1822 Pavilion!
After registration opens, you will have 60 days to complete your
registration to secure the super saver rate.
28Support Operation Homefront!
Earn Your 6 Sponsor Badges!
Splunk will donate $10 Dollars to Operation Homefront’s Holiday Meals for Military Families Program for
every attendee that completes their mission of earning 6 sponsor badges. The program will provide
meals to our local military families this holiday season.
Plus a bonus if we hit 350 number of completed missions. Splunk will double the $3,500 donation to
$7,000!
29Workshops: Get Splunk Hands-on Experience
Attend a Splunk Workshop
Upcoming Schedule
December 1: Introduction to Splunk Enterprise
December 14: Introduction to Splunk IT Troubleshooting
January 11: Introduction to Splunk Enterprise Security
January 11: NEW! Database Performance Tuning and
Capacity Planning Workshop
January 25: Introduction to Splunk IT Service Intelligence
January 25: NEW! Splunk for Application Developers
Location Visit the
Splunk Office McLean, VA Information
Kiosk in the
Solution
Pavilion!
Visit http://www.doyouknowsplunk.com/workshopsSplunk User Groups - Connect with Local Splunkers
Northern Virginia
Meets the last 3rd Thursday of every month
https://usergroups.splunk.com/group/northern-virginia-splunk-user-group.html Visit the
DC Information
Meets the last Wednesday of every month Kiosk in the
https://usergroups.splunk.com/group/washington-dc-splunk-user-group.html Solution
Baltimore Pavilion!
Meets the 3rd Monday of every month
https://usergroups.splunk.com/group/baltimore-splunk-user-group.htmlTake the GovSummit Post Event Survey!
We value your feedback!
Take the post event survey on the iPads in the foyer starting at 2:30pm!
32You can also read
