ELASTIC STACK SOLUTION ON DELL EMC VXFLEX FAMILY - ABSTRACT THIS WHITE PAPER HIGHLIGHTS THE DEPLOYMENT AND BEST PRACTICES OF ELASTIC STACK ON THE ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
White Paper
Elastic Stack Solution on Dell EMC VxFlex
family
Abstract
This white paper highlights the deployment and best practices of Elastic Stack on
the Dell EMC VxFlex family.
March 2020
000060
Revisions
Revisions
Date Description
March 2020 Initial release
Acknowledgements
This paper was produced by the following:
Author: Kailas Goliwadekar
Support: Keith Quebodeaux and Shashikiran Chidambara
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2020 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners. [3/16/2020] [White Paper] [000060]
2 Elastic Stack Solution on Dell EMC VxFlex family | 000060
Table of contents
Table of contents
Revisions............................................................................................................................................................................. 2
Acknowledgements ............................................................................................................................................................. 2
Table of contents ................................................................................................................................................................ 3
Executive summary............................................................................................................................................................. 5
1 Introduction ................................................................................................................................................................... 6
1.1 Objective ............................................................................................................................................................. 6
1.2 Audience ............................................................................................................................................................. 6
1.3 Terminology ........................................................................................................................................................ 7
2 Product overview .......................................................................................................................................................... 8
2.1 VxFlex family ...................................................................................................................................................... 8
2.1.1 VxFlex integrated rack................................................................................................................................ 8
2.1.2 VxFlex appliance ........................................................................................................................................ 9
2.1.3 VxFlex Ready Nodes.................................................................................................................................. 9
2.1.4 VxFlex OS .................................................................................................................................................. 9
2.1.5 VxFlex Manager .......................................................................................................................................10
2.1.6 VxFlex OS management ..........................................................................................................................10
2.2 Elastic Stack .....................................................................................................................................................11
3 Elastic Stack architecture overview ............................................................................................................................12
3.1 Elasticsearch ....................................................................................................................................................12
3.2 Logstash ...........................................................................................................................................................13
3.3 Kibana ...............................................................................................................................................................14
3.4 Beats .................................................................................................................................................................14
4 Solution architecture ...................................................................................................................................................15
4.1 Logical architecture ..........................................................................................................................................15
4.2 Network architecture .........................................................................................................................................16
5 Testing and Validation ................................................................................................................................................17
5.1 Test methodology .............................................................................................................................................17
5.2 Test results .......................................................................................................................................................18
6 Best practices .............................................................................................................................................................20
6.1 VxFlex integrated rack ......................................................................................................................................20
6.2 VxFlex OS .........................................................................................................................................................20
6.3 VxFlex network .................................................................................................................................................20
6.4 Elasticsearch ....................................................................................................................................................20
7 Deployment of Elastic Stack on VxFlex .....................................................................................................................21
3 Elastic Stack Solution on Dell EMC VxFlex family | 000060
Table of contents
8 Conclusion ..................................................................................................................................................................22
A Appendix A: Detailed results ......................................................................................................................................23
A.1 Use case1: EventData track .............................................................................................................................23
A.2 Use case1: Http_logs track ...............................................................................................................................25
B Appendix B: Configuration details ..............................................................................................................................28
C Technical support and resources ...............................................................................................................................29
C.1 Related resources.............................................................................................................................................29
C.2 Additional resources .........................................................................................................................................29
4 Elastic Stack Solution on Dell EMC VxFlex family | 000060
Executive summary
Executive summary
Teams responsible for managing large-scale and rapidly growing data center infrastructure constantly meet
challenges in terms of scalability and flexibility needs. Enterprises are transitioning from the traditional data
center to a hyperconverged infrastructure (HCI) approach to address these challenges. HCI based offerings
give enterprises the ability to support modern workload and provides a simplified solution to manage their
infrastructure needs. Organizations are switching to public cloud security and log analytics to provide
uninterrupted operations with minimum downtime and SLAs with huge amount of data being generated daily.
Since many organizations are switching to cloud, the necessity for public cloud security tools and log analytics
platforms is increasing rapidly.
A substantial amount of this data consists of logs like web server, http, application logs, and so on. With
careful and detail log analysis, organizations can tap on various business opportunities and threats
surrounding it. The Elastic Stack solution on Dell EMC VxFlex family provides complete log analysis that
helps in searching, analyzing, visualizing data from different machines, in addition helping to identify any
potential security threats.
This white paper provides guidelines and best practices for optimal deployment of Elastic Stack on Dell EMC
VxFlex family.
The Dell EMC VxFlex family for Elastic Stack solution offers unique advantages. The results from Elastic
stack performance benchmarking tests, executed using benchmarking tool Rally clearly showed that VxFlex
can support high performance requirements of Elastic stack. Other key benefits of deploying Elastic stack on
VxFlex include:
• Flexibility of deployment options.
• Faster indexing and searching performance.
• Easily scalable solution to multiple Elastic master and data nodes
5 Elastic Stack Solution on Dell EMC VxFlex family | 000060
Introduction
1 Introduction
This white paper outlines the design consideration of deploying Elastic Stack on VxFlex family. The primary
focus of this paper is to provide the reader with details of an Elastic Stack solution on VxFlex family. This
paper also provides more details on the best practices while deploying Elastic stack on VxFlex along with
some performance benchmarking results using Rally tool.
1.1 Objective
This white paper demonstrates
• VxFlex family and Elastic Stack Overview
• Elastic Stack Architecture
• Deployment of Elastic Stack on VxFlex integrated rack
• Validating Elastic Stack on VxFlex integrated rack
• Conclusion
1.2 Audience
This white paper is intended for Elastic Stack administrators, system engineers, partners, and members of
Dell EMC and partner professional service community who are looking to make real-time decisions, and
analysis of data to make the right marketing decisions.
The reader of this document must have a working knowledge of the following technologies:
• Dell EMC VxFlex portfolios
• Elastic Stack
• VMware vSphere
6 Elastic Stack Solution on Dell EMC VxFlex family | 000060
Introduction
1.3 Terminology
The following table defines acronyms and terms that are used throughout this document:
Terminology
Term Definition
HCI Hyperconverged Infrastructure
OOB Out-of-band management
SVM Storage Virtual Machine
SDS Storage Data Server
SDC Storage Data Client
VLAN Virtual Local Area Network
TOR Top of the Rack Switch
7 Elastic Stack Solution on Dell EMC VxFlex family | 000060
Product overview
2 Product overview
2.1 VxFlex family
VxFlex family helps transforming from a traditional three-tier architecture to a modern data center without any
trade-offs, meeting business requirements without compromise. The central software layer for VxFlex family is
VxFlex OS, scale-out block storage service that enables customers to create a scale-out Server SAN and/or
hyperconverged infrastructure. The VxFlex family currently include VxFlex Ready Nodes, VxFlex appliance,
and VxFlex integrated rack.
VxFlex family
2.1.1 VxFlex integrated rack
VxFlex integrated rack is a rack-scale engineered system, with integrated networking, that provides linear
scalability and enterprise-grade availability. VxFlex integrated rack is engineered, manufactured, managed,
supported, and sustained as one system for single end-to-end life cycle support.
VxFlex integrated systems create a server-based SAN by combining virtualization software, which is known
as VxFlex OS, with Dell EMC PowerEdge servers to deliver flexible, scalable performance and capacity on
demand. Local storage resources are combined to create a virtual pool of block storage with varying
performance tiers. An important building block of VxFlex integrated rack is the compute part, which consumes
the storage volumes from the system. The VxFlex integrated rack enables you to scale from a small
environment to enterprise scale with over a thousand nodes. In addition, it provides enterprise grade data
protection, multi-tenant capabilities, and add-on enterprise features such as QoS, thin provisioning, and
snapshots. VxFlex systems deliver the performance and time-to-value required to meet the demands of the
modern enterprise data center.
8 Elastic Stack Solution on Dell EMC VxFlex family | 000060Product overview
VxFlex integrated rack benefits
2.1.2 VxFlex appliance
VxFlex appliance is a preconfigured and validated for fast, easy deployment, VxFlex appliance offers a
turnkey experience in an economic form factor. With VxFlex appliance, customers benefit from a smaller
starting point, with massive scale potential, without having to compromise on performance and resiliency.
2.1.3 VxFlex Ready Nodes
VxFlex Ready Nodes combine Dell EMC PowerEdge servers that are powered by Intel® Xeon® Scalable
Processors and VxFlex OS software to create scalable, reliable, and easy-to-deploy building blocks for
hyperconverged or server SAN architecture, multi-hypervisor or bare metal environments, and high-
performance databases.
2.1.4 VxFlex OS
VxFlex OS is software that creates a server and IP-based SAN from direct-attached storage to deliver flexible
and scalable performance and capacity on demand. As an alternative to a traditional SAN infrastructure,
VxFlex OS combines HDD, SSD, and NVMe media to create virtual pools of block storage with varying
performance tiers. VxFlex OS provides enterprise-grade data protection, multi-tenant capabilities, and add-on
enterprise features such as QoS, thin provisioning, and snapshots. VxFlex OS supports physical and
virtualized servers, has been proven to deliver significant TCO savings vs. traditional SAN.
9 Elastic Stack Solution on Dell EMC VxFlex family | 000060Product overview
VxFlex OS consists of three primary components: Meta Data Manager (MDM), Storage Data Client (SDC)
and Storage Data Server (SDS). The MDMs work the brain of the system and are responsible for managing
metadata and core functions such as automated rebuild and rebalance, which ensure data access as media
and servers fail. The SDC runs like an agent or daemon on a server and acts like a highly available HBA to
connect to storage cluster to consume the storage as required by the application workload. The SDSs are
installed on every compute node and are daemons that contribute storage to the storage cluster. SDCs
communicate directly with the SDSs. When an SDC gets an I/O request from the application, it sees the
cached metadata map, and sends the request directly to the SDS, which contains the requested data.
2.1.5 VxFlex Manager
VxFlex Manager is a unified management and automation solution for the VxFlex integrated rack and the
VxFlex appliance that enables template-based provisioning of infrastructure and workloads. It provides
monitoring, alerting, and health status of hardware and services. It simplifies and automates lifecycle
management of diverse and heterogeneous IT resources. It enables users to respond rapidly and reliably to
dynamic business needs.
VxFlex Manager is a comprehensive IT Operations Management (ITOM) software purpose that is built for
VxFlex to automate and simplify implementation, expansion, and lifecycle management.
VxFlex Manager brings together multiple management consoles, workflow automation, and an intuitive
interface that allows customers to monitor, manage, deploy, and maintain physical and virtual resources with
the click of a button. Key tenets of the VxFlex Manager architecture include:
• System assurance: compliancy and non-disruptive remediation
• Insights: monitoring, alerting, and health checks
• Implementation simplification: simplified and automated system deployment and workflows
• Node serviceability: single-button take node out of and back into service
• Hypervisor update: single-button upgrade of hypervisor
2.1.6 VxFlex OS management
VxFlex OS management is available using a GUI, CLI, and REST clients. There is a VMware vSphere® plug-
in that allows VMware admins to deploy, upgrade, configure, and manage VxFlex OS in an ESXi environment
within VMware vSphere.
10 Elastic Stack Solution on Dell EMC VxFlex family | 000060Product overview
2.2 Elastic Stack
Elastic Stack is a group of open source products from Elastic. It was earlier known as the ELK Stack. It is an
amazing and powerful collection of three open source projects namely Elasticsearch, Logstash, and Kibana.
These three technologies work well with each other despite being separate projects.
Elastic Stack components
Elastic Stack is an end-to-end log analysis solution that helps in deep searching, analyzing, and visualizing
the logs generated from different machine. The Elastic Stack includes numerous capabilities that help users
easily build solutions focused in three areas:
• Enterprise Search -- application search, site search, and workplace search.
• Observability -- logging, APM, and metrics.
• Security -- SIEM, endpoint, and security analytics.
Value added features such as machine learning, alerting, maps services, canvas, cross cluster replication,
cross cluster search, field and document level security, and cluster authentication can all be leveraged to help
most of the information about your data. Many of these capabilities are free and others can be licensed
through an Elastic Subscription that includes support. For more information, contact Elastic team.
11 Elastic Stack Solution on Dell EMC VxFlex family | 000060Elastic Stack architecture overview
3 Elastic Stack architecture overview
Elastic stack is designed to help users to import data with any format and source; to search, analyze, and
visualize imported data in real time.
The following figure displays the components available in Elastic Stack:
• Beats are agents that ship the data from different systems.
• Logstash collects logs, and events data. It parses, filters, and transforms the data.
• Elasticsearch stores, searches, and analyzes the data.
• Kibana is used to visualize the data.
Elastic Stack architecture
3.1 Elasticsearch
Elasticsearch is a NoSQL database that is based on Lucene search engine and provides a restful interface for
different operations. It is easy to scale, flexible, schema-less distributed search and analytics engine. It
provides simple deployment maximum reliability and uses indexes for data storage. Elasticsearch can be
easily scaled horizontally. It is used to perform detailed analysis and stores the data centrally for quick search
of the documents.
An Elasticsearch cluster is a collection of one or more nodes (servers) that together holds the entire data and
provides federated indexing and search capabilities across all nodes. A node is a single server that is part of
the cluster, stores the data, and participates in the cluster’s indexing and search capabilities.
12 Elastic Stack Solution on Dell EMC VxFlex family | 000060Elastic Stack architecture overview
An index is a collection of documents that have similar characteristics. A document is a basic unit of
information that can be indexed. When an index is created, the number of shards can be defined. Each shard
is a fully functional and independent "index" that can be hosted on any node in the cluster. Sharding is
important for two primary reasons:
• It allows horizontally split or scale the content volume.
• It allows to distribute and parallelize operations across shards (potentially on multiple nodes) thus
increasing performance/throughput.
For more information about Elasticsearch, see Elastic Stack and Product Documentation .
3.2 Logstash
Logstash is the data collection pipeline tool through which input is taken from different sources and output to
different data sources. The data is cleansed through filter options and transformed before sending it to
Elasticsearch. Logstash has different adapters to handle different applications, and it normalizes the data into
different destinations. Processing in Logstash is organized into one or more pipelines.
Logstash
These pipelines receive data and place it in internal queue. The processing threads then use filter plugins,
and once the data is processed its send to the output plug-in.
13 Elastic Stack Solution on Dell EMC VxFlex family | 000060Elastic Stack architecture overview
3.3 Kibana
Kibana is an open-source analytics software data visualization tool. With this tool, different visualizations,
charts, maps, histograms, and dashboards can be created. Kibana reads data from Elasticsearch and helps
developers to have an immediate insight into it. Kibana enables creating and saving custom dashboards. It
can be used for search, view, and interact with data that is stored in Elasticsearch directories. There are
different methods for performing searches on the data.
A sample Kibana dashboard is shown here:
Sample Kibana Dashboard
The most common search types in Kibana are Free text searches, Field-level searches, Logical statements,
and Proximity searches.
Kibana offers loads of advantages like easy visualizing, real-time analysis, debugging capabilities, saving and
managing multiple dashboards, and it is completely integrated with Elasticsearch.
3.4 Beats
Beats are open-source data shippers that are installed as agents on the servers and send operational data to
Elasticsearch. They are lightweight and can send the data from hundreds or thousands of machines to Elastic
Common Schema (ECS) that normalizes the data. Each beat is a separately installable product.
For example, Metricbeat is used to collect metrics for memory usage, CPU usage, and disk space, whereas
Filebeat is used to send file data such as logs.
14 Elastic Stack Solution on Dell EMC VxFlex family | 000060Solution architecture
4 Solution architecture
For this paper, the Elastic Stack solution is deployed on Dell EMC VxFlex integrated rack. The VxFlex
integrated is configured in a two-layer configuration meaning that servers are either compute nodes, providing
the ESXi hypervisor or storage nodes running VxFlex OS.
4.1 Logical architecture
The below diagram shows logical architecture diagram of Elastic Stack on Dell EMC VxFlex integrated rack
two-layer setup that had four SDS and three SDC. The setup of VxFlex two-layer was configured using
VxFlex Manager which automates the entire deployment process.
Solution architecture
From VxFlex OS standpoint, there was a single VxFlex OS cluster with a protection domain, consisting of four
RHEL storage only nodes that played the role of SDS. Each host is populated with ten 1.92 TB Toshiba SSD
drives. These 40 disks are used to create a storage pool from which the usable storage volumes are created.
The VxFlex OS client software (SDC) is installed on each ESXi host in order to access the volumes created in
the storage pool, a VMware Datastore was created on each of these volumes. The Elastic Cluster VMs were
deployed onto these datastores. The Elastic cluster that is included one Master VM and two Data VMs.
On each of the three SDCs, a single CentOS VM was created. The first to host Logstash, the second
Elasticsearch and the third Kibana. All these three VMs could communicate with each other.
For detailed configuration of VxFlex integrated nodes, see Appendix 9.2 Configuration.
15 Elastic Stack Solution on Dell EMC VxFlex family | 000060Solution architecture
4.2 Network architecture
The following figure demonstrates high-level network architecture of the two-layer set up on VxFlex integrated
rack system:
Network architecture
VxFlex integrated rack networking details
Components Description
Cisco Nexus 93180YC-EX 10 Gbps & 25 Gbps TOR switches
Cisco Nexus 9332PQ 25 Gbps Aggregation switches
Cisco Nexus 3172TQ 1 Gbps & 10 Gbps Management switches
VxFlex storage traffic 2 x 25 Gbps links
16 Elastic Stack Solution on Dell EMC VxFlex family | 000060Testing and Validation
5 Testing and Validation
This section provides a detailed summary and description of the tests that are performed to validate the
Elastic Stack hosted on VxFlex integrated rack with benchmarking tool Rally.
5.1 Test methodology
Unit testing through System testing methods were used to perform various tests on Elastic Stack. Elastic
team used Rally, a benchmarking tool to run their nightly benchmarking tests.
Rally acts as load generator tool for the Elasticsearch. It also builds, sets up and tears down the Elasticsearch
cluster. This makes Rally a suitable tool to run in a basic environment. Rally also can be used on an existing
Elasticsearch cluster to manage benchmark configurations, compare results, and find any performance
issues. For more information about Rally, see Getting Started with Rally.
For this paper, most prevalent use cases of Elastic were chosen that demonstrates the VxFlex Family is one
of the best platforms to host Elasticsearch.
Use Case 1: This use case is based on the rally-eventdata-track that simulates event-based data use-cases.
This track can be used to create more complex and realistic simulations and benchmarks. More information
about this track use can be found at out at https://github.com/elastic/rally-eventdata-track.
Use Case 2: This use case is based on the webserver logs that are collected from a big event in 1998. These
are real logs and demonstrate elastic search indexes these logs and can be visualized in Kibana. More
information about this track can be found out at https://github.com/elastic/rally-tracks/tree/master/http_logs.
17 Elastic Stack Solution on Dell EMC VxFlex family | 000060Testing and Validation
5.2 Test results
Rally was run on a separate host to benchmark the Elasticsearch cluster hosted on a two-layer VxFlex
cluster. The two-layer configuration included three compute nodes and four storage nodes. The Elasticsearch
cluster includes one Master Node and two Data Nodes. The following figure shows the test set up for Rally:
Test setup
Use Case 1: A challenge ‘lasticlogs-1bn-load from EventData track was chosen to benchmark existing
Elasticsearch cluster. This challenge indexes 1 billion events into several indexes of two primary shards each
and results in around 200 GB of indexes being generated on disk. It can be used give an idea of how max
indexing performance behaves over an extended period. The benchmarking was done using the following
command:
esrally --track=eventdata --target-
hosts=192.168.105.71:9200,192.168.105.72:9200,192.168.105.73:9200 --track-
repository=eventdata --challenge=elasticlogs-1bn-load --pipeline=benchmark-only
18 Elastic Stack Solution on Dell EMC VxFlex family | 000060Testing and Validation
Results for challenge elasticlogs-1bn-load
Parameters Value
Median Throughput for index-append-1000-elasticlogs_q_write 93,673 docs/s
Median Throughput for node_storage 10.02 ops/s
Error Rate 0%
Total time to run the tests 10,844 s
Average CPU Utilization on each Data Node 58%
Memory Utilization on each Data Node 32%
Network Utilization on each Data Node 96,000 KBps
Use Case 2: http_logs track was chosen and run using Rally. This is rather a simple track based on the
webserver logs from the 1998 Football world cup. This track has five shards, and eight bulk indexing requests
are issued when the test is running. The benchmarking was done using the following command:
esrally --track=pmc --target-
hosts=192.168.105.71:9200,192.168.105.72:9200,192.168.105.73:9200
--pipeline=benchmark-only
The key results for track http_logs are represented in the following table:
Results for track http_logs
Parameters Value
Index Throughput 302,283 docs/s
Cumulative indexing time of primary shards 146 min
Total Young Gen GC 140.62 s
Total time to run the tests 90.31 min
Average CPU Utilization on each Data Node 30%
Memory Utilization on each Data Node 25%
For detailed results, see Appendix A.1.
19 Elastic Stack Solution on Dell EMC VxFlex family | 000060Best practices
6 Best practices
The following sections outline the best practices followed in this solution.
6.1 VxFlex integrated rack
• Minimum number of nodes for production workload is seven for a VxFlex two-layer set up.
• Homogenous node types are recommended for predictable performance.
• Maximum number of devices in a storage pool is 300.
• Maximum number of nodes in a protection domain is 32.
• Change the passwords for all default accounts.
• Use secure communication – HTTPS (TCP port 443) to remotely access VxFlex nodes.
• Ensure VxFlex rack is compliant to an RCM.
6.2 VxFlex OS
• Configure high-performance profile for MDM, SDS, and SDC.
• Disable Read Flash cache and Read RAM cache for all flash clusters.
• Check with VxFlex platform team to increase the per device queue length value to 256 per host for
improving the I/O concurrency.
• Ensure that the customize power plan is set to High Performance.
6.3 VxFlex network
• Confirm with VxFlex platform team to enable Jumbo frames for Windows VM, SVM, and at ESXi host.
• Use the Para virtual SCSI (PVSCSI) controller on guest VMs to achieve high performance.
• Enable secure network protocol options only (for example, HTTPS and Secure Shell (SSH).
• Separate management and control traffic from production application traffic. You can provide this
separation by using VLANs.
• Separate VMware vSphere vMotion traffic from production traffic according to VxFlex standard.
6.4 Elasticsearch
• RAM to be 32 GB or 64 GB for a virtual machine.
• JVM Heap size to 50% of RAM, max up to 32 GB. Keep at least 50% RAM available.
• If Elasticsearch must scale vertically, then add more vCPUs.
• Start with a proof of concept, and then test, optimize, iterate, and scale.
• Elasticsearch recommends increasing ulimit of File descriptors to 65536.
• Disable swap file.
• Increase the refresh interval of index from 1 s to more. For example, 20 s. This interval depends
on case to case basis.
• Benchmarking VM where Rally is supposed to run on a different VM which is not part of the
Elasticsearch cluster.
• Link followed for benchmarking https://www.elastic.co/blog/seven-tips-for-better-elasticsearch-
benchmarks.
Note: These best practices were used for the Elasticsearch configuration on a virtual machine and few
choices made were subjective.
20 Elastic Stack Solution on Dell EMC VxFlex family | 000060Deployment of Elastic Stack on VxFlex
7 Deployment of Elastic Stack on VxFlex
Elastic Stack can be deployed on different operating systems. For this white paper, Elastic Stack is deployed
on CentOS 7 VM on Dell EMC VxFlex integrated rack.
Elastic cluster deployment had one Master Node, and two Data Nodes. All these clusters were deployed as
Virtual Machines with CentOS 7 on different ESXi hosts.
1. Installed Elasticsearch 7.3 by following the procedure in the following link
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html
2. Once Elasticsearch is installed, then Logstash was installed using the procedure in the following link
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
3. Kibana was installed to access the dashboards using the procedure in the following link
https://www.elastic.co/guide/en/kibana/current/rpm.html
4. Installed other VMs with Filebeat to push log file into the Elasticsearch. Beats were installed using the
procedure in the following link https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-
installation.html
21 Elastic Stack Solution on Dell EMC VxFlex family | 000060Conclusion
8 Conclusion
The solution provided in this paper, demonstrates how you can deploy Elastic Stack on VxFlex family to meet
performance, resiliency, and scale. In addition, it states the best practices for deployment of VxFlex system
and the Elastic Stack. The solution showcases how VxFlex can address some of the key Elasticsearch
application challenges, for example the benchmark results demonstrate the capability of indexing one billion
documents in approximately 3 hours, highlighting the extreme performance capabilities of VxFlex.
Elastic customers are looking for extreme performance and scalability and flexibility to be deployed in multiple
operating environments such as Windows, Linux, Virtualized or Bare Metal. VxFlex provides the scalable
compute and storage that is required for Elastic stack with utmost flexibility.
Besides, Elastic is a GCP partner, available through the GCP Marketplace and VxFlex is already validated as
the infrastructure for Google Anthos solution.
22 Elastic Stack Solution on Dell EMC VxFlex family | 000060Appendix A: Detailed results
A Appendix A: Detailed results
A.1 Use case1: EventData track
The challenge elasticlogs-1bn-load was run on EventData track and the results are captured in this
section.
VxFlex Dashboard during test run
CPU Utilization (60%) on one of the Data Nodes, the other data node had similar CPU Utilization.
23 Elastic Stack Solution on Dell EMC VxFlex family | 000060Appendix A: Detailed results
Memory Utilization (47%) on Data Node
Network Utilization on Data Node
Iostat on Data Node
24 Elastic Stack Solution on Dell EMC VxFlex family | 000060Appendix A: Detailed results
A.2 Use case1: Http_logs track
The http_logs_track was run with Rally, and the results are captured in this section.
VxFlex Dashboard during test run
CPU Utilization on one of the Data Nodes, the other data node had similar CPU Utilization
25 Elastic Stack Solution on Dell EMC VxFlex family | 000060Appendix A: Detailed results
Memory Utilization on Data Node
Network Utilization
26 Elastic Stack Solution on Dell EMC VxFlex family | 000060Appendix A: Detailed results
Iostat on Data Node
27 Elastic Stack Solution on Dell EMC VxFlex family | 000060Appendix B: Configuration details
B Appendix B: Configuration details
Configuration details
Component Description
VxFlex integrated rack Compute Nodes
3 x VxFlex Nodes (R640 servers):
• VxFlex OS version: R3_0.1
• ESXi version: 6.7 U2
• CPU: 2 x Intel(R) Xeon(R) Gold 6140 CPU @ 2.30 GHz, 18 cores
• Memory: 384-GB RAM ((12 x 32-GB DIMMs)
4 x VxFlex Nodes (R640 servers):
• VxFlex OS version: R3_0.1
• VMware ESXi version: 6.7 U2
• CPU: 2 x Intel(R) Xeon(R) Gold 6126 CPU @ 2.60 GHz, 12 Cores
• Memory: 192-GB RAM (12 x 16-GB DIMMs)
Network 2 NIC cards, each having 2 ports 25 GbE connection
Elastic Stack • Elasticsearch: 7.3.2
• Kibana: 7.3.2
• Logstash:7.3.2
• FileBeat: 7.3.2
• Rally: 1.3
Elasticsearch Master VM • Operating system version: CentOS7
• vCPU: 16
• Memory: 32 GB
• HeapSize: 16 GB
Elasticsearch Data Node VM • Operating system version: CentOS7
• vCPU: 16
• Memory: 64 GB
• HeapSize: 32 GB
Rally VM • Operating system version: CentOS7
• vCPU: 8
• Memory: 16 GB
28 Elastic Stack Solution on Dell EMC VxFlex family | 000060Technical support and resources
C Technical support and resources
C.1 Related resources
See the following referenced or recommended resources that are related to this document:
Note: The following links are open to customers although some may require registration for access.
• VxFlex OS blog: https://blog.dellemc.com/en-us/tag/VxFlex OS/
• Elasticsearch: https://www.elastic.co/products/elasticsearch
• Rally: https://esrally.readthedocs.io
• Elasticsearch Tuning: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-
indexing-speed.html
• Deployment of Elastic Stack: https://www.howtoforge.com/tutorial/how-to-install-elastic-stack-on-
centos-7/
C.2 Additional resources
Referenced or recommended publications:
• Dell.com/support is focused on meeting customer needs with proven services and support.
• Dell EMC VxFlex product overview and White papers on DellEMC.com provides expertise that helps
to ensure customer success on Dell EMC VxFlex family.
29 Elastic Stack Solution on Dell EMC VxFlex family | 000060You can also read
