DELL EMC POWERSCALE ONEFS: MACOS NETWORK STORAGE USER EXPERIENCE AND PERFORMANCE OPTIMIZATIONS
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Best Practices
Dell EMC PowerScale OneFS: macOS Network
Storage User Experience and Performance
Optimizations
Abstract
This document describes performance and user-experience optimizations for
Apple® macOS® 10.14 (Mojave), 10.15 (Catalina), and 11 (Big Sur) with the Dell
EMC™ PowerScale™ OneFS operating system version 9.2.
June 2021
H17613.2
Revisions
Revisions
Date Description
February 2019 Initial release
March 2019 Minor updates
April 2019 Updates on .DS_Store behavior and flow control recommendations
June 2021 Updated for macOS 10.15 and 11. Added recommendations and troubleshooting advice.
Acknowledgments
Special Thanks: Thomas Berglund
The following members of Dell EMC storage engineering staff produced this paper:
Author: Gregory Shiff
Support: Simon Haywood, Rafal Szczesniak
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
This document may contain certain words that are not consistent with Dell's current language guidelines. Dell plans to update the document over
subsequent future releases to revise these words accordingly.
This document may contain language from third party content that is not under Dell's control and is not consistent with Dell's current guidelines for Dell's
own content. When such third party content is updated by the relevant third parties, this document will be revised accordingly.
Copyright © 2019–2021 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners. [6/4/2021] [Best Practices] [H17613.2]
2 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2
Table of contents
Table of contents
Revisions.............................................................................................................................................................................2
Acknowledgments ...............................................................................................................................................................2
Table of contents ................................................................................................................................................................3
Executive summary.............................................................................................................................................................5
Audience .............................................................................................................................................................................5
1 Introduction ...................................................................................................................................................................6
2 Network storage protocols and macOS .......................................................................................................................7
2.1 SMB multichannel in macOS 11.3 ......................................................................................................................7
3 Editing sysctl.conf .........................................................................................................................................................9
3.1 TCP delayed acknowledgment (delayed_ack) ...................................................................................................9
3.1.1 Delayed ack: macOS 10.14 and 10.15 ...............................................................................................................9
3.1.2 Delayed ack: macOS 11 .....................................................................................................................................9
3.1.3 Adjusting delayed ack .........................................................................................................................................9
4 Defaults and com.apple.desktopservices ...................................................................................................................11
4.1 .DS_Store files on network shares ...................................................................................................................11
5 Editing /etc/nsmb.conf ................................................................................................................................................12
5.1 SMB signing......................................................................................................................................................12
5.2 SMB session signing ........................................................................................................................................13
5.3 SMB change notifications .................................................................................................................................13
5.4 Force SMB protocol version .............................................................................................................................14
5.5 Enabling alternate data streams (named streams) ..........................................................................................14
5.6 Apple SMB extensions .....................................................................................................................................15
5.6.1 Disabling Apple extensions: macOS 10.14 ......................................................................................................15
5.6.2 Disabling Apple SMB extensions: macOS 10.15 and 11 .................................................................................15
5.7 Prefer wired connections for SMB multichannel...............................................................................................16
5.8 Example /etc/nsmb.conf ...................................................................................................................................16
6 Flow control, jumbo frames, and macOS ...................................................................................................................17
7 Setting boot arguments for improved network performance ......................................................................................19
7.1 Increasing network memory allocation .............................................................................................................19
7.2 Configuring ServerPerfMode ............................................................................................................................19
7.3 Appending Boot Arguments..............................................................................................................................20
8 Spotlight search of network volumes .........................................................................................................................22
8.1 Enabling Spotlight search .................................................................................................................................22
8.2 Performing Spotlight searches of network volumes .........................................................................................22
3 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2Table of contents
8.3 Performance implications of Spotlight and how to disable ...............................................................................23
9 Using Apple internal 10 GbE NICs with OneFS .........................................................................................................24
10 Recommended settings for PowerScale and macOS ................................................................................................25
11 Troubleshooting Apple performance issues ...............................................................................................................26
11.1 Reduce complexity ...........................................................................................................................................26
11.2 Simplify the client ..............................................................................................................................................26
11.3 Observe macOS network statistics...................................................................................................................27
12 Examining SMB mounts with smbutil .........................................................................................................................28
13 Conclusion ..................................................................................................................................................................29
A Technical support and resources ...............................................................................................................................30
4 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2Executive summary
Executive summary
Many environments and workflows that feature Dell EMC™ PowerScale™ systems as the primary shared
storage platform are highly reliant on systems running Apple® macOS® operating system. This guide provides
performance-tuning and user-experience configurations for macOS 10.14, 10.15, and 11 with the PowerScale
OneFS operating system, version 9.2.
The tunings in this guide provide advice for controlling macOS behavior when interacting with PowerScale
storage. It is meant to allow administrators the ability squeeze the maximum performance out of the macOS
client. For the most part, the settings in this guide will not “fix” an environment where performance is unstable
or unusable. In those cases, there are likely issues going on that are beyond the tuning advice in this guide.
See troubleshooting advice in section 11.
One notable exception is the poor performance of the onboard 10GBASE-T NICs in modern Apple
professional systems. There were significant driver issues with those NICs in macOS releases before macOS
11. When running earlier macOS releases, storage performance of macOS clients connecting to OneFS using
the onboard 10GBASE-T NICs is poor. A setting change in OneFS immediately fixes this issue, and Apple
resolved the driver problems in macOS 11. See section 9 for details.
Audience
This guide assumes the reader is proficient with macOS, the macOS command-line interface, and UNIX style
operating systems in general.
5 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2Introduction
1 Introduction
This paper focuses on the SMB protocol with macOS 10.14, 10.15, and 11 when connecting to PowerScale
OneFS 9.2.
Historically, many optimizations were necessary on macOS to achieve maximum performance from network
storage. These included adjustments such as Send and Receive Windows, Window Scaling, Inflight
Bandwidth Calculation, TCP Slow Start, and others. Fortunately, many such optimizations are no longer
necessary due to improvements in how macOS dynamically autotunes its network performance.
The following performance optimizations and user experience changes are covered in this document:
• Edits to /etc/sysctl.conf,
• Settings in com.apple.desktopservices
- DSDontWriteNetworkStores
• Edits to /etc/nsmb.conf
- SMB signing
- SMB session signing
- SMB notifications
- Force protocol version
- Alternate data streams (named Streams)
- Apple SMB extensions
- Force SMB multichannel to prefer wired connections
• Jumbo Frames
• Memory allocation to network stack, including changes to default nvram boot-args
• Server Performance mode
No single optimization is right for every environment. To take advantage of the tuning parameters outlined in
this paper, testing is critical to determine how each change affects the performance of storage clients and
their unique workloads.
6 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.22 Network storage protocols and macOS
Three network storage protocols are supported by macOS: AFP, SMB, and NFS. There is some support for
FTP with macOS, but that is out of scope for this document.
AFP is unsupported by PowerScale, and since macOS 10.9, AFP is not the default file-sharing protocol for
macOS. As such, it is not covered in this document.
Historically, NFS was the protocol of choice for connecting macOS systems to PowerScale. NFS is no longer
the best choice for macOS. Testing described in this paper concluded that SMB is as performant, and often
more performant, than NFS on macOS.
SMB is the protocol of choice for macOS clients connecting to PowerScale storage. This paper shows that
SMB consistently outperformed NFS on macOS, and describes additional topics:
• Alternate data streams (named streams) support
- PowerScale has robust support for macOS resource fork data stored through alternate data
streams over SMB.
- OneFS does not support macOS clients writing resource fork data to alternate data streams over
NFS.
• Full ACL support
- SMB fully supports file system ACLs for fine-grained control of file access permissions.
- OneFS supports NFSv4 ACLs which can be used on macOS, but can introduce further
complexity, especially in multiprotocol, multiplatform environments (and are not recommended).
• PowerScale support for Apple-specific extensions to the standard SMB protocol
- The Apple SMB implementation (SMBX) has added a few macOS-specific features. OneFS
supports some of these Apple-specific extensions to SMB.
2.1 SMB multichannel in macOS 11.3
SMB multichannel is a major advantage of SMB3. PowerScale OneFS 7.1.1+ supports SMB3 multichannel by
default. Apple has introduced SMB3 multichannel support as of macOS 11.3. Apple describes this
functionality in the following KB article: Configure SMB Multichannel behavior
SMB multichannel provides two features, failover and additional throughput.
Testing for this paper indicates that failover works as expected in macOS 11.3. A macOS client was
configured with two 40GbE links to a OneFS file system (using an ATTO™ Thunderlink™ adapter). While the
macOS client was playing back several streams of video from a OneFS storage volume, one of the network
links was pulled from the switch. The video continued playing smoothly with little to no dropped frames.
However, when that network link was reestablished, the macOS client did not recognize it. The network file
system had to be unmounted, and the user had to log out and back into their account for the macOS client to
reestablish the second link.
Multiple Ethernet connections to the storage only resulted in a small increase of aggregate throughput.
Testing with 2x 10 GbE links to a OneFS file system showed approximately a 10% read improvement and
20% write improvement. Testing with 2x 40 GbE links to the same file system showed little to no read and
7 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2write throughput improvement. SMB multichannel support is new in macOS, and it is likely that over time the
throughput benefits will improve.
The above Apple KB article on SMB multichannel behavior includes several /etc/nsmb.conf settings that
impact how macOS uses this feature. Since SMB multichannel is enabled by default in OneFS, it is
recommended to set macOS to use wired Ethernet links for SMB Multichannel. Using wireless connections for
SMB multichannel could incur significant performance penalties.
To force macOS 11.3 to prefer wired connections for SMB multichannel, add the following line to
/etc/nsmb.conf.
mc_prefer_wired=yes
See Section 5 in this document for more information on creating and editing /etc/nsmb.conf.
8 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.23 Editing sysctl.conf
On macOS, the /etc/sysctl.conf file can be edited to change some of the default settings in the macOS
kernel at boot time. There is also a CLI command, sysctl, that when run as root will make temporary changes
to those same settings. To make the changes persist across reboots, editing the /etc/sysctl.conf file is
required. If the file is not present on a system, a text editor can be used to create it.
3.1 TCP delayed acknowledgment (delayed_ack)
There are many explanations for TCP delayed acknowledgment (delayed_ack), such as the description in the
article TCP Performance problems caused by interaction between Nagle’s Algorithm and Delayed ACK. For
most uses of TCP, delayed acknowledgment is a good thing and makes network communication more
efficient. TCP acknowledgments are added to subsequent data packets. However, in practice on macOS,
particularly with network connections greater than 10 GbE, TCP delayed acknowledgment can cause
performance degradation.
There are four options in macOS for setting the characteristics of delayed_ack:
• delayed_ack=0: Responds after every packet (OFF)
• delayed_ack=1: Always employs delayed_ack; 6 packets can get 1 ack
• delayed_ack=2: Immediate ack after 2nd packet; 2 packets per ack (Compatibility Mode)
• delayed_ack=3: Should auto-detect when to employ delayed ack; 3 packets per ack
3.1.1 Delayed ack: macOS 10.14 and 10.15
The default setting for macOS 10.14 and 10.15 is delayed_ack=3. Testing has shown that in some
environments, particularly where the client is reading and writing simultaneously, delayed_ack=0 is typically
the ideal value. Rendering a video sequence is an example of such behavior.
3.1.2 Delayed ack: macOS 11
In macOS 11, the default value is delayed_ack=0. While it is possible to temporarily change this value with
the below sysctl -w command, the change does not persist across reboots. Values set for delayed_ack in
the sysctl.conf file are ignored in macOS 11. Thus it would appear that Apple has settled on delayed_ack=0
as being the ideal value.
3.1.3 Adjusting delayed ack
To query the current setting of the client, enter the following at the macOS command line:
$ sudo sysctl net.inet.tcp.delayed_ack
Example response:
net.inet.tcp.delayed_ack: 3
To change this setting, enter the following at the macOS command line:
$ sudo sysctl -w net.inet.tcp.delayed_ack=0
Example response:
net.inet.tcp.delayed_ack: 3 -> 0
9 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2To make the setting persist over a reboot (in macOS 10.14 and 10.15), edit the /etc/sysctl.conf file on
the macOS client:
• Create or edit the file at /etc/sysctl.conf
• Add the line net.inet.tcp.delayed_ack=0
10 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.24 Defaults and com.apple.desktopservices
The way macOS Finder treats directories accessed through SMB can be changed by using the CLI
defaults command to update preferences in com.apple.desktopservices.
Apple has outlined this behavior in the support article Adjust SMB browsing behavior in macOS High Sierra
10.13 and later.
Users must log out and log in again for changes to take effect.
4.1 .DS_Store files on network shares
The macOS Finder automatically creates .DS_Store files in every folder it accesses. This file stores
metadata about how to display that directory’s contents. The reading and writing of these files can slow down
performance when listing the contents of directories with high file counts.
It is possible to prevent macOS from creating .DS_Store files on network shares in macOS 10.13. As of
macOS 10.14, it is no longer possible to stop these files from being created on network shares. However, it is
possible to prevent macOS from reading the .DS_Store file before listing a directory’s contents. In the
absence of a .DS_Store file or if reading the .DS_Store file is suppressed (as in 10.14 and higher),
macOS will list the contents of a folder in alphanumeric order only upon initial open. Listing folders in this way
has been shown to significantly reduce the time it takes for macOS to display the contents of directories with
large numbers of files, such as those containing image sequences.
The following macOS CLI command prevents macOS 10.14 and higher from reading .DS_Store files on
network shares:
defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE
After running this command, the user will need to log out and log in for the changes to take effect.
For more information about .DS_Store files, see the .DS_Store Wikipedia article.
11 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.25 Editing /etc/nsmb.conf
The /etc/nsmb.conf file is a plain text file that offers configuration settings for how macOS treats SMB
shares. If this file is not present on a system, it will need to be created with a text editor. The
/etc/nsmb.conf file is divided into sections as outlined on its man page. For these edits, defaults for all
SMB shares need to be set. The first line in the file should be [default] followed by the parameters and their
values:
[default]
parameter=value
parameter2=value2
parameter3=value3
After updating /etc/nsmb.conf, unmount and remount SMB shares from the macOS system for changes to
take effect.
5.1 SMB signing
Packet signing increases the security of SMB connections by helping prevent man-in-the-middle attacks.
Essentially, a digital signature attached to each packet helps the client system confirm that data has not been
tampered with while in transit.
Packet signing is enabled by default in OneFS and is also the default behavior for SMB2 and SMB3
connections on macOS versions 10.11.5 through 10.13.3. Later versions of Apple’s documentation are a little
murky as to when SMB signing is enabled. This archived KB article HT205926 states:
Packet signing for SMB 2 or SMB 3 connections turns on automatically when needed if the server offers
it.
The extra overhead of packet signing can cause significant performance degradation on the latency-sensitive,
high-performance workloads common to PowerScale OneFS. The security risks posed by man-in-the-middle
attacks need to be assessed for each environment. When possible, it is recommended that these workflows
take place on private, secure networks, and that packet signing is disabled on the macOS client.
Testing for this paper showed that SMB signing was automatically enabled on the macOS client when
connecting to a native Apple file server. SMB signing did not get enabled automatically when connecting to
PowerScale OneFS (which also offers SMB signing).
Given the inconsistent nature of the SMB signing behavior in macOS it is recommended to manually disable
the feature using the /etc/nsmb.conf file.
Apple outlines disabling SMB signing in the support article Turn off packet signing for SMB 2 and SMB 3
connections.
To disable SMB signing on macOS, add the following entry to /etc/nsmb.conf:
signing_required=no
After updating /etc/nsmb.conf, unmount and remount SMB shares from the macOS system for the
changes to take effect.
12 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.25.2 SMB session signing
SMB3 introduces security enhancements that help prevent man-in-the-middle attacks during the initiation of
an SMB client connection to an SMB server. SMB session Signing is different from SMB signing which adds
a digital signature to each packet. SMB session signing can be described as a way to protect an SMB
session from being tampering with as it commences. When macOS client systems are bound anonymously to
a directory server, SMB session signing may cause authentication errors when a system tries to connect to an
SMB share.
In situations where proper network credentials are not working from macOS systems running version 10.13+,
disabling SMB session signing may resolve the issue. As with disabling SMB signing, disabling SMB session
signing reduces the security of an SMB connection and is recommended on systems running on private,
secure networks.
Apple describes session signing and how to disable it in the support article If you can't mount SMB share
hosted by a Mac bound to Open Directory.
Add the following line to the /etc/nsmb.conf file:
validate_neg_off=yes
After updating /etc/nsmb.conf, unmount and remount SMB shares from the macOS system for the
changes to take effect.
5.3 SMB change notifications
The SMB server provides macOS with updates or changes to mounted file shares. SMB notifications provided
by PowerScale provide the macOS client with this information in a OneFS environment. With a busy file
share, this may result in the macOS Finder refreshing itself frequently and users may notice their file listings
fluctuating or Finder windows reset to the top level directory while browsing. To avoid this behavior, it is
possible to disable the Finder from requesting SMB change notifications.
Applying this setting can break workflows that require SMB notifications for folder listings to be current. For
example, the collaborative Productions feature in Adobe™ Premiere Pro™ relies on change notifications to
be active on all clients participating in the Production. Disabling change notifications can also lead to data
corruption and other issues where multiple users are accessing the same files and directories.
Therefore, as with all macOS optimizations, great care and testing is required to make sure that enabling this
setting does not cause workflow problems.
To disable SMB notification, add the following line to the /etc/nsmb.conf file:
notify_off=yes
After updating /etc/nsmb.conf, unmount and remount SMB shares from the macOS system for the
changes to take effect.
13 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.25.4 Force SMB protocol version
Under certain circumstances, it may be desirable to force macOS to connect through a particular version of
SMB. The following edits to /etc/nsmb.conf force a particular SMB version. Note that macOS uses a
binary bitmap to specify which version of SMB to use. Add protocol_vers_map= and the appropriate value
to force protocol types. See the following examples.
# Protocol version is specified using binary bitmap
# 1 => 0001 => SMB 1 only
# 2 => 0010 => SMB 2 only
# 3 => 0011 => SMB 1 or 2
# 4 => 0100 => SMB 3 only
# 6 => 0110 => SMB 2 or 3
# To Force SMB 3 only:
protocol_vers_map=4
# To Force SMB 2 or 3 only
protocol_vers_map=6
5.5 Enabling alternate data streams (named streams)
PowerScale OneFS supports storing Apple resource fork data transparently in an alternate data stream, also
called a named stream. This prevents the creation of AppleDouble files, increases macOS performance
when writing to OneFS, and helps prevent file corruption.
To better understand this feature, refer to the following background information.
When macOS stores a file, that file is composed of two forks. There is a resource fork which contains
extended attributes about the file itself, such as the file’s icon. There is also a data fork which contains the file
data itself. When macOS writes data to local file systems, such as HFS+ or APFS-formatted volumes, the
resource fork data is stored in an alternate data stream and is not visible to users or systems accessing those
files; only the file itself is shown with all of its associated metadata.
When macOS writes to a storage system that does not support alternate data streams, the resource fork data
still needs to be stored somewhere. In this case, macOS stores the resource fork data in AppleDouble files.
These are files beginning in ._ followed by the file name. For example, saving a file called file.dat results in
two files being created on the file system: file.dat and ._file.dat. This degrades performance because it
increases the file operations on the target storage system. File corruption can also become an issue if a non-
macOS system overwrites or deletes the AppleDouble files.
As mentioned previously in this section, OneFS supports storing resource fork data in an alternate data
stream which prevents the creation of AppleDouble files. It is the default behavior of macOS 10.5+ to make
use of an alternate data stream on an SMB server if the feature is available (as it is on OneFS). When writing
through NFS, macOS will revert to storing resource fork data as AppleDouble files (another good reason to
always use SMB with macOS).
OneFS intelligently manages resource fork data so even non-Apple systems connected by NFS can interact
with files created or used by SMB-connected macOS systems with no corruption of the resource fork data and
no need for AppleDouble files.
14 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2While it is not necessary to force macOS to use alternate data streams (the default since macOS version
10.5), some administrators feel more comfortable explicitly enabling this feature in /etc/nsmb.conf by
adding the following line:
streams=yes
The following archived support document outlines this functionality: Mac OS X v10.5, v10.6: About named
streams on SMB-mounted NAS, Mac OS X, and Windows servers; "-36" or "-50" alerts may appear.
5.6 Apple SMB extensions
Apple has added several macOS-specific features to their implementation of SMB. Primarily, these features
address the handling of file metadata stored in alternate data streams as described previously. OneFS
supports these features:
ReadDirAttr: This feature changes how macOS handles reads of file metadata stored in alternate data
stream when listing the contents of large directories. Finder info, access rights, and resource fork size are
returned more efficiently for the files in the directory.
OsxCopyFile: With the SMB2 protocol, Microsoft implemented server-side optimizations when copying files
between directories on the file share. The extension introduced by Apple ensures that all Apple-specific file
metadata is properly copied along with the file itself. The copy process is also simplified as it is performed in
one request as opposed to splitting the requests into logical chunks. When performing a large server-side
copy, the macOS Finder ignores updates from the server. During these copies, the Finder copy progress bar
does not update and may appear to have stopped responding. When the copy finishes, the progress bar
snaps to complete.
In versions 10.13+, macOS has changed how it interacts with SMB servers to pull directory file listings.
Unintuitively, the Apple SMB extensions can slow performance in workflows that have directories with high file
counts or elaborate project files (such as complicated video edit projects). Disabling Apple extensions in
OneFS significantly improves performance in those situations.
5.6.1 Disabling Apple extensions: macOS 10.14
It is possible to disable Apple-specific SMB features on macOS clients running 10.14. To disable the macOS
client from using these extensions, add the following line to /etc/nsmb.conf:
aapl_off=true
5.6.2 Disabling Apple SMB extensions: macOS 10.15 and 11
It is not possible to disable Apple-specific SMB extensions in macOS 10.15 and 11. Instead, the Apple
extensions must be disabled on the OneFS storage cluster itself. To disable Apple SMB extensions in OneFS,
run the following command from the OneFS CLI:
isi_gconfig \
registry.Services.lwio.Parameters.Drivers.srv.smb2.EnableAppleExtensions=0
The following screen captures show the share information before and after Apple extensions were disabled
on the macOS client. Notice that before they are disabled, macOS identifies PowerScale as a fully native file
share, while afterwards, PowerScale shows up as a more generic share.
15 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.25.7 Prefer wired connections for SMB multichannel
macOS 11.3 adds SMB3 multichannel support as outlined in Apple KB HT212277 and Section 2.1 of this
document. Apple outlines a couple of settings that can be changed with /etc/nsmb.conf. Apple outlines
how to disable multichannel all together and force multichannel to prefer wired connections. Due to the
possible negative performance impacts of using Wi-Fi with SMB multichannel, it is recommended to force
macOS to prefer wired connections.
To force macOS 11.3 to prefer wired connections for SMB multichannel, add the following line to
/etc/nsmb.conf.
mc_prefer_wired=yes
5.8 Example /etc/nsmb.conf
The following shows an example of the /etc/nsmb.conf. Notice that the disabling of Apple-specific SMB
extensions is present in the file but has been commented out.
16 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.26 Flow control, jumbo frames, and macOS
Using jumbo frames (9,000 MTU) can increase macOS SMB performance on network connections greater
than 1 GbE. However, implementation of jumbo frames requires more than just setting a NIC port to use
9,000 MTU. The macOS client, network switch, and PowerScale storage must all be properly configured.
Keep in mind that changes to network settings on the switch, clients, and storage will cause connections to
drop. As such, these changes should not be undertaken during active production.
If jumbo frames are used, the network on which the macOS client connects to the PowerScale storage should
be dedicated to production. The network should not be used for non-storage traffic (such as email or Internet
browsing). Essentially, MTU sizes should be consistent on the network and that network should not be
cluttered with non-storage-related traffic. There should be no 1500 MTU traffic on the storage network running
at 9000 MTU.
The ports on the network switch itself must be configured to use jumbo frames, which means setting those
ports to greater than 9,000 MTU. On the switch used for testing in this document, the ports were set to
9,412 MTU. See to the documentation for the specific switch in use for optimal port MTU settings. The extra
overhead makes room for any additional data attached to the network packets, such as VLAN header
information. In testing, if the switch ports were set to exactly 9,000 MTU, macOS clients stopped responding
when connecting to network shares, even on a flat network. Administrators should refer to their switching
documentation for proper port configuration, since some switches may transparently add additional MTU to a
port and others need to be explicitly configured.
Another critical setting for using macOS with jumbo frames is to enable flow control on the switch. Flow
control for both RX and TX should be enabled on the network switch for all ports that are connected to both
the macOS client and PowerScale storage. In absence of switch side flow control, macOS jumbo frame
performance was lower than the standard 1,500 MTU. Other platforms tested were not as sensitive to flow
control. It made little to no difference in Windows or Linux performance but was required on macOS to
achieve good performance using jumbo frames.
Enabling flow control on switch ports in use by macOS and PowerScale is recommended, even when
running at 1,500 MTU. PowerScale storage is fast and can overflow the macOS network buffers causing
dropped packets and throughput degradation.
Once the ports are properly configured on the network switch, the external network ports on the PowerScale
storage must be set for 9,000 MTU. See the OneFS documentation on the PowerScale Info Hubs for setting
the external network ports on the PowerScale storage to 9,000 MTU.
Now that OneFS and the switch are set to jumbo frames, the NIC ports used by the macOS client can be
configured. There are two settings that must be configured on the macOS client: Jumbo (9000) MTU and
Duplex: full-duplex, flow-control. Setting Duplex to full-duplex with no flow-control resulted in poor
performance, even when flow control was disabled on the switch. To realize the benefit of jumbo frames on
macOS, flow control and MTU settings must be properly configured on both the client and the network switch.
17 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2In the macOS UI, this can be configured in System Settings > Network. Select the appropriate NIC port,
select Advanced Settings, and select the Hardware tab.
In the hardware tab, configure the port Manually and set the Duplex and MTU controls.
18 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.27 Setting boot arguments for improved network performance
Several boot arguments (boot-args) can be configured to increase the amount of system resources allocated
to network performance in macOS. These include increasing network buffer size and enabling
ServerPerfMode.
Configuring boot-args must be done from the macOS terminal while the system is booted into Recovery
Mode. Apple describes the process of booting to recovery mode in the article Mac startup key combinations.
To view currently set boot-args, run the following terminal command:
nvram boot-args
Setting boot-args explicitly from the command line will overwrite existing boot-args. Apple provides a sample
code snip-bit in the KB article about ServerPerfMode that uses sed and cut to append existing boot-args to
newly set ones. The example in that article is for ServerPerfMode, and can be adapted for other settings such
as ncl.
Client systems have a finite amount of resources. Allocating additional system resources to the network stack
means that there is less RAM and CPU available for other purposes. Administrators need to be aware of the
implications changing the defaults (as outlined below), and be prepared to roll the changes back if the client
systems becomes constrained as a result.
7.1 Increasing network memory allocation
Increasing the amount of memory that macOS allocates for network buffering can increase throughput for
network connections greater than 1 GbE.
Depending on the total system memory and the network connection type, various values can be set. 512 MB
may be the logical maximum.
Once booted into recovery mode, launch the Terminal application and use the following commands to add a
boot argument. Note this will overwrite any existing boot-args:
nvram boot-args="ncl=262144"
This setting is the number of 2 KB clusters assigned to the network buffer and is calculated as follows:
(262,144 * 2) / 1,024 = 512 MB
After setting this nvram boot argument, reboot the macOS system for the changes to take effect. Once
rebooted, run the following terminal command to verify the boot argument:
nvram boot-args
To remove this boot-args setting, run the following terminal command:
nvram -d boot-args
7.2 Configuring ServerPerfMode
ServerPerfMode updates various sysctl settings in macOS to increase system resources available to the
network stack. While the name suggests that this setting is aimed at Apple systems being used as servers,
testing has shown that client systems also benefit by enabling ServerPerfMode. In one particular example, a
19 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2video file in Canon’s™ C200 codec at 50 MBps (400mbps) would not play back smoothly until
ServerPerfMode was enabled. This was true despite both the Apple hardware and PowerScale storage
providing more than enough performance and other heavier codecs playing fine.
Apple does not provide specific documentation about all the changes enabled by ServerPerfMode. Only
documentation about how to enable it: Turn on performance mode for macOS Server
There is some discussion online about some of the sysctl changes that take place, such as this discussion on
apple.stackexchange.com. The exact changes may largely depend on the total amount of RAM in the Apple
system. One way to check would be to run: sysctl -a from the CLI before and after the change, dumping
the output to a text files to compare.
To enable ServerPerfMode, boot the Apple client system into Recovery Mode. Once in Recovery Mode, run
the following command from the Terminal application (existing boot arguments will be overwritten):
nvram boot-args=”serverperfmode=1”
7.3 Appending Boot Arguments
Apple provides terminal commands to add new boot-args without overwriting existing boot-args in this KB
article about enabling ServerPerfMode. Below are those commands adapted for both ServerPerfMode and
ncl. All commands assume the Apple system is booted into Recovery Mode.
To set both ServerPerfMode and ncl simultaneously, overwriting existing boot-args. Run the following
command from macOS Terminal:
nvram boot-args=”serverperfmode=1 ncl=262144”
To remove all boot-args simultaneously:
nvram -d boot-args
To enable ServerPerfMode and keep existing boot-args:
nvram boot-args="serverperfmode=1 $(nvram boot-args 2>/dev/null | cut -f 2-)"
To disable ServerPerfMode and keep existing boot-args:
nvram boot-args="$(nvram boot-args 2>/dev/null | sed -e \
$'s/boot-args\t//;s/serverperfmode=1//')"
To update ncl setting and keep existing boot-args:
nvram boot-args="ncl=262144 $(nvram boot-args 2>/dev/null | cut -f 2-)"
To remove ncl setting and keep existing boot args:
nvram boot-args="$(nvram boot-args 2>/dev/null | sed -e \
$'s/boot-args\t//;s/ncl=262144//')"
To update both ServerPerfMode and ncl simultaneously while keeping existing boot-args:
nvram boot-args="serverperfmode=1 ncl=262144 $(nvram boot-args 2>/dev/null \
| cut -f 2-)"
To remove both ServerPerfMode and ncl settings simultaneously while keeping other boot-args:
20 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2nvram boot-args="$(nvram boot-args 2>/dev/null | sed -e \
$'s/boot-args\t//;s/serverperfmode=1//;s/ncl=262144//')"
21 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.28 Spotlight search of network volumes
Spotlight search of network volumes (including PowerScale OneFS) is supported in macOS 10.14, 10.15 and
11. However there are some significant performance implications that need to be considered. Each individual
Apple client maintains its own local database of the network share. Having multiple Apple systems
simultaneously index a file system will degrade performance for all users until the index is complete.
To verify status of Spotlight search of the PowerScale OneFS volume, run the following terminal command
from the Apple client: mdutil -s /path/to/network/share/
8.1 Enabling Spotlight search
To enable Spotlight search on the network share, follow this process from the Apple client:
From the macOS UI, verify that the network volume is not listed:
System Preferences → Spotlight → Privacy
Next from the macOS terminal disable Spotlight for the network share with:
sudo mdutil -i off /path/to/network/share
Erase any existing Spotlight index for that share:
sudo mdutil -E /path/to/network/share
Reenable Spotlight indexing for the share:
sudo mdutil -i on /path/to/network/volume
If enabling Spotlight search on the network volume fails, examine the permissions on the share, they may
need to be made more permissive.
Once Spotlight indexing has begun, keep an eye on the mdworker process running on the Apple client.
The Apple client system stores a local index of the network volume on its internal storage. Separate indexes
are maintained for each network storage login and IP address. As a result, it is conceivable that multiple
indexes of the same network volume could be resident on a single Apple client. These indexes can grow large
depending on the size and complexity of the volume being scanned. The Spotlight index files for network
storage are located here:
/var/db/Spotlight-V100/LocalVolumes/
Occasionally, users may find that Spotlight search of a network volume stops returning fresh results. The only
fix for a “broken” Spotlight index is to disable search, erase the database, and reenable search, as outlined
above.
8.2 Performing Spotlight searches of network volumes
Spotlight search of network volumes is slightly different from how macOS users are used to using Spotlight.
• Spotlight search results for network volumes do not show up using the macOS key command
“Command+Space bar”.
22 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2• Spotlight search results for network volumes do not show up in searches using the Magnifying Glass
icon in the upper right of the macOS Desktop.
• Spotlight search results only show up when searching from the search field in the upper right of a
macOS Finder window with the network share selected.
Below is a screenshot of a Spotlight search for the term “autodesk” on the network volume “Space”.
8.3 Performance implications of Spotlight and how to disable
The macOS process which maintains the Spotlight index, mdworker, is aggressive in building its database.
Mdworker will begin creating this database quickly after being enabled. Even a few macOS systems could
cause significant storage traffic and degrade performance for other users. Spotlight search on Apple clients
should be initially enabled during off hours and in small batches in order to minimize performance impact
during initial database build.
When the OneFS file system is used for latency sensitive workflows, such as video editing, Spotlight search
should be disabled on network storage. For fast file system search in those environments, alterative tools
such as Dell DataIQ™ maybe more appropriate.
Spotlight indexing of a network volume can be disabled in several ways.
From the macOS UI, add the network volume to the list at:
System Preferences → Spotlight → Privacy
From the macOS command line:
mdutil -i off /path/to/network/volume
Or
mdutil -d /path/to/network/volume
23 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.29 Using Apple internal 10 GbE NICs with OneFS
Before macOS 11, there were significant driver issues with the integrated 10GBASE-T NICs on professional
Apple hardware. A change in OneFS to protect against certain kinds of Denial of Service (DOS) attacks
resulted in poor performance for Apple systems using integrated 10GBASE-T NICs to connect to OneFS.
Fortunately, the macOS driver issues are resolved in macOS 11.
However, for users of macOS 10.14 and 10.15, updating the PowerScale cluster setting of
net.inet.tcp.reass.maxqueulen to 2048 resolves these performance problems. This change is unnecessary for
macOS 11 users. The procedure for making the change is outlined in the following Dell KB article (login
required): Sysctl Security Change Can Affect TCP Performance On Degraded Networks OneFS v8.1.0+
Increasing net.inet.tcp.reass.maxqueulen can make the PowerScale cluster more vulnerable to certain DOS
attacks, but if the PowerScale cluster is on a secure, private network, the risk is minimal.
To view the current setting of net.inet.tcp.reass.maxqueuelen, on the PowerScale cluster run the following CLI
command (default value is 100):
sysctl net.inet.tcp.reass.maxqueuelen
Temporarily change the value on a particular node (the node to which the macOS 10.14 or 10.15 client is
connected):
sysctl net.inet.tcp.reass.maxqueuelen=2048
Have the clients check the network to confirm if the performance issue has been resolved. The change should
be immediate, the macOS clients do not need to unmount/remount the volume.
If the issue is resolved, change the value for the whole cluster on reboot by editing the following file:
vi /etc/mcp/defaults/sysctl.conf
If the sysctl exists, edit that line. If it is not in the file, add these lines to the bottom of the file:
net.inet.tcp.reass.maxqueuelen=2048
Push the change to all nodes in the cluster:
isi_sysctl_cluster net.inet.tcp.reass.maxqueuelen=2048
24 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.210 Recommended settings for PowerScale and macOS
The following recommendations are a good starting point. Having a second macOS system with a fresh install
of the operating system (and no optimizations) to compare against is ideal.
On the PowerScale OneFS cluster:
• Disable Apple extensions
o It is counterintuitive, but the current state of Apple extensions causes large directory listings
and opening complex projects to be slow. Disabling Apple extensions on the PowerScale
cluster disables them for all clients, avoiding the individual operating system behavior of
10.14 vs 10.15+.
• Increase the net.inet.tcp.reass.maxqueuelen setting to 2048.
o This change assumes the PowerScale cluster is on a private secure network. While
technically this change is unnecessary for macOS 11, at least at the time of this writing, most
macOS users are still running 10.14 or 10.15. Updating this setting does not negatively
impact macOS 11.
On the Apple client system:
• Boot into Recovery Mode, enable ServerPerfMode and increase network buffer allocation.
o The default settings of the macOS network stack are not optimized for network storage
performance. Making these changes means the Apple system will devote more resources to
networking.
• Use the defaults command to force macOS to ignore.DS_Store files on network shares
o In combination with disabling Apple extensions on the PowerScale cluster, this greatly
improves listing large directories in the macOS Finder.
• If connecting by 10 GbE or higher, change delayed_ack to 0 (macOS 10.14 and 10.15)
o The default setting of 3 can degrade performance with networks greater than 1 GbE
o macOS 11 defaults to delayed_ack=0
• Configure the following changes to /etc/nsmb.conf
o Disable SMB signing
▪ It is the default, but given the performance implications, best to hard code this setting
o Disable SMB Session signing
▪ Reduce incidence of password and connectivity problems
o Force SMB protocol version to SMB3 only
▪ SMB3 is the preferred protocol for macOS. If for some reason the client cannot
connect by SMB3, it may be better to have the mount fail and trigger further
investigation.
o Set SMB3 multichannel to prefer wired connections
▪ Having a macOS system try to connect by Wi-Fi when a wired connection is available
will negatively impact performance.
• Disable Spotlight search on the network share
o Unless necessary, Spotlight search can lead to performance degradation on network shares
with large numbers of macOS clients.
On the network switch:
• Enable flow control both RX and TX for all macOS clients and PowerScale front-end ports.
o This is a good idea for both 1500 MTU and 9000 MTU networks when used with macOS.
25 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.211 Troubleshooting Apple performance issues
Administrators often arrive at this tuning guide when trying to solve macOS performance problems. The
settings outlined in this document can be the difference between average macOS performance and
exceptional performance. However the recommendations in this paper are not intended to “fix” environments
where macOS systems are in an unusable state. An exception being the change of
net.inet.tcp.reass.maxqueuelen for older versions of macOS, which does make a huge difference.
The good news is that a fresh install of macOS 10.14 and higher connected to a PowerScale cluster works
reasonably well with no tuning whatsoever. This means browsing the file system, opening, reading, and
writing files and playing many video formats.
The question then arises of where to start when macOS systems are slow to browse the file system, will not
open or save files, and video playback is dropping frames. The trick here is to reduce complexity in the
system, get back to a known good state and work backwards to the user’s end state. Determining where the
problem lies (storage, network, client) is nontrivial in the best of circumstances. The following suggestions
have helped administrators track down their problems and answer those challenging questions.
11.1 Reduce complexity
The network itself is often the biggest source of complexity in a production environment. Removing as much
of the network as possible between the client and storage is a good first step.
• Connect the macOS client directly into one of the PowerScale front-end network ports.
o Otherwise connect the macOS system into the same switch as the PowerScale front-end
ports.
o If plugged through a switch, verify that flow control (RX and TX) is enabled for all macOS and
storage ports
• Verify that the macOS system is on the same subnet and VLAN as the PowerScale and that no
routing is required for the client system to access the storage.
• Use standard 1500 MTU size, and that ping times are between the macOS and PowerScale are low
(and conversely).
o 9000 MTU can provide extra performance in macOS for the most demanding workflows.
Unfortunately, 9000 MTU often causes more problems than it solves due to MTU mismatch
on the network. Having all devices work well using 1500 MTU first and then experimenting
with 9000 MTU is the recommended path.
If good performance is achieved with these steps, try advancing one network hop at a time to determine
where functionality breaks down and efforts must be focused.
11.2 Simplify the client
Did the system ever work as intended? Or did something change to “break” performance? Reimaging the
macOS client back to a known good state is another thing to try:
• Using Recovery Mode, reimage the client back to a base install of macOS
• Reinstall any third-party network drivers
o Verify the latest driver version and macOS compatibility. Apple is notorious for breaking
compatibility with new versions of macOS.
• Verify speed of Thunderbolt cables and which Thunderbolt ports are in use.
26 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2o Thunderbolt 1 and Thunderbolt 2 cables look physically identical but provide different types of
performance. Use of Thunderbolt 1 cables with Thunderbolt 2 compatible devices will not
realize the full performance capabilities of PowerScale or the Apple client.
▪ Thunderbolt speeds can be checked from Apple menu → About this mac → System
Report → Thunderbolt
o This ATTO technical bulletin shows which Thunderbolt ports to use on various Apple
hardware to avoid disconnects and performance issues: ATTO Technical Bulletin January
2020
11.3 Observe macOS network statistics
From the macOS terminal, the nettop command is an easy way to observe periodically updated network
statistics (default interval 1 second). The feedback provided by nettop can be helpful to understanding more
about how the macOS client is communicating with network storage during certain application workloads. The
following command show macOS TCP behavior for the kernel_task process, which is the parent process for
all SMB connections on TCP port 445:
nettop -m tcp -p kernel_task -J interface,bytes_in,bytes_out,rx_dupe,rx_ooo,re-
tx,rtt_avg,rcvsize,unacked,tx_win
Key metrics to look at, when analyzing network statistics with nettop are:
• rtt_avg – Round-Trip Time average (latency)
o Should be as low as possible. If there are spikes above 10 milliseconds, bandwidth and
performance will be negatively impacted. TCP and especially SMB traffic over TCP are
sensitive to latency.
• rx_ooo – Out of order TCP packets
o Can be a sign of storage server not being able to reassemble packets.
• re-tx – TCP Retransmits
o Can be a sign of latency or packet loss
• rx_dupe – TCP Duplicate ACKs
o The higher the latency is, there more Duplicate ACKs will appear.
High latency or large amounts of out of order packets will degrade performance. Either of these issues could
be the results of operating system or driver issues on the Apple client, network issues between the client and
storage, or latency issues within the storage file system on a software or hardware-related layer like
intermittently busy or queued up disks.
To confirm if the latency issues are related to SMB read or write requests, you can try capturing
Wireshark/tcpdump trace files with a capture filter for port 445, while reproducing issues. When the capture is
stopped, look at Statistics > Service Response Time > SMB2, and look out for high MAX SRT values.
Details about Wireshark and tcpdump are out of scope for this paper.
Before going down the rabbit hole of analyzing TCP packets, try to simplify the environment (see Section 11.1
above). Reducing the amount variables between the client and the storage system is a good starting point.
27 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2You can also read
