RAZORLAN TECHNICAL DOCUMENTATION - LAST MODIFIED: 19 JUL 2016 - RAZORLAN STATUS
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
RazorLAN technical documentation
Last modified: 19 Jul 2016Table of Contents
1 Network infrastructure........................................................................................................5
1.1 Internal......................................................................................................................... 5
1.1.1 OSI layer 1 (Physical)..........................................................................................................5
1.1.2 OSI layer 2 (Data link).........................................................................................................5
1.1.3 OSI layer 3 (Network).........................................................................................................7
1.1.4 Wireless connectivity.........................................................................................................8
1.1.5 DHCP...................................................................................................................................8
1.1.5.a TNet............................................................................................................................................. 8
1.1.5.b HNet............................................................................................................................................ 9
1.1.6 Router configuration..........................................................................................................9
1.1.6.a CM............................................................................................................................................... 9
1.1.6.b Zeus........................................................................................................................................... 10
1.1.6.c Hermes....................................................................................................................................... 11
1.2 External......................................................................................................................11
1.2.1 Domains...........................................................................................................................11
1.2.1.a razorlan.info............................................................................................................................... 11
1.2.1.b debark-vlieland.nl...................................................................................................................... 12
1.2.1.c jazzuzzy.nl................................................................................................................................... 12
1.2.2 Uplink...............................................................................................................................13
1.2.3 Quality of Service.............................................................................................................14
2 Procedures......................................................................................................................... 15
2.1 Emergency procedures...............................................................................................15
2.1.1 Unexpected downtime.....................................................................................................15
2.1.1.a No internet access...................................................................................................................... 15
2.1.1.b Service(s) unavailable................................................................................................................ 16
2.1.1.c Server offline.............................................................................................................................. 17
2.1.1.d Domain name failure................................................................................................................. 17
2.1.1.e External IP change...................................................................................................................... 17
2.1.1.f Denial of service attack............................................................................................................... 17
2.1.1.g Loss of passthrough on CM........................................................................................................ 17
2.1.1.h Post-incident.............................................................................................................................. 18
2.1.2 Hardware failure...............................................................................................................18
2.1.2.a Defective router......................................................................................................................... 18
2.1.2.b Defective switch......................................................................................................................... 18
2.1.2.c Secondary storage failure (non-RAID)......................................................................................... 18
2.1.2.d Secondary storage failure (RAID)................................................................................................ 18
2.1.3 Security breach.................................................................................................................18
2.1.3.a Compromised OpenVPN keys..................................................................................................... 18
2.1.3.b Compromised root CA................................................................................................................ 19
2.1.3.c Compromised certificate............................................................................................................ 19
2.2 Routine....................................................................................................................... 19
2.2.1 Calendar...........................................................................................................................19
2.2.2 Patches.............................................................................................................................20
2.2.3 Virus scans........................................................................................................................21
2.2.4 Data backups....................................................................................................................21
2.2.4.a Neon system backup.................................................................................................................. 21
\ 22.2.4.b Webserver backup..................................................................................................................... 22
2.2.5 Maintenance announcements.........................................................................................23
2.2.6 Stability review.................................................................................................................23
2.2.7 RAID integrity check.........................................................................................................23
2.2.8 UPS test............................................................................................................................23
2.3 Extraordinary maintenance........................................................................................23
2.3.1 Temporary TeamSpeak 3-server migration......................................................................23
2.3.2 Recovery from wireless network failure on TNet............................................................24
2.3.3 Recovery from crashed or terminated http daemon on Zeus router..............................25
2.3.4 OpenVPN certificate renewal...........................................................................................25
2.3.5 Adding an OpenVPN client...............................................................................................25
2.3.6 Allowing an OpenVPN client internet access via RazorLAN.............................................26
2.3.7 Creating a new certificate authority................................................................................27
2.3.8 Renewing the root certificate with the same key............................................................28
2.3.9 Renewing certificates.......................................................................................................28
2.3.10 Securing the certificate authority's private keys............................................................29
2.3.11 Deploying RDP certificates.............................................................................................29
2.3.12 Manual NetXMS server restart......................................................................................31
2.3.13 Changing a well-known hostname on TNet...................................................................31
2.3.14 Opening ports on Neon using iptables..........................................................................32
2.3.15 Accessing persistant Zeus logs.......................................................................................32
2.3.16 Setting up tamper-resistance for network shares.........................................................32
3 Server configuration..........................................................................................................33
3.1 Specifications.............................................................................................................33
3.1.1 Neon.................................................................................................................................33
3.1.2 Cobalt...............................................................................................................................33
3.1.3 Flerovium.........................................................................................................................34
3.2 Services......................................................................................................................34
3.2.1 Remote access..................................................................................................................34
3.2.1.a SSH............................................................................................................................................. 34
3.2.1.b RDP............................................................................................................................................ 36
3.2.2 HTTP.................................................................................................................................36
3.2.2.a Lighttpd...................................................................................................................................... 36
3.2.2.b ISS.............................................................................................................................................. 36
3.2.3 hMailServer......................................................................................................................36
3.2.3.a Client configuration.................................................................................................................... 36
3.2.3.b Server configuration................................................................................................................... 37
3.2.3.c External blacklists....................................................................................................................... 37
3.2.4 TeamSpeak 3....................................................................................................................37
3.2.5 OpenVPN..........................................................................................................................38
3.2.6 SMB/CIFS..........................................................................................................................39
3.2.7 Printing.............................................................................................................................40
3.2.8 Optional services..............................................................................................................40
3.2.8.a NetXMS...................................................................................................................................... 40
3.2.8.b FTP............................................................................................................................................. 40
3.2.8.c Minecraft................................................................................................................................... 40
3.2.8.d SWAT 4....................................................................................................................................... 41
3.3 Miscellaneous............................................................................................................. 41
\ 33.3.1 Cobalt RAID configuration................................................................................................41
3.3.1.a PERC 4/Di (Front panel SCSI drives)............................................................................................ 41
3.3.1.b Silicon Image 3124 (5,25'' bay SATA drives)................................................................................41
4 Appendix............................................................................................................................ 42
4.1 CentOS 6.5 install on Neon.........................................................................................42
4.2 Installation router Zeus..............................................................................................45
4.3 Neon iptables.............................................................................................................46
4.4 OpenVPN sample client configuration file..................................................................47
4.5 OpenVPN 'vars' script.................................................................................................47
4.6 ISP customer support.................................................................................................48
\ 41 Network infrastructure
1.1 Internal
1.1.1 OSI layer 1 (Physical)
Physical connections and locations of routers, switches, servers and oddities are drawn below.
Buildings are marked with thick lines and street addresses in the format: Country-Postal code-
Street number. Internal locations are grouped with coloured dashed lines.
1.1.2 OSI layer 2 (Data link)
Permanent hosts on the RazorLAN network have the following media access control addresses in
use. Henceforth, for brevity and ease of reading, a unique network interface identifier will be used
to refer to these MAC-addresses.
• Zeus
◦ br0 (LAN): B0:48:7A:E8:16:C6
▪ ath0: B0:48:7A:E8:16:C8
▪ eth0
▪ vlan1
▪ wifi0
◦ vlan2 (WAN): B0:48:7A:E8:16:C7
Network infrastructure\Internal 5• CM
◦ eth0: 90:6E:BB:D6:39:DF
◦ ath0: 64:27:37:26:17:0F
◦ WAN-router: 90:6E:BB:D6:E9:DD
◦ modem: 90:6E:BB:D6:E9:DC
◦ MTA: 90:6E:BB:D6:E9:DE
• Neon
◦ eth0: 1C:6F:65:43:1E:E1
◦ tun0: 8E:5C:B4:79:D8:D3
• Cobalt
◦ eth0: 00:06:5B:F7:D8:39
• Flerovium
◦ tun0: 00:FF:44:E3:69:5C
◦ eth0: 00:1B:38:56:0C:64
◦ ath0: 00:13:E8:8C:98:E1
• HPEA6F43
◦ ath0: 44:1E:A1:E9:6D:4C
• Hermes
◦ br0 (LAN): 00:1C:F0:F2:49:9E
▪ eth0
▪ ath0
◦ WAN: 00:1C:F0:F2:49:9F
• X-Book:
◦ ath0: 1C:4B:D6:4F:B9:45
• Kars-Hidding
◦ eth0: 00:90:F5:D9:A5:D5
• Alies-PC
◦ eth0: 00:19:B9:1F:4C:E7
• PS3
◦ ath0: 00:19:C5:31:D3:F1
• TV-Samsung
◦ eth0: 90:F1:AA:15:D5:E8
Below is an OSI layer 2 schematic showing some of RazorLAN's (semi-)permanent hosts.
Network infrastructure\Internal 61.1.3 OSI layer 3 (Network)
RazorLAN's internal network consists of the following subnets:
Name Range Accessibility
TNet 192.168.120.0/24 Wired and wifi (AP: TNet)
HNet 192.168.119.0/24 Wired and wifi (AP: HNet)
TNet_Modem 192.168.178.0/24 Via TNet
OpenVPN 10.120.120.0/24 OpenVPN
Network infrastructure\Internal 7Its most important hosts are shown in the diagram below. Arrows point to a DHCP-pushed
standard gateway.
'TNet' also serves as the default subnet for guests and non-production machines. 'HNet' is
available for compatibility with legacy wireless devices and has an access point with more liberal
security settings. 'TNet_Modem' was a subnet for administrative and emergency access to CM and
its own uplink, but is no longer in use as of May 1st 2015 due to the router on that subnet being
disabled.
1.1.4 Wireless connectivity
RazorLAN operates two wifi networks: TNet and HNet. Details listed below:
SSID BSSID AP Mode Tx Power Gain Channel Width Security
B0:48:7A: WPA2 Personal,
TNet Zeus NG 17 dBmW 3 dBi 1 (2412 MHz) 20 MHz
E8:16:C6 AES
00:1C:F0:F WPA Personal,
HNet Hermes NGB High 2 dBi 6 (2437 MHz) 20 MHz
2:49:9E TKIP & AES
64:27:37: WPA2 Personal,
TNet_Modem1 CM NGB 14 dBmW 0 dBi 11 (2462 MHz) 20 MHz
26:17:0F TKIP & AES
1.1.5 DHCP
1.1.5.a TNet
DHCP services on TNet are authoratitively provided by the Zeus router using DNSMasq. The range
192.168.120.64 to 192.168.120.127 is reserved for dynamic leases, which last by default for 1440
minutes. On the DD-WRT web configuration, the option 'Used Domain' (Services/Services/Services
1 No longer exists. Entry listed as historic record.
Network infrastructure\Internal 8Management/DHCP server) must be set to 'LAN & WLAN'. Local name resolution will fail otherwise; DNS will append the WAN domain and e.g. attempt to resolve 'neon.dynamic.ziggo.nl'. Relevant configuration files for DHCP and DNS from the DD-WRT router are added below. /etc/hosts 127.0.0.1 localhost 192.168.120.1 Zeus.r.lan 192.168.120.2 Neon.r.lan 192.168.120.6 HPEA6F43.r.lan 192.168.120.32 Flerovium.r.lan 192.168.120.3 Cobalt.r.lan 192.168.120.7 Hermes.r.lan 192.168.120.8 TV-Samsung.r.lan /tmp/dnsmasq.conf interface=br0 resolv-file=/tmp/resolv.dnsmasq domain=r.lan dhcp-leasefile=/tmp/dnsmasq.leases dhcp-lease-max=69 dhcp-option=lan,3,192.168.120.1 dhcp-authoritative dhcp-range=lan,192.168.120.64,192.168.120.127,255.255.255.0,1440m dhcp-host=1C:6F:65:43:1E:E1,Neon,192.168.120.2,86400m dhcp-host=44:1E:A1:E9:6D:4C,HPEA6F43,192.168.120.6,43200m dhcp-host=00:1B:38:56:0C:64,Flerovium,192.168.120.32,1440m dhcp-host=00:06:5B:F7:D8:39,Cobalt,192.168.120.3,86400m dhcp-host=00:1C:F0:F2:49:9F,Hermes,192.168.120.7,43200m dhcp-host=90:F1:AA:15:D5:E8,TV-Samsung,192.168.120.8,240m stop-dns-rebind /tmp/resolv.dnsmasq nameserver 208.67.222.222 nameserver 8.8.4.4 nameserver 208.67.220.220 1.1.5.b HNet DHCP services on HNet are authoratitively provided by Hermes. The range 192.168.119.64 to 192.168.119.254 is reserved for dynamic leases, which last for 1440 minutes. 1.1.6 Router configuration 1.1.6.a CM The cable modem is a Ziggo-provided, wireless-enabled modem-router-VoIP combination, manufactured by Ubee, type EVW3200. The modem-router is configured as a network bridge; the router functionality is disabled. Hardware version: 3.9.2 Network infrastructure\Internal 9
Firmware2 (as of 2015-04-29): 9.9.6007
Serial number: EVW320B50167065
1.1.6.b Zeus
The primary router is a TL-WR1043ND manufactured by TP-Link. It has a 400 MHz Atheros AR9132
rev 2 (0xb9) CPU and 32768 kB internal memory.
Hardware version: 1.8
Firmware: DD-WRT v24-sp2 (08/07/10) std – build 14896
List of important port forwards
Application Target Protocol Ports in Ports out
HTTP_Main Neon TCP 80 80
HTTP_Alt Cobalt TCP 8080 80
SSH_Neon Neon TCP 22 22
RDP_Cobalt Cobalt TCP 3389 3389
OpenVPN Neon UDP 1194 1194
TS3_All Neon UDP 9987-9990 9987-9990
TS3_Files Neon TCP 30033 30033
TS3_SQ Neon TCP 10011 10011
IMAP Cobalt TCP 143 143
SMTP_Main Cobalt TCP 25 25
SMTP_Alt Cobalt TCP 25025 25
Minecraft Neon TCP 25565-25568 25565-25568
SWAT4 Flerovium UDP 10480-10483 10480-10483
Custom scripts
Custom scripts are permanently stored on a USB flash drive attached to Zeus. On startup, the script
'/mnt/bootstrap.sh' is called, from which other scripts may be invoked that need to run
automatically. To ensure all services have finished starting, the bootstrap script starts after 125
seconds, by virtue of the following command stored in the startup variable in Zeus's nvram:
(sleep 125; /mnt/bootstrap.sh) 2> /var/log/customscripts &
All custom scripts that produce (error) messages to be logged are hereby redirected to a file in
'/var/log' which resides in volatile memory. This protects the USB flash drive against failure due to
excessive writing.
Important custom scripts are archived below.
/mnt/bootstrap.sh
#!/bin/sh
2 Firmware updates for CM are deployed automatically by Ziggo.
Network infrastructure\Internal 10touch /mnt/tempfile
/mnt/killigmprt.sh
/mnt/killigmprt.sh
#!/bin/sh
killall -SIGHUP igmprt
if [ $? -eq 0 ]
then
cp $(dirname "$0")/igmpproxy.conf.bak /tmp/igmpproxy.conf
igmprt -d /tmp/igmpproxy.conf
#This should fail, no interfaces enabled in config.
else
echo "$(date) Could not kill igmprt, no such process running." >&2
fi
1.1.6.c Hermes
The secondary router is a D-Link DIR-615. Its purpose is to provide internet access to legacy wifi
clients that are unable to connect to TNet due to its stricter security requirements, and to seperate
wired guest clients from the main network.
Hardware version: B2
Firmware: 2.25, 2008-05-16
Hermes is gateway to the following permanent hosts:
• PS3
1.2 External
1.2.1 Domains
The following domain names are assigned to or controlled by RazorLAN:
• razorlan.info
• debark-vlieland.nl
• jazzuzzy.nl
Additionally, RazorLAN provides hosting services for the following domains:
• piripiri-emmen.nl
1.2.1.a razorlan.info
Domain name registrar: GoDaddy (http://www.godaddy.com)
Username: 39855998
Date of expiry: November 14th 2016
Name servers: ns1.afraid.org
ns2.afraid.org
Network infrastructure\External 11ns3.afraid.org
ns4.afraid.org
DNS provider: FreeDNS (http://freedns.afraid.org)
Username: Marcks
Entries: razorlan.info A 217.122.60.241
razorlan.info MX 10:mail.razorlan.info
dsl.razorlan.info A 80.60.193.155
cm.razorlan.info A 217.122.60.13
mail.razorlan.info A 217.122.60.241
1.2.1.b debark-vlieland.nl
Domain name registrar: Domeinwinkel (http://www.domeinwinkel.nl)
Username: mark@razorlan.info
Date of expiry: November 15th 2015
Name servers: ns1.afraid.org
ns2.afraid.org
ns3.afraid.org
DNS provider: FreeDNS (http://freedns.afraid.org)
Username: Marcks
Entries: debark-vlieland.nl A 217.122.60.241
debark-vlieland.nl MX 10:mail.debark-vlieland.nl
www.debark-vlieland.nl URL http://debark-vlieland.nl
1.2.1.c jazzuzzy.nl
Domain name registrar: Domeinwinkel (http://www.domeinwinkel.nl)
Username: mark@razorlan.info
Date of expiry: December 5th 2015
Name servers: ns1.afraid.org
ns2.afraid.org
ns3.afraid.org
DNS provider: FreeDNS (http://freedns.afraid.org)
Network infrastructure\External 12Username: Marcks
Entries: jazzuzzy.nl A 217.122.60.241
jazzuzzy.nl MX 10:mail.jazzuzzy.info
www.jazzuzzy.nl URL http://jazzuzzy.nl
1.2.2 Uplink
ISP: Ziggo
Subscription: Alles-in-1 Plus
Maximum throughput: 150 Mb/s download, 15 Mb/s upload
Autonomous System Number: 9143
Gateway: D97A3C01.cm-3-3a.dynamic.ziggo.nl (217.122.60.1) at 00:14:F1:E8:E0:D9
CMTS 1: emn-rc0001-ubr014-te3-0-0-202.core.as9143.net (213.51.138.75)
CMTS 2: emn-rc0001-ubr014-te1-0-0-201.core.as9143.net (213.51.138.59)
CMTS subscriber-side: 10.210.0.1
WAN IP: 217.122.60.241
WAN subnet: 255.255.254.0
Typical DOCSIS status is listed below.
Downstream Bonded Channels
Channel Lock Status Modulation Frequency Power SNR Microreflections
1 Locked QAM256 241000000 Hz -2.8 dBmV 41.4 dB -29 dBc
2 Locked QAM256 249000000 Hz -3.5 dBmV 40.8 dB -33 dBc
3 Locked QAM256 257000000 Hz -4.0 dBmV 40.8 dB -33 dBc
4 Locked QAM256 265000000 Hz -4.2 dBmV 40.8 dB -31 dBc
5 Locked QAM256 273000000 Hz -2.9 dBmV 41.0 dB -31 dBc
6 Locked QAM256 281000000 Hz -3.0 dBmV 40.9 dB -34 dBc
7 Locked QAM256 289000000 Hz -3.3 dBmV 40.8 dB -34 dBc
8 Locked QAM256 297000000 Hz -3.5 dBmV 40.7 dB -32 dBc
Upstream Bonded Channels
Channel Lock Status US Channel Type Symbol Rate Frequency Power
1 Locked ATDMA 5120 Ksym/sec 58800000 Hz 44.5 dBmV
2 Locked ATDMA 5120 Ksym/sec 52000000 Hz 44.7 dBmV
3 Locked ATDMA 5120 Ksym/sec 44500000 Hz 44.2 dBmV
4 Locked ATDMA 5120 Ksym/sec 36000000 Hz 44.2 dBmV
Network infrastructure\External 131.2.3 Quality of Service
Because both personal computers for everyday use and production servers are connected through
the same uplink, RazorLAN uses a packet scheduler to prevent intensive use of the available
bandwith from imperiling the stability of services. The Quality of Service is managed on the WAN
port of router Zeus and extends to the entire network.
The DD-WRT firmware offers two packet schedulers: Hierarchical Fair Service Curve (HFSC) and
Hierarchical Token Bucket (HTB). RazorLAN uses the latter.
It is vital that QoS takes effect before the bandwidth limit is reached, so that the bottleneck is at
the packet scheduler where it can be controlled. To ensure this happens, the uplink and downlink
settings are set below the theoretical subscription rates. Headroom is granted to allow for natural
deviations between subscription rates and actual available rates.
Subscription (kb/s) QoS (kb/s) Headroom (kb/s) Potency
Uplink 15000 14000 1000 93.33%
Downlink 150000 146500 3500 97.67%
Quality of Service rules can be set by using MAC, netmask and services as criteria and take
precedence in that order. Traffic can be prioritised according to the table below.
Priority Minimum allocation Maximum allocation Mask
Exempt 60% 100% 100
Premium 25% 100% 10
Express 10% 100% 20
Standard 5% 100% 30
Bulk 1% 100% 40
The mask value is used internally to keep track of connections with different priorities and is useful
for actively monitoring QoS classification, for example through # cat
/proc/net/ip_conntrack.
The following rules are in effect:
1C:6F:65:43:1E:E1 Express
teamspeak Express
http Express
bittorrent Bulk
Note that internet traffic from and to Neon's eth0 interface always takes Express priority.
Therefore, this interface is not to be used for non-critical applications that may consume all
available bandwith. Should such applications be required to run on Neon, the QoS setting must be
temporarily adjusted or a different interface (e.g. OpenVPN) must be used.
QoS can be gracefully disabled and reenabled through SSH. To stop: # /usr/sbin/svqos stop 0
`get_wanface` 0 0, to start: # /usr/sbin/svqos `nvram get wshaper_downlink`
Network infrastructure\External 14`nvram get wshaper_uplink` `get_wanface` `nvram get wan_mtu` 0.
2 Procedures
2.1 Emergency procedures
2.1.1 Unexpected downtime
2.1.1.a No internet access
In case internet access appears unavailable from or to all major hosts on the RazorLAN network,
consult the following flowchart for an expedient recovery. Do not use this procedure if:
• only one host's internet access has ceased,
• the downtime is due to planned maintenance, or
• the downtime is caused by an internal power failure.
Procedures\Emergency procedures 15Start
Attempt wireless
FAILURE SUCCES Power off router DD-WRT
ping 192.168.120.1 connection to ping 192.168.178.1
for ten seconds
TNet_Modem
SUCCES FAILURE
FAILURE ping 192.168.120.1
Full router reboot
SUCCES
FAILURE Power off cable modem Lights US and DS NO External error, wait for
ping 192.168.178.1
for ten seconds green up? modem to reconnect
SUCCES YES
Online
SUCCES
SUCCES
SUCCES
ping razorlan.info ping external host ping 192.168.178.1
FAILURE FAILURE FAILURE
Full router reboot
FAILURE Check DOCSIS
ping external host
(http://192.168.178.1)
SUCCES
Check DNS
Domain failure
208.67.222.222
checklist
8.8.4.4
NO
Diagnosed?
YES
Solve if possible
2.1.1.b Service(s) unavailable
Services being unavailable can be a symptom of general infrastructure failures. Check whether
other emergency procedures apply.
Procedures\Emergency procedures 162.1.1.c Server offline
2.1.1.d Domain name failure
2.1.1.e External IP change
If the external IP of the Zeus router has changed to a link-local address or to an address in
TNet_Modem's range, execute procedures for 'Loss of passthrough on CM' (2.1.1.g) instead.
If reobtaining the external IP address previously assigned to RazorLAN appears impractible after a
full router reboot, all references to the IP address must be updated. Those references are listed
below, in order of priority.
1. DNS records for all domains, see documented domain list (1.2.1).
2. Privileged IP ranges in hMailServer (3.2.3).
3. Common third-party whitelists for preventing spam from end-user dynamic IPs (3.2.3.c).
4. NetXMS monitoring configuration (3.2.8.a).
5. Neon's '/etc/hosts.allow' (3.2.1.a).
6. OpenVPN client configuration files which contain hardcoded IP addresses to allow VPN
connections to be established without or before the use of DNS.
7. Update the OpenVPN sample client configuration file (4.4).
8. Clients' local references to IP addresses of RazorLAN's game servers and third party
references thereof.
2.1.1.f Denial of service attack
2.1.1.g Loss of passthrough on CM
If the passthrough functionality on the cable modem stops working, the Zeus router can no longer
obtain its WAN IP from the internet service provider at which RazorLAN's services are available.
Symptoms may include:
• router Zeus obtains a link-local address, a private range address or no address at all on its
WAN interface;
• RazorLAN servers are available locally, but not from the internet;
• CM no longer displays 'Router and Wireless disabled' on configuration page;
• internet connectivity remains.
With up-to-date firmware, the passthrough functionality can not be reinstated locally. Try the
following until availability of RazorLAN services is restored.
1. Power cycle the cable modem and renew the DHCP lease of router Zeus's WAN interface.
Procedures\Emergency procedures 172. Use the cable modem's built-in router. Configure Zeus as cascaded router with a static lease
and set up port-forwarding on CM. Contact the ISP to reinstate bridge mode. As an
alternative to port-forwarding, set up a DMZ.
If the external IP address has changed temporarily, update DNS records for all domains (1.2.1) and
flush DNS caches on routers. Leave other references to the previous address intact.
2.1.1.h Post-incident
As soon as the incident has been resolved and at least all essential services have resumed
operation, execute the following post-incident procedures.
• Document any maintenance performed in the maintenance log;
• write a brief statement on the RazorLAN status page (accessible via http://razorlan.info,
stored on Neon at 'srv/www/htdocs/index.php') containing:
◦ date and time of the incident,
◦ how users may have been affected, by listing affected services/machines and their
downtime,
◦ the cause of the incident,
◦ measures taken to prevent reoccurrence if applicable,
◦ whether or not normal operation has been fully restored;
• log in on Cobalt via RDP, access the NetXMS console and mark errors and warnings as
terminated or resolved.
2.1.2 Hardware failure
2.1.2.a Defective router
2.1.2.b Defective switch
2.1.2.c Secondary storage failure (non-RAID)
2.1.2.d Secondary storage failure (RAID)
2.1.3 Security breach
2.1.3.a Compromised OpenVPN keys
Log in on Neon and shut down the OpenVPN server by running sudo service openvpn stop.
• If a client's key files have been compromised, revoke his certificate through the script
below, replacing 'john_doe' with the common name of the compromised party. Root
privileges and a root environment are required. The 'revoke-full' command returns error 23
Procedures\Emergency procedures 18(verification certificate failed) to indicate the revocation has succeeded.
$ su -
# cd /etc/openvpn/easy-rsa
# . ./vars
# ./revoke-full john_doe
• If the server's key files have been compromised, the entire infrastructure needs to be
rebuilt. Revoke all certificates in the key directory. They can be displayed using: # ls
/etc/openvpn/easy-rsa/keys/*.crt and subsequently revoked using the script above.
The file 'crl.pem' will have been created or updated accordingly at this point. Copy it to a publicly
readable location (default '/etc/openvpn/easy-rsa') and make sure the server configuration file
incorporates this certificate revocation list with the option crl-verify crl.pem. Restart the
OpenVPN service.
In creating new certificates to replace the revoked ones, keep the following security concerns in
mind:
• create the new certificates in a secure environment, e.g. if the server keys were
compromised through an exploit that allowed an attacker remote access, fix this bug first;
• use different passwords to secure the client key files and distribution;
• make sure the distribution method is still secure and
• make sure the client can receive new secret files on a non-compromised machine.
2.1.3.b Compromised root CA
2.1.3.c Compromised certificate
2.2 Routine
2.2.1 Calendar
2016-03-16 VPN server certificate expires
2016-04-21 Cobalt's https certificate expires
2016-04-21 Cobalt's mail certificate expires
2016-04-22 Flerovium's RDP certificate expires
2016-04-22 Cobalt's RDP certificate expires
2016-07-15 RazorLAN root certificate expires
2016-11-14 Domain name registration for 'razorlan.info' expires
2016-11-15 Domain name registration for 'debark-vlieland.nl' expires
2016-12-05 Domain name registration for 'jazzuzzy.nl' expires
2016-12-19 Two week warning for FreeDNS account dormant status
2017-01-10 VPN root certificate expires
Procedures\Routine 192.2.2 Patches
Security updates and fixes for Microsoft operating systems are regularly released on every second
Tuesday of the month and automatically pushed via Windows Update. None of our Windows
servers are configured to install these updates automatically; they require manual installation.
After every 'Patch Tuesday', check whether updates are ready to be installed on the following
machines:
• Flerovium
• Cobalt
On Flerovium, updates classified 'important' can be installed at any time and at the discretion of
the user or administrator. By default, optional updates are not marked for installation, but they can
be marked as such after a brief inspection of their purpose. Rebooting the system, if required, may
be done at the earliest convenience.
On Cobalt, all important updates must be installed, with the exception of updates that may
interfere with security or operational concerns. Optional updates require a brief inspection of their
purpose. Updates that are not to be installed should be permanently hidden. If none of the
updates marked for installation require the system to reboot, the updates may be applied at the
earliest convience. Otherwise, execute the checklist below.
• Wait for off-peak hours
• Initiate downloading and installation of selected updates
• Wait for Windows Update to demand a system restart
• Ensure all running processes can be terminated safely (e.g., no backup operations, large file
transfers or virus scans in progress)
• Prepare for immediate on-site access in case reboot fails
• Start sending ICMP echo requests to Cobalt to monitor network availability
• Click 'restart now'. Alternatively, execute: shutdown /r /t 30 /d p:02:17 to reboot in
30 seconds for planned operating system hot fix.
• Wait for updates to install; if downtime exceeds 10 minutes, monitor progress on-site and
stand-by for emergency procedures: server offline
• Wait for server to go online
• Log in on Cobalt as Remote to view possible (error) messages from Windows Update and to
manually start required services and processes
• Verify all services on Cobalt have resumed normal operation
After completion of the checklist, run Windows Update again. Follow-up updates may be available.
If more updates that require a reboot are to be installed, postpone the procedure for at least 24
hours.
Updates for CentOS running on Neon do not cause downtime and can be installed at any time, but
Procedures\Routine 20at least once per Patch Tuesday. Root privileges are required, but are, for the purpose of applying updates, available through sudo to all members of the group 'remote', i.e. anyone with SSH access can update the system with the command: sudo yum update. 2.2.3 Virus scans Neon is not protected by a virus scanner. Flerovium is protected by Avira Antivir, but its configuration is not within the scope of this document. Cobalt is protected by the on-demand scanner ClamWin, scheduled to perform a full scan of the C:\-drive weekly at Monday 18:30. The virus database is updated daily at 21:16. Program updates need to be installed manually; ClamWin will notify when an update is available. Other drives on Cobalt are used for long-term storage; its contents should never be executed. They are therefore not part of the weekly scan. Positive hits will trigger an e-mail alert to admin@razorlan.info. ClamWin will log in as mark@razorlan.info on localhost:25, using hMailServer also running on Cobalt. Files will never be automatically deleted or quarantined. Reports of positive hits should be investigated manually. If the investigation confirms the presence of infected files, they should be deleted promptly, including possible backup copies of the file. 2.2.4 Data backups 2.2.4.a Neon system backup An image of Neon's root partition (/dev/sda2) is to be made every few months to allow for an expedient recovery after a hard disk failure or security breach. The image can be created at any time without interfering with production. The partition is 4.2 GB in size. Duplication, compression and storage will require a few minutes. Compression if the image is normally handled by gzip with default settings (LZ77-coding, compression strength 6). The output filename adheres to the following pattern: Neon_root_partition_yyyy-mm-dd.img.gz. To safeguard the image against tempering and accidental loss, it is stored on a different server, Cobalt. The code below serves as an example of the process. WARNING: The example script will generate errors if executed verbatim. Changing paths is mandatory. # dd if=/dev/sda2 of=/tmp/root.img $ gzip /tmp/root.img $ smbclient –-user samba //Cobalt/Disk1 smb: \> lcd /tmp smb: \> cd images smb: \images\> put root.img.gz Neon_root_partition_yyyy-mm-dd.img.gz smb: \images\> q $ rm /tmp/root.img.gz Despite the uncompressed image being a temporary file, the '/tmp' directory cannot be used; it resides on the partition being copied. The insufficient space error can be avoided by compressing the image before writing it to the file system, but this is less efficient. Use a different partition to store the temporary files instead. Use $ df --si to verify it has sufficient free space. The root partition is the only partition that needs to be backed up in full as part of the system Procedures\Routine 21
backup. Partition '/dev/sdb1' is used exclusively for non-system storage. Its data is part of a
different backup plan. The home partition (/dev/sda3) does contain files that need to be backed up
as part of the system backup, but also contains user data that are being backed up elsewhere.
The list below shows directories on the home partition. It is subject to change and may be
incomplete. Always check for important entries manually using # du --summarize –-si
/home/*.
Directory Approximate size Part of system backup?
/home/lighttpd 12 GB NO
/home/lost+found 16 KB NO
/home/Marcks 1.2 GB NO
/home/minecraft 5.2 GB NO
/home/samba 2.7 GB NO
/home/teamspeak3 35 MB YES
Execute the following commands to archive, compress and store the necessary directories from
the home partition.
# tar -cf /tmp/home.tar /home/directory1 /home/directory2 …
$ gzip /tmp/home.tar
$ smbclient –-user samba //Cobalt/Disk1
smb: \> lcd /tmp
smb: \> cd images
smb: \images\> put home.tar.gz Neon_home_partition_yyyy-mm-dd.tar.gz
smb: \images\> q
$ rm /tmp/home.tar.gz
Secure storage of images
To futher safegaurd the image against accidental corruption or deletion, once uploaded to Cobalt,
its file permissions are changed not to allow modification without administrative rights. Log in to
Cobalt, browse to the image file and in its advanced file permissions, create an entry for the user
'Everyone' and deny the following permissions:
• Create files / write data
• Create folders / append data
• Write attributes
• Write extended attributes
• Delete
2.2.4.b Webserver backup
Infrequently, contents of the main webserver at Neon are copied to the webserver at Cobalt, for
both data redundancy and availability of websites when Neon is down. Some content may be
excluded from these backups.
Procedures\Routine 22Backups are made from Cobalt. Log in via RDP, open a non-elevated command prompt, make sure
'\\Neon\webserver' is accessible (enter credentials if prompted). Directories that contain live
websites are mirrored using the command below:
robocopy \\Neon\webserver C:\inetpub\wwwroot *.* /MIR /R:2 /W:10 \
/XD aspnet* dir1 dir2 … /LOG:C:\inetpub\wwwroot\robocopy.log /TEE
The flag '/XD' excludes directories that do not carry live websites and need not be mirrored. It also
surpresses warnings due to aspnet directories missing on Neon; these directories are created by IIS
and only exist on Cobalt.
Next, all remaining files under 5 MiB in size can be copied automatically using:
robocopy \\Neon\webserver C:\inetpub\wwwroot *.* /E /XO /R:2 /W:10
/MAX:5000000 /XD aspnet* /LOG+:C:\inetpub\wwwroot\robocopy.log /TEE
Files larger than 5 MiB in size of which backups should be made, are to be copied manually using
the file explorer. Disk space on Cobalt's system drive (where the webserver resides) is not sufficient
to mirror the entire content of Neon's webserver.
2.2.5 Maintenance announcements
2.2.6 Stability review
2.2.7 RAID integrity check
2.2.8 UPS test
2.3 Extraordinary maintenance
2.3.1 Temporary TeamSpeak 3-server migration
The following procedure installs a temporary TeamSpeak server on Cobalt as a switchover so that
the regular service may be stopped on Neon with minimal inconvenience to clients. Because the
switchover server is not maintained, it is recreated from a fresh install on every use and deleted
afterwards.
Download the latest Win-32 version of the TeamSpeak 3 server software from:
http://www.teamspeak.com/?page=downloads. The software is portable; extract the archive to a
temporary location on Cobalt where the software will run and launch the executable. Server
administrator credentials will be generated. Keep these at hand. By default, the server will run on
'ts3server://Cobalt:9987'. Firewalls and NAT are pre-configured to allow remote access through
'ts3server://razorlan.info:9989'.
Attempt to log in on the newly created server. Gain server admin privileges by redeeming the
corresponding privilege key. Configure the virtual server to fit the needs of a temporary
switchover:
• create semi-permanent channels to accommodate all clients;
• set a descriptive welcome message identifying the virtual server as a temporary
switchover; and
Procedures\Extraordinary maintenance 23• perform optional configuration depending on the nature of the switchover.
Open a TeamSpeak client and connect in different tabs to all virtual servers involved in the
procedure to monitor the migration. Make sure to target RazorLAN's WAN address or domain
name rather than specific local server names, e.g. 'ts3server://razorlan.info' instead of
'ts3server://neon'. Update RazorLAN status information sources with a statement on the
procedure and an availability table.
If the regular servers are currently in use by clients, inform them of the migration through a text
message sent as 'serveradmin'. Clients will be redirected automatically. An example script for
Server Query (Neon:10011) is provided below.
login serveradmin password
serverlist
use sid=1
sendtextmessage targetmode=3 target=1 msg=Hello\sWorld!
logout
For more information on Server Query, consult the 'TeamSpeak 3 Server Query Manual', available
here: http://media.teamspeak.com/ts3_literature/TeamSpeak%203%20Server%20Query
%20Manual.pdf.
Log in on the Zeus router web configuration and go to the NAT pages. Rules forwarding TeamSpeak
ports to the switchover are already in place; enable them and save the page without applying.
Disable the forwarding rules for the server to be stopped and save the page. Apply the settings and
monitor the migration through the TeamSpeak client.
Once the regular service is back online, migrate back using the following steps:
• Test availability of the regular services.
• Monitor all services in individual tabs, including 'ts3server://razorlan.info' rather than
specific local server names.
• Inform clients, if applicable, of the remigration via a textmessage as serveradmin using
Server Query.
• Log in on the Zeus router, revert changes to the NAT pages described before and apply.
• Wait for clients to be redirected to the regular server.
• Update RazorLAN status information sources.
• Stop the switchover server.
• Delete the switchover server installation.
2.3.2 Recovery from wireless network failure on TNet
WARNING: Only attempt this procedure from a host with a stable, fully wired connection to the
router.
WARNING: This procedure may cause temporary instability of connections unaffected by the
wireless network failure.
Procedures\Extraordinary maintenance 24Log into the router Zeus. Temporarily disable the radio in the 'Wireless\Basic Settings' tab by switching the 'Wireless Network Mode' to 'Disabled'. Apply the new settings, refresh the page and restore 'Wireless Network Mode' to the documented setting. Apply the new settings. Wait for about ten seconds, during which some connections maintained by the router may be dropped momentarily. 2.3.3 Recovery from crashed or terminated http daemon on Zeus router This maintenance is to be performed to recover from a situation where the router Zeus does not appear to respond to requests on port 80, i.e. the configuration pages are inaccessible through the web interface, but is otherwise functioning normally. This means the router must respond appropriately to ICMP echo requests and provide internet access to hosts on TNet. Do not use this procedure if the router also shows symptoms not related to a crashed http deamon, unless: • access to the configuration pages through the web interface is immediately required, or • off-peak hours allow for a soft reboot of the router. Log into the router as root via SSH. Confirm the 'httpd' process is running by examining the output of 'ps'. If it is running, stop it by excuting the command 'stopservice httpd'. Subsequently, restart the http daemon with the command 'startservice httpd'. Confirm the process is running correctly through the 'ps' command and by visiting 'http://192.168.120.1' from another host. 2.3.4 OpenVPN certificate renewal When an OpenVPN client certificate expires, new key files must be generated and deployed. Log in on Neon as root and delete the client's certificate, the client's certificate signing request and the client's key. Then, remove the client's entry from the database, located at '/etc/openvpn/keys/index.txt'. Proceed as if adding a new OpenVPN client. When the Certificate Authority's root certificate expires, stop the OpenVPN service and follow the instructions from the official OpenVPN documentation: http://openvpn.net/index.php/open- source/documentation/howto.html#pki. Unlike creating or renewing client certificates, this procedure allows the use of the './clean-all' script; the CA's old certificate is no longer trusted, invalidating all client certificates signed by the CA up to now. New keys and certificates must be created and deployed for all clients. 2.3.5 Adding an OpenVPN client Commands used during this procedure are listed in a code block at the end of this section. Further documentation is available at 'http://openvpn.net/index.php/open- source/documentation/howto.html'. Generating key files Log in on Neon. Confirm the time and date are correct using the date command. An incorrect timestamp will write invalid values for 'valid from' and 'expires on' on the new certificates. Next, we need access to the keys directory. The use of sudo will not suffice as we need to set environmental variables from the './vars' script using 'source', a shell built-in rather than a Procedures\Extraordinary maintenance 25
command that can be elevated with sudo. Start a shell as root.
Go to the easy-rsa directory ('/etc/openvpn/easy-rsa') and source the './vars' script, which has
been configured at the time of installing OpenVPN. Per this configuration file, keys expire after 700
days. For clients with temporary access or key files that are not password protected, this value may
need to be overruled, e.g. by executing export KEY_EXPIRE=2.
Do not follow the OpenVPN documentation's recommendation of executing ./clean-all and
./build-ca! These commands are for first time use only and will clear the entire key directory,
permanently breaking all key files created and distributed thusfar.
Build the key using the client's common name as an argument for the build-key script. Use 'build-
key' for non-password protected keys, 'build-key-pass' otherwise. Most fields have a default value
set via the './vars' script. Use these with exception of the common name (defaults to argument of
the key building script), the name (enter full name of client) and if applicable, the key file password
(PEM pass phrase). Leave the challenge password for the key signing empty. Sign the certificate
and commit.
Preparing key files for distribution
The key is now valid for connecting to the OpenVPN network. Copy them from the keys directory
to the client using a secure channel. The default procedure is to store the necessary files in a
password protected .zip-archive and copy that to a publicly available directory on the RazorLAN
webserver. The client will require three files: 'ca.crt', 'client.crt' and 'client.key'. The .crt-files are
cryptographically considered public, the .key-file is secret and must not be distributed beyond the
intended client. Should the key file leak, revoke the certificate immediately.
WARNING: When using a password protected zip-archive to distribute secret keys, do not use the
--password flag of the zip command; the password will be maintained in plain text in the user's
command history, e.g.: '/root/.bash_history'. Use the non-echoing, interactive flag '--encrypt'.
Optionally, the zip-archive may include a pre-made configuration file. A sample configuration file is
also available at 'http://razorlan.info/ovpn/sample_config.ovpn'. The client will have to alter three
lines to point to the correct path of the key and certificate files. When creating a configuration file
specifically for a client, note that Windows and Unix operating systems use different conventions
for newlines.
$ date
$ su -
# cd /etc/openvpn/easy-rsa
# source ./vars
# export KEY_EXPIRE=700
# ./build-key-pass john_doe
# cd keys
# zip --encrypt /srv/www/htdocs/ovpn/john_doe.zip \
ca.crt john_doe.crt john_doe.key
# logout
2.3.6 Allowing an OpenVPN client internet access via RazorLAN
This procedure enables OpenVPN clients to route their internet traffic through RazorLAN. The
OpenVPN server provides network address translation. Through NAT, VPN clients can initiate
connections to other machines on the RazorLAN network as if the traffic originated from Neon.
Procedures\Extraordinary maintenance 26Understand the security implications, primarily:
• clients may harm the trustworthyness of RazorLAN by abusing the internet connection, e.g.
to send spam, leading to the blacklisting of our addresses;
• clients may circumvent whitelists that include the local 192.168.120.0/24 block;
• clients may harm the stability of the network by circumventing QoS on TNet, on which
Neon takes a high priority.
Log in on Neon and confirm the client has a fixed OpenVPN IP address as defined in
'/etc/openvpn/ipp.txt'. Traffic forwarded by NAT towards this address is automatically accepted by
the firewall via a permanent entry in iptables. Non-ICMP-traffic originating from this address must
be allowed manually. The example below assumes eth0 to be the server's main interface, tap0 to
be the OpenVPN interface and 10.120.120.254 to be the client's address.
# iptables -I FORWARD -i tap0 -o eth0 -s 10.120.120.254/32 -j ACCEPT
Additional criteria may be added, for instance to selectively allow http with '-p tcp --dport 80'.
Note that DHCP requests are not automatically accepted in that case. The following entry may
need to be added:
# iptables -I FORWARD -i tap0 -o eth0 -s 10.120.120.254/32 -p udp -–dport 53
-j ACCEPT
The firewall changes take immediate effect and are valid until the iptables service is restarted.
The client configuration file will need to include the line 'redirect-gateway def1' to automatically
set up the routing table to forward internet traffic over the virtual private network. Alternatively,
this option can be pushed to the client by appending the following line to the corresponding file in
the client configuration directory, e.g.: '/etc/openvpn/ccd/john_doe':
push “redirect-gateway def1”
Further instructions and caveats can be found in the official OpenVPN documentation:
http://openvpn.net/index.php/open-source/documentation/howto.html#redirect.
2.3.7 Creating a new certificate authority
When the root certificate's private keys are compromised, new certificates need to be created and
issued according to the procedure outlined here.
Log in on Neon as root. Ensure the date is set correctly and that OpenSSL is up to date. Create a
directory '/root/ca/keys' and create the root key in it. Self-sign the certificate. Example script
below.
# date
# yum update openssl
# cd /root/ca
# mkdir keys
# cd keys
# openssl genrsa -out ca.key 2048
# openssl req -x509 -new -nodes -key ca.key -days 2000 -out ca.pem
OpenSSL will start an interactive script. Use the following information.
Country Name NL
State or Province Name Drenthe
Procedures\Extraordinary maintenance 27You can also read
