NSX from Scratch

Learn VMware NSX – the SDN solution from VMware – from scratch.

تعلم VMware NSX من الصفر , باللغة العربية

Content of the course:

Introduction to SDN and NSX
Lab setup 3. NSX components
Architecture and sample designs
Preparing the physical network
Deploying NSX manager
Deploying controller(s)
Preparing vSphere clusters
Creating logical switches
Deploying DLR
L2 bridging
Exploring Security features
Configuring DFW
Security Groups
SpoofGuard
Application Rule Manager (ARM)
Deploy ESG
Configuring OSPF
Load balancer
SSL VPN-PLUS
Flow monitoring
Course Notes
vRealize Network Insight – Overview

Course Playlist – Language: Arabic

Presentations of the course

For NSX documentation guides, please click here

Preview my NSX quick notes, here.

*************************************************

=============================
VMware NSX – Important Notes
=============================

###############
TERMINOLOGY
###############

The terms “VXLAN segment”, “Virtual Network” (VN) and “Logical
Switch” (LS) all refer to the logical layer 2 domain created in the logical
network space and will be used interchangeably in this document. Which is the VLAN

BUM (Broadcast, Unknown Unicast, Multicast).

DNAT is “port forwarding”.

SNAT is the traditional NAT.

###########################
LICENSING
###########################

kb: 2135310
If you are using VMware vSphere 5.5 Update 3, vSphere 6.0.x or vSphere 6.5 and higher, VMware NSX for vSphere 6.x does not require a vSphere
Enterprise Plus license and can run on any vSphere edition including Essentials and Essentials Plus Kits.

NSX Licensing scheme >>> http://www.vmware.com/products/vsphere.html

######################
TECHNOLOGY PARTNERS
######################

PARTNERS: https://www.vmware.com/products/nsx/technology-partners.html

######################
TECHNICAL NOTES
######################

Authentication and user permissions:
– configure SSO via NSX VAMI.
– set user access permissions through the vSphere web client.

Installation notes:
– During NSX controller deployment, don’t tempt to create logical switches, or prepare hosts.

NSX Maximums:
– Do not configure more than 10,000 VNIs in a single vCenter because vCenter limits the number of
dvPortgroups to 10,000

The NSX layer 2 bridging data path is entirely performed in the ESXi
kernel rather than in user space. The control VM is only used to determine
on which ESXi host where a given bridging instance is active; not to
perform the actual bridging function

NSX GROUP OBJECT

NSX provides grouping mechanism criteria can include any of the following:
• vCenter Objects: VMs, Distributed Switches, Clusters, etc.
• VM Properties: vNICs, VM names, VM operating Systems, etc.
• NSX Objects: Logical Switches, Security Tags, Logical Routers, etc.

================
ESG
================
ESG >> Heartbeat keepalives are exchanged every second between the active and
standby edge instances to monitor each other’s health status. The messages exchanged on the internal port-group between the active and
standby NSX Edges are also used to sync up state information required for the
logical services (e.g., forwarding information base (FIB), NAT, firewall connection
state) to providing this service statefulness.

The Edge gateway can serve holistic need of small business – integrated
VPN, SSL client access and firewall services.

The ESG with HA option deploys two ESG VMs, one acting as primary
with the second in standby mode. In small data centers, ESG with HA is
the most deployed option because it allows organizations to enable
stateful services such as DHCP, NAT, and VPN.
With ESGs deployed in ECMP mode, up to eight VMs can be active
and participate in forwarding traffic in/out from the virtual network to
the physical undelay. Since all ESG VMs are active with ECMP, the
traffic may exit via one ESG VM and come back in through another.
Because of this , it is not possible to run stateful services on ESGs in
ECMP mode. Usually ESG with ECMP is deployed when more than 10G
of north-south bandwidth is required.

==================
SPOOFGUARD
==================
NSX provides a SpoofGuard feature to avoid spoofing of IP addresses. There are
two SpoofGuard mechanisms: trust on first use and manual authorization

— Guest Introspection extension framework must be
deployed to eable the advanced security inspection capibilities

— NSX Controller function: updating MAC, ARP, and VTEP
tables

— The DLR control VM is only needed if a dynamic routing protocol is required between the DLR and ESG

=======================
LOAD BALANCER
=======================

ONE-ARMED (PROXY MODE): only one NIC is required in the LB.

Downside: originating source IP is replaced by the LB IP, to workaround this, use “X-FORWARDED” for HTTP header.

In-line (transparent mode): the originating source IP is passed down to the server.
Downside: DLR cannot be used as the VM gateway must be set to the ESG IP.

########################
DESIGN CONSIDERATIONS
########################

– NSX manager outage may affect only specific functionalities such as identity
based firewall or flow monitoring collection

– Restoring a backup is only possible on a freshly
deployed NSX manager appliance that can access one of the previously backed
up instances.

– eBGP routing protocol is the right choice from Edge to physical

– NIC TEAMING: In summary, the “route based on originating port” is the recommended teaming
mode for VXLAN traffic for both compute and edge cluster

— EDGE HA >> etting the timer to 9
seconds is a safe best practice

— Edge >> If throughput greater than 10 GB is desired, convert the active-standby
gateway to ECMP mode, and add more nodes if needed.

— Small datacenter NSX design: all NSX and vSphere components as well as payload VMs are deployed
in a single cluster to provide rich set of NSX networking and security services.

— DFW does not require any changes to the physical infrastructure such as MTU or
routing. It also does not require VXLAN functionality.

— NSX Manager DFW “VM Exclusion List”
The NSX Manager VM exclusion list feature makes sure that any VM
placed in the list will never be affected by DFW rules. It is
recommended to add the vCenter VM to the exclusion list to prevent
losing access due to a misconfigured rule. as well as domain controllers, It is also a good approach to
add management VMs such as Log Insight and vROps in the exclusion
list as shown in Figure 4.5. If protection of vCenter and management
VMs is required by NSX

VTEP configuration: SMB data centers can choose the
NIC fail-over option as teaming policy which will create a single VTEP
per ESXi host

##############################
DESIGN GUIDES
##############################

VMware NSX Reference Design Guide
https://communities.vmware.com/docs/DOC-27683
• VMware Validated Design Guides
https://www.vmware.com/solutions/software-defined-datacenter/
validated-designs.html

#######################################
TROUBLESHOOTING TOOLS
#######################################

troubleshooting stage:
•
NSX monitoring dashboards – NSX and cluster health indication
•
NSX Traceflow – visual virtual network packet trace
•
NSX flow monitoring – quick packet flow
•
NSX endpoint monitoring – monitor OS processes running
inside a VM
•
vRealize Log Insight – presence of this or another syslog
server is a must
•
vRealize Network Insight – optional in a small data center

**************************************

Recommended teaming option for VDS on Edge hosts is
“route based on source ID”

*************************************

The recommendation is to use a subnet no larger than /22
(~1K IPs) for a logical switch

#######################################

NSX useful commands

################
Useful commands
################

–RUN FROM NSX-MGR CLI–

### List all logical router instance information.

nsxmgr-l-01a> show logical-router list all
Edge-id Vdr Name Vdr id #Lifs
edge-1 default+edge-1 0x00001388 3

### List the hosts that have received routing information for the logical router from the controller cluster.

nsxmgr-l-01a> show logical-router list dlr edge-1 host
ID HostName
host-25 192.168.210.52
host-26 192.168.210.53
host-24 192.168.110.53

### List the routing table information that is communicated to the hosts by the logical router. Routing table
entries should be consistent across all the hosts.
nsx-mgr-l-01a> show logical-router host host-25 dlr edge-1 route
VDR default+edge-1 Route Table
Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface]
Legend: [H: Host], [F: Soft Flush] [!: Reject] [E: ECMP]
Destination GenMask Gateway Flags Ref Origin UpTime Interface
———– ——- ——- —– — —— —— ———
0.0.0.0 0.0.0.0 192.168.10.1 UG 1 AUTO 4101 138800000002
172.16.10.0 255.255.255.0 0.0.0.0 UCI 1 MANUAL 10195 13880000000b
172.16.20.0 255.255.255.0 0.0.0.0 UCI 1 MANUAL 10196 13880000000a
192.168.10.0 255.255.255.248 0.0.0.0 UCI 1 MANUAL 10196 138800000002
192.168.100.0 255.255.255.0 192.168.10.1 UG 1 AUTO 3802 138800000002

### List additional information about the router from the point of view of one of the hosts. This output is
helpful to learn which controller is communicating with the host.

nsx-mgr-l-01a> show logical-router host host-25 dlr edge-1 verbose
VDR Instance Information :
—————————
Vdr Name: default+edge-1
Vdr Id: 0x00001388
Number of Lifs: 3
Number of Routes: 5
State: Enabled
Controller IP: 192.168.110.203
Control Plane IP: 192.168.210.52
Control Plane Active: Yes
Num unique nexthops: 1
Generation Number: 0
Edge Active: No

### Check the Controller IP field in the output of the show logical-router host host-25 dlr edge-1
verbose command.

### SSH to a controller, and run the following commands to display the controller’s learned VNI, VTEP, MAC,
and ARP table state information.

>> 192.168.110.202 # show control-cluster logical-switches vni 5000

VNI Controller BUM-Replication ARP-Proxy Connections
5000 192.168.110.201 Enabled Enabled 0

### The output for VNI 5000 shows zero connections and lists controller 192.168.110.201 as the owner
for VNI 5000. Log in to that controller to gather further information for VNI 5000.

>> 192.168.110.201 # show control-cluster logical-switches vni 5000

VNI Controller BUM-Replication ARP-Proxy Connections
5000 192.168.110.201 Enabled Enabled 3

### The output on 192.168.110.201 shows three connections. Check additional VNIs.

>> 192.168.110.201 # show control-cluster logical-switches vni 5001

VNI Controller BUM-Replication ARP-Proxy Connections
5001 192.168.110.201 Enabled Enabled 3

>> 192.168.110.201 # show control-cluster logical-switches vni 5002

VNI Controller BUM-Replication ARP-Proxy Connections
5002 192.168.110.201 Enabled Enabled 3

##########
##########

Check the MAC tables.
>> 192.168.110.201 # show control-cluster logical-switches mac-table 5000
VNI MAC VTEP-IP Connection-ID
5000 00:50:56:a6:23:ae 192.168.250.52 7

>> 192.168.110.201 # show control-cluster logical-switches mac-table 5001
VNI MAC VTEP-IP Connection-ID
5001 00:50:56:a6:8d:72 192.168.250.51 23

Check the ARP tables.

>> 192.168.110.201 # show control-cluster logical-switches arp-table 5000
VNI IP MAC Connection-ID
5000 172.16.20.10 00:50:56:a6:23:ae 7

>> 192.168.110.201 # show control-cluster logical-switches arp-table 5001
VNI IP MAC Connection-ID
5001 172.16.10.10 00:50:56:a6:8d:72 23

####################

### List all logical routers connected to this controller.

controller # show control-cluster logical-routers instance all

LR-Id LR-Name Universal Service-Controller Egress-Locale
0x1388 default+edge-1 false 192.168.110.201 local

### Note the LR-Id and use it in the following command.

controller # show control-cluster logical-routers interface-summary 0x1388

Interface Type Id IP[]
13880000000b vxlan 0x1389 172.16.10.1/24
13880000000a vxlan 0x1388 172.16.20.1/24
138800000002 vxlan 0x138a 192.168.10.2/29

controller # show control-cluster logical-routers routes 0x1388

Destination Next-Hop[] Preference Locale-Id Source
192.168.100.0/24 192.168.10.1 110 00000000-0000-0000-0000-000000000000 CONTROL_VM
0.0.0.0/0 192.168.10.1 0 00000000-0000-0000-0000-000000000000 CONTROL_VM

### From a host

[root@comp02a:~] esxcfg-route -l

VMkernel Routes:
Network Netmask Gateway Interface
10.20.20.0 255.255.255.0 Local Subnet vmk1
192.168.210.0 255.255.255.0 Local Subnet vmk0
default 0.0.0.0 192.168.210.1 vmk0

### Display the controller connections to the specific VNI.

192.168.110.203 # show control-cluster logical-switches connection-table 5000
Host-IP Port ID
192.168.110.53 26167 4
192.168.210.52 27645 5
192.168.210.53 40895 6

192.168.110.202 # show control-cluster logical-switches connection-table 5001
Host-IP Port ID
192.168.110.53 26167 4
192.168.210.52 27645 5
192.168.210.53 40895 6

#########################################
Traffic from controllers to hosts runs through
the management interfaces
#########################################

### On the host, you can view the controller network connection matched to the port number.

[root@192.168.110.53:~] #esxcli network ip connection list | grep 26167
tcp 0 0 192.168.110.53:26167 192.168.110.101:1234 ESTABLISHED
96416 newreno netcpa-worker

### Display active VNIs on the host. Observe how the output is different across hosts. Not all VNIs are
active on all hosts. A VNI is active on a host if the host has a VM that is connected to the logical
switch.

[root@192.168.210.52:~] # esxcli network vswitch dvs vmware vxlan network list –vds-name
Compute_VDS
VXLAN ID Multicast IP Control Plane Controller Connection
Port Count MAC Entry Count ARP Entry Count VTEP Count
——– ————————- ———————————– ———————
———- ————— ————— ———-
5000 N/A (headend replication) Enabled (multicast proxy,ARP proxy) 192.168.110.203
(up) 1 0 0 0
5001 N/A (headend replication) Enabled (multicast proxy,ARP proxy) 192.168.110.202
(up) 1 0 0 0

### First, ping from VM to another VM on a different subnet and then display the MAC table. Note that the
Inner MAC is the VM entry while the Outer MAC and Outer IP refer to the VTEP.

~ # esxcli network vswitch dvs vmware vxlan network mac list –vds-name=Compute_VDS –vxlan-id=5000
Inner MAC Outer MAC Outer IP Flags
—————– —————– ————– ——–
00:50:56:a6:23:ae 00:50:56:6a:65:c2 192.168.250.52 00000111

~ # esxcli network vswitch dvs vmware vxlan network mac list –vds-name=Compute_VDS –vxlan-id=5001
Inner MAC Outer MAC Outer IP Flags
—————– —————– ————– ——–
02:50:56:56:44:52 00:50:56:6a:65:c2 192.168.250.52 00000101
00:50:56:f0:d7:e4 00:50:56:6a:65:c2 192.168.250.52 00000111

— END OF COMMANDS —

Mo's Notes

NSX from Scratch

Like this:

About

Contact

Mo's Notes

NSX from Scratch

Share this:

Like this:

About

Subscribe

Contact