Cisco NEXUS switch and Virtualization

NX -OS  next generation operating system Featrue

Chapte 1 : New feature in Nexus

·         Modular

o   Enable particular module in nexus

o   E.g  Feature eigrp  or No feature eighrp to enable or disable eigrp

·         High availability à PSS - Persistence storage service: If BGP process crashes , it will not crash underlining operating system and that processes gracefully restart and it will have all state information before the crash.

·         Unified os for lan and san,

o    prior to this we had iOS  with IP protocol and SAN OS with Fiber channel protocl,

o   Reason for lan and san segregation

§  Security

§  Bandwidth

§  flow control

§  performance

o   Flow control:

§  LAN  - sender keep sending till receiver respond

§  SAN   - Receiver defines how much data transmitter can send

o   MX-OS 4.1 or higher run on MDS (multilayer director switch) as well as Nexus 7K as 5K.

o   Unification of lan and san using 10Gbps ethernet using FCoE

·         Role based access

o   Privilege levels access – method is in old ios

o   Views      - access based on view  - prior to nx-os

o   In nexus there is role based access

§  Eg:  username admin password Cisco 123 role {  network-admin | network-operaor | priv-0 -15 |vdc-admin | vdc-operator }

§  Vdc-admin or vdc operator option : In N7K we can have virtual Device  context (VDCs) and have separate admin for each context.

·         Cisco layered approach

o    scalability

o   resilient – failover

Chapter 2 : NEXUS Family:

The nexus 7000 series

·         Models:

o   7009  -2sup slots , 7 i/o module slots

o   7010 - 2sup slots, 8 i/o module slots

o   7018 - 2sup slots, 16 i/o module slots

·         L2/L3, DCP (data center bridging), FCoE

·         ISSu ( in service software upgrade)

·         VDCs (virtual device context)

·         Modularity

·         Separation of CP  (control plan) and DP (Data plan)

·         RBAC (role based Access control)

·         EEM  (Embedded event manager)

·         Call home

·         Dual supervision

·         Dual CMP (connectivity management processor) with own  memory , power and software. It provide lights out connectivity ( OOB )

·         Dual redundant central Arbiter of particular traffic – multiple path through architecture of devices.

·         Redundant fan module – hot swappable fan tray

Sup 1 is Eosale and EoL and replacement is Sup 2E ( dual quad-core cpu with 32GB RAM)

Licensing of software

1 default base license

2 enterprise lan license - for dynamic routing and multicast

3 advanced enterprise lan license - VDC, Cisco trust sec

4 mpls license  - mpls routing

5 transport service – OTV (Overlay transport virtualization)

6 enhance L2 services - fabric path

Command

Download license, store on boot flash and use following command

# install license    <-- install="" license="" o:p="" to="">

#Show license usage

Trial license: 120 day grace period for testing before buying license.

Modules: sup card, line card and fabric module

Fabric module support virtual output queuing.

Power Redundancy: with respect to  7010 we have similar option in UCS.

·         Combined power mode – no redundancy – all power supply work together.

·         Power supply redundancy  ( n +1)

·         Input source redundancy ( external power failure ) / grid power redundancy

·         Complete redundancy ( Power supply + input source

Nexus 7009 v6

Nexus 7009 v6.0(2) w/ SUP1, 10GE F1, 10GE M1

M1 module support L3 feature

F1 module is L2 feature

    Cisco fabric path similar to TRIL Transparent interconnection of lots of link

---------------------------------------------------------------------------

The Nexus 5000 series

5010   - throughput 520 gigbit per secon

5020

5548

5596  -  throughput  1.92 Tbps 

Common features in 5500 series

DCB –data center bridging

FCoE – Fiber channel over Ethernet

GEMS – generic expansion module slot – add FCoE

55XX L3 routing capability,

 routing can be enabled on

5548  with N55 –D160 L3 card.

5596 with N55-M160 L3

Port density is different in each model e.g in 5596 96 à 96 1 gig Ethernet port density

Nexus 5548 v5.1 (3)

Feature : unified port

------------------------------------------------------------------------------------------

The Nexus 2000 series

Function at Top or rack ToR, all c-series devices are connected to N2K device on ToR and N2K are managed by N5K device at EoR ( End of Rack)

2000 series are called a FEX – fabric extender

Redundancy using 2000 ToR and 5000 EoR

Model  nexus 2000 series  ( scalability ,oversubscription,  host port and fabric port )

·         2148  

o   4 X 10G fabric port,  1 port channel with max 4 port,  48 host port but no port channel, no FCoE

·         2224  

o   2 X 10G fabric port, 1 port channel port,  24 X 1G host port 24 port channel with max 8 port in one of the port channle, no FCoE

·         2248

o   4 X 10G fabric port, 1 port channel port,  48 X 1G host port 24 port channel , no FCoE

·         2232

o   8 X 10G fabric port, 1 port channel ,  32 X fiber optic host port 16port channel , only can supply FCoE on ToR location

---------------------------------------------------------------------------------------

Nexus 1000v v4.2   virtualized Ethernet module or supervisior moduel

Chapter 3: MDS family (multilayer director switch)

9500 series

9124

9148

9222i

9500

·         TABLE:

Model   --  FC port density

9506  --     192

9509   --    336

9513 --     528

·         nonblocking - virtual output queuing

Requirement  for  SAN - not packet loss and low latency

·         high bandwidth  2.2 Tbs internal bandwidh / 160 gbps (16 ISL bundle)

·         Low latency – less than 20microsecond per hop

·         Mulit protocol  ( FC, FC0E, FICON, FCIP, iSCSI)

·         Scalable – VSAN (virtual storage Area network ) Cisco invension

·         Secure - port security

·         High availability – dual sup, dual clock, dual power

·         sup2 (no FCoE) and sup2-A (FCoE),  Intercross bar Fabric (traffic cop) in 9506 and 9509 in 9513 will have separate intercross bar fabric module

·         Licensing types of MDS – feature based and module based.

o   feature based

§  ENt –security

§   sanover ip – FCIP

§  main frame – FICON

o   -module based

-----------------------------------------------------------------------------------------------------------------

Cisco MDS 9124

24 port – 8 default and 16 on-demand port

NPV (n port virtualization)

Cisco MDS 9148

  16-32-48 base license

Cisco MDS 9222i

Flexible with expansion slot support vide verity of module

18 FC 4gbps Ethernet port used for FCIP / iSCSI

-------------------------------------------------------------------------------------------------------------

NEXUS switch and NX-OS

NEXUS architecture:

# sh ver

software

·         BIOS    version 2.12.0

·         Kickstart image- contains linux           7K kernel  version 6.2(2), 5K 5.1(3)

·         System – contain software component of NX  multilayer director switch

Hardware

·         Supervisor : intel xenon  ,  12 gb memory

Plug-in

·         Core plugin – contain nx os software component

·         Ethernet – L2 and  L3 software component

·         In future we will see – Storage plugin – for FCoE

Chapter 4 : Monitoring Nexus Switch

Monitoring the Nexus

·         Rj 45 console is located on sup card.

·         Nexus 7000 with sup 1 engine with cmp (connectivity management processor) with dedicated O.S for OOB management access. Has led for notification and local authentication. Sup 2 don’t have this capability

·         To connect to cmp we use command # attach

o   Attach console | module | cmp

·         Remote acces:

o   Ssh v2 is enabled by default, ssh / telnet clinet & server capability , ip v4 and v6 supported

o   In cmp  use command  # ssh server enable or # telnet server enable

·         Management

o   Support concept of VRF, by default there are two VRF ( default and management vrf) exist in nexus switch.

o   Management interface is in management vrf. To test connectivity for management purpose is  # ping 10.10.10.10 vrf management

·         ISSU (in service software upgrade)

o   Upgrade with no disruption , data plane continue forwarding packet during upgrade proces

o   4.2.1

o   Kickstart, bios, system, fex (2000 series), i/o module bios and image

o   Started with 7000 series which has dual sup card, first upgrade standby sup engine, then upgrade second sup engine. This feature is also now supported in 5000 series which has single sup engine. Here control plane will be offline during upgrade. If 5500 series with L3 functionality it will not support ISSU.

o   Steps for 5000 series

§  Download appropriate software Cisco.com/go/fn (feature navigator)

§  Copy tftp à boot  flash

§  Show incompatibility  ( show incompatible with new image) ß pre upgrade command

§  Show install all impact ( shows impact of upgrade) ß pre upgrade command

§  Show install all status   ç post upgrade to verify installation status.

·         Control plane policing

o   Data plane

o   Mgmt plane – snmp

o   Control plan – L2 stp, LAcP, L3 ospf , bgp

§  CoPP control plan policing – restrict number of packet entering CP. There is  default CoPP, during installation it ask for strict , moderate, lenient or no default policy.

·         KEY  CLI command

o   Where – shows mode, which VDC

o   Show run ipqos | all   allà show everything including default

o   Sho run interface all

o   Show module

o   Show loggin

Chapter 5 vPC

vPC ( virtual port channel)

·          Used to bundle with uplink with two different uplink switch.

o   Virtual port channel peer

o   One of 7K become primary and other secondary

o   Orphan port – port not participating on vPC infrastructure

o   Member port – port participating in vPC

o   CFS – Cisco Fabric Service – used to synchronize stateful information

o   Peer keep-alive link – logical link work OOB path (no data or sycn message is sent over this link)

o   Limitation

§  Peer link should be 10 gig Ethernet port,  at least 2 used.

§  vPC is per VDC

§  L2 portcahnnel technology

o   Dual sided vPC 7K = 5K = 2K = c –series sever

o   Show vpc brief

§  Vpc domain id

§  Peer status

§  Keep alive status

§  vPC role

                      

o   show vpc peer-keepalive

§  keep alive tos 192 – binary representation of tos byte

§  Role – priority  ==== > lower is better

o   show vpc consistency-parameter interface port-channel 20

vPC (Virtual port-channel )
Overview
A virtual PortChannel (vPC) allows links that are physically connected to two different Cisco Nexus 7000 or 5000 Series devices to appear as a single PortChannel to a third device. The third device can be a Cisco Nexus 2000 Series Fabric Extender or a switch, server, or any other networking device. A vPC can provide Layer 2 multipathing, which allows you to create redundancy by increasing bandwidth, enabling multiple parallel paths between nodes and load-balancing traffic where alternative paths exist.
After you enable the vPC function, you create a peer keepalive link, which sends heartbeat messages between the two vPC peer devices.
The vPC domain includes both vPC peer devices, the vPC peer keepalive link, the vPC peer link, and all the PortChannels in the vPC domain connected to the downstream device. You can have only one vPC domain ID on each device.
A vPC provides the following benefits:
• Allows a single device to use a PortChannel across two upstream devices
• Eliminates Spanning Tree Protocol blocked ports
• Provides a loop-free topology
• Uses all available uplink bandwidth
• Provides fast convergence if either the link or a device fails
• Provides link-level resiliency
• Helps ensure high availability
The vPC not only allows you to create a PortChannel from a switch or server that is dual-homed to a pair of Cisco Nexus 7000 or 5000 Series Switches, but it can also be deployed along with Cisco Nexus 2000 Series Fabric Extenders.

The following list defines critical vPC concepts:
• vPC: vPC refers to the combined PortChannel between the vPC peer devices and the downstream device.
• vPC peer switch: The vPC peer switch is one of a pair of switches that are connected to the special PortChannel known as the vPC peer link. One device will be selected as the primary device, and the other will be the secondary device.
• vPC peer link: The vPC peer link is the link used to synchronize states between the vPC peer devices. The vPC peer link carries control traffic between two vPC switches and also multicast, broadcast data traffic. In some link failure scenarios, it also carries unicast traffic. You should have at least two 10 Gigabit Ethernet interfaces for peer links.
• vPC domain: This domain includes both vPC peer devices, the vPC peer keepalive link, and all the PortChannels in the vPC connected to the downstream devices. It is also associated with the configuration mode that you must use to assign vPC global parameters.
• vPC peer keepalive link: The peer keepalive link monitors the vitality of a vPC peer switch. The peer keepalive link sends periodic keepalive messages between vPC peer devices. The vPC peer keepalive link can be a management interface or switched virtual interface (SVI). No data or synchronization traffic moves over the vPC peer keepalive link; the only traffic on this link is a message that indicates that the originating switch is operating and running vPC.
• vPC member port: vPC member ports are interfaces that belong to the vPCs.

vPC configuration on the Cisco Nexus 5000 Series includes these steps:
• Enable the vPC feature.
• Create a vPC domain and enter vpc-domain mode.
• Configure the vPC peer keepalive link.
• (Optional) Configure system priority.
• (Optional) Configure vPC role priority.
• Create the vPC peer link.
• Move the PortChannel to vPC.

Ref: https://www.facebook.com/cciedclab

Chapter 6 : Fabric Path

·         Fabric Path Cisco version of TRILL

·         TRILL (transparent interconnection of lots of link) its replacement technology for STP.

In TRILL L3 routing intelligence is brought to L2 , thus keep L2 routing table.

·         IS-IS

·         Fabric path perfrom ECMP (Equal cost multiple path) , 16 – way equal cost multi path. ECMP with port channel technology 16 10gbps port channel will give 2.56 Tbs usable bandwidth.

·         Fabric path network appears as one single switch for STP running legacy  switch network.

·         Switch using legacy STP port is called as Classical Ethernet port and Fiber channel infrastructure port is called as FPP ( Fiber path port)

·         Enhancement in Fabric path ( modification from TRILL)

o   Conversational learning – learns only active mac address

·         Verification

o   Show mac address-table  will show destination hop fabric path device.

o   Show fabricpath route

Chapter 7 : OTV

OTV (Overlay Transport Virtualization)

Basic understanding of OTV
Today i am going to help you understand Why we need OTV?

Lets say, we have 3 switches (A,B,C). Switch A is connectec to B and Switch B is connected to Switch C. and Switch A has 2 vlans created on it, vlan 10 and 20. What if we want the the vlan 10 and 20 to be extended to Switch C over Switch B, We will have to simply create vlan 10 and 20 on both switch B and C and allow both the vlans on trunks connecting the switches, right? and its simple!!

If you look at this pic, we have two Datacenters, DC1 and DC2 which are geographicaly far away from each other, lets say one in Newyork and another one in Los Angles and there are some server which are there in both data centers,however, they sync their hearbeat over layer 2 only and doesnt work on layer 3. So,we have a requirment that we have to extend vlan 10 and 20 from DC1 to another data center, DC2!! You may call it Datacenter Interconnect (DCI).

can we do the same thing which we did to extend vlan from switch A to switch C in above example? Ofcourse Not!!, so what the are the solutions to achieve this?
Until OTV came into picture, we had few of the below options to achieve this:
-VPLS
-Dark Fiber (CWDM or DWDM)
-AToM
-L2TPv3

These are the services provided by Service Providers and they work on different mechanisms but basicaly what they do is, they provide you a layer 2 path between DC1 to DC2 similar to a trunk link between Switch A and Switch B. So what does that mean? If a broadcast is sent or a ARP request is sent, that will travel across the service provider to another data center in that VLAN? Ofcourse YES!! Your STP domain will also get extended over DCI. So, if a device in vlan 10 in DC1 is trying to communicate with another device which is also in DC1 but the ARP request will go all the way to DC2 switches on which that particular vlan is configured.

So, to avoid such problems, Cisco introduced OTV (Overlay Transport Virtualization) which is basicaly a DCI (data center interconnect) technology to be configured on Nexus Switches. Using OTV, we can extend Layer 2 between two or more datacenters over traditional L3 infrastructure provided by Service Provider, and we dont need a seperate L2 link for layer 2 extension and we will still be able to limit STP domain and unnecessary broadcast over WAN links. It can overlay multiple VLAN with a simple design. Basically what it does is that, Datacenters will be able to advertise their MAC addresses to each other(its called
Mac in IP" routing) and a decision can be made on the basis of MAC addresses whether that MAC address is local or in another data center and based on that, frame can be forwarded or limited to a particular data center only. OTV uses a control protocol to map MAC address destinations to IP next hops that are reachable through the normal L3 network core.
So, in Cisco's language "OTV can be thought of as MAC routing in which the destination is a MAC address, the next hop is an IP address, and traffic is encapsulated in IP so it can simply be carried to its MAC routing next hop over the core IP network. Thus a flow between source and destination host MAC addresses is translated in the overlay into an IP flow between the source and destination IP addresses of the relevant edge devices. This process is called encapsulation rather than tunneling as the encapsulation is imposed dynamically and tunnels are not maintained"

How this is implemented, that i will show in another simplified post!!Thank you!!

Ref: https://www.facebook.com/cciedclab

Chapter 8 :  Network Virtualization .

7K  with 5.X code

·         4 VDCs

·         VDCI ( default VDC, VDC 1) – control shared resources

·         VLANs / VRFs per VDC. Eg. Vlan 100 in each VDC will be separate.

·         Failure of protocol in one VDC will not affect same protocol in other VDC.

·         To create virtual interface we have to be that particular VDC.

·         Some IO module each port work independently and for some port grouping need to be done.

·         Role based access control.

o   Vdc admin from one user created context can not control other vdc.

o   To create vdc with advanced service license.

o   Configuration

§  Vdc test

§  Switch to vdc test

o   Verification

§  Show vdc

§  Show vdc membership (interface membership)

§  Show run vdc-all ( running configuration for all vdc)

§  Copy run start vdc-all

·         NVI ( network Interface virtualization)

o   Refers to FEX

o   VNtag à IEE802.1Qbh

2K – FEX

Physical port on 5K connecting to 2K = Fabric port

Bundle of 5 K connecting to 2K = F.P. Channle

Uplink on 2K connecting to 5K is called as FEX uplink

2K port connecting to server = FEX port

Virtualization in storage:

Device controlling virtualization.

1 host / server

2 physical disk system ( array-based)

3  network devices

Identify logical disk in SAN we use LUN ( logical unit number ) – sometime we refer logical disk as LUN but Lun is just numeric identifier.

In storage area we have to control the number of device accessing logical disk.  This id done by LUN masking using PWWN ( port world wide name). Also there is LUN mapping here we map logical disk to particular HBA host bus adapter in server.

Cisco developed LUN zoning, this technology is not dependant on storage vendor technology.

It is found in MDS switch and allows us to map logical disk to particular HBA.

Storage virtualization

Types: Block    Disk     Tape    File system     

Block: provide logical  volume of disk to user that is stored physically  somewhere.

Disk Virtualization  : providing disk out of large disk array,

Tape:

File system:  allow user to access a file.

File and record virtualization : presenting logical volume to particular user

It can be done : host level , network level or Array level

How we do it : in-band ( data and control going through same channel ) or  OOB ( data and control over separate channel)

Advantage of Network based

·         It is independent of server, OS, storage solution

·         Offloading virtualization.

Chapter 9 : Server virtualization

Before virtualization we had server sprawl (each entity or service had their own box or system), this is scalable but consume lots of power and administration work.

Aps      Aps      Aps

Os        Os        Os

--------------------------------------------------

Virtualization software – allocate hardware resources.

---------------------------------------------------

Rack mount server – CPU + RAM + NIC + Disk space

Benefit of Virtualization

-          Partitioning -  vm instance

-          Encapsulation  à everything is in set of files. We can take backup and manipulate this file to tweak the server

-          Isolation à

-          Hardware abstraction à we can move VM and move between hosts.

-          Cap EX / Op Ex

-          Capital expense – buying new hardware

-          Operational expense – spending for operation , like cooling and maintenance

Virtualization techniques: (based on how much modification done on guest OS)

Full – host os vs bare metal

Partial

Para-virtualization

VMware workstation 8 – type 2 – host OS, window 7 – VMware workstation 8

Hypervisor EXSi  - type 1

Bare metal - hypervisor –

EXS is old version of ESXi, EXS is Linux based kernel.

Vsphere 5.x software suite

   ESXi, Vsphere server, vsphere client, vshpere view product,

Microsoft Hyper – V server 2012

Citrix – Xen Server

Chapter 10 : Nexus 1000v

Chapter 11 : Storage Area Network

·         File-based vs Block-based

o   Eg. CIFS ( Common Internet File System)

o   Network File System (NFS)

o   This protocol has high latency and work over TCP/IP  and are chatty.

o   Suitable for MS office or print services

·         Block-based

o   SCSI

o   Many input output operations per second ( IOPS)

o   SCSI Parallel cable ( low latency, distance limitation)

o   iSCSI – transport of scsi over TCP (high latency)

o   FC – low latency and high B.W

o   FCoE – works on 10 gbps link

·         NFS ( network file system)

o   Client server environment

o   Unix client – service  mountd

o   Unix serve -  Volume   - portmatp / rpcbind

o   Mountd à portmap/rpcbind

o   Mount / automounter ( mount file on demand)

o   Once mound system we can use command like ls    à  NFSD

o   NFSv2 RFS 1094 32 bit stateless

o   NFSv3  rfs 1819 64 bit stateless

o   NFSv4 rfc 3530 statefull , security

·         SCSI

o   Host  à initiator

o   Storage à Target

o   Daisy chain but max 16 devices

o   Bus length 25 m

o   Max bw 320 mbps

·         FC

o   Overcome limitation of parallel scsi

o   16 million nodes

o   Loop ( token ring / modern day  switched transport (also called as fabric)

o   800 mbps  /  Fabric  8Gbps

o   6.2 miles

o   Multiple protocols.

·         SAN terminology

o   Initiator

o   HBA

o   Target

o   Smallest unit of data is word à encoded into 40 bit form by 8bit-10bit encoding process of FC, word get packaged into Frame (equivalent to ip packet in IP network), sequence is series of frame sending between nodes and are unidirectional.  Sequencing which happen during R / W operation are called as exchange ( similar to tcp session )

·         iSCSI

o   encapsulate scsi command and data over IP

o   TCP port 3260 – congestion control and in order deliver of  error free data in iSCSI environment

o   Distance concern is address

o   MDS 9222i and 9000 series provide transparent SCSI routing

·         DAS ( direct Access storage)

o   Limited mobility ( it often refer to capture storage)

·         SAN