NCS System Administration

NCS 3.0

October 26, 2016

Confidential

Agenda
1 Overview and Background
How NCS fits in with the rest of the
world, what it does, and why
2 NETCONF and YANG
Why NETCONF was invented and how
it is different from previous
management protocols
Walk-through of the YANG modeling
language
3 System Setup
Installing NCS, packages, NETSIM
device simulator
4 Device Manager
Work with devices, synchronization,
templates, policies, etc
October 26, 2016

5 Service Manager
Work with high-level services
6 Alarm Manager
The alarm model and how to work
with alarms
7 System Administration
Deeper topics about installation
choices, logging, trouble shooting,
high availability, clustering, backups,
disaster recovery, etc
8 NED Development
NED types, Yang models and Java
code

Confidential

Java VM

October 26, 2016

Confidential

Candidate database or not ?

NCS announces candidate support and also confirmed commit capability on it's
northbound NETCONF interface regardless of the capabilities of the managed
device
NCS divides devices into:
start_trans_running : Tail-f proprietary transaction mechanism towards running. May be more
efficient than candidate
lock_candidate - This mode is used for devices that support the candidate data store but
disallow direct writes to the running data store.
Continued

Device
Candidate

October 26, 2016

Confidential

Running

Candidate database or not, cont ?

NCS divides devices into:

start_trans_running
lock_candidate
lock_reset_candidate - This mode is used for devices that support the
candidate data and also allow direct writes to the running data store. This is
the default mode for Tail-f ConfD NETCONF agent. Since the running data
store is configurable, we must, prior to each configuration attempt, copy all
of running to the candidate.
startup - This mode is used for devices that have writable running, no
candidate but do support the startup data store. This is the typical mode for
Cisco like devices.
Running-only - This mode is used for extremely simple devices that only
support writable running.
> show status devices device lb0 state transaction-mode

October 26, 2016

Confidential

Confirmed-Commit ?

For NCS to support confirmed-commit northbound candidate is required.

Confirmed-Commit solves many tricky failure scenarios
Devices roll-back automatically if no commit within timeout or SSH closes

If Device does not support confimed-commit

NCS has the revert diff and simply sends the precise undo information to the device

If a Device rejects a config to its running NCS aborts the whole

transaction
If NCS looses SSH connection to the
device without Confirmed-Commit
we can not issue reverse diff
NCS
Escape with sync operations
2) Confirmed-Commit
1) Confirmed-Commit
4) Commit

3) Commit

Device
October 26, 2016

Confidential

Device
6

Installing NCS for deployment

NCS can be installed for production in a layout of your choice
LSB style: /opt/ncs, /etc/ncs, /var/ncs
Centralized style: in a directory of your choice /usr/local/ncs
The NCS installation in would often contain a versioned component and a soft link
/opt/ncs/current -> /opt/ncs/3.1
/opt/ncs/3.0
/opt/ncs/3.1
You also need a runtime data directory with the database, packages etc, e.g.
Runtime: /home/ncs

October 26, 2016

Confidential

NCS Start Phases Overview

NCS validates
& commits database content
NCS opens up
northbound interfaces

NCS starts
and reads schema

Phase 0

Phase 0U

User starts and connects

upgrader and validation
clients

October 26, 2016

Phase 1U

Phase 1

NCS
provides
Southbound
Service

Confidential

NCS
provides
Northbound
Service

Phase 2

User starts and connects

all other clients

NCS Start sequence

When starting NCS, additional flags may be given
--smp <number>
--ignore-initial-validation
Autostart, e.g. using
BSD style /etc/init.d/S90.ncs script
Ubuntu upstart /etc/init/ncs.conf script

October 26, 2016

Confidential

Managing NCS
Common management commands
ncs --status
ncs --version
ncs --reload
Reload packages at ncs start
--with-package-reload
export
NCS_RELOAD_PACKAGES=true
(At runtime: request packages
reload)

October 26, 2016

Take CDB backup

ncs-backup
NETCONF <get-config>
CLI save backup.cfg
With NCS stopped:
tar cvfz cdb-backup.tgz ncs-cdb/
Restore CDB backup
NETCONF <edit-config>
CLI load backup.cfg
With NCS stopped:
tar xvfz cdb-backup.tgz

Confidential

10

Troubleshooting NCS
Figuring out what is going on
Check log files
ncs.log
ncs-java-vm.log

devel.log

ncs --status
Verbose flag: ncs -v

October 26, 2016

When contacting support

Transcript
Log files
Enable debug error log in ncs.conf
Zip together a minimal project?
ncs --debug-dump
System call trace strace/ktrace/truss

Confidential

11

Disaster Management
NCS fails to start
Environment problem
Configuration mode change
NCS corruption
Data corruption
Internal error
NCS runtime problem
Out of memory, file descriptors, disk
Internal error

October 26, 2016

Transaction commit failure

External database problem
External validator problem
Internal error
Software upgrade failure
Upgrade client failure
Initial validation failure
Internal error

Confidential

12

AAA integration
Authentication
Stored in CDB
PAM (no roles/groups)
External auth:

Authorization
Authorization NACM rules
Authorization external
callback function

Auditing
Send to syslog
Send to files
Send to application

Tacacs+, Radius, LDAP,

Combination of the above

October 26, 2016

Confidential

13

High Availability

VIP

NCS - Master
CDB

NCS - Slave
Replication

CDB

Control /
Notifications

CDB

Multiple slaves are

possible

HAFW

October 26, 2016

NCS - Slave

Confidential

14

High Availability Geographically Distributed

NCS - Master

NCS - Slave

CDB

CDB

SEA

NCS - Slave

NCS - Slave

CDB

CDB

HAFW

Slave-slave replication
is possible

October 26, 2016

Confidential

15

NCS Cluster Geographically Distributed

NCS A
CDB

Dev
Dev
Dev

October 26, 2016

Cluster
Link

NCS B
SEA

Dev
Dev
Dev

CDB

Dev
Dev
Dev

Confidential

Dev
Dev
Dev

16

Geo Redundant Cluster

NCS HA Pair A
NCS - Master
CDB

NCS - Slave
CDB

Cluster
Link

NCS HA Pair B
SEA

NCS - Master
CDB

HAFW

Dev
Dev
Dev

October 26, 2016

NCS - Slave
CDB

HAFW

Dev
Dev
Dev

Dev
Dev
Dev

Confidential

Dev
Dev
Dev

17

Software Upgrade Campaign

Offline
Grace period, stop configuring
Stop NCS and applications
Backup configuration
Install new software
Start NCS and upgrade clients
Database upgrade performed
Start all applications
Allow configuration

October 26, 2016

In-service software upgrade

Grace period, stop configuring
Split brain mode
Stop slave NCS and applications
Install new software
Start slave NCS and upgrade clients
Database upgrade performed
Start all applications
Fail-over to upgraded system
Allow configuration
Upgrade other system
Join as slave

Confidential

18

Move NCS to a Bigger Machine

Offline
Grace period, stop configuring
Stop NCS and applications
Backup configuration
Restore backup on new machine
Start NCS and applications
Allow configuration

October 26, 2016

In-service move
Start slave NCS on new machine
Wait for database sync to complete
Force fail-over to new machine

Confidential

19

October 26, 2016

Confidential

20