Documente Academic
Documente Profesional
Documente Cultură
Page 1
Disclaimer
This document may contain product features and technology directions that are under
development or may be under development in the future.
Technical feasibility, market demand, user feedback, and the Apache Software Foundation
community development process can all effect timing and final delivery.
This documents description of these features and technology directions does not represent a
contractual commitment from Hortonworks to deliver these features in any generally available
product.
Product features and technology directions are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
Page 2
Agenda
Hadoop Security
Kerberos
Authorization and Auditing with Ranger
Gateway Security with Knox
Encryption
Page 3
HDP\PHD
Enterprise
Enterprise Services:
Services: Security
Security
Page 4
Authentication
Who am I/prove it?
Authorization
What can I do?
Audit
What did I do?
Data Protection
Kerberos
API security with
Apache Knox
Centralized
audit reporting
w/ Apache
Ranger
Wire encryption
in Hadoop
Native and
partner
encryption
Administration
Centrally management &
consistent security
Authentication
Authenticate users and systems
Authorization
Provision access to data
Audit
Maintain a record of data access
Data Protection
Protect data at rest and in motion
Page 5
Fall 2013
Largely silod deployments
with single workload clusters
2014
65% of clusters host
multiple workloads
A
HiveServer 2
Beeline
Client
Page 6
B
HDFS
Hive creates
map reduce
using NN
Service Ticket
Use Hive
Service T,icket
submit query
A
HiveServer 2
Beeline
Client
Page 7
Client
Requests a TGT
Receives TGT
Client dcrypts it with the password
hash
Sends the TGT and receives a Service
Ticket
Hive gets
Namenode
(NN) service
ticket
KDC
B
HDFS
Hive creates
map reduce
using NN ST
HiveServer 2
Beeline
Client
Client gets
service ticket for
Hive
Hive gets
Namenode
(NN) service
ticket
KDC
Page 8
B
HDFS
Original
request w/user
id/password
Apache
Use Hive ST,
submit query
Knox
Hive creates
map reduce
using NN ST
HiveServer 2
Client gets
query result
Beeline
Client
Knox gets
service ticket for
Hive
Hive gets
Namenode
(NN) service
ticket
KDC
Page 9
B
HDFS
SSL
Original
request w/user
id/password
SSL
Apache
Use Hive ST,
submit query
Knox
SASL
SSL
HiveServer 2
Beeline
Client
Knox gets
service ticket for
Hive
Hive gets
Namenode
(NN) service
ticket
KDC
Page 10
SSL
Hive creates
map reduce
using NN ST
Client gets
query result
Ranger
B
HDFS
Security Features
PHD/HDP Security
Authentication
Kerberos Support
Authorizations
Fine grained access control
Role base access control
Column level
Permission Support
Auditing
Resource access auditing
Policy auditing
Page 11
Extensive Auditing
Security Features
HDP/PHD Security w/ Ranger
Data Protection
Wire Encryption
Volume Encryption
File/Column Encryption
TDE
HDFS TDE & Partners
Reporting
Global view of policies and audit data
Manage
User/ Group mapping
Global policy manager, Web UI
Delegated administration
Page 12
Partner Integration
Security Integrations:
Ranger plugins: centralize authorization/audit of 3rd party s/w in Ranger
UI
Via Custom Log4J appender, can stream audit events to INFA infrastructure
Page 13
Authentication w/ Kerberos
Page 14
Page 14
Page 15
Page 22
Page 22
Audit
Extensive user access auditing in
HDFS, Hive and HBase
IP Address
Timestamp
Access
granted or denied
Hortonworks Inc. 2011 2014. All Rights Reserved
Page 23
Flexibility
in defining
policies
Control
access into
system
Centralizes administration of
security policy
Page 24
file level
access
control,
flexible
definition
Control
permissions
25
Page 25
26
Page 26
Page 27
Hadoop Components
Enterprise
Enterprise Services:
Services: Security
Security
Enterprise
Users
Ranger Audit
Server
HDFS
HBase
Ranger
Plugin
Hive Server2
Ranger
Plugin
HDFS
Ranger
Plugin
Storm
Ranger
Plugin
HDP 2.2 Additions
Page 28
Ranger Policy
Server
Legacy Tools
& Data
Governance
Integration API
Ranger
Plugin
Knox
Ranger
Plugin*
TBD
Installation Steps
Install PHD 3.0
Install Apache Ranger (https://tinyurl.com/mlgs3jy)
Verify http://<host>:6080/
-
Page 29
admin/admin
Ranger Plugins
HDFS
HIVE
KNOX
STORM
HBASE
Page 30
Ranger Console
31
Page 31
Repository Manager
Add New Repository
Edit Repository
Delete Repository
32
Page 32
Demo
33
Page 33
Page 34
Page 34
Page 35
Apache Knox
Knox can be used with both unsecured Hadoop clusters, and Kerberos secured clusters. In an enterprise
solution that employs Kerberos secured clusters, the Apache Knox Gateway provides an enterprise security
solution that:
Integrates well with enterprise identity management solutions
Protects the details of the Hadoop cluster deployment (hosts and ports are hidden from end users)
Simplifies the number of services with which a client needs to interact
Page 36
Falcon
Oozie
Scoop
Flume
Data
Operator
Page 37
App B
REST/HTTP
Application Tier
Load Balancer
App C
App N
JDBC/ODBC
Knox
Data Ingest
Hadoop Cluster
ETL
RPC Call
Bastian Node
SSH
Hadoop
Admin
Admin/
Operators
SSL
Original
request w/user
id/password
SSL
Apache
Use Hive ST,
submit query
Knox
SASL
SSL
HiveServer 2
Beeline
Client
Knox gets
service ticket for
Hive
Hive gets
Namenode
(NN) service
ticket
KDC
Page 38
SSL
Hive creates
map reduce
using NN ST
Client gets
query result
Ranger
B
HDFS
Why Knox?
Enhanced Security
Protect network details
SSL for non-SSL services
WebApp vulnerability filter
Simplified Access
Kerberos encapsulation
Extends API reach
Single access point
Multi-cluster support
Single SSL certificate
Page 39
Centralized Control
Central REST API auditing
Service-level authorization
Alternative to SSH edge node
Enterprise Integration
LDAP integration
Active Directory integration
SSO integration
Apache Shiro extensibility
Custom extensibility
Direct URL
Knox URL
WebHDFS http://namenode-host:50070/webhdfs
https://knox-host:8443/webhdfs
WebHCat
http://webhcat-host:50111/templeton
https://knox-host:8443/templeton
Oozie
http://ooziehost:11000/oozie
https://knox-host:8443/oozie
HBase
http://hbasehost:60080
https://knox-host:8443/hbase
http://hivehost:10001/cliservice
https://knox-host:8443/hive
http://yarn-host:yarn-port/ws
https://knox-host:8443/resourcemanager
Hive
YARN
Page 40
Masters could
be on many
different
hosts
Hortonworks
Inc. 2011 2014.
All Rights Reserved
One hosts,
one port
SSL config
at one host
Consistent
paths
Firewall
Firewall
DMZ
Masters
Masters
NN
NN
Edge
Node/Hado
op CLIs
RM
RM
RPC
Oozie
Oozie
Web
Web
HCat
HCat
Knox
Knox Gateway
Gateway
HTTP
LB
NM
NM
HTTP
HTTP
GW
GW
GW
GW
Hadoop
Hadoop Cluster
Cluster 22
Masters
Masters
NN
NN
RM
RM
LDAP
Page 41
HS2
HS2
Slaves
Slaves
DN
DN
REST
REST
Client
Client
HBase
HBase
Enterprise
Enterprise
Identity
Identity
Provider
Provider
LDAP/AD
LDAP/AD
Oozie
Oozie
HBase
HBase
Web
Web
HCat
HCat
Slaves
Slaves
DN
DN
Page 41
NM
NM
HS2
HS2
Page 42
Installation
Installed via Ambari
This can be done manually
Start the embeded ldap
Page 43
Data Protection
Wire and data at rest encryption
Page 44
Page 44
Data Protection
HDP allows you to apply data protection policy at
different layers across the Hadoop stack
Layer
What?
How ?
Storage and
Access
Transmission
Page 45
Page 53
http://hortonworks.com/kb/hdfs-transparent-data-encryption/
Page 54
Thank You
Page 55