PRERREQUISITS INSTALLATION

SUSE
1. Install SUSE Service Pack 2 or later
2. Install SUSE Linux Enterprise Software Development Kit 11 SP1

FILESYSTEM
3. Increase the ulimit to 32768 for users: mtxhdfs and mtxhbase

JAVA
4. Remove any IBM or default distribution version installed
5. Install Oracle 1.7.0_67 (update 67) JDK
(http://loninenghadoop36.uk.db.com/CDH5/cm/sles/5.2.0/RPMS/x86_64/oraclej2sdk1.7-1.7.0+update67-1.x86_64.rpm)

PYTHON
6. Install Python 2.7 (or 2.6) on the CDH Cloudera Manager Node

MySQL
7. Install MySQL 5.6 or 5.5 on the CDH Cloudera Manager Node mounted on
"/opt/mtxsys" directory
8. The InnoDB storage engine must be enabled in the MySQL server.
9. Create MySQL Root User and provide the password. Needed for Cloudera Manager
installation.
10. In SUSE make sure the JDBC connector for MySQL is installed
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_i
g_mysql.html?scroll=cmig_topic_5_5

NETWORKING

11. Cluster hosts must have a working network name resolution system and
correctly formatted /etc/hosts file. All cluster hosts must have properly configured
forward and reverse host resolution through DNS. The /etc/hosts files must
Contain consistent information about hostnames and IP addresses across all hosts
Not contain uppercase hostnames
Not contain duplicate IP addresses
12. Also, do not use aliases, either in /etc/hosts or in configuring DNS. A properly
formatted /etc/hosts file should be similar to the following example:

127.0.0.1

localhost.localdomain

localhost

192.168.1.1 cluster-01.example.com

cluster-01

192.168.1.2 cluster-02.example.com

cluster-02

192.168.1.3 cluster-03.example.com

cluster-03

13. The canonical name of each host in /etc/hosts must be the FQDN (for example
myhost-1.mynet.myco.com), not the unqualified hostname (for example myhost-1).
The canonical name is the first entry after the IP address.
14. Make sure the /etc/sysconfig/network file on each system contains the hostname
you have just set (or verified) for that system, for example myhost-1.
15. No blocking is done by Security-Enhanced Linux (SELinux).
16. IPv6 must be disabled.
17. No blocking by iptables or firewalls; port 7180 must be open because it is used
to access Cloudera Manager after installation. Cloudera Manager communicates
using specific ports, which must be open.
18. Set /proc/sys/vm/swappiness to 0.
Use the sysctl command to change this setting at runtime and edit /etc/sysctl.conf
for this setting to be saved after a reboot.

CLOUDERA HADOOP (INSTALL)

*** We would need a UNIX SA open support window also during our
setup steps to avoid get blocked due to root rights restrictions: ***

1. Create the file /etc/zypp/repos.d/cloudera-manager.repo with the following

contents in all nodes:
[cloudera-manager]
name=Cloudera Manager, Version 5.2.0
enabled=1
baseurl=http://loninenghadoop36.uk.db.com/CDH5/cm/sles/5.2.0/RPMS/x8
6_64/
gpgcheck = 0

2. zypper install cloudera-manager-server cloudera-manager-daemons clouderamanager-agent for the Cloudera Manager Nodes (longrtmxd52)
-

Once installed, modify the file /etc/cloudera-scm-agent/config.ini for all P

hosts (longrtmxd52 - longrtmxd56):

# Hostname of Cloudera SCM Server

server_host= longrtmxd52.uk.db.com
-

And for all B hosts (longrtmxd52 longrtmxd56):

# Hostname of Cloudera SCM Server

server_host= longrtmxd52.uk.db.com
3. Configure Cloudera Mananager /etc/cloudera-scm-server/db.properties
considering the following setup:
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql
# The database host
# If a non standard port is needed, use 'hostnameort'
com.cloudera.cmf.db.host=localhost

# The database name

com.cloudera.cmf.db.name=scm
# The database user
com.cloudera.cmf.db.user=scm
# The database user's password
com.cloudera.cmf.db.password=scm_password

4. Start creating the databases in mysql. Use the databases and username as below
since they are the Cloudera default names.
$ mysql -u root
#this is the most important user. Without it the cloudera server will not start
properly.
create database scm DEFAULT CHARACTER SET utf8;
grant all on scm.* TO 'scm'@'%' IDENTIFIED BY 'scm_password';
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
create database metastore DEFAULT CHARACTER SET utf8;
grant all on metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry_password';
create database nav DEFAULT CHARACTER SET utf8;
grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav_password';
create database navms DEFAULT CHARACTER SET utf8;
grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms_password';
For more info:
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v4latest/Cloudera-Manager-InstallationGuide/cmig_install_mysql.html#cmig_topic_5_5_2_unique_1

5.start the Cloudera Manager:

service cloudera-scm-server start

5. Install Cloudera Manager Agents on all the other boxes

zypper install cloudera-manager-agent

5. Copy the files /etc/cloudera-scm-agent/config.ini from the Cloudera manager box

to all the other boxes (longrtmxd52 longrtmxd56).

7. On the Cloudera Manager Hosts (longrtmxd52), start all the services (root
REQUIRED):
service cloudera-scm-server start
service cloudera-scm-agent start

8. On all the other boxes, start the Cloudera Cloudera Manager Agents (root
REQUIRED):
service cloudera-scm-agent start

CLOUDERA HADOOP (SETUP)

Once Cloudera Manager and Cloudera Manager and Agents are installed and
running:
First Log in with default admin credentials and you should see the beginning of the
first installation wizard.
Wizard:
1) Choose Cloudera Express Version.
2) If Agents are correctly running and configured. All the servers should appear
in the hosts list. Select them.
3) Configure parcel repository location
(http://loninenghadoop36.uk.db.com/CDH5/parcels/5.2.1/) and then the
wizard begins to install and distribute the several components in the servers
selected
4) After the components activation, a Hosts Inspector runs and detects
problems. If any resolve them.
Examples from earlier UAT-PRD setup:
Issue: mxhadb63.uk.db.com: Inspector did not run successfully
Try to restart the cloudera-scm-agent in mxhadb63.
Issue: There are mismatched versions across the system, which will cause failures. See below for
details on which hosts are running what versions of components.
Compare the mismatched versions and make them match.

Raised SR14633648:
- HOST: mxhadb59.uk.db.com
1) The group 'hadoop' does not include the user 'hdfs' on
mxhadb59.uk.db.com. Please add it.
2) The group 'hadoop' does not include the user 'mapred' on
mxhadb59.uk.db.com. Please add it.
2.bis) The group 'hadoop' does not include the user yarn on
mxhadb59.uk.db.com. Please add it.
- HOST: mxhadb60.uk.db.com
3) Uninstall Java 6 (java version "1.6.0_31") on mxhadb60.uk.db.com.
- HOST: mxhadb63.uk.db.com
4) Uninstall current Java 6 IBM version on mxhadb63.uk.db.com
5) Install "Oracle" JDK 1.7.0_67 on mxhadb63.uk.db.com
Location:
http://loninenghadoop36.uk.db.com/CDH5/cm/sles/5.2.0/RPMS/x86_64/oraclej2sdk1.7-1.7.0+update67-1.x86_64.rpm
6) Please restart the cloudera-scm-agent (sudo service cloudera-scm-agent

Skip the wizard and go to Cloudera Manager Home page.

5) Add the Cloudera Manager Services to the Cloudera Manager node x60
- Set the amon mysql database and its credentials
- Add /opt/mtxsys/ prefix to the default locations in the locations set
requested
- Change default log directories /var/log default value to
/opt/mtxsys/var/log/
- TODO: Set Cloudera Manager Server Local Data Storage Directory /var/lib/cloudera-scmserver to /opt/mtxsys /var/lib/cloudera-scm-server

6) After fix all the check problems, the hosts must be already be created or
added to a default cluster (if not create the cluster and add all the
provisioned cluster nodes to the cluster)
7) Rename the cluster name if desired
The next steps could generate configuration issues alerts in
cloudera. We will fix them after installing all the roles/services at the
end. Omit then for now.
8) Install Zookeeper (b61,b62,b63)
- Add /opt/mtxsys/ prefix to the default value /var/lib location requested
- Change default log directories /var/log default value to
/opt/mtxsys/var/log/
- Enable auto-creation of data directories = true (default false)
- Set maxSessionTimeout to 80000 (default 40000)
- Maximum Latency Monitoring Thresholds Warning 80% (Default 75%)
- Server Max Log Size 100MB (Default 200MB)
- Server Maximum Log File Backups 5 (Default 10)
- Restart the cluster including Re-Deploy configuration.
9) Install HDFS.
- Install the services properly (NameNode, SecondaryNN, HttpFS and
DataNodes)
- Change the data locations prefixing them with /opt/mtxdata
Add /opt/mtxsys/ prefix to the default value /var/lib location requested
- Change default log directories /var/log default value to
/opt/mtxsys/var/log/
- Disable by unchecking the dfs.permissions
- Set Default Replication Factor to 2.
- Set dfs.datanode.max.locked.memory to 2GB (Default 4GB)
- Set dfs.datanode.du.reserved to 20GB (Default 10GB)
- DataNode Max Log Size 100MB (Default 200MB)
- DataNode Maximum Log File Backups 5 (Default 10)
- Failover Controller Max Log Size 100MB (Default 200MB)
- Failover Controller Maximum Log File Backups 5 (Default 10)
- Check that Use Trash is enabled
- HttpFS Max Log Max Log Size 100MB (Default 200MB)

10)
-

HttpFS Max Log Maximum Log File Backups 5 (Default 10)

JournalNode Log Max Log Size 100MB (Default 200MB)
JournalNode Max Log Maximum Log File Backups 5 (Default 10)
NFS Gateway Max Log Size 100MB (Default 200MB)
NFS Gateway Max Log Maximum Log File Backups 5 (Default 10)
Java Heap Size of Namenode in Bytes 6GB (Default 1GB)
Java Heap Size of Secondary namenode in Bytes 6GB (Default 1GB)
Set dfs.namenode.handler.count to 60 (Default 30)
Set dfs.namenode.service.handler.count to 60 (Default 30)
NameNode Max Log Max Log Size 100MB (Default 200MB)
NameNode Max Log Maximum Log File Backups 5 (Default 10)
SecondaryNameNode Max Log Max Log Size 100MB (Default 200MB)
SecondaryNameNode Max Log Maximum Log File Backups 5 (Default 10)
Deploy configuration and restart HDFS
Install HBase
Install the services properly (Masters, HBase REST Serverand
RegionServers)
Restart the cluster including Re-Deploy configuration.
Change any /var directory prefixing them with /opt/mtxys
Set hbase.hregion.max.filesize to 2GB (default 10GB)
HBase REST Server Maximum Log Max Log Size 100MB (Default 200MB)
HBase REST Server Maximum Log File Backups 5 (Default 10)
HBase Thrift Server Maximum Log Max Log Size 100MB (Default 200MB)
HBase Thrift Server Maximum Log File Backups 5 (Default 10)
Master Maximum Log Max Log Size 100MB (Default 200MB)
Master Maximum Log File Backups 5 (Default 10)
Java Heap Size of HBase RegionServer in Bytes 3GB (Default 4GB)
RegionServer Max Log Max Log Size 100MB (Default 200MB)
RegionServer Maximum Log File Backups 5 (Default 10)
Healthy NodeManager Monitoring Thresholds, Waring 60%(Default 95%)
and Critical 60% (Default 90%)
ApplicationMaster Java Opts Base: -Djava.net.preferIPv4Stack=true
-Xmx768m
Set mapreduce.map.java.opts: -Djava.net.preferIPv4Stack=true
-Xmx1024m
JobHistory Server Max Log Max Log Size 100MB (Default 200MB)
JobHistory Server Maximum Log File Backups 5 (Default 10)
Set yarn.nodemanager.resource.memory-mb to 4GB (Default 1.76GB)
Check that yarn.nodemanager.resource.cpu-vcores = 4 (Default 8). CDH
sets this automatically.
NodeManager Max Log Max Log Size 100MB (Default 200MB)
NodeManager Server Maximum Log File Backups 5 (Default 10)
Check that yarn.scheduler.maximum-allocation-mb = 1,76GB
ResourceManager Max Log Max Log Size 100MB (Default 200MB)
ResourceManager Server Maximum Log File Backups 5 (Default 10)
Restart the cluster including Re-Deploy configuration.

11)
-

Install YARN(MR2 Included)

Install the services properly (ResourceManager, JobHistoryTracker and
NodeManagers)
Set the Name Node yarn.nodemanager.local-dirs to
/opt/mtxdata/yarn/nm only.
Change any /var directory prefixing them with /opt/mtxys
Restart the cluster including Re-Deploy configuration.

12)
TODO: MX Custom Scan Filter Configuration. Place Filter in
HBase Region Classpath:
- HBase Service Environment Advanced Configuration Snippet (Safety Valve):
HBASE_CLASSPATH=/opt/mtxapps/shared/lui/rawexport.lui.jar

13)
Hosts Configuration - Memory Overcommit Validation Threshold: 0.9
(Default 0.8)
14)
Host longrtmxd52 (CM Node) - Memory Overcommit Validation
Threshold: 0.95 (Only this single host has a different all host configuration)
15)
TODO: Filesystem Collection Exclusion Regex: ???? (Default ^$). Set it as desired.
16)
Configuring Period Stacks Collection:
- Check that for all roles HDFS, HBase, YARN, Cloudera Services the Stacks
Collection Enabled property is disabled.
- Stacks Collection feature requires an extra amount of local filesystem
storage capacity that we currently cannot afford.
17)
Add HDFS DataNode and HBase RegionServer Roles to the reserved
host x59 that will run the matrix components (StatisticsJOB, Housekeeper)
and deploy the client configuration on it.
18)
Remove the x59 reserved host from the cluster (including roles
decommission option) and add it again without any roles.
19)
Check that the x59 host is working properly by:
- Execute $> hadoop fs ls / and check that list the HDFS root directory
elements
- Start the HBase shell hbase shell and execute list. The table list
should be empty without throwing any exception or error.
20)
Now is time to fix any configuration issue that Cloudera Manager
informs. Please follow Cloudera Managers instructions or suggestions
to fix them, such as increasing heap space memories,
21)
Cloudera Hadoop default local users change to NIS users. Create UNIX
SA SR in dbsympohny.

UsersMappings.xlsx

The Unix SA steps performed at this point are:

Please do the next tasks (step by step, NOT Server by Server please):
1) For ALL listed PRE-PRD servers stop cloudera-scm-agents:
- sudo service clouder-scm-agent stop
2) Only for longrtmxd52.uk.db.com server stop cloudera-scm-server:
- sudo service clouder-scm-server stop
3) For ALL listed PRE-PRD servers add if needed the NIS Users to the servers:
mtxscm
mtxhadop
mtxflume
mtxhbase
mtxhdfs
mtxhive
mtxhive
mtxhtpfs
mtxhue
mtximpal
mtxkms
mtxllama
mtxmaprd
mtxoozie
mtxsolr
mtxspark
mtxsntry
mtxsqoop
mtxqoop2
mtxyarn
mtxzkpr
4) For ALL listed PRE-PRD, If the directory exists, delete any data content of the following
directories (only content, never the directory itself):
/tmp
/opt/mtxsys/var/log/hbase
/opt/mtxsys/var/lib/hadoop-hdfs
/opt/mtxdata/dfs/dn
/opt/mtxsys/var/log/hadoop-hdfs
/opt/mtxdata/dfs/snn
/opt/mtxdata/dfs/nn
/opt/mtxsys/var/lib/zookeeper
/opt/mtxsys/var/lib/hadoop-yarn
/opt/mtxsys/var/log/hadoop-mapreduce
/opt/mtxdata/yarn/nm
/opt/mtxsys/var/log/hadoop-yarn
/opt/mtxsys/var/log/zookeeper

5) For ALL listed PRE-PRD servers, change every filesystem user and group that matches the
users/groups list provided.
See attached old to new user/group mappings "UsersMappings.xlsx" SpreadSheet.

6) Change the cloudera services users also in /etc/init.d

Old:
USER=cloudera-scm
GROUP=cloudera-scm
New:
USER=mtxscm
GROUP=mtxscm
7) For ALL listed PRE-PRD servers start the cloudera-scm-agent:
- sudo service clouder-scm-agent start
8) Only for longrtmxd52.uk.db.com server start cloudera-scm-server:
- sudo service clouder-scm-server start
9) Please run with "sudo -u mtxhdfs":
hadoop fs -chmod 777 /user/history
hadoop fs -chmod 755 /user
10) Delete the default local users:
cloudera-scm
flume
hbase
hdfs
hive
httpfs
hue
impala
kms
llama
mapred
oozie
sentry
solr
spark
sqoop2
sqoop
yarn
zookeeper

22)
After starting CM change any user or group in CM configuration the NIS
users properly
23)
10. After changing the user names in order to have HBASE working
after restart you have to delete the table in zookeeper, so from the shell
$ hbase zkcli
$ rmr /hbase
24)
Start Zookeeper role. Ussualy without issues
25)
Start HDFS role, if Namenode exited with failure (format namenode and
delete datanodes data
hadoop namenode format (user: mtxhdfs and sudo)
And then if java.io.IOException: Incompatible clusterIDs in
/home/hadoop/dfs/data: namenode clustered -> solution: delete dn content
/opt/mtxdata/dfs/dn on all DataNodes nodes.
26)
Start HBase role and check Masters and Region Servers are running.
27)
Cluster-wide Advanced Configuration Snippet (Safety Valve) for coresite.xml field value change.
This is necessary later to be able to user the httpFS service.
In the property names you have to put the names of the new users:
<property>
<name>hadoop.proxyuser.mtxhtpfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxhtpfs.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxoozie.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxoozie.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxhdfs.hosts</name>
<value>*</value>
</property>

<property>
<name>hadoop.proxyuser.mtxhdfs.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxflume.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxflume.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxmaprd.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxmaprd.groups</name>
<value>*</value>
</property>

28)
HBase Service Advanced Configuration Snippet (Safety Valve) for coresite.xml field value change:
<property>
<name>hadoop.proxyuser.mtxhtpfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mtxhtpfs.groups</name>
<value>*</value>
</property>

ISSUES DURING SETUP AND SOLUTION

1) Zookeeper. Deploying client configuration and restarting cluster: Command

'DeployClusterClientConfig' failed for cluster 'MX2 UAT/DR'. Ignore if there
actually isnt any configuration to deploy.
2) After changing user, change the cloudera services users also in /etc/init.d
USER=cloudera-scm
GROUP=cloudera-scm

If you forget to change the user that start the cloudera services it wont
startup.
3) During Cloudera Manager Start you see this in the /var/log/clouderascm.server.log: Caused by: java.lang.ClassNotFoundException: Could not load
requested class : com.mysql.jdbc.Driver. This means that Cloudera Manager does not
find the mysql-connector.jar in the appropriate location specified above in the
instructions.

4) Running any job such as the Statistics Job:

13:18:23
6062 [main] INFO JobSubmitter.submitJobInternal - Cleaning up the
staging area /user/mtxprd/.staging/job_1435915774270_0001
13:18:23
6067 [main] WARN UserGroupInformation.doAs PriviledgedActionException as:mtxprd (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=mtxprd, access=EXECUTE,
inode="/user":mtxmaprd:supergroup:drwxrwx--at
org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFs
Permission(DefaultAuthorizationProvider.java:255)
Solve it by running with sudo u mtxhdfs:
hadoop fs -chmod 777 /user/history
hadoop fs -chmod 755 /user
5) Cloudera Manager suffering checkpoint test failure due to Namenode
checkpoint failure:
The filesystem checkpoint is 4 hour(s), 23 minute(s) old. This is 439.31% of the configured
checkpoint period of 1 hour(s). Critical threshold: 400.00%. 3,137 transactions have occurred
since the last filesystem checkpoint. This is 0.31% of the configured checkpoint transaction
target of 1,000,000.

http://stackoverflow.com/questions/21732988/error-org-apache-hadoop-hdfsserver-namenode-secondarynamenode-exception-in-doc
Solve it by deleting the SecondaryNameNode data directory snn content that
maybe is corrupt due to user changes or old data remaining

OTHER (LINKS)
CLOUDERA HADOOP DOCUMENTATION

http://www.cloudera.com/content/cloudera/en/documentation.html

CLOUDERA HADOOP INSTALLATION AND DOCUMENTATION

-

http://www.cloudera.com/content/cloudera/en/downloads/cdh/cdh-5-20.html

CLOUDERA MANAGER REQUIREMENTS

-

http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2x/topics/cm_ig_cm_requirements.html

NETWORK CONFIGURATION
-

http://www.cloudera.com/content/cloudera/en/documentation/core/v5-2x/topics/cdh_ig_networknames_configure.html?scroll=topic_11_1