1.4 HDFS Lab 1H

IBM Software
Information Management
HDFS
Hands-On Lab
IBM Software
Table of Contents
1 Introduction ....................................................................................................................................................................................... 3
2 About this Lab ................................................................................................................................................................................... 3
3 Environment Setup Requirements................................................................................................................................................... 3
3.1 Getting Started .............................................................................................................................................................................................. 3
4 Hadoop Administration..................................................................................................................................................................... 7
4.1 HDFS Disk Check.......................................................................................................................................................................................... 7
5 Exploring Hadoop Distributed File System (HDFS)........................................................................................................................ 8
5.1 Using the command line interface .................................................................................................................................................................. 9
6 Configuring Hadoop Default Settings (Advanced) ....................................................................................................................... 16

6.1 Increasing storage Block Size...................................................................................................................................................................... 16
6.2 Limit Data node’s disk usage .......................................................................................................................... Error! Bookmark not defined.
6.3 Configuring the replication factor ................................................................................................................................................................. 17
7 Launching the BigInsights Web Console...................................................................................................................................... 18
8 Working with Files in Web Console............................................................................................................................................... 18
9 Summary.......................................................................................................................................................................................... 22
Hadoop Core – HDFS

© Copyright IBM Corp. 2013. All rights reserved Page 2 of 23
IBM Software
1 Introduction
The overwhelming trend towards digital services, combined with cheap storage, has generated massive amounts of data that
enterprises need to effectively gather, process, and analyze. Techniques from the data warehousing and high-performance
computing communities are invaluable for many enterprises. However, often times their cost or complexity of scale-up
discourages the accumulation of data without an immediate need. As valuable knowledge may nevertheless be buried in this
data, related scaled-up technologies have been developed. Examples include Google’s MapReduce, and the open-source
implementation, Apache Hadoop.
Hadoop is an open-source project administered by the Apache Software Foundation. Hadoop’s contributors work for some of the
world’s biggest technology companies. That diverse, motivated community has produced a collaborative platform for
consolidating, combining and understanding data.
Technically, Hadoop consists of two key services: data storage using the Hadoop Distributed File System (HDFS) and large
scale parallel data processing using a technique called MapReduce
2 About this Lab

After completing this hands-on lab, you’ll be able to:
• Use Hadoop commands to explore the HDFS on the Hadoop system
• Use Infosphere BigInsights Web console to explore the HDFS on the Hadoop system
3 Environment Setup Requirements

To complete this lab you will need the following:
1. InfoSphere BigInsights Bootcamp VMware® image
2. VMware Player 2.x or VMware Workstation 5.x or later
For help on how to obtain these components please follow the instructions specified in VMware Basics and Introduction from
module 1.
3.1 Getting Started

To prepare for the contents of this lab, you must go through the process of getting all of the Hadoop components started.
1. Start the VMware image by clicking the button in VMware Workstation if it is not already on.
2. Log in to the VMware virtual machine using the following information:
• User: root
• Password: passw0rd

IBM Software
The first time you start the image a startup script might ask you a couple questions, please use the defaults for this.
You will be asked to login a second time after the script has run. This will not always be the case if your class
manager has already set up the images.
All userids in the image we have the password “passw0rd” ( the o being replaced with a zero ), We will use the root
user to login but we will be mostly working with the BigInsights Administrator biadmin. When you are supposed to
open a Terminal you should always use the user biadmin. There is a dedicated button for this.
3. Open Command Prompt Window selecting the “Terminal for biadmin” button
4. Change to the $BIGINSIGHTS_HOME (which by default is set to /opt/ibm/biginsights).

cd $BIGINSIGHTS_HOME/bin
or
cd /opt/ibm/biginsights/bin

IBM Software
5. Start the Hadoop components (daemons) on the BigInsights server. You can practice starting all components with these
commands. Please note they will take a few minutes to run:
./start-all.sh
If you want to start and stop BigInsights later there are also buttons available on the Desktop.
If you are prompted to authenticate the host, type ‘yes’ in order to proceed with the script:
The following figure shows the different Hadoop components starting.

IBM Software
Note: You may get an error that the server has not started, please be patient as it does take some time for the server to complete start.
6. Sometimes certain hadoop components may fail to start. You can start and stop the failed components one at a time by
using start.sh or stop.sh respectively. For example, to start and stop Hadoop use:
./start.sh hadoop
./stop.sh hadoop
In the following example, the console component failed. The particular component was then started again using
the ./start.sh console command. It then succeeded without any problems. This approach can be used for
any failed components. Once all components have started successfully, you can then move to the next section.

IBM Software
4 Hadoop Administration
4.1 HDFS Disk Check
There are various ways to monitoring the HDFS Disk, and this should be done occasionally to avoid space issues which can
arise if there is low disk storage remaining. One such issue can occur if the “hadoop healthcheck” or heartbeat as it is also
referred to sees that a node has gone offline. If a node is offline for a certain period of time, the data that the offline node was
storing will be replicated to other nodes (since there is a 3 node replication, the data is still available on the other 2 nodes). If
there is limited disk space, this can quickly cause an issue.
1. You can quickly access the HDFS report by executing the following command:
hadoop dfsadmin -report

IBM Software
5 Exploring Hadoop Distributed File System (HDFS)

Hadoop Distributed File System (HDFS), allows user data to be organized in the form of files and directories. It provides a
command line interface called FS shell that lets a user interact with the data in HDFS accessible to Hadoop MapReduce
programs.
There are two methods to interact with HDFS:
1. You can use the command-line approach and invoke the FileSystem (fs) shell using the format: hadoop fs <args>.
This is the method we will use first.
2. You can also manipulate HDFS using the BigInsights Web Console. You will explore the BigInsights Web Console later
on

IBM Software
5.1 Using the command line interface
In this part, we will explore some basic HDFS commands. All HDFS commands start with hadoop followed by dfs (distributed
file system) or fs (file system) followed by a dash, and the command. Many HDFS commands are similar to UNIX commands.
For details, refer to the Hadoop Command Guide and Hadoop FS Shell Guide.
Before we can explore some HDFS commands we must ensure that the HDFS daemons are running for the cluster. The start-
dfs.sh script is responsible for starting the HDFS daemons. We’ll also verify that the MapReduce daemons are running by
executing the start-mapred.sh script; this is required for later sections.
3. Start the HDFS daemons and the MapReduce daemons by executing the following commands:
start-dfs.sh
start-mapred.sh
Note: Since we just started all processes beforeyou should see the result below when running the scripts, showing that the HDFS and
mapreduce daemons are already running. In that case, you can proceed to the next step.
We will start with the hadoop fs –ls command which returns the list of files and directories with permission information.

IBM Software
Ensure the Hadoop components are all started, and from the same Gnome terminal window as before (and logged on as
biadmin), follow these instructions:
4. List the contents of the root directory.
hadoop fs -ls /
5. To list the contents of the /user/biadmin directory, execute:

hadoop fs -ls
or
hadoop fs -ls /user/biadmin
Note that in the first command there was no directory referenced, but it is equivalent to the second command where
/user/biadmin is explicitly specified. Each user will get its own home directory under /user. For example, in the case of
user biadmin, his home directory is /user/biadmin. Any command where there is no explicit directory specified will be
relative to the user’s home directory.

IBM Software
6. To create the directory myTestDir you can issue the following command:
hadoop fs -mkdir myTestDir
Where was this directory created? As mentioned in the previous step, any relative paths will be using the user’s home
directory.
7. Issue the ls command again to see the subdirectory myTestDir:
hadoop fs -ls
or
hadoop fs –ls /user/biadmin
Note: If you specify a relative path to hadoop fs commands, they will implicitly be relative to your user directory in HDFS. For
example when you created the directory myTestDir, it was created in the /user/biadmin directory.
To use HDFS commands recursively generally you add an “r” to the HDFS command (In the Linux shell this is generally
done with the “-R” argument).
8. For example, to do a recursive listing we’ll use the –lsr command rather than just –ls, like the examples below:
hadoop fs -ls /user

hadoop fs -lsr /user

IBM Software
9. You can pipe (using the | character) any HDFS command to be used with the Linux shell. For example, you can easily
use grep with HDFS by doing the following:
hadoop fs -mkdir /user/biadmin/myTestDir2

hadoop fs -ls /user/biadmin | grep Test
As you can see the grep command only returned the lines which had test in them (thus removing the “Found x items”
line and the .staging and oozie-biad directories from the listing
10. To move files between your regular Linux filesystem and HDFS you can use the put and get commands. For example,
move the text file README to the hadoop filesystem.
hadoop fs -put /home/biadmin/labs/hdfs/README README

hadoop fs -ls /user/biadmin
You should now see a new file called /user/biadmin/README listed as shown above. Note there is a ‘3’ highlighted in
the figure. This represents the replication factor so blocks of this file are replicated in 3 places.

IBM Software
11. In order to view the contents of this file use the –cat command as follows:
hadoop fs -cat README
You should see the output of the README file (that is stored in HDFS). We can also use the linux diff command to see
if the file we put on HDFS is actually the same as the original on the local filesystem.
12. Execute the commands below to use the diff command:
cd /home/biadmin/labs/hdfs
diff <( hadoop fs -cat README ) README
Since the diff command produces no output we know that the files are the same (the diff command prints all the lines in
the files that differ).
To find the size of files you need to use the –du or –dus commands. Keep in mind that these commands return the file
size in bytes.
13. To find the size of the README file use the following command:
hadoop fs -du README
In this example, the README file has 18 bytes.
14. To find the size of all files individually in the /user/biadmin directory use the following command:
hadoop fs -du /user/biadmin

IBM Software
15. To find the size of all files in total of the /user/biadmin directory use the following command:
hadoop fs -dus /user/biadmin
16. If you would like to get more information about hadoop fs commands, invoke –help as follows:
hadoop fs –help

IBM Software
17. For specific help on a command, add the command name after help. For example, to get help on the dus command
you’d do the following:
hadoop fs -help dus

IBM Software
6 Configuring Hadoop Default Settings
InfoSphere BigInsights has imported certain attributes from Apache Hadoop, and some of them have been changed to improve
performance. One such attribute is the default block size used for storing large files.
Consider the following short example. You have a 1GB file, on a 3-node replication cluster. With a block-size of 128MB, this file
will be split into 24 blocks (8 blocks, each replicated 3 times), and then stored on the Hadoop cluster accordingly by the master
node. Increasing and decreasing the block size can have very specific use-case implementations; however for the sake of this
lab we will not cover those Hadoop specific questions, but rather how to change these default values.
6.1 Increasing storage Block Size
Hadoop uses a standard block storage system to store the data across its data nodes. Since block size is slightly more of an
advanced topic, we will not cover the specifics as to what and why the data is stored as blocks throughout the cluster.
The default block size value for IBM BigInsights 2.0 is currently set at 128MB (as opposed to the Hadoop default of 64MB as you
will see in the steps below). If your specific use-case requires you to change this, it can be easily modified through Hadoop
configuration files.
1. When making any Hadoop core changes, it is good practice (and a requirement for most) to stop the services you are
changing before making any necessary changes. For the block size, you must stop the “Hadoop” and “Console”
services before proceeding if you have not done so in the step, and re-start them after you have made the changes.
2. Move to the directory where Hadoop configuration files are stored.

cd $BIGINSIGHTS_HOME/hadoop-conf
ls
3. Within this directory, you will see a file named “hdfs-site.xml”, one of the site-specific configuration files, which is on
every host in your cluster.
gedit hdfs-site.xml&
4. Navigate to the property named ‘dfs.block.size’, and you will see the value is set to 128MB, the default block size for
BigInsights. For the purpose of this lab, we will not change the value. If you wanted to change it you would need to

IBM Software
modify this setting and then restart your cluster. The new setting would be applied to all new created files. It is also
possible to change this settings for a specific file.
6.2 Configuring the replication factor
1. Navigate to the property named ‘dfs.replication’. If it is not specified in this file, then default replication value is used
which is specified in hdfs-default.xml.
2. hdfs-default.xml is stored at $BIGINSIGHTS_HOME/IHC/src/hdfs.
3. The current default replication factor value is 1. You can overwrite the default value by adding the following lines to
this file(hdfs-site.xml). The value will be the number of your choice.
For the purpose of this lab, we will not save this configuration change. This part of the lab is just to let you browse how to
change some of the configuration values when you need it later on. In a development environment a low replication factor of
1 may be applicable. For a production environment you shouldn’t change the factor.

IBM Software
7 Launching the BigInsights Web Console
1. Launching the web console is done by entering a URL into a web browser. The format for the URL is:
http://<host>:<port> or http://<host>:<port>/data/html/index.html
The default is:
http://bigdata:8080/data/html/index.html
For convenience, there is a shortcut on the biadmin's user Desktop, which will launch the web console when double-
clicked. It is called BigInsights Console.
2. If you use the link you will already have the username and password entered. If you enter the URL the login credentials
are
• User: biadmin
• Password: passw0rd
3. Verify that the BigInsights Web Console looks like this:
8 Working with Files in Web Console

The Files tab of the console enables you to explore the contents of your file system, create new subdirectories, upload small
files for test purposes, and perform other file-related functions. In this module, you’ll learn how to perform such tasks against the
Hadoop Distributed File System (HDFS) of BigInsights.
Examples in this section use the biadmin user which has a /user/biadmin directory in its distributed file system. If you're
accessing a BigInsights cluster using a different user ID, adapt the instructions in this exercise to work with your home directory
in HDFS.
1. Click the Files tab of the console to begin exploring the distributed file system.

IBM Software
2. Expand the directory tree shown in the pane at left. If files have been uploaded to HDFS before, you’ll be able to
navigate through the directory to locate them. In our case, the figure below shows /user/biadmin containing default
subdirectories.
3. Become familiar with the functions provided through the icons at the top of this pane, as we'll refer to some of these in
subsequent sections of this module. Simply hover your cursor over the icon to learn its function.
From left to right, the icons enable you to create a directory, upload a file to HDFS, download a file from HDFS to your
local file system, delete a file from HDFS, open a command window to launch HDFS shell commands, and refresh the
Web console page.
4. Position your cursor on the user/biadmin directory and click the Create Directory icon (at far left) to create a
subdirectory for test purposes.
5. When a pop-up window appears prompting you for a directory name, enter MyDirectory and click OK.

IBM Software
6. Expand the directory hierarchy to verify that your new subdirectory was created.
7. Click the Upload button to upload a small sample file from your local file system to HDFS.
8. When the pop-up window appears, click the Browse button to browse your local file system for a sample file.
9. Navigate through your local file system to the directory biadmin/labs/hdfs and select the CHANGES.txt file. Click OK.
10. Verify that the window displays the name of this file. Note that you can continue to click Choose file for additional files
to upload and that you can delete files as upload targets from the displayed list. However, for this exercise, simply click
OK.

IBM Software
11. When the upload completes, verify that the CHANGES.txt file appears in the directory tree at left. On the right, you
should see a subset of the file’s contents displayed in text format.
12. Let’s now download the same file from HDFS to your local file system. Highlight the CHANGES.txt file in your
MyDirectory directory and click the Download button. (This button is between the Upload and Delete buttons.)
13. When prompted, click the Save File button. Then select OK.

IBM Software
When prompted, identify the folder (directory) of your local file system where the file should be stored. Wait until the Web
browser indicates that the file download is complete.
9 Summary
You have just completed Lab 1 which focused on the basics of the Hadoop platform, including HDFS and a quick introduction to
the web console. You should now know how to perform the following basic tasks on the platform:
• Start/Stop the Hadoop components

• Interact with the data in the Hadoop Distributed File System (HDFS)
• Change Hadoop configuration values.
• Navigate within HDFS using both Hadoop commands and BigInsights Web Console

IBM Software
© Copyright IBM Corporation 2013

All Rights Reserved.
IBM Canada
8200 Warden Avenue
Markham, ON
L6G 1C7
Canada
IBM, the IBM logo, ibm.com and Tivoli are trademarks or registered
trademarks of International Business Machines Corporation in the
United States, other countries, or both. If these and other
IBM trademarked terms are marked on their first occurrence in this
information with a trademark symbol (® or ™), these symbols indicate
U.S. registered or common law trademarks owned by IBM at the time
this information was published. Such trademarks may also be
registered or common law trademarks in other countries. A current list
of IBM trademarks is available on the Web at “Copyright and
trademark information” at ibm.com/legal/copytrade.shtml
Other company, product and service names may be trademarks or

service marks of others.
References in this publication to IBM products and services do not

imply that IBM intends to make them available in all countries in which
IBM operates.
No part of this document may be reproduced or transmitted in any form

without written permission from IBM Corporation.
Product data has been reviewed for accuracy as of the date of initial
publication. Product data is subject to change without notice. Any
statements regarding IBM’s future direction and intent are subject to
change or withdrawal without notice, and represent goals and
objectives only.
THE INFORMATION PROVIDED IN THIS DOCUMENT IS

DISTRIBUTED “AS IS” WITHOUT ANY WARRANTY, EITHER
EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE OR NON-INFRINGEMENT.
IBM products are warranted according to the terms and conditions of

the agreements (e.g. IBM Customer Agreement, Statement of Limited
Warranty, International Program License Agreement, etc.) under which
they are provided.


1.4 HDFS Lab 1H

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

1.4 HDFS Lab 1H

Încărcat de

Drepturi de autor:

Formate disponibile

IBM Software

6 Configuring Hadoop Default Settings (Advanced) ....................................................................................................................... 16

Hadoop Core – HDFS

2 About this Lab

• Use Hadoop commands to explore the HDFS on the Hadoop system

3 Environment Setup Requirements

3.1 Getting Started

Hadoop Core – HDFS

4. Change to the $BIGINSIGHTS_HOME (which by default is set to /opt/ibm/biginsights).

Hadoop Core – HDFS

The following figure shows the different Hadoop components starting.

Hadoop Core – HDFS

Hadoop Core – HDFS

Hadoop Core – HDFS

5 Exploring Hadoop Distributed File System (HDFS)

There are two methods to interact with HDFS:

Hadoop Core – HDFS

5.1 Using the command line interface

Hadoop Core – HDFS

5. To list the contents of the /user/biadmin directory, execute:

Hadoop Core – HDFS

hadoop fs -ls /user

Hadoop Core – HDFS

hadoop fs -mkdir /user/biadmin/myTestDir2

hadoop fs -put /home/biadmin/labs/hdfs/README README

Hadoop Core – HDFS

hadoop fs -cat README

diff <( hadoop fs -cat README ) README

hadoop fs -du README

In this example, the README file has 18 bytes.

hadoop fs -du /user/biadmin

Hadoop Core – HDFS

hadoop fs -dus /user/biadmin

Hadoop Core – HDFS

hadoop fs -help dus

Hadoop Core – HDFS

6 Configuring Hadoop Default Settings

6.1 Increasing storage Block Size

2. Move to the directory where Hadoop configuration files are stored.

Hadoop Core – HDFS

6.2 Configuring the replication factor

Hadoop Core – HDFS

7 Launching the BigInsights Web Console

3. Verify that the BigInsights Web Console looks like this:

8 Working with Files in Web Console

Hadoop Core – HDFS

Hadoop Core – HDFS

Hadoop Core – HDFS

Hadoop Core – HDFS

• Start/Stop the Hadoop components

Hadoop Core – HDFS

© Copyright IBM Corporation 2013

Other company, product and service names may be trademarks or

References in this publication to IBM products and services do not

No part of this document may be reproduced or transmitted in any form

THE INFORMATION PROVIDED IN THIS DOCUMENT IS

IBM products are warranted according to the terms and conditions of

Hadoop Core – HDFS

S-ar putea să vă placă și