This is the RH342 Practice Exam.

Be sure to watch the introduction video for details regarding testing your progress.

Environment

LDAP/Kerberos authentication host:

ip-10-0-1-5.ec2.internal | 10.0.1.5

 LDAP Base: dc=example,dc=com

 Kerberos Realm: example.com

HostA:

ip-10-0-1-10.ec2.internal | 10.0.1.10

HostB:

ip-10-0-1-11.ec2.internal | 10.0.1.11

HostC:

ip-10-0-1-12.ec2.internal | 10.0.1.12

HostD:

ip-10-0-1-13.ec2.internal | 10.0.1.13

Changes requested in the tasks should persist through a restart of the system.

Solution
Start by logging in to the lab servers using the credentials provided on the hands-on lab
page:

ssh cloud_user@PUBLIC_IP_ADDRESS
Become the root user:

sudo su -

I recommend logging in to all four of the lab servers in separate tabs or windows to help
ensure we are running the right commands on the right hosts.

Also, keep in mind that there is a grading script that can be run on each host at any time
to help check your work. Be sure to replace the hostname with the name of the host you
are running the script on. Example:

score_HostA.sh

Task 1: Debug an application

On HostA, there is an application /root/example_app - troubleshoot and resolve this

application so that it will run successfully.

1. Install strace:

yum install -y strace

2. Run strace on the example_app:

strace example_app

3. Resolve the "No such file or directory" error by creating the necessary file:

touch /etc/example.conf

Task 2: Test an application for a memory leak

On HostA, test the application /root/memleak_test_app for a memory leak. If a

memory leak is detected, remove the executable bit from the application.

1. Install valgrind:

yum install -y valgrind

2. Run valgrind against memleak_test_app:

valgrind ./memleak_test_app

3. valgrind tells us there is indeed a memory leak. Remove the ability to execute
the application:

chmod -x memleak_test_app

Task 3: Manage a kernel module

On HostA, configure the module nf_conntrack to enable timestamping of network
flows. This change should persist through a reboot. Any services stopped to make this
change should be restarted.

1. Stop the firewall:

systemctl stop firewalld

2. Unload the module:

modprobe -r nf_conntrack

3. Load it w/ parameter:

modprobe nf_conntrack tstamp=1

4. Make it persist:

echo 'options nf_conntrack tstamp=1' >>

/etc/modprobe.d/nf_conntrack.conf

Task 4: Upgrade a package

On HostA, upgrade Apache (httpd) to the latest version and apply version control to
limit future upgrades.

1. Attempt to upgrade:

yum upgrade -y httpd

2. Edit /etc/yum.conf:

vim /etc/yum.conf

At the bottom of the file, remove the exclusion:

exclude=httpd

Save and close the file:

:wq

3. Upgrade Apache:

yum upgrade -y httpd

4. Install versionlock:

yum install -y yum-plugin-versionlock

5. Use versionlock:
 yum versionlock httpd

Task 5: Resolve SELinux issues

On HostA, resolve any issues starting the Apache (httpd) service, or issues viewing the
content. The existing web content and configuration should not be modified.

1. Start Apache:

systemctl start httpd

2. Enable the service to start on system boot:

systemctl enable httpd

3. Attempt to display the default webpage:

curl localhost

4. Use yum to determine which package provides sealert to help us troubleshoot

the "Permission Denied" error:

yum provides */sealert

5. Install the necessary packages:

yum install -y setroubleshoot stroubleshoot-server

6. Load the audit log into sealert:

sealert -a /var/log/audit/audit.log

7. Find the error and utilize the suggested fix, enabling the boolean to allow httpd
to read user content:

setsebool -P httpd_read_user_content 1

8. Let's go ahead and enable the httpd service to start on system boot:

systemctl enable httpd

Task 6: Resolve authentication issues

On HostA, the LDAP/Kerberos user testuser01 should be able to log in and obtain a
Kerberos ticket using the password welcome1.

1. View the contents of authconfig to ensure the necessary configuration is set:

cat /etc/sysconfig/authconfig

2. Launch authconfig-tui:
 authconfig-tui

o For the LDAP Settings:

 Server: ldap://10.0.1.5
o For Kerberos Settings:
 Realm: EXAMPLE.COM
 KDC: 10.0.1.5
 Admin Server: 10.0.1.5
3. Restart LDAP name scheduler:

systemctl restart nslcd

4. Log in to localhost as the testuser01 user:

ssh testuser01@localhost

5. Issue a Kerberos ticket:

kinit

6. List the Kerberos ticket:

klist

7. Log out as the testuser01 user:

exit

Task 7: Monitor system for vital characteristics

On HostA, there is a Performance Co-Pilot archive file. What was the 1 min kernel load
at 17:19:46?

Enter the value as a number into /root/load.txt. For example, if the load was 7.700,
the command would be:

echo "7.700" > /root/load.txt

1. Extract the archive:

tar xvzf archive.tgz

2. Install Performance Co-Pilot:

yum install -y pcp

3. Start Performance Co-Pilot:

systemctl start pmcd

 4. Use pmval to determine the kernel load at the specified time:

pmval -t1m kernel.all.load -a ip-10-0-1-

10.ec2.internal/20190324.17.08.0

5. Insert the requested value into load.txt:

echo 9.090 >> /root/load.txt

6. We are finished with the tasks for HostA. Let's run the grading script to ensure
all of the steps were completed successfully:

score_HostA

Reboot HostA:

shutdown -r now

When the server has finished rebooting, log back in:

ssh cloud_user@PUBLIC_IP_ADDRESS

Become the root user:

sudo su -

Run the grading script again to ensure all tasks are able to persist through a
reboot:

score_HostA

Task 8: Recover data from a corrupted file system

On HostB, recover the data from the filesystem mounted at /storage.

The integrity of the files on the filesystem must be preserved.

1. View the contents of /etc/fstab:

cat /etc/fstab

2. Recover the ext4 filesystem:

fsck.ext4 -y /dev/mapper/vg_2-lv_2

3. Attempt to mount /storage:

mount /storage

4. View the contents of /storage:

ll /storage
Task 9: Recover data from a broken LVM configuration

On HostB, troubleshoot and resolve the LVM issues preventing the data on /lvm_mount
from being accessed.
The integrity of the files on the filesystem must be preserved.

1. View the contents of /etc/fstab:

cat /etc/fstab

2. Attempt to mount /lvm_mount:

mount /lvm_mount/

3. Use vgcfrestore to determine what broke the LVM:

vgcfgrestore -l vg_1

4. Use the config file from before the volume group was resized:

vgcfgrestore -f /etc/lvm/archive/vg_1_00002-1291080026.vg vg_1

5. Disable the logical volume:

lvchange -an /dev/vg_1/lv_1

6. Active the logical volume:

lvchange -ay /dev/vg_1/lv_1

7. Mount the logical volume:

ll /lvm_mount

Task 10: Recover data from an encrypted file system

On HostB, troubleshoot and resolve the issues preventing the data located at /luks from
being accessed. The integrity of the files on the filesystem must be preserved.

1. View the contents of /etc/fstab:

cat /etc/fstab

2. View the contents of /etc/crypttab:

cat /etc/crypttab

3. View slots:

cryptsetup luksDump /dev/mapper/vg_2-lv_1

 4. Restore the header:

cryptsetup luksHeaderRestore --header-backup-file /root/vg_2-

lv_1.header /dev/mapper/vg_2-lv_1

5. View the contents of /etc/crypttab again:

cat /etc/crypttab

6. Manually unlock:

cryptsetup luksOpen /dev/mapper/vg_2-lv_1 luks-vg_2-lv_1 --key-

file /root/passphrase.key

7. View the contents of /luks:

ll /luks

Task 11: Troubleshoot issues affecting service start

On HostB, start the Performance Co-Pilot service and configure it to start automatically
upon reboot.

1. Attempt to start the pmcd service:

systemctl start pmcd

2. View the status of the service to determine what is causing the issue:

systemctl status pmcd

3. Resolve the cyclic dependency issue:

vi /etc/systemd/system/pmcd.service.d/dependency.conf

Be sure to change After to Before.

4. Start the pmcd service:

systemctl start pmcd

5. Enable the pmcd service to start automatically on server boot:

systemctl enable pmcd

6. We are finished with the tasks for HostB. Let's run the grading script to ensure
all of the steps were completed successfully:

score_HostB

Reboot HostB:
 shutdown -r now

When the server has finished rebooting, log back in:

ssh cloud_user@PUBLIC_IP_ADDRESS

Become the root user:

sudo su -

Run the grading script again to ensure all tasks are able to persist through a
reboot:

score_HostB

Task 12: Troubleshoot system resource consumption

On HostC, something is consuming an excessive amount of CPU load. Prevent

whatever it is from being executable in the future.

1. View the load average:

uptime

2. Attempt to use pcp atop:

pcp atop

3. Start the pmcd service:

systemctl start pmcd

4. Use pcp atop to determine what is causing the high CPU load:

pcp atop

5. View more information about the process, using the PID of the problematic
process identified in the previous step:

ps aux | grep <PID>;

6. Remove the ability to execute the problematic script:

chmod -x /usr/bin/polling.sh

7. Kill the process:

kill -9 <PID>;

Task 13: Troubleshoot an application

On HostC, troubleshoot and resolve the issue affecting the use of the dig command.

1. Attempt to use dig on google.com:

dig google.com

2. Determine which shared libraries are being used:

ldd $(which dig)

3. Determine which packages provide the missing shared library:

yum provides */libdns.so.100

4. Reinstall bind-libs:

yum reinstall -y bind-libs

5. Use dig on google.com to verify the application is working properly:

dig google.com

Task 14: Troubleshoot service authentication

On HostC, troubleshoot and resolve the issue preventing cloud_user from listing the
local Samba shares. Do not modify the existing Samba configuration file
/etc/samba/smb.conf.

Username: cloud_user

Password: welcome1

1. Use smbclient to attempt to list the local Samba shares:

smbclient -U cloud_user -L localhost

2. View the most recent logs:

tail /var/log/samba/log/smbd

3. Verify the samba package

rpm -V samba

4. Move the changed file to `/tmp/:

mv /etc/pam.d/samba /tmp/

5. Reinstall samba:

yum reinstall -y samba

 6. Verify that this has resolved the issue:

smbclient -U cloud_user -L localhost

Task 15: Troubleshoot iSCSI

HostC is an iSCSI target. Troubleshoot issues preventing HostD from accessing the
target. HostD should successfully be able negotiate the iSCSI connection and attach to
the block device.

On HostC

1. Start and enable the service to start automatically on server boot:

systemctl start target && systemctl enable target

2. Add a firewall rule to allow port 3260/tcp:

firewall-cmd --permanent --add-port=3260/tcp

3. Reload the firewall:

firewall-cmd --reload

4. View the current configuration:

echo "ls" | targetcli

On HostD

1. Verify the target from HostC matches the target on HostD:

vim /etc/iscsi/initiatorname.iscsi

Replace the initiator name in this file to match HostC.

2. Verify the authentication matches what is configured on HostC:

vim /etc/iscsi/iscsid.conf

Comment the following line:

node.session.auth.authmethod = CHAP

3. View any current records:

iscsiadm -m node

4. Discover the target:

iscsiadm -m discovery -t sendtargets -p 10.0.1.12

 5. Log in using the target shown from the previous step:

iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.ip-10-0-1-

10.x8664:sn.a3776832068c -l

6. Verify:

lsblk

Task 16: Configure remote logging

Configure HostC as a remote log server for HostD. The connection should be over the
standard TCP port.

On HostC

1. Edit the rsyslog.conf file:

vim /etc/rsyslog.conf

Uncomment the following lines:

$ModLoad imtcp
$InputTCPServerRun 514

2. Restart the rsyslog service:

systemctl restart rsyslog

3. Add a firewall rule to allow port 514/tcp:

firewall-cmd --permanent --add-port=514/tcp

4. Reload the firewall:

firewall-cmd --reload

On HostD:

1. Edit rsyslog.conf:

vim /etc/rsyslog.conf

Uncomment the following line:

*.* @@remote-host:514

Edit that line to look like this:

*.* @@10.0.1.12:514

2. Restart the rsyslog service:

 systemctl restart rsyslog

3. Send a log message:

logger "This is a test"

4. Run the grading script to ensure Task 15 and Task 16 are correctly configured:

score_HostD

Task 17: Troubleshoot RPM issues

HostD has a corrupt RPM database. Troubleshoot and resolve this issue.

1. Change to the rpm directory:

cd /var/lib/rpm

2. Attempt to verify the Packages file:

/usr/lib/rpm/rpmdb_verify Packages

3. Dump Packages and use it to load in the new Packages.new:

/usr/lib/rpm/rpmdb_dump Packages | /usr/lib/rpm/rpmdb_load

Packages.new

4. Verify the new Packages.new:

/usr/lib/rpm/rpmdb_verify Packages.new

5. Rename Packages to Packages.bad:

mv Packages Packages.bad

6. Rename Packages.new to Packages:

mv Packages.new Packages

7. Rebuild the RPM database:

rpm --rebuilddb

8. Clean yum:

yum clean all

Task 18: Troubleshoot networking

Step 1
Troubleshoot any networking issues preventing HostD from accessing the Apache
content on HostA at http://ip-10-0-1-10/index.html

On HostD

1. Attempt to view the Apache content on HostA:

curl http://ip-10-0-1-10/index.html

2. Attempt to view the Apache content directly:

curl 10.0.1.10

On HostA

1. Add a firewall rule to allow HTTP traffic:

firewall-cmd --permanent --add-service=http

2. Reload the firewall:

firewall-cmd --reload

On HostD

1. View the Apache content directly:

curl 10.0.1.10

2. Attempt to view the Apache content at the requested URL:

curl http://ip-10-0-1-10/index.html

3. View the contents of /etc/resolv.conf:

cat /etc/resolv.conf

4. Edit nsswitch.conf:

vim /etc/nsswitch.conf

Edit the hosts line to look like this:

hosts: files dns myhostname

5. View the Apache content at the requested URL:

curl http://ip-10-0-1-10/index.html

Step 2
There is a packet capture file located at /root/packet.cap. Examine this file and
discern the password used by testuser01. Enter this password as a single line into the
file /root/passwd.txt.

Example: echo "password" >> /root/passwd.txt

1. Install tcpdump:

yum -y install tcpdump

2. Use tcpdump to search the packet capture:

tcpdump -Ar packet.cap | less

Search the tcpdump output for testuser01:

/testuser01

3. Copy the password and save it in the file passwd.txt:

echo mypaSSword | /root/passwd.txt

Task 19: Examine and retrieve information from an sosreport

On HostD, examine the sosreport located at /root/sosreport-ip-10-0-1-11-2019-

03-22-wxoxhnk.tar.xz and determine the LDAP hostname the host attempted to bind
to.

Enter the name of this hostname as a single line in /root/test.txt.

Example: echo "server.example.com" >> /root/test.txt

1. Extract the sosreport:

tar xvf sosreport-ip-10-0-1-11-2019-03-22-wxoxhnk.tar.xz

2. Search the sosreport for LDAP:

cat sosreport-ip-10-0-1-11-2019-03-22-wxoxhnk/var/log/messages
| grep LDAP

3. Copy the LDAP server name and save it in the file /root/test:

echo ldap.example.com >> /root/test

Task 20: Assist in providing information for third-party investigation

On HostD, permit cloud_user to run example.stp as a module named cloud_tap.

1. Use SystemTap to run the file example.stp:

 stap example.stp

2. Compile the module:

stap -p4 -m cloud_tap example.stp

3. Make a directory for our module:

mkdir /lib/modules/$(uname -r)/systemtap

4. Move the module to the new directory:

mv cloud_tap.ko /lib/modules/$(uname -r)/systemtap/

5. Run the module:

staprun cloud_tap

6. Add cloud_user to the stapusr group:

usermod -aG stapusr cloud_user

7. Ensure the cloud_user user can run the cloud_tap module:

runuser -l cloud_user -c 'staprun cloud_tap'

8. Just as before, reboot the host:

shutdown -r now

Once the server has rebooted, log in again:

ssh cloud_user@PUBLIC_IP_ADDRESS

Become the root user:

sudo su -

Run the grading script to ensure all steps were completed correctly:

score_HostD.sh

Conclusion
Congratulations, you've completed this hands-on lab!