10/11/2020 Document 2545437.

1
Copyright (c) 2020, Oracle. All rights reserved. Oracle Confidential.

Important Service Issues When Upgrading to OS8.8.x (Doc ID 2545437.1)

In this Document

Details
IMPORTANT SERVICE ISSUES WHEN UPGRADING A ZFS STORAGE APPLIANCE(ZFSSA)TO OS8.8.x
Actions
Invalid stash objects must be removed before any update to OS 8.8.x
Backend multipathing does not recognize some newer drive models.
Some NDMP Backups Inconsistent And Problematic
ZFSSA Node with InfiniBand Hangs, NFS Shares Get Stuck on Clients
"Deadlock: cycle in blocking chain" panic
lockd: server <HOSTNAME> not responding
ZFSSA not responding over HTTP
ZFSSA reports it is an UNSUPPORTED platform
Veritas OST and Microsoft VSS plugins require workflows not included in OS8.8.0 through 8.8.3
Cluster Ethernet link intermittent issues
Problems mounting NFS Shares
Database crashes on NFS v3 & 4 shares after upgrading to OS 8.7.25 through 8.8.5
Upgrading to OS8.8.18 (from OS8.7 or earlier 8.8.X version) where the upgraded node sits at joining cluster.
Upgrade To OS8.8.18 May Fail Due To "replay svc tx for scrk property time_updatecheck not found" (Doc ID
2640228.1)
Post-upgrade to AK 8.8.X, ZFS Head Fails to Rejoin Cluster with 'exchangeFail Invoked' Error (Doc ID 2650954.1)
After upgrading to OS8.8.18 through OS8.8.24, a link aggregation(aggr) with VNIC/VLAN can no longer communicate
between appliance heads.
Upgrading to OS8.8.22 while using the Chrome Browser can cause akProgrammerError under Maintenance ->
System.
IMPORTANT Service issue with Upgrading OS8.8.x on 7120
IOM (I/O Controller Module) Fault During Cluster Upgrade to OS8.8.22 or Higher

Enabling suspend notifications during maintenance / service activities can prevent false failures from generating
unnecessary service requests. Oracle ZFS Storage Appliance: Suspend Notifications ( AK-8005-8W ) (Doc ID
2712745.1)
Contacts
References

APPLIES TO:

Sun ZFS Storage 7420 - Version All Versions and later

Internal ZFS ZS5-4 Rack - Version All Versions and later
Oracle ZFS Storage ZS5-4 - Version All Versions and later
Oracle ZFS Storage ZS4-4 - Version All Versions and later
Oracle ZFS Storage Appliance Racked System ZS7-2 for PCA/CatC - Version All Versions and later
Information in this document applies to any platform.

DETAILS

IMPORTANT SERVICE ISSUES WHEN UPGRADING A ZFS STORAGE APPLIANCE(ZFSSA)TO OS8.8.x

IMPORTANT SERVICE ISSUES WITH OS8.8.X WHICH REQUIRE THREE WORKFLOWS TO BE RUN

FOLLOW BOTH DOCS FOR INSTRUCTIONS ON APPLYING WORKFLOWS

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 1/10
10/11/2020 Document 2545437.1

ACTIONS

Issues effecting OS8.8.x

Below is a list of known bugs & documents related to OS8.8.x. containing fixes, workarounds or directions to identify
known issues.

To be sure the fix is in the OS8.8.x version you are planning on downloading and upgrading to, use our software
download page listed below.

Oracle ZFS Storage Appliance: Software Updates (Doc ID 2021771.1)

1. Select the "Release Notes" Link to the right of the OS8.8.x version you are planning on downloading, which will
open a pdf.
2. Navigate to "Closed Issues" which are fixed in this release. It will list the Synopsis & BUG #

Invalid stash objects must be removed before any update to OS 8.8.x

This issue is addressed once your ZFS Storage Appliance(ZFSSA) is running OS8.8.7; however, the proactive
workflows still need to be run prior to updating for an older OS release.

You will no longer need to run the clean_orpan_objects.akwf see BUG with fix below.

This issue is addressed in OS8.8.7 an newer

Unpublished Bug 30068279 - Backport 29705956 to AK-8.8-MICRO - Orphaned worksheets (owner deleted from
appliance) causes cluster rejoin to fail invalid stash object

Prior to an upgrade of a standalone or clustered ZFSSA node(s) to OS8.8.x, you should run both proactive
workflows listed below to identify and remove the invalid stash objects before you upgrade to prevent cluster
join issues.

Run these two proactive workflows on both nodes to identify and remove invalid stash objects before an upgrade
to OS8.8.x:

Workflow name: Filename:

Scrub bundles with invalid format (29055824) checkOldBundles.akwf

Identify and clean orphaned workflows and

clean_orphan_objects.akwf
worksheets

Reference:
How to Upload and Execute a Workflow on a ZFS Storage Appliance (Doc ID 2359446.1)

This is a proactive approach to detect and clean up the invalid stash objects before the effect the ZFSSA.

Failure to implement these two workflows prior to an OS update from OS8.7x to OS8.8x or OS8.8x to a higher
OS8.8.x version will result in this problem being seen:

244 error(s) have occurred during cluster rejoin.

The most recent error was: 'exchangeFail invoked'.
Cluster rejoin will be retried until successful.

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 2/10
10/11/2020 Document 2545437.1
Joining cluster ...

If this is encountered, reference:

ZFS Storage Appliance: Software Update to Version 8.8.x Fails at Cluster Join With 'exchangeFail invoked'. (Doc ID
2487089.1)

and engage Oracle support for assistance.

Backend multipathing does not recognize some newer drive models.

This issue is addressed in OS8.7.29 and newer, as well as OS8.8.6 and newer.

Unpublished Bug 29804424 - 14TB WDC ORA disk not using c0: 2013: workaround via scsi_vhci.conf

Unpublished Bug 29804414 - 14TB WDC ORA disk not using c0: 8.8: fix FISH inception_points.

Some newer drive models are not properly discovered by backend multipathing.

Support for the 14TB HDDs is now available in the 8.7.29 and 8.8.6 release or newer of the ZFSSA OS.

If your system is on 8.7.29 or 8.8.6 or higher or you do NOT have 14TB drives you can skip the document and
workflow below.

We recommend upgrading to our latest appliance firmware to fix this issue seen below.

If you are unable to upgrade to the latest appliance firmware to impliment the fix, run the workflow below.

If your ZFSSA is running OS8.8.0 thru OS8.8.5, run the workflow below in order to address this issue.

Workflow name: Filename:

Update system for new drive types patch_scsi_vhci.akwf

After Adding New DE3-24 Tray, Paths Report 0/2 and PCIe State Reports UKNOWN (Doc ID 2538243.1)

IMPORTANT NOTE:

You should run this workflow even if you do not see an issue with the paths as they can break if the boot
archive is updated.

After running the workflow, you must reboot, as the workflow directs you to. Perform this action on
both nodes of a ZFSSA Cluster, one at a time.

Some NDMP Backups Inconsistent And Problematic

This issue is addressed in OS8.8.5 and newer.

Unpublished Bug 28967351 - NDMP backup hangs for sparse files

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 3/10
10/11/2020 Document 2545437.1

Oracle ZFS Storage Appliance: Some NDMP Backups Do Not Complete After Upgrade To OS 8.8.0 Through 8.8.4 (Doc
ID 2510538.1)

ZFSSA Node with InfiniBand Hangs, NFS Shares Get Stuck on Clients

This issue is addressed in OS8.8.18 and newer.

Unpublished Bug 29695777 - Remove eib and sdp driver from AK.

If your ZFSSA is running OS8.7.x and experiences the above issue, you will need to run the workflow found in the
document below to remove the eoib driver.

Unpublished Bug 27159905 - "REMOVE EOIB AND EIBNX FROM AK" was closed as "Will not be fixed in OS8.7.x"

If your ZFS Storage Appliance contains Infiniband Host Channel Adapters, please read the document below and apply
the workflow to remove the Ethernet Over InfiniBand(eoib) driver.

The eoib driver is loaded by the ZFSSA; but should not be active on the ZFSSA nodes.

Oracle ZFS Storage Appliance: ZFSSA Node with InfiniBand Hangs, NFS Shares Get Stuck on Clients (Doc ID
2441188.1)

"Deadlock: cycle in blocking chain" panic

This issue is addressed in OS8.8.6 and newer.

Unpublished Bug 29743115 - Backport 29703361 to AK-8.8-MICRO - dnlc remove causes deadlock between znode
lock and hold lock

Oracle ZFS Storage Appliance Running OS8.8.0 Through 8.8.5 Panics With "Deadlock: cycle in blocking chain" (Doc
ID 2547005.1)

lockd: server <HOSTNAME> not responding

This issue is addressed in OS8.8.6 and newer.

Unpublished Bug 29592848 - Backport 29455971 to AK-8.8-MICRO - NFSv3/NLM lock request ignored with 11.4.4.0

Oracle ZFS Storage Appliance Running OS8.8.4 or 8.8.5 Not Responding to NFS lock Requests (Doc ID 2537395.1)

ZFSSA not responding over HTTP

This is addressed in OS8.8.5 and newer.

Unpublished Bug 28972766 - BUI access ceased due to apache threads in process_lingering_close()
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 4/10
10/11/2020 Document 2545437.1

Enterprise Manager With OS8.8.x -No BUI Access No New BUI Connections (SSH Works Fine) (Doc ID 2497490.1)

Oracle ZFS Storage Appliance Running OS8.8.0 through 8.8.4 Not Responding via HTTP but Is Responding Via SSH
(Doc ID 2547018.1)

ZFSSA reports it is an UNSUPPORTED platform

This warning message can be disregarded since it a cosmetic issue.

Oracle ZFS Storage Appliance: A Warning Message "Evaluation Platform" is Displayed on ZFSSA Running OS 8.8.x
Versions (Doc ID 2545695.1)

Veritas OST and Microsoft VSS plugins require workflows not included in OS8.8.0 through 8.8.3

This issue only effects systems initially installed with OS 8.8. A ZFSSA updated from a prior release is not effected.

Unpublished Bug 29043712 - s7K workflows broken in 8.8

Unpublished Bug 28890993 - OST plugin fails communicating with appliance after upgrading to OS 8.8

Unpublished Bug 28890968 - VSS plugin fails communicating with appliance after upgrading to OS 8.8

As a workaround, the missing workflows are provided in:

Plugins Incompatibility with OS 8.8 (Doc ID 2486048.1)

Cluster Ethernet link intermittent issues

This is addressed in OS8.8.6 and newer.

Unpublished Bug 29378043

The issue does not appear to cause any problems for the ZFSSA system other than reporting the alert.

The cluster Ethernet link is reported going down then back up in the alert log.

Cluster Link DLPI Reports Down/Up Repeatedly After Upgrade to AK 8.8.x (Doc ID 2532470.1)

Problems mounting NFS Shares

This is addressed in OS8.8.6 and newer.

Unpublished Bug 29702622 - mountd can fail to provide service over TCP

After a takeover, NFS clients are unable to mount shares for around 5 minutes.

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 5/10
10/11/2020 Document 2545437.1

We recommend you upgrade to our latest appliance firmware to fix this issue.

If you are running appliance firmware S8.8.0 to OS8.8.5, a workflow is available in the document below as a
workaround to this issue.

Oracle ZFS Storage Appliance Clients Are Intermittently Unable to Mount Shares Over TCP (Doc ID 2547327.1)

Database crashes on NFS v3 & 4 shares after upgrading to OS 8.7.25 through 8.8.5

This is addressed in:

OS8.8.7 and newer.

OS 8.7.29 Interim Diagnostic Relief (IDR) update 1.1 through 1.3.

Unpublished Bug 29506035 - 7420: OS8.8.3 panicked with Dead locked cycle in blocking chain

Unpublished Bug 30400370 - Backport 29754235 to AK-8.8-MICRO - NFS server can return incorrect error causing
DB crash in event client timeouts due to network partition

If you suspect your ZFS Storage Appliance that cannot be updated to OS 8.8.7 or newer has encountered this issue,
please engage Oracle support to confirm and provide the IDR.

Sun Storage 7000 Unified Storage System: How to install an IDR (Doc ID 1429544.1)

Upgrading a ZS5-2 to 8.8.x breaks LACP functionality.

This issue can be addressed in OS8.8.7 and newer but requires assistance from Oracle Support.

If you are not running OS 8.8.7 or above, you will need to upgrade prior to contacting Oracle Support.

Unpublished Bug 29350068 - ak-8.8.6 i40e driver loopback configuration incompatible with zs5 firmware

After upgrading a ZFSSA to OS8.8.0 through 8.8.9 issues are seen with ZS5-2 arrays with onboard network ports
used in LACP Aggregation.

Onboard network ports that are configured as part of an LACP Aggregation may not work properly and show as in
the loopback mode.

Current workarounds options are...

Rollback to pre-OS 8.8.x microcode.

Remove all but one interface from the Link Aggregation. WARNING: This will remove redundancy from the
link and leave only a single interface in the Link Aggregation.

The document below provides details for fix and workaround options.

If on OS 8.8.7 and above, open a Service Request to engage Oracle Support to manually upgrade the Lan On
Motherbord(LOM).

IMPORTANT: The LOM upgrade procedure will require someone onsite to remove the AC power cords for 60
seconds to complete the upgrade. Our ZFSSA team will work with you getting someone onsite to assist with
pulling the AC power cords.

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 6/10
10/11/2020 Document 2545437.1

Oracle ZFS Storage Appliance: LACP Errors with Loopback Condition seen on ZS5-2 (Doc ID 2607345.1)

Uploading / Staging OS8.8.8 from the BUI Fails.

This is addressed in OS8.8.9 and newer.

Unpublished Bug 30032290 - upload of update image to /api/system/v1/updates returns 408

When a software update is uploaded via the Browser User Interface(BUI) on ZFS Storage Appliance(ZFSSA) running
OS8.8.7 or OS8.8.8, the upload will appear to succeed and begin the next stage to decompress the file, but at that
time a dialog will appear:

Failed to decompress update image: <image name>

The software will fail to be loaded.

If the RESTful API is used to upload an image, the following error message will be returned:

Request Timeout
Server timeout waiting for the HTTP request from the client.

The software will fail to be loaded.

Solution:

The attached workflow can be used to update the Apache webserver configuration of the ZFSSA. After running the
workflow, restart the HTTPS service. The service can be restarted from the BUI by selecting Configuration->Services-
>HTTPS (in the Remote Access Section). The service cannot be restarted from the CLI.

See the document below for details.

Attempt to Upload OS on ZFS Storage Appliance Fails with "Failed to decompress update image" or "Server timeout
waiting for the HTTP request from the client" (Doc ID 2618948.1)

Using ZFS Encryption with Storage Appliance (ZFSSA) May Cause the System to Run Out of Memory.

This is addressed in OS8.8.8 and newer.

Unpublished Bug 25993455 - zebra striped dnode blocks

This issue may be seen on ZFSSA OS versions 8.8.6 through OS8.8.7

Below is the Service Alert for this issue:

Using ZFS Encryption in Solaris and Storage Appliance (ZFSSA) May Cause the System to Run Out of Memory (Doc ID
2629925.1)

Upgrading to OS8.8.18 (from OS8.7 or earlier 8.8.X version) where the upgraded node sits at joining cluster.

Below are two different cluster join issues with OS8.8.6 & OS8.8.18 #1 has a fix #2 is Proactive and or Reactive
depending.

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 7/10
10/11/2020 Document 2545437.1

Upgrade To OS8.8.18 May Fail Due To "replay svc tx for scrk property time_updatecheck not found" (Doc ID
2640228.1)

This is addressed in OS8.8.20 and newer.

Unpublished Bug 30821271 - 20610833 causes upgrade problems due to time_updatecheck

A reactive workaround at the fail safe shell needs to be performed by Oracle Support.

If you experience the issue above open a service request (SR) with the ZFSSA support team.

Post-upgrade to AK 8.8.X, ZFS Head Fails to Rejoin Cluster with 'exchangeFail Invoked' Error (Doc ID 2650954.1)

This issue requires a proactive workflow to be run before the ZFSSA is upgraded to OS8.8.6 & higher in order to
prevent the cluster join issue.

If you have upgraded to OS8.8.6 or higher and experience the above cluster join issue open a SR with our ZFSSA
Support Team and a ZFS Storage Appliance Technical Support Engineer will assist.

Unpublished Bug 30990403 - Post-upgrade to AK 8.8.18, ZFS head fails to rejoin cluster due to invalid trustedcerts
object

The proactive workaround is documented in:

Post-upgrade to AK 8.8.X, ZFS Head Fails to Rejoin Cluster with 'exchangeFail Invoked' Error (Doc ID 2650954.1)

The checkZeroTrustedcert_v1.1.akwf workflow can be run prior to upgrading to OS8.8.6 & higer versions.

After upgrading to OS8.8.18 through OS8.8.24, a link aggregation(aggr) with VNIC/VLAN can no longer
communicate between appliance heads.

This is addressed in OS8.8.25 and newer.

Unpublished Bug 31470350 - Aggr with VNIC/VLAN not working after upgrade to OS8.8.20

Unpublished Bug 31631412 - Remove VNICs MAC address persistence between reboots

The workaround is documented in:

ZFS Storage Appliance (ZFSSA) Software Unable to Communicate Between Cluster Heads (Doc ID 2686434.1)

Upgrading to OS8.8.22 while using the Chrome Browser can cause akProgrammerError under Maintenance -
> System.

Engineering is working on a solution in Bug below.

Unpublished Bug 31572434 - AKPROGRAMMERERROR USING CHROME AFTER UPGRADE TO OS8.8.22. ERRORS AT
MAINTENANCE -> SYSTEM

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 8/10
10/11/2020 Document 2545437.1

The workaround is documented in:

Error "akProgrammerError" Using Chrome After Upgrade to OS8.8.22. Errors at Maintenance -> System (Doc ID
2687231.1)

IMPORTANT Service issue with Upgrading OS8.8.x on 7120

Do not install OS8.8.x on a Sun ZFS Storage 7120. If there is a suspected security issue, and you believe
OS8.8.x is needed, open a SR with our ZFSSA TSE so we can review the issue and advise.

Oracle working on this issue in the Bugs listed below:

Unpublished Bug 29288699 - AK8.8 upgrade fails on appliance 7120 due to mpt device probe issue

Unpublished Bug 30165054 - OS8.8.x should not install on unsupported systems like 7120

We recommend upgrading to our recommended and latest appliance version OS8.7.29.

Our software downloads pages mentions the 7120 is not supported with OS8.8.x. See info below.

Oracle ZFS Storage Appliance: Software Updates (Doc ID 2021771.1)

Product Minimum Recommended Latest

Supported Version Version
Version

7120 OS8.6.0 (2013.1.6.0) OS8.7.29 OS8.7.29

(2013.1.7.29) (2013.1.7.29)

7420, 7320, and 7120 systems support both 2011.1 and 2013.1 releases, except 7120 systems do not support
OS8.8.x (2013.1.8.x).

7120 systems (OS8.8.x not supported)

____________________________________________________________________________________________________

IOM (I/O Controller Module) Fault During Cluster Upgrade to OS8.8.22 or Higher

IOM faults after upgrading AK to 8.8.22 and above. If upgrading a cluster, after upgrading the first controller, upon
reboot the upgraded controller will fault the IOM and the peer node can not be upgraded until the fault is addressed.
Check maintenance problems show for the event, problem and revision. If your issue matches follow steps in Doc below.

<EVENT_ID_MSG_ID> STORAGE-8000-Q2 Major

The new problem class is "fault.enclosure.controller.firmware-detected.failure"

Revision : 0000

ZFS Storage Appliance IOM Fault During Cluster Upgrade to AK 8.8.22 or Higher After First Head Reboots - STORAGE-
8000-Q2 Major (Doc ID 2711650.1)

Unpublished Bug 31906955 - DE2 tray firmware revision getting reset to invalid version

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 9/10
10/11/2020 Document 2545437.1

Unpublished Enh 31901536 - SES enclosure DE will fault IOM if it has invalid fw revision after upgrading 1st head

Enabling suspend notifications during maintenance / service activities can prevent false failures from generating
unnecessary service requests. Oracle ZFS Storage Appliance: Suspend Notifications ( AK-8005-8W ) (Doc ID
2712745.1)

____________________________________________________________________________________________________

CONTACTS

If you experience any of the known issues listed above running OS8.8.x, you can follow the reference documents to
address the issue.

If you still have issues, open a service request (SR) with the ZFSSA support team.

How to create a Software Service Request (SR) [VIDEO] (Doc ID 1949246.1)

Thank you

Didn't find what you are looking for?

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=py6xm6utr_4&id=2545437.1 10/10