Document 1607228.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-...

Resolving Common Weblogic Stdout Log Rotation Problems on UNIX (Doc ID 1607228.1)

In this Document

Purpose
Troubleshooting Steps
References

APPLIES TO:

Oracle WebLogic Server - Version 10.3 and later

Generic UNIX

PURPOSE

The goal is to troubleshoot WebLogic stdout log rotation and truncation problems; these become significant when the stdout file starts growing beyond a manageable size.

Specifically this note covers the usual problems, which are:

1) The stdout file is growing too big and cannot be truncated

2) The stdout file is growing too big and cannot be rotated
3) The stdout file is not growing too big but just isn't being rotated.

TROUBLESHOOTING STEPS

Review of the general principles

================================

a) stdout is a UNIX stream and weblogic cannot natively rotate it.

b) Weblogic can however rotate its own server log files, and there are command-line flags / admin console GUI option / WLST methods to redirect stdout to the server log file - the idea
is that then, if server log rotation is enabled, then stdout will also be rotated as it's going to be written to the same file.

The standard methods include:

i) Via the gui console:

Admin server (or the Managed Server) -> Logging -> General
-> Advanced Settings -> Redirect stdout logging enabled
-> enable the checkbox.

1 of 5 12/02/2015 01:57 PM
Document 1607228.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-...

ii) Setting redirect-to-serverlog flags in a startup or environment script,

for example Via JAVA_OPTIONS in commEnv.sh :

JAVA_OPTIONS=$JAVA_OPTIONS"-Dweblogic.log.RedirectStdoutToServerLogEnabled=true"

Some logging flags can be used in addition to the above, and some are mutually exclusive. These can be used in addition to the above:

-Dweblogic.StdoutDebugEnabled=true
-Dweblogic.log.StdoutSeverityLevel=Info
or
-Dweblogic.log.StdoutSeverityLevel=Debug

However the following are for naming specific files:

-Dweblogic.stdout=/tmp/mystdout.log
-Dweblogic.stderr=/tmp/mystderr.log

or on Windows, for example:

-Dweblogic.Stdout=C:\test\stdout.log

So if these get added by mistake, they may stop the RedirectStdoutToServerLogEnabled option from working.

iii) Via WLST. See Note 1344154.1 How to Redirect WLST stdout to a Specified File

iv) Via the linux logrotate system, ref Note 1273545.1: "WebLogic Server: Rotating stdout Log Files with UNIX OS Commands"

The same 'logrotate' command is not available on Solaris, but there is a Solaris equivalent, called 'logadm,' described here in the documentation: http://docs.oracle.com/cd/E23823_01
/html/817-0403/emcby.html

NOTE: When using this logrotate system, be sure to use the 'copytruncate' option -- this truncates the original log file in place after creating a copy, instead of moving the old log
file and optionally creating a new one.
- It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. (Note that there is a very small
time slice between copying the file and truncating it, so some logging data might be lost.) In other words, this says that some programs know how to cooperate with logrotate by
closing the log file and creating a new one when commanded. Logrotate can do this by trapping sighup.

c) The problem with the above standard methods is in how you start weblogic. Many choose to do:

nohup startweblogic.sh > customLogFile.log 2>1& &

this means that :

i) the UNIX redirect will *override* any redirect options you have set either in the console or via the command line

2 of 5 12/02/2015 01:57 PM
Document 1607228.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-...

ii) even if you take the UNIX redirect away, then you will also get a nohup.out file generated, which will also grow potentially huge in size. For example : nohup startweblogic.sh

The obvious workaround is to instead run it like this: startweblogic.sh

..but then the drawback is that this needs to be run either from the system console, or from a secure terminal somewhere that is constantly left running, options which are not always
feasible, because of remote security restrictions for console access.

The rest of this document describes some workarounds to the above problem.
--------------------------------------------------------------------------

Known issues:
-------------

Note 1562494.1: Log Rotation Does Not Happen When Rotated Log Directory And Current Log Are On Two Different Filesystems / mountpoints
Note 1074920.1: WebLogic Server Log Rotation Not Working When Writing to NFS Mount

-> both are fundamentally related to operating system limitations, so these are not bugs; for example the solution in the NFS note is simply to not use an NFS drive.

The problem with logrotate

--------------------------

There are some support notes that suggest that the solution to the stdout log rotation problem is to use the Linux utility 'logrotate' to rotate log files, which depending on certain factors,
may work or may not, and may sometimes cause binary corruption of the out file andnyou may see the file still growing in size, and not getting truncated. (Ref: Note 1215735.1 How to
Rotate the Standard Output (STDOUT) File for WebLogic Server -- This contains basic /etc/logrotate.conf setup.)

(If you are getting binary corruption, you can usually recreate the problem by trying to manually truncate the file:
cat /dev/null > your_log_file;
-> if the log file
a) shows binary characters in
b) is still huge in size

then this is generally the same issue affecting 'logrotate.')

Known workarounds/alternatives to the UNIX logrotate utility

--------------------------------------------------------------

1) instead of a redirect, when starting weblogic you can pipe the stdout to the UNIX 'split' command, so do something like this:

nohup startweblogic.sh | split -l 100000

-> what this does is separate the stdout into manageable 100000 line files. You can set this amount to anything you like, 1000 lines, 10000 lines etc. However for the problem of out
files growing huge, 100000 seems a reasonably good value. In its basic form, split creates the chunks as files called xaa xab xac xad etc, increasing the letter on the suffix, however if
you look at the man pages there are options to allow you to provide your own naming convention.

To test the principle without risking a change to WebLogic, you can do the following. Create a test UNIX script called myloop containing:

3 of 5 12/02/2015 01:57 PM
Document 1607228.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-...

while [ 2 -gt 1 ] ; do
echo "hello"
done

and then run the script like this:

./myloop | split -l 100000

from another terminal session, cd to the same directory and you will see the files being created: xaa xab etc.

- The administrator can now then write a cron job to archive these files out as and when needed to save space.

2) The logrotate mechanism is not reliable if the stdout is created via a 'non append' redirect symbol: ">". However changing to ">>" should allow logrotate and truncates to work:

nohup startweblogic.sh >> customLogFile.log 2>1 &&

This is because the file is created in "append" mode. The only drawback is that in some cases a new file may be required on each startup; in this case the administrator should simply
move the file to archive before starting up WebLogic.

3) Many UNIXes have the /usr/sbin/rotatelogs utility; this is related to Apache log rotation. You can pipe the output of startweblogic to rotatelogs, which handles it all for you. This basic
example rotates the file at 1 meg:

nohup startWebLogic.sh | /usr/sbin/rotatelogs /path/to/your/log/out.log 1M &

4) If in spite of using >>, you find that logrotate/truncate is still not working, it could be caused by the 'noclobber' option set in the startup environment. See: http://en.wikipedia.org
/wiki/Clobbering. If this is set, the logrotate will not work, and the solution is to unset it:

set +o noclobber

(set -o noclobber turns it on.)

5) Use a UNIX pipe: See Note 774807.1: How To Rotate The nohup.out File By Size. This is a similar solution to the 'split' suggestion, but involves creating a named 'UNIX pipe,' and
redirecting stdout to that on startup. You then use 'cat' to read from the pipe and redirect to a log file, and to stop the log file from being written you can kill the cat process, whereupon
the log can be archived off or rotated manually, and then cat is restarted to continue to read from the pipe.

6) Check that you are not redirecting stdout to the actual server log file name on the command line; this will likely cause all sorts of problems. You will likely see problems regardless of
whether you use > or >> So don't do either of these:

startWebLogic.sh > /path/user_projects/domains/domA/serverA/logs/serverA.log

startWebLogic.sh >> /path/user_projects/domains/domA/serverA/logs/serverA.log

7) For the extreme case where a quick solution to massive files is required, you can just destroy all stdout by starting up WebLogic like this:

4 of 5 12/02/2015 01:57 PM
Document 1607228.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-...

startWebLogic.sh > /dev/null

However no diagnostics are possible in the future, so this should be just a temporary measure, if used at all.

Didn't find what you are looking for?

5 of 5 12/02/2015 01:57 PM