newsyslog

Maintain system log files to manageable sizes

£inux uses logrotate

sudo newsyslog [-CFNnrsv] [-R tagname] [-a directory] [-d directory] [-f config_file] [file]

Examples:
sudo newsyslog -vn | grep -v skipping # verbose, norotation

newsyslog -r -s -v -f insuranceLog.conf -C -d /Volume/PRODUCTION/insuranceLogs # archive files by the insurance group.
(allow non-root, supress signaling, verbose, config file is insuranceLog.conf

A log will be archived because:

  1. it's large
  2. It's a good time   (if time is specified, size is not checked.
  3. it is Forced or archived by command options.
Archives generations by moving the current log to log.0̸ after moving log.0̸ to log.1

The archived logs can be compressed.
Initailzates log with specified attributes.

-v verbose.
-n Do not archive logs. With -v display what it would happen(example). For testing config file.
-f config_file default /etc/newsyslog.conf
-r run if not root. Don't send HUP to syslogd for use with nonprivledged users. ( will not violate file premissions!)
-d directory files are relative to directory.
Don't include / as the first character of the path.
With -r premits individual users to specify their own log directory.
-a directory for archive. May be relative to the path of log file
Subdirectorues are created as needed. -a directories are unaffected.
-R tagname archive a list of files.
tagname used in the messages written to log files which are archived.
-s supress signaling to caller. May inhibit compression.
Useful with -R
-C create files with C in their config entry.
syslog may not write to logfiles if they don't exist.
-CC only create files on the command-line or by Calling process. Intended for programs to trigger archiving.
Rules such as permissions, number of archive generations, compression … are followed.
When a daemon executes newsyslog -R tag , files must be are closed first and re-opened after newsyslog returns.
The caller should specify -s to prevent newsyslog from signaling itself and return sooner by avoiding a wait.
-N Do not archive anyfiles, only create files. Useful with -C or -CC
-F Force archiving. Provide fresh logs containg only recent entries.

Specifing files on the command line supresses processing other files.

Run periodically by launchd to archive log files.
Can be run by nonprivledged users with their group configuration directing archives to specific directories.

/System/Library/LaunchDaemons/com.apple.newsyslog.plist schedules newsyslog every 30 minutes.
<key>StartCalendarInterval</key> <dict> <key>Minute</key> <integer>30</integer>

The granularity is dependent on how often a newsyslog is scheduled, i.e. if not often enough a log may exceed size
If a time is specified, the file will only be archived if newsyslog is run within one hour of the specified time.
If an interval is specified, the file will be archived if that many hours have passed since the last archival.
When both an interval and a time are specified both conditions must be satisfied for the archiving.

Files

/etc/newsyslog.conf
/etc/newsyslog.d/* configuration files can be used with any files.

N.B. symbolic links (like from /var/log ) will end up in their "logfilename" location and become unlinked on the first archival, i.e. don't use sym links for /var/log files, however it's OK for subdirectories of /var/log like DiagnosticMessages, asl, Accounts, apache2!


configuration

Each line has five required fields and optional fields, separated with whitespace.
An astrix (*) indicates the field is ignored.
A # causes the rest of the line to be ignored, unless escaped with \ .

Required: - minus sign is a placeholder
<default>
or
logfile
<default> entry is used when a logfile is given with command which is not in the config files.
owner:group Names from /etc/passwd : /etc/group or numeric ID. Default: root:admin.
ugo numeric mode (permissions) when creating logfile. Example 640 : user read+write, group read, others none.(see chmod).
n number of archive generations in addition to the current log
nKB maximun size.
Rather than using a large size, consider retaining more generations.

Optional:
[hh]                                          interval
@[[[[[cc]yy]mm]dd][T[hh[mm[ss]]]]]   time           or both.

If interval contains an asterisk (*) archiving will not depend on time.

An interval in hours, followed by an @ (at-sign) and a time.

$ day of once a day, once a week, or once a month.

For time, the lead-in character is an @ at-sign.

Optional date fields default to the appropriate component of the current date; Optional time fields default to midnight; (not the best as many other things will trigger then as well)
For example if today is January 22, 2023, these are equivalent:

      20230122T000000
        230122T000000
          0122T000000
            22T000000
              T000000
              T0000
              T00
            22T
              T 
Optional time fields default to midnight.
  • hh hours, 0..23
  • w day of week, 0..6, 0 = Sunday
  • dd day of month, 1..31, or L or l the last day of the month.

Examples:

      $D0     daily at midnight (same as @T00)
      $D23    daily at 23:00 (same as @T23)
      $W0D23  weekly on Sunday at 23:00
      $W5D16  weekly on Friday at 16:00
      $M1D0   first day of every month at midnight (i.e., the start of the day; same as @01T00)
      $M5D6   every 5th day of month at 6:00 (same as @05T06)

B log is a binary or has special format.
… logfile turned over due to …
is NOT inserted into log indicating when and why the log file was archived.
C create log if -C was specified on the command line.
D set UF_NODUMP which causes dump to ignore this file.
G file_name is a shell glob pattern
J bzip2 is used to compress archive (15% smaller)
Z gzip is used to compress archive (faster)
N no process is signaled when this log file is archive. Requires that the first line in that file be a negative value to distinguish it from a process ID.
/pid_file file containing the daemon's process ID or
a group process ID if U was specified.
A signal_number is sent the process ID contained in this file.
If not present, then SIGHUP will be sent to syslogd, unless N has been specified.
U pid_file contains the ID for a process group instead of a process.
signal_number sent to the daemon process (or to all processes in a process group, if U was specified). default SIGHUP

See also

bzip2, gzip, syslog, newsyslog.conf, chown, syslogd
rotlogs.sh

Example of config file

#Contents of maillog, messages, and lpd-errs may be confidential.

#   sudo newsyslog -vn|egrep --invert-match --line-buffered "skip|will"    # show only actions expected to be taken
#         verbose --'`--- no archiving

#                                                       @ interval, $ day...
#                                                                         J:Compress; B:dont add "new" message
# logfilename                       [owner:group] mode count size when  flags [/pid_file] [sig_num]
/var/log/01_alert.log                 root:staff  640  3     100    *      J
/var/log/02_crit.log                  root:staff  640  3     100    *      J
/var/log/03_err.log                   root:staff  640  3     100    *      J
/var/log/04_warn.log                  root:staff  640  3     100    *      J
/var/log/appfirewall.log              root:staff  640  3     100    *      J
/var/log/install.log                  root:staff  640  3     100    *      J
/var/log/mail.log                     root:staff  640  1     100    *      J
/var/log/secure.log                   root:staff  640  5     100    *      J
# seems that if both  size and time are specified, size is ignored! so here's 2 entries
# which confuses newsyslog which archives logs assigning odd numbers and only compressing sometimes
/var/log/system.log                   root:staff  640  7     200    *      J
/var/log/system.log                   root:staff  640  7     200    @T0007 J
# added 7/31/13 after 165KB
/var/log/kernel.log                   root:staff  640  5     200    *      J
# wtmp doesn;t exist ??!!
/var/log/wtmp                         root:staff  644  3     *      @01T05 B

# added 7/31/13 ; previously listed as secret recently growing FAST trim Previously expected 00:00 8/31/13
/var/log/wifi.log                     root:staff  640  3     100    *      J

#added 6/23/13:
/var/log/opendirectoryd.log           root:staff  644  3     100    *      J

/Volumes/DATA/log/05_notice.log       root:staff  644  3     200    *      J
/Volumes/DATA/log/06_info.log         root:staff  644  3     200    *      J
/Volumes/DATA/log/07_debug.log        root:staff  644  3     200    *      J

/var/log/apache2/access_log           root:staff  644  3     *    $W6D23   J  #Saturday night
/var/log/apache2/error_log            root:staff  644  3     *    $W6D23   J


Examples

sudo newsyslog -vn  # test new configuration
/var/log/01_alert.log <3J>: size (Kb): 4 [100] --> skipping
/var/log/02_crit.log <3J>: size (Kb): 4 [100] --> skipping
/var/log/03_err.log <3J>: size (Kb): 24 [100] --> skipping
/var/log/04_warn.log <3J>: size (Kb): 16 [100] --> skipping
/var/log/appfirewall.log <3J>: size (Kb): 8 [100] --> skipping
/var/log/install.log <3J>: size (Kb): 68 [100] --> skipping
/var/log/mail.log <1J>: size (Kb): 24 [100] --> skipping
/var/log/secure.log <5J>: size (Kb): 80 [100] --> skipping
/var/log/system.log <7J>: size (Kb): 8 [200] --> skipping
/var/log/system.log <7J>: --> will trim at Thu Aug  1 00:07:00 2013
/var/log/kernel.log <5J>: size (Kb): 164 [200] --> skipping
/var/log/wtmp <3>: does not exist, skipped.
/var/log/wifi.log <3J>: size (Kb): 20 [100] --> skipping
/var/log/opendirectoryd.log <3J>: size (Kb): 4 [100] --> skipping
/Volumes/DATA/log/05_notice.log <3J>: size (Kb): 184 [200] --> skipping
/Volumes/DATA/log/06_info.log <3J>: size (Kb): 152 [200] --> skipping
/Volumes/DATA/log/07_debug.log <3J>: size (Kb): 160 [200] --> skipping
/var/log/apache2/access_log <3J>: --> will trim at Sat Aug  3 23:00:00 2013
/var/log/apache2/error_log <3J>: --> will trim at Sat Aug  3 23:00:00 2013
/Library/Logs/slapconfig.log <10J>: does not exist, skipped.
/var/log/kernel.log <5J>: size (Kb): 164 [1000] --> skipping
/Library/Logs/named.log <5J>: does not exist, skipped.
/var/log/wifi.log <3J>: --> will trim at Thu Aug  1 00:00:00 2013
sudo newsyslog -vF # Force in preparation for new activities

/var/log/appfirewall.log <3J>: size (Kb): 20 [100] --> trimming log....
/var/log/ftp.log <1J>: does not exist, skipped.
/var/log/hwmond.log <3J>: does not exist, skipped.
/var/log/install.log <3J>: size (Kb): 12 [100] --> trimming log....
/var/log/ipfw.log <3J>: does not exist, skipped.
/var/log/lookupd.log <1J>: does not exist, skipped.
/var/log/lpr.log <1J>: does not exist, skipped.
/var/log/mail.log <1J>: size (Kb): 4 [100] --> trimming log....
/var/log/ppp.log <1J>: does not exist, skipped.
/var/log/secure.log <5J>: size (Kb): 4 [100] --> trimming log....
/var/log/system.log <7J>: --> trimming log....
/var/log/wtmp <3>: does not exist, skipped.
/Library/Logs/slapconfig.log <10J>: does not exist, skipped.
/var/log/kernel.log <5J>: size (Kb): 512 [1000] --> trimming log....
/Library/Logs/named.log <5J>: does not exist, skipped.
/var/log/wifi.log <3J>: does not exist, skipped.
Signal all daemon process(es)...
Notified daemon pid 20 = /var/run/syslog.pid
Pause 10 seconds to allow daemon(s) to close log file(s)
Compress all rotated log file(s)...  



2/12/18 note: no nn-llll.log files rotated. /var/log/ftp.log <5J>: size (Kb): 4 [1000] --> trimming log.... /var/log/hwmond.log <5J>: size (Kb): 4 [1000] --> trimming log.... /var/log/ipfw.log <5J>: size (Kb): 4 [1000] --> trimming log.... /var/log/lpr.log <5J>: size (Kb): 4 [1000] --> trimming log.... /var/log/ppp.log <5J>: size (Kb): 4 [1000] --> trimming log.... /var/log/wtmp <3>: size (Kb): 0 [1000] --> trimming log.... /Library/Logs/slapconfig.log <10J>: size (Kb): 4 [1000] --> trimming log.... /var/log/slapd.log <10J>: size (Kb): 4 [1000] --> trimming log.... /var/log/xscertd.log <5J>: size (Kb): 4 [10000] --> trimming log.... /Library/Logs/named.log <5J>: --> trimming log.... /Library/Logs/named.log <5J>: --> trimming log.... /var/log/wifi.log <2>: size (Kb): 8 [10] --> trimming log.... /var/log/corecaptured.log <3J>: size (Kb): 4 [10] --> trimming log.... /Volumes/DATA/dgerman/Library/Logs/DiskUtility.log <1J>: size (Kb): 4 [100] --> trimming log.... /Volumes/DATA/dgerman/Library/Logs/GoogleSoftwareUpdateAgent.log <1J>: size (Kb): 4 [50] --> trimming log.... /Volumes/DATA/dgerman/log/acctlg <2J>: size (Kb): 4 [50] --> trimming log.... /Volumes/DATA/dgerman/log/sshgd.sshvvv.log <1J>: size (Kb): 4 [10] --> trimming log.... /var/db/diagnostics/logdata.Persistent.* <1>: size (Kb): 4 [10] --> trimming log.... /var/log/authpriv.log <2J>: size (Kb): 40 [50] --> trimming log.... /Library/Logs/DiagnosticReports <2J>: size (Kb): 0 [50] --> trimming log.... newsyslog: error reading from /Library/Logs/DiagnosticReports: Is a directory log >lt 1484 07:40 04_warn.log 9849 07:40 03_err.log 8169 07:40 02_crit.log 2861 07:40 01_alert.log 2861 07:40 00_em.log 115018 07:41 secure.log 142 07:41 ftp.log.0 79 07:41 ftp.log 142 07:41 hwmond.log.0 79 07:41 hwmond.log 142 07:41 ipfw.log.0 79 07:41 ipfw.log 142 07:41 lpr.log.0 79 07:41 lpr.log 142 07:41 ppp.log.0 79 07:41 ppp.log 0 07:41 wtmp 142 07:41 slapd.log.0 79 07:41 slapd.log 142 07:41 xscertd.log.0 79 07:41 xscertd.log 7919 07:41 wifi.log.0 79 07:41 wifi.log 156 07:41 corecaptured.log.0 79 07:41 corecaptured.log 37166 07:41 authpriv.log.0 79 07:41 authpriv.log 5157 07:41 system.log

Launchctl file

/System/Library/LaunchDaemons/com.apple.newsyslog.plist

rearranged by DGG for clarity

   <key>Label</key> <string>com.apple.newsyslog</string>
    <key>ProgramArguments</key> <array> <string>/usr/sbin/newsyslog</string> </array>
    <key>LowPriorityIO</key> <true/>
    <key>Nice</key> <integer>1</integer>
    <key>StartCalendarInterval</key> <dict> <key>Minute</key> <integer>30</integer> </dict>

Errors

newsyslog: must have root privs or specify -r to run as nonproviledged user.
newsyslog: chmod(/var/log/wifi.log.1.bz2) in change_attrs: No such file or directory or rotation failed perhaps due to insufficient access rights.

see also

Apple Syslog Manager
logrotate