I'm trying to setup a secondary log rotation mechanism on on of my production systems. Right now, the system is setup to rotate logs on a weekly basis. In addition to the weekly log rotation, I also need to take hourly snapshots (i.e. just the deltas between snapshots). As part of this process, I can not delete the original file since that is being handled by the weekly log rotation and archival process. These files get pretty big, so IO and CPU are both concerns. Any suggestions?
On Tue, May 26, 2009 at 9:31 AM, Tal Cohen <wlug@cohen123.com> wrote:
I'm trying to setup a secondary log rotation mechanism on on of my production systems.
Right now, the system is setup to rotate logs on a weekly basis.
In addition to the weekly log rotation, I also need to take hourly snapshots (i.e. just the deltas between snapshots).
And do what? Put them on another computer? If so, try plan9port's venti archival storage system or rdiff-backup. If not, could you specify?
As part of this process, I can not delete the original file since that is being handled by the weekly log rotation and archival process.
These files get pretty big, so IO and CPU are both concerns.
Any suggestions? _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
On May 26, 2009, at 9:31 AM, "Tal Cohen" <wlug@cohen123.com> wrote:
I'm trying to setup a secondary log rotation mechanism on on of my production systems.
Right now, the system is setup to rotate logs on a weekly basis.
In addition to the weekly log rotation, I also need to take hourly snapshots (i.e. just the deltas between snapshots).
Is the weekly rotation really needed, or do you just need to retain a certain amount of data? Switching to hourly rotation and retaining a weeks worth or more of data would be the simplest solution.
As part of this process, I can not delete the original file since that is being handled by the weekly log rotation and archival process.
These files get pretty big, so IO and CPU are both concerns.
Do you have plenty of space? If you do you can have syslog log the data into two files, one with a weekly rotation and one daily. -- Greg
Ive always just written a script to append the log file to a file name messages.MMDDYY.log and then delete the log file. and then created an init script to run it at boot time. That way everything is neatly sorted for later viewing, and i can delete the ones i no longer need manually. On Tue, May 26, 2009 at 10:56 AM, Gregory Boyce <gregory.boyce@gmail.com> wrote:
On May 26, 2009, at 9:31 AM, "Tal Cohen" <wlug@cohen123.com> wrote:
I'm trying to setup a secondary log rotation mechanism on on of my production systems.
Right now, the system is setup to rotate logs on a weekly basis.
In addition to the weekly log rotation, I also need to take hourly snapshots (i.e. just the deltas between snapshots).
Is the weekly rotation really needed, or do you just need to retain a certain amount of data? Switching to hourly rotation and retaining a weeks worth or more of data would be the simplest solution.
As part of this process, I can not delete the original file since that is being handled by the weekly log rotation and archival process.
These files get pretty big, so IO and CPU are both concerns.
Do you have plenty of space? If you do you can have syslog log the data into two files, one with a weekly rotation and one daily. -- Greg _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
On Sat, May 30, 2009 at 08:40:08AM -0400, Jason Couture wrote:
Ive always just written a script to append the log file to a file name messages.MMDDYY.log and then delete the log file. and then created an init script to run it at boot time. That way everything is neatly sorted for later viewing, and i can delete the ones i no longer need manually.
logrotate does this automatically: /etc/logrotate.conf: # see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # use date as a suffix of the rotated file dateext # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp and btmp -- we'll rotate them here /var/log/wtmp { monthly create 0664 root utmp minsize 1M rotate 1 } /var/log/btmp { missingok monthly create 0600 root utmp rotate 1 } # system-specific logs may be also be configured here. /etc/logrotate.d/syslog: /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true endscript } Resulting rotated log files: -rw-r-----. 1 root wheel 703684 2009-05-03 03:42 messages-20090503 -rw-r-----. 1 root wheel 814677 2009-05-10 11:42 messages-20090510 -rw-r-----. 1 root wheel 779478 2009-05-17 16:06 messages-20090517 -rw-r-----. 1 root wheel 721265 2009-05-24 15:03 messages-20090524 -rw-r-----. 1 root wheel 338888 2009-05-30 10:01 messages
Guys, thanks for all the replies. In the end it looks like I will just end up doing regular log rotation. That said, I still think it is an interesting question: How to maintain snapshots of logs without both truncating them when taking the snapshot, and without putting excessive load on the system. Any thoughts? -----Original Message----- From: wlug-bounces@mail.wlug.org [mailto:wlug-bounces@mail.wlug.org] On Behalf Of Chuck Anderson Sent: Saturday, May 30, 2009 10:24 AM To: wlug@mail.wlug.org Subject: Re: [Wlug] Log Rotation (Snapshot) On Sat, May 30, 2009 at 08:40:08AM -0400, Jason Couture wrote:
Ive always just written a script to append the log file to a file name messages.MMDDYY.log and then delete the log file. and then created an init script to run it at boot time. That way everything is neatly sorted for later viewing, and i can delete the ones i no longer need manually.
logrotate does this automatically: /etc/logrotate.conf: # see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # use date as a suffix of the rotated file dateext # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp and btmp -- we'll rotate them here /var/log/wtmp { monthly create 0664 root utmp minsize 1M rotate 1 } /var/log/btmp { missingok monthly create 0600 root utmp rotate 1 } # system-specific logs may be also be configured here. /etc/logrotate.d/syslog: /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true endscript } Resulting rotated log files: -rw-r-----. 1 root wheel 703684 2009-05-03 03:42 messages-20090503 -rw-r-----. 1 root wheel 814677 2009-05-10 11:42 messages-20090510 -rw-r-----. 1 root wheel 779478 2009-05-17 16:06 messages-20090517 -rw-r-----. 1 root wheel 721265 2009-05-24 15:03 messages-20090524 -rw-r-----. 1 root wheel 338888 2009-05-30 10:01 messages _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
Seriously, check out venti. You can delete the logs all you want as long as you do it after the venti backup runs and you'll keep them forever. Venti stores sha1-addressed blocks of data, so multiple copies of the same data take up a smaller amount of space in venti. Also, blocks are LZO compressed for permanent storage. Venti splits the archive into 500MB "arenas" that you can easily burn to CD when full and keep safe. Venti can be read online with several clients with no slowdown, and doing an actual backup (with the vac program) causes virtually no system load increase and only takes a couple of minutes even on things that are appended to a lot (e.g., a maildir that subscribes to LKML). The coolest online-access to a venti server is via vacfs, which will give you a directory tree like this under the mount point: /YYYY/MMDD/path/to/backedup/file You can bind-mount one of the /YYYY/MMDD directories over your current working directory and pretend that it's 2 years ago and access all your files. Only caveat is that it's hard to set up, but I have some scripts. On Sat, May 30, 2009 at 2:39 PM, Tal Cohen <wlug@cohen123.com> wrote:
Guys, thanks for all the replies. In the end it looks like I will just end up doing regular log rotation.
That said, I still think it is an interesting question: How to maintain snapshots of logs without both truncating them when taking the snapshot, and without putting excessive load on the system. Any thoughts?
-----Original Message----- From: wlug-bounces@mail.wlug.org [mailto:wlug-bounces@mail.wlug.org] On Behalf Of Chuck Anderson Sent: Saturday, May 30, 2009 10:24 AM To: wlug@mail.wlug.org Subject: Re: [Wlug] Log Rotation (Snapshot)
On Sat, May 30, 2009 at 08:40:08AM -0400, Jason Couture wrote:
Ive always just written a script to append the log file to a file name messages.MMDDYY.log and then delete the log file. and then created an init script to run it at boot time. That way everything is neatly sorted for later viewing, and i can delete the ones i no longer need manually.
logrotate does this automatically:
/etc/logrotate.conf:
# see "man logrotate" for details # rotate log files weekly weekly
# keep 4 weeks worth of backlogs rotate 4
# create new (empty) log files after rotating old ones create
# use date as a suffix of the rotated file dateext
# uncomment this if you want your log files compressed #compress
# RPM packages drop log rotation information into this directory include /etc/logrotate.d
# no packages own wtmp and btmp -- we'll rotate them here /var/log/wtmp { monthly create 0664 root utmp minsize 1M rotate 1 }
/var/log/btmp { missingok monthly create 0600 root utmp rotate 1 }
# system-specific logs may be also be configured here.
/etc/logrotate.d/syslog:
/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true endscript }
Resulting rotated log files:
-rw-r-----. 1 root wheel 703684 2009-05-03 03:42 messages-20090503 -rw-r-----. 1 root wheel 814677 2009-05-10 11:42 messages-20090510 -rw-r-----. 1 root wheel 779478 2009-05-17 16:06 messages-20090517 -rw-r-----. 1 root wheel 721265 2009-05-24 15:03 messages-20090524 -rw-r-----. 1 root wheel 338888 2009-05-30 10:01 messages _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
_______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
Maybe it's just me, but rsync would probably do the right thing for you unless you have incredibly large logs and/or are already dealing with large amounts of i/o, etc. Otherwise, it's not hard to manually do something like ... $current_log file is $X bytes and $snapshot_log file is $Y bytes, so if $Y > $X just seek($Y) and grab the remainder of the file. The biggest issue is knowing when you need to start over. ie: if the log file is rotated out, then you can't just go by size but may have to validate log contents or whatever. If you write out logs in append-only mode, then the above method works fine. On Sat, May 30, 2009 at 2:39 PM, Tal Cohen <wlug@cohen123.com> wrote:
Guys, thanks for all the replies. In the end it looks like I will just end up doing regular log rotation.
That said, I still think it is an interesting question: How to maintain snapshots of logs without both truncating them when taking the snapshot, and without putting excessive load on the system. Any thoughts?
-----Original Message----- From: wlug-bounces@mail.wlug.org [mailto:wlug-bounces@mail.wlug.org] On Behalf Of Chuck Anderson Sent: Saturday, May 30, 2009 10:24 AM To: wlug@mail.wlug.org Subject: Re: [Wlug] Log Rotation (Snapshot)
On Sat, May 30, 2009 at 08:40:08AM -0400, Jason Couture wrote:
Ive always just written a script to append the log file to a file name messages.MMDDYY.log and then delete the log file. and then created an init script to run it at boot time. That way everything is neatly sorted for later viewing, and i can delete the ones i no longer need manually.
logrotate does this automatically:
/etc/logrotate.conf:
# see "man logrotate" for details # rotate log files weekly weekly
# keep 4 weeks worth of backlogs rotate 4
# create new (empty) log files after rotating old ones create
# use date as a suffix of the rotated file dateext
# uncomment this if you want your log files compressed #compress
# RPM packages drop log rotation information into this directory include /etc/logrotate.d
# no packages own wtmp and btmp -- we'll rotate them here /var/log/wtmp { monthly create 0664 root utmp minsize 1M rotate 1 }
/var/log/btmp { missingok monthly create 0600 root utmp rotate 1 }
# system-specific logs may be also be configured here.
/etc/logrotate.d/syslog:
/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true endscript }
Resulting rotated log files:
-rw-r-----. 1 root wheel 703684 2009-05-03 03:42 messages-20090503 -rw-r-----. 1 root wheel 814677 2009-05-10 11:42 messages-20090510 -rw-r-----. 1 root wheel 779478 2009-05-17 16:06 messages-20090517 -rw-r-----. 1 root wheel 721265 2009-05-24 15:03 messages-20090524 -rw-r-----. 1 root wheel 338888 2009-05-30 10:01 messages _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
_______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
participants (6)
-
Chuck Anderson -
Gregory Boyce -
J.R. Mauro -
Jason Couture -
Tal Cohen -
Theo Van Dinter