Someone at the cook-out mentioned they were using bacula for backups. I have just installed it and I have a DLT tape drive. I have it confugered OK and i can connect to my fd (client) and my sd (storage device). To keep things simple, lets say that I am doing everything on one server. I can label the tapes and even backup to my "File" device. My question is, what/where do I build a list of jobs to run? Lets say I want a "Full" backup of the system, and a seperate backup of just /home. Where do these jobs get defined, etc etc etc. This seems to be the only piece I am missing in the doc's. thanks mike
On Sat, Sep 02, 2006 at 10:42:10PM -0700, Mike Leo wrote:
My question is, what/where do I build a list of jobs to run?
Lets say I want a "Full" backup of the system, and a seperate backup of just /home.
All of that kind of policy based config gets defined in the director config file. You create filesets to define what files get backed up, and schedules to define when. -- Frank Sweetser fs at wpi.edu | For every problem, there is a solution that WPI Network Engineer | is simple, elegant, and wrong. - HL Mencken GPG fingerprint = 6174 1257 129E 0D21 D8D4 E8A3 8E39 29E3 E2E8 8CEC
Mike> Someone at the cook-out mentioned they were using bacula for Mike> backups. I have just installed it and I have a DLT tape drive. Same here, I've got Bacula running right now at home to a DLT7k drive. Mike> I have it confugered OK and i can connect to my fd (client) and Mike> my sd (storage device). Mike> To keep things simple, lets say that I am doing everything on Mike> one server. Yup. Mike> I can label the tapes and even backup to my "File" device. Mike> My question is, what/where do I build a list of jobs to run? This is one of the weaknesses of Bacula in my mind. You need to setup a JobDefs { ... } section in your /etc/bacula/bacula-dir.conf file. Mike> Lets say I want a "Full" backup of the system, and a seperate Mike> backup of just /home. Mike> Where do these jobs get defined, etc etc etc. This seems to be Mike> the only piece I am missing in the doc's. The docs are actually pretty good, it just sometimes takes a bit of re-reading to make it work. For example, here's what I have for one client. JobDefs { Name = "jack to tape" Type = Backup Level = Differential Client = jack-fd FileSet = "jack Full Set" Schedule = "MonthlyCycle" Storage = jfsnew-DLT Messages = Standard Pool = Backups Priority = 10 } Notice the Schedule and FileSet lines. Those are OTHER sections in the same file which define which schedule to use, and which filesets to run. Then you need to define a Job {...} section too: Job { Name = "jack-fd" JobDefs = "jack to tape" Write Bootstrap = "/var/lib/bacula/jack.bsr" } So here's most of what I've got defined for this one client. It's a pain, and I wish he would just dump this into the DB so it's easier to manage. Or maybe having a tool to write these files would be useful. :] FileSet { Name = "jack Full Set" #noatime = yes Ignore FileSet Changes = yes Include { Options { wilddir = \.thumbnails exclude = yes } File = @/etc/bacula/FileSet-jack } Exclude { File = /sys File = /proc File = /tmp File = ".journal" File = ".fsck" File = "/var/lib/mysql" File = ".thumbnails" } } Schedule { Name = "MonthlyCycle" Run = Level=Full 1st sun at 1:05 Run = Level=Differential 2nd-5th sun at 1:05 Run = Level=Incremental mon-sat at 1:05 } Client { Name = jack-fd Address = jack FDPort = 9102 Catalog = MyCatalog Password = "" # password for FileDaemon File Retention = 60 days # 30 days Job Retention = 12 months # six months AutoPrune = yes # Prune expired Jobs/Files } Let me know if this helps or not. John
Thanks a bunch. I had found this out 5 minutes after my first email, but your examples will come in handy i'm sure! Now i just need to start building my own backup FileSet's etc. One question, however... i have a 35/70Gb DLT drive with IV tapes, but I am only getting less than 30 gigs' per tape. my bacula-sd.conf lists " Media Type = DLT"....if this is not accurate, could bacula think I have only a 15/30Gb drive? Should it read DTL7000 or something? Also, does bacula compress naturally or do i have to tell it too? thanks again mike John Stoffel <john@stoffel.org> wrote: Mike> Someone at the cook-out mentioned they were using bacula for Mike> backups. I have just installed it and I have a DLT tape drive. Same here, I've got Bacula running right now at home to a DLT7k drive. Mike> I have it confugered OK and i can connect to my fd (client) and Mike> my sd (storage device). Mike> To keep things simple, lets say that I am doing everything on Mike> one server. Yup. Mike> I can label the tapes and even backup to my "File" device. Mike> My question is, what/where do I build a list of jobs to run? This is one of the weaknesses of Bacula in my mind. You need to setup a JobDefs { ... } section in your /etc/bacula/bacula-dir.conf file. Mike> Lets say I want a "Full" backup of the system, and a seperate Mike> backup of just /home. Mike> Where do these jobs get defined, etc etc etc. This seems to be Mike> the only piece I am missing in the doc's. The docs are actually pretty good, it just sometimes takes a bit of re-reading to make it work. For example, here's what I have for one client. JobDefs { Name = "jack to tape" Type = Backup Level = Differential Client = jack-fd FileSet = "jack Full Set" Schedule = "MonthlyCycle" Storage = jfsnew-DLT Messages = Standard Pool = Backups Priority = 10 } Notice the Schedule and FileSet lines. Those are OTHER sections in the same file which define which schedule to use, and which filesets to run. Then you need to define a Job {...} section too: Job { Name = "jack-fd" JobDefs = "jack to tape" Write Bootstrap = "/var/lib/bacula/jack.bsr" } So here's most of what I've got defined for this one client. It's a pain, and I wish he would just dump this into the DB so it's easier to manage. Or maybe having a tool to write these files would be useful. :] FileSet { Name = "jack Full Set" #noatime = yes Ignore FileSet Changes = yes Include { Options { wilddir = \.thumbnails exclude = yes } File = @/etc/bacula/FileSet-jack } Exclude { File = /sys File = /proc File = /tmp File = ".journal" File = ".fsck" File = "/var/lib/mysql" File = ".thumbnails" } } Schedule { Name = "MonthlyCycle" Run = Level=Full 1st sun at 1:05 Run = Level=Differential 2nd-5th sun at 1:05 Run = Level=Incremental mon-sat at 1:05 } Client { Name = jack-fd Address = jack FDPort = 9102 Catalog = MyCatalog Password = "" # password for FileDaemon File Retention = 60 days # 30 days Job Retention = 12 months # six months AutoPrune = yes # Prune expired Jobs/Files } Let me know if this helps or not. John _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
On Mon, Sep 04, 2006 at 03:11:14PM -0700, Mike Leo wrote:
i have a 35/70Gb DLT drive with IV tapes, but I am only getting less than 30 gigs' per tape.
my bacula-sd.conf lists " Media Type = DLT"....if this is not accurate, could bacula think I have only a 15/30Gb drive? Should it read DTL7000 or something?
Bacula doesn't actually parse the media type - it's nothing more than a unique identifier string that bacula uses to determine what volumes can be read by what drives. You could call file volumes "blue", DLT tapes "red", and DVDRs "green" and it would work fine. Bacula doesn't try to pre-determine the amount of space available on a tape. It just writes until the drive reports that it's hit the end of media, re-reads the last succesfully written block to make sure it got there okay, marks the volume full and asks for another one. The fact that you're getting about 30G of data per tape implies that you have hardware tape compression off. I believe the 'mt' command will tell you the compression status - the online manual has an example. The other 5G or so is probably taken up by volume metadata, such as the volume label and file attributes.
Also, does bacula compress naturally or do i have to tell it too?
No, you have to explicitly enable gzip compression if you want to use it. It's typically reccomended to use drive hardware compression instead, though, as a) drive hardware compression is usually faster, especially on older machines, and b) it's possible to run into file daemon incompatabilities that can make it difficult to restore data, especially if you're trying to restore the data onto a different machine that the one it was backed up on. -- Frank Sweetser fs at wpi.edu | For every problem, there is a solution that WPI Network Engineer | is simple, elegant, and wrong. - HL Mencken GPG fingerprint = 6174 1257 129E 0D21 D8D4 E8A3 8E39 29E3 E2E8 8CEC
Mike> Thanks a bunch. I had found this out 5 minutes after my first Mike> email, but your examples will come in handy i'm sure! No problem. Bacula is good, the manual is excellent for what it is, but the config files are Baroque... Kern's style I guess. Mike> i have a 35/70Gb DLT drive with IV tapes, but I am only getting Mike> less than 30 gigs' per tape. How is the tape drive hooked up to the system, and how fast is the system you're running on? Can you share some details on a completed job maybe? Mike> my bacula-sd.conf lists " Media Type = DLT"....if this is not Mike> accurate, could bacula think I have only a 15/30Gb drive? Mike> Should it read DTL7000 or something? Nah, doesn't really matter. Mike> Also, does bacula compress naturally or do i have to tell it too? I'm not aware of Bacula compressing data from the client when sending it to the tape drive unless you explicitly enable it. And it's generally not worth doing. The key thing with DLT is to push data to the drive at a high enough rate to keep it from shoe-shining. That both wears down the drive, and kills performance/storage capacity. Since the native speed is 5MB/s, you really really really wnat to be able to push 10MB/s to the drive to let it compress the data well. Unfortunately, that can be hard to do, esp if you have lots of small files to backup. Also, check your dmesg output to see how the drive is detected and at what speed it's running. When you put a tape into the drive, does the compression light come on as well? There's a known limitation (performance bottleneck actually) where Bacula uses transactions for inserting records into the DB one at a time. If they could insert more records at a time, performance would go up since the transaction overhead would be amortized across more INSERTS. Also, which DB are you using? And which version of Bacula? The newest released version I think (and I'd have to check) now uses sqlite3 as the default DB, which has a big performance loss compared to older 1.36.x versions which defaulted to sqlite which had better performance. Now one of the things I did to speed up my backups was to put my DB onto another host, since my tape drive and data were all one one system, so adding mysql updates ontop of that really just brought it down to it's knees. Anyway, more data on your backups and how well they currently run would be good. On my system, I get around 24gb to 68gb per tape, so it's certainly possible to get more data on there, esp since I'm using a dual CPU Xeon 550mhz box to do my backups. Old and not very fast. John
Just for grins, here's my output of 'mt -f /dev/nst0 status' on my DLT7k driver: > mt -f /dev/st0 status SCSI 2 tape drive: File number=0, block number=0, partition=0. Tape block size 0 bytes. Density code 0x1b (DLT 35GB). Soft error count since last status=0 General status bits on (41010000): BOT ONLINE IM_REP_EN You can also run the bacula command 'btape' on there to make sure you've got things configured properly. John
Ok...I have it all figured out in my head, now i just need to start configuring the jobs. I was able to get a 2nd 35/70 drive, now DLT1 and DLT2 are options for my backups. I can't figure out how to have 1 job write to both either concurrently, or sequentailly as the job only allows me to pick 1 or the other. I tried using: Storage = DLT1, DLT2 and by having 2 Storage lines in my bacula-dir.conf, but neither worked. Any idea's? John Stoffel <john@stoffel.org> wrote: Just for grins, here's my output of 'mt -f /dev/nst0 status' on my DLT7k driver:
mt -f /dev/st0 status SCSI 2 tape drive: File number=0, block number=0, partition=0. Tape block size 0 bytes. Density code 0x1b (DLT 35GB). Soft error count since last status=0 General status bits on (41010000): BOT ONLINE IM_REP_EN
You can also run the bacula command 'btape' on there to make sure you've got things configured properly. John _______________________________________________ Wlug mailing list Wlug@mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug
Mike> Ok...I have it all figured out in my head, now i just need to Mike> start configuring the jobs. When you think you have it figured out, re-read the manuals and examples. Mike> I was able to get a 2nd 35/70 drive, now DLT1 and DLT2 are Mike> options for my backups. Right, that will help. Mike> I can't figure out how to have 1 job write to both either Mike> concurrently, or sequentailly as the job only allows me to pick Mike> 1 or the other. Correct, a single job can only write to one device at a time. It will automatically write to the next available device (I think...) if it's available. So if you have TapeA in drive1 and TapeB in drive2, and they are both in the same Pool, and you have one big job that runs and fills up TapeA, it should start using TapeB to continue the save. If you have multiple jobs running at the same time, you can tweak the "Maximum Concurent Jobs = " Parameter in the bacula-sd.conf file, and also tweak it in the Bacula Director config file. Re-reading the manual again, I'm thinking that I really need to look into Spooling my jobs (only have four total) so that I don't shoe-shine the drive. Of course this means having 10-20gb of free space for this, which is harder. Mike> I tried using: Storage = DLT1, DLT2 and by having 2 Storage Mike> lines in my bacula-dir.conf, but neither worked. Join the bacula-users mailing list for more detailed questions like this. I know that Kern has been improving this aspect of Bacula lately. Whcih version are you running? As an aside, after running Legato(EMC) Networker for many years, bacula's user interface (cli or gui) feels clunky and painful. It's the biggest problem with the software. In general, it works quite well. I just did a restore of data last night spanned across three tapes and besides having to change them, it ran wonderfully. I just wish I had known ahead of time that I would need those tapes.
participants (3)
-
Frank Sweetser -
John Stoffel -
Mike Leo