- WLUG - wlug.mailman3.com

Re: [Wlug] NFS question

by John Stoffel

>>>>> "Jeff" == Jeff Moyer <jmoyer(a)redhat.com> writes: Jeff> Don, are you able to use OCFS? If not, you could at least Jeff> cluster the NFS server with 2 RHEL boxes. I believe this will Jeff> be cheaper than buying a redundant NAS (and serves precisely the Jeff> same purpose). In fact, I believe that the cluster software was Jeff> included in the AS2.1 license. This is too funny, I didn't even see this reply before I wrote the other one suggesting just this type of idea. Heh! Since you can buy 2U boxes with a bunch of mirrored storage four around 3k/per, you should be able to setup a cluster providing NFS services to the other hosts pretty cheaply and quickly. John

Oct. 13, 2004

RE: [Wlug] NFS question

by John Stoffel

Don> Thanx for the info John. Unfortunately I can't use the Red Hat Don> GFS, it requires v3.0 and we cannot upgrade to that yet. Since it's open source, can't you get it from http://sources.redhat.com/cluster/gfs/ and see if it will install on your systems? Don> I have been pushing for it, but Oracle won't give me an definite Don> answer as to whether their apps are certified for RH 3.0. That's silly, but very understandable. From looking at your corp website, I can see why you're interested in only deploying supportable systems, esp in such a production environment. Don> We could go with Veritas' Storage Foundation for Oracle RAC, but Don> that would require considerable change, and no one here knows the Don> software. For the time being I'll probably have to go with a NAS Don> device and make it as redundant as possible. I know Veritas. :] And I'm looking for a job. :] Well, I know VxVM/VxFS quite well and I've been exposed to their Clustering software as well. Good stuff all around. But that doesn't solve the question here, which is how to get a good reliable NFS file storage (would the storage be on the SAN, or local to the server) for a good price. Here's a thought, but a pair of cheap 2U servers and install RHEL 3.0 along with the GFS filesystem and clustering software. Export via NFS to the other servers. If one node goes down, you've got a backup and failover. And it would give you exposure/experience with RHEL 3.0, GFS and clustering so you would be working to migrate the Oracle instances to the same type of setup down the road. John

Oct. 13, 2004

Re: [Wlug] NFS question

by Jeff Moyer

==> Regarding Re: [Wlug] NFS question; "John Stoffel" <stoffel(a)lucent.com> adds: Don> I have an Oracle 9i RAC set up consisting of 3 database servers, two Don> apps servers and an EMC Clarion CX700 SAN. The servers are all Don> running Red Hat 2.1, the db servers are at kernel .34 and the apps Don> servers are at .49. I have 5 nfs mount points shared out by the first Don> database server. stoffel> This is the problem, the fact that you're using NFS in a clustered stoffel> environment. Have you looked into using a Cluster Filesystem for stoffel> your needs? Some options would be: stoffel> http://www.redhat.com/software/rha/gfs/ stoffel> It even mentions Oracle 9i in the blurb. Don, are you able to use OCFS? If not, you could at least cluster the NFS server with 2 RHEL boxes. I believe this will be cheaper than buying a redundant NAS (and serves precisely the same purpose). In fact, I believe that the cluster software was included in the AS2.1 license. Don> One mount point is for logs from the database servers (only they Don> connect to it), stoffel> Why do they all need to log to one filesystem? Can't they log stoffel> locally and if you want to aggregate the logs, you could rsync stoffel> them hourly from the servers to a central log host. Don> and the others are for miscellaneous files and storage. We have been Don> doing failover testing, shutting down a database server and seeing if Don> the other two keep processing requests. We have a problem when we Don> shut down the first database server, the one sharing the nfs volumes. Don> The other two servers just hang trying to connect to the shares. I Don> have tried every option in my fstab file to make the servers stop Don> trying to reach the nfs shares if the server is not available, but Don> nothing has worked. Here is one of the lines from the fstab file Don> where I mount a share: stoffel> Yeah, what you're trying to do isn't going to work without either: stoffel> 1. moving to a Cluster Filesystem so that all hosts and read/write stoffel> to the same filesystem on the Clarrion SAN concurrently. stoffel> 2. moving to a Cluster setup where you have TWO servers in a stoffel> cluster, and the cluster provides the NFS file service to other stoffel> servers. Right. This is an easy option, I think, if you can afford 1 or 2 more boxes (Depending on whether you want to have a dedicated file server not running database apps). stoffel> 3. Getting a NAS box which provides NFS service to the DB servers stoffel> and which has the require reliability you need. stoffel> I personally like NetApps, but they can be pricey. But even with stoffel> a single head, they're reliable and run well. I had one box with stoffel> an uptime of almost 500 days. Oh yeah, UPS and Generator backup stoffel> helps as well. *grin* stoffel> You could get away with using another cheaper NAS, but since you stoffel> spent the money on the SAN, why not just use that and GFS to stoffel> provide the storage you need. Esp if most of the filesystems you stoffel> provide are read (mostly) then you shouldn't have many problems. I do believe RHEL 3 has been certified with Oracle 9i. I'm not sure about the particular apps you are using, but the db itself should work just fine. I would ask your contact at Red Hat. -Jeff

Oct. 13, 2004

RE: [Wlug] NFS question

by Don Peterson

Thanx for the info John. Unfortunately I can't use the Red Hat GFS, it requires v3.0 and we cannot upgrade to that yet. I have been pushing for it, but Oracle won't give me an definite answer as to whether their apps are certified for RH 3.0. We could go with Veritas' Storage Foundation for Oracle RAC, but that would require considerable change, and no one here knows the software. For the time being I'll probably have to go with a NAS device and make it as redundant as possible. -----Original Message----- From: wlug-bounces(a)mail.wlug.org [mailto:wlug-bounces@mail.wlug.org] On Behalf Of John Stoffel Sent: Wednesday, October 13, 2004 9:28 AM To: Worcester Linux Users Group Subject: Re: [Wlug] NFS question Don> I have an Oracle 9i RAC set up consisting of 3 database servers, Don> two apps servers and an EMC Clarion CX700 SAN. The servers are all Don> running Red Hat 2.1, the db servers are at kernel .34 and the apps Don> servers are at .49. I have 5 nfs mount points shared out by the Don> first database server. This is the problem, the fact that you're using NFS in a clustered environment. Have you looked into using a Cluster Filesystem for your needs? Some options would be: http://www.redhat.com/software/rha/gfs/ It even mentions Oracle 9i in the blurb. Don> One mount point is for logs from the database servers (only they Don> connect to it), Why do they all need to log to one filesystem? Can't they log locally and if you want to aggregate the logs, you could rsync them hourly from the servers to a central log host. Don> and the others are for miscellaneous files and storage. We have Don> been doing failover testing, shutting down a database server and Don> seeing if the other two keep processing requests. We have a Don> problem when we shut down the first database server, the one Don> sharing the nfs volumes. The other two servers just hang trying to Don> connect to the shares. I have tried every option in my fstab file Don> to make the servers stop trying to reach the nfs shares if the Don> server is not available, but nothing has worked. Here is one of Don> the lines from the fstab file where I mount a share: Yeah, what you're trying to do isn't going to work without either: 1. moving to a Cluster Filesystem so that all hosts and read/write to the same filesystem on the Clarrion SAN concurrently. 2. moving to a Cluster setup where you have TWO servers in a cluster, and the cluster provides the NFS file service to other servers. 3. Getting a NAS box which provides NFS service to the DB servers and which has the require reliability you need. I personally like NetApps, but they can be pricey. But even with a single head, they're reliable and run well. I had one box with an uptime of almost 500 days. Oh yeah, UPS and Generator backup helps as well. *grin* You could get away with using another cheaper NAS, but since you spent the money on the SAN, why not just use that and GFS to provide the storage you need. Esp if most of the filesystems you provide are read (mostly) then you shouldn't have many problems. John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies stoffel(a)lucent.com - http://www.lucent.com - 978-952-7548 _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug

Oct. 13, 2004

Re: [Wlug] NFS question

by John Stoffel

Don> I have an Oracle 9i RAC set up consisting of 3 database servers, Don> two apps servers and an EMC Clarion CX700 SAN. The servers are Don> all running Red Hat 2.1, the db servers are at kernel .34 and the Don> apps servers are at .49. I have 5 nfs mount points shared out by Don> the first database server. This is the problem, the fact that you're using NFS in a clustered environment. Have you looked into using a Cluster Filesystem for your needs? Some options would be: http://www.redhat.com/software/rha/gfs/ It even mentions Oracle 9i in the blurb. Don> One mount point is for logs from the database servers (only they Don> connect to it), Why do they all need to log to one filesystem? Can't they log locally and if you want to aggregate the logs, you could rsync them hourly from the servers to a central log host. Don> and the others are for miscellaneous files and storage. We have Don> been doing failover testing, shutting down a database server and Don> seeing if the other two keep processing requests. We have a Don> problem when we shut down the first database server, the one Don> sharing the nfs volumes. The other two servers just hang trying Don> to connect to the shares. I have tried every option in my fstab Don> file to make the servers stop trying to reach the nfs shares if Don> the server is not available, but nothing has worked. Here is one Don> of the lines from the fstab file where I mount a share: Yeah, what you're trying to do isn't going to work without either: 1. moving to a Cluster Filesystem so that all hosts and read/write to the same filesystem on the Clarrion SAN concurrently. 2. moving to a Cluster setup where you have TWO servers in a cluster, and the cluster provides the NFS file service to other servers. 3. Getting a NAS box which provides NFS service to the DB servers and which has the require reliability you need. I personally like NetApps, but they can be pricey. But even with a single head, they're reliable and run well. I had one box with an uptime of almost 500 days. Oh yeah, UPS and Generator backup helps as well. *grin* You could get away with using another cheaper NAS, but since you spent the money on the SAN, why not just use that and GFS to provide the storage you need. Esp if most of the filesystems you provide are read (mostly) then you shouldn't have many problems. John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies stoffel(a)lucent.com - http://www.lucent.com - 978-952-7548

Oct. 13, 2004

Reminder: WLUG meeting Tonight (Wed Oct 13)

by Andy Stewart

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 HI everybody, We'll be having a WLUG meeting tonight (Wed Oct 13) at 7:00 PM in the usual place. There isn't a single topic on the agenda, but here's what I'll cover to at least get a discussion started: 1) Update on the install fest with Worcester Computer Co-op, which was held this past Saturday 2) Update on the Linux adult education class I'm teaching 3) GNU FDL license and Debian's critique thereof (this could almost be a meeting by itself - maybe it should be???) 4) Upcoming meetings and guest speaker announcement 5) Any/all topics of interest brought up by other meeting attendees See ya tonight, Andy - -- Andy Stewart, Founder Worcester Linux Users' Group Worcester, MA USA http://www.wlug.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFBbK81Hl0iXDssISsRAsY8AJ4wmo8AuS43XWgO4bPYLwR7Xb/BDwCaAn3D HDTqh9qAp7Rxx5o/EQ8wmSI= =mQlq -----END PGP SIGNATURE-----

Oct. 13, 2004

Re: [Wlug] NFS question

by Adam Keck

Hello Don, True, but if you must use NAS for this, a redundant NAS device takes care of the single point of failure issue. Look at NAS solutions that provide clustered heads attached with redundant paths to redundant disks. Clustered Linux w/ SAN solutions would work or perhaps a blackbox like a Network Appliance filer cluster ($$$!) depending on your budget and time constraints. Moving the nfs shares off the first database server is important, because, in order to have redundancy in the cluster, members each must have the same function. Right now the first database server provides an additional unique function. Nothing can take over for it, as you have seen. -Adam On Oct 12, 2004, at 11:48 AM, Don Peterson wrote: > I'm not sync'ing data, all three servers are writing logs to the share > constantly (the applcsf share). We didn't buy Veritas volume manager, > and we are not running Red Hat 3.0 yet, so I had to go with NFS for a > shared file system. Believe me, I wish I didn't have to use NFS. > I am looking into buying a NAS device to store the shares on, that way > any of the servers can go down and the others won't be affected. But > the NAS will still be a single point of failure.

Oct. 12, 2004

RE: [Wlug] NFS question

by Don Peterson

I'm not sync'ing data, all three servers are writing logs to the share constantly (the applcsf share). We didn't buy Veritas volume manager, and we are not running Red Hat 3.0 yet, so I had to go with NFS for a shared file system. Believe me, I wish I didn't have to use NFS. I am looking into buying a NAS device to store the shares on, that way any of the servers can go down and the others won't be affected. But the NAS will still be a single point of failure. -----Original Message----- From: wlug-bounces(a)mail.wlug.org [mailto:wlug-bounces@mail.wlug.org] On Behalf Of Charles R. Anderson Sent: Tuesday, October 12, 2004 10:22 AM To: wlug(a)mail.wlug.org Subject: Re: [Wlug] NFS question On Tue, Oct 12, 2004 at 11:15:11AM -0400, Don Peterson wrote: > I mount a share: > shoprod1:/applcsf /applcsf nfs > soft,rsize=8192,wsize=8192,retrans=6,timeo=14,intr > Can you see anything wrong with this line, or can anyone suggest > something else to try. I can't even forceably unmount the shares with > an umount -f. > And if we try to kill any processes accessing the shares the kill > command just hangs (even with a kill -9). > Any suggestions? I don't think there is any good solution. Using the "soft" and "intr" mount options is an extremely bad idea. Do it only if you don't care about losing/corrupting your data. Important data should always be mounted "hard,nointr". Personally, I avoid NFS like the plague. I only use it for readonly mounts (where soft,intr is less of a problem). Instead, I've standardized on rsync over ssh with RSA keys to get data between servers that need to be synchronized. _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug

Oct. 12, 2004

Re: [Wlug] NFS question

by Charles R. Anderson

On Tue, Oct 12, 2004 at 11:15:11AM -0400, Don Peterson wrote: > I mount a share: > shoprod1:/applcsf /applcsf nfs > soft,rsize=8192,wsize=8192,retrans=6,timeo=14,intr > Can you see anything wrong with this line, or can anyone suggest > something else to try. I can't even forceably unmount the shares with > an umount -f. > And if we try to kill any processes accessing the shares the kill > command just hangs (even with a kill -9). > Any suggestions? I don't think there is any good solution. Using the "soft" and "intr" mount options is an extremely bad idea. Do it only if you don't care about losing/corrupting your data. Important data should always be mounted "hard,nointr". Personally, I avoid NFS like the plague. I only use it for readonly mounts (where soft,intr is less of a problem). Instead, I've standardized on rsync over ssh with RSA keys to get data between servers that need to be synchronized.

Oct. 12, 2004

NFS question

by Don Peterson

Hello All, I've been a slient member for awhile, now I have a question/problem I hope someone can answer. I have an Oracle 9i RAC set up consisting of 3 database servers, two apps servers and an EMC Clarion CX700 SAN. The servers are all running Red Hat 2.1, the db servers are at kernel .34 and the apps servers are at .49. I have 5 nfs mount points shared out by the first database server. One mount point is for logs from the database servers (only they connect to it), and the others are for miscellaneous files and storage. We have been doing failover testing, shutting down a database server and seeing if the other two keep processing requests. We have a problem when we shut down the first database server, the one sharing the nfs volumes. The other two servers just hang trying to connect to the shares. I have tried every option in my fstab file to make the servers stop trying to reach the nfs shares if the server is not available, but nothing has worked. Here is one of the lines from the fstab file where I mount a share: shoprod1:/applcsf /applcsf nfs soft,rsize=8192,wsize=8192,retrans=6,timeo=14,intr Can you see anything wrong with this line, or can anyone suggest something else to try. I can't even forceably unmount the shares with an umount -f. And if we try to kill any processes accessing the shares the kill command just hangs (even with a kill -9). Any suggestions? Thanx, Don Peterson Network/Telecomm Mgr. Sterilite Corporation

Oct. 12, 2004