- WLUG - wlug.mailman3.com

Re: [Wlug] NFS Trouble

by Chuck Haines

I will try all of these things and get back to everyone on what I find out. Thanks a bunch for your help. Chuck On Thu, 9 Sep 2004 14:39:21 -0400, John Stoffel <stoffel(a)lucent.com> wrote: > > Chuck> The curent line from the fstab is as follows: > Chuck> hostname:/home /home nfs > Chuck> rsize=4096,wsize=4096,bg,intr,rw,actimeo=15 0 0 > > Here's one of the problems, you're not specifying TCP here, and you're > using pretty low numbers. Try changing to use 32768 and tcp for your > numbers. > > Also, have you checked the duplex on server to make sure it's ok? You > mention you're using RAID on the server, I assume RAID5? If so, what > kind of stripe size are you using? > > What does 'vmstat 1' (on the server) say from about 15 seconds before > the copy kicks in, to about 15 seconds after the copy is done? > > Does the time scale up linearly when you copy a 100mb file? Or drop > by half when you copy a 25mb file? > > What filesystem are you using on the NFS server and do you have quotas > or anything else like that setup? How full is the filesystem? > > What happens if you write a pair of 50 mb files at the same time from > two different clients? Does the time double? Does the load double? > > Basically, I don't know what's going on here, but I suspect: > > 1. network speed/duplex mismatch - that you're seeing lots of timeouts > and retries on writes, but not reads. > > 2. NFS needs tuning on the clients to write in bigger chunks > > 3. You're RAID stinks. Which reminds me, how much time does a write > from a non-raid disk on the server to the home directory/raid disk > take? Can you time the write time of a 50mb file and if it's still > ugly, then you've narrowed down the issue. > > Basically, there's alot of potential problems here and we need more > details. > > John > John Stoffel - Senior Unix Systems Administrator - Lucent Technologies > stoffel(a)lucent.com - http://www.lucent.com - 978-952-7548 > -- Chuck Haines chaines(a)gmail.com ------------------------------------------- Tau Kappa Epsilon Fraternity WPI Class of 2005 ------------------------------------------- AIM: CyberGrex YIM: CyberGrex_27 ICQ: 3707881 -------------------------------------------

Sept. 9, 2004

Re: [Wlug] NFS Trouble

by John Stoffel

Chuck> The curent line from the fstab is as follows: Chuck> hostname:/home /home nfs Chuck> rsize=4096,wsize=4096,bg,intr,rw,actimeo=15 0 0 Here's one of the problems, you're not specifying TCP here, and you're using pretty low numbers. Try changing to use 32768 and tcp for your numbers. Also, have you checked the duplex on server to make sure it's ok? You mention you're using RAID on the server, I assume RAID5? If so, what kind of stripe size are you using? What does 'vmstat 1' (on the server) say from about 15 seconds before the copy kicks in, to about 15 seconds after the copy is done? Does the time scale up linearly when you copy a 100mb file? Or drop by half when you copy a 25mb file? What filesystem are you using on the NFS server and do you have quotas or anything else like that setup? How full is the filesystem? What happens if you write a pair of 50 mb files at the same time from two different clients? Does the time double? Does the load double? Basically, I don't know what's going on here, but I suspect: 1. network speed/duplex mismatch - that you're seeing lots of timeouts and retries on writes, but not reads. 2. NFS needs tuning on the clients to write in bigger chunks 3. You're RAID stinks. Which reminds me, how much time does a write from a non-raid disk on the server to the home directory/raid disk take? Can you time the write time of a 50mb file and if it's still ugly, then you've narrowed down the issue. Basically, there's alot of potential problems here and we need more details. John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies stoffel(a)lucent.com - http://www.lucent.com - 978-952-7548

Sept. 9, 2004

Re: [Wlug] NFS Trouble

by Chuck Haines

It is all software raid, so I assumed slightly more intensive, however I didn't expect a load of 7 either. The raid array is 7 73.6 GB Ultra320 SCSI 15k RPM drives connected to the machine via Ultra320 SCSI (it's a dell powervault connected to a dell poweredge). Chuck On Thu, 9 Sep 2004 13:54:47 -0400, Keller, Tim <tim.keller(a)stratus.com> wrote: > What's the disk like on the nfs server? > > If it's all software based RAID, writing is going to be really expensive CPU > wise vs. reading. Though even with that said, I can't see where a 50 meg > file would put a load of 7 on a box. > > Tim. > > > -----Original Message----- > From: Dwight A. Ernest [mailto:dwight@significant.com] > Sent: Thursday, September 09, 2004 12:24 PM > To: Chuck Haines; Worcester Linux Users Group > Subject: Re: [Wlug] NFS Trouble > > With what options did you mount the volume on the client? > > Chuck Haines wrote: > > > I have an interesting problem that is occuring. I have a /home > > directory on a file server and I have several login servers that mount > > this /home directory via NFS. Now this /home directory has a ton of > > files and directorys (about 2000 user accounts). I noticed that when > > I perform a copy of a large file ( I tested with a 50 MB file) on a > > login server from something like /tmp to the NFS mounted /home, the > > load average on the file server jumps up to over 7, and seems to take > > forever (around 3-4 minutes). If I copy a large file (once again I > > tested with 50 MB) from the NFS mounted /home to like /tmp on the > > login server, there is no noticable load average increase and it > > occurs in approximately 10-15 seconds. So it seems that writes to the > > NFS mounted /home are taking longer than they should and are spiking > > the load average. Both the file server and the login servers are dual > > P3's with 2 gigs of ram. Has anyone ever seen anything like this, or > > knows of any solutions to this problem? > > > > Thanks, > > Chuck > > > > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > _______________________________________________ > Wlug mailing list > Wlug(a)mail.wlug.org > http://mail.wlug.org/mailman/listinfo/wlug > -- Chuck Haines chaines(a)gmail.com ------------------------------------------- Tau Kappa Epsilon Fraternity WPI Class of 2005 ------------------------------------------- AIM: CyberGrex YIM: CyberGrex_27 ICQ: 3707881 -------------------------------------------

Sept. 9, 2004

RE: [Wlug] NFS Trouble

by Keller, Tim

What's the disk like on the nfs server? If it's all software based RAID, writing is going to be really expensive CPU wise vs. reading. Though even with that said, I can't see where a 50 meg file would put a load of 7 on a box. Tim. -----Original Message----- From: Dwight A. Ernest [mailto:dwight@significant.com] Sent: Thursday, September 09, 2004 12:24 PM To: Chuck Haines; Worcester Linux Users Group Subject: Re: [Wlug] NFS Trouble With what options did you mount the volume on the client? Chuck Haines wrote: > I have an interesting problem that is occuring. I have a /home > directory on a file server and I have several login servers that mount > this /home directory via NFS. Now this /home directory has a ton of > files and directorys (about 2000 user accounts). I noticed that when > I perform a copy of a large file ( I tested with a 50 MB file) on a > login server from something like /tmp to the NFS mounted /home, the > load average on the file server jumps up to over 7, and seems to take > forever (around 3-4 minutes). If I copy a large file (once again I > tested with 50 MB) from the NFS mounted /home to like /tmp on the > login server, there is no noticable load average increase and it > occurs in approximately 10-15 seconds. So it seems that writes to the > NFS mounted /home are taking longer than they should and are spiking > the load average. Both the file server and the login servers are dual > P3's with 2 gigs of ram. Has anyone ever seen anything like this, or > knows of any solutions to this problem? > > Thanks, > Chuck > _______________________________________________ Wlug mailing list Wlug(a)mail.wlug.org http://mail.wlug.org/mailman/listinfo/wlug

Sept. 9, 2004

Fwd: [Wlug] NFS Trouble

by Chuck Haines

Meant to sent this to the whole list. ---------- Forwarded message ---------- From: Chuck Haines <chaines(a)gmail.com> Date: Thu, 9 Sep 2004 13:44:53 -0400 Subject: Re: [Wlug] NFS Trouble To: jmoyer(a)redhat.com Server: Kernel = 2.4.21-15.ELsmp nfs-util = 1.0.6-21EL Client: Kernel = 2.4.21-15.ELsmp nfs-util = 1.0.6-21EL Both machines are running RedHat Advance Server 3.0 ( Taroon Update 2). When performing the copy, top on the server loops like this after about 75% of the transfer: 13:43:19 up 26 days, 6:05, 2 users, load average: 7.19, 3.30, 1.62 125 processes: 124 sleeping, 1 running, 0 zombie, 0 stopped CPU states: cpu user nice system irq softirq iowait idle total 0.2% 0.0% 1.6% 0.0% 0.4% 48.3% 49.3% cpu00 0.0% 0.0% 1.6% 0.0% 0.0% 66.6% 31.6% cpu01 0.4% 0.0% 1.6% 0.0% 0.8% 30.0% 67.0% Mem: 2061576k av, 2044428k used, 17148k free, 0k shrd, 351044k buff 1425940k actv, 270704k in_d, 31124k in_c Swap: 1052248k av, 19780k used, 1032468k free 1371100k cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 24242 bkmcd 15 0 4648 4220 3012 S 0.4 0.2 0:03 1 smbd 165 root 17 0 0 0 0 SW 0.2 0.0 26:10 0 raid5d 4783 root 15 0 0 0 0 DW 0.2 0.0 80:48 0 nfsd 4790 root 15 0 0 0 0 DW 0.2 0.0 80:29 1 nfsd 4788 root 15 0 0 0 0 DW 0.2 0.0 82:35 0 nfsd 4785 root 15 0 0 0 0 DW 0.2 0.0 80:40 0 nfsd 4784 root 15 0 0 0 0 DW 0.2 0.0 81:18 1 nfsd 24515 chaines 15 0 1256 1256 900 R 0.2 0.0 0:00 0 top 1 root 15 0 508 468 448 S 0.0 0.0 0:24 0 init 2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0 As you can see, most of the CPU is in an IO wait state. Hope this helps, Chuck On Thu, 9 Sep 2004 12:54:28 -0400, Jeff Moyer <jmoyer(a)redhat.com> wrote: > ==> Regarding [Wlug] NFS Trouble; Chuck Haines <chaines(a)gmail.com> adds: > > chaines> I have an interesting problem that is occuring. I have a /home > chaines> directory on a file server and I have several login servers that > chaines> mount this /home directory via NFS. Now this /home directory has > chaines> a ton of files and directorys (about 2000 user accounts). I > chaines> noticed that when I perform a copy of a large file ( I tested with > chaines> a 50 MB file) on a login server from something like /tmp to the > chaines> NFS mounted /home, the load average on the file server jumps up to > chaines> over 7, and seems to take forever (around 3-4 minutes). If I copy > chaines> a large file (once again I tested with 50 MB) from the NFS mounted > chaines> /home to like /tmp on the login server, there is no noticable load > chaines> average increase and it occurs in approximately 10-15 seconds. So > chaines> it seems that writes to the NFS mounted /home are taking longer > chaines> than they should and are spiking the load average. Both the file > chaines> server and the login servers are dual P3's with 2 gigs of ram. > chaines> Has anyone ever seen anything like this, or knows of any solutions > chaines> to this problem? > > Kernel version (client and server)? Version of nfs-utils? What does top > show as taking up CPU when you do the copy and your load average spikes? > > -Jeff > -- Chuck Haines chaines(a)gmail.com ------------------------------------------- Tau Kappa Epsilon Fraternity WPI Class of 2005 ------------------------------------------- AIM: CyberGrex YIM: CyberGrex_27 ICQ: 3707881 ------------------------------------------- -- Chuck Haines chaines(a)gmail.com ------------------------------------------- Tau Kappa Epsilon Fraternity WPI Class of 2005 ------------------------------------------- AIM: CyberGrex YIM: CyberGrex_27 ICQ: 3707881 -------------------------------------------

Sept. 9, 2004

Re: [Wlug] NFS Trouble

by Jeff Moyer

==> Regarding [Wlug] NFS Trouble; Chuck Haines <chaines(a)gmail.com> adds: chaines> I have an interesting problem that is occuring. I have a /home chaines> directory on a file server and I have several login servers that chaines> mount this /home directory via NFS. Now this /home directory has chaines> a ton of files and directorys (about 2000 user accounts). I chaines> noticed that when I perform a copy of a large file ( I tested with chaines> a 50 MB file) on a login server from something like /tmp to the chaines> NFS mounted /home, the load average on the file server jumps up to chaines> over 7, and seems to take forever (around 3-4 minutes). If I copy chaines> a large file (once again I tested with 50 MB) from the NFS mounted chaines> /home to like /tmp on the login server, there is no noticable load chaines> average increase and it occurs in approximately 10-15 seconds. So chaines> it seems that writes to the NFS mounted /home are taking longer chaines> than they should and are spiking the load average. Both the file chaines> server and the login servers are dual P3's with 2 gigs of ram. chaines> Has anyone ever seen anything like this, or knows of any solutions chaines> to this problem? Kernel version (client and server)? Version of nfs-utils? What does top show as taking up CPU when you do the copy and your load average spikes? -Jeff

Sept. 9, 2004

Re: [Wlug] NFS Trouble

by Chuck Haines

The curent line from the fstab is as follows: hostname:/home /home nfs rsize=4096,wsize=4096,bg,intr,rw,actimeo=15 0 0 Where hostname is replaced with the name of the machine I am mounting the /home from. Chuck On Thu, 09 Sep 2004 12:23:52 -0400, Dwight A. Ernest <dwight(a)significant.com> wrote: > With what options did you mount the volume on the client? > > > > Chuck Haines wrote: > > > I have an interesting problem that is occuring. I have a /home > > directory on a file server and I have several login servers that mount > > this /home directory via NFS. Now this /home directory has a ton of > > files and directorys (about 2000 user accounts). I noticed that when > > I perform a copy of a large file ( I tested with a 50 MB file) on a > > login server from something like /tmp to the NFS mounted /home, the > > load average on the file server jumps up to over 7, and seems to take > > forever (around 3-4 minutes). If I copy a large file (once again I > > tested with 50 MB) from the NFS mounted /home to like /tmp on the > > login server, there is no noticable load average increase and it > > occurs in approximately 10-15 seconds. So it seems that writes to the > > NFS mounted /home are taking longer than they should and are spiking > > the load average. Both the file server and the login servers are dual > > P3's with 2 gigs of ram. Has anyone ever seen anything like this, or > > knows of any solutions to this problem? > > > > Thanks, > > Chuck > > > > -- Chuck Haines chaines(a)gmail.com ------------------------------------------- Tau Kappa Epsilon Fraternity WPI Class of 2005 ------------------------------------------- AIM: CyberGrex YIM: CyberGrex_27 ICQ: 3707881 -------------------------------------------

Sept. 9, 2004

Re: [Wlug] NFS Trouble

by Dwight A. Ernest

With what options did you mount the volume on the client? Chuck Haines wrote: > I have an interesting problem that is occuring. I have a /home > directory on a file server and I have several login servers that mount > this /home directory via NFS. Now this /home directory has a ton of > files and directorys (about 2000 user accounts). I noticed that when > I perform a copy of a large file ( I tested with a 50 MB file) on a > login server from something like /tmp to the NFS mounted /home, the > load average on the file server jumps up to over 7, and seems to take > forever (around 3-4 minutes). If I copy a large file (once again I > tested with 50 MB) from the NFS mounted /home to like /tmp on the > login server, there is no noticable load average increase and it > occurs in approximately 10-15 seconds. So it seems that writes to the > NFS mounted /home are taking longer than they should and are spiking > the load average. Both the file server and the login servers are dual > P3's with 2 gigs of ram. Has anyone ever seen anything like this, or > knows of any solutions to this problem? > > Thanks, > Chuck >

Sept. 9, 2004

Re: [Wlug] NFS Trouble

by Mike Frysinger

On Thursday 09 September 2004 11:59 am, Chuck Haines wrote: > Has anyone ever seen anything like this, or > knows of any solutions to this problem? i havent seen anything like that before, but i have had random NFS issues ... i found using NFSv3 over TCP resolved pretty much all of the problems i experienced while using NFSv2 or NFSv3 over UDP you could also try tweaking the rsize and wsize parameters for the mount -mike

Sept. 9, 2004

NFS Trouble

by Chuck Haines

I have an interesting problem that is occuring. I have a /home directory on a file server and I have several login servers that mount this /home directory via NFS. Now this /home directory has a ton of files and directorys (about 2000 user accounts). I noticed that when I perform a copy of a large file ( I tested with a 50 MB file) on a login server from something like /tmp to the NFS mounted /home, the load average on the file server jumps up to over 7, and seems to take forever (around 3-4 minutes). If I copy a large file (once again I tested with 50 MB) from the NFS mounted /home to like /tmp on the login server, there is no noticable load average increase and it occurs in approximately 10-15 seconds. So it seems that writes to the NFS mounted /home are taking longer than they should and are spiking the load average. Both the file server and the login servers are dual P3's with 2 gigs of ram. Has anyone ever seen anything like this, or knows of any solutions to this problem? Thanks, Chuck -- Chuck Haines chaines(a)gmail.com ------------------------------------------- Tau Kappa Epsilon Fraternity WPI Class of 2005 ------------------------------------------- AIM: CyberGrex YIM: CyberGrex_27 ICQ: 3707881 -------------------------------------------

Sept. 9, 2004