Apparently random performance issue processing large data sets with Perl
Hi, We have an HP rack server dl380 G7 with dual x5650 Xeon processors, 48GB of RAM, a RAID 1(2-146GB SAS disks)/5(6-500GB SAS disks) setup of eight physical drives with Windows 2008 R2 Enterprise installed on RAID 1 and data that is being processed stored on RAID 5. We are using Perl scripts to do processing of a lot of text. We are using ActiveState ActivePerl v5.10.1 x64. For some reason, occasionally, the processing takes a lot longer than other times. The data is always about the same size, growing slowly over time, so it is not like there is suddenly more data to process. The whole process should take around 13 hours, but i saw it go for as long as two or three days. When the slowdown happened, I watched the resource monitor disk report, and it had the RAID5 drive at almost 100% activity with the queue length of around 1.0 and very slow reads and writes of around 1MB/second. When other disk operations happen such as copying of a lot of data, the disk IO is at around 200MB/second and the queue is at up to 6, so this seems to be normal. I also see the memory usage, and i see that most of the memory is used for standby file cache by the system with very little free memory. I understand that the standby cache gets released to allocate memory for use by active processes, but I think that this may be affecting the script performance because the script works on very large size text files and these are probably getting cached by the system. Is there a way to adjust/limit how much memory is used for standby file cache? I found a hotfix (KB976700) that seems to address my problem, but it didn't seem to help, as I still see occasional very slow processing. I am not sure what is going on, because this happens occasionally and randomly. Any help will be greatly appreciated. Regards, Tim
February 23rd, 2011 5:15pm

Here are some performance articles you should look at, most are written for 2003 32bit, but still apply, just subsitute the 2008 R2/64bit values/limitations as needed. First verify how much memory is being assigned to the VAS of the process, then detemine whats in physical memory vs page file for this process. In gerneal Workingset = physical memory, the rest is paged memory. http://networkadminkb.com/kb/Knowledge%20Base/Windows2003/How%20to%20view%20the%20Virtual%20Address%20Space%20currently%20in%20use%20by%20a%20process.aspx Verify, the Disk IO is not overly burdened. Raid access time for the LUN should be very low (.030 or less consistently), if your script mostly reads from disk focus on reads, if it writes, then focus on write. http://networkadminkb.com/Shared%20Documents/Disk%20IO%20Tutorial%20Part%201.aspx Monitoring file system cache http://technet.microsoft.com/en-us/library/cc757398(WS.10).aspx Setup perfmon logs to monitor daily and try and see if there are significant differences between the good and the bad.
Free Windows Admin Tool Kit Click here and download it now
February 23rd, 2011 11:34pm

Here are some performance articles you should look at, most are written for 2003 32bit, but still apply, just subsitute the 2008 R2/64bit values/limitations as needed. First verify how much memory is being assigned to the VAS of the process, then detemine whats in physical memory vs page file for this process. In gerneal Workingset = physical memory, the rest is paged memory. http://networkadminkb.com/kb/Knowledge%20Base/Windows2003/How%20to%20view%20the%20Virtual%20Address%20Space%20currently%20in%20use%20by%20a%20process.aspx Verify, the Disk IO is not overly burdened. Raid access time for the LUN should be very low (.030 or less consistently), if your script mostly reads from disk focus on reads, if it writes, then focus on write. http://networkadminkb.com/Shared%20Documents/Disk%20IO%20Tutorial%20Part%201.aspx Monitoring file system cache http://technet.microsoft.com/en-us/library/cc757398(WS.10).aspx Setup perfmon logs to monitor daily and try and see if there are significant differences between the good and the bad. Review standard performance settings http://networkadminkb.com/kb/Knowledge%20Base/Windows2003/How%20to%20performance%20tune%20a%20Windows%202003%20Server.aspx
February 23rd, 2011 11:34pm

any updates?
Free Windows Admin Tool Kit Click here and download it now
March 4th, 2011 10:33am

Hi, Yes, thanks. I tried doing monitoring, but I didn't find anything particularly interesting in the logs in regards to the slowdown. It did help me improve my script performance overall, though, so that was helpful. I found elsewhere that in Windows Serve 2008 R2 (and maybe other versions as well) scheduled tasks have a lower prioirity then normal for CPU, IO, and Memory. I found out how to set these priorities to a normal level, and that seemed to have helped though I am not yet 100% sure of this helping. Thanks for your help. Tim
March 9th, 2011 6:43pm

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics