[nSLUG] Load spikes, X bogs down. Why? Fix?

Dop Ganger nslug at fop.ns.ca
Sat Apr 9 09:09:37 ADT 2016


On Sat, 9 Apr 2016, Mike Spencer wrote:

> Something of that kind is probably right.  It's increasingly clear
> that the problem has to do with HD access.  With a music file
> playing (correctly, from HD file) with mplayer:
>
>     USBi  == Big file on ith mounted USB thumb drive
>     HDn   == Dir on hard drive partition n
>
>  cp USB1      -> HD3   X, music stutter; cp heads top(1) except some
>                        bdflush;   xload(1) spikes
>
>  cp HD3/file  -> HD4   Same problem
>
>  cp USB1      -> USB2  Music, X unaffected, cp heads top(1), no bdflush,
>                        smaller spike in xload(1), no problem

You may have excessive buffering going on - again, with current settings 
things can be a little excessive. /proc/sys/vm is the directory, files to 
tinker with are:

dirty_background_ratio: % of system memory when bdflush gets a prod to 
start writing out, default is 10, so on my system with 8 gig of RAM that's 
800 meg of data cached before it starts asking to write.

dirty_ratio: % of system memory when a commit is forced rather than just 
requested, default is 20, so on my system when this is hit I have 1.6 gig 
of data suddenly wanting to get written out asap

dirty_expire_centisecs: how long something can stay in write cache before 
it's forced out to disk, regardless of dirty ratio settings. Default is 30 
seconds.

dirty_writeback_centisecs: Another tuning knob that says how frequently 
the flush process gets called to check if there's anything to write 
(either dirty ratio filled or expire timeout hit).

A quick fix might be to set dirty_background_ratio to 1 and dirty_ratio to 
2 in order to minimise the amount of write behind going on. The other 
alternative is to prod bdflush into running more often but ramp up the 
cache to absorb all writes until they're ready, using values of 1 and 90 
respectively. Reducing dirty_expire_centisecs can also help get data 
written in a more timely fashion.

egrep 'dirty|writeback' /proc/vmstat will show you the current usage. 
Larding some syntactic sugar on, watch -n .5 $(echo egrep 
\'dirty\|writeback\' /proc/vmstat) will display it updated every half a 
second.

> But I don't have the /sys hierarchy you mention on this system, only
> /proc and I don't find anything there that resembles the
> /sys/devices/[yadda,yadda]/block/sda/queue/read_ahead_kb
> present on my much newer laptop.  Is there a similar place with an
> older kernel to achieve this?  Or a boot param?

I would imagine mount -t sys /sys should do it, possibly with mkdir /sys 
first if it's not there already.

> hdparm -a reports "readahead = 8  (on)" and the manpage says,
[snip]
> Which, if it has the same effect as your suggestion, also has the same
> value.

Yes, I just tested it and this is the same thing. Note that the 
nr_requests is the number of requests in flight which is something else. 
Reducing the number of requests in flight can help with latency caused by 
heavy disk use.

> But the hdparm manpage continues:
>
>       Also, many IDE drives also have a separate built-in read-ahead
>       function, which alleviates the need for a filesystem read-ahead
>       in many situations.
>
> so may there be something in the HD itself that somehow forces a
> larger readahead?

Possible but unlikely. Linux settings are tuned for a range of different 
scenarios, from a server down to a phone; the settings are often just a 
reasonable default for most people but will need tuning for the local 
system. The read-ahead in the hard drive will be tuned based on the drive 
size so is more likely to be an appropriate value.

> --- Begin vmstat output, start before cp, end after cp done ---
>
> bogus% vmstat 1
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
> 1  0      0  13332   4708 383860    0    0    78    38  142   92  1  2 98  0
> 0  0      0  13328   4708 383860    0    0     0     0  105   49  0  1 99  0
> 0  0      0  13328   4708 383860    0    0     0     0  157  306  0  0 100  0
> 0  0      0  13328   4708 383860    0    0     0     0  101   33  1  0 99  0
> 0  2      0  11928   4712 385188    0    0   748     0  300  422  1  1 98  0
> 1  1      0   5332   4720 391616    0    0 18825     0 6094 11956  1 27 72  0
> 1  1      0   5232   4748 388540    0    0 18441     0 6031 11765  1 25 74  0
> 1  1      0   4708   4772 385860    0    0 18697     0 6068 11829  0 27 73  0
> 1  1      0   5180   4800 382160    0    0 18697     0 6111 11897  0 24 76  0
> 2  0      0   4472   4828 379628    0    0 18826     0 6064 11825  1 28 71  0
> 1  1      0   4836   4860 376092    0    0 18445     0 6031 11774  1 31 68  0
> 5  2      0   5336   4860 375340    0    0  1792  6764 1104 1198  4 91  5  0
> 3  0      0   4512   4892 372812    0    0 19082 32145 12148 12368 14 79  6  0
> 2  3      0   5344   4916 368936    0    0 17672  4160 5801 11212  4 29 67  0
> 0  2      0   5000   4936 366136    0    0 19082 34752 12337 12278 19 75  6  0

Too much data is being written to disk and processes are blocking.

> 1  1      0   4888   4964 366264    0    0 18569     0 6070 11886  1 22 77  0
> 6  1      0   5320   4964 365804    0    0   384  7232  778  248 16 83  1  0
> 3  0      0   4604   4992 366512    0    0 18477 30396 11608 12055 15 79  6  0
> 1  1      0   4484   5020 366572    0    0 18789     0 6102 11958  1 29 70  0
> 5  1      0   5264   5020 365828    0    0   128  7040  626  110 18 82  0  0
> 3  0      0   4464   5048 366588    0    0 18829 32000 11822 12137 15 79  6  0

Now there's contention between read and write processes - note how reads 
and writes are alternating between dropping to near zero.

> 2  1      0   4432   5724 365964    0    0 18697 34624 12156 12125 19 74  6  0
> 0  0      0   4920   5740 365516    0    0  9175     0 3090 5949  0 14 86  0
> 1  0      0   4900   5740 365516    0    0     0     0  101   27  1  0 99  0
> 1  1      0   4900   5740 365516    0    0     0  6696  458   27 11 27 62  0
> 3  0      0   4900   5740 365516    0    0     0 27200 4352  203 29 71  0  0
> 0  0      0   4900   5740 365516    0    0     0  1560  846   74 25 35 40  0
> 0  0      0   4900   5740 365516    0    0     0     0  101   21  0  0 100  0

And now the copy is finished and data is flushed out over time.

Something that may help us using ionice and schedtool to set priorities; 
set the copy to run low priority, eg:

(ionice -c 3 schedtool -D -n 10 -e tar -cf - ./totallylegitfile.mp4) | (cd 
/home/media && ionice -c 3 schedtool -D -n 10 -e tar -xvf - )

There's also some options for mplayer you can use; for example on my 4 
core machine I tell mplayer to use 4 threads, 128 meg of cache, pre-fille 
the cache to 50% before starting to play, and drop frames on overruns:

mplayer --lavdopts=threads=4 -cache 131072 -cache-min 50 --framedrop 
totallylegitfile.mp4

Cheers... Dop.


More information about the nSLUG mailing list