Linux overcommit of memory

VM with limited memory fell over today, so I had to learn a bit about linux memory allocation. Notes for future reference:

The Linux kernel generally responds to memory requests positively. By default, it uses a heuristic algorithm that will deny blatantly unserviceable requests, but allow a degree of overcommitment on the grounds that most processes ask for more than they need most of the time. This works reasonably well usually but if you’re overallocating there’s an inevitable risk of an out of memory error. When this happens, the oom-killer is invoked and will gleefully traipse through your system slaughtering processes according to its own arcane whims*. This can properly mess up your system, so once it’s been invoked you probably want to consider a full reboot in case it’s knocked out any fundamental processes (hopefully once you’ve investigated and fixed whatever is causing you to run out of memory).

In some situations it might be better to be more strict about over-commitment of memory in the first place. The parameter vm.overcommit_memory controls how the kernel responds to requests for memory and takes one of three values:

0: The default. Use the heuristic thing. Allocate memory unless a process is obviously taking the piss.

1: Just hand out memory, don’t worry about overcommitment (so big risk of oom, but potentially better performance for memory intensive tasks)

2: Don’t hand out memory if the request is bigger than the total available swap + vm.overcommit_ratio% RAM

The default vm.overcommit_ratio setting is 50%.

* It uses some heuristic to determine which processes would be least damaging to kill off, but it can still take out vital stuff.

Advertisements

dhcp, resolv.conf and dhclient

I have a (RHEL 6) VM that needs a DHCP IP. I’ve added search options into resolv.conf manually but every time dhcp renews its lease, these are overwritten. I also want it to request a specific IP address. Turns out dhclient config files are handy:

1. Don’t use NetworkManager:

chkconfig NetworkManager off
service NetworkManager stop

Set NM_CONTROLLED=no in ifcfg-eth0

2. Edit (create if it doesn’t exist) the file: /etc/dhcp/dhclient-eth0.conf (or whatever interface you’re configuring) to contain something like:

send dhcp-requested-address XXX.XXX.XXX.XXX;
prepend domain-search "foo.bar.com","bar.com";

3. Test new config

ifdown eth0
ifup eth0

dhclient.conf and dhcp-options man pages for more.

Computational Biology Primers

Useful list of Nature Biotech and PLOS Comp Bio primer articles: http://www.liacs.nl/~hoogeboo/mcb/nature_primer.html

genome co-ords

Because I keep forgetting these and having to check:

Ensembl co-ords are 1-based, fully closed: The range [1,5] means positions 1,2,3,4,5.

UCSC data co-ords are 0-based, half-open: The range [0,5) means positions 1,2,3,4,5

Confusingly, their genome browser uses 1-based, fully closed (ie they add 1 to the internally stored start (but not end) position of any interval before displaying it in the browser).

Also see this useful biostar cheatsheet: http://www.biostars.org/p/84686/

Lua compile errors

Attempting to install Lua on the cluster (Centos 5).
make linux gives various “libreadline.so: undefined reference” errors.
Just add -lncurses to the “linux:” line in src/Makefile

WP comprehensive google maps plugin jquery version error

I just updated wordpress and got an unlikely sounding error about my jquery being too old for the comprehensive google maps plugin. It isn’t, it’s just the test in the plugin doesn’t work properly: http://wordpress.org/support/topic/jquerey-error-after-wordpress-36-update

thing.

ls training/ | sort -R | head -n45 | xargs -I file mv training/file test/