Xerosphere

Its authors describe it best: "Suhosin is an advanced protection system for PHP installations."

What the Suhosin patch does is it sets some limits for particular resource usages, prevents some possible buffer overflows, protects against some common vulnerabilities, and so on and so forth.  Full feature list is here.  For the most part, it prevents others (and you!) from being able to do bad things on and to your server.

If you install your LAMP stack from the standard Debian repositories, guess what -- you have Suhosin installed.  A quick peek at your phpinfo() will tell you whether or not you are running it.  This may create a problem which may be difficult to detect and difficult to solve, and here's why:

• The default configuration for suhosin is fairly restrictive on a couple of fronts.  It limits the number of POST variables allowed per request, and some of the more burly Drupal administration forms may exceed those limits.
• When those limits are exceeded, Suhosin's default behavior is to log its errors to syslog, not the Apache error log.
• Also, when the limit for maximum post values is exceeded, the variable list is truncated at the max number of variables and just gets passed along to PHP, incomplete as it may be.  This could potentially cause all manner of unpredictable results.
• Removing Suhosin from your PHP install is a non-trivial task which involves rebuilding PHP from source packages.
• If you are not a real nerdy PHP power user or Linux administrator, you are accustomed to changing PHP settings only in php.ini, (or perhaps in Drupal's .htaccess file).  Suhosin has it's own separate configuration file (as it should, to be sure) which may not be in an obvious location.

The first three items on this list added up to me being stumped. In my case, I was working on a site which had a whole lot of blocks defined.  The (very long) block admin page would not save my changes -- I'd assign a block to a new region, hit save, and the page would return as if all was fine, except the change was not saved.  No error in Drupal's watchdog log.  Checking the Apache error log revealed no errors.  I fiddled with my Apache log level to see if there was anything worth noting, and there was not.  Turned on devel to check memory usage; plenty of memory to spare.  It would appear that saving the page was just silently failing.

After barking up a few more wrong trees I took a glance at my server's system log (syslog).  Nothing jumped out.  I watched it in real time while saving the block page.  Aha!  Suhosin was spitting out a couple errors.  At that point, I had really only ever heard of Suhosin, and had a general idea of what it did (again, Suhosin is installed by default when you install PHP using apt on Debian)

So.  What was happening was that Suhosin imposed a limit on the maximum number of POST variables that PHP will accept on a given request.  The default is 200.  This page with its giant form had well over a couple hundred POST vars.

Suhosin's configuration file (on Debian, anyway) is /etc/php5/apache2/conf.d/suhosin.ini

Ultimate solution was to adjust the following values:

   suhosin.post.max_vars = 1000
  suhosin.request.max_vars = 1000

References:

• The Hardened PHP Project
• Instructions on removing Suhosin patch from PHP (Chris Blunt)
• Drupal security: Using Suhosin to secure PHP installations (Dynamite Heads) - Drupal-friendly suhosin.ini