17 March 2011

install and integrate memcached in drupal

Hi Every one,
If performance is the issue for your Drupal site then one thing you are missing is memcache. This article explains you step by step process to install and memcache and how memcache helps to boost your drupal website.

What is memcache ?
Memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
Memcached's magic lies in its two-stage hash approach. It behaves as though it were a giant hash table, looking up key = value pairs. Give it a key, and set or get some arbitrary data.

When doing a memcached lookup, first the client hashes the key against the whole list of servers. Once it has chosen a server, the client then sends its request, and the server does an internal hash key lookup for the actual item data.

For example, if we have clients 1, 2, 3, and servers A, B, C:

Client 1 wants to set key "foo" with value "barbaz". Client 1 takes the full list of servers (A, B, C), hashes the key against them, then lets say ends up picking server B. Client 1 then directly connects to server B, and sets key "foo" with value "barbaz". Next, client 2 wants to get key "foo". Client 2 runs the same client library as client 1, and has the same server list (A, B, C). It is able to use the same hashing process to figure out key "foo" is on server B. It then directly requests key "foo" and gets back "barbaz".

Installing Memcache on Linux Machine :

Step 1: Run this command
~$> sudo aptitude install memcached
Step 2: Run this command
~$> sudo aptitude install php5-dev
Step 3: Installing PECL memcache

~$> cd /usr/local/src
~$> wget http://pecl.php.net/get/memcache-2.2.5.tgz
~$> tar zxvf memcache-2.2.5.tgz
~$> cd memcache-2.1.2
~$> phpize
~$> ./configure
~$># make && make install
Step 4: Open your php.ini file at /etc/php5/apache2/php.ini and add the below line
extension = memcache.so
Step 5: Restart Apache ~$>sudo /etc/init.d/apache2 restart
Step 6: If your memcache is installed properly you can see memcache configurations at http://localhost/info.php
Basically info.php is a file at /var/www folder with code <?php echo phpinfo(); ?>
Step7: The optimal number of daemons depends on your needs, but generally you want one for each cache table in your Drupal database because this makes clearing the cache on any of those tables less disruptive to the rest of your cache. So try to add these lines in memcached file in /etc/init.d/

memcached -u www-data -p 11211 -m 2 -d
memcached -u www-data -p 11212 -m 2 -d
memcached -u www-data -p 11213 -m 2 -d
memcached -u www-data -p 11214 -m 2 -d
memcached -u www-data -p 11215 -m 2 -d
memcached -u www-data -p 11216 -m 2 -d
Step 8: Restart memcache and try this command to verify the memcached daemons that are running.

~$>ps -A | grep memcached
23846 ?        00:00:00 memcached
23848 ?        00:00:00 memcached
23850 ?        00:00:00 memcached
23852 ?        00:00:00 memcached
23854 ?        00:00:00 memcached
23856 ?        00:00:00 memcached
Now memcache is installed and running on your system. Its time to integrate with drupal now.
Step 9: Download the drupal memcache module from here http://drupal.org/project/memcache .
Step 10: Open your php.ini file at /etc/php5/apache2/php.ini and add the below line
memcache.hash_strategy="consistent"
Step 11: Put your site offline.
Step 12: Edit your settings.php file in drupal /sites/default directory with below code.

$conf = array(
'cache_inc' => './sites/all/modules/contrib/memcache/memcache.db.inc',
'memcache_servers' => array(
  'localhost:11211' => 'default',
  'localhost:11212' => 'filter',
  'localhost:11213' => 'menu',
  'localhost:11214' => 'page',
  'localhost:11215' => 'form',
  'localhost:11216' => 'block',
),
'memcache_bins' => array(
  'cache' => 'default',
  'cache_filter' => 'filter',
  'cache_menu' => 'menu',
  'cache_page' => 'page',
  'cache_form' => 'form',
  'cache_block' => 'block',
  ),
);
Step 13: Enable the memcache module that you downloaded at step 9.
Step 14: Now if you enable memcache statistics at http://project_folder/admin/settings/memcache then you can see the statistics something like below if your memcache is properly installed and running.

* get:
 variables
   theme_registry:garland
   links:navigation:page-cid:admin/settings/memcache:1
   links:navigation:tree-data:3c28d675de5c9165717c8bc9be1c436c
   links:primary-links:page-cid:admin/settings/memcache:1
   links:primary-links:tree-data:5d6d3aaaaef5fba302ce62698fa37bbe
   links:secondary-links:page-cid:admin/settings/memcache:1
   links:secondary-links:tree-data:5d6d3aaaaef5fba302ce62698fa37bbe
* set:
  variables
    theme_registry:garland
* hit:
    links:navigation:page-cid:admin/settings/memcache:1
    links:navigation:tree-data:3c28d675de5c9165717c8bc9be1c436c
    links:primary-links:page-cid:admin/settings/memcache:1
    links:primary-links:tree-data:5d6d3aaaaef5fba302ce62698fa37bbe
    links:secondary-links:page-cid:admin/settings/memcache:1
    links:secondary-links:tree-data:5d6d3aaaaef5fba302ce62698fa37bbe
* bins:
    cache
    cache
    cache
    cache
    cache_menu
    cache_menu
    cache_menu
    cache_menu
    cache_menu
    cache_menu
Thats it you are done and installed memcache.!!
Cheers,



04 March 2011

schedule tasks on linux using crontab


If you've got a website that's heavy on your web server, you might want to run some processes like generating thumbnails or enriching data in the background. This way it can not interfere with the user interface. Linux has a great program for this called cron. It allows tasks to be automatically run in the background at regular intervals. You could also use it to automatically create backups, synchronize files, schedule updates, and much more. Welcome to the wonderful world of crontab.

Crontab

The crontab (cron derives from chronos, Greek for time; tab stands for table) command, found in Unix and Unix-like operating systems, is used to schedule commands to be executed periodically. To see what crontabs are currently running on your system, you can open a terminal and run:
sudo crontab -l

To edit the list of cronjobs you can run:

sudo crontab -e

This wil open a the default editor (could be vi or pico, if you want you can change the default editor) to let us manipulate the crontab. If you save and exit the editor, all your cronjobs are saved into crontab. Cronjobs are written in the following format:

* * * * * /bin/execute/this/script.sh

Scheduling explained

As you can see there are 5 stars. The stars represent different date parts in the following order:
  1. minute (from 0 to 59)
  2. hour (from 0 to 23)
  3. day of month (from 1 to 31)
  4. month (from 1 to 12)
  5. day of week (from 0 to 6) (0=Sunday)

Execute every minute

If you leave the star, or asterisk, it means every. Maybe that's a bit unclear. Let's use the the previous example again:

* * * * * /bin/execute/this/script.sh

They are all still asterisks! So this means execute
/bin/execute/this/script.sh:

  1. every minute
  2. of every hour
  3. of every day of the month
  4. of every month
  5. and every day in the week.

In short: This script is being executed every minute. Without exception.

Execute every Friday 1AM

So if we want to schedule the script to run at 1AM every Friday, we would need the following cronjob:

0 1 * * 5 /bin/execute/this/script.sh

Get it? The script is now being executed when the system clock hits:

  1. minute: 0
  2. of hour: 1
  3. of day of month: * (every day of month)
  4. of month: * (every month)
  5. and weekday: 5 (=Friday)

Execute on workdays 1AM

So if we want to schedule the script to Monday till Friday at 1 AM, we would need the following cronjob:

0 1 * * 1-5 /bin/execute/this/script.sh

Get it? The script is now being executed when the system clock hits:

  1. minute: 0
  2. of hour: 1
  3. of day of month: * (every day of month)
  4. of month: * (every month)
  5. and weekday: 1-5 (=Monday til Friday)

Execute 10 past after every hour on the 1st of every month

Here's another one, just for practicing

10 * 1 * * /bin/execute/this/script.sh

Fair enough, it takes some getting used to, but it offers great flexibility.

Neat scheduling tricks

What if you'd want to run something every 10 minutes? Well you could do this:

0,10,20,30,40,50 * * * * /bin/execute/this/script.sh

But crontab allows you to do this as well:

*/10 * * * * /bin/execute/this/script.sh

Which will do exactly the same. Can you do the the math? ;)

Special words

If you use the first (minute) field, you can also put in a keyword instead of a number:

@reboot     Run once, at startup
@yearly     Run once  a year     "0 0 1 1 *"
@annually   (same as  @yearly)
@monthly    Run once  a month    "0 0 1 * *"
@weekly     Run once  a week     "0 0 * * 0"
@daily      Run once  a day      "0 0 * * *"
@midnight   (same as  @daily)
@hourly     Run once  an hour    "0 * * * *

Leave the rest of the fields empty so this would be valid:

@daily /bin/execute/this/script.sh

Storing the crontab output

By default cron saves the output of /bin/execute/this/script.sh in the user's mailbox (root in this case). But it's prettier if the output is saved in a separate logfile. Here's how:

*/10 * * * * /bin/execute/this/script.sh 2>&1 >> /var/log/script_output.log

Explained

Linux can report on different levels. There's standard output (STDOUT) and standard errors (STDERR). STDOUT is marked 1, STDERR is marked 2. So the following statement tells Linux to store STDERR in STDOUT as well, creating one datastream for messages & errors:

2>&1

Now that we have 1 output stream, we can pour it into a file. Where > will overwrite the file, >> will append to the file. In this case we'd like to to append:

>> /var/log/script_output.log

Mailing the crontab output

By default cron saves the output in the user's mailbox (root in this case) on the local system. But you can also configure crontab to forward all output to a real email address by starting your crontab with the following line:

MAILTO="yourname@yourdomain.com"

Mailing the crontab output of just one cronjob


If you'd rather receive only one cronjob's output in your mail, make sure this
package is installed:

aptitude install mailx

And change the cronjob like this:

*/10 * * * * /bin/execute/this/script.sh 2>&1 | mail -s "Cronjob ouput" yourname@yourdomain.com

Trashing the crontab output


Now that's easy:

*/10 * * * * /bin/execute/this/script.sh 2>&1 > /dev/null

Just pipe all the output to the null device, also known as the black hole. On Unix-like operating systems, /dev/null is a special file that discards all data written to it.