Installing ElasticSearch on mac os x or linux

These instructions are specifically for Mac Os X , but they should work fine on linux too.

Download the latest version of Elastic search from here: http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.5.tar.gz The current version is 0.18.5 but a later version should be ok

Unpack the tar file in the download directory (doubleclick on the tar file) this should create the directory ~/Downloads/elasticsearch-0.18.5

open a console in the download directory and pull down the elastic search service wrapper from github.

    $cd ~/Downloads
    $git clone https://github.com/elasticsearch/elasticsearch-servicewrapper.git elasticsearch-servicewrapper

Now we will move the service into place

    $cd ~/Downloads
    $sudo mv elasticsearch-0.18.5 /usr/local/share
    $cd elasticsearch-servicewrapper
    $sudo mv service /usr/local/share/elasticsearch-0.18.5/bin
    $cd /usr/local/share
    $sudo ln -s elasticsearch-0.18.5 elasticsearch
    $sudo chown -R root:wheel elasticsearch

Now we can start up the service

 
    $cd /usr/local/share/elasticsearch
    $sudo bin/service/elasticsearch start 

You should see the following output

   $ sudo bin/service/elasticsearch start
    Starting ElasticSearch. Detected Mac OSX and installed launchd daemon.
    Waiting for ElasticSearch...
    .
    .
    .
    .
    .
    .
    running: PID:57767
    $

Note: The pid number will be different on your system

Check the service is ok by opening up a browser on http://localhost:9200, you should see the following output

    {
      "ok" : true,
      "name" : "Bova",
      "version" : {
        "number" : "0.18.5",
        "date" : "2011-03-01T19:07:42",
        "snapshot_build" : false
      },
      "tagline" : "You Know, for Search",
      "cover" : "DON'T PANIC",
      "quote" : {
        "book" : "The Hitchhiker's Guide to the Galaxy",
        "chapter" : "Chapter 30",
        "text1" : "\"The chances of finding out what's really going on in the universe are so remote, the only thing to do is hang the sense of it and keep yourself occupied. Look at me, I design fjords. I'd far rather be happy than right any day.\"",
        "text2" : "\"And are you?\"",
        "text3" : "\"No, that's where it all falls apart I'm afraid.\"",
        "text4" : "\"Pity, it sounded like quite a nice lifestyle otherwise.\""
      }
    } 

Stopping Elastic search

use a similar command to starting as shown above.

 
    $cd /usr/local/share/elasticsearch
    $sudo bin/service/elasticsearch stop

Zend Framework, Dojo and MVC layout

Recently I had a real nasty problem which had me scratching my head for several days, I have recently started using the Zend Framework and Dojo framework together to allow me to create forms which have rich controls in them, such as calendar controls etc.

Initially all worked well, setting up a simple ZF application, Using Zend_Dojo to "Dojo" enable my forms and views all went without a hitch. For each control i instantiated in my forms the Zend_Dojo_View_Container attached to the view would automatically create all the required script and library calls needed to make sure the right components where integrated with the page. Everything was fantastic. Then I tried to apply the same approach to my actual application.

The difference between my test rig and my Application is that in most of the views in my application we deliberately disable the MVC layout, and render the pages using the Zend_Controller_Action::renderScript() method. The reason for this is that our view scripts are pre-assembled with all components preprocessed into single files per page, including all the layout elements.

Typically we use the following in the Controller baseclass preDispatch method

        $layout = Zend_Layout::getMvcInstance();
        if($layout) {
            $layout->disableLayout();
        }
        $this->_helper->viewRenderer->setNoRender();

Generally my Controller actions look like this.

public function registerAction() {
        /*
         * Register for new account
         */
        $register = new Form_User_Registration();
        if (Zend_Auth::getInstance()->hasIdentity()) {
            $this->_redirect('/');
        } else if ($this->getRequest()->isPost()) {
            // do the registration action here 
            //    redirect when done 
        }

        $this->view->register = $register;
 
        echo $this->renderScript('pages/user/register.phtml');
    }

However doing things this way, I discovered that none of the form dijit specifications had been injected into the Zend_Dojo_View_Container, so none of my controls rendered as Dojo rich controls.

The reason why this was happening became clear after a lot of debugging. In the conventional MVC mode, the "contents" of the page are rendered before the layout scripts are run. The form is rendered into the page contents and then the layout script is executed to create the page around the contents. However in my case the contents where executing in the same script as the layout, so the Dojo container was being output at the top of the script before the contents where rendered, hence the missing specifications.

The solution was simple, but non-obvious. change the assignment of the form to the view from

        $this->view->register = $register;

to:

        $this->view->register = (string)$register;

Which forced the __ToString() method of the Zend_Dojo_Form class to be called before the Controller renderScript() function executed, causing the dijit specs to be registered first.

This fixed the problem, and finally all my forms where being rendered correctly with Dojo Rich controls..

Messing around with SEO engines

Checkout a new SEO engine at http://www.gold2cashxchange.co.uk . This is an automated portal system we are experimenting with.

Java: Bad CPU id in executable

Here is a weird one:

Recently i have been doing a lot of java work, and many of the packages I use need to have Java_HOME set. So normally
i just add this line to my /etc/profile

JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home; export JAVA_HOME

Works just fine on my Macbook, and my recent 2009 Mac Mini. However i have an older 2006 Intel Mac Mini, the older Core Duo type, rather than the newer Core 2 Duo versions. Its been out of action for a few months due to a failed Hard Drive, but recently I revived it by buying a suitable hard drive from PCWorld and attacking the machine with a putty knife. (upgrading the ram whilst i was in there).

Everything seemed to be working fine, re-installed the OS, and I took the opportunity whilst i had a virgin system to document and automate the installation of my EC2 management scripts. Everything swam along fine until I actually tried to use the EC2 management scripts where upon I got the dreaded message "java: Bad CPU Id in executable" WTF....

Googling the error did not seem to help, so back to the drawing board, and take a microscope to the machine.

I checked the target directories, they are fine, in the "versions" directory under the java framework I can see 1.42, 1.50 and 1.60. so what is happening.

The first clue is that the java preferences applet does not list the 1.6.0 version as available.... Hmm why not????

The second clue is that switching to version 1.5.0 in the JAVA_HOME statement makes it all work.

The answer is that Apple never released a 32bit version of Java 1.6, but to rub salt into the wounds they load a perfectly non-functional 64bit version onto all 32bit only boxes. Ouch...

Come on Apple, there is no reason to limit java 1.6 to 64bit only, and even if there is, why on earth load a nonfunctional 64bit framework onto 32bit machines?

Shakes head in amazement.......

EBS, Flash drives for EC2 machines

Recently i had a bizarre situation where i had been running a load of stuff through Hadoop on one of our EC2 clusters, and the job had failed, but it failed in such a way that if i could save away the data, (and the logs for diagnostic purposes), and restart the cluster with some changed parameters i would be able to recover the data and carry on, having invested about $60 in machine time and hours of head scratching already on the failed run, I thought this was a good idea.

So no problem just use hadoop's distcp to move the data up to s3n://bucketname, hmmm that did not work out: OK, how about hadoop dfs -cp to a local directory of all the data and use s3cmd to move it to s3 storage, hmm that did not work out either, something odd is going on here.

Then it dawned on me that some of the files in the run where very large, as where the intermediate products, around 10G on average and there where lots of them. Now s3 has a limit of 2G for objects stored in its file system. I did not want to use the hadoop s3:// non native filesystem as that is difficult to verify if all the data had arrived safely, as nothing else would read it, so i had a brainwave, and successively mounted and unmounted an EBS volume onto each machine , saved away the relevant bits and pieces, and then moved onto the next box. All of our base EC2 images have xfs support wired into the system, as well as /mnt/ebs as a mountpoint for an attached ebs volume. So it was simply a case of using ElasticFox to attach the EBS volume to the instance and the issuing a "mount/mnt/ebs" and then "umount /mnt/ebs" once i was done.

So effectively using an EBS volume as a virtual flash-drive.

Fixing External monitors on the Aspire One and Fedora 10

I was recently forced to spend some time using my Aspire One as a primary development machine, having left my macbook PSU at work, and being too lazy to go all the way back in to retrieve it. 

So I thought, thats ok, I have this old monitor kicking around here, and a spare keyboard and mouse, just bang them into my Aspire One and off we go. But.......

Fedora 10 uses the new configless Xserver and the new Intel GEM enabled driver, a configless Xserver will happily run X without all the tedious fiddling with /etc/X11/xorg.conf. In fact if you take a peek in that directory you will see that there is no xorg.conf. 

So when I plugged my external monitor in, I was encouraged to see it display everything up to the login screen, logged in and was presented with a tiny 640x480 desktop on both monitors. 

Ok, no problem, pop up the Display properties dialog, set the sizes of the screens back to thier proper setup, 1024x600 for the builtin LCD, 1024x768 for the ext monitor. Press apply and ..... and .... and .... Nothing happened...  WHAT!!!!! 

OK it turns out that there is one vital piece of configuration that is missing in a configless Xserver boot, and thats the Virtual screen size.  You need a Virtual screen size that is bigger than the sum of both your monitor surfaces to run dual head (internal + external monitor). 

I tried everything, creating a minimal xorg.conf that just added that line, using sample xorg.confs for closely speced machines, nothing worked. And those that did, resulted in the disabling of DRI which produced syrupy screens on the external monitor. 

So the solution is to create an xorg.conf that is exactly what the autoconfig would produce, plus the extra line. 

So here are the steps

1. Reboot the Aspire and press RETURN when the initial bootloader screen comes up. 

2. press "a" and add a space then "3" to the end of the kernel boot line, press RETURN to boot the machine. 

3. The machine will boot to a command line instead of the usual graphical login screen. 

4. Login as root

5. Run the following

$cd /root
$Xorg -configure

The system will flash the screen and produce a file called xorg.conf.new in the current directory. This file contains all of the details that the configless startup would use.

$cp xorg.conf.new  /etc/X11/xorg.conf

Now locate the section at the end of that file that shows as below, and add the line indicated in

SubSection "Display"
   Viewport        0 0
   Depth          24
   Virtual        2048 2048           # Add this line in 
EndSubSection

6. Reboot the machine and you will now be able to reconfigure the monitors to support 1024x600 internal and 1024x768 external

Note: Because of limitations in the driver, if you use a value of more than 2048 in either dimension, it will disable DRI and the screen will become very slow again. This unfortunately limits the external display to 1024 width. If you want it to go wider then turn off the internal LCD panel in the Display Properties. This is being fixed in the next version of the driver, and this whole proceedure may become redundant by then as it will also support configless virtual spaces. 

Comparing Ubuntu 8.10 (Intrepid Ibix) and Fedora 10 on the Aspire One

I have been trying out several linux distributions on my Aspire One to find one that suites the machine best, I have tried everything, including a copy of PCBSD that was well "less than successful".

The two top runners are currently Fedora 10 (beta) and Ubuntu 8.10 (beta).

First a warning, Beta software is not for everybody, you can end up with a dead machine if you hit an issue with an update, and have to know how to recover your machine when that happens. You also have to have a good set of backups.

There is a superb tool called remastersys that creates boot able backups of not just your data, but your entire operating system, which I use in conjunction with an external FAT formatted USB drive.

One thing I have found is that trying to find a distro that supports the very recent hardware and chipsets found inside the Aspire One is hard, and I have narrowed my evaluations to those distros shipping the brand new 2.6.27 kernel or later, as that is the only one that seems to support the chipsets out of the box. I also need to have networkmanager 0.7, to support my Mobile USB broadband modem (a Huawei 169G).

I was using Ubuntu at first but needed to change to a RPM/YUM based system due to us using Centos 5 everywhere at work, and since I am building code that needs to be run and installed in that enviroment, the differences in the package managers was just too great for me to be comfortable with. (Yes I write software on my Aspire netbook, its quite capable of it, I was very surprised at how well Eclipse performs on this platform, almost as fast as my 2008 macbook).

Basics:

Installation of both distros was relatively easy, I used unetbootin to create a bootable 1G USB thumbdrive directly from the distributed ISO's and booted from that, I did not notice any issues with installation of either distro. Out of the box Fedora was slightly better in this respect, with a few caveats. Choose the gnome distros for each, as I have found that the newer KDE setups are somewhat less functional, in particular current KDE incarnations (4.1+) seem to have issues with saving settings.

Video:

Ubunto produced the more complete setup here, but only after I deleted the /etc/X11/xorg.conf file and allowed the new xorg 7.4 system to work its magic with configless boots.

glxgears turned in a performance of about 350 fps, which is fast enough to enable compiz desktop effects, however since that is just eye-candy its debatable whether it is worth enabling it on a device of this class. A minor irritation was that in order to support multiple screens, the screen resolution app has to create an xorg.conf with a virtual screen that encompasses both physical screens, and it initially gets it wrong which means that the external monitor resolution is low until you hand edit it to up the required size, I just doubled the dimensions in both directions and rebooted and was then able to select the 1280x1024 res I was looking for on my monitor.

Fedora was a mixed bag, it again needed the delete my xorg.conf trick, but try as I might I could not get it to properly support an external monitor at a reasonable resolution. However the performance of the driver in glxgears is significantly better, getting 550-600 fps. Im still trying to determine the reasons for this performance difference.

Wireless:

Due to the inclusion of networkmanager 0.7 in both distros, wireless was a doddle, both wifi and usb modem worked out of the box, however both distros suffered from the same issues re DHCP and wifi performance.

Every now and then they refused to acquire either a wired or wireless ip address due to dhcp timeouts, rebooting the machine seemed to clear the problem. The wifi is using the new ath5k driver for the Atheros chipset in both cases, and I have found that this driver seems to effect the sensitivity of of the wifi, with far lower signal strengths than under the older madwifi driver, and frequent dropouts and stalls.

Also under the madwifi driver there where a set of sysctls that would enable the wifi led which dont work on the ath5k driver, and I have not found any substitutes. The driver binds to the led_class module, and looking at the source has functions for enabling/disabling this mode, but I cant find any documentation on how to enable it.

laptop use:

Ubuntu has working suspend and resume, the sound sometimes does not restore properly coming out of sleep of hibernate but that is a minor annoyance, on Fedora both modes where a bust, resulting in a locked up machine requiring a hard reset.

Software:

I must admit my needs are probably different from average, I need to enable a full local LAMP stack and software development tools (yes it runs fine, and no the machine is not slow after doing so). The Aspire has dual 1.6Ghz cores, up to 1.5G of ram and a 120G HDD so its quite capable of handling this load. It should be noted that the spec of the machine is amost identical to the perfomance of an instance running on the Amazon EC2 cluster (1.6Ghz, 1.6G Ram, 160G HDD).

With Ubuntu setting up the stack was hard, on fedora this was a Breeze, Fedora even has a full Eclipse 3.4.x install in the repository, and has installable packages for eclipse PDT, subclipse and Xdebug. So my usual fight to get a working PHP dev enviroment working was eliminated.

I was able to setup up the entire machine for running our web app, including checking out the code from our subversion repository in under 30 mins, vs the 3-4 hour battle that I had with ubuntu. The Fedora packages even setup the correct SVN provider interfaces for subclipse which really impressed me.

I had some trouble setting up netbeans on Fedora, but mainly because it could not find the JRE directory, once that was sorted out, it installed fine. I tend to use Eclipse for PHP development, and Netbeans for C++ development as I have never really got on with the Eclipse CDT.

Overall:

Ubuntu is defiantly the more polished distro for general use, my specialised needs tend to lean me towards Fedora where im willing to put up with the shortcomings, I also like the faster more responsive feel to the fedora distro.

One final tip, if you are playing with beta software and hit issues, then engage with the community around the distro, they are normaly very responsive, and make sure that any problems you find are submitted as bug tickets, or they will never get fixed. Dont just sit back and wait for somebody else to report the problems...