HDInsight: Installation on Windows platform

Guide for starting with HDInsight on windows

Installing HDInsight on windows:

  • HDInsight installer is powered by Microsoft Web Platform Installer. To download it you can use the following link:
    http://www.microsoft.com/web/gallery/install.aspx?appid=HDINSIGHT-PREVIEW
  • After installing Microsoft WPI (Web Platform installer), run it with administrator privileges.
  • Search Hadoop in the search box. It will locate the HDInsight preview service installer for windows.

    Installing hadoop with WPI 1.1

    Installing hadoop with WPI 1.1

  • Click add to add installer to Microsoft WPI installation cart.

    Installing hadoop with WPI 2.1

    Installing hadoop with WPI 2.1

  • Accept the license and click install and wait for WPI to do the installation for you.

    Installing hadoop with WPI, Pic 2.2

    Installing hadoop with WPI, Pic 2.2

  • The installer also includes the hadoop package and IIS components.

Verifications of HDInsight Installation Success:

  • Four shortcuts will be created on the desktop, when the installer is finished.

    Installation Verification Pic3.1

    Installation Verification Pic3.1

  • Click on Hadoop Command Line link. Hadoop commands can be run in this command line.

    Hadoop Command Line Pic4.1

    Hadoop Command Line Pic4.1

  • Double Click on Microsoft HDInsight Dashboard.  This will open the dashboard for managing the Map reduce jobs, HIVE Queries , Hadoop commands, etc .

    HDInsight Dashboard Pic5.1

    HDInsight Dashboard Pic5.1

  • Goto Windows task manager, click on services and check if all Microsoft Hadoop’s Services are running.

    HDInsight Services in task manager, Pic6.1

    HDInsight Services in task manager, Pic6.1

Anomalies during setup we encountered:

While installing on our system we encountered some issues.

ISSUE1: Failed installation of IIS Components 

  • The IIS components failed to install while the hadoop services installed successfully. Tried reinstalling it and found only these IIS components missing.

    Failed IIS Components highlighted during re installing, Pic7.1.

    Failed IIS Components highlighted during re installing, Pic7.1.

FIX

  • By exploring IIS and its components, we found that they are tuned off by Windows features but they are present on the machine. This is why setup failed to install it. To fix this go to:
    Control Panel –> Turn Windows features on or off –> Internet Information Services
    Install all components from here(Check all required components) and try again with the same installation procedure. It will try to install hadoop but as all other hadoop services are already present, so it nwill will configure IIS components and exit with success.

ISSUE2: _Management_Engine_Components\DAL was unexpected at this time Error

When we installed HDInstall, we found all four Icons on desktop. But when we tried to run Hadoop Command Line, we found this:

_Management_Engine_Components\DAL was unexpected at this time

We were also unable to run any hadoop command as the command line kept saying- “invalid storage or unformatted storage”. When we tried to format Namenode, we were unsuccessful. We looked into the log file and checked the namenode service logs in:

  • /bin/namenode.err.log
  • /logs/name****

We found that namenode had not been formatted and it said invalid /nn directory when we tried to format it.

While debugging, we found that name had not started but was trying to start again and again. We checked the log /bin/namenode.err.log  and found that the error was  due to the bad formatting of the HDFS directory that path /nn is pointing to.

Fix:
The source of this issue was that the loader is unable to locate the binaries for running namenode. So we opened the Local-Drive\Hadoop\hadoop-1.1.0-SNAPSHOT\binhadoop.cmd in notepad and found that HADOOP_BIN_PATH was being determined dynamically and there was somthing happening wrong with this path. So, we edited it manually and replaced the set HADOOP_BIN_PATH script with the hard coded script that includes the local path to the bin.

set HADOOP_BIN_PATH=C:\hadoop\hadoop-1.1.0-SNAPSHOT\bin

We tried to run the hadoop NameNode from the services explorer and it started successfully. The dashboard also started with more options like interactive console, etc.

Now Hadoop is ready with Single node on Windows.

Leave a Reply

Your email address will not be published. Required fields are marked *


seven × = 56

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>