Guide for starting with HDInsight on windows
Installing HDInsight on windows:
- HDInsight installer is powered by Microsoft Web Platform Installer. To download it you can use the following link:
- After installing Microsoft WPI (Web Platform installer), run it with administrator privileges.
- Search Hadoop in the search box. It will locate the HDInsight preview service installer for windows.
- Click add to add installer to Microsoft WPI installation cart.
- Accept the license and click install and wait for WPI to do the installation for you.
- The installer also includes the hadoop package and IIS components.
Verifications of HDInsight Installation Success:
- Four shortcuts will be created on the desktop, when the installer is finished.
- Click on Hadoop Command Line link. Hadoop commands can be run in this command line.
- Double Click on Microsoft HDInsight Dashboard. This will open the dashboard for managing the Map reduce jobs, HIVE Queries , Hadoop commands, etc .
- Goto Windows task manager, click on services and check if all Microsoft Hadoop’s Services are running.
Anomalies during setup we encountered:
While installing on our system we encountered some issues.
ISSUE1: Failed installation of IIS Components
- The IIS components failed to install while the hadoop services installed successfully. Tried reinstalling it and found only these IIS components missing.
- By exploring IIS and its components, we found that they are tuned off by Windows features but they are present on the machine. This is why setup failed to install it. To fix this go to:
Control Panel –> Turn Windows features on or off –> Internet Information Services
Install all components from here(Check all required components) and try again with the same installation procedure. It will try to install hadoop but as all other hadoop services are already present, so it nwill will configure IIS components and exit with success.
ISSUE2: _Management_Engine_Components\DAL was unexpected at this time Error
When we installed HDInstall, we found all four Icons on desktop. But when we tried to run Hadoop Command Line, we found this:
_Management_Engine_Components\DAL was unexpected at this time
We were also unable to run any hadoop command as the command line kept saying- “invalid storage or unformatted storage”. When we tried to format Namenode, we were unsuccessful. We looked into the log file and checked the namenode service logs in:
We found that namenode had not been formatted and it said invalid /nn directory when we tried to format it.
While debugging, we found that name had not started but was trying to start again and again. We checked the log /bin/namenode.err.log and found that the error was due to the bad formatting of the HDFS directory that path /nn is pointing to.
The source of this issue was that the loader is unable to locate the binaries for running namenode. So we opened the Local-Drive\Hadoop\hadoop-1.1.0-SNAPSHOT\binhadoop.cmd in notepad and found that HADOOP_BIN_PATH was being determined dynamically and there was somthing happening wrong with this path. So, we edited it manually and replaced the set HADOOP_BIN_PATH script with the hard coded script that includes the local path to the bin.
We tried to run the hadoop NameNode from the services explorer and it started successfully. The dashboard also started with more options like interactive console, etc.
Now Hadoop is ready with Single node on Windows.