`Install openssh and if available cmake , make

cmake install

install make

After installation run cygwin as Admin important other wise it won't work

Arun@Arun-PC:ssh-host-config

click yes to all prompt questions and enter

Arun@Arun-PC:net start sshd

check whether 22 port is in use by below command

Arun@Arun-PC:net stat -a|grep 22

connect to putty

UserName Arun and passwd is case sensitive enter username same case as enter in the user creation

From Apache Hadoop 2.2.0 release hadoop officially supports for running Hadoop on Microsoft Windows as well. But the bin distribution ofApache Hadoop 2.2.0 release does not contain some windows native components (like winutils.exe, hadoop.dll etc). As a result, if we try to run Hadoop in windows, we'll encounter ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path.

To fix this follow the steps:

STEP 1: INSTALL TOOLS TO RUN HADOOP ON WINDOWS

Install cygwin and during installation search openssh and check openssh checkbox and install.

once sshd is started port 22 is open and can be used by hadoop namenode for communicating with the hdfs.

Secure Shell, or SSH, is a cryptographic (encrypted) network protocol to allow remote login and other network services to operate securely over an unsecured network.[1]

SSH provides a secure channel over an unsecured network in a client-server architecture, connecting an SSH client application with an SSH server.[2] Common applications include remote command-line login and remote command execution, but any network service can be secured with SSH. The protocol specification distinguishes between two major versions, referred to as SSH-1 and SSH-2.

The most visible application of the protocol is for access to shell accounts on Unix-like operating systems, but it sees some limited use on Windows as well. In 2015, Microsoft announced that they would include native support for SSH in a future release. [3]

SSH was designed as a replacement for Telnet and for unsecured remote shell protocols such as the Berkeley rlogin, rsh, and rexec protocols. Those protocols send information, notably passwords, in plaintext, rendering them susceptible to interception and disclosure using packet analysis.[4] The encryption used by SSH is intended to provide confidentiality and integrity of data over an unsecured network, such as the Internet,although files leaked by Edward Snowden indicate that the National Security Agency can sometimes decrypt SSH, allowing them to read the content of SSH sessions

SSH uses public-key cryptography to authenticate the remote computer and allow it to authenticate the user, if necessary. [2] There are several ways to use SSH; one is to use automatically generated public-private key pairs to simply encrypt a network connection, and then use password authentication to log on.

Another is to use a manually generated public-private key pair to perform the authentication, allowing users or programs to log in without having tospecify a password. In this scenario, anyone can produce a matching pair of different keys (public and private). The public key is placed on all computers that must allow access to the owner of the matching private key (the owner keeps the private key secret). While authentication is basedon the private key, the key itself is never transferred through the network during authentication. SSH only verifies whether the same person offering the public key also owns the matching private key. In all versions of SSH it is important to verify unknown public keys, i.e. associate the public keys with identities, before accepting them as valid. Accepting an attacker's public key without validation will authorize an unauthorized attacker as a valid user.

SSH is typically used to log into a remote machine and execute commands, but it also supports tunneling, forwarding TCP ports and X11 connections; it can transfer files using the associated SSH file transfer (SFTP) or secure copy (SCP) protocols.[2] SSH uses the client-server model.

The standard TCP port 22 has been assigned for contacting SSH servers.[7]

An SSH client program is typically used for establishing connections to an SSH daemon accepting remote connections. Both are commonly present on most modern operating systems, including Mac OS X, most distributions of GNU/Linux, OpenBSD, FreeBSD, NetBSD, Solaris and OpenVMS. Notably, Windows is one of the few modern desktop/server OSs that does not include SSH by default. Proprietary, freeware and open source (e.g. PuTTY,[8] and the version of OpenSSH which is part of Cygwin [9]) versions of various levels of complexity and completeness exist. Native Linux file managers (e.g. Konqueror) can use the FISH protocol to provide a split-pane GUI with drag-and-drop. The open source Windows program WinSCP [10] provides similar file management (synchronization, copy, remote delete) capability using PuTTY as a back-end. Both WinSCP [11] and PuTTY[12] are available packaged to run directly off of a USB drive, without requiring installation on the client machine. Setting up an SSH server in Windows typically involves installation (e.g. via installing Cygwin[13]).

SSH is important in cloud computing to solve connectivity problems, avoiding the security issues of exposing a cloud-based virtual machine directly on the Internet. An SSH tunnel can provide a secure path over the Internet, through a firewall to a virtual machine. [1

If above service stopped error occursIf above service stopped error occur, when try to login to putty error occurs. Also make sure there is no firefall or antivirus blocking.

Points to remember: 1. Install MS Windows SDK v7.v full with all components. 2. For me patch was required in one of the pom3. Connect to internet4. Follow this post step by step

Architecture The SSH-2 protocol has an internal architecture (defined in RFC 4251) with well-separated layers, namely:

The transport layer (RFC 4253). This layer handles initial key exchange as well as server authentication, and sets up encryption, compression and integrity verification. It exposes to the upper layer an interface for sending and receiving plaintext packets with sizes of up to 32,768 bytes each (more can be allowed by the implementation). The transport layer also arranges for key re-exchange, usually after 1 GB of data has been transferred or after 1 hour has passed, whicheveroccurs first.

The user authentication layer (RFC 4252). This layer handles client authentication and provides a number of authentication methods. Authentication is client-driven: when one is prompted for a password, it may be the SSH client prompting, not the server. The server merely responds to the client's authentication requests. Widely used user-authentication methods include the following:

password: a method for straightforward password authentication, including a facility allowing a password to be changed. Not all programs implement this method.

publickey: a method for public key-based authentication, usually supporting at least DSA or RSA keypairs, with other implementations also supporting X.509certificates.

keyboard-interactive (RFC 4256): a versatile method where the server sends one or more prompts to enter information and the client displays them and sends back responses keyed-in by the user. Used to provide one-time password authentication such as S/Key or SecurID. Used by some OpenSSH configurations when PAM is the underlying host-authentication provider to effectively provide password authentication, sometimes leading to inability to log in with a client that supports just the plain password authentication method.

GSSAPI authentication methods which provide an extensible scheme to perform SSH authentication using external mechanisms such as Kerberos 5 or NTLM, providing single sign-on capability to SSH sessions. These methods are usually implemented by commercial SSH implementations for use in organizations, though OpenSSH does have a working GSSAPI implementation.

The connection layer (RFC 4254). This layer defines the concept of channels, channel requests and global requests using which SSH services are provided. A single SSH connection can host multiple channels simultaneously, each transferring data in both directions. Channel requests are used to relay out-of-band channel-specific data, such as the changed size of a terminal window or the exit code of a server-side process. The SSH client requests a server-side port to be forwarded using a global request. Standard channel types include:

shell for terminal shells, SFTP and exec requests (including SCP transfers)

direct-tcpip for client-to-server forwarded connections

forwarded-tcpip for server-to-client forwarded connections

The SSHFP DNS record (RFC 4255) provides the public host key fingerprints in order to aid in verifying the authenticity of the host.

This open architecture provides considerable flexibility, allowing the use of SSH for a variety of purposes beyond a secure shell. The functionality of the transport layer alone is comparable to Transport Layer Security(TLS); the user-authentication layer is highly extensible with custom authentication methods; and the connection layer provides the ability to multiplex many secondary sessions into a single SSH connection, a feature comparable to BEEP and not available in TLS.

Uses

Logging into OpenWrt via SSH usingPuTTY running on Windows.

SSH is a protocol that can be used for many applications across many platforms including most Unix variants (Linux, the BSDs including Apple's OS X, and Solaris), as well as Microsoft Windows. Some of the applications below may require features that are only available or compatible with specific SSH clients or servers. For example, using the SSH protocol to implement a VPN is possible, but presently only with the OpenSSH server and client implementation.

Uses

SSH is a protocol that can be used for many applications across many platforms including most Unix variants (Linux, the BSDs including Apple's OS X, and Solaris), as well as Microsoft Windows. Some of the applications below may require features that are only available or compatible with specific SSH clients or servers. For example, using the SSH protocol to implement a VPN is possible, but presently only with the OpenSSH server and client implementation.

1) For login to a shell on a remote host (replacing Telnet and rlogin)

2) For executing a single command on a remote host (replacing rsh)

3) For setting up automatic (passwordless) login to a remote server (for example, using OpenSSH [23])

4) Secure file transfer

5) In combination with rsync to back up, copy and mirror files efficiently and securely

6) For forwarding or tunneling a port (not to be confused with a VPN, which routes packets between different networks, or bridges two broadcast domains into one).

7) For using as a full-fledged encrypted VPN. Note that only OpenSSH server and client supports this feature.

8) For forwarding X from a remote host (possible through multiple intermediate hosts)

9) For browsing the web through an encrypted proxy connection with SSH clients that support the SOCKS protocol.

10) For securely mounting a directory on a remote server as a filesystem on a local computer using SSHFS.

11) For automated remote monitoring and management of servers through one or more of the mechanisms discussed above.

12) For development on a mobile or embedded device that supports SSH.

Hadop common failed since need sdk in windows mvn package -Pdist,native-win -DskipTests –Dtar or just mvn package -Pdist,native-win –DskipTests if any encoding error occurs with Dtar. Not Dtar switch not used it just creates a folder inside \hadoop-2.5.2-src\hadoop-dist\target\ hadoop-2.5.2 else hadoop-2.5.2.tar

[WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": CreateProcess error=2, The system cannot find the file specified[ERROR] stdout: []

install protoc buffer to environment variable and run.Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, andsimpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the "old" format.

need to run hadoop src with maven in Microsoft sdk if above error occurs.

To run in Microsoft sdk:

Select Start --> All Programs --> Microsoft Windows SDK v7.1 and open Windows SDK 7.1 Command Prompt. Change directory to Hadoop source code folder (c:\hdfs). Execute mvn package with options -Pdist,native-win -DskipTests -Dtar to create Windows binary tar distribution.

issue is protoc buff is not set correctly in environment path:

After setting c:/protoc32/ in system environment variable path variable , below is output in window ordinary command prompt.

STEP 2: GENERATE HADOOP FROM MAVEN SRC IN WINDOWS SDK

Run in windows sdk command prompt:

NOTE:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Cannot run program "msbuild" (in directory "C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common"): CreateProcess error=2, The system cannot find the file specified -> [Help 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project had

Environment variables:%SystemRoot%;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Java\jdk1.7.0_80\bin;C:\Mongo 2.6\bin;C:\Windows\SysWOW64;C:\mule-standalone-3.7.0\bin;C:\apache-maven-3.1.1\bin;C:\Program Files\Java\jdk1.7.0_80\bin;C:\cygwin64\bin;C:\cygwin64\home\admin\hadoop-2.5.2\bin;C:\cygwin64\home\admin\hadoop-2.5.2\sbin;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\protoc32;C:\hadooputils;C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin;C:\Windows\Microsoft.NET\Framework64\v3.5

C:\Windows\Microsoft.NET\Framework64\v3.5 –MSBuild.exe

util.NativeCrc32"[INFO][INFO] --- exec-maven-plugin:1.2:exec (compile-ms-winutils) @ hadoop-common ---Build started 1/14/2016 10:43:27 PM.Project "C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" on node 0 (default targets). Building solution configuration "Release|Win32".C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln.cache(245,5): error MSB3411: Could not load the Visual C++ component "VCBuild.exe". If the component is not installed, either 1) install the Microsoft Windows SDK for Windows Server 2008 and .NET Framework 3.5, or 2) install Microsoft Visual Studio 2008.Done Building Project "C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" (default targets) -- FAILED.

Build FAILED.

Run again by downloading vcbuild.exe manually from git but version is not compatible.

Build started 1/14/2016 11:05:33 PM.Project "C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" on node 0 (default targets). Building solution configuration "Release|Win32".C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln.cache(245,5): error MSB6003: The specified task executable "VCBuild.exe" could not be run. This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information to see whether you need a x86 (32-bit) or x64 (64-bit) version of the program, and then contact the software publisherDone Building Project "C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" (default targets) -- FAILED.

Installed to fix this issue:

1. dotNetFx45_Full_setup standalone.2. .net Framework 4.3. VC-Compiler-KB2519277

NotFoundExceptionC:\Hadoop\hadoop-2.5.2-src>mvn -X package -Pdist,native-win -DskipTests -DtarApache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 20:52:22+0530)Maven home: C:\apache-maven-3.1.1\bin\..Java version: 1.7.0_80, vendor: Oracle CorporationJava home: C:\Program Files\Java\jdk1.7.0_80\jre

Default locale: en_US, platform encoding: Cp1252OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"[INFO] Error stacktraces are turned on.[DEBUG] Reading global settings from C:\apache-maven-3.1.1\bin\..\conf\settings.xml[DEBUG] Reading user settings from C:\Users\admin\.m2\settings.xml[DEBUG] Using local repository at C:\Users\admin\.m2\repository[DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for C:\Users\admin\.m2\repository[INFO] Scanning for projects...\NOTE:make sure encoding is UTF-8

if above error occurs install .net runtime framework 4.0 standalone and vc-compiler

see now it shows up fine.

Done Building Project "C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" (default targets) -- FAILED.error : You are attempting to build a Win32 application from an x64 environment. If using theWindows 7.1 SDK build environment, type setenv /x86 [C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj]

NOTE:Microsoft does not have 64 bit of vc compiler hence to compile it in 64 followThis steps:1. Download and install the Windows Software Development Kit version 7.1. Visual C++ 2010 Express does not include a 64 bit compiler, but the SDK does. A link to the SDK:http://msdn.microsoft.com/en-us/windowsserver/bb980924.aspx2. Change your project configuration. Go to Properties of your project. On the top of the dialog box there will be a "Configuration" drop-down menu. Make sure that selects "All Configurations." There will also be a "Platform" drop-down that will read "Win32." Finally on the right there is a "Configuration Manager" button - press it. In the dialog that comes up, find your project, hit the Platform drop-down, select New, then select x64. Now change the "Active solution platform" drop-down menu to "x64." When you return to the Properties dialog box, the "Platform" drop-down should now read "x64."

3. Finally, change your toolset. In the Properties menu of your project, under Configuration Properties | General, change Platform Toolset from "v100" to "Windows7.1SDK".These steps have worked for me, anyway

Change win32 to x64 as mentioned above since vc++ editor is not there just change in the below project configuration file.

C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln

AS MENTIONED in the above desc uncomment all 32 Active solution platform.Change win32 to x64 (find and replaced)NOTE:As like below it wont accept comment (i.e #) in global section(preSolution)GlobalSection(SolutionConfigurationPlatforms) = preSolution

Debug|Win32 = Debug|Win32Debug|x64 = Debug|x64Release|Win32 = Release|Win32Release|x64 = Release|x64

EndGlobalSection

Installed vcbuild dir:

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Platforms\Win32\PlatformToolsets\Windows7.1SDK

Error : You are attempting to build a Win32 application from an x64 environment. If using theWindows 7.1 SDK build environment, type setenv /x86 [C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj]Above error coming from Microsoft.Cpp.Win32.Windows7.1SDK.targets file.

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Platforms\Win32\PlatformToolsets\Windows7.1SDK\ Microsoft.Cpp.Win32.Windows7.1SDK.targets

Above Condition:"'$(WindowsSDKVersionOverride)' != ''and '$(TARGET_CPU)' == 'x64'" causes this error . hence set the Environment like below.

Environment variable to set:

set WindowsSdkDir=C:\Program Files\Microsoft SDKs\Windows\v7.1set PATH=%WindowsSdkDir%\Bin;%PATH%set LIB=%WindowsSdkDir%\Lib;%LIB%set INCLUDE=%WindowsSdkDir%\Include;%INCLUDE%set TARGET_CPU=x64

After this all error gone but below error:"C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default target) (1) ->"C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (default target) (2) ->(GetReferenceAssemblyPaths target) ->C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.Targets(847,9): warning MSB3644: The reference assemblies for framework ".NETFramework,Version=v4.0" were not found. To resolve this, install the SDK or Targeting Pack for this framework version or retarget your application to a version of the frameworkfor which you have the SDK or Targeting Pack installed. Note that assemblies will be resolved from the Global Assembly Cache (GAC) and will be used in place ofreference assemblies. Therefore your assembly may not be correctly targeted forthe framework you intend. [C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]"C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default target) (1) ->"C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (default target) (2) ->(ClCompile target) -> C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\intrin.h(26): fatal error C1083: Cannot open include file: 'ammintrin.h': No such file or directory [C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj] C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\intrin.h(26): fatal error C1083: Cannot open include file: 'ammintrin.h': No such file or directory [C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]

1 Warning(s) 2 Error(s)

to fix the above error fix is available in hadoop

I can not compile my C++ projects after installing ‘Security Update for Microsoft Visual Studio 2010 (KB2455033)’, see my yesterday’s post:

Security Update for Microsoft Visual Studio 2010 (KB2455033) failed with Code 643.

Visual Studio 2010 shows the following error:

1>------ Build started: Project: some_project_name_i_am_hiding, Configuration: Debug Win32 ------

1> stdafx.cpp

1>c:\program files (x86)\microsoft visual studio 10.0\vc\include\intrin.h(26): fatal error C1083: Cannot open include file: 'ammintrin.h': No such file or directory

========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

Error is caused because file intrin.h have reference to ammintrin.h, and after this update there is no such file. Relevant fragment from intrin.h:

#if defined(_M_IX86) || defined(_M_X64)

#include <immintrin.h>

#include <ammintrin.h>

i have reinstalled Visual Studio since : Windows Update KB2455033 breaks build with missing ammintrin.h.

see post by visual studio

https://connect.microsoft.com/VisualStudio/feedback/details/660584/windows-update-kb2455033-breaks-build-with-missing-ammintrin-h

Windows Update KB2455033 breaks build with missing ammintrin.h - by

The workaround is to install Visual Studio 2010 SP1. Download it form Microsoft Website: Microsoft Visual Studio 2010 Service Pack 1 (Installer).

File to install:VS10sp1-KB983509.exeOr just download ammintrin.m from http://in.mathworks.com/matlabcentral/answers/uploaded_files/735/ammintrin.m

After this still getting below error obj linker error

to fix this when tried with setenv /x86 it clearly says try x64

hence try with setenv /x64 itself to fix linker error

Change win32 to x64 (find and replaced) as in winutils.sln, winutils.vcxproj, libwinutils.vcxproj

run again

NOTE:In native.sln comment even mixed mode too. It shows native obj linker error previously above file.

finally issue fixed – working !!!.

see hadoop-2.5.2 gets generated.

STEPS:1) download hadoop-src from the apache.2) Download Microsoft .net Runtime 4.0 and VC-compiler.exe

3) change native,winutils folder inside \hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main to match the 64 bit version.changed version available under:

download the https://github.com/arunsadhasivam/hadoop/edit/master/README.md4) download ammintrin.m from below url.http://in.mathworks.com/matlabcentral/answers/uploaded_files/735/ammintrin.melse update with latest version of VC-compiler.exe The workaround is to install Visual Studio 2010 SP1. Download it form Microsoft Website: Microsoft Visual Studio 2010 Service Pack 1 (Installer).

5) check whether the below environment variable is there%SystemRoot%;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Java\jdk1.7.0_80\bin;C:\Mongo 2.6\bin;C:\Windows\SysWOW64;C:\mule-standalone-3.7.0\bin;C:\apache-maven-3.1.1\bin;C:\Program Files\Java\jdk1.7.0_80\bin;C:\cygwin64\bin;C:\cygwin64\home\admin\hadoop-2.5.2\bin;C:\cygwin64\home\admin\hadoop-2.5.2\sbin;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\protoc32;C:\hadooputils;C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin;C:\Windows\Microsoft.NET\Framework64\v3.5

6) install windows sdk 1.7 version and make sure that it open without any error like below

7) set the below environment variable and then run

All are very important especially setenv else it uses /x86 and shows “trying to run win32 in 64”setenv /x64 set WindowsSdkDir=C:\Program Files\Microsoft SDKs\Windows\v7.1set PATH=%WindowsSdkDir%\Bin;%PATH%set LIB=%WindowsSdkDir%\Lib;%LIB%set INCLUDE=%WindowsSdkDir%\Include;%INCLUDE%set TARGET_CPU=x64

it open new windows then enter all others

STEP 3: RUN MAVEN TO GENERATE NATIVE LIBRARY

After successful run , hadoop dist did not have lib folder , since lib folder is important for running hadoop in windows .

when run with out native lib it shows below error. Since windows need native libraries

TO get the native libraries use the below commandmvn package -Pdist,native –DskipTests

download CMakesince hadoop requires this file to compile header file

Requirements:

1. Windows System2. JDK 1.6+3. Maven 3.0 or later4. Findbugs 1.3.9 (if running findbugs)5. ProtocolBuffer 2.5.06. CMake 2.6 or newer7. Make8. Windows SDK or Visual Studio 2010 Professional9. Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip10. zlib headers (if building native code bindings for zlib)11. Internet connection for first build (to fetch all Maven and Hadoop dependencies)

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012). Do not use Visual Studio Express. It does not support compiling for 64-bit, which is problematic if running a 64-bitsystem. The Windows SDK is free to download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

Building:

Keep the source code tree in a short path to avoid running into problems related to Windows maximum path length limitation. (For example, C:\hdc).

Run builds from a Windows SDK Command Prompt. (Start, All Programs, Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt.)

JAVA_HOME must be set, and the path must not contain spaces. If the full path would contain spaces, then use the Windows short path instead.

You must set the Platform environment variable to either x64 or Win32 depending on whether you're running a 64-bit or 32-bit system. Note that this is case-sensitive. It must be "Platform", not "PLATFORM" or "platform". Environment variables on Windows are usually case-insensitive, but Maven treats them as case-sensitive. Failure to set this environment variable correctly will cause msbuild to fail while building the native code in hadoop-common.

set Platform=x64 (when building on a 64-bit system) set Platform=Win32 (when building on a 32-bit system)

Several tests require that the user must have the Create Symbolic Links privilege.

All Maven goals are the same as described above with the exception that native code is built by enabling the 'native-win' Maven profile. -Pnative-win is enabled by default when building on Windows since the native components are required (not optional) on Windows.

If native code bindings for zlib are required, then the zlib headers must bedeployed on the build machine. Set the ZLIB_HOME environment variable to the directory containing the headers.

set ZLIB_HOME=C:\zlib-1.2.7

At runtime, zlib1.dll must be accessible on the PATH. Hadoop has been tested with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in

the zlib 1.2.7 source tree.

Building distributions:

Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]

see now it looks to add native lib to the dist folder with native lib.mingw-w64 –Make file to compile c ,c++ header filer

The mingw-w64 project is a complete runtime environment for gcc to support binaries native to Windows 64-bit and 32-bit operating systems.

CMAKE VS MAKE

Programmers have been using CMake and Make for a long time now. When you join a big company or start workingon a project with a large codebase, there are all these builds that you need to take care of. You must have seenthose “CMakeLists.txt” files floating around. You are supposed to run “cmake” and “make” commands on theterminal. A lot of people just follow the instructions blindly, not really caring about why we need to do things in acertain way. What is this whole build process and why is it structured this way? What are the differences betweenCMake and Make? Does it matter? Are they interchangeable? As it turns out, they are quite different. It is important to understand the differences between them to make sure youdon’t get yourself in trouble. Before getting into the differences, let’s first see what they are.

MakeThe way in which we design a software system is that we first write code, then the compiler compiles it and createsexecutable files. These executable files are the ones that carry out the actual task. “Make” is a tool that controls thegeneration of executables and other non-source files of a program from the program’s source files.

The “Make” tool needs to know how to build your program. It gets its knowledge of how to build your program from afile called the “makefile”. This makefile lists each of the non-source files and how to compute it from other files. Whenyou write a program, you should write a makefile for it, so that it is possible to use “Make” to build and install theprogram. Simple stuff! If you didn’t understand it, go back and read the paragraph again because it’s important for thenext part.

Why do we need “Make”?The reason we need “Make” is because it enables the end user to build and install your package without knowing thedetails of how it’s done. Every project comes with its own rules and nuances, and it can get quite painful every timeyou have a new collaborator. That’s the reason we have this makefile. The details of the build process are actuallyrecorded in the makefile that you supply. “Make” figures out automatically which files it needs to update, based onwhich source files have changed. It also automatically determines the proper order for updating the files, in case onenon-source file depends on another non-source file.

Recompiling the entire program every time we change a small part of the system would be inefficient. Hence, if youchange a few source files and then run “Make”, it doesn’t recompile the whole thing. It updates only those non-sourcefiles that depend directly or indirectly on the source files that you changed. Pretty neat! “Make” is not limited to anyparticular language. For each non-source file in the program, the makefile specifies the shell commands to computeit. These shell commands can run a compiler to produce an object file, the linker to produce an executable, ar toupdate a library, Makeinfo to format documentation, etc. “Make” is not limited to just building a package either. Youcan also use “Make” to control installing or uninstalling a package, generate tags tables for it, or anything else youwant to do often enough to make it worth while writing down how to do it.

CMakeCMake stands for Cross-platform Make. CMake recognizes which compilers to use for a given kind of source. In caseyou didn’t know, you can’t use the same compiler to build all the different kinds of sources. You can do this manuallyevery time you want to build your project, but it would be tedious and painful. CMake invokes the right sequence ofcommands for each type of target. Therefore, there is no explicit specification of commands like $(CC).

For coding junkies who really want the gory details, read on. If you are not into all that, you can skip to the nextsection. All the usual compiler/linker flags dealing with the inclusion of header files, libraries, etc are replaced byplatform independent and build system independent commands. Debugging flags are included by either setting thevariable CMAKE_BUILD_TYPE to “Debug”, or by passing it to CMake when invoking the program:

cmake -DCMAKE_BUILD_TYPE:STRING=Debug.

CMake also offers the platform independent inclusion of the ‘-fPIC’ flag (via the POSITION_INDEPENDENT_CODEproperty) and many others. Still, more obscure settings can be implemented by hand in CMake just as well as in aMakefile (by using COMPILE_FLAGS and similar properties). Of course CMake really starts to shine when third partylibraries (like OpenGL) are included in a portable manner.

What is the difference?The build process has one step if you use a Makefile, namely typing “make” at the command line. For CMake, thereare two steps: First, you need to setup your build environment (either by typing cmake <source_dir> in your builddirectory or by running some GUI client). This creates a makefile or something equivalent, depending on the buildsystem of your choice (e.g. Make on *nix, VC++ or MinGW on Windows, etc). The build system can be passed toCMake as a parameter. However, CMake makes reasonable default choices depending on your systemconfiguration. Second, you perform the actual build in the selected build system.

We are going to jump into the GNU build system territory here. If you are not familiar with that, this paragraph mightlook like jibber-jabber to you. Alright, now that I have given the statutory warning, let’s move on! We can compareCMake with Autotools. When we do that, we can see the shortcomings of Make, and they form the reason for thecreation of Autotools. We can also see the obvious advantages of CMake over Make. Autoconf solves an importantproblem i.e. reliable discovery of system-specific build and runtime information. But this is only a small part in thedevelopment of portable software. To this end, the GNU project has developed a suite of integrated utilities to finishthe job Autoconf started: the GNU build system, whose most important components are Autoconf, Automake, andLibtool.“Make” can’t do that, at least not without modifying it anyway! You can make it do all that stuff but it would take a lotof time maintaining it across platforms. CMake solves the same problem, but at the same time, it has a fewadvantages over the GNU Build System:

The language used to write CMakeLists.txt files is readable and easier to understand. It doesn’t only rely on “Make” to build the project. It supports multiple generators like Xcode, Eclipse, Visual Studio, etc.

When comparing CMake with Make, there are several advantages of using CMake:

Cross platform discovery of system libraries. Automatic discovery and configuration of the toolchain. Easier to compile your files into a shared library in a platform agnostic way, and in general easier to use than make.

CMake does more than just “make”, so it can be more complex. In the long run, it’s better to learn how to use it. If youhave just a small project on only one platform, then maybe “Make” can do a better job.

STEPS FOR INSTALLING HADOOP ONWINDOWS

Steps to generate hadoop executable and native files

1. JDK 1.6+2. Maven 3.0 or later3. Findbugs 1.3.9 (if running findbugs)4. ProtocolBuffer 2.5.0 - marshalling5. CMake(cross platform make) 2.6 or newer - give instructions commands to be

compiled by make 6. Make - to build executables by running the cmake scripts.7. Windows SDK or Visual Studio 2010 Professional8. Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip – to

connect to port 22 since hadoop name node listen to port 22.9. zlib headers (if building native code bindings for zlib)- make sure zlib1.dll on class

path.10. Internet connection for first build (to fetch all Maven and Hadoop dependencies)11. Make sure try you set all the below environment variable before run maven or atleast

setenv /x64 . since setenv is very important without which it wont work.

setenv /x64set WindowsSdkDir=C:\Program Files\Microsoft SDKs\Windows\v7.1set PATH=%WindowsSdkDir%\Bin;%PATH%set LIB=%WindowsSdkDir%\Lib;%LIB%set INCLUDE=%WindowsSdkDir%\Include;%INCLUDE%set TARGET_CPU=x64

12. make sure set hadooputils - C:\hadooputils can be downloaded from https://github.com/arunsadhasivam/hadoop/tree/master/hadooputils%20-complete

13. download hadood src and change hadood-common project (native,winutils) folder in C:\Hadoop\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main with corrected files according to 64 bit compilation from github. https://github.com/arunsadhasivam/hadoop/hadoopWindows .

14. run maven with below command to compile src initially. mvn package -Pdist,native-win –DskipTests

15. next run to get native file separately for that need CMake and Make exe in classpath.mvn package -Pdist,native –DskipTest

NOTE: Software needed to be installed in order are:

1.dotNetFx40_Full_x86_x64.exe2. winsdk_web.exe

3. VC-Compiler-KB251927.exe4. cmake-3.3.2-win32-x86.exe

5. mingw-w64-install.exe after install with 64 format rename mingw-make to make

Environment variable:

%SystemRoot%;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Java\jdk1.7.0_80\bin;C:\Mongo 2.6\bin;C:\Windows\SysWOW64;C:\mule-standalone-3.7.0\bin;C:\apache-maven-3.1.1\bin;C:\Program Files\Java\jdk1.7.0_80\bin;C:\cygwin64\bin;C:\cygwin64\home\admin\hadoop-2.5.2\bin;C:\cygwin64\home\admin\hadoop-2.5.2\sbin;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\protoc32;C:\hadooputils;C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin;C:\Windows\Microsoft.NET\Framework64\v3.5;C:\Program Files (x86)\Git\bin;C:\Program Files (x86)\CMake\bin;C:\zlib128

C:\hadooputils –some exe like winutils.exe need to be downloaded manuallyAlso better to have git.exe and tortoise svn.exe in environment variable path since it shows warning.

mingw-w64:

NOTE:Renamed ming64-make to make since maven needs executable make.

make sure it points to only one version of .net framework. See it refer 4.0 but only 3.5 is in environment PATH system variable.

As you can see older version does not impact only the latest version causes the problem. Since like in below it take precedence.Although .net 3.5 is needed to run by Microsoft sdk 7.1 below but all above v4.0.30319 gets installed during .net runtime 4.0 installtion.which is needed.removing that show error during compilation. So don’t remove v4.0.30319 . see below screen shot showing error.

since windows sdk uses below batch you can see it use .net 3.5 only. Since above I installed C:\Windows\System32\cmd.exe /E:ON /V:ON /T:0E /K "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd"

others not needed in redistributable pacakage

Environment Variable:%SystemRoot%;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Java\jdk1.7.0_80\bin;C:\Mongo 2.6\bin;C:\Windows\SysWOW64;C:\mule-standalone-3.7.0\bin;C:\apache-maven-3.1.1\bin;C:\Program Files\Java\jdk1.7.0_80\bin;C:\cygwin64\bin;C:\cygwin64\home\hadoop-2.2.0\bin;C:\cygwin64\home\hadoop-2.2.0\sbin;C:\protoc32;C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin;C:\Windows\Microsoft.NET\Framework64\v3.5;C:\Program Files (x86)\Git\bin;

works fine with hadoop-2.2.0 and native files gets generated

in hadoop-2.5.2 some files native files are missing

NOTE:important to note that bin folder should contains .dll and .exe file (winutils.exe for hadoop 2.2) which gets generated only for hadoop 2.2

[WARNING] C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\security\SecurityUtil.java:[596,20] IPAddressUtil is internal proprietary API and may be removed in a future release

[INFO]

[INFO] --- native-maven-plugin:1.0-alpha-7:javah (default) @ hadoop-common ---

[INFO] cmd.exe /X /C ""C:\Program Files\Java\jdk1.7.0_80\bin\javah" -d C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\native\javah -classpath "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\classes;C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-annotations\target\hadoop-annotations-2.2.0.jar;C:\Program Files\Java\jdk1.7.0_80\jre\..\lib\tools.jar;C:\Users\admin\.m2\repository\com\google\guava\guava\11.0.2\guava-11.0.2.jar;C:\Users\admin\.m2\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;C:\Users\admin\.m2\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;C:\Users\admin\.m2\repository\org\apache\commons\commons-math\2.1\commons-math-2.1.jar;C:\Users\admin\.m2\repository\xmlenc\xmlenc\0.52\xmlenc-0.52.jar;C:\Users\admin\.m2\repository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;C:\Users\admin\.m2\repository\commons-codec\commons-codec\1.4\commons-codec-1.4.jar;C:\Users\admin\.m2\repository\commons-io\commons-io\2.1\commons-io-2.1.jar;C:\Users\admin\.m2\repository\commons-net\commons-net\3.1\commons-net-3.1.jar;C:\Users\admin\.m2\repository\javax\servlet\servlet-api\2.5\servlet-api-2.5.jar;C:\Users\admin\.m2\repository\org\mortbay\jetty\jetty\6.1.26\jetty-6.1.26.jar;C:\Users\admin\.m2\repository\org\mortbay\jetty\jetty-util\6.1.26\jetty-util-6.1.26.jar;C:\Users\admin\.m2\repository\com\sun\jersey\jersey-core\1.9\jersey-core-1.9.jar;C:\Users\admin\.m2\repository\com\sun\jersey\jersey-json\1.9\jersey-json-1.9.jar;C:\Users\admin\.m2\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;C:\Users\admin\.m2\repository\stax\stax-api\1.0.1\stax-api-1.0.1.jar;C:\Users\admin\.m2\repository\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar;C:\Users\admin\.m2\repository\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar;C:\Users\admin\.m2\repository\javax\activation\activation\1.1\activation-1.1.jar;C:\Users\admin\.m2\repository\org\codehaus\jackson\jackson-jaxrs\1.8.8\jackson-jaxrs-1.8.8.jar;C:\Users\admin\.m2\repository\org\codehaus\jackson\jackson-xc\1.8.8\jackson-xc-1.8.8.jar;C:\Users\admin\.m2\repository\com\sun\jersey\jersey-server\1.9\jersey-server-1.9.jar;C:\Users\admin\.m2\repository\asm\asm\3.2\asm-3.2.jar;C:\Users\admin\.m2\repository\commons-logging\commons-logging\1.1.1\commons-logging-1.1.1.jar;C:\Users\admin\.m2\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;C:\Users\admin\.m2\repository\net\java\dev\jets3t\jets3t\0.6.1\jets3t-0.6.1.jar;C:\Users\admin\.m2\repository\commons-lang\commons-lang\2.5\commons-lang-2.5.jar;C:\Users\admin\.m2\repository\commons-configuration\commons-configuration\1.6\commons-configuration-1.6.jar;C:\Users\admin\.m2\repository\commons-collections\commons-collections\3.2.1\commons-collections-3.2.1.jar;C:\Users\admin\.m2\repository\commons-digester\commons-digester\1.8\commons-digester-1.8.jar;C:\Users\admin\.m2\repository\commons-beanutils\commons-beanutils\1.7.0\commons-beanutils-1.7.0.jar;C:\Users\admin\.m2\repository\commons-beanutils\commons-beanutils-core\1.8.0\commons-beanutils-core-1.8.0.jar;C:\Users\admin\.m2\repository\org\slf4j\slf4j-api\1.7.5\slf4j-api-1.7.5.jar;C:\Users\admin\.m2\repository\org\codehaus\jackson\jackson-core-asl\1.8.8\jackson-core-asl-1.8.8.jar;C:\Users\admin\.m2\repository\org\codehaus\jackson\jackson-mapper-asl\1.8.8\jackson-mapper-asl-1.8.8.jar;C:\Users\admin\.m2\repository\org\apache\avro\avro\1.7.4\avro-1.7.4.jar;C:\Users\admin\.m2\repository\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;C:\Users\admin\.m2\repository\org\xerial\snappy\snappy-java\1.0.4.1\snappy-java-1.0.4.1.jar;C:\Users\admin\.m2\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-auth\target\hadoop-auth-2.2.0.jar;C:\Users\admin\.m2\repository\com\jcraft\jsch\0.1.42\jsch-0.1.42.jar;C:\Users\admin\.m2\repository\org\apache\zookeeper\zookeeper\3.4.5\zookeeper-3.4.5.jar;C:\Users\admin\.m2\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;C:\Users\admin\.m2\repository\org\tukaani\xz\1.0\xz-1.0.jar" org.apache.hadoop.io.compress.zlib.ZlibCompressor org.apache.hadoop.io.compress.zlib.ZlibDecompressor org.apache.hadoop.security.JniBasedUnixGroupsMapping org.apache.hadoop.io.nativeio.NativeIO org.apache.hadoop.security.JniBasedUnixGroupsNetgroupMapping org.apache.hadoop.io.compress.snappy.SnappyCompressor org.apache.hadoop.io.compress.snappy.SnappyDecompressor org.apache.hadoop.io.compress.lz4.Lz4Compressor org.apache.hadoop.io.compress.lz4.Lz4Decompressor org.apache.hadoop.util.NativeCrc32"

[INFO]

[INFO] --- exec-maven-plugin:1.2:exec (compile-ms-winutils) @ hadoop-common ---

Build started 2/13/2016 6:55:01 PM.

Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" on node 1 (default targets).

ValidateSolutionConfiguration:

Building solution configuration "Release|x64".

Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" (1) is building "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj.metaproj" (2) on node 1 (default targets).

Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj.metaproj" (2) is building "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj" (3) on node 1 (default targets).

PrepareForBuild:

Creating directory "..\..\..\target\winutils\x64\Release\".

Creating directory "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/".

InitializeBuildStatus:

Creating "..\..\..\target\winutils\x64\Release\libwinutils.unsuccessfulbuild" because "AlwaysCreate" was specified.

ClCompile:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\CL.exe /c /Zi /nologo /W3 /WX- /O2 /Oi /GL /D WIN32 /D NDEBUG /D _CONSOLE /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Fo"..\..\..\target\winutils\x64\Release\\" /Fd"..\..\..\target\winutils\x64\Release\vcWindows7.1SDK.pdb" /Gd /TC /errorReport:queue libwinutils.c

libwinutils.c

Lib:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\Lib.exe /OUT:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/libwinutils.lib" /NOLOGO /LTCG ..\..\..\target\winutils\x64\Release\libwinutils.obj

libwinutils.vcxproj -> C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/libwinutils.lib

FinalizeBuildStatus:

Deleting file "..\..\..\target\winutils\x64\Release\libwinutils.unsuccessfulbuild".

Touching "..\..\..\target\winutils\x64\Release\libwinutils.lastbuildstate".

Done Building Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj" (default targets).

Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj.metaproj" (2) is building "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj" (4) on node 1 (default targets).

InitializeBuildStatus:

Creating "..\..\..\target\winutils\x64\Release\winutils.unsuccessfulbuild" because "AlwaysCreate" was specified.

ClCompile:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\CL.exe /c /Zi /nologo /W3 /WX- /O2 /Oi /GL /D WIN32 /D NDEBUG /D _CONSOLE /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Fo"..\..\..\target\winutils\x64\Release\\" /Fd"..\..\..\target\winutils\x64\Release\vcWindows7.1SDK.pdb" /Gd /TC /errorReport:queue readlink.c symlink.c systeminfo.c chmod.c chown.c groups.c hardlink.c task.c ls.c main.c

readlink.c

symlink.c

systeminfo.c

chmod.c

chown.c

groups.c

hardlink.c

task.c

ls.c

main.c

Link:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\link.exe /ERRORREPORT:QUEUE /OUT:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/winutils.exe" /INCREMENTAL:NO /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"..\..\..\target\winutils\x64\Release\winutils.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\bin\winutils.pdb" /SUBSYSTEM:CONSOLE /OPT:REF /OPT:ICF /LTCG /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/winutils.lib" /MACHINE:X64 ..\..\..\target\winutils\x64\Release\readlink.obj

..\..\..\target\winutils\x64\Release\symlink.obj

..\..\..\target\winutils\x64\Release\systeminfo.obj

..\..\..\target\winutils\x64\Release\chmod.obj

..\..\..\target\winutils\x64\Release\chown.obj

..\..\..\target\winutils\x64\Release\groups.obj

..\..\..\target\winutils\x64\Release\hardlink.obj

..\..\..\target\winutils\x64\Release\task.obj

..\..\..\target\winutils\x64\Release\ls.obj

..\..\..\target\winutils\x64\Release\main.obj

"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\bin\libwinutils.lib"

Generating code

Finished generating code

winutils.vcxproj -> C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/winutils.exe

Manifest:

C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\x64\mt.exe /nologo /verbose /outputresource:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/winutils.exe;#1" /manifest ..\..\..\target\winutils\x64\Release\winutils.exe.intermediate.manifest

FinalizeBuildStatus:

Deleting file "..\..\..\target\winutils\x64\Release\winutils.unsuccessfulbuild".

Touching "..\..\..\target\winutils\x64\Release\winutils.lastbuildstate".

Done Building Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj" (default targets).

Done Building Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj.metaproj" (default targets).

Done Building Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln" (default targets).

Build succeeded.

0 Warning(s)

0 Error(s)

Time Elapsed 00:00:13.96

[INFO]

[INFO] --- exec-maven-plugin:1.2:exec (compile-ms-native-dll) @ hadoop-common ---

Build started 2/13/2016 6:55:16 PM.

Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" on node 1 (default targets).

ValidateSolutionConfiguration:

Building solution configuration "Release|x64".

Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (1) is building "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (2) on node 1 (default targets).

PrepareForBuild:

Creating directory "..\..\..\target\native\Release\".

InitializeBuildStatus:

Creating "..\..\..\target\native\Release\native.unsuccessfulbuild" because "AlwaysCreate" was specified.

ClCompile:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\CL.exe /c /I..\winutils\include /I..\..\..\target\native\javah /I"C:\Program Files\Java\jdk1.7.0_80\include" /I"C:\Program Files\Java\jdk1.7.0_80\include\win32" /I.\src /Zi /nologo /W3 /WX- /O2 /Oi /GL /D WIN32 /D NDEBUG /D _WINDOWS /D _USRDLL /D NATIVE_EXPORTS /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Fo"..\..\..\target\native\Release\\" /Fd"..\..\..\target\native\Release\vcWindows7.1SDK.pdb" /Gd /TC /wd4244 /errorReport:queuesrc\org\apache\hadoop\io\compress\lz4\lz4.c src\org\apache\hadoop\io\compress\lz4\Lz4Compressor.c src\org\apache\hadoop\io\compress\lz4\Lz4Decompressor.c src\org\apache\hadoop\io\nativeio\file_descriptor.c src\org\apache\hadoop\io\nativeio\NativeIO.c src\org\apache\hadoop\security\JniBasedUnixGroupsMappingWin.c src\org\apache\hadoop\util\bulk_crc32.csrc\org\apache\hadoop\util\NativeCodeLoader.c src\org\apache\hadoop\util\NativeCrc32.c

lz4.c

Lz4Compressor.c

Lz4Decompressor.c

file_descriptor.c

NativeIO.c

JniBasedUnixGroupsMappingWin.c

src\org\apache\hadoop\security\JniBasedUnixGroupsMappingWin.c(48): warning C4047: 'function' : 'LPSTR' differs in levels of indirection from 'LPSTR *' [C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]

src\org\apache\hadoop\security\JniBasedUnixGroupsMappingWin.c(48): warning C4024: 'FormatMessageA' : different types for formal and actual parameter 5 [C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]

bulk_crc32.c

NativeCodeLoader.c

NativeCrc32.c

Link:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\link.exe /ERRORREPORT:QUEUE /OUT:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/hadoop.dll" /INCREMENTAL:NO /NOLOGO /LIBPATH:..\..\..\target\bin Ws2_32.lib libwinutils.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /ManifestFile:"..\..\..\target\native\Release\hadoop.dll.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\bin\hadoop.pdb" /SUBSYSTEM:WINDOWS /OPT:REF /OPT:ICF /LTCG /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/hadoop.lib" /MACHINE:X64 /DLL ..\..\..\target\native\Release\lz4.obj

..\..\..\target\native\Release\Lz4Compressor.obj

..\..\..\target\native\Release\Lz4Decompressor.obj

..\..\..\target\native\Release\file_descriptor.obj

..\..\..\target\native\Release\NativeIO.obj

..\..\..\target\native\Release\JniBasedUnixGroupsMappingWin.obj

..\..\..\target\native\Release\bulk_crc32.obj

..\..\..\target\native\Release\NativeCodeLoader.obj

..\..\..\target\native\Release\NativeCrc32.obj

Creating library C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/hadoop.lib and object C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/hadoop.exp

Generating code

Finished generating code

native.vcxproj -> C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/hadoop.dll

Manifest:

C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\x64\mt.exe /nologo /verbose /outputresource:"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target/bin/hadoop.dll;#2" /manifest ..\..\..\target\native\Release\hadoop.dll.intermediate.manifest

FinalizeBuildStatus:

Deleting file "..\..\..\target\native\Release\native.unsuccessfulbuild".

Touching "..\..\..\target\native\Release\native.lastbuildstate".

Done Building Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (default targets).

Done Building Project "C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default targets).

Build succeeded.

"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default target) (1) ->

"C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (default target) (2) ->

(ClCompile target) ->

src\org\apache\hadoop\security\JniBasedUnixGroupsMappingWin.c(48): warning C4047: 'function' : 'LPSTR' differs in levels of indirection from 'LPSTR*' [C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]

src\org\apache\hadoop\security\JniBasedUnixGroupsMappingWin.c(48): warning C4024: 'FormatMessageA' : different types for formal and actual parameter 5 [C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]

2 Warning(s)

0 Error(s)

Time Elapsed 00:00:04.47

[INFO]

[INFO] --- avro-maven-plugin:1.7.4:schema (generate-avro-test-sources) @ hadoop-common ---

[INFO]

[INFO] --- hadoop-maven-plugins:2.2.0:protoc (compile-test-protoc) @ hadoop-common ---

[INFO]

[INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-common ---

[INFO] Using default encoding to copy filtered resources.

[INFO]

[INFO] --- maven-antrun-plugin:1.6:run (create-log-dir) @ hadoop-common ---

[INFO] Executing tasks

main:

[delete] Deleting directory C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\test\data

[mkdir] Created dir: C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\test\data

[mkdir] Created dir: C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\log

[copy] Copying 6 files to C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\test-classes

[INFO] Executed tasks

[INFO]

[INFO] --- maven-antrun-plugin:1.6:run (copy-test-tarballs) @ hadoop-common ---

[INFO] Executing tasks

main:

[copy] Copying 2 files to C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\test-classes

[INFO] Executed tasks

[INFO]

[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-common ---

[INFO] Compiling 341 source files to C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\target\test-classes

[WARNING] C:\Hadoop\hadoop-2.2.0-src\hadoop-common-project\hadoop-common\src\test\java\org\apache\hadoop\security\ssl\KeyStoreTestUtil.java:[108,15] X509CertImpl is internal proprietary API and may be removed in a future release

[INFO]

[INFO] --- maven-surefire-plugin:2.12.3:test (default-test) @ hadoop-common ---

[INFO] Tests are skipped.

[INFO] --- maven-jar-plugin:2.3.1:jar (prepare-jar) @ hadoop-common ---Apache Hadoop Main ................................ SUCCESS [2.839s]

[INFO] Apache Hadoop Project POM ......................... SUCCESS [1.389s]

[INFO] Apache Hadoop Annotations ......................... SUCCESS [2.867s]

[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.191s]

[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [1.808s]

[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [3.997s]

[INFO] Apache Hadoop Auth ................................ SUCCESS [3.365s]

[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.511s]

[INFO] Apache Hadoop Common .............................. SUCCESS [2:19.517s]

[INFO] Apache Hadoop NFS ................................. SUCCESS [7.271s]

[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.052s]

[INFO] Apache Hadoop HDFS ................................ SUCCESS [1:48.356s]

[INFO] Apache Hadoop HttpFS .............................. SUCCESS [29.764s]

[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [15.483s]

[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [5.892s]

[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.116s]

[INFO] hadoop-yarn ....................................... SUCCESS [0.755s]

[INFO] hadoop-yarn-api ................................... SUCCESS [58.341s]

[INFO] hadoop-yarn-common ................................ SUCCESS [41.616s]

[INFO] hadoop-yarn-server ................................ SUCCESS [0.222s]

[INFO] hadoop-yarn-server-common ......................... SUCCESS [12.719s]

[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [18.352s]

[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [4.325s]

[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [24.415s]

[INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.457s]

[INFO] hadoop-yarn-client ................................ SUCCESS [6.260s]

[INFO] hadoop-yarn-applications .......................... SUCCESS [0.163s]

[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [2.488s]

[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.098s]

[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [33.843s]

[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [2.359s]

[INFO] hadoop-yarn-site .................................. SUCCESS [0.175s]

[INFO] hadoop-yarn-project ............................... SUCCESS [1.226s]

[INFO] hadoop-mapreduce-client-common .................... SUCCESS [25.130s]

[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [3.742s]

[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [15.882s]

[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [8.723s]

[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [7.939s]

[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [2.220s]

[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [7.636s]

[INFO] hadoop-mapreduce .................................. SUCCESS [0.597s]

[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [5.471s]

[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [12.711s]

[INFO] Apache Hadoop Archives ............................ SUCCESS [2.580s]

[INFO] Apache Hadoop Rumen ............................... SUCCESS [9.520s]

[INFO] Apache Hadoop Gridmix ............................. SUCCESS [5.555s]

[INFO] Apache Hadoop Data Join ........................... SUCCESS [3.696s]

[INFO] Apache Hadoop Extras .............................. SUCCESS [4.345s]

[INFO] Apache Hadoop Pipes ............................... SUCCESS [0.067s]

[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [1.350s]

[INFO] Apache Hadoop Tools ............................... SUCCESS [0.055s]

[INFO] Apache Hadoop Distribution ........................ SUCCESS [15.598s]

[INFO] Apache Hadoop Client .............................. SUCCESS [5.230s]

[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.438s]

[INFO] Total time: 11:14.078s

[INFO] Finished at: Sat Feb 13 19:05:29 IST 2016

[INFO] Final Memory: 80M/395M

C:\cygwin64\home\hadoop-2.2.0\etc\hadoop\hadoop-env.cmdset JAVA_HOME=C:\Java\jdk1.7.0_80

Note:

make sure java home without space I.e avoid like c:/program files

C:\cygwin64\home\hadoop-2.2.0\etc\hadoop\core-site.xml<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

C:\cygwin64\home\hadoop-2.2.0\etc\hadoop\mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

C:\cygwin64\home\hadoop-2.2.0\etc\hadoop\hdfs-site.xml

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>/yarn_data/dfs/namenode</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/yarn_data/dfs/datanode</value>

</property>

</configuration>

NOTE:

dont give c:/cygwin it throws error whatever you give it takes the root from cygwin install directory.

C:\cygwin64\home\hadoop-2.2.0\etc\hadoop\yarn-site.xm l

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.application.classpath</name>

<value>

%HADOOP_HOME%\etc\hadoop,

%HADOOP_HOME%\share\hadoop\common\*,

%HADOOP_HOME%\share\hadoop\common\lib\*,

%HADOOP_HOME%\share\hadoop\mapreduce\*,

%HADOOP_HOME%\share\hadoop\mapreduce\lib\*,

%HADOOP_HOME%\share\hadoop\hdfs\*,

%HADOOP_HOME%\share\hadoop\hdfs\lib\*,

%HADOOP_HOME%\share\hadoop\yarn\*,

%HADOOP_HOME%\share\hadoop\yarn\lib\*

</value>

</property>

</configuration>

16/02/14 13:51:29 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED;

cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize LocalizationServiceorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize Loc

alizationService

cygwin output:

ERROR:

if you run hadopp on cygwin it won't work. Please work on windows cmd it works

************************************************************/

16/02/13 19:40:14 WARN common.Util: Path /yarn_data/dfs/datanode should be speci

fied as a URI in configuration files. Please update hdfs configuration.

16/02/13 19:40:15 WARN util.NativeCodeLoader: Unable to load native-hadoop libra

ry for your platform... using builtin-java classes where applicable

16/02/13 19:40:17 FATAL datanode.DataNode: Exception in secureMain

java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.a

ccess0(Ljava/lang/String;I)Z

at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)

at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)

at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskCheck

Jars need to run

avro-1.7.4.jar

commons-configuration-1.6.jar

commons-httpclient-3.1.jar

commons-lang-2.5.jar

commons-logging-1.1.1.jar

guava-11.0.2.jar

hadoop-auth-2.2.0.jar

hadoop-common-2.2.0.jar

hadoop-hdfs-2.2.0.jar

hadoop-mapreduce-client-common-2.2.0.jar

hadoop-mapreduce-client-core-2.2.0.jar

hadoop-nfs-2.2.0.jar

hadoop-yarn-api-2.2.0.jar

hadoop-yarn-common-2.2.0.jar

jackson-core-asl-1.8.8.jar

jackson-mapper-asl-1.8.8.jar

log4j-1.2.17.jar

slf4j-api-1.7.5.jar

c:\users\admin>hdfs dfs -mkdir /input

see the output

C:\Users\admin>yarn jar C:/HADOOPOUTPUT/mapreduce.jar test.WordCount /input/wordcount.txt /output

hadoopmapreduce in ubuntu commandsclient-core-2.5.jar in class path.hdadmin@ubuntu:/jars$ hadoop fs -lshdadmin@ubuntu:/jars$ hadoop fs -mkdir /aruncreate a directory arun in hdfs access by hdfs://arunhdadmin@ubuntu:/jars$ hadoop fs -copyFromLocal sales.csv /arun/salesArun.csvcopy a file from /home/hdadmin/jars/sales.csv to hdfs with new name /arun/salesArun.csvhdadmin@ubuntu:/jars$ hadoop jar firsthadoop.jar mapreduce.SalesCountryDriver/arun/salesArun.csv /outputrun the hadoop mapreduce jar with name firsthadoop with main class SalesCountry Driver programAll the default port no will be inside /HADOOP_HOME/share/doc/hadoop webapps

To force safe mode:

hadoop namenode -recover -forcetemp files files will be under c:/tmp/hadoop-admin directoryTo change tmp Directory

To change the local temp directory use:

-D dfs.data.dir=/tmp

To specify additional local temp directories use:

-D mapred.local.dir=/tmp/local -D mapred.system.dir=/tmp/system -D mapred.temp.dir=/tmp/temp

Hadoop Streaming