HBase僞分布式環境下,HBase的API操作,遇到的問題

在hadoop2.5.2伪分布式上,安装了hbase1.0.1.1的伪分布式

利用HBase的API创建个testapi的表时,提示

 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
        at com.wonderad.mr.ADReport2HBase.main(ADReport2HBase.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        ... 6 more

原因:jar包在handoop上运行的时候,要调用hbase/lib下的jar包
解决方法:
      第一种:在hadoop的hadoop-env.sh中添加   export HADOOP_CLASSPATH=/usr/local/hbase/lib/*
      其实,还有一种方法就是将hbase/lib/下的jar拷贝到hadoop/lib下,从道理上说这种肯定可以
     但是我试了试,没成功
    原因是:我的本地库文件压根就没有加载成功,因为我的是32位的,hadoop2.5.2中的库文件是64的
 
当然可以自己编译一下就行了,按照src下的BUILDING.text即可!如下:

Build instructions for Hadoop

-
Requirements:

* Unix System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

-
Maven main modules:

hadoop (Main Hadoop project)
- hadoop-project (Parent POM for all Hadoop Maven modules. )
(All plugins & dependencies versions are defined here.)
- hadoop-project-dist (Parent POM for modules that generate distributions.)
- hadoop-annotations (Generates the Hadoop doclet used to generated the Javadocs)
- hadoop-assemblies (Maven assemblies used by the different modules)
- hadoop-common-project (Hadoop Common)
- hadoop-hdfs-project (Hadoop HDFS)
- hadoop-mapreduce-project (Hadoop MapReduce)
- hadoop-tools (Hadoop tools like Streaming, Distcp, etc.)
- hadoop-dist (Hadoop distribution assembler)

-
Where to run Maven from?

It can be run from any module. The only catch is that if not run from utrunk
all modules that are not part of the build run must be installed in the local
Maven cache or available in a Maven repository.

-
Maven build goals:

* Clean : mvn clean
* Compile : mvn compile [-Pnative]
* Run tests : mvn test [-Pnative]
* Create JAR : mvn package
* Run findbugs : mvn compile findbugs:findbugs
* Run checkstyle : mvn compile checkstyle:checkstyle
* Install JAR in M2 cache : mvn install
* Deploy JAR to Maven repo : mvn deploy
* Run clover : mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
* Run Rat : mvn apache-rat:check
* Build javadocs : mvn javadoc:javadoc
* Build distribution : mvn package [-Pdist][-Pdocs][-Psrc][-Pnative][-Dtar]
* Change Hadoop version : mvn versions:set -DnewVersion=NEWVERSION

Build options:

* Use -Pnative to compile/bundle native code
* Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist)
* Use -Psrc to create a project source TAR.GZ
* Use -Dtar to create a TAR with the distribution (using -Pdist)

Snappy build options:

Snappy is a compression library that can be utilized by the native code.
It is currently an optional component, meaning that Hadoop can be built with
or without this dependency.

* Use -Drequire.snappy to fail the build if libsnappy.so is not found.
If this option is not specified and the snappy library is missing,
we silently build a version of libhadoop.so that cannot make use of snappy.
This option is recommended if you plan on making use of snappy and want
to get more repeatable builds.

* Use -Dsnappy.prefix to specify a nonstandard location for the libsnappy
header files and library files. You do not need this option if you have
installed snappy using a package manager.
* Use -Dsnappy.lib to specify a nonstandard location for the libsnappy library
files. Similarly to snappy.prefix, you do not need this option if you have
installed snappy using a package manager.
* Use -Dbundle.snappy to copy the contents of the snappy.lib directory into
the final tar file. This option requires that -Dsnappy.lib is also given,
and it ignores the -Dsnappy.prefix option.

Tests options:

* Use -DskipTests to skip tests when running the following Maven goals:
'package', 'install', 'deploy' or 'verify'
* -Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,....
* -Dtest.exclude=<TESTCLASSNAME>
* -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java

-
Building components separately

If you are building a submodule directory, all the hadoop dependencies this
submodule has will be resolved as all other 3rd party dependencies. This is,
from the Maven cache or from a Maven repository (if not available in the cache
or the SNAPSHOT 'timed out').
An alternative is to run 'mvn install -DskipTests' from Hadoop source top
level once; and then work from the submodule. Keep in mind that SNAPSHOTs
time out after a while, using the Maven '-nsu' will stop Maven from trying
to update SNAPSHOTs from external repos.

-
Protocol Buffer compiler

The version of Protocol Buffer compiler, protoc, must match the version of the
protobuf JAR.

If you have multiple versions of protoc in your system, you can set in your
build shell the HADOOP_PROTOC_PATH environment variable to point to the one you
want to use for the Hadoop build. If you don't define this environment variable,
protoc is looked up in the PATH.
-
Importing projects to eclipse

When you import the project to eclipse, install hadoop-maven-plugins at first.

$ cd hadoop-maven-plugins
$ mvn install

Then, generate eclipse project files.

$ mvn eclipse:eclipse -DskipTests

At last, import to eclipse by specifying the root directory of the project via
[File] > [Import] > [Existing Projects into Workspace].

-
Building distributions:

Create binary distribution without native code and without documentation:

$ mvn package -Pdist -DskipTests -Dtar

Create binary distribution with native code and with documentation:

$ mvn package -Pdist,native,docs -DskipTests -Dtar

Create source distribution:

$ mvn package -Psrc -DskipTests

Create source and binary distributions with native code and documentation:

$ mvn package -Pdist,native,docs,src -DskipTests -Dtar

Create a local staging version of the website (in /tmp/hadoop-site)

$ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

-

Handling out of memory errors in builds

-

If the build process fails with an out of memory error, you should be able to fix
it by increasing the memory used by maven -which can be done via the environment
variable MAVEN_OPTS.

Here is an example setting to allocate between 256 and 512 MB of heap space to
Maven

export MAVEN_OPTS="-Xms256m -Xmx512m"

-

Building on OS/X

-

A one-time manual step is required to enable building Hadoop OS X with Java 7
every time the JDK is updated.
see: https://issues.apache.org/jira/browse/HADOOP-9350

$ sudo mkdir `/usr/libexec/java_home`/Classes
$ sudo ln -s `/usr/libexec/java_home`/lib/tools.jar `/usr/libexec/java_home`/Classes/classes.jar

-

Building on Windows

-
Requirements:

* Windows System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* Windows SDK or Visual Studio 2010 Professional
* Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
which is problematic if running a 64-bit system. The Windows SDK is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

-
Building:

Keep the source code tree in a short path to avoid running into problems related
to Windows maximum path length limitation. (For example, C:\hdc).

Run builds from a Windows SDK Command Prompt. (Start, All Programs,
Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt.)

JAVA_HOME must be set, and the path must not contain spaces. If the full path
would contain spaces, then use the Windows short path instead.

You must set the Platform environment variable to either x64 or Win32 depending
on whether you're running a 64-bit or 32-bit system. Note that this is
case-sensitive. It must be "Platform", not "PLATFORM" or "platform".
Environment variables on Windows are usually case-insensitive, but Maven treats
them as case-sensitive. Failure to set this environment variable correctly will
cause msbuild to fail while building the native code in hadoop-common.

set Platform=x64 (when building on a 64-bit system)
set Platform=Win32 (when building on a 32-bit system)

Several tests require that the user must have the Create Symbolic Links
privilege.

All Maven goals are the same as described above with the exception that
native code is built by enabling the 'native-win' Maven profile. -Pnative-win
is enabled by default when building on Windows since the native components
are required (not optional) on Windows.

If native code bindings for zlib are required, then the zlib headers must be
deployed on the build machine. Set the ZLIB_HOME environment variable to the
directory containing the headers.

set ZLIB_HOME=C:\zlib-1.2.7

At runtime, zlib1.dll must be accessible on the PATH. Hadoop has been tested
with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in
the zlib 1.2.7 source tree.

http://www.zlib.net/

-
Building distributions:

* Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]

更多相关文章
  • 在Ubuntu14.04环境下,编译android程序时候,运行ant debug的时候出现如下异常: [aapt] /opt/Android/adt-bundle-linux-x86_64-20140624/sdk
  • 今天在公司的windows2003服务器上使用IIS搭建站点,语言是asp.net和PHP,两种需要同时支持,我还是第1次遇到这样有挑战性的要求. asp.net没问题,配置起来很顺利.但是加载PHP支持的时候,遇到
  • 异常信息:java.net.UnknownHostException: Host is unresolved. 异常发生场景:GRPS网络环境. 解决方法:1.检查Androidmanifest是否添加了相应权限.
  • 在Win7虛擬機下搭建Hadoop2.6.0僞分布式環境
    近几年大数据越来越火热.由于工作需要以及个人兴趣,最近开始学习大数据相关技术.学习过程中的一些经验教训希望能通过博文沉淀下来,与网友分享讨论,作为个人备忘. 第一篇,在win7虚拟机下搭建hadoop2.6.0伪分布式环境. 1. 所需要的软件 使用Vmware 11.0搭建虚拟机,安装Ubuntu ...
  • spark1.2.0版本搭建僞分布式環境
    1.下载scala2.11.5版本,下载地址为:http://www.scala-lang.org/download/2.11.5.html 2.安装和配置scala: 第一步:上传scala安装包 并解压 第二步 配置SCALA_HOME环境变量到bash_profile 第三步 source 使
  • hadoop2.6.0版本搭建僞分布式環境
    由于个人需要在自己的笔记本上搭建hadoop伪分布环境,为了方便自己使用,如想看机器也看之前的一篇博客:hadoop2.6.0版本集群环境搭建 一台虚拟机,配置信息如下: 内存:1G,cpu:一个core,硬盘:15G 1.修改下主机名为master sudo vi /etc/sysconfig/n ...
  • Ubuntu 14.04 (32位)上搭建Hadoop 2.5.1單機和僞分布式環境
    引言 一直用的Ubuntu 32位系统(准备下次用Fedora,Ubuntu越来越不适合学习了),今天准备学习一下Hadoop,结果下载Apache官网上发布的最新的封装好的2.5.1版,配置完了根本启动不起来,查看错误日志发现是native库的版本和系统不一致,使用file命令可以发现封装版的na ...
  • 集群环境搭建请见:http://blog.csdn.net/jediael_lu/article/details/45145767 一.环境准备 1.安装linux.jdk 2.下载hadoop2.6.0,并解压 3
一周排行