Discussion:
troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
(too old to reply)
Andrew Purtell
2012-01-11 02:11:41 UTC
Permalink
Hi,


I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but have run into a couple of issues. Perhaps I'm doing something wrong.

What I've done:

  - Checked out Hadoop branch-0.23 and installed artifacts into the local maven cache:

     $ cd /path/to/hadoop
     $ mvn clean install -DskipTests

  - Checked out HBase branch-0.92 and attempted to test under the maven profile for 0.23:
 
     $ cd /path/to/hbase
     $ mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce

The first issue I encountered was that the MiniMRCluster would not initialize:

2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153): Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying ClientProtocolProvider : org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "test"
2012-01-10 18:01:02,494 ERROR [main] security.UserGroupInformation(1160): PriviledgedActionException as:apurtell (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
2012-01-10 18:01:02,497 INFO  [main]
hbase.HBaseTestingUtility(1173): Stopping mini mapreduce cluster...
2

I filed MAPREDUCE-3654 (https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be more going on here, but the patch on that issue let me move forward.

Now I am seeing this:

2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked org.apache.hadoop.mapred.LocalClientProtocolProvider as the ClientProtocolProvider
2012-01-10 17:51:35,243 INFO  [main] mapreduce.TableOutputFormat(197): Created table instance for mrtest
2012-01-10 17:51:35,249 DEBUG [main] mapreduce.JobSubmitter(347): Configuring job job_local_0001 with file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001 as the submit dir
2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding the following namenodes' delegation tokens:[hdfs://localhost.localdomain:54525]
2012-01-10 17:51:35,251 WARN  [main]
mapreduce.JobSubmitter(139): Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default FileSystem: file:///
2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning up the staging area file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
2012-01-10 17:51:35,268 ERROR [main] security.UserGroupInformation(1160): PriviledgedActionException as:apurtell (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
2012-01-10 17:51:35,273 INFO  [main] hbase.HBaseTestingUtility(1173): Stopping mini mapreduce
cluster...

Of course that FileNotFoundException is for a file that does indeed exist, in the local maven cache:

    $ ls -l /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
    -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56 /home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar

I set permissions on all directories from /home/apurtell to the jar to 'rwxr-xr-x' just to make sure there was not some kind of odd permissions problem.

Any idea what is going on here?


Best regards,


- Andy


Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Mahadev Konar
2012-01-11 02:30:58 UTC
Permalink
Hi Andrew,
Thanks for filing MAPREDUCE-3654. Ill take a look. As for the
filenotfound issue
mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce

on hbase 0.92. (latest svn branch) should be able to recreate this?
(Assuming your patch applied on MR-3654)

I can try that on my setup and see what the issue is.

thanks
mahadev
Post by Andrew Purtell
Hi,
I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but
have run into a couple of issues. Perhaps I'm doing something wrong.
     $ cd /path/to/hadoop
     $ mvn clean install -DskipTests
     $ cd /path/to/hbase
     $ mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153): Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying
org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"test"
PriviledgedActionException as:apurtell (auth:SIMPLE)
cause:java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
Stopping mini mapreduce cluster...
2
I filed MAPREDUCE-3654
(https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be more
going on here, but the patch on that issue let me move forward.
2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked
org.apache.hadoop.mapred.LocalClientProtocolProvider as the
ClientProtocolProvider
Created table instance for mrtest
Configuring job job_local_0001 with
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
as the submit dir
2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding the
following namenodes' delegation tokens:[hdfs://localhost.localdomain:54525]
2012-01-10 17:51:35,251 WARN  [main] mapreduce.JobSubmitter(139): Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default
FileSystem: file:///
2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job jar
file set.  User classes may not be found. See Job or Job#setJar(String).
2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning
up the staging area
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
PriviledgedActionException as:apurtell (auth:SIMPLE)
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
Stopping mini mapreduce cluster...
Of course that FileNotFoundException is for a file that does indeed exist,
    $ ls -l
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
    -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
I set permissions on all directories from /home/apurtell to the jar to
'rwxr-xr-x' just to make sure there was not some kind of odd permissions
problem.
Any idea what is going on here?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Andrew Purtell
2012-01-11 17:52:26 UTC
Permalink
Hi Mahadev,
Post by Andrew Purtell
mvn -Dhadoop.profile=23 clean test
   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
Yes, but as Hitesh noted on MAPREDUCE-3654, it seems the Yarn client provider is not found or used. (I don't know yet how all of this works...) So the issue with LocalClientProvider not liking the MASTER_ADDRESS set to "test" by the mini Yarn cluster is merely a symptom. Likewise perhaps the FNFE problem.
Post by Andrew Purtell
I can try that on my setup and see what the issue is.
If you could spare a few minutes, that would be very helpful I think. Perhaps our POM needs updating, some artifact is perhaps not being pulled in for testing, and while this was fine for 0.23.0, that is no longer the case for 0.23.1+?


Best regards,


   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Post by Andrew Purtell
________________________________
Sent: Tuesday, January 10, 2012 6:30 PM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Andrew,
Thanks for filing MAPREDUCE-3654. Ill take a look. As for the
filenotfound issue
mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn  branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
I can try that on my setup and see what the issue is.
thanks
mahadev
Post by Andrew Purtell
Hi,
I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but
have run into a couple of issues. Perhaps I'm doing something wrong.
     $ cd /path/to/hadoop
     $ mvn clean install -DskipTests
  - Checked out HBase branch-0.92 and attempted to test under the maven
     $ cd /path/to/hbase
     $ mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153): Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying
org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"test"
PriviledgedActionException as:apurtell (auth:SIMPLE)
cause:java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
Stopping mini mapreduce cluster...
2
I filed MAPREDUCE-3654
(https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be more
going on here, but the patch on that issue let me move forward.
2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked
org.apache.hadoop.mapred.LocalClientProtocolProvider as the
ClientProtocolProvider
Created table instance for mrtest
Configuring job job_local_0001 with
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
as the submit dir
2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding the
following namenodes' delegation tokens:[hdfs://localhost.localdomain:54525]
2012-01-10 17:51:35,251 WARN  [main] mapreduce.JobSubmitter(139): Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default
FileSystem: file:///
2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job jar
file set.  User classes may not be found. See Job or Job#setJar(String).
2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning
up the staging area
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
PriviledgedActionException as:apurtell (auth:SIMPLE)
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
Stopping mini mapreduce cluster...
Of course that FileNotFoundException is for a file that does indeed exist,
    $ ls -l
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
    -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
I set permissions on all directories from /home/apurtell to the jar to
'rwxr-xr-x' just to make sure there was not some kind of odd permissions
problem.
Any idea what is going on here?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Andrew Purtell
2012-01-12 23:23:53 UTC
Permalink
Hi Mahadev,

Was this reproducible?

 
Best regards,


  - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Post by Andrew Purtell
________________________________
Sent: Wednesday, January 11, 2012 9:52 AM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Mahadev,
Post by Andrew Purtell
mvn -Dhadoop.profile=23 clean test
   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
Yes, but as Hitesh noted on MAPREDUCE-3654, it seems the Yarn client provider is not found or used. (I don't know yet how all of this works...) So the issue with LocalClientProvider not liking the MASTER_ADDRESS set to "test" by the mini Yarn cluster is merely a symptom. Likewise perhaps the FNFE problem.
Post by Andrew Purtell
I can try that on my setup and see what the issue is.
If you could spare a few minutes, that would be very helpful I think. Perhaps our POM needs updating, some artifact is perhaps not being
pulled in for testing, and while this was fine for 0.23.0, that is no longer the case for 0.23.1+?
Post by Andrew Purtell
Best regards,
   - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Post by Andrew Purtell
________________________________
Sent: Tuesday, January 10, 2012 6:30 PM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Andrew,
Thanks for filing MAPREDUCE-3654. Ill take a look. As for the
filenotfound issue
mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn  branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
I can try that on my setup and see what the issue is.
thanks
mahadev
Post by Andrew Purtell
Hi,
I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but
have run into a couple of issues. Perhaps I'm doing something wrong.
     $ cd
/path/to/hadoop
Post by Andrew Purtell
Post by Andrew Purtell
Post by Andrew Purtell
     $ mvn clean install -DskipTests
  - Checked out HBase branch-0.92 and attempted to test under the maven
     $ cd /path/to/hbase
     $ mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153): Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying
org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"test"
PriviledgedActionException as:apurtell (auth:SIMPLE)
cause:java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
Stopping mini mapreduce cluster...
2
I filed MAPREDUCE-3654
(https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be more
going on here, but the patch on that issue let me move forward.
2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked
org.apache.hadoop.mapred.LocalClientProtocolProvider as the
Post by Andrew Purtell
Post by Andrew Purtell
Post by Andrew Purtell
ClientProtocolProvider
Created table instance for mrtest
Configuring job job_local_0001 with
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
as the submit dir
2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding the
following namenodes' delegation tokens:[hdfs://localhost.localdomain:54525]
2012-01-10 17:51:35,251 WARN  [main] mapreduce.JobSubmitter(139): Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default
FileSystem: file:///
2012-01-10 17:51:35,261
WARN  [main] mapreduce.JobSubmitter(241): No job jar
Post by Andrew Purtell
Post by Andrew Purtell
Post by Andrew Purtell
file set.  User classes may not be found. See Job or Job#setJar(String).
2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning
up the staging area
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
PriviledgedActionException as:apurtell (auth:SIMPLE)
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
Stopping mini mapreduce cluster...
Of course that FileNotFoundException is for a file that does indeed exist,
    $ ls -l
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
Post by Andrew Purtell
Post by Andrew Purtell
Post by Andrew Purtell
    -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
I set permissions on all directories from /home/apurtell to the jar to
'rwxr-xr-x' just to make sure there was not some kind of odd permissions
problem.
Any idea what is going on here?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Mahadev Konar
2012-01-13 02:02:15 UTC
Permalink
HI Andrew,
Sorry got side tracked with something else. Taking a look now.

mahadev
Post by Andrew Purtell
Hi Mahadev,
Was this reproducible?
Best regards,
  - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
________________________________
Sent: Wednesday, January 11, 2012 9:52 AM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Mahadev,
Post by Andrew Purtell
mvn -Dhadoop.profile=23 clean test
   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
Yes, but as Hitesh noted on MAPREDUCE-3654, it seems the Yarn client
provider is not found or used. (I don't know yet how all of this works...)
So the issue with LocalClientProvider not liking the MASTER_ADDRESS set to
"test" by the mini Yarn cluster is merely a symptom. Likewise perhaps the
FNFE problem.
Post by Andrew Purtell
I can try that on my setup and see what the issue is.
If you could spare a few minutes, that would be very helpful I think.
Perhaps our POM needs updating, some artifact is perhaps not being pulled in
for testing, and while this was fine for 0.23.0, that is no longer the case
for 0.23.1+?
Best regards,
   - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
________________________________
Sent: Tuesday, January 10, 2012 6:30 PM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Andrew,
Thanks for filing MAPREDUCE-3654. Ill take a look. As for the
filenotfound issue
mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn  branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
I can try that on my setup and see what the issue is.
thanks
mahadev
Post by Andrew Purtell
Hi,
I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but
have run into a couple of issues. Perhaps I'm doing something wrong.
     $ cd /path/to/hadoop
     $ mvn clean install -DskipTests
  - Checked out HBase branch-0.92 and attempted to test under the maven
     $ cd /path/to/hbase
     $ mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153): Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying
org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"test"
PriviledgedActionException as:apurtell (auth:SIMPLE)
cause:java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
Stopping mini mapreduce cluster...
2
I filed MAPREDUCE-3654
(https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be more
going on here, but the patch on that issue let me move forward.
2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked
org.apache.hadoop.mapred.LocalClientProtocolProvider as the
ClientProtocolProvider
Created table instance for mrtest
Configuring job job_local_0001 with
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
as the submit dir
2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding the
following namenodes' delegation
tokens:[hdfs://localhost.localdomain:54525]
2012-01-10 17:51:35,251 WARN  [main] mapreduce.JobSubmitter(139): Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default
FileSystem: file:///
2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job jar
file set.  User classes may not be found. See Job or Job#setJar(String).
2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning
up the staging area
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
PriviledgedActionException as:apurtell (auth:SIMPLE)
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
Stopping mini mapreduce cluster...
Of course that FileNotFoundException is for a file that does indeed exist,
    $ ls -l
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
    -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
I set permissions on all directories from /home/apurtell to the jar to
'rwxr-xr-x' just to make sure there was not some kind of odd permissions
problem.
Any idea what is going on here?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Mahadev Konar
2012-01-13 03:53:18 UTC
Permalink
Andrew,
I just checked out 0.92 branch and tried running:

mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce

Looks like a compilation issue:

------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR] /Users/mahadev/workspace/hbase-workspace/hbase-git/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[341,33]
cannot find symbol
[ERROR] symbol : variable dnRegistration
[ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
[ERROR] -> [Help 1]
[ERROR]
--------------

Looks like this commit caused the issue:

=====================
commit 26eb39b424838717cf878332db8c837b02117c5b
HBASE-5163 TestLogRolling#testLogRollOnDatanodeDeath fails
sometimes on Jenkins or hadoop QA
("The directory is already locked.") (N Keywal)
===================

Reverting that patch locally, we have a fix to get the tests working
(thanks to Hitesh). Mostly changes to pom files. Should we open a
HBase jira for the compilation issue? I suppose 0.92 wont be the
target fix or would it?

thanks
mahadev
Post by Mahadev Konar
HI Andrew,
 Sorry got side tracked with something else. Taking a look now.
mahadev
Post by Andrew Purtell
Hi Mahadev,
Was this reproducible?
Best regards,
  - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
________________________________
Sent: Wednesday, January 11, 2012 9:52 AM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Mahadev,
Post by Andrew Purtell
mvn -Dhadoop.profile=23 clean test
   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
Yes, but as Hitesh noted on MAPREDUCE-3654, it seems the Yarn client
provider is not found or used. (I don't know yet how all of this works...)
So the issue with LocalClientProvider not liking the MASTER_ADDRESS set to
"test" by the mini Yarn cluster is merely a symptom. Likewise perhaps the
FNFE problem.
Post by Andrew Purtell
I can try that on my setup and see what the issue is.
If you could spare a few minutes, that would be very helpful I think.
Perhaps our POM needs updating, some artifact is perhaps not being pulled in
for testing, and while this was fine for 0.23.0, that is no longer the case
for 0.23.1+?
Best regards,
   - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
________________________________
Sent: Tuesday, January 10, 2012 6:30 PM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Andrew,
Thanks for filing MAPREDUCE-3654. Ill take a look. As for the
filenotfound issue
mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn  branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
I can try that on my setup and see what the issue is.
thanks
mahadev
Post by Andrew Purtell
Hi,
I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but
have run into a couple of issues. Perhaps I'm doing something wrong.
     $ cd /path/to/hadoop
     $ mvn clean install -DskipTests
  - Checked out HBase branch-0.92 and attempted to test under the maven
     $ cd /path/to/hbase
     $ mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
2012-01-10 18:01:02,489 INFO  [main] mapreduce.TestTableMapReduce(153): Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying
org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"test"
PriviledgedActionException as:apurtell (auth:SIMPLE)
cause:java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
Stopping mini mapreduce cluster...
2
I filed MAPREDUCE-3654
(https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be more
going on here, but the patch on that issue let me move forward.
2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked
org.apache.hadoop.mapred.LocalClientProtocolProvider as the
ClientProtocolProvider
Created table instance for mrtest
Configuring job job_local_0001 with
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
as the submit dir
2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding the
following namenodes' delegation
tokens:[hdfs://localhost.localdomain:54525]
2012-01-10 17:51:35,251 WARN  [main] mapreduce.JobSubmitter(139): Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default
FileSystem: file:///
2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job jar
file set.  User classes may not be found. See Job or Job#setJar(String).
2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning
up the staging area
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
PriviledgedActionException as:apurtell (auth:SIMPLE)
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
Stopping mini mapreduce cluster...
Of course that FileNotFoundException is for a file that does indeed exist,
    $ ls -l
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
    -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
I set permissions on all directories from /home/apurtell to the jar to
'rwxr-xr-x' just to make sure there was not some kind of odd permissions
problem.
Any idea what is going on here?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Andrew Purtell
2012-01-14 19:19:12 UTC
Permalink
Hi Mahadev,


As for the fixes to get the HBase 0.92.x unit tests working with Hadoop 0.23.x, we can fix our POM in 0.92.1, no problem. My thanks to you and Hitesh. I'd like to be running 0.92 under 0.23 on test clusters soon. Successful unit testing is the first step.


Best regards,


    - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Post by Andrew Purtell
________________________________
Sent: Thursday, January 12, 2012 8:10 PM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
I logged https://issues.apache.org/jira/browse/HBASE-5191
I suggest fixing it on HBase TRUNK.
Thanks
Andrew,
Post by Andrew Purtell
mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile
(default-testCompile) on project hbase: Compilation failure
[ERROR] /Users/mahadev/workspace/hbase-workspace/hbase-git/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java:[341,33]
cannot find symbol
[ERROR] symbol  : variable dnRegistration
[ERROR] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
[ERROR] -> [Help 1]
[ERROR]
--------------
=====================
commit 26eb39b424838717cf878332db8c837b02117c5b
   HBASE-5163  TestLogRolling#testLogRollOnDatanodeDeath fails
sometimes on Jenkins or hadoop QA
                  ("The directory is already locked.") (N Keywal)
===================
Reverting that patch locally, we have a fix to get the tests working
(thanks to Hitesh). Mostly changes to pom files. Should we open a
HBase jira for the compilation issue? I suppose 0.92 wont be the
target fix or would it?
thanks
mahadev
Post by Mahadev Konar
HI Andrew,
 Sorry got side tracked with something else. Taking a look now.
mahadev
Post by Andrew Purtell
Hi Mahadev,
Was this reproducible?
Best regards,
  - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
________________________________
Sent: Wednesday, January 11, 2012 9:52 AM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Mahadev,
Post by Andrew Purtell
mvn -Dhadoop.profile=23 clean test
   -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
Yes, but as Hitesh noted on MAPREDUCE-3654, it seems the Yarn client
provider is not found or used. (I don't know yet how all of this works...)
So the issue with LocalClientProvider not liking the MASTER_ADDRESS set to
"test" by the mini Yarn cluster is merely a symptom. Likewise perhaps the
FNFE problem.
Post by Andrew Purtell
I can try that on my setup and see what the issue is.
If you could spare a few minutes, that would be very helpful I think.
Perhaps our POM needs updating, some artifact is perhaps not being pulled in
for testing, and while this was fine for 0.23.0, that is no longer the case
for 0.23.1+?
Best regards,
   - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
________________________________
Sent: Tuesday, January 10, 2012 6:30 PM
Subject: Re: troubles with HBase unit tests using MiniMRCluster on 0.23.1-SNAPSHOT
Hi Andrew,
Thanks for filing MAPREDUCE-3654. Ill take a look. As for the
filenotfound issue
mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
on hbase 0.92. (latest svn  branch)  should be able to recreate this?
(Assuming your patch applied on MR-3654)
I can try that on my setup and see what the issue is.
thanks
mahadev
Post by Andrew Purtell
Hi,
I've been trying to test HBase 0.92 (prerelease) with 0.23.1-SNAPSHOT but
have run into a couple of issues. Perhaps I'm doing something wrong.
  - Checked out Hadoop branch-0.23 and installed artifacts into the local
     $ cd /path/to/hadoop
     $ mvn clean install -DskipTests
  - Checked out HBase branch-0.92 and attempted to test under the maven
     $ cd /path/to/hbase
     $ mvn -Dhadoop.profile=23 clean test
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Started mrtest
2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying
org.apache.hadoop.mapred.LocalClientProtocolProvider
2012-01-10 18:01:02,494 INFO  [main] mapreduce.Cluster(116): Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"test"
PriviledgedActionException as:apurtell (auth:SIMPLE)
cause:java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
Stopping mini mapreduce cluster...
2
I filed MAPREDUCE-3654
(https://issues.apache.org/jira/browse/MAPREDUCE-3654). There could be more
going on here, but the patch on that issue let me move forward.
2012-01-10 17:51:35,225 DEBUG [main] mapreduce.Cluster(106): Picked
org.apache.hadoop.mapred.LocalClientProtocolProvider as the
ClientProtocolProvider
Created table instance for mrtest
Configuring job job_local_0001 with
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
as the submit dir
2012-01-10 17:51:35,251 DEBUG [main] mapreduce.JobSubmitter(565): adding the
following namenodes' delegation
tokens:[hdfs://localhost.localdomain:54525]
2012-01-10 17:51:35,251 WARN  [main] mapreduce.JobSubmitter(139): Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2012-01-10 17:51:35,252 DEBUG [main] mapreduce.JobSubmitter(158): default
FileSystem: file:///
2012-01-10 17:51:35,261 WARN  [main] mapreduce.JobSubmitter(241): No job jar
file set.  User classes may not be found. See Job or Job#setJar(String).
2012-01-10 17:51:35,266 INFO  [main] mapreduce.JobSubmitter(388): Cleaning
up the staging area
file:/tmp/hadoop-apurtell/mapred/staging/apurtell-129099179/.staging/job_local_0001
PriviledgedActionException as:apurtell (auth:SIMPLE)
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
Stopping mini mapreduce cluster...
Of course that FileNotFoundException is for a file that does indeed exist,
    $ ls -l
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
    -rw-rw-r-- 1 apurtell apurtell 764555 2012-01-05 18:56
/home/apurtell/.m2/repository/org/apache/zookeeper/zookeeper/3.4.2/zookeeper-3.4.2.jar
I set permissions on all directories from /home/apurtell to the jar to
'rwxr-xr-x' just to make sure there was not some kind of odd permissions
problem.
Any idea what is going on here?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Loading...