Setting extra classpath and library path for Yarn M/R Application

[Henning Blohm]

Hi,

I am having a hard time figuring out how to add command line options, an
extra class path and library path to a map reduce application running on
Hadoop 2.6.0 (CDH 5.6).

Unfortunately this application is using a native library with a
pre-installed wrapper JAR that is installed on all data nodes. So what I
am trying to achieve is to pass an LD_LIBRARY_PATH and an extra
classpath to the application runner.

Previously, I used

<property>
<name>mapred.child.java.opts</name>
<value>-XX:-UseSplitVerifier -Xmx500M -XX:MaxPermSize=256M
-Dnosql.master=${nosql.master}</value>
</property>

in the client config file for command line options and set

export HADOOP_CLASSPATH=/pathToWrapperJar
export JAVA_LIBRARY_PATH=/nativepath/

in my Hadoop environment. None of this seems to work with Hadoop 2.6.0
(Yarn).

Here is what I have now:

I managed to add to LD_LIBRARY_PATH by specifying

<property>
<name>yarn.app.mapreduce.am.env</name>
<value>LD_LIBRARY_PATH=/nativepath</value>
</property>

<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Da=1 -Db=2</value>
</property>

in the client config file. I can tell that by cat'ing the
launch_container.sh while it is present.

I do however fail to add anything to the application's class path. Plus
I would rather not specify the native library path and wrapper jar in
the client config but rather globally.

It would be great, if somebody can give me a hint first and foremost on
the class path question and secondly some orientation on whether using
the new configuration properties is the way to go.

Thanks,
Henning