Discussion:
NameNode low on available disk space
(too old to reply)
Andy Isaacson
2013-01-22 21:08:15 UTC
Permalink
Moving from general@ to ***@. The general list is not for technical
questions, it's to discuss project-wide issues.

On Tue, Jan 22, 2013 at 1:03 PM, Mohit Vadhera
Namenode switches into safemode when it has low disk space on the root fs /
i have to manually run a command to leave it. Below are log messages for
low space on root / fs. Is there any parameter so that i can reduce
reserved amount.
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode is ON.
If your Namenode filesystem really only has 10MB available, you are
running well into a red zone for data loss. You cannot safely operate
a NN when the filesystem is that close to being out of space and going
to safemode is appropriate.

The default reserve of 100MB is a good margin of safety for a small
cluster. Instead of decreasing your safety margin, fix your machine so
that there is more than 100MB available on the NN filesystem.

-andy
Mohit Vadhera
2013-01-23 03:43:50 UTC
Permalink
Ok Steve. I am forwarding my issue again to the list that you said. The
version is

Hi,

Namenode switches into safemode when it has low disk space on the root fs /
i have to manually run a command to leave it. Below are log messages for
low space on root / fs. Is there any parameter so that i can reduce
reserved amount.Hadoop 2.0.0-cdh4.1.2

2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode is ON.
Harsh J
2013-01-23 04:42:33 UTC
Permalink
Edit your hdfs-site.xml (or whatever place of config your NN uses) to lower
the value of property "dfs.namenode.resource.du.reserved". Create a new
property if one does not exist, and set the value of space to a suitable
level. The default itself is pretty low - 100 MB in bytes.
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you said. The
version is
Hi,
Namenode switches into safemode when it has low disk space on the root fs
/ i have to manually run a command to leave it. Below are log messages for
low space on root / fs. Is there any parameter so that i can reduce
reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode is ON.
--
Harsh J
Harsh J
2013-01-23 05:43:58 UTC
Permalink
Hi again,

Yes, you need to add it to hdfs-site.xml and restart the NN.
Thanks Harsh, Do I need to add parameters in hdfs-site.xml and restart
service namenode.
+ public static final String DFS_NAMENODE_DU_RESERVED_KEY =
"dfs.namenode.resource.du.
reserved";
+ public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT = 1024 *
1024 * 100; // 100 MB
Edit your hdfs-site.xml (or whatever place of config your NN uses) to
lower the value of property "dfs.namenode.resource.du.reserved". Create a
new property if one does not exist, and set the value of space to a
suitable level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you said. The
version is
Hi,
Namenode switches into safemode when it has low disk space on the root fs
/ i have to manually run a command to leave it. Below are log messages for
low space on root / fs. Is there any parameter so that i can reduce
reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode is ON.
--
Harsh J
--
Harsh J
Mohit Vadhera
2013-01-23 06:14:10 UTC
Permalink
Thanks Guys, As you said the level is already pretty low i.e 100 MB but in
my case the root fs / has 14 G available. What can be the root cause then ?

/dev/mapper/vg_operamast1-lv_root
50G 33G 14G 71% /

As per logs.
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
is below the configured reserved amount 104857600
Post by Harsh J
Hi again,
Yes, you need to add it to hdfs-site.xml and restart the NN.
Thanks Harsh, Do I need to add parameters in hdfs-site.xml and restart
service namenode.
+ public static final String DFS_NAMENODE_DU_RESERVED_KEY =
"dfs.namenode.resource.du.
reserved";
+ public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT = 1024 *
1024 * 100; // 100 MB
Edit your hdfs-site.xml (or whatever place of config your NN uses) to
lower the value of property "dfs.namenode.resource.du.reserved". Create a
new property if one does not exist, and set the value of space to a
suitable level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you said. The
version is
Hi,
Namenode switches into safemode when it has low disk space on the root
fs / i have to manually run a command to leave it. Below are log messages
for low space on root / fs. Is there any parameter so that i can reduce
reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode is ON.
On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
--
Harsh J
--
Harsh J
Mohit Vadhera
2013-01-23 15:31:36 UTC
Permalink
Can somebody answer me on this plz ?


On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
Post by Mohit Vadhera
Thanks Guys, As you said the level is already pretty low i.e 100 MB but in
my case the root fs / has 14 G available. What can be the root cause then ?
/dev/mapper/vg_operamast1-lv_root
50G 33G 14G 71% /
As per logs.
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
is below the configured reserved amount 104857600
Post by Harsh J
Hi again,
Yes, you need to add it to hdfs-site.xml and restart the NN.
Thanks Harsh, Do I need to add parameters in hdfs-site.xml and restart
service namenode.
+ public static final String DFS_NAMENODE_DU_RESERVED_KEY =
"dfs.namenode.resource.du.
reserved";
+ public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT = 1024
* 1024 * 100; // 100 MB
Edit your hdfs-site.xml (or whatever place of config your NN uses) to
lower the value of property "dfs.namenode.resource.du.reserved". Create a
new property if one does not exist, and set the value of space to a
suitable level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you said. The
version is
Hi,
Namenode switches into safemode when it has low disk space on the root
fs / i have to manually run a command to leave it. Below are log messages
for low space on root / fs. Is there any parameter so that i can reduce
reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
2013-01-21 01:22:52,218 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode is ON.
On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
--
Harsh J
--
Harsh J
Harsh J
2013-01-23 15:40:23 UTC
Permalink
Mohit,

When do you specifically get the error at the NN? Does your NN consistently
not start with that error?

Your local disk space availability can certainly fluctuate if you use the
same disk for MR and other activity which creates temporary files.
Post by Mohit Vadhera
Can somebody answer me on this plz ?
On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
Post by Mohit Vadhera
Thanks Guys, As you said the level is already pretty low i.e 100 MB but
in my case the root fs / has 14 G available. What can be the root cause
then ?
/dev/mapper/vg_operamast1-lv_root
50G 33G 14G 71% /
As per logs.
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
is below the configured reserved amount 104857600
Post by Harsh J
Hi again,
Yes, you need to add it to hdfs-site.xml and restart the NN.
Thanks Harsh, Do I need to add parameters in hdfs-site.xml and restart
service namenode.
+ public static final String DFS_NAMENODE_DU_RESERVED_KEY =
"dfs.namenode.resource.du.
reserved";
+ public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT = 1024
* 1024 * 100; // 100 MB
Edit your hdfs-site.xml (or whatever place of config your NN uses) to
lower the value of property "dfs.namenode.resource.du.reserved". Create a
new property if one does not exist, and set the value of space to a
suitable level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you said.
The version is
Hi,
Namenode switches into safemode when it has low disk space on the root
fs / i have to manually run a command to leave it. Below are log messages
for low space on root / fs. Is there any parameter so that i can reduce
reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
STATE* Safe mode is ON.
On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
--
Harsh J
--
Harsh J
--
Harsh J
Mohit Vadhera
2013-01-23 15:54:17 UTC
Permalink
NN switches randomly into the safemode then I run command to leave safemode
manually. I never got alerts for low disk space on machine level and i
didn't see the space fluctuates GBs into MBs .
Post by Harsh J
Mohit,
When do you specifically get the error at the NN? Does your NN
consistently not start with that error?
Your local disk space availability can certainly fluctuate if you use the
same disk for MR and other activity which creates temporary files.
On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <
Post by Mohit Vadhera
Can somebody answer me on this plz ?
On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
Post by Mohit Vadhera
Thanks Guys, As you said the level is already pretty low i.e 100 MB but
in my case the root fs / has 14 G available. What can be the root cause
then ?
/dev/mapper/vg_operamast1-lv_root
50G 33G 14G 71% /
As per logs.
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
is below the configured reserved amount 104857600
Post by Harsh J
Hi again,
Yes, you need to add it to hdfs-site.xml and restart the NN.
Thanks Harsh, Do I need to add parameters in hdfs-site.xml and
restart service namenode.
+ public static final String DFS_NAMENODE_DU_RESERVED_KEY =
"dfs.namenode.resource.du.
reserved";
+ public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT =
1024 * 1024 * 100; // 100 MB
Edit your hdfs-site.xml (or whatever place of config your NN uses) to
lower the value of property "dfs.namenode.resource.du.reserved". Create a
new property if one does not exist, and set the value of space to a
suitable level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you said.
The version is
Hi,
Namenode switches into safemode when it has low disk space on the
root fs / i have to manually run a command to leave it. Below are log
messages for low space on root / fs. Is there any parameter so that i can
reduce reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
STATE* Safe mode is ON.
On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
--
Harsh J
--
Harsh J
--
Harsh J
Harsh J
2013-01-23 16:42:54 UTC
Permalink
A random switching behavior can only be explained by a fluctuating disk
space I'd think. Are you running MR operations on the same disk (i.e. is it
part of mapred.local.dir as well)?
Post by Mohit Vadhera
NN switches randomly into the safemode then I run command to leave
safemode manually. I never got alerts for low disk space on machine level
and i didn't see the space fluctuates GBs into MBs .
Post by Harsh J
Mohit,
When do you specifically get the error at the NN? Does your NN
consistently not start with that error?
Your local disk space availability can certainly fluctuate if you use the
same disk for MR and other activity which creates temporary files.
On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <
Post by Mohit Vadhera
Can somebody answer me on this plz ?
On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
Post by Mohit Vadhera
Thanks Guys, As you said the level is already pretty low i.e 100 MB but
in my case the root fs / has 14 G available. What can be the root cause
then ?
/dev/mapper/vg_operamast1-lv_root
50G 33G 14G 71% /
As per logs.
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
is below the configured reserved amount 104857600
Post by Harsh J
Hi again,
Yes, you need to add it to hdfs-site.xml and restart the NN.
Thanks Harsh, Do I need to add parameters in hdfs-site.xml and
restart service namenode.
+ public static final String DFS_NAMENODE_DU_RESERVED_KEY =
"dfs.namenode.resource.du.
reserved";
+ public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT =
1024 * 1024 * 100; // 100 MB
Edit your hdfs-site.xml (or whatever place of config your NN uses) to
lower the value of property "dfs.namenode.resource.du.reserved". Create a
new property if one does not exist, and set the value of space to a
suitable level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you said.
The version is
Hi,
Namenode switches into safemode when it has low disk space on the
root fs / i have to manually run a command to leave it. Below are log
messages for low space on root / fs. Is there any parameter so that i can
reduce reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
STATE* Safe mode is ON.
On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
--
Harsh J
--
Harsh J
--
Harsh J
--
Harsh J
Mohit Vadhera
2013-01-23 16:55:24 UTC
Permalink
MR operation are running on the same machine. i checked the parameter "
mapred.local.dir" in my installed directory /etc/hadoop/ but didn't find .
One question the disk space reserved size displayed in logs in KB or MB ?
I am layman on hadoop. The link I followed to install is given below

https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode

Thanks,
Post by Harsh J
A random switching behavior can only be explained by a fluctuating disk
space I'd think. Are you running MR operations on the same disk (i.e. is it
part of mapred.local.dir as well)?
On Wed, Jan 23, 2013 at 9:24 PM, Mohit Vadhera <
Post by Mohit Vadhera
NN switches randomly into the safemode then I run command to leave
safemode manually. I never got alerts for low disk space on machine level
and i didn't see the space fluctuates GBs into MBs .
Post by Harsh J
Mohit,
When do you specifically get the error at the NN? Does your NN
consistently not start with that error?
Your local disk space availability can certainly fluctuate if you use
the same disk for MR and other activity which creates temporary files.
On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <
Post by Mohit Vadhera
Can somebody answer me on this plz ?
On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
Post by Mohit Vadhera
Thanks Guys, As you said the level is already pretty low i.e 100 MB
but in my case the root fs / has 14 G available. What can be the root
cause then ?
/dev/mapper/vg_operamast1-lv_root
50G 33G 14G 71% /
As per logs.
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
is below the configured reserved amount 104857600
Post by Harsh J
Hi again,
Yes, you need to add it to hdfs-site.xml and restart the NN.
Thanks Harsh, Do I need to add parameters in hdfs-site.xml and
restart service namenode.
+ public static final String DFS_NAMENODE_DU_RESERVED_KEY =
"dfs.namenode.resource.du.
reserved";
+ public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT =
1024 * 1024 * 100; // 100 MB
Edit your hdfs-site.xml (or whatever place of config your NN uses)
to lower the value of property "dfs.namenode.resource.du.reserved". Create
a new property if one does not exist, and set the value of space to a
suitable level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you said.
The version is
Hi,
Namenode switches into safemode when it has low disk space on the
root fs / i have to manually run a command to leave it. Below are log
messages for low space on root / fs. Is there any parameter so that i can
reduce reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
STATE* Safe mode is ON.
On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
--
Harsh J
--
Harsh J
--
Harsh J
--
Harsh J
Harsh J
2013-01-23 16:58:35 UTC
Permalink
The logs display it in simple bytes. If the issue begins to occur when you
start using Hadoop, then its most certainly MR using up the disk space
temporarily.

You could lower the threshold, or you could perhaps use a bigger disk for
your trials/more nodes.


On Wed, Jan 23, 2013 at 10:25 PM, Mohit Vadhera <
Post by Mohit Vadhera
MR operation are running on the same machine. i checked the parameter "
mapred.local.dir" in my installed directory /etc/hadoop/ but didn't find .
One question the disk space reserved size displayed in logs in KB or MB ?
I am layman on hadoop. The link I followed to install is given below
https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode
Thanks,
Post by Harsh J
A random switching behavior can only be explained by a fluctuating disk
space I'd think. Are you running MR operations on the same disk (i.e. is it
part of mapred.local.dir as well)?
On Wed, Jan 23, 2013 at 9:24 PM, Mohit Vadhera <
Post by Mohit Vadhera
NN switches randomly into the safemode then I run command to leave
safemode manually. I never got alerts for low disk space on machine level
and i didn't see the space fluctuates GBs into MBs .
Post by Harsh J
Mohit,
When do you specifically get the error at the NN? Does your NN
consistently not start with that error?
Your local disk space availability can certainly fluctuate if you use
the same disk for MR and other activity which creates temporary files.
On Wed, Jan 23, 2013 at 9:01 PM, Mohit Vadhera <
Post by Mohit Vadhera
Can somebody answer me on this plz ?
On Wed, Jan 23, 2013 at 11:44 AM, Mohit Vadhera <
Post by Mohit Vadhera
Thanks Guys, As you said the level is already pretty low i.e 100 MB
but in my case the root fs / has 14 G available. What can be the root
cause then ?
/dev/mapper/vg_operamast1-lv_root
50G 33G 14G 71% /
As per logs.
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_operamast1-lv_root' is 10653696, which
is below the configured reserved amount 104857600
Post by Harsh J
Hi again,
Yes, you need to add it to hdfs-site.xml and restart the NN.
Thanks Harsh, Do I need to add parameters in hdfs-site.xml and
restart service namenode.
+ public static final String DFS_NAMENODE_DU_RESERVED_KEY =
"dfs.namenode.resource.du.
reserved";
+ public static final long DFS_NAMENODE_DU_RESERVED_DEFAULT =
1024 * 1024 * 100; // 100 MB
Edit your hdfs-site.xml (or whatever place of config your NN uses)
to lower the value of property "dfs.namenode.resource.du.reserved". Create
a new property if one does not exist, and set the value of space to a
suitable level. The default itself is pretty low - 100 MB in bytes.
On Wed, Jan 23, 2013 at 9:13 AM, Mohit Vadhera <
Post by Mohit Vadhera
Ok Steve. I am forwarding my issue again to the list that you
said. The version is
Hi,
Namenode switches into safemode when it has low disk space on the
root fs / i have to manually run a command to leave it. Below are log
messages for low space on root / fs. Is there any parameter so that i can
reduce reserved amount.Hadoop 2.0.0-cdh4.1.2
2013-01-21 01:22:52,217 WARN
org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space
available on volume '/dev/mapper/vg_lv_root' is 10653696, which is below
the configured reserved amount 104857600
2013-01-21 01:22:52,218 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on
available disk space. Entering safe mode.
STATE* Safe mode is ON.
On Wed, Jan 23, 2013 at 2:50 AM, Steve Loughran <
--
Harsh J
--
Harsh J
--
Harsh J
--
Harsh J
--
Harsh J
Harsh J
2013-01-23 17:11:41 UTC
Permalink
On Wed, Jan 23, 2013 at 10:37 PM, Mohit Vadhera <
51200
51200 *bytes* is 50 KB. 50 MB is 50*1024*1024, which is 52428800. You can
verify changes to config by visiting the http://NNHOST:50070/conf page and
searching for the config key name to see if the NN has picked it up.
--
Harsh J
Mohit Vadhera
2013-01-23 17:45:06 UTC
Permalink
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For verifying i
checked the http link that you gave and i saw the property their but on
http://NNHOST:50070 <http://NNHOST:50070/conf> i noticed warning( *WARNING
: There are 4 missing blocks. Please check the logs or run fsck in order to
identify the missing blocks.* <http://172.20.3.119:50070/corrupt_files.jsp>)
when i clicked on this link i can see file names . Do I need to reboot the
machine to run fsck on root fs/ or is there hadoop command fsck that i can
run on the running hadoop ?

Thanks,
Harsh J
2013-02-28 08:29:26 UTC
Permalink
Hi,

The exact error is displayed on your log and should be somewhat self
explanatory:

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.

Please check this one's availability, permissions (the NN user should
be able to access it).

On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
Please find below logs for shutting down the namenode service. Can anybody
check this
2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 02:07:51,910 INFO
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
more than 10 times
2013-02-28 02:07:52,466 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.extension = 0
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory
/mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 02:08:48,913 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/192.168.1.3
On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
Hi Guys,
I have space on other partition. Can I change the path for cache files on
other partition ? I have below properties . Can it resolve the issue ? If i
change the path to other directories and restart services I get the below
error while starting the service namenode. I didn't find anything in logs so
far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
Hadoop datanode is running [ OK ]
Hadoop namenode is dead and pid file exists [FAILED]
Hadoop secondarynamenode is running [ OK ]
Thanks,
On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For verifying
i checked the http link that you gave and i saw the property their but on
http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
blocks. Please check the logs or run fsck in order to identify the missing
blocks.) when i clicked on this link i can see file names . Do I need to
reboot the machine to run fsck on root fs/ or is there hadoop command fsck
that i can run on the running hadoop ?
Thanks,
--
Harsh J
Mohit Vadhera
2013-02-28 10:58:13 UTC
Permalink
Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and
set permission . when i restart hadoop services. It removes and I find
below logs.

Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?

2013-02-28 05:57:50,902 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
81133-2zbGk5zsN3cvsa84sZI4N06D/***@public.gmane.org
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.

Command to format the NN.

sudo -u hdfs hdfs namenode -format

Thanks,
After creating the directory and setting permission I tried to restart the
services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
being started.
Need to check few logs from below logs.
===================================
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN
tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 05:06:26,015 INFO
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.extension = 0
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 05:06:26,394 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
If I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
Post by Harsh J
Hi,
The exact error is displayed on your log and should be somewhat self
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.
Please check this one's availability, permissions (the NN user should
be able to access it).
On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
Please find below logs for shutting down the namenode service. Can
anybody
check this
2013-02-28 02:07:51,752 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due
to lack
of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
maxReplication
= 512
2013-02-28 02:07:51,910 INFO
minReplication
= 1
2013-02-28 02:07:51,910 INFO
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
hadmin
2013-02-28 02:07:51,920 INFO
isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring
more than 10 times
2013-02-28 02:07:52,466 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics
system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
join
Directory
/mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
with
status 1
2013-02-28 02:08:48,913 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
192.168.1.3
On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
Hi Guys,
I have space on other partition. Can I change the path for cache
files on
other partition ? I have below properties . Can it resolve the issue
? If i
change the path to other directories and restart services I get the
below
error while starting the service namenode. I didn't find anything in
logs so
far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/namesecondary</value>
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
done
Hadoop datanode is running [ OK ]
Hadoop namenode is dead and pid file exists [FAILED]
Hadoop secondarynamenode is running [ OK ]
Thanks,
On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For
verifying
Post by Mohit Vadhera
i checked the http link that you gave and i saw the property their
but on
Post by Mohit Vadhera
http://NNHOST:50070 i noticed warning( WARNING : There are 4
missing
Post by Mohit Vadhera
blocks. Please check the logs or run fsck in order to identify the
missing
Post by Mohit Vadhera
blocks.) when i clicked on this link i can see file names . Do I
need to
Post by Mohit Vadhera
reboot the machine to run fsck on root fs/ or is there hadoop
command fsck
Post by Mohit Vadhera
that i can run on the running hadoop ?
Thanks,
--
Harsh J
shashwat shriparv
2013-02-28 12:25:04 UTC
Permalink
mpl: Stopping NameNo
Please check how disk space uses is fluctuation while running MR Jobs.



∞
Shashwat Shriparv
shashwat shriparv
2013-02-28 12:29:51 UTC
Permalink
And alos check for this parameter

dfs.namenode.safemode.threshold-pct

check the file system if health status is correct.

and if underreplication is there??



∞
Shashwat Shriparv



On Thu, Feb 28, 2013 at 5:55 PM, shashwat shriparv <
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera <
mpl: Stopping NameNo
Please check how disk space uses is fluctuation while running MR Jobs.
∞
Shashwat Shriparv
Mohit Vadhera
2013-03-01 06:41:56 UTC
Permalink
Hi,

May i know the answer plz ?

Thanks,
Post by Mohit Vadhera
Even i created the
file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock and set permission .
when i restart hadoop services. It removes and I find below logs.
Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
Command to format the NN.
sudo -u hdfs hdfs namenode -format
Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera <
After creating the directory and setting permission I tried to restart
the services and i get error
"/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
Need to check few logs from below logs.
===================================
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN
tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 05:06:26,015 INFO
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO
org.apache.hadoop.hdfs.server.common.Storage: Lock on
/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
with status 1
2013-02-28 05:06:26,394 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera <
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created
. If I do compare with the older path the permissions are same on the
parent directories.
Do I need to create this this directory manually and set the permission ?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
Post by Harsh J
Hi,
The exact error is displayed on your log and should be somewhat self
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.
Please check this one's availability, permissions (the NN user should
be able to access it).
On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
Please find below logs for shutting down the namenode service. Can
anybody
check this
2013-02-28 02:07:51,752 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due
to lack
of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
maxReplication
= 512
2013-02-28 02:07:51,910 INFO
minReplication
= 1
2013-02-28 02:07:51,910 INFO
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
hadmin
2013-02-28 02:07:51,920 INFO
isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring
more than 10 times
2013-02-28 02:07:52,466 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics
system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
namenode join
Directory
storage
directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
with
status 1
2013-02-28 02:08:48,913 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
192.168.1.3
On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
Hi Guys,
I have space on other partition. Can I change the path for cache
files on
other partition ? I have below properties . Can it resolve the issue
? If i
change the path to other directories and restart services I get the
below
error while starting the service namenode. I didn't find anything in
logs so
far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/namesecondary</value>
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
status; done
Hadoop datanode is running [ OK ]
Hadoop namenode is dead and pid file exists [FAILED]
Hadoop secondarynamenode is running [ OK ]
Thanks,
On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For
verifying
Post by Mohit Vadhera
i checked the http link that you gave and i saw the property their
but on
Post by Mohit Vadhera
http://NNHOST:50070 i noticed warning( WARNING : There are 4
missing
Post by Mohit Vadhera
blocks. Please check the logs or run fsck in order to identify the
missing
Post by Mohit Vadhera
blocks.) when i clicked on this link i can see file names . Do I
need to
Post by Mohit Vadhera
reboot the machine to run fsck on root fs/ or is there hadoop
command fsck
Post by Mohit Vadhera
that i can run on the running hadoop ?
Thanks,
--
Harsh J
Harsh J
2013-03-01 06:44:35 UTC
Permalink
I believe I already responded to another one of your multiple threads
on this problem, that you have to move contents of the old dir to the
new dir and not format. Thats how one changes the dfs.name.dir
properly.

On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
Post by Mohit Vadhera
Hi,
May i know the answer plz ?
Thanks,
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
Post by Mohit Vadhera
Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
and set permission . when i restart hadoop services. It removes and I find
below logs.
Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
Command to format the NN.
sudo -u hdfs hdfs namenode -format
Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
After creating the directory and setting permission I tried to restart
the services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
being started.
Need to check few logs from below logs.
===================================
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN
tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 05:06:26,015 INFO
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
more than 10 times
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO
org.apache.hadoop.hdfs.server.common.Storage: Lock on
/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
with status 1
2013-02-28 05:06:26,394 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
If I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
Post by Harsh J
Hi,
The exact error is displayed on your log and should be somewhat self
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.
Please check this one's availability, permissions (the NN user should
be able to access it).
On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
Please find below logs for shutting down the namenode service. Can anybody
check this
2013-02-28 02:07:51,752 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 02:07:51,910 INFO
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
hadmin
2013-02-28 02:07:51,920 INFO
isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
more than 10 times
2013-02-28 02:07:52,466 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory
/mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 02:08:48,913 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at
OPERA-MAST1.ny.os.local/192.168.1.3
On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
Hi Guys,
I have space on other partition. Can I change the path for cache files on
other partition ? I have below properties . Can it resolve the issue ? If i
change the path to other directories and restart services I get the below
error while starting the service namenode. I didn't find anything in logs so
far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
Hadoop datanode is running [ OK ]
Hadoop namenode is dead and pid file exists [FAILED]
Hadoop secondarynamenode is running [ OK ]
Thanks,
On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For verifying
i checked the http link that you gave and i saw the property their but on
http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
blocks. Please check the logs or run fsck in order to identify the missing
blocks.) when i clicked on this link i can see file names . Do I need to
reboot the machine to run fsck on root fs/ or is there hadoop command fsck
that i can run on the running hadoop ?
Thanks,
--
Harsh J
--
Harsh J
Harsh J
2013-03-01 07:54:26 UTC
Permalink
Actually, I did: http://search-hadoop.com/m/qvn7EhQQHm1. Perhaps you aren't
subscribed to the lists and wouldn't have received anything sent back.


On Fri, Mar 1, 2013 at 1:21 PM, Mohit Vadhera
thanks Harsh but you didn't answer on this before, I will try to move old
directory "name" to new location and restart services . Hope it will not
loose any data .
old Location
$ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary
New location
$ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
total 8
drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary
Thanks,
Post by Harsh J
I believe I already responded to another one of your multiple threads
on this problem, that you have to move contents of the old dir to the
new dir and not format. Thats how one changes the dfs.name.dir
properly.
On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
Post by Mohit Vadhera
Hi,
May i know the answer plz ?
Thanks,
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
Post by Mohit Vadhera
Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
and set permission . when i restart hadoop services. It removes and I
find
Post by Mohit Vadhera
Post by Mohit Vadhera
below logs.
Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?
2013-02-28 05:57:50,902 INFO
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
nodename
Post by Mohit Vadhera
Post by Mohit Vadhera
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics
Post by Mohit Vadhera
Post by Mohit Vadhera
system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
join
Post by Mohit Vadhera
Post by Mohit Vadhera
java.io.IOException: NameNode is not formatted.
Command to format the NN.
sudo -u hdfs hdfs namenode -format
Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
After creating the directory and setting permission I tried to restart
the services and i get error
"/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
not
Post by Mohit Vadhera
Post by Mohit Vadhera
being started.
Need to check few logs from below logs.
===================================
2013-02-28 05:06:24,905 WARN
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
in
Post by Mohit Vadhera
Post by Mohit Vadhera
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
in
Post by Mohit Vadhera
Post by Mohit Vadhera
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
storage
Post by Mohit Vadhera
Post by Mohit Vadhera
directory (dfs.namenode.name.dir) configured. Beware of dataloss due
to lack
Post by Mohit Vadhera
Post by Mohit Vadhera
of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
namespace
Post by Mohit Vadhera
Post by Mohit Vadhera
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN
org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period
Post by Mohit Vadhera
Post by Mohit Vadhera
at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
started
2013-02-28 05:06:24,905 WARN
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
in
Post by Mohit Vadhera
Post by Mohit Vadhera
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI
in
Post by Mohit Vadhera
Post by Mohit Vadhera
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
storage
Post by Mohit Vadhera
Post by Mohit Vadhera
directory (dfs.namenode.name.dir) configured. Beware of dataloss due
to lack
Post by Mohit Vadhera
Post by Mohit Vadhera
of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
namespace
Post by Mohit Vadhera
Post by Mohit Vadhera
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
maxReplication
Post by Mohit Vadhera
Post by Mohit Vadhera
= 512
2013-02-28 05:06:26,015 INFO
minReplication
Post by Mohit Vadhera
Post by Mohit Vadhera
= 1
2013-02-28 05:06:26,015 INFO
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
Post by Mohit Vadhera
Post by Mohit Vadhera
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
Post by Mohit Vadhera
Post by Mohit Vadhera
hadmin
2013-02-28 05:06:26,022 INFO
isPermissionEnabled =
Post by Mohit Vadhera
Post by Mohit Vadhera
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
true
Post by Mohit Vadhera
Post by Mohit Vadhera
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring
Post by Mohit Vadhera
Post by Mohit Vadhera
more than 10 times
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO
org.apache.hadoop.hdfs.server.common.Storage: Lock on
/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics
Post by Mohit Vadhera
Post by Mohit Vadhera
system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
namenode join
Post by Mohit Vadhera
Post by Mohit Vadhera
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
Post by Mohit Vadhera
Post by Mohit Vadhera
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
with status 1
2013-02-28 05:06:26,394 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
created .
Post by Mohit Vadhera
Post by Mohit Vadhera
If I do compare with the older path the permissions are same on the
parent
Post by Mohit Vadhera
Post by Mohit Vadhera
directories.
Do I need to create this this directory manually and set the
permission
Post by Mohit Vadhera
Post by Mohit Vadhera
?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
Post by Harsh J
Hi,
The exact error is displayed on your log and should be somewhat self
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.
Please check this one's availability, permissions (the NN user
should
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
be able to access it).
On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
Please find below logs for shutting down the namenode service. Can
anybody
check this
2013-02-28 02:07:51,752 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
image
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss
due
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
to lack
of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured.
Beware
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
of
dataloss due to lack of redundant storage directories!
2013-02-28 02:07:51,884 INFO
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
maxReplication
= 512
2013-02-28 02:07:51,910 INFO
minReplication
= 1
2013-02-28 02:07:51,910 INFO
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
hadmin
2013-02-28 02:07:51,920 INFO
isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
names
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
occuring
more than 10 times
2013-02-28 02:07:52,466 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
exist.
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
NameNode
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
metrics
system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
metrics
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode
metrics
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
namenode join
Directory
/mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
Exiting
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
with
status 1
2013-02-28 02:08:48,913 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at
OPERA-MAST1.ny.os.local/192.168.1.3
On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
Hi Guys,
I have space on other partition. Can I change the path for cache
files on
other partition ? I have below properties . Can it resolve the
issue
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
? If i
change the path to other directories and restart services I get
the
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
below
error while starting the service namenode. I didn't find
anything in
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
logs so
far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/name</value>
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/namesecondary</value>
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
status; done
Hadoop datanode is running [ OK
]
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Hadoop namenode is dead and pid file exists
[FAILED]
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Hadoop secondarynamenode is running [ OK
]
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Thanks,
On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For
verifying
i checked the http link that you gave and i saw the property
their
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Post by Mohit Vadhera
but on
http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
blocks. Please check the logs or run fsck in order to identify
the
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Post by Mohit Vadhera
missing
blocks.) when i clicked on this link i can see file names .
Do I
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Post by Mohit Vadhera
need to
reboot the machine to run fsck on root fs/ or is there hadoop
command fsck
that i can run on the running hadoop ?
Thanks,
--
Harsh J
--
Harsh J
--
Harsh J
Mohit Vadhera
2013-03-01 07:51:18 UTC
Permalink
thanks Harsh but you didn't answer on this before, I will try to move old
directory "name" to new location and restart services . Hope it will not
loose any data .

old Location

$ sudo ls -l /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 11:28 namesecondary


New location

$ sudo ls -l /mnt/san1/hdfs/hdfs/dfs/
total 8
drwx------. 3 hdfs hdfs 4096 Feb 28 11:28 data
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 11:28 namesecondary


Thanks,
Post by Harsh J
I believe I already responded to another one of your multiple threads
on this problem, that you have to move contents of the old dir to the
new dir and not format. Thats how one changes the dfs.name.dir
properly.
On Fri, Mar 1, 2013 at 12:11 PM, Mohit Vadhera
Post by Mohit Vadhera
Hi,
May i know the answer plz ?
Thanks,
On Thu, Feb 28, 2013 at 4:28 PM, Mohit Vadhera
Post by Mohit Vadhera
Even i created the file /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
and set permission . when i restart hadoop services. It removes and I
find
Post by Mohit Vadhera
Post by Mohit Vadhera
below logs.
Do I need to format the NN?
Below is the command to format the NN ?
Any kind of loss while formatting ?
Is there any way to avoid formatting and change the cache path ?
2013-02-28 05:57:50,902 INFO
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
nodename
Post by Mohit Vadhera
Post by Mohit Vadhera
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics
Post by Mohit Vadhera
Post by Mohit Vadhera
system...
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
stopped.
2013-02-28 05:57:50,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
shutdown complete.
2013-02-28 05:57:50,905 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
join
Post by Mohit Vadhera
Post by Mohit Vadhera
java.io.IOException: NameNode is not formatted.
Command to format the NN.
sudo -u hdfs hdfs namenode -format
Thanks,
On Thu, Feb 28, 2013 at 3:47 PM, Mohit Vadhera
After creating the directory and setting permission I tried to restart
the services and i get error
"/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
not
Post by Mohit Vadhera
Post by Mohit Vadhera
being started.
Need to check few logs from below logs.
===================================
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
storage
Post by Mohit Vadhera
Post by Mohit Vadhera
directory (dfs.namenode.name.dir) configured. Beware of dataloss due
to lack
Post by Mohit Vadhera
Post by Mohit Vadhera
of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
************************************************************/
2013-02-28 05:06:23,385 WARN
org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate
tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period
Post by Mohit Vadhera
Post by Mohit Vadhera
at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
started
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
storage
Post by Mohit Vadhera
Post by Mohit Vadhera
directory (dfs.namenode.name.dir) configured. Beware of dataloss due
to lack
Post by Mohit Vadhera
Post by Mohit Vadhera
of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
maxReplication
Post by Mohit Vadhera
Post by Mohit Vadhera
= 512
2013-02-28 05:06:26,015 INFO
minReplication
Post by Mohit Vadhera
Post by Mohit Vadhera
= 1
2013-02-28 05:06:26,015 INFO
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
Post by Mohit Vadhera
Post by Mohit Vadhera
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
Post by Mohit Vadhera
Post by Mohit Vadhera
hadmin
2013-02-28 05:06:26,022 INFO
isPermissionEnabled =
Post by Mohit Vadhera
Post by Mohit Vadhera
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
true
Post by Mohit Vadhera
Post by Mohit Vadhera
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring
Post by Mohit Vadhera
Post by Mohit Vadhera
more than 10 times
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO
org.apache.hadoop.hdfs.server.common.Storage: Lock on
/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics
Post by Mohit Vadhera
Post by Mohit Vadhera
system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
Post by Mohit Vadhera
Post by Mohit Vadhera
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
join
Post by Mohit Vadhera
Post by Mohit Vadhera
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
Post by Mohit Vadhera
Post by Mohit Vadhera
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
Post by Mohit Vadhera
Post by Mohit Vadhera
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting
with status 1
2013-02-28 05:06:26,394 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
On Thu, Feb 28, 2013 at 3:18 PM, Mohit Vadhera
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being
created .
Post by Mohit Vadhera
Post by Mohit Vadhera
If I do compare with the older path the permissions are same on the
parent
Post by Mohit Vadhera
Post by Mohit Vadhera
directories.
Do I need to create this this directory manually and set the
permission
Post by Mohit Vadhera
Post by Mohit Vadhera
?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
Post by Harsh J
Hi,
The exact error is displayed on your log and should be somewhat self
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.
Please check this one's availability, permissions (the NN user should
be able to access it).
On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
Please find below logs for shutting down the namenode service. Can anybody
check this
2013-02-28 02:07:51,752 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.common.Util: Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss
due
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
to lack
of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
2013-02-28 02:07:51,884 INFO
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 02:07:51,910 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 02:07:51,910 INFO
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
hadmin
2013-02-28 02:07:51,920 INFO
isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring
more than 10 times
2013-02-28 02:07:52,466 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not
exist.
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
NameNode
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
metrics
system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
Directory
/mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
Exiting
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
with
status 1
2013-02-28 02:08:48,913 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at
OPERA-MAST1.ny.os.local/192.168.1.3
On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
Hi Guys,
I have space on other partition. Can I change the path for cache files on
other partition ? I have below properties . Can it resolve the
issue
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
? If i
change the path to other directories and restart services I get
the
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
below
error while starting the service namenode. I didn't find anything
in
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
logs so
far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/name</value>
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/namesecondary</value>
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service
status; done
Hadoop datanode is running [ OK
]
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Hadoop namenode is dead and pid file exists
[FAILED]
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Hadoop secondarynamenode is running [ OK
]
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Thanks,
On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For
verifying
i checked the http link that you gave and i saw the property
their
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Post by Mohit Vadhera
but on
http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
blocks. Please check the logs or run fsck in order to identify
the
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Post by Mohit Vadhera
missing
blocks.) when i clicked on this link i can see file names . Do
I
Post by Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
Post by Mohit Vadhera
need to
reboot the machine to run fsck on root fs/ or is there hadoop
command fsck
that i can run on the running hadoop ?
Thanks,
--
Harsh J
--
Harsh J
Mohit Vadhera
2013-02-28 10:17:08 UTC
Permalink
After creating the directory and setting permission I tried to restart the
services and i get error "/mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock
acquired by nodename 7275-2zbGk5zsN3cvsa84sZI4N06D/***@public.gmane.org" and services are not
being started.

Need to check few logs from below logs.
===================================
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!


************************************************************/
2013-02-28 05:06:23,385 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:
Cannot locate configuration: tried
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-02-28 05:06:23,556 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,905 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack of redundant sto
rage directories!
2013-02-28 05:06:24,906 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of re
dundant storage directories!
2013-02-28 05:06:25,618 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2013-02-28 05:06:25,623 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=false
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication = 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 512
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 1
2013-02-28 05:06:26,015 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 2
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 3000
2013-02-28 05:06:26,016 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer = false
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs (auth:SIMPLE)
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadmin
2013-02-28 05:06:26,022 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true
2013-02-28 05:06:26,023 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 05:06:26,026 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 05:06:26,359 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 05:06:26,361 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 0
2013-02-28 05:06:26,378 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /mnt/san1/hdfs/cache/hdfs/dfs/name/in_use.lock acquired by nodename
7275-2zbGk5zsN3cvsa84sZI4N06D/***@public.gmane.org
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2013-02-28 05:06:26,381 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-02-28 05:06:26,382 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 05:06:26,385 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-02-28 05:06:26,394 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localtest/192.168.1.3
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created .
If I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?
Older Path
# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs
# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary
New Path
# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs
# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary
Thanks,
Post by Harsh J
Hi,
The exact error is displayed on your log and should be somewhat self
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.
Please check this one's availability, permissions (the NN user should
be able to access it).
On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
Please find below logs for shutting down the namenode service. Can
anybody
check this
Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack
of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
maxReplication
= 512
2013-02-28 02:07:51,910 INFO
minReplication
= 1
2013-02-28 02:07:51,910 INFO
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
hadmin
2013-02-28 02:07:51,920 INFO
isPermissionEnabled =
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring
more than 10 times
2013-02-28 02:07:52,466 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics
system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
join
Directory
/mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
with
status 1
2013-02-28 02:08:48,913 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
192.168.1.3
On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
Hi Guys,
I have space on other partition. Can I change the path for cache files
on
other partition ? I have below properties . Can it resolve the issue ?
If i
change the path to other directories and restart services I get the
below
error while starting the service namenode. I didn't find anything in
logs so
far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/namesecondary</value>
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
done
Hadoop datanode is running [ OK ]
Hadoop namenode is dead and pid file exists [FAILED]
Hadoop secondarynamenode is running [ OK ]
Thanks,
On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For
verifying
Post by Mohit Vadhera
i checked the http link that you gave and i saw the property their
but on
Post by Mohit Vadhera
http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
blocks. Please check the logs or run fsck in order to identify the
missing
Post by Mohit Vadhera
blocks.) when i clicked on this link i can see file names . Do I
need to
Post by Mohit Vadhera
reboot the machine to run fsck on root fs/ or is there hadoop command
fsck
Post by Mohit Vadhera
that i can run on the running hadoop ?
Thanks,
--
Harsh J
Mohit Vadhera
2013-02-28 09:48:55 UTC
Permalink
Thanks Harsh, /mnt/san1/hdfs/cache/hdfs/dfs/name is not being created . If
I do compare with the older path the permissions are same on the parent
directories.
Do I need to create this this directory manually and set the permission ?

Older Path

# ll /var/lib/hadoop-hdfs/cache/hdfs/
total 4
drwxr-xr-x. 5 hdfs hdfs 4096 Dec 27 11:34 dfs

# ll /var/lib/hadoop-hdfs/cache/hdfs/dfs/
total 12
drwx------. 3 hdfs hdfs 4096 Dec 19 02:37 data
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 name
drwxr-xr-x. 3 hdfs hdfs 4096 Feb 28 02:36 namesecondary


New Path

# ll /mnt/san1/hdfs/cache/hdfs/
total 4
drwxr-xr-x 3 hdfs hdfs 4096 Feb 28 02:08 dfs


# ll /mnt/san1/hdfs/cache/hdfs/dfs/
total 4
drwxr-xr-x 2 hdfs hdfs 4096 Feb 28 02:36 namesecondary


Thanks,
Post by Harsh J
Hi,
The exact error is displayed on your log and should be somewhat self
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.
Please check this one's availability, permissions (the NN user should
be able to access it).
On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
Please find below logs for shutting down the namenode service. Can
anybody
check this
Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
Path
/mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
configuration files. Please update hdfs configuration.
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image
storage
directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
lack
of redundant storage directories!
2013-02-28 02:07:51,754 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
edits storage directory (dfs.namenode.edits.dir) configured. Beware of
dataloss due to lack of redundant storage directories!
Refreshing hosts (include/exclude) list
2013-02-28 02:07:51,890 INFO
dfs.block.invalidate.limit=1000
2013-02-28 02:07:51,909 INFO
dfs.block.access.token.enable=false
2013-02-28 02:07:51,910 INFO
defaultReplication = 1
2013-02-28 02:07:51,910 INFO
maxReplication
= 512
2013-02-28 02:07:51,910 INFO
minReplication
= 1
2013-02-28 02:07:51,910 INFO
maxReplicationStreams = 2
2013-02-28 02:07:51,910 INFO
shouldCheckForEnoughRacks = false
2013-02-28 02:07:51,910 INFO
replicationRecheckInterval = 3000
2013-02-28 02:07:51,910 INFO
encryptDataTransfer = false
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
=
hdfs (auth:SIMPLE)
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
=
hadmin
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
=
true
2013-02-28 02:07:51,920 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-02-28 02:07:51,925 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-02-28 02:07:52,462 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring
more than 10 times
2013-02-28 02:07:52,466 INFO
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.min.datanodes = 0
2013-02-28 02:07:52,467 INFO
dfs.namenode.safemode.extension = 0
2013-02-28 02:07:52,469 INFO
Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
2013-02-28 02:07:52,471 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics
system...
2013-02-28 02:07:52,472 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
stopped.
2013-02-28 02:07:52,473 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system
shutdown complete.
2013-02-28 02:07:52,473 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode
join
Directory
/mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
directory does not exist or is not accessible.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting
with
status 1
2013-02-28 02:08:48,913 INFO
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/
192.168.1.3
On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
Hi Guys,
I have space on other partition. Can I change the path for cache files
on
other partition ? I have below properties . Can it resolve the issue ?
If i
change the path to other directories and restart services I get the
below
error while starting the service namenode. I didn't find anything in
logs so
far. Can you please suggest something ?
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/lib/hadoop-hdfs/cache/${user.name
}/dfs/namesecondary</value>
</property>
<property>
Service namenode is failing
# for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status;
done
Hadoop datanode is running [ OK ]
Hadoop namenode is dead and pid file exists [FAILED]
Hadoop secondarynamenode is running [ OK ]
Thanks,
On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
Post by Mohit Vadhera
Post by Harsh J
http://NNHOST:50070/conf
Harsh, I changed the value as said & restarted service NN. For
verifying
Post by Mohit Vadhera
i checked the http link that you gave and i saw the property their but
on
Post by Mohit Vadhera
http://NNHOST:50070 i noticed warning( WARNING : There are 4 missing
blocks. Please check the logs or run fsck in order to identify the
missing
Post by Mohit Vadhera
blocks.) when i clicked on this link i can see file names . Do I
need to
Post by Mohit Vadhera
reboot the machine to run fsck on root fs/ or is there hadoop command
fsck
Post by Mohit Vadhera
that i can run on the running hadoop ?
Thanks,
--
Harsh J
Continue reading on narkive:
Loading...