Discussion:
Can set different dfs.replication for different dirs
(too old to reply)
Meng QingPing
2014-04-28 08:42:18 UTC
Permalink
Hi,

I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 for
dfs /user. How to configure? Both /tmp and /user are generated by
mapreduce job or hive or sqoop.

Thanks,
Jack
Nitin Pawar
2014-04-28 08:59:24 UTC
Permalink
DFS replication is set on the file level (block level) or at cluster level
(if you do not specify the replication factor while writing the file then
this one is picked).
As per my understanding, there is nothing for directories.
Post by Meng QingPing
Hi,
I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 for
dfs /user. How to configure? Both /tmp and /user are generated by
mapreduce job or hive or sqoop.
Thanks,
Jack
--
Nitin Pawar
sudhakara st
2014-04-28 12:22:25 UTC
Permalink
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.

Example:

- hadoop fs -setrep -w 3 -R /user/hadoop/dir1

hadoop dfs -setrep -R -w 1 /dir/
Post by Nitin Pawar
DFS replication is set on the file level (block level) or at cluster level
(if you do not specify the replication factor while writing the file then
this one is picked).
As per my understanding, there is nothing for directories.
Post by Meng QingPing
Hi,
I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
for dfs /user. How to configure? Both /tmp and /user are generated by
mapreduce job or hive or sqoop.
Thanks,
Jack
--
Nitin Pawar
--
Regards,
...sudhakara
Nitin Pawar
2014-04-28 12:41:14 UTC
Permalink
Sudhakar,

will this set for the new files being written to those directories?
Post by sudhakara st
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.
- hadoop fs -setrep -w 3 -R /user/hadoop/dir1
hadoop dfs -setrep -R -w 1 /dir/
Post by Nitin Pawar
DFS replication is set on the file level (block level) or at cluster
level (if you do not specify the replication factor while writing the file
then this one is picked).
As per my understanding, there is nothing for directories.
Post by Meng QingPing
Hi,
I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
for dfs /user. How to configure? Both /tmp and /user are generated by
mapreduce job or hive or sqoop.
Thanks,
Jack
--
Nitin Pawar
--
Regards,
...sudhakara
--
Nitin Pawar
sudhakara st
2014-04-29 08:40:33 UTC
Permalink
Hello Nitin,

HDFS replication factor is always associated with file level. When your
copying or creating file for any directory will set to default replicas.
but you can specify your replication when creating or copying files
hadoop fs -D dfs.replication=2 -put foo.txt fsput

and in java

FileSystem fs = FileSystem.get(new Configuration());
fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
Post by Nitin Pawar
Sudhakar,
will this set for the new files being written to those directories?
Post by sudhakara st
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.
- hadoop fs -setrep -w 3 -R /user/hadoop/dir1
hadoop dfs -setrep -R -w 1 /dir/
Post by Nitin Pawar
DFS replication is set on the file level (block level) or at cluster
level (if you do not specify the replication factor while writing the file
then this one is picked).
As per my understanding, there is nothing for directories.
Post by Meng QingPing
Hi,
I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
for dfs /user. How to configure? Both /tmp and /user are generated by
mapreduce job or hive or sqoop.
Thanks,
Jack
--
Nitin Pawar
--
Regards,
...sudhakara
--
Nitin Pawar
--
Regards,
...sudhakara
Meng QingPing
2014-04-29 08:56:10 UTC
Permalink
Thanks for all replies.

The files in /tmp most are generated by hadoop jobs. Can set the
yarn/mapreduce to specify one repication for tmp files?

Thanks,
Jack
Post by sudhakara st
Hello Nitin,
HDFS replication factor is always associated with file level. When your
copying or creating file for any directory will set to default replicas.
but you can specify your replication when creating or copying files
hadoop fs -D dfs.replication=2 -put foo.txt fsput
and in java
FileSystem fs = FileSystem.get(new Configuration());
fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
Post by Nitin Pawar
Sudhakar,
will this set for the new files being written to those directories?
Post by sudhakara st
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.
- hadoop fs -setrep -w 3 -R /user/hadoop/dir1
hadoop dfs -setrep -R -w 1 /dir/
Post by Nitin Pawar
DFS replication is set on the file level (block level) or at cluster
level (if you do not specify the replication factor while writing the file
then this one is picked).
As per my understanding, there is nothing for directories.
Post by Meng QingPing
Hi,
I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
for dfs /user. How to configure? Both /tmp and /user are generated by
mapreduce job or hive or sqoop.
Thanks,
Jack
--
Nitin Pawar
--
Regards,
...sudhakara
--
Nitin Pawar
--
Regards,
...sudhakara
--
Thanks,
Qingping
Meng QingPing
2014-04-29 09:15:13 UTC
Permalink
Another question
Can set expired time for /tmp or set yarn/mapreduce to remove the expired
tmp files periodically?

Thanks,
Jack
Post by Meng QingPing
Thanks for all replies.
The files in /tmp most are generated by hadoop jobs. Can set the
yarn/mapreduce to specify one repication for tmp files?
Thanks,
Jack
Hello Nitin,
Post by sudhakara st
HDFS replication factor is always associated with file level. When your
copying or creating file for any directory will set to default replicas.
but you can specify your replication when creating or copying files
hadoop fs -D dfs.replication=2 -put foo.txt fsput
and in java
FileSystem fs = FileSystem.get(new Configuration());
fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
Post by Nitin Pawar
Sudhakar,
will this set for the new files being written to those directories?
Post by sudhakara st
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.
- hadoop fs -setrep -w 3 -R /user/hadoop/dir1
hadoop dfs -setrep -R -w 1 /dir/
Post by Nitin Pawar
DFS replication is set on the file level (block level) or at cluster
level (if you do not specify the replication factor while writing the file
then this one is picked).
As per my understanding, there is nothing for directories.
Post by Meng QingPing
Hi,
I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as
3 for dfs /user. How to configure? Both /tmp and /user are generated by
mapreduce job or hive or sqoop.
Thanks,
Jack
--
Nitin Pawar
--
Regards,
...sudhakara
--
Nitin Pawar
--
Regards,
...sudhakara
--
Thanks,
Qingping
--
Thanks,
Qingping
Continue reading on narkive:
Loading...