Geek Answers Handbook

In which folder or where are the fsimage and edit log files actually stored for reading and merging namenode at startup?

When the namenode starts, it reads the HDFS state from the image file, fsimage, and then applies the changes from the edit log file.

If I'm not mistaken, the name node starts when we write start -all.sh. So during this run, I think it reads fsimage and edits the logs and combines them. But from which folder or from which place does she really read both of these things?

+6
source share
3 answers

In hadoop-1.x, the start -all.sh script internally performs two operations, start-dfs.sh and start-mapred.sh . start-dfs.sh will start all the daemons necessary for hdfs, i.e.: datanode, namenode, secondary namenode

The checkpoint operation (using edit logs for fsimage) occurs during the launch of namenode, and this activity can be configured during namenode runs by setting the hdfs-site.xml --> dfs.namenode.checkpoint.period .

When namenode is entered, namenode daemons load fsimage from the directory specified in hdfs-site.xml -> dfs.name.dir. This property must be overridden, otherwise it will take the default value ( file:///tmp/dfs/name/ )

The location of the editing logs can be found by checking the value hdfs-site.xml -> dfs.name.edits.dir . The default value of dfs.name.edits.dir is $ {dfs.name.dir}.

Above property names changed in hadoop-2.0

+6
source

When you start the daemons, namenode will check the xml configuration file with the name

core-site.xml

located in the confo folder. On my system, it is located in usr / lib / hadoop / conf, which is the installation directory installed by hadoop.

In this configuration file you can see

 <configuration> <property> <name>hadoop.tmp.dir</name> <value>/var/lib/hadoop-0.20/cache/${user.name}</value> </property> </configuration>

This code / var / lib / hadoop-0.20 / cache / indicates the location of the fsimage, fstime, and edits file.

If, in the event of a namenode failure, for some specific time, the fsimage data will be temporarily stored in secondynamenode, and after the namenode is restored, the temporary data will be stored in fsimage.

0
source

below is the place where you can find patches and fsimage files.

/ application / Hadoop / TMP / Dpp / name / current

[core-site.xml]

 <property><name>hadoop.tmp.dir</name><value>/app/hadoop/tmp</value>

I believe this will help you.

0
source

More articles:


All Articles