Set heapsize for an hdfs datanode

Description

There doesn't seem to be any way to specify max heap size for an hdfs service.

The only way I found is to add the hadoop_datanode_opts env var to the env:

"hadoop_env" : {
"hadoop_datanode_opts": "-Xmx2048m"
},

but I end up with two -Xmx's in the command line.

There will be a default -Xmx1000 and then my -Xmx2048m. There's no guarantee from the jvm that my setting -Xmx2048m will take precedence.

Am I missing something?

Release Notes

None

Activity

Show:
vbuciuc
October 27, 2017, 3:56 PM

Hi,

Thanks for the answer.

Indeed I am using hadoop 2.5.3.

But looking through the cookbook I found that I can set heapsize just for the datanode in the /etc/default/hadoop-hdfs-datanode if I modify the cookbook to pass a "hadoop_datanode_heapsize" attribute in there. Maybe that's something you might wanna consider implementing.

You can close the ticket.

Regards,
Victor.

Derek Wood
October 27, 2017, 1:42 AM

Hi ,
Which hadoop distro/version are you using?

Please see https://issues.apache.org/jira/browse/HADOOP-9870 regarding the multiple -Xmx flags. This is addressed in https://issues.apache.org/jira/browse/HADOOP-9902 for Hadoop 3.0, which afaik is not in any of the popular distributions yet.

Until then, I would first suggest setting HADOOP_HEAPSIZE as follows:

HADOOP_HEAPSIZE is not component-specific, but does prevent the multiple -Xmx flags, whereas HADOOP_DATANODE_OPTS are just generic strings appended to the end of the cmdline.

Let me know if that helps, but I believe this is an issue with Hadoop itself and not this cookbook.

Fixed
Your pinned fields
Click on the next to a field label to start pinning.

Assignee

Derek Wood

Reporter

vbuciuc