We're updating the issue view to help you get more done. 

Set heapsize for an hdfs datanode

Description

There doesn't seem to be any way to specify max heap size for an hdfs service.

The only way I found is to add the hadoop_datanode_opts env var to the env:

"hadoop_env" : {
"hadoop_datanode_opts": "-Xmx2048m"
},

but I end up with two -Xmx's in the command line.

There will be a default -Xmx1000 and then my -Xmx2048m. There's no guarantee from the jvm that my setting -Xmx2048m will take precedence.

Am I missing something?

Release Notes

None

Activity

Show:
Derek Wood
October 27, 2017, 1:42 AM

Hi ,
Which hadoop distro/version are you using?

Please see https://issues.apache.org/jira/browse/HADOOP-9870 regarding the multiple -Xmx flags. This is addressed in https://issues.apache.org/jira/browse/HADOOP-9902 for Hadoop 3.0, which afaik is not in any of the popular distributions yet.

Until then, I would first suggest setting HADOOP_HEAPSIZE as follows:

HADOOP_HEAPSIZE is not component-specific, but does prevent the multiple -Xmx flags, whereas HADOOP_DATANODE_OPTS are just generic strings appended to the end of the cmdline.

Let me know if that helps, but I believe this is an issue with Hadoop itself and not this cookbook.

vbuciuc
October 27, 2017, 3:56 PM

Hi,

Thanks for the answer.

Indeed I am using hadoop 2.5.3.

But looking through the cookbook I found that I can set heapsize just for the datanode in the /etc/default/hadoop-hdfs-datanode if I modify the cookbook to pass a "hadoop_datanode_heapsize" attribute in there. Maybe that's something you might wanna consider implementing.

You can close the ticket.

Regards,
Victor.

Fixed

Assignee

Derek Wood

Reporter

vbuciuc

Labels

None

Docs Impact

None

UX Impact

None

Components

Priority

Major
Configure