Unable to run the recipe


Installing Cookbook Gems:
Compiling Cookbooks...
[2017-08-22T17:59:15+05:30] FATAL: HDFS NameNode HA requires node['hadoop']['hdfs_site']['dfs.nameservices'] to be set

Recipe Compile Error in /var/chef/cache/cookbooks/hadoop/recipes/_hadoop_hdfs_ha_checkconfig.rb

---------- exit

Cookbook Trace:
--------------- /var/chef/cache/cookbooks/hadoop/recipes/_hadoop_hdfs_ha_checkconfig.rb:25:in `from_file'

Relevant File Content:
---------------------- /var/chef/cache/cookbooks/hadoop/recipes/_hadoop_hdfs_ha_checkconfig.rb:

18: #
19: # HDFS HA requires dfs.nameservices
22: if node['hadoop'].key?('hdfs_site') && node['hadoop']['hdfs_site'].key?('dfs.nameservices')
23: dfs_nameservices = node['hadoop']['hdfs_site']['dfs.nameservices'].split(',')
24: else
25>> Chef::Application.fatal!("HDFS NameNode HA requires node['hadoop']['hdfs_site']['dfs.nameservices'] to be set")
26: end
28: # We have dfs.nameservices, and now need to check them
29: dfs_nameservices.each do |ns|
30: # Start namenode checks
31: next unless node['hadoop']['hdfs_site'].key?("dfs\.ha\.namenodes\.#{ns}")
32: # We need two and only two NameNodes
33: namenodes = node['hadoop']['hdfs_site']["dfs\.ha\.namenodes\.#{ns}"].split(',')
34: if namenodes.size != 2

System Info:
------------ chef_version=13.2.20
ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
program_name=chef-client worker: ppid=8621;start=17:59:14;

Running handlers:
[2017-08-22T17:59:15+05:30] ERROR: Running exception handlers
Running handlers complete
[2017-08-22T17:59:15+05:30] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 01 seconds
[2017-08-22T17:59:15+05:30] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2017-08-22T17:59:15+05:30] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2017-08-22T17:59:15+05:30] ERROR: exit
[2017-08-22T17:59:15+05:30] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Release Notes



Derek Wood
August 24, 2017, 5:32 PM

You would probably want to set it in a role, or another cookbook which "wraps" the hadoop cookbook. We have some notes on wrapping the cookbook here.

Since this recipe is being called, it means you are either 1) already setting node['hadoop']['hdfs_site']['dfs.ha.automatic-failover.enabled'] somewhere, or 2) directly running this recipe or hadoop_hdfs_zkfc via the runlist. If 1) you should set the other attributes in the same place. If 2), and also as a general recommendation, I would suggest not worrying about HA yet and first get it working with the simplest setup as you become more familiar with chef.

I'm going to go ahead and close this as it is not a bug.

saif ahmad
August 23, 2017, 4:46 AM


Thanks for the reply.
Where do I set this value?

In which file? I know this question is very lame but I'm just a week old using chef and cookbooks.

Derek Wood
August 22, 2017, 9:58 PM

This isn't an actual cookbook error, but rather a check to prevent it from rendering an invalid hdfs-site.xml configuration that you have provided. It seems you have specified

but have not specified the other necessary attributes for HA namenode. In particular

must be set.

I suggest taking a look at the _hadoop_hdfs_ha_checkconfig.rb recipe, or the Apache docs to see the other necessary configurations for HA Namenode.

Won't Fix
Your pinned fields
Click on the next to a field label to start pinning.


Derek Wood


saif ahmad