How can we connect cluster-based software with a chef?

Question

As part of setting up platform orchestration, we use our python package to install various software packages on a cluster of machines in the cloud.

We have the following scenario:

• of many software products, one of our programs is Ambari (which helps manage the hadoop platform).
• it works as follows: "n" is the number of cluster machines reporting 1 ambari server.
• for each machine in the cluster, in order to report, we must install ambari-agent on each machine in the cluster and change its properties file with the ambari server, which should inform and run ambari-agent.

what can we do - we were able to install ambari servers and ambari agents separately in our cluster machines using separate cookbooks.

what we cannot do is how we can change each property file of the machine’s ambari-agent file so that it points to our ambari server IP address. in general, what is an elegant way to connect cluster-based programs as part of a chef’s orchestration?

NB :. ambari-server is created on the fly, and therefore its IP address is obtained at runtime.

Is it possible? Are there any alternatives to the above problem?

thanks

Answer 1

This is the script for Chef-server and search .

You will need to change your recipe, which the ambari agent installs to dynamically obtain the IP address of the ambari server.

First, you run recipes that set up your ambari server. When Chef successfully works there, it fills in some information about the node for the Chef server, including recipes and roles that have been applied to this particular node. You can go to the chef server and check the node attributes, especially the "recipes" attribute.

Now change the ambari agent recipe. I do not know what the agent configuration file looks like, but we are only interested in one line where the IP address of the main server is set.

Create a configuration file template and add it to cooking. Replace the main IP binding value with <%= @master_ip %>

Change the ambari agent recipe to set this value correctly:

 # search for the server node. I expect server node was configured with # ambari::server recipe. If not, change it to the appropriate value and # don't fortget to escape colons. ambari_server_node = search( :node, 'recipes:ambari\:\:server' ) # now create the configuration file on ambari agent node from # the previously created template and pass the value for the @master_ip # variable template '/right/path/on/target/node/config.file' do [...] variables( :master_ip => ambari_server_node['ipaddress'] ) end 

Answer 2

If you have a chef server, you can search for a node that launches the ambari server recipe. Then you use the IP address of this machine. Alternatively, you can use the DNS name for the ambari server, and then update the DNS record to point to the new server when it is available.

Other options include using confd with etcd or using consul. Each of them will allow you to update your post-chef using an ip server.

Answer 3

This sounds like the perfect use for Consul .

Flexible key / value storage for dynamic configuration, feature tagging, coordination, leaderboards, and more. Long polling for almost instant notification of configuration changes.

We started using it to solve a problem that is very similar to yours. Historically, any configuration has been handled by a large Python codebase with many cook chefs. We worked on replacing some of its parts with a consul.

What really impressed me was how quickly the information transmitted through the cluster bubbled up to visible. I can see the equipment of the machines and go online before the cloud service control panel is accessible thanks to the Consuls healthchecks.

This is much more than just playing as a key / value store. It can also handle service discovery and perform any routine health checks that you can currently track through something like Sensu.

I highly recommend checking it out.

Hope this helps. Good luck with the deployment!