Take the Guesswork Out of Installing DolphinScheduler—Here’s How to Do It | HackerNoon

Apache DolphinScheduler is a distributed, decentralized, and scalable visual DAG workflow task scheduling systeåm. It is committed to solving the complex dependencies in data processing workflows, making the scheduling system ready-to-use out of the box. Ambari is a tool for managing Hadoop clusters. Although Ambari does not directly support the installation of DolphinScheduler, we can integrate it into Ambari through a custom service. This article will describe how to use Ambari to install DolphinScheduler.

1. Ambari Service Installation Package

Place the service installation package that needs to be integrated into Ambari in the following directory:

/var/lib/ambari-server/resources/stacks/HDP/3.1/services

2. Restart the Ambari-server Service

sudo ambari-server restart

After the integration, you can proceed with the service installation on the Ambari web page.

1. Add Service and Select DolphinScheduler
2. DolphinScheduler Service Configuration Choose to use MySQL as the metadata database for the DolphinScheduler service.
3. Service Selection for Installation Select two DS-master nodes, at least two DS-ALERTS, DS-APIs, and deploy the remaining nodes as DS-worker nodes.
4. Error Handling for hue-desktop-ini If you have configured namenode-ha before installing DolphinScheduler, you may encounter errors related to ‘hue-desktop-ini’. You need to modify the params.py script.

Script location: /var/lib/ambari-server/resources/stacks/HDP/3.3/services/DOLPHINSCHEDULER/package/scripts/params.py

Modify the content:
# hdfs_httpfs_host = config['configurations']['hue-desktop-ini']['hdfs_httpfs_host']
hdfs_httpfs_host = "dfs_ha_enabled"  # Comment out the original line and assign any value to hdfs_httpfs_host

After making the changes, save and restart the ambari-server service to reinstall DolphinScheduler.

5. Copy MySQL Driver to All Nodes’ libs Directory

cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/alert-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/api-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/master-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/current/dolphinscheduler/worker-server/libs/
cp /tmp/mysql-connector-j-8.0.33.jar /usr/hdp/3.3.1.0-004/dolphinscheduler/tools/libs/

6. Change Ownership of All Services to hdfs User

chown -R hdfs:hdfs /var/log/dolphinscheduler
chown -R hdfs:hdfs /usr/hdp/3.3.1.0-004/dolphinscheduler
chown -R hdfs:hdfs /etc/dolphinscheduler/
ln -s /usr/hdp/3.1.4.0-315/dolphinscheduler dolphinscheduler

After making these changes, start DolphinScheduler from the Ambari management platform. The metadata will be automatically initialized and tables will be created during the startup process.