The trifacta flows which either read from or write to Databricks Delta Tables requires an All Purpose Cluster when executed.
When an existing All Purpose Cluster is configured to be used by Trifacta and if this cluster is either in a terminated or starting state , then the execution of the Trifacta Flows fail with below message.
Since All Purpose Clusters are expensive, we cannot keep it running throughout the day.
" PublishException
Cluster with Id 0613-075842-ii603z2y needs restart. Please try again after restarting it.
"
If we do not configure any cluster, Trifacta tries to spin up a new All Purpose Cluster but again fails with below message
"Job failed:
Invalid SQL generated due to Run Parameter values (DatabricksClusterException: Cluster with Id 0708-100950-kqlkz2fw is starting. Please try again after some time.) "
The question is can Trifacta trigger and wait for a Databricks All Purpose cluster similar ?
