Updated a month ago
Manish (@manishatgit) started discussion #113
a year ago · 1 comment
A Spark driver (aka an application’s driver process) is a JVM process that hosts SparkContext for a Spark application. It is the master node in a Spark application.
Driver (Edit this file)
You're right. It's a bit confusing and am going to fix the confusion. When I said "the master node" I didn't mean to refer to the Spark infrastructure (the master and workers) but to the distinction between the driver and executors. You're right that the driver can be hosted on any worker in a cluster if you use --deploy-mode cluster command-line option.
Thanks for bringing it up! Very appreciated.
You’re not receiving notifications from this thread.