jaceklaskowski
Mastering Apache Spark

Updated a month ago

Palash Gupta (@palashgupta) started discussion #104

a year ago · 2 comments

Open

Requesitng few probable Reasons to get "Failed to get broadcast variable" error in Spark 2.0.0

Hello Sir,

I started building an application using Spark 2.0.0 and some times I faced "Failed to get broadcast variable" error from Spark & it stopped processing. Currently I'm resolving to delete all running application, blockmanager directory, lib file created in /tmp directory and re-running the spark applications. My experience tells me that when I run multiple spark applications or one long running spark applications hang then it gives me the error.

As I don't understand the root cause of the problem with confidence, I want to know probable reasons to look into it.

Thanks for your support & will be waiting for your reply.

Best Regards, Palash Gupta

Jacek Laskowski @jaceklaskowski commented a year ago

Can you send me the entire stack track to start from? Do you have a Spark app to reproduce the issue? I highly recommend moving our discussion to StackOverflow to get more support from the Spark community on SO. Let's do this and continue our investigation on SO. Deal?

Palash Gupta @palashgupta commented a year ago

Hello Sir,

Thanks a lot for your response.

I'm trying to reproduce the problem & as soon as I make it I will send you the full stack trace. And I will also open a case in StakeOverflow. Though previously I created two similar cases but the response from StakeOverflow mates were not helpful for me.

Best Regards, Palash Gupta

Palash Gupta @palashgupta commented a year ago

Hello Sir,

I was able to take some low level trace for the problem and it happened when we are running multiple separate spark applications at a time but resources are distributed carefully.

Trace link: https://www.dropbox.com/s/zfrp0k953bzaj7z/kpienginensn_20161218093235.rar?dl=0

Palash Gupta @palashgupta commented a year ago

Hello Sir,

Greetings!

I have created a thread in Stakeoverflow site as well as you suggested with more specific log.

http://stackoverflow.com/questions/41236661/failed-to-get-broadcast-1-piece0-of-broadcast-1-in-pyspark-application

Thanks for your suggestion & I would appreciate if you comment over there.

Best Regards, Palash Gupta


to join this conversation on GitBook. Already have an account? Sign in to comment
Notifications

You’re not receiving notifications from this thread.


1 participant