jupyter sparkmagic on hdp

1、安装jupyter

2、安装sparkmagic

3、设置超时

"livy_server_heartbeat_timeout_seconds": 0,

4、设置集群模式

"session_configs": {
 	"driverMemory": "2G",
 	"executorCores": 4,
 	"proxyUser": "bernhard",
 	"conf": {
 		"spark.master": "yarn-cluster",
 		"spark.jars.packages": "com.databricks:spark-csv_2.10:1.5.0"
 	}
 }
5、配置hdp

livy.server.csrf_protection.enabled ==> false
6、启动notebook，配饰spark session

%load_ext sparkmagic.magics

%manage_spark

参考文档
https://github.com/jupyter-incubator/sparkmagic
https://github.com/bernhard-42/Sparkmagic-on-HDP