pyspark AttributeError: 'NoneType' object has no attribute 'setCallSite'
pyspark:
AttributeError: 'NoneType' object has no attribute 'setCallSite'
我草,是pyspark的bug。解决方法:
print("Approximately joining on distance smaller than 0.6:")
distance_min = model.approxSimilarityJoin(imsi_proc_df, imsi_proc_df, 1e6, distCol="JaccardDistance") \
.select(col("datasetA.id").alias("idA"),
col("datasetB.id").alias("idB"),
col("JaccardDistance")) #.filter("idA=idB")
print(distance_min.show())
print("*"*88)
print(imsi_proc_df.show()) key = Vectors.sparse(53, [1, 3], [1.0, 1.0])
print(model.approxNearestNeighbors(imsi_proc_df, key, 2).show())
print("start calculate find botnet!")
print("*"*99)
print("time start:", time.time())
print(type(distance_min), dir(distance_min))
print(dir(distance_min.toLocalIterator)) ############################################## add this line to solve
distance_min.sql_ctx.sparkSession._jsparkSession = spark_app._jsparkSession
distance_min._sc = spark_app._sc
############################################# similarity_val_rdd = distance_min.toLocalIterator #.collect()
print("time end:", time.time())
print(similarity_val_rdd)
print("*"*99)
try:
G = ConnectedGraph()
ddos_ue_list = []
for item in similarity_val_rdd():
imsi, imsi2, jacard_similarity_val = item["idA"], item["idB"], item["JaccardDistance"]
print("???", imsi, imsi2, jacard_similarity_val)
Description
reproducing the bug from the example in the documentation:
import pyspark
from pyspark.ml.linalg import Vectors
from pyspark.ml.stat import Correlation
spark = pyspark.sql.SparkSession.builder.getOrCreate()
dataset = [[Vectors.dense([1, 0, 0, -2])],
[Vectors.dense([4, 5, 0, 3])],
[Vectors.dense([6, 7, 0, 8])],
[Vectors.dense([9, 0, 0, 1])]]
dataset = spark.createDataFrame(dataset, ['features'])
df = Correlation.corr(dataset, 'features', 'pearson')
df.collect()
This produces the following stack trace:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-92-e7889fa5d198> in <module>()
11 dataset = spark.createDataFrame(dataset, ['features'])
12 df = Correlation.corr(dataset, 'features', 'pearson')
---> 13 df.collect() /opt/spark/python/pyspark/sql/dataframe.py in collect(self)
530 [Row(age=2, name=u'Alice'), Row(age=5, name=u'Bob')]
531 """
--> 532 with SCCallSiteSync(self._sc) as css:
533 sock_info = self._jdf.collectToPython()
534 return list(_load_from_socket(sock_info, BatchedSerializer(PickleSerializer()))) /opt/spark/python/pyspark/traceback_utils.py in __enter__(self)
70 def __enter__(self):
71 if SCCallSiteSync._spark_stack_depth == 0:
---> 72 self._context._jsc.setCallSite(self._call_site)
73 SCCallSiteSync._spark_stack_depth += 1
74 AttributeError: 'NoneType' object has no attribute 'setCallSite'
Analysis:
Somehow the dataframe properties `df.sql_ctx.sparkSession._jsparkSession`, and `spark._jsparkSession` do not match with the ones available in the spark session.
The following code fixes the problem (I hope this helps you narrowing down the root cause)
df.sql_ctx.sparkSession._jsparkSession = spark._jsparkSession
df._sc = spark._sc df.collect() >>> [Row(pearson(features)=DenseMatrix(4, 4, [1.0, 0.0556, nan, 0.4005, 0.0556, 1.0, nan, 0.9136, nan, nan, 1.0, nan, 0.4005, 0.9136, nan, 1.0], False))]
pyspark AttributeError: 'NoneType' object has no attribute 'setCallSite'的更多相关文章
- python3 AttributeError: 'NoneType' object has no attribute 'split'
from wsgiref.simple_server import make_server def RunServer(environ, start_response): start_response ...
- AttributeError: 'NoneType' object has no attribute 'split' 报错处理
报错场景 social_django 组件对原生 django 的支持较好, 但是因为 在此DRF进行的验证为 JWT 方式 和 django 的验证存在区别, 因此需要进行更改自行支持 JWT 方式 ...
- python提示AttributeError: 'NoneType' object has no attribute 'append'【转发】
在写python脚本时遇到AttributeError: 'NoneType' object has no attribute 'append' a=[] b=[1,2,3,4] a = a.appe ...
- python提示AttributeError: 'NoneType' object has no attribute 'append'
在写python脚本时遇到AttributeError: 'NoneType' object has no attribute 'append' a=[] b=[1,2,3,4] a = a.appe ...
- Keras AttributeError 'NoneType' object has no attribute '_inbound_nodes'
问题说明: 首先呢,报这个错误的代码是这行代码: model = Model(inputs=input, outputs=output) 报错: AttributeError 'NoneType' o ...
- AttributeError: 'NoneType' object has no attribute 'extend'
Python使用中可能遇到的小问题 AttributeError: 'NoneType' object has no attribute 'extend' 或者AttributeError: 'Non ...
- 解决opencv:AttributeError: 'NoneType' object has no attribute 'copy'
情况一: 路径中有中文,更改即可 情况二:可以运行代码,在运行结束时显示 AttributeError: 'NoneType' object has no attribute 'copy' 因为如果是 ...
- appium 报错:AttributeError:"NoneType' object has no attribute 'XXX'
报错截图如下: 问题原因: 根据以上报错提示可已看到问题的原因为:logger中没有info此方法的调用,点击"具体报错的位置"上面的链接,可直接定位到具体的报错位置.根据分析所得 ...
- PIL中分离通道发生“AttributeError: 'NoneType' object has no attribute 'bands'”
解决方法: 这个貌似是属于一个bug 把Image.py中的1500行左右的split函数改成如下即可: def split(self): "Split image into bands&q ...
随机推荐
- [LeetCode] 507. Perfect Number 完美数字
We define the Perfect Number is a positive integer that is equal to the sum of all its positive divi ...
- [LeetCode] 531. Lonely Pixel I 孤独的像素 I
Given a picture consisting of black and white pixels, find the number of black lonely pixels. The pi ...
- consul删除无效实例
consul删除无效实例删除无效服务删除无效节点删除无效服务http://127.0.0.1:8500/v1/agent/service/deregister/test-9c14fa595ddfb8f ...
- [转] ABP框架Web API跨域问题的解决方案
原文地址:https://www.cnblogs.com/farb/p/ABPWebAPICrossDomain.html 1.在Web Api 项目下安装 Microsoft.AspNet.Web ...
- PHP计算两个坐标之间的距离
<?php /** * 计算两点之间的距离 * @param $lng1 经度1 * @param $lat1 纬度1 * @param $lng2 经度2 * @param $lat2 纬度2 ...
- PHP_MySQL高并发加锁事务处理
1.背景: 现在有这样的需求,插入数据时,判断test表有无username为‘mraz’的数据,无则插入,有则提示“已插入”,目的就是想只插入一条username为‘mraz’的记录. 2.一般程序 ...
- Java基础笔试练习(十二)
1.(C#.JAVA)扩展方法能访问被扩展对象的public成员 A.能 B.不能 答案: A 解析: 翻译一下,子类方法是否能够访问父类中的public成员. 2.如果子类要调用父类的构造函数,则通 ...
- 20191210-RobotFramework常见问题解决
附加-问题解决 1. 执行robot用例的时候提示WebDriverException: Message: invalid argument: can't kill an exited proce ...
- Microsoft的考验――查找第二大的数
#include<stdio.h> int main() { int n,m,t,max,max1; scanf("%d",&n); while(n--) { ...
- mysql中sum与if,case when 结合使用
1.sum与if结合使用 如图:数据表中,count_money 字段可为正,可为负.为正表示收入,负表示支出. 统计总收入,总支出. select sum(if(count_money > 0 ...