from: https://www.linkedin.com/pulse/html-canvas-testing-selenium-opencv-maciej-kusz

Since HTML <canvas> become more and more popular for creating interactive content on any web page like games (especially since Adobe Flash technology is dying), there is a big problem with testing it using pure Selenium. If you have never seen <canvas> you may be wondering why? Mostly because <canvas> (like old Flash element) is seen in DOM structure just like element without any content even if there is a complex game inside, eg.

<canvas id="myCanvas" width="200" height="100"></canvas>

Using just Selenium you will be only able to locate <canvas> element and get its positionsize and some state, like isElementVisible, etc., but you will not be able to see what's inside and test internal behavior.

What can we do to test HTML <canvas>?

Since the <canvas> element is a container for graphics elements (with additional logic written in JavaScript) we can try to perform manual mouse actions using Selenium Action Chains. We have there a few useful action, like:

By combining only those 2 actions you will be able to click any button inside <canvas>element. But you will face 2 big problems:

  1. What are (x, y) coordinates of the center of button to be clicked?
  2. What is the current state of the game?

Get (x, y) coordinates the button center

We can approach this problem from 2 different directions:

  1. Prepare static (x, y) coordinate of the button center inside <canvas> element and use move_to_element_with_offset from Selenium Action Chain
  2. Get button center dynamically

Point 1 is quite easy to prepare using any graphics editing tool and we will not talk about it (going and easy path is not the way we follow at XCaliber, especially when the path is short and ends with a cliff). Reason for it is quite easy: we will need to implement dynamic method if we want to know a state of the game.

So how we can obtain button center coordinates dynamically?

We "just" need to "see" what's happening inside the <canvas> element. You can think: "easy to say, harder to do", but you will see that it's not that hard.

The best approach to "just see" problem is to use computer vision. Since Python has a very good binding for widely use the library called OpenCV, we can use it to solve this problem. In short, OpenCV is an image processing tool that will allow us to see what's happening inside <canvas> element.

In my previous article about Page Object Pattern, I have described how to prepare the object for XPath element locator. Let's use the same approach for a graphical locator.

Graphical locator

import cv2
import numpy
from io import BytesIO
from PIL import Image class GraphicalLocator(object): def __init__(self, img_path):
self.locator = img_path
# x, y position in pixels counting from left, top corner
self.x = None
self.y = None
self.img = cv2.imread(img_path)
self.height = self.img.shape[0]
self.width = self.img.shape[1]
self.threshold = None @propertydef center_x(self):return self.x + int(self.width / 2) \
if self.x and self.width else None @propertydef center_y(self):return self.y + int(self.height / 2) \
if self.y and self.height else None def find_me(self, drv):# Clear last found coordinates
self.x = self.y = None
# Get current screenshot of a web page
scr = drv.get_screenshot_as_png()
# Convert img to BytesIO
scr = Image.open(BytesIO(scr))
# Convert to format accepted by OpenCV
scr = numpy.asarray(scr, dtype=numpy.float32).astype(numpy.uint8)
# Convert image from BGR to RGB format
scr = cv2.cvtColor(scr, cv2.COLOR_BGR2RGB) # Image matching works only on gray images
# (color conversion from RGB/BGR to GRAY scale)
img_match = cv2.minMaxLoc(
cv2.matchTemplate(cv2.cvtColor(scr, cv2.COLOR_RGB2GRAY),
cv2.cvtColor(self.img, cv2.COLOR_BGR2GRAY),
cv2.TM_CCOEFF_NORMED)) # Calculate position of found element
self.x = img_match[3][0]
self.y = img_match[3][1] # From full screenshot crop part that matches template image
scr_crop = scr[self.y:(self.y + self.height),
self.x:(self.x + self.width)] # Calculate colors histogram of both template# and matching images and compare them
scr_hist = cv2.calcHist([scr_crop], [0, 1, 2], None,
[8, 8, 8], [0, 256, 0, 256, 0, 256])
img_hist = cv2.calcHist([self.img], [0, 1, 2], None,
[8, 8, 8], [0, 256, 0, 256, 0, 256])
comp_hist = cv2.compareHist(img_hist, scr_hist,
cv2.HISTCMP_CORREL) # Save treshold matches of: graphical image and image histogram
self.threshold = {'shape': round(img_match[1], 2),'histogram': round(comp_hist, 2)} # Return image with blue rectangle around match
return cv2.rectangle(scr, (self.x, self.y),
(self.x + self.width, self.y + self.height),
(0, 0, 255), 2)

  



The code above should be self-explained. The main reason for this is to provide the same interface as for XPath element locator. Having above you can do something like this if you want to test if given image is present on a web page:

img_check = GraphicalLocator("/path/to/image.png")
img_check.find_me(webdriver_instance)

Problem with above code is that it can give you false positive matches.

How to defend against false positive matches?

Take a look at GraphicalLocator object and its threshold attribute. It contains 2 values:

  1. The threshold for image shape match is telling us how similar both images are (the one you are looking for and found one). If value equals 1 then images shapes are identical.
  2. The threshold for image colors histogram match is telling us how similar colors of both images are. If value equals 1 then images colors histograms are identical.

Why do we need those 2 thresholds? Take a look at pictures below:

Both images present the same button, but in 2 different states (enabled and disabled). When you will try to find the first image and the second one will be present, shape threshold will be set to 1. It's happened because OpenCV image matching algorithm works on grey scaled images. In grey scale, both images shape is the same. Because of that, when you want to be sure that image you are looking for, is an image you can see, you need to check not only if the shape is identical but also colors of the images are the same. It's why there is also color histogram threshold calculated during image finding. This way code for checking is image present, should lock like this:

img_check = GraphicalLocator("/path/to/img.png")
img_check.find_me(webdriver_instance) is_found = True if img_check.threshold['shape'] >= 0.8 and \
img_check.threshold['histogram'] >= 0.4 else False

Values of thresholds to compare should be chosen during some experiments (those are working for me).

How to click?

Now the best part. Just take a look at this code snippet:

from selenium.webdriver.common.action_chains import ActionChains

img_check = GraphicalLocator("/path/to/img.png")
img_check.find_me(webdriver_instance) is_found = True if img_check.threshold['shape'] >= 0.8 and \
img_check.threshold['histogram'] >= 0.4 else False if is_found:
action = ActionChains(webdriver_instance)
action.move_by_offset(img_check.center_x, img_check.center_y)
action.click()
action.perform()

Conclusion

As you can see it's not so hard to check if the image is visible in the <canvas> element and click on it. Extending this approach with allow you check the current state of the game, because of state checking will be based on visibility or invisibility of some elements.

PS. It also works with Flash elements ;)

HTML <​canvas> testing with Selenium and OpenCV的更多相关文章

  1. Functional testing - python, selenium and django

    Functional testing  - python selenium django - Source Code : from selenium import webdriverfrom sele ...

  2. 使用Selenium和openCV对HTML5 canvas游戏进行自动化功能测试(一)

    上一篇讲了HTML5 canvas游戏的基本工作原理,接下来讲如何进行自动化功能测试. Selenium是一个跨平台的跨浏览器的对网页进行自动化测试的工具.从Selenium 2.0开始Seleniu ...

  3. 300+ Manual Testing and Selenium Interview Questions and Answers

    Manual testing is a logical approach and automation testing complements it. So both are mandatory an ...

  4. How to click on a point on an HTML5 canvas in Python selenium webdriver

    https://stackoverflow.com/questions/29624949/how-to-click-on-a-point-on-an-html5-canvas-in-python-se ...

  5. HTML5 canvas游戏工作原理

    HTML5已经不是一个新名词.它看上去很cool,有很多feature,大多数人普遍看好它的发展.对于我来说,最感兴趣的是它的canvas标签,可以结合Javascript来绘制游戏画面. 我们可以在 ...

  6. 简单的HTML5 canvas游戏工作原理

    HTML5已经不是一个新名词.它看上去很cool,有很多feature,大多数人普遍看好它的发展.对于我来说,最感兴趣的是它的canvas标签,可以结合Javascript来绘制游戏画面. 我们可以在 ...

  7. 所有selenium相关的库

    通过爬虫 获取 官方文档库 如果想获取 相应的库 修改对应配置即可 代码如下 from urllib.parse import urljoin import requests from lxml im ...

  8. JavaScript(Node.js)+ Selenium自动化测试

    Selenium is a browser automation library. Most often used for testing web-applications, Selenium may ...

  9. [转] 以后再有人问你selenium是什么,你就把这篇文章给他

    本文转自:https://blog.csdn.net/TestingGDR/article/details/81950593 写在最前面:目前自动化测试并不属于新鲜的事物,或者说自动化测试的各种方法论 ...

随机推荐

  1. 冒泡排序(Python实现)

    目录 1. while版本--冒泡排序 2. for版本--冒泡排序 3. 测试用例 4. 算法时间复杂度分析 1. while版本--冒泡排序 def bubble_sort_while(a_lis ...

  2. python-lambda、filter、reduce、map

    python-lambda.map.filter.reduce lamdba python关键字,用于在表达式中创建匿名函数. 注意:lambda函数的定义体只能用纯表达式,不能赋值,不能使用whil ...

  3. H5页面分享微信自定义分享title和img

    前端开发H5 需分享到朋友圈和发给好友,想自定义分享的title和图表还有简短一句话,还需调用微信的api 首先需获取到微信的appId,timestamp,nonceStr,signature 微信 ...

  4. SQL Server之获取下周一的日期

    今天项目中需要得到下周一的日期,故想到了一种解决办法,用slq语句解决了.当然实现方法肯定不只有这一种. -(select DATEPART(weekday,getdate())) /*下周一差几天 ...

  5. 新节点在线加入PXC

    环境 192.168.139.151 新增节点 192.168.139.148-150 集群节点 192.168.139.151 已经安装好PXC软件 计划: 选用192.168.139.150 节点 ...

  6. java资料共享

    1.javascript视频教程 链接: http://pan.baidu.com/s/1gd57FVH 密码: d9ei 2.JPA视频教程 链接: http://pan.baidu.com/s/1 ...

  7. java中的锁之Lock接口与Condition接口

    一.Lock源码. 1.是一个接口.一共有6个方法. 2.方法详细如下: (1)当前线程尝试获取锁.结果分两种情况,一是成功获取到锁,则返回:二是获取锁失败,则一直等待.不响应中断请求. (2)当前线 ...

  8. Mockito/PowerMockito Straige Issues

    http://blog.csdn.net/xiaoyaoyulinger/article/details/52415494 http://breezylee.iteye.com/blog/208843 ...

  9. html5-颜色的表示

    div{width: 100%;height: 100px;}body{background: url(../pic/2.png);}/*#div1{background: #ff0000;}#div ...

  10. 【转】Requests 官方中文文档 - 快速上手

    迫不及待了吗?本页内容为如何入门 Requests 提供了很好的指引.其假设你已经安装了 Requests.如果还没有,去安装一节看看吧. 首先,确认一下: Requests 已安装 Requests ...