Skip to content

1.2.0
Compare
Choose a tag to compare
@yimelia yimelia released this 18 Jun 03:05
· 313 commits to master since this release
v1.2.0
1a206a0

Airtest1.2.0更新

English Version

新增：图像识别算法mstpl

我们新增了一个改良版本的模板匹配识别算法，能够大幅度增加原有的模板匹配的成功率，更好地适配分辨率变化的场景。

将Airtest更新至1.2.0以上版本即可使用，如果您正在使用AirtestIDE，也请同步更新至1.2.10以上版本，最新的版本已经将该算法设置为首选使用的算法。

请参考以下代码示例：

from airtest.core.api import *

touch(Template("search.png", scale_max=800, scale_step=0.005))

在Template图片对象初始化时，与图像识别有关的参数如下：

  • threshold：用于设置图像识别的筛选阈值，默认为0.8，因此需要图像识别效果大于0.8时，才会认为此次识别是成功的
  • rgb：是否要进行彩色识别，默认为False，意思是airtest默认会将图片处理为灰度图片，因此如果需要加入彩色三通道的识别，需要将它设置为True
  • 新加入scale_max：新算法专用参数，用于调节匹配的最大范围，如果要查找的目标UI很小的话，可以适当增大这个数值，默认值 800， 推荐值 740, 800, 1000 取值范围 [700 , 2000]
  • 新加入scale_step：新算法专用参数，用于控制搜索比例步长，如果要查找的目标UI很小的话，可以适当减小这个数值，默认值0.005，推荐值 0.02, 0.005, 0.001 取值范围 [0.001, 0.1]

如果在图像识别时遇到问题或异常报错，请将当前界面截图、与脚本中使用的图片一起提交issue。

接口改动：连接手机时无需设置参数

在Airtest 1.2.0以上版本中，我们不再希望大家连接手机时，手动指定各连接参数了，包括cap_method，touch_method和ori_method。

因此我们进行了一些改动，在初始化手机时，如果遇到问题，Airtest将会自动尝试进行兼容，并选取备用方案进行连接，直到成功为止。

需要关注的重要改动如下：

  • 目前Android12暂时还没有很好地支持，屏幕画面效果很差

  • 新增Android.screen_proxy，类似于之前的touch_proxy，统一使用它作为屏幕截图方法的调用入口

  • 初始化Android()对象时，可以不再强制指定cap_method="MINICAP"或"JAVACAP"，screen_proxy会自动按照顺序进行初始化，直到选取到可以成功初始化的截图方法。

    • 例如，之前在连接模拟器时，需要在IDE上勾选use javacap，或是在脚本运行时指定cap_method=JAVACAP，现在可以不指定，也能成功初始化
    • 删除了cap_method=MINICAP_STREAM这个参数，请不要再传MINICAP_STREAM这个值
    • 初始化的优先顺序为：自定义的截图方法 > MINICAP > JAVACAP > ADBCAP，其中，如果调用到了ADBCAP，将会打印一个warning，告诉用户此时截图效率很低
    • 保留了旧的一些调用入口，方便兼容旧的代码
      • 例如dev.minicap，将会返回dev.screen_proxy对象，依然可以使用dev.minicap.get_frame_from_stream()接口
      • 例如dev.cap_method，将会返回当前使用的截图方法名称（比如"MINICAP"），同样也可以使用dev.cap_method = "JAVACAP"来强制指定截图方法

  • 在获取图片时，再也不需要进行大量的类型判断然后调用对应的截图方法了

    • 直接使用dev.screen_proxy.get_frame_from_stream()即可

    • 即以前的这种代码可以废弃了：

    • if dev.cap_method == CAP_METHOD.MINICAP:
    dev.minicap.get_frame_from_stream()
elif dev.cap_method == xxx:
    ...

  • 如果希望实现自定义的新截图方法：在airtest.core.android.cap_methods.base_cap中，实现了所有截图方法的基类BaseCap，可以通过继承它，并实现get_frame_from_stream这个接口，就能新增一个自定义的截图方法，将它注册到ScreenProxy中就可以使用了

    • from airtest.core.android.cap_methods.base_cap import BaseCap
class TestCap(BaseCap):
    def get_frame_from_stream(self):
        return b"frame"

ScreenProxy.register_method("TESTCPY", TestCap)
# 默认优先初始化为自定义的TestCap
cap = ScreenProxy.auto_setup(dev.adb)

仅airtest开发者需要关注的改动如下：

  • 原先的android.get_display_info()默认会返回minicap.get_display_info()，现在改为默认直接返回adb.get_display_info() 43a9c56

    • 因为如果默认没有使用minicap，在这里还会强制触发一次minicap的初始化，很浪费时间
    • 希望能够让ori_method=ADBORI这个参数真正只与屏幕旋转方向做关联，默认情况下使用rotation_watcher.jar来获取屏幕方向，如果失败了，则可以指定此参数改为每隔2秒使用ADB获取一次当前屏幕旋转方向（一般是一些车机设备/特殊平板等设备）。原先的逻辑里，ADBORI的指定有可能是因为该设备minicap初始化失败，也可能是rotation_watcher拿到的屏幕方向不对，会造成问题难以排查。

  • 所有截图方法都包含有snapshot()这个接口，默认会处理截取的图片，返回一个cv2的图片，去掉了android.snapshot里的大量类型判断。

    • minicap额外增加了一个projection参数，用来调整屏幕图的大小，方便IDE中使用该接口拿到手机原始图片进行截图 d3fb6a6

  • 与screen_proxy类似，为touch_proxy添加了auto_setup逻辑，在初始化手机时，会先尝试初始化minitouch，如果失败会尝试maxtouch，最后选择adbtouch，这样可以避免一些机型虽然不是android10，但是也需要使用maxtouch的问题 e4efe9e

  • 为上述改动增加了对应的单元测试用例

  • 解决了在连接过程中，由于失败可能导致的socket未关闭或subprocess未关闭输入输出流的问题

其他BUG修复

  • nbsp增加了一个auto_kill参数，在手机断开连接时，能自动杀掉socket，避免内存泄露的问题 b3ff533
  • 解决了几个结束时未完全关闭io流的warning
  • 兼容了在脚本运行时传入的脚本路径末尾带有/的情况
  • SIFT算法专利已经解禁，因此重新加入了SIFT算法作为默认算法，但是受限于opencv的版本号，不是所有的版本号都可以直接使用 c8e9bba
  • 修复了在部分新型号手机上，在横屏状态时获取到的display_info尺寸相反的问题，这个问题会导致在IDE上的横屏状态下，屏幕画面显示错误 9bcccdf
    • 目前已测试过不少机型均修复了此问题（例如vivo iQOO Neo5，华为Mate40Pro等）
    • 如果仍有机型出现屏幕横竖旋转时显示错误的问题，请将具体的机型告诉我们

Airtest 1.2.0 update

New: image recognition algorithm mstpl

We have added an improved version of the template matching recognition algorithm, which can greatly increase the success rate of the original template matching and better adapt to scenes with changing resolutions.

Update Airtest to version 1.2.0 or higher. If you are using AirtestIDE, please also update to version 1.2.10 or higher. The latest version has set this algorithm as the preferred algorithm.

Please refer to the following code example:

from airtest.core.api import *

touch(Template("search.png", scale_max=800, scale_step=0.005))

When the Template image object is initialized, the parameters related to image recognition are as follows:

  • threshold: used to set the screening threshold of image recognition, the default is 0.8, so the recognition effect is required to be greater than 0.8 before the recognition is considered successful
  • rgb: Whether to perform color recognition, the default is False, which means that airtest will process the picture as a grayscale picture by default, so if you need to add color three-channel recognition, you need to set it to True
  • Newly added scale_max: new algorithm special parameter, used to adjust the maximum range of matching, If the target UI to be searched is small, you can increase this value appropriately, the default value is 800, the recommended value is 740, 800 , 1000 Value range [700, 2000]
  • Newly added scale_step: a new algorithm dedicated parameter, used to control the search scale step size, If the target UI to be found is small, you can appropriately reduce this value, the default value is 0.005, the recommended value is 0.02, 0.005 , 0.001 Value range [0.001, 0.1]

If you encounter problems or abnormal errors during image recognition, please take a screenshot of the current interface and submit an issue together with the pictures used in the script.

Interface changes: No need to set parameters when connecting to a mobile phone

In Airtest 1.2.0 and above, we no longer want everyone to manually specify the connection parameters when connecting to the phone, including cap_method, touch_method, and ori_method.

Therefore, we have made some changes. When initializing the phone, if there is a problem, Airtest will automatically try to be compatible and select a backup plan to connect until it succeeds.

The important changes that need attention are as follows:

  • Currently Android12 is not yet well supported, the screen effect is very poor
  • Added Android.screen_proxy, which is similar to the previous touch_proxy and uses it as the call entry of the screenshot method.
  • When initializing the Android() object, you can no longer force to specify cap_method="MINICAP" or "JAVACAP", screen_proxy will automatically initialize in order until the screenshot method that can be successfully initialized is selected .
    • For example, when connecting to the simulator before, you need to check use javacap on the IDE, or specify cap_method=JAVACAP when the script is running. Now it is not necessary to specify it, and it can be initialized successfully.
    • Deleted the parameter cap_method=MINICAP_STREAM, please do not pass the value of MINICAP_STREAM again
    • The priority order of initialization is custom screenshot method > MINICAP > JAVACAP > ADBCAP. If ADBCAP is called, a warning will be printed to tell the user that the screenshot efficiency is very low at this time.
    • Keep some of the old call entry points, which is convenient to be compatible with the old code
      • For example, dev.minicap will return the dev.screen_proxy object, and the dev.minicap.get_frame_from_stream() interface can still be used
      • For example, dev.cap_method will return the name of the currently used screenshot method (such as "MINICAP"), and you can also use dev.cap_method = "JAVACAP" to force the specified screenshot method
  • When acquiring pictures, there is no need to perform a lot of type judgment and then call the corresponding screenshot method
    • Use dev.screen_proxy.get_frame_from_stream() directly
    • That is, the previous code can be discarded:
    • if dev.cap_method == CAP_METHOD.MINICAP:
    dev.minicap.get_frame_from_stream()
elif dev.cap_method == xxx:
    ...
  • If you want to implement a new custom screenshot method: In airtest.core.android.cap_methods.base_cap, the base class BaseCap, which implements all screenshot methods, can be inherited and implement the interface get_frame_from_stream. A custom screenshot method can be added, and it can be used by registering it in ScreenProxy 
    • from airtest.core.android.cap_methods.base_cap import BaseCap
class TestCap(BaseCap):
    def get_frame_from_stream(self):
        return b"frame"

ScreenProxy.register_method("TESTCPY", TestCap)
# The default priority is initialized to the custom TestCap
cap = ScreenProxy.auto_setup(dev.adb)

Other bug fixes

  • nbsp added an auto_kill parameter, which can automatically kill the socket when the phone is disconnected to avoid memory leaks
  • Solved several warnings that did not completely close the io stream at the end
  • Compatible with / at the end of the script path passed in when the script is running
  • The SIFT algorithm patent has been lifted, so the SIFT algorithm has been re-added as the default algorithm, but it is limited by the version number of OpenCV. Not all version numbers can be used directly c8e9bba
  • Fixed an issue where the display_info size obtained in landscape mode was reversed on some new models of mobile phones. This issue would cause the screen to display incorrectly in landscape mode on the IDE 9bcccdf
    • Many models have been tested to fix this problem (such as vivo iQOO Neo5, Huawei Mate40Pro, etc.)
    • If there are still models with the wrong display when the screen is rotated horizontally and vertically, please tell us the specific model
Assets 2
Loading