New feature: image detection strategy "edge"

[simonmeggle]

Hi,
with this pull request I want to share the work of my valued colleague @gautamilango and me.

It introduces another strategy edgeto detect images in situations when too much pixels are different and further loweringconfidence would lead to false (or no) results.

The key here is to pre-process both images (reference image and screenshot) with canny edge detection (https://scikit-image.org/).

This reduces the images to the most relevant parts before they are compared.

It makes image detection extremely rebust against

• inpredictable colour nuances (example: "hover"-colors for buttons)
• compression artifacts (e.g. when testing over RDP/Citrix, which compress screen information depending on the available bandwidth)
• pixel deviations of dynamic image sources (e.g. map from 3rd party providers)
• ...

Debug Image is a helper keyword and starts a UI (written by Gautam) which gives worthful insights about

• the best strategy (default oder edge),
• its parameters (confidence, egde detection params),
• the number of matches (unknown so far)
• the match score of the best match (also unknown, done with try&error).

There is also a detailled debugger. It shows the edge detection results in oder to fine-tune the parameters.

We are already using the edge strategy successfully in production (synthetic e2e monitoring with Checkmk/Robotmk) and it works pretty well.

To make this strategy as integrated as possible, we have put a lot of effort into this, without breaking any existing functionality.

Thanks to ABRAXAS Informatik AG in Switzerland, which spent the ressources to make this possible at all! 👍

That's all for now - we are looking forward to hearing from you.
Any suggestion is highly appreciated! :-)

Best regards,
Simon
(simon.meggle at elabit.de)

PS: See also https://blog.robotmk.org/

[Tattoo]

Hi @simonmeggle,

This on the first glance seems an awesome addtition to the library!

I need to check this thoroughly which I haven't had time yet. I hope to check this in the coming weeks.

[simonmeggle]

Hi Tattoo, glad to hear that you find this useful.
I understand that this is a big commit which needs to be carefully reviewed.
Take your time, I can provide you any assistance you need!

[simonmeggle]

Hi Tattoo, I just want to ask if I can be of any help? Regards, Simon

[kleinski]

It would be awesome to have the edge strategy included in the imagehorzizonlibrary to avoid the problems of compression artifacts while trying to test through RDP environments.

[PhilippLemke]

I I am already looking for such a feature, as I often struggle with (nested) remote desktop connections.

[a9038889]

This function is very good~ I look forward to its appearance

[maxiking203]

I also took a look at @simonmeggle new features to the Library. We could really improve our testcases and developing process with the implemented debugger and the edge detection.
Definitely looking forward to use them as a Component of the ImageHorizonLibrary in the future.

[roland-gsell]

This would be very useful in our environment. Hope it gets implemented soon.

[7meis]

Happy new year!
Is there already a timeline to implement this.
This function would be very helpful for my use cases.
Many thanks.

[simonmeggle]

Is there already a timeline to implement this.

It will be implemented soon, but obviously not in this library (the pr still hangs, no one has ever reviewed it).