Make your USE gui program

Question

I would like to write a program that can “use” other programs by controlling the mouse / keyboard and being able to “see” what is on the screen.

I used AutoIt to do something similar, but I had to cheat sometimes because the language is not so powerful, or maybe it is just that I suck, and I can not do so much with it: P

So ... I need:

• Take screenshots, then I will compare them so that the program "understands", but it should "see"
• Use mouse: move, click and release, simple, right?
• Using the keyboard: pressing certain keys or key combinations, including special keys such as Alt , Ctrl , etc.

How can i do this in python?
Does it work on both Linux and windows? (it can be really cool, but it is not necessary)

Answer 1

I was lucky with similar tasks using PyWinAuto .

pywinauto is a set of python modules for automating the Microsoft Windows GUI. At the same time, the simplest one allows you to send mouse and keyboard for windows dialogs and controls.

It also has some support for capturing dialog images and such using the Python PIL image library.

Answer 2

You can use WATSUP under Windows.

Answer 3

AutoIt is fully capable of doing everything you mentioned. When I want to do some automation, but use Python functions, the easiest way for me is to use AutoItX , which is a DLL / COM control.

Taken from my answer :

import win32com.client oAutoItX = win32com.client.Dispatch( "AutoItX3.Control" ) oAutoItX.Opt("WinTitleMatchMode", 2) #Match text anywhere in a window title width = oAutoItX.WinGetClientSizeWidth("Firefox") height = oAutoItX.WinGetClientSizeHeight("Firefox") print width, height 

Answer 4

If you are comfortable with pascal, a really powerful keyboard / mouse / screen reader is SCAR: http://freddy1990.com/index.php?page=product&name=scar This can do OCR, search for bitmaps, determine color and etc. It is often used to automate online games, but it can be used for any situation when you want to simulate a person reading a screen and entering information.

Answer 5

I used the Windows (only) input API to write a VNC remote control application in the past. This allows you to easily fake a keyboard and mouse at the system level (i.e. not just send events to a single application).

If you are trying to do some kind of automatic testing of entire systems at the GUI level, this is a great USENIX article describing the automatic test response - a must-read.