Automation Tools Screenshot Ignores Software-Added Controls

Question

I save the screenshot in Automation Tools, but some of the controls are missing from this screenshot. Software-added controls seem to be "ignored."

How can i fix this?

Manual screenshot in Simulator: (pay attention to the yellow frame)

Screenshot of tool automation:

Automation script:

var target = UIATarget.localTarget(); target.delay(0.5) target.captureScreenWithName( "screenshot1.png" ); 

In Xcode (universal, objective-c), I created a new application with one view. I added a button and a tag with some restrictions for automatic layout in the storyboard.

I added this code to programmatically add a yellow button:

 - (void)viewDidLoad { [super viewDidLoad]; // Do any additional setup after loading the view, typically from a nib. UIButton *b = [[UIButton alloc] init]; b.backgroundColor = [UIColor yellowColor]; [b setTitle:@"Extra" forState:UIControlStateNormal]; [b setTranslatesAutoresizingMaskIntoConstraints:NO]; [self.view addSubview:b]; [self.view addConstraint:[NSLayoutConstraint constraintWithItem:b attribute:NSLayoutAttributeTop relatedBy:NSLayoutRelationEqual toItem:self.view attribute:NSLayoutAttributeTopMargin multiplier:1.0 constant:0.0]]; [self.view addConstraint:[NSLayoutConstraint constraintWithItem:b attribute:NSLayoutAttributeTrailing relatedBy:NSLayoutRelationEqual toItem:self.view attribute:NSLayoutAttributeTrailingMargin multiplier:1.0 constant:0.0]]; [self.view setNeedsUpdateConstraints]; // Correction: - initially I used the line below, but that was wrong // The problem is not solved with using setNeedsUpdateConstraints. // [self.view updateConstraints]; } 

I am using Xcode Version 7.1.1 (7B1005), my OS is El Capitan 10.11.1 (15B42). I know that it worked in earlier versions, because I use ui-screen-shooter to take screenshots of my application, and problems arise now, and they worked before that. This is not a problem with ui-screen-shooter, because I can only play it using automation tools.

What can i do with this?

EDIT:

One of the differences between the controls in storyboards and manually created ones can be the identifier of the object, which can be seen in the source of the storyboard. In case this is a problem, can I set an id for manually created controls? (Or is it lost once the scene is read from the storyboard?)

EDIT 2:

I selected all the properties of the elements in the view controller view (via objc / runtime) and compared the properties of the two buttons. There are slight differences: (will not include identical entries)

  Button from Storyboard |  Manual button
 -------------------------------------------------- ---------------
 "_defaultRenderingMode" = 2;  |  "_defaultRenderingMode" = 1;
 text = Button;  |  text = Extra;
                                  |
         (several position values ​​slightly different)
                                  |
 description contains: |
 "autoresize = RM + BM;"  | 
                                  |  backgroundColorSystemColorName = yellowColor;

EDIT 3:

Screenshot of target.logElementTree() output:

EDIT 4:

I added

 [UIButton appearance].backgroundColor = [UIColor yellowColor]; 

and all buttons now have a yellow background, but the Extra button is not in the screenshot.

Basically, I am not focused on appearance. I also tried

 UIButton *b = [UIButton buttonWithType:UIButtonTypeSystem]; 

so that the button looks like typical blue buttons. However, the button does not appear in the automation screenshot.

EDIT 5:

In case it matters: I can access the controls from a script automation and make taps - for example (from another application, and not from this test code):

 target.frontMostApp().mainWindow().buttons()[25].tap() 

Debugging is handled, so script automation can access the controls.

EDIT 6: I created a bug report on the Apple bugreport system. If you can reproduce it, it may be useful to do the same (at least, as I understand the intended use of the error report in Apple).

EDIT 7: (thanks quellish for your answer - handle my first wave of thoughts here :) I fixed the error in the code - I called [self.view updateConstraints]; but it should be [self.view setNeedsUpdateConstraints]; . This, however, did not affect the results of the automation screenshots.

I was wondering if the restrictions were updated and if calls were made to updateViewConstraints and some others. This is registered during application loading:

 viewDidLoad viewWillAppear updateViewConstraints viewWillLayoutSubviews viewWillLayoutSubviews viewDidAppear 

Therefore, when I add a button and restrictions to viewDidLoad and updateViewConstraints is updateViewConstraints , I would assume that all restrictions will be resolved when viewDidAppear called.

I did a quick check using this in viewDidAppear : (Still not sure if this is the way - I have little experience with accessibility.)

 _extraButton.accessibilityFrame = UIAccessibilityConvertFrameToScreenCoordinates(_extraButton.frame, self.view); 

The screenshot of the automation remains white.

It is interesting, however, that in the element tree, part of the screenshot of the additional button shows the correct location. Therefore, I would suggest that automation has raised the right position.

Answer 1

The problem you see is due to several factors. As stated in your question, your problem is that:

• Screenshots of UIAutomation will not include your stylized button, taking into account the test project you included and the automation script.

In different conditions, you see a different behavior:

• When a button is launched through a thread or a storyboard, the problem does not arise only when it is declared and added to the hierarchy of the code view (in particular, the viewDidLoad controller).

• The button appears in screenshots of UIAutomation when launched in earlier versions of iOS (i.e. iOS 8).

However, if you remove the restrictions added to viewDidLoad , you will see that a stylish button begins to appear in screenshots. This should be a hint - limitations are part of the problem.

With a constraint on the solution of the layout engine, he tries to do as much work as possible in batches by combining performance constraints. When the button and constraints are described in nib or storyboard, the constraint mechanism will receive them all at once and much earlier in the life cycle of the view hierarchy. Because of this, the steps of the layout process can be greatly simplified.

When a constraint is added later to the life cycle of a view hierarchy, however, you must redefine the constraints. Think about the added constraints that are added to the queue, and the linking mechanism will reevaluate them the next time the queue is “full”. UIAutomation intercepts the UIAccessibility API to do this - and accessibility may not know or care about what happens with the layout of the visual presentation. The visual presentation may change, but not in a way that affects the availability and thus the presentation of UIAutomation in the world.

iOS 9 has changed a few things about how the layout works under the hood, so you see this problem now.

There are several ways you can take action:

• Create a hierarchy of your presentation in the bank or storyboard. This will give you the most consistent experience and has performance and other benefits.

• Use the accessibility API to hint that the position (accessibility frame) of the button has changed after passing the layout. Unfortunately, you can still see how UIAutomation captures it during layout if the availability, visibility, or user interaction are not disabled before the layout is complete.

• Instead of an arbitrary delay in the script automation, use the UIAutomation timeouts effectively to wait for the element to appear. This is described in the documentation , and is best practice. Make it unavailable or invalid until the layout is complete.

Using these methods, I was able to easily get your sample project, so as not to reproduce the problem that you see. Some or all of these solutions may be appropriate for your project - or not.

Answer 2

The problem with instrumental automation most likely will not be fixed.

Part 1: Xcode 7 UI Tests - This is the new Tool Automation.

Part of the solution: fastlane / snapshot is the new ui-screen-shooter . (ui-screen-shooter is deprecated by the author, and he recommends switching to fastlane / snapshot.)

So, I will return to fastlane / snapshot and my problem will hopefully be resolved.