Android cannot read window contents on multiple devices using accessibility service

Question

My requirement: Reading text from a popup, dialog, etc. for a specific application.

I have implemented an accessibility service and I receive the proper events and data as per my requirement. However, during testing, I realized that on some devices, instead of using AlertDialog or Dialog, they used an action (thematic as a dialogue). Therefore, in my accessibility event, I only get the activity header, is there a way to find the text displayed by this particular pop-up action?

I searched pretty hard, but couldn't get much help on this topic, and the documentation was not good at this issue. There is not much in the accessibility service code, but if you still need it, I'll post it later.

thank

Answer 1

Use accessiblityNodeInfo to get information even when the phone returns, it will receive a ussd response, and it will reject the dialog when it is possible to enter several options.

First of all, in the case of [pohne], the event class name is returned as ussdalertactivty, so I used only a “warning” to identify the warning dialog from the ussd response.

public void onAccessibilityEvent(AccessibilityEvent event) { if (event.getPackageName().toString().equals("com.android.phone") && event.getClassName().toString().toLowerCase() .contains("alert")) { AccessibilityNodeInfo source = event.getSource(); if (source != null) { String pcnResponse = fetchResponse(source); } } 

Now I have created the fetchResponse function, which will return the answer from pcn as a string and also cancel the dialog, so you need to execute the GlobalAction (GLOBAL_ACTION_BACK) function.

 private String fetchResponse(AccessibilityNodeInfo accessibilityNodeInfo) { String fetchedResponse = ""; if (accessibilityNodeInfo != null) { for (int i = 0; i < accessibilityNodeInfo.getChildCount(); i++) { AccessibilityNodeInfo child = accessibilityNodeInfo.getChild(i); if (child != null) { CharSequence text = child.getText(); if (text != null && child.getClassName().equals( Button.class.getName())) { // dismiss dialog by performing action click in normal // cases if (isUSSDFromApp && (text.toString().toLowerCase().equals("ok") || text .toString().toLowerCase() .equals("cancel"))) { child.performAction(AccessibilityNodeInfo.ACTION_CLICK); return fetchedResponse; } } else if (text != null && child.getClassName().equals( TextView.class.getName())) { // response of normal cases if (text.toString().length() > 10) { fetchedResponse = text.toString(); } } else if (child.getClassName().equals( ScrollView.class.getName())) { // when response comes as phone then response can be // retrived from subchild for (int j = 0; j < child.getChildCount(); j++) { AccessibilityNodeInfo subChild = child.getChild(j); CharSequence subText = subChild.getText(); if (subText != null && subChild.getClassName().equals( TextView.class.getName())) { // response of different cases if (subText.toString().length() > 10) { fetchedResponse = subText.toString(); } } else if (subText != null && subChild.getClassName().equals( Button.class.getName())) { // dismiss dialog by performing action click in // different // cases if (isUSSDFromApp && (subText.toString().toLowerCase() .equals("ok") || subText .toString().toLowerCase() .equals("cancel"))) { subChild.performAction(AccessibilityNodeInfo.ACTION_CLICK); return fetchedResponse; } } } } } } } return fetchedResponse; } 

Hope this solves the problem regardless of devices.

Answer 2

This is the code I use and it works for me:

 public class USSDService extends AccessibilityService { public static String TAG = USSDService.class.getSimpleName(); @Override public void onAccessibilityEvent(AccessibilityEvent event) { Log.d(TAG, "onAccessibilityEvent"); AccessibilityNodeInfo source = event.getSource(); if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED && !String.valueOf(event.getClassName()).contains("AlertDialog")) { return; } if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED && (source == null || !source.getClassName().equals("android.widget.TextView"))) { return; } if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED && TextUtils.isEmpty(source.getText())) { return; } List<CharSequence> eventText; if(event.getEventType() == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED) { eventText = event.getText(); } else { eventText = Collections.singletonList(source.getText()); } String text = processUSSDText(eventText); if( TextUtils.isEmpty(text) ) return; // Close dialog performGlobalAction(GLOBAL_ACTION_BACK); // This works on 4.1+ only Log.d(TAG, text); // Handle USSD response here } private String processUSSDText(List<CharSequence> eventText) { for (CharSequence s : eventText) { String text = String.valueOf(s); // Return text if text is the expected ussd response if( true ) { return text; } } return null; } @Override public void onInterrupt() { } @Override protected void onServiceConnected() { super.onServiceConnected(); Log.d(TAG, "onServiceConnected"); AccessibilityServiceInfo info = new AccessibilityServiceInfo(); info.flags = AccessibilityServiceInfo.DEFAULT; info.packageNames = new String[]{"com.android.phone"}; info.eventTypes = AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED | AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED; info.feedbackType = AccessibilityServiceInfo.FEEDBACK_GENERIC; setServiceInfo(info); } } 

Declare it in Android manifest

 <service android:name=".USSDService" android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"> <intent-filter> <action android:name="android.accessibilityservice.AccessibilityService" /> </intent-filter> <meta-data android:name="android.accessibilityservice" android:resource="@xml/ussd_service" /> 

Create an XML file that describes the accessibility service called ussd_service

 <?xml version="1.0" encoding="utf-8"?> <accessibility-service xmlns:android="http://schemas.android.com/apk/res/android" android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged" android:accessibilityFeedbackType="feedbackGeneric" android:accessibilityFlags="flagDefault" android:canRetrieveWindowContent="true" android:description="@string/accessibility_service_description" android:notificationTimeout="0" android:packageNames="com.android.phone" />