Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instrumented test execution with ANDROIDX_TEST_ORCHESTRATOR does not work with Android 11 #743

Closed
mateuszkwiecinski opened this issue Sep 15, 2020 · 21 comments
Assignees

Comments

@mateuszkwiecinski
Copy link

@mateuszkwiecinski mateuszkwiecinski commented Sep 15, 2020

Description

Similar to #355, but the execution hangs

Steps to Reproduce

  1. Create new project using AS wizard
  2. Enable Android orchestrator following the https://developer.android.com/training/testing/junit-runner
  3. Run tests on Android Emulator with API 30

Expected Results

The test passes

Actual Results

  • The test hangs
  • logcat shows:
2020-09-15 21:28:39.486 12791-12791/com.example.myapplication E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.myapplication, PID: 12791
    java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo{com.example.myapplication.test/androidx.test.runner.AndroidJUnitRunner}: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6709)
        at android.app.ActivityThread.access$1300(ActivityThread.java:237)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:223)
        at android.app.ActivityThread.main(ActivityThread.java:7656)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
     Caused by: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
        at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.connect(OrchestratedInstrumentationListener.java:93)
        at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:309)
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6704)
        at android.app.ActivityThread.access$1300(ActivityThread.java:237) 
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913) 
        at android.os.Handler.dispatchMessage(Handler.java:106) 
        at android.os.Looper.loop(Looper.java:223) 
        at android.app.ActivityThread.main(ActivityThread.java:7656) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947) 
2020-09-15 21:28:39.486 12791-12791/com.example.myapplication E/AndroidJUnitRunner: An unhandled exception was thrown by the app.
2020-09-15 21:28:39.493 12791-12791/com.example.myapplication E/InstrumentationResultPrinter: Failed to mark test No Tests as finished after process crash
  • (might be unrelated, but doesn' happen on api 29) my private app shows in addition: mkdir: ‘/storage/emulated/0/Android/data/my.app.package’: Permission denied which points at DeviceProviderInstrumentTestTask and getAdditionalTestOutputEnabled which tries to save some log file. disabling the android.enableAdditionalTestOutput property removes the mkdir error but the test still hangs

AndroidX Test and Android OS Versions

    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    androidTestImplementation 'androidx.test:runner:1.3.0'
    androidTestUtil 'androidx.test:orchestrator:1.3.0'

Android OS Versions: the issue exists only on API 30. The test passes on lower apis.

Link to a public git repo demonstrating the problem:

https://github.com/mateuszkwiecinski/orchestrator_doesnt_work
task: ./gradlew connectedDebugAndroidTest hangs and never ends
image

Github action run confirming the issue: https://github.com/mateuszkwiecinski/orchestrator_doesnt_work/pull/1/checks?check_run_id=1119649523

@prfarlow1
Copy link

@prfarlow1 prfarlow1 commented Sep 16, 2020

I am also experiencing this bug. I also made a test application that experiences the same indefinite hanging on Android 11: https://github.com/prfarlow1/UiTestApplication

@LmKupke
Copy link

@LmKupke LmKupke commented Sep 21, 2020

Same issue is occurring when compiling and targeting SDK 30 and running against an emulator with 30.

2020-09-21 14:26:44.942 31419-31419/com.scompany.consumer E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.scompany.consumer, PID: 31419
java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo
java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6709)
at android.app.ActivityThread.access$1300(ActivityThread.java:237)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loop(Looper.java:223)
at android.app.ActivityThread.main(ActivityThread.java:7656)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
Caused by: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService

  • no issue when compiling and targeting 30 and running on an emulator with 29.
  • no issue when compiling and targeting 29 and running on an emulator with 30.
@prfarlow1
Copy link

@prfarlow1 prfarlow1 commented Sep 22, 2020

Can we get an update from the Android test team when this will be addressed? This is a really bad bug. We rely on the orchestrator for our tests to pass and right now we just can't run our tests on Android 11 test devices.

@nimeacuerdo
Copy link

@nimeacuerdo nimeacuerdo commented Sep 28, 2020

Same here. Everything was working fine with emulator api 29 and Orchestrator until we migrated to api 30.

@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 5, 2020

Hey all, sorry for delayed response. We are looking into this, but the fix is proving more difficult to get released than anticipated. We hope to have something out soon to address this.

@tillchen
Copy link

@tillchen tillchen commented Oct 15, 2020

Same here. Switched to Android 10 and it's working fine for me.

@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 15, 2020

Please give 1.3.1-alpha01 a try. This should hopefully be fixed.
See https://github.com/android/android-test/releases/tag/androidx-test-1.3.1-alpha01

@mateuszkwiecinski
Copy link
Author

@mateuszkwiecinski mateuszkwiecinski commented Oct 19, 2020

Can't confirm it works :/ It doesn't hang indeed, but still fails with a message:

Task :app:connectedDebugAndroidTest
Starting 0 tests on test(AVD) - 11
Tests on test(AVD) - 11 failed: Instrumentation run failed due to 'Process crashed.'

com.android.build.gradle.internal.testing.ConnectedDevice > No tests found.[test(AVD) - 11] FAILED
No tests found. This usually means that your test classes are not in the form that your test runner expects (e.g. don't inherit from TestCase or lack @test annotations).

Failed run:
https://github.com/mateuszkwiecinski/orchestrator_doesnt_work/pull/3/checks?check_run_id=1274143298

It works on devices with api <30
image

@prfarlow1
Copy link

@prfarlow1 prfarlow1 commented Oct 19, 2020

I also could not get my tests to work with the new 1.3.1-alpha01

@brettchabot brettchabot reopened this Oct 19, 2020
@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 19, 2020

It looks like there is an incompatibility with the testInstrumentationRunnerArguments clearPackageData: 'true' setting. I was able to get @mateuszkwiecinski sample working by commenting out that line.

brettchabot added a commit to android/testing-samples that referenced this issue Oct 19, 2020
Enable clearPackageData
@brettchabot brettchabot self-assigned this Oct 19, 2020
@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 19, 2020

A workaround for the clearPackageData issue is to declare a

application android:forceQueryable="true"

in the androidTest manifest

@prfarlow1
Copy link

@prfarlow1 prfarlow1 commented Oct 19, 2020

A workaround for the clearPackageData issue is to declare a

application android:forceQueryable="true"

in the androidTest manifest

What does android:forceQueryable="true" actually do? I can't find any documentation for it

@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 19, 2020

Check out the docs on the new package visibility restrictions on API 30 here
https://developer.android.com/training/basics/intents/package-visibility

IIUC using forceQueryable basically lets other apps (like the orchestrator app) communicate with the test package.

copybara-service bot pushed a commit that referenced this issue Oct 19, 2020
This fixes using 'clearPackageData' on API 30 by making test apps (that build androidx.test.runner) callable from androidx.test.orchestrator.

Fixes #743

PiperOrigin-RevId: 337944463
copybara-service bot pushed a commit that referenced this issue Oct 20, 2020
This fixes using 'clearPackageData' on API 30 by making test apps (that build androidx.test.runner) callable from androidx.test.orchestrator.

Fixes #743

PiperOrigin-RevId: 337944463
@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 20, 2020

Please give 1.3.1-alpha02 a try

@mateuszkwiecinski
Copy link
Author

@mateuszkwiecinski mateuszkwiecinski commented Oct 20, 2020

Thank you @brettchabot 👍
I can confirm it works in my sample project 🚀

@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 20, 2020

whew thanks for confirming @mateuszkwiecinski

@tadfisher
Copy link

@tadfisher tadfisher commented Oct 22, 2020

android:forceQueryable isn't recognized by AAPT2 4.1.0 and fails the build:

Execution failed for task ':Mobile-Android:processDevDebugAndroidTestResources'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
   > Android resource linking failed
     /home/tad/simple/mobile/Mobile-Android/Mobile-Android/build/intermediates/packaged_manifests/devDebugAndroidTest/AndroidManifest.xml:22: AAPT: error: attribute android:forceQueryable not found.
         
@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 22, 2020

@tadfisher can you be more specific about your project setup? I don't know where a 'AAPT2 4.1.0' comes from.

The aapt2 version in build tools 30.0.2 appears to be
~/Android/Sdk/build-tools/30.0.2$ ./aapt2 version
Android Asset Packaging Tool (aapt) 2.19-6739378

@tadfisher
Copy link

@tadfisher tadfisher commented Oct 22, 2020

@brettchabot My bad, this is on a nonstandard setup using aapt2 from maven.google.com. With the aapt2 packaged with build-tools 30.0.2, this works.

@brettchabot
Copy link
Collaborator

@brettchabot brettchabot commented Oct 22, 2020

@tadfisher cool thanks for confirming.

@mateuszkwiecinski
Copy link
Author

@mateuszkwiecinski mateuszkwiecinski commented Nov 2, 2020

@brettchabot Just in case if that's related: I faced another issue with 1.3.1-alpha02 - our tests don't fail anymore. Apparently new alpha version changes when org.junit.runner.notification.RunListener callbacks are called, which breaks LeakCanary, which throws an exception and for an unknown reason makes the test to pass.
I reported an issue there (square/leakcanary#1986) but as well that might be an unexpected behavior change in the android-test library

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

7 participants
You can’t perform that action at this time.