Introduce the devil_util binary to replace the md5sum binary.
In this devil_util binary, we use the crc32 hashing function
instead of md5 for better efficiency. We also use mmap rather than
directly reading the file for better efficiency.
In the future, we will add more functionalities to devil_util,
such as the ability to uncompress the transferred file.
Bug: 397452007
Change-Id: I3cc2c135969578413d2b500b9c2ca8b50978864c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6283025
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Martin Kong <martinkong@google.com>
Cr-Commit-Position: refs/heads/main@{#1425736}
This reverts commit 64a901a69b.
Reason for reland: Added hasattr check
Original change's description:
> Revert "Android: Use prebuilt devil tools for most scripts"
>
> This reverts commit d832d48946.
>
> Reason for revert: "AttributeError: 'AvdArgs' object has no attribute 'use_local_devil_tools'"
> https://ci.chromium.org/ui/p/chromium/builders/ci/android-nougat-x86-rel/4277/overview
> https://chromium-swarm.appspot.com/task?id=61f77da6f79dd610
>
> Original change's description:
> > Android: Use prebuilt devil tools for most scripts
> >
> > It still uses chromium's copies of prebuilts, but no longer defaults
> > to chromium's copy of tools that need to be built. Namely:
> > md5sum, forwarder, and dex_commands.jar
> >
> > The reason for using local copies of these is to be able to test
> > changes to them. This coverage is maintained by still using them
> > for non-component builds within gtests & instrumentation tests.
> >
> > The main motivation for this change is to fix a frequently
> > occurring error where component builds of the tools do not get
> > pushed properly to the devices, and then do not work. md5sum also
> > runs faster when built in release mode, so it doesn't make sense
> > to use a slow copy of it when building a debug version of chrome.
> >
> > This is the error output I (and others) see frequently:
> >
> > W 4.817s TimeoutThread-1-for-Install(FA6AK0304925) Error calculating md5: (device: FA6AK0304925) shell command run via adb failed on the device:
> > command: a=/data/local/tmp/md5sum/md5sum_bin;! [[ $(ls -l $a) = *14076* ]]&&exit 2;export LD_LIBRARY_PATH=/data/local/tmp/md5sum;$a -gz H4sIADX5rWMC/63RMQ6DMAwF0BORVGxVxcDQqVKrSu2MTGIgIolDEoK4fRFXqAcv1vf7g6WGDBJCkIqcGIlGiwK8jmS0ONZJqCmSw2p+VPh09D7mu93d53Vd2kvbNLKHdFyE+Sb/plKwJncKeoul7mDNE/psFGSKvA1nipckP5hRoGdVNZZuNazkgKhLzUqmNRbcWcnC+/ANews7nugPZnbMzvICAAA=;
> > exit status: 1
> > output:
> > - CANNOT LINK EXECUTABLE "/data/local/tmp/md5sum/md5sum_bin": library "libabsl.cr.so" not found
> >
> > Bug: 1404180
> > Change-Id: Ie93b1a28caca4adfc492726450324ea90b198adf
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4491904
> > Commit-Queue: Andrew Grieve <agrieve@chromium.org>
> > Reviewed-by: Ben Pastene <bpastene@chromium.org>
> > Cr-Commit-Position: refs/heads/main@{#1138561}
>
> Bug: 1404180
> Change-Id: I94d2aa0e50328afefe15a3d164432b14dc901a38
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4503307
> Auto-Submit: Ben Pastene <bpastene@chromium.org>
> Commit-Queue: Ben Pastene <bpastene@chromium.org>
> Owners-Override: Stephanie Kim <kimstephanie@google.com>
> Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
> Cr-Commit-Position: refs/heads/main@{#1138931}
Include-Ci-Only-Tests: true
Bug: 1404180
Change-Id: Ib2a04d291509c58325a10d22f9d07c56c39023c2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4500324
Auto-Submit: Andrew Grieve <agrieve@chromium.org>
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1139184}
This reverts commit d832d48946.
Reason for revert: "AttributeError: 'AvdArgs' object has no attribute 'use_local_devil_tools'"
https://ci.chromium.org/ui/p/chromium/builders/ci/android-nougat-x86-rel/4277/overviewhttps://chromium-swarm.appspot.com/task?id=61f77da6f79dd610
Original change's description:
> Android: Use prebuilt devil tools for most scripts
>
> It still uses chromium's copies of prebuilts, but no longer defaults
> to chromium's copy of tools that need to be built. Namely:
> md5sum, forwarder, and dex_commands.jar
>
> The reason for using local copies of these is to be able to test
> changes to them. This coverage is maintained by still using them
> for non-component builds within gtests & instrumentation tests.
>
> The main motivation for this change is to fix a frequently
> occurring error where component builds of the tools do not get
> pushed properly to the devices, and then do not work. md5sum also
> runs faster when built in release mode, so it doesn't make sense
> to use a slow copy of it when building a debug version of chrome.
>
> This is the error output I (and others) see frequently:
>
> W 4.817s TimeoutThread-1-for-Install(FA6AK0304925) Error calculating md5: (device: FA6AK0304925) shell command run via adb failed on the device:
> command: a=/data/local/tmp/md5sum/md5sum_bin;! [[ $(ls -l $a) = *14076* ]]&&exit 2;export LD_LIBRARY_PATH=/data/local/tmp/md5sum;$a -gz H4sIADX5rWMC/63RMQ6DMAwF0BORVGxVxcDQqVKrSu2MTGIgIolDEoK4fRFXqAcv1vf7g6WGDBJCkIqcGIlGiwK8jmS0ONZJqCmSw2p+VPh09D7mu93d53Vd2kvbNLKHdFyE+Sb/plKwJncKeoul7mDNE/psFGSKvA1nipckP5hRoGdVNZZuNazkgKhLzUqmNRbcWcnC+/ANews7nugPZnbMzvICAAA=;
> exit status: 1
> output:
> - CANNOT LINK EXECUTABLE "/data/local/tmp/md5sum/md5sum_bin": library "libabsl.cr.so" not found
>
> Bug: 1404180
> Change-Id: Ie93b1a28caca4adfc492726450324ea90b198adf
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4491904
> Commit-Queue: Andrew Grieve <agrieve@chromium.org>
> Reviewed-by: Ben Pastene <bpastene@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#1138561}
Bug: 1404180
Change-Id: I94d2aa0e50328afefe15a3d164432b14dc901a38
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4503307
Auto-Submit: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Owners-Override: Stephanie Kim <kimstephanie@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Cr-Commit-Position: refs/heads/main@{#1138931}
It still uses chromium's copies of prebuilts, but no longer defaults
to chromium's copy of tools that need to be built. Namely:
md5sum, forwarder, and dex_commands.jar
The reason for using local copies of these is to be able to test
changes to them. This coverage is maintained by still using them
for non-component builds within gtests & instrumentation tests.
The main motivation for this change is to fix a frequently
occurring error where component builds of the tools do not get
pushed properly to the devices, and then do not work. md5sum also
runs faster when built in release mode, so it doesn't make sense
to use a slow copy of it when building a debug version of chrome.
This is the error output I (and others) see frequently:
W 4.817s TimeoutThread-1-for-Install(FA6AK0304925) Error calculating md5: (device: FA6AK0304925) shell command run via adb failed on the device:
command: a=/data/local/tmp/md5sum/md5sum_bin;! [[ $(ls -l $a) = *14076* ]]&&exit 2;export LD_LIBRARY_PATH=/data/local/tmp/md5sum;$a -gz H4sIADX5rWMC/63RMQ6DMAwF0BORVGxVxcDQqVKrSu2MTGIgIolDEoK4fRFXqAcv1vf7g6WGDBJCkIqcGIlGiwK8jmS0ONZJqCmSw2p+VPh09D7mu93d53Vd2kvbNLKHdFyE+Sb/plKwJncKeoul7mDNE/psFGSKvA1nipckP5hRoGdVNZZuNazkgKhLzUqmNRbcWcnC+/ANews7nugPZnbMzvICAAA=;
exit status: 1
output:
- CANNOT LINK EXECUTABLE "/data/local/tmp/md5sum/md5sum_bin": library "libabsl.cr.so" not found
Bug: 1404180
Change-Id: Ie93b1a28caca4adfc492726450324ea90b198adf
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4491904
Commit-Queue: Andrew Grieve <agrieve@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1138561}
This reverts commit 9bdc5ccb7f.
Reason for revert: Reverting until I can fix the `import six` issues more cleanly.
Original change's description:
> Roll catapult to 532e564 and update for changes in it.
>
> This CL rolls //third_party/catapult to
> 532e564f622742fc93a7f91dd3556d711e778a7e and picks up
>
> https://chromium-review.googlesource.com/c/catapult/+/2450450
>
> which removes zipfile_2_7_13.py as a dependency. This CL
> also updates the corresponding Chromium .pydeps files that mentioned
> it.
>
> That change also added a dependency on `six`, which seems to be
> pervasively assumed in other parts of the catapult codebase as
> present in the environment; however, it isn't guaranteed to be
> there in the chromium compile-time python build, so this adds
> that as needed as well.
>
> TBR=bsheedy@chromium.org
>
> Bug: 1112471
> Change-Id: I849355a5bdc18847dcbf39f3d3a9e9fbf428e3be
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2449938
> Commit-Queue: Dirk Pranke <dpranke@google.com>
> Reviewed-by: Dirk Pranke <dpranke@google.com>
> Reviewed-by: John Chen <johnchen@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#814062}
TBR=dpranke@google.com,bsheedy@chromium.org,johnchen@chromium.org
Change-Id: I8dc0ccc9819eb5cd7de6d9fab4ec9a23c485ca09
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 1112471, 1135502
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2453150
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Cr-Commit-Position: refs/heads/master@{#814255}
This CL rolls //third_party/catapult to
532e564f622742fc93a7f91dd3556d711e778a7e and picks up
https://chromium-review.googlesource.com/c/catapult/+/2450450
which removes zipfile_2_7_13.py as a dependency. This CL
also updates the corresponding Chromium .pydeps files that mentioned
it.
That change also added a dependency on `six`, which seems to be
pervasively assumed in other parts of the catapult codebase as
present in the environment; however, it isn't guaranteed to be
there in the chromium compile-time python build, so this adds
that as needed as well.
TBR=bsheedy@chromium.org
Bug: 1112471
Change-Id: I849355a5bdc18847dcbf39f3d3a9e9fbf428e3be
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2449938
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: Dirk Pranke <dpranke@google.com>
Reviewed-by: John Chen <johnchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#814062}
This adds a presubmit check to ensure scripts initialize devil via
devil_chromium.Initialize() rather than other ways
(script_common.InitializeEnvironment, devil_env.config.Initialize).
devil_chromium.Initialize() is preferable to either because it uses
better defaults for dependencies (ex. adb version v40 rather than v32).
This also adds docs devil_chromium.Initialize() itself, since the
adb_path parameter was undocumented (but is related to the primary
motivation for this change).
The check intentionally skips third_party/, as these projects won't have
access to devil_chromium.Initialize(). The check also skips
devil_chromium.py itself, since the Initialize() method legitimately
calls devil_env.config itself.
Bug: 978697
Test: git cl presubmit
Change-Id: Ia699ad8f2ddeef46057baeb6708cf09b254b756d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1707458
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Nate Fischer <ntfschr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#678859}
No change to logic, just a refactor.
This swaps out hardcoded string literals for the newly added ABI
constants for a few scripts in chromium.
This updates pydeps for the scripts this transitively affects, which
conveniently includes the pydeps files which previously blocked a
catapult autoroll.
Bug: 922854, 925237, 924778
Test: vpython android_webview/tools/run_cts_test.py
Test: vpython build/android/list_class_verification_failures_test.py
Change-Id: I9c754809f00606bad19120e869d5c7f47b8dba02
Reviewed-on: https://chromium-review.googlesource.com/c/1436876
Reviewed-by: Tobias Sargeant <tobiasjs@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Nate Fischer <ntfschr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#626122}
Without this, we were downloading chromium_commands.dex.jar from GCS every
time we tried to use it. This is both unnecessary and potentially problematic,
as catapult/devil's downloading mechanism is currently prone to hanging with
lock failures.
With this change, we will instead use the locally built version of
chromium_commands.dex.jar when available.
BUG=674172
Review-Url: https://codereview.chromium.org/2575003003
Cr-Commit-Position: refs/heads/master@{#439537}
This change will hopefully allow us to configure devil more based
on the build config. For example, it will let us use the same
android_sdk_tools to run tests that we use to build with.
BUG=
Review URL: https://codereview.chromium.org/1812383003
Cr-Commit-Position: refs/heads/master@{#386461}
Adds md5sum and forwarder device for mips and mips64.
BUG=missing md5sum and forwarder for mips devices
Review URL: https://codereview.chromium.org/1507233005
Cr-Commit-Position: refs/heads/master@{#364079}
