0

Reland "Add new ANGLE Linux and Mac tester configs."

This is a reland of fab66bae04

Includes necessary change to luci-scheduler.cfg and updated docs.

Original change's description:
> Add new ANGLE Linux and Mac tester configs.
>
> These configs will replace linux_angle_rel_ng and mac_angle_rel_ng.
> They initially mirror the configuration but will soon be updated to
> run only the subsets of tests that are most relevant to ANGLE. This
> should reduce flakiness in ANGLE and reduce load for Chrome.
>
> Also updates related configuration files to include linux-angle-rel
> and mac-angle-rel.
>
> Bug: 822310
> Change-Id: I2b40ed45f5d9e105c088ab166bd2207c96280151
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1551463
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#647854}

Bug: 822310
Tbr: jbudorick@chromium.org
Change-Id: Iaeeffc7b089b05d379bb353d627d13b39fd43704
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1554296
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#648727}
This commit is contained in:
Jamie Madill
2019-04-08 17:19:17 +00:00
committed by Commit Bot
parent e9ae8d506a
commit da894ce9c9
8 changed files with 2787 additions and 49 deletions

@ -535,63 +535,55 @@ chrome-infra team if this doesn't work as expected.)
[go/chromecals]: http://go/chromecals
### How to add a new "optional" try bot
### How to add a new try bot that runs a subset of tests or extra tests
TODO(kbr): the naming of the "optional" try bots is confusing and
unfortunate. They should probably be renamed to something like "extratests" or
"extra_tests", so perhaps a new naming convention of "gpu_win_extratests_rel" or
"win_gpu_extratests_rel". Unfortunately making this change at this point
requires touching tons of files across many workspaces and is unlikely to happen
unless someone highly motivated wants to pick up the task.
Several projects (ANGLE, Dawn) run custom tests using the Chromium recipes. They
use try bot bot configs that run subsets of Chromium or additional slower tests
that can't be run on the main CQ.
The "optional" GPU try bots are a concession to the reality that there are some
long-running GPU test suites that simply can not run against every Chromium CL.
They run some additional tests that are usually run only on the
chromium.gpu.fyi waterfall. Some of these tests, like the WebGL 2.0 conformance
suite, are intended to be run on the normal try bots once hardware capacity is
available. Some are not intended to ever run on the normal try bots.
These try bots are a little different because they mirror waterfall bots that
don't actually exist. The waterfall bots' specifications exist only to tell
these try bots which tests to run.
The optional try bots are a little different because they mirror waterfall bots
that don't actually exist. The waterfall bots' specifications exist only to
tell the optional try bots which tests to run.
Let's say that you intended to add a new such optional try bot on Windows. Call
it `win_new_optional_tests_rel` for example. Now, if you wanted to just add
this GPU type to the existing `win_optional_gpu_tests_rel` try bot, you'd
just follow the instructions above
([How to start running tests on a new GPU type on an existing try bot](#How-to-start-running-tests-on-a-new-GPU-type-on-an-existing-try-bot)). The steps below describe how to spin up
an entire new optional try bot.
Let's say that you intended to add a new such custom try bot on Windows. Call it
`win-myproject-rel` for example. You will need to add a "fake" mirror bot for
each GPU family the tests you will need to run. For a GPU type of
"CoolNewGPUType" in this example you could add a "fake" bot named "MyProject GPU
Win10 Release (CoolNewGPUType)".
1. Allocate new virtual machines for the bots as described in [How to set up
new virtual machine
instances](#How-to-set-up-new-virtual-machine-instances).
1. Make sure that you have some swarming capacity for the new GPU type. Since
it's not running against all Chromium CLs you don't need the recommended 30
minimum bots, though ~10 would be good.
1. Create a CL in the Chromium workspace:
1. Add your new bot (for example, "Optional Win7 Release
1. Create a CL in the Chromium workspace the does the following. Here's an
[example CL](https://crrev.com/c/1554296).
1. Add your new bot (for example, "MyProject GPU Win10 Release
(CoolNewGPUType)") to the chromium.gpu.fyi waterfall in
[waterfalls.pyl]. (Note, this is a bad example: the
"optional" bots have special semantics in this script. You'd probably
want to define some new category of bot if you didn't intend to add
this to `win_optional_gpu_tests_rel`.)
1. Re-run the script to regenerate the JSON files.
1. Land the above CL.
1. Create a CL in the tools/build workspace:
1. Modify `masters/master.tryserver.chromium.win`'s [master.cfg] and
[slaves.cfg] to add the new tryserver. Follow the pattern for the
existing `win_optional_gpu_tests_rel` tryserver. Namely, add the new
entry to master.cfg, and add the new tryserver to the
`optional_builders` list in `slaves.cfg`.
1. Modify [`chromium_gpu_fyi.py`][chromium_gpu_fyi.py] to add the new
"Optional Win7 Release (CoolNewGPUType)" entry.
1. Modify [`trybots.py`][trybots.py] to add
the new `win_new_optional_tests_rel` try bot, mirroring "Optional
Win7 Release (CoolNewGPUType)".
1. Land the above CL and request an off-hours restart of the
tryserver.chromium.win waterfall.
1. Now you can send CLs to the new bot with:
`git cl try -m tryserver.chromium.win -b win_new_optional_tests_rel`
[master.cfg]: https://chromium.googlesource.com/chromium/tools/build/+/master/masters/master.tryserver.chromium.win/master.cfg
[slaves.cfg]: https://chromium.googlesource.com/chromium/tools/build/+/master/masters/master.tryserver.chromium.win/slaves.cfg
[waterfalls.pyl].
1. Re-run [`src/testing/buildbot/generate_buildbot_json.py`][generate_buildbot_json.py] to regenerate the JSON files.
1. Update [`cr-buildbucket.cfg`][cr-buildbucket.cfg] to add `win-myproject-rel`.
1. Update [`luci-milo.cfg`][luci-milo.cfg] to include `win-myproject-rel`.
1. Update [`luci-scheduler.cfg`][luci-scheduler.cfg] to include "MyProject GPU Win10 Release
(CoolNewGPUType)".
1. Update [`src/tools/mb/mb_config.pyl`][mb_config.pyl] to include `win-myproject-rel`.
1. Also add your fake bot to [`src/testing/buildbot/generate_buildbot_json.py`][generate_buildbot_json.py] in the list of `get_bots_that_do_not_actually_exist` section.
1. *After* the Chromium-side CL lands and the bot is on the console, create a CL
in the [`tools/build`][tools/build] workspace which does the
following. Here's an [example CL](https://crrev.com/c/1554272).
1. Adds "MyProject GPU Win10 Release
(CoolNewGPUType)" to [`chromium_gpu_fyi.py`][chromium_gpu_fyi.py] in
`scripts/slave/recipe_modules/chromium_tests/`. You can copy a similar
step.
1. Adds `win-myproject-rel` to [`trybots.py`][trybots.py] in the same folder.
This is where you associate "MyProject GPU Win10 Release
(CoolNewGPUType)" with `win-myproject-rel`. See the sample CL for an example.
1. Get this reviewed and landed. This step tells the Chromium recipe about
the newly-deployed waterfall bot, so it knows which JSON file to load
out of src/testing/buildbot and which entry to look at.
1. After your CLs land you should be able to find and run `win-myproject-rel` on CLs
using Choose Trybots in Gerrit.
### How to test and deploy a driver update

@ -3506,6 +3506,7 @@ buckets {
name: "gpu-manual-try-linux-nvidia-tsn"
}
builders { mixins: "linux-try" name: "leak_detection_linux" }
builders { mixins: "linux-angle-try" name: "linux-angle-rel" }
builders { mixins: "linux-angle-try" name: "linux_angle_compile_dbg_ng" }
builders { mixins: "linux-angle-try" name: "linux_angle_dbg_ng" }
builders { mixins: "linux-angle-try" name: "linux_angle_deqp_rel_ng" }
@ -3625,6 +3626,7 @@ buckets {
builders { mixins: "ios-try" name: "ios-simulator-eg" }
builders { mixins: "ios-try" name: "ios-simulator-xcode-clang" }
builders { mixins: "ios-try" name: "ios-slimnav" }
builders { mixins: "mac-angle-try" name: "mac-angle-rel" }
builders { mixins: "mac-angle-try" name: "mac_angle_compile_dbg_ng" }
builders { mixins: "mac-angle-try" name: "mac_angle_dbg_ng" }
builders { mixins: "mac-angle-try" name: "mac_angle_rel_ng" }

@ -4477,6 +4477,9 @@ consoles {
builders {
name: "buildbucket/luci.chromium.try/android_angle_vk64_rel_ng"
}
builders {
name: "buildbucket/luci.chromium.try/linux-angle-rel"
}
builders {
name: "buildbucket/luci.chromium.try/linux_angle_dbg_ng"
}
@ -4489,6 +4492,9 @@ consoles {
builders {
name: "buildbucket/luci.chromium.try/linux_angle_rel_ng"
}
builders {
name: "buildbucket/luci.chromium.try/mac-angle-rel"
}
builders {
name: "buildbucket/luci.chromium.try/mac_angle_dbg_ng"
}
@ -4718,6 +4724,9 @@ consoles {
builders {
name: "buildbucket/luci.chromium.try/gpu-manual-try-win10-nvidia-rel"
}
builders {
name: "buildbucket/luci.chromium.try/linux-angle-rel"
}
builders {
name: "buildbucket/luci.chromium.try/linux-blink-heap-concurrent-marking-tsan-rel"
}
@ -4857,6 +4866,9 @@ consoles {
builders {
name: "buildbucket/luci.chromium.try/ios12-sdk-simulator"
}
builders {
name: "buildbucket/luci.chromium.try/mac-angle-rel"
}
builders {
name: "buildbucket/luci.chromium.try/mac-jumbo-rel"
}

@ -1311,6 +1311,22 @@ job {
# builder-Builders.
################################################################################
job {
id: "ANGLE GPU Linux Release (Intel HD 630)"
# Triggered by "GPU FYI Linux Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job {
id: "ANGLE GPU Linux Release (NVIDIA)"
# Triggered by "GPU FYI Linux Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job {
id: "Cast Audio Linux"
acl_sets: "default"
@ -1984,6 +2000,30 @@ job {
# Mac Builders. Sorted alphabetically.
################################################################################
job {
id: "ANGLE GPU Mac Release (Intel)"
# Triggered by "GPU FYI Mac Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job {
id: "ANGLE GPU Mac Retina Release (NVIDIA)"
# Triggered by "GPU FYI Mac Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job {
id: "ANGLE GPU Mac Retina Release (AMD)"
# Triggered by "GPU FYI Mac Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job {
id: "Dawn GPU Mac Release (Intel)"
# Triggered by "GPU FYI Mac Builder".

File diff suppressed because it is too large Load Diff

@ -920,6 +920,11 @@ class BBJSONGenerator(object):
# are defined only to be mirrored into trybots, and don't actually
# exist on any of the waterfalls or consoles.
return [
'ANGLE GPU Linux Release (Intel HD 630)',
'ANGLE GPU Linux Release (NVIDIA)',
'ANGLE GPU Mac Release (Intel)',
'ANGLE GPU Mac Retina Release (AMD)',
'ANGLE GPU Mac Retina Release (NVIDIA)',
'ANGLE GPU Win10 Release (Intel HD 630)',
'ANGLE GPU Win10 Release (NVIDIA)',
'Dawn GPU Linux Release (Intel HD 630)',

@ -2070,6 +2070,67 @@
'name': 'chromium.gpu.fyi',
'machines': {
# BEGIN Fake builder used as mirror targets for ANGLE's GPU tryservers
'ANGLE GPU Linux Release (Intel HD 630)': {
'os_type': 'linux',
'browser_config': 'release',
'mixins': [
'linux_intel_hd_630',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_linux_release_gtests',
'gpu_telemetry_tests': 'gpu_fyi_linux_intel_and_nvidia_release_telemetry_tests',
},
},
'ANGLE GPU Linux Release (NVIDIA)': {
'os_type': 'linux',
'browser_config': 'release',
'mixins': [
'linux_nvidia_quadro_p400',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_linux_release_gtests',
'isolated_scripts': 'gpu_angle_perftests',
'gpu_telemetry_tests': 'gpu_fyi_linux_intel_and_nvidia_release_telemetry_tests',
},
},
'ANGLE GPU Mac Release (Intel)': {
'os_type': 'mac',
'browser_config': 'release',
'mixins': [
'mac_mini_intel_gpu',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_mac_release_gtests',
'gpu_telemetry_tests': 'gpu_fyi_mac_release_telemetry_tests',
},
},
'ANGLE GPU Mac Retina Release (AMD)': {
'os_type': 'mac',
'browser_config': 'release',
'mixins': [
'mac_retina_amd_gpu',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_mac_release_gtests',
'gpu_telemetry_tests': 'gpu_fyi_mac_release_telemetry_tests',
},
},
'ANGLE GPU Mac Retina Release (NVIDIA)': {
'os_type': 'mac',
'browser_config': 'release',
'mixins': [
'mac_retina_nvidia_gpu',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_mac_release_gtests',
'gpu_telemetry_tests': 'gpu_fyi_mac_release_telemetry_tests',
},
},
'ANGLE GPU Win10 Release (Intel HD 630)': {
'os_type': 'win',
'browser_config': 'release',

@ -669,10 +669,12 @@
'android_angle_deqp_rel_ng': 'deqp_android_release_trybot_arm64',
'android_angle_vk32_deqp_rel_ng': 'deqp_android_vulkan_release_trybot',
'android_angle_vk64_deqp_rel_ng': 'deqp_android_vulkan_release_trybot_arm64',
'linux-angle-rel': 'gpu_fyi_tests_release_trybot',
'linux_angle_ozone_rel_ng': 'gpu_fyi_tests_ozone_linux_system_gbm_libdrm_release_trybot',
'linux_angle_dbg_ng': 'gpu_fyi_tests_debug_trybot',
'linux_angle_deqp_rel_ng': 'deqp_release_trybot',
'linux_angle_rel_ng': 'gpu_fyi_tests_release_trybot',
'mac-angle-rel': 'gpu_fyi_tests_release_trybot',
'mac_angle_dbg_ng': 'gpu_fyi_tests_debug_trybot',
'mac_angle_rel_ng': 'gpu_fyi_tests_release_trybot',
'win-angle-rel': 'gpu_fyi_tests_release_trybot_x86',