[tools/perf] Remove memory.top_10_mobile
Benchmark has been superseded by system_health.memory_mobile. Also take the chance to update and fix some broken links in memory benchmark docs. TBR=crouleau@chromium.org Bug: 761014 Change-Id: I8bd548bff0bc35cbba751ce2745e5162fbbcbec2 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1688785 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#674997}
This commit is contained in:

committed by
Commit Bot

parent
869c7f26a4
commit
2097a1d966
docs/memory-infra
tools/perf
benchmark.csv
benchmarks
core
bot_platforms.py
shard_maps
android-go-perf_map.jsonandroid-go_webview-perf_map.jsonandroid-nexus5x-perf_map.jsonandroid-pixel2-perf_map.jsonandroid-pixel2_webview-perf_map.jsonandroid_nexus5_perf_map.jsonandroid_nexus5x_webview_perf_map.jsonandroid_nexus6_webview_perf_map.json
undocumented_benchmarks.pyexamples
pinpoint_cli
page_sets
@ -34,15 +34,12 @@ System health memory benchmarks are:
|
||||
* [system_health.memory_desktop][system_health] -
|
||||
user stories running on desktop platforms.
|
||||
|
||||
These benchmarks are run continuously on the [chromium.perf][] waterfall,
|
||||
These benchmarks are run continuously on the [chrome.perf][] waterfall,
|
||||
collecting and reporting results on the
|
||||
[Chrome Performance Dashboard][chromeperf].
|
||||
|
||||
Other benchmarks maintained by the memory-infra team are discussed in the
|
||||
[appendix](#Other-benchmarks).
|
||||
|
||||
[system_health]: https://chromium.googlesource.com/chromium/src/+/master/tools/perf/page_sets/system_health/
|
||||
[chromium.perf]: https://build.chromium.org/p/chromium.perf/waterfall
|
||||
[chrome.perf]: https://ci.chromium.org/p/chrome/g/chrome.perf/console
|
||||
[chromeperf]: https://chromeperf.appspot.com/report
|
||||
|
||||
### User stories
|
||||
@ -138,7 +135,7 @@ create a new job, and fill in the required details:
|
||||
If you have more specific needs, or need to automate the creation of jobs, you
|
||||
can also consider using [pinpoint_cli][].
|
||||
|
||||
[pinpoint_cli]: https://cs.chromium.org/chromium/src/third_party/catapult/experimental/soundwave/bin/pinpoint_cli
|
||||
[pinpoint_cli]: https://cs.chromium.org/chromium/src/tools/perf/pinpoint_cli
|
||||
|
||||
### How to run locally
|
||||
|
||||
@ -204,61 +201,3 @@ where:
|
||||
`private_dirty_size`).
|
||||
|
||||
[memory-infra]: /docs/memory-infra/README.md
|
||||
|
||||
## Appendix
|
||||
|
||||
There are a few other benchmarks maintained by the memory-infra team.
|
||||
These also use the same set of metrics as system health, but have differences
|
||||
on the kind of stories that they run.
|
||||
|
||||
### memory.top_10_mobile
|
||||
|
||||
The [memory.top_10_mobile][memory_py] benchmark is in the process of being deprecated
|
||||
in favor of system health benchmarks. This process, however, hasn't been
|
||||
finalized and currently they are still the reference benchmark used for
|
||||
decision making in the Android release process. Therefore, **it is important
|
||||
to diagnose and fix regressions caught by this benchmark**.
|
||||
|
||||
The benchmark's work flow is:
|
||||
|
||||
- Cycle between:
|
||||
|
||||
- load a page on Chrome, wait for it to load, [force garbage collection
|
||||
and measure memory][measure];
|
||||
- push Chrome to the background, force garbage collection and measure
|
||||
memory again.
|
||||
|
||||
- Repeat for each of 10 pages *without closing the browser*.
|
||||
|
||||
- Close the browser, re-open and repeat the full page set a total of 5 times.
|
||||
|
||||
- Story groups are either `foreground` or `background` depending on the state
|
||||
of the browser at the time of measurement.
|
||||
|
||||
The main difference to watch out between this and system health benchmarks is
|
||||
that, since a single browser instance is kept open and shared by many
|
||||
individual stories, they are not independent of each other. In particular, **do
|
||||
not use the `--story-filter` argument when trying to reproduce regressions**
|
||||
on these benchmarks, as doing so will affect the results.
|
||||
|
||||
[memory_py]: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/memory.py
|
||||
[measure]: https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/internal/actions/action_runner.py#L133
|
||||
|
||||
### Dual browser benchmarks
|
||||
|
||||
Dual browser benchmarks are intended to assess the memory implications of
|
||||
shared resources between Chrome and WebView.
|
||||
|
||||
* [memory.dual_browser_test][memory_extra_py] - cycle between doing Google
|
||||
searches on a WebView-based browser (a stand-in for the Google Search app)
|
||||
and loading pages on Chrome. Runs on Android devices only.
|
||||
|
||||
Story groups are either `on_chrome` or `on_webview`, indicating the browser
|
||||
in foreground at the moment when the memory measurement was made.
|
||||
|
||||
* [memory.long_running_dual_browser_test][memory_extra_py] - same as above,
|
||||
but the test is run for 60 iterations keeping both browsers alive for the
|
||||
whole duration of the test and without forcing garbage collection. Intended
|
||||
as a last-resort net to catch memory leaks not apparent on shorter tests.
|
||||
|
||||
[memory_extra_py]: https://cs.chromium.org/chromium/src/tools/perf/contrib/memory_extras/memory_extras.py
|
||||
|
@ -31,7 +31,6 @@ media.desktop,dalecurtis@chromium.org,Internals>Media,,"aac,audio_only,audio_vid
|
||||
media.mobile,dalecurtis@chromium.org,Internals>Media,,"aac,audio_only,audio_video,background,beginning_to_end,busyjs,cns,h264,mp3,mse,opus,seek,src,video_only,vorbis,vp9"
|
||||
media_perftests,"crouleau@chromium.org, dalecurtis@chromium.org",Internals>Media,,
|
||||
memory.desktop,erikchen@chromium.org,,,
|
||||
memory.top_10_mobile,perezju@chromium.org,,,
|
||||
net_perftests,net-dev@chromium.org,Internals>Network,,
|
||||
octane,hablich@chromium.org,Blink>JavaScript,,
|
||||
passthrough_command_buffer_perftests,net-dev@chromium.org,Internals>Network,,
|
||||
|
Can't render this file because it has a wrong number of fields in line 3.
|
@ -99,23 +99,3 @@ class MemoryBenchmarkDesktop(_MemoryInfra):
|
||||
@classmethod
|
||||
def ShouldAddValue(cls, name, _):
|
||||
return DefaultShouldAddValueForMemoryMeasurement(name)
|
||||
|
||||
|
||||
@benchmark.Info(emails=['perezju@chromium.org'])
|
||||
class MemoryBenchmarkTop10Mobile(_MemoryInfra):
|
||||
"""Measure foreground/background memory on top 10 mobile page set.
|
||||
|
||||
This metric provides memory measurements for the System Health Plan of
|
||||
Chrome on Android.
|
||||
"""
|
||||
page_set = page_sets.MemoryTop10Mobile
|
||||
options = {'pageset_repeat': 5}
|
||||
SUPPORTED_PLATFORMS = [story.expectations.ALL_MOBILE]
|
||||
|
||||
@classmethod
|
||||
def Name(cls):
|
||||
return 'memory.top_10_mobile'
|
||||
|
||||
@classmethod
|
||||
def ShouldAddValue(cls, name, _):
|
||||
return DefaultShouldAddValueForMemoryMeasurement(name)
|
||||
|
@ -107,7 +107,6 @@ _WIN_10_LOW_END_HP_CANDIDATE_BENCHMARK_NAMES = frozenset([
|
||||
_WIN_7_BENCHMARK_NAMES = _OFFICIAL_EXCEPT_DISPLAY_LOCKING
|
||||
_WIN_7_GPU_BENCHMARK_NAMES = _OFFICIAL_EXCEPT_DISPLAY_LOCKING
|
||||
_ANDROID_GO_BENCHMARK_NAMES = frozenset([
|
||||
'memory.top_10_mobile',
|
||||
'system_health.memory_mobile',
|
||||
'system_health.common_mobile',
|
||||
'startup.mobile',
|
||||
|
@ -1,17 +1,9 @@
|
||||
{
|
||||
"0": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"end": 9
|
||||
}
|
||||
}
|
||||
"benchmarks": {}
|
||||
},
|
||||
"1": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"begin": 9
|
||||
}
|
||||
}
|
||||
"benchmarks": {}
|
||||
},
|
||||
"2": {
|
||||
"benchmarks": {
|
||||
|
@ -1,25 +1,12 @@
|
||||
{
|
||||
"0": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"end": 8
|
||||
}
|
||||
}
|
||||
"benchmarks": {}
|
||||
},
|
||||
"1": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"begin": 8,
|
||||
"end": 14
|
||||
}
|
||||
}
|
||||
"benchmarks": {}
|
||||
},
|
||||
"2": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"begin": 14
|
||||
}
|
||||
}
|
||||
"benchmarks": {}
|
||||
},
|
||||
"3": {
|
||||
"benchmarks": {}
|
||||
|
@ -58,7 +58,6 @@
|
||||
"media.mobile": {
|
||||
"begin": 4
|
||||
},
|
||||
"memory.top_10_mobile": {},
|
||||
"octane": {}
|
||||
}
|
||||
},
|
||||
|
@ -106,25 +106,14 @@
|
||||
},
|
||||
"11": {
|
||||
"benchmarks": {
|
||||
"media.mobile": {},
|
||||
"memory.top_10_mobile": {
|
||||
"end": 5
|
||||
}
|
||||
"media.mobile": {}
|
||||
}
|
||||
},
|
||||
"12": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"begin": 5,
|
||||
"end": 16
|
||||
}
|
||||
}
|
||||
"benchmarks": {}
|
||||
},
|
||||
"13": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"begin": 16
|
||||
},
|
||||
"octane": {}
|
||||
}
|
||||
},
|
||||
|
@ -85,17 +85,11 @@
|
||||
"loading.mobile": {
|
||||
"begin": 92
|
||||
},
|
||||
"media.mobile": {},
|
||||
"memory.top_10_mobile": {
|
||||
"end": 9
|
||||
}
|
||||
"media.mobile": {}
|
||||
}
|
||||
},
|
||||
"9": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"begin": 9
|
||||
},
|
||||
"octane": {}
|
||||
}
|
||||
},
|
||||
|
@ -55,17 +55,11 @@
|
||||
"loading.mobile": {
|
||||
"begin": 69
|
||||
},
|
||||
"media.mobile": {},
|
||||
"memory.top_10_mobile": {
|
||||
"end": 11
|
||||
}
|
||||
"media.mobile": {}
|
||||
}
|
||||
},
|
||||
"5": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"begin": 11
|
||||
},
|
||||
"octane": {},
|
||||
"rasterize_and_record_micro.partial_invalidation": {},
|
||||
"rasterize_and_record_micro.top_25": {},
|
||||
|
@ -47,17 +47,11 @@
|
||||
"loading.mobile": {
|
||||
"begin": 43
|
||||
},
|
||||
"media.mobile": {},
|
||||
"memory.top_10_mobile": {
|
||||
"end": 3
|
||||
}
|
||||
"media.mobile": {}
|
||||
}
|
||||
},
|
||||
"4": {
|
||||
"benchmarks": {
|
||||
"memory.top_10_mobile": {
|
||||
"begin": 3
|
||||
},
|
||||
"octane": {}
|
||||
}
|
||||
},
|
||||
|
@ -48,7 +48,6 @@
|
||||
"begin": 70
|
||||
},
|
||||
"media.mobile": {},
|
||||
"memory.top_10_mobile": {},
|
||||
"octane": {}
|
||||
}
|
||||
},
|
||||
|
@ -19,7 +19,6 @@ UNDOCUMENTED_BENCHMARKS = {
|
||||
'media.mobile',
|
||||
'media_perftests',
|
||||
'memory.desktop',
|
||||
'memory.top_10_mobile',
|
||||
'net_perftests',
|
||||
'octane',
|
||||
'passthrough_command_buffer_perftests',
|
||||
|
@ -1,16 +1,16 @@
|
||||
{
|
||||
"name": "Example bisect job",
|
||||
"bug_id": "893896",
|
||||
"target": "performance_test_suite",
|
||||
"configuration": "android-go-perf",
|
||||
"benchmark": "memory.top_10_mobile",
|
||||
"bug_id": "944965",
|
||||
"target": "performance_webview_test_suite",
|
||||
"configuration": "android-go_webview-perf",
|
||||
"benchmark": "system_health.memory_mobile",
|
||||
"comparison_mode": "performance",
|
||||
"story": "http.m.youtube.com.results.q.science",
|
||||
"chart": "memory:chrome:all_processes:reported_by_os:system_memory:private_footprint_size",
|
||||
"tir_label": "foreground",
|
||||
"trace": "http_m_youtube_com_results_q_science",
|
||||
"story": "browse.media.facebook.photos",
|
||||
"chart": "memory:webview:all_processes:reported_by_os:system_memory:private_dirty_size",
|
||||
"tir_label": "browse_media",
|
||||
"trace": "browse:media:facebook_photos",
|
||||
"statistic": "avg",
|
||||
"repository": "chromium",
|
||||
"start_git_hash": "daa361c6e17be19c30de3896efeb63ca0034e9b8",
|
||||
"end_git_hash": "30588560782b21f7910332d49796948a3b482c19"
|
||||
"start_git_hash": "dd0f551fcd96d46c6ec2b2a7c543cb2083f1831d",
|
||||
"end_git_hash": "e6808e8d7e973f43b0796505a09ee03caf6ec976"
|
||||
}
|
||||
|
@ -1,87 +0,0 @@
|
||||
# Copyright 2015 The Chromium Authors. All rights reserved.
|
||||
# Use of this source code is governed by a BSD-style license that can be
|
||||
# found in the LICENSE file.
|
||||
|
||||
import re
|
||||
|
||||
from telemetry.page import page as page_module
|
||||
from telemetry.page import shared_page_state
|
||||
from telemetry import story as story_module
|
||||
|
||||
from devil.android.sdk import keyevent # pylint: disable=import-error
|
||||
|
||||
from page_sets import top_10_mobile
|
||||
|
||||
|
||||
class Top10MobileSharedState(shared_page_state.SharedMobilePageState):
|
||||
def ShouldStopBrowserAfterStoryRun(self, story):
|
||||
# Close the browser after each background story.
|
||||
return isinstance(story, BackgroundPage)
|
||||
|
||||
|
||||
class MemoryMeasurementPage(page_module.Page):
|
||||
"""Abstract class for measuring memory on a story unit."""
|
||||
|
||||
_PHASE = NotImplemented
|
||||
|
||||
def __init__(self, story_set, name, url):
|
||||
super(MemoryMeasurementPage, self).__init__(
|
||||
page_set=story_set, name=name, url=url,
|
||||
shared_page_state_class=Top10MobileSharedState,
|
||||
grouping_keys={'phase': self._PHASE})
|
||||
|
||||
|
||||
class ForegroundPage(MemoryMeasurementPage):
|
||||
"""Take a measurement after loading a regular webpage."""
|
||||
|
||||
_PHASE = 'foreground'
|
||||
|
||||
def __init__(self, story_set, name, url):
|
||||
super(ForegroundPage, self).__init__(story_set, name, url)
|
||||
|
||||
def RunPageInteractions(self, action_runner):
|
||||
action_runner.tab.WaitForDocumentReadyStateToBeComplete()
|
||||
action_runner.MeasureMemory(self.story_set.DETERMINISTIC_MODE)
|
||||
|
||||
|
||||
class BackgroundPage(MemoryMeasurementPage):
|
||||
"""Take a measurement while Chrome is in the background."""
|
||||
|
||||
_PHASE = 'background'
|
||||
|
||||
def __init__(self, story_set, name):
|
||||
super(BackgroundPage, self).__init__(story_set, name, 'about:blank')
|
||||
|
||||
def RunNavigateSteps(self, action_runner):
|
||||
# Launch clock app, pushing Chrome to the background.
|
||||
android_browser = action_runner.tab.browser
|
||||
android_browser.Background()
|
||||
|
||||
def RunPageInteractions(self, action_runner):
|
||||
# Take measurement.
|
||||
action_runner.MeasureMemory(self.story_set.DETERMINISTIC_MODE)
|
||||
|
||||
# Go back to Chrome.
|
||||
android_browser = action_runner.tab.browser
|
||||
android_browser.platform.android_action_runner.InputKeyEvent(
|
||||
keyevent.KEYCODE_BACK)
|
||||
|
||||
|
||||
class MemoryTop10Mobile(story_module.StorySet):
|
||||
"""User story to measure foreground/background memory in top 10 mobile."""
|
||||
DETERMINISTIC_MODE = True
|
||||
|
||||
def __init__(self):
|
||||
super(MemoryTop10Mobile, self).__init__(
|
||||
archive_data_file='data/memory_top_10_mobile.json',
|
||||
cloud_storage_bucket=story_module.PARTNER_BUCKET)
|
||||
|
||||
for url in top_10_mobile.URL_LIST:
|
||||
# We name pages so their foreground/background counterparts are easy
|
||||
# to identify. For example 'http://google.com' becomes
|
||||
# 'http_google_com' and 'after_http_google_com' respectively.
|
||||
# This also allows to use e.g. '--story-filter http_google_com' to
|
||||
# match and run both parts of the story.
|
||||
name = re.sub(r'\W+', '_', url)
|
||||
self.AddStory(ForegroundPage(self, name, url))
|
||||
self.AddStory(BackgroundPage(self, 'after_' + name))
|
Reference in New Issue
Block a user