0

[tools/perf] Remove memory.top_10_mobile

Benchmark has been superseded by system_health.memory_mobile.

Also take the chance to update and fix some broken links in memory
benchmark docs.

TBR=crouleau@chromium.org

Bug: 761014
Change-Id: I8bd548bff0bc35cbba751ce2745e5162fbbcbec2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1688785
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Cr-Commit-Position: refs/heads/master@{#674997}
This commit is contained in:
Juan Antonio Navarro Perez
2019-07-05 10:42:58 +00:00
committed by Commit Bot
parent 869c7f26a4
commit 2097a1d966
15 changed files with 23 additions and 246 deletions

@ -34,15 +34,12 @@ System health memory benchmarks are:
* [system_health.memory_desktop][system_health] -
user stories running on desktop platforms.
These benchmarks are run continuously on the [chromium.perf][] waterfall,
These benchmarks are run continuously on the [chrome.perf][] waterfall,
collecting and reporting results on the
[Chrome Performance Dashboard][chromeperf].
Other benchmarks maintained by the memory-infra team are discussed in the
[appendix](#Other-benchmarks).
[system_health]: https://chromium.googlesource.com/chromium/src/+/master/tools/perf/page_sets/system_health/
[chromium.perf]: https://build.chromium.org/p/chromium.perf/waterfall
[chrome.perf]: https://ci.chromium.org/p/chrome/g/chrome.perf/console
[chromeperf]: https://chromeperf.appspot.com/report
### User stories
@ -138,7 +135,7 @@ create a new job, and fill in the required details:
If you have more specific needs, or need to automate the creation of jobs, you
can also consider using [pinpoint_cli][].
[pinpoint_cli]: https://cs.chromium.org/chromium/src/third_party/catapult/experimental/soundwave/bin/pinpoint_cli
[pinpoint_cli]: https://cs.chromium.org/chromium/src/tools/perf/pinpoint_cli
### How to run locally
@ -204,61 +201,3 @@ where:
`private_dirty_size`).
[memory-infra]: /docs/memory-infra/README.md
## Appendix
There are a few other benchmarks maintained by the memory-infra team.
These also use the same set of metrics as system health, but have differences
on the kind of stories that they run.
### memory.top_10_mobile
The [memory.top_10_mobile][memory_py] benchmark is in the process of being deprecated
in favor of system health benchmarks. This process, however, hasn't been
finalized and currently they are still the reference benchmark used for
decision making in the Android release process. Therefore, **it is important
to diagnose and fix regressions caught by this benchmark**.
The benchmark's work flow is:
- Cycle between:
- load a page on Chrome, wait for it to load, [force garbage collection
and measure memory][measure];
- push Chrome to the background, force garbage collection and measure
memory again.
- Repeat for each of 10 pages *without closing the browser*.
- Close the browser, re-open and repeat the full page set a total of 5 times.
- Story groups are either `foreground` or `background` depending on the state
of the browser at the time of measurement.
The main difference to watch out between this and system health benchmarks is
that, since a single browser instance is kept open and shared by many
individual stories, they are not independent of each other. In particular, **do
not use the `--story-filter` argument when trying to reproduce regressions**
on these benchmarks, as doing so will affect the results.
[memory_py]: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/memory.py
[measure]: https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/internal/actions/action_runner.py#L133
### Dual browser benchmarks
Dual browser benchmarks are intended to assess the memory implications of
shared resources between Chrome and WebView.
* [memory.dual_browser_test][memory_extra_py] - cycle between doing Google
searches on a WebView-based browser (a stand-in for the Google Search app)
and loading pages on Chrome. Runs on Android devices only.
Story groups are either `on_chrome` or `on_webview`, indicating the browser
in foreground at the moment when the memory measurement was made.
* [memory.long_running_dual_browser_test][memory_extra_py] - same as above,
but the test is run for 60 iterations keeping both browsers alive for the
whole duration of the test and without forcing garbage collection. Intended
as a last-resort net to catch memory leaks not apparent on shorter tests.
[memory_extra_py]: https://cs.chromium.org/chromium/src/tools/perf/contrib/memory_extras/memory_extras.py

@ -31,7 +31,6 @@ media.desktop,dalecurtis@chromium.org,Internals>Media,,"aac,audio_only,audio_vid
media.mobile,dalecurtis@chromium.org,Internals>Media,,"aac,audio_only,audio_video,background,beginning_to_end,busyjs,cns,h264,mp3,mse,opus,seek,src,video_only,vorbis,vp9"
media_perftests,"crouleau@chromium.org, dalecurtis@chromium.org",Internals>Media,,
memory.desktop,erikchen@chromium.org,,,
memory.top_10_mobile,perezju@chromium.org,,,
net_perftests,net-dev@chromium.org,Internals>Network,,
octane,hablich@chromium.org,Blink>JavaScript,,
passthrough_command_buffer_perftests,net-dev@chromium.org,Internals>Network,,

Can't render this file because it has a wrong number of fields in line 3.

@ -99,23 +99,3 @@ class MemoryBenchmarkDesktop(_MemoryInfra):
@classmethod
def ShouldAddValue(cls, name, _):
return DefaultShouldAddValueForMemoryMeasurement(name)
@benchmark.Info(emails=['perezju@chromium.org'])
class MemoryBenchmarkTop10Mobile(_MemoryInfra):
"""Measure foreground/background memory on top 10 mobile page set.
This metric provides memory measurements for the System Health Plan of
Chrome on Android.
"""
page_set = page_sets.MemoryTop10Mobile
options = {'pageset_repeat': 5}
SUPPORTED_PLATFORMS = [story.expectations.ALL_MOBILE]
@classmethod
def Name(cls):
return 'memory.top_10_mobile'
@classmethod
def ShouldAddValue(cls, name, _):
return DefaultShouldAddValueForMemoryMeasurement(name)

@ -107,7 +107,6 @@ _WIN_10_LOW_END_HP_CANDIDATE_BENCHMARK_NAMES = frozenset([
_WIN_7_BENCHMARK_NAMES = _OFFICIAL_EXCEPT_DISPLAY_LOCKING
_WIN_7_GPU_BENCHMARK_NAMES = _OFFICIAL_EXCEPT_DISPLAY_LOCKING
_ANDROID_GO_BENCHMARK_NAMES = frozenset([
'memory.top_10_mobile',
'system_health.memory_mobile',
'system_health.common_mobile',
'startup.mobile',

@ -1,17 +1,9 @@
{
"0": {
"benchmarks": {
"memory.top_10_mobile": {
"end": 9
}
}
"benchmarks": {}
},
"1": {
"benchmarks": {
"memory.top_10_mobile": {
"begin": 9
}
}
"benchmarks": {}
},
"2": {
"benchmarks": {

@ -1,25 +1,12 @@
{
"0": {
"benchmarks": {
"memory.top_10_mobile": {
"end": 8
}
}
"benchmarks": {}
},
"1": {
"benchmarks": {
"memory.top_10_mobile": {
"begin": 8,
"end": 14
}
}
"benchmarks": {}
},
"2": {
"benchmarks": {
"memory.top_10_mobile": {
"begin": 14
}
}
"benchmarks": {}
},
"3": {
"benchmarks": {}

@ -58,7 +58,6 @@
"media.mobile": {
"begin": 4
},
"memory.top_10_mobile": {},
"octane": {}
}
},

@ -106,25 +106,14 @@
},
"11": {
"benchmarks": {
"media.mobile": {},
"memory.top_10_mobile": {
"end": 5
}
"media.mobile": {}
}
},
"12": {
"benchmarks": {
"memory.top_10_mobile": {
"begin": 5,
"end": 16
}
}
"benchmarks": {}
},
"13": {
"benchmarks": {
"memory.top_10_mobile": {
"begin": 16
},
"octane": {}
}
},

@ -85,17 +85,11 @@
"loading.mobile": {
"begin": 92
},
"media.mobile": {},
"memory.top_10_mobile": {
"end": 9
}
"media.mobile": {}
}
},
"9": {
"benchmarks": {
"memory.top_10_mobile": {
"begin": 9
},
"octane": {}
}
},

@ -55,17 +55,11 @@
"loading.mobile": {
"begin": 69
},
"media.mobile": {},
"memory.top_10_mobile": {
"end": 11
}
"media.mobile": {}
}
},
"5": {
"benchmarks": {
"memory.top_10_mobile": {
"begin": 11
},
"octane": {},
"rasterize_and_record_micro.partial_invalidation": {},
"rasterize_and_record_micro.top_25": {},

@ -47,17 +47,11 @@
"loading.mobile": {
"begin": 43
},
"media.mobile": {},
"memory.top_10_mobile": {
"end": 3
}
"media.mobile": {}
}
},
"4": {
"benchmarks": {
"memory.top_10_mobile": {
"begin": 3
},
"octane": {}
}
},

@ -48,7 +48,6 @@
"begin": 70
},
"media.mobile": {},
"memory.top_10_mobile": {},
"octane": {}
}
},

@ -19,7 +19,6 @@ UNDOCUMENTED_BENCHMARKS = {
'media.mobile',
'media_perftests',
'memory.desktop',
'memory.top_10_mobile',
'net_perftests',
'octane',
'passthrough_command_buffer_perftests',

@ -1,16 +1,16 @@
{
"name": "Example bisect job",
"bug_id": "893896",
"target": "performance_test_suite",
"configuration": "android-go-perf",
"benchmark": "memory.top_10_mobile",
"bug_id": "944965",
"target": "performance_webview_test_suite",
"configuration": "android-go_webview-perf",
"benchmark": "system_health.memory_mobile",
"comparison_mode": "performance",
"story": "http.m.youtube.com.results.q.science",
"chart": "memory:chrome:all_processes:reported_by_os:system_memory:private_footprint_size",
"tir_label": "foreground",
"trace": "http_m_youtube_com_results_q_science",
"story": "browse.media.facebook.photos",
"chart": "memory:webview:all_processes:reported_by_os:system_memory:private_dirty_size",
"tir_label": "browse_media",
"trace": "browse:media:facebook_photos",
"statistic": "avg",
"repository": "chromium",
"start_git_hash": "daa361c6e17be19c30de3896efeb63ca0034e9b8",
"end_git_hash": "30588560782b21f7910332d49796948a3b482c19"
"start_git_hash": "dd0f551fcd96d46c6ec2b2a7c543cb2083f1831d",
"end_git_hash": "e6808e8d7e973f43b0796505a09ee03caf6ec976"
}

@ -1,87 +0,0 @@
# Copyright 2015 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
import re
from telemetry.page import page as page_module
from telemetry.page import shared_page_state
from telemetry import story as story_module
from devil.android.sdk import keyevent # pylint: disable=import-error
from page_sets import top_10_mobile
class Top10MobileSharedState(shared_page_state.SharedMobilePageState):
def ShouldStopBrowserAfterStoryRun(self, story):
# Close the browser after each background story.
return isinstance(story, BackgroundPage)
class MemoryMeasurementPage(page_module.Page):
"""Abstract class for measuring memory on a story unit."""
_PHASE = NotImplemented
def __init__(self, story_set, name, url):
super(MemoryMeasurementPage, self).__init__(
page_set=story_set, name=name, url=url,
shared_page_state_class=Top10MobileSharedState,
grouping_keys={'phase': self._PHASE})
class ForegroundPage(MemoryMeasurementPage):
"""Take a measurement after loading a regular webpage."""
_PHASE = 'foreground'
def __init__(self, story_set, name, url):
super(ForegroundPage, self).__init__(story_set, name, url)
def RunPageInteractions(self, action_runner):
action_runner.tab.WaitForDocumentReadyStateToBeComplete()
action_runner.MeasureMemory(self.story_set.DETERMINISTIC_MODE)
class BackgroundPage(MemoryMeasurementPage):
"""Take a measurement while Chrome is in the background."""
_PHASE = 'background'
def __init__(self, story_set, name):
super(BackgroundPage, self).__init__(story_set, name, 'about:blank')
def RunNavigateSteps(self, action_runner):
# Launch clock app, pushing Chrome to the background.
android_browser = action_runner.tab.browser
android_browser.Background()
def RunPageInteractions(self, action_runner):
# Take measurement.
action_runner.MeasureMemory(self.story_set.DETERMINISTIC_MODE)
# Go back to Chrome.
android_browser = action_runner.tab.browser
android_browser.platform.android_action_runner.InputKeyEvent(
keyevent.KEYCODE_BACK)
class MemoryTop10Mobile(story_module.StorySet):
"""User story to measure foreground/background memory in top 10 mobile."""
DETERMINISTIC_MODE = True
def __init__(self):
super(MemoryTop10Mobile, self).__init__(
archive_data_file='data/memory_top_10_mobile.json',
cloud_storage_bucket=story_module.PARTNER_BUCKET)
for url in top_10_mobile.URL_LIST:
# We name pages so their foreground/background counterparts are easy
# to identify. For example 'http://google.com' becomes
# 'http_google_com' and 'after_http_google_com' respectively.
# This also allows to use e.g. '--story-filter http_google_com' to
# match and run both parts of the story.
name = re.sub(r'\W+', '_', url)
self.AddStory(ForegroundPage(self, name, url))
self.AddStory(BackgroundPage(self, 'after_' + name))