[tools/perf] Remove memory.top_10_mobile

Benchmark has been superseded by system_health.memory_mobile.

Also take the chance to update and fix some broken links in memory
benchmark docs.

TBR=crouleau@chromium.org

Bug: 761014
Change-Id: I8bd548bff0bc35cbba751ce2745e5162fbbcbec2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1688785
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Cr-Commit-Position: refs/heads/master@{#674997}

docs/memory-infra/memory_benchmarks.md

@@ -34,15 +34,12 @@ System health memory benchmarks are:
 *   [system_health.memory_desktop][system_health] -
     user stories running on desktop platforms.
 
-These benchmarks are run continuously on the [chromium.perf][] waterfall,
+These benchmarks are run continuously on the [chrome.perf][] waterfall,
 collecting and reporting results on the
 [Chrome Performance Dashboard][chromeperf].
 
-Other benchmarks maintained by the memory-infra team are discussed in the
-[appendix](#Other-benchmarks).
-
 [system_health]: https://chromium.googlesource.com/chromium/src/+/master/tools/perf/page_sets/system_health/
-[chromium.perf]: https://build.chromium.org/p/chromium.perf/waterfall
+[chrome.perf]: https://ci.chromium.org/p/chrome/g/chrome.perf/console
 [chromeperf]: https://chromeperf.appspot.com/report
 
 ### User stories
@@ -138,7 +135,7 @@ create a new job, and fill in the required details:
 If you have more specific needs, or need to automate the creation of jobs, you
 can also consider using [pinpoint_cli][].
 
-[pinpoint_cli]: https://cs.chromium.org/chromium/src/third_party/catapult/experimental/soundwave/bin/pinpoint_cli
+[pinpoint_cli]: https://cs.chromium.org/chromium/src/tools/perf/pinpoint_cli
 
 ### How to run locally
 
@@ -204,61 +201,3 @@ where:
     `private_dirty_size`).
 
 [memory-infra]: /docs/memory-infra/README.md
-
-## Appendix
-
-There are a few other benchmarks maintained by the memory-infra team.
-These also use the same set of metrics as system health, but have differences
-on the kind of stories that they run.
-
-### memory.top_10_mobile
-
-The [memory.top_10_mobile][memory_py] benchmark is in the process of being deprecated
-in favor of system health benchmarks. This process, however, hasn't been
-finalized and currently they are still the reference benchmark used for
-decision making in the Android release process. Therefore, **it is important
-to diagnose and fix regressions caught by this benchmark**.
-
-The benchmark's work flow is:
-
-- Cycle between:
-
-  - load a page on Chrome, wait for it to load, [force garbage collection
-    and measure memory][measure];
-  - push Chrome to the background, force garbage collection and measure
-    memory again.
-
-- Repeat for each of 10 pages *without closing the browser*.
-
-- Close the browser, re-open and repeat the full page set a total of 5 times.
-
-- Story groups are either `foreground` or `background` depending on the state
-  of the browser at the time of measurement.
-
-The main difference to watch out between this and system health benchmarks is
-that, since a single browser instance is kept open and shared by many
-individual stories, they are not independent of each other. In particular, **do
-not use the `--story-filter` argument when trying to reproduce regressions**
-on these benchmarks, as doing so will affect the results.
-
-[memory_py]: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/memory.py
-[measure]: https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/internal/actions/action_runner.py#L133
-
-### Dual browser benchmarks
-
-Dual browser benchmarks are intended to assess the memory implications of
-shared resources between Chrome and WebView.
-
-*   [memory.dual_browser_test][memory_extra_py] - cycle between doing Google
-    searches on a WebView-based browser (a stand-in for the Google Search app)
-    and loading pages on Chrome. Runs on Android devices only.
-
-    Story groups are either `on_chrome` or `on_webview`, indicating the browser
-    in foreground at the moment when the memory measurement was made.
-
-*   [memory.long_running_dual_browser_test][memory_extra_py] - same as above,
-    but the test is run for 60 iterations keeping both browsers alive for the
-    whole duration of the test and without forcing garbage collection. Intended
-    as a last-resort net to catch memory leaks not apparent on shorter tests.
-
-[memory_extra_py]: https://cs.chromium.org/chromium/src/tools/perf/contrib/memory_extras/memory_extras.py

tools/perf/benchmark.csv

@@ -31,7 +31,6 @@ media.desktop,dalecurtis@chromium.org,Internals>Media,,"aac,audio_only,audio_vid
 media.mobile,dalecurtis@chromium.org,Internals>Media,,"aac,audio_only,audio_video,background,beginning_to_end,busyjs,cns,h264,mp3,mse,opus,seek,src,video_only,vorbis,vp9"
 media_perftests,"crouleau@chromium.org, dalecurtis@chromium.org",Internals>Media,,
 memory.desktop,erikchen@chromium.org,,,
-memory.top_10_mobile,perezju@chromium.org,,,
 net_perftests,net-dev@chromium.org,Internals>Network,,
 octane,hablich@chromium.org,Blink>JavaScript,,
 passthrough_command_buffer_perftests,net-dev@chromium.org,Internals>Network,,

tools/perf/benchmarks/memory.py

@@ -99,23 +99,3 @@ class MemoryBenchmarkDesktop(_MemoryInfra):
   @classmethod
   def ShouldAddValue(cls, name, _):
     return DefaultShouldAddValueForMemoryMeasurement(name)
-
-
-@benchmark.Info(emails=['perezju@chromium.org'])
-class MemoryBenchmarkTop10Mobile(_MemoryInfra):
-  """Measure foreground/background memory on top 10 mobile page set.
-
-  This metric provides memory measurements for the System Health Plan of
-  Chrome on Android.
-  """
-  page_set = page_sets.MemoryTop10Mobile
-  options = {'pageset_repeat': 5}
-  SUPPORTED_PLATFORMS = [story.expectations.ALL_MOBILE]
-
-  @classmethod
-  def Name(cls):
-    return 'memory.top_10_mobile'
-
-  @classmethod
-  def ShouldAddValue(cls, name, _):
-    return DefaultShouldAddValueForMemoryMeasurement(name)

tools/perf/core/bot_platforms.py

@@ -107,7 +107,6 @@ _WIN_10_LOW_END_HP_CANDIDATE_BENCHMARK_NAMES = frozenset([
 _WIN_7_BENCHMARK_NAMES = _OFFICIAL_EXCEPT_DISPLAY_LOCKING
 _WIN_7_GPU_BENCHMARK_NAMES = _OFFICIAL_EXCEPT_DISPLAY_LOCKING
 _ANDROID_GO_BENCHMARK_NAMES = frozenset([
-    'memory.top_10_mobile',
     'system_health.memory_mobile',
     'system_health.common_mobile',
     'startup.mobile',

tools/perf/core/shard_maps/android-go-perf_map.json

@@ -1,17 +1,9 @@
 {
     "0": {
-        "benchmarks": {
+        "benchmarks": {}
-            "memory.top_10_mobile": {
-                "end": 9
-            }
-        }
     },
     "1": {
-        "benchmarks": {
+        "benchmarks": {}
-            "memory.top_10_mobile": {
-                "begin": 9
-            }
-        }
     },
     "2": {
         "benchmarks": {

tools/perf/core/shard_maps/android-go_webview-perf_map.json

@@ -1,25 +1,12 @@
 {
     "0": {
-        "benchmarks": {
+        "benchmarks": {}
-            "memory.top_10_mobile": {
-                "end": 8
-            }
-        }
     },
     "1": {
-        "benchmarks": {
+        "benchmarks": {}
-            "memory.top_10_mobile": {
-                "begin": 8,
-                "end": 14
-            }
-        }
     },
     "2": {
-        "benchmarks": {
+        "benchmarks": {}
-            "memory.top_10_mobile": {
-                "begin": 14
-            }
-        }
     },
     "3": {
         "benchmarks": {}

tools/perf/core/shard_maps/android-nexus5x-perf_map.json

@@ -58,7 +58,6 @@
             "media.mobile": {
                 "begin": 4
             },
-            "memory.top_10_mobile": {},
             "octane": {}
         }
     },

tools/perf/core/shard_maps/android-pixel2-perf_map.json

@@ -106,25 +106,14 @@
     },
     "11": {
         "benchmarks": {
-            "media.mobile": {},
+            "media.mobile": {}
-            "memory.top_10_mobile": {
-                "end": 5
-            }
         }
     },
     "12": {
-        "benchmarks": {
+        "benchmarks": {}
-            "memory.top_10_mobile": {
-                "begin": 5,
-                "end": 16
-            }
-        }
     },
     "13": {
         "benchmarks": {
-            "memory.top_10_mobile": {
-                "begin": 16
-            },
             "octane": {}
         }
     },

tools/perf/core/shard_maps/android-pixel2_webview-perf_map.json

@@ -85,17 +85,11 @@
             "loading.mobile": {
                 "begin": 92
             },
-            "media.mobile": {},
+            "media.mobile": {}
-            "memory.top_10_mobile": {
-                "end": 9
-            }
         }
     },
     "9": {
         "benchmarks": {
-            "memory.top_10_mobile": {
-                "begin": 9
-            },
             "octane": {}
         }
     },

tools/perf/core/shard_maps/android_nexus5_perf_map.json

@@ -55,17 +55,11 @@
             "loading.mobile": {
                 "begin": 69
             },
-            "media.mobile": {},
+            "media.mobile": {}
-            "memory.top_10_mobile": {
-                "end": 11
-            }
         }
     },
     "5": {
         "benchmarks": {
-            "memory.top_10_mobile": {
-                "begin": 11
-            },
             "octane": {},
             "rasterize_and_record_micro.partial_invalidation": {},
             "rasterize_and_record_micro.top_25": {},

tools/perf/core/shard_maps/android_nexus5x_webview_perf_map.json

@@ -47,17 +47,11 @@
             "loading.mobile": {
                 "begin": 43
             },
-            "media.mobile": {},
+            "media.mobile": {}
-            "memory.top_10_mobile": {
-                "end": 3
-            }
         }
     },
     "4": {
         "benchmarks": {
-            "memory.top_10_mobile": {
-                "begin": 3
-            },
             "octane": {}
         }
     },

tools/perf/core/shard_maps/android_nexus6_webview_perf_map.json

@@ -48,7 +48,6 @@
                 "begin": 70
             },
             "media.mobile": {},
-            "memory.top_10_mobile": {},
             "octane": {}
         }
     },

tools/perf/core/undocumented_benchmarks.py

@@ -19,7 +19,6 @@ UNDOCUMENTED_BENCHMARKS = {
   'media.mobile',
   'media_perftests',
   'memory.desktop',
-  'memory.top_10_mobile',
   'net_perftests',
   'octane',
   'passthrough_command_buffer_perftests',

tools/perf/examples/pinpoint_cli/bisect_job.json

@@ -1,16 +1,16 @@
 {
   "name": "Example bisect job",
-  "bug_id": "893896",
+  "bug_id": "944965",
-  "target": "performance_test_suite",
+  "target": "performance_webview_test_suite",
-  "configuration": "android-go-perf",
+  "configuration": "android-go_webview-perf",
-  "benchmark": "memory.top_10_mobile",
+  "benchmark": "system_health.memory_mobile",
   "comparison_mode": "performance",
-  "story": "http.m.youtube.com.results.q.science",
+  "story": "browse.media.facebook.photos",
-  "chart": "memory:chrome:all_processes:reported_by_os:system_memory:private_footprint_size",
+  "chart": "memory:webview:all_processes:reported_by_os:system_memory:private_dirty_size",
-  "tir_label": "foreground",
+  "tir_label": "browse_media",
-  "trace": "http_m_youtube_com_results_q_science",
+  "trace": "browse:media:facebook_photos",
   "statistic": "avg",
   "repository": "chromium",
-  "start_git_hash": "daa361c6e17be19c30de3896efeb63ca0034e9b8",
+  "start_git_hash": "dd0f551fcd96d46c6ec2b2a7c543cb2083f1831d",
-  "end_git_hash": "30588560782b21f7910332d49796948a3b482c19"
+  "end_git_hash": "e6808e8d7e973f43b0796505a09ee03caf6ec976"
 }

tools/perf/page_sets/memory_top_10_mobile.py

@@ -1,87 +0,0 @@
-# Copyright 2015 The Chromium Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-import re
-
-from telemetry.page import page as page_module
-from telemetry.page import shared_page_state
-from telemetry import story as story_module
-
-from devil.android.sdk import keyevent # pylint: disable=import-error
-
-from page_sets import top_10_mobile
-
-
-class Top10MobileSharedState(shared_page_state.SharedMobilePageState):
-  def ShouldStopBrowserAfterStoryRun(self, story):
-    # Close the browser after each background story.
-    return isinstance(story, BackgroundPage)
-
-
-class MemoryMeasurementPage(page_module.Page):
-  """Abstract class for measuring memory on a story unit."""
-
-  _PHASE = NotImplemented
-
-  def __init__(self, story_set, name, url):
-    super(MemoryMeasurementPage, self).__init__(
-        page_set=story_set, name=name, url=url,
-        shared_page_state_class=Top10MobileSharedState,
-        grouping_keys={'phase': self._PHASE})
-
-
-class ForegroundPage(MemoryMeasurementPage):
-  """Take a measurement after loading a regular webpage."""
-
-  _PHASE = 'foreground'
-
-  def __init__(self, story_set, name, url):
-    super(ForegroundPage, self).__init__(story_set, name, url)
-
-  def RunPageInteractions(self, action_runner):
-    action_runner.tab.WaitForDocumentReadyStateToBeComplete()
-    action_runner.MeasureMemory(self.story_set.DETERMINISTIC_MODE)
-
-
-class BackgroundPage(MemoryMeasurementPage):
-  """Take a measurement while Chrome is in the background."""
-
-  _PHASE = 'background'
-
-  def __init__(self, story_set, name):
-    super(BackgroundPage, self).__init__(story_set, name, 'about:blank')
-
-  def RunNavigateSteps(self, action_runner):
-    # Launch clock app, pushing Chrome to the background.
-    android_browser = action_runner.tab.browser
-    android_browser.Background()
-
-  def RunPageInteractions(self, action_runner):
-    # Take measurement.
-    action_runner.MeasureMemory(self.story_set.DETERMINISTIC_MODE)
-
-    # Go back to Chrome.
-    android_browser = action_runner.tab.browser
-    android_browser.platform.android_action_runner.InputKeyEvent(
-        keyevent.KEYCODE_BACK)
-
-
-class MemoryTop10Mobile(story_module.StorySet):
-  """User story to measure foreground/background memory in top 10 mobile."""
-  DETERMINISTIC_MODE = True
-
-  def __init__(self):
-    super(MemoryTop10Mobile, self).__init__(
-        archive_data_file='data/memory_top_10_mobile.json',
-        cloud_storage_bucket=story_module.PARTNER_BUCKET)
-
-    for url in top_10_mobile.URL_LIST:
-      # We name pages so their foreground/background counterparts are easy
-      # to identify. For example 'http://google.com' becomes
-      # 'http_google_com' and 'after_http_google_com' respectively.
-      # This also allows to use e.g. '--story-filter http_google_com' to
-      # match and run both parts of the story.
-      name = re.sub(r'\W+', '_', url)
-      self.AddStory(ForegroundPage(self, name, url))
-      self.AddStory(BackgroundPage(self, 'after_' + name))