This includes removing most direct references and uses from telemetry.
BUG=267773,476709
Review URL: https://codereview.chromium.org/1132993004
Cr-Commit-Position: refs/heads/master@{#329963}
Motivation: when I make benchmark_runner._PrintBenchmarkList a public method,
I want to provide unittest coverage for the method. I would want to reuse the
logic of TestStreamOutput for writing that unittest.
BUG=482090
Review URL: https://codereview.chromium.org/1112773003
Cr-Commit-Position: refs/heads/master@{#327610}
For the full list of changes, see
94d7e2a...4e4d540.
WebPageReplay moved the method has_sni() from certutils to platformsettings,
and renamed it to HasSniSupport(). This CL fixes the one location where
has_sni() is used, and also fixes the stubs.
BUG=477130
Review URL: https://codereview.chromium.org/1087093002
Cr-Commit-Position: refs/heads/master@{#325313}
As part of the Telemetry public API refactor, I wrote a tool that sorts imports.
The tool will be checked in separately, but here's all the import changes in Telemetry and its dependencies.
It'll be easier to test the refactor tool if we do the import sorting in advance, so it doesn't clutter the diffs for the module moves.
Style guide rule: "Within each grouping, imports should be sorted lexicographically, ignoring case, according to each module's full package path."
BUG=449308
TEST=trybots (tools/telemetry/run_tests; tools/perf/run_tests)
Review URL: https://codereview.chromium.org/1029263003
Cr-Commit-Position: refs/heads/master@{#322211}
This CL removes the is_smooth flag from Telemetry, and refactors existing tests
and usages on a case-by-case basis to take this into account.
BUG=444697
R=nednguyen,sullivan
Review URL: https://codereview.chromium.org/1013803003
Cr-Commit-Position: refs/heads/master@{#321015}
1) Move the serving_dir related logic to one place and add some comments to improve hackability and stability for cloud_storage in telemetry.
2) Make serving_dirs available for user stories as part of the refactor to remove page_sets.
This is the second part of https://codereview.chromium.org/794493004/, broken out for reviewer ease.
BUG=454531
BUG=435063
Committed: https://crrev.com/eed596dced7b9e3f7d21c84f695a39b946c62bf2
Cr-Commit-Position: refs/heads/master@{#320523}
Review URL: https://codereview.chromium.org/838253005
Cr-Commit-Position: refs/heads/master@{#320990}
1) Move the serving_dir related logic to one place and add some comments to improve hackability and stability for cloud_storage in telemetry.
2) Make serving_dirs available for user stories as part of the refactor to remove page_sets.
This is the second part of https://codereview.chromium.org/794493004/, broken out for reviewer ease.
BUG=454531
BUG=435063
Review URL: https://codereview.chromium.org/838253005
Cr-Commit-Position: refs/heads/master@{#320523}
Also adds more fields to environment to narrow the scope of
benchmark and user story set discovery. This should avoid problems
with adding Python files to unrelated directories, and hides PageTests
from external Telemetry benchmark runners like run_gpu_tests.py and
chrome_proxy's run_benchmark.
R=dtu,nednguyen,sullivan,kbr@chromium.org,bolian
BUG=460181
TEST=tools/perf/run_benchmark; content/test/gpu/run_gpu_tests.py; tools/chrome_proxy/run_benchmark # All return a full and correct test list.
Committed: https://crrev.com/1da5f7f70ea6dc7dd0667ea78637802c76305f5a
Cr-Commit-Position: refs/heads/master@{#318149}
Review URL: https://codereview.chromium.org/942663002
Cr-Commit-Position: refs/heads/master@{#318531}
This change helps move the Telemetry framework towards saner exception
handling.
BUG=460625
Review URL: https://codereview.chromium.org/952693003
Cr-Commit-Position: refs/heads/master@{#318349}
Reason for revert:
crbug.com/462063
Original issue's description:
> [Telemetry] Pass test_runner environment in local args instead of a global variable
>
> Also adds more fields to environment to narrow the scope of
> benchmark and user story set discovery. This should avoid problems
> with adding Python files to unrelated directories, and hides PageTests
> from external Telemetry benchmark runners like run_gpu_tests.py and
> chrome_proxy's run_benchmark.
>
> R=dtu,nednguyen,sullivan,kbr@chromium.org,bolian
> BUG=460181
> TEST=tools/perf/run_benchmark; content/test/gpu/run_gpu_tests.py; tools/chrome_proxy/run_benchmark # All return a full and correct test list.
>
> Committed: https://crrev.com/1da5f7f70ea6dc7dd0667ea78637802c76305f5a
> Cr-Commit-Position: refs/heads/master@{#318149}
TBR=bolian@chromium.org,dtu@chromium.org,kbr@chromium.org,nednguyen@google.com,sullivan@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=460181
Review URL: https://codereview.chromium.org/955183003
Cr-Commit-Position: refs/heads/master@{#318175}
Also adds more fields to environment to narrow the scope of
benchmark and user story set discovery. This should avoid problems
with adding Python files to unrelated directories, and hides PageTests
from external Telemetry benchmark runners like run_gpu_tests.py and
chrome_proxy's run_benchmark.
R=dtu,nednguyen,sullivan,kbr@chromium.org,bolian
BUG=460181
TEST=tools/perf/run_benchmark; content/test/gpu/run_gpu_tests.py; tools/chrome_proxy/run_benchmark # All return a full and correct test list.
Review URL: https://codereview.chromium.org/942663002
Cr-Commit-Position: refs/heads/master@{#318149}
This CL also exposes the property tabs on the class TabTestCase.
BUG=442546
Review URL: https://codereview.chromium.org/940153002
Cr-Commit-Position: refs/heads/master@{#317662}
Reason for revert:
Original CL was ok. Failures were most likely due to https://codereview.chromium.org/914303007/ instead.
Original issue's description:
> Revert of [Telemtry] Move url lists of alexa1-10000 page to a json file. (patchset #1 id:1 of https://codereview.chromium.org/915303002/)
>
> Reason for revert:
> Build broke on many bots:
>
> Example:
> https://build.chromium.org/p/chromium.win/builders/Win7%20Tests%20%281%29/builds/36194
>
> Running ['E:\\b\\depot_tools\\python276_bin\\python.exe', u'E:\\b\\build\\scripts\\tools\\runit.py', '--show-path', 'E:\\b\\depot_tools\\python276_bin\\python.exe', u'E:\\b\\build\\scripts\\slave\\runtest.py', '--target', 'Release', '--xvfb', '--builder-name', u'Win7 Tests (1)', '--slave-name', u'vm801-m1', '--build-number', '36194', '--build-properties', '{"buildnumber": 36194, "slavename": "vm801-m1", "buildername": "Win7 Tests (1)"}', '--annotate', 'gtest', '--test-type', 'telemetry_unittests', '--run-python-script', 'E:\\b\\build\\slave\\Win7_Tests__1_\\build\\src\\tools\\telemetry\\run_tests', '--browser', 'release', '--retry-limit', '3', '--write-full-results-to', 'c:\\users\\chrome~2\\appdata\\local\\temp\\tmpo3eir6']
> Set PYTHONPATH: E:\b\build\..\build_internal\site_config;E:\b\build\third_party\httplib2\python2;E:\b\build\third_party\zope;E:\b\build\third_party\uritemplate;E:\b\build\third_party\twisted_8_1;E:\b\build\third_party\tempita_0_5;E:\b\build\third_party\swarm_client;E:\b\build\third_party\sqlalchemy_migrate_0_7_1;E:\b\build\third_party\sqlalchemy_0_7_1;E:\b\build\third_party\simplejson;E:\b\build\third_party\setuptools-0.6c11;E:\b\build\third_party\pytz;E:\b\build\third_party\psutils;E:\b\build\third_party\oauth2client;E:\b\build\third_party\mock-1.0.1;E:\b\build\third_party\mock-0.7.2;E:\b\build\third_party\mock-0.6.0;E:\b\build\third_party\markupsafe;E:\b\build\third_party\lighttpd;E:\b\build\third_party\jinja2;E:\b\build\third_party\infra;E:\b\build\third_party\httplib2;E:\b\build\third_party\gsutil;E:\b\build\third_party\google_api_python_client;E:\b\build\third_party\depot_tools_patch;E:\b\build\third_party\depot_tools;E:\b\build\third_party\decorator_3_3_1;E:\b\build\third_party\Crypto;E:\b\build\third_party\coverage-3.7.1;E:\b\build\third_party\cherrypy;E:\b\build\third_party\buildbot_8_4p1;E:\b\build\third_party\buildbot_7_12;E:\b\build\third_party\.svn;E:\b\build\site_config;E:\b\build\scripts;E:\b\build\scripts\release;E:\b\build\third_party;E:\b\build\third_party\requests_1_2_3;E:\b\build_internal\site_config;E:\b\build_internal\symsrc;E:\b\build\slave;E:\b\build\third_party\buildbot_slave_8_4;E:\b\build\third_party\twisted_10_2;
> File "E:\b\depot_tools\python276_bin\python.exe", line 1
> SyntaxError: Non-ASCII character '\x90' in file E:\b\depot_tools\python276_bin\python.exe on line 1, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
> Command ['E:\\b\\depot_tools\\python276_bin\\python.exe', u'E:\\b\\build\\scripts\\tools\\runit.py', '--show-path', 'E:\\b\\depot_tools\\python276_bin\\python.exe', u'E:\\b\\build\\scripts\\slave\\runtest.py', '--target', 'Release', '--xvfb', '--builder-name', u'Win7 Tests (1)', '--slave-name', u'vm801-m1', '--build-number', '36194', '--build-properties', '{"buildnumber": 36194, "slavename": "vm801-m1", "buildername": "Win7 Tests (1)"}', '--annotate', 'gtest', '--test-type', 'telemetry_unittests', '--run-python-script', 'E:\\b\\build\\slave\\Win7_Tests__1_\\build\\src\\tools\\telemetry\\run_tests', '--browser', 'release', '--retry-limit', '3', '--write-full-results-to', 'c:\\users\\chrome~2\\appdata\\local\\temp\\tmpo3eir6'] returned exit code 1
> Traceback (most recent call last):
> File "E:\b\build\slave\Win7_Tests__1_\build\src\testing\scripts\telemetry_unittests.py", line 54, in <module>
> sys.exit(common.run_script(sys.argv[1:], funcs))
> File "E:\b\build\slave\Win7_Tests__1_\build\src\testing\scripts\common.py", line 49, in run_script
> return args.func(args)
> File "E:\b\build\slave\Win7_Tests__1_\build\src\testing\scripts\telemetry_unittests.py", line 31, in main_run
> results = json.load(f)
> File "E:\b\depot_tools\python276_bin\lib\json\__init__.py", line 290, in load
>
> @@@STEP_CURSOR telemetry_unittests@@@
>
> Original issue's description:
> > [Telemtry] Move url lists of alexa1-10000 page to a json file.
> >
> > Existing implementation of alexa1-10000 embeds the huge url list in
> > the python file, which is a terrible. It makes it super hard to navigate
> > through the file, or review its content, and make it harder to update the
> > alexa1-10000 pages if needed.
> >
> > Committed: https://crrev.com/fec4b0793f9d7eb6bef66f83687f10e6391dff62
> > Cr-Commit-Position: refs/heads/master@{#316012}
>
> TBR=sullivan@chromium.org,dtu@chromium.org,nednguyen@google.com
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/c0cee90848513819e5a80c38f0337e56f611e02f
> Cr-Commit-Position: refs/heads/master@{#316049}
TBR=sullivan@chromium.org,dtu@chromium.org,nednguyen@google.com
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/924513004
Cr-Commit-Position: refs/heads/master@{#316055}
Reason for revert:
Build broke on many bots:
Example:
https://build.chromium.org/p/chromium.win/builders/Win7%20Tests%20%281%29/builds/36194
Running ['E:\\b\\depot_tools\\python276_bin\\python.exe', u'E:\\b\\build\\scripts\\tools\\runit.py', '--show-path', 'E:\\b\\depot_tools\\python276_bin\\python.exe', u'E:\\b\\build\\scripts\\slave\\runtest.py', '--target', 'Release', '--xvfb', '--builder-name', u'Win7 Tests (1)', '--slave-name', u'vm801-m1', '--build-number', '36194', '--build-properties', '{"buildnumber": 36194, "slavename": "vm801-m1", "buildername": "Win7 Tests (1)"}', '--annotate', 'gtest', '--test-type', 'telemetry_unittests', '--run-python-script', 'E:\\b\\build\\slave\\Win7_Tests__1_\\build\\src\\tools\\telemetry\\run_tests', '--browser', 'release', '--retry-limit', '3', '--write-full-results-to', 'c:\\users\\chrome~2\\appdata\\local\\temp\\tmpo3eir6']
Set PYTHONPATH: E:\b\build\..\build_internal\site_config;E:\b\build\third_party\httplib2\python2;E:\b\build\third_party\zope;E:\b\build\third_party\uritemplate;E:\b\build\third_party\twisted_8_1;E:\b\build\third_party\tempita_0_5;E:\b\build\third_party\swarm_client;E:\b\build\third_party\sqlalchemy_migrate_0_7_1;E:\b\build\third_party\sqlalchemy_0_7_1;E:\b\build\third_party\simplejson;E:\b\build\third_party\setuptools-0.6c11;E:\b\build\third_party\pytz;E:\b\build\third_party\psutils;E:\b\build\third_party\oauth2client;E:\b\build\third_party\mock-1.0.1;E:\b\build\third_party\mock-0.7.2;E:\b\build\third_party\mock-0.6.0;E:\b\build\third_party\markupsafe;E:\b\build\third_party\lighttpd;E:\b\build\third_party\jinja2;E:\b\build\third_party\infra;E:\b\build\third_party\httplib2;E:\b\build\third_party\gsutil;E:\b\build\third_party\google_api_python_client;E:\b\build\third_party\depot_tools_patch;E:\b\build\third_party\depot_tools;E:\b\build\third_party\decorator_3_3_1;E:\b\build\third_party\Crypto;E:\b\build\third_party\coverage-3.7.1;E:\b\build\third_party\cherrypy;E:\b\build\third_party\buildbot_8_4p1;E:\b\build\third_party\buildbot_7_12;E:\b\build\third_party\.svn;E:\b\build\site_config;E:\b\build\scripts;E:\b\build\scripts\release;E:\b\build\third_party;E:\b\build\third_party\requests_1_2_3;E:\b\build_internal\site_config;E:\b\build_internal\symsrc;E:\b\build\slave;E:\b\build\third_party\buildbot_slave_8_4;E:\b\build\third_party\twisted_10_2;
File "E:\b\depot_tools\python276_bin\python.exe", line 1
SyntaxError: Non-ASCII character '\x90' in file E:\b\depot_tools\python276_bin\python.exe on line 1, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
Command ['E:\\b\\depot_tools\\python276_bin\\python.exe', u'E:\\b\\build\\scripts\\tools\\runit.py', '--show-path', 'E:\\b\\depot_tools\\python276_bin\\python.exe', u'E:\\b\\build\\scripts\\slave\\runtest.py', '--target', 'Release', '--xvfb', '--builder-name', u'Win7 Tests (1)', '--slave-name', u'vm801-m1', '--build-number', '36194', '--build-properties', '{"buildnumber": 36194, "slavename": "vm801-m1", "buildername": "Win7 Tests (1)"}', '--annotate', 'gtest', '--test-type', 'telemetry_unittests', '--run-python-script', 'E:\\b\\build\\slave\\Win7_Tests__1_\\build\\src\\tools\\telemetry\\run_tests', '--browser', 'release', '--retry-limit', '3', '--write-full-results-to', 'c:\\users\\chrome~2\\appdata\\local\\temp\\tmpo3eir6'] returned exit code 1
Traceback (most recent call last):
File "E:\b\build\slave\Win7_Tests__1_\build\src\testing\scripts\telemetry_unittests.py", line 54, in <module>
sys.exit(common.run_script(sys.argv[1:], funcs))
File "E:\b\build\slave\Win7_Tests__1_\build\src\testing\scripts\common.py", line 49, in run_script
return args.func(args)
File "E:\b\build\slave\Win7_Tests__1_\build\src\testing\scripts\telemetry_unittests.py", line 31, in main_run
results = json.load(f)
File "E:\b\depot_tools\python276_bin\lib\json\__init__.py", line 290, in load
@@@STEP_CURSOR telemetry_unittests@@@
Original issue's description:
> [Telemtry] Move url lists of alexa1-10000 page to a json file.
>
> Existing implementation of alexa1-10000 embeds the huge url list in
> the python file, which is a terrible. It makes it super hard to navigate
> through the file, or review its content, and make it harder to update the
> alexa1-10000 pages if needed.
>
> Committed: https://crrev.com/fec4b0793f9d7eb6bef66f83687f10e6391dff62
> Cr-Commit-Position: refs/heads/master@{#316012}
TBR=sullivan@chromium.org,dtu@chromium.org,nednguyen@google.com
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
Review URL: https://codereview.chromium.org/920953003
Cr-Commit-Position: refs/heads/master@{#316049}
Existing implementation of alexa1-10000 embeds the huge url list in
the python file, which is a terrible. It makes it super hard to navigate
through the file, or review its content, and make it harder to update the
alexa1-10000 pages if needed.
Review URL: https://codereview.chromium.org/915303002
Cr-Commit-Position: refs/heads/master@{#316012}
(pyOpenSSL 0.13 or greater).
This relands https://crrev.com/855603002/,
"Roll Web Page Replay to pick-up HTTPS certificate fixes," and
picks-up one additional Web Page Replay change to test for SNI support.
BUG=431686
Review URL: https://codereview.chromium.org/903153002
Cr-Commit-Position: refs/heads/master@{#315333}
still supporting UserStories using local data.
Just log warnings when trying to download archives without a specified bucket
or where the user doesn't have permission to access the archive's bucket but
has a local copy of the archive. Raises cloud storage errors when there is no
local copy of the archive and the user doesn't have permission to access the
archive's bucket.
Also removes the ignore_archive parameter since it is only used in the smoke
test and doesn't provide any information that can't be gleaned from the
UserStory not having a bucket.
BUG=
Review URL: https://codereview.chromium.org/794493004
Cr-Commit-Position: refs/heads/master@{#310542}
Prior to this commit, you could not specify a path to an Android APK to run a
Telemetry test against. Desktop was the only platform that supported it.
This commit provides support support for running reference builds for the
Android platform in buildbot.
To run a Telemetry test against an exact APK, you can run the following
in tools/perf:
./run_benchmark --browser=exact --browser-executable=/path/to/my.apk benchmark
This also adds myself (mstrum@amazon.com) to AUTHORS.
R=tonyg@chromium.org
BUG=435179
Review URL: https://codereview.chromium.org/745733002
Cr-Commit-Position: refs/heads/master@{#310346}
We "white list" some exceptions which user story runner can continue running
the rest of user stories even if they are raised during the test run.
Those exceptions are:
page_test.Failure
util.TimeoutException
exceptions.LoginException
exceptions.ProfilingException
page_action.PageActionNotSupported
exceptions.AppCrashException (except if this is thrown inside
shared_user_story_state.state.TearDownState(), which indicates a failure of
recovering the test state)
BUG=437735
Review URL: https://codereview.chromium.org/755323012
Cr-Commit-Position: refs/heads/master@{#308258}
XP was the only platform that hung last time we tried this and the other
platforms sped up considerably.
BUG=417812
Review URL: https://codereview.chromium.org/752393005
Cr-Commit-Position: refs/heads/master@{#307139}
