Name: tecken-loadtests
Owner: Mozilla Services
Description: Bombard Mozilla Symbol Server's Symbolication service
Created: 2017-04-03 19:21:35.0
Updated: 2017-05-24 18:47:03.0
Pushed: 2018-02-12 16:34:55.0
Homepage: null
Size: 21916
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Ability to bombard Mozilla Symbol Server's
Symbolication service with tonnes of stacks as stored in .json files
and the Download service with tonnes of symbol URL requests.
Download or clone this repo.
Run a Python that has
requests installed.
Type something like python download.py http://localhost:8000 symbol-queries-groups.csv
assuming you have the download server running at localhost:8000
Sit and watch it or kill it with Ctrl-C. If you kill it before it
finishes (finishing is likely to take hours) stats are printed out with
what's been accomplished so far.
Alternatively you can do the same but add another CSV file that contains
looks for code_file and code_id. For example:
on download.py http://localhost:8000 downloading/symbol-queries-groups.csv downloading/socorro-missing.csv
That second file is expected to have the following header:
g_file,debug_id,code_file,code_id
The results look like this after running for a while:
DONE SO FAR 302
FOR 173.957s
AGE RATE 1.74 requests/s
US CODE COUNT MEDIAN (INTERNAL) AVERAGE (INTERNAL) % RIGHT
274 0.644s 0.651s 0.564s 0.660s 95.62
28 0.657s 0.639s 0.693s 0.663s 100.00
That means that 302 URLs were sent in. In 95.62% of the cases, Tecken also found that the symbol file didn't exist (compared with what was the case when the CSV file was made). And there were 28 requests where the symbol existed and was able to redirect to an absolute S3 URL.
The (INTERNAL) is the median and average of the seconds it took the server,
internally, to make the lookup. So if a look up took 0.6 seconds and
0.5 seconds internally, it means there was an 0.1 second overhead of
making the request to Tecken. In that case, the 0.5 is basically purely
the time it takes Tecken to talk to S3. But mind you, Tecken can iterate
over a list of S3 sources. It's not possible to tell if it
Builds are made on TaskCluster, as an artifact it builds symbols zip files. To get a list of recent ones of these for local development or load testing run the script:
python list-firefox-symbols-zips.py
Each URL can be used to test symbol upload by URL.
Download or clone this repo.
Run a Python that has
requests installed.
Type something like python symbolicate.py stacks http://localhost:8000
assuming you have the symbolication server running at localhost:8000
Sit and watch it or kill it with Ctrl-C. If you kill it before it
finishes (finishing is likely to take hours) stats are printed out with
what's been accomplished so far.
When you finish (or kill it unfinished) it will print out something that looks like this:
L SO FAR 369 JOBS DONE
SUM AVG MEDIAN STD-DEV
e_lookups.count 1083 2 3 1.73
e_lookups.size 1.8GB 5.0MB 0.0B 8.7MB
e_lookups.time 8.505s 0.023s 0.001s 0.038s
loads.count 941 2 2 1.73
loads.size 23.9GB 66.3MB 76.5MB 40.4MB
loads.time 3179.728s 8.594s 10.257s 4.432s
er_time 3350.723s 9.056s 10.758s 4.484s
les.count 1083 2 3 1.73
ks.count 8497 22 25 5.00
ks.real 8334 22 25 5.39
3351.656s 9.059s 10.725s 4.443s
ONCLUSION...
l Download Speed 7.7MB/s
l Cache Speed 219.0MB/s
Here's the same output but annotated with comments:
L SO FAR 369 JOBS DONE
SUM AVG MEDIAN STD-DEV
w many times we've tried to look up a module in the LRU cache.
e_lookups.count 1083 2 3 1.73
w much data we have successfully extracted out of the LRU cache.
e_lookups.size 1.8GB 5.0MB 0.0B 8.7MB
e time spent doing lookups on the LRU cache (hits or misses).
e_lookups.time 8.505s 0.023s 0.001s 0.038s
w many distinct URLs that have had to be downloaded.
loads.count 941 2 2 1.73
e amount of data that has been downloaded from URLs (uncompressed).
loads.size 23.9GB 66.3MB 76.5MB 40.4MB
e time spent doing URL downloads.
loads.time 3179.728s 8.594s 10.257s 4.432s
special one. This wraps the 'downloads.time' plus the time it
kes to make the and getting the response. Should be marginally
gger than than 'downloads.time'
er_time 3350.723s 9.056s 10.758s 4.484s
stinct number of modules that have been come across. Note
at this number is the same as 'cache_lookups.count' above.
les.count 1083 2 3 1.73
tal number of individual stacks symbolicated.
ks.count 8497 22 25 5.00
me as 'stacks.count' except sometimes the module index is -1 so we
ow we don't have to symbolicate it and can just insert its hex offset
rectly.
ks.real 8334 22 25 5.39
tal time spent symbolicating all stacks. This spans cache misses and
che hits.
3351.656s 9.059s 10.725s 4.443s
ONCLUSION...
e download speed doing downloads. But note! this is UNcompressed so it's
kely to be much higher (how much? roughly the average gzip size of a
mbol text file) than what you get for your broadband when you open
tp://fast.com.
l Download Speed 7.7MB/s
e speed at which the web service can extract data out of the LRU cache.
is is a really important number if you want to optimize how the LRU
ta pipelining works.
l Cache Speed 219.0MB/s
This script picks sample JSON stacks to send in randomly. Every time. That means that if you start it, kill it and start again, it's unlikely that you'll be able to benefit much from the cache of the first run.
symbol-queries-groups.csv file was madeFirst of all, you need to enable logging on the
org.mozilla.crash-stats.symbols-public and
org.mozilla.crash-stats.symbols-private S3 buckets. Make the logging
go to the bucket peterbe-symbols-playground-deleteme-in-2018 and for
each make the prefix be public-symbols/ and private-symbols/
respectively.
The file symbol-queries-groups.csv was created by running
generate-csv-logs.py a bunch of ways:
AWS_ACCESS_KEY=... AWS_SECRET_ACCESS_KEY=... python generate-csv-logs.py download
python generate-csv-logs.py summorize
python generate-csv-logs.py group
To start a molotov testing there's a
loadtest.py script. Basic usage:
molotov --max-runs 10 -cx loadtest.py
By default the base URL for this will be http://localhost:8000. If you
want to override that, change the environment variable URL_SERVER.
For example:
URL_SERVER=https://symbols.dev.mozaws.net molotov --max-runs 10 -cx loadtest.py
To load test Tecken with realistic .zip uploads, you can simulate the
uploads sent to Socorro in the past.
The make-symbol-zip.py script will look at the logs, pick a
recent one (uploaded by Mozilla RelEng)
and then download each and every file from S3 and make a .zip file in
your temp directory (e.g. /tmp/massive-symbol-zips/symbols-2017-06-09T04_01_45.zip).
Simply run it like this::
python make-symbol-zip.py
In the stdout, it should say where it was saved.
Now you can use that to upload. For example:
curl -X POST -H "Auth-Token: YYYYYYY" --form myfile.zip=@/tmp/massive-symbol-zips/symbols-2017-06-09T04_01_45.zip http://localhost:8000/upload/
First you have to make a bunch of .zip files. See the section above on
“Make Symbol Zips”. That script uses the same default save directory
as upload-symbol-zips.py. This script picks random .zip files from
that directory where they're temporarily saved. This script will actually
go ahead and make the upload.
First try:
python upload-symbol-zips.py --help
By default, it will upload 1 random .zip file to
http://localhost:8000/upload. All the uploads are synchronous.
This does require an Auth-Token (aka. “API token”) in the environment
called AUTH_TOKEN. Either export it or use like this:
AUTH_TOKEN=7e353c4f34644ef6ba1cfb02b3c3662d python upload-symbol-zips.py
If you do the testing using localhost:8000 but actually depend on uploading
the to an S3 bucket that is on the Internet, the uploads can become really
slow. Especially on a home broad band. To limit it to .zip files that
aren't too large you can add --max-size option. E.g.
python upload-symbol-zips.py --max-size 100m
That will pick (randomly) only from .zip files that are 100Mb or less.
symbols-uploaded/YYYY-MM-DD.json.gzGet an API token from Crash-stats
with the View all Symbol Uploads permission. Then run:
AUTH_TOKEN=bdf6effac894491a8ebd0d1b15f3ab5a python generate-symbols-uploaded.py
There's a script called analyze-symbol-uploads-times.py which gives insight
into symbol upload times. Use it to analyze how concurrent uploads
work/optimize. You need an auth token with the “View All Symbols Uploads”
permission. Then run:
AUTH_TOKEN=66...92e python analyze-symbol-uploads-times.py --domain=symbols.stage.mozaws.net --limit=10
If you run python list-firefox-symbols-zips.py 3 it will find 3 recent
symbols builds URLs on TaskCluster. You can actually pipe them into the
the upload-symbol-zips.py script. For example, this is how you do it
for stage:
export AUTH_TOKEN=xxxxxxxStageAPITokenxxxxxxxxx
python list-firefox-symbols-zips.py 1 | python upload-symbol-zips.py https://symbols.stage.mozaws.net --download-urls-from-stdin --max-size=2gb