NaturalHistoryMuseum/mysql-slow

Name: mysql-slow

Owner: Natural History Museum

Description: A pipeline for processing records in mysql's slow query log files, generated by Scratchpads servers

Created: 2017-04-18 21:38:34.0

Updated: 2017-04-19 08:09:24.0

Pushed: 2017-04-19 08:23:14.0

Homepage:

Size: 11

Language: R

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

mysql-slow

A pipeline for processing records in mysql's slow query log files, generated by Scratchpads servers sp-data-1 and sp-data-2.

Setup

You will need Python 3.6.1 or later and R 3.3.3 or later.

rtualenv mysql-slow
install -r requirements.pip
ipt --slave --no-save --no-restore-history -e "install.packages('xts')"

Set the locations of the python3 and Rscript binaries in stages.py.

Run the analysis

1. Compress mysql-slow logs on sp-data-1 and sp-data-2.

   ssh to each server and run

    cp /var/log/mysql/mysql-slow.log . && \
   sudo chown `id --user --name`:`id --group --name` mysql-slow.log && \
   mv mysql-slow.log mysql-slow-`hostname`.log && \
   gzip mysql-slow-`hostname`.log
   

2. scp compressed logs from to 0_data.

   On your workstation

   sp-data-1:/home/lawh/mysql-slow-sp-data-1.log.gz 0_data
   sp-data-2:/home/lawh/mysql-slow-sp-data-2.log.gz 0_data
   

3. Remove the compressed files from both sp-data-1 and sp-data-2

   ssh to each server and run

   ysql-slow-`hostname`.log
   

4. Run the pipeline and view the results

   On your workstation

   on mysql-slow
   rset.py -R && somerset.py all
    4_visualise/*png