b. Tools and Procedures
1. Slow query log
2. General log
3. Error log
In deciding the best approach to scaling your system, one of the most important concepts is that of bottlenecks. Bottlenecks are the limitations of your current system that have been reached first and prevent the maximization of other resources as the load on your system increases. So, obviously, the identification of bottlenecks is critical to knowing what improvements, if any, will allow your system to perform faster and handle more data. In this section we will discuss how to analyse your system to identify these bottlenecks and the types of solutions that certain bottlenecks suggest to us.
MySQL has a few different logs that can be utilized to identify bottlenecks and monitor server status. Each of these is enabled and managed through the MySQL configuration file, my.cnf. This file in linux will most likely be located in /etc/mysql or just /etc. On windows, look for my.ini rather in the windows directory, c:, or the MySQL instalation directory. MySQL actually looks for the file in this order so make sure you modify the one that it finds first. The first type of log we will discuss is the slow query log. To enable this log in your my.cnf file add this in the [mysqld] section.log_slow_queries = 1;
slow_query_log_file = <path to slow log>/mysql-slow.log
Additionally you can specify long_query_time=<some number of seconds> to tell it how long a query has to run for you to consider it slow. Other options are available in the mysql documentation. One of the most helpful is log-queries-not-using-indexes. Monitor this log over a period of time to identify indexes that need to be added and queries being submitted that need to be optimized. MySQL is fast so long as the amount of data being operated upon at any given time stays within memory. Once swapping to disk begins, it’s performance declines dramatically. The size of your available ram in relation to the amount of data being processed is a common source of bottlenecks. Using this log to identify the queries that are taking a lot of time will give you hints as to whether this type of paging to disk is going on. Increasing ram and adjusting queries to minimize paging will allow you to process much larger tables in a reasonable amount of time.
The general log is enabled by adding general_log_file=<path to general log/mysql.log or whatever file name you wish. This contains information about all queries that have been run including selects and who, that is, which database user, attempted to execute them. Whatever directory you chose for your logs, it is usually a good practice to put them all in the same place make sure that mysql has the correct permissions necessary to write to that directory. Otherwise, it is not critical to name or place these files in any particular manner. Just choose a strategy that works for you.
Finally, you can log errors with the error log by adding log-error=<path to error log>/mysql-error.log This, as can be expected, logs errors and if configured to, warnings generated by the database. Otherwise, errors will be logged to /var/syslog in linux. The error log helps in diagnosing startup and shutdown issues as well as failures and warnings that impact performance and stability. When dealing with large tables, disk space limitations and slow queries due to memory issues, failures resulting in logged errors will be common until your system is properly tuned.
Use these logs to identify the source of trouble and once issues have been resolved and everything is running smoothly, you can turn them off if you like. After all, logging is an expensive operation too. With MySQL, disk space issues related to operational and informational logs are more common than those due to actual data storage. Only enable logging that you need. Due to the wide range of errors and warnings that can be logged in these files, I will not go into detail regarding the interpretation of any particular one. When looking for the cause of an error or performance problem, check the logs and search on google for more information. MySQL is open source and the internet will have answers to most any problem you encounter.