JSPWiki JUnit test performance management#
Table of Contents
- JSPWiki JUnit test performance management
- DDL testcaseresult table
- Sample content testcaseresult
- DDL testsuiteresult table
- Sample content testsuiteresult
- Some useful queries
- Overall statistics
- All testsuites per tag having errors or failures
- 10 most time taking testsuites
- Classes with most testcases
- 15 most time taking testcase methods
- Testsuites that have increased their test time the most between releases
The idea is to use the information created by the JUnit tests.
Every time you run an " ant tests", a reports directory is created containing a test report for each test.
each JUnit test produces an XML format report
containing response times, errors and failures. (See the "Test Report" tab)
I wrote some code that gathers this information and stores it in an SQL database.
It takes two parameters :
- the name of the directory (i.e. "/home/metskem/workspace/JSPWiki/build/tests/reports")
- a tag that identifies this test run (i.e. "3.0.0-svn-17")
If we gather this information for each JSPWiki level that has significant changes, we could analyze if, and where performance differences occur, and also keep track of a lot of other things.
DDL testcaseresult table#
DROP TABLE IF EXISTS `testcaseresult`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `testcaseresult` ( `tag` varchar(32) NOT NULL, `hostname` varchar(32) NOT NULL, `timestamp` timestamp NOT NULL, `classname` varchar(128) NOT NULL, `name` varchar(128) NOT NULL, `time` double(8,3) NOT NULL default '0.000', PRIMARY KEY (`hostname`,`classname`,`name`,`timestamp`) ) TYPE=InnoDB DEFAULT CHARSET=utf8; SET character_set_client = @saved_cs_client;
Sample content testcaseresult#
mysql> select * from testcaseresult limit 10; +--------------+----------+---------------------+---------------------------------------------+-------------------------------------+-------+ | tag | hostname | timestamp | classname | name | time | +--------------+----------+---------------------+---------------------------------------------+-------------------------------------+-------+ | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testDeleteGroup | 1.990 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testSaveExistingGroup | 1.968 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testSaveNewGroup | 2.011 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testView | 2.398 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testViewNonExistent | 1.972 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testViewNullGroup | 1.967 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | testRename | 1.975 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | testRenameReferences | 1.972 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | testRenameReferencesChangeRefsFalse | 1.969 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | testRenameReferencesChangeRefsTrue | 1.975 | +--------------+----------+---------------------+---------------------------------------------+-------------------------------------+-------+ 10 rows in set (0.01 sec)
DDL testsuiteresult table#
DROP TABLE IF EXISTS `testsuiteresult`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `testsuiteresult` ( `tag` varchar(32) NOT NULL, `hostname` varchar(32) NOT NULL, `timestamp` timestamp NOT NULL, `name` varchar(128) NOT NULL, `errors` int(11) NOT NULL default '0', `tests` int(11) NOT NULL default '0', `failures` int(11) NOT NULL default '0', `time` double(8,3) NOT NULL default '0.000', PRIMARY KEY (`hostname`,`name`,`timestamp`) ) TYPE=InnoDB DEFAULT CHARSET=utf8; SET character_set_client = @saved_cs_client;
Sample content testsuiteresult#
mysql> select * from testsuiteresult; +--------------+----------+---------------------+--------------------------------------------------------------+--------+-------+----------+--------+ | tag | hostname | timestamp | name | errors | tests | failures | time | +--------------+----------+---------------------+--------------------------------------------------------------+--------+-------+----------+--------+ | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | 0 | 6 | 0 | 12.310 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | 0 | 6 | 0 | 11.867 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:54 | org.apache.wiki.action.WikiContextFactoryTest | 0 | 5 | 0 | 6.423 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:17:21 | org.apache.wiki.auth.acl.DefaultAclManagerTest | 0 | 3 | 0 | 1.245 |
Some useful queries#
Here are a couple of examples of queries we could run.
Overall statistics#
mysql> select tag,sum(tests),avg(time),sum(time) from testsuiteresult group by tag order by tag; +--------------+------------+-----------+-----------+ | tag | sum(tests) | avg(time) | sum(time) | +--------------+------------+-----------+-----------+ | 2.8.2-svn-13 | 948 | 1.9197143 | 174.694 | | 3.0.0-svn-17 | 944 | 5.5338652 | 492.514 | | 3.0.0-svn-71 | 1050 | 4.4532762 | 467.594 | | 3.0.0-svn-72 | 1055 | 4.5185849 | 478.970 | +--------------+------------+-----------+-----------+ 4 rows in set (0.13 sec)
All testsuites per tag having errors or failures#
mysql> select tag,name,tests,errors,failures from testsuiteresult where errors>0 or failures > 0 group by tag,name order by name,tag; +--------------+------------------------------------------------+-------+--------+----------+ | tag | name | tests | errors | failures | +--------------+------------------------------------------------+-------+--------+----------+ | 3.0.0-svn-71 | org.apache.wiki.content.ContentManagerTest | 4 | 4 | 0 | | 3.0.0-svn-72 | org.apache.wiki.content.ContentManagerTest | 4 | 4 | 0 | | 2.8.2-svn-13 | org.apache.wiki.dav.AttachmentDavProviderTest | 3 | 0 | 1 | | 3.0.0-svn-72 | org.apache.wiki.dav.AttachmentDavProviderTest | 3 | 0 | 1 | | 3.0.0-svn-71 | org.apache.wiki.ui.migrator.BundleMigratorTest | 8 | 1 | 0 | | 3.0.0-svn-72 | org.apache.wiki.ui.migrator.BundleMigratorTest | 8 | 1 | 0 | | 2.8.2-svn-13 | org.apache.wiki.WikiEngineTest | 44 | 0 | 1 | | 3.0.0-svn-17 | org.apache.wiki.WikiEngineTest | 44 | 0 | 1 | | 3.0.0-svn-71 | org.apache.wiki.WikiEngineTest | 44 | 0 | 1 | | 3.0.0-svn-72 | org.apache.wiki.WikiEngineTest | 44 | 0 | 1 | +--------------+------------------------------------------------+-------+--------+----------+ 10 rows in set (0.01 sec)
10 most time taking testsuites#
mysql> select tag,name,tests,time from testsuiteresult order by time desc,name,tag limit 10; +--------------+------------------------------------------------+-------+--------+ | tag | name | tests | time | +--------------+------------------------------------------------+-------+--------+ | 3.0.0-svn-17 | stress.MassiveRepositoryTest | 1 | 84.526 | | 3.0.0-svn-71 | stress.MassiveRepositoryTest | 1 | 78.442 | | 3.0.0-svn-72 | stress.MassiveRepositoryTest | 1 | 76.723 | | 3.0.0-svn-17 | org.apache.wiki.parser.JSPWikiMarkupParserTest | 205 | 70.827 | | 2.8.2-svn-13 | stress.MassiveRepositoryTest | 1 | 69.571 | | 3.0.0-svn-72 | org.apache.wiki.parser.JSPWikiMarkupParserTest | 205 | 67.173 | | 3.0.0-svn-71 | org.apache.wiki.parser.JSPWikiMarkupParserTest | 205 | 66.344 | | 3.0.0-svn-17 | org.apache.wiki.search.SearchManagerTest | 6 | 43.761 | | 3.0.0-svn-71 | org.apache.wiki.search.SearchManagerTest | 6 | 42.688 | | 3.0.0-svn-72 | org.apache.wiki.search.SearchManagerTest | 6 | 42.035 | +--------------+------------------------------------------------+-------+--------+ 10 rows in set (0.08 sec)
Classes with most testcases#
mysql> select distinct name,tests from testsuiteresult order by tests desc limit 15; +-----------------------------------------------------------+-------+ | name | tests | +-----------------------------------------------------------+-------+ | org.apache.wiki.parser.JSPWikiMarkupParserTest | 205 | | org.apache.wiki.parser.CreoleToJSPWikiTranslatorTest | 83 | | org.apache.wiki.WikiEngineTest | 44 | | org.apache.wiki.TextUtilTest | 35 | | org.apache.wiki.TextUtilTest | 34 | | org.apache.wiki.workflow.WorkflowTest | 24 | | org.apache.wiki.content.PageRenamerTest | 21 | | org.apache.wiki.ReferenceManagerTest | 18 | | org.apache.wiki.auth.AuthorizationManagerTest | 17 | | org.apache.wiki.htmltowiki.HtmlStringToWikiTranslatorTest | 17 | | org.apache.wiki.ReleaseTest | 17 | | org.apache.wiki.plugin.PluginManagerTest | 16 | | org.apache.wiki.VariableManagerTest | 16 | | org.apache.wiki.workflow.SimpleDecisionTest | 16 | | org.apache.wiki.dav.DavPathTest | 15 | +-----------------------------------------------------------+-------+ 15 rows in set (0.03 sec)
15 most time taking testcase methods#
mysql> select tag,classname,name,time from testcaseresult order by time desc,name,tag limit 15; +--------------+------------------------------------------+------------------------+--------+ | tag | classname | name | time | +--------------+------------------------------------------+------------------------+--------+ | 3.0.0-svn-17 | stress.MassiveRepositoryTest | testMassiveRepository1 | 84.308 | | 3.0.0-svn-71 | stress.MassiveRepositoryTest | testMassiveRepository1 | 78.440 | | 3.0.0-svn-72 | stress.MassiveRepositoryTest | testMassiveRepository1 | 76.721 | | 2.8.2-svn-13 | stress.MassiveRepositoryTest | testMassiveRepository1 | 69.569 | | 3.0.0-svn-17 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch | 10.530 | | 3.0.0-svn-17 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch2 | 10.474 | | 3.0.0-svn-72 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch | 10.433 | | 3.0.0-svn-17 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch3 | 10.396 | | 3.0.0-svn-71 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch | 10.360 | | 3.0.0-svn-72 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch2 | 10.340 | | 3.0.0-svn-71 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch2 | 10.322 | | 3.0.0-svn-71 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch3 | 10.316 | | 3.0.0-svn-72 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch3 | 10.316 | | 2.8.2-svn-13 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch | 10.070 | | 2.8.2-svn-13 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch2 | 10.029 | +--------------+------------------------------------------+------------------------+--------+ 15 rows in set (0.24 sec)
Testsuites that have increased their test time the most between releases#
mysql> select name,(max(time)-min(time))/avg(time)*100 as percdelta,min(time),max(time),avg(time) from testsuiteresult where tag='3.0.0-svn-71' or tag='3.0.0-svn-72' group by name order by 2 desc limit 25; +-----------------------------------------------------------+----------------+-----------+-----------+-----------+ | name | percdelta | min(time) | max(time) | avg(time) | +-----------------------------------------------------------+----------------+-----------+-----------+-----------+ | org.apache.wiki.auth.permissions.PagePermissionTest | 76.92307692308 | 0.004 | 0.009 | 0.0065000 | | org.apache.wiki.dav.DavPathTest | 73.68421052632 | 0.006 | 0.013 | 0.0095000 | | org.apache.wiki.auth.permissions.WikiPermissionTest | 66.66666666667 | 0.004 | 0.008 | 0.0060000 | | org.apache.wiki.util.ClassUtilTest | 66.66666666667 | 0.003 | 0.006 | 0.0045000 | | org.apache.wiki.workflow.SimpleDecisionTest | 60.00000000000 | 0.007 | 0.013 | 0.0100000 | | org.apache.wiki.parser.CreoleToJSPWikiTranslatorTest | 49.81412639405 | 0.101 | 0.168 | 0.1345000 | | org.apache.wiki.PageManagerTest | 47.19887955182 | 4.364 | 7.060 | 5.7120000 | | org.apache.wiki.util.CommentedPropertiesTest | 46.40000000000 | 0.048 | 0.077 | 0.0625000 | | org.apache.wiki.ui.migrator.JspDocumentTest | 40.00000000000 | 0.002 | 0.003 | 0.0025000 | | org.apache.wiki.workflow.TaskTest | 40.00000000000 | 0.006 | 0.009 | 0.0075000 | | org.apache.wiki.ui.migrator.JspParserTest | 38.46153846154 | 0.042 | 0.062 | 0.0520000 | | org.apache.wiki.auth.permissions.GroupPermissionTest | 35.29411764706 | 0.007 | 0.010 | 0.0085000 | | org.apache.wiki.auth.authorize.WebContainerAuthorizerTest | 34.02489626556 | 1.400 | 1.974 | 1.6870000 | | org.apache.wiki.auth.authorize.GroupTest | 32.66381297333 | 2.541 | 3.533 | 3.0370000 | | org.apache.wiki.util.PriorityListTest | 28.57142857143 | 0.003 | 0.004 | 0.0035000 | | org.apache.wiki.parser.MarkupParserTest | 28.57142857143 | 0.003 | 0.004 | 0.0035000 | | org.apache.wiki.content.WikiNameTest | 28.57142857143 | 0.003 | 0.004 | 0.0035000 | | org.apache.wiki.PropertyReaderTest | 25.00000000000 | 0.014 | 0.018 | 0.0160000 | | org.apache.wiki.attachment.AttachmentManagerTest | 24.69245065935 | 4.952 | 6.347 | 5.6495000 | | org.apache.wiki.auth.acl.DefaultAclManagerTest | 24.02159244265 | 0.978 | 1.245 | 1.1115000 | | org.apache.wiki.FileUtilTest | 23.85964912281 | 0.251 | 0.319 | 0.2850000 | | org.apache.wiki.util.UtilJ2eeCompatTest | 22.22222222222 | 0.004 | 0.005 | 0.0045000 | | org.apache.wiki.auth.authorize.JDBCGroupDatabaseTest | 21.84782608696 | 1.639 | 2.041 | 1.8400000 | | org.apache.wiki.util.CryptoUtilTest | 18.18181818182 | 0.005 | 0.006 | 0.0055000 | | org.apache.wiki.ReleaseTest | 18.18181818182 | 0.015 | 0.018 | 0.0165000 | +-----------------------------------------------------------+----------------+-----------+-----------+-----------+ 25 rows in set (0.03 sec)