JSPWiki JUnit test performance management#
Table of Contents
- JSPWiki JUnit test performance management
- DDL testcaseresult table
- Sample content testcaseresult
- DDL testsuiteresult table
- Sample content testsuiteresult
- Some useful queries
- Overall statistics
- All testsuites per tag having errors or failures
- 10 most time taking testsuites
- Classes with most testcases
- 15 most time taking testcase methods
- Testsuites that have increased their test time the most between releases
The idea is to use the information created by the JUnit tests.
Every time you run an " ant tests", a reports directory is created containing a test report for each test.
each JUnit test produces an XML format report
containing response times, errors and failures. (See the "Test Report" tab)
I wrote some code that gathers this information and stores it in an SQL database.
It takes two parameters :
- the name of the directory (i.e. "/home/metskem/workspace/JSPWiki/build/tests/reports")
- a tag that identifies this test run (i.e. "3.0.0-svn-17")
If we gather this information for each JSPWiki level that has significant changes, we could analyze if, and where performance differences occur, and also keep track of a lot of other things.
DDL testcaseresult table#
DROP TABLE IF EXISTS `testcaseresult`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `testcaseresult` ( `tag` varchar(32) NOT NULL, `hostname` varchar(32) NOT NULL, `timestamp` timestamp NOT NULL, `classname` varchar(128) NOT NULL, `name` varchar(128) NOT NULL, `time` double(8,3) NOT NULL default '0.000', PRIMARY KEY (`hostname`,`classname`,`name`,`timestamp`) ) TYPE=InnoDB DEFAULT CHARSET=utf8; SET character_set_client = @saved_cs_client;
Sample content testcaseresult#
mysql> select * from testcaseresult limit 10; +--------------+----------+---------------------+---------------------------------------------+-------------------------------------+-------+ | tag | hostname | timestamp | classname | name | time | +--------------+----------+---------------------+---------------------------------------------+-------------------------------------+-------+ | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testDeleteGroup | 1.990 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testSaveExistingGroup | 1.968 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testSaveNewGroup | 2.011 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testView | 2.398 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testViewNonExistent | 1.972 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | testViewNullGroup | 1.967 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | testRename | 1.975 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | testRenameReferences | 1.972 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | testRenameReferencesChangeRefsFalse | 1.969 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | testRenameReferencesChangeRefsTrue | 1.975 | +--------------+----------+---------------------+---------------------------------------------+-------------------------------------+-------+ 10 rows in set (0.01 sec)
DDL testsuiteresult table#
DROP TABLE IF EXISTS `testsuiteresult`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `testsuiteresult` ( `tag` varchar(32) NOT NULL, `hostname` varchar(32) NOT NULL, `timestamp` timestamp NOT NULL, `name` varchar(128) NOT NULL, `errors` int(11) NOT NULL default '0', `tests` int(11) NOT NULL default '0', `failures` int(11) NOT NULL default '0', `time` double(8,3) NOT NULL default '0.000', PRIMARY KEY (`hostname`,`name`,`timestamp`) ) TYPE=InnoDB DEFAULT CHARSET=utf8; SET character_set_client = @saved_cs_client;
Sample content testsuiteresult#
mysql> select * from testsuiteresult; +--------------+----------+---------------------+--------------------------------------------------------------+--------+-------+----------+--------+ | tag | hostname | timestamp | name | errors | tests | failures | time | +--------------+----------+---------------------+--------------------------------------------------------------+--------+-------+----------+--------+ | 3.0.0-svn-71 | bismarck | 2009-02-19 19:15:55 | org.apache.wiki.action.GroupActionBeanTest | 0 | 6 | 0 | 12.310 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:18 | org.apache.wiki.action.RenameActionBeanTest | 0 | 6 | 0 | 11.867 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:16:54 | org.apache.wiki.action.WikiContextFactoryTest | 0 | 5 | 0 | 6.423 | | 3.0.0-svn-71 | bismarck | 2009-02-19 19:17:21 | org.apache.wiki.auth.acl.DefaultAclManagerTest | 0 | 3 | 0 | 1.245 |
Some useful queries#
Here are a couple of examples of queries we could run.
Overall statistics#
mysql> select tag,sum(tests),avg(time),sum(time) from testsuiteresult group by tag order by tag; +--------------+------------+-----------+-----------+ | tag | sum(tests) | avg(time) | sum(time) | +--------------+------------+-----------+-----------+ | 2.8.2-svn-13 | 948 | 1.9197143 | 174.694 | | 3.0.0-svn-17 | 944 | 5.5338652 | 492.514 | | 3.0.0-svn-71 | 1050 | 4.4532762 | 467.594 | +--------------+------------+-----------+-----------+ 3 rows in set (0.02 sec)
All testsuites per tag having errors or failures#
mysql> select tag,name,tests,errors,failures from testsuiteresult where errors>0 or failures > 0 group by tag,name order by name,tag; +--------------+------------------------------------------------+-------+--------+----------+ | tag | name | tests | errors | failures | +--------------+------------------------------------------------+-------+--------+----------+ | 3.0.0-svn-71 | org.apache.wiki.content.ContentManagerTest | 4 | 4 | 0 | | 2.8.2-svn-13 | org.apache.wiki.dav.AttachmentDavProviderTest | 3 | 0 | 1 | | 3.0.0-svn-71 | org.apache.wiki.ui.migrator.BundleMigratorTest | 8 | 1 | 0 | | 2.8.2-svn-13 | org.apache.wiki.WikiEngineTest | 44 | 0 | 1 | | 3.0.0-svn-17 | org.apache.wiki.WikiEngineTest | 44 | 0 | 1 | | 3.0.0-svn-71 | org.apache.wiki.WikiEngineTest | 44 | 0 | 1 | +--------------+------------------------------------------------+-------+--------+----------+ 6 rows in set (0.01 sec)
10 most time taking testsuites#
mysql> select tag,name,tests,time from testsuiteresult order by time desc,name,tag limit 10; +--------------+------------------------------------------------+-------+--------+ | tag | name | tests | time | +--------------+------------------------------------------------+-------+--------+ | 3.0.0-svn-17 | stress.MassiveRepositoryTest | 1 | 84.526 | | 3.0.0-svn-71 | stress.MassiveRepositoryTest | 1 | 78.442 | | 3.0.0-svn-17 | org.apache.wiki.parser.JSPWikiMarkupParserTest | 205 | 70.827 | | 2.8.2-svn-13 | stress.MassiveRepositoryTest | 1 | 69.571 | | 3.0.0-svn-71 | org.apache.wiki.parser.JSPWikiMarkupParserTest | 205 | 66.344 | | 3.0.0-svn-17 | org.apache.wiki.search.SearchManagerTest | 6 | 43.761 | | 3.0.0-svn-71 | org.apache.wiki.search.SearchManagerTest | 6 | 42.688 | | 2.8.2-svn-13 | org.apache.wiki.search.SearchManagerTest | 6 | 40.240 | | 3.0.0-svn-17 | org.apache.wiki.WikiEngineTest | 44 | 37.644 | | 3.0.0-svn-71 | org.apache.wiki.WikiEngineTest | 44 | 35.495 | +--------------+------------------------------------------------+-------+--------+ 10 rows in set (0.01 sec)
Classes with most testcases#
mysql> select distinct name,tests from testsuiteresult order by tests desc limit 15; +-----------------------------------------------------------+-------+ | name | tests | +-----------------------------------------------------------+-------+ | org.apache.wiki.parser.JSPWikiMarkupParserTest | 205 | | org.apache.wiki.parser.CreoleToJSPWikiTranslatorTest | 83 | | org.apache.wiki.WikiEngineTest | 44 | | org.apache.wiki.TextUtilTest | 35 | | org.apache.wiki.TextUtilTest | 34 | | org.apache.wiki.workflow.WorkflowTest | 24 | | org.apache.wiki.content.PageRenamerTest | 21 | | org.apache.wiki.ReferenceManagerTest | 18 | | org.apache.wiki.auth.AuthorizationManagerTest | 17 | | org.apache.wiki.htmltowiki.HtmlStringToWikiTranslatorTest | 17 | | org.apache.wiki.ReleaseTest | 17 | | org.apache.wiki.plugin.PluginManagerTest | 16 | | org.apache.wiki.VariableManagerTest | 16 | | org.apache.wiki.workflow.SimpleDecisionTest | 16 | | org.apache.wiki.dav.DavPathTest | 15 | +-----------------------------------------------------------+-------+ 15 rows in set (0.00 sec)
15 most time taking testcase methods#
mysql> select tag,classname,name,time from testcaseresult order by time desc,name,tag limit 15; +--------------+-----------------------------------------------+------------------------------+--------+ | tag | classname | name | time | +--------------+-----------------------------------------------+------------------------------+--------+ | 3.0.0-svn-17 | stress.MassiveRepositoryTest | testMassiveRepository1 | 84.308 | | 3.0.0-svn-71 | stress.MassiveRepositoryTest | testMassiveRepository1 | 78.440 | | 2.8.2-svn-13 | stress.MassiveRepositoryTest | testMassiveRepository1 | 69.569 | | 3.0.0-svn-17 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch | 10.530 | | 3.0.0-svn-17 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch2 | 10.474 | | 3.0.0-svn-17 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch3 | 10.396 | | 3.0.0-svn-71 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch | 10.360 | | 3.0.0-svn-71 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch2 | 10.322 | | 3.0.0-svn-71 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch3 | 10.316 | | 2.8.2-svn-13 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch | 10.070 | | 2.8.2-svn-13 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch2 | 10.029 | | 2.8.2-svn-13 | org.apache.wiki.search.SearchManagerTest | testSimpleSearch3 | 10.029 | | 3.0.0-svn-71 | org.apache.wiki.providers.RCSFileProviderTest | testMillionChanges | 7.179 | | 3.0.0-svn-17 | org.apache.wiki.WikiEngineTest | testExternalModificationRefs | 6.418 | | 3.0.0-svn-17 | org.apache.wiki.WikiEngineTest | testExternalModification | 6.350 | +--------------+-----------------------------------------------+------------------------------+--------+ 15 rows in set (0.11 sec)
Testsuites that have increased their test time the most between releases#
mysql> select name,(max(time)-min(time))/avg(time)*100 as percdelta,min(time),max(time) from testsuiteresult where tag='3.0.0-svn-17' or tag='2.8.2-svn-13' group by name order by 2 desc limit 25; +-----------------------------------------------------------+-----------------+-----------+-----------+ | name | percdelta | min(time) | max(time) | +-----------------------------------------------------------+-----------------+-----------+-----------+ | org.apache.wiki.util.ClassUtilTest | 197.54098360656 | 0.003 | 0.485 | | org.apache.wiki.util.SerializerTest | 196.02649006623 | 0.003 | 0.299 | | org.apache.wiki.workflow.FactTest | 194.95798319328 | 0.003 | 0.235 | | org.apache.wiki.parser.MarkupParserTest | 194.78260869565 | 0.003 | 0.227 | | org.apache.wiki.ui.RedirectCommandTest | 194.61883408072 | 0.003 | 0.220 | | org.apache.wiki.util.UtilJ2eeCompatTest | 194.36619718310 | 0.005 | 0.350 | | org.apache.wiki.util.PriorityListTest | 192.88888888889 | 0.004 | 0.221 | | org.apache.wiki.auth.acl.AclEntryImplTest | 192.72727272727 | 0.004 | 0.216 | | org.apache.wiki.workflow.OutcomeTest | 192.19143576826 | 0.031 | 1.557 | | org.apache.wiki.ui.GroupCommandTest | 191.42300194932 | 0.044 | 2.008 | | org.apache.wiki.auth.permissions.AllPermissionTest | 190.90909090909 | 0.005 | 0.215 | | org.apache.wiki.filters.FilterManagerTest | 190.89715536105 | 0.052 | 2.233 | | org.apache.wiki.plugin.GroupsTest | 190.70307960488 | 0.040 | 1.681 | | org.apache.wiki.util.MailUtilTest | 190.67110899572 | 0.049 | 2.052 | | org.apache.wiki.util.CryptoUtilTest | 190.62500000000 | 0.006 | 0.250 | | org.apache.wiki.ui.PageCommandTest | 190.48977650975 | 0.050 | 2.053 | | org.apache.wiki.auth.login.WebContainerLoginModuleTest | 190.36973344798 | 0.056 | 2.270 | | org.apache.wiki.auth.permissions.WikiPermissionTest | 189.47368421053 | 0.006 | 0.222 | | org.apache.wiki.dav.DavPathTest | 189.39393939394 | 0.007 | 0.257 | | org.apache.wiki.workflow.TaskTest | 189.18918918919 | 0.006 | 0.216 | | org.apache.wiki.ui.WikiCommandTest | 188.67187500000 | 0.058 | 1.990 | | org.apache.wiki.auth.login.CookieAssertionLoginModuleTest | 188.39694656489 | 0.057 | 1.908 | | org.apache.wiki.plugin.UndefinedPagesPluginTest | 188.21630347054 | 0.073 | 2.405 | | org.apache.wiki.render.WysiwygEditingRendererTest | 188.12260536398 | 0.062 | 2.026 | | org.apache.wiki.auth.login.UserDatabaseLoginModuleTest | 187.73869346734 | 0.061 | 1.929 | +-----------------------------------------------------------+-----------------+-----------+-----------+ 25 rows in set (0.02 sec)