\subsection{Empirical analysis of patterns} 

As a by-product of Aibolit's ML and recommendation
engine, we get a tool for empirically analysing different patterns' impacts on the target quality metric. Just
like in the main recommendation algorithm (Algorith~\ref{fig:recsys_alg}, Section~\ref{sec:recommendation_algorithm}), we can estimate whether a particular pattern has a positive or negative impact on the quality metric by considering modifications of source code where pattern count is decreased or increased. We perform such a procedure on a heldout set, which allows to estimate the average impact of a particular pattern on quality. 

In Table~\ref{tab:pattern_analysis} we present a case study of the 34 patterns used at training of Aibolit (see Appendix for the pattern descriptions). On a separate test set (see details in Section~\ref{sec:dataset}), for each pattern, we considered increasing and decreasing the pattern's count by 1. We used the pretrained Aibolit's regression model to predict the corresponding change in quality metric.


\begin{table}[ht]
\footnotesize
\begin{tabular}{lllllll}
patterns                                 &   p- m-   &   p+ m+   &   p- m+   &   p+ m-   &   p- m=   &   p+ m=   \\ 
\\ \hline
Asserts                                	&	92	&	61	&	186	&	219	&	5	&	3	\\
Setters                                		&	245	&	160	&	901	&	976	&	21	&	31	\\
Empty Rethrow                          		&	77	&	65	&	25	&	36	&	0	&	1	\\
Prohibited class name                  		&	617	&	311	&	218	&	522	&	0	&	2	\\
Force Type Casting                     		&	2363	&	2313	&	1742	&	1790	&	14	&	16	\\
Count If Return                        		&	969	&	883	&	214	&	298	&	0	&	2	\\
Implements Multi                       		&	459	&	320	&	264	&	403	&	0	&	0	\\
Instance of                            		&	1396	&	1374	&	151	&	173	&	6	&	6	\\
Many primary constructors              		&	19	&	19	&	604	&	605	&	2	&	1	\\
Method chain                           		&	573	&	574	&	2217	&	2214	&	43	&	45	\\
Multiple try                           		&	371	&	239	&	259	&	391	&	0	&	0	\\
Non final attribute                    		&	1249	&	1185	&	5835	&	5839	&	127	&	187	\\
Null check                             		&	5863	&	5978	&	575	&	457	&	9	&	12	\\
Partial synchronized                   		&	46	&	49	&	155	&	149	&	0	&	3	\\
Redundant catch                        		&	84	&	46	&	38	&	79	&	4	&	1	\\
Return null                            		&	1290	&	813	&	926	&	1401	&	4	&	6	\\
String concat                          		&	1596	&	2089	&	1506	&	1012	&	23	&	24	\\
Super Method                           		&	212	&	275	&	1012	&	949	&	10	&	10	\\
This in constructor                    		&	15	&	42	&	72	&	45	&	0	&	0	\\
Var declaration distance for 5 lines   		&	1976	&	1464	&	738	&	1244	&	19	&	25	\\
Var declaration distance for 7 lines   		&	1190	&	959	&	733	&	949	&	16	&	31	\\
Var declaration distance for 11 lines  		&	703	&	603	&	365	&	466	&	10	&	9	\\
Var in the middle                      		&	3372	&	3117	&	2799	&	3046	&	37	&	45	\\
Array as function argument             		&	882	&	768	&	351	&	464	&	1	&	2	\\
Joined validation                      		&	232	&	226	&	14	&	26	&	8	&	2	\\
Non final class                        		&	7141	&	3201	&	3723	&	7663	&	0	&	0	\\
Private static method                  		&	529	&	541	&	667	&	648	&	2	&	9	\\
Public static method                   		&	1166	&	1212	&	1142	&	1088	&	4	&	12	\\
Null Assignment                        		&	1245	&	803	&	717	&	1150	&	6	&	15	\\
Multiple While                         		&	120	&	97	&	16	&	39	&	0	&	0	\\
Protected Method                       		&	868	&	402	&	1147	&	1613	&	7	&	7	\\
Send Null                              		&	556	&	325	&	1510	&	1733	&	5	&	13	\\
Nested Loop                            		&	580	&	527	&	29	&	81	&	1	&	2	\\
\hline
\end{tabular}
\centering
\caption{Empirical analysis of patterns. For each source code in the test set, we consider encreasing (\textbf{p+}) and decreasing (\textbf{p-}) pattern count by 1. And we recorded whether the maintainability metric increased (\textbf{m+}) or decreased (\textbf{m-}) as a result of that (where lower is better). In some cases the metric did no change (\textbf{m=}). The values in the cells are counts of cases.} 
\label{tab:pattern_analysis}
\captionsetup{font=scriptsize}
% \caption*{
%     We use the following notation into named columns:
%     ($p-$): decrease pattern by $\frac{1}{ncss}$; 
%     ($p+$): increase pattern by $\frac{1}{ncss}$;
%     ($c-$): complexity has been decreased; $c+$: complexity has been increased;
%     ($c=$): complexity has been not changed; 
%     (\emph{-1(top1)}): decreasing of pattern shows best \emph{CogC} improvement;
%     (\emph{+1(top1)}): increasing of pattern shows best \emph{CogC} improvement.
% }
% \caption*{
%     We use the following notation into named columns: \\
%     \\
%     \centering
%     \begin{tabular}{rl}
%         $p-$ & decrease pattern by $\frac{1}{ncss}$ \\
%         $p+$ & increase pattern by $\frac{1}{ncss}$ \\
%         $c-$ & complexity has been decreased \\
%         $c+$ & complexity has been increased \\
%         $c=$ & complexity has been not changed \\
%         \emph{-1(top1)} & decreasing of pattern shows best \emph{CogC} improvement   \\
%         \emph{+1(top1)} & increasing of pattern shows best \emph{CogC} improvement \\
%     \end{tabular}
% }
\end{table}

Based on the statistics in Table~\ref{tab:pattern_analysis}, it appears that \emph{Prohibited class name}, \emph{Count If Return}, \emph{Instance of}, \emph{Null check}, \emph{Nested Loop}, 
\emph{Array as function argument}, \emph{Joined validation} are \textbf{anti-patterns}, since their count decrease tends to improve  the metric. Another group of patterns are \emph{Setters}, \emph{Many primary constructors}, \emph{Method chain}, \emph{Non final attribute}, \emph{Super Method}, \emph{Send Null patterns}: for them we observe that decresing them causes the metric to deteriorate, and increasing causes the metric to improve. We consider them \textbf{pro-patterns}. The third group (the rest of the patterns) can both improve and deteriorate the metric. We restrain oursevles from calling them either anti- or pro-patterns.

Attempting to interpret the results, we observe that ``true'' anti-patterns usually have \emph{if/else condition} or cycle in their definition.
E.g., \emph{Null check} always checks for a null, \emph{Count If Return}, \emph{Instance of}, 
\emph{Joined validation} have always \emph{if condition}, \emph{Nested Loop} has always at least 1 loop inside. Given that so far we have worked with the Cognitive Complexity metric, it is no surprise that those patterns affect it (\emph{if/else condition} or cycle are the main contributors into \emph{CogC}). Despite this limitation of the present analysis, we believe the proposed \textit{method} itself can be very useful in software engineering practice and research.




% ORIG TABLE:
% patterns                                & -1(top1)  & +1(top1)  &   p- m-   &   p+ m+   &   p- m+   &   p+ m-   &   p- m=   &   p+ m=   \\ 
% \\ \hline
% Asserts                                 &   0   &   100 &   92  &   61  &   186 &   219 &   5   &   3   \\
% Setters                                 &   1   &   113 &   245 &   160 &   901 &   976 &   21  &   31  \\
% Empty Rethrow                           &   1   &   4   &   77  &   65  &   25  &   36  &   0   &   1   \\
% Prohibited class name                   &   80  &   24  &   617 &   311 &   218 &   522 &   0   &   2   \\
% Force Type Casting                      &   69  &   24  &   2363    &   2313    &   1742    &   1790    &   14  &   16  \\
% Count If Return                         &   311 &   26  &   969 &   883 &   214 &   298 &   0   &   2   \\
% Implements Multi                        &   24  &   244 &   459 &   320 &   264 &   403 &   0   &   0   \\
% Instance of                             &   211 &   6   &   1396    &   1374    &   151 &   173 &   6   &   6   \\
% Many primary constructors               &   0   &   343 &   19  &   19  &   604 &   605 &   2   &   1   \\
% Method chain                            &   3   &   203 &   573 &   574 &   2217    &   2214    &   43  &   45  \\
% Multiple try                            &   156 &   180 &   371 &   239 &   259 &   391 &   0   &   0   \\
% Non final attribute                     &   34  &   485 &   1249    &   1185    &   5835    &   5839    &   127 &   187 \\
% Null check                              &   1573    &   14  &   5863    &   5978    &   575 &   457 &   9   &   12  \\
% Partial synchronized                    &   1   &   93  &   46  &   49  &   155 &   149 &   0   &   3   \\
% Redundant catch                         &   6   &   2   &   84  &   46  &   38  &   79  &   4   &   1   \\
% Return null                             &   104 &   40  &   1290    &   813 &   926 &   1401    &   4   &   6   \\
% String concat                           &   43  &   126 &   1596    &   2089    &   1506    &   1012    &   23  &   24  \\
% Super Method                            &   1   &   174 &   212 &   275 &   1012    &   949 &   10  &   10  \\
% This in constructor                     &   2   &   891 &   15  &   42  &   72  &   45  &   0   &   0   \\
% Var declaration distance for 5 lines    &   396 &   14  &   1976    &   1464    &   738 &   1244    &   19  &   25  \\
% Var declaration distance for 7 lines    &   25  &   95  &   1190    &   959 &   733 &   949 &   16  &   31  \\
% Var declaration distance for 11 lines   &   16  &   686 &   703 &   603 &   365 &   466 &   10  &   9   \\
% Var in the middle                       &   118 &   50  &   3372    &   3117    &   2799    &   3046    &   37  &   45  \\
% Array as function argument              &   86  &   25  &   882 &   768 &   351 &   464 &   1   &   2   \\
% Joined validation                       &   63  &   2   &   232 &   226 &   14  &   26  &   8   &   2   \\
% Non final class                         &   1056    &   2370    &   7141    &   3201    &   3723    &   7663    &   0   &   0   \\
% Private static method                   &   35  &   133 &   529 &   541 &   667 &   648 &   2   &   9   \\
% Public static method                    &   37  &   408 &   1166    &   1212    &   1142    &   1088    &   4   &   12  \\
% Null Assignment                         &   103 &   35  &   1245    &   803 &   717 &   1150    &   6   &   15  \\
% Multiple While                          &   65  &   7   &   120 &   97  &   16  &   39  &   0   &   0   \\
% Protected Method                        &   71  &   151 &   868 &   402 &   1147    &   1613    &   7   &   7   \\
% Send Null                               &   13  &   109 &   556 &   325 &   1510    &   1733    &   5   &   13  \\
% Nested Loop                             &   412 &   35  &   580 &   527 &   29  &   81  &   1   &   2   \\