%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
-\title{Improving plagiarism detection}
+\title{Neco Simonovo and Feature Type Selection for Pairwise Document Comparison}
%%% Please do not remove the subtitle.
\subtitle{Notebook for PAN at CLEF 2013}
This paper describes approaches used for the Plagiarism Detection task in PAN 2013 international competition
on uncovering plagiarism, authorship, and social software misuse.
We present modified three-way search methodology for Source Retrieval subtask and analyse snippet similarity performance.
-Next, we show changes in selected feature for text alignement which led to plagdet score improvement.
-The results of source retrieval show, that presented approach is adaptable in real-world plagiarism situations.
-Improved results for text alignment achieved in the competition overall third place.
+The results show, that presented approach is adaptable in real-world plagiarism situations.
+For the detailed comparison task, we discuss feature type selection,
+global postprocessing. We have significantly improved the pairwise comparison
+results with even further optimizations possible.
\end{abstract}
\section{Introduction}
In PAN 2013 competition on plagiarism detection we participated in both the Source Retrieval
-and the Text Alignment subtask. In both tasks we adapted methodology used in PAN 2012.
+and the Text Alignment subtask. In both tasks we adapted methodology used in PAN 2012\footnote{%
+See \cite{pan2012} for an overview of PAN 2012 plagiarism detection campaign.} \cite{suchomel_kas_12}.
Section~\ref{source_retr} describes querying approach for source retrieval, where we used three different
types of queries. We present a new type of query based on text paragraphs.
The query execution were controled by its type and by preliminary similarities
discovered during the searches.
-In section~\ref{text_alignment} we present modified common text feature fot text alignment.
-We also compare performance of both the previous and the modified algorithms.
-
+In Section~\ref{text_alignment} we describe our approach for the text alignment
+(pairwise comparison) subtask. We briefly introduce our system,
+and then we discuss the feature types, which are usable for pairwise comparison,including the evaluation of their feasibility for this purpose. We then describe
+the global (corpus-wide) optimizations used, and finally we discuss
+the results achieved and further development.
\input{simon-source_retrieval}
\input{yenya-text_alignment}
Unfortunately the ChatNoir search engine does not support phrasal search, therefore it
is possible that evaluated results may be quite distorted in this manner.
+In the text alignment subtask, we have achieved a significant improvement
+with respect to our system from PAN 2012. Further development in this
+area is still possible. For a real-world system, however, a completely
+different set of parameters and heuristics needs to be used, as a result
+of plagdet score together with the structure of the competition corpus
+being too different from the real world.
+
\bibliographystyle{splncs03}
\begin{raggedright}
\bibliography{pan13-notebook}
\cite{stamatatos2011plagiarism}. From those common features (each of which\r
can occur multiple times in both source and suspicious document), we form\r
{\it valid intervals}\footnote{%\r
-We describe the algorithm for computing valid intervals in \cite{Kasprzak2009a},\r
-and a similar approach is also used in \cite{stamatatos2011plagiarism}.}\r
+See \cite{Kasprzak2009a} for the algorithm for computing valid intervals;\r
+a similar approach is also used in \cite{stamatatos2011plagiarism}.}\r
of characters\r
from the source and suspicious documents, where the interval in both\r
of these documents is covered ``densely enough'' by the common features.\r
\r
For the training corpus,\r
our unmodified software from PAN 2012 gave the following results\footnote{%\r
-See \cite{potthastframework} for definition of {\it plagdet} and the rationale for this type of scoring.}:\r
+See \cite{potthastframework} for definition of {\it plagdet} and the rationale behind this type of scoring.}:\r
\r
\def\plagdet#1#2#3#4{\par{\r
$\textit{plagdet}=#1, \textit{recall}=#2, \textit{precision}=#3, \textit{granularity}=#4$}\hfill\par}\r
\plagdet{0.7235}{0.6306}{0.8484}{1.0000}\r
\r
We take the above as the baseline for further improvements.\r
-In the next sections, we summarize the modifications we did for PAN 2013,\r
-including approaches tried but not used.\r
+In the next sections, we summarize the modifications we did for PAN 2013.\r
\r
\subsection{Alternative Features}\r
\label{altfeatures}\r
\subsection{Global Postprocessing}\r
\r
For PAN 2013, the algorithm had access to all of the source and suspicious\r
-documents at once. It was not limited to a single document pair, as in\r
-2012. Because of this, we have rewritten our software to handle\r
+documents at once. It was not limited to a single document pair, as it was\r
+in 2012. We have rewritten our software to handle\r
all of the documents in one run, in order to be able to do cross-document\r
optimizations and postprocessing, similar to what we did for PAN 2010.\r
-This required refactorization of most of the code. We are able to handle\r
-most of the computation in parallel in per-CPU threads, with little\r
-synchronization needed. The parallelization was used especially\r
-for development, where it has provided a significant performance boost.\r
-The official performance numbers are from single-threaded run, though.\r
+%This required refactorization of most of the code. We are able to handle\r
+%most of the computation in parallel in per-CPU threads, with little\r
+%synchronization needed. The parallelization was used especially\r
+%for development, where it has provided a significant performance boost.\r
+%The official performance numbers are from single-threaded run, though.\r
\r
For PAN 2010, we have used the following postprocessing heuristics:\r
If there are overlapping detections inside a suspicious document,\r
of overlaps led to a new heuristics: we unconditionally drop overlapping\r
detections with up to 250 characters both, but if at least one of them\r
is longer, we keep both detections. This is probably a result of\r
-plagdet being skewed too much to recall (because the percentage of\r
+plagdet being skewed too much towards recall (because the percentage of\r
plagiarized cases in the corpus is way too high compared to real world),\r
so it is favourable to keep the detection even though the evidence\r
for it is rather low.\r
\plagdet{0.7448}{0.7659}{0.7251}{1.0003}\r
\r
This is quite similar to what we have seen on a training corpus,\r
-only the granularity different from 1.000 is a bit surprising, given\r
-the aggressive joining of neighbouring detections we perform.\r
+with only the granularity different from 1.000 being a bit surprising.\r
+%, given\r
+%the aggressive joining of neighbouring detections we perform.\r
Compared to the other participants, our algorithm performs\r
especially well for human-created plagiarism (the 05-summary-obfuscation\r
sub-corpus), which is where we want to focus for our production\r
systems\footnote{Our production systems include the Czech National Archive\r
of Graduate Theses, \url{http://theses.cz}}.\r
\r
- After the final evaluation, we did further experiments\r
-with feature types, and discovered that using stop-word 8-grams,\r
-word 4-grams, {\it and} contextual $n$-grams as described in\r
-Section \ref{altfeatures} performs even better (on a training corpus):\r
-\r
-\plagdet{0.7522}{0.7897}{0.7181}{1.0000}\r
+% After the final evaluation, we did further experiments\r
+%with feature types, and discovered that using stop-word 8-grams,\r
+%word 4-grams, {\it and} contextual $n$-grams as described in\r
+%Section \ref{altfeatures} performs even better (on a training corpus):\r
+%\r
+%\plagdet{0.7522}{0.7897}{0.7181}{1.0000}\r
\r
We plan to experiment further with combining more than two types\r
of features, be it continuous $n$-grams or contextual features.\r
our software is prototyped in a scripting language (Perl), so it is not\r
the fastest possible implementation of the algorithm used. The code\r
contains about 800 non-comment lines of code, including the parallelization\r
-of most parts and debugging/logging statements. The only language-dependent\r
+of most parts and debugging/logging statements.\r
+\r
+ The system is mostly language independent. The only language dependent\r
part of the code is the list of English stop-words for stop-word $n$-grams.\r
We use no stemming or other kinds of language-dependent processing.\r
\r
https://www.fi.muni.cz/~kas/git / pan13-paper.git / commitdiff