-This paper describes our approaches for the Plagiarism Detection task
-of PAN 2013.
+This paper describes our approaches for the Plagiarism Detection task of PAN 2013. We present modified three-way search methodology for source retrieval subtask. We introduce new query type – the paragraph based queries. Their purpose is to check some parts of suspicious text in more depth. The other two types of queries are: the keywords based for retrieval of documents concerning the same theme; and the intrinsic plagiarism based for retrieval sources which contain text detected as different, in a manner of writing style, from other parts of the suspicious document. The query execution was controlled by its type and by preliminary similarities discovered during the searches. We discuss 2-tuples snippet similarity measurement for decision making over search result download, which indicates how many neighbouring word pairs coexist in the snippet and in the suspicious document. Our tests indicate advantages setting of snippet similarity threshold. The results show that our approach had the second best ratio of recall to the number of used queries, which tells about the query efficacy. Our approached achieved low precision probably due to reporting many results which were not considered as correct hits. Nonetheless those results contained some textual similarity according to text alignment subtask score, which we believe is still worthwhile to report.
+For the text alignment subtask, we use the similar approach as in PAN 2012.We detect common features of various types between the suspicious and source documents. We experimented with more types of features. The best results had the combination of sorted word 4-grams with unsorted stop-word 8-grams. From the common features we compute valid intervals, which map passages from the suspicious document to the passages of the source document, such that these passages are covered “densely enough” with corresponding common features. For PAN 2013, we modified the post-processing phase: the fact that the algorithm had access to the whole corpus of source and suspicious documents at once allowed us to process the documents in one batch and to perform a global post-processing, handling the overlapping detections not only between the given suspicious and source document, but also between all the detections from a given suspicious document. The modifications brought a significant improvement compared to PAN 2012 on a training corpus, and the results from the competition corpus are similar enough to claim that these improvements are usable in general.
-We present modified three-way search methodology for source retrieval subtask.
-TODO Neco podrobnejsiho.
-
-For the text alignment subtask, we use the similar approach as in PAN 2012.
-We detect common features of various types between the suspicious and source
-documents. We have experimented with more types of features. The best
-results had the combination of sorted word 4-grams with unsorted stop-word
-8-grams. From the common features we compute valid intervals, which map
-passages from the suspicious document to the passages of the source document,
-such that these passages are covered ``densely enough'' with corresponding
-common features. For PAN 2013, we have modified the postprocessing phase:
-the fact that the algorithm had access to the whole corpus of source and
-suspicious documents at once allowed us to process the documents in one
-batch and to perform a global post-processing, handling the overlapping
-detections not only between the given suspicious and source document,
-but also between all the detections from a given suspicious document.
-The modifications brought a significant improvement compared to PAN 2013
-on a training corpus, and the results from the competition corpus
-are similar enough to claim that these improvements are usable in general.
\section{Introduction}
In PAN 2013 competition on plagiarism detection we participated in both the Source Retrieval
-and the Text Alignment subtask. In both tasks we adapted methodology used in PAN 2012\footnote{%
+and the Text Alignment subtasks. In both tasks we adapted methodology used in PAN 2012\footnote{%
See \cite{pan2012} for an overview of PAN 2012 plagiarism detection campaign.} \cite{suchomel_kas_12}.
Section~\ref{source_retr} describes querying approach for source retrieval, where we used three different
types of queries. We present a new type of query based on text paragraphs.
-The query execution were controled by its type and by preliminary similarities
+The query execution was controlled by its type and by preliminary similarities
discovered during the searches.
Section~\ref{text_alignment} describes our approach for the text alignment
(pairwise comparison) subtask. We briefly introduce our system,
\section{Conclusions}
-We introduces querying strategy with snippet similarity measure which approved to be
-competitive. In source retrieval subtask the strategy performed with the second best ratio
+We introduced querying strategy with snippet similarity measure. %which approved to be competitive.
+In source retrieval subtask the strategy performed with the second best ratio
of recall to the number of used queries.
We focused our queries on selected parts of text
and on parts with no discovered external similarities.
The source retrieval is a subtask in a plagiarism detection process during\r
which only a relatively small subset of documents are retrieved from the\r
large corpus. Those candidate documents are usually further compared in detail with the\r
-suspicious document. In the PAN 2013 source retrieval subtask the main goal was to\r
+suspicious document. In PAN 2013 source retrieval subtask the main goal was to\r
identify web pages which have been used as a source of plagiarism for test corpus creation.\r
\r
The test corpus contained 58 documents each discussing only one theme.\r
\r
%In the real-world scenario the corpus is the whole Web and the search engine can be a contemporary commercial search engine which scales to the size of the Web.\r
\r
-The same methodology -- utilizing a search engine is use for the source retrieval. \r
+The same methodology -- utilizing a search engine; is used for source retrieval. \r
This approach is based on the fact that we do not\r
possess enough resources to download and effectively process the whole corpus.\r
In the case of PAN 2013 competition we utilized the ChatNoir~\cite{chatnoir} \r
and in decision what to actually download and what to report as a plagiarism case from the search results.\r
\r
These first two stages are shown in Figure~\ref{fig:source_retr_process} as Querying and Selecting. Selected results \r
-from the search engine are forthwith textually aligned with the suspicious document (see section~\ref{text_alignment} for more details).\r
+from the search engine are then textually aligned with the suspicious document (see section~\ref{text_alignment} for more details).\r
%This is the last decision phase -- what to report.\r
If there is any continuous passage of reused text detected, the result document is reported\r
and the continuous passages in the suspicious document are marked as {\it discovered} and no further processing\r
i) keywords based queries, ii) intrinsic plagiarism\r
based queries, and iii) paragraph based queries. Three main properties distinguish each type of query: i) Positional; ii) Phrasal; iii) Deterministic.\r
Positional queries carry extra information about a textual interval in the suspicious document which the query represents.\r
-A phrasal query aims for retrieval of documents containing the same small piece of a text. They are usually created from closely coupled words. \r
+A phrasal query aims for retrieval of documents containing the same small piece of text. They are usually created from closely coupled words. \r
Deterministic queries for specific suspicious document are always the same no matter how many times we run the software. \r
%On the contrary the software can create in two runs potentially different nondeterministic queries.\r
\r
Our approach had the second best ratio of recall to the number of used queries, which\r
tells about the query efficacy. The approach with the best ratio used few queries (4.9 queries per document which\r
was 0.4 of the amount we used), but also obtained the lowest recall (0.65 of our recall).\r
-The approach with highest recall (and also lowest precision) achieved 2.8 times higher recall with 3.9 times more queries compared to ours.\r
+The approach with the highest recall (and also lowest precision) achieved 2.8 times higher recall with 3.9 times more queries compared to ours.\r
\r
Our approach achieved also low precision, which means we reported many more results and they\r
were not considered as correct hits. On the other hand each reported result contained some\r
\subsection{Alternative Features}\r
\label{altfeatures}\r
\r
-In PAN 2012, we have used word 5-grams and stop-word 8-grams.\r
-This year we have experimented with different word $n$-grams, and also\r
+In PAN 2012, we used word 5-grams and stop-word 8-grams.\r
+This year we experimented with different word $n$-grams, and also\r
with contextual $n$-grams as described in \cite{torrejondetailed}.\r
Modifying the algorithm to use contextual $n$-grams created as word\r
5-grams with the middle word removed (i.e. two words before and two words\r
\r
\plagdet{0.7421}{0.6721}{0.8282}{1.0000}\r
\r
-We have then made tests with plain word 4-grams, and to our surprise,\r
+We then made tests with plain word 4-grams, and to our surprise,\r
it gave even better score than contextual $n$-grams:\r
\r
\plagdet{0.7447}{0.7556}{0.7340}{1.0000}\r
of the corpus (in terms of plagdet score), except the 02-no-obfuscation\r
part.\r
\r
-In our final submission, we have used word 4-grams and stop-word 8-grams.\r
+In our final submission, we used word 4-grams and stop-word 8-grams.\r
\r
\subsection{Global Postprocessing}\r
\r
%for development, where it has provided a significant performance boost.\r
%The official performance numbers are from single-threaded run, though.\r
\r
-For PAN 2010, we have used the following postprocessing heuristics:\r
+For PAN 2010, we used the following postprocessing heuristics:\r
If there are overlapping detections inside a suspicious document,\r
keep the longer one, provided that it is long enough. For overlapping\r
-detections up to 600 characters, drop them both. We have implemented\r
-this heuristics, but have found that it led to a lower score than\r
+detections up to 600 characters, drop them both. We implemented\r
+this heuristics, but found that it led to a lower score than\r
without this modification. Further experiments with global postprocessing\r
of overlaps led to a new heuristics: we unconditionally drop overlapping\r
detections with up to 250 characters both, but if at least one of them\r
is longer, we keep both detections. This is probably a result of\r
plagdet being skewed too much towards recall (because the percentage of\r
-plagiarized cases in the corpus is way too high compared to real world),\r
+plagiarized cases in the corpus is way too high compared to real-world),\r
so it is favourable to keep the detection even though the evidence\r
for it is rather low.\r
\r
\r
\subsection{Evaluation Results and Future Work}\r
\r
- The evaulation on the competition corpus had the following results:\r
+ The evaluation on the competition corpus had the following results:\r
\r
\plagdet{0.7448}{0.7659}{0.7251}{1.0003}\r
\r
\r
We plan to experiment further with combining more than two types\r
of features, be it continuous $n$-grams or contextual features.\r
-This should allow us to tune down the aggresive heuristics for joining\r
+This should allow us to tune down the aggressive heuristics for joining\r
neighbouring detections, which should lead to higher precision,\r
-hopefully without sacrifying recall.\r
+hopefully without sacrificing recall.\r
\r
As for the computational performance, it should be noted that\r
our software is prototyped in a scripting language (Perl), so it is not\r
https://www.fi.muni.cz/~kas/git / pan13-paper.git / commitdiff