=pod

=encoding utf8

    wiklean.pl - Perl script that cleans Wikipedia code (ie converts to plain text).
    Copyright (C) 2016  Rogelio Nazar

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.


Documentation: 

There have been different attempts to extract plain text from Wikipedia files, 
such as:
https://blog.afterthedeadline.com/2009/12/04/generating-a-plain-text-corpus-from-wikipedia/
or:
http://medialab.di.unipi.it/wiki/Wikipedia_Extractor


wiklean.pl is just another approach to the subject.

Variables needed to be set:

There are some words that you need to translate to the target language.
For instance:

# Archivo => Fichier
# Categoría => Catégorie



Regarding the corpus: sometimes you will have a single input file. 
Someother times, you may have a set:
(number of lines indicated for each file)
   29967548 frwiki-20160501-pages-meta-current1.xml
   56190910 frwiki-20160501-pages-meta-current2.xml
   95000601 frwiki-20160501-pages-meta-current3.xml
 233863618 frwiki-20160501-pages-meta-current4.xml

A loop is activated for this cases


------------------------------------

In case of encoding problems (ie, always)
simply apply:

iconv -c -f utf8 -t Latin1 file.txt > fileiso.txt

and then back to utf8:

iconv -c -f Latin1 -t utf8 fileiso.txt > file.txt

Of course this will make you loose the corrupted
characters.

You may also find it convenient to change the names of the files to 
something more comfortable:

mv frwiki-20160501-pages-meta-current1.xml-p000000003p000412301 frw1.xml
mv  frwiki-20160501-pages-meta-current2.xml-p000412304p001647888 frw2.xml
mv  frwiki-20160501-pages-meta-current3.xml-p001647895p004419859 frw3.xml
mv  frwiki-20160501-pages-meta-current4.xml-p004419860p009976943 frw4.xml

So this would mean:

iconv -c -f utf8 -t Latin1 frw1.xml > frw1iso.xml
iconv -c -f Latin1 -t utf8 frw1iso.xml > frw1.xml
iconv -c -f utf8 -t Latin1 frw2.xml > frw2iso.xml
iconv -c -f Latin1 -t utf8 frw2iso.xml > frw2.xml
iconv -c -f utf8 -t Latin1 frw3.xml > frw3iso.xml
iconv -c -f Latin1 -t utf8 frw3iso.xml > frw3.xml
iconv -c -f utf8 -t Latin1 frw4.xml > frw4iso.xml
iconv -c -f Latin1 -t utf8 frw4iso.xml > frw4.xml

And then:

rm frw1iso.xml
rm frw2iso.xml
rm frw3iso.xml
rm frw4iso.xml

You can do this manually if it is a single file or just a few.
But with more files you can use this simple script:

perl convertFiles.pl inputfolder outputfolder

And then again but this time changing the order of the arguments.

For initial tests, it is best to try with a fragment of the text:

head -10000 enwiki-20160801-pages-meta-current.xml > wikipediafile.xml 

Proceed then to 

perl wiklean.pl wikipediafile.xml testhead.txt

 ~/treetagger/cmd/tree-tagger-french testhead.txt > taghead.txt

Of course there are errors, plenty of them. 
For example:

grep » taghead.txt | grep -v cit

grep "unknown" taghead.txt | more

grep "unknown" taghead.txt > errors.txt

This will show you the errors you have and see which ones are frequent.

We even have a script for that: errorchecker.pl

To be used like this:

grep "unknown" taghead.txt > errors.txt
perl errorchecker.pl errores.txt

Once we have made all the tests and we are sure errors have been corrected,
proceed with the cleaning of all the corpus

A "secret" function of this script is the extraction of synonyms from Wikipedia's metada.
If interested, run like this:

perl wiklean.pl wikipediafile.xml outputFile.txt synonyms




=cut

use strict;
$|++;

my ($file, $outname, $synonyms) = @ARGV;

# Here, a two level stoplist was needed.
# This should be translated to the target language.
my @kill = ( 
#'Fichier',
#'Notes et références|Bibliographie primaire|Bibliographie secondaire|Catégorie:'

'File',
'Notes et références|Bibliographie primaire|Bibliographie secondaire|Category:'



);





# In the case of French, we had four files. 
# Here, numbers indicate the number of lines of the files, 
# in order to have some sense of progress.
# my @sizes = qw(29967548 56190910 95000601 233863618);

# in the French version we had this:

#foreach my $ef (1 .. 4) { # and this should have not been 4 but scalar(@files)
#  my $file = "frw".$ef.".xml";
#  my $outname = "frw".$ef."clean.txt";
#  my $outag = "frw".$ef.".tag";
#  my $wc = $sizes[($ef-1)];

print "\nStarting process with file $file. ";


# print "Calculating size with wc... (if you know it already, edit this script to skip this part)\n";

## Now we assume we have a single file. 
# my $wc = `wc -l $file`;
#chomp $wc;
#$wc =~ s/ +.+//g;

# Once you know this, you can skip this step
 my $wc = 1763197667;

my $sino = 0; # if active, will only provide a list of synonyms.
$sino++ if ($synonyms =~ /synonym/i);

print "\nOk. Size in lines is: [$wc]\n";

my $outag = $outname.".tag";
my %sin;
my $count;
open(FILE, "<", $file) or die "I cannot open file  $file";
open OUT, ">", $outname or die "\nWarning: I cannot open file $outname for writing:".$!;
while (my $line = <FILE>)  {
	$count++;
	chomp $line;
	# [[sector servicios|sector terciario]] 
	while (	$line =~ s/\[\[([^\|\[\]]+)\|([^\|\[\]]+)\]\]/$2/) {
		my $uno = $1;
		my $dos = $2;
		next if ($uno =~ /^$kill[0]/ ) ;
		if ($sin{$uno."\t".$dos}) {
			$sin{$uno."\t".$dos}++;
		} else {
			$sin{$dos."\t".$uno}++;
		}
	}; 

	# we delete all the references:
	# &lt;ref&gt;{{cita web|url=http://www.iho-ohi.net/mtg_docs/circular_letters/spanish/2001/Cl55s.pdf|título=Circular C55 de 2001 de la OHI|editorial=iho-ohi.net|formato=pdf|fechaacceso=17 de mayo de 2014}}&lt;/ref&gt;
	# &lt;/ref&gt;
	$line =~ s/&lt;ref&gt;\{\{[^\}]+\}\}&lt;\/ref&gt;//g;
	$line =~ s/https?:\/\/[^ ]+/URL/g;
 	$line =~ s/[\[\]]+//g;
	$line =~ s/\{\{[^\}]+\}\}//g;
#	$line =~ s/''+/"/g;
#	$line =~ s/[]/"/g;
#	« L"""Arrestation d"""Arsène Lupin »
	$line =~ s/([^ ])([«»]+)/$1 $2/g;
	$line =~ s/([«»]+)([^ ])/$1 $2/g;
	$line =~ s/''+/"/g;
	$line =~ s/""+/"/g;
	$line =~ s/ ?===* ?//g;
	$line =~ s/http:\/\/[^ ]+//g;
	# next if ($line =~ /^Archivo/ ) ;
	next if ($line =~ /^$kill[0]/ ) ;
	next if ($line =~ /.jpg\|/i);
	$line =~ s/(<\/text>|&amp;|nbsp;|ref name=|&quot;|\/gallery )//g;
	$line =~ s/\/?(ref)?\&[lgtnbsp]+;/ /g;
#	a le coupable comme narrateur. .
	$line =~ s/\. \.$/\./g;
	$line =~ s/[\|\}\^\*=\{\#]+/ /g;
	$line =~ s/(['\(\)]+)/ $1 /g;
	$line =~ s/ (br|URL|CET|ref) / /g;
	$line =~ s/(align|style|class|scope|group|cellspacing|border|mode|url)=[^ ]+/ /g;
	$line =~ s/(fontsize|$kill[0]|shift|align|user_Talk|color|data):[^ ]+/ /g;
	$line =~ s/ \/small / /g;
	$line =~ s/  +/ /g;
	next if ($line =~ /^ *[\*#=_!<\|\{\}]+/ # 0-9
	|| length ($line) < 30
	|| length ($line) > 10000
#	|| $line =~ /^(Categoría:| td style=background|data:image\/)/
	|| $line =~ /^($kill[1])/ 
	|| $line =~ /^image_/
	|| $line =~ /^ ?\/?center ?$/
	# || $line =~ /^:  small .+ \/small$/
	);
	print OUT $line."\n" if $line && !$sino;
	# The <p> tag indicates end of sentence
	# and is used by Treetagger.

	if ($count =~ /00000$/) {
		print $file ." ".int (100* ($count/$wc))."%\n";
	}
}

close FILE;

if ($sino) { # they don't want the list of synonyms
  foreach my $v ( sort { $sin{$b} <=> $sin{$a} } keys %sin) {
	print OUT "\n$sin{$v}\t$v" if ($sin{$v});
	$v =~ s/([^\t]+)\t([^\t]+)/$2\t$1/;
	delete $sin{$v};
  }
}

close OUT;
print "\nOk!. Now you can issue command: \n ~/treetagger/cmd/tree-tagger-french $outname > $outag \n\n";