Have Geek Brain

How do I parse an HTML site using Perl?

Could you give me some tips for parsing HTML in Perl? I am planning on parsing keywords (including URL links) and storing them in a MySQL database. I am using Windows XP.

Also, do I need to download some of the website pages to my local hard drive first using some kind of offline explorer? If so, can you point me to a good download tool?

+2


a source to share


4 answers


You can use LWP to fetch pages to be analyzed. There are many ways to parse HTML. You can use regular expressions to find links and keywords (although this is usually not a good practice) or modules like HTML :: TokeParser or HTML :: TreeBuilder.



+4


a source


You can use one of the many HTML parser modules. If you are familiar with jQuery, the pQuery module is a good choice as it carries over most of the easy-to-use jQuery features of Perl for parsing and scraping HTML.



+2


a source


The HTTrack copier / downloader website has a lot more features than any Perl library available.

+1


a source


To navigate and save the whole site locally, you can use wget -r -np http://localhost/manual/

(wget is available on Windows, standalone or partly Cygwin / MinGW). That said, if you want to both forward data as well , Mojolicious can be used to create a simple parallel web crawler that is very light on dependencies:

#!/usr/bin/env perl
use feature qw(say);
use strict;
use utf8;
use warnings qw(all);

use Mojo::UserAgent;

# FIFO queue
my @urls = (Mojo::URL->new('http://localhost/manual/'));

# User agent following up to 5 redirects
my $ua = Mojo::UserAgent->new(max_redirects => 5);

# Track accessed URLs
my %uniq;

my $active = 0;
Mojo::IOLoop->recurring(
    0 => sub {

        # Keep up to 4 parallel crawlers sharing the same user agent
        for ($active .. 4 - 1) {

            # Dequeue or halt if there are no active crawlers anymore
            return ($active or Mojo::IOLoop->stop) unless my $url = shift @urls;

            # Fetch non-blocking just by adding a callback and marking as active
            ++$active;
            $ua->get(
                $url => sub {
                    my (undef, $tx) = @_;

                    say "\n$url";
                    say $tx->res->dom->at('html title')->text;

                    # Extract and enqueue URLs
                    for my $e ($tx->res->dom('a[href]')->each) {
                        # Validate href attribute
                        my $link = Mojo::URL->new($e->{href});
                        next if 'Mojo::URL' ne ref $link;

                        # "normalize" link
                        $link = $link->to_abs($tx->req->url)->fragment(undef);
                        next unless $link->protocol =~ /^https?$/x;

                        # Access every link once
                        next if ++$uniq{$link->to_string} > 1;

                        # Don't visit other hosts
                        next if $link->host ne $url->host;

                        push @urls, $link;
                        say " -> $link";
                    }

                    # Deactivate
                    --$active;
                }
            );
        }
    }
);

# Start event loop if necessary
Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
0


a source






More articles:


All Articles