[wellylug] Automatically retrieving certificate and webpage via HTTPS

Ewen McNeill wellylug at ewen.mcneill.gen.nz
Wed Dec 28 21:46:36 NZDT 2005


In message <43B21754.7010708 at gmail.com>, David Antliff writes:
>When I was on Paradise.net, I wrote and maintained a script that 
>downloaded my current cable Internet data usage data. [...]
>I have since moved to Ihug and things are a little different [...]
>Usage page requires an SSL authenticated session to be established [...]
>I don't really know much about SSL and HTTPS yet, so my question is - 
>can this login process be automated somehow so that I can modify my 
>script to collect usage data from Ihug? 

Yes it should be possible to automate the collection from iHug, although
you may be better off using something like Perl and LWP to do the
collection, especially if you get a cookie once you log in which you
have to send back at the next stage (I do this to, eg, collect the
prepay balance off Vodafone's website).

>How do I go about obtaining a certificate from their server? 

The server certificate will be sent to you as part of the HTTPS
negotiation (along with the SSL chain that authorises it back to a "well
known" certificate authority).  Providing you don't care about
"proving" that you're collecting the information from the correct site
(ie, verifying the SSL certificate is as expected) you can simply ignore
this.

SSL does support client certificates as well (which is, I assume, what
wget offers as an option), but it's extremely unlikly that iHug assigns
you a client certificate and requires you to send it as part of the
login process.   (SSL client certificates are almost never used outside
some very closed systems.)

I'll include the perl script I use to get paradise cable data (from the
https service), which may give you some hints on what is required.  If
you need to track cookies look at HTTP::Cookies and friends.

Ewen

-=- cut here -=-
#! /usr/bin/perl 
# Request current traffic information from Paradise
# Works for Cable Traffic plans, but it appears not for Citylink Plans.
# Written by Ewen McNeill <ewen at naos.co.nz>, 2003/04/23 based on an idea
# by Michael Robinson <michael at diaspora.gen.nz>
# Rewritten by Ewen McNeill <ewen at naos.co.nz>, 2004/02/15 to use newer
# CGIs at paradise.
# Free for you to use.  If you break it, you own both parts.
BEGIN { delete $ENV{'https_proxy'}; }
use LWP::UserAgent;
use HTTP::Request::Common;
$OUTPUT_FORMAT="%-25s %15s %15s %15s\n";
@HEADINGS=('Billing Period','International','Domestic','Total');
$name=$ENV{'PARADISE_USER'} || 'ewenmcneill';
if (-t STDIN)
{ 
  print "Enter password: ";
  system "stty", "-echo";
  chomp($password=<>);
  system "stty", "echo";
  print "\n";;
}
else
{
  chomp($password=<>);
}
$ua = new LWP::UserAgent;
$req = GET "https://www.paradise.net.nz/cgi-bin/members/usage/show-volume-usage";
$req->authorization_basic($name,$password);
$res = $ua->request($req) or die "Cannot make request";
@ans = split(/\n/, $res->content());
my ($daterange, at totals);
my $index = 0;
foreach (@ans)
{
  if (/(\d\d\d\d-\d\d-\d\d to \d\d\d\d-\d\d-\d\d)/) { $daterange=$1; $index=0;}
  if (/<td align="RIGHT".*?>(\d+\.\d+)</)           { $totals[$index++]=$1;   }
}
printf $OUTPUT_FORMAT, @HEADINGS;
printf $OUTPUT_FORMAT, $daterange, at totals[0..2];
-=- cut here -=-




More information about the wellylug mailing list