Hi Graham, just need to check something. I found a little code to list links. It also checks whether that link exists. Another thing that happens is a time out. If there is no response from a server after so much time, it continues and won't show that particular link. Links are in a small text file (not many about 10 or so - some are added occassionally)

The time out code is this:
Code:

$ua->timeout(10); ## time out setting

My question is - what is the 10? 10 Seconds? 10 milliseconds? or what?

This works fine and fairly quickly, when I tested it from my Stateside server to UK server based links.

Hope you can shed some light on it for me, many thanks - Ted

Hi, Ted ... I'm afraid I'm going to answer a question with a question. "What sort of object id $ua?"

Timeout settings vary in their units ... our Tomcat deployment course always springs to mind when I mention this, as there are two settings in the main configuration files - one in milliseconds and one in minutes!

Hi Graham, does this help? It's a script I ran across and tried out. I assume the larger the time setting the longer  before the "time out" for a URL.

Code:

#!/usr/bin/perl use CGI::Carp qw(warningsToBrowser fatalsToBrowser); print "Content-type: text/html\n\n"; use LWP::Simple qw($ua head);   $dbfile = "osites.txt"; $ua->timeout(10); ## time out setting

There's the bit that says $ua head, so I guess that's some kind of reference in the LWP Simple module. It's not as long as a minute because the links get listed pretty quickly. I'm presuming it's milliseconds, but just wanted to be sure.

Cheers - Ted

Ted,

The code you have posted should do nothing except return an error or errors. Are you sure that is the exact code you are trying to use?

The timeout() methods interval is measured in seconds unless you tell your script to do something different.

But timeout() is not a method of LWP::Simple, which is not even a OO module. Maybe you are using LWP::UserAgent?

Hi Kevin,

I get no errors on my Local testbed server. Tried online on my live server it ran a treat. Both Apache 2.0/Perl 5.8

Here's the whole thing

Code:

The file just has a 2 field record in (for instance): http://www.somesite.com|Somesite http://www.thatwebsite.com|That Web Site http://anothersite.net|Another Site ---------------------------- #!/usr/bin/perl # Links to other site # Also check if link exists or times out. use CGI::Carp qw(warningsToBrowser fatalsToBrowser); print "Content-type: text/html\n\n"; use LWP::Simple qw($ua head);   $dbfile = "osites.txt"; $ua->timeout(10); ## time out setting ## read link list open (ORGDB,"$dbfile") || die("Cannot open $dbfile");   flock(ORGDB, 2); @ODB = <ORGDB>;   close (ORGDB); foreach $rec (@ODB){ ($surl,$snme)=split(/\|/,$rec); $url = "http://$surl"; if (head($url)) {print qq ~<a href="http://$surl" target="_blank">$snme</a><br>~;} else {} } exit;

Seconds should be fine then, I may lessen it some. It handles URLs that are not there okay, if a server's bogged down though it would slow stuff a little. Seeing as there won't be very many links it should suffice.

- Ted

OK, I See what is going on, quoted from LWP::Simple:

The module will also export the LWP::UserAgent object as $ua if you ask for it explicitly.

That was a new one to me. What that does is this:

Quote:

require LWP;    require LWP::UserAgent;    require HTTP::Status;    require HTTP:ate;    $ua = new LWP::UserAgent;  # we create a global UserAgent object    my $ver = $LWP::VERSION = $LWP::VERSION;  # avoid warning    $ua->agent("LWP::Simple/$LWP::VERSION");    $ua->env_proxy;

So it is using LWP::UserAgent and creates a $ua object sort of automatically. So, ignoring syntax errors inthe original code you posted, the last code you posted should run fine although it could maybe be pared down by using LWP::UserAgent directly and not loading other LWP stuff you're not using.


Thanks Kevin,

Quote:

could maybe be pared down by using LWP::UserAgent directly and not loading other LWP stuff you're not using.

I'll give that a try and see how it goes.

Ended up leaving as it was.

Tried a live test with 15 calls on and it's quite slow. The object is to have a links list so that if the URL is not there, it won't show. Too many links lists have sites that are no longer valid and I wanted to avoid that by testing the links first (without massive amounts of code).

The quest continues....

Update:
Now having done the test and seeing it's inherent slowness, I can see this needs to be approached from a different direction. Probably where I have a list of links in a file that comes up as normal (quickly). Then, do a regular check in my admin script (or a cron job) to see whether or not they are all still valid and write code to delete the out of date links.

Otherwise I end up with complaints that links aren't working. I hadn't realized the process it takes to check things like this (inexperience).  Learn something new every day

on 03/17/09 at 17:14:07, TedH wrote:

The object is to have a links list so that if the URL is not there, it won't show. Too many links lists have sites that are no longer valid and I wanted to avoid that by testing the links first (without massive amounts of code). The quest continues....

I've been there, Ted ... (but I do need to update my procedures and be there again).

Since links don't tend to die and com eback again quickly, there's no need to check all the URLs every time, or indeed at the time that the user is on the page - a script on the server that runs (say) once every 24 hours and builds up a local file or database of what exists and what has gone away should be sufficient in frequency, I would have thought.

I need one of those too - perhaps you should come down one evening / weekend and we could work on it (do I feel another Geekmas coming on)?

Graham







Whle I may not be able to get down there, I'll email you a copy when I've done it.

Been quite busy here lately, which is good, all things considered.

Get back to you later - Ted

on 03/17/09 at 17:14:07, TedH wrote:

Ended up leaving as it was. Tried a live test with 15 calls on and it's quite slow. The object is to have a links list so that if the URL is not there, it won't show. Too many links lists have sites that are no longer valid and I wanted to avoid that by testing the links first (without massive amounts of code). The quest continues.... Update: Now having done the test and seeing it's inherent slowness, I can see this needs to be approached from a different direction. Probably where I have a list of links in a file that comes up as normal (quickly). Then, do a regular check in my admin script (or a cron job) to see whether or not they are all still valid and write code to delete the out of date links. Otherwise I end up with complaints that links aren't working. I hadn't realized the process it takes to check things like this (inexperience).  Learn something new every day

It should speed up if you shorten the timeout() interval but it might also be less accurate.  Personally I would do something like Graham suggests or implement a "report a bad link" feature so your users can report bad links to you and you can investigate them on an as needed basis. Links not working one day might work the next just because of the unpredictable nature of the internet. Overall, this is not something you want to do in real time, you want to do it as a cron job or task or periodically check the links manually or use a reporting system.

Yeah, I think it is best to do it as a cron job or as an admin function. I prefer the admin function as that would give time to double check in case a link comes back (i.e. - server down at time of check).

Right now I've got to the point of putting the result into an array and figuring out how to get it written to a file. Standard type array didn't work (only wrote the first line), so I'm going to have a look at your hashes suggestion (from another posting).

Oh what fun we have

Update:
Tried to get too fancy. It's just basic Perl. I simply used a form for the results and saved them. Kills a number of birds with one stone.

Hey Kevin, been tearing my hair out for hours trying stuff before I figured out that LWP Simple throws in an extra, blinkin' carriage return. There's no smiley for Aaaaaaargh!!

Happened after the valid URLs got read into the textarea and I saved them. The save grabbed it. Solved with,
Code:

~ s/\r\n/\n/g;

I think I'll sit down and watch NCIS now - I need a break.

I've been bald for years, so welcome to the club.

---