AndreaPlanet.com - My door always open to you * Login   * Register * FAQ    * Search
It is currently Mon Sep 01, 2014 2:57 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 15 posts ] 
Author Message
 Post subject: Google Images Photo Grabber
PostPosted: Sun Mar 20, 2005 9:04 pm 
I wrote a small Perl program to help collect images for use in my mosaics. It takes a search term then grabs as many images from Google Images as you want. It only saves the thumbnails, which are all I need for a mosaic.

You can read the description can get the source code here

It's pretty simple, so you could easily modify it to get images from yahoo, or to download the full-size images, or organize them differently.


Top
  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Sun Mar 20, 2005 9:56 pm 
:o This is exactly what I have been lookin for. Only if I know what to do with this source code. Can you elaborate a little more how I can get this to work? I'm not that savy with web programming. Thanks.
badsegue wrote:
I wrote a small Perl program to help collect images for use in my mosaics. It takes a search term then grabs as many images from Google Images as you want. It only saves the thumbnails, which are all I need for a mosaic.

You can read the description can get the source code here

It's pretty simple, so you could easily modify it to get images from yahoo, or to download the full-size images, or organize them differently.


Top
  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Tue Mar 22, 2005 11:44 am 
thisisgood wrote:
:o This is exactly what I have been lookin for. Only if I know what to do with this source code. Can you elaborate a little more how I can get this to work? I'm not that savy with web programming. Thanks.


It may be a little awkward for those who aren't comfortable with running scripts. First you'll need to install perl. You can download it from ActiveState
I think it will add perl to your command path if you let it, then you can just run the script as I described in the instructions, e.g.:
Code:
perl get.pl "starry night" 1 100

If you let it associate ".pl" files with perl then you can try
Code:
get.pl "starry night" 1 100


Give it a try and let me know how it works, and I can try to help out if there are problems.


Top
  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Wed Mar 23, 2005 6:28 am 
Offline
Site Admin
User avatar

Joined: Mon Apr 15, 2002 9:12 am
Posts: 1257
Location: Italy
I tried the script and it works well. :)
But as you said it just print out the filename. Do you think it is complicated for you to change the script to save to a subdirectory all the thumbnails directly?


Top
 Profile  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Wed Mar 23, 2005 12:12 pm 
Andrea wrote:
I tried the script and it works well. :)
But as you said it just print out the filename. Do you think it is complicated for you to change the script to save to a subdirectory all the thumbnails directly?


It should be saving the thumbnails. You probably need to change the destination folder (I should have changed that earlier!). Change this line to point to your desired directory:
Code:
my $in    = "F:/Images";


If you've done that and and it's still not working then let me know and I'll try to figure out what is wrong.

I'm also working on a small Windows program to do the same thing. That should be less trouble than a script. I'm able to get images from google and yahoo. I also am looking at grabbing images from a newsgroup service, like this one:
http://archive.xusenet.com/f/Urecommend/index.html


Top
  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Wed Mar 23, 2005 1:03 pm 
Offline
Site Admin
User avatar

Joined: Mon Apr 15, 2002 9:12 am
Posts: 1257
Location: Italy
oops...I didnt check the destination folder...
I changed it to "./" so the program save the images to the local subdirectory. I think it's the best default setting. I just gave command

get.pl "flowers" 0 10000

and it is running very well. Just somewhat slow since it save/download one image at a time but this is not a problem. In the worst case I can run multiple commands at the same time, something like this:

get.pl "flowers" 0 1000
get.pl "flowers" 1000 2000
get.pl "flowers" 2000 3000
get.pl "flowers" 3000 4000


Top
 Profile  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Wed Mar 23, 2005 8:43 pm 
Andrea wrote:
oops...I didnt check the destination folder...
I changed it to "./" so the program save the images to the local subdirectory. I think it's the best default setting. I just gave command

get.pl "flowers" 0 10000

and it is running very well. Just somewhat slow since it save/download one image at a time but this is not a problem. In the worst case I can run multiple commands at the same time, something like this:

get.pl "flowers" 0 1000
get.pl "flowers" 1000 2000
get.pl "flowers" 2000 3000
get.pl "flowers" 3000 4000


I guess it would be better to make it multi-threaded for speed. I usually only grab about 500 images or so for a single search--after that the relevance of the images is usually pretty low. For something like flowers I'd do separate searches for "flowers", "rose", "tulip", etc., until I get enough good images. I'm looking at the various search engines, some seem better than others at finding images that closely match what I'm looking for. I'd like to get it to the point where you can pick the engines you want to use then it will fetch your search from all of them.


Top
  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Thu Mar 24, 2005 3:57 am 
Offline
Site Admin
User avatar

Joined: Mon Apr 15, 2002 9:12 am
Posts: 1257
Location: Italy
I agree completeley. Also the command

get.pl "flowers" 0 10000

stopped at 622 images... as I remember google limit the number of results....


Top
 Profile  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Thu Mar 24, 2005 4:26 am 
Offline
Site Admin
User avatar

Joined: Mon Apr 15, 2002 9:12 am
Posts: 1257
Location: Italy
An idea is to have for the input a text file with all search terms

flower
red flower
blu flower
etc

Then we run the get.pl with the text file, something like:

multiget.pl "myfile.txt" 0 200 10

and the script run the different get.pl specifiyng the search parameter.
The last parameter is the maximum number of simultaneous open processes.


Top
 Profile  
 
 Post subject: Re: Google Images Photo Grabber
PostPosted: Mon Apr 11, 2005 8:36 pm 
badsegue wrote:
I wrote a small Perl program to help collect images for use in my mosaics. It takes a search term then grabs as many images from Google Images as you want. It only saves the thumbnails, which are all I need for a mosaic.

You can read the description can get the source code here

It's pretty simple, so you could easily modify it to get images from yahoo, or to download the full-size images, or organize them differently.


The link here is broken. I must have changed some configuration option by mistake. Sorry to anyone that was unable to find it. If anyone is still looking for this it's here: http://badsegue.org/archives/2005/03/21/photomosaics-and-a-google-image-grabber

I'm almost ready to make available a windows app that does what the script does and more. I'll be sure to post a notice here when it's finally ready, in a few days.

-----------------
http://badsegue.org


Top
  
 
 Post subject:
PostPosted: Mon Apr 11, 2005 10:03 pm 
Offline
Fan

Joined: Mon Jan 10, 2005 11:33 pm
Posts: 23
Hmm, I really want to use this google image finder, but I am baffeled by how it works. I honestly don't know where to begin. Please help, because I would like to use this feature ASAP lol.

THANKS!


Top
 Profile  
 
 Post subject: Ahh well...
PostPosted: Tue Apr 12, 2005 9:55 am 
Offline
Fan

Joined: Mon Jan 10, 2005 11:33 pm
Posts: 23
Wells, it's kinda too late now, but I still EXTREMELY WANT TO KNOW how to use this google image grabber, lol.

I'm trying to grab as many pictures of Rene Magritte's artwork and make a moasic from them, but doing this by hand and copying and pasting them each individually is just being waaaaaaay too time consuming.

Please post here as soon as ou possibly can with like instructions on how I can run your program. It's probably really simple, I just can't get to the right place to type in my search querey of "Rene Magritte" or "Magritte."

I have already downloaded Perl using the link you gave me, but I have no idea what that program is either, therefore I have no idea how to use your "script" or whatever, lol.

Please please please please please help as soon as you can, lol, I need this mosaic done soon. Thanks so much!


Top
 Profile  
 
 Post subject: Re: Ahh well...
PostPosted: Tue Apr 12, 2005 1:02 pm 
Offline
Site Admin
User avatar

Joined: Mon Apr 15, 2002 9:12 am
Posts: 1257
Location: Italy
just follow the instructions on this page:
http://badsegue.org/archives/2005/03/21 ... ge-grabber
in other words:

1. download this text and save it into a file called get.pl

Code:
use HTML::Parser;
use HTTP::Request::Common;
use LWP;
use URI::Escape;
   
use strict;
   
$|=1;
   
my $client = LWP::UserAgent->new(agent=>'Mozilla', timeout=>'0', keep_alive=>1);
my $ua    = "Mozilla";
my $in    = "./";
my $query = shift; chomp($query);
my $start_idx = shift; chomp($start_idx);
my $end_idx = shift; chomp($start_idx);
my $url   = "http://images.google.com/images?q=$query+filetype:jpg\&safe=off";
my $start = $start_idx || "0";
my $stop = $end_idx || 0;
my $dest_dir = "$in/" . uri_escape ($query);
   
my $count = 1;
   
my $p = new HTML::Parser (
 api_version => 3,
 start_h     => [\&tag, "tagname, attr"],
);
   
print "Start = $start, Stop = $stop, Query = $query\n";
mkdir $in || die "Couldn't make $in ($!)\n";
mkdir $dest_dir || die "Couldn't make $dest_dir ($!)\n";
   
while (1) {
  my $test = $start;
   
  # Get the search results page
  my $request = HTTP::Request->new('GET', "${url}\&start=${start}");
  my $response = $client->request($request);
   
  $p->parse( $response->content );
  # See if we are out of images
 if ($test == $start || ($stop && ($start >= $stop))) {
  print "Done.\n";
  exit 0;
 }
}
   
sub tag {
  my ($tagname, $attr) = (@_);
   
  # Found the next page graphic, increment counter to continue grabbing
  if ($attr->{'src'} && ($attr->{'src'} eq "/nav_next.gif" )) {
        $start += 20;
  }
   
  return unless ($tagname eq 'img');
  return unless ($attr->{'src'} && $attr->{'src'} =~ /images\?q=tbn:.*\.jpg/i);
  my $filename = $attr->{'src'};
  $filename =~ s/\/images\?q=tbn:.*://;
  $filename = uri_escape($filename);
   
  if (-e "$dest_dir/$filename") {
    print "Skipping ";
  } else {
    my $request = HTTP::Request->new('GET', "http://images.google.com$attr->{'src'}");
    my $response = $client->request($request, "${dest_dir}/${filename}");
  }
  print "$filename (", $count++, ")\n";
}



2. install the program ActivePerl and say Yes for the associations of the .PL extension to ActivePerl.

3. from the command line where the get.pl is located execute the command:

get.pl \"flower\" 0 100

Note: to get the command line prompt use Start/Run... and insert "cmd.exe"

byeee
Andrea


Top
 Profile  
 
 Post subject:
PostPosted: Tue Apr 12, 2005 1:03 pm 
Offline
Site Admin
User avatar

Joined: Mon Apr 15, 2002 9:12 am
Posts: 1257
Location: Italy
...an idea...why not use perl2exe for to create a windows exectuable of this script?


Top
 Profile  
 
 Post subject:
PostPosted: Tue Apr 12, 2005 9:13 pm 
Offline
Fan

Joined: Mon Jan 10, 2005 11:33 pm
Posts: 23
Lol

I guess I just have to accept the fact that I'll never figure this out.

Thanks for all your help, but it is still very vague to me. Lol, unless you are willing to go EXTREMELY in depth about how I can use your tool, I wont be able to use it LOL (because I'm not good with scripts at all).

That's ok though, I don't want to steal all of your time, but if you are willing to I will DEFINITLY LISTEN! LOL

So thanks either way!!!!!
-Chris


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 15 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: Yahoo [Bot] and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group