FOSSology  4.4.0
Open Source License Compliance by Open Source Software
Wget Agent

Table of Contents

About Wget_agent

Wget agent is use to get files from FTP, FTPS, HTTP, HTTPS, GIT and SVN. It supports major flags of the GNU Wget.

Ways to use Wget agent

There are 3 ways to use the wget_agent:

  1. Command Line download: download one file or one directory from the command line
  2. Agent Based download: run from the scheduler
  3. Command Line locally to import the file(directory):
    • Import one file or one directory from the command line, used by upload from file and upload from server

Command Line download

To download one file or one directory from the command line: example:

./wget_agent http://www.aaa.com/bbb

Agent Based

To download one file or one directory (one URL )from the scheduler: example:

  1. part 1 parameters from the scheduler:
    19 - http://g.org -l 1 -R index.html*
    19 is uploadpk, 'http://g.org' is downloadfile url, '-l 1 -R index.html*' is several parameters used by wget_agent
  2. part 2 parameters from wget_agent.conf:
    `-d /var/local/lib/fossology/agents`
    '/var/local/lib/fossology/agent' is directory for downloaded file(directory) storage temporarily, after all file(directory) is dowloaded, move them into repo

Command Line locally to import the file(directory)

To Import one file or one directory from the command line into repo: example:

./wget_agent -g fossy -k $uploadpk '$UploadedFile'

Supported actions

Command line flag Description
-h Help (print this message), then exit
-i Initialize the DB connection then exit (nothing downloaded)
-g group Set the group on processed files (e.g., -g fossy)
-G Do NOT copy the file to the gold repository
-d dir Directory for downloaded file storage
-k key Upload key identifier (number)
-A acclist Specify comma-separated lists of file name suffixes or patterns to accept
-R rejlist Specify comma-separated lists of file name suffixes or patterns to reject
-l depth Specify recursion maximum depth level depth. The default maximum depth is 5
-c configdir Specify the directory for the system configuration
-C Run from command line
-v Verbose (-vv = more verbose)
-V Print the version info, then exit
OBJ if a URL is listed, then it is retrieved
if a file is listed, then it used
if OBJ and Key are provided, then it is inserted into
the DB and repository
no file process data from the scheduler

Agent source