wget_agent: Retrieve a file and put it in the database. More...

#include "wget_agent.h"

Include dependency graph for wget_agent.c:

Macros
#define	_GNU_SOURCE

#define	ASPRINTF_MEM_ERROR 88

#define	ASPRINTF_MEM_ERROR_LOG LOG_FATAL("Not enough memory for asprintf before line %d", __LINE__)

#define	PREFIXMAX 10

Functions
int	IsFile (char *Fname, int Link)
	Given a filename, is it a file? More...

void	SafeExit (int rc)
	Closes the connection to the server, free the database connection, and exit. More...

int	GetPosition (char *URL)
	Get the position (ending + 1) of http\|https\|ftp:// of one url. More...

void	DBLoadGold ()
	Insert a file into the database and repository. More...

int	TaintURL (char Sin, char Sout, int SoutSize)
	Given a URL string, taint-protect it. More...

char *	PrepareWgetDest (char TempFile, char TempFileDir, char *TempFileDirectory)
	Prepare directory for wget. More...

int	GetURL (char TempFile, char URL, char *TempFileDir)
	Do the wget. More...

int	GetVersionControl ()
	Get source code from version control system. More...

void	SetEnv (char S, char TempFileDir)
	Convert input pairs into globals. More...

char *	PathCheck (char *DirPath)
	Check if path contains a "%H", "%R". More...

int	Archivefs (char Path, char TempFile, char *TempFileDir, struct stat Status)
	Copy downloaded files to temporary directory. More...

void	GetProxy ()
	Get proxy from fossology.conf. More...

void	Usage (char *Name)
	Here are some suggested options. More...

void	replace_url_with_auth ()
	Translate authentication of git clone. More...

void	MaskPassword ()
	Get the username from GlobalParam and create new parameters without password.

char *	GetVersionControlCommand (int withPassword)
	get the command to run to get files from version control system More...

Variables
char	SQL [STRMAX]
	For DB. More...

PGconn *	pgConn = NULL
	For the DB. More...

long	GlobalUploadKey =-1
	Input for this system.

char	GlobalTempFile [STRMAX]
	Temp file to be used.

char	GlobalURL [URLMAX]
	URL to download.

char	GlobalType [STRMAX]
	Type of download (FILE/version control)

char	GlobalParam [STRMAX]
	Additional parameters.

char *	GlobalProxy [6]
	Proxy from fossology.conf.

char	GlobalHttpProxy [STRMAX]
	HTTP proxy command to use.

int	GlobalImportGold =1
	Set to 0 to not store file in gold repository.

gid_t	ForceGroup =-1
	Set to group id to be used for download files.

Detailed Description

wget_agent: Retrieve a file and put it in the database.

Definition in file wget_agent.c.

Function Documentation

◆ Archivefs()

int Archivefs	(	char *	Path,
		char *	TempFile,
		char *	TempFileDir,
		struct stat	Status
	)

Copy downloaded files to temporary directory.

If the path(fs) is a directory, create a tar file from files(dir) in a directory to the temporary directory.

If the path(fs) is a file, copy the file to the temporary directory

Parameters

Path	The fs will be handled, directory(file) you want to upload from server
TempFile	The tar(regular) file name
TempFileDir	Temporary directory path
Status	The status of Path

Returns: 1 on sucess, 0 on failure

Definition at line 819 of file wget_agent.c.

◆ DBLoadGold()

void DBLoadGold ( )

Insert a file into the database and repository.

Copy the downloaded file to gold repository according to the checksum and change the owner if ForceGroup is set. Then insert in upload and pfile tables.

Definition at line 81 of file wget_agent.c.

◆ GetPosition()

int GetPosition ( char * URL )

Get the position (ending + 1) of http|https|ftp:// of one url.

Parameters

URL The URL

Returns: the position (ending + 1) of http|https|ftp:// of one url E.g. http://fossology.org return 7

Definition at line 66 of file wget_agent.c.

◆ GetProxy()

void GetProxy ( )

Get proxy from fossology.conf.

Get proxy from fossology.conf and copy in GlobalProxy array

Definition at line 916 of file wget_agent.c.

◆ GetURL()

int GetURL	(	char *	TempFile,
		char *	URL,
		char *	TempFileDir
	)

Do the wget.

Parameters

TempFile	Used when upload from URL by the scheduler, the downloaded file(directory) will be archived as this file. When running from command, this parameter is null, e.g. /srv/fossology/repository/localhost/wget/wget.32732
URL	The url you want to download
TempFileDir	Where you want to store your downloaded file(directory)

Returns: int, 0 on success, non-zero on failure.

set proxy

Definition at line 328 of file wget_agent.c.

◆ GetVersionControl()

int GetVersionControl ( )

Get source code from version control system.

Returns: int - 0: successful; others: fail

for user fossy

git: git config --global http.proxy web-proxy.cce.hp.com:8088; git clone http://github.com/schacon/grit.git

svn: svn checkout --config-option servers:global:http-proxy-host=web-proxy.cce.hp.com --config-option servers:global:http-proxy-port=8088 https://svn.code.sf.net/p/fossology/code/trunk/fossology/utils/

Definition at line 566 of file wget_agent.c.

◆ GetVersionControlCommand()

char* GetVersionControlCommand ( int withPassword )

get the command to run to get files from version control system

Parameters

int	withPassword true to make command with actual or false to mask password

Returns: char* null terminated string

save each upload files in /srv/fossology/repository/localhost/wget/wget.xxx.dir/

Definition at line 1118 of file wget_agent.c.

◆ IsFile()

int IsFile	(	char *	Fname,
		int	Link
	)

Given a filename, is it a file?

Parameters

Link	Should it follow symbolic links?

Returns: int 1=yes, 0=no.

Definition at line 38 of file wget_agent.c.

◆ PathCheck()

char* PathCheck ( char * DirPath )

Check if path contains a "%H", "%R".

Check if path contains a "%U" or "%H". If so, substitute a unique ID for U.

Substitute the "%H" with hostname and "%R" with repo location. \parm DirPath Directory path.

Returns: new directory path

Definition at line 772 of file wget_agent.c.

◆ PrepareWgetDest()

char* PrepareWgetDest	(	char *	TempFile,
		char *	TempFileDir,
		char *	TempFileDirectory
	)

Prepare directory for wget.

Parameters

TempFile
TempFileDir
TempFileDirectory	Internal helper function for function GetURL

Returns: destination for wget download or NULL

Definition at line 300 of file wget_agent.c.

◆ replace_url_with_auth()

void replace_url_with_auth ( )

Translate authentication of git clone.

Translate authentication of git clone from http://git.code.sf.net/p/fossology/fossology.git –username –password password (input) to http://username:password@git.code.sf.net/p/fossology/fossology.git

Definition at line 1006 of file wget_agent.c.

◆ SafeExit()

void SafeExit ( int rc )

Closes the connection to the server, free the database connection, and exit.

Close scheduler and database connections, then exit.

Parameters

rc	Exit value

Definition at line 53 of file wget_agent.c.

◆ SetEnv()

void SetEnv	(	char *	S,
		char *	TempFileDir
	)

Convert input pairs into globals.

This functions taints the parameters as needed.

Parameters

S	The parameters for wget_aget have 2 parts, one is from scheduler, that is S
TempFileDir	The parameters for wget_aget have 2 parts, one is from wget_agent.conf, that is TempFileDir

This will be removed when the jobqueue is changed.

Definition at line 683 of file wget_agent.c.

◆ TaintURL()

int TaintURL	(	char *	Sin,
		char *	Sout,
		int	SoutSize
	)

Given a URL string, taint-protect it.

Parameters

[in]	Sin	The source URL
[out]	Sout	The tainted URL
[in]	SoutSize	The capacity of Sout

Returns: 1=tainted, 0=failed to taint

Definition at line 269 of file wget_agent.c.

◆ Usage()

void Usage ( char * Name )

Here are some suggested options.

Say how to run this program.

Parameters

Name	The name of the executable, usually it is wget_agent

Definition at line 976 of file wget_agent.c.

Variable Documentation

◆ pgConn

PGconn* pgConn = NULL

For the DB.

DB connection.

Database connection.

Definition at line 22 of file wget_agent.c.

◆ SQL

char SQL[STRMAX]

For DB.

SQL query to execute.

Are there any jobs running?

usage: $ck4j = new check4jobs(); $count = $ck4j->getJobCount; $count = $ck4j->Check;

NOTE: this program depends on the UI testing infrastructure at this point.

Returns: boolean (TRUE or FALSE)

Version: "$Id: $"

@TODO add method documentation @TODO create a subclass for doing longer waits (CheckandWait?)

Created on Jan. 15, 2009

Definition at line 20 of file wget_agent.c.

Macros

Functions

Variables

Detailed Description

Function Documentation

◆ Archivefs()

◆ DBLoadGold()

◆ GetPosition()

◆ GetProxy()

◆ GetURL()

◆ GetVersionControl()

◆ GetVersionControlCommand()

◆ IsFile()

◆ PathCheck()

◆ PrepareWgetDest()

◆ replace_url_with_auth()

◆ SafeExit()

◆ SetEnv()

◆ TaintURL()

◆ Usage()

Variable Documentation

◆ pgConn

◆ SQL