Public Member Functions
def	__init__ (self, timeout=5, max_tries=5, verbose=False, clobber=False)

def	open_url (self, url, ntries=None, get=False)

def	needs_download (self, url, filepath, check_times=False, response=None)

def	download_file (self, url, filepath)

def	list_pageurls (self, url, regex='')

def	download_allfiles (self, url, dirpath, regex='', check_times=False, response=None, clobber=False, dry_run=False)

def	spider (self, url, level=0, visited=None)

Public Attributes
	timeout

	max_tries

	verbose

	clobber

	session

Detailed Description

Definition at line 137 of file SessionUtils.py.

Constructor & Destructor Documentation

◆ init()

def __init__	(	self,
		timeout = `5`,
		max_tries = `5`,
		verbose = `False`,
		clobber = `False`
	)

Definition at line 139 of file SessionUtils.py.

Member Function Documentation

◆ download_allfiles()

def download_allfiles	(	self,
		url,
		dirpath,
		regex = `''`,
		check_times = `False`,
		response = `None`,
		clobber = `False`,
		dry_run = `False`
	)

Downloads all available files from a remote url into a local dirpath.
Default is to download only if local file doesn't match remote size;
set clobber=True to always download.

Definition at line 271 of file SessionUtils.py.

◆ download_file()

def download_file	(	self,
		url,
		filepath
	)

Definition at line 235 of file SessionUtils.py.

◆ list_pageurls()

def list_pageurls	(	self,
		url,
		regex = `''`
	)

Returns a sorted, unique set of links from a given url.
Optionally specify regex to filter for acceptable files;
default is to list only links starting with url.

Definition at line 246 of file SessionUtils.py.

◆ needs_download()

def needs_download	(	self,
		url,
		filepath,
		check_times = `False`,
		response = `None`
	)

Returns False if filepath is present and size matches remote url;
True otherwise.  Optionally check timestamp as well.

Definition at line 205 of file SessionUtils.py.

◆ open_url()

def open_url	(	self,
		url,
		ntries = `None`,
		get = `False`
	)

Return requests.Session object for specified url.
Retries up to self.max_tries times if server is busy.
By default, retrieves header only.

Definition at line 146 of file SessionUtils.py.

◆ spider()

def spider	(	self,
		url,
		level = `0`,
		visited = `None`
	)

Demo crawler

Definition at line 302 of file SessionUtils.py.

Member Data Documentation

◆ clobber

clobber

Definition at line 143 of file SessionUtils.py.

◆ max_tries

max_tries

Definition at line 141 of file SessionUtils.py.

◆ session

session

Definition at line 144 of file SessionUtils.py.

◆ timeout

timeout

Definition at line 140 of file SessionUtils.py.

◆ verbose

verbose

Definition at line 142 of file SessionUtils.py.

The documentation for this class was generated from the following file:

/gfs-oceanweb/web/ocssw/ocssw_src/src/scripts/seadasutils/SessionUtils.py

Public Member Functions

Public Attributes