Accessing Internet with Libwww

To start using the package, you must load it first:

    :- [libwww].

The general form of a Web call is as follows:

    :- libwww_request([request1, request2, ..., request_n]).

Each request has the following syntax:

    request_type(+URL, +RequestParams, -ResponseParams, -Result, -Status)

The request type functor must be either htmlparse, xmlparse, fetch, or header. The first two are requests to parse HTML/XML pages, respectively. Fetch is a request to bring in a page without parsing, nd header is a request to retrieve only the header information (which is returned in the ResponseParams argument--see below). The URL must be an atom or a string (list of characters).^6.1Request parameters must be either a variable (in which case the request is considered to not have special parameters) or a list. The following terms are allowed in that list:

• timeout(+Secs) - request timeout. If it is not specified, a default value (5 seconds) is used. Only the first request in a list should have the timeout value set. Timeouts that appear in subsequent requests are ignored.
• authentication([c(+Realm,+Username,+Pasword),...]) - If the site requires authentication, you should specify it in a list as an argument to the authentication/1 functor. Realm is a string that the servers return to let applications know which username/password pair to send (in case the application works with several pages that require different authentication). They are used as page identifiers for the authentication purposes. If Realm is an atom ( e.g., authentication('FooSite', boo, moo)), then when a Web server requests authentication for the FooSite realm, the Libwww package will send the foo/moo user-password pair.

  If Realm is a variable, then it is considered to match every realm. The Libwww package searches for matching authentication triples in the order they appear in the authentication list. Thus, the triple where Realm is a variable should appear last and serve as a default username/password pair.

• formdata([attval_pair1, attval_pair2, ...]) -- list of attribute/value pairs to fill out a form (in case URL is a CGI script). Each attribute/value pair must be an atom of the form attr=val.
• selection(Taglist1,Taglist2,Taglist3) -- if the request is htmlparse or xmlparse, then this term provides control over which tags to parse. Taglist1 is a list of tags that specifies inside which tags to parse. For instance, if it is [ul,form] than parsing will be done only inside these elements. Other elements will be ignored. Taglist2 tells the system to stop parsing inside the corresponding elements. For instance, [table] means that parsing should be done only inside ul and form elements. However, if we hit a table during parsing, then parsing should stop unless we hit ul or form inside the table element. This switching of parsing on and off can continue to arbitrary depth. Taglist3 is a list of tags that are to be ignored completely. That is, the parsing process will simply strip these tags (but not the text inside them). For instance, if Taglist3 is [p,i] and the page contains `` <p>foo <i>moo</i>'' then parsing will be done as if the page contained just `` foo moo''.
• if_modified_since(Date) -- fetch the document only if it has been modified after the specified date. The date must be given as an atom string in the GMT format, e.g., 'Tue, 21 Sep 1999 14:46:36 GMT'. Otherwise, the status code WWW_EXPIRED_DOC is returned.

The ResponseParams argument is a list of terms returned by the libwww_request call. It contains two kinds of information: header information and sub-request information. The header information consists of terms like: header('Content-Type', 'text/html'), header('Server', 'Netscape-Enterprise/3.6 SP2'), etc., as defined by the HTTP protocol ( header/2 is a functor and its arguments are atoms). The sub-request information consists of terms of the form: subrequest('http://www.foo.org/test/file.html',-401). It indicates that during processing of the current request, it was necessary to access another page, http://www.foo.org/test/file.html, but the server responded with the error code -401 (authentication error). Such sub-requests might be spawned during XML parsing.

The Result of a libwww_request call depends on the request type. In case of fetch it is an atom or a list of characters (depending on whether URL was specified as an atom or a list of characters), or it might be an unbound variable in case of an error. For header requests, Result is always an unbound variable.

For htmlparse and xmlparse, Result is a variable in case of an error and a complex term otherwise. In the latter case, it is a list of the form [elt1,...,elt_n], where each elt_i is of the form:

    elt(tag, [attval(attrname,value),...], [elt1',...,elt'_m])

The second argument here represents the list of attribute-value pairs. In HTML, some attributes, like checked, can be binary, in which case the corresponding value will be unbound. The third argument represents HTML or XML elements that are within the scope of tag. These elements have the same syntax as the parent element: elt(tag',attrs,sub-elements). If a tag has no attributes or if it does not have sub-elements, the corresponding lists will be empty. One special tag, pcdata, is introduced to represent pieces of text that appear in the document. This tag is our own creation--neither HTML nor XML use tags to represent text. One important difference between pcdata and other tags is that the third argument in elt(pcdata,...,...) is an atom or a list of characters, not a list (unlike other tags). If URL was specified as an atom, then the third argument of the pcdata-element is an atom as well. If URL is a character list, then so is the corresponding argument in the pcdata-element.

Finally, Status is bound to an integer that represents the return code from the HTTP request. A complete list of return codes is given in XSB/prolog_includes/http_errors.h. If you need to refer to error codes in your Prolog application, it is advisable to use symbolic notation. To make this happen, put the following lines at the top of your program:

    :- compiler_options([xpp_on]).
    #include "http_errors.h"

The Libwww package also includes a predicate that is convenient for providing English language explanations to the errors:

    :- import http_liberr/3 from usermod.

The first argument of this predicate is the error code, the second is an explanation in English, and the last is the class of the error ( e.g., internal, server error, client error, etc.). For full details see XSB/packages/libwww/http_liberr.P. Note that the code for a successful call is HT_LOADED (=200), not zero or one!