2024 π Daylatest newsbuy art
In your hiding, you're alone. Kept your treasures with my bones.Coeur de Piratecrawl somewhere bettermore quotes
very clickable
data + munging

The Perl Journal

Volumes 1–6 (1996–2002)

Code tarballs available for issues 1–21.

I reformatted the CD-ROM contents. Some things may still be a little wonky — oh, why hello there <FONT> tag. Syntax highlighting is iffy. Please report any glaring issues.

(2001) Finance::Quotelets. The Perl Journal, vol 5(5), issue #21, Fall 2001.

Finance::Quotelets


If you've read the article on Finance::Quote in TPJ (#19), then you may have downloaded the module and played with it. You may also have found it useful in keeping track of stocks you're interested in. Wouldn't it be nice if it could do also keep track of that managed fund you invested in, or tell you how much the gold bullion you've been hoarding in that sock is worth? Perhaps you're thinking of investing in camels, and want to know the market price? Maybe you'd even like to know the ping time to your favorite quake server to determine whether you should be day-trading or fragging? Unfortunately, Finance::Quote can't do those things, or can it?

Finance::Quote provides a mechanism for loading custom data-source modules at run-time. These Quotelets allow you to add arbitrary data look-up methods to Finance::Quote at run-time, which means you can keep track of all these extra things using your existing Finance::Quote applications. (Thanks go to Nathan Torkington for coining the term "Quotelet". It sounds much better than "pluggable Finance::Quote module", which I was using earlier.)

If you're not already familiar with Finance::Quote, then you might want to read the excellent introduction in TPJ (#19), or alternatively hop on over to the Finance::Quote Web site (https://finance-quote.sourceforge.net/) and follow the Documentation links from there.

What is a Quotelet?

A Quotelet is a special module that can be loaded by Finance::Quote at run-time to provide access to extra data sources. Quotelets are just like other Perl modules, although they follow a few specific conventions that allow the Finance::Quote framework to load and use them.

If you're going to be writing your own Quotelet, you'll need to make sure you follow these requirements if you want them to work properly.

In no particular order, these requirements are:

  • The module must be in the Finance::Quote namespace, for example: Finance::Quote::MyModule. The framework relies upon this when looking for modules to load.

  • The module must define a function called "methods". This registers which data sources the module provides, and the functions that provide them.

  • All functions registered by the "methods" function must conform to the standards described later in the section "Getting down to business".

  • The module must define a function called "labels". This returns information about which labels are returned. These labels are used to determine whether your Quotelet can supply specific information requested by the user.

Quotelets can do more than is required of them, including providing extra functions that are not used by Finance::Quote. This makes it possible to write modules that can also be used in different environments, or even as standalone libraries.

Requirements in Detail

The Package Name

All Quotelets must exist in the Finance::Quote namespace. Therefore, a module that fetches information from "Perl Securities" might be called "Finance::Quote::PerlSec".

The package name doesn't need to be at the top of the Finance::Quote namespace. For example, "Finance::Quote::Asia::HongKong" is a perfectly acceptable package name.

When you're developing your package, you may wish to place it in the "Finance::Quote::Quotelet::" namespace, which is guaranteed not to be used by the official Finance::Quote distribution. This makes it possible to upgrade your version of Finance::Quote without having to worry about namespace clashes.

Explaining the Methods

All Finance::Quote modules need to define a methods subroutine, which registers which methods your module provides, and how to access them. The methods subroutine takes no arguments, and returns a list of market and subroutine-ref pairs. Typically, your subroutine would look something like this:

 
 package Finance::Quote::Quotelet::MyModule;
 sub methods { return (myfunds => \&funds,
 myloans => \&loans); }

When your module is loaded, Finance::Quote calls this function and remembers the result. In this case, it knows that "myfunds" can be accessed by calling the funds() subroutine, and "myloans" can be accessed by calling the loans subroutine.

Depending upon the market you're dealing with, there may be a standard name that you should use for your function. These standard market names are used by the failover mechanism, so an alternative method can be used if the preferred method fails.

Current standard market names include:

Method-Name Source

australia Australian Stocks

canada Canadian Stocks

dwsfunds Deutsche Bank Gruppe

europe European Stocks

fidelity Fidelity Investments

nasdaq NASDAQ

nyse New York Stock Exchange

tiaacref TIAA-CREF

troweprice T. Rowe. Price

usa USA Stocks

uk_unit_trusts UK Unit Trusts

vwd Vereinigte Wirtschaftsdienste GmbH

If your market or source does not exist in the list above, then you can always post a message to finance-quote-devel@lists.sourceforge.net and ask whether a standard name has been assigned.

Method names must be all lower case, consist of alphanumeric characters (including underscore) and begin with a letter. Although this is not enforced, future versions of Finance::Quote may rely upon it.

Sometimes it would be nice to access specific look up functions directly, bypassing any failover mechanisms, rather than just specifying the market from which you want to obtain information. To this end, it is strongly recommended that you provide a unique name in addition to any standard names for accessing your functions.

# "australia" is the standard name for
# looking up stocks in Australia.  However
# people can also access our function directly
# by using the unique name of "skippy".
sub methods { return (skippy => \&funds,
             australia=> \&funds); }
Describing the Information Returned

One of the features that Finance::Quote provides is the ability for a programmer to require certain information which they want returned, without having to worry about which specific modules are called to obtain that. To facilitate this, each Quotelet needs to register which labels it will attempt to provide. A list of common labels is provided in the sidebar.

The syntax for this is simple. Each Quotelet needs to have a function that returns a list of (method, listref of labels) pairs, like this:

sub labels {
    return (myfunds => [date,yield,price,currency],
            myloans => [date,price,currency]);
}

Often, you'll find that you're returning the same list of labels. A useful shortcut for this is:

sub labels {
    my @labels = qw/date yield price name currency/;
    return (myfunds => \@labels, myloans =>\@labels);
}

The list of labels should be those that your Quotelet can be reasonably sure of obtaining from a successful lookup, as this is a way of your module promising that it will return that information if it's available and meaningful to do so. It's okay if your Quotelet doesn't return a label if it doesn't make sense for a given stock (e.g., some quotes don't return dividends), but you should not list a label here that you will never return.

There are a few conventions that are good to conform to here. All Quotelets should return a label called "price", which is traditionally the most interesting value for a given symbol. For shares, it's usually the last traded price; for funds, it's usually the yield; for indexes, it's the index value; and for loans, it's the interest rate. By having a "price" label, stock-tickers and other software can track worth without worrying about the nature of the underlying symbol.

It's also polite to return a "method" label, indicating which method returned the information. This is useful if different sites place different licenses on their information, and the calling program wants to know what it can and cannot do. It's also useful in failover situations where many different modules may have attempted to retrieve information, and you want to know which one succeeded. The value of the "method" label should be the same as the unique method-name that set it.

Changing Currencies

The Finance::Quote framework can arrange for automatic conversion between currencies. To do this, it needs to know the currency returned by your Quotelet.

If your Quotelet is returning information that is in a specific currency, it should ensure that it sets the "currency" label to a ISO 3-letter currency code. For example, if you were dealing with Australian dollars you'd use "AUD". If the given symbol does not have a currency (for example, an index such as the Dow Jones), then this label should remain blank.

Finance::Quote uses a default list of labels that are usually returned in a currency format. This works under most circumstances. However, you can explicitly set which labels have contents that should undergo currency conversion. You do this by defining a "currency_fields" sub-routine. This should return a list of labels that have a currency associated with them:

sub currency_fields {
       return qw/price nav high low bid/;
}

Typically, you'll want to make use of the default list, but maybe add one or two of your own. The currency_fields function is always passed a valid Finance::Quote object as its first argument, and the defaults list can be accessed by calling the default_currency_fields() method. As such, your currency_fields might look like this:

sub currency_fields {
        my $q = shift;
      return ($q->default_currency_fields,
             qw/commission interest/);
}

It's perfectly acceptable to return label names from currency_fields that your Quotelet will never return -- in this case they just won't be used. Likewise, it's okay to return a label twice, so you don't have to worry about some of your labels overlapping with those in the default list.

The currency conversion in Finance::Quote is pretty intelligent. It can handle not only straight integer and floating point numbers, but also fields like "53.5 - 98.4", or "5.3M". One thing to remember is that Finance::Quote has trouble using a decimal point other than a period and doesn't handle numbers with thousand separators such as "2,345".

Getting Down to Business

The functions referenced by the methods() subroutine are usually the ones that will do all the hard work, and these will be called by Finance::Quote when the information that your Quotelet can provide is needed. The following discussion applies to each function referenced by the methods() subroutine.

The first argument to your function will be a Finance::Quote object. This will be followed by a list of zero or more symbol names. This means most of your functions will probably start like this:

sub myloans {
     my ($quoter, @symbols) = @_;
     # Remainder of function...
}

The Finance::Quote object that your function is passed provides a number of useful methods that you may find useful. By far the most used method is user_agent, which returns a preconfigured LWP::UserAgent that's set up with the user's proxy, timeout, and other settings. For this reason, if you need to make a Web, FTP, gopher, or other request that an LWP::UserAgent object can handle, you should make use of the one supplied by the Finance::Quote object:

 
 sub myloans {
 my ($quoter, @symbols) = @_;
 my $ua = $quoter->user_agent;
 
 # Fetch symbols from somewhere...
 
 my $response = $ua->request(GET $someurl);
 }

Changing the settings on the user-agent will also change them for the rest of Finance::Quote, so don't do that. In most cases, there should not be a reason for you to change any settings.

The Finance::Quote object also provides a parse_csv function, which takes a string of comma-separated values and returns them as a list, and scale_field(), which takes a string and a number by which to scale, and multiplies the two in an intelligent fashion. This is the same function that gets used when we're doing currency conversion, so it does the sensible thing with fields like "103.5 - 105.1".

Your function should return a "two-dimensional hash", of the style you would normally expect to get from Finance::Quote. For example:

$hash{$symbol,"last"} = $last_price;
$hash{$symbol,"name"} = $stock_name;

It is mandatory for you to set the "success" label to a true value if your lookup for that symbol succeeded, or a false value if it failed. It is also polite to set the "errormsg" label if a particular symbol look-up failed.

Finally, you need to check whether Finance::Quote wants to have returned a hash or a hash-reference. This is almost always done like this:

 return wantarray() ? %hash : \%hash;

Putting It Together -- An Example

Let's look at a simple example. The code below shows the essential parts of a Finance::Quote module:

#!/usr/bin/perl -w
 use strict;
 
 package Finance::Quote::Quotelet::Example;
 use HTTP::Request::Common;
 
 # The "methods" function registers two modes, "australia" and
 # "my_example". Both of these call the function &myfunc.
 
 sub methods {return (australia => \&myfunc, my_example => \&myfunc)}
 
 # These are the labels my Quotelet guarantees to return.
 
 my @labels = qw/name last volume price method/;
 
 sub labels { return (australia => \@labels,
 my_example => \@labels); }

 # Here's the function that does all the hard work.
 
 sub myfunc {
 my $quoter = shift; # F::Q object.
 my @stocks = @_;
 my %info; # Stock info goes in this hash.
 
 my $EXAMPLE_URL =
 "https://www.example.oz.au/info.csv?";
 
 # Fetch my information. In this case, our pretend
 # source wants the symbols appended to the end
 # of the URL.
 
 my $ua = $quoter->user_agent;
 my $response = $ua->request(GET $EXAMPLE_URL.join("+",@stocks));
 
 # If there's an error, return an appropriate hash
 # with all failures.
 unless ($response->is_success) {
 foreach my $stock (@stocks) {
 $info{$stock,"success"} = 0;
 $info{$stock,"errormsg"} = "HTTP failure";
 }
 return wantarray() ? %info : \%info;
 }
 
 # Parse our CSV file that's been returned, and load
 # into the hash. We'll split lines up on CRLF or
 # just LF. The first element is the stock symbol.
 
 foreach my $line (split('\015?\012',$response->content)) {
 my @data = $quoter->parse_csv($line);
 my $stock = $data[0];
 
 # Set the success flag. If this is unset then
 # data may not be returned.
 $info{$stock,"success"} = 1;
 
 $info{$stock,"name"} = $data[1];
 $info{$stock,"last"} = $data[2];
 $info{$stock,"volume"} = $data[3];
 
 # Our "price" is the same as the last
 # price for the stock.
 $info{$stock,"price"} = $info{$stock,"last"};
 
 # Method is always our own unique name.
 $info{$stock,"method"} = "my_example";
 }
 
 # All done. Return the array with data loaded in it.
 
 return wantarray ? %info : \%info;
 }
 
 

Putting It Together -- A Real-Life Example

Using HTML::TableExtract

A lot of information appears on Web sites, and many Web sites use tables to present that information. While it is possible to extract this data using some clever regular expressions or HTML parsing, a module already exists which does most of the hard work for you. Matt Sisk's HTML::TableExtract lets you specify the headings in a table that you're looking for, and it will do the rest.

HTML::TableExtract is described in more detail in TPJ (#19), and is available from your local CPAN store.

BT Funds

Some friends of mine have investments with "BT Australia". BT provides information on their unit prices on their Web site at: https://btfunds.com.au/

The actual page containing the information is accessible via the rather lengthy URL: https://online.btfunds.com.au/btonline/default.asp?request=MFUnitPriceHistory&FundType=RETAIL

The page uses tables to present the information, so we'll make use of HTML::TableExtract. See Listing 1 for the Quotelet code. (Listings are available from the TPJ Web site: https://www.sysadminmag.com/tpj.)

For this Quotelet, the symbols we will pass are the full name of the funds we're interested in, so when we fetch data from "btfunds" it will probably look like this:

 
 my $q = Finance::Quote->new(qw/-defaults Quotelet::BTAust/);
 my %funds = $q->fetch("btfunds","BT Cash Management Trust",
 "BT Future Goals");

Using Your New Quotelet

To use your new Quotelet in code, simply pass its name (without the "Finance::Quote prefix) to the new method of Finance::Quote, like this:

my $quoter = Finance::Quote->new("Quotelet::BTAust");

This arranges for Finance::Quote to load and register your module. In most cases, you probably want to load the default modules as well. This is done by passing "-defaults" as the first argument to new:

my $quoter = Finance::Quote->new("-defaults","Quotelet::BTAust");

If you have an existing Finance::Quote application that does not allow you to pass in extra modules at startup, then it's also possible to set the "FQ_LOAD_QUOTELET" to a space-separated list of modules to load. If the application creates a new Finance::Quote object without passing any arguments, this environment variable will be used. This feature exists in Finance::Quote version 1.05 and later.

export FQ_LOAD_QUOTELET="-defaults Quotelet::BTAust";

Other Uses of Quotelets

While the Finance::Quote framework was written to obtain information about financial information in a robust manner, there is no reason why it cannot be used to obtain other sorts of information as well. For example, you might define a Quotelet which interprets its symbols as hostnames and returns ping-times. Alternatively, you may have a Quotelet that uses the symbols as geographical locations and returns weather information. While this is a little odd, it means that you could display both your portfolio, beach conditions, and the ping-time to your favorite Quake server all using the same ticker.

Conclusion

Now that I've explained the requirements to write your own Quotelets, I hope you feel filled with the knowledge required to add arbitrary data sources to your favorite Finance::Quote tickers and programs.

If you need help in developing your Quotelet, or just want to hang around other Finance::Quote developers, there is a link from the Finance::Quote Web page to support forums, mailing lists, bug-tracking systems, and other useful things. Most of these can be found at: https://sourceforge.net/projects/finance-quote/

If you do write your own module and think that others may find it useful, then I urge you to consider submitting it for inclusion in the Finance::Quote standard distribution. Send mail to: finance-quote-devel@lists.sourceforge.net if you have a Quotelet you'd like to make available.

Until then, you can enjoy knowing the value of that gold in your sock, the temperature outside, the price of camels, and most importantly, whether it's a good day for frags.

Paul Fenwick is a senior consultant with the Obsidian Consulting group. He has interests in cycling, homebrew, edible plants, and permaculture. He's also available for hire, and with the Australian dollar as it is, he's even quite affordable. He can be contacted at: pjf@obsidian.com.

Standard Finance::Quote Labels

These labels are commonly used by Finance::Quote. Only success and price are really required, all the rest are optional. It is perfectly okay to return labels that are not in this list.

name	Company or Mutual Fund Name
last	Last Price
high	Highest trade today
low	Lowest trade today
date	Last Trade Date (MM/DD/YY format)
time	Last Trade Time
net	Net Change
p_change	Percent Change from previous day's close
volume	Volume
avg_vol	Average Daily Vol
bid	Bid
ask	Ask
close	Previous Close
open	Today's Open
day_range	Day's Range
year_range	52-Week Range
eps	Earnings per Share
pe	P/E Ratio
div_date	Dividend Pay Date
div	Dividend per Share
div_yield	Dividend Yield
cap	Market Capitalization
ex_div	Ex-Dividend Date.
nav	Net Asset Value
yield	Yield (usually 30 day avg)

exchange	 The exchange the information was obtained from.
success	 Did the stock successfully return information? (true/false)
errormsg	 If success is false, this field may contain the reason why.
method	 The module (as could be passed to fetch) which found this information.

listing 1

(2001) Finance::Quotelets. The Perl Journal, vol 5(5), issue #21, Fall 2001.
#!/usr/bin/perl -w
use strict;
# BTAust Quotelet.
# Paul Fenwick <pjf@cpan.org>, November 2000
# Insert GPL here.
package Finance::Quote::Quotelet::BTAust;
use HTML::TableExtract;
use HTTP::Request::Common;
use Data::Dumper;
# In perl 5.6 we could use 'our', but we'll try to stay 5.005 compatible.
use vars qw/$VERSION/;
$VERSION = 1.00;
# The btfunds_direct method is our unique name.  We've used
# _direct because it gets the information directly from BT's
# website, and not through a different provider.
sub methods { return (btfunds => \&btfunds, btfunds_direct => \&bt_funds); }
# The labels we guarantee to return (if they make sense):
sub labels {
    my @labels = qw/price name date entry exit yield/;
    return (btfunds => \@labels, btfunds_direct => \@labels);
}
# Which fields should undergo currency conversion:
sub currency_fields { return qw/price entry exit/; }
# These variables don't need to be calculated every time the function
# is called, so we define them in the top scope.
my $BTFUNDS_URL = "https://online.btfunds.com.au/btonline/default.asp?request= \
MFUnitPriceHistory&FundType=RETAIL"; my %month; @month{qw/Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec/} = 1..12; # The actual function that does the hard work. sub btfunds { my ($quoter, @symbols) = @_; return unless @symbols; # No point doing work without symbols. # Drop our symbols into a hash for fast lookup later on. my %symbol_lookup; @symbol_lookup{@symbols} = (); my $ua = $quoter->user_agent; my %funds; # Where we'll store the data. my $response = $ua->request(GET $BTFUNDS_URL); # Unless we have a success in the HTTP session, then # everything is a failure. unless ($response->is_success) { foreach my $symbol (@symbols) { $funds{$symbol, "errormsg"} = "HTTP failure"; $funds{$symbol, "success"} = 0; } return wantarray() ? %funds : \%funds; } # Use HTML::TableExtract to find the information we're # looking for. my $te = HTML::TableExtract->new(headers => [ "Product/Fund Name", "As At", "Entry Price", "Exit Price"]); $te->parse($response->content); # Step through each row, placing the info into our hash # if required. foreach my $row (($te->table_states)[0]->rows) { my ($name, $date, $entry, $exit) = @$row; next unless $date; # Skips bogus rows. # Strip unprintables and undesirables. $name =~ s/[^\s\w#]/ /g; $date =~ s/[^\s\w]/-/g; # A hash on the end means this fund has a yield, # not a price. We strip the hash so we can # compare names later on. $name =~ s/\s*(#?)\s*$//; my $yield_fund = $1; # Skip the row if we're not interested. next unless exists $symbol_lookup{$name}; $funds{$name,"success"} = 1; $funds{$name,"name"} = $name; # Set ourselves as the module that found this info. $funds{$name,"method"} = "btfunds"; # Convert the date into US format. $date =~ s#(\d+)-(\w+)-(\d+)#$month{$2}/$1/$3#; $funds{$name,"date"} = $date; # Note that yield funds do not have a currency, # only a percentage. if ($yield_fund) { $entry =~ s/%$//; # Strip trailing percent $funds{$name,"yield"} = $entry; $funds{$name,"price"} = $entry; } else { $exit =~ s/^\$//; # Strip leading $'s $entry =~ s/^\$//; $funds{$name,"entry"} = $entry; $funds{$name,"exit"} = $exit; $funds{$name,"currency"} = "AUD"; $funds{$name,"price"} = $exit; } } # Anything not found is a failure... foreach my $symbol (@symbols) { unless ($funds{$symbol,"success"}) { $funds{$symbol,"success"} = 0; $funds{$symbol,"errormsg"} = "Symbol not found"; } } return wantarray() ? %funds : \%funds; }
Martin Krzywinski | contact | Canada's Michael Smith Genome Sciences CentreBC Cancer Research CenterBC CancerPHSA
Google whack “vicissitudinal corporealization”
{ 10.9.234.151 }