This article discusses how to create a Vista Sidebar Gadget using a technique called "screen scraping" to fetch data from an external web page. I'm not sure what the actual protocols are for using data from another site but at the very least I think that you should first gain the permission of the owner of the web page before doing so. If you would like to learn more about screen scraping then this Wikipedia article is a great place to start: Screen Scraping article on Wikipedia.
The code that accompanies this article can be downloaded from here.
Getting Started
To get things started create a folder named ScraperGadget and add the following things into it:
- An HTML file named ScraperGadget.html
- An HTML file named Flyout.html
- An XML manifest file named Gadget.xml
- A JavaScript file named Main.js
- A CSS file named Main.css
- A CSS file named Flyout.css
- Create a folder named "Images" and add an icon for yourself and one for the gadget.
In my images folder I have the following images for my icons:
Adding a Manifest
Open the Gadget.xml file and add the following content for your manifest definition:
<?xml version="1.0" encoding="utf-8"?>
<gadget>
<name>Dilbert Homepage Scraper Gadget</name>
<namespace>MarkItUp.Gadgets</namespace>
<version>1.0</version>
<author name="Darren Neimke">
<info url="http://MarkItUp.com" />
<logo src="images/MarkItUpIcon.jpg"/>
</author>
<copyright>MarkItUp.com 2006</copyright>
<description>
A Gadget allows a user to display content from the Dilbert home page
which is located here: http://dilbertblog.typepad.com/.
</description>
<icons>
<icon height="48" width="48" src="images/scraper.jpg"/>
</icons>
<hosts>
<host name="sidebar">
<base type="HTML" apiVersion="1.0.0" src="scrapergadget.html"/>
<permissions>Full</permissions>
<platform minPlatformVersion="1.0"/>
<defaultImage src="images/scraper.jpg"/>
</host>
</hosts>
</gadget>
To learn more about the makeup of a Gadget manifest file, view the following MSDN article:
http://msdn2.microsoft.com/en-us/library/aa965879.aspx
Defining the HTML User Interface
Finally, add some HTML into the ScraperGadget.html file to get things started:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head>
<title>Dilbert Homepage Scraper Gadget - Main</title>
<script src="main.js" language="javascript" type="text/javascript"></script>
<link href="main.css" rel="stylesheet" type="text/css" />
</head>
<body onload="initializeMain();">
<ul id="myList"></ul>
<div id="errorView"></div>
</body>
</html>
That's all that we need at this stage for the main view of our gadget which will ultimately display a list of the titles of the posts on the main page of the Dilbert website. You can see how the gadget has a reference to both the main.js file and also the main.css file and if you take a look at the body tag, you can see that we call a function named initializeMain() to get things rolling.
A pinch of CSS
To make our gadget look a little nice we can add some CSS code to spice up the UL and LI elements and alter the default font sizes to get more text into our small viewing space. Here is the code for Main.css:
body {
margin: 4px;
width: 130px;
height: 245px;
font: 9pt Calibri;
color: #000;
background-color:#cccccc;
}
a {
color: #ffffff;
text-decoration: none;
text-overflow:ellipsis;
}
a:hover {
color: red;
text-decoration: underline;
}
ul {
width: 90%;
border: 1px solid #000;
background-color: #8aa;
padding: 0px 5px ;
cursor: default;
margin-left: 0px;
}
ul li {
list-style-type: none;
margin: 0px;
position: relative;
}
ul li:hover {
background-color: #ffa;
color: #000;
}
The Javascript behavior code
The initializeMain function is responsible for grabbing the source of the page that we are scraping, breaking it down into an object model, and then displaying elements of that object model onto the page. Here is the content of the initializeMain function:
function initializeMain() {
try {
var scraper = new WebScraper() ;
gPageString = scraper.Scrape(gURL) ;
parseDocument() ;
renderDocument() ;
} catch ( ex ) {
displayError( ex.message ) ;
}
}
There's some error handling and other stuff in there but it's mostly just fetching data and formatting it on the page. You can learn about the underlying screen scraping code by reading the following article and by downloading the source files for this article and going through them:
http://markitup.com/Posts/Post.aspx?postId=96d2941f-591f-4363-b5a5-0535ef710f1b
One interesting thing about the parsing of the HTML is that we break it down into an object model before we display it - this makes the data much simpler to work with when the gadget is running. Here's the code for the Article data class that we store each article's data in:
function Article(title, body, index) {
this.Title = title ;
this.Body = body ;
this.Index = index ;
}
Each instance of this class will have a Title, Body, and Index property that we can access when we want to use the data. In the parseDocument function we actually grab each article from the page, create an Article object for each one, and store the whole lot of them in a globally-scoped Array so that we can use and access the data on-demand quite easily. The code for the parseDocument function is shown here:
function parseDocument() {
var titlePattern = "<h3 class=\"entry-header\">(([^<]|.)+?)</h3>" ;
var bodyPattern = "<div class=\"entry-body\">(([^<]|.)+?)</div>" ;
var matches = new RegexHelper().Matches( gPageString, titlePattern ) ;
var titles = new Array() ;
for( i=0; i<matches.length; i++ ) {
titles[i] = matches[i].Groups[0] ;
}
matches = new RegexHelper().Matches( gPageString, bodyPattern ) ;
for( i=0; i<matches.length; i++ ) {
var body = matches[i].Groups[0] ;
var article = new Article(titles[i], body, i) ;
gArticles[i] = article ;
}
}
Once we have our array (gArticles) of Article objects, all that remains is to render a list of their titles in our gadget - this task is performed by the renderDocument function:
function renderDocument() {
for( i=0; i<gArticles.length; i++ ) {
var article = gArticles[i] ;
var text = document.createTextNode(article.Title);
var e = document.createElement("li");
var a = document.createElement("a");
a.dataIndex = article.Index; // custom prop
a.attachEvent('onclick', handleClick);
a.appendChild(text);
e.appendChild(a);
myList.appendChild(e);
}
}
It's important to highlight that the way that this code wires up the handleClick event hander is not by any means cross-browser compatible, but it's running in the gadget runtime which is the same as IE so that's not important. The handleClick event handler is a simple function that will be invoked whenever the user clicks on a title and will display the content in a gadget Flyout window. For now simply add the following code for handleClick function in the main.js file and we'll test our gadget:
function handleClick(event){
var selectedIndex = event.srcElement.dataIndex ;
errorView.innerHTML = "You clicked on article " + selectedIndex ;
}
As you can see, this code will simply display the index of the article that was clicked on in our errorView DIV element.
An Initial Test of our Gadget
The simplest way to test the gadget is to open the ScraperGadget.html file in a browser to check that everything runs OK. Running the page and clicking on an article heading show give you a result similar to this:
Once you've got it working in the browser it's time to package up the gadget and get it running in the Vista Sidebar. Open the ScraperGadget folder and create a .zip file of all the contents. When the .zip package is created, rename it to ScraperGadget.gadget (NOTE the .gadget extension!). Upon renaming to a .gadget file, the icon should change to indicate that the package is now a gadget package. Double-click on this file to begin the installation.
Double-click on the gadget and press 'Install' when asked. After doing that the gadget should now appear in the Sidebar like so:
The Dilbert screen scraping gadget is displayed at the head of the list here - just above my custom Cricket Gadget
If you open the Sidebar gallery (by pressing the + icon at the top of the Sidebar, you will see how the gadget has used the data in the manifest file to display information about itself:
Adding the Flyout code
Now we've tested out gadget and can see that the important bits are all working as they should be. We've rendered the UI for the default view and have our object model stored nicely in memory. It's time to open that Flyout.html file and get started with creating the flyout behavior. When we've finished we'll have a flyout that will display the stories behind those headlines, here's a peek at the finished product:
The behaviour here is similar to the RSS Gadget that comes as a standard gadget with Windows Vista. In the picture you can see that the user has clicked on the first article in the list and this has then displayed the entire post content in the flyout window. The title of article is displayed at the top of the flyout and the body of the article is displayed in a scrollable DIV element. If the user clicks on the same article title in the list, the flyout will close but if they click on another title then its content will be displayed in the flyout.
Just as with the main window, the HTML for the flyout is dirt simple:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head>
<title>Dilbert Homepage Scraper Gadget - Main</title>
<script src="main.js" language="javascript" type="text/javascript"></script>
<link href="flyout.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="flyoutBackground">
<h4 id="title"></h4>
<div id="bodyContent"></div>
</div>
</body>
</html>
You can find the CSS for those elements in the download for this article.
Loading the Flyout Window
It's now time to revisit that handleClick function which receives the clicks from the links in the list and add the code for displaying the Flyout.
var gSelectedIndex = -1 ;
function handleClick(event) {
var idx = event.srcElement.dataIndex ;
if( idx == gSelectedIndex ) {
gSelectedIndex = -1 ;
System.Gadget.Flyout.show = false ;
}else{
gSelectedIndex = idx ;
if(System.Gadget.Flyout.show) {
addContentToFlyout();
} else {
System.Gadget.Flyout.show = true;
System.Gadget.Flyout.onShow = function() {
addContentToFlyout();
}
System.Gadget.Flyout.onHide = function() {
gSelectedIndex = -1 ;
}
}
}
}
Here we can see that the handleClick function gets the index of the link that was clicked and compares it against the previously clicked item in the list; if it's the same as the previous selection then we reset the selected index and hide the flyout. If the index is different to the previously selected one we check to see whether the flyout window is open and then load the content into it with our addContentToFlyout function. You can also see that whenever the flyout is shown we wire up event handlers for the onShow and onHide events.
Finally, the code for adding content to the flyout is dirt simple:
function addContentToFlyout() {
try {
if(System.Gadget.Flyout.show) {
var flyoutDoc = System.Gadget.Flyout.document;
var article = gArticles[gSelectedIndex] ;
flyoutDoc.getElementById("title").innerHTML = article.Title ;
flyoutDoc.getElementById("bodyContent").innerHTML = article.Body ;
}
} catch ( ex ) {
displayError( ex.message ) ;
}
}
We simply get a handle to the document in the Flyout window, grab the current article object from our list of articles and then bind the Title and Body properties of that object to some HTML elements in the window.
I encourage you to download the code that accompanies this article and to play around with customizing it for your own purposes.