Archive

Archive for the ‘Ajax’ Category

PhantomJS A Scriptable Headless Browser

October 19, 2012 Leave a comment

October 19th, 2012 | Ken Payson

PhantomJS A Scriptable Headless Browser

More and more, modern web applications are moving away from post-back driven pages and embracing ajax intensive sites that make use of client-side view-models. While this leads to great user-experience, it raises challenges in writing tests that cover the complex functionality on the web page. Web automation tools have been around for a long time to help automate web tasks. Selenium is one of the most popular web automation frameworks. Selenium has webdrivers for all of the major browsers. By using these drivers we can automate tasks such as opening a browser, navigating to a page, filling out a form, submitting it and checking the results. This is a very usefully thing and it can be fun to watch a browser performing like a player piano; running through set of tasks without you. There is a major drawback to the current suite of web drivers though, they are slow. The time it takes to load and render pages is too slow when we have a suite of tests to run. Most of the time, the questions we are asking can be phrased as a whether a certain element is in the DOM once some other action is complete. Actually seeing things on the screen isn’t really necessary. This is doubly so when these tests are being run on a build server where we will not be watching the test run. What we want is a “headless” browser – a browser that internally does the same things that a standard browser does, makes requests, parses html, builds a DOM, understands javascript, handles cookies, and session. In shorts it behaves like a browser does except it doesn’t actually render pages.

Enter PhantomJS. PhantomJS is a headless javascript scriptable webbrowser built using the WebKit engine. There have been other attempts at headless browsers in the past. The HTML Browser remote driver with Selenium, is one example. However, earlier headless browsers did not have a proper javascript engine backing them and as a result were limited to use with very simple pages. Because PhantomJs is based on webkit, and uses the WebKit javascript engine it does not have this problem.

To get started with PhantomJS, download the latest version from PhantomJS.org Working with PhantomJS directly can be challenging because PhantomJS is rather low level. To make working with PhantomJS easy, also download CasperJS from CasperJS.org. CasperJS is a navigation scripting & testing utility written to work with PhantomJS. It enhances the PhantomJS API so the coding is easier.

Scripting with Phantom/Casper is very easy once you learn to avoid a few of the pitfalls. With Phantom/Casper you can write javascript that is injected into the webpage you are testing. Casper has a utility class for selecting and modifying elements via css selectors.

If you need, it is also be possible to use JQuery. If JQuery is not already part of the page, it can be dynamically injected and used. However, usually it is easiest to use the document.querySelector method that is natively part of the latest version of javascript.

Phantom scripts are server side javascript. We can send client side javascript to the browser. We can also do things server side that we cannot do in a browser. There is a File System module that lets us read and write to files. There is a System module that lets us work with command line arguments and environment variables.

Here is a simple example that will query google using supplied command line arguments. A report on the results will be written to a file.

Here is the PhantomJS script using Casper

phantom.casperPath = 'C:\\CasperJs\\casperjs-1.0.0-RC1';
phantom.injectJs(phantom.casperPath + '\\bin\\bootstrap.js');

var casper = require('casper').create();

var system = require('system');
var page = require('webpage').create();
var utils = require('utils');
var fs = require('fs');

var Debug = function(message) {
    casper.echo("\n" + message + "\n");
}

var googleHome= "http://www.google.com";

casper.start();

casper.userAgent('Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89');
casper.thenOpen(googleHome, function() {
    casper.echo("url: " + this.getCurrentUrl());
});

casper.then(function() {
    Debug("SearchTerm1: " + system.args[1]);
    Debug("SearchTerm2: " + system.args[2]);
});


casper.then(function() {
    casper.evaluate(function(searchTerm1,searchTerm2) {
        
        document.querySelector('input[name="q"]').setAttribute('value', searchTerm1 + ' ' + searchTerm2);
        document.querySelector('form[action="/search"]').submit(); 
    }, {
        searchTerm1: system.args[1],
        searchTerm2: system.args[2]
    });
});


casper.then(function() {
    Debug("new url: " + this.getCurrentUrl());
});


casper.then(function() {

    var secondLink = this.evaluate(function() {
        return  __utils__.findAll('h3.r a')[1].href; 
    });
    
    var numResultsOnPage = this.evaluate(function() {
        return __utils__.findAll('h3.r a').length;
    });
    
    var fstream = fs.open('C:\\temp\\searchResults.txt', 'w');
    fstream.write("There are " + secondLink + " results on the page\r\n\r\n");
    fstream.write("The second link url is " + numResultsOnPage);
    fstream.close();
    
});

casper.run(function() {
    this.exit(); 
});

PhantomJS is run from the command line: PhantomJs …

In order to stay away from the dos prompt, I usually create a one or two line batch file to run my PhantomJs script. Here is the batch file to run the example program.
cd c:\\phantomjs\\scripts
phantomjs GoogleSearch2.js stinky cheese

The future of PhantomJS with Selenium
One thing exciting development to keep an eye on is GhostDriver. GhostDriver is a Selenium Web Driver for PhantomJS. It is still in development and Selenium doesn’t fully support it, but when it is available (mostly in the next release 2.26) it will be possible to write Selenium tests in C# and have them run against PhantomJS. Initial reports say that the GhostDriver with PhantomJS could be twice as fast as the Selenium Chrome Driver.

 

David Cooksey is a Senior .NET developer at LogicBoost, an agile software services and product development company based in Washington DC.

The AJAX UpdatePanel Is it your worst enemy?

November 17, 2009 1 comment

November 17th 2009 | Kevin Jones

The AJAX UpdatePanel: Is it your worst enemy?

A recent code camp presentation I did sparked a debate about the “UpdatePanel” in AJAX. It’s no secret I believe it barely qualifies as real AJAX at all.

Most AJAX Frameworks have the UpdatePanel, and they are all similar in function, but I’m referring specifically to the one built into the Microsoft AJAX Framework. If you haven’t used AJAX before, it’s a useful tool to know and understand, but I wouldn’t recommend going much further than that.

The UpdatePanel is a control that simply designates its contents as an asynchronous callback, or “AJAX” call. Take a look at this ubiquitous example of updating a label with the servers time – sans a postback:

<asp:ScriptManager runat="server" ID="MyScriptMap" EnablePartialRendering="true" />
<asp:UpdatePanel runat="server" ID="TimeUpdatePanel">
    <ContentTemplate>
        <asp:Label runat="server" ID="TimeLabel" />
    </ContentTemplate>
    <Triggers>
        <asp:AsyncPostBackTrigger ControlID="UpdateTimeButton" EventName="Click" />
    </Triggers>
</asp:UpdatePanel>
<asp:Button runat="server" ID="UpdateTimeButton" OnClick="UpdateTimeButton_Click" Text="Update Time" />

The code behind it looks like this:

protected void Page_Load(object sender, EventArgs e) {
    SetTime();
}

private void SetTime() {
    TimeLabel.Text = DateTime.Now.ToString("G");
}

protected void UpdateTimeButton_Click(object sender, EventArgs e) {
    SetTime();
}

Pretty simple. When we click the UpdateTimeButton the label updates the time with the current time, and without causing a postback. Great, right? It’s easy and requires little knowledge of advanced AJAX functionality.

Then what’s the problem?

Firstly, it’s slow and cumbersome. Secondly, it’s not very flexible. Let’s use our trusty tool – Firebug – to see what the AJAX Framework is really doing under the covers.

Here’s the post:

MyScriptMap MyScriptMap|UpdateTimeButton
UpdateTimeButton Update Time
__ASYNCPOST True
__EVENTARGUMENT
__EVENTTARGET
__EVENTVALIDATION /wEWAgLsi+LQBwLJqc+YBE5RYQbIcM8reKtnUnkQkSFh4HRl
__VIEWSTATE /wEPDwULLTE4MzQ0ODUyNDIPZBYCAgMPZBYCAgMPZBYCZg9kF

gICAQ8PFgIeBFRleHQFFDExLzgvMjAwOSA3OjEzOjEwIFBNZG

RktDZ2TbaSW4PmU5zi/+6LeFQPeOA=

According to IIS, this is about 344 bytes. That’s a lot of data to post just to get a new date. So why did we need to post so much information?

The ViewState is the biggest part of this. ASP.NET fundamentally uses the ViewState for a whole slew of reasons, but in order for ASP.NET to process the request it needs the ViewState. There is also a field called the EventValidation. It contains information that ensures the ViewState hasn’t been tampered with. Since the EventValidation goes together with the ViewState, the AJAX call needs to send this information too. The rest of the data tells the ASP.NET runtime exactly what happened: which button was clicked; the name of the ScriptManager; etc.

That’s quite a bit of information. This is what the server sent back:

55|updatePanel|TimeUpdatePanel|<span id=”TimeLabel”>11/8/2009 7:13:35 PM</span>|128|hiddenField|__VIEWSTATE|/wEPDwULLTE4M

zQ0ODUyNDIPZBYCAgMPZBYCAgMPZBYCZg9kFgICAQ8PFgI

eBFRleHQFFDExLzgvMjAwOSA3OjEzOjM1IFBNZGRkbHZ0Gxr67

EAa/bNdqZfXC6A7YPA=|48|hiddenField|__EVENTVALIDATION|

/wEWAgLUiOrpBgLJqc+YBK3kgBaC8Pr4C9SpWuCECT5qBkBH|16|

asyncPostBackControlIDs||UpdateTimeButton|0|postBackC

ontrolIDs|||16|updatePanelIDs||tTimeUpdatePanel

|0|childUpdatePanelIDs|||15|panelsToRefreshIDs||TimeUpda

tePanel|2|asyncPostBackTimeout||90|20|formAction|

|UpdatePanelTime.aspx|

Much more than just a simple date and time. ASP.NET is sending back a lot of information in the form of a single string that is delimited by pipes, and parsing it. The total size of it as IIS is 556 bytes. After the third pipe you’ll find the content of the label we were updating – HTML and all. This is the essence of the UpdatePanel. It works by sending its content to the server and the server sends back new content. The client framework then replaces the old content with the new content.

Clever enough, but it can lead to poor performance and bad design.

Is there a different way?

Yes, and though it can be a little inconvenient you gain more in the long term.

AJAX is Asynchronous JavaScript and XML. Ironically, XML is becoming the less popular means of AJAX communication. JSON (pronounced Jason) has become the primary means of AJAX based requests. JSON is easy to work with, and more compact than XML. Here is a simple example of a key and value pair.

[
    {"Key":"The Key 1","Value":"The Value 1"},
    {"Key":"The Key 2","Value":"The Value 2"}
]

In .NET, it’s fairly easy to return JSON data from a web service so long as all of the types returned can be serialized. If we were to create a simple web service that returns the date and time, we could use JavaScript to request it from the web service. The Microsoft AJAX Framework includes a framework to make this much easier than hand coding requests from JavaScript. Here is what our web service method looks like:

[WebMethod]
[ScriptMethod(UseHttpGet = false)]
public string GetDateAndTime()
{
    return DateTime.Now.ToString("G");
}

It looks like a typical web service method, but notice the addition of the ScriptMethod attribute on the method. In order to wire AJAX into a web service, additions to the web.config file are required. Visual Studio 2008 will create a new web.config file with all of the correct settings in place to get you started with AJAX.

Calling it from our client side script is very simple:

<div>
    <asp:ScriptManager runat="server" ID="MyScriptMap" EnablePartialRendering="true">
        <Services>
            <asp:ServiceReference Path="~/DataSource.asmx" />
        </Services>
    </asp:ScriptManager>
</div>
<script language="javascript" type="text/javascript">
    function updateTime() {
        var service = new Ajax4Samples.DataSource();
        service.GetDateAndTime(iHaveDateAndTime, null, null);
    }

    function iHaveDateAndTime(theTime) {
        var myTimeSpan = $get('MyTimeSpan');
        myTimeSpan.innerHTML = theTime;
    }
</script>
<span>The Server's Time is: </span>
<span id="MyTimeSpan"><%= DateTime.Now.ToString("G") %></span>
<input type="button" onclick="updateTime();" value="Update Time" />

This is a page that updates a span—MyTimeSpan—with the response from what the web service returns. We do this by adding our web service as a ServiceReference. It generates a large piece of JavaScript that allows communication with the web service. From there, a few lines of JavaScript request the data from the service. When we click Update Time, we don’t actually send any data to the server. And why should we?

The response is as simple as this:

{“d”:”11/8/2009 8:22:26 PM”}

Just a date and a time that totals 28 bytes. The “d” is a little bit of magic that the AJAX Framework itself is using. However when you consume it you needn’t worry about the “d”. The AJAX Framework includes some metadata in responses to make things easier, such as type names, to allow for a typing system.

Sure there’s a little more coding to it but it allows a greater degree of freedom and functionality with AJAX. There are other frameworks that complement the Microsoft AJAX one, including jQuery which allows for easy manipulation of the DOM and finding controls.

My conclusion: avoid the update panel at all costs. It’s cheap and easy to implement, but it’s slow and doesn’t allow adequate flexibility.

Kevin Jones is a Team Lead at Thycotic Software, an agile software services and product development company based in Washington DC. Secret Server is our flagship password management software product. On Twitter? Follow Kevin

Categories: Ajax, ASP.NET