Why I like Pipe-formatted tests

Robot framework supports several different formats for its test cases. You can write tests in HTML, reStructuredText, tab-separated values (eg: export from a spreadsheet program), and plain text with either pipe-separated fields or space-separated fields.

Each format has advantages and disadvantages, mostly centered around how easy it is to write the tests vs how easy it is to read the test after it has been created. These two goals are sometimes in conflict with each other.

This article gives a very brief rundown of the advantages of each, and covers why I eventually came to embrace the pipe-separated plain text format as the best of the available formats.

HTML

HTML is my least favorite format for test cases. This is due to two reasons: it can be difficult to write the tests, and it can be difficult comparing two versions of a test.

The tests are difficult to write because HTML editors aren’t optimized for writing robot, so you’re trying to write test cases in a tool designed to create websites. The tool has a different goal than the user. Of course, you can use a plain text editor, but then you’re cursed with having to hand-edit HTML tags (though, smart editors can help ease that pain).

Because your test case data is wrapped in HTML tags, it can make the test cases hard to diff and merge simply because of all of the extra markup. Worse, in my experience, different web editors may introduce differences in the markup by adding or removing white space or line breaks. If the people in an organization use different tools, it can cause problems for comparing versions of a test case — a single line change to a test case in a different tool may end up reformatting the whole file.

reStructuredText

Using reStructuredText has an advantage similar to HTML in that you can create test documents that are very easy to read. Writing reStructured text is at least arguably easier to write than HTML since the markup doesn’t rely so much on tags as it does simple conventions, so you get the best of both words — fairly easy to write, and definitely easy to read.

What I don’t like about reStructuredText is that it’s still a bit verbose. While you can “draw” test tables that indeed look like tables, you have to draw test tables that look like tables. Since the focus of writing tests should be on the logic and data of the test, having to spend extra effort to make the tables look like tables is onerous.

There’s another option where you can use code blocks. Within a code block you can use the space-separated format. However, if you’re going to use the space-separated format, why add the extra complexity of reStructuredText on top of it?

I would be interested in hearing from someone that is successfully using reStructuredText for their tests. Maybe I can be convinced it’s the way to go.

Tab-separated values

My first exposure to robot was at a company where we settled on the tab-separated format. This is arguably the easiest to type when using a standard text editor, since it’s just a single keypress to start a new cell. The problem with tab-separated files, in my opinion, is the fact that it’s hard to distinguish between a tab and just a bunch of spaces. And, depending on your tab stops, a single tab may often look like a single space, leading to ambiguity.

You also have the same (dis?)advantage as using tabs in software — different people have different settings for tab stops. Some see this as an advantage, but often results in code that lines up and reads properly to one person but not another.

Plain text, space-separated format

The plain text, space-separated format seems to be the most popular. This is the format that the robot framework inventor uses, and many examples I see on the internet use this format. It has the advantage that test cases can read very much like English language, which definitely is one of the main appeals of using a keyword driven testing language.

The Achilles heal of this format, in my opinion, is that it leads to ambiguity. Two spaces can separate two cells in a row, but in print and on websites it’s can be difficult to visually distinguish one space from two. To work around that, the robot framework user guide suggest using four spaces, but then you lose some of the readability, plus it’s just a pain to type four spaces between each cell.

Plain text pipe-separated format

This is the format I prefer, because I think it strikes a nice balance between being easy to read and write. It has many of the same advantages as the space- and tab-separated formats, but it has one distinct advantage in that it is much less ambiguous. There’s no denying the presence of a pipe between two fields.

The downside to the pipe separated format is that pressing space-pipe-space is tedious, and also difficult to say out loud when teaching others. Another downside is that the presence of a pipe removes some of the “english-like” features of the space separated performance.

In fact, having to type a pipe is a pretty big downside. Just as I would not want to type four spaces between two cells, typing space-pipe-space is also somewhat annoying. Not only that, but on many non-US keyboards, the pipe character is particularly hard to type. If I had one of those keyboards I would definitely gravitate toward the space-separated format.

That, to me, is just about the only real downside to using the pipe-separated format. In fact, I liked the format enough, and hated typing the pipe enough, that I wrote my own editing tools to eliminate the pain.

A few years ago I wrote a desktop app that let you type a pipe simply by pressing the tab key. You then get the ease of editing that you would get with the tab-separated-value format, but with the clarity of the pipe-separated format.

This proved to be a fantastic solution. However, I was spending a lot of my time writing a basic text editor around this single feature, rather than focusing on adding even more features.

That’s when I switched to writing a robot extension for brackets. My extension has a couple of key features that I believe gives the best editing experience for robot framework tests. You have the ease of entering separators by pressing tab, the clarity that comes with pipes rather than spaces, yet the ease of reading because the pipes are rendered in a dim color, making them less dominant. And, because it is an extension to an existing editor, I can focus on adding additional features such as in-line keyword documentation, code hints,
and so on.

Summary

I prefer the pipe-separated plain text format, and use that at work, and in my blog posts and answers on stackoverflow. It’s what I recommend to anyone just getting started. But honestly, all of the supported formats are great and have their place. I think that the fact they are all plain text gives robot a lot of its appeal, because the tests play very well with development tools we already use — text editors, source code control, diff, etc.

Advertisements
Why I like Pipe-formatted tests

Brackets robot framework extension 0.9.5 is out

I’ve pushed a new version of the robot framework plugin for brackets. This version has improved coding hinting support. Code hints now behave a bit more like other editors.

There is also now some basic support for hinting variables. If you type "${" (without the quotes), a list of all variables in the current file will be presented. I’ll be improving this in future releases to include variables defined in resource files.

Another feature is a special shortcut vv — type vv at the start of a cell to automatically insert “${}“, move the cursor inside the curly braces, and bring up the list of variables. It’s a seemingly simple thing, but seems to add to the editing experience by cutting down the number of times you have to type cumbersome characters.

Finally, keyword hints (when you press control-space in a place where a keyword is likely) now includes local keywords. Prior to this change it only picked up keywords in external libraries and resource files.

For more information, see the Changelog

Brackets robot framework extension 0.9.5 is out

Brackets as an outlining tool

For years (and years and years) I’ve used emacs as my primary editing tool. I sometimes use its built-in outline mode, but more often then not I just type into a never-ending buffer named “notes.txt”.

Lately I’ve been working on an extension to the brackets text editor for the robot framework. In order to eat my own dog food I’m slowing starting to use brackets instead of emacs for my go-to text editor. Emacs is a hard habit to break!

Today I needed to do a little outlining, so I pulled up brackets and created a new file named “notes.md” (.md is an extension for the markdown format). I was pleasantly surprised to see that this works quite well as an outliner. The markdown mode provides both syntax highlighting and code folding for headers and lists. This makes it easy to collapse and expand the sections of my outline as
I work. I’m not a “power outliner”, so this is really all I need.

Brackets is really starting to grow on me. Maybe one of these days I’ll write a more comprehensive review. If you aren’t married to your text editor — or even if you are — brackets is worth checking out.

Brackets as an outlining tool

Introducing Robot Framework extension for Brackets

Looking for a good text editor for creating robot framework plain text files? I’ve been working on an extension for the fantastic brackets editor. To install, simply install the brackets editor, go to the extension manager, and search for “robot”.

Project page: https://github.com/boakley/brackets-robotframework
Project wiki: https://github.com/boakley/brackets-robotframework/wiki

Features of the extension

The robot extension for brackets provides the following features:

  • syntax highlighting
  • code folding
  • smart handling of the tab key
  • code completion (if you have robotframework-hub installed)
  • in-editor keyword documentation (again, if you have robotframework-hub installed)

What is “smart handling of the tab key”?

I recommend using the pipe-delimited plain text format for test cases. I think this is the best combination of easy-to-read and easy-to-write. Unlike the tab- and space-separated formats, the separation of cells is more evident, which I think is important.

One downside of this format is that it’s tedious typing “| |” on most lines. Even worse is a continuation line which begins with “| | ... |“.

With this extension, pressing tab on a blank line automatically inserts pipes, spaces and continuation characters intelligently, removing some of the tedium with this format.

Want to start a new test case? A single tab on a blank line gives you the opening “|” (pipe-space) sequence.

Want to add a new statement to the current test case? Two tabs will get you the leading “| |” (pipe-space-pipe-space).

Want to add a new cell at the end of the current statement? If the cursor is at the last character and you press tab, it will insert “|” to start a new cell. Pressing tab immediately again will remove the trailing “|“, start a new line, and then insert the same number of pipes and spaces as the line you were just editing.

The general idea is to try and get the tab key to “do the right thing”. I’m still refining exactly what that means, but what it has now seems to work pretty well.

Why another extension for another editor?

There are already extensions for several popular text editors. Why create another one?

If I’m honest, it’s partly due to my own hubris. I have some slightly radical ideas for where I want this extension to go, and these ideas might not mesh with existing projects. So, why not write my own?

Another factor is that brackets is based on the remarkable codemirror widget, and I had recently spent a lot of time working with codemirror at my previous job. I’m wanting to strengthen my own javascript skills (which are extremely weak), so this is a chance for me to grow, professionally.

Considering all that, it was easier for me to create a new extension rather than to fork an existing extension and become a contributor.

Integration with robotframework-hub

Another project I’ve been working on is [robotframework-hub (https://github.com/boakley/robotframework-hub/wiki). The hub is a web app and api for serving up information about keywords.

If you are running the hub on your machine, the brackets extension can use the hub API to provide autocomplete and in-editor keyword documentation. Soon you’ll also be able to search your keywords right within the editor.

Summary

My goal with this extension is to give automation engineers a high quality editing tool, so they can focus more on the test case and less on the mechanics of editing the test.

Acknowledgements

A huge thank-you to my current employer, Echo Global Logistics who generously allows me to work on this project as part of my day job.

Introducing Robot Framework extension for Brackets

I’ve been working on a new project lately, called Robot Framework
Hub. It’s a lame name, I know. Suggest a better name and maybe I’ll
use it!

The project is now to the point where it’s ready to be used by more
people than just me and my team. In invite you to take a look. If
you’re impatient and want to see some screenshots before going to the
trouble of installing it, visit the rfhub wiki here:
https://github.com/boakley/robotframework-hub/wiki

What is robot framework hub?

What is the hub (I’ll use “rfhub” from here on out)? In short, rfhub
is a light weight, zero configuration web application and api for
serving up documentation for all of the robot framework keywords on
your system.

Tired of hitting the internet for library documentation? rfhub solves
that problem. Also, you can get documentation for all of the keywords
in your local resource files and private libraries in addition to
documentation for built-in keywords and installed libraries. Rfhub
also allows you search across all your keywords at once.

Want to try it out? it’s crazy easy to get started. All you need is
python and the robot framework, and a working internet
connection. Open up a terminal or command prompt and issue these two commands:

$ pip install robotframework-hub
$ python -m rfhub

That’s it! Now you can browse to <http://localhost:7070/doc&gt; to see the
documentation for the keywords on your system. Want to use the api?
<http://localhost:7070/api/keywords&gt; will return all of the keywords as
a JSON object. You can also search by name, or get a list of keywords
for a particularly library.

If you want to tell it where to find your own keywords, just add one
or more directories as arguments. For example, if you have keywords in
/src/tests/keywords, you can run the hub like this to see the
documentation for all of the resource files in that directory:

$ python -m rfhub /src/tests/keywords

Although serving documentation is all it does now,
I have pretty big plans for the hub — I want it to have a
dashboard to control your testing activities, a way to view robot
framework documentation, and a UI for running tests and viewing test
results amoung other ideas.

What rfhub is not

There’s another robot framework documentation system out there called
rfdocs (http://www.rfdocs.org). rfhub is not a replacement for
rfdocs. It is, however, an alternative. Whereas rfdocs is an
enterprise-class tool for managing the documentation for multiple
versions of libraries, rfdoc is a tool specifically for viewing the
documentation actually installed on your workstation.

Where can I find it?

rfhub is available on github: https://github.com/boakley/robotframework-hub

There’s a bit of information on that landing page, along with links to
the source. Be sure to check out the wiki which you can get to from
that landing page.

Want to contribute?

Would you like to contribute? I would love to have help growing rfhub
into an indispensable part of your testing toolkit. rfhub is written
in python and uses the flask web application framework. For the front
end I use bootstrap.

Send me an email and we can work out the details.

Acknowledgements

A huge thank-you to my current employer, Echo Global Logistics
(http://www.echo.com) who generously allow me to work on this
project as part of my day job.

What is Robot Framework?

In my current job we’re using robot framework. My team is tasked with teaching others how to use it, and to build additional tools to make the process of test automation easier. I’m going to be writing some on the blog about this framework, so I thought I would give a very brief introduction and explain why robot is so compelling.

Robot framework is a generic keyword-driven testing framework specifically targeted at acceptance test driven development. Tests are plain text files using a simple tabular syntax. Because the test cases are plain text, you are free to use whatever tools you’re most comfortable with (Eclipse, Visual Studio,emacs, vim, Notepad++, etc).

Robot framework is written in python, is robust and mature, and under active development. There are several extensions you can use in your test cases (eg: selenium, database, etc), and there are robot-specific extensions for other popular tools (eg: jenkins, sublime text,
brackets, etc).

What do the tests look like?

There is no canonical example of a robot framework test. It supports multiple formats, including HTML, reStructuredText, plain text, and tab-separated data. Personally I prefer the plain text, pipe-delimited format. It’s very diff-and-merge friendly, easy to write and easy to read. Unlike with tab- and space-separated plain text, the division between cells in a test case are clearly visible.

Here’s an example from the robotframework.org web site, reformatted to the pipe-delimited format:

*** Settings ***
| Documentation
| ... | A test suite with a single test for valid login.
| ... |
| ... | This test has a workflow that is created using keywords in
| ... | the imported resource file.
|
| Resource | resource.txt

*** Test Cases ***
| Valid Login
| | Open Browser To Login Page
| | Input Username | demo
| | Input Password | mode
| | Submit Credentials
| | Welcome Page Should Be Open
| | [Teardown] | Close Browser

Under the *** Test Cases *** heading, each line that begins with a single pipe denotes the name of a test case. Lines that begin with pipe-space-pipe-space are statements. Typically the first word will be a keyword (either a built-in keyword or a custom keyword), and remainging cells are keyword arguments.

That’s about all there is to it — very straight-forward, easy to read and easy to write.

What can the framework be used for?

The robot framework isn’t just for testing python code. It can be used for a variety of testing tasks, including:

  • RESTful API testing
  • SOAP service testing
  • Website testing
  • Database testing
  • Desktop application testing
  • Mobile application testing (with selenium and 3rd party drivers)
  • Unit and integration testing (though, there are often better tools
    for that job)
  • Manual testing – scenarios can be written as if they were automated,
    they can prompt the user rather than perform a task.

As you can see, it can be useful for a wide variety of testing tasks. Now, it might not always be the best solution for any one of those areas but it’s one of the few tools that can be used for just about all testing tasks. This means you have a single test syntax to learn, a single reporting structure, a single way to tag and manage tests, and so on.

Other things I like about the framework:

  • Robot framework is written in python, and libraries can be written
    in python. Anything you can do with python, you can do in a test
    case. This makes it very easy to extend to do a variety of tasks.
  • Robot framework has a remote library interface, which means you can write keywords
    in any language that can open a socket (.NET, java, C, etc).
  • Robot test suites are plain utf-8 text. This means they play nicely
    with source control, making it easy to diff and merge
  • You have the choice of writing tests in a BDD (given/when/then)
    style, a procedural style, or in a data driven style. Work the way
    you want, rather than be forced into a particular style of test.
  • it has a powerful tagging mechanism
  • it has a powerful system for streaming test results to other tools
    and dashboards (called “listeners” if you want to look it up in the
    user guide).

Summary

Robot framework is a really good testing framework, if your goals are
to use a single framework for a variety of testing and if you want to
use a keyword driven approach. The fact that it’s easily extensible
with python and that it plays well with tools you probably already use makes
it a very pragmatic choice for a large variety of testing tasks.

The robot framework project has fairly comprehensive documentation. To
get started, visit http://www.robotframework.org

What is Robot Framework?