Lua Unit Testing: First Impressions of Busted

Lately Jeff and I have talked about unit testing the Lua code in our game. My recent work on a library we use gave me an excuse to try out Busted: a unit-testing framework for Lua. I have a list of such frameworks that I want to experiment with using and Busted was at the top, thanks to the magic of alphabetical ordering. (Sucks for you Telescope.) So today I wanted to share my initial impressions of Busted thus far.

The Good

I like the way the tests read semi-naturally. When I first posted about Luvent a reader expressed his dislike of Busted for the same reason. The framework does encourage the use of superfluous, technically unnecessary table names and such. Not only that, Busted provides enough synonyms to create good odds that you will not write a test the same way as someone else, e.g. assert.is.equal() and assert.are.equal() and assert.equal() are all exactly the same. Personally I think that flexibility benefits Busted. But I also enjoy using Perl, so I understand why some people prefer there to be only one way to do things. I don’t think Busted’s various synonyms for assertions represent an inherently superior design decision; I just enjoy the expressiveness.

Busted’s implementation of ‘spies’ has been a useful testing tool. You can wrap a spy around a function in order to test the use of that function, e.g. how many times the code called the function, what arguments it received, like so:

local foo = spy.new(realFooFunction)

--[[--
-- Now we use 'foo' the same way we would use 'realFooFunction'.  The
-- spy we created keeps track of information we can examine later in
-- the unit test.
--]]--

-- We called foo() three times.
assert.spy(foo).was.called(3)

-- We called foo(bar), giving it that argument.
assert.spy(foo).was.called_with(bar)

Spies have thus far been great for testing something like Luvent where I am passing around a lot of callbacks and want to make sure they’re all used properly.

Finally, I like Busted’s various output options, and its default output style. My experience with Perl has lead me to use various tools based around the Test Anything Protocol, which Busted supports. The program can also spit out results in JSON; I’ve had personal reason to use that, but I do appreciate the value in having an output format I can feed easily into custom utilities if I ever have to create any. Maybe it is obvious at this point, but I like having options.

The Bad

Busted’s documentation about the actual assertions it provides is a useless, infinitely recursive loop of hyperlinks. The Busted website says it uses luassert and directs you there for information about the types of assertions you can write in unit tests. The site for luassert points you back towards the Busted page for examples. To get a list of the available assertions I had to resort to reading the damned source code (scroll to the bottom).

Busted allows you to ‘tag’ tests. From the command-line you can choose to only run tests that have certain tags. That is nice, but I found myself wanting the inverse option: running all tests except those with certain tags. For example, I have a few tests which intentionally pause execution for a few seconds so that I can make sure some clock-related functionality of my library operates properly. It would be great if I could tag those tests and then tell Busted to skip them unless I explicitly say to run them.

(Update 2 September 2013): Busted provides the --exclude-tags option for skipping tests based on tags as described above.

You define individual tests with the it() function. However, if you do not plan to write the test immediately you can use pending() instead, effectively leaving yourself a note to come back and fill that test in later. However, Busted accepts empty it() tests and reports them as successful tests, e.g.:

describe("How empty it()'s succeed", function ()
    it("Passes as a successful test", function () end)
end)

If you run Busted on the above it will report one successful test even though the it() clearly does nothing. That feels like an odd design choice to me. I would prefer it if Busted reported the above as an error and forced me to use pending() for the tests I mean to come back and finish later.

Overall

Right now I feel like the pros of Busted out-weigh its cons. That said, I do still intend to try out other Lua unit-testing libraries and frameworks when I get the chance. There’s more that one testing methodology out there so I want to experiment with other options. In mean time though I would still recommend Busted to Lua developers because I have a very positive overall opinion of its quality and usefulness.

Advertisements

3 thoughts on “Lua Unit Testing: First Impressions of Busted

  1. Hey! Jack here (@ajacksified). This is an awesome post and makes it really clear that we need to improve our assertions documentation; I’ll get on that.

    We’ve had some discussion about how to handle empty `it`s as well.

    Please feel free to leave any feedback in the form of issues at https://github.com/Olivine-Labs/busted/issues as well, so they don’t go forgotten! I’ve already added two issues that I hope cover what you’ve mentioned.

  2. Hey Jack, thanks for reading the post. It’s been a month and I have looked at some alternatives, but nothing so far has had features or a design that compelled me to switch away from using Busted; so I probably will end up making more pull-requests and such as I continue using it, heh.

    And seeing a program in active development with an active community of developers and users always gives me a lot more confidence in that program over others, which I should have stressed in the article, as that is something I really like about Busted.

  3. Thanks for the post, Eric! I don’t know if you can help me with this question. Perhaps it’s better suited for Jack. However, I couldn’t find any way to reach out to the authors of Busted through their website.

    I’m looking to use Busted for my project. Is there any way that I could include Busted (and all of it’s dependencies) in my repo such that users of my repo don’t have to install anything on their machines? Currently, it seems that the only way to use Busted is to install it via luarocks. The requirement to not install anything is especially important since we have a server farm that builds our project. The main platforms I’d want to target are OS X and Linux.

    I’d greatly appreciate any help.

    Edit: For the sake of posterity, my question above can be found on the Busted forum.

Add Your Thoughts

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s