Testing

Table of contents

Anyone who uses Sandstorm can test out new Sandstorm apps and features! Consider it our open beta program. Join the email list: sandstorm-dev.

The feedback that you provide will help make Sandstorm and the apps better, easier to install, and more lightweight. There is room for technical feedback, usability feedback, performance feedback, and platform consistency feedback, just to name a few. Therefore truly anyone who has used any app on Sandstorm can be helpful in making other apps better, even if you have no programming background or formal technology education.

This page has three sections:

• How to find apps that need testing
• What to look for when testing a Sandstorm app
• How to deliver feedback to app packagers/authors

How to find apps that need testing

The most straightforward ways to find apps that need testing is to look in the Sandstorm App List. Install an app, and try it. You can do so on the demo server or your own.

If you're interested in testing alpha-level apps which may have bugs, but which may also have exciting functionality that you can't get anywhere else, join the sandstorm-dev email list. If you're an app author or app packager looking for people to test your new package, please send a link to sandstorm-dev for feedback!

What to look for, when testing a Sandstorm app

Make sure the app is as Sandstorm-y as possible. See the Sandstorm App Developer Handbook for tips on how great Sandstorm apps work. Particularly, make sure the app:

• Integrates with Sandstorm login.

• Has a sensible granularity for each instance, so that sharing instances is a sensible thing to do

• Avoids implementing internal access control, relying on Sandstorm for that.

Check for features in the interface which don't make sense under Sandstorm, like:

• Login buttons, since Sandstorm should automatically log the user in.

Check for HTML security issues that will eventually be disallowed:

• The app should not embed third-party resources. Note that if it does need to embed some, this suggests that Sandstorm should grow support for that third-party resource. One example of this might be Gravatar.

Check for excessive resource use:

• If you are a self-hoster, install it on your machine, click around, and run top in the background. Does the app seem like it's using more CPU than it should?

• Create some data using the app. Does it seem that the instance is using a surprisingly high amount of disk space?

Check if the app be made more Sandstorm-y by relying on platform features:

• Consider if the app could take advantage of web publishing.

Download a backup and look through the contents to see if it is storing only data that should be there.

• The backup should only contain user data. This is important for the following reasons:

  • When the app is upgraded, it only gets the content of /var -- inversely, if the app stores its own code in /var then an upgrade will have half new code, half old code.
  • The data is duplicated for every instance, wasting space. On managed hosting, this means the data is charged against the user of the app once per instance!

How to deliver feedback

Here are some important social tips to maximize the chance that people fix the issues you find.

Find the best venue, usually a Sandstorm-specific issue tracker

If you are testing an app, the best place to give feedback is an issue tracker specific to the Sandstorm package of that app. An example of such an issue tracker is the tracker for the MediaWiki package that belongs to Jason Paryani, the maintainer of the MediaWiki package in Sandstorm. You can typically find these issue trackers by:

• Going to the Sandstorm app list, and

• Finding the app you are looking for, and

• Clicking on the Code: On GitHub link, and

• Clicking on Issues on the right. If you don't see an Issues link, try to find an icon that looks like !

If you cannot find such an issue tracker, it might be OK to send the author a personal email, but note that those are easily lost.

Express gratitude

When people make a Sandstorm package, it took them time. If you show that you respect the time it took, then they will be more likely to make the changes you ask for.

Good example:

The packaging for this app is a good start, and the app itself is really pretty! Thanks for building it and taking the time to prepare a package for Sandstorm. I'm sure when the app is done, lots of people will be excited to use it.

Provide a suggestion for how to improve something

Good example:

I see that the app has a "Log In" button, even though when I use it, I am already logged in (because you took the time to support X-Sandstorm-Username). Is there a chance you can remove the "Log In" button? I think it would confuse people if it stayed in.

State the limits of your perspective -- start sentences with "I"

I find that people are more receptive to my feedback if I start the sentence with "I"! This way, they can identify which parts of my request are based on my unique aesthetic preferences and which parts are probably something they might experience. For example:

When I click "Save", the app opens a modal, and I can't find a way to dismiss the modal, so I couldn't figure out how to keep using the app. I reloaded the browser tab instead.

In the above example, there might have been a way to dismiss the modal. If you had written this (bad example):

The modal that comes up after clicking "Save" has now way to dismiss it.

then the author might remark that snapping your fingers three times dismisses it, which would presumably lead you to say that you had no way to guess that. In this case, the point is not the existence of a feature but the fact that you were unable to find it, so state your experience and the point becomes clear.

Stay casual

The sandstorm-dev email list is a pretty casual place. It's OK to use shorthand etc. so long as it's clear what you mean.