Statement Design Guide

This guide is intended to help Activity Providers send properly-structured statements to Watershed so that they can take full advantage of Watershed’s report cards. 

It should be useful not only for Activity Providers working with us as part of a custom client project but also for clients working on Watershed compatibility in order to improve their product. For those working with us on client projects, we will normally work with you much more closely during your statement design. There will also often be potential for us to improve the functionality of specific cards or even create new cards as part of the project scope.

Please note: In order to use most the information in this guide, you should have a good understanding of the Experience API and the properties of statements. We recommend taking the time to explore and experiement with the developer section of experienceapi.com before reading this guide.

This guide consists of 6 main sections:

Getting Started with the Statement Design Guide

To use this guide, we recommend you first select which cards you'd like to optimize for and then design statements around the chosen cards. If you need help while doing this, please let us know.

  1. Card selection

Use this document to assess which cards are most likely to work best with which sets of data from your application. You should try to identify at least one card that will display the detailed activity data from each feature of your application and either the same or a different card type to display high level data.

It may be that you can use multiple card types with your data, but you should not expect every data set to work with every card. If multiple features of your application are tracked, the card selection decision should be made independently for each feature.

  1. Statement design

Having identified the card(s) your statements will target, use the guidelines in this document to design statements to suit these cards. You can use a statement generator to craft test statements to send to the Watershed sandbox before implementing these statements in your application. 

Data Expected by Core Watershed Cards

All cards pull human readable display text from statements to display to the user viewing the report. For this reason you should populate the verb display and activity definition name and description properties. Watershed is only available in English so “en” or “en-US” should be used.

Interactions Card

The Interactions Card simply displays interactions as a list. Aside from ensuring verb display and activity name and description fields are completed as outlined above, there are no special rules for statements displayed by this card.

Activity Card

The Activity Card provides in depth analysis of learners taking an assessment (or similar activity) and of the assessment itself.  It reads statements about assessments and statements about questions. We’ll look at each separately:

The Assessment

The Activity Card will look for any statements where the configured activity is the object and try to pull data from those statements. The following statement properties are of note:

Property

How it’s used

verb

The verb is ignored by the Activity Card. You can use any verb to convey information about the assessment.

result.duration

The Activity Card will add up the total of all durations received and count the number of times it receives a duration value to calculate an average. The total and average duration of the assessment are displayed. You should therefore only issue statements about the assessment with a non-zero duration at the end of the assessment attempt. You can issue statements with a zero duration at the start of the assessment attempt without interfering with the average.

result.score.scaled

The Activity Card will use the scaled score to show the best, last and first scores for each learner, show a percentage of attempts with scores in each bracket and show how average scores changed over time.

result.score.raw

result.score.min

result.score.max

In the event that scaled score is not provided, but raw, min and max are, then the Activity Card will make a best guess at calculating the scaled score.

If raw, min and max are not provided or their values are such that the Activity Card is not able to calculate a scaled score then no score will be recorded and the statement will not count as an attempt in terms of scores.

It is recommended that scaled score is always used.

result.success

The Activity Card will count the number of statements where success is true and display this. Statements where the success property is false are counted todays the failure total. Statements without a success property are not counted in this total.

result.completion

The Activity Card will count the number of statements where completion is true and display this. Statements where the completion property is false are counted towards the incomplete attempts total, so avoid issuing statements with a completion status of false unless you want to record a finished but incomplete attempt.

Please note:
  • Activity Providers that issue a statement at the start of each attempt with a completion status of false will cause this card to display misleading incompletion data.
  • Statements without a completion property are not counted in the total of incomplete attempts.
  • It is important to record that completion is true for both passed and failed attempts so that the difference between completions and successes can be observed in the reports.

context.registration

The registration property is used by the Activity Card to separate out attempts. Several of the reports on the card include attempt filters that show:

  • all attempts
  • best attempt
  • first attempt
  • last attempt

When any filter other than all attempts is selected, the report will only show data from one registration per user.

If no registration value is provided, the report assumes all activity has occurred within a single attempt.

Please note: If the activity provider sends multiple statements about the same user with the same registration, then either data from all of these statements will be displayed within the one attempt, or only data from the first statement will be represented (depending on the report). This can lead to confusing data. It is recommended that activity providers only send results with one statement per registration/learner combination (also note: registrations are normally unique to each learner).

To ensure that the number of attempts is the same by all measures (i.e. the number of completions matches the number of scores), all result properties should be included on a single statement at the end of the attempt.

Please note: In statements about the assessment, you should not include the assessment itself within any of  the context activities or else the assessment will be considered one of the questions.

Example statement:

{
    "timestamp": "2015-05-29T06:48:04.758Z", 
    "object": {
        "definition": {
            "type": "http://adlnet.gov/expapi/activities/assessment/", 
            "name": {
                "en-US": "Example Assessment"
            }, 
            "description": {
                "en-US": "This is a fictional, example assessment."
            }
        }, 
        "id": "http://example.com/activities/example-assessment", 
        "objectType": "Activity"
    }, 
    "actor": {
        "account": {
            "homePage": "http://example.com", 
            "name": "Neta Flemming"
        }, 
        "name": "Neta Flemming", 
        "objectType": "Agent"
    }, 
    "verb": {
        "id": "http://adlnet.gov/expapi/verbs/completed", 
        "display": {
            "en-US": "completed"
        }
    }, 
    "result": {
        "completion": true, 
        "duration": "PT0H11M43S", 
        "score": {
            "scaled": 0.6
        }, 
        "success": false
    }, 
    "context": {
        "registration": "5b9ad34c-c257-4426-83fe-27417f2e2e53"
    }
}

Assessment Questions

The Activity Card will filter all statements where the assessment appears as in any of the context activity arrays and consider these to be statements about questions of the assessment. This is true regardless of the other content of the statement, for example there is no requirement for the activity type to be “http://adlnet.gov/expapi/activities/cmi.interaction”. You should therefore take care when using an assessment activity within the context activities property of a statement.

The following statement properties are of note:

Property

How it’s used

verb

The verb is ignored by the Activity Card. You can use any verb to convey information about assessment questions.

definition.description

definition.name

The Activity Card assumes that the question text is included as the description of the question activity and will display that on reports. If a description is not provided, it will use the name instead. If neither are provided, it will display the activity id.

We recommend you use the description property.

definition.type

The type of all question activities should be "http://adlnet.gov/expapi/activities/cmi.interaction" so that you can make use of the interaction properties that tell Watershed what type of question this is and the correct response. This isn’t a requirement of the Activity Card , but of the xAPI specification itself.  

You should include all of the optional interaction activity properties defined in the interaction activities section of the specification as they contain important information for Watershed to display the question data.  

definition.correctResponsesPattern

The Activity Card will display all correct response patterns for each question on the report.

Please note: Changing the correct response pattern after statements have already been issued can cause odd behaviour and is not recommended.

result.duration

The Activity Card displays the average time to complete each question. It gets this data from the duration property of the statement about the question.

result.response

The Activity Card displays the number of times each response was given on a graph. The graph will include all available options for the question, plus any responses that were not listed as options.

result.success

The Activity Card does not mark answers and relies on the statement to tell it whether or not the question was answered correctly. Even if the response matches one of the provided correct response patterns, you must still include the success property.

Marking a question as correct will cause watershed to display the answer given as correct (by coloring it blue on the graph), even if it is not listed as a correct response pattern. You should therefore only mark answers correct if they are correct and should include them in the correct response pattern.

result.score

The Activity Card does not currently make use of score data for questions, however it is recommended that if you do score questions that you include this information in case it is displayed in a future version.

context.registration

The registration property is used by the Activity Card to separate out attempts. The question detail report include attempt filters that show:

  • all attempts
  • best attempt
  • first attempt
  • last attempt

When any filter other than all attempts is selected, the report will only show data from one registration per user.

If no registration value is provided, the report assumes all activity has occurred within a single attempt.

Please note: If the activity provider sends multiple statements about the same user with the same registration, then data from all of these statements will be displayed within the one attempt. This can be common for assessments that allow multiple tries at a question. Watershed does not currently support filtering data by try at a question within an attempt. Activity Providers might be tempted to use a different registration id for each question try in order to get the best results from Watershed, however this is not recommended as it may make the data harder to use for other purposes such as path analysis in future.

Example statement:

{
    "timestamp": "2015-07-20T18:35:48.742Z", 
    "object": {
        "definition": {
            "interactionType": "true-false", 
            "correctResponsesPattern": [
                "true"
            ], 
            "type": "http://adlnet.gov/expapi/activities/cmi.interaction", 
            "description": {
                "en-US": "Does the xAPI include the concept of statements?"
            }
        }, 
        "id": "http://example.com/activities/example-assessment/questions/true-false", 
        "objectType": "Activity"
    }, 
    "actor": {
        "account": {
            "homePage": "http://example.com", 
            "name": "Larry Lucero"
        }, 
        "objectType": "Agent"
    }, 
    "verb": {
        "id": "http://adlnet.gov/expapi/verbs/completed", 
        "display": {
            "en-US": "completed"
        }
    }, 
    "result": {
        "completion": true, 
        "duration": "PT0H0M57S", 
        "response": "false", 
        "success": false
    }, 
    "context": {
        "contextActivities": {
            "parent": [
                {
                    "definition": {
                        "type": "http://adlnet.gov/expapi/activities/assessment/", 
                        "name": {
                            "en-US": "Example Assessment"
                        }, 
                        "description": {
                            "en-US": "This is a fictional, example assessment."
                        }
                    }, 
                    "id": "http://example.com/activities/example-assessment", 
                    "objectType": "Activity"
                }
            ]
        }, 
        "registration": "2526b18a-ae98-450c-807e-80e04afcb666"
    }
}

Measure Based Cards

There are a number of cards in Watershed that make use of configurable measures. This include barchart, linechart, correlation, leaderboard and many more. These measures can be configured to work with any statement property so as long as your statement data is presented in a logical way, it should work with those cards.

Program Card

The program card is also very flexible and can work with any data set with one caveat: you should ensure that any scores and durations are not reported multiple times to avoid duplicate counting of that data.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

If you can't find what you need or you want to ask a real person a question, please contact customer support.