diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 0f6dfc7..3b60683 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.9.3","generation_timestamp":"2023-09-28T10:00:13","documenter_version":"1.0.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.9.3","generation_timestamp":"2023-09-29T12:26:40","documenter_version":"1.0.0"}} \ No newline at end of file diff --git a/dev/api/index.html b/dev/api/index.html index 1ddaafe..5b911aa 100644 --- a/dev/api/index.html +++ b/dev/api/index.html @@ -1,2 +1,2 @@ -API reference · ComputerAdaptiveTesting.jl
GitHub

API reference

ComputerAdaptiveTesting.Aggregators — Module

This module takes care of integrating and optimizing over the ability/difficulty space. It includes TrackedResponses, which can store cumulative results during a test.

source
ComputerAdaptiveTesting.NextItemRules — Module

This module implements the next item selection rules, which form the main part of CAT.

Bibliography

[1] Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. In Elements of adaptive testing (pp. 3-30). Springer, New York, NY.

source
ComputerAdaptiveTesting.NextItemRules.AbilityVarianceStateCriterion — Type
struct AbilityVarianceStateCriterion{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.NextItemRules.StateCriterion

  • dist_est::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

  • integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

This StateCriterion returns the variance of the ability estimate given a set of responses.

source
ComputerAdaptiveTesting.NextItemRules.DistributionExpectationBasedItemCriterion — Type
struct DistributionExpectationBasedItemCriterion{DistributionAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, AbilityIntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator, StateCriterionT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion

  • ability_estimator::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

  • integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

  • state_criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion

This ItemCriterion wraps a StateCriterion and looks at its expected value for a particular item 1-ply ahead by integrating over an ability curve.

source
ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule — Type
struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

  • strategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

  • criterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion

ItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.

ItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch1Ply when no NextItemStrategy is given.

source
ComputerAdaptiveTesting.NextItemRules.NextItemRule — Type
abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type for all item selection rules. All descendants of this type are expected to implement the interface (rule::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int

NextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule.

source
ComputerAdaptiveTesting.NextItemRules.PointExpectationBasedItemCriterion — Type
struct PointExpectationBasedItemCriterion{PointAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator, StateCriterionT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion

  • ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

  • state_criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion

This ItemCriterion wraps a StateCriterion and looks at its expected value for a particular item 1-ply ahead based on a point ability estimate.

source
ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule — Type
struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule
  • rng::Random.AbstractRNG: Default: Xoshiro()

This is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.

source
ComputerAdaptiveTesting.NextItemRules.UrryItemCriterion — Type
struct UrryItemCriterion{AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

This item criterion just picks the item with the raw difficulty closest to the current ability estimate.

source
ComputerAdaptiveTesting.CatConfig.CatRules — Type
struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase
  • next_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.
  • termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.
  • ability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()

Configuration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.

This may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.

CatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)

Explicit constructor for CatRules.

CatRules(bits...)

Implicit constructor for CatRules.

source
CATPlots.CATPlots — Module

This module contains helpers for creating CAT/IRT related plots. This module requires the optional depedencies AlgebraOfGraphics, DataFrames and Makie to be installed.

source
+API reference · ComputerAdaptiveTesting.jl
GitHub

API reference

ComputerAdaptiveTesting.Aggregators — Module

This module takes care of integrating and optimizing over the ability/difficulty space. It includes TrackedResponses, which can store cumulative results during a test.

source
ComputerAdaptiveTesting.NextItemRules — Module

This module implements the next item selection rules, which form the main part of CAT.

Bibliography

[1] Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. In Elements of adaptive testing (pp. 3-30). Springer, New York, NY.

source
ComputerAdaptiveTesting.NextItemRules.AbilityVarianceStateCriterion — Type
struct AbilityVarianceStateCriterion{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.NextItemRules.StateCriterion

  • dist_est::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

  • integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

This StateCriterion returns the variance of the ability estimate given a set of responses.

source
ComputerAdaptiveTesting.NextItemRules.DistributionExpectationBasedItemCriterion — Type
struct DistributionExpectationBasedItemCriterion{DistributionAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, AbilityIntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator, StateCriterionT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion

  • ability_estimator::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator

  • integrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator

  • state_criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion

This ItemCriterion wraps a StateCriterion and looks at its expected value for a particular item 1-ply ahead by integrating over an ability curve.

source
ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule — Type
struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

  • strategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

  • criterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion

ItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.

ItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch1Ply when no NextItemStrategy is given.

source
ComputerAdaptiveTesting.NextItemRules.NextItemRule — Type
abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type for all item selection rules. All descendants of this type are expected to implement the interface (rule::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int

NextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule.

source
ComputerAdaptiveTesting.NextItemRules.PointExpectationBasedItemCriterion — Type
struct PointExpectationBasedItemCriterion{PointAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator, StateCriterionT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion

  • ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

  • state_criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion

This ItemCriterion wraps a StateCriterion and looks at its expected value for a particular item 1-ply ahead based on a point ability estimate.

source
ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule — Type
struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule
  • rng::Random.AbstractRNG: Default: Xoshiro()

This is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.

source
ComputerAdaptiveTesting.NextItemRules.UrryItemCriterion — Type
struct UrryItemCriterion{AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator

This item criterion just picks the item with the raw difficulty closest to the current ability estimate.

source
ComputerAdaptiveTesting.CatConfig.CatRules — Type
struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase
  • next_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.
  • termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.
  • ability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()

Configuration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.

This may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.

CatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)

Explicit constructor for CatRules.

CatRules(bits...)

Implicit constructor for CatRules.

source
CATPlots.CATPlots — Module

This module contains helpers for creating CAT/IRT related plots. This module requires the optional depedencies AlgebraOfGraphics, DataFrames and Makie to be installed.

source
diff --git a/dev/contributing/index.html b/dev/contributing/index.html index b758867..6de03d3 100644 --- a/dev/contributing/index.html +++ b/dev/contributing/index.html @@ -1,2 +1,2 @@ -Contributing · ComputerAdaptiveTesting.jl
GitHub

Contributing

Contributions are welcome. Please discuss any larger changes in the issues before making a pull request to avoid wasted work.

Running tests

You can run tests locally like so:

julia --project=test test/runtests.jl

Unfortunately, Pkg.test() does not work properly at the moment. See this issue.

+Contributing · ComputerAdaptiveTesting.jl
GitHub

Contributing

Contributions are welcome. Please discuss any larger changes in the issues before making a pull request to avoid wasted work.

Running tests

You can run tests locally like so:

julia --project=test test/runtests.jl

Unfortunately, Pkg.test() does not work properly at the moment. See this issue.

diff --git a/dev/creating_a_cat/index.html b/dev/creating_a_cat/index.html index 39c912b..2b7abf1 100644 --- a/dev/creating_a_cat/index.html +++ b/dev/creating_a_cat/index.html @@ -1,5 +1,5 @@ -Creating a CAT · ComputerAdaptiveTesting.jl
GitHub

Creating a CAT

This guide gives a brief overview of how to create a CAT using the configuration structs in ComputerAdaptiveTesting.jl.

API design

The configuration of a CAT is built up as a tree of configuration structs. These structs are all subtypes of CatConfigBase.

The constructors for the configuration structs in this package tend to have smart defaults. In general most constructors have two forms. The first is an explicit keyword constructor form, where all arguments are given:

ConfigurationObject(
+Creating a CAT · ComputerAdaptiveTesting.jl
GitHub
Creating a CAT
This guide gives a brief overview of how to create a CAT using the configuration structs in ComputerAdaptiveTesting.jl.
API design
The configuration of a CAT is built up as a tree of configuration structs. These structs are all subtypes of CatConfigBase.
The constructors for the configuration structs in this package tend to have smart defaults. In general most constructors have two forms. The first is an explicit keyword constructor form, where all arguments are given:
ConfigurationObject(
   field1=value1,
   field1=value2,
-)
The second is an implicit form, where arguments are given in any order. If possible, they will be used together will appropriately guessed defaults to construct the configuration:
ConfigurationObject(value2, value1)
The implicit form is particularly useful for quick prototyping. Implicit form constructors are also available for some abstract types. In this case, they will return a concrete type that is a reasonable default for the abstract type.
After using the implicit form, you can print the object to see what values have been filled in. This may be useful in case you want to modify some of the defaults or switch to the explicit form.
Item banks
Item banks are the source of items for the test. The basic definitions are provided by FittedItemBanks.jl and can be fit to data using RIrtWrappers.jl. See the documentation pages of those packages for more information.
CatRules
This is the main type for configuring a CAT. It contains the item bank, the next item selection rule, and the stopping rule. CatRules has explicit and implicit constructors.
ComputerAdaptiveTesting.CatConfig.CatRules — Type
struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase
  • next_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.
  • termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.
  • ability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level.  Default: NullAbilityTracker()
Configuration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.
This may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.
CatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)
Explicit constructor for CatRules.
CatRules(bits...)
Implicit constructor for CatRules.
source
Next item selection with NextItemRule
The next item selection rule is the most important part of the CAT. Each rule extends the NextItemRule abstract type.
ComputerAdaptiveTesting.NextItemRules.NextItemRule — Type
abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase
Abstract base type for all item selection rules. All descendants of this type are expected to implement the interface (rule::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int
NextItemRule(bits...; ability_estimator=nothing, parallel=true)
Implicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule.
source
A sort of null hypothesis next item selection rule is RandomNextItemRule, which 
ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule — Type
struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule
  • rng::Random.AbstractRNG: Default: Xoshiro()
This is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.
source
Other rules are created by combining a ItemCriterion – which somehow rates items according to how good they are – with a NextItemStrategy using an ItemStrategyNextItemRule, which acts as an adapter. The default NextItemStrategy (and currently only) is ExhaustiveSearch1Ply. When using the implicit constructors, ItemCriterion can therefore be used directly without wrapping in any place an NextItemRule is expected.
ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule — Type
struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule
  • strategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy
  • criterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion
ItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.
ItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)
Implicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch1Ply when no NextItemStrategy is given.
source
Evaluating item and state merit with ItemCriterion and StateCriterion
The ItemCriterion abstract type is used to rate items according to how good they are as a candidate for the next item. A typical example is InformationItemCriterion, which using the current ability estimate $\theta$ and the item response function irf to calculate each item's information $\frac{irf_θ'^2}{irf_θ * (1 - irf_θ)}$.
Within this, there are two types of ExpectationBasedItemCriterion which act as adapters: PointExpectationBasedItemCriterion and DistributionExpectationBasedItemCriterion. These take a StateCriterion, which evaluates how good a particular state is in terms getting a good estimate of the test takers ability. They look one ply ahead to get the expected value of the $StateCriterion$ after selecting the given item. The AbilityVarianceStateCriterion looks at the variance of the ability $\theta$ estimate at that state.
Stopping rules with TerminationCondition
Currently the only TerminationCondition is FixedItemsTerminationCondition, which ends the test after a fixed number of items.

+)

The second is an implicit form, where arguments are given in any order. If possible, they will be used together will appropriately guessed defaults to construct the configuration:

ConfigurationObject(value2, value1)

The implicit form is particularly useful for quick prototyping. Implicit form constructors are also available for some abstract types. In this case, they will return a concrete type that is a reasonable default for the abstract type.

After using the implicit form, you can print the object to see what values have been filled in. This may be useful in case you want to modify some of the defaults or switch to the explicit form.

Item banks

Item banks are the source of items for the test. The basic definitions are provided by FittedItemBanks.jl and can be fit to data using RIrtWrappers.jl. See the documentation pages of those packages for more information.

CatRules

This is the main type for configuring a CAT. It contains the item bank, the next item selection rule, and the stopping rule. CatRules has explicit and implicit constructors.

ComputerAdaptiveTesting.CatConfig.CatRules — Type
struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase
  • next_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.
  • termination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.
  • ability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.
  • ability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()

Configuration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.

This may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.

CatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)

Explicit constructor for CatRules.

CatRules(bits...)

Implicit constructor for CatRules.

source

Next item selection with NextItemRule

The next item selection rule is the most important part of the CAT. Each rule extends the NextItemRule abstract type.

ComputerAdaptiveTesting.NextItemRules.NextItemRule — Type
abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase

Abstract base type for all item selection rules. All descendants of this type are expected to implement the interface (rule::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int

NextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule.

source

A sort of null hypothesis next item selection rule is RandomNextItemRule, which

ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule — Type
struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule
  • rng::Random.AbstractRNG: Default: Xoshiro()

This is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.

source

Other rules are created by combining a ItemCriterion – which somehow rates items according to how good they are – with a NextItemStrategy using an ItemStrategyNextItemRule, which acts as an adapter. The default NextItemStrategy (and currently only) is ExhaustiveSearch1Ply. When using the implicit constructors, ItemCriterion can therefore be used directly without wrapping in any place an NextItemRule is expected.

ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule — Type
struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule

  • strategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy

  • criterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion

ItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.

ItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)

Implicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch1Ply when no NextItemStrategy is given.

source

Evaluating item and state merit with ItemCriterion and StateCriterion

The ItemCriterion abstract type is used to rate items according to how good they are as a candidate for the next item. A typical example is InformationItemCriterion, which using the current ability estimate $\theta$ and the item response function irf to calculate each item's information $\frac{irf_θ'^2}{irf_θ * (1 - irf_θ)}$.

Within this, there are two types of ExpectationBasedItemCriterion which act as adapters: PointExpectationBasedItemCriterion and DistributionExpectationBasedItemCriterion. These take a StateCriterion, which evaluates how good a particular state is in terms getting a good estimate of the test takers ability. They look one ply ahead to get the expected value of the $StateCriterion$ after selecting the given item. The AbilityVarianceStateCriterion looks at the variance of the ability $\theta$ estimate at that state.

Stopping rules with TerminationCondition

Currently the only TerminationCondition is FixedItemsTerminationCondition, which ends the test after a fixed number of items.

diff --git a/dev/examples/examples/ability_convergence_3pl/index.html b/dev/examples/examples/ability_convergence_3pl/index.html index cb77aa9..6b44e7c 100644 --- a/dev/examples/examples/ability_convergence_3pl/index.html +++ b/dev/examples/examples/ability_convergence_3pl/index.html @@ -49,4 +49,4 @@ abs_err = abs(θ - true_θ) end

Make a plot showing how the estimated value evolves during the CAT. We also plot the 'true' values used to generate the responses.

conv_lines_fig = ability_evolution_lines(recorder; abilities=abilities)
 conv_lines_fig

Make an interactive plot, showing how the distribution of the ability likelihood evolves.

conv_dist_fig = lh_evolution_interactive(recorder; abilities=abilities)
-conv_dist_fig

This page was generated using DemoCards.jl and Literate.jl.

+conv_dist_fig

This page was generated using DemoCards.jl and Literate.jl.

diff --git a/dev/examples/examples/ability_convergence_mirt/index.html b/dev/examples/examples/ability_convergence_mirt/index.html index 54e5f1d..67e04f9 100644 --- a/dev/examples/examples/ability_convergence_mirt/index.html +++ b/dev/examples/examples/ability_convergence_mirt/index.html @@ -48,4 +48,4 @@ abs_err = sum(abs.(θ .- true_θ)) @info "convergence" true_θ θ abs_err end

Make a plot showing how the estimated value converges during the CAT.

conv_lines_fig = ability_convergence_lines(recorder; abilities=abilities)
-conv_lines_fig

This page was generated using DemoCards.jl and Literate.jl.

+conv_lines_fig

This page was generated using DemoCards.jl and Literate.jl.

diff --git a/dev/examples/examples/irfs.ipynb b/dev/examples/examples/irfs.ipynb index 7001f41..83a1c49 100644 --- a/dev/examples/examples/irfs.ipynb +++ b/dev/examples/examples/irfs.ipynb @@ -42,7 +42,7 @@ " const uuid = event.data[0];\n", " const width = event.data[1];\n", " const height = event.data[2];\n", - " const iframe = document.getElementById('df3812ba-1065-4143-b730-ce2eb8ff57ee');\n", + " const iframe = document.getElementById('78e3f2db-1794-4956-8748-dee0a6fb4b02');\n", " if (iframe) {\n", " iframe.style.width = width + \"px\";\n", " iframe.style.height = height + \"px\";\n", @@ -56,7 +56,7 @@ " }\n", " register_resize_handler('http://127.0.0.1:9284')\n", "\n", - "\n" + "\n" ] }, "metadata": {}, diff --git a/dev/examples/examples/irfs/index.html b/dev/examples/examples/irfs/index.html index 6fc7944..fe69bf8 100644 --- a/dev/examples/examples/irfs/index.html +++ b/dev/examples/examples/irfs/index.html @@ -17,7 +17,7 @@ const uuid = event.data[0]; const width = event.data[1]; const height = event.data[2]; - const iframe = document.getElementById('8fed4ad0-bf7d-4da5-9373-f33067d8aeec'); + const iframe = document.getElementById('ec67b63b-d8af-410d-861d-5083881c8466'); if (iframe) { iframe.style.width = width + "px"; iframe.style.height = height + "px"; @@ -31,5 +31,5 @@ } register_resize_handler('http://127.0.0.1:9284') - -

This page was generated using DemoCards.jl and Literate.jl.

+ +

This page was generated using DemoCards.jl and Literate.jl.

diff --git a/dev/examples/examples/perf_plots.ipynb b/dev/examples/examples/perf_plots.ipynb index 5cb92e5..c7809e5 100644 --- a/dev/examples/examples/perf_plots.ipynb +++ b/dev/examples/examples/perf_plots.ipynb @@ -16,7 +16,7 @@ " const uuid = event.data[0];\n", " const width = event.data[1];\n", " const height = event.data[2];\n", - " const iframe = document.getElementById('cce16d69-f937-4711-aa18-bb4a399d5faf');\n", + " const iframe = document.getElementById('c381005f-9e8d-4909-b506-d8dd5b30c8b2');\n", " if (iframe) {\n", " iframe.style.width = width + \"px\";\n", " iframe.style.height = height + \"px\";\n", @@ -30,7 +30,7 @@ " }\n", " register_resize_handler('http://127.0.0.1:9284')\n", "\n", - "\n" + "\n" ] }, "metadata": {}, diff --git a/dev/examples/examples/perf_plots/index.html b/dev/examples/examples/perf_plots/index.html index ecbcd38..83bd034 100644 --- a/dev/examples/examples/perf_plots/index.html +++ b/dev/examples/examples/perf_plots/index.html @@ -133,7 +133,7 @@ const uuid = event.data[0]; const width = event.data[1]; const height = event.data[2]; - const iframe = document.getElementById('32cf40f5-64d2-4852-9fdc-e633adbcd527'); + const iframe = document.getElementById('2dff6e25-a150-4b33-ba83-a5eedf85e52e'); if (iframe) { iframe.style.width = width + "px"; iframe.style.height = height + "px"; @@ -147,5 +147,5 @@ } register_resize_handler('http://127.0.0.1:9284') - -

This page was generated using DemoCards.jl and Literate.jl.

+ +

This page was generated using DemoCards.jl and Literate.jl.

diff --git a/dev/examples/examples/vocab_iq/index.html b/dev/examples/examples/vocab_iq/index.html index 5fb13a8..7f5b32b 100644 --- a/dev/examples/examples/vocab_iq/index.html +++ b/dev/examples/examples/vocab_iq/index.html @@ -56,4 +56,4 @@ run_cat(loop_config, item_bank) end -run_vocab_iq_cat()

This page was generated using DemoCards.jl and Literate.jl.

+run_vocab_iq_cat()

This page was generated using DemoCards.jl and Literate.jl.

diff --git a/dev/examples/index.html b/dev/examples/index.html index ca57fb9..8db74c3 100644 --- a/dev/examples/index.html +++ b/dev/examples/index.html @@ -31,4 +31,4 @@

This example shows how to run a CAT end-to-end on real data.

- + diff --git a/dev/index.html b/dev/index.html index 265ead2..d0037b1 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,2 +1,2 @@ -Getting started · ComputerAdaptiveTesting.jl
GitHub

Getting started

This is the documentation for ComputerAdaptiveTesting. For information about the purpose of the package and its installation, please see the README.md.

Contents

  • Creating a CAT: A guide to creating a CAT using the configuration structs in ComputerAdaptiveTesting.jl
  • Using your CAT: A collection of mini topical guides about: simulating CATs, integrating into a user-facing applications, evaluating CATs, visualising CATs, and constructing decision trees from CATs
  • Examples: A collection of example code which you can run and modify to get a feel for how to use the package
  • API reference: The full API reference for the package
+Getting started · ComputerAdaptiveTesting.jl
GitHub

Getting started

This is the documentation for ComputerAdaptiveTesting. For information about the purpose of the package and its installation, please see the README.md.

Contents

  • Creating a CAT: A guide to creating a CAT using the configuration structs in ComputerAdaptiveTesting.jl
  • Using your CAT: A collection of mini topical guides about: simulating CATs, integrating into a user-facing applications, evaluating CATs, visualising CATs, and constructing decision trees from CATs
  • Examples: A collection of example code which you can run and modify to get a feel for how to use the package
  • API reference: The full API reference for the package
diff --git a/dev/search_index.js b/dev/search_index.js index 82015f3..26423c3 100644 --- a/dev/search_index.js +++ b/dev/search_index.js @@ -1,3 +1,3 @@ var documenterSearchIndex = {"docs": -[{"location":"contributing/#Contributing","page":"Contributing","title":"Contributing","text":"","category":"section"},{"location":"contributing/","page":"Contributing","title":"Contributing","text":"Contributions are welcome. Please discuss any larger changes in the issues before making a pull request to avoid wasted work.","category":"page"},{"location":"contributing/#Running-tests","page":"Contributing","title":"Running tests","text":"","category":"section"},{"location":"contributing/","page":"Contributing","title":"Contributing","text":"You can run tests locally like so:","category":"page"},{"location":"contributing/","page":"Contributing","title":"Contributing","text":"julia --project=test test/runtests.jl","category":"page"},{"location":"contributing/","page":"Contributing","title":"Contributing","text":"Unfortunately, Pkg.test() does not work properly at the moment. See this issue.","category":"page"},{"location":"using_your_cat/#Using-your-CAT","page":"Using your CAT","title":"Using your CAT","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"Now you've created your cat according to Creating a CAT, there are a number of ways you can use it. This section covers a few. See also the Examples.","category":"page"},{"location":"using_your_cat/#Simulating-CATs","page":"Using your CAT","title":"Simulating CATs","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"using_your_cat/#Integrating-into-a-user-facing-applications","page":"Using your CAT","title":"Integrating into a user-facing applications","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"using_your_cat/#Evaluating-CATs","page":"Using your CAT","title":"Evaluating CATs","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"using_your_cat/#Visualising-CATs","page":"Using your CAT","title":"Visualising CATs","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"using_your_cat/#Constructing-decision-trees-from-CATs","page":"Using your CAT","title":"Constructing decision trees from CATs","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/perf_plots.jl\"","category":"page"},{"location":"examples/examples/perf_plots/#Perf-plots","page":"Perf plots","title":"Perf plots","text":"","category":"section"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"Plots for performance of different integrators for ability estimators","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"using Makie\nimport Pkg\nusing CSV\nusing DataFrames\nusing CategoricalArrays: levels\nusing IterTools: chain\nusing ComputerAdaptiveTesting\nusing CATPlots\nusing CATPlots: LabelledToggleGrid, MenuGrid\n\nDOCS_DATA = pkgdir(ComputerAdaptiveTesting) * \"/docs/data/\"\n\n@automakie()\n\ndf = CSV.read(\n\t(DOCS_DATA * \"integrator_benchmark_results.csv\"),\n\tDataFrame;\n\ttypes=Dict(\n\t\t\"value\" => Float64,\n\t\t\"err\" => Float64,\n\t\t\"bytes\" => Int64,\n\t\t\"response\" => Int32,\n\t\t\"item_bank\" => Int32,\n\t\t\"dim\" => Int32,\n\t\t\"trial\" => Int32,\n\t\t\"time\" => Float64,\n\t\t\"gctime\" => Float64,\n\t\t\"rtol\" => Float64,\n\t)\n)\n\nfunction plot(df)\n\tfig = Figure()\n\tax = Axis(fig[1, 1])\n\n\tgroupables = [\"item_bank\", \"response\", \"trial\", \"dim\", \"integrator\"]\n\tregressables = [\"integrator\", \"dim\", \"rtol\", \"value\", \"err\", \"bytes\", \"gctime\"]\n\n\tMenu(fig, options = [\"none\", groupables...], default = \"none\")\n\ttoggles = []\n\tsliders = []\n\tslider_idxs = []\n\tmenus = []\n\tmenu_idxs = []\n\tfor (idx, groupable) in enumerate(groupables)\n\t\tpush!(toggles, (\n\t\t\tlabel = groupable,\n\t\t))\n\t\topts = sort(levels(df[!, groupable]))\n\t\tif eltype(opts) <: AbstractString\n\t\t\tpush!(slider_idxs, idx)\n\t\t\tpush!(menus, (\n\t\t\t\tlabel = groupable,\n\t\t\t\toptions = opts\n\t\t\t))\n\t\telse\n\t\t\tpush!(menu_idxs, idx)\n\t\t\tpush!(sliders, (\n\t\t\t\tlabel = groupable,\n\t\t\t\trange = opts\n\t\t\t))\n\t\tend\n\tend\n\ttgrid = LabelledToggleGrid(\n\t\tfig,\n\t\ttoggles...,\n\t\twidth = 350,\n\t\ttellheight = false\n\t)\n\tlsgrid = SliderGrid(\n\t\tfig,\n\t\tsliders...,\n\t\twidth = 350,\n\t\ttellheight = false\n\t)\n\tmgrid = MenuGrid(\n\t\tfig,\n\t\tmenus...,\n\t\twidth = 350,\n\t\ttellheight = false\n\t)\n\n\tworking_df = Observable(nothing)\n\tfunction update_working_df(_)\n\t\tdf_prime = df\n\t\tslider_menu_vals = Vector{Any}(undef, length(sliders) + length(menus))\n\t\tslider_menu_vals[slider_idxs] = [slider.value for slider in lsgrid.sliders]\n\t\tslider_menu_vals[menu_idxs] = [menu.selection for menu in mgrid.menus]\n\t\tfor (groupable, toggle, value) in zip(groupables, tgrid.toggles, slider_menu_vals)\n\t\t\tif !toggle.value\n\t\t\t\tcontinue\n\t\t\tend\n\t\t\tdf_prime = filter(working_df, groupable => value)\n\t\tend\n\t\tworking_df[] = df_prime\n\tend\n\n\tfor observable in chain(\n\t\t(t.active for t in tgrid.toggles),\n\t\t(s.value for s in lsgrid.sliders),\n\t\t(m.selection for m in mgrid.menus)\n\t)\n\t\ton(update_working_df, observable)\n\tend\n\n\tx_var_menu = Menu(fig, options = regressables)\n\ty_var_menu = Menu(fig, options = regressables)\n\tcoded_menu = Menu(fig, options = regressables)\n\n\txs = @lift($working_df !== nothing && $(x_var_menu.selection) !== nothing ? $working_df[$(x_var_menu.selection)] : nothing)\n\tys = @lift($working_df !== nothing && $(y_var_menu.selection) !== nothing ? $working_df[$(y_var_menu.selection)] : nothing)\n\n\tscatter = @lift(\n\t\t$xs !== nothing && $ys !== nothing && $(coded_menu.selection) !== nothing ?\n\t\t\tlines!(\n\t\t\t\tax,\n\t\t\t\t$xs,\n\t\t\t\t$ys;\n\t\t\t\tcolor=$(coded_menu.selection)\n\t\t\t) : nothing\n\t)\n\n\tfig\nend\n\nplot(df)","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"
\n","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/index.md\"","category":"page"},{"location":"examples/#demo-page","page":"Examples","title":"Examples and demos","text":"","category":"section"},{"location":"examples/","page":"Examples","title":"Examples","text":"You can browse through the examples below. Each example has code and outputs, as well as links to download a script to run locally or to view as a Jupyter notebook on NBViewer. Some of the examples have interactive plots, and these will not currently work unless the scripts are run locally. TODO: Add binder links to examples.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"If you would like to run the examples as Jupyter notebooks locally to see interactive plots inline with the code, you can clone the repository and run docs/build_nbs.jl like so:","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"$ git clone https://github.com/JuliaPsychometricsBazaar/ComputerAdaptiveTesting.jl.git\n$ cd ComputerAdaptiveTesting\n$ julia --project=. docs/build_nbs.jl","category":"page"},{"location":"examples/#Examples","page":"Examples","title":"Examples","text":"","category":"section"},{"location":"examples/","page":"Examples","title":"Examples","text":"
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"How abilities converge on simulated 3PL data","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"This example shows how to run a CAT based on a synthetic correct/incorrect 3PL IRT model.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"How abilities converge on simulated MIRT data","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"This example shows how to run a CAT based on a synthetic correct/incorrect MIRT model.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Item Response Functions","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Typically, the logistic c.d.f. is used as the transfer function in IRT. However, it in an IRT context, a scaled version intended to be close to a normal c.d.f. is often used. The main advantage is that this is usually faster to compute. ComputerAdaptiveTesting provides normalscaledlogistic, which is also used by default, for this purpose:","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Perf plots","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Perf plots","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Running a CAT based based on real response data","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"This example shows how to run a CAT end-to-end on real data.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"CurrentModule = ComputerAdaptiveTesting","category":"page"},{"location":"creating_a_cat/#Creating-a-CAT","page":"Creating a CAT","title":"Creating a CAT","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"This guide gives a brief overview of how to create a CAT using the configuration structs in ComputerAdaptiveTesting.jl.","category":"page"},{"location":"creating_a_cat/#API-design","page":"Creating a CAT","title":"API design","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The configuration of a CAT is built up as a tree of configuration structs. These structs are all subtypes of CatConfigBase.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.CatConfig.CatConfigBase","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.ConfigBase.CatConfigBase-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.ConfigBase.CatConfigBase","text":"abstract type CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The constructors for the configuration structs in this package tend to have smart defaults. In general most constructors have two forms. The first is an explicit keyword constructor form, where all arguments are given:","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ConfigurationObject(\n field1=value1,\n field1=value2,\n)","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The second is an implicit form, where arguments are given in any order. If possible, they will be used together will appropriately guessed defaults to construct the configuration:","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ConfigurationObject(value2, value1)","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The implicit form is particularly useful for quick prototyping. Implicit form constructors are also available for some abstract types. In this case, they will return a concrete type that is a reasonable default for the abstract type.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"After using the implicit form, you can print the object to see what values have been filled in. This may be useful in case you want to modify some of the defaults or switch to the explicit form.","category":"page"},{"location":"creating_a_cat/#Item-banks","page":"Creating a CAT","title":"Item banks","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"Item banks are the source of items for the test. The basic definitions are provided by FittedItemBanks.jl and can be fit to data using RIrtWrappers.jl. See the documentation pages of those packages for more information.","category":"page"},{"location":"creating_a_cat/#CatRules","page":"Creating a CAT","title":"CatRules","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"This is the main type for configuring a CAT. It contains the item bank, the next item selection rule, and the stopping rule. CatRules has explicit and implicit constructors.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.CatConfig.CatRules","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.CatConfig.CatRules-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.CatConfig.CatRules","text":"struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\nnext_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.\n\ntermination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.\n\nability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()\n\nConfiguration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.\n\nThis may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.\n\nCatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)\n\nExplicit constructor for CatRules.\n\nCatRules(bits...)\n\nImplicit constructor for CatRules.\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/#Next-item-selection-with-NextItemRule","page":"Creating a CAT","title":"Next item selection with NextItemRule","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The next item selection rule is the most important part of the CAT. Each rule extends the NextItemRule abstract type.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.NextItemRule","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.NextItemRule-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.NextItemRule","text":"abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\nAbstract base type for all item selection rules. All descendants of this type are expected to implement the interface (rule::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int\n\nNextItemRule(bits...; ability_estimator=nothing, parallel=true)\n\nImplicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule.\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"A sort of null hypothesis next item selection rule is RandomNextItemRule, which ","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule","text":"struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule\n\nrng::Random.AbstractRNG: Default: Xoshiro()\n\nThis is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"Other rules are created by combining a ItemCriterion – which somehow rates items according to how good they are – with a NextItemStrategy using an ItemStrategyNextItemRule, which acts as an adapter. The default NextItemStrategy (and currently only) is ExhaustiveSearch1Ply. When using the implicit constructors, ItemCriterion can therefore be used directly without wrapping in any place an NextItemRule is expected.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule","text":"struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule\n\nstrategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy\ncriterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion\n\nItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.\n\nItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)\n\nImplicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch1Ply when no NextItemStrategy is given.\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.ItemCriterion","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.ItemCriterion-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.ItemCriterion","text":"abstract type ItemCriterion <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.NextItemStrategy","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.NextItemStrategy-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.NextItemStrategy","text":"abstract type NextItemStrategy <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply","text":"struct ExhaustiveSearch1Ply <: ComputerAdaptiveTesting.NextItemRules.NextItemStrategy\n\nparallel::Bool\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/#Evaluating-item-and-state-merit-with-ItemCriterion-and-StateCriterion","page":"Creating a CAT","title":"Evaluating item and state merit with ItemCriterion and StateCriterion","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The ItemCriterion abstract type is used to rate items according to how good they are as a candidate for the next item. A typical example is InformationItemCriterion, which using the current ability estimate theta and the item response function irf to calculate each item's information fracirf_θ^2irf_θ * (1 - irf_θ).","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"Within this, there are two types of ExpectationBasedItemCriterion which act as adapters: PointExpectationBasedItemCriterion and DistributionExpectationBasedItemCriterion. These take a StateCriterion, which evaluates how good a particular state is in terms getting a good estimate of the test takers ability. They look one ply ahead to get the expected value of the StateCriterion after selecting the given item. The AbilityVarianceStateCriterion looks at the variance of the ability theta estimate at that state.","category":"page"},{"location":"creating_a_cat/#Stopping-rules-with-TerminationCondition","page":"Creating a CAT","title":"Stopping rules with TerminationCondition","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"Currently the only TerminationCondition is FixedItemsTerminationCondition, which ends the test after a fixed number of items.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.TerminationConditions.TerminationCondition","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.TerminationConditions.TerminationCondition-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.TerminationConditions.TerminationCondition","text":"abstract type TerminationCondition <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition","text":"struct FixedItemsTerminationCondition <: ComputerAdaptiveTesting.TerminationConditions.TerminationCondition\n\nnum_items::Int64\n\n\n\n\n\n","category":"type"},{"location":"api/","page":"API reference","title":"API reference","text":"CurrentModule = ComputerAdaptiveTesting","category":"page"},{"location":"api/#API-reference","page":"API reference","title":"API reference","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"","category":"page"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [ComputerAdaptiveTesting, ComputerAdaptiveTesting.Aggregators, ComputerAdaptiveTesting.Responses, ComputerAdaptiveTesting.Sim, ComputerAdaptiveTesting.TerminationConditions, ComputerAdaptiveTesting.NextItemRules, ComputerAdaptiveTesting.CatConfig]","category":"page"},{"location":"api/#ComputerAdaptiveTesting.Aggregators","page":"API reference","title":"ComputerAdaptiveTesting.Aggregators","text":"This module takes care of integrating and optimizing over the ability/difficulty space. It includes TrackedResponses, which can store cumulative results during a test.\n\n\n\n\n\n","category":"module"},{"location":"api/#ComputerAdaptiveTesting.Aggregators.EnumerationOptimizer","page":"API reference","title":"ComputerAdaptiveTesting.Aggregators.EnumerationOptimizer","text":"Argmax + max over the ability likihood given a set of responses with a given coefficient using exhaustive search.\n\nTODO: Add item bank trait for enumerable item banks.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.Aggregators.RiemannEnumerationIntegrator","page":"API reference","title":"ComputerAdaptiveTesting.Aggregators.RiemannEnumerationIntegrator","text":"In case an item bank is enumerable (e.g. GriddedItemBank), then this method integrates over the ability likihood given a set of responses with a given coefficient function using a Riemann sum (aka the rectangle rule).\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.Aggregators.maybe_tracked_ability_estimate-Tuple{ComputerAdaptiveTesting.Aggregators.TrackedResponses, Any}","page":"API reference","title":"ComputerAdaptiveTesting.Aggregators.maybe_tracked_ability_estimate","text":"This method returns a tracked point estimate if it is has the given ability estimator, otherwise it computes it using the given ability estimator.\n\n\n\n\n\n","category":"method"},{"location":"api/#ComputerAdaptiveTesting.Sim.auto_responder-Tuple{Any}","page":"API reference","title":"ComputerAdaptiveTesting.Sim.auto_responder","text":"This function constructs a next item function which automatically responds according to responses.\n\n\n\n\n\n","category":"method"},{"location":"api/#ComputerAdaptiveTesting.Sim.prompt_response-Tuple{Any, Any}","page":"API reference","title":"ComputerAdaptiveTesting.Sim.prompt_response","text":"This response callback simply prompts \n\n\n\n\n\n","category":"method"},{"location":"api/#ComputerAdaptiveTesting.Sim.run_cat-Tuple{CatLoopConfig, FittedItemBanks.AbstractItemBank}","page":"API reference","title":"ComputerAdaptiveTesting.Sim.run_cat","text":"Run a given CatLoopConfig\n\n\n\n\n\n","category":"method"},{"location":"api/#ComputerAdaptiveTesting.Sim.run_random_comparison","page":"API reference","title":"ComputerAdaptiveTesting.Sim.run_random_comparison","text":"This function compares a given CAT configuration with a CAT using a random next item selection rule.\n\n\n\n\n\n","category":"function"},{"location":"api/#ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition","page":"API reference","title":"ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition","text":"struct FixedItemsTerminationCondition <: ComputerAdaptiveTesting.TerminationConditions.TerminationCondition\n\nnum_items::Int64\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.TerminationConditions.TerminationCondition","page":"API reference","title":"ComputerAdaptiveTesting.TerminationConditions.TerminationCondition","text":"abstract type TerminationCondition <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules","text":"This module implements the next item selection rules, which form the main part of CAT.\n\nBibliography\n\n[1] Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. In Elements of adaptive testing (pp. 3-30). Springer, New York, NY.\n\n\n\n\n\n","category":"module"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.catr_next_item_aliases","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.catr_next_item_aliases","text":"This mapping provides next item rules through the same names that they are available through in the catR R package. TODO compability with mirtcat\n\n\n\n\n\n","category":"constant"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.AbilityVarianceStateCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.AbilityVarianceStateCriterion","text":"struct AbilityVarianceStateCriterion{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.NextItemRules.StateCriterion\n\ndist_est::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator\nintegrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator\n\nThis StateCriterion returns the variance of the ability estimate given a set of responses.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.DistributionExpectationBasedItemCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.DistributionExpectationBasedItemCriterion","text":"struct DistributionExpectationBasedItemCriterion{DistributionAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, AbilityIntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator, StateCriterionT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator\nintegrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator\nstate_criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion\n\nThis ItemCriterion wraps a StateCriterion and looks at its expected value for a particular item 1-ply ahead by integrating over an ability curve.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply","text":"struct ExhaustiveSearch1Ply <: ComputerAdaptiveTesting.NextItemRules.NextItemStrategy\n\nparallel::Bool\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.ItemCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.ItemCriterion","text":"abstract type ItemCriterion <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule","text":"struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule\n\nstrategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy\ncriterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion\n\nItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.\n\nItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)\n\nImplicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch1Ply when no NextItemStrategy is given.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.NextItemRule","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.NextItemRule","text":"abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\nAbstract base type for all item selection rules. All descendants of this type are expected to implement the interface (rule::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int\n\nNextItemRule(bits...; ability_estimator=nothing, parallel=true)\n\nImplicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.NextItemStrategy","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.NextItemStrategy","text":"abstract type NextItemStrategy <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.PointExpectationBasedItemCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.PointExpectationBasedItemCriterion","text":"struct PointExpectationBasedItemCriterion{PointAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator, StateCriterionT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator\nstate_criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion\n\nThis ItemCriterion wraps a StateCriterion and looks at its expected value for a particular item 1-ply ahead based on a point ability estimate.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule","text":"struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule\n\nrng::Random.AbstractRNG: Default: Xoshiro()\n\nThis is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.StateCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.StateCriterion","text":"abstract type StateCriterion <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.UrryItemCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.UrryItemCriterion","text":"struct UrryItemCriterion{AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator\n\nThis item criterion just picks the item with the raw difficulty closest to the current ability estimate.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.CatConfig.CatLoopConfig","page":"API reference","title":"ComputerAdaptiveTesting.CatConfig.CatLoopConfig","text":"Configuration for a simulatable CAT.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.CatConfig.CatRules","page":"API reference","title":"ComputerAdaptiveTesting.CatConfig.CatRules","text":"struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\nnext_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.\n\ntermination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.\n\nability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()\n\nConfiguration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.\n\nThis may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.\n\nCatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)\n\nExplicit constructor for CatRules.\n\nCatRules(bits...)\n\nImplicit constructor for CatRules.\n\n\n\n\n\n","category":"type"},{"location":"api/","page":"API reference","title":"API reference","text":"CurrentModule = CATPlots","category":"page"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [CATPlots]","category":"page"},{"location":"api/#CATPlots.CATPlots","page":"API reference","title":"CATPlots.CATPlots","text":"This module contains helpers for creating CAT/IRT related plots. This module requires the optional depedencies AlgebraOfGraphics, DataFrames and Makie to be installed.\n\n\n\n\n\n","category":"module"},{"location":"api/#CATPlots.LabelledToggleGrid","page":"API reference","title":"CATPlots.LabelledToggleGrid","text":"For information about attributes, use attribute_help(LabelledToggleGrid).\n\n\n\n\n\n","category":"type"},{"location":"api/#CATPlots.MenuGrid","page":"API reference","title":"CATPlots.MenuGrid","text":"For information about attributes, use attribute_help(MenuGrid).\n\n\n\n\n\n","category":"type"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/ability_convergence_3pl.jl\"","category":"page"},{"location":"examples/examples/ability_convergence_3pl/#ability_convergence_3pl","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"","category":"section"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"How abilities converge on simulated 3PL data","category":"page"},{"location":"examples/examples/ability_convergence_3pl/#Running-a-CAT-based-on-a-synthetic-correct/incorrect-3PL-IRT-model","page":"How abilities converge on simulated 3PL data","title":"Running a CAT based on a synthetic correct/incorrect 3PL IRT model","text":"","category":"section"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"This example shows how to run a CAT based on a synthetic correct/incorrect 3PL IRT model.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Import order is important. We put ComputerAdaptiveTesting last so we get the extra dependencies.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"using Makie\nimport Pkg\nimport Random\nusing Distributions: Normal, cdf\nusing AlgebraOfGraphics\nusing ComputerAdaptiveTesting\nusing ComputerAdaptiveTesting.Sim: auto_responder\nusing ComputerAdaptiveTesting.NextItemRules: AbilityVarianceStateCriterion\nusing ComputerAdaptiveTesting.TerminationConditions: FixedItemsTerminationCondition\nusing ComputerAdaptiveTesting.Aggregators: PriorAbilityEstimator, MeanAbilityEstimator, LikelihoodAbilityEstimator\nusing FittedItemBanks\nusing ComputerAdaptiveTesting.Responses: BooleanResponse\nimport PsychometricsBazaarBase.IntegralCoeffs\nusing PsychometricsBazaarBase.Integrators\nusing PsychometricsBazaarBase.ConstDistributions: normal_scaled_logistic\nusing CATPlots\n\n@automakie()","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Now we are read to generate our synthetic data using the supplied DummyData module. We generate an item bank with 100 items and fake responses for 3 testees.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"using FittedItemBanks.DummyData: dummy_full, std_normal, SimpleItemBankSpec, StdModel3PL\n(item_bank, abilities, responses) = dummy_full(\n Random.default_rng(42),\n SimpleItemBankSpec(StdModel3PL(), OneDimContinuousDomain(), BooleanResponse());\n num_questions=100,\n num_testees=3\n)","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Simulate a CAT for each testee and record it using CatRecorder. CatRecorder collects information which can be used to draw different types of plots.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"max_questions = 99\nintegrator = FixedGKIntegrator(-6, 6, 80)\ndist_ability_est = PriorAbilityEstimator(std_normal)\nability_estimator = MeanAbilityEstimator(dist_ability_est, integrator)\nrules = CatRules(\n ability_estimator,\n AbilityVarianceStateCriterion(dist_ability_est, integrator),\n FixedItemsTerminationCondition(max_questions)\n)\n\npoints = 500\nxs = range(-2.5, 2.5, length=points)\nraw_estimator = LikelihoodAbilityEstimator()\nrecorder = CatRecorder(xs, responses, integrator, raw_estimator, ability_estimator)\nfor testee_idx in axes(responses, 2)\n tracked_responses, θ = run_cat(\n CatLoopConfig(\n rules=rules,\n get_response=auto_responder(@view responses[:, testee_idx]),\n new_response_callback=(tracked_responses, terminating) -> recorder(tracked_responses, testee_idx, terminating),\n ),\n item_bank\n )\n true_θ = abilities[testee_idx]\n abs_err = abs(θ - true_θ)\nend","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Make a plot showing how the estimated value evolves during the CAT. We also plot the 'true' values used to generate the responses.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"conv_lines_fig = ability_evolution_lines(recorder; abilities=abilities)\nconv_lines_fig","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Make an interactive plot, showing how the distribution of the ability likelihood evolves.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"conv_dist_fig = lh_evolution_interactive(recorder; abilities=abilities)\nconv_dist_fig","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/irfs.jl\"","category":"page"},{"location":"examples/examples/irfs/#Item-Response-Functions","page":"Item Response Functions","title":"Item Response Functions","text":"","category":"section"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"Item Response Functions","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"using Makie\nimport Pkg\nusing Distributions: Normal, cdf\nusing PsychometricsBazaarBase.ConstDistributions: normal_scaled_logistic\nusing CATPlots\n\n@automakie()","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"Typically, the logistic c.d.f. is used as the transfer function in IRT. However, it in an IRT context, a scaled version intended to be close to a normal c.d.f. is often used. The main advantage is that this is usually faster to compute. ComputerAdaptiveTesting provides normalscaledlogistic, which is also used by default, for this purpose:","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"xs = -8:0.05:8\nlines(xs, cdf.(Normal(), xs))\nlines!(xs, cdf.(normal_scaled_logistic, xs))\ncurrent_figure()","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"
\n","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/vocab_iq.jl\"","category":"page"},{"location":"examples/examples/vocab_iq/#vocab_iq","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"","category":"section"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"Running a CAT based based on real response data","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"This example shows how to run a CAT end-to-end on real data.","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"First a 1-dimensional IRT model is fitted based on open response data to the vocabulary IQ test using the IRTSupport package which internally, this uses the mirt R package. Next, the model is used to administer the test interactively.","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"using Base.Filesystem\nusing ComputerAdaptiveTesting\nusing FittedItemBanks.DummyData: std_normal\nusing ComputerAdaptiveTesting.Sim\nusing ComputerAdaptiveTesting.NextItemRules\nusing ComputerAdaptiveTesting.TerminationConditions\nusing ComputerAdaptiveTesting.Aggregators\nusing FittedItemBanks\nusing FittedItemBanks: item_params\nimport PsychometricsBazaarBase.IntegralCoeffs\nusing PsychometricsBazaarBase.Integrators\nusing PsychometricsBazaarBase.Optimizers\nusing ItemResponseDatasets: prompt_readline\nusing ItemResponseDatasets.VocabIQ\nusing RIrtWrappers.Mirt\n\nfunction get_item_bank()\n fit_4pl(get_marked_df_cached(); TOL=1e-2)\nend\n\nfunction run_vocab_iq_cat()\n item_bank, labels = get_item_bank()\n integrator = FixedGKIntegrator(-6, 6, 61)\n ability_integrator = AbilityIntegrator(integrator)\n dist_ability_est = PriorAbilityEstimator(std_normal)\n optimizer = AbilityOptimizer(OneDimOptimOptimizer(-6.0, 6.0, NelderMead()))\n ability_estimator = ModeAbilityEstimator(dist_ability_est, optimizer)\n @info \"run_cat\" ability_estimator\n rules = CatRules(\n ability_estimator,\n AbilityVarianceStateCriterion(dist_ability_est, ability_integrator),\n FixedItemsTerminationCondition(45)\n )\n function get_response(response_idx, response_name)\n params = item_params(item_bank, response_idx)\n println(\"Parameters for next question: $params\")\n prompt_readline(VocabIQ.questions[response_idx])\n end\n function new_response_callback(tracked_responses, terminating)\n if tracked_responses.responses.values[end] > 0\n println(\"Correct\")\n else\n println(\"Wrong\")\n end\n ability = ability_estimator(tracked_responses)\n var = variance_given_mean(ability_integrator, dist_ability_est, tracked_responses, ability)\n println(\"Got ability estimate: $ability ± $var\")\n println(\"\")\n end\n loop_config = CatLoopConfig(\n rules=rules,\n get_response=get_response,\n new_response_callback=new_response_callback\n )\n run_cat(loop_config, item_bank)\nend\n\nrun_vocab_iq_cat()","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/ability_convergence_mirt.jl\"","category":"page"},{"location":"examples/examples/ability_convergence_mirt/#ability_convergence_mirt","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"","category":"section"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"How abilities converge on simulated MIRT data","category":"page"},{"location":"examples/examples/ability_convergence_mirt/#Running-a-CAT-based-on-a-synthetic-correct/incorrect-MIRT-model","page":"How abilities converge on simulated MIRT data","title":"Running a CAT based on a synthetic correct/incorrect MIRT model","text":"","category":"section"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"This example shows how to run a CAT based on a synthetic correct/incorrect MIRT model.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"Import order is important. We put ComputerAdaptiveTesting last so we get the extra dependencies.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"using Makie\nimport Pkg\nimport Random\nusing Distributions: Normal, cdf\nusing AlgebraOfGraphics\nusing ComputerAdaptiveTesting\nusing ComputerAdaptiveTesting.Sim: auto_responder\nusing ComputerAdaptiveTesting.NextItemRules: DRuleItemCriterion\nusing ComputerAdaptiveTesting.TerminationConditions: FixedItemsTerminationCondition\nusing ComputerAdaptiveTesting.Aggregators: PriorAbilityEstimator, MeanAbilityEstimator, LikelihoodAbilityEstimator\nusing FittedItemBanks\nimport PsychometricsBazaarBase.IntegralCoeffs\nusing PsychometricsBazaarBase.Integrators\nusing PsychometricsBazaarBase.ConstDistributions: normal_scaled_logistic\nusing CATPlots\n\n@automakie()","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"Now we are read to generate our synthetic data using the supplied DummyData module. We generate an item bank with 100 items and fake responses for 3 testees.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"dims = 2\nusing FittedItemBanks.DummyData: dummy_full, std_mv_normal, SimpleItemBankSpec, StdModel4PL\nusing ComputerAdaptiveTesting.Responses: BooleanResponse","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"TODO: pass in dims","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"(item_bank, abilities, responses) = dummy_full(\n Random.default_rng(42),\n SimpleItemBankSpec(StdModel4PL(), VectorContinuousDomain(), BooleanResponse()),\n dims;\n num_questions=10,\n num_testees=2\n)","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"Simulate a CAT for each testee and record it using CatRecorder. CatRecorder collects information which can be used to draw different types of plots.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"max_questions = 9\nintegrator = CubaIntegrator([-6.0, -6.0], [6.0, 6.0], CubaVegas(); rtol=1e-2)\nability_estimator = MeanAbilityEstimator(PriorAbilityEstimator(std_mv_normal(dims)), integrator)\nrules = CatRules(\n ability_estimator,\n DRuleItemCriterion(ability_estimator),\n FixedItemsTerminationCondition(max_questions)\n)","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"XXX: We shouldn't need to specify xs here since the distributions are not used – rework","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"points = 3\nxs = repeat(range(-2.5, 2.5, length=points)', dims, 1)\nraw_estimator = LikelihoodAbilityEstimator()\nrecorder = CatRecorder(xs, responses, integrator, raw_estimator, ability_estimator, abilities)\nfor testee_idx in axes(responses, 2)\n @debug \"Running for testee\" testee_idx\n tracked_responses, θ = run_cat(\n CatLoopConfig(\n rules=rules,\n get_response=auto_responder(@view responses[:, testee_idx]),\n new_response_callback=(tracked_responses, terminating) -> recorder(tracked_responses, testee_idx, terminating),\n ),\n item_bank\n )\n true_θ = abilities[:, testee_idx]\n abs_err = sum(abs.(θ .- true_θ))\n @info \"convergence\" true_θ θ abs_err\nend","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"Make a plot showing how the estimated value converges during the CAT.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"conv_lines_fig = ability_convergence_lines(recorder; abilities=abilities)\nconv_lines_fig","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"#Getting-started","page":"Getting started","title":"Getting started","text":"","category":"section"},{"location":"","page":"Getting started","title":"Getting started","text":"This is the documentation for ComputerAdaptiveTesting. For information about the purpose of the package and its installation, please see the README.md.","category":"page"},{"location":"#Contents","page":"Getting started","title":"Contents","text":"","category":"section"},{"location":"","page":"Getting started","title":"Getting started","text":"Creating a CAT: A guide to creating a CAT using the configuration structs in ComputerAdaptiveTesting.jl\nUsing your CAT: A collection of mini topical guides about: simulating CATs, integrating into a user-facing applications, evaluating CATs, visualising CATs, and constructing decision trees from CATs\nExamples: A collection of example code which you can run and modify to get a feel for how to use the package\nAPI reference: The full API reference for the package","category":"page"}] +[{"location":"contributing/#Contributing","page":"Contributing","title":"Contributing","text":"","category":"section"},{"location":"contributing/","page":"Contributing","title":"Contributing","text":"Contributions are welcome. Please discuss any larger changes in the issues before making a pull request to avoid wasted work.","category":"page"},{"location":"contributing/#Running-tests","page":"Contributing","title":"Running tests","text":"","category":"section"},{"location":"contributing/","page":"Contributing","title":"Contributing","text":"You can run tests locally like so:","category":"page"},{"location":"contributing/","page":"Contributing","title":"Contributing","text":"julia --project=test test/runtests.jl","category":"page"},{"location":"contributing/","page":"Contributing","title":"Contributing","text":"Unfortunately, Pkg.test() does not work properly at the moment. See this issue.","category":"page"},{"location":"using_your_cat/#Using-your-CAT","page":"Using your CAT","title":"Using your CAT","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"Now you've created your cat according to Creating a CAT, there are a number of ways you can use it. This section covers a few. See also the Examples.","category":"page"},{"location":"using_your_cat/#Simulating-CATs","page":"Using your CAT","title":"Simulating CATs","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"using_your_cat/#Integrating-into-a-user-facing-applications","page":"Using your CAT","title":"Integrating into a user-facing applications","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"using_your_cat/#Evaluating-CATs","page":"Using your CAT","title":"Evaluating CATs","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"using_your_cat/#Visualising-CATs","page":"Using your CAT","title":"Visualising CATs","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"using_your_cat/#Constructing-decision-trees-from-CATs","page":"Using your CAT","title":"Constructing decision trees from CATs","text":"","category":"section"},{"location":"using_your_cat/","page":"Using your CAT","title":"Using your CAT","text":"TODO","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/perf_plots.jl\"","category":"page"},{"location":"examples/examples/perf_plots/#Perf-plots","page":"Perf plots","title":"Perf plots","text":"","category":"section"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"Plots for performance of different integrators for ability estimators","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"using Makie\nimport Pkg\nusing CSV\nusing DataFrames\nusing CategoricalArrays: levels\nusing IterTools: chain\nusing ComputerAdaptiveTesting\nusing CATPlots\nusing CATPlots: LabelledToggleGrid, MenuGrid\n\nDOCS_DATA = pkgdir(ComputerAdaptiveTesting) * \"/docs/data/\"\n\n@automakie()\n\ndf = CSV.read(\n\t(DOCS_DATA * \"integrator_benchmark_results.csv\"),\n\tDataFrame;\n\ttypes=Dict(\n\t\t\"value\" => Float64,\n\t\t\"err\" => Float64,\n\t\t\"bytes\" => Int64,\n\t\t\"response\" => Int32,\n\t\t\"item_bank\" => Int32,\n\t\t\"dim\" => Int32,\n\t\t\"trial\" => Int32,\n\t\t\"time\" => Float64,\n\t\t\"gctime\" => Float64,\n\t\t\"rtol\" => Float64,\n\t)\n)\n\nfunction plot(df)\n\tfig = Figure()\n\tax = Axis(fig[1, 1])\n\n\tgroupables = [\"item_bank\", \"response\", \"trial\", \"dim\", \"integrator\"]\n\tregressables = [\"integrator\", \"dim\", \"rtol\", \"value\", \"err\", \"bytes\", \"gctime\"]\n\n\tMenu(fig, options = [\"none\", groupables...], default = \"none\")\n\ttoggles = []\n\tsliders = []\n\tslider_idxs = []\n\tmenus = []\n\tmenu_idxs = []\n\tfor (idx, groupable) in enumerate(groupables)\n\t\tpush!(toggles, (\n\t\t\tlabel = groupable,\n\t\t))\n\t\topts = sort(levels(df[!, groupable]))\n\t\tif eltype(opts) <: AbstractString\n\t\t\tpush!(slider_idxs, idx)\n\t\t\tpush!(menus, (\n\t\t\t\tlabel = groupable,\n\t\t\t\toptions = opts\n\t\t\t))\n\t\telse\n\t\t\tpush!(menu_idxs, idx)\n\t\t\tpush!(sliders, (\n\t\t\t\tlabel = groupable,\n\t\t\t\trange = opts\n\t\t\t))\n\t\tend\n\tend\n\ttgrid = LabelledToggleGrid(\n\t\tfig,\n\t\ttoggles...,\n\t\twidth = 350,\n\t\ttellheight = false\n\t)\n\tlsgrid = SliderGrid(\n\t\tfig,\n\t\tsliders...,\n\t\twidth = 350,\n\t\ttellheight = false\n\t)\n\tmgrid = MenuGrid(\n\t\tfig,\n\t\tmenus...,\n\t\twidth = 350,\n\t\ttellheight = false\n\t)\n\n\tworking_df = Observable(nothing)\n\tfunction update_working_df(_)\n\t\tdf_prime = df\n\t\tslider_menu_vals = Vector{Any}(undef, length(sliders) + length(menus))\n\t\tslider_menu_vals[slider_idxs] = [slider.value for slider in lsgrid.sliders]\n\t\tslider_menu_vals[menu_idxs] = [menu.selection for menu in mgrid.menus]\n\t\tfor (groupable, toggle, value) in zip(groupables, tgrid.toggles, slider_menu_vals)\n\t\t\tif !toggle.value\n\t\t\t\tcontinue\n\t\t\tend\n\t\t\tdf_prime = filter(working_df, groupable => value)\n\t\tend\n\t\tworking_df[] = df_prime\n\tend\n\n\tfor observable in chain(\n\t\t(t.active for t in tgrid.toggles),\n\t\t(s.value for s in lsgrid.sliders),\n\t\t(m.selection for m in mgrid.menus)\n\t)\n\t\ton(update_working_df, observable)\n\tend\n\n\tx_var_menu = Menu(fig, options = regressables)\n\ty_var_menu = Menu(fig, options = regressables)\n\tcoded_menu = Menu(fig, options = regressables)\n\n\txs = @lift($working_df !== nothing && $(x_var_menu.selection) !== nothing ? $working_df[$(x_var_menu.selection)] : nothing)\n\tys = @lift($working_df !== nothing && $(y_var_menu.selection) !== nothing ? $working_df[$(y_var_menu.selection)] : nothing)\n\n\tscatter = @lift(\n\t\t$xs !== nothing && $ys !== nothing && $(coded_menu.selection) !== nothing ?\n\t\t\tlines!(\n\t\t\t\tax,\n\t\t\t\t$xs,\n\t\t\t\t$ys;\n\t\t\t\tcolor=$(coded_menu.selection)\n\t\t\t) : nothing\n\t)\n\n\tfig\nend\n\nplot(df)","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"
\n","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"","category":"page"},{"location":"examples/examples/perf_plots/","page":"Perf plots","title":"Perf plots","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/index.md\"","category":"page"},{"location":"examples/#demo-page","page":"Examples","title":"Examples and demos","text":"","category":"section"},{"location":"examples/","page":"Examples","title":"Examples","text":"You can browse through the examples below. Each example has code and outputs, as well as links to download a script to run locally or to view as a Jupyter notebook on NBViewer. Some of the examples have interactive plots, and these will not currently work unless the scripts are run locally. TODO: Add binder links to examples.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"If you would like to run the examples as Jupyter notebooks locally to see interactive plots inline with the code, you can clone the repository and run docs/build_nbs.jl like so:","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"$ git clone https://github.com/JuliaPsychometricsBazaar/ComputerAdaptiveTesting.jl.git\n$ cd ComputerAdaptiveTesting\n$ julia --project=. docs/build_nbs.jl","category":"page"},{"location":"examples/#Examples","page":"Examples","title":"Examples","text":"","category":"section"},{"location":"examples/","page":"Examples","title":"Examples","text":"
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"How abilities converge on simulated 3PL data","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"This example shows how to run a CAT based on a synthetic correct/incorrect 3PL IRT model.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"How abilities converge on simulated MIRT data","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"This example shows how to run a CAT based on a synthetic correct/incorrect MIRT model.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Item Response Functions","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Typically, the logistic c.d.f. is used as the transfer function in IRT. However, it in an IRT context, a scaled version intended to be close to a normal c.d.f. is often used. The main advantage is that this is usually faster to compute. ComputerAdaptiveTesting provides normalscaledlogistic, which is also used by default, for this purpose:","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Perf plots","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Perf plots","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n\n \n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"Running a CAT based based on real response data","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"This example shows how to run a CAT end-to-end on real data.","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
\n
\n
","category":"page"},{"location":"examples/","page":"Examples","title":"Examples","text":"
","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"CurrentModule = ComputerAdaptiveTesting","category":"page"},{"location":"creating_a_cat/#Creating-a-CAT","page":"Creating a CAT","title":"Creating a CAT","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"This guide gives a brief overview of how to create a CAT using the configuration structs in ComputerAdaptiveTesting.jl.","category":"page"},{"location":"creating_a_cat/#API-design","page":"Creating a CAT","title":"API design","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The configuration of a CAT is built up as a tree of configuration structs. These structs are all subtypes of CatConfigBase.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.CatConfig.CatConfigBase","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.ConfigBase.CatConfigBase-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.ConfigBase.CatConfigBase","text":"abstract type CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The constructors for the configuration structs in this package tend to have smart defaults. In general most constructors have two forms. The first is an explicit keyword constructor form, where all arguments are given:","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ConfigurationObject(\n field1=value1,\n field1=value2,\n)","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The second is an implicit form, where arguments are given in any order. If possible, they will be used together will appropriately guessed defaults to construct the configuration:","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ConfigurationObject(value2, value1)","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The implicit form is particularly useful for quick prototyping. Implicit form constructors are also available for some abstract types. In this case, they will return a concrete type that is a reasonable default for the abstract type.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"After using the implicit form, you can print the object to see what values have been filled in. This may be useful in case you want to modify some of the defaults or switch to the explicit form.","category":"page"},{"location":"creating_a_cat/#Item-banks","page":"Creating a CAT","title":"Item banks","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"Item banks are the source of items for the test. The basic definitions are provided by FittedItemBanks.jl and can be fit to data using RIrtWrappers.jl. See the documentation pages of those packages for more information.","category":"page"},{"location":"creating_a_cat/#CatRules","page":"Creating a CAT","title":"CatRules","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"This is the main type for configuring a CAT. It contains the item bank, the next item selection rule, and the stopping rule. CatRules has explicit and implicit constructors.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.CatConfig.CatRules","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.CatConfig.CatRules-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.CatConfig.CatRules","text":"struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\nnext_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.\n\ntermination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.\n\nability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()\n\nConfiguration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.\n\nThis may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.\n\nCatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)\n\nExplicit constructor for CatRules.\n\nCatRules(bits...)\n\nImplicit constructor for CatRules.\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/#Next-item-selection-with-NextItemRule","page":"Creating a CAT","title":"Next item selection with NextItemRule","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The next item selection rule is the most important part of the CAT. Each rule extends the NextItemRule abstract type.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.NextItemRule","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.NextItemRule-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.NextItemRule","text":"abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\nAbstract base type for all item selection rules. All descendants of this type are expected to implement the interface (rule::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int\n\nNextItemRule(bits...; ability_estimator=nothing, parallel=true)\n\nImplicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule.\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"A sort of null hypothesis next item selection rule is RandomNextItemRule, which ","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule","text":"struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule\n\nrng::Random.AbstractRNG: Default: Xoshiro()\n\nThis is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"Other rules are created by combining a ItemCriterion – which somehow rates items according to how good they are – with a NextItemStrategy using an ItemStrategyNextItemRule, which acts as an adapter. The default NextItemStrategy (and currently only) is ExhaustiveSearch1Ply. When using the implicit constructors, ItemCriterion can therefore be used directly without wrapping in any place an NextItemRule is expected.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule","text":"struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule\n\nstrategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy\ncriterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion\n\nItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.\n\nItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)\n\nImplicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch1Ply when no NextItemStrategy is given.\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.ItemCriterion","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.ItemCriterion-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.ItemCriterion","text":"abstract type ItemCriterion <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.NextItemStrategy","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.NextItemStrategy-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.NextItemStrategy","text":"abstract type NextItemStrategy <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply","text":"struct ExhaustiveSearch1Ply <: ComputerAdaptiveTesting.NextItemRules.NextItemStrategy\n\nparallel::Bool\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/#Evaluating-item-and-state-merit-with-ItemCriterion-and-StateCriterion","page":"Creating a CAT","title":"Evaluating item and state merit with ItemCriterion and StateCriterion","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"The ItemCriterion abstract type is used to rate items according to how good they are as a candidate for the next item. A typical example is InformationItemCriterion, which using the current ability estimate theta and the item response function irf to calculate each item's information fracirf_θ^2irf_θ * (1 - irf_θ).","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"Within this, there are two types of ExpectationBasedItemCriterion which act as adapters: PointExpectationBasedItemCriterion and DistributionExpectationBasedItemCriterion. These take a StateCriterion, which evaluates how good a particular state is in terms getting a good estimate of the test takers ability. They look one ply ahead to get the expected value of the StateCriterion after selecting the given item. The AbilityVarianceStateCriterion looks at the variance of the ability theta estimate at that state.","category":"page"},{"location":"creating_a_cat/#Stopping-rules-with-TerminationCondition","page":"Creating a CAT","title":"Stopping rules with TerminationCondition","text":"","category":"section"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"Currently the only TerminationCondition is FixedItemsTerminationCondition, which ends the test after a fixed number of items.","category":"page"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.TerminationConditions.TerminationCondition","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.TerminationConditions.TerminationCondition-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.TerminationConditions.TerminationCondition","text":"abstract type TerminationCondition <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"creating_a_cat/","page":"Creating a CAT","title":"Creating a CAT","text":"ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition","category":"page"},{"location":"creating_a_cat/#ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition-creating_a_cat","page":"Creating a CAT","title":"ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition","text":"struct FixedItemsTerminationCondition <: ComputerAdaptiveTesting.TerminationConditions.TerminationCondition\n\nnum_items::Int64\n\n\n\n\n\n","category":"type"},{"location":"api/","page":"API reference","title":"API reference","text":"CurrentModule = ComputerAdaptiveTesting","category":"page"},{"location":"api/#API-reference","page":"API reference","title":"API reference","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"","category":"page"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [ComputerAdaptiveTesting, ComputerAdaptiveTesting.Aggregators, ComputerAdaptiveTesting.Responses, ComputerAdaptiveTesting.Sim, ComputerAdaptiveTesting.TerminationConditions, ComputerAdaptiveTesting.NextItemRules, ComputerAdaptiveTesting.CatConfig]","category":"page"},{"location":"api/#ComputerAdaptiveTesting.Aggregators","page":"API reference","title":"ComputerAdaptiveTesting.Aggregators","text":"This module takes care of integrating and optimizing over the ability/difficulty space. It includes TrackedResponses, which can store cumulative results during a test.\n\n\n\n\n\n","category":"module"},{"location":"api/#ComputerAdaptiveTesting.Aggregators.EnumerationOptimizer","page":"API reference","title":"ComputerAdaptiveTesting.Aggregators.EnumerationOptimizer","text":"Argmax + max over the ability likihood given a set of responses with a given coefficient using exhaustive search.\n\nTODO: Add item bank trait for enumerable item banks.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.Aggregators.RiemannEnumerationIntegrator","page":"API reference","title":"ComputerAdaptiveTesting.Aggregators.RiemannEnumerationIntegrator","text":"In case an item bank is enumerable (e.g. GriddedItemBank), then this method integrates over the ability likihood given a set of responses with a given coefficient function using a Riemann sum (aka the rectangle rule).\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.Aggregators.maybe_tracked_ability_estimate-Tuple{ComputerAdaptiveTesting.Aggregators.TrackedResponses, Any}","page":"API reference","title":"ComputerAdaptiveTesting.Aggregators.maybe_tracked_ability_estimate","text":"This method returns a tracked point estimate if it is has the given ability estimator, otherwise it computes it using the given ability estimator.\n\n\n\n\n\n","category":"method"},{"location":"api/#ComputerAdaptiveTesting.Sim.auto_responder-Tuple{Any}","page":"API reference","title":"ComputerAdaptiveTesting.Sim.auto_responder","text":"This function constructs a next item function which automatically responds according to responses.\n\n\n\n\n\n","category":"method"},{"location":"api/#ComputerAdaptiveTesting.Sim.prompt_response-Tuple{Any, Any}","page":"API reference","title":"ComputerAdaptiveTesting.Sim.prompt_response","text":"This response callback simply prompts \n\n\n\n\n\n","category":"method"},{"location":"api/#ComputerAdaptiveTesting.Sim.run_cat-Union{Tuple{RulesT}, Tuple{CatLoopConfig{RulesT}, FittedItemBanks.AbstractItemBank}} where RulesT<:CatRules","page":"API reference","title":"ComputerAdaptiveTesting.Sim.run_cat","text":"Run a given CatLoopConfig\n\n\n\n\n\n","category":"method"},{"location":"api/#ComputerAdaptiveTesting.Sim.run_random_comparison","page":"API reference","title":"ComputerAdaptiveTesting.Sim.run_random_comparison","text":"This function compares a given CAT configuration with a CAT using a random next item selection rule.\n\n\n\n\n\n","category":"function"},{"location":"api/#ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition","page":"API reference","title":"ComputerAdaptiveTesting.TerminationConditions.FixedItemsTerminationCondition","text":"struct FixedItemsTerminationCondition <: ComputerAdaptiveTesting.TerminationConditions.TerminationCondition\n\nnum_items::Int64\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.TerminationConditions.TerminationCondition","page":"API reference","title":"ComputerAdaptiveTesting.TerminationConditions.TerminationCondition","text":"abstract type TerminationCondition <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules","text":"This module implements the next item selection rules, which form the main part of CAT.\n\nBibliography\n\n[1] Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. In Elements of adaptive testing (pp. 3-30). Springer, New York, NY.\n\n\n\n\n\n","category":"module"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.catr_next_item_aliases","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.catr_next_item_aliases","text":"This mapping provides next item rules through the same names that they are available through in the catR R package. TODO compability with mirtcat\n\n\n\n\n\n","category":"constant"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.AbilityVarianceStateCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.AbilityVarianceStateCriterion","text":"struct AbilityVarianceStateCriterion{DistEst<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, IntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator} <: ComputerAdaptiveTesting.NextItemRules.StateCriterion\n\ndist_est::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator\nintegrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator\n\nThis StateCriterion returns the variance of the ability estimate given a set of responses.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.DistributionExpectationBasedItemCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.DistributionExpectationBasedItemCriterion","text":"struct DistributionExpectationBasedItemCriterion{DistributionAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator, AbilityIntegratorT<:ComputerAdaptiveTesting.Aggregators.AbilityIntegrator, StateCriterionT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.DistributionAbilityEstimator\nintegrator::ComputerAdaptiveTesting.Aggregators.AbilityIntegrator\nstate_criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion\n\nThis ItemCriterion wraps a StateCriterion and looks at its expected value for a particular item 1-ply ahead by integrating over an ability curve.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.ExhaustiveSearch1Ply","text":"struct ExhaustiveSearch1Ply <: ComputerAdaptiveTesting.NextItemRules.NextItemStrategy\n\nparallel::Bool\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.ItemCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.ItemCriterion","text":"abstract type ItemCriterion <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.ItemStrategyNextItemRule","text":"struct ItemStrategyNextItemRule{NextItemStrategyT<:ComputerAdaptiveTesting.NextItemRules.NextItemStrategy, ItemCriterionT<:ComputerAdaptiveTesting.NextItemRules.ItemCriterion} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule\n\nstrategy::ComputerAdaptiveTesting.NextItemRules.NextItemStrategy\ncriterion::ComputerAdaptiveTesting.NextItemRules.ItemCriterion\n\nItemStrategyNextItemRule which together with a NextItemStrategy acts as an adapter by which an ItemCriterion can serve as a NextItemRule.\n\nItemStrategyNextItemRule(bits...; ability_estimator=nothing, parallel=true)\n\nImplicit constructor for ItemStrategyNextItemRule. Will default to ExhaustiveSearch1Ply when no NextItemStrategy is given.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.NextItemRule","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.NextItemRule","text":"abstract type NextItemRule <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\nAbstract base type for all item selection rules. All descendants of this type are expected to implement the interface (rule::NextItemRule)(responses::TrackedResponses, items::AbstractItemBank)::Int\n\nNextItemRule(bits...; ability_estimator=nothing, parallel=true)\n\nImplicit constructor for NextItemRule. Uses any given NextItemRule or delegates to ItemStrategyNextItemRule.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.NextItemStrategy","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.NextItemStrategy","text":"abstract type NextItemStrategy <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.PointExpectationBasedItemCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.PointExpectationBasedItemCriterion","text":"struct PointExpectationBasedItemCriterion{PointAbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator, StateCriterionT<:ComputerAdaptiveTesting.NextItemRules.StateCriterion} <: ComputerAdaptiveTesting.NextItemRules.ExpectationBasedItemCriterion\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator\nstate_criterion::ComputerAdaptiveTesting.NextItemRules.StateCriterion\n\nThis ItemCriterion wraps a StateCriterion and looks at its expected value for a particular item 1-ply ahead based on a point ability estimate.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.RandomNextItemRule","text":"struct RandomNextItemRule{RandomT<:Random.AbstractRNG} <: ComputerAdaptiveTesting.NextItemRules.NextItemRule\n\nrng::Random.AbstractRNG: Default: Xoshiro()\n\nThis is the most basic rule for choosing the next item in a CAT. It simply picks a random item from the set of items that have not yet been administered.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.StateCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.StateCriterion","text":"abstract type StateCriterion <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.NextItemRules.UrryItemCriterion","page":"API reference","title":"ComputerAdaptiveTesting.NextItemRules.UrryItemCriterion","text":"struct UrryItemCriterion{AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator} <: ComputerAdaptiveTesting.NextItemRules.ItemCriterion\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.PointAbilityEstimator\n\nThis item criterion just picks the item with the raw difficulty closest to the current ability estimate.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.CatConfig.CatLoopConfig","page":"API reference","title":"ComputerAdaptiveTesting.CatConfig.CatLoopConfig","text":"Configuration for a simulatable CAT.\n\n\n\n\n\n","category":"type"},{"location":"api/#ComputerAdaptiveTesting.CatConfig.CatRules","page":"API reference","title":"ComputerAdaptiveTesting.CatConfig.CatRules","text":"struct CatRules{NextItemRuleT<:ComputerAdaptiveTesting.NextItemRules.NextItemRule, TerminationConditionT<:ComputerAdaptiveTesting.TerminationConditions.TerminationCondition, AbilityEstimatorT<:ComputerAdaptiveTesting.Aggregators.AbilityEstimator, AbilityTrackerT<:ComputerAdaptiveTesting.Aggregators.AbilityTracker} <: ComputerAdaptiveTesting.ConfigBase.CatConfigBase\n\nnext_item::ComputerAdaptiveTesting.NextItemRules.NextItemRule: The rule to choose the next item in the CAT given the current state.\n\ntermination_condition::ComputerAdaptiveTesting.TerminationConditions.TerminationCondition: The rule to choose when to terminate the CAT.\n\nability_estimator::ComputerAdaptiveTesting.Aggregators.AbilityEstimator: The ability estimator, which estimates the testee's current ability.\n\nability_tracker::ComputerAdaptiveTesting.Aggregators.AbilityTracker: The ability tracker, which tracks the testee's current ability level. Default: NullAbilityTracker()\n\nConfiguration of the rules for a CAT. This all includes all the basic rules for the CAT's operation, but not the item bank, nor any of the interactivity hooks needed to actually run the CAT.\n\nThis may be more a more convenient layer to integrate than CatLoopConfig if you want to write your own CAT loop rather than using hooks.\n\nCatRules(; next_item=..., termination_condition=..., ability_estimator=..., ability_tracker=...)\n\nExplicit constructor for CatRules.\n\nCatRules(bits...)\n\nImplicit constructor for CatRules.\n\n\n\n\n\n","category":"type"},{"location":"api/","page":"API reference","title":"API reference","text":"CurrentModule = CATPlots","category":"page"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [CATPlots]","category":"page"},{"location":"api/#CATPlots.CATPlots","page":"API reference","title":"CATPlots.CATPlots","text":"This module contains helpers for creating CAT/IRT related plots. This module requires the optional depedencies AlgebraOfGraphics, DataFrames and Makie to be installed.\n\n\n\n\n\n","category":"module"},{"location":"api/#CATPlots.LabelledToggleGrid","page":"API reference","title":"CATPlots.LabelledToggleGrid","text":"For information about attributes, use attribute_help(LabelledToggleGrid).\n\n\n\n\n\n","category":"type"},{"location":"api/#CATPlots.MenuGrid","page":"API reference","title":"CATPlots.MenuGrid","text":"For information about attributes, use attribute_help(MenuGrid).\n\n\n\n\n\n","category":"type"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/ability_convergence_3pl.jl\"","category":"page"},{"location":"examples/examples/ability_convergence_3pl/#ability_convergence_3pl","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"","category":"section"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"How abilities converge on simulated 3PL data","category":"page"},{"location":"examples/examples/ability_convergence_3pl/#Running-a-CAT-based-on-a-synthetic-correct/incorrect-3PL-IRT-model","page":"How abilities converge on simulated 3PL data","title":"Running a CAT based on a synthetic correct/incorrect 3PL IRT model","text":"","category":"section"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"This example shows how to run a CAT based on a synthetic correct/incorrect 3PL IRT model.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Import order is important. We put ComputerAdaptiveTesting last so we get the extra dependencies.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"using Makie\nimport Pkg\nimport Random\nusing Distributions: Normal, cdf\nusing AlgebraOfGraphics\nusing ComputerAdaptiveTesting\nusing ComputerAdaptiveTesting.Sim: auto_responder\nusing ComputerAdaptiveTesting.NextItemRules: AbilityVarianceStateCriterion\nusing ComputerAdaptiveTesting.TerminationConditions: FixedItemsTerminationCondition\nusing ComputerAdaptiveTesting.Aggregators: PriorAbilityEstimator, MeanAbilityEstimator, LikelihoodAbilityEstimator\nusing FittedItemBanks\nusing ComputerAdaptiveTesting.Responses: BooleanResponse\nimport PsychometricsBazaarBase.IntegralCoeffs\nusing PsychometricsBazaarBase.Integrators\nusing PsychometricsBazaarBase.ConstDistributions: normal_scaled_logistic\nusing CATPlots\n\n@automakie()","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Now we are read to generate our synthetic data using the supplied DummyData module. We generate an item bank with 100 items and fake responses for 3 testees.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"using FittedItemBanks.DummyData: dummy_full, std_normal, SimpleItemBankSpec, StdModel3PL\n(item_bank, abilities, responses) = dummy_full(\n Random.default_rng(42),\n SimpleItemBankSpec(StdModel3PL(), OneDimContinuousDomain(), BooleanResponse());\n num_questions=100,\n num_testees=3\n)","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Simulate a CAT for each testee and record it using CatRecorder. CatRecorder collects information which can be used to draw different types of plots.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"max_questions = 99\nintegrator = FixedGKIntegrator(-6, 6, 80)\ndist_ability_est = PriorAbilityEstimator(std_normal)\nability_estimator = MeanAbilityEstimator(dist_ability_est, integrator)\nrules = CatRules(\n ability_estimator,\n AbilityVarianceStateCriterion(dist_ability_est, integrator),\n FixedItemsTerminationCondition(max_questions)\n)\n\npoints = 500\nxs = range(-2.5, 2.5, length=points)\nraw_estimator = LikelihoodAbilityEstimator()\nrecorder = CatRecorder(xs, responses, integrator, raw_estimator, ability_estimator)\nfor testee_idx in axes(responses, 2)\n tracked_responses, θ = run_cat(\n CatLoopConfig(\n rules=rules,\n get_response=auto_responder(@view responses[:, testee_idx]),\n new_response_callback=(tracked_responses, terminating) -> recorder(tracked_responses, testee_idx, terminating),\n ),\n item_bank\n )\n true_θ = abilities[testee_idx]\n abs_err = abs(θ - true_θ)\nend","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Make a plot showing how the estimated value evolves during the CAT. We also plot the 'true' values used to generate the responses.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"conv_lines_fig = ability_evolution_lines(recorder; abilities=abilities)\nconv_lines_fig","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"Make an interactive plot, showing how the distribution of the ability likelihood evolves.","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"conv_dist_fig = lh_evolution_interactive(recorder; abilities=abilities)\nconv_dist_fig","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"","category":"page"},{"location":"examples/examples/ability_convergence_3pl/","page":"How abilities converge on simulated 3PL data","title":"How abilities converge on simulated 3PL data","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/irfs.jl\"","category":"page"},{"location":"examples/examples/irfs/#Item-Response-Functions","page":"Item Response Functions","title":"Item Response Functions","text":"","category":"section"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"Item Response Functions","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"using Makie\nimport Pkg\nusing Distributions: Normal, cdf\nusing PsychometricsBazaarBase.ConstDistributions: normal_scaled_logistic\nusing CATPlots\n\n@automakie()","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"Typically, the logistic c.d.f. is used as the transfer function in IRT. However, it in an IRT context, a scaled version intended to be close to a normal c.d.f. is often used. The main advantage is that this is usually faster to compute. ComputerAdaptiveTesting provides normalscaledlogistic, which is also used by default, for this purpose:","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"xs = -8:0.05:8\nlines(xs, cdf.(Normal(), xs))\nlines!(xs, cdf.(normal_scaled_logistic, xs))\ncurrent_figure()","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"
\n","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"","category":"page"},{"location":"examples/examples/irfs/","page":"Item Response Functions","title":"Item Response Functions","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/vocab_iq.jl\"","category":"page"},{"location":"examples/examples/vocab_iq/#vocab_iq","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"","category":"section"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"Running a CAT based based on real response data","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"This example shows how to run a CAT end-to-end on real data.","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"First a 1-dimensional IRT model is fitted based on open response data to the vocabulary IQ test using the IRTSupport package which internally, this uses the mirt R package. Next, the model is used to administer the test interactively.","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"using Base.Filesystem\nusing ComputerAdaptiveTesting\nusing FittedItemBanks.DummyData: std_normal\nusing ComputerAdaptiveTesting.Sim\nusing ComputerAdaptiveTesting.NextItemRules\nusing ComputerAdaptiveTesting.TerminationConditions\nusing ComputerAdaptiveTesting.Aggregators\nusing FittedItemBanks\nusing FittedItemBanks: item_params\nimport PsychometricsBazaarBase.IntegralCoeffs\nusing PsychometricsBazaarBase.Integrators\nusing PsychometricsBazaarBase.Optimizers\nusing ItemResponseDatasets: prompt_readline\nusing ItemResponseDatasets.VocabIQ\nusing RIrtWrappers.Mirt\n\nfunction get_item_bank()\n fit_4pl(get_marked_df_cached(); TOL=1e-2)\nend\n\nfunction run_vocab_iq_cat()\n item_bank, labels = get_item_bank()\n integrator = FixedGKIntegrator(-6, 6, 61)\n ability_integrator = AbilityIntegrator(integrator)\n dist_ability_est = PriorAbilityEstimator(std_normal)\n optimizer = AbilityOptimizer(OneDimOptimOptimizer(-6.0, 6.0, NelderMead()))\n ability_estimator = ModeAbilityEstimator(dist_ability_est, optimizer)\n @info \"run_cat\" ability_estimator\n rules = CatRules(\n ability_estimator,\n AbilityVarianceStateCriterion(dist_ability_est, ability_integrator),\n FixedItemsTerminationCondition(45)\n )\n function get_response(response_idx, response_name)\n params = item_params(item_bank, response_idx)\n println(\"Parameters for next question: $params\")\n prompt_readline(VocabIQ.questions[response_idx])\n end\n function new_response_callback(tracked_responses, terminating)\n if tracked_responses.responses.values[end] > 0\n println(\"Correct\")\n else\n println(\"Wrong\")\n end\n ability = ability_estimator(tracked_responses)\n var = variance_given_mean(ability_integrator, dist_ability_est, tracked_responses, ability)\n println(\"Got ability estimate: $ability ± $var\")\n println(\"\")\n end\n loop_config = CatLoopConfig(\n rules=rules,\n get_response=get_response,\n new_response_callback=new_response_callback\n )\n run_cat(loop_config, item_bank)\nend\n\nrun_vocab_iq_cat()","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"","category":"page"},{"location":"examples/examples/vocab_iq/","page":"Running a CAT based based on real response data","title":"Running a CAT based based on real response data","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"EditURL = \"/home/runner/work/ComputerAdaptiveTesting.jl/ComputerAdaptiveTesting.jl/docs/examples/examples/ability_convergence_mirt.jl\"","category":"page"},{"location":"examples/examples/ability_convergence_mirt/#ability_convergence_mirt","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"","category":"section"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"(Image: Source code) (Image: notebook)","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"How abilities converge on simulated MIRT data","category":"page"},{"location":"examples/examples/ability_convergence_mirt/#Running-a-CAT-based-on-a-synthetic-correct/incorrect-MIRT-model","page":"How abilities converge on simulated MIRT data","title":"Running a CAT based on a synthetic correct/incorrect MIRT model","text":"","category":"section"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"This example shows how to run a CAT based on a synthetic correct/incorrect MIRT model.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"Import order is important. We put ComputerAdaptiveTesting last so we get the extra dependencies.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"using Makie\nimport Pkg\nimport Random\nusing Distributions: Normal, cdf\nusing AlgebraOfGraphics\nusing ComputerAdaptiveTesting\nusing ComputerAdaptiveTesting.Sim: auto_responder\nusing ComputerAdaptiveTesting.NextItemRules: DRuleItemCriterion\nusing ComputerAdaptiveTesting.TerminationConditions: FixedItemsTerminationCondition\nusing ComputerAdaptiveTesting.Aggregators: PriorAbilityEstimator, MeanAbilityEstimator, LikelihoodAbilityEstimator\nusing FittedItemBanks\nimport PsychometricsBazaarBase.IntegralCoeffs\nusing PsychometricsBazaarBase.Integrators\nusing PsychometricsBazaarBase.ConstDistributions: normal_scaled_logistic\nusing CATPlots\n\n@automakie()","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"Now we are read to generate our synthetic data using the supplied DummyData module. We generate an item bank with 100 items and fake responses for 3 testees.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"dims = 2\nusing FittedItemBanks.DummyData: dummy_full, std_mv_normal, SimpleItemBankSpec, StdModel4PL\nusing ComputerAdaptiveTesting.Responses: BooleanResponse","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"TODO: pass in dims","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"(item_bank, abilities, responses) = dummy_full(\n Random.default_rng(42),\n SimpleItemBankSpec(StdModel4PL(), VectorContinuousDomain(), BooleanResponse()),\n dims;\n num_questions=10,\n num_testees=2\n)","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"Simulate a CAT for each testee and record it using CatRecorder. CatRecorder collects information which can be used to draw different types of plots.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"max_questions = 9\nintegrator = CubaIntegrator([-6.0, -6.0], [6.0, 6.0], CubaVegas(); rtol=1e-2)\nability_estimator = MeanAbilityEstimator(PriorAbilityEstimator(std_mv_normal(dims)), integrator)\nrules = CatRules(\n ability_estimator,\n DRuleItemCriterion(ability_estimator),\n FixedItemsTerminationCondition(max_questions)\n)","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"XXX: We shouldn't need to specify xs here since the distributions are not used – rework","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"points = 3\nxs = repeat(range(-2.5, 2.5, length=points)', dims, 1)\nraw_estimator = LikelihoodAbilityEstimator()\nrecorder = CatRecorder(xs, responses, integrator, raw_estimator, ability_estimator, abilities)\nfor testee_idx in axes(responses, 2)\n @debug \"Running for testee\" testee_idx\n tracked_responses, θ = run_cat(\n CatLoopConfig(\n rules=rules,\n get_response=auto_responder(@view responses[:, testee_idx]),\n new_response_callback=(tracked_responses, terminating) -> recorder(tracked_responses, testee_idx, terminating),\n ),\n item_bank\n )\n true_θ = abilities[:, testee_idx]\n abs_err = sum(abs.(θ .- true_θ))\n @info \"convergence\" true_θ θ abs_err\nend","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"Make a plot showing how the estimated value converges during the CAT.","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"conv_lines_fig = ability_convergence_lines(recorder; abilities=abilities)\nconv_lines_fig","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"","category":"page"},{"location":"examples/examples/ability_convergence_mirt/","page":"How abilities converge on simulated MIRT data","title":"How abilities converge on simulated MIRT data","text":"This page was generated using DemoCards.jl and Literate.jl.","category":"page"},{"location":"#Getting-started","page":"Getting started","title":"Getting started","text":"","category":"section"},{"location":"","page":"Getting started","title":"Getting started","text":"This is the documentation for ComputerAdaptiveTesting. For information about the purpose of the package and its installation, please see the README.md.","category":"page"},{"location":"#Contents","page":"Getting started","title":"Contents","text":"","category":"section"},{"location":"","page":"Getting started","title":"Getting started","text":"Creating a CAT: A guide to creating a CAT using the configuration structs in ComputerAdaptiveTesting.jl\nUsing your CAT: A collection of mini topical guides about: simulating CATs, integrating into a user-facing applications, evaluating CATs, visualising CATs, and constructing decision trees from CATs\nExamples: A collection of example code which you can run and modify to get a feel for how to use the package\nAPI reference: The full API reference for the package","category":"page"}] } diff --git a/dev/using_your_cat/index.html b/dev/using_your_cat/index.html index 5b656dd..4bd2574 100644 --- a/dev/using_your_cat/index.html +++ b/dev/using_your_cat/index.html @@ -1,2 +1,2 @@ -Using your CAT · ComputerAdaptiveTesting.jl
GitHub
+Using your CAT · ComputerAdaptiveTesting.jl
GitHub