- Security considerations
- Installation and requirements
- Avoiding
|> ignore
after assertion chains - Writing your own assertions
- Customizing the format
- Configuring options
- Assertion list
- FAQ
Treat assertion exception messages (and therefore test failure messages) as securely as you treat your source code.
Faqt derives subject names from your source code. Known existing limitations (described later in this document) as well as bugs can cause Faqt to use a lot more of your code in the subject name than intended (up to entire source files). Therefore, do not give anyone access to Faqt assertion failure messages if they should not have access to your source code.
-
Install Faqt from NuGet. Faqt supports .NET 5.0 and higher.
-
If you use path mapping (e.g., CI builds with
DeterministicSourcePaths
enabled) or want to execute assertions where source files are not available (e.g. in production), enable the following settings on all projects that call assertions (either in the.fsproj
files or inDirectory.Build.props
):<DebugType>embedded</DebugType> <EmbedAllSources>true</EmbedAllSources>
Alternatively, enable them by passing the following parameters to your
dotnet build
/test
/publish
commands:-p:DebugType=embedded -p:EmbedAllSources=true
Note that
DebugType=embedded
is automatically set by DotNet.ReproducibleBuilds if you use that.
Since assertions return And
or AndDerived
, F# may warn you in some cases if an assertion chain is not ignored
using |> ignore
.
For convenience, you can open Faqt.Operators
and use the %
prefix operator instead:
%x.Should().Be("a")
%y.Should().Be("b")
Note that the %
operator is simply an alias for ignore
and is defined like this:
let inline (~%) x = ignore x
If you want to use another operator, you can define your own just as easily.
See this StackOverflow answer for valid prefix operators. However, your
custom operator will then be shown in the subject name (whereas %
is automatically removed).
Writing your own assertions is easy! Custom assertions are implemented exactly like Faqt’s built-in assertions, so you
can always look at those for inspiration (see all files ending with Assertions
in this folder).
All the details are further below, but first, we'll get a long way just by looking at some examples.
Here is Faqt’s simplest assertion, Be
:
open System.Runtime.CompilerServices
open Faqt
open Faqt.AssertionHelpers
[<Extension>]
type Assertions =
/// Asserts that the subject is equal to the specified value.
[<Extension>]
static member Be(t: Testable<'a>, expected: 'a, ?because) : And<'a> =
use _ = t.Assert()
if t.Subject <> expected then
t.With("Expected", expected).With("But was", t.Subject).Fail(because)
And(t)
Simple, right? Using the default YAML-based formatter, it will result in this message:
Subject: <expression from user code>
Because: <as specified by user, skipped if None>
Should: Be
Expected: <expected value>
But was: <subject value>
As you can see, Faqt automatically adds Subject
, Because
(if supplied by the user), and Should
(the name of the
method where you call t.Assert()
). After that, any additional key-value pairs you specify are displayed in order.
Now let's look at an assertion that's just as simple, but uses derived state, where you return
AndDerived
instead of And
:
open System
open System.Runtime.CompilerServices
open Faqt
open Faqt.AssertionHelpers
[<Extension>]
type Assertions =
/// Asserts that the subject has a value.
[<Extension>]
static member HaveValue(t: Testable<Nullable<'a>>, ?because) : AndDerived<Nullable<'a>, 'a> =
use _ = t.Assert()
if not t.Subject.HasValue then
t.With("But was", t.Subject).Fail(because)
AndDerived(t, t.Subject.Value)
This allows users to continue asserting on the derived state (the inner value, in this case), for example like
this: nullableInt.Should().HaveValue().That.Should(()).Be(2)
.
Finally, let's look at a more complex assertion - a higher-order assertion that calls user assertions and which also asserts for every item in a sequence:
open System
open System.Runtime.CompilerServices
open Faqt
open Faqt.AssertionHelpers
open Faqt.Formatting
type private SatisfyReportItem = { Index: int; Failure: FailureData }
[<Extension>]
type Assertions =
/// Asserts that all items in the collection satisfy the supplied assertion.
[<Extension>]
static member AllSatisfy(t: Testable<#seq<'a>>, assertion: 'a -> 'ignored, ?because) : And<_> =
use _ = t.Assert(true, true)
if isNull (box t.Subject) then
t.With("But was", t.Subject).Fail(because)
let failures =
t.Subject
|> Seq.indexed
|> Seq.choose (fun (i, x) ->
try
use _ = t.AssertItem()
assertion x |> ignore
None
with :? AssertionFailedException as ex ->
Some { Index = i; Failure = ex.FailureData }
)
|> Seq.toArray
if failures.Length > 0 then
t.With("Failures", failures).With("Subject value", t.Subject).Fail(because)
And(t)
Note that in this case we use t.Assert(true, true)
at the top. Both parameters are optional. The first true
indicates that this is a higher-order assertion, and the second true
indicates that the assertions are run for each
item in a sequence. Note also that we call use _ = t.AssertItem()
before the assertion of each item.
The most significant thing not demonstrated in the examples above is that if your assertion calls Should
, make sure
to use the Should(t)
overload instead of Should()
.
If you want all the details, here they are:
-
Open
Faqt.AssertionHelpers
. -
If needed, open
Faqt.Formatting
to get access toTryFormat
(described below) andFailureData
(mostly useful for higher-order assertions as demonstrated above). -
Implement the assertion as an extension method for
Testable
(the first argument), with whatever constraints you need. The constraints could be implicitly imposed by F#, as withBe
where it requiresequality
on'a
due to the use of the inequality operator (<>
), or they could be explicitly specified, for example by specifying more concrete types (such asTestable<'a option>
in order to have your extension only work foroption
-wrapped types). -
Accept whichever arguments you need for your assertion, and end with an optional
?because
parameter. -
First in your method, call
use _ = t.Assert()
. This is needed to track important state necessary for subject names to work. The method has two optional boolean parameters. If your assertion is a higher-order assertion that calls user code that is expected to call other assertions (likeSatisfy
), uset.Assert(true)
. If your assertion additionally calls the same user assertion(s) for each item in a sequence (likeAllSatisfy
), callt.Assert(true, true)
, and additionally calluse _ = t.AssertItem()
before the assertion of each item. -
If your condition is not met and the assertion should fail, call
t.Fail(because)
, optionally with any number ofWith(key, value)
orWith(condition, key, value)
beforeFail
:t.With("Key 1", value1).With("Key 2", value2).Fail(because)
Note that the keys
"Subject"
,"Because"
, and"Should"
are reserved by Faqt.Which key-value pairs, if any, to add to the message is up to you. Most assertion failure messages are more helpful if they display value being tested (
t.Subject
). Faqt mostly places this as the last value, so that if its rendering is very large, it does not push other important details far down. If you do rendert.Subject
and there is no logical key that fits (such as"But was"
in the assertionBe
), the recommended key is"Subject value"
.If you add values where you wrap user-supplied data (e.g., in a record, list or similar), then you should wrap the user values in the single-case DU
TryFormat
. This will ensure that if the value fails serialization, a fallback formatter will be used for that value. IfTryFormat
is not used, only the top-level items added usingWith
will have this behavior, which may cause a (less useful) fallback formatter to be used for your top-level values instead of its (user-supplied) constituent parts.If you use anonymous type values in your assertion, note that anonymous type members seems to appear in alphabetical order, not declaration order. If this is not desired, use a normal record.
-
If your assertion extracts derived state that can be used for further assertions, return
AndDerived(t, derivedState)
. Otherwise returnAnd(t)
. PreferAndDerived
overAnd
if at all relevant, since it strictly expands what the user can do. -
If your assertion calls
Should
at any point, make sure you use the overload that takes the originalTestable
as an argument (.Should(t)
), since it contains important state relating to the end user’s original assertion call. -
If your assertion calls other assertions, consider how your assertion method name will read when used with the assertion message from the called assertion. For example,
Be
has a message like:Subject: x Should: Be Expected: true But was: false
If the
bool
assertionBeTrue
just calledBe(true)
internally, it would read like:Subject: x Should: BeTrue Expected: true But was: false
The
Expected: true
line is superfluous given the name of the assertion. Therefore, it's better with a separate assertion message forBeTrue
that does not includeExpected
, thereby producing this improved message:Subject: x Should: BeTrue But was: false
Faqt's formatter is implemented as a simple function with signature FailureData -> string
. You can implement your own
formatter from scratch (the FailureData
members correspond to the keys in the default assertion messages and are
hopefully self-explanatory), or easily configure the built-in YAML-based formatter to your liking as shown below:
open Faqt
open Faqt.Formatting
let myFormatter : FailureData -> string =
// You can implement your own formatter from scratch, or modify the default one as shown here
YamlFormatterBuilder.Default
// Override System.Text.Json options
.ConfigureJsonSerializerOptions(fun opts -> opts.MaxDepth <- 5)
// Override FSharp.SystemTextJson options
.ConfigureJsonFSharpOptions(fun opts -> opts.WithUnionAdjacentTag())
// Add custom System.Text.Json converters
.AddConverter(MyJsonConverter())
// Easily transform values before serializing without needing a custom converter
.SerializeAs(fun (t: System.Type) -> t.FullName)
// Same as above, but does not apply to subtypes
.SerializeExactAs(...)
// Set how values wrapped in TryFormat are formatted when serialization fails
.TryFormatFallback(fun _ex obj -> obj.ToString())
// Set the YamlDotNet visitor (inheriting from YamlDotNet.RepresentationModel.YamlVisitorBase)
// that is used after loading the serialized JSON into a YAML document
.SetYamlVisitor(MyYamlVisitor)
// Build the formatter
.Build()
// Set the default formatter
Formatter.Set(myFormatter)
// Set the config for a certain scope (until the returned value is disposed)
use _ = Formatter.With(myConfig)
Faqt contains some configurable options, which are adjusted similarly to the formatter:
open Faqt
open Faqt.Configuration
// Create a configuration
let myConfig =
FaqtConfig.Default
// Set the maximum length of rendered HttpContent in assertion failure output
.SetHttpContentMaxLength(1024 * 1024)
// Disable formatting of HttpContent (such as indenting JSON for readability)
.SetFormatHttpContent(false)
// Transform HTTP header values
.SetMapHttpHeaderValues(fun name value ->
if name.Equals("Authorization", StringComparison.OrdinalIgnoreCase) then "***" else value
)
// Set the default config
Config.Set(myConfig)
// Set the config for a certain scope (until the returned value is disposed)
use _ = Config.With(myConfig)
// Config.Current is available globally and can be used in your own converters/formatters.
myFormatter Config.Current
Satisfy
NotSatisfy
SatisfyAny
SatisfyAll
Be
: Structural or custom equalityNotBe
: Structural or custom equalityBeOneOf
: Structural equality with multiple candidates, optionally with a mapping for the derived valueNotBeOneOf
: Structural equality with multiple candidatesBeSameAs
: Reference equalityNotBeSameAs
: Reference equalityBeNull
NotBeNull
Transform
: Parsing or other transformations that can throwTryTransform
: Same asTransform
, but for functions returningOption
,Result
, orbool * 'a
( likeInt32.TryParse
)BeOfType
: Exact type checkBeAssignableTo
: Polymorphic type check
BeCloseTo
: Same asBe
, but with a toleranceNotBeCloseTo
: Same asNotBe
, but with a toleranceBeGreaterThan
BeGreaterThanOrEqualTo
BeLessThan
BeLessThanOrEqualTo
BePositive
BeNegative
BeNonNegative
BeNonPositive
BeInRange
: Inclusive range
BeOfCase
: Assert DU case and continue asserting on the inner valueBeSome
BeNone
BeOk
BeError
BeTrue
BeFalse
Imply
: If the subject is true, the specified value must also be trueBeImpliedBy
: If the specified value is true, the subject must also be true
HaveValue
NotHaveValue
BeNull
NotBeNull
BeUpperCase
: Case check with invariant or specified cultureBeLowerCase
: Case check with invariant or specified cultureBe
: Equality with specified comparison type (the normalBe
uses ordinal comparison)NotBe
: Equality with specified comparison type (the normalNotBe
uses ordinal comparison)Contain
: Substring check with ordinal or specified comparison typeNotContain
: Substring check with ordinal or specified comparison typeStartWith
: Prefix check with ordinal or specified comparison typeNotStartWith
: Prefix check with ordinal or specified comparison typeEndWith
: Suffix check with ordinal or specified comparison typeNotEndWith
: Suffix check with ordinal or specified comparison typeMatchRegex
NotMatchRegex
MatchWildcard
: Simplified wildcard check with*
(zero or more characters) and?
(one character)NotMatchWildcard
: Simplified wildcard check with*
(zero or more characters) and?
(one character)BeJsonEquivalentTo
: Checks that two JSON strings are equivalent (ignoring formatting)DeserializeTo
: Checks that a string is deserializable to a specified target type- All
seq<_>
assertions, including:HaveLength
BeEmpty
NotBeEmpty
BeNullOrEmpty
Contain
: Member check with key and valueNotContain
: Member check with key and valueHaveSameItemsAs
ContainKey
NotContainKey
ContainKeys
ContainValue
NotContainValue
- All
seq<_>
assertions, including:AllSatisfy
SatisfyRespectively
HaveLength
BeEmpty
NotBeEmpty
BeNullOrEmpty
Contain
: Member check withKeyValuePair<_, _>
NotContain
: Member check withKeyValuePair<_, _>
ContainExactlyOneItem
ContainExactlyOneItemMatching
ContainAtLeastOneItem
ContainAtLeastOneItemMatching
ContainAtMostOneItem
ContainAtMostOneItemMatching
ContainItemsMatching
NotContainItemsMatching
BeSupersetOf
BeProperSupersetOf
BeSubsetOf
BeProperSubsetOf
IntersectWith
NotIntersectWith
AllSatisfy
: Higher-order assertion where all values must satisfy the supplied assertionSatisfyRespectively
: Higher-order assertion where each value must satisfy the respective supplied assertionHaveLength
BeEmpty
NotBeEmpty
BeNullOrEmpty
Contain
: Member check with structural equalityNotContain
: Member check with structural equalityAllBe
: Identical items check with structural equalityAllBeMappedTo
: Identical mapped items check with structural equalityAllBeEqual
AllBeEqualBy
SequenceEqual
: Item-wise check with structural equalityHaveSameItemsAs
: Order-ignoring items check with structural equalityContainExactlyOneItem
ContainExactlyOneItemMatching
ContainAtLeastOneItem
ContainAtLeastOneItemMatching
ContainAtMostOneItem
ContainAtMostOneItemMatching
ContainItemsMatching
NotContainItemsMatching
BeDistinct
BeDistinctBy
BeAscending
BeAscendingBy
BeDescending
BeDescendingBy
BeStrictlyAscending
BeStrictlyAscendingBy
BeStrictlyDescending
BeStrictlyDescendingBy
BeSupersetOf
BeProperSupersetOf
BeSubsetOf
BeProperSubsetOf
IntersectWith
NotIntersectWith
Be
: Equality check againststring
NotBe
: Equality check againststring
BeEmpty
NotBeEmpty
HaveFlag
NotHaveFlag
Throw
: Polymorphic exception check for top-level exceptionThrowInner
: Polymorphic exception check for top-level or inner exception on any level (including any exception in anAggregateException
)ThrowExactly
: Exact exception check for top-level exceptionNotThrow
Roundtrip
: Check that a (potentiallyOption
orResult
-returning) function returns the input value. The function is typically a composition of parsing and extracting a value, e.g.,(fromX >> toX).Should().Roundtrip(value)
.
All assertion failure messages contain the full response and the original request.
HaveStatusCode
Be1XXInformational
Be2XXSuccessful
Be3XXRedirection
Be4XXClientError
Be5XXServerError
Be100Continue
Be101SwitchingProtocols
Be200Ok
- (etc. for other status codes)
HaveHeader
: Check for the existence of a header (and continue asserting on the header value(s))HaveHeaderValue
: Check for the existence of a header with a specific valueHaveStringContentSatisfying
: Check for string content satisfying a specified inner assertion
All of them. XUnit, NUnit, MSTest, NSpec, MSpec, Expecto, you name it. Faqt is agnostic to the test framework (and can also be used in non-test production code); it simply throws a custom exception when an assertion fails.
The automatic subject name (the first part of the assertion message) is correct in most situations, but there are edge cases where it may produce unexpected results:
- Multi-line strings literals will be concatenated.
- Lines starting with
//
in multi-line string literals will be removed. - Subject names will be truncated if they are too long (currently 1000 characters, though that may change without notice). This is because it is then likely that a limitation or a bug is causing Faqt to use too large parts of the source code as the subject name.
- The subject name may be incorrect under the following conditions:
- Assertion chains not starting on a new line or at the start of a lambda (
fun ... ->
or_.
) - Assertion chains containing lambdas (
fun ... ->
or_.
) outside an assertion - Nested
Satisfy
,AllSatisfy
or other higher-order assertions SatisfyAny
or similar with multiple assertion chains all on the same line containing the same assertion- Assertion chains not fully completing on a single thread
- Assertion chains containing non-assertion methods with the same name as an assertion
- Situations where assertions are not invoked in source order, such as for assertions chained after
AllSatisfy
if the sequence is empty
- Assertion chains not starting on a new line or at the start of a lambda (
If you have encountered a case not covered above, please raise an issue. If I can't or won't fix it, I can at the very least document it as a known limitation.
These limitations are due to the implementation of automatic subject names. The implementation is based on clever use of caller info attributes, parsing source code from either local files or embedded resources, thread-local state, and simple regex-based processing/replacement of the call chain based on which assertions have been encountered so far.
If you would like to help make the automatic subject name functionality more robust, please raise an issue. You can find the relevant code in SubjectName.fs.
This is due to how subject names are implemented, and the solution was chosen as the lesser of several evils. The
details are probably boring, but in short, when an assertion fails, Faqt needs to know the chain of assertions
encountered in the source code in order to derive the correct subject name. This chain is stored in thread-local state,
and has to be reset when a new assertion chain starts. This is done in Should()
. However, that would ruin the subject
name for assertions after subsequent Should()
calls in the chain.
Alternative solutions would either require making the assertion syntax more verbose (e.g. by enclosing entire assertion
chains in some method call, or wrapping them in a use
statement in order to reset the thread-local state or avoid it
entirely), or make the subject name incorrect in many more cases (e.g. by removing the tracking of the encountered
assertion history altogether, thereby only giving correct subject names up to the first assertion of any given name in a
chain).
Note: I recognize that the below is not the only way to look at the issue. If you fundamentally disagree with this policy, I am open to discussing it. Please raise an issue.
It really boils down to assumptions about Faqt users would expect and find useful. For example, I assume that
making HaveLength(0)
pass for null
values would be a surprise for many users, and therefore be a bad idea. On the
other hand, allowing null values in assertions makes the assertions more composable, since it is trivial to
add .NotBeNull()
to the start of your assertion chain if you want to require a non-null
value for an assertion that
allows it (and somewhat harder to allow a null
in an assertion that requires a non-null
value, where you'd have to
use something like SatisfyAny
).
That being said, in order to find some guiding principles, the general policy on allowing or disallowing null
subject
values is based on the following:
null
is separate from "empty". Values that arenull
do not have properties like "length" and "contents", whereas empty values do.- Negative assertions (like
NotBeEmpty
orNotContain
) essentially assert the lack of a property, e.g., the lack of a specific length.
With that in mind, null
subject values are generally allowed in negative assertions and disallowed in positive
assertions. For example, HaveLength(0)
will fail for null
, because a null
value does not have any length (zero or
otherwise). Contrariwise, NotHaveLength(0)
(if it existed) would assert the lack of having the length 0
, and will
pass for null
values since they, indeed, do not possess the property of having that specific length.
Another way to look at it is that negative assertions could be thought of conceptually as e.g. not (HaveLength(0))
,
i.e., just an inversion of the corresponding positive assertion. In this light, anything that fails the positive
assertion (including null
) should pass the negative assertion.
The only exceptions are for assertions that check equality, such as Be
or BeSameAs
. Here, null
is considered equal
to null
(which is consistent with the default F# implementations of structural and reference equality). This also
extends to SequenceEqual
and HaveSameItemsAs
, which will pass if both sequences are null
.
FluentAssertions is a fantastic library, and very much the inspiration for Faqt. Unfortunately, its API design causes trouble for F#. Here are the reasons I decided to make Faqt instead of just using FluentAssertions:
- The
because
parameter cannot be omitted when used from F# (#2225). - Several assertions (specifically, those that accept an
Action<_>
) requireignore
when used from F# (#2226). - The subject name does not consider transformations in the assertion chain (#2223).
- Improving F# usage issues (particularly the point about the
because
parameter) was deemed out of scope for FluentAssertions. - The relatively free-form assertion messages of FluentAssertions are harder to parse than more structured output, especially for complex objects and collections.
- Some assertions run contrary to expectations of F# (or even C#) developers (discussion).
Note that Faqt does not aim for feature parity with FluentAssertions. For example, Faqt does not execute and report on multiple assertions simultaneously; like almost all assertion libraries, it stops at the first failure ("monadic" instead of "applicative" behavior).
I will admit I have not used Shouldly myself, but its feature set (ignoring the actual assertions) seems to be a subset of that of FluentAssertions. For example, it does not support chaining assertions. However, I like its easy-to-read assertion failure messages, and have used those as inspiration for Faqt's assertion messages.
Unquote is a great library built on a great idea: Use code quotations with
arbitrary bool
-returning F# expressions as your assertions, and Unquote will display step-by-step evaluations if the
assertion fails. This allows you to assert whatever you want without needing custom-made assertions.
Unfortunately, I stopped using it because of several issues:
- Its assertion messages are not very helpful for non-trivial objects or expressions. The rendering is confusing, and it can be very hard to see what the actual error is. Often, I resorted to debugging instead of reading the assertion message, because that turned out to be quicker.
- It is based around F# quotations, which have several limitations, for example regarding generic functions and mutable values. Simply put, not all F# code can be used in a quotation.
- It can not be used to extract values for further testing or similar (which is supported by Faqt's
BeSome
and similar assertions). - I need assertions that can work in production code, too. I assume that evaluating quotations has a significant performance impact. (I have admittedly not measured this, since I stopped using it for the reasons above anyway.)
Faqt is designed only for F#. The subject names only work correctly for F#, and the API design and assertion choices are based on F# idioms and expected usage. Any support for C# is incidental, and improving or even preserving C# support is out of scope for Faqt. You are likely better off with FluentAssertions for C#.