Testing

A virtual clock instead of a real one, fuzzing at every parser, the specification's own published vectors, and mutation testing for the rules that have no vector.

On this page

Half of Matter is deadlines: the reliability layer backs off over four seconds, a fail-safe expires, an intermittently connected device sleeps for an hour, an mDNS name is claimed in 750 milliseconds. Testing that against a real clock means waiting, which means those paths get tested rarely and flakily.

Time is a parameter

platform::sim is a supported platform, not a test fixture: an in-process IPv6 network with a virtual clock that jumps to the next deadline instead of waiting for it, plus loss, duplication and jitter you set per test.

use matter_kit::platform::sim::{Impairment, SimNet, block_on};
use matter_kit::platform::{Timer, Duration};

let net = SimNet::new(42);              // seeded: same run, same result, every time
net.impair(Impairment::lossy(30));      // drop three datagrams in ten

let start = Timer::now(&net);
block_on(&net, async {
    net.sleep(Duration::from_secs(3600)).await;   // returns immediately
});
// An hour of virtual time has passed and no wall-clock time has.
assert_eq!(
    Timer::now(&net).saturating_duration_since(start),
    Duration::from_secs(3600)
);

The whole reliability layer runs through it: a lost message retransmitted until it lands, a duplicate acknowledged but delivered once, a replay rejected by the counter window, a dead peer abandoned after exactly five transmissions. Commissioning does too, including the cases that matter most — a wrong passcode, a forged confirmation, and keys used in the wrong direction.

Published vectors, not self-consistency

Two implementations agreeing proves only that they share an opinion. Wherever the specification prints a value, that value is the test: the TLV tables, the SPAKE2+ vectors, the privacy nonce, the onboarding payload, the compressed fabric identifier, the operational group key and IPK, the destination identifier, the RCAC/ICAC/NOC in both their Matter TLV and X.509 forms, the Certification Declarations with their CMS wrappers, two DNS-SD record listings, and the worked examples for network reordering and event encoding.

One check is stronger still, and was not designed. The first published Certification Declaration and the published DAC chain come from different chapters, were written independently, and happen to describe the same device. So the rules for relating a declaration to a chain can be run on two documents that were never meant to be used together — and they agree.

Fuzzing

Twenty-six targets cover every parser that faces the network, and each asserts more than “no panic”: that iteration terminates, that borrowed payloads really are slices of the input, that anything produced decodes again, and that a responder never answers a response.

They run for a minute on every pull request and for ten minutes nightly. The discovery target is the one that matters most — it runs against code executing earliest in a device’s life, before commissioning and therefore before any Matter security exists at all.

messaging fuzzes the reception path — the code that runs first on anything arriving from the network, and the only exposed parser with state worth attacking: a replay window, an exchange table, MRP timers. It asserts that no datagram ever installs a session, that the exchange table stays within its configured capacity, and that after any amount of garbage a real exchange still opens and delivers.

Not every target is a parser. acl fuzzes the cluster that decides what every other cluster will allow, where the failure mode is a quiet grant rather than a crash: it asserts that no sequence of bytes ever creates an entry on a fabric other than the writer’s, and that a PASE entry — which §6.6.2.1 forbids — never enters the list.

chunking fuzzes a liveness property instead: it lets the fuzzer choose the size and shape of every attribute value and the message budget, then checks that splitting a report across messages terminates, that MoreChunkedMessages is set on exactly the messages that are not the last, and that the chunks together carry each path exactly once. The failure it exists to catch is not a crash — it is a device that promises a continuation it never sends, or retries a value that will never fit.

Mutation testing where there is no vector

Most of the specification is prose, and prose has no vector. For those rules the test is: break the rule in the source, confirm a test fails, restore.

Every mutation that survives is a weak test, and each one found this way has been strengthened — a test comparing a constant against itself, an iterator silently swallowing decode errors, an encoder checked only against itself. A sample of what is pinned this way:

RuleSectionBreaking it
a concrete path gets a status, an expanded one is discarded§8.4.3.2a wildcard read enumerates what a subject may not see
the first access check comes before the existence checks§8.4.3.2the difference between two status codes is a map of the node
the cumulative fail-safe timer is never extended§11.10.7.2an administrator holds a device hostage forever
the NOC’s public key must match the device’s own§11.18.6.7an administrator impersonates the node it commissioned
the network list’s order is precedence, and every operation preserves it§11.9.7.10a device silently prefers a different access point
a lower event priority can never displace a higher one§7.14.2a Debug flood loses the Critical record a safety application depends on
a name is not answered for until 250 ms after the third probeRFC 6762 §8.1two devices both claim one name
a Timed window is consumed by the first request on it§8.7.4one Timed Request pays for a second command, later
a disallowed element present is a defect, not a spare§7.3a device advertises an attribute its feature map forbids
the OffWaitTime guard holds against a second OnWithTimedOff§1.5.6.5the lights come back on behind the person who just left the room
a groupcast is acted on and never answered§1.3.7.1.2one multicast to twenty lights draws twenty unicast replies at once
no fabric takes more than half the Scene Table§1.4.6whoever commissions first fills the device; the second ecosystem gets nothing
removing a group removes its scenes§1.3.7.4scenes accumulate for groups the endpoint has left, and nothing can reclaim them
a second KeepActive never shortens the first§9.13.6.1one controller’s request cancels most of another’s
ReachableChanged fires on the change, not on the poll§9.13.7.1a thirty-second poll pushes every other record out of the event ring
a device type’s mandatory elements are demanded, not just its clusters§9.2the app draws a scene button that does nothing
a Level Control command without On/Off does nothing while the light is off§1.6.6.9a dimmer winds a switched-off lamp up, and turning it on blinds somebody
a ‘with On/Off’ command turns the light on before the fade, not after§1.6.7.6the lamp snaps to full brightness instead of fading up
a slow Move accumulates tenths rather than rounding each tick§1.6.7.2.2a rate of 1 rounds to zero every tick and never moves
a NOC’s subject is the authority’s to choose, not the device’s§11.18a device names itself, and can name itself as somebody else
the attestation and CSR nonces must come back unchanged§11.18.6.1, §11.18.6.6a prior session’s response is replayable
a good chain does not rescue a signature that did not verify§6.2.3the chain is evidence about a key nothing proved possession of
the publisher chooses MaxInterval, and the subscriber uses it§8.5.3.2a live subscription is torn down and rebuilt for ever
a subscriber allows a retransmission past MaxInterval§8.5.4a working subscription dies on the first MRP retry
§4.5’s length prefix does not count itself§4.5.1two nodes agree until either talks to a third
a stream that lost its framing stays lost, and the connection closes§4.15.2.3the prefix is the only message boundary there is, so everything after it is read as a header
MRP does not run on a TCP peer§4.15the sender retransmits into a stream that had already delivered
BDX blocks arrive in ascending sequential counter order§11.22.6.1a gap in a firmware image nobody notices until it is flashed
a definite-length transfer ends on exactly the length it promised§11.22.6.5a truncated image is accepted as complete
a receiver-driven Sender waits for a query before sending§11.22.6.2a sleepy node is woken by blocks it never asked for
an AnnounceOTAProvider never rewrites DefaultOTAProviders§11.20.7.5any administrator on the fabric redirects every later update, including after its own access is removed
a TLS certificate an endpoint still names cannot be removed§14.4.6.7the endpoint authenticates nothing, discovered at connection time
a fabric-scoped miss in chapter 14 is NOT_FOUND, never UNSUPPORTED_ACCESS§14.4.6one administrator counts another’s certificates and learns their ids
a Large Message command is refused on a transport that cannot carry one§8.8.2.3, §14.4a 3000-octet certificate is attempted over UDP
every candidate key for a Group Session ID is tried, not just the first§4.17.3.6one group’s traffic is dropped whenever two keys collide
the group peer table is never recycled§4.16.1an attacker evicts a peer and then replays that peer’s traffic
an epoch key never comes back out of a KeySetRead§11.2.7.2any administrator walks off with the key to every group message the node sends
the root endpoint is never in a group§11.27.7.1a multicast reaches the clusters that manage the node itself
an NTL chain may not deliver more than it announced§4.21.4.2the reassembly buffer fills on a length the sender chose — found by fuzzing
a recovery node waits 120 seconds before it advertises§5.9.3the device broadcasts through every brief outage its access point has
an ICAC must answer this administrator’s CSR§11.25.6.3the anchor picks the key, and the joining ecosystem installs a CA it cannot sign with
a datastore group may not claim the Administrator or Anchor CAT§11.24.7.40xFFFE is inside the permitted range, so every member of the group gets the fabric
a thermostat with no clock has no current suggestion§4.3.7the thermostat follows a window it can no longer tell it has left

Two transcriptions of one source

The cheapest useful test in the repository, and the one that found the most: every cluster written by hand here was transcribed from the specification PDF, and every cluster in clusters::generated was produced by cargo xtask from the CSA’s machine-readable data model. The two were derived independently from the same source, so where they disagree one of them is wrong.

tests/conformance.rs runs that comparison over every hand-written cluster. It is the only thing that catches a ClusterRevision left a revision behind, an element list that does not move when its feature map does, or a mistake in the validator itself — each of those is consistent with every test written for the code that has it, which is the whole argument for keeping a second, independent description of the same thing.

tests/commissioner.rs is the third instance and the largest: the commissioner was written from §5.5 and the device’s clusters from §11.18, and the test runs one against the other through the whole flow. It found that AddTrustedRootCertificate has no response command — a commissioner expecting one fails on every conformant device — and that a certificate authority must be told whether it is a root or an intermediate, because §6.5.6.3 names them by different DN attributes and the wrong one produces certificates that are individually valid and chain to nothing.

tests/device_types.rs does the same for device types. It is what holds a validator to the elements a device type makes mandatory and not merely its clusters — an On/Off Light without TriggerEffect and CopyScene is not one — and to counting client entries, without which a light is indistinguishable from the switch that controls it.

A CI job regenerates the library from a fresh checkout of the data model and fails on any difference, so the comparison stays honest as the specification moves.

The gates

Every change runs: the suite under --all-features, under std and under --no-default-features; Clippy over every target in both modes; rustdoc with warnings denied; builds for two bare-metal targets with and without defmt; the full feature powerset; the minimum supported Rust version; a regeneration of the cluster library, diffed; licence and advisory checks; the fuzzers; a link of a whole light for an nRF52840, against a size budget; and two checks that the documents still describe the repository — every number they state (cargo xtask stats --check) and the specification index generated from the source (cargo xtask coverage --check).

Meeting other implementations

Everything above is matter-kit checking its own reading of the specification. Two things check it against somebody else’s, because a suite that only ever meets itself measures agreement rather than correctness.

interop/ depends on rs-matter 0.3.0 and compares the two crates directly on the onboarding payload (§5.1) and TLV (Appendix A) — over the published examples, over every boundary value, and over arbitrary inputs with proptest. It is a separate workspace, so rs-matter’s 217 crates never enter matter-kit’s dependency graph, its feature powerset or its MSRV.

interop/chip/run.sh has chip-tool — the controller every certified Matter product is paired by — commission examples/light through the whole of §5.5:

ReadCommissioningInfo   ArmFailSafe        ConfigRegulatory      ConfigureTCAcknowledgments
SendPAICertificateRequest                  SendDACCertificateRequest
SendAttestationRequest  AttestationVerification                   AttestationRevocationCheck
SendOpCertSigningRequest                   ValidateCSR
GenerateNOCChain        SendTrustedRootCert                       SendNOC
EvictPreviousCaseSessions                  FindOperationalForStayActive
FindOperationalForCommissioningComplete    SendComplete
                                                   → Device commissioning completed with success

The last two lines are the ones that cost the most to reach. They are not credentials: they are mDNS resolution of _matter._tcp, a CASE handshake on the identity just issued, and an interaction-model command served over the operational session on a fabric that did not exist a second earlier.

Attestation is verified, not bypassed: the device generates a §6.2.2 chain at start-up and writes its PAA out, and chip-tool is pointed at that as its trust anchor, so the DAC chain, the Certification Declaration and the NOCSR signature are all checked by somebody else’s code. AddTrustedRootCertificate and AddNOC both succeed — the crate accepts an operational identity issued by the reference implementation.

interop/chip/python.sh runs the CSA Test Harness’s own certification cases — the same TC_*.py scripts an authorised test lab executes — out of the same image:

# All eleven pass.
./interop/chip/python.sh TC_CGEN_2_1 TC_OPCREDS_3_1 TC_ACL_2_2 TC_ACL_2_4 TC_ACL_2_6 \
    TC_ACL_2_10 TC_IDM_1_2 TC_IDM_1_4 TC_IDM_2_2 TC_IDM_2_3 TC_IDM_4_2
./interop/chip/python.sh --list   # all 618

The checks you run yourself

cargo xtask cite    # every § reference and every normative quotation, against the PDFs
cargo xtask api     # every pub fn against every call site: what nothing calls
./footprint/run.sh  # link a light for an nRF52840 and read the sections out

cargo xtask api runs anywhere; the other two need something a hosted job does not have.

cargo xtask cite needs the specification PDFs, which are free from csa-iot.org and not redistributable, so they are not in the repository. It reads these guides too — a > blockquote here is a claim about the specification exactly as a /// block in the source is. It is the first thing to run before a specification uplift: a renumbering between revisions is exactly what it finds, and fifteen citations in this crate have been wrong, three of them quotations that were real but abridged past the point of being verbatim.

cargo xtask api is cheaper and blunter. A public function nothing calls is usually spare API and occasionally the missing half of a rule — four of this crate’s defects were exactly that, including a session table that answered “no space” where §4.11.1.1 requires an eviction. Its second list matters as much: called from tests or examples but from nothing in the library, which is where a rule the application has to remember shows up.

A skip is reported as a skip and exits non-zero. Most TC_*.py carry a has_attribute or has_feature decorator, so a device that does not implement the thing a case tests does not fail it — the case declines to run, and Mobly’s summary still reads Failed 0. The verdict is therefore read whole: Passed equal to Requested, and Skipped zero.

TC_OPCREDS_3_1 is the most searching of them by a long way: eighty-five steps that commission a second and a third fabric through the Enhanced Commissioning Method, roll one back with ArmFailSafe(0), reconnect over PASE and read every credential list along the way.

A case is run with the arguments it declares for itself. Every TC_*.py carries an === BEGIN CI TEST ARGUMENTS === header naming the script-args the SDK’s own CI gives it, and a large minority add a --endpoint or a PIXIT that the case refuses to run without. The runner parses that header and passes it through, minus the flags it sets itself — and minus --enable-spec-errata-ci-only-disallowed-for-certification, which relaxes a case in a way its own name says is inadmissible.

Each case commissions the device itself over on-network discovery and opens a wildcard subscription before its first assertion, so every run exercises mDNS, PASE, attestation, CASE and the interaction model on a fabric that did not exist a moment earlier.

Both ends run in containers on an IPv6 Docker network, so it needs no radio, no second machine and no C++ toolchain. It is the only gate here that has ever found a defect the others could not, because every other gate has matter-kit on both ends — and two implementations of the same misreading agree perfectly. Three patterns account for most of what it finds:

  • Correct for one of something, wrong for two. One fabric, one subscriber, one report cursor. A fixture with one of a thing never exercises the code that has to tell two apart.
  • A library piece whose documented caller does not exist. Every part built, tested, and joined to nothing. The symptom is silence, not an error.
  • A fixture that pins what production varies. Three fixed key seeds hid a certificate serial number that was invalid for one key in 256.

The runner itself is held to the same standard, and has been wrong in the same ways — most sharply by counting a skipped case as a pass, which is why the verdict is now read whole.