Change Log

All notables changes to this project are documented in this file.

The format is inspired by Keep a Changelog.

24.3 (2024-10-14)

Protocol	Version
Consensus	7.0.0
Runtime Host	5.1.0
Runtime Committee	5.0.0

Configuration Changes

The pathbadger storage backend is now the default for new nodes (#5876)

When a node is provisioned into an empty data directory it will default to using the pathbadger storage backend.

For existing nodes, the storage backend is automatically detected based on the data directory. When multiple storage directories exist, the one most recently modified is used.

In case one does not want this new behavior, it is still possible to set the storage.backend to badger/pathbadger to explicitly configure the desired storage backend and disable autodetection.

Features

go/runtime: Support bundle components without ELF binary (#5838)
Add support for verifying Intel TDX quotes (#5840)
Add QEMU-based Intel TDX provisioner (#5861)
runtime/src/enclave_rpc: Support peer feedback for concurrent requests (#5872)
go/storage: Add automatic storage backend detection (#5876)

The new default storage backend is "auto" which attempts to detect the storage backend that should be used based on existing data directories. When none exist, "pathbadger" is used. When multiple exist, the most recently modified one is used.

This should make newly deployed nodes default to pathbadger.
go/common/grpc: allow non-tls connections to loopback addresses (#5878)
runtime/src/enclave_rpc/client: Support concurrent sessions (#5879)
go/runtime/client: Expose state proofs via gRPC (#5888)

Bug Fixes

runtime/src/dispatcher: Propagate panics during status/policy update (#5851)

Internal Changes

buildkite: Enable debug assertions in runtimes (#5836)
go: Bump go-libp2p to 0.36.3 (#5844)
keymanager: Add mock trusted signers for debug mock SGX builds (#5852)
go: Bump github.com/dgraph-io/badger to v4.3.1 (#5856)
keymanager/src/client: Fetch churp key shares concurrently (#5863)
keymanager: Expose runtime ID and RSK in the key manager client (#5865)
runtime: Remove obsolete feature rpc_peer_id (#5867)
go: Bump go-libp2p to 0.36.5 (#5890)
go: Bump CometBFT to 0.37.11-oasis1 (#5891)
secret-sharing/src/churp/switch: Verify combined bivariate polynomial (#5897)

After all bivariate shares are collected and the switch either creates a new shareholder or proactivates the share of an existing one, the new share should be verified to ensure that the verification matrix of the combined bivariate polynomial satisfies the non-zero leading term requirements.

24.2 (2024-08-20)

Protocol	Version
Consensus	7.0.0
Runtime Host	5.1.0
Runtime Committee	5.0.0

Configuration Changes

Support only entity_id to be specified in configuration (#5764)

Instead of the need to specify the path to the entire entity descriptor, the node configuration can now specify only registration.entity_id, the entity public key.

Features

keymanager/src/churp: Fetch key shares and recover key (#5715)
go/consensus: add GasUsed to transaction results and Size to block header (#5761)
go/worker/keymanager/churp: Add metrics (#5773)

The following metrics have been added:
- oasis_worker_keymanager_churp_threshold_number is the degree of the secret-sharing polynomial.
- oasis_worker_keymanager_churp_extra_shares_number is the minimum number of extra shares.
- oasis_worker_keymanager_churp_handoff_number is the epoch number of the last handoff.
- oasis_worker_keymanager_churp_next_handoff_number is the epoch number of the next handoff.
- oasis_worker_keymanager_churp_handoff_interval is the handoff interval.
- oasis_worker_keymanager_churp_committee_size is the number of nodes in the committee.
- oasis_worker_keymanager_churp_submitted_applications_total is the number of submitted applications.
- oasis_worker_keymanager_churp_confirmed_applications_total is the number of confirmed applications.
- oasis_worker_keymanager_churp_enclave_rpc_latency_seconds is the latency of enclave rpc calls.
- oasis_worker_keymanager_churp_enclave_rpc_failures_total is the number of failed enclave rpc calls.
go/upgrade/migrations: Prepare handler for version 24.2.0 (#5777)
go/runtime: Automatically enable all configured detached components (#5785)

Since they are explicitly configured there should be no need to enable them twice. This just defaults detached components to be enabled and one needs to explicitly disable them.
go/oasis-node: Add identity show-address subcommand (#5820)

This subcommand makes it easier to get the address associated with the node identity public key.

Bug Fixes

go/runtime/registry/host: Ignore key manager quote policy update feature (#5759)

If the key manager policy and status update watcher started before the runtime active version was ready, it failed to fetch the runtime info and stopped. Therefore, the key manager status and quote policy were never updated, causing the key manager runtime client to reject incoming Noise session requests since the policy was not set.
go/runtime/bundle: Use manifest hash at bundle load time (#5785)
go/runtime: Add timeout for HostSubmitTx from ROFL apps (#5813)
go/runtime/config: Fix incorrect error message for invalid prune config (#5821)

Internal Changes

go/consensus: Add the latest consensus-breaking software feature version (#5760)

Useful for enabling new consensus-breaking features with upcoming releases. A feature is enabled if the latest consensus-breaking software feature version is not lower than the version in which the feature is to be enabled.
Bump Rust toolchain to 2024-07-09 (#5770, #5772)
go: Bump CometBFT to 0.37.9-oasis1 (#5782)
secret-sharing: Make sensitive functions constant time (#5800)
go/common/sgx: Make quote policy YAML serializable (#5804)

24.1 (2024-06-28)

Protocol	Version
Consensus	7.0.0
Runtime Host	5.1.0
Runtime Committee	5.0.0

Features

secret-sharing/src/churp: Implement key derivation center (#5698)
keymanager/src/churp: Implement key derivation function (#5699)
Support detached ROFL components (#5707)

Previously each bundle that contained one or more ROFL components also needed to contain the exact version of the RONL component it was attaching to.

This is somewhat awkward to use when we assume a more decentralized development and deployment of ROFL applications. This commit adds support for detached ROFL components where the bundle only contains the ROFL and oasis-node then automatically gets the appropriate RONL component from another bundle.

Bug Fixes

go/worker/common: Always take the latest epoch for deployments (#5693)

Fix for an issue which could have prevented runtimes from being initialized in case the runtime was suspended before the first node was started.
go/worker/common: Retry provisioning on epoch transitions when suspended (#5694)
go/runtime/host: Ignore stale abort requests (#5702)
keymanager: Increase MAX_FRESH_HEIGHT_AGE (#5703)

This avoids issues where key managers get out of sync during operation and start rejecting otherwise valid requests. It should still be safe especially since all new runtimes now perform freshness checks.
go/runtime/host/sgx: Fix SGX bundle validation (#5706)
go/registry/api: update ProveFreshness tx body type (#5708)
go/worker/storage: Do not add any particular roles (#5725)

Since the storage node is always coupled with another role, make sure to not add any particular role as otherwise this could cause observer nodes to also register as compute nodes and then misbehave.

Internal Changes

go: Reverse order of TCB update fetching (#5704)

Early TCB updates are now tried first when updating TCB info. If validation for an early update fails, the mechanism falls back to the standard update.
runtime: Add VerifiedAttestation with more metadata (#5728)

Since verified attestations generated from TEE capabilities can include additional metadata like the enclave's view of consensus layer height at time of attestation, this allows such data to be used by callers.
go: Bump CometBFT to 0.37.7-oasis1 (#5730)

24.0 (2024-05-13)

Protocol	Version
Consensus	7.0.0
Runtime Host	5.1.0
Runtime Committee	5.0.0

Configuration Changes

Add num_light_blocks_kept configuration option (#5466)

Located under consensus.prune, it allows configuring the number of light blocks that are kept in the local trusted store (defaulting to 10000).
Remove may_generate configuration option (#5530)

Key manager worker no longer needs this configuration option, previously located under worker.keymanager, as master secrets can be generated by all key manager committee nodes.
go/consensus: Change default sync trust period to 30 days (#5665)
config: Add option to override internal unix socket path (#5427)

Previously the UNIX socket path could only be overriden via a debug-only option. Since this option can be generally useful in production environments it is now supported in the config file. The socket path can be set under common.internal_socket_path, and is not considered a debug option anymore.

Features

go/storage/mkvs: Add PathBadger storage backend (#5559)

Instead of using trie node hashes as keys in the underlying Badger store, this new backend instead uses a combination of version and index within the batch of trie nodes as keys which leads to improved locality when iterating over the trie while at the same time making the database smaller and compactions faster.

The new backend makes some (reasonable) assumptions, specifically that only one root per type may be finalized in any version and that there may be no child roots within the same version.

The new backend is experimental and can be enabled by setting the configuration option storage.backend to pathbadger. Note that this will start with an empty database so will require a complete resync.
Implement ROFL (Runtime OFf-chain Logic) (#5602)

This extends runtimes with support for arbitrary off-chain logic that uses the existing decentralized execution, distribution and remote attestation infrastructure provided by Oasis. See ADR 24 for more details.

Use cases include oracles, bridges, light clients, complex data workloads, intent solvers, autonomous AI agents, etc.
Implement proactive secret sharing in key manager (CHURP) (#5551, #5562, #5568, #5569, #5570, #5572, #5575, #5590, #5592, #5616, #5617, #5628, #5640, #5653, #5658, #5663, #5684)

This adds support for proactive secret sharing in the key manager runtimes, which makes it possible for key manager nodes to only hold shares of keys in encrypted memory. Even if the TEE is compromised on a given node, only a single share can be recovered, opening the way for storing secrets with even stronger security guarantees.

Client-side support for share reconstruction will be part of an upcoming release.
go/governance: Support for allowing voting without an entity (#5644)
go/governance: Add support for proposal metadata (#5650)
go/consensus: Add global minimum gas price (#5657)
go/vault: Add simple consensus layer vault (#5662)

The vault service is a simple multi-sig where multiple parties vote to perform actions on behalf of the vault account. This feature is disabled by default and needs to be enabled via a governance vote.
go/consensus/roothash: Filter executor commitments by runtime ID (#5436)

Compute executor committee workers no longer have to verify the signatures of observed commitments simply to identify them as invalid.
go/oasis-node/cmd/genesis: Make attestation age/interval configurable (#5449)

A new flag registry.tee_features.sgx.default_max_attestation_age was added to the genesis command to specify the default maximum attestation age when SGX RAK-signed attestations are enabled. Additionally, within the runtime registry configuration, one can now set the attestation interval for periodic runtime re-attestation.
runtime: Add roothash round roots state wrappers in rust (#5457)
go/roothash: expose RoundRoots in the roothash client (#5467)
Build EPID and DCAP versions of the AESMD docker image (#5496)
runtime/sgx/pcs: Make useful attestation verification methods public (#5503)
go/storage: Add storage metrics for GetDiff method (#5512)
runtime: Implement MKVS storage proof builder (#5514)

Introduces get_proof method to retrieve MKVS proofs for specific tree entries.
go/runtime/host: Add separate metric for RHP call timeouts (#5519)
go/runtime: Add load-balancer runtime provisioner (#5528)
Support for Proofs Without Implicit Internal Leaf Nodes (#5531)

Previously, internal MKVS nodes in proofs included full leaf nodes implicitly. With this update, leaf nodes are explicitly added as regular child nodes within the proof structure. This modification optimizes proof sizes by avoiding inclusion of potentially large values associated with leaf nodes that are not directly relevant to the proof's target node.

This change maintains backward compatibility. Existing proofs are unmarshaled as version 0, while version 1 proofs adopt the new scheme.
go/oasis-net-runner: Add ability to set node log level and format (#5534)

Previously, every node started by the net runner had the default log level of debug, while now it's possible to set it to other levels. The log format can also be changed.
go/runtime/host: Add GetActiveVersion to Runtime interface (#5535)
go/worker/keymanager: Authorize noise session connect calls (#5539)

A peer is granted permission to connect if it is authorized to invoke at least one secure enclave RPC method.
go/worker/keymanager/p2p: Remove retries and sticky peers (#5546)

Since retry and peer selection is now handled in the runtimes having it also done outside is detrimental to latency. The runtime knows better when to actually retry and which peers to select.
go/common/keyformat: Ensure prefixes are unique per namespace (#5548)
sgx: Support early updates for ECDSA TCB infos (#5578)
go/upgrade: Adjust MaxTxSize and MaxBlockSize in consensus240 handler (#5588)

This is needed as DCAP quotes are larger and nodes running multiple confidential runtimes may otherwise exceed the max transaction size.
go/runtime/client: Add the GetUnconfirmedTransactions method (#5589)

Similarly to GetUnconfirmedTransactions in the consensus API, this new method returns the currently pending runtime transactions from the runtime transaction pool.
go/worker/storage: Make fetch pool per-runtime (#5595)

This should speed up storage sync in case of nodes that have multiple runtimes configured.
Add observer nodes automatically to the keymanager's access list (#5606)

Observer nodes for a given paratime had to be added manually. This change brings observer nodes in line with compute nodes, which were added automatically.
go/staking: Read token information from state when available (#5648)
go/upgrade: Support upgrades without stopping the node (#5659)

Bug Fixes

go/oasis-net-runner: Fix SkipPolicy flag in default fixture (#5394)
go/worker/storage: Disable storage worker when no runtimes configured (#5394)
go/worker/keymanager: Fix race condition when accessing runtime status (#5398)
keymanager: Fix public key decoding for legacy key manager clients (#5398)
go/worker/common: Refresh current epoch for suspended runtimes (#5400)
go/worker/client: Fix nil dereference on early Query (#5403)
go/worker/compute/executor: Use local time for batch scheduling (#5404)
go/worker/compute/executor: Estimate pool rank from observed commitments (#5404)
go/worker/compute/executor: Start processing once all txs are fetched (#5404)
go/worker/compute/executor: Schedule only if higher ranks didn't propose (#5404)
Fix stuck control status on runtime nodes before initialization (#5407)
Fix showing empty peer IDs in the Consensus light client status output (#5408)
config/migrate: Automatically configure external P2P addresses for validators (#5410)
go/consensus: Do not crash on nil result from Commit (#5423)

The Commit function can return both a nil error and a nil result in case the given block is not available yet.
go/worker/compute/executor: Clear channels when not in the committee (#5426)
go/runtime/host/multi: Propagate special requests to next version (#5433)

Previously periodic consensus sync requests were not propagated to the next (e.g. upcoming) runtime version. This could result in the runtime's consensus view going stale which would make the attestations too old so they would be rejected during scheduling.

Additionally, key manager update requests should also be propagated to ensure the runtime is ready immediately when activated, avoiding any potential race conditions.
go/runtime/host/sandbox: Release lock before calling into runtime (#5438)

Similar to how this is handled in the multi runtime host, we need to release the lock before calling into the runtime as otherwise this could lead to a deadlock in certain situations.
go/worker/compute: Bound batch execution time (#5438)
go/runtime/host/protocol/connection: Cancel call if connection is closed (#5442)
go/worker/compute: Use correct context when aborting runtime (#5446)
go/worker/compute: Also abort in case deadline exceeded (#5446)
go/worker/compute/executor: Propose promptly upon detecting discrepancy (#5447)
go/runtime: Fix zombie channel pipe leak on runtime restarts (#5447)

Pipes created by a call to channels.Unwrap spawned new goroutines that were not terminated during runtime restarts. These zombie pipes also intercepted one value from the newly created pipes, causing them to block indefinitely.
go/runtime/host/multi: Release lock early to avoid blocking (#5448)
runtime: Create controller in RpcClient constructor (#5450)

Previously if no RPC calls were initiated by the runtime, the client controller task was never spawned which caused quote policy update requests to pile up in the command queue, eventually blocking the entire runtime from processing requests.

Since the async runtime is now available early on during initialization, we can spawn the controller in the RpcClient constructor, avoiding these problems.
go/worker/compute: Abort runtimes only on timeouts (#5453)
go/p2p: Increase incoming connection limit for seed nodes (#5456)
go/p2p: Close connection to seed node after every request (#5456)

Bootstrap client, which is responsible for peer discovery and advertisement, now terminates connection to the seed node after every request. This action should free up recourses (e.g. inbound/outbound connections) on both sides without affecting performance since discovered peers are cached (see retention period) and advertisement is done infrequently (see TTL).
runtime: Improve error reporting if DeoxysII unsealing fails (#5458)

Previously, if the CPU changed between runs of the Oasis node, the error reported was a cryptic "ciphertext is corrupted" (because the sealed SGX secrets were invalidated). Now we add a bit more context to make it easier for the end-user.
go/consensus/cometbft/light: Don't crash when signed header unavailable (#5462)
go/consensus/cometbft/light: Try multiple sources when fetching blocks (#5466)
go/p2p/peermgmt/backup: Prevent overwriting TTL when restoring peers (#5469)

If the peer address of a seed node was added to the libp2p address book before peer manager restored backup peer addresses, its permanent TTL was replaced with the TTL for recently connected peers.
governance: Fix pretty priting of ChangeParametersProposal (#5472)
go/oasis-node: Remove obsolete db section during config migration (#5473)
go/p2p/discovery: Close only idle connections to seed node (#5476)
governance: fix delegator-validator vote match check (#5479)
go/p2p/peermgmt: Find peers and connect only when needed (#5480)

If we are already connected to a sufficient number of peers for a given topic or protocol, there's no need to retrieve additional peers from the registry or the seed node.
go/consensus/cometbft/light: Only fetch from light store for now (#5481)

In practice the previously introduced fetch from light client caused the light client to fall back to slow backwards verification due to target blocks being in the past, below the pruning window.
rhp: don't prepend 'failed to read response' to runtime module errors (#5488)
storage/checkpoints: Ignore i/o root in genesis checkpoint (#5492)
go/genesis: Suppress misleading genesis sanity check logs (#5495)
metrics: fix labels for disk read and write metrics (#5510)

Fixes oasis_node_disk_written_bytes and oasis_node_disk_read_bytes which were mistakenly reversed.
go/runtime/transaction: Use node database directly when possible (#5520)

Previously accessing the transaction artifacts tree would always use the slower read syncer interface meant for communicating with untrusted db instances. This is now short-circuited in case a local db is available.
go/worker/keymanager: Fix race conditions when accessing status fields (#5529)
Include the mode of the node in control status output (#5537)
go/runtime/txpool: Remove rechecked transactions from seen cache (#5542)

In case a transaction is rejected because it fails a re-check pass, it should also be removed from the seen cache as it may be resubmitted later when it could become valid.
go/worker/client: Fix observer node registration (#5545)

Previously a node configured as an observer node would forget to register for all of its configured runtimes, causing the registration to fail.
governance/ChangeParametersProposal: Display only changed parameters (#5566)

Omit the unchanged parameter values when pretty-printing the ChangeParameterProposal.
runtime: Verify freshness with RAK against latest state (#5567)
go/runtime/host: Wait for readiness instead of failing immediately (#5577)
go/consensus/cometbft/beacon: Fix GetEpochBlock implementation (#5605)
go/archive: fix runtime queries on archive nodes (#5622)

Fixes storage worker initialization on archive nodes which was causing runtime queries to fail.
runtime: Add missing support for per-role admission policy decoding (#5623)
go/runtime/txpool: Don't abort runtime if node is not synced yet (#5630)

If the node hasn't finished syncing, checkTxBatch previously caused the runtime to be aborted, even though it wasn't the runtime's fault. Now the checks are retried after the node is finished syncing.
go/worker/compute/executor/committee: Retry scheduling on failure (#5660)
keymanager: Allow one epoch in the future during validation (#5686)

This avoids an issue where a key manager node that is slightly behind would return an error during an epoch transition.

Documentation Improvements

License the documentation under CC BY 4.0 (#5509)
docs: Fix broken link anchors (#5515)
docs: Fix broken link to CometBFT metrics page (#5550)

Internal Changes

go: Bump go to 1.22.2 (#5556, #5586, #5626)

Also updates go tooling:
- golangci-lint to 1.56.1
- gofumpt to 0.6.0
- goimports to 0.18.0
go: Bump github.com/quic-go/quic-go to v0.42.0 (#5625)
go: Bump go-libp2p to v0.33.2 (#5625)
go/common/sync: Add IsRunning function to sync.One (#5536)
keymanager/src/runtime: Remove legacy version of init request (#5205)
makefile: Exit with an error if go mod tidy fails (#5270)
go: Bump golang.org/x/net to v0.17.0 (#5403)
Install setuptools to make Towncrier fork work with Python 3.12 (#5421)
go/common/sgx: implement GetPCKCertificateChain PCS API client (#5465)
Bump Rust toolchain to 2024-03-04 (#5513, #5583)
go: Bump go-libp2p to 0.32.2 (#5521)
runtime: Increase number of processing threads in SGX (#5523)
rust: bump shlex to v 1.3.0 (#5533)

RUSTSEC-2024-0006
go/p2p/rpc: Remove support for sticky peers (#5546)
runtime/enclave_rpc: Support caller to provide peer feedback (#5546)
go/worker/compute: Simplify I/O root commit (#5553)

This also avoids an intermediate committed IO root which complicates the required database layout.
keymanager/runtime: Use insecure RPC requests for ephemeral public keys (#5554)
runtime/src/enclave_rpc: Simplify RPC context (#5555)

The RPC context now contains only essential data for secure RPC methods. Identity, consensus verifier, and storage have been removed and are now available to methods responsible for master and ephemeral secrets after initialization.
docker: Bump cargo-tarpaulin to 0.27.3 (#5558)
runtime/src/storage: Move untrusted in-memory key/value storage (#5561)

The untrusted in-memory key/value storage is now accessible to all tests.
runtime: Bump tendermint-rs to 0.35.0 (#5604)
ci: Update SGX tests to run DCAP (#5610)
go: Make sure packages compile on Windows (#5614)
go: Bump CometBFT to 0.37.6-oasis1 (#5615, #5673)
Build runtime binaries in release mode (#5641)

Running SGX tests with binaries built in debug mode can be extremely slow, so build everything in release mode. This should also speed up E2E tests.
Add support for mock SGX builds (#5642)

This makes it easier to test various features even when SGX hardware is not available.
.github/dependabot: group rust dependency updates (#5654)

23.0 (2023-10-10)

Protocol	Version
Consensus	7.0.0
Runtime Host	5.1.0
Runtime Committee	5.0.0

Removals and Breaking Changes

go/registry: Remove support for deprecated PVSS based random beacon (#4394)

Remove the backward compatibility placeholder DeprecatedBeacon field from the node descriptor. The PVSS backend is no longer present in 22.x and so the field is now removed, and even genesis registrations without a VRF signing key will be rejected.
go/consensus/tendermint/apps/beacon: Do gas accounting earlier (#4667)

The VRF proof submission tx has gas use happen too late, and can in certain circumstances underestimate the amount of gas the tx will cost.
go/common/crypto/signature: Use ECVRF v16 (#4668)

The IETF draft was updated, so use the newer method of calculating proofs. This is incompatible with the v10 proof derivation/verification, however beta values for a given input will be identical.
go/staking/api: Add burn address (#4755)

Transfers to oasis1qzq8u7xs328puu2jy524w3fygzs63rv3u5967970 will be treated as an explicit token burn (transfer amount deducted from the source's general balance and destroyed).

The private key of the address is unknown, and the address is marked as reserved so the address can not be used as the source address for any transactions.
go/consensus/tendermint: Fix liveness tracking for primary+backup (#4798)

In case a node is both in primary and backup compute committee and doesn't submit a commit but there was no discrepancy, the node should still be treated as failed.
go/registry: Validate SoftwareVersion field in node descriptor (#5012)
go/staking/api: Specify slashed debonding amount in TakeEscrowEvent (#5016)

The event now contains the DebondingAmount field.
go/roothash/api: Support governance actions via roothash messages (#5021)

This allows runtimes to vote on and submit consensus governance proposals.
go/staking: Make DelegationsFor() query efficient (#5023)

A reverse delegation mapping is added to the staking state that makes querying outgoing delegations efficient.
go/p2p: Include chain context in p2p protocol names (#5025)

Chain context was included in p2p protocol and topic names as until now it was impossible to distinguish Mainnet and Testnet names from each other. Unique names will also ease peer discovery as now we can use one seed node for multiple networks.
governance: Support delegator votes (#5034)

Adds support for delegators to vote and override the validator votes as described in ADR 0020.
go/common/sgx/pcs: Add support for blacklisting FMSPCs (#5038)

PCS quote policy now supports blacklisting of FMSPCs.

Quotes for systems with blacklisted FMSPC description of the TCB won't get verified and remote attestation will fail.

This is specially useful if we want to block CPUs or platforms for which a new security vulnerability has just been disclosed from running a runtime enclave.

FMSPCs can be added or removed from the global default quote policy by changing registry parameter TEEFeatures via change parameters proposals. The blacklist can also be overridden per-runtime in SGX constraints.
go/scheduler/api: Add EntityID to consensus validator's state (#5050)

Validator lists in scheduler consensus state now include entity and node identifiers.
go/consensus: Always return ErrInvalidArgument on tx deserialization fails (#5055)

Previously, some modules were returning non-specific errors on tx deserialization failures, which resulted in them being assigned to module "unknown", code 1.
go/staking: Limit maximum allowance amount based on total supply (#5056)

Request for updating allowance above total supply now fails with ErrAllowanceGreaterThanSupply.
go/consensus/keymanager: Skip expired nodes (#5066)
go/consensus: Implement libp2p backed light client sync protocol (#5085)
go/consensus: Remove consensus.LightClientBackend (#5085)

All methods were merged into the consensus.ClientBackend.
go/registry: Rename RuntimeEvent event to RuntimeStartedEvent (#5088)
go/registry: Add RuntimeSuspendedEvent event (#5088)

The event is emitted whenever an active runtime is suspended. On resume, the existing RuntimeStartedEvent is emitted.
go/registry: Remove TLS addresses from node's descriptor (#5100)

As of #5085, the TLS addresses are unused. Albeit also unused, the TLS public keys remain par of the node's descriptor.
go/registry: Remove Consensus RPC role from node's descriptor (#5100)

As of #5085, the role is unused.
go/staking: Add MinCommissionRate parameter (#5102)

This parameter enforces the network-wide minimum commission rate an account can configure.

Also include support for updating the parameter via ConsensusParameterChanges governance proposal.
go/staking: Add escrow requirement for updating commission schedule (#5102)

To prevent bloating the commission schedule state, the commission schedule can only be updated for entities with enough stake to register a validator node.
go/registry: Add optional bundle checksum to runtime deployments (#5112)

Currently, it is not used, but it could be used for verification of automatically distributed runtime bundles in the future.
go/sgx/ias: Add support for blacklisting GIDs (#5113)

GID black list is a list of blocked platform EPID group IDs.
go/registry: Support changing a runtime's owner (#5114)

The current owner can sign and submit a registry.RegisterRuntime txn which sets a new entity as the owner of the runtime.
go/common/node: Cleanup node's VRF configuration as it is mandatory (#5127)

Simplify types by removing the ability for a node's VRF to be nil. Note that in practice (e.g. in all existing deployments) it was already mandatory as registering a node without it set would always fail.
go/keymanager/api: Move key manager gas costs (#5166)

Consensus parameters were added to the key manager state and key manager gas costs were moved from the registry state to the key manager state.
go/roothash: Treat failure indications as stragglers (#5261)

If it is already known that the number of valid commitments will not exceed the required threshold, there is no need to wait for the discrepancy timer to expire. Instead, proceed directly to the discrepancy resolution mode, regardless of any additional commitments.
go/roothash: Optimize and refactor commitment pool processing (#5274)

The commitment pool processing has been optimized and refactored to improve code readability:
- The discrepancy detection has been modified to immediately switch to the resolution mode when two commits differ, eliminating the necessity to wait for the proposer's commitment.
- The discrepancy resolution process was redesigned to fail as soon as it becomes evident that no group of votes can attain the majority, such as when there are too many failures.
go/roothash/commitment: Refactor executor commitment header (#5282)

The embedded compute results header struct has been changed to a field.
go/upgrade: Check all pending upgrades if we reached the upgrade epoch (#5287)
go/consensus: Add block metadata to allow same-block state validation (#5292)
go/consensus: Add provable events (#5294)

Provable events are event types which can be proven and included in the event proof tree.
go/consensus/roothash: Track runtime proposer liveness (#5334)

The roothash application now monitors the runtime proposer liveness, which runtimes can utilize to penalize proposers with insufficient commitments. To activate penalties for such nodes, the executor committee parameters need to be updated by configuring the following setting:
- MaxMissedProposalsPercent: The maximum percentage of proposed rounds in an epoch that can fail for a node to be considered live. Zero means that all proposed rounds can fail.
go/staking: Enable changing the reward schedule (#5352)

Add ability to change the staking reward schedule through a governance vote.
go/worker/compute/executor/committee: Support backup proposers (#5354)

All executor committee workers are now permitted to schedule transactions, each with distinct per-round priority. Priority dictates the time after which a worker can propose a new batch. The consensus layer tracks all published executor commitments and tries to build a new runtime block on a proposal with the highest priority.
go/common/node: Add observer node role (#5357)

The observer role is meant for so called client or read-only nodes that follow the chain (consensus and/or runtimes) and can also access the confidential state.

Previously, such nodes didn't register and access to confidential state was guarded by key managers white-listing specific client nodes.

With the introduction of observer node role, confidential state access is now transparently managed in-protocol, allowing confidential observer nodes to register and be subject to the usual constraints.
go/roothash/api: Store past runtime state and I/O roots in consensus state (#5359)

Add MaxPastRootsStored roothash consensus parameter which specifies the maximum number of past runtime state and I/O roots to store in the consensus state.

This enables easier cross-runtime communication.
go/oasis-node/cmd: Move debug fix-genesis command to genesis migrate (#5360)
go/scheduler/api: Add ability to change voting power distribution (#5365)

The voting power distribution can now be changed via consensus governance.

The two options currently available are:
- VotingPowerDistributionLinear (voting power is linearly proportional to the stake, i.e. the existing distribution),
- VotingPowerDistributionSqrt (voting power is proportional to the square root of the stake, a new option added now).

Configuration and Operational Changes

go/consensus/tendermint: Refactor seed node and peer addresses (#4964)

Previously, a seed node was a Tendermint backend-specific construct which only supported a single service. To provide support for other services, the seed node has been refactored and peer addresses have been generalized.

This brings the following configuration changes.

Seed nodes:
- The consensus.tendermint.mode setting was removed.
  
  To configure a node to be a seed node, set the new top-level mode setting to seed.
Non-seed nodes:
- The consensus.tendermint.seed setting was removed and replaced with p2p.seeds setting.
- The consensus.tendermint.p2p.unconditional_peer_ids setting was renamed to consensus.p2p.unconditional_peers.
General:
- Persistent peers and seed nodes' addresses were generalized and now have the following form: pubkey@IP:port, where pubkey represents a node's p2p public key (the previous form was ID@IP:port where ID represents a node's Tendermint ID which is a truncated sha256 hash of a node's p2p public key).
  
  This affects the p2p.seeds, consensus.p2p.persistent_peers and consensus.sentry_upstream_addresses settings.
- Similarly, peers are now identified through p2p public keys which affects the consensus.p2p.unconditional_peers setting (previously, they used Tendermint IDs).
Tendermint-specific configuration of a seed node stayed the same. The node can still be tuned through consensus.p2p.* settings.
go/worker/common/p2p: Refactor P2P package (#4996)

In preparation for extending the use of our P2P network to consensus-related services (previously, it was only used for runtime services), move the P2P package to the root of the hierarchy and rename its configuration settings.

The following changes were made to the p2p configuration:
- The worker. prefix has been dropped.
- Settings pertaining to the same feature were grouped under the same prefix.
- The worker.client.addresses setting was removed (it is no longer needed).
Below is the list of modified settings that can be used to configure the P2P network:
- p2p.port
- p2p.registration.addresses
- p2p.gossipsub.peer_outbound_queue_size
- p2p.gossipsub.validate_queue_size
- p2p.gossipsub.validate_concurrency
- p2p.gossipsub.validate_throttle
- p2p.connection_manager.max_num_peers
- p2p.connection_manager.peer_grace_period
- p2p.connection_manager.persistent_peers
- p2p.connection_gater.blocked_peers
- p2p.peer_manager.connectedness_low_water
go/oasis-node: Always start libp2p node (#4995)

As validators now participate in the P2P network, the configuration of publicly routable p2p addresses through worker.p2p.addresses setting is now mandatory in a production setting.
go/consensus: Use libp2p-backed light client for Tendermint state sync (#5085)

Remove the consensus.tendermint.state_sync.consensus_node setting as it has become obsolete. P2P peers for state sync are automatically selected.
Rework configuration handling and apply consistent setting naming (#5070)

The node now supports a configuration file that holds all the settings that were previously available via command-line options.

The command-line options themselves have been removed in favor of the config file.

The config file supports environment variable substitution using the envsubst package (usage information).

The following settings have been removed:
- consensus.tendermint.mode in favor of using the global node mode (mode).
- runtime.mode in favor of using the global node mode (mode).
- worker.client.port as it is no longer needed.
- worker.registration.force_register as it has been deprecated.
The following settings have been renamed:
- datadir to common.data_dir.
- log.file to common.log.file.
- log.format to common.log.format.
- log.level to common.log.level.
  
  The log level is now a map of module to log level.
  
  Use the default module to map the default log level.
- debug.rlimit to common.debug.rlimit.
- debug.allow_root to common.debug.allow_root.
- pprof.bind to pprof.bind_address.
- consensus.tendermint.* to consensus.*.
- consensus.tendermint.core.listen_address to consensus.listen_address.
- consensus.tendermint.core.external_address to consensus.external_address.
- consensus.tendermint.log.debug to consensus.log_debug.
- consensus.tendermint.light_client.trust_period to consensus.state_sync.trust_period.
- consensus.tendermint.seed.debug.disable_addr_book_from_genesis to consensus.debug.disable_addr_book_from_genesis.
- consensus.tendermint.sentry.upstream_address to consensus.sentry_upstream_addresses.
- consensus.tendermint.upgrade.stop_delay to consensus.upgrade_stop_delay.
- consensus.tendermint.supplementarysanity.* to consensus.supplementary_sanity.*.
- consensus.tendermint.p2p.persistent_peer to consensus.p2p.persistent_peers.
- consensus.tendermint.p2p.unconditional_peer to consensus.p2p.unconditional_peers.
- ias.proxy.address to ias.proxy_addresses.
- ias.debug.skip_verify to ias.debug_skip_verify.
- runtime.sandbox.binary to runtime.sandbox_binary.
- runtime.sgx.loader to runtime.sgx_loader.
- runtime.history.pruner.* to runtime.history_pruner.*.
- worker.sentry.addresses to runtime.sentry_addresses.
- worker.tx_pool.* to runtime.tx_pool.*.
- worker.keymanager.* to keymanager.*.
- worker.keymanager.runtime.id to keymanager.runtime_id.
- worker.registration.* to registration.*.
- worker.sentry.* to sentry.*.
- worker.sentry.control.authorized_pubkey to sentry.control.authorized_pubkeys.
- worker.storage.* to storage.*.
- worker.storage.public_rpc.enabled to storage.public_rpc_enabled.
- worker.storage.checkpoint_sync.disabled to storage.checkpoint_sync_disabled.
go/oasis-node/cmd/config: Support configuration file migration (#5237)

To migrate a node's YAML config file from the old (i.e. Oasis Core 22.2.x) to the new (i.e. Oasis Core 23.0) format, you can use the newly added oasis-node config migrate command.

Pass the path to the old configuration file with --in flag and specify the desired path to the new configuration file with --out flag, for example:
```
oasis-node config migrate --in old.yaml --out new.yaml
```
The oasis-node config migrate command logs the various changes it makes and warns if a config option is no longer supported, etc.

At the end, any unknown sections of the input config file are printed to the terminal so one has a chance to review them and make the manual changes (if needed).

Note that the oasis-node config migrate command does not preserve comments and the order of sections in the inputted YAML file. One needs to copy/amend the new configuration file manually.

One should always carefully read the output of this command, as well as compare the generated config file with the original before using it.
go/genesis: Move halt epoch from genesis to node-local configuration (#5200)

It can be configured through the new consensus.halt_epoch setting.
go/oasis-node/cmd/debug: Update `oasis-node debug fixgenesis`` command (#5286)

Make the oasis-node debug fixgenesis command update a few more consensus parameters.
go: Transition to CometBFT, a fork of Tendermint (#5033, #5268, #5280, #5291, #5299, #5306)

Transition to a more maintained fork of the Tendermint BFT protocol. Also update Tendermint / CometBFT version to 0.37.2.

As part of the transition, use more generic data directory names. The following subdirectories under the node's data directory have been renamed:
- tendermint to consensus
- tendermint/abci-state to consensus/state
- tendermint-seed to seed (on seed nodes only)
go: Remove TLS certificate rotation (#5318)

We use libp2p for all communication now, so TLS certificate rotation is no longer needed.

Deprecations

go/oasis-node/cmd: Deprecate oasis-node subcommands in favor of Oasis CLI (#5376)

The following oasis-node subcommands have been deprecated in favor of the new Oasis CLI and will be removed in a future Oasis Core release:
- consensus *
- control runtime-stats
- debug bundle *
- governance *
- keymanager *
- registry *
- signer *
- stake *

Features

go/p2p: Bootstrap libp2p peers using seed nodes (#4981)

Seed nodes can now bootstrap libp2p peers. Bootstrapping can be enabled or disabled using a new configuration setting named p2p.discovery.bootstrap.enable which can be used on seed nodes as well as non-seed nodes (e.g. clients, key managers). The latter can also configure how frequently peers are fetched from the seed nodes with the p2p.discovery.bootstrap.retention_period setting.
go/oasis-node: Always start libp2p node (#4995)

Validator nodes were previously not part of our P2P network as it consisted only of nodes that had runtimes configured. Always starting the libp2p node makes them available for consensus-related services.

As validators now participate in the P2P network, the configuration of publicly routable p2p addresses through worker.p2p.addresses setting is now mandatory in a production setting.
go/p2p/rpc: Add support for consensus-wide libp2p protocols (#5000)
go/staking: Reduce DelegationsTo() scanning (#5011)

The escrow address is the first part of the key so we can seek to it.
go/staking: Specify slashed debonding amount in TakeEscrowEvent (#5016)

The newly introduced DebondingAmount field lets observers distinguish how much was slashed from the active escrow pool and how much from the debonding escrow pool.
go/staking: Reduce DebondingDelegationsFor() scanning (#5022)

DebondingDelegationsFor keys are ordered by delegatorAddr. Once past it, it is unnecessary to scan further.
go/registry: Add MaxRuntimeDeployments parameter (#5049)

It specifies the maximum number of runtime deployments.
go/worker/keymanager: Show current key manager policy in the node's status (#5079)
go/worker/keymanager: Show global key manager status in the node's status (#5080)
go/registry: Add WatchEvents() method (#5088)

It can be used to follow the emitted registry events.
runtime/src/enclave_rpc: Verify RPC quotes with key manager quote policy (#5092)
keymanager/src/client: Fetch public keys using insecure RPC requests (#5101)
go/staking: Add CommissionScheduleAddresses() method (#5102)

It can be used to obtain the addresses of accounts with a non-empty commission schedule.
go/consensus: Add missing early exits when simulating transactions (#5104)

Some transactions were missing the early exit after gas accounting when simulating transactions.
go/common/sgx/pcs: Add support for PCS v4 and TCB info v3 (#5108)
go/common/node: Add runtime encryption key (REK) (#5125)

The new key allows enclaves to publish encrypted data on-chain to an enclave instance.
runtime/src/enclave_rpc: Support calls to explicit key manager members (#5156)

Key manager enclaves can now request a host to talk not only to a randomly chosen key manager instance, but also to a specific instance. The identity of the remote node is verified only in Noise sessions. In these, the enclave obtains the other instance's trusted RAK from the consensus layer and compares it to the one used throughout the session.
keymanager: Add forward-secrecy to ephemeral keys (#5158)

Deriving ephemeral keys from the key manager's master secret did not guarantee forward secrecy. In order to fulfill this requirement, we needed ephemeral secrets that are randomly generated on every epoch and distributed securely amongst enclave executors.
go/runtime: Reduce downtime for TEE runtime upgrades (#5188)
keymanager/src/runtime: Support master secret rotations (#5196)

Key managers now have the ability to rotate the master secret at predetermined intervals. Each rotation introduces a new generation, or version, of the master secret that is sequentially numbered, starting from zero. These rotations occur during key manager status updates, which typically happen during epoch transitions. To perform a rotation, one of the key manager enclaves must publish a proposal for the next generation of the master secret, which must then be replicated by the majority of enclaves. If the replication process is not completed by the end of the epoch, the proposal can be replaced with a new one.

The following metrics have been added:
- oasis_worker_keymanager_consensus_ephemeral_secret_epoch_number is the epoch number of the latest ephemeral secret.
- oasis_worker_keymanager_consensus_master_secret_generation_number is the generation number of the latest master secret.
- oasis_worker_keymanager_consensus_master_secret_rotation_epoch_number is the epoch number of the latest master secret rotation.
- oasis_worker_keymanager_consensus_master_secret_proposal_generation_number is the generation number of the latest master secret proposal.
- oasis_worker_keymanager_consensus_master_secret_proposal_epoch_number is the epoch number of the latest master secret proposal.
- oasis_worker_keymanager_enclave_ephemeral_secret_epoch_number is the epoch number of the latest ephemeral secret loaded into the enclave.
- oasis_worker_keymanager_enclave_master_secret_generation_number is the generation number of the latest master secret as seen by the enclave.
- oasis_worker_keymanager_enclave_master_secret_proposal_generation_number is the generation number of the latest master secret proposal loaded into the enclave.
- oasis_worker_keymanager_enclave_master_secret_proposal_epoch_number is the epoch number of the latest master secret proposal loaded into the enclave.
- oasis_worker_keymanager_enclave_generated_master_secret_generation_number is the generation number of the latest master secret generated by the enclave.
- oasis_worker_keymanager_enclave_generated_master_secret_epoch_number is the epoch number of the latest master secret generated by the enclave.
- oasis_worker_keymanager_enclave_generated_ephemeral_secret_epoch_number is the epoch number of the latest ephemeral secret generated by the enclave.
The following metrics have had runtime labels added:
- oasis_worker_keymanager_compute_runtime_count,
- oasis_worker_keymanager_policy_update_count.
keymanager/src/runtime: Verify and modify init request (#5204)

The init request was never verified against the consensus layer state and, therefore, was not trustworthy. To make this request more informative and easily verifiable against consensus, it was extended to include all key manager status fields.
go/oasis-node/cmd/config: Add oasis-node config migrate command (#5237)

Add oasis-node config migrate command to help migrate a node's YAML config file from the old (i.e. Oasis Core 22.2.x) to the new (i.e. Oasis Core 23.0) format introduced in commit [2a132b3].

The command logs the various changes it makes and warns if a config option is no longer supported, etc.

At the end, any unknown sections of the input config file are printed to the terminal so one has a chance to review them and make the manual changes (if needed).
go/runtime/host/sgx: Cache TCB bundles and lower refreshing frequency (#5245)

The node will now make significantly fewer requests for fetching TCB bundles. Because these are now stored locally in the persistent store, this mechanism also enables a degree of tolerance against transient fetch failures in cases where a stored bundle is still valid.
go/runtime/host/sgx: Add metrics for attestations (#5254)
go/control: Improve node registration status clarity (#5256)

Three new fields have been added to the oasis-node control status's output under the registration status section:
- last_attempt_successful - true if the last registration attempt succeeded.
- last_attempt_error_message - error message if the last registration attempt failed.
- last_attempt - time of the last registration attempt.
Also, if the registration descriptor has expired, it is no longer shown in the output.
go/control: Add storage worker status message to control status (#5262)

A status message that shows the current state of the storage worker has been added to the oasis-node control status's storage worker status report.

This enables the node operator to quickly check if the storage worker is still initializing, syncing checkpoints, or syncing rounds.
go/worker/compute: Optimize backup worker commit submission (#5264)

Backup compute workers now observe any gossiped commitments and preempt consensus when it is obvious that there will be a discrepancy declared.
tests: Support cross-version upgrade tests (#5267)
runtime/consensus/tendermint/verifier: Support same-block validation (#5300)

The post-execution state of the latest consensus block is now verified using the block metadata transaction, effectively eliminating any block delay for state verification.
go/control: Add runtime provisioner type to control status (#5301)

Add provisioner field to oasis-node control status's output under runtimes.<RUNTIME-ID>.
go/oasis-test-runner: Run the test runner and scenarios for limited time (#5304)

The test runner and scenarios can now be configured to run for a limited duration using the following flags:
- timeout: the maximum allowable total duration for all scenarios,
- scenario_timeout: the maximum allowable duration for an individual scenario.
go/worker/keymanager/status: Show active version of the key manager runtime (#5320)

Running oasis-node control status command for a key manager node now includes the active_version field which stores the version number of the currently deployed key manager runtime. If no deployment is active, the value is set to null.
go/p2p: implement various metrics collection (#5327)

The following Prometheus metrics were added:
- oasis_p2p_peers
- oasis_p2p_blocked_peers
- oasis_p2p_connections
- oasis_p2p_topics
- oasis_p2p_protocols
See metrics documentation for descriptions of metrics.
go/sgx/ias: Add support for v5 AVR (#5372)

Bug Fixes

go/runtime/txpool: Republish sooner if republish limit is reached (#5003)

This fixes a case where some portion of a batch of transaction would take a long time to be published if there are no new transactions incoming.
go/worker/common: Reorder state determination checks (#5005)

Otherwise the shown state would be misleading, e.g. showing that it is waiting for runtime host being provisioned while it is actually blocked in initialization like storage sync.
go/p2p/rpc: Fix multi call dispatch to different peers (#5005)
go/storage/mkvs/checkpoint: Exclude initial version when pruning (#5005)
go/worker/storage: Fix case when checkpoint sync disabled but forced (#5005)

If checkpoint sync is disabled but sync has been forced (e.g. because the state at genesis is non-empty), we must request to sync the checkpoint at genesis as otherwise we will jump to a later state which may not be desired given that checkpoint sync has been explicitly disabled via configuration.
go/p2p/rpc: Fix peer grading when context is canceled (#5007)

When method CallMulti() finishes early, the requests in progress are canceled and unfairly recorded as failed.
go/p2p/rpc: Fix memory leak when RPC multi call finishes early (#5007)

When method CallMulti() finishes early, the result channel is never cleared. Therefore, the channel never closes and leaves one go routine hanging.
go/common/workerpool: Fix memory leak when workerpool is stopped early (#5008)

When workerpool is stopped, the job channel might still contain jobs which haven't been processed. Therefore, the channel never closes and leaves one Go routine hanging.
runtime: Properly handle state root verification on backup nodes (#5053)
runtime/consensus/tendermint/verifier: Correctly compare headers (#5068)

Since the store may have an earlier (non-canonical, but valid) version of the block available, we need to only compare the actual header and not the commits/signatures.

This is because it can happen that during the immediate sync the light block does not yet contain all of the commits (but only just enough to be valid, e.g. 2/3+) and this gets stored in the light block store. Later on (e.g. during a query) the presented light block may have the full set of commits.
go/control/status: Take storage into account for last retained round (#5074)

When local storage is available (e.g. in stateful nodes), the oasis-node control status's output should only include a round for which storage is available.
go/runtime/registry: Fix watching policy updates (#5092)

When multiple key managers were running, the last known status of the runtime's key manager was overwritten with each status update. On runtime (re)starts, this resulted in the wrong policy being set.
go/consensus: Ensure state has the correct chain context (#5107)

Previously, one could accidentally copy state from one network but use a genesis document from a different one, causing state corruption during Tendermint block replay.

There is now a check to ensure we abort early.
go/runtime/registry: Fix key manager (quote) policy updates (#5111)

When a key manager (quote) policy update fails, the host should retry the update until the policy is updated. For example, when using Tendermint as a backend service, the first update will always fail because the consensus verifier sees new blocks with a one-block delay.
go/tendermint: Change order of events returned from GetEvents() (#5117)

The new order reflects the order in which the events were generated during block execution.
go/worker/client: Better handle latest round queries with verification (#5123)

When a query is requesting to be executed against the latest round and the runtime reports a consensus verifier error, use an earlier round instead as the latest round may not yet be verifiable by the light client as it needs to wait for the validator signatures.
runtime: Fix verification of blocks between two trusted heights (#5134)
runtime: Clear verification trace after block verification (#5148)
go/consensus/tendermint/apps/keymanager: Fix committee construction (#5159)

Previously, a node was added to the key manager committee if the node's first registered key manager runtime passed validation and matched the key manager status. From now on, all supported versions of the key manager runtime must pass this check.
go/worker/client: Ensure block round is synced to storage (#5160)

Previously, the transaction inclusion checks could attempt to inspect a block that the node has not yet synced, triggering an error.
go/worker/compute: Do not drop valid proposals (#5161)

Previously, valid proposals could be dropped instead of being forwarded via the P2P gossip when the local node's consensus view was slightly behind even though the proposal was valid. With smaller committees and certain topologies this could result in some nodes not getting the proposals.
go/registry/api: Fix node registration when TEE not available (#5177)

Previously, registration skipped the TEE hardware verification if a node registered without TEE capability.
go/runtime: Also re-attest based on MaxAttestationAge (#5187)
go/runtime/host/sgx: Update Quoting Enclave target info during re-attestation (#5239)

This allows the node to continue working in case aesmd is upgraded while the node is running. In this case the Quoting Enclave identity can change and this requires the target info to be updated.
go/worker/client: Also treat literal latest round as latest (#5248)
go/worker/compute/executor: Do not propose batch on epoch transition (#5260)

Previously, a compute node could propose a new batch just before the epoch transition happened, resulting in computation that will be discarded anyway.
go/oasis-node/cmd/ias: Regenerate TLS certificate on startup (#5289, #5375)
go/staking: Fix reward distribution when common pool is exhausted (#5319)

Handle scenario where reward schedule hasn't completed yet, but the common pool has already been exhausted.
go/storage/mkvs: Fix commit of nil entries (#5321)
go/worker/storage/sync: Backoff when storage sync starts failing (#5341)

Fixes the case where if storage requests start failing (e.g. due to network errors) the storage worker would crazily retry requests - using lots of CPU and filling up the logs.
go/common/cbor: Relax CBOR decoding for gRPC/RHP endpoints (#5335)
go/worker/common: Reset runtime provisioned status on failure (#5361)
go/registry: Do not verify node TEE capabilities during genesis (#5389)
go/runtime/host/sgx/epid: Ensure consistent IAS proxy usage for attestation (#5390)

Refactors the IAS proxy client to expose separate clients for each configured IAS proxy, instead of load-balancing internally between endpoints on a per-request basis.

This is required because the attestation procedure requires three calls to the IAS endpoint (GetSPIDInfo, GetSigRL, VerifyEvidence) which should all interact with the same endpoint.

Documentation Improvements

docs: Use "chapter" and "section" terms consistently (#5384)

Internal Changes

go/p2p: Improve peer manager's peer handling (#5002)

Previously, peer manager connected to all nodes in the registry which had multiple drawbacks (connections got pruned, no protocol selection, no peer discovery). This was changed so that peers are connected depending on the supported protocols and the number of peers in a protocol/topic is regulated in a controlled fashion.
go/p2p/rpc: Refactor RPC calls (#5007)

Previously, peer manager and RPC client were too tightly coupled. The client also didn't support simple RPC calls which call exactly one peer.
runtime/src/enclave_rpc: Add support for insecure key manager RPC requests (#5075)

The key manager uses encrypted sessions to authenticate clients and protect sensitive data. The number of sessions is limited, thus susceptible to DoS attacks. A malicious client can establish multiple sessions in parallel, preventing other clients from making requests. Furthermore, since each session is encrypted, the exchanged messages cannot be read or modified. For public key requests this is not ideal as one would want to cache the responses locally and serve them to other clients to unburden the manager. Large quotes also cannot be removed from the exchanged messages if we are sure that the recipient can obtain them through some other means. Supporting insecure key manager RPC requests solves some of the before mentioned problems and leaves space for further optimizations.
go/p2p/rpc: Enable subscribing to peer updates (#5083)

Adds WatchUpdates() method to the PeerManager which allows subscribing to peer updates (peers being added or removed).
runtime/src/protocol: Deserialize unknown RHP messages as invalid (#5094)

Runtime-host protocol terminated the reader thread when failed to deserialize a runtime message on the Rust side (e.g. when Body enum contained an unknown field). Decoding is now more robust as these messages are deserialized as invalid and latter discarded and logged as malformed by the handler.
go/grpc: Remove unused gRPC code (#5100)

Removes a lot of unneeded gRPC code since no runtime or consensus protocols use it for communication anymore.
crypto/x25519: Add type-safe X25519 private/public key types (#5121)
runtime: Refactor consensus verifier predicates (#5128)
ci: Explicitly use Ubuntu 22.04 for release builds (#5131)

This avoids the situation when ubuntu-latest gets changed to something else, possibly imposing a different set of dependencies.
runtime/src/protocol: Remove consensus version compatibility check (#5135)

Consensus version check was a sanity check which didn't allow dump-restore upgrades. The removal did no harm as the consensus version was never authenticated and light clients use the verifier to check state compatibility and authenticity.
runtime: Attempt to flush buffers before aborting (#5146)
runtime: Simplify the LRU-backed consensus verifier light store (#5149)
runtime: Reduce the size of the light block LRU store (#5150)
go/consensus/supplementarysanity: Fix checks for legacy validators (#5168)
keymanager: Add support for master secret generations (#5198)

Refactored key manager's init method to be able to support multiple generations of the master secret.
go/keymanager/api: Add signed encrypted master secret (#5199)
runtime: Always use host node ID when verifying freshness with RAK (#5201)

This avoids an initial scan over all the nodes in the registry.
runtime: Use a limited multi-threaded Tokio runtime in SGX (#5214)
keymanager: Support policies in unsafe builds (#5215)

The key manager settings configured in the policy can now be tested on non-SGX builds as well. This simplifies tests, as default values can be configured in the test fixtures (e.g. master secret rotation period).
go/worker/keymanager: Optimize enclave initialization (#5218)

Enclave initialization was moved into its own goroutine to avoid blocking the main loop of the key manager worker. Once initialization is completed, the resulting state of the enclave is compared to the latest key manager status. If the latter has changed, initialization is performed again.

This will be useful when we deploy master secret rotation since new secrets may be generated while old secrets are being replicated which can result in an outdated state once initialization finishes.
runtime: Refactor Tokio runtime constructor (#5220)
docker: Switch to ghcr.io for container registry (#5224)
runtime/consensus/roothash: Implement executor commitment structures (#5282)

Structures and functions related to executor commitments were added in order to be used later for executor commitment verification.
go/consensus/tendermint: Implement {Prepare,Process}Proposal (#5285)

This also makes the nodes execute the proposal in the prepare/process phase such that advanced modification (e.g. including meta transactions based on results) and validation (e.g. rejecting blocks with invalid transactions) becomes possible.
go/oasis-test-runner: Build key manager runtime with trust root (#5307)

The runtime trust-root scenarios now build not only the simple key/value runtime but also the key manager runtime with an embedded trust root.
tests/upgrade: Test master secrets (#5325)

The upgrade scenario was enhanced to test runtime and key manager upgrades, trust roots and master secret rotations.
Bump Go to 1.21.0 (#5030, #5228, #5356)
go: Remove timecache replace directive in go.mod file (#5272)

The replace directive for github.com/whyrusleeping/timecache has been removed since the go-libp2p-pubsub library version 0.9.3 no longer utilizes it.
go: Remove flatbuffers replace directive in go.mod file (#5273)

The replace directive for github.com/google/flatbuffers has been removed since the badger library version 3.2103.4 uses the same version 1.12.1.
go: Ignore CVE-2022-44797 until tendermint uses newer btcd (#5024)
go: Bump golang.org/x/net to 0.13.0 (#5095, #5337)
go: Bump go-libp2p-pubsub to 0.9.1 (#5180, #5195)
go: Bump go-libp2p to 0.30.0 (#5026, #5087, #5180, #5297, #5338, #5339, #5355)
runtime: Bump serde_json to 1.0.87 (#5031)
runtime: Bump futures to 0.3.25 (#5031)
runtime: Bump arbitrary to 1.2.0 (#5031)
runtime: Bump oasis-cbor to 0.5.1 (#5035)
runtime: Bump tendermint-rs to 0.30.0 (#5037, #5106, #5190, #5234)
runtime: Bump tokio to 1.29.1 (#5120, #5330)
runtime: Bump tempfile to 3.4.0 (#5213)
runtime: Bump ed25519-dalek to 2.0.0 (#5351)
runtime: Bump rsa to 0.9.2 (#5351)
runtime: Bump sha2 to 0.10.7, hmac to 0.12.1 (#5351)
runtime-loader: Bump clap to 4.0.22 (#5031)
runtime-loader: Bump futures to 0.3.25 (#5031)
keymanager: Bump futures to 0.3.25 (#5031)
tools: Bump clap to 4.0.22 (#5031)
docker: Bump base image to Ubuntu 22.04 (#5141)
docker: Bump cargo-tarpaulin to 0.25.0 (#5191)

22.2 (2022-10-13)

Protocol	Version
Consensus	6.0.0
Runtime Host	5.1.0
Runtime Committee	4.0.0

Removals and Breaking Changes

go/staking/grpc: rename misnamed GovernanceDeposits method (#4652)
go/runtime: Mandate the use of runtime schedule control (#4665)

Runtimes that do not support the schedule control feature will no longer work with executor nodes.

Features

Add support for PCS attestation (#4806, #4789, #4720, #4752)
go/governance: Add change parameters proposal (#4938)

Introducing a new governance proposal for changing consensus parameters. Until now, this was possible only with an upgrade governance proposal which was not very efficient.
Add archive mode support (#4539)

Node started in archive mode only serves existing consensus and runtime states. The node has all unneeded consensus and P2P functionality disabled so it wont participate in the network. Archive mode can be set using the consensus.tendermint.mode setting.
go/control: Show the debug option statuses in the control output (#4634)
runtime: Make persistent transaction check state configurable (#4640)
go/runtime/txpool: Limit outstanding transactions per sender (#4665)
go/control/status: Add fields for quick overview of node status (#4669)
go/worker/storage: Make checkpoint sync peer selection smarter (#4671)
runtime: verify epoch in dispatcher using consensus verifier (#4677)
go/worker/compute: Ensure trust root is verified before registering (#4678)
go/worker/compute: Start batch resolution early for backup worker (#4679)
go/runtime/txpool: Add roothash incoming messages' data as transactions (#4681)

Roothash incoming messages can provide a piece of data for the runtime. With this change, the data is now treated as a transaction.

Along with this change, we're splitting the txpool into multiple queues. The transactions collected from roothash incoming messages go in a special queue that does not undergo checking or broadcasting.

We also make another queue for a node's own transactions, so that a proposer can prioritize its own transactions.
go/worker/registration: add node status metrics (#4686)
runtime: Enable dispatcher to verify state integrity for queries (#4694, #4830, #4904)
go/worker/common/p2p: Add support for persistent and blocked peers (#4713)
go/worker/common: Properly handle dynamic key manager configuration (#4715)

Since the runtime can go from having no key manager configured to having one, the worker node should handle this correctly.
go/runtime/txpool: Add txpool metric for rejected runtime transactions (#4724)
runtime: Verify RAK in consensus state when serving requests (#4741)
runtime: Add support for reporting EnclaveRPC peer feedback (#4757)

This makes EnclaveRPC more robust as the higher-level layer in the runtime can trigger peer replacement on high level errors.
archive-mode: disable runtime P2P if archive mode is used (#4775)
go/consensus/tendermint: Add consensus.tendermint.halt_height (#4793)

This is equivalent to the cosmos halt-height option, to be used to bring a node down gracefully at a specific height.
net-runner: Add --fixture.default.runtime.version flag (#4813)
go/oasis-node: allow km to have private peers (#4821)
go/runtime/registry: allow client nodes to run sgx runtimes (#4832)

Client nodes can now run runtimes in SGX, which enables them to execute signed queries if peered with a keymanager.
go/worker/keymanager: Add key manager worker metrics (#4877)

The following metrics were added:
- oasis_worker_keymanager_compute_runtime_count
- oasis_worker_keymanager_enclave_rpc_count
- oasis_worker_keymanager_policy_update_count
go/control/status: Add key manager worker status to node status (#4883)
keymanager: Add support for ephemeral keys (#4888)
oasis-net-runner: Set default max_allowances, increase test account balance (#4902)

The max_allowances staking setting in the default oasis-net-runner fixture has been changed from 0 to 16 matching the testnet and mainnet setting. This enables the allowance transactions locally which are required for doing deposits to runtimes.

The initial balance of the default test account generated by the oasis-net-runner was increased from 100 tokens to 1 million. This makes the account directly suitable for deploying and running confidential smart contracts (e.g. on the local Cipher instance) with more realistic transaction fees.
runtime/consensus/verifier: Support trust root consensus layer upgrades (#4903)
runtime: Support consensus event queries (#4904)
keymanager: Validate latest trust root height in key manager requests (#4910)
registry: Add ProveFreshness consensus layer transaction (#4916)

Introducing new transaction that accepts a fixed-size binary blob of 32 bytes and always succeeds without doing any processing or state changes. Transaction is needed for client node TEE freshness verification and enabled via freshness_proofs parameter located in tee_features consensus parameter group.
runtime: Add client node TEE freshness verification (#4922)
keymanager: Verify that policy was published in the consensus layer (#4925)
Bind TEE attestations to nodes and enforce freshness (#4926)
go/genesis: Cache computed genesis document hash (#4919)
go/oasis-node/cmd: unsafe-reset preserve local storage by default (#4700)
go/oasis-node/cmd: unsafe-reset check datadir validity (#4702)
runtime: Emit runtime logs as oasis-node logs (#4709, #4924)

Previously, runtime logs used a slightly different format. Also, they were written to stdout in a manner that was not synchronized with node logs, so the two sets of logs sometimes intertwined mid-line. Those annoyances are gone, plus runtime logs are now annotated with the runtime ID.

Bug Fixes

go/worker/compute: Always include resolved transactions (#4633)
go/runtime/txpool: Fix crash on early access (#4638)
go/worker/compute: Improve proposed transaction handling (#4640)
go/runtime/host: Always emit StoppedEvent on stop (#4647)

Previously the StoppedEvent was only emitted in case the runtime was previously running. In case multihost was performing a version switch when a runtime was not yet started, this resulted in a deadlock.
go/worker/common/p2p: Make sure P2P stops before service cleanup runs (#4650)

Otherwise this may result in a crash during shutdown when P2P requests are processed while database is already closed.
go/oasis-node/cmd/control: Fix runtime-stats arg (#4659)
Fix waiting for deregister event on RequestShutdown (#4662)
go/worker/compute: Only advertise active version for TEE runtimes (#4683)

Previously this caused additional downtime on upgrades due to capability updates not being allowed.
go/runtime/host/sandbox: Properly handle clone3 in seccomp policy (#4687, #4861, #4867)

We need to handle the clone3 syscall in a special manner as there are several complications to its handling:
- Newer glibc versions will try clone3 first and if they see EPERM they will instantly fail making the program unable to spawn threads.
- The clone3 syscall is much more complex than clone and so we can't simply inspect its flags as we do for clone.
Therefore we need to reject the syscall with ENOSYS, causing fallback to clone.
go/runtime/config: report error if SGX loader required but not configured (#4710)
go/worker/keymanager: Fix crash on capability TEE updates (#4725)
go/worker/registration: Only register once epoch is known (#4726)
go/runtime/registry: Refresh key manager policy on runtime changes (#4729)

Since the runtime can change dynamically (due to version upgrades), we need to make sure that we notify the new runtime as well.
go/runtime/config: fix SGX provisioner check for runtimes without SGX (#4734)

This fixes a bug in 22.1.5 where an SGX provisioner was required even for non-SGX runtimes.
runtime/common/version: fix from u64 conversion (#4741)
go/common/quantity: Avoid side-effects on errors (#4790)
net-runner: Fix regression introduced in #4564 (#4808)

Fix the missing Deployments field when oasis-net-runner generating the genesis state and no key manager is provided.
go/registry/api/runtime: ensure no nil deployments in ValidateDeployments (#4822)
go/registry/api: ensure no nil runtimes in VerifyRegisterNodeArgs (#4822)
go/consensus/tendermint: Make sure DBs are only closed during cleanup (#4823)
go/oasis-node/cmd/genesis: Fix reading of height argument when dumping state (#4892)

Command line argument for block height was mistakenly read from an invalid source, which caused all state dumps to be made at height 0 (the most recent block height).
go/runtime/txpool: Add block propagation delay before tx publish (#4959)
go/runtime: Fix configuring the SGX-based provisioner (#4960)

Previously there was an edge case when the SGX loader was configured and remapping to non-SGX was forced (e.g. on client nodes). This would result in an invalid SGX configuration that resulted in a strange error message about a missing SIGSTRUCT.

This has now been changed so that remapping happens independent of whether an SGX loader is configured or not.
go/worker/compute: Only process keymanager client initialization once (#4962)

Documentation Improvements

Document gcc-multilib dependency for Ubuntu/Debian systems (#4475)

It is needed for building oasis-core-runtime.
doc: Replace community slack links with discord (#4751)
doc: Migrate ADRs to oasisprotocol/adrs repo (#4770)

Internal Changes

oasis-net-runner: support configuring runtime state and state root (#4843)
go/worker/common/p2p/rpc: Add support for per-call peer limits (#4671)
docker: Add oasisprotocol/aesmd Docker image (#4690)
runtime: Bump tendermint-rs, x509-parser and oid-registry (#4706)
ci: Build OpenSSL statically for Oasis Core releases (#4707)

This makes sure that oasis-core-loader is more portable between distributions.
go/common/sgx/aesm: Add support for newer methods (#4711)
rust-toolchain: Migrate to TOML version of rust-toolchain (#4727)
go/common/node/address: use custom type instead of net.TCPAddr (#4744)
go: Bump go-libp2p to 0.23.2 (#4765, #4782, #4906, #4944)
go: Bump viper to 1.12.0 (#4766)
go/oasis-test-runner/fixture: support configuring runtime messages (#4769)
runtime: Bump oasis-cbor to 0.5.0 (#4784, #4835)
go/oasis-test-runner/byzantine: Add primary+backup liveness test (#4799)
Bump Go to 1.18.3 (#4820)
go/runtime/host: Make sure runtime-reported version matches bundle (#4825)
go: remove synced storage wrapper (#4837)
go/common/sgx/sigstruct: Add support for offline signing (#4844)
go: Drop go-fuzz in favor of Go 1.18 fuzzing support (#4875)
keymanager-client: Skip policy when OASIS_UNSAFE_SKIP_KM_POLICY is set (#4878)
go/common/grpc: advancedtls options moved to IdentityOptions (#4886)
runtime: Refactor dispatcher initializer and expose consensus_verifier (#4893)
runtime: Add state_at to consensus Verifier trait (#4893)

This allows fetching historic consensus layer state in addition to the latest state.
Bump Rust toolchain to 2022-08-22 (#4896)
runtime: Add epoch_state query and EPOCH_INVALID constant (#4897)
go: Bump Tendermint to v0.34.21 (#4899)
go: Add consensus transaction prioritization (#4911)
go/worker/keymanager: Use common runtime host handler (#4926)
Bump tokio to 1.20.x (#4928)
keymanager: Refactor and reorganize runtime-related crates (#4930)
runtime/consensus/tendermint/verifier: Refactor verifier (#4934)
runtime: Add current sender sequence number to CheckTxMeta (#4956)
runtime-loader: Remove unused ELF loader (#4969)
mkvs/proof: Make sure that all entries in the proof have been used (#4973)

Note that this does not impact verification correctness as only visited nodes are part of the proof computation and the verified tree returned by the verifier.

But in case future code may want to do something with raw entries of a verified proof this removes a possible footgun where unverified entries could be considered.

22.1 (2022-04-01)

Protocol	Version
Consensus	6.0.0
Runtime Host	5.0.0
Runtime Committee	4.0.0

Upgrading From Previous Versions

Previous versions allowed running Oasis Node (i.e. the oasis-node binary) as root (effective user ID of 0). As of this version, this is no longer allowed and attempting to run the oasis-node process as the root user will now terminate immediately on startup.

Removals and Breaking Changes

governance: Allow unregistered nodes in votes (#4607)

The intent from the comments is to allow votes to be cast as long as the entity has at least 1 validator in the active set. This fixes entities with nodes that are not currently registered having their votes rejected.
go/consensus/tendermint: Record node liveness successes (#4615)
scheduler: Improve runtime committee election fairness (#4623)

Prune the per-entity nodes down to the per-runtime cap if configured, prior to doing the election, to improve fairness when entities opt to operate multiple nodes for the same runtime.

Configuration Changes

go/oasis-node/cmd/node: Do not allow running Oasis Node as root (#4464)

Nothing in Oasis Node will ever require elevated privileges. Attempting to run the oasis-node process as the root user will now terminate immediately on startup. While there may be specific circumstances where it is safe to run network services with the effective user ID set to 0, the overwhelming majority of cases where this is done is a misconfiguration.

If the previous behavior is required, the binary must be run in unsafe/debug mode (via the intentionally undocumented flag), and debug.allow_root must also be set.
go/worker/storage: Storage checkpoints are now disabled by default (#4561)
- The worker.storage.checkpointer.disabled flag is removed.
- The worker.storage.checkpointer.enabled flag is added to allow enabling storage checkpoints.

Features

go/oasis-node/cmd/stake: Allow querying historical account info (#4416, #4620)

The oasis-node stake account info CLI command now accepts --height flag which allows one to query an account's info at an arbitrary height.
go/oasis-node/cmd/node: Do not allow running Oasis Node as root (#4464)

Running network accessible services as the root user is extremely bad for system security as a general rule. While it would be "ok" if we can drop privileges, syscall.AllThreadsSyscall does not work if the binary uses cgo at all.
go/oasis-node/cmd/debug/bundle: Add info command (#4546)

Add oasis-node debug bundle info CLI command for inspecting runtime bundles.
Randomize storage checkpoints wall-clock interval (#4561)
go/worker/registration: Add a random re-registration delay (#4574)
go/consensus/tendermint: Bump default max number of inbound peers (#4576)
go/worker: Add txsync protocol (#4579)

No longer clear tx pool on epoch transitions. Add the txsync protocol for syncing transaction between nodes.
go/oasis-node/cmd/control: Add clear-deregister command (#4580)

Instead of using --worker.registration.force_register flag to clear the persisted deregister flag manually, there now is an explicit oasis-node control clear-deregister CLI command that will do so.
runtime/host/sgx: Log the advisory URLs on attestation failure (#4582)
go/oasis-node/cmd: Preserve MKVS database by default in unsafe-reset command (#4588)

Preserving the MKVS database is becoming the more common workflow (i.e. keeping runtimes' state while wiping consensus state), so we're making that the default for oasis-node unsafe-reset.

Use --preserve.mkvs_database=false to wipe the MKVS database as the previous default did.
go/signature: Apply options on registered contexts (#4591)
go/oasis-node/cmd/control: Add runtime-stats command (#4596)

The new oasis-node control runtime-stats CLI command allows querying historical runtime statistics.
Implements liveness and some other useful executor metrics (#4610)

The following Prometheus metrics were added:
- oasis_worker_executor_is_worker
- oasis_worker_executor_is_backup_worker
- oasis_worker_executor_committee_p2p_peers
- oasis_worker_executor_liveness_total_rounds
- oasis_worker_executor_liveness_live_rounds
- oasis_worker_executor_liveness_live_ratio
See metrics documentation for descriptions of metrics.
go/oasis-node/cmd: Allow using non local gRPC connections (#4617)

Require server-side TLS for non unix-socket gRPC connections.
go/worker/common: Lower default transaction recheck interval (#4618)

Bug Fixes

go/common/identity: Save re-generated sentry client TLS certificate (#4382)

Sentry client TLS certificate is always re-generated from the private key when the Oasis Node starts.

Previously, the re-generated sentry client TLS certificate was not saved to disk, which caused confusion since the on-disk certificate file (i.e. sentry_client_tls_identity_cert.pem) had incorrect/outdated expiry date.
go/common/identity: Save re-generated node's persistent TLS certificate (#4382)

If a node's TLS certificate is persistent, it is always re-generated from the private key when the Oasis Node starts.

Previously, the re-generated node's persistent TLS certificate was not saved to disk, which caused confusion since the on-disk certificate file (i.e. tls_identity_cert.pem) had incorrect/outdated expiry date.
go/worker/common: Don't crash if current descriptor is not ready (#4534)
go/worker/beacon: Don't submit VRF proofs if registration disabled (#4536)
go/runtime/host/multi: Release lock before calling into runtime (#4538)
go/worker/common: Watch for all deployment updates when suspended (#4541)
go/worker/common: Properly refresh liveness statistics (#4543)
go/worker/storage: Only attempt checkpoint sync on initial startup (#4544)
go/oasis-node/cmd/ias: Refresh runtimes on every epoch (#4548)
go/worker/common: Propagate CheckTx errors (#4551)
consensus/sanity-checks: Skip suspended runtimes for computing stake claims (#4556)
runtime: Commit check transaction results for subsequent batches (#4557)
go/runtime/txpool: Abort runtime in case it times out during checks (#4563)
oasis-net-runner: Fix fixtures without a keymanager (#4564)
go/worker/registration: Clear deregister flag when deregistered (#4580)

When gracefully halting the node after the node's registration expires, to ensure that the deregistration and shutdown occurs, the node will persist the fact that it is mid-shutdown in a flag.

Previously, this flag had to be cleared manually by the node operator which, while serving to ensure that the node does not get restarted and re-register, is sub-optimal as it required manual intervention.

Instead, if the node is deregistered cleanly, we will clear the flag under the assumption that the operator can configure whatever automation they are using to do the right thing.
go/registry: Fix invalid sanity check (#4585)
go/worker/registration: Fix shutdown if registration never succeeded (#4595)

This fixes a case where node would never shutdown if the initial registration never succeeded.
go/registry: Fix node stake claim sanity check with suspended runtimes (#4603)

Internal Changes

go: The libp2p logs are now emitted via Oasis Node's logging system (#4531)
go/runtime/bundle: Verify SGX signature if present (#4542)
go: Bump libp2p to 0.18.0 (#4562, #4568, #4578)
go/worker/storage: Remove separate storage sync status store (#4565)

Previously the worker maintained a separate store that kept information about the progress of storage sync. Since it was a separate store this could cause problems if it got out of sync (e.g. due to partial manual copies). This should make the process more robust as there is only one source of truth.
Remove worker.registration.debug.private_key flag (#4595)

The flag was unused.
ci: Show dirtiness after regenerating protos (#4598)
go/common/crypto/signature: Improve the batch verification API (#4602)
go/upgrade: Add empty upgrade handler (#4616)
go: Bump github.com/opencontainers/image-spec to 1.0.2 (#4629)
go: Bump github.com/opencontainers/runc to 1.0.3 (#4629)

22.0 (2022-03-01)

Protocol	Version
Consensus	5.0.0
Runtime Host	5.0.0
Runtime Committee	4.0.0

Removals and Breaking Changes

go/registry/api: Lower minimum runtime round timeout (#4090)

The minimum runtime round timeout was previously 5 consensus blocks, but any non-zero positive value is now allowed.
go/common/node: Add SoftwareVersion to the node descriptor (#4212)

This is an opaque string that is totally ignored by the consensus layer, populated with go/common/version.SoftwareVersion for diagnostic purposes.
go/beacon: Add support for, and switch to VRF based elections defined in ADR 0010 (#4248)
Remove runtime genesis state parameter (#4308)

Runtimes are now expected to initialize genesis state on their own when processing the first block. Existing runtimes that use the Oasis SDK already do that.
Remove separate storage committee (#4308)

This removes the need for a separate storage committee and instead treats the compute executor committee as the storage committee, greatly simplifying the protocol and opening a way towards further performance optimizations.
go/governance: Replace Quorum/Threshold parameters with StakeThreshold (#4309)

This replaces the old quorum + threshold based governance proposal verification with a single unified threshold that represents the percentage of yes votes in terms of total voting power for a governance proposal to pass.

The oasis-node debug fix-genesis command's conversion logic will convert the existing parameters to a stake threshold of 68%.
go/roothash: Persist last round results in consensus state (#4316)

Previously last normal round results were only available as events at the height where the runtime block was emitted. This caused problems when consensus state sync was used as that block may not be available. Last rounds results are now persisted in consensus state so they remain always available.
go/roothash: Add incoming runtime messages as defined in ADR 0011 (#4415)

go/staking: DebondEndTime is added to the DebondingStartEscrowEvent (#4437)

DebondEndTime is the epoch at the start of which the debonding escrow will be reclaimed.

type DebondingStartEscrowEvent struct {
        Owner           Address           `json:"owner"`
        Escrow          Address           `json:"escrow"`
        Amount          quantity.Quantity `json:"amount"`
        ActiveShares    quantity.Quantity `json:"active_shares"`
        DebondingShares quantity.Quantity `json:"debonding_shares"`

        // Added.
        DebondEndTime   beacon.EpochTime `json:"debond_end_time"`
}

go/staking: Add minimum transfer amount (#4456)

A new consensus parameter MinTransferAmount (similar to the existing MinDelegationAmount) allows the network to reject small transfers.
go/worker/storage: Transition storage sync to P2P (#4459)
go/staking: Add minimum transact balance (#4461)

In addition to a transaction fee, making a transaction requires that the sender has at least a minimum general staking balance.

A new consensus parameter MinTransactBalance allows the network to enforce this behavior.

Some staking methods now abort when they previously would have caused accounts to go below this minimum general staking balance.
consensus: Base64 encode all tendermint event values (#4465)

Tendermint 0.35 requires all events to be actual strings and doesn't support binary data anymore. It also requires no dashes in attribute keys.

Although Oasis Core still uses Tendermint 0.34, events are updated now to avoid breaking changes later.
Add support for node suspension due to runtime liveness failures (#4470)
go/runtime/txpool: Refactor runtime scheduling queue (#4477)

Remove the algorithm field from the runtime descriptor as it is unused. Refactor the runtime transaction pool as it can be simplified now that there is just one algorithm.
go/common/cbor: Bump decoding limits (#4478)
go/worker/keymanager: Transition keymanager protocol to P2P (#4490)
go/worker/storage: Transition public storage RPC to P2P (#4500)

Configuration Changes

runtime: Change OASIS_STRICT_AVR_VERIFY to OASIS_UNSAFE_LAX_AVR_VERIFY (#4055)

Previous builds would default to lax verification. After this change the default verify behavior will be the strict (requires OK quote status), unless OASIS_UNSAFE_LAX_AVR_VERIFY is set.

NOTE: As with previously, this option has no effect if AVR verification is disabled entirely.
Introducing the runtime mode (#4308, #4520)

Instead of separately configuring various roles for a node, there is now a single configuration flag called runtime.mode which enables the correct roles as needed. It should be set to one of the following values:
- none (runtime support is disabled, only consensus layer is enabled)
- compute (node is participating as a runtime compute node for all the configured runtimes)
- keymanager (node is participating as a keymanager node)
- client (node is a stateful runtime client)
- client-stateless (node is a stateless runtime client and connects to remote nodes for any state queries)
Nodes that have so far been participating as compute nodes should set the mode to compute and nodes that have been participating as clients for querying and transaction submission should set it to client.

The following configuration flags have been removed:
- runtime.supported (existing runtime.paths is used instead)
- worker.p2p.enabled (now automatically set based on runtime mode)
- worker.compute.enabled (now set based on runtime mode)
- worker.keymanager.enabled (now set based on runtime mode)
- worker.storage.enabled (no longer needed)
- worker.storage.root_cache.apply_lock_lru_slots (no longer needed)
Switch to unified runtime bundles (#4469)

Instead of distributing runtime binaries separately and having the node operators configure the appropriate binary based on runtime ID, runtime artifacts are now distributed under a single self-describing "fat" bundle file.

To configure runtimes the new syntax is:
```
runtime:
  paths:
    - examplepath/runtime.orc
    - anotherExamplePath/anotherRuntime.orc
```
If the commandline is being used runtime.paths now only takes the path to the runtime bundle(s).

This requires downloading new builds of the runtime binaries that are packaged in this format.

Features

go/oasis-node/cmd/consensus: Add next_block_state command (#4241)

This adds the oasis-node consensus next_block_state CLI command that shows the consensus voting status of the next block, primarily intended for debugging and monitoring upgrades.
go/runtime/client: Add missing GetTransactionsWithResults method (#4295)
go/common: Use entire error as error message (#4298)
go/runtime/host/sgx: Autodetect SGX device name (#4333)
go/runtime/client: Add last retained round query (#4334)

This allows clients to query what is the earliest runtime round that the give node is storing.
go/runtime/scheduling: Allow a higher priority tx even if queue is full (#4431)
go/worker/common: Report locally supported runtime host versions (#4436)
Add support for runtime schedule control (#4438)

This feature gives runtimes control over scheduling transactions inside a block, so the runtimes may implement their own more advanced policies.
Add support for runtime message results as defined in ADR 0012 (#4443)
go/runtime: Downtime-less runtime updates (#4449)

Runtime descriptors now include information regarding supported versions, and the epoch from which they are valid, so that updates can happen without incurring downtime by having upgrades and the descriptor change pre-staged well in advance of the upgrade epoch.

NOTE: In the initial implementation, SGX enclaves will take an unavoidable 1 epoch of downtime (the epoch when the upgrade takes effect), as allowing attestation for provisional (i.e. may never become valid) enclaves seems moderately risky.
go/oasis-node: Implement unified runtime bundles (#4469)

This changes runtime bundles to be unified across all supported TEE types and self describing so that configuring runtimes is only a matter of passing in the runtime bundle file.
tendermint: Bump to version 0.34.15-oasis1 (#4481)
go/worker/common: Report liveness statistics in runtime status (#4524)

The control status output will now show something like the following for nodes which are currently in the executor committee:
```
  "liveness": {
    "total_rounds": 78,
    "live_rounds": 77
  },
```
The information comes from the consensus layer's liveness statistics for the current epoch. One can derive the node's liveness percentage as live_rounds / total_rounds.

Bug Fixes

go/common/identity: Don't forget to refresh TLS certificate (#4289)
go/worker/storage: Fix potential memory leak through context (#4290)
go/storage/api: Handle context cancellation in receiveWriteLogIterator (#4292)
go/worker/storage: Configure GetDiff timeout (#4296)
go: Make sure correct namespace/hash text serialization is used (#4303)
go/worker/storage: Handle state sync before runtime is operational (#4311)

Previously, there was an edge case that was not handled when the runtime was registered in the consensus layer but not yet operational (e.g. there have been no normal blocks yet). If a new node used state sync to quickly catch up with the consensus layer and sync to a height after runtime's genesis, the node would never register as it would keep waiting for storage checkpoints (which wouldn't exist yet).
go/oasis-node/cmd: Add missing namespace to LatestStateRoot (#4317)
go/worker/compute: Handle round results at genesis (#4317)
go/consensus/tendermint: Only reset executor pool after emitting block (#4323)

Make sure to only reset the executor pool after any timeouts have been cleared (e.g. when an empty block is emitted) as otherwise there could be a stale timeout.
go/runtime/client: SubmitTxNoWait should propagate CheckTx error (#4328)
go/staking: Fix incorrect non-existence check (#4340)
go/runtime/history: Limit batch size when pruning rounds (#4345)
go/consensus/tendermint/roothash: Only index the correct runtime (#4351)

In case multiple runtimes were being tracked, the indexing process could incorrectly set the last indexed round which could make it skip to index a round.
oasis-node: Make Semver parsing less strict (#4366)

Semantic version parsing behavior changed in #4343 was too strict. This fix changes parsing of versions so that only major component is required whereas minor and patch are optional and any remaining components are ignored.
go/storage/mkvs: Don't fail sanity check for equal roots (#4381)

In case the state root did not change, a write log is empty and so the sanity check should not require that it exists.
runtime: Propagate panics during transaction/local RPC/policy dispatch (#4386)

A panic during transaction/local RPC/policy dispatch signals a serious problem so it should be propagated and the runtime should crash to force state reset.
go/runtime/client: Take storage into account in GetLastRetainedBlock (#4387)
go/worker/storage: Checkpoint correct round on consensus checkpoint (#4387)
go/storage/client: Catch GetDiff errors early (#4397)
go/worker/storage: Limit number of rounds to fetch before applying them (#4403)

Previously, when a node was syncing from genesis, it would try to fetch all unapplied rounds before applying them. This could mean trying to fetch 100k+ rounds before applying them. In combination with failing to fetch rounds and random retrying, this could make the syncing process unbearably slow.
go/storage/api: Make metrics wrapper return proper types (#4411)

Previously, the metrics wrapper tried to support all of the different backend interfaces, causing problems with places that perform type checks to determine which backend type is in use.

The metrics wrapper is now changed to return a type matching the wrapped backend type.
go/runtime/transaction: Return transactions in batch order when queried (#4429)

Previously, when runtime transactions were queried via a GetTransactions call, they were returned ordered by transaction hash instead of in execution order.
runtime: Switch to the upstream jsonrpc package (#4439)
go/runtime/scheduling: Attempt to continue scheduling (#4453)
runtime: Fix registry.Roothash serialization (#4460)

Brings the staking_params_are_empty method up to date and adds some additional serialization tests.
Allow launching enclaves with SW_HARDENING_NEEDED quote status (#4491)

Documentation Improvements

ADR 0010: VRF-based Committee Elections (#4248)
ADR 0011: Incoming Runtime Messages (#4327)
ADR 0012: Runtime Message Results (#4392)
Migrate Oasis Core Docs to Docusaurus (#4501)

Internal Changes

Use typed attributes in all consensus services (#4034)

Beacon, keymanager, registry, roothash and scheduler events are updated to use the typed event attribute API.
go: Bump fxamacker/cbor to 2.3.0 (#3986)
common/node: Verify AVR quote status (#4055)

When verifying a TEECapabilities structure, additionally verify the quote status against the new SGXConstraints.AllowedQuoteStatuses vector so that nodes that have an invalid quote status can be omitted from scheduling entirely.

NOTE: QuoteOK is ALWAYS allowed, as disallowing it is nonsensical.
Setup rust code lints (#4170)
go/roothash: Remove unused commitment.SignatureVerifier (#4308)
go/worker/client: Move runtime client to a worker (#4308)
runtime: Bump tendermint-rs to 0.23.1 (#4365)
Bump Go to 1.17.3 and Rust to nightly-2021-11-04 (#4375)
tests/workload/runtime: Wait for 3rd epoch for compute nodes to be up (#4395)

Since the VRF-based scheduler was introduced the compute nodes only become eligible for election in the third epoch after genesis.
go/consensus/tendermint/api: Add transaction context (#4426)

The old way of doing checkpoints is removed as the new transaction context supports nesting and event isolation.
go/common/crypto/signature: Add static entropy provider for P2P roles (#4459)
go/worker/common/p2p: Add support for RPC protocols over P2P (#4459)
go/common/dynlib: Embed dynlib (#4513)

21.3 (2021-09-27)

Protocol	Version
Consensus	4.0.0
Runtime Host	4.0.0
Runtime Committee	3.0.0

Removals and Breaking Changes

runtime: Remove runtime, tags and messages context fields (#3572)

The test runtime is refactored to stop relying on those fields and behave like a simplified version of the dispatcher in the SDK.
runtime: Remove storage::StorageContext (#3572)

The storage context was deprecated and has now been removed.
go/runtime: Remove tag indexer (#3888)

The tag indexer is slow, buggy and unused by the SDK. Runtime transaction indexing and querying capabilities should be provided as part of the new event streaming API.
runtime: Verify consensus layer state by using a light client (#3952)
go/runtime/client: Remove deprecated GetTxs method (#3983)

The GetTxs method has been deprecated and removed. Users should migrate to use GetTransactions instead.
oasis-node: Remove opentracing and jaeger support (#4011)

This is breaking as it deprecates a feature from the node binary, and alters both the P2P and runtime/host communication protocols.
go/ias-proxy: remove the debug use genesis option (#4098)
go/oasis-node: Prevent startup when file descriptor limit is too low (#4192)

Before, a warning was emitted if file descriptor limit was low (below 1024). Since low file descriptor limit can cause problems with BadgerDB, a high enough limit is now required on node startup (at least 50000).

Follow the File Descriptor Limit documentation page for details on how to increase the limit on your system.
runtime: Deprecate transaction::MethodDispatcher (#4198)
runtime: Deprecate the old MethodDispatcher related types (#4201)
go/roothash: Remove GetLatestBlock from BlockHistory interface (#4217)

Instead developers should use GetBlock or GetAnnotatedBlock with the special RoundLatest value to get the latest block or annotated block.
go/scheduler: Remove debug_static_validators consensus parameter (#4222)

The option was only used in unit tests and required some special handling.
go/runtime: Remove obsolete transaction types (#4228)

The deprecated types TxnCall, TxnOutput and TxnCheckResult have been removed. Any users should consider migrating to the Runtime SDK or copying the type definitions in case they still rely on them.
runtime: Add MaxMessages to CheckTxBatch and Query (#4231)

This allows transaction checks and queries to access the maximum number of runtime messages that can be emitted per round. Previously this information was only available in ExecuteTxBatch.
keymanager-api-common: Implement Zeroize for secret types (#4242)
Remove EnclaveRPC client gRPC interface (#4244)

Since the SDK now supports using the key manager, the idea is that EnclaveRPC calls should be done from the runtime and exposed via queries.
runtime: Make storage cache capacity configurable (#4264)

Also changes the start_runtime method signature to accept a Config struct so that in the future adding various configuration is easier.
runtime: Make dispatcher core async (#4273)

Note that this only affects the core and allows for parallel query processing which still happens in blocking threads. It does open the way for fully async query dispatch.
runtime: Refactor protocol type, expose via transaction::Context (#4276)
go/common: Change Namespace's text (un)marshaling to use the hex form (#4279)
go/common/crypto/hash: Change Hash's text (un)marshaling to use the hex form (#4279)
runtime: Pass query arguments as arbitrary bytes (#4280)

This makes it more in line with execute/check requests which already accepted arbitrary bytes and it prevents malformed CBOR in query arguments from breaking the Runtime Host Protocol.
go/consensus/api: Change Block's Hash field type to the common Hash type (#4281)

Also change the type of Status' hash-related fields (LatestHash, GenesisHash, LastRetainedHash) to the common Hash type to allow nicer text/JSON serialization.

Configuration Changes

go/runtime: Remove tag indexer (#3888)

The runtime.history.tag_indexer.backend configuration option has been removed.
go/worker/common/p2p: Add flags controlling libp2p validator concurrency (#4025)
- worker.p2p.validate_concurrency: libp2p gossipsub per topic validator concurrency limit
- worker.p2p.validate_throttle: libp2p gossipsub validator concurrency limit
badger/migration: badger.migrate.num_go_routines flag (#4037, #4041)

The flag enables controlling the number of go routines badger uses when doing the v2 -> v3 migration. Use the flag to lower memory pressure during the migration, by lowering the amount of workers to e.g. 1.

Features

go/worker/executor: Batch runtime CheckTx transactions (#2548)
go/staking: Add DebondingStartEscrowEvent (#4014)

This makes it easier to reason about the debonded amounts, both in shares and in base units.
go/oasis-net-runner: Allow setting runtime provisioner (#4024)

On systems that do not support Bubblewrap, the runtime provisioner can be set to unconfined.
go/oasis-net-runner: Allow running without a key manager (#4024)

This feature makes it easier to run debug runtimes that are not (yet) associated with a key manager.
runtime-loader: Allow building for non-SGX use on platforms that aren't Linux (#4024)
oasis-net-runner: Support setting custom node arguments in fixtures (#4025)
Build oasis-node with jemalloc tag (used by BadgerDB) (#4040, #4046)

In BadgerDB v3, using jemalloc seems to be recommended and better supported option (1, 2). Based on our experience, using jemalloc reduces BadgerDB memory usage.

To build oasis-node without jemalloc requirement, set the OASIS_BADGER_NO_JEMALLOC="1" environment variable before invoking the Makefile.
upgrade/migrations: Add max allowances upgrade handler (#4064)

Add "consensus-staking-max-allowances-16" upgrade handler which updates the consensus staking parameter MaxAllowances to 16.
go/common/errors: Expose runtime query error details to clients (#4067)
go/worker/storage: Synchronize checkpoints with consensus layer (#4080)
runtime/client: Add SubmitTx variant that includes metadata (#4162)
go/registry/api: Add its own body type to MethodDeregisterEntity (#4177)
upgrade/migrations: Add consensus-params-update-2021-08 upgrade handler (#4180)
go/roothash/api/block: Use custom Timestamp type for block's header (#4183)

This enables prettier Oasis Node's control status CLI command's output for runtimes' latest_time field and matches the format of consensus' latest_time field.
go/roothash/api: Add ConsensusParameters method (#4187)
go/consensus/tendermint: Request state refresh on consensus init (#4237)

Previously pending upgrade descriptors were only refreshed during governance proposal execution or after finishing a state sync. This is changed so that the MessageStateSyncCompleted message is also emitted after consensus initialization on already synced nodes so pending upgrades get installed even in case state was synced manually (out of band).
go/common/node: Add custom text (un)marshaler for RolesMask type (#4243)

This will result in easy to understand "roles" fields in various CLI commands that output JSON.
go/common/entity: Use pretty-printed JSON when saving entity to file (#4266)
go/oasis-node/cmd: Use pretty-printed variant when outputting JSON (#4266)
go/common/crypto/hash: Add LoadFromHexBytes() method to Hash type (#4281)
go/common/crypto/hash: Add Hex() method to Hash type (#4281)

Bug Fixes

go/oasis-net-runner: Always honor fixture.default.setup_runtimes (#4009)
go/common/badger: Fix v2->v3 migration for managed mode (#4010)
go/storage/mkvs/checkpoint: Checkpoint in descending order (#4010)

Previously the checkpointer would generate checkpoints in ascending order which meant that it could generate many checkpoints only to garbage collect them in the next step.
go/consensus/tendermint/apps/staking: Fix emitted reward events (#4033)
badger/migration: Keep only last version when migrating non-managed DB (#4037)

Even though Oasis Core always configures non-managed badger databases to keep a single version of keys, some databases in the wild contain multiple versions of the same key.

Skip migrating more than the latest version when migrating non-managed badger databases.
go/runtime/host/sandbox: Prevent restart storm after 10 minutes (#4059)
go/consensus/tendermint: Fix last retained version query (#4060)

Previously, the reported version was incorrect when the node used state sync and pruning was disabled.
go/worker/executor: Do not route local storage applies via gRPC (#4061)
go/upgrade/api: Validate upgrade migration handler name on registration (#4065)
go/worker/storage: Add missing timeout for operations (#4072)
go/worker/storage: Don't crash on early query (#4072)
go/worker/storage: Force checkpoint sync when block info unavailable (#4080)
ias/proxy/client: Make GetSigRL not panic if IAS proxy not configured (#4081)
go/worker/executor: Skip proposer timeout if batch was received (#4083)
go/consensus/tendermint: Handle validator fetching error in GetStatus() (#4085)
go/worker/common: Always use naked local storage (#4093)

Previously the committee group could incorrectly use the synced version of the local backend which could block operations resulting in failure to fetch data.
go/storage/api/mux: Don't return an error on a successful read (#4093)
go/storage/mkvs/checkpoint: Determine next offset correctly (#4096)
go/worker/compute: Replicate input batch locally (#4107)

Previously, storage commit could fail in case the node was both an executor and a storage node but not in the storage committee.
go/consensus/tendermint: Correctly propagate errors (#4110)

Not propagating the state unavailable error could lead to corruption when the database becomes unavailable (e.g., due to running out of space or file descriptors).
go/runtime/host/sandbox: Add deadline to Bubblewrap pipes (#4111)

This prevents the constructor from blocking forever in case something is wrong with the sandbox setup.
go/runtime/client: Fail SubmitTx early for unsupported runtimes (#4202)

Make sure that the runtime is actually among the supported runtimes as otherwise we will not be able to actually get any results back.
go/runtime: Fix ignored round in runtime Query (#4210)
go/runtime/client: Wait for hosted runtime provisioning (#4214)

Previously an early runtime Query or CheckTx could fail if the locally hosted runtime was just being provisioned. This makes it wait for the provisioning to complete instead.
go/common/identity: Refresh sentry TLS certificates (#4239)

Since we are using public keys for TLS authentication, we make sure that sentry TLS certificates are refreshed to avoid them expiring.
go/runtime: Add key manager EnclaveRPC support to client runtime hosts (#4244)

Previously, runtimes hosted on client nodes were not able to issue EnclaveRPC requests as the endpoint was not implemented in the handler. This unifies the handler implementations between the client and compute nodes and makes sure that key manager requests are possible (of course client nodes will only be able to request public keys).
go/upgrade: EnsureCompatible should only check consensus (#4267)
go/common/grpc: Configure max message size on the client (#4268)

Documentation Improvements

Update example output of oasis-node control status CLI command (#4281)

Internal Changes

go: Bump github.com/dgraph-io/badger/v3 from 3.2011.1 to 3.2103.1 (#4003, #4127)
go: Bump github.com/prometheus/common from 0.25.0 to 0.28.0 (#4007)
go: Bump github.com/prometheus/client_golang from 1.10.0 to 1.11.0 (#4008)
runtime: Stop using webpki to validate the IAS cert chain (#4021)
runtime: Do not require OASIS_STORAGE_PROTOCOL_SERVER_BINARY (#4023)
go: Bump gitlab.com/yawning/dynlib to the latest version (#4026)
go/oasis-node/cmd/debug: Remove the consim sub-command (#4030)
go: Bump github.com/spf13/viper from 1.7.1 to 1.8.0 (#4032)
go/consensus/tendermint: Add typed attribute API (#4033)

Using typed events makes it slightly harder to use an incorrect key/value combination. Unfortunately, we cannot use this in all of the cases due to events in registry/roothash not being nicely structured and changing this would require existing nodes to resync due to internal event key changes.
build: Stuff to make CI faster (#4038)
ci: Fix dirty Git state in the release GitHub Actions workflow (#4050)
github: Unify jemalloc installation with GoReleaser installation in workflows (#4052)
go/common/backoff: Add NewExponentialBackoff() with sane defaults (#4059)
go/worker/common: Move committee storage to Group (#4061)
go/consensus/tendermint/apps/governance: Ensure upgrade is in progress (#4070)
go/storage/mkvs/checkpoint: Add ForceCheckpoint() and WatchCheckpoints() (#4080)
go/runtime: Delay subscriptions until after consensus sync (#4080)

Previously, if the node used consensus state sync it would fail to receive any updates for the various descriptors until the descriptors were updated after the state sync checkpoint.
go/consensus: Expose consensus state checkpointer (#4080)
runtime: Bump bech32 to 0.8.1 (#4089)
runtime: Bump tendermint to 0.20.0 (#4089)
ci: Make jemalloc builds reproducible (#4095)
runtime: Use JSON-RPC for the mkvs interop helper (#4101)
go/worker/compute: Add runtime abort timeout to avoid blocking forever (#4111)
go/common/logging: Switch to go-kit/log module (#4117)

This is the only part of go-kit that we are using so it makes sense to switch to the smaller package.
runtime: Switch from serde-cbor for oasis-cbor (#4133, #4278)
rust: Bump rsa from 0.4.0 to 0.5.0 (#4147)
runtime/simple-keyvalue: Check nonces (#4154)
storage/checkpoint_sync: Add timeout for fetching checkpoints from nodes (#4155)
workload/runtime: Ensure correct round for validating events (#4161)
go/registry/gen_vectors: Stop generating register transactions for v1 entities (#4167)
rust: Bump x509-parser from 0.9.2 to 0.10.0 (#4173)
rust: Bump intrusive-collections to 0.9.2 (#4174)
go: Update gofumpt to 0.1.1 (#4176)
go/registry/gen_vectors: Add vectors for deregister entity transactions (#4177)
go/oasis-test-runner: Add support for testing upgrade handlers (#4179)
go: Update golangci-lint to 1.41.1 (#4190)
rust/ci: Bump tarpaulin to 0.18.0 (#4197)
rust: Update crossbeam-deque to 0.7.4/0.8.1 (#4209)
rust: Bump tokio from 1.8.1 to 1.10.1 (#4213)
go/oasis-test-runner: Add an E2E test for (un)freezing a slashed validator (#4223)
Bump Rust toolchain to 2021-08-17 (#4235)
go: Bump github.com/libp2p/go-libp2p to 0.15.0 (#4257)
go: Bump grpc to 1.40.0 (#4260)
go/fixgenesis: Remove the not anymore relevant migration (#4265)
go/oasis-node/cmd/common: Add PrettyJSONMarshal() helper (#4266)
rust: Bump crypto-bigint to 0.2.10 (#4278)
rust: Bump base64ct to 1.1.0 (#4278)
rust: Bump zeroize_derive to 1.2.0 (#4282)

21.2 (2021-06-07)

Protocol	Version
Consensus	4.0.0
Runtime Host	3.0.0
Runtime Committee	2.0.0

Upgrading From Previous Versions

Due to a database backend upgrade introduced in this version (see below for details), initial startup time will be longer as a full database migration needs to be performed for each of the internal databases. The migration may take several minutes and requires enough free disk space to accommodate a copy of the node's data directory.

Removals and Breaking Changes

runtime: add support for transaction priority and weights (#3760)

Adds support for runtime transaction priority and weights that can be set by runtime in CheckTx response. Transactions are scheduled by priority (higher first). Transaction weights are used by scheduler when creating batches to ensure no batch passes the per round weight limit. Supported limits are:
- count - number of transactions - per round limit is a runtime parameter in registry (already existed prior to this change).
- size_bytes - size of transactions in bytes - per round limit is a runtime parameter in registry (already existed prior to this change).
- consensus_messages - number of emitted consensus messages - per round limit is a consensus parameter (already existed prior to this change).
- Custom, runtime specific weights. internal.BatchWeightLimits method is queried for custom batch weight limits on every round.
runtime: Ensure compatible consensus versions (#3823)

Runtime now ensures host is running a compatible consensus version. This is a breaking runtime host protocol change as the version types in protocol are changed.
runtime: Add support for block tags (#3848)
runtime: Add epoch to ExecuteTx/CheckTx/Query requests (#3892)
go/cmd/registry: simplify register runtime command (#3903)

The registry runtime gen_register command now accepts a JSON runtime descriptor, which should simplify generating runtime registration transactions. The registy runtime init_genesis command is removed as runtime descriptors in genesis are not singed since version 21.0+ making the command obsolete.
staking: Include share amounts in AddEscrow and ReclaimEscrow events (#3939)
Update emitted events when disbursing rewards (#3944)

Before, a single AddEscrow event was emitted which did not correctly represent the actual state changes when rewards were disbursed.

Now the following events are emitted:
- Transfer(CommonPool -> Recipient, commissionAmount) event for the commissioned part of the reward
- AddEscrow(Recipient -> Recipient, commissionAmount) event for the automatically escrowed commission reward
- AddEscrow(CommonPool -> Recipient, restAmount) for the non-commissioned part of the reward (which only increases existing shares prices)

Configuration Changes

Change IAS proxy client authentication configuration (#3958)

Previously the full TLS certificate needed to be passed when configuring the IAS proxy client, but now only the public key needs to be included.

The ias.tls configuration option is removed and ias.proxy_addr is renamed to ias.proxy.address. One can now configure multiple proxy addresses and each address needs to contain the Base64-encoded public key, for example: <pubkey>@ias.oasis.dev.
go/worker/compute: remove schedule_check_tx.enabled flag (#3965)

Runtime executor workers now always call CheckTx before scheduling runtime transactions. The option to disable CheckTx for runtime transactions is removed.

Features

Bump Badger database to v3 (#3859)

The old database will automatically be converted during the initial node startup. For this reason the following should be taken into account when upgrading:
- Initial startup time will be longer as a full database migration needs to be performed for each of the internal databases. The migration may take several minutes.
- In order to ensure a backup exists in the unlikely case that anything goes wrong with the migration, the old databases will be kept in the same location but with a .backup suffix. For this reason, make sure that you have enough free disk space to accommodate a copy of the node's data directory.
runtime: Implement consensus staking layer state accessors (#3564)
Add support for local runtime-specific configuration (#3776)

Runtimes can now accept configuration local to the compute node that is running them. This is possible via the following configuration schema (can only be configured via a config file):
```
runtime:
  config:
    "8000000000000000000000000000000000000000000000000000000000000000":
      core:
        min_gas_price: 1
```
This is useful to specify runtime-specific parameters like minimum gas price accepted by the node. Note that this configuration can only be used during non-deterministic transactions checks as it would otherwise result in a discrepancy.
go/runtime/client: Add GetTransactions method, deprecate GetTxs (#3812)

A new method called GetTransactions has been added to replace the (now deprecated) GetTxs. The new method does not require the caller to pass the IO root as that is an internal detail. This makes it consistent with the existing GetEvents API.

Users of GetTxs should migrate to GetTransactions as the former may be removed in a future release.
go/roothash/api: Add gRPC service (#3827)
common/crypto/signature: Use curve25519-voi (#3829)

This Ed25519/X25519 implementation is faster and more maintainable that the previous in-house implementation.
common/crypto/slip10: Add a SLIP-0010 implementation (#3829)

Needed for some wallet stuff, and it is the least-awful hierarchical derivation scheme for Ed25519.
go/oasis-node/cmd/genesis: Display doc's hash and SHA256 checksum with check (#3858)
go/worker/compute: Propose block on runtime genesis (#3894)
go/oasis-node: Add storage namespace rename subcommand (#3908)
go/oasis-net-runner: default fixture support staking genesis (#3914)
go/oasis-net-runner: support multiple runtimes in default fixture (#3914)
go/common/crypto/sakg: Add ADR 0008 implementation (#3918)
go/worker/compute: Propose block on new epoch (#3987)
go/oasis-node/cmd/debug/beacon: Initial import (#3897)

Add a debug beacon status command which queries the current beacon and PVSS backend state.

Bug Fixes

go/oasis-node: Dump correct block height on halt (#3755)
go/storage/mkvs: Skip already pruned nodes in migration (#3857)
Apply upgrade proposals directly from governance consensus application (#3870)

This should make it more robust in light of fast sync as otherwise the worker could miss upgrade events in certain edge cases.
Runtime workers should wait for history reindex (#3889)

If a node first syncs consensus without any runtimes configured and a runtime is configured later, the workers should wait for historic runtime block reindexing to complete before continuing with initialization. Otherwise historic block queries may fail and prevent the worker from operating normally.
go/consensus/tendermint: Update nonce post-execution during CheckTx (#3896)

This now correctly handles multiple transactions from the same signer being checked in the same block as previously subsequent transactions were rejected.
worker/storage: Fix heartbeat stall (#3899)
go/runtime/client: Properly propagate CheckTx error context (#3973)

Documentation Improvements

ADR 0008: Standard Account Key Generation (#3656)
Update Security document (#3853)
Note that RELEASE_BRANCH variable needs to be exported in Release Process (#3891, #3993)
ADR 0009: Ed25519 Signature Verification Semantics (#3925)
Add missing gRPC services and beacon service (#3991)

Internal Changes

go/worker: Make compute and client node use local storage if available (#3251)
Runtime consensus state interoperability tests (#3833)
go: bump github.com/hashicorp/go-hclog from 0.15.0 to 0.16.1 (#3834, #3927)
go/storage: Remove unused ForceFinalize RPC call (#3863)
e2e/runtime: Add test for storage sync after messy/incorrect cleanup (#3863)
rust: bump tendermint from 0.18.1 to 0.19.0 (#3866)
Build and upload release artifacts on every master/stable merge (#3875)
go: Ignore jwt-go vulns since we're not using the features (#3877)
go: bump github.com/hashicorp/go-plugin from 1.4.0 to 1.4.2 (#3880)
Bump Go to 1.16.3 (#3886)
changelog: Automatically add protocol versions table (#3891)
rust: bump hmac from 0.10.1 to 0.11.0 (#3906)
go: bump github.com/grpc-ecosystem/go-grpc-middleware from 1.2.2 to 1.3.0 (#3915)
go/staking/gen_account_vectors: Add ADR 0008 test vectors (#3918, #3960)
go: bump github.com/libp2p/go-libp2p from 0.13.0 to 0.14.0 (#3928)
common/crypto/slip-10: Relax the seed-length check (#3945)

The SLIP-0010 spec, was updated after approximately 5 years to be coherent, so support S that range from 128-bits to 512-bits, and add in the formerly nonsensical test vector.
runtime/common/crypto/signature: Switch Ed25519 to our consensus flavor (#3946)
rust: bump curve25519-dalek from 3.0.0 to 3.1.0 (#3949)
runtime: Bump futures/tokio/grpcio (#3950)
go: bump github.com/prometheus/common from 0.20.0 to 0.25.0 (#3951)
go: bump google.golang.org/grpc to 1.38.0 (#3953)
Bump Rust toolchain to 2021-05-20 (#3955)
workload/governance: fix case with multiple equal active proposals (#3959)
go/runtime/scheduling: priority queue txpool (#3965)

Priority queue transaction pool (based on B-Trees) is implemented to support scheduling based on the newly added runtime transaction priorities.
go: bump github.com/uber/jaeger-client-go to 2.29.1+incompatible (#3966)
Bump grpcio to 0.9.0 (#3968)
go/common/errors: Support context propagation (#3973)

This adds WithContext and Context in order to be able to add an error context in a structured way. Such contexts are then correctly propagated over gRPC and consensus layer transaction results in order to improve the error messages.
rust: bump crossbeam from 0.8.0 to 0.8.1 (#3977)
go: bump github.com/multiformats/go-multiaddr from 0.3.1 to 0.3.2 (#3978)
go: bump github.com/cenkalti/backoff/v4 from 4.1.0 to 4.1.1 (#3979)
go: bump github.com/libp2p/go-libp2p from 0.14.0 to 0.14.1 (#3980)
rust/Cargo.toml: specify exact ring depedency version (#3981)
runtime: Stop using the ring backend for snow (#3990)
rust: use 0.0.0 as crate versions (#3995)

Crates are following Oasis Core versioning scheme. The crates are not published and GitHub tags should be used as version definitions. Therefore bumping versions on the crates themselves is omitted. Update crate authors.
go/worker/common: Wait for all child workers to initialize (#4001)

21.1 (2021-04-12)

Protocol	Version
Consensus	4.0.0
Runtime Host	2.0.0
Runtime Committee	2.0.0

Removals and Breaking Changes

go/registry: Allow update from no key manager to with key manager (#3802)
go/common/entity: Rename LatestEntityDescriptorVersion (#3845)

Rename it to LatestDescriptorVersion.
go/upgrade: Limit Descriptor's field values (#3845)

Add variables defining Descriptor's field value limits: MinDescriptorVersion, MaxDescriptorVersion, MinUpgradeHandlerLength, MaxUpgradeHandlerLength, MinUpgradeEpoch, MaxUpgradeEpoch.
storage: Drop the MKVS node version field (#3500)
go/common/version: Move ConsensusProtocol ver to top of ProtocolVersions (#3845)

Features

oasis-net-runner: Configure a client node even without runtimes (#3794)
go/staking: Improve pretty-print for Escrow/ReclaimEscrow transactions (#3819)
go/common/entity: Add {Min,Max}DescriptorVersion constants (#3845)
go/common/version: Add ValidateBasic() to Version and ProtocolVersions (#3845)
go/worker/compute/executor: Enable scheduler transaction checks by default (#3851)

Bug Fixes

oasis-net-runner: Configure a runtime in default fixture (#3805)
go/worker/storage: Fix Finalize and Apply failure handling (#3820)
go/storage: Fix failure handling in checkpoint syncing (#3830)

In some cases, the database could be left in a corrupt state after a checkpoint chunk failed to be restored.
go/consensus/tendermint: Bump Tendermint Core to v0.34.9-oasis2 (#3844)

Internal Changes

go: bump github.com/golang/protobuf from 1.4.3 to 1.5.1 (#3799)
go: bump github.com/prometheus/client_golang from 1.9.0 to 1.10.0 (#3803)
go: bump github.com/prometheus/common from 0.19.0 to 0.20.0 (#3811)
go: bump google.golang.org/grpc from 1.36.0 to 1.36.1 (#3816)
go/storage/client: Implement node blacklisting (#3820)
go/worker/storage: Add independent heartbeat for syncing retries (#3820)
go: bump github.com/golang/protobuf from 1.5.1 to 1.5.2 (#3821)
go/worker/common: Iterate over all handlers on unhandled peer message (#3838)
go/runtime/client: Also recheck blocks periodically (#3842)
go/consensus/api/transaction/testvectors: Extend meaning of the Valid field (#3845)

Extend meaning of TestVector's Valid field to also indicate that the given test vector's transaction passes basic static validation (besides having a valid signature and being correctly serialized).

Update MakeTestVector() and MakeTestVectorWithSigner() functions to take the valid argument indicating whether the transaction is valid or not.

Update go/registry/gen_vectors, go/staking/gen_vectors and go/governance/gen_vectors packages.

Expand go/governance/gen_vectors package to cover more (invalid) transaction cases.

21.0 (2021-03-18)

Protocol	Version
Consensus	3.0.0
Runtime Host	2.0.0
Runtime Committee	2.0.0

Removals and Breaking Changes

go/consensus/tendermint: Handle LIGHT_CLIENT_ATTACK evidence (#3156)
go/beacon: Add a PVSS based beacon backend (#3180)

This is a significant improvement over the existing beacon, in that it at least attempts to be secure.

For more details, see ADR 0007.

go: Use human readable identifiers for slashing reasons (#3276, #3646)

When serialized into JSON the slashing reason identifiers are now human readable (e.g. consensus-equivocation instead of 0).

Old:

"slashing": {
    "0": {
        "amount": "100",
        "freeze_interval": 1
    }
}

New:

"slashing": {
    "consensus-equivocation": {
        "amount": "100",
        "freeze_interval": 1
    }
}

go: Improve error message when delegating less than MinDelegationAmount (#3308)

Previously, an ErrInvalidArgument was returned in this case, which is not very descriptive, so a separate error was created: ErrUnderMinDelegationAmount.
go/worker/p2p: Include chain ID and remove patch version from topic IDs (#3311)
go: Move storage backend initialization to storage worker (#3323)

Notably, this also means that node command line options for storage configuration have been renamed and that the backend type option isn't needed anymore unless the storage worker is enabled.
Compare protocol versions according to SemVer 2.0.0 (#3360)

As described in our Versioning document, we bumped the protocol versions to version 1.0.0 with the release of Oasis Core 20.10.

Hence, we also need to modify how we compare and detect backward-incompatible changes to follow SemVer 2.0.0 rules.

From now onwards, only a change in a protocol's major version signifies a backward-incompatible change.
go/common/version: Remove MajorMinor() method from Version type (#3360)
roothash: Add support for failure indicating commitments (#3391)

Failure-indicating executor commits are introduced in order to give the compute nodes a possibility to vote for failure when they cannot execute the given batch (e.g., due to unavailability of storage). Such commits will always trigger a discrepancy during discrepancy detection and will vote for failing the round in discrepancy resolution phase.

For more details, see ADR 0005.
Introduce storage root types (#3415)

Each storage root now has an associated type (i.e. IO or state). In particular, this impacts the Apply and ApplyBatch calls and also changes the storage format of the Badger NodeDB.
go/staking: Add support for beneficiary allowances (#3428)

The staking service now supports setting allowances on general account balances as specified by ADR 0003.
Add runtime accounts in the consensus layer (#3429)

As described in ADR 0003 this adds support for runtimes having accounts in the consensus layer. Runtimes can manipulate those accounts by emitting newly introduced transfer and withdraw runtime messages.
go/roothash: Refactor runtime messages (#3430)

Several modifications are made to the runtime message representation and processing as specified by ADR 0003.
go/roothash: Renamed FinalizedEvent field in roothash.Event (#3448)

The FinalizedEvent field in the roothash.Event structure has been renamed to just Finalized in order to be consistent with other fields.

Note that the serialization already used just finalized so that remains unchanged.
Remove SignedRuntime (#3450)

As part of the work on ADR 0004, support for signed runtime descriptors has been removed, since they are no longer required. All methods that used to take signed runtime descriptors now take non-signed runtime descriptors instead.

This also affects the genesis file. Use the oasis-node debug fix-genesis command to convert a genesis file from the old format to the new one.
Modify runtime descriptors to support ADR 0004 (#3450)

As described in ADR 0004, the runtime descriptors are modified to enable runtime governance.
go/consensus/tendermint: Use canonical chain context in InitChain (#3477)
go/roothash/genesis: Restore state from suspended runtimes (#3484)

Roothash state of suspended runtimes in genesis was previously not correctly restored, causing the suspended runtime state to reset once the runtime was resumed.
Implement Consensus Governance backend (#3502)

As described in ADR 0006 this adds the consensus governance backend implementation.

See ADR 0006 and Governance backend documentation for more details.
runtime: Move consensus layer structures to its own module (#3511)

Everything under common::{address, registry, roothash, staking} in the oasis-core-runtime crate has been moved under the consensus module.
go/consensus/api: Rename ConsensusVersion to Version in Status type (#3511)
go/upgrade: Support multiple pending upgrades (#3536)

Upgrade module now supports submitting multiple upgrades.
go/registry/api: Allow updating more fields when updating an expired node (#3539)

Before, registering an expired (but not yet deleted) node was subject to the same update restrictions as an active node being updated.
go/oasis-node/cmd/stake: Enhance account info's output (#3557)

It now also displays an account's incoming and outgoing active/debonding delegations, total and available balance.
go/consensus: Include consensus genesis parameters in GetParameters (#3565)

Consensus GetParameters method response now includes the consensus genesis parameters.
runtime: Make CheckTx result more generic (#3573)

The previous definition made it hard for the runtime host to reason about the returned status code of CheckTx unless it was tied to the result format defined by the runtime.

This is a breaking change to the Runtime Host Protocol as the RuntimeCheckTxBatchResponse structure has changed. It also breaks the transaction::Dispatcher API.
go/upgrade: Use ProtocolVersions as internal upgrade identifier (#3579)

Before this change the internal upgrades used node binary hashes to ensure upgrade version compatibility. To make it more practical use software versions instead.
Add ability to set a non-zero initial commission schedule (#3601)

When setting the initial commission schedule for an account, the RateBoundLead restriction is ignored. Therefore the initial bounds and rates can be set for the first next epoch aligned with RateChangeInterval.

This effectively means that accounts without a configured commission schedule should be considered as having bounds: 0%-100%, as the initial schedule can be set at any next epoch.
go/scheduler: Check runtime versions for node eligibility (#3611)
go/roothash/api/message: Add RegistryMessage runtime message (#3628)

A new runtime message was added that enables runtimes to submit updated runtime descriptors for their runtime to the registry.
runtime: Remove common::runtime::RuntimeId (#3633)

The Go API has been using Namespace for a while so this makes it consistent in the Rust runtime as well.
go/roothash: Support slashing runtime compute nodes (#3640)

Runtimes can configure whether compute nodes should be slashed for submitting incorrect results or equivocating.

For more details, see ADR 0005.
runtimes: Support for Add/ReclaimEscrow runtime messages (#3641)

Escrow runtime messages need to be explicitly enabled via the AllowEscrowMessages consensus parameter.
go/roothash: Runtime domain separation context in executor commitments (#3643)
Add runtime scheduling constraints (#3644)

Each runtime can now impose additional scheduling constraints for its committee nodes (separately for primary/backup):
- Validator set membership.
- Maximum number of nodes per entity.
- Minimum candidate node pool size.
go/slashing/api: Change SlashReason type to uint8 (#3646)
go/staking/api: Rename DoubleSigning to ConsensusEquivocation (#3646)
go/registry: Fix node sanity checks (#3660)
Remove support for entity-signed node registrations (#3661)

Support for entity-signed node registrations has been removed, as they were only used in tests and not on any real network. Only node-signed node registrations are supported now.
go/registry: Change minimum ProposerTimeout from 5 to 2 blocks (#3682)
go/genesis/api: Change canonical form of genesis file to end with a newline (#3709)
go/staking/state: Remove nonce as the debonding delegations disambiguator (#3717)

Account nonce is removed from the debonding delegations state key. Instead, if there are multiple debonding delegations for the same account and debond end epoch, those get merged into a single debonding delegation record.
runtime: Introduce RoundResults to allow adding other information (#3726)

In addition to the existing runtime message execution results we want to provide additional information about the last successful round to runtimes. This introduces a RoundResults structure for this purpose.
go/staking/api: Rename Delegations() method to DelegationsFor() (#3774)
go/staking/api: Rename DebondingDelegations() method (#3774)

Rename it to DebondingDelegationsFor().
go/upgrade/api: Version upgrade Descriptor (#3781)
go/consensus: Store runtime state roots in a single key (#3782)

That makes it easier to construct proofs starting at the consensus state root that prove what the state root of a specific runtime is. You can then chain these proofs to prove something about runtime state.
go/staking/api/token: Print token's symbol after the amount (#3784)

This is more consistent with how others present token amounts.
go/consensus/api/transaction: Print txn's method and body before the rest (#3784)

This is more intuitive and enables people to discover the transaction's main information quicker.
go/consensus/tendermint: Bump Tendermint Core to v0.34.8-oasis1 (#3786)
go/roothash: Finish commitment processing early when possible (#3790)

When the set of current commitments already determines the fate of the process (e.g., when there is a discrepancy or a majority of votes indicate success), proceed with the process instead of waiting for the remaining commitments.
go/upgrade/api: Simplify upgrade Descriptor (#3793)

Remove UpgradeMethod type and Method field, rename Name field to Handler and replace Identifier field with Target field which corresponds to an upgrade's target version.
go/common/version: Remove Toolchain field from ProtocolVersions (#3793)

Configuration Changes

go: Change storage backend configuration options (#3323)

All the --storage.* options have been renamed to --worker.storage.*.

Nodes that don't have the storage worker enabled don't need to configure the storage backend anymore, since it will be chosen correctly automatically.
Change compute worker runtime configuration flags (#3653)

All the flags under worker.runtime.* have been moved under just runtime.*.

For example the worker.runtime.paths flag needs to be renamed to runtime.paths.

Features

go/common/version: Add MaskNonMajor() method to Version type (#3360)
go/worker/storage: Add validation for IO roots in Apply operations (#3440)
go/runtime/client: Add SubmitTxNoWait method (#3444)

SubmitTxNoWait method publishes the runtime transaction and doesn't wait for results.
Runtime client should fail SubmitTx calls until consensus is synced (#3452)
Runtime storage sync should use any storage node (#3454)

Before, storage node sync only used nodes from the current storage committee. Now it also syncs (with lower priority) from other storage nodes registered for the runtime.
go/oasis-node/cmd: Add oasis-node stake account validate_address CLI command (#3493, #3589)

It can be used to validate a staking account's address.
runtime: Add mkvs::Iterator trait and MKVS::iter trait method (#3499)
runtime: Add mkvs::OverlayTree similar to its Go counterpart (#3499)
go/storage/mkvs: Generate proofs even if Tree.Position is not found (#3499)
go/consensus/api: Add Registry()/Staking()/Scheduler() to client backend (#3511)

This makes it more convenient to use the consensus client backend.
go/oasis-node/cmd/stake: Add commands for managing allowances (#3511)

Add oasis-node stake account gen_allow and oasis-node stake account gen_withdraw CLI commands.
go/consensus/tendermint: Also dump state when shutting down for upgrade (#3516)
go/common/version: Add ConvertGoModulesVersion() function (#3546)

It can be used to convert a Go Modules compatible version defined in ADR 0002 (i.e. a Go Modules compatible Git tag without the go/ prefix) to the canonical Oasis Core version.
go/oasis-node/cmd: Add oasis-node stake account nonce CLI command (#3559)

It can be used to get a staking account's current nonce.
go/scheduler: Add ConsensusParameters method to scheduler backend (#3565)
go/epochtime: Add ConsensusParameters method to epochtime backend (#3565)
runtime/common/version: Make Version::new a const function (#3573)

Also expose members publicly.
go/runtime/client: Add GetEvents to fetch emitted runtime events (#3573)
go/worker/executor: Propose a new batch if there are message results (#3645)

The runtime executor now proposes a new batch if there are message results from previous round even if there are no transactions.
go/runtime/client: Perform local CheckTx when a hosted runtime exists (#3653)
runtime: Add support for runtime-specific queries (#3662)

Runtimes can now implement a handler for RuntimeQueryRequest messages to expose arbitrary queries to runtime clients. Query routing is performed using method names.
go/runtime/client: Also expose CheckTx (#3662)

The CheckTx functionality is now also exposed via the runtime client API.
go/worker/storage: Add StorageRPC role and change access policy (#3696)

Getting checkpoints and diffs is now allowed for any connecting node, which eliminates a race condition and initialization difficulty in the storage committee node startup.

State access is made gated but optionally public as before, depending on the value of the worker.storage.public_rpc.enabled configuration option.
go/common/diff: Add package implementing helpers for comparing objects (#3709)

Add UnifiedDiffString() which returns unified diff between two strings.
go/genesis/api: Update WriteFileJSON() to create files in the canonical form (#3709)

Consequentially, all its users (most notably the dump genesis halt hook) now produce genesis files in the canonical form.
go/genesis/api: Add CanonicalJSON() method to Document type (#3709)

It can be used to obtain the canonical form of the genesis document serialized into a file.
go/staking/api/token: Generalize PrettyPrintAmount() (#3725)

Support passing the amount as either quantity.Quantity or prettyprint.Quantity.
go/staking/api: Add StakeForShares() method to SharePool type (#3725)

It can be used to compute the amount of base units for the given amount of shares.
go/common/prettyprint: Add Quantity type (#3725)
runtime: Provide sets of good and bad compute nodes in round results (#3726)
go/control/GetStatus: Include the latest epoch (#3743)
go/runtime/registry: Add methods returning active runtime descriptors (#3771)

Runtime registry now includes methods returning the currently active runtime descriptor. Active descriptor is the runtime descriptor valid for the current epoch.
go/beacon: Add GetFutureEpoch method to beacon API (#3773)
go/consensus/tendermint/apps/staking/state: Add incoming delegation methods (#3774)

Add new methods to ImmutableState type for querying incoming (debonding) delegations to an escrow account: DelegationsTo() and DebondingDelegationsTo().
go/staking/api: Add DelegationsTo()/DebondingDelegationsTo() methods (#3774)

They can be used to obtain all incoming (debonding) delegations to the given account.
go/staking/api: Add DelegationInfo and DebondingDelegationInfo types (#3774)

They are (debonding) delegation descriptors with additional information about the share pool they belong to.
go/staking/api: Add DelegationInfosFor()/DebondingDelegationInfosFor() (#3774)

They can be used to obtain incoming (debonding) delegations with additional information for the given account.
runtime: Provide consensus layer chain domain separation context (#3778)

This adds a way for the runtime to be informed about the chain domain separation context that the consensus layer is using. It can be used by the runtime to perform domain separation for cryptographic signatures.
go/consensus: Add GetChainContext (#3778)

This makes it easier for clients to fetch the chain domain separation context as previously they needed to fetch the genesis document and compute the chain context from it. It also adds ChainContext to the node's consensus status report.
go/staking/api/token: Add support for passing token's value sign via context (#3780)
go/control/api: Add node status to registration status response (#3783)
go/oasis-node/cmd: Improve transaction preview when generating transactions (#3792)

Display amounts in tokens and display genesis document's hash when previewing transactions with various oasis-node * gen_* CLI commands.
go/oasis-node/cmd/common/context: Add GetCtxWithGenesisInfo() helper (#3792)
go/upgrade: Implement PrettyPrint interface for Descriptor (#3800)
go/governance: Implement PrettyPrint interface for various types (#3800)

Implement PrettyPrinter for ProposalContent, UpgradeProposal, CancelUpgradeProposal and ProposalVote types.
go/common/version: Implement PrettyPrint interface for ProtocolVersions (#3800)

Bug Fixes

go/consensus: Include only top K validators at genesis time (#3177)

Previously, all the eligible validators were included in the validator set at genesis time.

This has been fixed to include only the top MaxValidators as specified in the scheduler consensus parameters.

Additionally, the MaxValidatorsPerEntity scheduler consensus parameter is now enforced.
go/genesis: Fix epoch-based sanity checks (#3477)
go/storage/mkvs/checkpoint: Fix handling of non-zero earliest version (#3480)
go/consensus/tendermint: Fix early GetStatus() with initial height > 1 (#3481)
go: Add missing size checks to UnmarshalBinary() methods (#3497)

Note that in practice these will never currently be triggered as the caller always checks the overall size before calling the more specific UnmarshalBinary() method.
go/storage/mkvs: Fix edge case in overlay iterator (#3499)
runtime: Remove incorrect Sync impl on mkvs::Tree (#3499)
go/oasis-node/cmd/registry/entity: Reuse signer instead of creating a new one (#3505)

The oasis-node registry entity register CLI command previously always created two signer factories, one for signing the entity descriptor and one for signing the entity registration transaction.

Some signers assign exclusive access to an underlying resource (e.g., HSM) to the given factory. In that case, all operations on the second signer factory would fail.
Bump Go to 1.15.5 (#3512)

This fixes a recently disclosed security vulnerability in the Go runtime library.
go/worker/registration: Fix RequestShutdown for nodes with no registration (#3524)

In cases where the registration worker was started with a dummy goroutine that just waits for the service to stop, deregistration requests would hang, because the dummy wasn't watching the deregistration channel.
go/worker/storage: Fill in additional versions when restoring state (#3525)

When a storage database from one network is used in a new network (e.g. when the consensus layer did a dump/restore upgrade) properly handle the case where there were additional rounds after the runtime has stopped (e.g., due to epoch transitions).
go/consensus/tendermint: Report peers and validator status only after started (#3534)

When accessing the node status in very early stages of initialization when a Tendermint node structure is not available, the status RPC would make the node panic. Leave the peers and validator status blank instead.
go/worker/storage: Force checkpoint sync when replication is needed (#3538)

Previously a freshly initialized storage node with no genesis state would fall back to incremental sync even though there was no chance of that succeeding.
go/roothash/api/commitment: Return proposer commit in DR (#3545)

A subtle bug introduced in #3448 and discovered by daily long-term tests.
go/tendermint: Fix GetLightBlock,GetParameters queries for latest height (#3565)
go/runtime/scheduling: Consumer should pull from scheduler (#3569)

Previously the scheduler would push batches to the consumer (e.g. compute node) which makes no sense as the consumer knows when it's ready to accept new batches. This changes the model so that the consumer pulls batches from the scheduler.
go/consensus/tendermint/roothash: Fix latest block reindexing (#3594)

Only commit the block in case it was not already committed during reindex. This can happen when reindexing after a crash (e.g., the before_index crash point) since the initial height at which the reindex happens does not necessarily contain a round finalization event so reindexing up to height-1 doesn't help.

Discovered during long-term tests.
go/worker/common/committee: Wait for initial consensus sync (#3599)
go/consensus/tendermint: Fix nil dereference in EstimateGas (#3648)
go/oasis-node/cmd/node: Fix error handling when writing in dumpGenesis() (#3714)
go/cmd/fixgenesis: Add default beacon and governance parameters (#3728)
go/consensus: Move upgrade logic to governance, gracefully handle halt (#3755)
go/consensus: Exclude expired nodes from validator set at genesis time (#3763)

The validator set selection algorithm now properly ignores expired nodes.
go/upgrade: Ensure upgrade handler exists (#3768)
Allow switching binary while an upgrade is pending or in progress (#3768)

Removes RunningVersion/SubmittingVersion internal pending upgrade fields. Binary can now be switched mid-upgrade as long as it remains compatible with the in-progress upgrade.
Progress the startup upgrade stage only after a successful startup step (#3768)

Before, the startup state was progressed before the startup stage was run, therefore in case of a failed startup upgrade the stage would be skipped after the node restart.
runtime/scheduling: Fix ordered_map.UpdateConfig (#3771)

Before, UpdateConfig did not correctly update the list element reference causing an inconsistency of the underlying map and queue.
go/roothash/reindexBlocks: Return latest known round if no new rounds indexed (#3791)

This fixes a case where a storage node would not register if restarted while synced and there were no new runtime rounds (e.g. the runtime is suspended).
go/worker/executor: Avoid holding lock while applying processed batch (#3801)

Before, the executor worker would hold the CrossNode lock, while doing requests to storage.

Documentation Improvements

ADR 0003: Consensus/Runtime Token Transfer (#3262)
ADR 0004: Runtime Governance (#3284)
Document genesis document's hash (#3312)
Update Release Process documentation (#3327)
ADR 0005: Runtime Compute Node Slashing (#3339)
Rename Versioning scheme document to Versioning and include it in the index (#3360)
docs: Fix some links that break with gitbook (#3389)
ADR 0006: Consensus Governance (#3423)
ADR 0007: Improved Random Beacon (#3439)
Update Deploying a Runtime document's example (#3789)

Internal Changes

ci: Setup daily longterm tests with longer epoch periods (#3301)
Omit node operator link from release notes (#3313)
go: Bump github.com/prometheus/procfs from 0.1.3 to 0.3.0 (#3314, #3616)
Make: Output messages to stderr instead of stdout (#3327)
Make: Refactor Change Log handling (#3327)
internal: Refactor project's version handling (#3327, #3345, #3492)

Simplify version computation logic in Make files and start using Punch's version file to track version.
Make: Refactor release-related targets (#3327, #3368)
- Rename tag-next-release target to release-tag.
- Rename release target to release-build.
- Add release-stable-branch target that creates and pushes a stable branch for the current release.
github: Bump GoReleaser in ci-reproducibility and release workflows to 0.152.0 (#3327, #3368, #3602)
go/e2e/workload-queries: Allow historical QueryTxs to fail few times (#3338)
go/nancy: Remove golang.org/x/net false positive ignores (#3340)
go/dependencies/etcd: Update etcd with replace directive (#3341)
ci: Bump actions/setup-python from v2.1.2 to v2.2.1 (#3349, #3405, #3582)
rust: Bump base64 from 0.12.3 to 0.13.0 (#3353)
Prioritize nodes that signed storage receipts (#3354)

The compute executor and tag indexer now prioritize reads from storage nodes that signed the corresponding storage receipts.
go/storage: Support node prioritization in read requests (#3354)

The following new functions are added to the storage API package to help storage backend implementations:
- WithNodePriorityHint
- WithNodePriorityHintFromSignatures
- NodePriorityHintFromContext
ci: Bump actions/setup-node from v2.1.1 to v2.1.4 (#3364, #3563, #3575)
ci: Bump actions/setup-go from v2.1.2 to v2.1.3 (#3365)
ci: Bump actions/upload-artifact from v2.1.4 to v2.2.2 (#3367, #3520, #3596)
go/oasis-test-runner: Fix e2e/consensus-state-sync scenario (take two) (#3369)

Previous "fix" actually made the test silently skip actually performing state sync because it used the wrong validator index.
go/e2e/queries: Correctly ignore UnexpectedEOF errors (#3372)
ci: Make build-fuzz failures fatal now that upstream is fixed (#3374)
go: Bump github.com/blevesearch/bleve from 1.0.10 to 1.0.14 (#3384, #3530, #3558)
ci: Daily long-term test minor fixes (#3387)
- Don't fail on UnexpectedEOF during GetTransactionsWithResults.
- Enable "RecoverCorruptedWAL" for all nodes.
go/roothash: Add consistent hash test for ComputeResultsHeader (#3391)
go: Bump github.com/libp2p/go-libp2p-pubsub from 0.3.5 to 0.4.1 (#3392, #3510, #3624)
go/e2e/runtime/runtime-upgrade: Wait for old compute nodes to expire (#3404)
go: Bump github.com/golang/protobuf from 1.4.2 to 1.4.3 (#3408)
go/e2e: Add storage byzantine E2E test cases (#3414)
go: Bump github.com/prometheus/client_golang from 1.7.1 to 1.9.0 (#3419, #3576)
go: Bump github.com/spf13/cobra from 1.0.0 to 1.1.1 (#3427)
go: Bump google.golang.org/grpc from 1.32.0 to 1.36.0 (#3437, #3495, #3543, #3617, #3631, #3730)
rust: Bump rustracing (#3446)
- Bump rustracing from 0.4.0 to 0.5.0.
- Bump rustracing_jaeger from 0.5.0 to 0.6.0.
go/consensus/tendermint: Remove ForeignExecuteTx (#3448)

Use recently introduced message passing between apps instead.
go/consensus/tendermint: Add message passing between mux applications (#3448)
go/common/crash: Improvements to the debug crash framework (#3449)
- Threadsafe global crasher.
- Ability to configure default probability for all crash points.
- Fix reported caller information on crash.
- Crash with a specific exit code.
Add GetEntityNodes() query to registry state (#3450)
go: Bump github.com/hashicorp/go-plugin from 1.3.0 to 1.4.0 (#3494)
go: Bump github.com/hashicorp/go-hclog from 0.14.1 to 0.15.0 (#3496)
go/oasis-node/cmd/common/consensus: Augment SignAndSaveTx() with signer (#3506)

Add ability to pass a pre-existing signature.Signer as signer parameter to the SignAndSaveTx() function.
go: Bump github.com/libp2p/go-libp2p-core from 0.6.1 to 0.8.5 (#3510, #3711, #3718)
go: Bump github.com/libp2p/go-libp2p from 0.11.0 to 0.13.0 (#3510, #3588)
rust: Bump arc-swap to 0.4.8 (#3511)
ci: Bump actions/download-artifact from v2.0.5 to v2.0.8 (#3519, #3571, #3597)
github: Replace deprecated set-env workflow command with environment file (#3522)
go/storage/mkvs/checkpoint: Add initial version parameter (#3538)

Previously, if the local database contained a version earlier than the genesis version, the checkpointer would attempt to create a new checkpoint at that earlier version (and fail). Now the version is clamped at the initial version.
ci: Bump docker/build-push-action from v1 to v2.2.2 (#3549, #3605)
go/oasis-node/cmd/common: Add LoadEntitySigner() helper (#3556)

Replace LoadEntity() helper with LoadEntitySigner() that requires no arguments and loads the entity directory and obtains the signer backend by itself.
go/oasis-node/txsource: Increase submission timeout (#3568)

Sometimes it can take up to two minutes for a transaction to get included in a block due to high load.
dependencies: Update bubblewrap to 0.4.1 (#3577)
go/oasis-node/txsource: Fix event validation in queries workload (#3593)

A single transaction can now emit multiple identical events when performing runtime message execution emitting events.
go/e2e/txsource: Add txsource SGX scenario using fewer nodes (#3595)
go/oasis-test-runner/txsource: Add governance workload (#3603)
rust: Bump smallvec to 0.6.14 (#3604)
runtime/storage/mkvs: Add MKVS impl for &mut T (#3604)
go/e2e/governance-upgrade: Wait for compute nodes to be ready (#3618)
rust: Bump honggfuzz from 0.5.51 to 0.5.52 (#3621)
rust: Bump slog-scope from 4.3.0 to 4.4.0 (#3622)
go/common/cbor: Re-export Marshaler and Unmarshaler interfaces (#3630)
go: Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 (#3632)
runtime: Validate block namespace against runtime ID (#3633)
go/oasis-test-runner/txsource: Decrease MinPoolSize for restarts (#3636)

When restarts are enabled there can be one less node so we should make sure to decrease the MinPoolSize in this case.
go/runtime: Move host configuration to runtime registry (#3653)
go/runtime/host: Add helpers for common requests (#3653)

An additional interface RichRuntime is added which provides wrappers for common requests (currently only CheckTx).
go: Bump ed25519 for fixes on 32-bit architectures (#3657)

Note that 32-bit architectures are not supported by Oasis Core.
runtime: Refactor the dispatcher (#3662)

Previously, each request handler performed its own response submission which repeated a lot of code. This has now been changed by all handlers returning a proper result type and the main dispatch loop sending the responses.
tests: Fix potential key clash in the confidential test KV runtime (#3663)

This was responsible for some E2E test flakiness.
beacon: Add a minimal gRPC service (#3666)
go/worker/beacon: Fix a state recovery edge case (#3668)
go/consensus/tendermint/abci/mux: Reject early EstimateGas calls (#3669)

With the new beacon, EstimateGas will not work till either InitChain or BeginBlock have been called successfully at least once.
go/consensus: Add support for method priority (#3675)
go/oasis-node/cmd/common/metrics: Fix label escaping (#3678)
Make sure tracing is only enabled in debug mode (#3679)
rust: Bump bech32 from 0.7.2 to 0.8.0 (#3699)
go/oasis-test-runner: Add support for configuring pprof (#3719)

Adds support for configuring pprof on the test nodes.
go/oasis-test-runner: Configurable supplementary sanity checker (#3719)

Previously, supplementary sanity checker was always automatically started on the validator-0.
Make: Remove build-go target's go alias (#3722)
Make: Add lint-go-mod-tidy target (#3722)
go: Bump github.com/golang/snappy from 0.0.2 to 0.0.3 (#3735)
rust: Bump aesm-client from 0.5.1 to 0.5.2 (#3742)
runtime/common/quantity: Add Mul/Div implementations (#3744)
go/nodes/grpc: Tweak backoff timeouts (#3757)

Before, the default storage commit timeout was less than grpcBackoffMaxDelay which made the storage commit request retries ineffective whenever the max delay was reached (e.g. first few requests after a storage node was restarted).
go/consensus: Validate mode in SetContextParameters (#3773)

This makes sure consensus parameters can only be updated in InitChain and EndBlock.

Previously, we only did this check for the core consensus parameters but it makes sense to extend it to all of the parameters.
go/staking/tests/debug: Move into go/staking/tests package (#3774)
go/staking/tests: Generalize account handling and add delegations (#3774)
go: Bump github.com/hashicorp/go-multierror from 1.1.0 to 1.1.1 (#3777)
go: Bump github.com/prometheus/common to 0.19.0 (#3779)

20.12 (2020-11-03)

Protocol	Version
Consensus	2.0.0
Runtime Host	1.0.0
Runtime Committee	1.0.0

Removals and Breaking Changes

go/common/crypto: Bump ed25519 version (#3458, #3470, #3471)

Configuration Changes

go/runtime/scheduling: Max queue size flag rename (#3434)
- worker.executor.schedule_max_queue_size -> worker.executor.schedule_max_tx_pool_size
go/runtime/client: Add max transaction age (#3443)

Added runtime.client.max_transaction_age flag to configure number of consensus blocks after which a submitted runtime transaction is considered expired. Expired transactions are dropped by the client.

Features

go/runtime/scheduler: Switch to an ordered queue tx pool implementation (#3434)

Bug Fixes

go/runtime/client: Runtime client should retry processing any failed blocks (#3412)
go/storage/client: Retry ErrNodeNotFound errors on write requests (#3424)
go/worker/common: check if active epoch exists in HandlePeerMsg (#3432)

Fixes nil pointer dereference that can happen if the executor node tries to publish a message before it is synced
go/runtime/client: Wait for initial consensus block and group version (#3443)

Before, the runtime client would publish invalid messages before obtaining the initial group version. The messages were correctly retired upon receiving the group version, but this resulted in needless messages.
Storage node should update access control policy on new node registrations (#3453)

Before, the storage node only updated policy when existing nodes updated registrations or committee changed. This missed the case when new storage node registered mid-epoch.
rust: Bump futures to 0.3.7 (#3460)

Fixes RUSTSEC-2020-0059.

20.11.3 (2020-10-16)

Protocol	Version
Consensus	1.0.0
Runtime Host	1.0.0
Runtime Committee	1.0.0

Bug Fixes

go/storage/mkvs: Fix Finalize after checkpoint restore (#3330)

Internal Changes

go/storage/mkvs: Add node tracking to chunk restore (#3241)
go: Add ability to set initial block height at genesis (#3416)

20.11.2 (2020-10-13)

Protocol	Version
Consensus	1.0.0
Runtime Host	1.0.0
Runtime Committee	1.0.0

Features

go/runtime/client: remove unneeded P2P message republishing (#3325)

Bug Fixes

go/worker/p2p: Skip peer authentication for our own messages (#3319)

In practice this fixes a bug in a setup where executor nodes are used to submit runtime transactions. The executor nodes that are not part of the active committee, would end up self-rejecting transaction messages.
go/worker/executor: don't unnecessarily hold the lock during storage requests (#3320)
go/worker/executor: only remove incoming transactions on finalized round (#3332)

Simplifies the executor to only remove transactions from the incoming queue once a round is successfully finalized. Before, the proposing executor also removed transactions when proposing a batch, which was an unneeded leftover from before the transaction scheduler committee was merged with into executor.

Also fixes an edge case where batch was not reinserted on failed rounds.
go/worker/common/committee: group version should be the epoch block (#3347)

When node is restarted, EpochTransition is called on the first received block, which is not necessary the actual epoch transition block. Therefore the epoch block needs to be queried to obtain the correct group version.
go: Seed node support configuring tendermint P2P settings (#3356)

Seed node now supports configuring relevant tendermint P2P settings via existing tendermint configuration flags.
go: Seed node panics on oasis-node control CLI (#3356)

Seed node now implements the missing consensus backend methods and supports oasis-node control CLI.
go/worker/p2p: Correctly update peers on WatchNodeList events (#3378)
go/registry.WatchNodeList: return current node list upon subscription (#3379)
go/worker/p2p: Relay some permanent errors (#3386)

Before the P2P worker would not relay any messages failing with a permanent error. However there are cases where the client should permanently fail the dispatch, but still relay the message to peers.

Adds Relayable error wrapper which can be used in handlers to notify that a message should be relayed regardless if the error is permanent or not.
go/worker/p2p: Retry messages we consider permanent (#3386)

p2pError.IsPermanent notion of a permanent error differs from the upstream cenkalti/backoff/v4 notion. Correctly retry on context.Canceled errors as we don't consider them permanent.
go/consensus/tendermint: Backport bug fixes (#3398)

Internal Changes

go/backoff: Update cenkalti/backoff to 4.1.0 (#3390)
rust: bump crossbeam from 0.7.3 to 0.8.0 (#3394)

20.11.1 (2020-10-07)

Protocol	Version
Consensus	1.0.0
Runtime Host	1.0.0
Runtime Committee	1.0.0

Features

go/consensus: add last retained block information (#3348)

New in the consensus API, the status struct now has information about the earliest block available on this node. This differs from the genesis block when pruning is enabled.

Internal Changes

go: Update to build on macOS (#3333)

20.11 (2020-09-23)

Protocol	Version
Consensus	1.0.0
Runtime Host	1.0.0
Runtime Committee	1.0.0

Configuration Changes

go/worker/p2p: Configurable libp2p buffer sizes (#3264)

Added worker.p2p.peer_outbound_queue_size and worker.p2p.validate_queue_size flags for configuring libp2p buffer sizes.

Features

Introduce the canonical form of a Genesis file (#2757)

This form is the pretty-printed JSON file with 2-space indents, where:
- Struct fields are encoded in the order in which they are defined in the corresponding struct definitions.
- Maps have their keys converted to strings which are then encoded in lexicographical order.
For more details, see the Genesis Document documentation.

The oasis-node genesis init and oasis-node genesis dump CLI commands are updated to output Genesis file in this canonical form.
go/staking/api: Add PrettyPrintCommissionScheduleIndexInfixes() helper (#3265)
go/staking/api: Update commission schedule rate and rate bound pretty prints (#3265)

Pretty print rates and rate bounds as enumerated lists to enable easier inspection of commission schedule (amendments) in combination with a hardware-based signer plugin.
go/worker/executor: Cache last seen runtime transactions (#3274)

To enable a basic form of runtime transaction replay prevention, the transaction scheduler maintains a LRU cache of last seen runtime transactions keyed by transaction hash.
go/worker/p2p: Use hash of the message payload as message ID (#3275)
go/oasis-node: Make oasis-node genesis check command check if form canonical (#3298)

Bug Fixes

runtime/client: Skip not yet published requests in retry check (#3264)
go/consensus/tendermint: Don't swallow recheck result errors (#3267)
go/runtime/client: Fix possible panic on shutdown (#3271)
go/worker/registration: Verify entity exists before node registration (#3286)

This avoids some cases of failed node registration transactions when the entity under which the node is being registered does not actually exist.
go/runtime/host/protocol: Set connection write deadline (#3289)

The RHP connection should have a write deadline to make sure that the connection is closed in case the runtime is not working.
go/worker/executor: Clear tx queue before returning to committee (#3300)

Fixes a bug where an executor returning to committee would propose stale transactions that it received right before exiting committee in previous epoch, due to a race condition between adding transaction to the queue and clearing the queue. Clearing the incoming queue is now done before node starts being a compute worker instead of after it stops being.
go/oasis-node: Start internal gRPC server before consensus backend (#3303)

Documentation Improvements

Document oasis-node genesis CLI commands (#3298)

Internal Changes

ci/txsource: Shut down nodes for longer period of time (#3223)
rust: Update tiny-keccak, hmac, sha2 dependencies (#3260)
go/extra/stats: Remove the unmaintained extra/stats tool (#3270)
go/oasis-test-runner: Add e2e/genesis-file scenario (#3278)
go/ci: Update nancy to 1.0.0 (#3283)
go/consensus/tests: Use explicit latest height when comparing query results (#3287)

Fixes flakiness in the test that occurred when a different height was used for the GetTransactions and GetTransactionsWithResults queries.
go/e2e/sentry: Sanity check consensus peers (#3291)

Sentry E2E test now ensures that validators running behind sentries only connect to their sentry nodes.
rust: Bump serde crates (#3294)
- serde_cbor@0.10.2 -> serde_cbor@0.11.1
- serde@1.0.71 -> serde@1.0.116
- serde_bytes@0.10.5 -> serde@0.11.5
rust: Bump snow version from 0.6.2 to 0.7.1 (#3296)
go: Bump github.com/golang/snappy from 0.0.1 to 0.0.2 (#3297)
rust: bump ed25519-dalek from 1.0.0 to 1.0.1 (#3299)

20.10 (2020-09-09)

Protocol	Version
Consensus	1.0.0
Runtime Host	1.0.0
Runtime Committee	1.0.0

Removals and Breaking Changes

go/staking: Remove WatchTransfers/Burns/Escrows in favor of WatchEvents (#3080)

The separate WatchTransfers/Burns/Escrows methods provided less information than the more general WatchEvents, namely they were missing the height and tx hash. There is no good reason to maintain both as the individual methods can be easily replaced with WatchEvents.
go/roothash: Drop support for multiple committees (#3179)

Since there is currently no transaction scheduler implementation which would support multiple committees, there is no sense in having the merge node as it could be a source of bugs.

The merge node is also the only client for the Merge* storage operations, so they can just be removed in order to reduce the exposed API surface.
go/worker/executor: Support multiple transaction schedulers (#3184)

Transaction scheduling committee is removed and the transaction scheduler worker is merged into the executor worker. Transaction scheduling gRPC service is removed and runtime transaction submission is now done via libp2p's gossipsub.

Each active executor worker now also acts as a transaction scheduler. Each round one of the executors acts as the scheduler and is expected to propose a batch for scheduling. Nodes switch between schedulers in round-robin fashion.

Metric changes:

Existing transaction scheduler incoming queue size metrics was renamed:
- oasis_worker_txnscheduler_incoming_queue_size -> oasis_worker_incoming_queue_size
go/runtime/scheduling: Rename "batching" algorithm to "simple" (#3184)

Note: Existing deployments will need to alter the state dump and fix the scheduling algorithm in all registered compute runtimes.
go/consensus/tendermint: Bump Tendermint Core to v0.34-rc4-oasis2 (#3186, #3229)
go/roothash: Add support for executor triggered timeouts (#3199)

Executors can now trigger a roothash timeout in case there are new transactions waiting to be proposed, but the current transaction scheduler is not proposing a batch. The timeout can only happen once sufficient (configurable via a new ProposerTimeout runtime parameter) consensus blocks pass since the round started. The proposer timeout causes the current round to fail.

Note: Existing deployments will need to alter the state dump to fix existing runtimes that do not have ProposerTimeout configured.
go/common/cbor: Bump fxamacker/cbor to bafca87fa6db (#3204)

This should result in some smaller payloads as omitempty should work better for our use cases now.
go/roothash: Change runtime round timeout to be in blocks (#3210)

Note: Existing deployments will need to alter the state dump and update the executor round timeout in all registered compute runtimes.
go/common/crypto/signature: Remove SignerRole.FromString() method (#3225)

Use the newly added SignerRole.UnmarshalText() method instead.
go/consensus: Simplify light client API (#3229)

Methods GetSignedHeader and GetValidatorSet have been replaced with GetLightBlock which provides both the signed header and the validator set. This makes sense as the two are commonly used together so this saves a round-trip.
release: Wrap files in the release tarball in a single directory (#3232)
release: Use SHA256SUMS-<VERSION>.txt name template for checksum files (#3232)
Bump protocol versions to 1.0.0 in preparation for the Mainnet (#3249)

As described in our Versioning scheme, we will bump the protocol versions (Consensus, Runtime Host, Runtime Committee) to version 1.0.0 when preparing an Oasis Core release for the Mainnet, which signifies they are ready for production use.

Configuration Changes

Added worker.p2p.enabled flag to explicitly enable P2P worker (#3184)

Note: compute workers will automatically enable the P2P worker. The P2P worker needs to be manually enabled on runtime-client nodes that want to submit runtime transactions.
go/worker/compute/executor: Remove transaction scheduler worker (#3184)

The worker.txn_scheduler.check_tx.enabled flag has been renamed to worker.executor.schedule_check_tx.enabled. The worker.txnscheduler.batching.max_queue_size flag has been renamed to worker.executor.schedule_max_queue_size.
go/runtime/scheduling: Rename "batching" algorithm to "simple" (#3184)

The following oasis-node registry runtime command line flags have been renamed:
- runtime.txn_scheduler.batching.max_batch_size to runtime.txn_scheduler.max_batch_size.
- runtime.txn_scheduler.batching.max_batch_size_bytes to runtime.txn_scheduler.max_batch_size_bytes.
go/common/crypto/signature: Use descriptive names for Signer roles (#3225)

The --signer.composite.backends CLI flag previously accepted integer-indexed Signer roles, e.g:
```
--signer.composite.backends 1:file,2:file,3:file,4:plugin
```
Now, it only accepts descriptive string names for Signer roles, e.g.:
```
--signer.composite.backends entity:file,node:file,p2p:file,consensus:plugin
```

Features

go/consensus/api/transaction: Pretty-print transaction's fee amount in tokens (#3151)
go/consensus/api/transaction: Implement PrettyPrinter interface for Fee (#3167)
go/staking/api/token: Initial implementation (#3167)
go/consensus/tendermint: Support configurable initial height (#3186)
go/consensus: Add the GetUnconfirmedTransactions method (#3187)

The new method allows the caller to query the current set of transactions in the mempool (e.g., known transactions which have not yet been included in a block).
go/consensus: Add ErrDuplicateTx error for duplicate transactions (#3192)
go/common/crypto/signature: Add new methods to SignerRole type (#3225)

Add String(), MarshalText() and UnmarshalText() methods to SignerRole type.

Add SignerEntityNode, SignerNodeName, SignerP2PName, SignerConsensusName constants that represent the names of the corresponding Signer roles.
release: Add oasis-remote-signer binary to the release tarball (#3226)
go: Unify CLI tools' version display and also display the Go toolchain version (#3233)
go/oasis-node/cmd/common: Add SetBasicVersionTemplate() function (#3233)

It can be used to set a basic custom version template for the given cobra command that shows the version of Oasis Core and the Go toolchain.
go/control: Add per-runtime status report (#3234)

An additional runtimes field has been added to the output of the control.GetStatus method (e.g., as reported by control status subcommand). The field contains a map of supported runtime IDs to their runtime-specific status reports.

Bug Fixes

go/common/crypto/drbg: Also consider empty slices as Null values (#3165)

The wrong handling of an edge case in the HMAC_DRBG implementation has been corrected. An update with empty additional_data now behaves the same as an update with nil additional data. While the spec is not 100% clear around how this is to be treated, supplemental documentation suggests that this is the correct way to handle it.

Oasis code never uses HMAC_DRNG with a non-nil empty additional_data parameter, so nothing changes for Oasis users.
go/consensus/tendermint: Sync state database before discarding versions (#3173)

Otherwise a crash can cause the state database to be rolled back to a version that has already been discarded from Tendermint's state stores which would prevent replay on restart.

Discovered during long-term tests.
go/genesis: Remove time sanity check (#3178)

Previously the genesis document sanity check rejected genesis documents with future timestamps which made it awkward to prepare the node in advance. Since the only supported consensus backend (Tendermint) can handle future timestamps by delaying the consensus process, allow such genesis documents.
go/registry/runtime: Validate runtime transaction scheduler parameters (#3184)

Note: Existing deployments might need to alter the runtime state dump in case existing registered runtimes have invalid parameters configured.
runtime: Use separate cache for checking transactions (#3191)

This allows calling both check and execute methods against the same runtime instance.
Executor should refresh runtime scheduling parameters (#3203)

Fixes the executor node to watch for runtime scheduling parameter changes and if needed update its scheduling configuration.
go/roothash: Make the parent block check earlier (#3206)
go/registry: Correctly propagate lookup errors (#3209)
go/oasis-node: Omit existing entity check for non-file signers (#3215)

The "registry entity init" subcommand previously always performed a check whether an entity already exists. It did that by creating an additional signer factory to perform this check.

Some signers assign exclusive access to an underlying resource (e.g., HSM) to the given factory. In that case, all operations on the second signer factory would fail. Thus we now omit the existing entity check for non-file signers.
go: Fix CLI tools to not print error messages twice (#3230)
go/runtime/transaction: Ensure consistent batch order indices (#3248)
go/worker/storage: Always use fresh nodes for policy updates (#3252)
go/worker/common: Use group-synced storage client (#3253)

Previously the runtime worker(s) used the common storage client which was not synced with any particular committee version. This could cause an executor node to use a stale storage node for storing updates.

Documentation Improvements

Add documentation example on the process of registering and running a runtime (#3081, #3207, #3228)
Expand documentation on test vectors (#3205)
Fix staking account address derivation description to match what the code does (#3240)
ADR 0002: Go Modules Compatible Git Tags (#3242)

Go Modules only allow Semantic Versioning 2.0.0 for versioning of the modules which makes it hard to work with Oasis Core's CalVer (calendar versioning) scheme.

Design a scheme for tagging Oasis Core releases with Go Modules compatible Git tags (in addition to the ordinary Git tags).

Internal Changes

go/storage/mkvs: Add NoPersist commit option (#2186)

Using the NoPersist commit option makes the Commit only compute all the hashes but does not persist any roots in the database.
go/oasis-test-runner: Add support for funding entities in genesis (#3081)

Additionally, make the default fixture command a bit more configurable.
go/worker/storage: Add initial sync from checkpoints (#3181)

Instead of relying on the slow per-block root sync, the worker now tries syncing from checkpoints, if any suitable are found.
Update Badger version to v2.2007.2 (#3182, #3227)
go/oasis-test-runner: Fix e2e/consensus-state-sync scenario (#3194)

Instead of terminating the validator-to-be-synced immediately and restarting it later, do not even start it. Early stopping could result in state that prevents proper state sync later.
Bump Go to 1.15.1 (#3197)
go: Explicitly use public key pinning for certificate verification (#3197)

While we only ever use public key pinning for authenticating TLS connections, some places still used the regular TLS config with a single certificate in the certificate pool. This causes failures on Go 1.15+ due to CommonName checks being deprecated, even if we never used hostnames for authentication.

This changes all cases to use our explicit public key pinning credentials for gRPC connections.
tests/txsource: Run queries workload against all nodes (#3209)
tests/txsource/queries: Verify consensus state integrity (#3209)
go/common/grpc: Add IsErrorCode helper method (#3209)
go/registry: Refactor runtime descriptor validity checks (#3210)
go/common/keyformat: Add support for int64 (#3210)
go/roothash/tester: Fix flaky RoundTimeoutWithEpochTransition (#3214)
ci: run the daily test for 12 hours (#3219)
Bump Rust toolchain to nightly-2020-08-29 for LVI mitigation speedups (#3231)
go: Bump bleve to 1.0.10 (#3231)
Update Go dependencies (#3238, #3255)
- cenkalti/backoff/v4: v4.0.0 -> v4.0.2
- grpc-ecosystem/go-grpc-middleware: f849b5445de4 -> v1.2.1
- hashicorp/go-multierror: v1.0.0 -> v1.1.0
- libp2p/go-libp2p: v0.10.2 -> v0.11.0
- libp2p/go-libp2p-pubsub: v0.3.3 -> v0.3.5
- multiformats/go-multiaddr: v0.2.2 -> v0.3.1
- prometheus/common: v0.10.0 -> v0.13.0
- google.golang.org/grpc: v1.31.0 -> v1.32.0
- google.golang.org/protobuf: v1.23.0 -> v1.24.0
Also updates the Go version in go.mod to 1.15.
Configure dependabot (#3239)

Configures dependabot for Go, Rust and Github Actions.
internal: Document how to include protocol versions in the Change Log (#3249)
go/runtime/committee: Support filtering nodes by tags (#3253)
Make: Augment tag-next-release to create a Go Modules compatible Git tag (#3258)

This implements the tagging scheme described in ADR 0002: Go Modules Compatible Git Tags.

20.9 (2020-08-05)

Process

Introduce Architectural Decision Records (#2924)
Remind to bump protocol versions before release (#3097)

Removals and Breaking Changes

go/registry: Add MinWriteReplication to runtime storage parameters (#1821)

The MinWriteReplication specifies the minimum number of nodes to which any writes must be replicated before being assumed to be committed.
go/common/cbor: Reject CBOR blobs with unknown fields (#2020)
go/consensus: Enable periodic state checkpoints (#2880)

This adds the following consensus parameters which control how state checkpointing is to be performed (currently not enforced):
- state_checkpoint_interval is the interval (in blocks) on which state checkpoints should be taken.
- state_checkpoint_num_kept is the number of past state checkpoints to keep.
- state_checkpoint_chunk_size is the chunk size that should be used when creating state checkpoints.
go/consensus/tendermint: Bump Tendermint Core to 0.34 (#2882)
go/worker/common/p2p: Use libp2p's gossipsub (#3010)
go/registry: Add up node thresholds for each runtime (#3027)
go/registry: Make TLS address required for RoleConsensusRPC (#3038)
go/consensus/tendermint: Fix node update verification (#3040)
runtime: Cleanup some runtime host protocol messages (#3055)
go: Rename existing staking token to base unit (#3061)

This allows introducing the concept of staking token which is defined as a given number of base units.

Additionally, rename fields of the following go/staking/api types: TransferEvent, BurnEvent, AddEscrowEvent, TakeEscrowEvent, ReclaimEscrowEvent, Transfer, Burn, Escrow, ReclaimEscrow.
go/staking: Increase reward amount denominator (#3065)

This allows the genesis block writer to specify finer reward rates.

Associated genesis controls analysis:
```
RewardAmountDenominator
|- AddRewards
|  |- RewardFactorEpochElectionAny
|  '- RewardFactorEpochSigned
'- AddRewardSingleAttenuated
   '- RewardFactorBlockProposed
```
Note to the genesis block writer: scale rewards factors up by a factor of 1,000.
go/registry/api: Remove GetNodeList method (#3067)

The GetNodeList method was unused and is therefore removed. Any code using this method can be migrated to use GetNodes instead.
go/roothash/api: Include RuntimeID field in roothash events (#3073)
go/tendermint/roothash: Include RuntimeID attribute in events (#3073)

All tendermint roothash events now include "runtime-id" attribute.
go/roothash/api: Add FinalizedEvent type to roothash events (#3073)
go/registry/api: Include transaction hash and height in all registry events (#3073)
go/backends: Return pointers in GetEvents methods (#3092)

To avoid unnecessary copying and to make the methods more unified with rest of the APIs, return pointers to events in backend GetEvents methods.
go/consensus/tendermint: Use P2P key for Tendermint P2P (#3101)

Previously the Tendermint consensus backend used the node's identity key for Tendermint P2P connections. This has now been changed to use the node's P2P key instead as that can be made ephemeral in the future.
go/oasis-test-runner/env: Rename types, fields, functions to refer to scenario (#3108)

Rename TestInstanceInfo type to ScenarioInstanceInfo and its Test field to Scenario.

Rename Env's TestInfo() and WriteTestInfo() methods to ScenarioInfo() and WriteScenarioInfo() for consistency.
go/oasis-test-runner/cmd: Rename --test flag to --scenario flag (#3108)

Rename the short version from -t to -s.
go/oasis-node/cmd/common/metrics: Rename "scenario" metrics label to "test" (#3108)
go/common/prettyprint: Augment PrettyPrinter.PrettyPrint() with Context (#3111)
go/oasis-node/cmd: Remove --retry flag from staking CLI commands (#3113)

Remove obsolete --retry flag from oasis-node stake info, oasis-node stake list and oasis-node stake account info CLI commands.
go: Remove the integrated ledger support (#3128)

The signer and associated enumeration tool have been moved to oasisprotocol/oasis-core-ledger.
go/oasis-node/cmd/common/consensus: Add context.Context to SignAndSaveTx() (#3137)
go/registry: Require SGX for non-test keymanager runtimes (#3146)

Note: Existing deployments might need to alter the state dump to fix any existing keymanager runtimes that registered without SGX hardware.
go/registry: Require SGX for non-test compute runtimes using a key manager (#3159)

Note: Existing deployments might need to alter the state dump to fix any existing compute runtimes that registered without SGX hardware and have keymanager runtime configured.

Configuration Changes

Remove explicit evidence-related consensus parameters (#2882)

The following evidence-related consensus parameters have been removed as they are now derived based on the debonding period and other parameters:
- max_evidence_age_blocks
- max_evidence_age_time
Make sure to update the genesis file.
go/consensus/tendermint: Make configuration options consistent (#3082)

All Tendermint configuration option names have been changed to be consistently placed under consensus.tendermint.. This requires any previous options that started with tendermint. to be changed to start with consensus.tendermint..
Change the way seed-only nodes are configured (#3116)

The previous --consensus.tendermint.p2p.seed_mode configuration flag has been removed. In its place there is now a more general --consensus.tendermint.mode flag which should be set to seed in order to make the node a seed-only node.

Features

go/storage/client: Use MinWriteReplication when waiting for writes (#1821)
registry: Allow nodes to opt-in to more runtimes (#2179)

Previously, when a node updated its registration, the list of runtimes had to be identical. It is now possible to add new runtimes.
e2e/tests: Added runtime upgrade test (#2520)
go oasis-node: Add CLI command for estimating gas cost of a transaction (#2723)

This adds oasis-node consensus estimate_gas.

This adds --transaction.unsigned.
go/oasis-node/cmd: Show Genesis document's hash when displaying transactions (#2871)

Show Genesis document's hash when generating staking transactions with oasis-node stake account gen_* CLI commands and when showing transactions' pretty prints with the oasis-node consensus show_tx CLI command.
go/consensus/tendermint: Add support for state sync (#2880)
go/registry/api: Add GetNodeByConsensusAddress method (#2961)

GetNodeByConsensusAddress can be used to query nodes by their Consensus address.
go/identity/cli: Add show TLS pubkey commands (#3015)

Adds following CLI helpers for displaying TLS public keys:
- oasis-node identity show-tls-pubkey --datadir <datadir> for displaying the public key used in the external node gRPC endpoints.
- oasis-node identity show-sentry-client-pubkey --datadir <datadir> for displaying the public key used by the upstream nodes when connecting to the sentry control endpoint.
go/oasis-node/cmd/stake: Support showing stake amounts in tokens (#3037)

The oasis-node stake info and oasis-node stake account info CLI commands show stake amounts (e.g. account balances, staking thresholds) in tokens.
go/staking/api: Pretty-print account balances in tokens (#3037)

If a PrettyPrinter's context carries appropriate values for the token's ticker symbol and token's value base-10 exponent, print the balance in tokens instead of base units.
go/control: Add registration status to node status (#3038)

This updates the response returned by the GetStatus method exposed by the node controller service to include a Registration field that contains information about the node's current registration.
go/control: Add identity status to node status (#3038)

This updates the response returned by the GetStatus method exposed by the node controller service to include an Identity field that contains information about the public keys used to identify a node in different contexts.
go/consensus: Add IsValidator to reported node status (#3038)
go/consensus/api: Add GetTransactionsWithResults method (#3047)

GetTransactionsWithResults returns a list of transactions and their execution results, contained within a consensus block at a specific height.
go/common/crypto/signature/signers/ledger: Descriptive error on user reject (#3050)

Make Ledger signer return a more descriptive error message when a user rejects a transaction on the Ledger device.
go/common/crypto/signature/signers/ledger: Add support for consensus signer (#3056)

Add support for consensus signer that can be used with the Oasis Validator Ledger app.
go/oasis-node/cmd/genesis: Allow setting staking token symbol and value exp (#3061)

Allow setting staking token's ticker symbol and token value's base-10 exponent in oasis-node genesis init CLI command via --staking.token_symbol and --staking.token_value_exponent flags.
go/staking: Add TokenSymbol and TokenValueExponent fields to Genesis (#3061)

They allow denominating stake amounts in tokens (besides base units used internally). For more details, see Staking Developer Docs.
go/control: List all valid TLS public keys in identity status (#3062)

This changes the Identity field in the reposnse of the GetStatus method exposed by the node controller service to include all valid TLS public keys for the node. This change affects nodes using automatic certificate rotation, which at any point use 2 valid TLS public keys.
go/oasis-node: CLI support for the GetStatus endpoint (#3064)

See docs/oasis-node/cli.md#status.
go/consensus: Expose read-only state via light client interface (#3077)

Nodes configured as consensus RPC services workers now expose read-only access to consensus state via the usual MKVS ReadSyncer interface, allowing light clients to remotely query state while transparently verifying proofs.
go/oasis-node/cmd/stake: Show pretty-printed account info (#3087)

Change oasis-node stake account info CLI command to output pretty-printed account info instead of raw JSON data.
go/staking/api: Implement PrettyPrinter interface for various types (#3087, #3132)

Implement PrettyPrinter for Transfer, Burn, Escrow, ReclaimEscrow, AmendCommissionSchedule, SharePool, StakeThreshold, StakeAccumulator, GeneralAccount, EscrowAccount, Account, CommissionRateStep, CommissionRateBoundStep and CommissionSchedule types.
go/worker/storage: Added round sync metrics (#3088)
go/registry: Add support for querying suspended runtimes (#3093)

Registry GetRuntimes method now accepts a parameter to enable querying for suspended runtimes. WatchRuntimes will now always include suspended runtimes in the initial response.
go/consensus/tendermint: Support consensus backend modes (#3116)
go/common/crypto/signature: Add a plugin backed signer implementation (#3120)

Bloating the repository with a ton of different HSM (etc) signing backends doesn't make sense, and is a maintenance burden. Use the go-plugin package to allow for externally distributed signer plugins.
go/common/grpc: Client verbose logging and metrics (#3121)

Adds option to enable verbose logging for gRPC client and adds basic gRPC client instrumentation.

Verbose gRPC client logging can be enabled with the existing grpc.log.debug flag.

Metric changes:

Existing gRPC server metrics were renamed:
- oasis_grpc_calls -> oasis_grpc_server_calls
- oasis_grpc_latency -> oasis_grpc_server_latency
- oasis_grpc_stream_writes -> oasis_grpc_server_stream_writes
Added corresponding metrics:
- oasis_grpc_client_calls gRPC client calls metric.
- oasis_grpc_client_latency gRPC client call latencies.
- oasis_grpc_client_stream_writes gRPC client stream writes.
go/common/prettyprint: Add QuantityFrac() (#3129, #3131)
go/oasis-node/cmd/common/consensus: Ask for confirmation before signing (#3134)

If file signer is used, ask for confirmation before signing a transaction, unless --assume_yes flag is set.
go/oasis-node/cmd/common/consensus: Pretty print transaction before signing it (#3134)

Adapt oasis-node stake account gen_* CLI commands to display stake amount in pretty printed transactions in token values.
go/oasis-node/cmd/common/flags: Add --assume_yes flag (#3137)
go/oasis-node/cmd/common: Add GetUserConfirmation() (#3137)
go/oasis-node/cmd/common/consensus: Print helper text about transaction review (#3149)

If one uses the signer plugin and a TTY, print a helper text to notify him that he may need to review the transaction on the device if a hardware-based signer plugin is used.
go/consensus: Add SubmitTxNoWait method (#3152)

The new method allows submitting a transaction without waiting for it to be included in a block.
go/consensus/api/transaction: Add PrettyPrintBody() to Transaction type (#3157)
go/worker/keymanager: Ignore runtimes not in policy document (#3162)
go/consensus: Move SubmitEvidence to LightClientBackend (#3169)

This allows light clients to submit evidence of Byzantine behavior.

Bug Fixes

runtime: Reduce maximum RHP message size to 16 MiB (#2213)
go/worker/storage: Fix storage node policy (#3021)
runtime/dispatcher: Break recv loop on abort request (#3023)
go/runtime/host/sandbox: Fix possible data race (#3024)

The data race existed because the cancel function that is referenced inside a goroutine waiting for initialization to complete was unintentionally overwritten.
go/consensus/tendermint: Properly handle no committed blocks (#3032)
runtime/committee/client: Reduce gRPC max backoff timeout (#3035)

Committee nodes are expected to be available and this timeout is more in line with timeouts used in the clients using these connections.
go/worker/common/p2p: Don't treat context cancellation as permanent (#3075)

Context cancellation errors should not count as permanent for P2P dispatch as the cancelled context may be due to the round advancing in which case dispatch should actually be retried.
go/oasis-test-runner/cmd: Sort scenarios for correct parallel execution (#3104)
oasis-node/cmd/unsafe-reset: Fix globs for runtime files (#3105)
go/oasis-node/cmd/common: ExportEntity should use the entity ctor (#3117)

Instead of using an entity populated with the zero values and a manually filled in public key, use the entity constructor that can fill in sensible values for things like the version.
go/consensus/tendermint/roothash: Ignore non-tracked runtimes when reindexing (#3124)
go/tendermint/roothash: Skip pruned heights when reindexing (#3127)
go/worker/common: Treat stale unauthorized peer error as permanent (#3133)

If the message's group version indicates that the message is stale and an authorization check fails, treat the error as permanent as a stale message will never become valid.
go/worker/compute: Enforce maximum batch sizes from txn scheduler (#3139)
Bump libp2p to 0.10.2 (#3150)
go/storage/mkvs/checkpoint: Remove empty version directories (#3160)

When all root checkpoints are removed for a specific version, the version dir itself should also be removed.
go/registry/metrics: Fix oasis_registry_runtimes metric (#3161)

Metric was counting runtime events, which does not correctly take into account the case where runtime is suspended and resumed. The metric is now computed by querying the registry.

Documentation Improvements

ADR 0000: Architectural Decision Records (#2924)

Introduce architectural decision records (ADRs) for keeping track of architecture decisions in a transparent way.
docs/runtime: Update the Runtime Host Protocol section (#2978)
docs: Add the Accounts section of the Staking docs (#3005)
go/staking/api: Document commission-related public types (#3042)
Add Governance Model (#3130)
ADR 0001: Multiple Roots Under the Tendermint Application Hash (#3136)

Currently the Tendermint ABCI application hash is equal to the consensus state root for a specific height. In order to allow additional uses, like proving to light clients that specific events have been emitted in a block, we should make the application hash be derivable from potentially different kinds of roots.
docs: Add cryptography section (#3147)

Internal Changes

worker/sentry: Replace policy watcher with UpdatePolicies API method (#2820)

Previously, the sentry gRPC worker watched for policy changes, but now all policy changes are pushed to the sentry node via a new sentry API method, UpdatePolicies.
go/oasis-test-runner: Use common cli package for provisioning runtimes (#3021)
go/common/encoding/bech32: Replace to-be-removed dependency (#3030)
ci: Don't automatically retry timed out jobs (#3036)
go/oasis-test-runner: Refactor E2E scenarios (#3043)

Previously there were some scenarios which incorrectly used the e2e/runtime base even though they do not need any runtimes to work. These have now been changed to use the e2e base instead.
go/oasis-test-runner/cmd/common: Give verbose error when scenario is not found (#3044)

List all available scenarios if oasis-test-runner command is invoked with an unknown scenario.
go/worker/compute/executor: Defer fetching the batch from storage (#3049)

There is no need to attempt to fetch the batch immediately, we can defer it to when we actually need to start processing the batch. This makes fetching not block P2P dispatch.
go/worker/compute/merge: Defer finalization attempt (#3049)

There is no need for the finalization attempt to block handling of an incoming commitment as any errors from that are not propagated. This avoids blocking P2P relaying as well.
go/oasis-test-runner: Support regexps for matching scenario names (#3051)
go/common/cbor: Add support for generic versioned blobs (#3052)
go/registry: Add transaction test vector generator (#3059)
go/common/prettyprint: Add PrettyType for use in test vectors (#3059)
go/consensus/tendermint/epochtime_mock: Fix initial notify (#3060)
go/tests/sentry: Add access control sanity checks (#3062)
ci/longtests: Setup prometheus monitoring (#3090)
go/consensus/tendermint: Refactor internal event handling (#3091)

Previously each consensus service client implemented its own event processing loop. All service clients now have a unified event loop implementation (each still runs in its own goroutine) that takes care of query subscriptions and event dispatch.
go: Use gofumpt instead of gofmt to format Go (#3095)
go/oasis-test-runner/cmd: Limit scenario name regex matching (#3103)

Prevent oasis-test-runner to match too many scenarios for a given scenario name regex by ensuring the given scenario name regex matches the whole scenario name.
go/tests/e2e: Add missing Clone() override to late-start (#3106)
go/oasis-test-runner: Refactor code to refer to scenario(s) consistently (#3108)
go/oasis-test-runner/oasis: Purge most of tests/fixture-data (#3110)
go/consensus/tendermint: Make New return consensus.Backend (#3115)
go/consensus/tendermint: Move GenesisProvider to api package (#3115)
go/consensus/tendermint: Move service.TendermintService to api.Backend (#3115)
go: Use cbor.Versioned for descriptor versioning (#3119)
go/e2e/txsource: Periodically restart runtime nodes (#3124)
ci: Extract go artifacts upload code into a script (#3125)
go/tests/e2e/history_reindex: Scenario that triggers roothash reindexing (#3127)
go/badger: bump badger version (#3142)
go/oasis-node: Fix the interactive prompt to be correct (#3143)

20.8 (2020-06-16)

Process

release: Create a stable branch when preparing a new release (#2993)

Currently we create a stable release branch only once we need to backport some changes. This PR changes the release process to create the stable branch when creating a new release. This will ensure CI jobs hooked to stable/ branches, such as building a release tagged CI docker image, will be run for every release.

Removals and Breaking Changes

Change staking account ids/addresses to truncated hash of the public key (#2928)

Previously, staking account identifiers were called ids and were represented by a corresponding entity's public key.

Now, they are called addresses and are represented by a truncated hash of a corresponding entity's public key, prefixed by a 1 byte address version.

Furthermore, the new staking account addresses use the newly added Bech32 encoding for text serialization with oasis as their human readable part (HRP) prefix.
go/common/crypto/signature: Rename NewBlacklistedKey() function (#2940)

Rename it to NewBlacklistedPublicKey() for consistency with the newly added NewPublicKey() function.
go/staking/api: Rename Accounts() and AccountInfo() functions (#2940)

Rename Accounts() function to Addresses() and AccountInfo() function to Account() to avoid ambiguity and better reflect their return value.
go/consensus/api: Rename Caller field in EstimateGasRequest type (#2940)

Rename EstimateGasRequest's Caller field to Signer to better describe the field's value which is the public key of the transaction's signer.
go/consensus/api: Use staking.Address in GetSignerNonceRequest type (#2940)

Replace GetSignerNonceRequest's ID field with AccountAddress field to reflect the recent staking account id/address change.
Disallow v0 entity/node/runtime descriptors (#2992)
registry: Add runtime-specific staking thresholds (#2995)
go/staking: Rename fields in Event structure (#3013)

Fields in the Event structure are renamed to drop the Event suffix, breaking Go API compatibility. This has the following effect:
- TransferEvent field is renamed to Transfer.
- BurnEvent field is renamed to Burn.
- EscrowEvent field is renamed to Escrow.
The wire format of the event structure is unchanged.
go/oasis-node/cmd/stake: Make list command's verbose output consistent (#3016)

Change oasis-node stake list --verbose CLI command's output to not list all accounts' information as a single-line JSON string but rather output each account's JSON string on a separate line.

Configuration Changes

Change staking account ids/addresses to truncated hash of the public key (#2928)

Due to this breaking change described above, the following configuration changes are needed:
- In oasis-node staking account info CLI, the --stake.account.id option has been renamed to --stake.account.address and now accepts a Bech32 encoded account address.
- In oasis-node staking account gen_transfer CLI, the --stake.transfer.destination option now accepts a Bech32 encoded account address.
- In oasis-node staking account gen_escrow and oasis-node staking account gen_reclaim_escrow CLI, the --stake.escrow.account option now accepts a Bech32 encoded account address.
go/worker/sentry: Do client authentication on sentry control grpc (#3018)

The following configuration changes are needed due to this change:
- Added: worker.sentry.control.authorized_pubkey option to configure allowed upstream nodes. This should be set to sentry client TLS keys of upstream nodes.
- Renamed: worker.sentry.control_port option to worker.sentry.control.port.

Features

go/worker/common: Allow specifying the path to the bwrap binary (#1599)

This adds a new config option --worker.runtime.sandbox_binary that allows overriding the path to the sandbox support binary (ie: bwrap).
worker/compute/executor: Independently submit commit in case of faults (#1807)
go/common/encoding: Add bech32 package implementing Bech32 encoding (#2940)
go/consensus/tendermint/apps/staking: Forbid txs for reserved addresses (#2940)

Prevent reserved staking addresses (e.g. the common pool address) from being used as the from address in staking transactions.
go/common/crypto: Add address package implementing a generic crypto address (#2940)

It supports versioning and context separation and can be used to implement specific addresses, e.g. the staking account address.
go/staking: Forbid reserved addresses in the Genesis (#2940)

Prevent reserved staking addresses (e.g. the common pool address) from being used as an account in the Genesis' staking ledger or as an address in the Genesis' (debonding) delegations.
go/common/crypto/signature: Add NewPublicKey() function (#2940)

It creates a new public key from the given hex representation or panics.
go/consensus/tendermint/api: Add OwnTxSignerAddress() to ApplicationState (#2940)

It returns the transaction signer's staking address of the local node.
go/staking/api: Add Address type for representing staking account addresses (#2940)
go/common/crypto/hash: Add Truncate() method to Hash (#2940)

It returns the first n bytes of a hash.
runtime: Handle runtime interruptions gracefully (#2991)
go/oasis-node/cmd/stake: Add pubkey2address commmand (#3003)

Add oasis-node stake pubkey2address CLI command for converting a public key (e.g. an entity's ID) to a staking account address.
go/roothash: Add commitment events (#3013)

The following two kinds of events are added to the roothash service:
- ExecutorCommittedEvent emitted when the roothash service processes a commitment from an executor node.
- MergeCommittedEvent emitted when the roothash service processes a commitment from a merge node.

Bug Fixes

staking: Don't emit zero-amount staking events (#2983)

If fees were set to 0, staking events related to the fee accumulator would still get emitted with zero amounts, which is pointless.

This fix affects only events related to the internal fee accumulator and common pool accounts, manual transfers with 0 as the amount will still get emitted.
Bump Rust toolchain to 2020-06-09 for LVI mitigations (#2987)
worker/compute: Retry local dispatch between workers (#3000)

If a node has multiple roles simultaneously, the local submissions don't go through P2P, but via a direct function call. This function call is now retried in case of errors.
go/txsource: fix queries workload earliest committee epoch (#3001)
go/oasis-node: Support non-file signers in registry node init (#3011)

Documentation Improvements

docs: Refactor setup instructions, add single validator node setup (#3006)

Internal Changes

client: Rename the rpc crate to enclave_rpc (#2469)
runtime: Rename the rpc crate to enclave_rpc (#2469)
rust: Remove the use of the failure crate (#2755)

Using this crate is no longer considered best practice, and bugs in it have broken the build before.

Note: Unfortunately the runtime-loader crate still uses this failure nonsense since the external Intel AESM client crate uses it for error handling. Since it is a stand-alone binary it has been left as is.
Makefile: update docker-shell target (#2985)
test/e2e/sentry: don't set seed node for nodes running with sentries (#2989)
ci: Remove push path filters for dockerfile build action (#2997)
go/consensus/tendermint: Refactor roothash event handling (#3013)

This makes roothash event handling similar to staking event handling, with common code paths for pubsub and polling-based calls.

It also adds Height and TxHash to roothash events.
ci: Make Buildkite fail if desired tag of Docker CI image doesn't exist (#3016)

20.7 (2020-06-08)

Removals and Breaking Changes

go/registry: Avoid storing full TLS certificates (#2556)

Previously the node registry descriptor contained full TLS certificates for talking with nodes via gRPC. This changes it so that only TLS public keys are used when verifying peer certificates for TLS authentication.

This makes the registry descriptors smaller and also makes it easier to pass around TLS identities (as public keys are much shorter).

Obviously, this change BREAKS the consensus protocol and all previously signed node descriptors.

The following configuration changes are needed due to this change:
- In oasis-node registry node CLI, the --node.committee_address option has been renamed to --node.tls_address and the format has changed from <certificate>@ip:port to <pubkey>@ip:port.
- For configuring sentry nodes on the workers, the --worker.sentry.cert_file has been removed. Instead, the --worker.sentry.address now takes the same address format as specified above (<pubkey>@ip:port).
Previously signed node descriptors (v0) are considered valid at genesis time iff the node is exclusively a validator node as indicated by the role bits. Other nodes will need to be removed from genesis.
Make oasis_ Prometheus metrics help consistent (#2602)

Help messages for metrics starting with oasis_ were revisited and made consistent. If you are using Prometheus and/or Push Gateway, you will need to clear Prometheus data/ directory and restart the services to avoid inconsistency warnings.
Refactor the runtime host APIs (#2801)

Several changes were made to the runtime host APIs used to provision and communicate with runtimes:
- The runtime host implementation has been moved to go/runtime/host.
- Some of the runtime host protocol types have been changed, all references to Worker in messages were renamed to Runtime to make it more clear what they refer to. Additionally, the Error message type has been changed to include additional fields (module and code) to make it easier to remap errors automatically.
  
  This makes it a BREAKING change for any existing runtimes.
- Provisioning of a runtime is now performed by a Provisioner which is an interface. Implementations exist for (sandboxed) ELF binaries and Intel SGX enclaves. The implementations have been refactored to be more composable, so for example the SGX implementation only implements the SGX-related bits but uses the regular provisioner otherwise.
- Configuration options for hosted runtimes have changed so existing configs will need to be updated as follows:
  - The --worker.runtime.backend option has been renamed to --worker.runtime.provisioner.
  - The --worker.runtime.loader option has been renamed to --worker.runtime.sgx.loader and is now only required for supporting SGX runtimes. Non-SGX runtimes no longer need a loader.
  - The --worker.runtime.binary option has been renamed to --worker.runtime.paths and the value format has changed to be either a YAML map or a set of comma-separated key-value pairs separated by = (e.g., <runtime-ID>=/path/to/binary).
- The key manager worker has been slightly changed to use the common runtime provisioning code. The following configuration options have been removed:
  - --worker.keymanager.tee_hardware as the TEE hardware is inferred from the runtime descriptor.
  - --worker.keymanager.runtime.loader and --worker.keymanager.runtime.binary as the common options mentioned above should be used instead.
go/consensus/tendermint: Clean up indices in SetNode (#2910)
go/registry: Drop support for v0 node descriptor (#2918)

Configuration Changes

go/oasis-node: Fix signer configuration via YAML (#2949)

The --signer argument has been renamed to --signer.backend to allow signer configuration to be passed via a YAML config. Previously, this would be impossible as signer would need to be both a string and a map at the same time when set via a YAML config.

Features

go/runtime/host/sgx: Add support for SIGSTRUCTs (#1707)

For now this will just generate one, signed with the same key that runtime-loader used to use (the Fortanix dummy key), but this will also support using file backed signatures, once we have an idea on how we are going to handle the process for such things.
Add metrics for inter-node communcation (#1771)
- New module label was added to oasis_codec_size metric which contains information of the caller. Currently p2p value denotes a peer-to-peer message among Oasis nodes and runtime-host a message from/to enclave.
- New oasis_rhp_latency summary metric for measuring Runtime Host communication latency was added.
- New oasis_rhp_successes and oasis_rhp_failures counter metrics for counting number of successful and failed Runtime Host calls respectively were added.
go/oasis-node/cmd/debug: Add the dumpdb command (#2359)

This command will attempt to extract the ABCI state from a combination of a shutdown node's on-disk database and the genesis document currently being used by the network, and will write the output as a JSON formatted genesis document.

Some caveats:
- It is not guaranteed that the dumped output will be usable as an actual genesis document without manual intervention.
- Only the state that would be exported via a normal dump from a running node will be present in the dump.
- The epochtime base will be that of the original genesis document, and not the most recent epoch (different from a genesis dump).
e2e/tests: added keymanager runtime upgrade test (#2517)
Version signed entity, node and runtime descriptors (#2581)

This introduces a DescriptorVersion field to all entity, node and runtime descriptors to support future updates and handling of legacy descriptors at genesis.

All new registrations only accept the latest version while initializing from genesis is also allowed with an older version to support a dump/restore upgrade.
Add new consensus-related Prometheus metrics (#2842)

Four new metrics have been added:
- oasis_worker_epoch_number is the current epoch number as seen by the worker.
- oasis_worker_node_registered is a binary metric which denotes, if the node is registered.
- oasis_consensus_proposed_blocks is the number of proposed Tendermint blocks by the node.
- oasis_consensus_signed_blocks is the number of Tendermint blocks the node voted for.
keymanager: Rename APIs referencing "contracts" (#2844)

Each runtime does not neccecarily have a notion for contracts, so the key manager now operates in terms of KeyPairIds that identify a given KeyPair.
oasis-net-runner: add support for running networks with non-mock IAS (#2883)
go/stats: New availability v3 formula (#2896)

Here's the implementation of the new availability score v3 for May.
consensus: Add GetStatus method to API (#2902)

A new GetStatus method has been added to the consensus API. It returns useful information about the latest block, the genesis block, and the node itself.
control: Add GetStatus method to API (#2902)

A new GetStatus method has been added to the control API. It returns the software version and the status of the consensus layer.
go/extra/stats: Give ties the same rank (#2908)

Previously records tied by availability score would get different ranks, which was wrong.
go/oasis-node/cmd/common/metrics: Deprecate formal pushgateway support (#2936)

The prometheus authors do not recommend using it for most situations, it appears to be somewhat fragile, and we shouldn't be using it internally, so the functionality is now only usable if the correct debug-only flags are set.
runtime/storage/mkvs: Add method for checking local key existence (#2938)

Adds a method to probe the local cache for key existence, guaranteeing that no remote syncing will be done.
staking: Add WatchEvents (#2944)

A new method was added to the staking API, WatchEvents. It returns a channel that produces a stream of staking Events. The Event structure was also extended to include the block height.
go/consensus/genesis: Add a public key blacklist (#2948)

This change adds a public key blacklist to the consensus parameters. All signatures made by public keys in the blacklist will be rejected.

WARNING: For now the node will panic on startup if the genesis staking ledger has entries for blacklisted public keys. By the time this feature is actually put to use (hopefully never), the staking ledger address format will be changed, resolving this caveat.

Bug Fixes

go/worker/keymanager: retry initialization in case of failure (#2517)

The keymanager worker registers only after the initialization either fails or succeeds. In case the worker needs to replicate the first initialization will always fail, since other nodes' access control prevents it from replicating. In that case the initialization should be retried.
ias: support for IAS API v4 (#2883)

IAS-proxy now proxies to IAS v4 endpoint. Attestation code now works with v4 IAS API spec.
common/flags: Fix parsing of metrics.labels, if provided in config .yml (#2905)

Bug in viper library was fixed upstream and drop-in replacement for GetStringMapString was removed.
staking: Emit events when disbursing fees and rewards (#2909)

Staking events are now generated when disbursing fees and rewards. There are two new special account IDs -- CommonPoolAccountID and FeeAccumulatorAccountID (both defined in go/staking/api/api.go), which are used only in events to signify the common pool and the fee accumulator respectively. These account IDs are invalid by design to prevent misusing them anywhere else.
runtime: Notify runtimes of its key manager policy updates (#2919)

Before runtimes were unaware of any key-manager policy updates. The runtime only queried for the active key-manager policy at startup. This is now changed so that the host notifies runtimes of any key-manager policy changes and runtime updates the policies.
go/consensus: Hash user-controlled storage key elements (#2929)
go/oasis-node/cmd/common/metrics: Re-create the pusher on failure (#2936)

When using prometheus' push client, any single failure causes the client to be unusable for future requests. Re-create the client on failure, so that metrics might start working again.
go/oasis-node: Fix parsing of signer configuration (#2950)
go/oasis-node: Make sure the node only tries to stop once (#2964)

This could previously result in a panic during shutdown.
go/oasis-node: Don't bypass Stop in newNode failure handler (#2966)
go/runtime/host/sandbox: Retry Call in case the runtime is not yet ready (#2967)
go/oasis-node: Properly handle shutdown during startup (#2975)
go/consensus/tendermint: Don't panic on context cancellation (#2982)

Documentation Improvements

docs: Add documentation around our gRPC over TLS flavor (#2556)
Document all Prometheus metrics produced by oasis-node (#2602)

List of metrics including the description, metric type, metric-specific labels, and location in the source is now available in docs/oasis-node/metrics.md Markdown file. To automate generation of this list, a new go/extra/extract-metric tool was introduced. To update the list of metrics, execute make update-docs in the project root. Documentation needs to be up to date for lint rule to succeed.

Internal Changes

ci: Automatically rebuild development/CI Docker images (#295)
go: Replace the use of pkg/error with fmt (#2057)

I also tried to make some of the error messages more consistent but gave up when I got to the staking/byzantine node related ones because none of them are following our convention.
go/control: Add IsReady() and WaitReady() RPC methods (#2130)

Beside IsSynced() and WaitSynced() which are triggered when the consensus backend is synced, new IsReady() and WaitReady() methods have been added to the client protocol. These are triggered when all node workers have been initialized (including the runtimes) and the hosted processes are ready to process requests.

In addition new oasis-node debug control wait-ready command was added which blocks the client until the node is ready.
ci: Add CI test for running E2E tests with IAS development API (#2883)
oasis-test-runner: Refactor initialization of scenario flags (#2897)

Implementations of Parameters() function defined in test-runner's scenario interface have been revised. All scenario-settable flags are now explicitly initialized and scenarios call standard FlagSet accessors to fetch scenario-specific parameters.
ci: automatically rety exit status 125 failures (#2900)
go/ias: update test vector to a v4 report (#2901)
go: Bump Go to 1.14.3 (#2913)
go: update dependencies depending on websocket@v1.4.0 (#2927)

Due to a vulnerability in websocket@1.4.0: CWE-190.

Updated libraries:
- github.com/libp2p/go-libp2p@v0.1.1 to github.com/libp2p/go-libp2p@v0.9.1
- github.com/spf13/viper@v1.6.3 to github.com/spf13/viper@v1.7.0
- replace github.com/gorilla/websocket with github.com/gorilla/websocket v1.4.2
changelog: Add a changelog fragment type for configuration changes (#2952)
runtime-loader: Bump Fortanix EDP crate versions (#2955)

Also bumps Rust nightly to 2020-05-15 as that is required to build the new versions.
go/worker/registration: Add SetAvailableWithCallback to RoleProvider (#2957)

The new method allows the caller to register a callback that will be invoked on a successful registration that includes the node descriptor updated by the passed hook.
Transfer oasis-core repository to oasisprotocol (#2968)
ci: don't retry jobs that timeout (#2969)
go: update dynlib dep (#2974)

20.6 (2020-05-07)

Removals and Breaking changes

go/consensus/tendermint: Use MKVS for storing application state (#1898)
oasis-node: Refactor metrics parameters (#2687)
- --metrics.push.job_name renamed to --metrics.job_name.
- --metrics.push.interval renamed to --metrics.interval.
- --metrics.push.instance_label replaced with more general --metrics.labels map parameter where instance is a required key, if metrics are enabled. For example --metrics.push.instance_label abc now becomes --metrics.labels instance=abc. User can also set other arbitrary Prometheus labels, for example --metrics.labels instance=abc,cpu=intel_i7-8750.
go/consensus/tendermint: Store consensus parameters in ABCI state (#2710)
go: Bump tendermint to v0.33.3-oasis1 (#2834)

This is breaking as the tendermint block format has changed.
go/consensus/genesis: Make max evidence age block and time based (#2834)
- Rename max_evidence_age -> max_evidence_age_blocks
- Add max_evidence_age_time (default 48h)
This is obviously breaking.
keymanager-lib: Bind persisted state to the runtime ID (#2843)

It is likely prudent to bind the persisted master secret to the runtime ID. This change does so by including the key manager runtime ID as the AAD when sealing the master secret.

This is backward incompatible with all current key manager instances as the existing persisted master secret will not decrypt.
go/runtime/enclaverpc: Refactor gRPC endpoint routing (#2844)

Previously each endpoint required its own gRPC service. But since all EnclaveRPC requests already include an "endpoint" field, it is better to use that for routing requests.

This commit adds a new enclaverpc.Endpoint interface that is used as an endpoint descriptor. All endpoints must be registered in advance (e.g., during init). It also changes the key manager EnclaveRPC support to use the new API.
oasis-net-runner: --net.* flags renamed to --fixture.default.* (#2848)

For example --net.node.binary mynode/oasis-node becomes --fixture.default.node.binary mynode/oasis-node.
go/consensus: Stake weighted voting (#2868)

That is, validator voting power proportional to entity stake (previously: "flat" all-validators-equal voting power). Radical!
go/common/node: Add RoleConsensusRPC role bit (#2881)

Features

go/worker/consensusrpc: Add public consensus RPC services worker (#2440)

A public consensus services worker enables any full consensus node to expose light client services to other nodes that may need them (e.g., they are needed to support light clients).

The worker can be enabled using --worker.consensusrpc.enabled and is disabled by default. Enabling the public consensus services worker exposes the light consensus client interface over publicly accessible gRPC.
go/consensus: Add basic API for supporting light consensus clients (#2440)
oasis-node: Add benchmarking utilities (#2687)
- New Prometheus metrics for:
  - datadir space usage,
  - I/O (read/written bytes),
  - memory usage (VMSize, RssAnon, RssFile, RssShmem),
  - CPU (utime and stime),
  - network interfaces (rx/tx bytes/packets),
- Bumps prometheus/go_client to latest version which fixes sending label values containing non-url characters.
- Bumps spf13/viper which fixes IsSet() behavior.
Add GetEvents to backends (#2778)

The new GetEvents call returns all events at a specific height, without having to watch for them using the Watch* methods. It is currently implemented for the registry, roothash, and staking backends.
go/keymanager/api: Add a gRPC endpoint for status queries (#2843)

Mostly so that the test cases can query statuses.
go/oasis-test-runner/oasis: Add a keymanager replication test (#2843)
oasis-net-runner: Add support for fixtures in JSON file (#2848)

New flag --fixture.file allows user to load default fixture from JSON file. In addition dump-fixture command dumps configured JSON-encoded fixture to standard output which can serve as a template.
go/consensus/tendermint: Expose new config options added in Tendermint 0.33 (#2855)

Tendermint 0.33 added the concept of unconditional P2P peers. Support for setting the unconditional peers via tendermint.p2p.unconditional_peer_ids configuration flag is added. On sentry node, upstream nodes will automatically be set as unconditional peers.

Tendermint 0.33 added support for setting maximum re-dial period when dialing persistent peers. This adds support for setting the period via tendermint.p2p.persistent_peers_max_dial_period flag.
go/consensus/tendermint: Signal RetainHeight on Commit (#2863)

This allows Tendermint Core to discard data for any heights that were pruned from application state.
go/consensus/tendermint: Bump Tendermint Core to 0.33.4 (#2863)
go/consensus/tendermint: sync-worker additionally check block timestamps (#2873)

Sync-worker relied on Tendermint fast-sync to determine if the node is still catching up. This PR adds aditional condition that the latest block is not older than 1 minute. This prevents cases where node would report as caught up after stopping fast-sync, but before it has actually caught up.
go/consensus: Add GetGenesisDocument (#2889)

The consensus client now has a new method to return the original genesis document.
go/staking: Add event hashes (#2889)

Staking events now have a new TxHash field, which contains the hash of the transaction that caused the event (or the empty hash in case of block events).

Bug Fixes

go: Extract and generalize registry's staking sanity checks (#2748)

Augment the checks to check if an entity has enough stake for all stake claims in the Genesis document to prevent panics at oasis-node start-up due to entities not having enough stake in the escrow to satisfy all their stake claims.
go/oasis-node/cmd/ias: Fix WatchRuntimes retry (#2832)

Previously the IAS proxy could incorrectly panic during shutdown when the context was cancelled.
go/worker/keymanager: Add an enclave rpc handler (#2843)
go/worker/keymanager: Actually allow replication to maybe work (#2843)

Access control forbidding replication may be more secure, but is not all that useful.
go/keymanager/client: Support km->km connections (#2843)
go/common/crypto/mrae/deoxysii: Use SHA512/256 for the KDF (#2853)

Following 73aacaa73d7116a6be0443e70f2d10d0c7a4b76e, this should also use the correct hash algorithm for the KDF.
go/extra/stats: fix & simplify node-entity mapping (#2856)

Instead of separately querying for entities and nodes, we can get Entity IDs from nodes directly.

This change also fixes a case that previous variant missed: node that was removed from entity list of nodes, but has not yet expired.
go/extra/stats: fix heights at which missing nodes should be queried (#2858)

If a missing signature is encountered, the registry should be queried at previous height, since that is the height at which the vote was made.
client/rpc: Change session identifier on reset (#2872)

Previously the EnclaveRPC client did not change the session identifier on reset, resulting in unnecessary round-trips during a transport error. The EnclaveRPC client now changes the session identifier whenever resetting the session.
go/worker/storage: Correctly apply genesis storage state (#2874)

Previously genesis storage state was only applied at consensus genesis which did not support dynamically registered runtimes. Now genesis state is correctly applied when the storage node initializes for the first time (e.g., when it sees the registered runtime).

This also removes the now unused RegisterGenesisHook method from the consensus backend API.
worker/registration: use WatchLatestEpoch when watching for registrations (#2876)

By using WatchLatestEpoch the worker will always try to register for latest known epoch, which should prevent cases where registration worker fell behind and was trying to register for past epochs.
go/runtime/client: Actually store the created key manager client (#2885)
go/runtime/committee: Restore previously picked node in RR selection (#2885)

Previously the round-robin node selection policy would randomize the order on every update ignoring the currently picked node. This would cause the current node to flip on each update causing problems with EnclaveRPC which is stateful.

The fix makes the round-robin node selection policy attempt to restore the currently picked node on each update. This means that in case the node is still in the node list, it will not change.
go/scheduler: Increase tokens per voting power (#2892)

We'll need this to fit under tendermint's maximum total voting power limit.

Documentation improvements

Refactor documentation, add architecture overview (#2791)

Internal changes

oasis-test-runner: Add benchmarking utilities (#2687)
- oasis-test-runner now accepts --metrics.address and --metrics.interval parameters which are forwarded to oasis-node workers.
- oasis-test-runner now signals oasis_up metric to Prometheus when a test starts and when it finishes.
- --num_runs parameter added which specifies how many times each test should be run.
- basic E2E test was renamed to runtime.
- Scenario names now use corresponding namespace. e.g. halt-restore is now e2e/runtime/halt-restore.
- Scenario parameters are now exposed and settable via CLI by reimplementing scenario.Parameters() and setting it with --<test_name>.<param>=<val>.
- Scenario parameters can also be generally set, for example --e2e.node.binary will set node.binary parameter for all E2E tests and --e2e/runtime.node.binary will set it for tests which inherit runtime.
- Multiple parameter values can be provided in form --<test_name>.<param>=<val1>,<val2>,.... In this case, oasis-test-runner combines them with other parameters and generates unique parameter sets for each test.
- Each scenario is run in a unique datadir per parameter set of form oasis-test-runnerXXXXXX/<test_name>/<run_id>.
- Due to very long datadir for some e2e tests, custom internal gRPC socket names are provided to oasis-node.
- If metrics are enabled, new labels are passed to oasis-nodes and pushed to Prometheus for each test:
  - instance,
  - run,
  - test,
  - software_version,
  - git_branch,
  - whole test-specific parameter set.
- New version.GitBranch variable determined and set during compilation.
- Current parameter set, run number, and test name dumped to test_info.json in corresponding datadir. This is useful when packing whole datadir for external debugging.
- New cmp command for analyzing benchmark results has been added which fetches the last two batches of benchmark results from Prometheus and compares them. For more information, see README.md in go/oasis-test-runner folder.
ci: New benchmarks pipeline has been added (#2687)

benchmarks.pipeline.yml runs all E2E tests and compares the benchmark results from the previous batch using the new oasis-test-runner cmp command.
oasis-node: Add custom internal socket path flag (for E2E tests only!) (#2687)

--debug.grpc.internal.socket_name flag was added which forces oasis-node to use the given path for the internal gRPC socket. This was necessary, because some E2E test names became very lengthy and original datadir exceeded the maximum unix socket path length. oasis-test-runner now generates shorter socket names in /tmp/oasis-test-runnerXXXXXX directory and provides them to oasis-node. Due to security risks never ever use this flag in production-like environments. Internal gRPC sockets should always reside in node datadir!
go/registry/api: Extend NodeLookup and RuntimeLookup interfaces (#2748)

Define Nodes() and AllRuntimes() methods.
go/staking/tests: Add escrow and delegations to debug genesis state (#2767)

Introduce stakingTestsState that holds the current state of staking tests and enable the staking implementation tests (StakingImplementationTest, StakingClientImplementationTests) to always use this up-to-date state.
go/runtime/committee: Don't close gRPC connections on connection refresh (#2826)
go: Refactor E2E coverage integration test wrapper (#2832)

This makes it possible to easily have E2E coverage instrumented binaries for things other than oasis-node.
go/oasis-node: Move storage benchmark subcommand under debug (#2832)
keymanager-runtime: replace with test/simple-keymanager (#2837)

Common keymanager initalization code is extracted into the keymanager-lib crate. This enables for the actual key manager implementation to only provide a set of key manager policy signers. Aditionally the keymanager-runtime crate is removed and replaced with a test simple-keymanager runtime that is used in E2E tests.
docker: remove docker image build pipelines and cleanup testing image (#2838)
go/oasis-test-runner: Generate a new random seed on each run (#2849)
go/storage/mkvs/checkpoint: Refactor restorer interface (#2860)
go/storage/mkvs/checkpoint: Add common checkpointer implementation (#2860)

Previously there was a checkpointer implemented in the storage worker but since this may be useful in multiple places, the checkpointer implementation is generalized and moved to the checkpoint package.
go/oasis-test-runner: Configure consensus state pruning (#2866)
go: Start using new protobuf module location (#2867)

The previous location has been deprecated.
go/common/pubsub: support subscriptions based on bounded ring channels (#2876)
go/epochtime: add WatchLatestEpoch method (#2876)

The method is similar to the existing WatchEpochs method, with the change that unread epochs get overridden with latest epoch.
go/common/crypto/hash: Add NewFrom and NewFromBytes functions (#2890)
ci: automatically retry jobs due to host agent failures (#2894)

20.5 (2020-04-10)

Process

Include oasis-core-runtime-loader and oasis-net-runner in releases (#2780)

Removals and Breaking changes

storage: Rename "round" to "version" (#2734)

Previously the MKVS used the term "round" to mean a monotonically increasing version number. This choice was due to the fact that it was initially used to only store runtime state which has a concept of rounds.

As we expand its use it makes more sense to generalize this and call it version.
go staking: Remove locked accounts (#2753)

We expect not to need this feature.
go/consensus: Introduce gas cost based on tx size (#2761)
storage/mkvs: Only nil value should mean deletion (#2775)

Previously an empty value in the write log signalled that the given entry is a delete operation instead of an insert one. This was incorrect as inserting an empty byte string is allowed. The value is now wrapped in an Option, with None (nil in Go) meaning delete and Some(vec![]) ([]byte{} in Go) meaning insert empty value.

This change is BREAKING as it changes write log semantics and thus it breaks the runtime worker-host protocol.
go/staking: include LastBlockFees in genesis (#2777)

Previosuly last block fees in the block of the genesis dump were lost. In the case when these fees were non-zero this also caused a missmatch in total token supply. Last block fees are now exported and during initialization of the new chain moved to the common pool.
go/staking: Three-way fee split (#2794)

We should give more to the previous block proposer, which is the block that first ran the transactions that paid the fees in the LastBlockFees. Currently they only get paid as a voter.

See oasis-core#2794 for a description of the new fee split.

Instructions for genesis document maintainers:
1. Rename fee_split_vote to fee_split_weight_vote and fee_split_propose to fee_split_weight_next_propose and add fee_split_weight_propose in .staking.params.

Features

node: Add automatic TLS certificate rotation support (#2098)

It is now possible to automatically rotate the node's TLS certificates every N epochs by passing the command-line flag worker.registration.rotate_certs. Do not use this option on sentry nodes or IAS proxies.
go/storage/mkvs: Use Badger to manage versions (#2674)

By restricting how Prune behaves (it can now only remove the earliest round) we can leverage Badger's managed mode to have it manage versions for us. This avoids the need to track node lifetimes separately.
go/common/crypto/signature/signer/composite: Initial import (#2684)

This adds a composite signer factory that can aggregate multiple signer factories. This could be used (for example), to use multiple signer backends simultaneously, depending on the key role.

Eg: The P2P link signer could use a local file, while the consensus signer can be backed by a remote HSM.
e2e tests: Test debonding entries from genesis (#2747)

Here's an e2e test scenario that exercises debonding delegation records from the genesis document.
Add support for custom runtime dispatchers (#2749)

This reorganizes the dispatching code to work with a trait rather than a concrete dispatcher object, enabling runtimes to have their own dispatchers.
txsource: delegation workload (#2752)
txsource: add a runtime workload (#2759)

The added runtime workload submits simiple-keyvalue runtime requests.
go/txsource: add a commission schedule amendments workload (#2766)

The added workload generated commission schedule amendment requests.
go/staking: add LastBlockFees query method (#2769)

LastBlockFees returns the collected fees for previous block.
txsource/queries: workload doing historical queries (#2769)

Queries workload continuously performs various historic queries using the exposed APIs and makes sure the responses make sense.
go/txsource/transfer: inlcude burn transactions in transfer workload (#2773)
go/oasis-node/cmd/debug/consim: Initial import (#2784)

Add the ability to exercise some backends without tendermint, while attempting to preserve some of the semantics.
go/runtime/client: expose GetGenesisBlock method in runtime client (#2796)

Bug Fixes

staking/state: fix DelegationsFor queries (#2756)

DelegationFor and DebondingDelegationsFor would stop traversing the state to soon in some cases.
staking/api/commission: fix possible panic in validation check (#2763)

The validation check would panic whenever the number of bound steps was greater than rate_steps + 2.
go/storage/mkvs: Don't forget to include siblings in SyncGet proof (#2775)
storage/mkvs: Don't try to sync dirty keys (#2775)
go/storage/client: Refresh connections when retrying (#2783)

Previously the storage client did not refresh connections on each retry, so in case a committee change happened while an operation was in progress, all operations continued to use the old connection (which was closed) and thus failed. We now refresh connections on each retry.
go/storage/client: Don't treat "no nodes" as a permanent error (#2785)
go/consensus/tendermint: Use our notion of latest height (#2786)

Do not let Tendermint determine the latest height as that completely ignores ABCI processing so it can return a block for which local state does not yet exist.
go/runtime/client: use history for GetBlock(latest) (#2795)

Using history for all GetBlock requests prevents the case where the latest block would already be available but not yet in history, leading to inconsistent results compared to querying by specific block number.
go/storage/mkvs: Use cbor.UnmarshalTrusted for internal metadata (#2800)
go/consensus: Shorten gas import (#2802)

Switch to more concise FromUint64.
go/consensus/tendermint/apps/staking: Propagate error in initTotalSupply (#2809)
go/staking: Use uint16 for limits in CommissionScheduleRules (#2810)
go/txsource/queries: Wait for indexed blocks before GetBlockByHash (#2814)
go/worker/registration: Fix crash when failing to query sentry addresses (#2825)
go/consensus/tendermint: Bump Tendermint Core to 0.32.10 (#2833)

Documentation improvements

README: Update Rust-related installation instructions (#2745)

Internal changes

Move storage/mkvs/urkel to just storage/mkvs (#2657)

The MKVS implementation has been changed from the initial "Urkel tree" structure and it is no longer an actual "Urkel tree" so having "urkel" in its name is just confusing.
runtime: Add hongfuzz fuzzing targets (#2705)
go/staking/api: Add more details to sanity check errors (#2760)
Make: Allow running Go unit tests and node tests independently (#2776)

They can be run using the new test-unit and test-node targets.
txsource/workload/queries: Add num last kept versions argument (#2783)
go/runtime/client: Use prefetch (GetTransactionMultiple) in QueryTxs (#2783)

Previously if multiple transactions in the same block were returned as QueryTxs results, each transaction was queried independently, resulting in multiple round trips. Now we use GetTransactionMultiple to prefetch multiple transactions in a single round trip which should improve latency.
txsource/queries: increase the odds of querying the latest height (#2787)
go/txsource/queries: obtain earliest runtime round from runtime genesis (#2796)
go/common/cbor: Add UnmarshalTrusted for trusted inputs (#2800)

The new method relaxes some decoding restrictions for cases where the inputs are trusted (e.g., because they are known to be generated by the local node itself).
oasis-test-runner/txsource: increase number of validators (#2815)

Increase the number of validators used in txsource tests so that consensus can keep making progress when one of the nodes is restarted.
go/consensus/tendermint: support DebugUnsafeReplayRecoverCorruptedWAL (#2815)

Adds support for setting tendermint DebugUnsafeReplayRecoverCorruptedWAL and enables it in daily txsource test runs.
oasis-test-runner/txsource: disable LogAssertNoTimeouts if restarts (#2817)
go/extra/stats: Update availability score formula (#2819)

As per </operators/the-quest-rules.html#types-of-challenges>, the availability score formula has changed from "Blocks Signed + 50 x Blocks Proposed" to "Blocks Signed + 50 x Blocks Proposed in Round 0".
oasis-test-runner/txsource: disable Merge Discrepancy checker (#2821)

Timeout due to validator restarting also causes a merge discrepancy. Since timeouts can happen, also disable the Merge discrepancy checker.
go/runtime/committee: Introduce close delay when rotating connections (#2822)

Previously a connection was immediately closed, interrupting any in-flight requests. This introduces a configurable (via WithCloseDelay option) close delay so rotated connections are only closed after some time.
go/common/grpc: Remove manual resolver hack (#2822)

Since gRPC supports the WithResolvers option to specify local resolvers there is no need to use the global resolver registry hack.
go/runtime/committee: Don't reset committees when they don't change (#2822)

Previously each committee election triggered a reset of all connections for that committee. This changes the logic to just bump the committee version in case all the committee members are the same.
tests/fixture/txsources: add initial balance for validator-3 (#2830)

20.4 (2020-03-04)

Removals and Breaking changes

go/registry: Enable non-genesis runtime registrations by default (#2406)
Optionally require a deposit for registering a runtime (#2638)
go/staking: Add stateful stake accumulator (#2642)

Previously there was no central place to account for all the parts that need to claim some part of an entity's stake which required approximations all around.

This is now changed and a stateful stake accumulator is added to each escrow account. The stake accumulator is used to track stake claims from different apps (currently only the registry).

This change also means that node registration will now always require the correct amount of stake.
go/registry: Add explicit EntityID field to Runtime descriptors (#2642)
go/staking: More reward for more signatures (#2647)

We're adjusting fee distribution and the block proposing staking reward to incentivize proposers to create blocks with more signatures.
Send and check expected epoch number during transaction execution (#2650)

Stress tests revealed some race conditions during transaction execution when there is an epoch transition. Runtime client now sends expectedEpochNumber parameter in SubmitTx call. The transaction scheduler checks whether the expected epoch matches its local one. Additionally, if state transition occurs during transaction execution, Executor and Merge committee correctly abort the transaction.
go/staking: Add per-account lockup (#2672)

With this, we'll be able to set up special accounts in the genesis document where they're not permitted to transfer staking tokens until a the specified epoch time. They can still delegate during that time.
Use --stake.shares flag when specifying shares to reclaim from an escrow (#2690)

Previously, the oasis-node stake account gen_reclaim_escrow subcommand erroneously used the --stake.amount flag for specifying the amount of shares to reclaim from an escrow.

Features

Implement node upgrade mechanism (#2607)

The node now accepts upgrade descriptors which describe the upgrade to carry out.

The node can shut down at the appropriate epoch and then execute any required migration handlers on the node itself and on the consensus layer.

Once a descriptor is submitted, the old node can be normally restarted and used until the upgrade epoch is reached; the new binary can not be used at all until the old binary has had a chance to reach the upgrade epoch. Once that is reached, the old binary will refuse to start.
go/common/crypto/signature/signers/remote: Add experimental remote signer (#2686)

This adds an experimental remote signer, reference remote signer implementation, and theoretically allows the node to be ran with a non-file based signer backend.
go/extra/stats: Figure out how many blocks entities propose (#2693)

Cross reference which node proposed each block and report these per-entity as well.
go/extra/stats: Availability ranking for next Quest phase (#2699)

A new availability score will take into account more than the number of block signatures alone.

This introduces the mechanism to compute a score and print the rankings based on that. This also implements a provisional scoring formula.
go/oasis-node/txsource: Add registration workload (#2718)
go/oasis-node/txsource: Add parallel workload (#2724)
go/scheduler: Validators now returns validators by node ID (#2739)

The consensus ID isn't all that useful for most external callers, so querying it should just return the validators by node ID instead.
go/staking: Add a Delegations() call, and expose it over gRPC (#2740)

This adds a Delegations() call in the spirit of DebondingDelegations() that returns a map of delegations for a given delegator.
go/oasis-node/txsource: Use a special account for funding accounts (#2744)

Generate and fund a special account that is used for funding accounts during the workload instead of hard-coding funding for fixed addresses.

Additionally, start using fees and non-zero gas prices in workloads.

Bug Fixes

go/storage/client: Retry storage ops on specific errors (#1865)
go/tendermint: Unfatalize seed populating nodes from genesis (#2554)

In some cases we'd prefer to include some nodes in the genesis document even when they're registered with an invalid address.

This makes the seed node ignore those entries and carry on, while keeping those entries available for the rest of the system.
go: Re-enable signature verification disabled for migration (#2615)

Now that the migration has hopefully been done, re-enable all of the signature verification that was disabled for the sake of allowing for migration.
go: Don't scan the whole keyspace in badger's tendermint DB implementation (#2664)

When we run off the end of a range iteration.
runtime: Bump ring to 0.16.11, snow to 0.6.2, Rust to 2020-02-16 (#2666)
go/staking/api: Fix genesis sanity check for nonexisting accounts (#2671)

Detect when a (debonding) delegation is specified for a nonexisting account and report an appropriate error.
go/storage/mkvs: Fix iterator bug (#2691)
go/storage/mkvs: Fix bug in key.Merge operation with extra bytes (#2698)
go/storage/mkvs: Fix removal crash when key is too short (#2698)
go/storage/mkvs: Fix node unmarshallers (#2703)
go/storage/mkvs: Fix proof verifier (#2703)
go/consensus/tendermint: Properly cache consensus parameters (#2708)
go/badger: Enable truncate to recover from corrupted value log file (#2732)

Apparently badger is not at all resilient to crashes unless the truncate option is enabled.
go/oasis-net-runner/fixtures: Increase scheduler max batch size to 16 MiB (#2741)

This change facilitates RPCs to larger, more featureful runtimes.
go/common/version: Allow omission of trailing numbers in parSemVerStr() (#2742)

Go's runtime.Version() function can omit the patch number, so augment parSemVerStr() to handle that.

Internal changes

github: Bump GoReleaser to 0.127.0 and switch back to upstream (#2564)
github: Add new steps to ci-lint workflow (#2572, #2692, #2717)

Add Lint git commits, Lint Markdown files, Lint Change Log fragments and Check go mod tidy steps to ci-lint GitHub Actions workflow.

Remove Lint Git commits step from Buildkite's CI pipeline.
ci: Skip some steps for non-code changes (#2573, #2702)

When one makes a pull request that e.g. only adds documentation or assembles the Change Log from fragments, all the heavy Buildkite pipeline steps (e.g. Go/Rust building, Go tests, E2E tests) should be skipped.
go/common/cbor: Bump fxamacker/cbor to v2.2 (#2635)
go/storage/mkvs: Fuzz storage proof decoder (#2637)
ci: merge coverage files per job (#2644)
go/oasis-test-runner: Update multiple-runtimes E2E test (#2650)

Reduce all group sizes to 1 with no backups, use EpochtimeMock to avoid unexpected blocks, add numComputeWorkers parameter.
Replace redundant fields with Consensus accessors (#2650)

Backend in go/consensus/api contains among others accessors for Beacon, EpochTime, Registry, RootHash, Scheduler, and KeyManager. Use those instead of direct references. The following structs were affected:
- Node in go/cmd/node,
- Node in go/common/committee,
- Worker in go/common,
- clientCommon in go/runtime/client,
- Group in go/worker/common/committee.
go/storage: Refactor checkpointing interface (#2659)

Previously the way storage checkpoints were implemented had several drawbacks, namely:
- Since the checkpoint only streamed key/value pairs this prevented correct tree reconstruction as tree nodes also include a Round field which specifies the round at which a given tree node was created.
- While old checkpoints were streamed in chunks and thus could be resumed or streamed in parallel from multiple nodes, there was no support for verifying the integrity of a single chunk.
This change introduces an explicit checkpointing mechanism with a simple file-based backend reference implementation. The same mechanism could also be used in the future with Tendermint's app state sync proposal.
changelog: Use Git commit message style for Change Log fragments (#2662)

For more details, see the description in Change Log fragments.
Make: Add lint targets (#2662, #2692)

Add a general lint target that depends on the following lint targets:
- lint-go: Lint Go code,
- lint-git: Lint git commits,
- lint-md: Lint Markdown files (except Change Log fragments),
- lint-changelog: Lint Change Log fragments.
Add sanity checks for stake accumulator state integrity (#2665)
go/consensus/tendermint: Don't use UnsafeSigner (#2670)
rust: Update ed25519-dalek and associated dependencies (#2678)

This change updates ed25519-dalek, rand and x25519-dalek.
Bump minimum Go version to 1.13.8 (#2689)
go/storage/mkvs: Add overlay tree to support rolling back state (#2691)
go/storage/mkvs: Make Tree an interface (#2691)
gitlint: Require body length of at least 20 characters (if body exists) (#2692)
Make build-fuzz work again, test it on CI (#2695)
changelog: Reference multiple issues/pull requests for a single entry (#2697)

For more details, see the description in Change Log fragments.
go/storage/mkvs: Add MKVS fuzzing (#2698)
github: Don't trigger ci-reproducibility workflow for pull requests (#2704)
go/oasis-test-runner: Improve txsource E2E test (#2709)

This adds the following general txsource scenario features:
- Support for multiple parallel workloads.
- Restart random nodes on specified interval.
- Ensure consensus liveness for the duration of the test.
It also adds an oversized txsource workload which submits oversized transactions periodically.
go/consensus/tendermint: Expire txes when CheckTx is disabled (#2720)

When CheckTx is disabled (for debug purposes only, e.g. in E2E tests), we still need to periodically remove old transactions as otherwise the mempool will fill up. Keep track of transactions were added and invalidate them when they expire.
runtime: Remove the non-webpki/snow related uses of ring (#2733)

As much as I like the concept of ring as a library, and the implementation, the SGX support situation is ridiculous, and we should minimize the use of the library for cases where alternatives exist.

20.3 (2020-02-06)

Removals and Breaking changes

Add gRPC Sentry Nodes support (#1829)

This adds gRPC proxying and policy enforcement support to existing sentry nodes, which enables protecting upstream nodes' gRPC endpoints.

Added/changed flags:
- worker.sentry.grpc.enabled: enables the gRPC proxy (requires worker.sentry.enabled flag)
- worker.sentry.grpc.client.port: port on which gRPC proxy is accessible
- worker.sentry.grpc.client.address: addresses on which gRPC proxy is accessible (needed so protected nodes can query sentries for its addresses)
- worker.sentry.grpc.upstream.address: address of the protected node
- worker.sentry.grpc.upstream.cert: public certificate of the upstream grpc endpoint
- worker.registration.sentry.address renamed back to worker.sentry.address
- worker.registration.sentry.cert_file renamed back to worker.sentry.cert_file
go/common/crypto/tls: Use ed25519 instead of P-256 (#2058)

This change is breaking as the old certificates are no longer supported, and they must be regenerated. Note that this uses the slower runtime library ed25519 implementation instead of ours due to runtime library limitations.
All marshalable enumerations in the code were checked and the default Invalid = 0 value was added, if it didn't exist before (#2546)

This makes the code less error prone and more secure, because it requires the enum field to be explicitly set, if some meaningful behavior is expected from the corresponding object.
go/registry: Add ConsensusParams.MaxNodeExpiration (#2580)

Node expirations being unbound is likely a bad idea. This adds a consensus parameter that limits the maximum lifespan of a node registration, to a pre-defined number of epochs (default 5).

Additionally the genesis document sanity checker is now capable of detecting if genesis node descriptors have invalid expirations.

Note: Existing deployments will need to alter the state dump to configure the maximum node expiration manually before a restore will succeed.
go/common/crypto/signature: Use base64-encoded IDs/public keys (#2588)

Change String() method to return base64-encoded representation of a public key instead of the hex-encoded representation to unify CLI experience when passing/printing IDs/public keys.
go/registry: Disallow entity signed node registrations (#2594)

This feature is mostly useful for testing and should not be used in production, basically ever. Additionally, when provisioning node descriptors, --node.is_self_signed is now the default.

Note: Breaking if anyone happens to use said feature, but enabling said feature is already feature-gated, so this is unlikely.
go/registry: Ensure that node descriptors are signed by all public keys (#2599)

To ensure that nodes demonstrate proof that they posses the private keys for all public keys contained in their descriptor, node descriptors now must be signed by the node, consensus, p2p and TLS certificate key.

Note: Node descriptors generated prior to this change are now invalid and will be rejected.
Rewards and fees consensus parameters (#2624)

Previously things like reward "factors" and fee distribution "weights" were hardcoded. But we have pretty good support for managing consensus parameters, so we ought to move them there.
Special rewards for block proposer (#2625)
- a larger portion of the fees
- an additional reward
go/consensus/tendermint: mux offset block height consistently (#2634)

We've been using blockHeight+1 for getting the epoch time except for on blockHeight=1. The hypothesis in this change is that we don't need that special case.
Tendermint P2P configuration parameters (#2646)

This allows configuring P2P parameters:
- MaxNumInboundPeers,
- MaxNumOutboundPeers,
- SendRate and
- RecvRate
through their respective CLI flags:
- tendermint.p2p.max_num_inbound_peers,
- tendermint.p2p.max_num_outbound_peers,
- tendermint.p2p.send_rate, and
- tendermint.p2p.recv_rate.
It also increases the default value of MaxNumOutboundPeers from 10 to 20 and moves all P2P parameters under the tendermint.p2p.* namespace.
Add oasis-node identity tendermint show-{node,consensus}-address subcommands (#2649)

The show-node-address subcommmand returns node's public key converted to Tendermint's address format. It replaces the oasis-node debug tendermint show-node-id subcommand.

The show-consensus-address subcommand returns node's consensus key converted to Tendermint's address format.

Features

Refresh node descriptors mid-epoch (#1794)

Previously node descriptors were only refreshed on an epoch transition which meant that any later updates were ignored until the next epoch. This caused stale RAKs to stay in effect when runtime restarts happened, causing attestation verification to fail.

Enabling mid-epoch refresh makes nodes stay up to date with committee member node descriptor updates.
go/worker/storage: Add configurable limits for storage operations (#1914)
Flexible key manager policy signers (#2444)

The key manager runtime has been split into multiple crates to make its code reusable. It is now possible for others to write their own key managers that use a different set of trusted policy signers.
Add oasis-node registry node is-registered subcommand (#2508)

It checks whether the node is registered.
Runtime node admission policies (#2513)

With this, each runtime can define its node admission policy.

Currently only two policies should be supported:
- Entity whitelist (only nodes belonging to whitelisted entities can register to host a runtime).
- Anyone with enough stake (currently the only supported policy).
The second one (anyone with enough stake) can introduce liveness issues as long as there is no slashing for compute node liveness (see #2078).
go/keymanager: Support policy updates (#2516)

This change adds the ability for the key manager runtime owner to update the key manger policy document at runtime by submitting an appropriate transaction.

Note: Depending on the nature of the update it may take additional epoch transitions for the key manager to be available to clients.
Tooling for runtimes' node admission policy (#2563)

We added a policy type where you can whitelist the entities that can operate compute nodes. This adds the tooling around it so things like the registry runtime genesis init tool can set them up.
Export signer public key to entity (#2609)

We added a command to export entities from existing signers, and a check to ensure that the entity and signer public keys match.

This makes it so that a dummy entity cannot be used for signers backed by Ledger.
go/registry: Handle the old and busted node descriptor envelope (#2614)

The old node descriptor envelope has one signature. The new envelope has multiple signatures, to ensure that the node has access to the private component of all public keys listed in the descriptor.

The correct thing to do, from a security standpoint is to use a new set of genesis node descriptors. Instead, this change facilitates the transition in what is probably the worst possible way by:
- Disabling signature verification entirely for node descriptors listed in the genesis document (Technically this can be avoided, but there are other changes to the node descriptor that require no verification to be done if backward compatibility is desired).
- Providing a conversion tool that fixes up the envelopes to the new format.
- Omitting descriptors that are obviously converted from state dumps.
Note: Node descriptors that are using the now deprecated option to use the entity key for signing are not supported at all, and backward compatibility will NOT be maintained.
go/oasis-node/cmd/debug/fixgenesis: Support migrating Node.Roles (#2620)

The node.RolesMask bit definitions have changed since the last major release deployed to the wild, so support migrating things by rewriting the node descriptor.

Note: This assumes that signature validation in InitChain is disabled.

Bug Fixes

go/registry: deduplicate registry sanity checks and re-enable address checks (#2428)

Existing deployments had invalid P2P/Committee IDs and addresses as old code did not validate all the fields at node registration time. All ID and address validation checks are now enabled.

Additionally, separate code paths were used for sanity checking of the genesis and for actual validation at registration time, which lead to some unexpected cases where invalid genesis documents were passing the validation. This code (at least for registry application) is now unified.
Make oasis-node binaries made with GoReleaser via GitHub Actions reproducible again (#2571)

Add -buildid= back to ldflags to make builds reproducible again.

As noted in 60641ce, this should be no longer necessary with Go 1.13.4+, but there appears to be a specific issue with GoReleaser's build handling.
go/worker/storage: Fix sync deadlock (#2584)
go/consensus/tendermint: Always accept own transactions (#2586)

A validator node should always accept own transactions (signed by the node's identity key) regardless of the configured gas price.
go/registry: Allow expired registrations at genesis (#2598)

The dump/restore process requires this to be permitted as expired registrations are persisted through an entity's debonding period.
go/oasis-node/cmd/registry/runtime: Fix loading entities in registry runtime subcommands (#2606)
go/storage: Fix invalid memory access crash in Urkel tree (#2611)
go/consensus/tendermint/apps/staking: Fix epochtime overflow (#2627)
go/tendermint/keymanager: Error in Status() if keymanager doesn't exist (#2628)

This fixes panics in the key-manager client if keymanager for the specific runtime doesn't exist.
go/oasis-node/cmd/stake: Make info subcommand tolerate invalid thresholds (#2632)

Change the subcommand to print valid staking threshold kinds and warn about invalid ones.
go/staking/api: Check if thresholds for all kinds are defined in genesis (#2633)
go/cmd/registry/runtime: Fix provisioning a runtime without keymanager (#2639)
registry/api/sanitycheck: Move genesis stateroot check into registration (#2643)

Runtime genesis check should only be done when registering, not during the sanity checks.
Make oasis control is-synced subcommand more verbose (#2649)

Running oasis-node control is-synced will now print a message indicating whether a node has completed initial syncing or not to stdout in addition to returning an appropriate status code.

Internal changes

go/genesis: Fix genesis tests and registry sanity checks (#2589)
github: Add ci-reproducibility workflow (#2590)

The workflow spawns two build jobs that use the same build environment, except for the path of the git checkout. The oasis-node binary is built two times, once directly via Make's go build invocation and the second time using the GoReleaser tool that is used to make the official Oasis Core releases. The last workflow job compares both checksums of both builds and errors if they are not the same.
go/consensus/tendermint/abci: Add mock ApplicationState (#2629)

This makes it easier to write unit tests for functions that require ABCI state.

20.2 (2020-01-21)

Removals and Breaking changes

go node: Unite compute, merge, and transaction scheduler roles (#2107)

We're removing the separation among registering nodes for the compute, merge, and transaction scheduler roles. You now have to register for and enable all or none of these roles, under a new, broadened, and confusing--you're welcome--term "compute".
Simplify tendermint sentry node setup. (#2362)

Breaking configuration changes:
- worker.sentry.address renamed to: worker.registration.sentry.address
- worker.sentry.cert_file renamed to: worker.registration.sentry.cert_file
- tendermint.private_peer_id removed
- added tendermint.sentry.upstream_address which should be set on sentry node and it will set tendermint.private_peer_id and tendermint.peristent_peer for the configured addresses
Charge gas for runtime transactions and suspend runtimes which do not pay periodic maintenance fees (#2504)

This introduces gas fees for submitting roothash commitments from runtime nodes. Since periodic maintenance work must be performed on each epoch transition (e.g., electing runtime committees), fees for that maintenance are paid by any nodes that register to perform work for a specific runtime. Fees are pre-paid for the number of epochs a node registers for.

If the maintenance fees are not paid, the runtime gets suspended (so periodic work is not needed) and must be resumed by registering nodes.
go: Rename compute -> executor (#2525)

It was proposed that we rename the "compute" phase (of the txnscheduler, compute, merge workflow) to "executor".

Things that remain as "compute":
- the registry node role
- the registry runtime kind
- the staking threshold kind
- things actually referring to processing inputs to outputs
- one of the drbg contexts
So among things that are renamed are fields of the on-chain state and command line flags.
RuntimeID is not hard-coded in the enclave anymore (#2529)

It is passed when dispatching the runtime. This enables the same runtime binary to be registered and executed multiple times with different RuntimeID.

Features

Consensus simulation mode and fee estimator (#2521)

This change allows the compute nodes to participate in networks which require gas fees for various operations in the network. Gas is automatically estimated by simulating transactions while gas price is currently "discovered" manually via node configuration.

The following configuration flags are added:
- consensus.tendermint.submission.gas_price should specify the gas price that the node will be using in all submitted transactions.
- consensus.tendermint.submission.max_fee can optionally specify the maximum gas fee that the node will use in submitted transactions. If the computed fee would ever go over this limit, the transaction will not be submitted and an error will be returned instead.
Optimize registry runtime lookups during node registration (#2538)

A performance optimization to avoid loading a list of all registered runtimes into memory in cases when only a specific runtime is actually needed.

Bug Fixes

Don't allow duplicate P2P connections from the same IP by default (#2558)
go/extra/stats: handle nil-votes and non-registered nodes (#2566)
go/oasis-node: Include account ID in stake list -v subcommand (#2567)

Changes stake list -v subcommand to return a map of IDs to accounts.
Use a newer version of the oasis-core tendermint fork (#2569)

The updated fork has additional changes to tendermint to hopefully prevent the node from crashing if the file descriptors available to the process get exhausted due to hitting the rlimit.

While no forward progress can be made while the node is re-opening the WAL, the node will now flush incoming connections that are in the process of handshaking, and retry re-opening the WAL instead of crashing with a panic.

Documentation improvements

Document versioning scheme used for Oasis Core and its Protocols (#2457)

See Versioning Scheme.
Document Oasis Core's release process (#2565)

See Release process.

20.1 (2020-01-14)

Features

go/worker/txnscheduler: Check transactions before queuing them (#2502)

The transaction scheduler can now optionally run runtimes and check transactions before scheduling them (see issue #1963). This functionality is disabled by default, enable it with worker.txn_scheduler.check_tx.enabled.

Bug Fixes

go/runtime/client: Return empty sequences instead of nil (#2542)

The runtime client endpoint should return empty sequences instead of nil as serde doesn't know how to decode a NULL when the expected type is a sequence.
Temporarily disable consensus address checks at genesis (#2552)

20.0 (2020-01-10)

Removals and Breaking changes

Use the new runtime ID allocation scheme (#1693)

This change alters the runtime ID allocation scheme to reserve the first 64 bits for flags indicating various properties of the runtime, and to forbid registering runtimes that have test runtime IDs unless the appropriate consensus flag is set.
Remove staking-related roothash messages (#2377)

There is no longer a plan to support direct manipulation of the staking accounts from the runtimes in order to isolate the runtimes from corrupting the consensus layer.

To reduce complexity, the staking-related roothash messages were removed. The general roothash message mechanism stayed as-is since it may be useful in the future, but any commits with non-empty messages are rejected for now.
Refactoring of roothash genesis block for runtime (#2426)
- RuntimeGenesis.Round field was added to the roothash block for the runtime which can be set by --runtime.genesis.round flag.
- The RuntimeGenesis.StorageReceipt field was replaced by StorageReceipts list, one for each storage node.
- Support for base64 encoding/decoding of Bytes was added in rust.
When registering a new runtime, require that the given key manager ID points to a valid key manager in the registry (#2459)
Remove oasis-node debug dummy sub-commands (#2492)

These are only useful for testing, and our test harness has a internal Go API that removes the need to have this functionality exposed as a sub-command.
Make storage per-runtime (#2494)

Previously there was a single storage backend used by oasis-node which required that a single database supported multiple namespaces for the case when multiple runtimes were being used in a single node.

This change simplifies the storage database backends by removing the need for backends to implement multi-namespace support, reducing overhead and cleanly separating per-runtime state.

Due to this changing the internal database format, this breaks previous (compute node) deployments with no way to do an automatic migration.

Features

Add oasis-node debug storage export sub-command (#1845)
Add fuzzing for consensus methods (#2245)

Initial support for fuzzing was added, along with an implementation of it for some of the consensus methods. The implementation uses oasis-core's demultiplexing and method dispatch mechanisms.
Add storage backend fuzzing (#2246)

Based on the work done for consensus fuzzing, support was added to run fuzzing jobs on the storage api backend.
Add oasis-node unsafe-reset sub-command which resets the node back to a freshly provisioned state, preserving any key material if it exists (#2435)
Add txsource (#2478)

The so-called "txsource" utility introduced in this PR is a starting point for something like a client that sends transactions for a long period of time, for the purpose of creating long-running tests.

With this change is a preliminary sample "workload"--a DRBG-backed schedule of transactions--which transfers staking tokens around among a set of test accounts.
Add consensus block and transaction metadata accessors (#2482)

In order to enable people to build "network explorers", we exposed some additional methods via the consensus API endpoint, specifically:
- Consensus block metadata.
- Access to raw consensus transactions within a block.
- Stream of consensus blocks as they are finalized.
Make maximum in-memory cache size for runtime storage configurable (#2494)

Previously the value of 64mb was always used as the size of the in-memory storage cache. This adds a new configuration parameter/command-line flag --storage.max_cache_size which configures the maximum size of the in-memory runtime storage cache.
Undisable transfers for some senders (#2498)

Ostensibly for faucet purposes while we run the rest of the network with transfers disabled, this lets us identify a whitelist of accounts from which we allow transfers when otherwise transfers are disabled.

Configure this with a map of allowed senders' public keys -> true in the new undisable_transfers_from field in the staking consensus parameters object along with "disable_transfers": true.
Entity block signatures count tool (#2500)

The tool uses node consensus and registry API endpoints and computes the per entity block signature counts.

Bug Fixes

Reduce Badger in-memory cache sizes (#2484)

The default is 1 GiB per badger instance and we use a few instances so this resulted in some nice memory usage.

19.0 (2019-12-18)

Process

Start using the new Versioning and Release process for Oasis Core. (#2419)

Adopt a CalVer (calendar versioning) scheme for Oasis Core (as a whole) with the following format:
```
YY.MINOR[.MICRO][-MODIFIER]
```
where:
- YY represents short year (e.g. 19, 20, 21, ...),
- MINOR represents the minor version starting with zero (e.g. 0, 1, 2, 3, ...),
- MICRO represents (optional) final number in the version (sometimes referred to as the "patch" segment) (e.g. 0, 1, 2, 3, ...).
  
  If the MICRO version is 0, it is be omitted.
- MODIFIER represents (optional) build metadata, e.g. git8c01382.
The new Versioning and Release process will be described in more detail in the future. For more details, see #2457.

Files

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

Change Log

24.3 (2024-10-14)

Configuration Changes

Features

Bug Fixes

Internal Changes

24.2 (2024-08-20)

Configuration Changes

Features

Bug Fixes

Internal Changes

24.1 (2024-06-28)

Features

Bug Fixes

Internal Changes

24.0 (2024-05-13)

Configuration Changes

Features

Bug Fixes

Documentation Improvements

Internal Changes

23.0 (2023-10-10)

Removals and Breaking Changes

Configuration and Operational Changes

Deprecations

Features

Bug Fixes

Documentation Improvements

Internal Changes

22.2 (2022-10-13)

Removals and Breaking Changes

Features

Bug Fixes

Documentation Improvements

Internal Changes

22.1 (2022-04-01)

Upgrading From Previous Versions

Removals and Breaking Changes

Configuration Changes

Features

Bug Fixes

Internal Changes

22.0 (2022-03-01)

Removals and Breaking Changes

Configuration Changes

Features

Bug Fixes

Documentation Improvements

Internal Changes

21.3 (2021-09-27)

Removals and Breaking Changes

Configuration Changes

Features

Bug Fixes

Documentation Improvements

Internal Changes

21.2 (2021-06-07)

Upgrading From Previous Versions

Removals and Breaking Changes

Configuration Changes

Features

Bug Fixes

Documentation Improvements

Internal Changes

21.1 (2021-04-12)

Removals and Breaking Changes

Features

Bug Fixes

Internal Changes

21.0 (2021-03-18)

Removals and Breaking Changes

Configuration Changes

Features

Bug Fixes