Testnet DayLight | Release Cycle Update #21
Our development planning features a Release Cycle of three weeks, providing a consistent stream of updates to the community, developers, and businesses relying on Dusk technology.
The publicly available Dusk Network GitHub contains over 18 active repositories, each focused on a different technical subject. Progress in each is ongoing and can be followed in real-time. The Release Cycle process is currently applied to some of our most active repositories.
Release Cycle planning has not yet been applied to all repositories. When they are, future installments of the Release Cycle updates will see an expansion of repositories in the spotlight.
For the community's benefit, we’ve chosen a selection of some (but not all) of the work that is being done in our repositories, including recent advancements.
merkle
Closed issues
Use only one EMPTY_SUBTREE #49
The Aggregate trait makes the user provide an array of values of Self which represent the empty subtrees at each level. We should only use a single EMPTY_SUBTREE. This allows for a much more ergonomic API.
Change Aggregate to take an array of references instead of an iterator #44
Currently Aggregate aggregates over an Iterator, but it is actually much nicer to aggregate over an array of width arity of references.
This also makes sure that Aggregate aggregates over exactly arity items.
The file 'lib.rs' is too big and can be splitted. I propose to move all Tree related code to form its own module and the same for Node. Helper functions like capacity can stat in 'lib.rs`.
Implement Standardized CI Workflow #39
We've recently introduced a set of reusable workflow files in our .github repository to standardize our CI/CD process for Rust projects. These new workflows use up-to-date, maintained actions and also introduce caching, leading to significant performance improvements in subsequent runs. This repo is a prime candidate to benefit from this new setup.
In this crate the tree is constant over a generic height. To mimic a behavior where a tree is as small as possible and grows with the insertion of leaves, we can add the functionality to get the smallest node in the tree that holds all leaves that have been inserted so far. This smallest subtree can be seen as the root of a tree that can grow with the insertion of new leaves, even though the underlying data structure always keeps the same height.
Lazy computation of level hashes #37
At the moment, batch inserting of leaves leads to redundant re-calculations of the level hashes. We might want to compute the level hashes only when we return an opening.
use criterion for the benchmarks #32
Implement the benchmarks using criterion.
Add detailed explanation for the public types, traits and methods in this crate #31
Flesh up the documentation on public types, traits and methods. Especially the poseidon implementation and opening gadget.
Poseidon252
Open issues
Implement Standardized CI Workflow #219
We've recently introduced a set of reusable workflow files in our .github repository to standardize our CI/CD process for Rust projects. These new workflows use up-to-date, maintained actions and also introduce caching, leading to significant performance improvements in subsequent runs. This repo is a prime candidate to benefit from this new setup.
Add sponge framework over fixed size input #215
Add a sponge function that is generic over the input length and therefore doesn't need to add a padding at the end of the input, saving a full computation when the input length can be divided by the rate.
rusk
Open issues
Resolve any incompatibility issues between rusk and golang node implementations #921
We're wrapping up the full migration of dusk-blockchain into rusk repository (golang -> rust). Before moving on, we need to ensure that both developments are the same implementation of the protocol without any breaking change.
Reduce block allocation in ledger database #916
On persisting a block, any block transaction is stored in duplicate. To enhance the storage of transaction data without duplications, it is necessary to improve the current data schema utilized in persisting a single block. The suggested approach is to adopt the data schema employed by the dusk-blockchain.
Reload most recent block from persisted ledger state on startup #915
Upon startup, chain component must retrieve the latest block from the persistent ledger database. This allows the node to trigger catch-up procedure from lastly persisted state.
Mempool: replace transaction by paying more #914
Currently, if a transaction pay very small gas price, it might get stuck in the mempool, with high volumes of transactions. If this happens, there is no way to get it unstuck: if the user submits a new transaction spending the same notes but paying more gas price, it gets rejected.
Remove delay in generate_block #913
Currently, generate_block waits CONSENSUS_DELAY_MS to avoid avoid consensus-split situation.
This should not be needed anymore because we include iteration in the block header.
Implement greedy transaction inclusion for new blocks #912
Currently, when creating a new candidate block, we follow this strategy:
- We include txs summing their gas limit until the sum reaches the block limit + 10% (slippage_gas_limit).
Pseudocode:
mempool_txs = get_txs_by_gas_price()
total_gas = 0 ; txs_list = []
for tx in mempool_txs:
if total_gas + tx.gas_limit < slippage_gas_limit
txs_list.push(tx)- We execute the state transition with these txs, and exclude txs that were not executed by the vm.
This approach has the advantage of speeding up block creation, but includes transactions only based on their gas_limit. This might produce situations like the following one:
- suppose we had a block limit of 1000 and a mempool with 100 transaction with gas_limit=100, but which actually spend 10; in this case we would only include 10 transactions, instead of 100, because we consider the gas_limit instead of the actually spent gas;
Replace gas_price with gas_limit in get_txs_sorted_by_fee #911
In get_txs_sorted_by_fee, total_gas should add tx's gas_limit instead of gas_price. Also, when exceeding slippage_gas_limit, loop should continue instead break, so to fetch another tx.
Make Sortition Hash compliant with specification #907
Update consensus nomenclature #905
Following discussions in the past months, we should update several names in the consensus protocol.
Closed issues
OoM in get_provisioners RPC #908
When there are enough stakes managed by the transfer contract, the get_provisioners will fail due to the argument buffer being too small to hold all of them at the same time.
node: Add Iteration field to Block header #848
This is needed in order to reflect protocol changes made in dusk-network/dusk-blockchain#1500
node: Add transactions merkle tree root to the block header #847
This is needed in order to reflect protocol changes made in dusk-network/dusk-blockchain#1499
Upgrade tar-rs dependency #832
Current used version of tar-rs has 2 major vulnerabilities:
jubjub
Closed issues
Merge our implementation with upstream implementation #115
Our implementation has diverted significantly from the zkcrypto upstream original. In order to get the upstream improvements it is good to have them merged.
Implement Standardized CI Workflow #113
We've recently introduced a set of reusable workflow files in our .github repository to standardize our CI/CD process for Rust projects. These new workflows use up-to-date, maintained actions and also introduce caching, leading to significant performance improvements in subsequent runs. This repo is a prime candidate to benefit from this new setup.
Separate our custom implementations from the originally forked #110
Currently the original implementation and our additions are intertwined all across the crate. We want to get an overview on the differences (and similarities) between our fork and the original. For this to happen we need a clear separation between the two.
Remove canonical #109
Remove all canonical references.
wallet-cli
Open issues
Populate the cache database on startup with all the CF #149
Populate the database on startup with all the CF(s) because we know them before we fetch_notes.
Benchmark fetch_notes in clients.rs with MAX_ADDRESSES #146
Benchmark fetch_notes in clients.rs and reduce the number of max addresses to see if there's significant overhead.
Use nullifier as key for note's store #144
With the upcoming implementation of the cache, there is the need to store notes in a unique way (actually we use the hash of the notes).
A nice feature to have, to shrink the database size and improve the performances, is to store only the unspent notes, and get rid of all the notes that are then useless (those already spent).
Instead of using the note's hash as key for the note, we should use the nullifier.
The uniqueness property is not touch, but the nullifiers give us some technical way to improve the fetch notes handler.
- We can easily check if a notes is spent, just checking the key of the stored value (no need to compute nullifier on the fly)
- We can then remove the spent notes from the database (or at least we can move it to a different column family)
Closed issues
Wrong network fallback to default #143
If you run the wallet-cli specifying a network that doesn't exist, the network used is the default one.
bls12_381
Closed issues
Separate our custom implementations from the originally forked #109
Currently the original implementation and our additions are intertwined all across the crate. We want to get an overview on the differences (and similarities) between our fork and the original. For this to happen we need a clear separation between the two.
dusk-blockchain
Open issues
Invalid state root after restart #1530
Closed issues
Panic while deserializing transaction #1529
When a malformed transaction is received, dusk-blockchain will panic while calculating hash.
Make CommitteeSize consistent with size #1520
The CommitteeSize function decides the committee size depending on the number of provisioners.
This is somewhat inconsistent with the committee 'size' as number of voting credits.
In particular, given how this function behaves, the committee size is set to less than ConsensusMaxCommitteeSize (64) if the number of provisioners is lower than this value.
citadel
Open issues
Specify and implement lifecycle mechanisms for License data #53
License data needs a mechanism to enforce its lifecycle, some elements need to expire, or compensation is needed should the license not be issued. Such mechanisms need to be specified and implemented.
Specify and implement mechanism for delegation of the license ZK proof calculation #52
ZK proof calculation is either done by the user (of Citadel protocol) or it could be delegated to the node. Details for this delegation solution, as well as implementation, are subject of this issue.
Closed issues
Research on the computation of k_lic #35
Right now, we compute a value k_lic = G · H(lsk), where lsk is the license secret key, and H() is a hash function. k_lic is a symmetric key used to encrypt data using Poseidon, and shared between two parties. Likewise, the other party should never learn lsk.
We should research if this approach is correct, and in that case, if using a truncated hash function (either Blake or Poseidon) is secure in this scenario.
Plus, it is important to remember that this approach was followed because, if we provide kdh as done in Phoenix, the SP can recover the public key of the user from kdh and lpk.
As you can see, development is continuous and this list is not exhaustive. For example, minor issues and advancements concerning the Kadcast element have also been opened and resolved. The complete series of repositories can be found on GitHub.
Breakthrough developments will receive a separate spotlight, such as our latest deployment of Daylight. Thank you very much for your understanding and feedback as we continue to find the best way to provide the community with transparency and development information.
In A Nutshell: Release Cycle planning
Release Cycle development planning has been adopted by major companies including Google, Mozilla, and during the development of products such as Ubuntu, Kubernetes, and many more. The reason for this is clear: Release Cycle planning improves the predictability of software development for developers and the community alike.
For a more detailed explanation of the concept (along with frequently asked questions) please scroll down to the bottom of this article.
The below noted additions/changes to the repository can be followed on GitHub, for further details on their status and their function in the stack.
Piecrust v0.5.0
Added
Add Session::call_raw allowing for deferred (de)serialization [#218]
Add MAP_NORESERVE flag to mmap syscall [#213]
Changed
Include points_limit in c import [#216]
jubjub v0.13.0
Added
Add more tests for wnaf computation [#104]
Changed
Merge upstream changes from zkcrypto [#115]
Removed
Remove canonical and canonical_derive dependency [#109]
wallet-cli v0.15.0
Changed
Cache implementation now uses rocksdb instead of microkelvin [#56]
Fixed
Throw an error there when specifying a network that does not exist [#143]
bls12_381 v0.12.0
Added
Align crate with upstream zkcrypto/bls12_381 [#117]
Removed
Remove scalar field generator [#100]
Remove canonical and canonical_derive dependency [#108]
Changed
Rename serde_req feature to serde [#117]
Separate Dusk's additions from the original crate [#109]
merkle v0.4.0
Added
Documented poseidon related types [#31]
Changed
Modify Aggregate to include only one EMPTY_SUBTREE [#49]
Relax Aggregate bounds to not be Copy [#49]
Change return of Tree::smallest_subtree to Ref<T> as opposed to Option<Ref<T>> [#49]
FAQ
We’ve included a small FAQ section below to make sure the community understands our intention with Release Cycles.
What is Release Cycle planning?
Release Cycle planning means that developmental updates are published at consistent intervals on GitHub; in the case of Dusk Network, every three weeks. These releases describe the latest additions, changes, fixes, and assets added to the tech stack by the development team.
Dusk Network currently has over 18+ active repositories on GitHub, each repository covering a different technical project. Release Cycle planning has not yet been applied to all repositories; the current focus on a single GitHub repository for clarity does not mean there are no ongoing developmental efforts occurring in other areas.
Does being featured in a Release mean Deployment?
The Release Cycle updates does not mean immediate deployments on our testnet. It is a release detailing additions, changes, and fixes to the repositories that we are ready to share with the public. All released content is considered stable, consistent, reviewed, and cross-checked with other repositories.
Are Release Cycle updates a spotlight for major deliverables?
No. The Release Cycle approach is strictly a way to provide a consistent shape for publication of our work in the clearest terms possible. They are not deadlines to be made, nor are they tied to specific development sprints. Release Cycle updates aim to raise attention with the community, and give proper coverage to, publishable GitHub releases. Major deliverables will be given their proper publication in dedicated articles.
