Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ln/replace best chain with block index #1413

Draft
wants to merge 48 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
fc9a0d3
start by refactoring block index into a read-through cache
lazynina Sep 23, 2024
f3e8e4c
best chain and header chain with read through cache
lazynina Sep 24, 2024
a2e5b4b
check bestHeaderChain.ChainMap ONLY when processing pos headers
lazynina Sep 24, 2024
ff6ee24
updates for locating headers and faster cache usage
lazynina Sep 24, 2024
71b1eaa
more fixes
lazynina Sep 25, 2024
17a56bd
more fixes
lazynina Sep 26, 2024
d153927
more fix, specifically around .Ancestor
lazynina Sep 26, 2024
47e1958
bump test timeout
lazynina Sep 26, 2024
5e5c0c1
load all blocks after committed tip in block index
lazynina Sep 26, 2024
9a32c26
fix GetBlockNodeByHashAndHeight - no need for checking height > maxHe…
lazynina Sep 26, 2024
5359be8
bump timeout to 60m
lazynina Sep 27, 2024
89a845e
try hashicorp lru cache
lazynina Sep 27, 2024
4f7ff12
Fix up some more tests and remove logging
lazynina Sep 27, 2024
722dcc0
fix string slice workaround, fix calc next difficulty
lazynina Sep 27, 2024
1941487
use best header chain instead of best chain for calc next difficulty …
lazynina Sep 27, 2024
627e882
fix tests
lazynina Sep 28, 2024
a469c3a
refactor to just use a block hash for best block chain and best heade…
lazynina Oct 3, 2024
c6ad24a
move block index migration to NewBlockchain instead of NewServer, add…
lazynina Oct 3, 2024
216fd31
merge main into branch
lazynina Oct 3, 2024
1f80f19
Merge branch 'ln/read-through-cache-for-block-index-and-best-chain' i…
lazynina Oct 3, 2024
8b73b1c
handle case where we haven't hit PoS cutover yet in block index migra…
lazynina Oct 3, 2024
7ac67bd
cleanup
lazynina Oct 4, 2024
7675765
do some sampling for is fully stored and check archival mode
lazynina Oct 7, 2024
148bad3
fix get blocks to store
lazynina Oct 7, 2024
8aa6a5b
cleanup
lazynina Oct 7, 2024
53b2d29
make a bunch of methods on block index private and expose block index…
lazynina Oct 7, 2024
91e22ff
don't bother healing orphan pointers if we're syncing
lazynina Oct 8, 2024
9abe238
refactor to reduce number of times we have to call hash and don't bot…
lazynina Oct 8, 2024
23991c4
upgrade cloudflare/circl
lazynina Oct 8, 2024
8adc624
Ln/replace best chain plus replace old lru cache (#1414)
lazynina Oct 8, 2024
2608ec5
Merge branch 'ln/replace-best-chain-with-block-index' of github.com:d…
lazynina Oct 8, 2024
dbad649
go mod tidy
lazynina Oct 8, 2024
ebb1175
expose SetBestChain
lazynina Oct 8, 2024
ae34cff
fix exit condition for GetBlocksToStore
lazynina Oct 8, 2024
b48c52d
use bytes.Equal instead of reflect.DeepEqual for comparing block hashes
lazynina Oct 8, 2024
925e44e
use get block from best chain by height to speed up syncing
lazynina Oct 8, 2024
dddd64c
fix condition for returning from get block by hash
lazynina Oct 8, 2024
62503c1
revert using get block from best chain by height
lazynina Oct 8, 2024
131f389
add height check to reduce calls to compare hashes
lazynina Oct 9, 2024
f57a699
fix exit condition for height in get block from best chain by hash
lazynina Oct 9, 2024
2bae1a2
allow early return from GetBlockFromBestChainByHash if we're using he…
lazynina Oct 9, 2024
ea243f3
minor enhancement to isTipCurrent
lazynina Oct 9, 2024
3f73bb6
fix txindex issue w/ genesis block
lazynina Oct 9, 2024
e200407
fix process block pow logic for genesis block
lazynina Oct 9, 2024
3ad68be
fix - get next block after tip hash in update process of txindex
lazynina Oct 9, 2024
78034d4
reduce direct access to BlockNode.Parent and get rid of heal orphan p…
lazynina Oct 9, 2024
02e9e15
fix postgres get best chain logic
lazynina Oct 9, 2024
dec4675
merge in main
lazynina Oct 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions cmd/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ type Config struct {
func GetStringSliceWorkaround(flagName string) []string {
value := viper.GetString(flagName)
if value == "" || value == " " {
return []string{}
values := viper.GetStringSlice(flagName)
return values
}
return strings.Split(value, ",")
}
Expand Down Expand Up @@ -147,7 +148,6 @@ func LoadConfig() *Config {

// Peers
config.ConnectIPs = GetStringSliceWorkaround("connect-ips")
glog.V(2).Infof("Connect IPs read in: %v", config.ConnectIPs)
config.AddIPs = GetStringSliceWorkaround("add-ips")
config.AddSeeds = GetStringSliceWorkaround("add-seeds")
config.TargetOutboundPeers = viper.GetUint32("target-outbound-peers")
Expand Down Expand Up @@ -266,6 +266,7 @@ func (config *Config) Print() {
glog.Infof("MaxSyncBlockHeight: %v", config.MaxSyncBlockHeight)
}

glog.Infof("Connect IPs: %s", config.ConnectIPs)
if len(config.ConnectIPs) > 0 {
glog.Infof("Connect IPs: %s", config.ConnectIPs)
}
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ require (
github.com/bxcodec/faker v2.0.1+incompatible
github.com/cloudflare/circl v1.5.0
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
github.com/decred/dcrd/container/lru v1.0.0
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0
github.com/deso-protocol/go-deadlock v1.0.1
github.com/deso-protocol/go-merkle-tree v1.0.0
Expand All @@ -27,6 +26,7 @@ require (
github.com/go-pg/pg/v10 v10.13.0
github.com/golang/glog v1.2.2
github.com/google/uuid v1.6.0
github.com/hashicorp/golang-lru/v2 v2.0.3
github.com/mitchellh/go-homedir v1.1.0
github.com/oleiade/lane v1.0.1
github.com/onflow/crypto v0.25.2
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,6 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/decred/dcrd/container/lru v1.0.0 h1:7foQymtbu18aQWYiY9RnNIeE+kvpiN+fiBQ3+viyJjI=
github.com/decred/dcrd/container/lru v1.0.0/go.mod h1:vlPwj0l+IzAHhQSsbgQnJgO5Cte78+yI065V+Mc5PRQ=
github.com/decred/dcrd/crypto/blake256 v1.0.0/go.mod h1:sQl2p6Y26YV+ZOcSTP6thNdn47hh8kt6rqSlvmrXFAc=
github.com/decred/dcrd/crypto/blake256 v1.1.0 h1:zPMNGQCm0g4QTY27fOCorQW7EryeQ/U0x++OzVrdms8=
github.com/decred/dcrd/crypto/blake256 v1.1.0/go.mod h1:2OfgNZ5wDpcsFmHmCK5gZTPcCXqlm2ArzUIkw9czNJo=
Expand Down Expand Up @@ -207,6 +205,8 @@ github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 h1:kes8mmyCpxJsI7FTwtzRqEy9
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2/go.mod h1:Gou2R9+il93BqX25LAKCLuM+y9U2T4hlwvT1yprcna4=
github.com/hashicorp/go-sockaddr v1.0.7 h1:G+pTkSO01HpR5qCxg7lxfsFEZaG+C0VssTy/9dbT+Fw=
github.com/hashicorp/go-sockaddr v1.0.7/go.mod h1:FZQbEYa1pxkQ7WLpyXJ6cbjpT8q0YgQaK/JakXqGyWw=
github.com/hashicorp/golang-lru/v2 v2.0.3 h1:kmRrRLlInXvng0SmLxmQpQkpbYAvcXm7NPDrgxJa9mE=
github.com/hashicorp/golang-lru/v2 v2.0.3/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/hashicorp/hcl v1.0.1-vault-5 h1:kI3hhbbyzr4dldA8UdTb7ZlVVlI2DACdCfz31RPDgJM=
github.com/hashicorp/hcl v1.0.1-vault-5/go.mod h1:XYhtn6ijBSAj6n4YqAaf7RBPS4I06AItNorpy+MoQNM=
github.com/hinshun/vt10x v0.0.0-20220119200601-820417d04eec h1:qv2VnGeEQHchGaZ/u7lxST/RaJw+cv273q79D81Xbog=
Expand Down
4 changes: 2 additions & 2 deletions lib/block_producer.go
Original file line number Diff line number Diff line change
Expand Up @@ -367,8 +367,8 @@ func (desoBlockProducer *DeSoBlockProducer) _getBlockTemplate(publicKey []byte)
blockRet.Header.TransactionMerkleRoot = merkleRoot

// Compute the next difficulty target given the current tip.
diffTarget, err := CalcNextDifficultyTarget(
lastNode, CurrentHeaderVersion, desoBlockProducer.params)
diffTarget, err := desoBlockProducer.chain.CalcNextDifficultyTarget(
lastNode, CurrentHeaderVersion)
if err != nil {
return nil, nil, nil, errors.Wrapf(err, "DeSoBlockProducer._getBlockTemplate: Problem computing next difficulty: ")
}
Expand Down
1 change: 1 addition & 0 deletions lib/block_view.go
Original file line number Diff line number Diff line change
Expand Up @@ -5114,6 +5114,7 @@ func (bav *UtxoView) GetSpendableDeSoBalanceNanosForPublicKey(pkBytes []byte,
// but we do have the header. As a result, this condition always evaluates to false and thus
// we only process the block reward for the previous block instead of all immature block rewards
// as defined by the params.
// NOTE: we are not using .GetParent here as it changes the meaning of this code.
if blockNode.Parent != nil {
nextBlockHash = blockNode.Parent.Hash
} else {
Expand Down
4 changes: 2 additions & 2 deletions lib/block_view_lockups_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2462,7 +2462,7 @@ func TestLockupBlockConnectsAndDisconnects(t *testing.T) {
require.NoError(t, utxoView.FlushToDb(blk2.Header.Height))

// Update the tip
testMeta.chain.bestChain = testMeta.chain.bestChain[:len(testMeta.chain.bestChain)-1]
testMeta.chain.blockIndex.tip = testMeta.chain.blockIndex.tip.Parent

// Validate the state update
utxoView = NewUtxoView(
Expand Down Expand Up @@ -2517,7 +2517,7 @@ func TestLockupBlockConnectsAndDisconnects(t *testing.T) {
require.NoError(t, utxoView.FlushToDb(blk1.Header.Height))

// Update the tip
testMeta.chain.bestChain = testMeta.chain.bestChain[:len(testMeta.chain.bestChain)-1]
testMeta.chain.blockIndex.setTip(testMeta.chain.blockIndex.tip.Parent)

// Verify we return back to the initial state
utxoView = NewUtxoView(
Expand Down
15 changes: 7 additions & 8 deletions lib/block_view_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ import (
"github.com/deso-protocol/core/bls"

"github.com/btcsuite/btcd/btcec/v2"
"github.com/decred/dcrd/container/lru"
"github.com/dgraph-io/badger/v4"
embeddedpostgres "github.com/fergusstrange/embedded-postgres"
"github.com/golang/glog"
"github.com/hashicorp/golang-lru/v2"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
Expand Down Expand Up @@ -702,8 +702,8 @@ func (tes *transactionTestSuite) testDisconnectBlock(tm *transactionTestMeta, te
require.NoError(err)
// sanity-check that the last block hash is the same as the last header hash.
require.Equal(true, bytes.Equal(
tm.chain.bestChain[len(tm.chain.bestChain)-1].Hash.ToBytes(),
tm.chain.bestHeaderChain[len(tm.chain.bestHeaderChain)-1].Hash.ToBytes()))
tm.chain.blockIndex.GetTip().Hash.ToBytes(),
tm.chain.blockIndex.GetHeaderTip().Hash.ToBytes()))
// Last block shouldn't be nil, and the number of expectedTxns should be the same as in the testVectorBlock + 1,
// because of the additional block reward.
require.NotNil(lastBlock)
Expand Down Expand Up @@ -791,15 +791,14 @@ func (tes *transactionTestSuite) testDisconnectBlock(tm *transactionTestMeta, te
// TODO: if ever needed we can call tm.chain.eventManager.blockDisconnected() here.

// Update the block and header metadata chains.
tm.chain.bestChain = tm.chain.bestChain[:len(tm.chain.bestChain)-1]
tm.chain.bestHeaderChain = tm.chain.bestHeaderChain[:len(tm.chain.bestHeaderChain)-1]
delete(tm.chain.bestChainMap, *lastBlockHash)
delete(tm.chain.bestHeaderChainMap, *lastBlockHash)
tm.chain.blockIndex.setTip(tm.chain.BlockTip().GetParent(tm.chain.blockIndex))
tm.chain.blockIndex.setHeaderTip(tm.chain.HeaderTip().GetParent(tm.chain.blockIndex))

// We don't pass the chain's snapshot above to prevent certain concurrency issues. As a
// result, we need to reset the snapshot's db cache to get rid of stale data.
if tm.chain.snapshot != nil {
tm.chain.snapshot.DatabaseCache = *lru.NewMap[string, []byte](DatabaseCacheSize)
tm.chain.snapshot.DatabaseCache, err = lru.New[string, []byte](int(DatabaseCacheSize))
require.NoError(err)
}

// Note that unlike connecting test vectors, when disconnecting, we don't need to verify db entries.
Expand Down
Loading
Loading