Stellar to keep only 1 year of historic data on horizon nodes #162

New Issue

thabeta · 2025-01-09T14:57:19Z

thabeta commented

2025-01-09 14:57:19 +00:00

Description

Stellar changes to 1year only of historic data https://stellar.org/blog/foundation-news/sdf-s-horizon-limiting-data-to-1-year

summary from lee for paths forward

hosting our own horizon, we tried that before, didn't work. At this point that is over 40TB of data and you really need nVME storage (or a clustered setup, not sure about that). Anyway we're talking at least 10K worth of current gen server hardware, not to mention the time to manage it.
rent access to a full archive node from an operator. This incurs a monthly cost most likely, but we just need to adapt our existing codebases to use that horizon instance, so thats almost no work
rewrite our codebases which do historic queries to use stellars bigtable data lake which contains the historic data. We'd need an account to fund our queries (all queries are paid), and of course there is the cost of rewriting our code

## Description Stellar changes to 1year only of historic data https://stellar.org/blog/foundation-news/sdf-s-horizon-limiting-data-to-1-year ## summary from lee for paths forward - hosting our own horizon, we tried that before, didn't work. At this point that is over 40TB of data and you really need nVME storage (or a clustered setup, not sure about that). Anyway we're talking at least 10K worth of current gen server hardware, not to mention the time to manage it. - rent access to a full archive node from an operator. This incurs a monthly cost most likely, but we just need to adapt our existing codebases to use that horizon instance, so thats almost no work - rewrite our codebases which do historic queries to use stellars bigtable data lake which contains the historic data. We'd need an account to fund our queries (all queries are paid), and of course there is the cost of rewriting our code

thabeta added the

Issue

label 2025-01-09 14:57:39 +00:00

thabeta added this to the tfgrid_3_16 project 2025-01-09 14:57:42 +00:00

despiegk was assigned by thabeta

2025-01-09 14:58:00 +00:00

thabeta commented

2025-01-13 15:04:31 +00:00

also on effect on the bridges

quoting Lee

we maintain a cursor in a local file. The problem is, if this cursor file is gone, we start from scratch again. Its mainly a problem for transactions going back to stellar (so withdraws from other networks). We use the memo field to encode transaction hashes and check before we send the tft back if there is no tx on stellar with that memo. So if now we start from scratch because the bridge persistence file is modified, for instance, then all bsc/eth (and in the future solana) txes which have been done more than a year ago would be done again, becuase the bridge doesn't find the withdraw transaction and thus thinks its missed it and it hasn't happened yet

also on effect on the bridges quoting Lee ``` we maintain a cursor in a local file. The problem is, if this cursor file is gone, we start from scratch again. Its mainly a problem for transactions going back to stellar (so withdraws from other networks). We use the memo field to encode transaction hashes and check before we send the tft back if there is no tx on stellar with that memo. So if now we start from scratch because the bridge persistence file is modified, for instance, then all bsc/eth (and in the future solana) txes which have been done more than a year ago would be done again, becuase the bridge doesn't find the withdraw transaction and thus thinks its missed it and it hasn't happened yet ```

Sign in to join this conversation.