feat: Add code generator for transactions and ledger entries

[a1q123456]

Quick note for reviewers: files under protocol_autogen/transactions and protocol_autogen/ledger_entries are auto-generated. I don't think it's worth putting so much effort on those files unless you'd like to check if they're identical after re-generation, or if you want to see how those things look like.

High Level Overview of Change

This PR adds code generators to generate type wrappers for transactions and ledger entries. The generators are python scripts and we generate code during configuration. This approach provides 2 benefits:

1. It's friendly to IDEs and humans
2. We always get the latest generated code after we modify transactions.macro, ledger_entries.macro, sfields.macro, or the template files.
3. First time when you build the project before the venv is created, you'll need network connection, but once you have your venv, cmake will only give you a warning if you have no network connection.
4. You can customise the venv by setting the build option CODEGEN_VENV_DIR

How we'll use the wrappers

Building a transaction

auto ammClawBack = AMMClawbackBuilder{bob, alice, XRP, USD}
    .setHolder(bobAccountId)
    .setAmount(XRP(10))
    .build();

Wrapping a transaction

auto ammCreate = AMMCreate{<your STTx>};

ammCreate.getAccount();
// ...

Building a ledger entry

auto amm = AMMBuilder{bobAccount, XRP(10), XRP, IOU, ownerNode}
    .setPreviousTxnID(txnId)
    .build();

Modifying a ledger entry

auto ammNew = AMMBuilder{amm}
    .setOwnerNode(newOwnerNode)
    .build();

Wrapping a ledger entry

auto amm = AMM{view.read(keylet::amm(...))};

amm.getAsset();
// ...

Context of Change

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (non-breaking change that only restructures code)
• Performance (increase or change in throughput and/or latency)
• Tests (you added tests for code that already exists, or your new feature included in this PR)
• Documentation update
• Chore (no impact to binary, e.g. .gitignore, formatting, dropping support for older tooling)
• Release

API Impact

• Public API: New feature (new methods and/or new fields)
• Public API: Breaking change (in general, breaking changes should only impact the next api_version)
• libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
• Peer protocol change (must be backward compatible or bump the peer protocol version)

Future Tasks

1. We should continue making SF_ARRAY typed so that we can return STArrayProxy instead of STArray const&
2. I believe we can make Keylet typed so that we'll retrieve a typed ledger entry wrapper from view.read()

Q&A

Q1: Why add Python as a build dependency?

A: Python is already used elsewhere in the project (e.g., rippled-workload). The code generation runs at configure-time and takes ~1 second. Generated files are committed to version control, so builds work offline — Python is only needed when modifying .macro files or templates.

Q2: Why configure-time generation instead of build-time or a manual step?

A: Configure-time ensures generated headers exist before IDE indexing occurs, providing proper code completion and navigation. A pre-commit hook runs too late (after editing), and a manual step risks stale files.

Q3: What will consume these generated wrappers?

A:

A new JTx test framework (next PR) using builders instead of JSON/functors
Future: typed Keylets, typed transactor wrappers
Example of the new pattern:

auto payment = PaymentBuilder{accountId, destination, amount}
    .setFlags(...)
    .sign(pk, sk)
    .build();
env.submit(payment);

Q4: Why not use X-macros instead of Python?

A: An X-macro approach was attempted but rejected due to complexity. Mapping sfAccount => SF_ACCOUNT::type::value_type and handling optional vs required fields required extensive template metaprogramming (partial specialisations, SFINAE). The Python approach is significantly more readable and maintainable.

Q5: Why not parse TxFormats.cpp directly instead of maintaining .macro files?

A: The .macro files ARE the source of truth. TxFormats.cpp includes them via #include transactions.macro"`. Parsing C++ would require libclang and correct build parameters, adding significant complexity.

Q6: Why Mako over Jinja2?

A: Because we use it in ripple-workload already, we do not have to learn another template syntax. Both are functionally equivalent for this use case; Mako allows more natural Python embedding if needed.

Q7: What happens if code generation fails?

A: The build fails with FATAL_ERROR. There's no silent fallback to stale files, explicit failure is preferred over hidden staleness.

Q8: Why are generated files committed to version control?

A:

Allows builds without Python/network access
Provides visibility into what changed when .macro files are modified
Acts as a cache for faster CI
IDE can index files immediately after clone

Q9: Are there tests for the code generator?

A: Yes, we have unit test templates and auto-generated unit tests.

Q10: What's the timeline for consumer code?

A: The new JTx PR (using these wrappers) is the next focus after this PR is merged.

Q11: Is there any existing tool that can do this?

A: Not really, lots of the code generation tools want to take over the serialisation part, and we're doing it ourselves. e.g. protoc could be a candidate, but it comes with a couple of problems unless we overhaul the system completely:

1. Serialisation and deserialisation like I mentioned previously. It's mainly for defining structures and hassle-free binary serialisation and deserialisation.
2. Hard to integrate with the internal type system. i.e. When we say there's an sfAccount field, we imply that the type is SF_ACCOUNT and the value type is SF_ACCOUNT::type::value_type. It's currently impossible to do it without a plugin in protoc.
3. Apart from typed wrappers, we also want to generate some definitions of transactions and ledger entries. However, protoc currently only generates structs and it doesn't generate anything else.

Q12: How do we add a new code generator for another macro file?

A: As most of the functionality is implemented in macro_parser_common.py (like cleaning up the macro file to remove #define and #include), we can do it fairly simply.

1. Copy an existing generator (like scripts/generate_tx_classes.py)
2. Modify the parser keyword and how we parse the parameters
3. Write a new template file
4. Call the new generator in cmake/XrplProtocolAutogen.cmake

[mvadari]

Is there a good strategy for adding new non-autogen functions to these objects? e.g for #6408

[a1q123456]

Is there a good strategy for adding new non-autogen functions to these objects? e.g for #6408

I think as the first step, we can use those typed ledger entries to build those views.

If we want to polish it a bit further, I think it'd be nice to inherit from those ledger entries to provide functionalities. i.e. anything under protocol_autogen is purely generated code and it doesn't come with any functionalities, and things in xrpl/ledger are more transactor-facing, and then, we'll end up with things like protocol_autogen::RippleState and class ledger::RippleState : public protocol_autogen::RippleState so that developers will see both getHighLimit() (from protocol_autogen::RippleState) and getMyLimit() (from ledger::RippleState).

[mvadari]

Is there a good strategy for adding new non-autogen functions to these objects? e.g for #6408

I think as the first step, we can use those typed ledger entries to build those views.

If we want to polish it a bit further, I think it'd be nice to inherit from those ledger entries to provide functionalities. i.e. anything under protocol_autogen is purely generated code and it doesn't come with any functionalities, and things in xrpl/ledger are more transactor-facing, and then, we'll end up with things like protocol_autogen::RippleState and class ledger::RippleState : public protocol_autogen::RippleState so that developers will see both getHighLimit() (from protocol_autogen::RippleState) and getMyLimit() (from ledger::RippleState).

Ok yeah I like that solution.

[codecov]

Codecov Report

❌ Patch coverage is 99.97568% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.4%. Comparing base (57e4cbb) to head (9306692).
⚠️ Report is 4 commits behind head on develop.

Files with missing lines	Patch %	Lines
include/xrpl/protocol_autogen/LedgerEntryBase.h	94.1%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #6443     +/-   ##
=========================================
+ Coverage     79.7%   81.4%   +1.7%     
=========================================
  Files          878     988    +110     
  Lines        68116   74411   +6295     
  Branches      7563    7559      -4     
=========================================
+ Hits         54300   60599   +6299     
+ Misses       13816   13812      -4     

Files with missing lines	Coverage Δ
...ude/xrpl/protocol_autogen/LedgerEntryBuilderBase.h	100.0% <100.0%> (ø)
include/xrpl/protocol_autogen/STObjectValidation.h	100.0% <100.0%> (ø)
include/xrpl/protocol_autogen/TransactionBase.h	100.0% <100.0%> (ø)
...ude/xrpl/protocol_autogen/TransactionBuilderBase.h	100.0% <100.0%> (ø)
include/xrpl/protocol_autogen/ledger_entries/AMM.h	100.0% <100.0%> (ø)
...xrpl/protocol_autogen/ledger_entries/AccountRoot.h	100.0% <100.0%> (ø)
.../xrpl/protocol_autogen/ledger_entries/Amendments.h	100.0% <100.0%> (ø)
...lude/xrpl/protocol_autogen/ledger_entries/Bridge.h	100.0% <100.0%> (ø)
...clude/xrpl/protocol_autogen/ledger_entries/Check.h	100.0% <100.0%> (ø)
.../xrpl/protocol_autogen/ledger_entries/Credential.h	100.0% <100.0%> (ø)
... and 100 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kuznetsss]

How does cmake understand that it doesn't need to run this each time? Shouldn't we use add_custom_command() here instead because it allows to specify outputs and mark the process as code generation?

[a1q123456]

We currently are only able to regenerate when any of the dependent files change.

[kuznetsss]

Could you please explain where we set what cmake should check to decide whether to call this execute_process() or not?

[a1q123456]

execute_process always runs but cmake won't reconfigure automatically if the files on this list get modified.

set_property(
        DIRECTORY
        APPEND
        PROPERTY
            CMAKE_CONFIGURE_DEPENDS
                "${TRANSACTIONS_MACRO}"
                "${LEDGER_ENTRIES_MACRO}"
                "${SFIELDS_MACRO}"
                "${GENERATE_TX_SCRIPT}"
                "${GENERATE_LEDGER_SCRIPT}"
                "${SCRIPTS_DIR}/macro_parser_common.py"
                "${SCRIPTS_DIR}/templates/Transaction.h.mako"
                "${SCRIPTS_DIR}/templates/TransactionTests.cpp.mako"
                "${SCRIPTS_DIR}/templates/LedgerEntry.h.mako"
                "${SCRIPTS_DIR}/templates/LedgerEntryTests.cpp.mako"
                "${REQUIREMENTS_FILE}"
    )

[kuznetsss]

Does it mean that if I clean build directory and call cmake it will always overwrite existing generated files?

[a1q123456]

yeah, it currently always runs the code generator when we configure the project, the code above only ensures that cmake will automatically reconfigure the project when those files get modified (it works the same as how it works after we add or remove a source file - it automatically reconfigures)

[kuznetsss]

LGTM 🚀