Really weird thing happened on Dec, 6th, 2021, between 4 and 7 AM UTC. Somehow miners generate blocks which failed validation (except of few ones generated with no transactions and finished in the widely recognized blockchain).
In short, the issue was about a transaction which passed checks during block generation but then failed verification on other nodes with an error during AOT costing, the most complex part of the ErgoTree interpreter. Unfortunately, logs did not contain full information to replay the issue with confidence. After some time the tx got into a block which passed validation.
While we’re still doing investigation, we recommend miners to update to a minor release 4.0.16.1 Release Ergo Protocol Reference Client 4.0.16.1 · ergoplatform/ergo · GitHub , which tries to use interpreter during candidate generation in the same way as during block verification, also fixing possible context inconsistency issues. This version also dumping execution context into logs when transaction verification fails, to make verification easily reproducible with logs given.
However, complexity of interpreter with AOT costing is a problem per se, thus we will propose 5.0 soft-fork introducing simplified interpreter with JIT costing soon. Another long-term direction to follow is to incentivize developing alternative block assemblers.
