At OM2, database health was part of the normal rhythm of running the trading platform. We tracked growth and performance over time, and during one of those reviews the architects raised a concern about a core transaction table that had grown beyond 1.6 billion records.
A table that large clearly deserved attention, but the number itself did not explain the problem. I took the product lead on the investigation alongside Architecture, working closely with our DBA and engineering teams to understand what was driving the growth, what pressure it was creating, and what we could safely change without causing a different problem elsewhere.
The number was only the first clue
The investigation showed that the growth was not evenly distributed. Almost half of the table consisted of what we called house transactions. For many external transactions, the platform also created a corresponding internal transaction. The behavior had existed for years and had become part of the ordinary shape of the data.
At that scale, a legacy convention had accumulated into hundreds of millions of records.
The natural next step was to ask whether we could stop creating them and remove the history that had already accumulated. Answering that question took more work than identifying the opportunity, because the records had been around long enough that their original purpose was only part of what mattered.
What had the platform learned to depend on?
Production systems accumulate dependencies gradually. Reporting starts reading a field because it is available. An operational process builds a query around a familiar record shape. Customer-specific logic grows around existing data. Code that has not been touched for years keeps working because the assumptions underneath it remain undisturbed.
That was the main concern in this case. Before we changed the write path or removed historical data, we needed to determine whether the house transactions were genuinely redundant in the current platform or had quietly become part of something else.
Working with Architecture and the DBA, we traced where the transactions were written and where they were consumed. The review extended beyond the core database into reporting, support and quantitative-analysis flows, monitoring, customer-specific behavior and other dependent tables. Some of those paths were old enough that recent team memory was not a reliable guide, so we had to follow the behavior through the system and establish what would actually happen if the records stopped appearing.
Once we were satisfied that removing the house transactions would preserve data integrity and would not break dependent flows, the nature of the work changed. We could move from investigation into a controlled migration.
The implementation separated future behavior from historical cleanup. We first changed the write path so unnecessary house transactions would stop accumulating. The existing records could then be removed without the table continuing to refill with the same pattern.
The cleanup itself was handled as a production migration. The plan included backups, controlled handling of indexes and foreign keys, a defined deletion procedure, dedicated QA and a rollback path. That discipline mattered because of the size of the dataset, but also because the age of the behavior increased the likelihood of dependencies that were no longer obvious.
Both parts were completed. The historical house transactions were cleaned out, and the redundant write behavior was stopped. In practical terms, we removed roughly half of a 1.6-billion-record table and reduced a source of load that had already become meaningful for platform performance.
From immediate relief to the next constraint
The cleanup gave us useful headroom, and stopping the redundant writes changed the rate at which the table would continue to grow. It also made the next structural constraint easier to see.
The transaction model still depended on an identifier range with a finite ceiling. Continued platform growth would eventually run into that limit. In the following planning increment, I continued to lead the scalability work around moving the relevant transaction identifiers and database structures to BIGINT, expanding the numeric range the platform could support and removing that capacity constraint.
That initiative was implemented as well.
The cleanup addressed scale that had already accumulated inside the platform. Changing the write behavior prevented the same legacy pattern from continuing to consume capacity. The BIGINT migration removed a separate structural limit that would otherwise have become increasingly important as transaction volume kept growing.
What stayed with me from the work was how much an implementation choice can change simply by surviving long enough.
By the time we examined the house transactions, the question was no longer whether the original behavior still made sense in isolation. Years of production use had built data, dependencies and operating expectations around it. Changing it safely meant understanding which of those relationships still mattered, removing what had genuinely become unnecessary, and then carrying the scalability work far enough that the platform was not simply waiting for the next constraint to arrive.
That is the part of platform work that often matters most: the system you are changing is not just the architecture that was originally designed. It is also everything that has learned to depend on it since.