The ability to analyze data and draw insights from it is crucial for business success today. The prerequisite for this is a solid and agile data warehouse (DWH). However, in many companies, the central data platform is already outdated and can no longer meet current requirements.

If specialist departments have to wait days for critical reports and work with outdated data because performance or stability leave something to be desired, this is not acceptable. Companies are now faced with the question of whether it is better to modernize the existing DWH or build a completely new one. There are both technical and strategic aspects to consider. First of all, you should check whether your data warehouse is still up to date and meets the requirements of the specialist departments. If there are frequent complaints that BI reports are taking too long, this indicates a lack of performance. This can be due to outdated hardware as well as problems with the database or data modeling.

Even if there are stability problems, this hinders day-to-day work. This is because the data warehouse must be continuously fed with new information so that specialist departments can access the latest figures. If the reloading process stops, no up-to-date BI reports can be created. Poor performance and instability are the most common reasons why a data warehouse needs to be modernized.

Less obvious, but nevertheless serious, are problems with data integrity and historization. This is because they lead to falsified results. Have you ever noticed that data in the data warehouse does not match the data in the source systems? If, for example, the total turnover displayed in the DWH differs from the aggregated individual postings in the financial accounting system, discount deductions or bonuses may not have been mapped correctly. Data integrity problems are often related to the fact that key figures and processes in the data warehouse are inadequately documented. Particularly in the case of complex structures that have developed over the years, it is then no longer possible to understand how a key figure actually came about.

Problems with historization occur when master data changes but no time reference is noted in the data warehouse. This leads to historical data being incorrectly assigned. For example, if a customer moves from Cologne to Berlin, then suddenly all of their previously recorded actions apply to the new Berlin address, even though they took place in Cologne. Errors in historization are always related to data modelling. To prevent this from happening, those responsible must create the appropriate attributes for the master data in the data warehouse and also take the time reference into account.

Excessive complexity as a knock-out criterion

One major problem with an old data warehouse is the complex structures that have grown over the years. They affect performance, stability and data integrity as well as future development opportunities. When the data warehouse was set up at the time, it met current requirements. Over the years, however, new requirements have continuously been added. As a result, the DWH has been constantly expanded and adapted. However, just like a house once it has been built, you cannot tear down the walls of a data warehouse at will. Sometimes complicated workarounds are required to integrate a new key figure into the existing structure. This takes time, and specialist departments may have to wait a long time until their new requirement is implemented.

With every change, the structure also becomes more complex and confusing. Sometimes data is also stored twice because it cannot be reused. This makes it difficult to understand which data is used for which calculation. If this data is also not in the same state, the results will differ. If a data warehouse has become very complex and confusing, it may make more sense to completely rebuild it rather than modernize it. In principle, a data warehouse should be structured in such a way that it can be expanded and is clearly documented. All the data you need should always be in the same place. If these requirements are met, future requirements of the specialist departments can also be integrated without great effort. In addition, it is then possible to set up or replace BI front-end tools on the data warehouse as required, so that specialist departments can create reports independently at the click of a mouse. This saves time and takes the pressure off the IT department.

Future viability of the DWH depends on strategic issues

Whether the data warehouse is fit for the future also depends on strategic questions. Where does your company want to develop in the coming years? Anyone planning IoT projects needs the ability to store and process huge amounts of data from different sources and in different formats - including both structured data and unstructured data such as text documents or videos. This may require connecting the data warehouse with a data lake and using tools that access both data platforms.

Anyone planning to change their business model must also take this into account in the data warehouse. For example, if you no longer want to sell machines in future, but offer them for rent "as a service", you will need to collect completely different data and map it in the DWH - for example, information on where a machine is currently located, how many hours it has been in operation at the customer's premises, what condition it is in and when it needs to be serviced.

Data warehouse must be a matter for the boss

The question of future requirements in particular shows that the data warehouse is not just an IT issue, but is closely linked to the corporate strategy. For this reason, a member of management should always be involved when it comes to the future of the data platform. It is worth bringing in the expertise of an experienced consultant to provide support. They will determine current and future requirements in joint workshops and interviews. A thorough analysis of the current and target status will make it clear whether it is worth modernizing the existing data warehouse or whether it is better to build a new one from scratch.

Martin Clement, Senior Solution Architect Analytics & Data at Axians IT Solutions / ag