Lessons from Cloud Data Projects

A project that one of our team members reviewed involved building complex algorithms to personalise daily offers for retail customers.

This is a common scenario, where entry-level cloud services are cost-effective but limited. Typically, advanced compute and storage options are priced at a premium, while being harder to predict. This uncertainty arises from the wide range of service configurations that are possible, where a small change can significantly increase fees.

Some organisations tackle this issue by limiting options. In the above scenario, this was implemented through “templates” that bundled services for specific purposes - such as data cleaning, insights generation or rapid prototyping – with predictable costs. When a team member required cloud services for one of these purposes, the cloud administration team activated them using the relevant template, subject to approval of the cost.

Another best practice is to incentivise responsible use through reporting. Several organisations have put in place cost-monitoring and cross-charging mechanisms. These allow cloud administrators to apportion costs to each team and provide transparency on usage

On another personalisation project, the team could measure changes in customer spend but also wanted qualitative feedback regarding the perceived quality and relevance of content. Several people working on the project volunteered as test subjects, giving permission to use their loyalty and transaction data, while believing that their information would be protected.

Although the cloud services used by the project complied with the organisation’s information security policy, this did not adequately control the use of data. Team members could copy data across storage areas inside the cloud platform. They also combined data from multiple sources, making it difficult to understand provenance and control usage.

To deal with this issue, some organisations are adding data-centric security controls to their information security policies, implemented through cloud “data zones”. A data zone provides granular control over data stores, even at the level of individual data fields, automatically enforcing rules on access and replication. For example, storing Australian personally identifiable information (PII) in a data zone that only allows employees of the organisation’s Australian entity to see PII fields and triggers an approval workflow when the team would like to use personal information for personalisation applications.

In reviewing cloud platform implementations, we have observed some organisations excluding the ingestion of data from their scope. As a result, they have delivered “empty shells” of technology for future initiatives to work with. We believe that integration with source data systems would improve the adoption rate of cloud platforms once implemented.

The most common argument in support of separating the build of a cloud platform from the ingestion of data is that the data required by each project is different. Our rebuttal is that, although each initiative may require data in specific formats, this data is likely to originate from the same sources. Typically, these are enterprise systems that support core business processes, such as the Customer Relationship Management (CRM) platform.

We recommend that cloud data platform projects build interfaces to these systems as well as setting up cloud infrastructure. This would save future initiatives the effort and time of profiling data sources, obtaining approvals to connect to them and re-implementing interfaces that could be built once and used across projects. As part of this, we suggest maintaining a “data catalogue” to provide common definitions for, and visibility of, data already ingested into the cloud.

In addition, organisations undergoing constant change may wish to establish a dedicated data integration team to support ingestion of new data into the cloud. In our experience, identifying and profiling data sources can represent 40 to 60 percent of the expense in a typical data project. A data integration team that existed outside of individual projects could be tasked with maintaining a knowledge base of data sources and integrations, significantly reducing this expense.

We believe that organisations undertaking cloud data projects should consider automating cost monitoring, data privacy controls and integrations with key systems. With these capabilities in place, they will be able to make effective and efficient use of cloud services – typically taking months out of the development time for digital assets.