3 steps to improve Test Data Management. Download Gartner's report.
The concept of data retention emerges at a time when data architectures are developing to become sophisticated ecosystems that promote full visibility and control over data.
Data retention policies refer to the procedures that govern how long data is kept in a database. For what purpose and what techniques are applied for deletion at the end of the data lifecycle. As such, it represents a crucial part of compliance under regulations such as the GDPR or HIPAA which, among other things, regulate data deletion and archiving.
In this context, organizations must balance the need to retain data for business operations and legal requirements. At the same time, they must comply with privacy obligations that prescribe certain data deletion requirements, all without sacrificing efficiency.
Achieving such an equilibrium involves designing advanced data architectures where true information governance is a continuous practice. In the case of data retention in particular, systems must be put into place that classify, archive, and delete data according to retention policies and relevant legal norms.
In this article, we take a look at the legal framework regulating data retention, including the GDPR retention period. We also explore key aspects for building data architectures that support efficient data retention and deletion through the use of the right tools and technical design.
The term data retention architecture refers to a framework capable of supporting the archiving and safe deletion of data when the end of the data lifecycle is reached.
This framework goes beyond the technical, incorporating data retention policies and systems that ensure the right procedures are executed.
Across the full data lifecycle, from creation to deletion, organizations must ensure retention policies are taken into account. A process that can be divided in the following stages:
Across all these stages, data retention is thus closely tied to the entire lifecycle of data: in every step, compliance requirements related to retention must be observed and data managed systematically to ensure consistent results.
For deeper insights, you can explore our guide on personal data processing in development environments.
In this process, two key concepts emerge that must be paid close attention to when it comes to the data lifecycle:
Together, these two concepts put transparency at the forefront of data retention across the full lifecycle. On top of it all, the right data governance strategy can establish policies as well as quality and security standards. This allows the design of a consistent architecture where all these concepts and procedures are fully integrated.
The principle of storage limitation is described in Article 5 of the GDPR. More specifically, this article states that data should be “kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which the personal data are processed.”
As such, when it comes to understanding the GDPR retention period, it’s important to note this norm doesn’t prescribe a specific time limit for retaining personal data: it’s up for each organization to determine what the appropriate period is according to the type of data, the purpose it has been collected for and potential industry-relevant regulations, all while attending to the principle of data not being stored for longer than necessary.
Meanwhile, Recital 39 insists that organizations should ensure that “time limits should be established by the controller for erasure or for a periodic review.”
This clause highlights the need to generate data retention guidelines that establish the justification for data collection and the retention periods to be applied. It also covers potential industry requirements that may justify this timeline.
Other GDPR articles and principles work together to generate a regulatory ecosystem around data retention and the data lifecycle. This includes the principles of data minimization, purpose limitation, data accuracy, and data subject rights (such as the “Right to be forgotten” in article 17).
Meanwhile, for organizations to present evidence of compliance, consistent documentation around data retention must be ensured. This includes, among others, documents related to:
Because of its particular nature, data retention around medical information is specifically regulated across different countries. HIPAA is a U.S. federal law that regulates data privacy around Protected Health Information and, among other aspects, it establishes how long medical data should be retained, and what security measures are to be implemented.
HIPAA does establish minimum retention periods for certain types of documents, including a six-year period for documentation of policies and procedures designed to comply with HIPAA from the date of creation or the date when it was last in effect, whichever is later.
Generating data retention policies should start by identifying and classifying the types of data held by the organization, in order to determine which data is sensitive or has potential to be regulated. This first classification also guides data retention timelines as well as the specific measures to be implemented for protection.
With visibility across all data in their possession, the organization must review what laws and regulations are applicable considering the industry it belongs to and the country it operates in. This might lead to requirements to comply with the European GDPR, the UK data privacy rules or US data protection rules, among other options.
A third step to guarantee compliant data retention policies is to establish roles and responsibilities when it comes to data governance and privacy. This includes data controllers and DPOs, whose clear responsibilities must be described regarding data retention, among other issues.
Automation and control procedures must be defined that enforce retention rules, from data archiving to data deletion. This reduces human error and minimizes the chances of compliance risks, with software being able to schedule data retention procedures including deletion. At the same time, control mechanisms include those for data access, but also having the possibility of initiating manual procedures for responding to user requests requiring the deletion of their data.
Data retention policies should be audited regularly to guarantee compliance and allow for opportunities for continuous improvement.
Retention without access is yet another option with its own set of pros and cons: while the formula offers the possibility to retain data for regulation purposes out of active systems, organizations are not able to access it easily in case it’s needed for audits or legal needs.
Cloud providers like AWP, Azure and GCP offer automated retention capacities, including:
| AWS | Azure | GCP | |
| Features | Transitions data automatically between storage classes and offers the possibility for automated deletion based on timelines | Presents “hot”, “cold” and “archive” categories for data and offers automated lifecycle management | Offers Cloud Storage Lifecycle Management that automatically transitions data to archive-like categories based on rules |
These automated management options are guided by configuration options such as the generation of buckets and blobs inside containers, for which general rules and policies around permissions or retention policies can be configured.
As the importance of solid data retention policies is more and more understood, technologies are emerging to help organizations comply. Such is the case of AI and automation technologies, which allow for the automatic detection of expired data, thus reducing human errors and streamlining operations around data retention.
At the same time, these tools can be integrated with platforms such as icaria Data Privacy, generating a consistent ecosystem across databases and cloud storage.
As a result, it’s possible to achieve traceability and consistency in enforcing data retention, and solid data architectures where scalability across multiple storage platforms is achieved.
Data retention policies of data storage promote organizations’ procedures to eliminate unnecessary data and thus reduce costs related to storage, such as the need for specialized hardware and cloud costs.
Examples of organizations sanctioned for wrong data retention practices are useful to understand the importance of paying close attention to this issue. For instance, along the list of the sanctioned organizations by the French data protection authority CNIL is a penalty of 100,000 euros for the PAP website “for failing to comply with its obligations in terms of data retention periods and data security”, or a distance learning apprentice training centre, which was fined with 10,000 euros for wrong data retention, among other reasons.
Clear, compliant data retention policies are a straightforward measure to protect organizations against sanctions and lawsuits, but also against the reputational crisis related to not following data privacy policies. Today, keeping data for unnecessarily long periods of time can have important reputational consequences, including a higher risk in case of breaches and a perception of negligence by the public. On the contrary, a company capable of demonstrating responsible data management is in a better position to build trust with the public as well as to respond to potential regulatory inquiries.
As seen across the article, a well-built data retention architecture is capable of mitigating risks, guaranteeing compliance and improving efficiency across an organization’s processes.
When well designed, data retention doesn’t merely engage technical aspects: it must also be rooted in cultural and policy aspects as well as automation.
In this quest, tools like icaria Data Privacy are becoming key allies for facilitating total data control across all business operations.
Want to learn more about icaria Technology and the software solutions promoting control, traceability and automation in data retention towards better data governance? Get in touch with us and speak to our team about how we can help you.
