GDPR What you need to know about Consent

GDPR – What you need to know about Consent

Who should read this?

This post is written for Data Controllers and anyone who needs to understand what the obligations of the Data Controller under GDPR are for obtaining consent. We’re talking about customers here, but the same obligations apply to any other personal data that you hold.

What do you need to know?

Over the last several years, data-driven marketing has been increasingly adopted as companies strive to make more effective use of their customer data. However the advent of GDPR has led many to question whether this trend can continue. The main problem is the increased burden for obtaining consent from the customer that GDPR places on you, if you are the data controller. Before we look at this in more detail though, let’s examine when consent must be explicit and when it need not be.

Although GDPR makes several mentions of “explicit consent” rather than just “consent”, it does not define what it means by “explicit”. Explicit consent is required for automated decision-making including profiling e.g. for the purpose of deciding whether to extend an offer. If you’re relying on consent (rather than one of the other provisions of the GDPR) for processing of sensitive personal data or transferring personal data outside of the EEA, then it must be explicit.

You don’t need explicit consent if, for example:

  • You need to use the customer data to fulfil your obligations under a contract
  • It’s required by law
  • It’s in the customer’s vital interests
  • It’s in your legitimate interests, except where such interests are overridden by the interests, rights or freedoms of the customer.

Under the current Data Protection Act, consent must be freely given, specific and informed but it can be implied. As now, under the GDPR, sensitive personal data requires explicit consent. Under the GDPR, consent must also be unambiguous and you must be able to demonstrate that consent was given. It must be clearly distinguishable, intelligible and easily accessible and the customer must be informed of their right to withdraw consent – it needs to be as easy to withdraw consent as it is to give consent. There is also (under GDPR) the much discussed “Right to be forgotten” – the right to request the deletion or removal of personal data where there is no compelling reason for its continued processing. When a customer withdraws consent, they may be more likely to exercise the right to be forgotten, so systems need to be designed with this in mind.

Under GDPR it’s no longer acceptable to bundle permission into the terms and conditions governing supply, they must be unbundled from the contract. Each aspect for which consent is being sought also has to be explained in granular detail. To be “freely given”, consent should not be required if it is not necessary for the performance of the contract, which could affect some providers of free web services – as discussed in more detail in this post.

Do I need consent for Direct Marketing?

The legitimate interest test needs some clarification. GDPR includes an explicit mention of Direct Marketing as a legitimate interest, but as Steve Henderson points out in this interesting post on the DMA website, this must be seen in the context of how you come to obtain that data in the first place, as well as how you’re going to use it. For example if you have obtained the contact details electronically then you continue to be bound by the Privacy Electronic Communications Regulations (PECR) and the EU E-Privacy Directive, irrespective of the legitimate interest test. Obviously if the personal data is used for direct mail and the data has been obtained in store or via a coupon, the legitimate interest test is applicable.

How is it going to work in the UK?

Now I want to look at how GDPR may be enshrined in UK regulations. There has been a lot of reaction from the industry to the Information Commissioner’s Office consultation on its GDPR consent guidance; it’s good to see TechUK representing these views to the Department for Culture Media and Sport (DCMS) robustly. The draft guidance states that consent requires a positive opt-in and that pre-ticked boxes or any other method of consent by default should not be used. However, the guidance also recognises that other opt-in methods might include signing a consent statement, oral confirmation, a binary choice presented with equal prominence, or switching a setting away from the default.

In the draft guidance there is a tendency to treat opt-in consent and “explicit consent” as the same, which GDPR itself does not. One potential issue which could arise is the effect that requiring opt-in consent for non-sensitive personal data could have on suppression processing: our comment to TechUK (which it included in its response to DCMS) was:

“One example of where an over-reliance on opt-in consent for non-sensitive personal data could have unintended consequences is where customers have moved to a new address. Currently, there are a handful of providers of lists of people who have moved which are used by many companies, either to suppress correspondence to that person at the old address, or (if the new address is available and it’s for a permitted purpose) to update correspondence records to the new address. The information for these lists is typically provided by financial services companies, who will not provide the data if they believe that it may be contrary to GDPR. Without the new address, suppliers would not know that their customer has moved and would be unable to contact them for permission to send correspondence – it would rely on the customer notifying each company that they have done business with of their new address. An inevitable result therefore of requiring an opt-in consent for non-sensitive personal data would be more mail for the old occupant sent to the new occupant at the customer’s old address.”

The bottom line

Although it’s easy to get bogged down in the detail of what you need to do regarding consent and to look on its tightening up by GDPR as a negative, DQM GRC research this year shows that an increasing proportion of consumers are ready to share their personal data if it’s properly explained why they should do so and the benefit to them: “two-thirds of consumers said they are willing to share their personal information in the right circumstances – a positive shift since 2016 when only half said this was the case”. The fundamental truth is still the same: an engaged customer who feels that they are in charge of the relationship is more likely to be a valuable customer.

Further reading

Some more information, including tips on how to handle the subject of consent, is in this article from Kemp Little.

In the next post, we’ll look at the differing obligations of Data Controllers and Data Processors.

GDPR Key Definitions

GDPR Key Definitions and Terminology

This week, as part of our series on GDPR, we are looking at the key definitions in the EU General Data Protection Regulation to help you decide which of the obligations of GDPR do indeed apply to your business.

Data Controller

Like the existing Data Protection Act (DPA), the GDPR applies to Data Controllers who process personal data. So first, who is the Data Controller? This is a person who decides the purpose for which any personal data is to be processed and the way in which it is to be processed. This can be decided by one person alone or jointly with other people.

Data Processor

Unlike the DPA, the GDPR introduces specific responsibilities for the Data Processor. These are third parties that process data on behalf of the Data Controller and includes IT service providers (many of which are among our clients). In a later post, we’ll look at the specific responsibilities of Data Processors, especially when processing is subcontracted to other Data Processors.

By the way, an employee of a company which decides what and how personal data is to be processed is a Data Controller, not a Data Processor.

Personal Data

The GDPR has a broader definition of what constitutes personal data than the DPA, by incorporating reference to identifiers such as name, identification numbers, IP address and location. Each person to which the personal data refers is known as a Data Subject.

Sensitive Personal Data

Again, the GDPR definition of sensitive personal data is slightly broader than under the DPA. The main addition is biometric data, for the purposes of uniquely identifying a person. Actually, the GDPR talks about a special category of personal data rather than sensitive personal data but the definition is almost the same. The table below illustrates what is sensitive and what isn’t, and what isn’t personal data – but as this excellent article discussing the subject suggests, if you are asking yourself if it’s personal data or not, why not err on the side of caution and treat it as if it is?

What is personal or sensitive data?

Right to be forgotten

The right to erasure of personal data or ‘the right to be forgotten’ enables an individual to request the deletion or removal of personal data whether there is no compelling reason for its continued processing. We’ll talk more about this in the next post when we discuss consent.

Data Protection Officer

A Data Protection Officer is someone who is given formal responsibility for data protection compliance within a business. Not every business will need to appoint a data protection officer – you need to do so if:

  • Your organisation is a public authority; or
  • You carry out large scale systematic monitoring of individuals (for example, online behaviour tracking); or
  • You carry out large scale processing of special categories of data or data relating to criminal convictions and offences.

Data Protection Authority

The Data Protection Authority in the UK will still be the UK Information Commissioner, who is tasked by the EU with the monitoring and enforcement of the GDPR within the UK. The European Data Protection Board is the “super regulator” consisting of the heads of each national supervisory authority. The Queen’s Speech last week announced a new Data Protection Bill to remove any doubt that the UK will implement GDPR so that it continues to be in force after Brexit takes effect.


Derogation, meaning an exemption from the regulations, is something under active discussion within the DCMS at the moment (Department for Culture Media and Sport, the relevant government department). I’m a member of the GDPR working group of TechUK, the software and IT services industry body whose views DCMS seeks while drafting guidelines and derogations for GDPR.

Adequacy and Transfer of Data outside the EEA

If the UK leaves not only the EU but also the EEA, crucially the GDPR allows transferring data outside the EEA to any country or territory in respect of which the Commission has made a “positive finding of adequacy” i.e. is the transfer to a country on the EU Commission’s list of countries or territories providing adequate protection for the rights and freedoms of data subjects in connection with the processing of their personal data? Achieving this “positive finding of adequacy” is one of the main aims of the consultation that DCMS is holding at the moment.


Pseudonymisation is a method by which personal data is processed such that it can no longer be tied to an individual data subject without linking to additional data. This does offer scope for some forms of data processing to avoid the obligations attendant on processing personal data, as long as the data being provided for processing doesn’t include the additional dataset(s).

Privacy Impact Assessments

A Privacy Impact Assessment (PIA) is an obligatory method of identifying and reducing privacy risks to individuals through the misuse of their personal information when you are undertaking new projects handling personal data.


Profiling means automated processing of personal data for evaluation analysis or prediction. When processing personal data for profiling purposes, you must ensure that appropriate safeguards are in place.

Further reading

This is by no means a complete list of the definitions used in the GDPR but it is the most important ones, other than terms like consent and subject access right which we will discuss in later posts. A brief but more complete list of definitions is available here.

In the next post, we’ll look at when you need consent.

uk election week

Suppression screening, terrorism and your vote

The election today and the aftermath of the appalling events at London Bridge and Borough Market have disrupted my momentum for writing about EU GDPR and what you need to know to get ready for next May 25th. I’m a frequent visitor to Borough Market and often walk across London Bridge, so like many others, this is the first time that terrorism has seemed so close to me.

Top of my mind this week is the news that the third mass murderer at London Bridge was an Italian/Moroccan whose name is apparently on the Schengen Information System – according to the BBC, “An Italian police source has confirmed to the BBC that Zaghba had been placed on a watch list, which is shared with many countries, including the UK.” Both the Westminster Bridge and London Bridge attacks were conducted using hired vehicles, the first a car and the second a van. Last month, the U.S. Transportation Security Administration announced that it wants truck rental agencies to be more vigilant in efforts to prevent these attacks and according to the same article, Penske (a nationwide truck leasing company in the US) screens customers using a watch list.

So, the first question that springs to my mind after London is “Should vehicle rental companies in Europe be screening customers against the Schengen list?” Obviously, not all such attacks are committed using hired vehicles, but many (if not most) are committed using hired or stolen vehicles – and stolen vehicles are likely to be on a police database with an active watch being kept out for them. The larger the vehicle, the more dangerous it is, the more likely it is to be able to crash through barriers and kill and maim people – and the more likely it is to be hired or stolen rather than owned.

The next question that rose to my mind was “Will the UK still have access to the Schengen list after Brexit?” Hopefully, however “hard” Brexit turns out to be, UK and EU negotiators will have cooperation on terrorism at the top of their list and such information will continue to be shared, so increasing systematic use of this data should be top of many people’s agendas.

Last, I worried whether the increased responsibilities for protection of personal data (and vastly increased fines) being introduced with GDPR next May will lead to companies putting their own interests first when it comes to (not) sharing information about suspicious persons with the authorities, or whether there need to be exemptions written into the guidance to ensure that individuals and organisations don’t get fined for breaches of GDPR through trying to do the right thing to help protect the public? I can ask this at next week’s techUK Data Protection Group, where one of the people developing the legislation and guidance from the Department for Culture, Media & Sport will be in attendance.

One other thought concerning data about people seems particularly relevant today – last Tuesday’s Telegraph “fears that thousands of postal ballots could have been sent out to voters who have died, putting their vote at risk of being used by somebody else”. Of course, speaking from personal experience, potentially a much bigger fraud could be all the residents of care homes, especially those with Alzheimer’s, being sent postal votes. Are additional precautions taken in checking that these votes are being filled in by the residents themselves? I know that in at least some cases, the postal vote addressee is not screened against the Power of Attorney registers. Given that GDPR obliges organisations to make sure the personal data that they keep is accurate and up-to-date, I wonder how the formula for fining an organisation 2-4% of global gross revenue under GDPR applies to a taxpayer-funded body such as a local authority!?

Bridging the gap

Bridging the skills gap

TechMarketView’s UKHotViews© post Are you hiding from your skills crisis? last week really struck a chord. Kate Hanaghan gave some interesting feedback about Microsoft’s Cloud Skills Report (which surveyed 250 mid-sized to large UK organisations) but in our experience, many of the same issues apply to moving from proprietary in house systems or legacy packaged software to industry-standard data platforms such as SQL Server.

According to Kate, “individuals themselves are not always keen to move away from the technologies they have spent years working with” and suppliers need to “convince technologists (who tend to be middle aged and highly experienced) they must re-skill in certain areas to support the business as it attempts to grow in digital areas”.

Although as Kate says, many legacy technologies will be around for many years to come, I think that with the increasing pace of technological change, individuals are unwise if they ignore opportunities to embrace new technologies. Movement to the cloud is now so rapid that cost and competitive pressures will force many organisations that are currently steadfastly “on premise” to start moving across sooner rather than later – particularly marketing services companies where demand is elastic. Companies and individuals who try and move from 20 year old, non-standard technology straight to the cloud will struggle, whereas companies with more modern infrastructure and techies with more modern skills will have more of an evolutionary, beaten path .

Apart from competitive pressures, there are many other sound reasons for moving from such aging systems to industry-standard data platforms, as we wrote in Data cleansing – inside the database or on your desktop? One of the key reasons is that using a platform like SQL Server is much more extensible – for example, in the marketing services sector, our matchIT SQL package can connect directly with MIS products upstream and document composition products downstream using shared data, so all the data is processed within SQL Server. For the company, data is more secure and both errors and turnaround time are greatly reduced. For IT staff, it means they can enhance their CV’s with sought-after skills and be ready to embrace the next opportunity a rapidly changing world gives them – such as using Microsoft Azure or Apache Spark for cloud deployment.

I’ll leave the last word to Kate, who wrote to me about her post: “In some ways I just find it so hard to understand. Who wouldn’t want to future-proof their career?! I mean, we’re going to be working till we’re 80!!”


Driving Business with a Real-Time Single Customer View

Since we blogged about the challenges we overcame to deliver a Single Customer View for a major retailer a few years ago, we’ve found a lot of the same challenges repeated across other industry sectors such as non-profit, financial services and education, as well as marketing services providers managing marketing databases for companies in many different sectors. So if that’s more of the same, what’s different? In a word, time. It’s no longer good enough to have a Single Customer View that is only up to date every night, it should be up to date as quickly as your customer switches from one device to another – that is, in real time.

What are the benefits of a real-time Single Customer View?


Let’s stick with the multi-channel retail example both for continuity and because increasingly any product can be viewed through the eyes of a shopper, whether it is a scarf, a phone, a take-out meal, an insurance policy or a credit card account. It is widely recognized that the key to success in retail is a positive customer experience, so let’s look at some research findings:

To illustrate, if a customer orders online using their home computer for collection in store, and then after leaving home they want to change the order (using the browser on their phone or by calling the central customer service line), they expect the vendor to have the latest information about the order immediately available – otherwise, the potential for customer disenchantment is spelt out in the JDA research quoted above. If the info is all up to date, the new visit/call from the customer is an opportunity for the vendor to pitch an additional purchase, based on a 360° view of the customer’s account.

So how can you deliver a real time Single Customer View?


To answer this question, we first need to review where the moving data that we discussed before is coming from: keeping with the multi-channel retail example, it’s from Point-of-Sale systems in store, customers entering orders on the web site and call center operatives entering and looking up orders. These may be feeding into multiple corporate databases (ERP, Accounts, different subsidiary businesses etc.)  The challenge is: how do we perform the standardization, verification and matching that is required, classify misleading data etc. all on the fly, given that there can be as many as a dozen different data flows to handle? And how do we do all this quickly enough to ensure that the operator always has a current and complete view of the customer?

The key to meeting the challenge posed by the need for a real time Single Customer View is to accept that traditional disk-based database technology is too slow – we can’t afford the time to write a dozen or more transactions to disk, standardize and link all these by writing more records to disk and then read it all back from various disks to give the information to the operator – we can’t expect them to have a coffee break between every transaction!

To us the answer was obvious – all the data needs to be kept in computer memory, updated in memory and read back from memory, so getting away from the limitations placed by conventional hard disks and even solid state disks. But, you may say, that’s fine for small volumes of data but what if we’re streaming thousands of transactions a minute into databases with tens (or even hundreds) of millions of customers? The good news is that computer memory is so cheap these days that it’s extremely cost-effective to provision enough memory to handle even a billion customer accounts, with failover to a mirror of the data in the event of a problem.

Now it’s all very well to say “just use a lot of memory”, but can you find software that will run on all the different varieties of hardware, server technology and database systems that make up the corporate data sets? And will this software allow for the different kinds of error and discrepancy that arise when people enter name, company name, mailing address, email and multiple phone numbers? Even more challenging, will it allow for misleading data such as in store purchases being entered using a store address as the customer address, or a customer giving their partner’s phone number along with their own name and email address?

Once you’ve successfully managed to process the data real-time, you can begin to organize, understand and make use of it in real-time. To use the retail example one final time, now you can take the call from the customer on their way to collect their order and (by finding the order linked to their mobile number) enable them easily to add an item they’ve forgotten plus another item prompted by their purchase history. If the branch near home doesn’t have all the items in stock, you can direct them to the branch which does have the stock near their office – based on an up to date work address linked to the customer. With a real-time, 360° Single Customer View, it’s easy!

How Ashley Madison Can Inspire Your Business

As each new name and every illicit detail is revealed, the 37 million members of Ashley Madison, a website promoting extramarital affairs, are scrambling to save their marriages, careers, and reputations.  This list, which is now available to anyone aware ofthe existence of Google, reportedly includes the names and sexual fantasies of members of the armed services, United Nations, and even the Vatican.  Looks like someone’s prayers weren’t heard this week.

As the extent of the contact information becomes more easily accessible, a new breed of data analyst is emerging.  Creative thinkers are using the information to win custody battles, deduce which cities have the most cheaters, and even get a leg up over another candidate for a job promotion.

If everyone from neglected housewives to tawdry tabloid writers is capable of using data to form opinions and make well-informed decisions, the question is… why aren’t you?

Now I’m not talking about crawling through Ashley Madison’s troves of cheaters, I’m talking about your company.  Your data.  Demographics, geographic locations, purchasing behavior… your contact records say a million things about your customers.  A million patterns are lying in wait, holding the key to better marketing, better operations, and better business decisions.  Whereas for Ashley Madison data spelled disaster, for you it should spell potential.

Customer data, when compromised, can be a company’s worst nightmare.  When used intelligently, customer data can increase profits and reduce the guessing game so many businesses play on a day-to-day basis.

In order to use your data intelligently, you must be confident that it is accurate and up-to-date.  If your records indicate you have 14 Jeremiah Whittinglys living in Chicago, you can either double your production of Jeremiah Whittingly personalized baseball caps, or perhaps take a closer look at how clean your data is.  I’m personally leaning towards the second option.

However, beefing up marketing efforts in Juneau, where your database says 10 percent of your client base is located, is a smart idea.  Unless your data entry employee didn’t realize ‘AK’ was the postal code abbreviation for Alaska rather than Arkansas.  In which case, polar bears stand a better chance of appreciating your new billboard than your target market.

Ridding your database of duplicate, incorrect, or incomplete records is the first step in recognizing the power of customer data.  The next step is figuring out what this data means for you and your company, and if every talk show host and dark web hacker can do it with the right tools, so can you.

UK Regulatory Pressure to Contact Customers Increases

In recent weeks, UK government and financial services organisations have received increasing political and regulatory pressure to make greater efforts to proactively notify policy holders and account owners of their rights and savings information. To avoid the threat of regulatory fines, organisations have quickly prioritised data quality initiatives to the top of the list but in reality, the benefits of data suppression and enhancement go far beyond avoiding fines and in fact will make for stronger business models, more trustworthy brands and better customer service.

What’s New

A report in July by the House of Commons Public Accounts Committee quoted Treasury estimates that from 200,000 to 236,000 victims of the collapse of Equitable Life may miss out on compensation payments because it may not be able to trace between 17%-20% of policyholders by that date. The committee urged the Treasury to take urgent action to track down as many former policyholders of the failed insurer as possible (many of whom are elderly) before the March 2014 deadline. Payments totalling £370 million are due to be made by that date.

More recently still, there has been discussion of the huge number of interest rate reductions affecting savers without them being notified – banks and building societies last month announced a further 120 cuts to rates on savings accounts, some as high as 0.5%, on top of 750 made to existing easy access accounts this year. According to the Daily Telegraph, “around 17 million households are believed to have cash in an easy access account”.  While savings providers are able to make cuts of up to 0.25% without notifying customers, a spokesman for the regulator, the Financial Conduct Authority (FCA), told The Telegraph that “it is keeping a close eye on the activity of banks as the blizzard of rate reductions continues.”

Case in Point

To avoid the risk of potentially massive future penalties, a variety of organisations have taken up the challenge of contacting large numbers of customers, to provide the requisite communication. In fact, a financial services organisation which was recently advised by the FCA to make reasonable efforts to contact all its customers, retained a helpIT client to run a suppression job which netted significant savings: of the initial mailing file consisting of over seven million customers, half a million new addresses were supplied, half a million gone aways were removed and over 200 thousand deceased names suppressed. In this instance, the actual and potential savings for the organisation were enormous and went well beyond the cost of non-compliance – to say nothing of the savings to brand reputation in the eyes of new occupants and relatives of the deceased.

Easy Options

Fortunately, the right software makes it easy to compare customer data to an assortment of third party suppression files in different formats, keyed according to different standards. In fact, huge savings can be achieved by employing standard “gone away” and suppression screening, as well increasing the success rate in contacting old customers by finding their new addresses. While there used to be only a couple of broad coverage “gone away” files, these days there is a wealth of data available to mailers to enable them to reach their customers, going far beyond Royal Mail’s NCOA (National Change of Address) and Experian’s Absolute Movers lists. This “new address” data is in many cases pooled by financial services companies via reference agencies such as Equifax (in the reConnect file) and by property agencies via firms such as Wilmington Millennium (Smartlink). Similarly, deceased data is now much more comprehensive and more readily available than ever before.

New address, gone away and deceased data is also easy to access, either as a web-based service or downloaded onto the organisation’s own servers. Costs have come down with competition, so it’s certainly cheaper now to run gone away and deceased suppression than it is to print and mail letters to the “disappeared”.

Although it is never going to be 100%, data and software tools do exist to make it easy for the organisation to take reasonable steps to cost-effectively fulfil their obligations, even on names that might be considered low value, that an organisation might ordinarily have forgotten about.

Bottom Line

These numbers should give pause for thought to organisations of any type that are tempted to “spray and pray” or decide to keep silent about something their customers would really like to know about, regardless of regulation. What’s more, the value to the business, the customers and the brand goes far beyond the regulations with which they need to comply.

helpIT Feedback to Royal Mail PAF® Consultation

On 14 June 2013, Royal Mail launched a consultation on proposed changes to the Postcode Address File (PAF®) licensing scheme and invited contributions from anyone affected. Said to “simplify…the licensing and pricing regime”, helpIT has concerns that the proposed changes would negatively impact direct mailers. As a provider of data quality software to more than 100  organisations that would be affected by such changes, helpIT systems notified customers, collated their input and drafted a response on their behalf. The Consultation is now closed but you can read more about the PAF® licensing options here.

Below is a summary of the feedback submitted to Royal Mail and the kind of feedback received from our customers which mirrors our own concerns.

Q.1: Do you agree with the principles underpinning PAF® Licence simplification?

We are a major provider of PAF address verification software for batch usage – our users are a mixture of service providers and end users who use PAF software embedded within our broader data cleansing solutions. Our feedback includes feedback from many of our users who have replied directly to our notification of the consultation, rather than reply via your portal.

We agree with the principles except for no. 6, “to ensure that current levels of income derived from PAF® licensing are maintained for Royal Mail”. In addition, although we support no. 8, “to seek swift deployment of a PAF® Public Sector Licence”, we feel that free usage should be extended to the private sector, or at least made available to all private sector organisations at a small flat fee of no more than is necessary to cover administration of the licence and to discourage users without a real need.

Q.2 Are there other principles that you believe should underpin PAF® licence simplification?

Royal Mail should follow the example of postal providers in other countries who have made PAF free for users, which (unsurprisingly) is proven to result in improved address quality  and lower sortation and delivery costs through higher levels of automation. We believe that in the UK too, these reduced costs will far outweigh the loss of income by eliminating or reducing the income received from PAF licensing.

Q.3 Do you agree that these are an accurate reflection of market needs?

The market needs an efficient and cost-effective mail system – this principle is not mentioned! Royal Mail’s approach should be to encourage use of direct mail and delivery of goods by mail. It should focus on reduction in handling costs to more effectively compete with other carriers, rather than increase prices in a vain effort to improve profitability.

Q.5 Is the emergence of ‘Licensee by Usage’ as a preferred model reasonable when assessed against the principles, market needs and evaluation criteria?

For reasons stated above, this model does not fit the market needs, or Royal Mail and the UK economy’s fundamental interests. If a usage-based charging model is adopted for batch use of PAF, at the least we would not expect to see a transaction charge applied to a record whose address and postcode are not changed as part of a batch process, as in our opinion this will deter usage of PAF for batch cleansing and directly lead to a lower return on investment for use of mail. Even if this refinement is accepted, this will increase work for solutions and service providers, end users and Royal Mail in recording changed addresses/postcodes and auditing. We have a large, established user base that has made use of PAF, particularly for batch address verification, essential to maintenance of data quality standards. Any increase in charges to our user base will result in decreased usage and the more significant any increase, the higher the dropout rate will be amongst our current users and the lower the take-up from new users.

Typical feedback from an end user is as follows:

We currently use a Marketing Data Warehouse which is fed from transactional databases for Web, Call Centre and Shop transactions. The addresses captured in these different systems are of variable quality, and includes historical data from other systems since replaced. Much of it is unmailable without PAF enhancement, but we are unable to load enhanced/corrected address data back to the transactional systems for operational reasons. This Marketing Data Warehouse is used to mail around 6 million pieces a year via Royal Mail, in individual mailings of up to 600,000, as well as smaller mailings. The quality of the data is crucial to us in making both mailings and customer purchases deliverable. Our Marketing Data Warehouse is built each weekend from the transactional systems, and as a part of this build we PAF process all records each weekend, and load the corrected data into the database alongside the original data. It’s not an ideal solution, but is a pragmatic response to the restrictions of our environment, and enables us to mail good quality addresses, and to remove duplicate records (over 100,000). If we simply count the number of addresses processed per week, at 1p per unit, this would be completely unaffordable. Should this happen we would have to re-engineer our operations to remove redundant processing. Also, when a new PAF file was available we would still have to process the whole file (currently around 2.6 million records), at a cost of £26,000 assuming the minimum cost of 1p per record. This is again unaffordable. It is not in Royal Mail’s interests to price users out of PAF processing records in this way. We therefore urge Royal Mail to reconsider their proposals to ensure our costs do not rise significantly.

Typical feedback from a service provider is as follows:

95% of our PAF usage is to achieve maximum postage discount for our clients. We would either enhance an address or add a DPS suffix to an address.  Therefore, the primary principle of PAF is to assist with the automation of the postal process.  Reading through the consultation document there is very little discussion surrounding PAF and postal system. All the working examples are for call centres. In paragraph 10 of the consultation document, Royal Mail acknowledges the wider use of PAF in areas such as database marketing, e-commerce and fraud management.  However, these areas have no additional benefits to Royal Mail.  On the traditional mail side, Royal Mail directly benefits from the automation of the
postal system through the use of PAF validated addresses.  If Royal Mail wish to promote mail and strive for full automation in the postal system then they should be encouraging the use of PAF validation by mail customers.

There is also a potential conflict of interest for Royal Mail. The more changes they make to PAF then the more revenue they could generate from address updates.  Worthwhile having some limits on the number of addresses that can be changed in a year or at least some authority checking on the necessity of the address changes. I believe there is a conflict of interest with Royal Mail being both the provider and an end user of PAF (through mailing system).  It would be better to have the administration and selling of PAF as an independent organisation.

Q.6 Do you believe that a different model would better meet the principles that underpin licence simplification?

Yes, a flat rate payment model.

Q.9 Are there any further simplification or changes that might be required?

Due to the short notice for the consultation period, during a holiday period, and the lack of notice provided proactively to us as a solutions provider, we can’t currently comment on this except to say that it is probable that changes will be required.

Q.10 Are the ways you use PAF® covered by the proposed terms?

Same answer as Q9.

Q.13 Do you think Transactional pricing is an appropriate way to price PAF®?

As explained above and made crystal clear in the typical responses from two of our users, transactional pricing is NOT an appropriate way to price PAF for batch usage. It will simply lead to a large exodus by batch users of PAF and a significant reduction in the use of direct mail and delivery by mail.

Q.14 Do you think ‘by Transaction’ is an appropriate way of measuring usage?

There are significant systems and auditing problems associated with measuring usage by transaction.

Q.15 Does your organisation have the capability to measure ‘Usage by Transaction’?

Our software does not measure volume of usage and it will not be possible to do this in a foolproof way. It will also lead to significant challenges for audit.

Q.16 Are there situations or Types of Use that you don’t think suit transactional measurement?

Batch database and mailing list cleansing.


Remembering the helpIT Legacy

View ““You’ve come a long way, Baby”: Remembering the world’s first stored program computer

Last Friday was the 65th anniversary of the first successful execution of the world’s first software program and it was great to see the occasion marked by a post and specially commissioned video on Google’s official blog, complete with an interview earlier this month with my father, Geoff Tootill. The Manchester Small-Scale Experimental Machine (SSEM), nicknamed Baby, was the world’s first stored-program computer i.e. the first computer that you could program for different tasks without rewiring or physical reconfiguration. The program was a routine to determine the highest proper factor of any number. Of course, because nobody had written one before, the word “program” wasn’t used to describe it and “software” was a term that nobody had coined. The SSEM was designed by the team of Frederic C. Williams, Tom Kilburn and Geoff Tootill, and ran its first program on 21st June 1948.

I have heard first hand my father’s stories about being keen to work winter overtime as it was during post-war coal rationing and the SSEM generated so much heat that it was much the cosiest place to be! Also, his habit of keeping one hand in his pocket when touching any of the equipment to prevent electric shocks. Before going to work on the Manchester machine, my Geoff Tootill Notebookfather worked on wartime development and commissioning of radar, which he says was the most responsible job he ever had (at the age of just 21), despite his work at Manchester and (in the 60’s) as Head of Operations at the European Space Research Organisation. Although he is primarily an engineer, a hardware man, my father graduated in Mathematics from Cambridge University and had all the attributes to make an excellent programmer. I like to think that my interest in and aptitude for software stemmed from him in both nature and nurture – although aptitude for hardware and electronics didn’t seem to rub off on me. He was extremely interested in the software that I initially wrote for fuzzy matching of names and addresses as it appealed to him both as a computer scientist and as a linguist. My father then went on to design the uniquely effective phonetic algorithm, soundIT, which powers much of the fuzzy matching in helpIT’s software today, as I have written about in my blog post on the development of our phonetic routine.

The Manchester computing pioneers have not had enough recognition previously, and I’m delighted that Google has paid tribute to my father and his colleagues for their contribution to the modern software era – and to be able to acknowledge my father’s place in the evolution of our company.

Additional Resources:

6 Reasons Companies Ignore Data Quality Issues

When lean businesses encounter data quality issues, managers may be tempted to leverage existing CRM platforms or similar tools to try and meet the perceived data cleansing needs. They might also default to reinforcing some existing business processes and educating users in support of good data. While these approaches might be a piece of the data quality puzzle, it would be naive to think that they will resolve the problem. In fact, ignoring the problem for much longer while trying some half-hearted approaches, can actually amplify the problem you’ll eventually have to deal with later. So why do they do it? Here are some reasons we have heard about why businesses have stuck their heads in the proverbial data quality sand:

1. “We don’t need it. We just need to reinforce the business rules.”

Even in companies that run the tightest of ships, reinforcing business rules and standards won’t prevent all your problem. First, not all data quality errors are attributable to lazy or untrained employees. Consider nicknames, multiple legitimate addresses and variations on foreign spellings just to mention a few. Plus, while getting your process and team in line is always a good habit, it still leaves the challenge of cleaning up what you’ve got.

2. “We already have it. We just need to use it.”

Stakeholders often mistakenly think that data quality tools are inherent in existing applications or are a modular function that can be added on. Managers with sophisticated CRM or ERP tools in place may find it particularly hard to believe that their expensive investment doesn’t account for data quality. While customizing or extending existing ERP applications may take you part of the way, we are constantly talking to companies that have used up valuable time, funds and resources trying to squeeze a sufficient data quality solution out of one of their other software tools and it rarely goes well.

3. “We have no resources.”

When human, IT and financial resources are maxed out, the thought of adding a major initiative such as data quality can seem foolhardy. Even defining business  requirements is challenging unless a knowledgeable data steward is on board. With no clear approach, some businesses tread water in spite of mounting a formal assault. It’s important to keep in mind though that procrastinating a data quality issue can cost more resources in the long run because the time it takes staff to navigate data with inherent problems, can take a serious toll on efficiency.

4. “Nobody cares about data quality.”

Unfortunately, when it comes to advocating for data quality, there is often only one lone voice on the team, advocating for something that no one else really seems to care about. The key is to find the people that get it. They are there, the problem is they are rarely asked. They are usually in the trenches, trying to work with the data or struggling to keep up with the maintenance. They are not empowered to change any systems to resolve the data quality issues and may not even realize the extent of the issues, but they definitely care because it impacts their ability to do their job.

5. “It’s in the queue.”

Businesses may recognize the importance of data quality but just can’t think about it until after some other major implementation, such as a data migration, integration or warehousing project. It’s hard to know where data quality fits into the equation and when and how that tool should be implemented but it’s a safe bet to say that the time for data quality is before records move to a new environment. Put another way: garbage in = garbage out. Unfortunately for these companies, the unfamiliarity of a new system or process compounds the challenge of cleansing data errors that have migrated from the old system.

6. “I can’t justify the cost.”

One of the biggest challenges we hear about in our industry is the struggle to justify a data quality initiative with an ROI that is difficult to quantify. However, just because you can’t capture the cost of bad data in a single number doesn’t mean that it’s not affecting your bottom line. If you are faced with the dilemma of ‘justifying’ a major purchase but can’t find the figures to back it up, try to justify doing nothing. It may be easier to argue against sticking your head in the sand, then to fight ‘for’ the solution you know you need.

Is your company currently sticking their head in the sand when it comes to data quality? What other reasons have you heard?

Remember, bad data triumphs when good managers do nothing.