Bridging the gap

Bridging the skills gap

TechMarketView’s UKHotViews© post Are you hiding from your skills crisis? last week really struck a chord. Kate Hanaghan gave some interesting feedback about Microsoft’s Cloud Skills Report (which surveyed 250 mid-sized to large UK organisations) but in our experience, many of the same issues apply to moving from proprietary in house systems or legacy packaged software to industry-standard data platforms such as SQL Server.

According to Kate, “individuals themselves are not always keen to move away from the technologies they have spent years working with” and suppliers need to “convince technologists (who tend to be middle aged and highly experienced) they must re-skill in certain areas to support the business as it attempts to grow in digital areas”.

Although as Kate says, many legacy technologies will be around for many years to come, I think that with the increasing pace of technological change, individuals are unwise if they ignore opportunities to embrace new technologies. Movement to the cloud is now so rapid that cost and competitive pressures will force many organisations that are currently steadfastly “on premise” to start moving across sooner rather than later – particularly marketing services companies where demand is elastic. Companies and individuals who try and move from 20 year old, non-standard technology straight to the cloud will struggle, whereas companies with more modern infrastructure and techies with more modern skills will have more of an evolutionary, beaten path .

Apart from competitive pressures, there are many other sound reasons for moving from such aging systems to industry-standard data platforms, as we wrote in Data cleansing – inside the database or on your desktop? One of the key reasons is that using a platform like SQL Server is much more extensible – for example, in the marketing services sector, our matchIT SQL package can connect directly with MIS products upstream and document composition products downstream using shared data, so all the data is processed within SQL Server. For the company, data is more secure and both errors and turnaround time are greatly reduced. For IT staff, it means they can enhance their CV’s with sought-after skills and be ready to embrace the next opportunity a rapidly changing world gives them – such as using Microsoft Azure or Apache Spark for cloud deployment.

I’ll leave the last word to Kate, who wrote to me about her post: “In some ways I just find it so hard to understand. Who wouldn’t want to future-proof their career?! I mean, we’re going to be working till we’re 80!!”

 

Driving Business with a Real-Time Single Customer View

Since we blogged about the challenges we overcame to deliver a Single Customer View for a major retailer a few years ago, we’ve found a lot of the same challenges repeated across other industry sectors such as non-profit, financial services and education, as well as marketing services providers managing marketing databases for companies in many different sectors. So if that’s more of the same, what’s different? In a word, time. It’s no longer good enough to have a Single Customer View that is only up to date every night, it should be up to date as quickly as your customer switches from one device to another – that is, in real time.

What are the benefits of a real-time Single Customer View?

 

Let’s stick with the multi-channel retail example both for continuity and because increasingly any product can be viewed through the eyes of a shopper, whether it is a scarf, a phone, a take-out meal, an insurance policy or a credit card account. It is widely recognized that the key to success in retail is a positive customer experience, so let’s look at some research findings:

To illustrate, if a customer orders online using their home computer for collection in store, and then after leaving home they want to change the order (using the browser on their phone or by calling the central customer service line), they expect the vendor to have the latest information about the order immediately available – otherwise, the potential for customer disenchantment is spelt out in the JDA research quoted above. If the info is all up to date, the new visit/call from the customer is an opportunity for the vendor to pitch an additional purchase, based on a 360° view of the customer’s account.

So how can you deliver a real time Single Customer View?

 

To answer this question, we first need to review where the moving data that we discussed before is coming from: keeping with the multi-channel retail example, it’s from Point-of-Sale systems in store, customers entering orders on the web site and call center operatives entering and looking up orders. These may be feeding into multiple corporate databases (ERP, Accounts, different subsidiary businesses etc.)  The challenge is: how do we perform the standardization, verification and matching that is required, classify misleading data etc. all on the fly, given that there can be as many as a dozen different data flows to handle? And how do we do all this quickly enough to ensure that the operator always has a current and complete view of the customer?

The key to meeting the challenge posed by the need for a real time Single Customer View is to accept that traditional disk-based database technology is too slow – we can’t afford the time to write a dozen or more transactions to disk, standardize and link all these by writing more records to disk and then read it all back from various disks to give the information to the operator – we can’t expect them to have a coffee break between every transaction!

To us the answer was obvious – all the data needs to be kept in computer memory, updated in memory and read back from memory, so getting away from the limitations placed by conventional hard disks and even solid state disks. But, you may say, that’s fine for small volumes of data but what if we’re streaming thousands of transactions a minute into databases with tens (or even hundreds) of millions of customers? The good news is that computer memory is so cheap these days that it’s extremely cost-effective to provision enough memory to handle even a billion customer accounts, with failover to a mirror of the data in the event of a problem.

Now it’s all very well to say “just use a lot of memory”, but can you find software that will run on all the different varieties of hardware, server technology and database systems that make up the corporate data sets? And will this software allow for the different kinds of error and discrepancy that arise when people enter name, company name, mailing address, email and multiple phone numbers? Even more challenging, will it allow for misleading data such as in store purchases being entered using a store address as the customer address, or a customer giving their partner’s phone number along with their own name and email address?

Once you’ve successfully managed to process the data real-time, you can begin to organize, understand and make use of it in real-time. To use the retail example one final time, now you can take the call from the customer on their way to collect their order and (by finding the order linked to their mobile number) enable them easily to add an item they’ve forgotten plus another item prompted by their purchase history. If the branch near home doesn’t have all the items in stock, you can direct them to the branch which does have the stock near their office – based on an up to date work address linked to the customer. With a real-time, 360° Single Customer View, it’s easy!

How Ashley Madison Can Inspire Your Business

As each new name and every illicit detail is revealed, the 37 million members of Ashley Madison, a website promoting extramarital affairs, are scrambling to save their marriages, careers, and reputations.  This list, which is now available to anyone aware ofthe existence of Google, reportedly includes the names and sexual fantasies of members of the armed services, United Nations, and even the Vatican.  Looks like someone’s prayers weren’t heard this week.

As the extent of the contact information becomes more easily accessible, a new breed of data analyst is emerging.  Creative thinkers are using the information to win custody battles, deduce which cities have the most cheaters, and even get a leg up over another candidate for a job promotion.

If everyone from neglected housewives to tawdry tabloid writers is capable of using data to form opinions and make well-informed decisions, the question is… why aren’t you?

Now I’m not talking about crawling through Ashley Madison’s troves of cheaters, I’m talking about your company.  Your data.  Demographics, geographic locations, purchasing behavior… your contact records say a million things about your customers.  A million patterns are lying in wait, holding the key to better marketing, better operations, and better business decisions.  Whereas for Ashley Madison data spelled disaster, for you it should spell potential.

Customer data, when compromised, can be a company’s worst nightmare.  When used intelligently, customer data can increase profits and reduce the guessing game so many businesses play on a day-to-day basis.

In order to use your data intelligently, you must be confident that it is accurate and up-to-date.  If your records indicate you have 14 Jeremiah Whittinglys living in Chicago, you can either double your production of Jeremiah Whittingly personalized baseball caps, or perhaps take a closer look at how clean your data is.  I’m personally leaning towards the second option.

However, beefing up marketing efforts in Juneau, where your database says 10 percent of your client base is located, is a smart idea.  Unless your data entry employee didn’t realize ‘AK’ was the postal code abbreviation for Alaska rather than Arkansas.  In which case, polar bears stand a better chance of appreciating your new billboard than your target market.

Ridding your database of duplicate, incorrect, or incomplete records is the first step in recognizing the power of customer data.  The next step is figuring out what this data means for you and your company, and if every talk show host and dark web hacker can do it with the right tools, so can you.

UK Regulatory Pressure to Contact Customers Increases

In recent weeks, UK government and financial services organisations have received increasing political and regulatory pressure to make greater efforts to proactively notify policy holders and account owners of their rights and savings information. To avoid the threat of regulatory fines, organisations have quickly prioritised data quality initiatives to the top of the list but in reality, the benefits of data suppression and enhancement go far beyond avoiding fines and in fact will make for stronger business models, more trustworthy brands and better customer service.

What’s New

A report in July by the House of Commons Public Accounts Committee quoted Treasury estimates that from 200,000 to 236,000 victims of the collapse of Equitable Life may miss out on compensation payments because it may not be able to trace between 17%-20% of policyholders by that date. The committee urged the Treasury to take urgent action to track down as many former policyholders of the failed insurer as possible (many of whom are elderly) before the March 2014 deadline. Payments totalling £370 million are due to be made by that date.

More recently still, there has been discussion of the huge number of interest rate reductions affecting savers without them being notified – banks and building societies last month announced a further 120 cuts to rates on savings accounts, some as high as 0.5%, on top of 750 made to existing easy access accounts this year. According to the Daily Telegraph, “around 17 million households are believed to have cash in an easy access account”.  While savings providers are able to make cuts of up to 0.25% without notifying customers, a spokesman for the regulator, the Financial Conduct Authority (FCA), told The Telegraph that “it is keeping a close eye on the activity of banks as the blizzard of rate reductions continues.”

Case in Point

To avoid the risk of potentially massive future penalties, a variety of organisations have taken up the challenge of contacting large numbers of customers, to provide the requisite communication. In fact, a financial services organisation which was recently advised by the FCA to make reasonable efforts to contact all its customers, retained a helpIT client to run a suppression job which netted significant savings: of the initial mailing file consisting of over seven million customers, half a million new addresses were supplied, half a million gone aways were removed and over 200 thousand deceased names suppressed. In this instance, the actual and potential savings for the organisation were enormous and went well beyond the cost of non-compliance – to say nothing of the savings to brand reputation in the eyes of new occupants and relatives of the deceased.

Easy Options

Fortunately, the right software makes it easy to compare customer data to an assortment of third party suppression files in different formats, keyed according to different standards. In fact, huge savings can be achieved by employing standard “gone away” and suppression screening, as well increasing the success rate in contacting old customers by finding their new addresses. While there used to be only a couple of broad coverage “gone away” files, these days there is a wealth of data available to mailers to enable them to reach their customers, going far beyond Royal Mail’s NCOA (National Change of Address) and Experian’s Absolute Movers lists. This “new address” data is in many cases pooled by financial services companies via reference agencies such as Equifax (in the reConnect file) and by property agencies via firms such as Wilmington Millennium (Smartlink). Similarly, deceased data is now much more comprehensive and more readily available than ever before.

New address, gone away and deceased data is also easy to access, either as a web-based service or downloaded onto the organisation’s own servers. Costs have come down with competition, so it’s certainly cheaper now to run gone away and deceased suppression than it is to print and mail letters to the “disappeared”.

Although it is never going to be 100%, data and software tools do exist to make it easy for the organisation to take reasonable steps to cost-effectively fulfil their obligations, even on names that might be considered low value, that an organisation might ordinarily have forgotten about.

Bottom Line

These numbers should give pause for thought to organisations of any type that are tempted to “spray and pray” or decide to keep silent about something their customers would really like to know about, regardless of regulation. What’s more, the value to the business, the customers and the brand goes far beyond the regulations with which they need to comply.

helpIT Feedback to Royal Mail PAF® Consultation

On 14 June 2013, Royal Mail launched a consultation on proposed changes to the Postcode Address File (PAF®) licensing scheme and invited contributions from anyone affected. Said to “simplify…the licensing and pricing regime”, helpIT has concerns that the proposed changes would negatively impact direct mailers. As a provider of data quality software to more than 100  organisations that would be affected by such changes, helpIT systems notified customers, collated their input and drafted a response on their behalf. The Consultation is now closed but you can read more about the PAF® licensing options here.

Below is a summary of the feedback submitted to Royal Mail and the kind of feedback received from our customers which mirrors our own concerns.

Q.1: Do you agree with the principles underpinning PAF® Licence simplification?

We are a major provider of PAF address verification software for batch usage – our users are a mixture of service providers and end users who use PAF software embedded within our broader data cleansing solutions. Our feedback includes feedback from many of our users who have replied directly to our notification of the consultation, rather than reply via your portal.

We agree with the principles except for no. 6, “to ensure that current levels of income derived from PAF® licensing are maintained for Royal Mail”. In addition, although we support no. 8, “to seek swift deployment of a PAF® Public Sector Licence”, we feel that free usage should be extended to the private sector, or at least made available to all private sector organisations at a small flat fee of no more than is necessary to cover administration of the licence and to discourage users without a real need.

Q.2 Are there other principles that you believe should underpin PAF® licence simplification?

Royal Mail should follow the example of postal providers in other countries who have made PAF free for users, which (unsurprisingly) is proven to result in improved address quality  and lower sortation and delivery costs through higher levels of automation. We believe that in the UK too, these reduced costs will far outweigh the loss of income by eliminating or reducing the income received from PAF licensing.

Q.3 Do you agree that these are an accurate reflection of market needs?

The market needs an efficient and cost-effective mail system – this principle is not mentioned! Royal Mail’s approach should be to encourage use of direct mail and delivery of goods by mail. It should focus on reduction in handling costs to more effectively compete with other carriers, rather than increase prices in a vain effort to improve profitability.

Q.5 Is the emergence of ‘Licensee by Usage’ as a preferred model reasonable when assessed against the principles, market needs and evaluation criteria?

For reasons stated above, this model does not fit the market needs, or Royal Mail and the UK economy’s fundamental interests. If a usage-based charging model is adopted for batch use of PAF, at the least we would not expect to see a transaction charge applied to a record whose address and postcode are not changed as part of a batch process, as in our opinion this will deter usage of PAF for batch cleansing and directly lead to a lower return on investment for use of mail. Even if this refinement is accepted, this will increase work for solutions and service providers, end users and Royal Mail in recording changed addresses/postcodes and auditing. We have a large, established user base that has made use of PAF, particularly for batch address verification, essential to maintenance of data quality standards. Any increase in charges to our user base will result in decreased usage and the more significant any increase, the higher the dropout rate will be amongst our current users and the lower the take-up from new users.

Typical feedback from an end user is as follows:

We currently use a Marketing Data Warehouse which is fed from transactional databases for Web, Call Centre and Shop transactions. The addresses captured in these different systems are of variable quality, and includes historical data from other systems since replaced. Much of it is unmailable without PAF enhancement, but we are unable to load enhanced/corrected address data back to the transactional systems for operational reasons. This Marketing Data Warehouse is used to mail around 6 million pieces a year via Royal Mail, in individual mailings of up to 600,000, as well as smaller mailings. The quality of the data is crucial to us in making both mailings and customer purchases deliverable. Our Marketing Data Warehouse is built each weekend from the transactional systems, and as a part of this build we PAF process all records each weekend, and load the corrected data into the database alongside the original data. It’s not an ideal solution, but is a pragmatic response to the restrictions of our environment, and enables us to mail good quality addresses, and to remove duplicate records (over 100,000). If we simply count the number of addresses processed per week, at 1p per unit, this would be completely unaffordable. Should this happen we would have to re-engineer our operations to remove redundant processing. Also, when a new PAF file was available we would still have to process the whole file (currently around 2.6 million records), at a cost of £26,000 assuming the minimum cost of 1p per record. This is again unaffordable. It is not in Royal Mail’s interests to price users out of PAF processing records in this way. We therefore urge Royal Mail to reconsider their proposals to ensure our costs do not rise significantly.

Typical feedback from a service provider is as follows:

95% of our PAF usage is to achieve maximum postage discount for our clients. We would either enhance an address or add a DPS suffix to an address.  Therefore, the primary principle of PAF is to assist with the automation of the postal process.  Reading through the consultation document there is very little discussion surrounding PAF and postal system. All the working examples are for call centres. In paragraph 10 of the consultation document, Royal Mail acknowledges the wider use of PAF in areas such as database marketing, e-commerce and fraud management.  However, these areas have no additional benefits to Royal Mail.  On the traditional mail side, Royal Mail directly benefits from the automation of the
postal system through the use of PAF validated addresses.  If Royal Mail wish to promote mail and strive for full automation in the postal system then they should be encouraging the use of PAF validation by mail customers.

There is also a potential conflict of interest for Royal Mail. The more changes they make to PAF then the more revenue they could generate from address updates.  Worthwhile having some limits on the number of addresses that can be changed in a year or at least some authority checking on the necessity of the address changes. I believe there is a conflict of interest with Royal Mail being both the provider and an end user of PAF (through mailing system).  It would be better to have the administration and selling of PAF as an independent organisation.

Q.6 Do you believe that a different model would better meet the principles that underpin licence simplification?

Yes, a flat rate payment model.

Q.9 Are there any further simplification or changes that might be required?

Due to the short notice for the consultation period, during a holiday period, and the lack of notice provided proactively to us as a solutions provider, we can’t currently comment on this except to say that it is probable that changes will be required.

Q.10 Are the ways you use PAF® covered by the proposed terms?

Same answer as Q9.

Q.13 Do you think Transactional pricing is an appropriate way to price PAF®?

As explained above and made crystal clear in the typical responses from two of our users, transactional pricing is NOT an appropriate way to price PAF for batch usage. It will simply lead to a large exodus by batch users of PAF and a significant reduction in the use of direct mail and delivery by mail.

Q.14 Do you think ‘by Transaction’ is an appropriate way of measuring usage?

There are significant systems and auditing problems associated with measuring usage by transaction.

Q.15 Does your organisation have the capability to measure ‘Usage by Transaction’?

Our software does not measure volume of usage and it will not be possible to do this in a foolproof way. It will also lead to significant challenges for audit.

Q.16 Are there situations or Types of Use that you don’t think suit transactional measurement?

Batch database and mailing list cleansing.

 

Remembering the helpIT Legacy

View ““You’ve come a long way, Baby”: Remembering the world’s first stored program computer

Last Friday was the 65th anniversary of the first successful execution of the world’s first software program and it was great to see the occasion marked by a post and specially commissioned video on Google’s official blog, complete with an interview earlier this month with my father, Geoff Tootill. The Manchester Small-Scale Experimental Machine (SSEM), nicknamed Baby, was the world’s first stored-program computer i.e. the first computer that you could program for different tasks without rewiring or physical reconfiguration. The program was a routine to determine the highest proper factor of any number. Of course, because nobody had written one before, the word “program” wasn’t used to describe it and “software” was a term that nobody had coined. The SSEM was designed by the team of Frederic C. Williams, Tom Kilburn and Geoff Tootill, and ran its first program on 21st June 1948.

I have heard first hand my father’s stories about being keen to work winter overtime as it was during post-war coal rationing and the SSEM generated so much heat that it was much the cosiest place to be! Also, his habit of keeping one hand in his pocket when touching any of the equipment to prevent electric shocks. Before going to work on the Manchester machine, my Geoff Tootill Notebookfather worked on wartime development and commissioning of radar, which he says was the most responsible job he ever had (at the age of just 21), despite his work at Manchester and (in the 60’s) as Head of Operations at the European Space Research Organisation. Although he is primarily an engineer, a hardware man, my father graduated in Mathematics from Cambridge University and had all the attributes to make an excellent programmer. I like to think that my interest in and aptitude for software stemmed from him in both nature and nurture – although aptitude for hardware and electronics didn’t seem to rub off on me. He was extremely interested in the software that I initially wrote for fuzzy matching of names and addresses as it appealed to him both as a computer scientist and as a linguist. My father then went on to design the uniquely effective phonetic algorithm, soundIT, which powers much of the fuzzy matching in helpIT’s software today, as I have written about in my blog post on the development of our phonetic routine.

The Manchester computing pioneers have not had enough recognition previously, and I’m delighted that Google has paid tribute to my father and his colleagues for their contribution to the modern software era – and to be able to acknowledge my father’s place in the evolution of our company.

Additional Resources:

6 Reasons Companies Ignore Data Quality Issues

When lean businesses encounter data quality issues, managers may be tempted to leverage existing CRM platforms or similar tools to try and meet the perceived data cleansing needs. They might also default to reinforcing some existing business processes and educating users in support of good data. While these approaches might be a piece of the data quality puzzle, it would be naive to think that they will resolve the problem. In fact, ignoring the problem for much longer while trying some half-hearted approaches, can actually amplify the problem you’ll eventually have to deal with later. So why do they do it? Here are some reasons we have heard about why businesses have stuck their heads in the proverbial data quality sand:

1. “We don’t need it. We just need to reinforce the business rules.”

Even in companies that run the tightest of ships, reinforcing business rules and standards won’t prevent all your problem. First, not all data quality errors are attributable to lazy or untrained employees. Consider nicknames, multiple legitimate addresses and variations on foreign spellings just to mention a few. Plus, while getting your process and team in line is always a good habit, it still leaves the challenge of cleaning up what you’ve got.

2. “We already have it. We just need to use it.”

Stakeholders often mistakenly think that data quality tools are inherent in existing applications or are a modular function that can be added on. Managers with sophisticated CRM or ERP tools in place may find it particularly hard to believe that their expensive investment doesn’t account for data quality. While customizing or extending existing ERP applications may take you part of the way, we are constantly talking to companies that have used up valuable time, funds and resources trying to squeeze a sufficient data quality solution out of one of their other software tools and it rarely goes well.

3. “We have no resources.”

When human, IT and financial resources are maxed out, the thought of adding a major initiative such as data quality can seem foolhardy. Even defining business  requirements is challenging unless a knowledgeable data steward is on board. With no clear approach, some businesses tread water in spite of mounting a formal assault. It’s important to keep in mind though that procrastinating a data quality issue can cost more resources in the long run because the time it takes staff to navigate data with inherent problems, can take a serious toll on efficiency.

4. “Nobody cares about data quality.”

Unfortunately, when it comes to advocating for data quality, there is often only one lone voice on the team, advocating for something that no one else really seems to care about. The key is to find the people that get it. They are there, the problem is they are rarely asked. They are usually in the trenches, trying to work with the data or struggling to keep up with the maintenance. They are not empowered to change any systems to resolve the data quality issues and may not even realize the extent of the issues, but they definitely care because it impacts their ability to do their job.

5. “It’s in the queue.”

Businesses may recognize the importance of data quality but just can’t think about it until after some other major implementation, such as a data migration, integration or warehousing project. It’s hard to know where data quality fits into the equation and when and how that tool should be implemented but it’s a safe bet to say that the time for data quality is before records move to a new environment. Put another way: garbage in = garbage out. Unfortunately for these companies, the unfamiliarity of a new system or process compounds the challenge of cleansing data errors that have migrated from the old system.

6. “I can’t justify the cost.”

One of the biggest challenges we hear about in our industry is the struggle to justify a data quality initiative with an ROI that is difficult to quantify. However, just because you can’t capture the cost of bad data in a single number doesn’t mean that it’s not affecting your bottom line. If you are faced with the dilemma of ‘justifying’ a major purchase but can’t find the figures to back it up, try to justify doing nothing. It may be easier to argue against sticking your head in the sand, then to fight ‘for’ the solution you know you need.

Is your company currently sticking their head in the sand when it comes to data quality? What other reasons have you heard?

Remember, bad data triumphs when good managers do nothing.

Tamerlan Tsarnaev escaped monitoring due to simple misspelling?

Over the past several weeks in particular, there has been much discussion in our data quality community about the costs of poor data quality. Poor customer service, wasted mailings, inefficient delivery service – it often frustrates us to see the issues caused by poor data quality when we all know the importance of resolving those concerns.

Unfortunately, the recent events in Boston may have just demonstrated that our inefficient data management practices may have been partly to blame for the failure to keep tabs on one of the bombers. According to recent reports, Senator Lindsey Graham suggests a possible reason that a suspected terrorist was able to travel abroad to a location with possible ties to terrorist activity could have been due to a simple misspelling.  According to Senator Graham, “The reason we didn’t know he went over to Russia is because his name was misspelled.”

Is it possible?

The world is going to be asking this question a lot over the next few days and yes, it is very possible.  People misspell names and email addresses on a regular basis, either when data is captured or when they are searching for it. In this case though, airlines and the FBI do have software for fuzzy matching against terrorist watch lists, so what went wrong in this case? I guess the software may not have been in place everywhere it needs to be, or maybe the error (be it a typo, deliberate misspelling or phonetic variation) was too different from the correct name to be detected – however good the software, there are always going to be inconsistencies that can’t be picked up without a huge number of “false positives”. If Tamerlan Tsarnaev had a green card but was not a citizen, he would have had to present his green card on re-entry to the USA, but not on exit – is there a system in place to capture and check foreign passport numbers against a watch list when people leave the country? Clearly, if the system has the right capability, immigration and law enforcement officials have an opportunity to question the individual both before travel and on return to the USA.

It’s unfortunate and horrifying that such a tragic event may in part be due to poor data quality. In most cases, we have only minor ramifications such as mail going astray or being duplicated, or longer customer service calls, to prove to  organizations the importance of this very necessary technology. We may never know for sure if a simple misspelling was really to blame for lack of monitoring, but it is clear that data quality and identity resolution has important implications for terrorist watch activities.

The New Paradigm in Healthcare Data Quality

There is no higher importance in managing customer information than when making decisions on health care. While most markets are busy striving for a ‘single customer view’ to improve customer service KPIs or marketing campaign results, healthcare organizations must focus on establishing  a ‘single patient view’, making sure a full patient history is attached to a single, correct contact.  Unlike in traditional CRM solutions, healthcare data is inherently disparate
and is managed by a wide variety of patient systems that, in addition to collecting and managing contact data, also tracks thousands of patient data points including electronic health records, insurance coverage, provider names,  prescriptions and more. Needless to say, establishing the relationships between patients and their healthcare providers, insurers, brokers, pharmacies and the like or even grouping families and couples together, is a significant challenge. Among them are issues with maiden/married last names, migration of individuals between family units and insurance plans, keying errors at point of entry or even deliberate attempts by consumers to defraud the healthcare system.

In many cases, the single patient view can be handled through unique identifiers , such as those for group health plans or for individuals within their provider network. This was an accepted practice at a recent Kaiser Permanente location I visited, where a gentleman went to the counter and reeled off his nine digit patient number before saying “hello”. But while patient ID numbers are standard identifiers, they will differ between suppliers and patients can’t be relied on to use it as their first method of identification. This is where accuracy and access to other collected data points (I.e. SSN, DOB and current address) becomes critical.

While healthcare organizations have done a decent job so far of attempting to establish and utilize this ‘single patient view’, the healthcare data quality paradigm is shifting once again. For example, The Patient Protection and Affordable Care Act (PPACA) means that healthcare organizations will now have to deal with more data, from more sources and face tougher regulations on how to manage and maintain that data.  The ObamaCare Health Insurance Exchange Pool means that more Americans can potentially benefit from health insurance coverage, increasing the number with coverage by around 30 million. Through these new initiatives, consumers will also have greater choice for both coverage and services  – all further distributing the data that desperately needs to be linked.

With such inherent change – how do you effectively service patients at the point-of-care? And, do you want your trained medics and patient management team to be responsible for the data quality audit before such care can even begin?

So what are the new dynamics that healthcare companies need to plan for?

  • Addition of new patients into a system without prior medical coverage or records
  • Frequent movement of consumers between healthcare plans under the choice offered by the affordable care scheme
  • Increased mobility of individuals through healthcare systems as they consume different vendors and services

This increased transactional activity means healthcare data managers must go beyond the existing efforts of linking internal data and start to look at how to share data across systems (both internal and external) and invest in technology that will facilitate this critical information exchange. Granted, this will be a significant challenge given the fact that many organizations have several proprietary systems, contract requirements and privacy concerns but oddly enough, this begins with best practices in managing contact data effectively.

Over the last year, I’ve worked with an increasing number of customers on the issue of managing the introduction of new data into healthcare databases.  Just like healthcare, data quality is both preventative and curative. Curative measures include triage on existing poor quality data, and investigating the latent symptoms of unidentified relationships in the data. The preventative measures are to introduce a regimen of using DQ tools to accurately capture new information at
point of entry efficiently, and to help identify existing customers quickly and accurately.

For healthcare customers, we’ve managed to do just this by implementing helpIT systems’ technology, matchIT SQL to deal with the backend data matching, validation and merging and findIT S2 to empower users to quickly and accurately identify existing patients or validate new patient details with the minimum of keystrokes. This complementary approach gives a huge return on investment allowing clinical end-users to focus on the task at hand, rather than repeatedly dealing with data issues.

Whenever there is movement in data or new sources of information, data quality issues will arise. But when it comes to healthcare data quality, I’m sure healthcare DBA’s and other administrators are fully aware of the stakes at hand. Improving and streamlining data capture plus tapping into the various technology connectors that will give physicians and service providers access to all patient data will have a profound effect on patient care, healthcare costs, physician workloads and access to relevant treatment. Ultimately, this is the desired outcome.

I’m delighted to be engaged further on this subject so if you have more insight to share, please comment on this or drop me a line.


Click & Collect – How To Do It Successfully?

In the UK this Christmas, the most successful retailers have been those that sell online but allow collection by the shopper – in fact, these companies have represented a large proportion of the retailers that had a good festive season. One innovation has been the rise of online retailers paying convenience stores to take delivery and provide a convenient collection point for the shopper, but two of the country’s biggest retailers, John Lewis and Next, reckon that click and collect has been the key to their Christmas sales figures – and of course they both have high volume e-commerce sites as well as many bricks and mortar stores.

The article here by the Daily Telegraph explains why “click and collect” is proving so popular, especially in a holiday period. The opportunities for major retailers are  obvious, especially as they search for ways to respond to the Amazon threat – but how do they encourage their customers to shop online and also promote in store shopping? The key is successful data-driven marketing: know your customer, incentivize them to use loyalty programs and target them with relevant offers. However, this also presents a big challenge – the disparity and inconsistency in the data that the customer provides when they shop in these different places.

In store, they may not provide any information, or they may provide name and phone number, or they may use a credit card and/or their loyalty card. Online they’ll provide name, email address and (if the item is being delivered), credit card details and their address. If they are collecting in store, they may just provide name and email address and pay on collection – and hopefully they’ll enter their loyalty card number, if they have one. To complicate matters further, people typically have multiple phone numbers (home, office, mobile), multiple addresses (home and office, especially if they have items delivered to their office) and even multiple email addresses. This can be a nightmare for the marketing and IT departments in successfully matching this disparate customer data in order to establish a Single Customer View. To do this, they need software that can fulfill multiple sophisticated requirements, including:

  • Effective matching of customer records without being thrown off by data that is different or missing.
  • Sophisticated fuzzy matching to allow for keying mistakes and inconsistencies between data input by sales representatives in store and in call centers, and customers online.
  • The ability to recognize data that should be ignored – for example, the in-store purchase records where the rep keyed in the address of the store because the system demanded an address and they didn’t have time to ask for the customer’s address, or the customer didn’t want to provide it.
  • Address verification using postal address files to ensure that when the customer does request delivery, the delivery address is valid – and even when they don’t request delivery, to assist the matching process by standardizing the address.
  • The ability to match records (i) in real-time, in store or on the website (ii) off-line, record by record as orders are fed though for fulfillment and (iii) as a batch process, typically overnight as data from branches is fed through. The important point to note here is that the retailer needs to be able to use the same matching engine in all three matching modes, to ensure that inconsistencies in matching results don’t compromise the effectiveness of the processing.
  • Effective grading of matches so that batch and off-line matching can be fully automated without missing lots of good matches or mismatching records. With effective grading of matching records, the business can choose to flag matches that aren’t good enough for automatic processing so they can be reviewed by users later.
  • Recognition of garbage data, particularly data collected from the web site, to avoid it entering the marketing database and compromising its effectiveness.
  • Often, multiple systems are used to handle the different types of purchase and fulfillment. The software must be able to connect to multiple databases storing customer data in different formats for the different systems

With a wide range of data quality solutions on the market, it’s often difficult to find a company that can check all of these boxes. That’s where helpIT systems comes in. If you are a multi-channel retailer currently facing these challenges, contact helpIT systems for a Free Data Analysis and an in depth look at how you can achieve a Single Customer View.