Wanted: Data Engineers

What did you want to be when you grew up? Personally, I've always loved sport and for some time I thought about being a rugby player. Admittedly, this is a far cry from my role at BAE Systems but, in my defence, but when I wasn't dreaming of running out at Twickenham I was also interested in computers and technology.

After 15 years in the data and technology industry, through a time of unprecedented change, growth and innovation, it feels like a good decision. But of course, job choices are not confined to childhood. Even as we progress through our chosen career path there are decisions to take and options to explore.

And in the rapidly evolving field of technology entirely new roles can sometimes emerge - like data engineering, for example.

Data engineering - what, why and how

You might be wondering what a 'data engineer' actually is but you'll almost certainly have a perception about what a data scientist does.

Comparing the two, a data scientist uses scientific methods, creativity and domain understanding to extract knowledge and insights from various forms of data, whilst a data engineer is more focused on constructing architecture, systems and data pipelines which can turn such insights into better customer engagement or strengthened large scale processes for millions of customers or decisions every day.

So they're similar roles working in the same areas but with different skills and both critical to the success of delivering business value from data insights.

Data science has been a 'hot job' for a few years now as organisations seek to build capability in this new area. The good news is that organisations are increasingly recognising the value of data engineering, too, in the process of delivering value. After all, a mass consumer business like a bank, utility company or telecommunications company has millions of customer interactions - scaling insights on such a scale requires engineers, and plenty of them. The bad news, though, is that much remains to be done.

Closing the delivery gap

In the recent report we commissioned from Forrester, we found that although demand for data insights across all industries is at an all-time high, significant investments in data science and analytics capabilities have not guaranteed that organisations can derive business value from this wealth of new-found information. Quite the opposite, actually.

According to our research, 72 per cent of firms have sought to execute data insights projects, but only 10 per cent have actually managed to deliver business value from those projects today. Put another way, nine out of 10 firms today do not currently see value from their insights projects.

Now, it's important to note that delivery challenges are hardly infrequent. Turning something on paper or PowerPoint deck into positive impact is easy to say, far harder to do - for both private and public sectors alike.

But that's exactly why data engineers are so important. It is they who possess the nous and knowledge to scale up data insights - particularly in organisations with a large volume of customer interactions or operational decisions - and ensure that decision-makers can do more than rely on reports or data sheets.

Our Forrester research found that 38 per cent of firms are looking to invest in improved engineering capabilities. That's an encouraging start but that's all it is. In reality, every firm needs to recognise the value of such a role in order to actually use the data now available to them.

Those that choose to do so are already reaping the rewards.

Find out more;Download the Forrester Research Paper 'The Data Insights Gap: Take Your Firm From Insight to Action'.