How to Close the Big Data Talent Gap at Your Organisation

Big data offers many benefits for organisations in all industries, but unfortunately, a lot of companies don’t reap these benefits yet. The reason is not that they don’t want to start with big data, nor that they don’t understand what big data is. The challenge many companies face is attracting the right big data talent.

Big data talent is scarce and what is scarce is expensive. Finding the right data professionals for a big data project remains difficult for a lot of organisations. This does not come as a surprise if we look at the figures around global big data talent.

A Growing Global Big Data Talent Shortage

Back in 2011, McKinsey already estimated that in 2018 there would be a shortage of 290.000 data scientists in the United States alone. Globally, demand for data scientists is projected to exceed supply by more than 50 percent by 2018. In the UK, the expected shortage is 56.000 data scientists by 2020. Currently, there are over 500.000 big data jobs listed globally.

As it may be clear, demand is skyrocketing, and this will only increase in the future. On the other hand, big data master programs have only started to appear in the past five years or so. Currently, there are only a few hundred data science master programs, barely enough to cater for future demand.

The result is scarcity of available big data talent, which is unfortunately not the only challenge for organisations. Most organisations have no idea who to look for. The term ‘Big Data Scientist’ is over-hyped and almost every organisation wanting to deal, or dealing, with big data, is looking to hire this illustrative big data scientist, while they have no clue what the Big Data scientist should do! To make matters worse, there are many ‘ordinary’ data analysts out there who have taught themselves a programming language or something else and now call themselves ‘Big Data Scientist’, to make an extra buck of this hype.

Understanding The Required Big Data Skills

These two challenges, therefore, make it difficult for organisations looking to hire big data talent. It is important to know what they really need and which skills they should really hire. This will, on the one hand, prevent hiring ‘fake’ data scientists and, on the other hand, will help you hire a better fit for your business.

Of course, what skills are required all depends on the big data projects being developed, the industry, the type of organisation and many other variables. However, as Louis Colombus writes on Forbes, we can distinguish ten skills that are most in demand by organisations. The top three of this list includes Python programming, Linux expertise and Structured Query Language, with Apache Hadoop currently in position eight.

Of course, this list of skills is constantly changing, especially since we are moving to the algorithmic business and AI is rapidly gaining more ground. It is therefore quite likely that within a few years the list includes important topics such as machine learning, algorithm design or artificial intelligence. As a result, the search will become even harder for organisations.

Finding the Right Big Data Talent

Knowing that there is a shortage and that many organisations don’t know what to look for, how can organisations close this big data gap and hire the best talent for their projects?

First of all, you can look internally to see which employees show the potential to become a big data professional. Without knowing it, you might have a great big data talent in the making within your organisation who, with the right training, can become a top big data scientist.

In case you already have some great big data professionals within your organisation, but are looking for more, another option can be to optimise and amplify existing big data talent through a Centre of Excellence. Within this Centre of Excellence, the existing big data talent is grouped together and works for the entire organisation instead of a single department. An additional advantage is that top Centres of Excellence, if executed correctly, can attract additional talent that was difficult to attract before.

If these two options don’t work for you, it is time to start looking externally. Obviously, you can publish your job posts on Datafloq, but before you do so, understand why you need big data professionals and what skills they should have. As mentioned, this depends on the big data project and the better you know the exact skills and technologies you are looking for, the better you will be able to find the right big data professional. In addition, provide critical input on talent needs – who, what, when, where, how – to better facilitate the talent deployment and better attract and retain the big data professionals.

But having a large pool of data scientists to choose from is not the end game. In the end, it is making sure that the right big data professional works for the right company and works on the right big data project. After all, the manufacturing industry requires a different type of Big Data talent, with different skills, then, for example, the financial services industry.

Finally, if you manage to attract big data professionals, ensure that you treat them well. Restricting big data professionals too much will very likely result in an empty big data department. On the other hand, offering them interesting and challenging projects and giving them the freedom to bring solutions to the table you would not have thought of will likely attract more big data talent. Good luck!

Image: alphaspirit/Shutterstock