The advance in digital information and the use of artificial intelligence (A.I.) have improved the quantity and quality of new labour market information substantially in the last two decades. In part, this development has been driven by increasing low-cost computing power and the availability of real-time demand and supply labour market information. In addition, these two factors have been accompanied by the need for closer links between training provision and labour market outcomes, the emergence of new forms of employment, and how the global supply chains are organised via digital connectivity. ¹

While traditional labour market information could only deal with a limited range of (aggregated and administratively collected) information, e.g., participation rates, levels of unemployment, wages, work hours, the incidence of labour disputes, accident rates, unionisation and occupational changes, current forms of labour market analysis is increasingly viewed as ‘intelligence’ because of its ability to generate insights that are often hidden behind the complexity of real-time and voluminous databases. Thus, with the use of artificial intelligence, labour market analysis can now tackle areas of enquiry (some routinely) that were hard to achieve before:

• Using ‘demand’ (e.g., recruitment) data to predict future growth across different occupations
• The extent, nature, and types of skills mismatch
• Future skills demand
• What skills may be relevant to occupational mobility
• Which jobs are really new, and which ones are just a re-combination of old/existing jobs that require new or additional skills?
• What is the impact of digitalisation within occupations?
• The market value of skills
• New skills required by employers
• Identification and visualisation of atypical career pathways

Because of the potential of these ‘new digital tool’, both public and private sector bodies are busy implementing their own labour market intelligence projects. For example, the European Commission constructed its skills agenda around “Strengthe­ning skills intelligence”, which led to an ambitious OVATE online tool that will identify emerging occupations and skills trends. This is then linked to other mobility tools, such as the Europass, an EU-wide platform that aims at supporting learning and careers. In New Zealand, the authorities build the ‘Occupational Outlooks’ platform for career development based on future job trends. The private sector has also been building similar tools. The list is too long to cite here. Some good examples can be found in an extensive Cedefop report in 2021

Similar efforts are also seen in middle-income countries. For example, the Malaysian government used big data to identify a “Critical Occupations List” representing a set of occupations in demand. The platform is intended to form the primary instrument to promote better coordination of human capital policies and matching skills supply to employer demand. Likewise, various forms of labour market intelligence projects have been implemented in Tunisia, Malawi, Myanmar, and Albania³.

Not all are ‘plain sailing’! New tools also bring new challenges. For starters, we are still in the transitional phase in which the vast majority of labour market researchers are unfamiliar with A.I. techniques and the analytic paradigm associated with big data analytics. It will take time to build up a useful capacity to take advantage of new digital tools. Secondly, many of the big or platform-based data sets are not open to the public, and most of them have been curated by commercial companies. Thirdly, there is a research ethic hurdle to overcome. A lot of the big data sets, especially when it is linked to administrative databases, require consent or privacy protection of some sort. In some cases, it may require a change in legislation for research to have access to the data. However, the most challenging issues concern statistical biases and representativeness. A large part of the new/big data are ‘messy’ or unstructured that needs ‘cleaning’ (e.g., the ‘extraction’, ‘transformation’ and ‘loading’ (ETL) stage). In addition, many of the big data sets are derived from sources wherewe are uncertain about the characteristics of the underlying populations. As a result, cross-country comparisons arising from the use of A.I. and big data are rare.

Despite challenges of some magnitude, we can draw some conclusion regarding the current stage of development. A.I. and labour market intelligence are here to stay and are expected to progress much further. For instance, web-based big data are expected to become an integral part of the data infrastructure that supports countries, regions, employers, learners, and education and training providers. Another area of growth is the need to train more domain experts in using A.I. because big data analysis is only as good as the underlying ontologies that build and maintain the data source. Here, it seems that the need for domain-specific knowledge and expertise has been underestimated so far. The current progress also highlights the fact that the application of A.I. to labour market intelligence still requires policymakers to be cautious when interpreting the findings of big data analysis. In this sense, ‘human intelligence’ and domain expertise will remain essential. It is the combination of artificial and human intelligence that will be central to developing big data’s role in shaping effective technical and vocational education and training (TVET) and skills policies in the coming years.

¹ See Mezzanzanica, M. and Mercorio, F., ‘Big Data Enables Labor Market Intelligence’, in Encyclopedia of Big Data Technologies, Springer International Publishing, 2018, pp. 1–11.

² Cedefop; European Commission; ETF; ILO; OECD; UNESCO (2021). Perspectives on policy and practice: tapping into the potential of big data for skills policy. Luxembourg: Publications Office.

³ See Cedefop 2021, ibid.