Datalore
Collaborative data science platform for teams
5 Emerging Data Job Trends in 2024
This is a guest blog post by Thu Vu.
The long-term outlook for data jobs – like data scientists, statisticians, and market research analysts – predicts explosive growth, much faster than most other fields. But here’s the catch: Despite the promising stats, data professionals are struggling to find jobs. AI is also shaking things up with capable LLMs, coding assistants, and loads of smart tools for data analytics.
So, what does this mean for you as a data professional? What exactly has changed in the past one to two years? To try and find out, I spent 24 hours researching the data job market.
In this article, I’m going to share five key trends I’ve noticed in the market right now and detail what I would do to adapt to these changes. To view all the graphs in one place, click the link below:
1. Data job postings are getting competitive but remain stable
First up, let’s talk about the job market. You might be thinking – with all the tech layoffs we’ve been hearing about, surely data jobs must be taking a hit, right? Well, you might be surprised.
Since the end of 2022, the number of jobs for data scientists, data analysts, and data engineers has decreased by about 15% but has remained quite stable since the beginning of this year.
This is pretty interesting, especially when you consider that the tech industry has experienced two big waves of layoffs since the COVID-19 pandemic – one at the beginning of 2023 and another at the beginning of 2024.
But if we lay this trend next to the tech layoffs, we can see that the data job posts didn’t seem to be affected much by the tech layoff trends.
This trend can also be observed per job title.
Compared to the last two years, the job market is certainly more competitive, with companies being more selective. The uncertain economic growth and high interest rates are making companies tighten their budgets and seek skills that focus on efficiency and cost reduction. This might explain why it feels harder to land a job today.
Don’t let that discourage you, though! It just means we need to be smarter about how we position ourselves in the market. Which brings me to my next point…
2. Technical skills
If you’re in the data field, you’ve probably noticed that certain programming languages are becoming more dominant. Technical skills are consolidating. And the data backs this up.
A whopping 86% of data scientists said that Python is the main language they use for current projects. Another 10% said they use it as a secondary language.
And Python is being used for everything from data analysis to machine learning and web development.
A few years ago, if you asked me whether you should learn R or Python, I would’ve said it doesn’t matter which language you start with. But today, I’d definitely say Python.
But Python isn’t the only player in town. Looking at job postings data over the past two years, SQL consistently appears in up to 60% of all job posts, right alongside Python, month after month. This shows that Python and SQL are and will remain the dominant languages for the foreseeable future.
So, if you’re trying to become a data scientist from scratch and are wondering what programming language to learn, I’d say it’s smart to focus on mastering Python and SQL first.
In fact, you might be surprised to hear this, but Wes McKinney, the creator of the pandas package, was recently asked to give advice for data scientists. His shocking response? “Learn SQL.”
He said, “Learning SQL is actually a really good skill. It’s not just learning SQL, the language, but learning the concepts of relational algebra and how to think about data sets, designing schemas, and organizing data.”
As much as I love Python, I must agree on this. I once worked on a machine learning project at a large bank for anti-money laundering. We used PySpark SQL to query bank transaction data, think billions of rows. I remember at the beginning it’d take me up to 10 minutes to even run a command to check the number of rows in the dataset. Only afterward, I learned about concepts such as partitioning and load balancing to help me optimize my code.
So, if you’re looking to level up your skills, don’t overlook SQL. It’s more than just a query language – it’s a way of thinking about data that can make you a better data professional overall.
3. Emergence of AI engineering
Now, here’s where things get really exciting. While data scientists and data analyst roles are always in demand, there’s a new kid on the block: AI engineers.
These roles have emerged with the rapid development of large language models in the past two years. And here’s the interesting part – they don’t require a PhD. Rather, they require an in-depth knowledge of LLMs, prompt engineering, and AI agent workflow engineering.
This role is still very new, and there are different opinions about what the job exactly entails.
But in general, you can think of it this way: If we put data science research on one end of the spectrum and AI applications on the other, AI engineers lean towards the product and user end. They build applications that use pre-trained AI models, or foundation models, to solve a specialized business problem. You can perhaps think of AI engineers as software engineers specializing in AI products.
For example, let’s say a company wants to develop an AI application like a specialized customer chatbot using LLMs. AI Engineers will be the ones who put the AI model in place, do some prompt engineering, fine-tuning the model if necessary, tailoring the workflow to the use case, and also evaluating the application to make sure it works as it should.
Andrej Karpathy, a big name in the AI world, predicts that:
“In numbers, there’s probably going to be significantly more AI engineers than there are ML engineers/LLM engineers. One can be quite successful in this role without ever training anything.” – Andrej Karpathy
And the job post data backs this up. In the past two years, AI engineer jobs have been growing much faster than ML engineer jobs, with the former actually surpassing the latter in May 2023, according to Hacker News Hiring trends.
You might be asking what’s the key difference between AI engineers and ML engineers: Well, it’s quite simple, creating LLM applications rely heavily on prompt engineering, which of course doesn’t exist in a traditional machine learning model. Evaluating LLM applications also requires a very different approach to using precision and recall metrics or mean squared error like in machine learning models.
Recently, PwC, one of the big four companies, landed a deal with OpenAI to become its first resale partner. This partnership helps PwC scale AI capabilities across businesses to help drive accelerated impact for clients. What this basically means is more and more businesses will be able to build and incorporate AI into their solutions. And who will implement this? AI engineers, that’s who.
So, if you have a data science background, how do you become an AI engineer?
I’m certainly not an expert in this field. But a popular post on Hacker News suggested focusing on these areas:
- Mathematical foundations
- Basic statistics
- Python programming
- Courses such as:
- ML specialization and deep learning specialization by Andrew Ng on Coursera
- fast.ai deep learning courses
- And then you can choose a specific area of AI to focus on, for example:
- Natural language processing (NLP)
- Computer vision (CV)
- Reinforcement learning (RL)
- Other specializations
There are other emerging new roles being talked about, such as quality assurance business analyst (or QA analyst for short), who investigate LLM outputs, design A/B tests, and build dashboards to monitor the overall performance of AI products. This role is relatively business-focused.
This is yet to be seen in the data for the number of new vacancies being posted. But, this role does not require significant qualifications, and I think anyone with an analytical mind and data science skills can adapt themselves to this new role.
4. Freelancing trends
Now, let’s talk about another interesting trend: the rise of freelancing in the data world.
In 2024, there seems to have been a significant increase in the number of job posts looking for contractors and freelancers. This is exciting news for those of you who might be interested in a part-time or flexible job.
Becoming a freelancer is a great way to learn new skills fast and build a diverse portfolio because you’ll get to work on real business problems with clients, sometimes in completely new domain areas, with different types of data and analytics tools.
In the US, most freelancers find their work through previous clients, friends and family, social media, and professional contacts.
The question is, if you don’t have a previous client already, how do you find your first client?
Here’s my advice: Start small and utilize your own network. Post about your relevant projects on LinkedIn, showcase your skills, and mention that you’re looking for freelance work in this and that area. I used to be scared to post stuff on Linkedin, but a little bit of fear is a good sign that you’re getting out of your comfort zone.
Even better, you can approach your neighbors, friends, and family members directly. Small business owners around the corner might need your help, and friends and relatives might have interesting work for you.
Once you have 2–3 projects like this (even for free or at very small commissions), you can start collecting reviews, building your process, and refining your services to offer them at a larger scale. You can even start going on online job boards like Upwork and Fiverr to find more jobs.
5. Automation – The rise of low-code and no-code tools
Last but definitely not least, let’s talk about a trend that’s changing the game: low-code and no-code tools. It’s not an exaggeration to say that, in the future, anyone could become a data analyst without specialized training. And I’m talking about doing much more than just creating pivot tables in Excel.
Low-code and no-code development platforms, enabled by AI, are becoming increasingly popular. The low-code market is forecast to grow at an annual average rate of 22.9% from 2023 to 2030.
These tools are making data analytics more accessible to people who might not have traditional coding skills. They provide simplified interfaces that let anyone plug in their data and do tasks like data preparation, statistical analysis, visualization, and even build machine learning models without significant coding effort.
Now, I know what some of you might be thinking. This sounds too good to be true, right? Well, you’re not entirely wrong: AI’s smart, but it still needs human expertise to guide it in complex tasks. And at the end of the day, someone needs to make sense of the results, evaluate them, and make a decision.
So, it’s all about finding that sweet spot where tech and expertise work together.
As low-code platforms improve to automate a lot of entry-level data analysis tasks, data jobs will likely become more specialized in the future. So instead of just looking for “data analyst” or “data scientist” job titles, you might consider a wider range of titles such as marketing analyst, sales analyst, risk analyst, psychometrician, quality assurance analyst, data governor, or perhaps even data cleaning ninja.
Conclusions
No matter what your next career move will be, the market will always be in constant flux. The key is to stay adaptable, keep learning, and make valuable connections with people.
If you want to dive deeper into the data behind this video, you can find the full report in a Datalore notebook on the JetBrains Datalore website.
Subscribe to Datalore News and Updates
