Eleven years ago, the fate of data scientists’ careers changed across the globe with the publication of a Harvard Business Review blog titled: “Data Scientist: The Sexiest Job of the 21st Century”. The article painted a compelling portrait of a highly proficient “data hacker” capable of coding, dissecting data, and effectively conveying insights to corporate leaders. The core message resonated and soon Google searches for the career exploded.
Peak of inflated expectations
This blog was published at the height of the “big data” era of data science, where companies were awash in millions of data points on their customers and competitors, with little idea about what to do with all of it. The rapid development of machine learning algorithms, along with cloud computing tools, brought about data scientists: A skilled professional who can write thousands of lines of code and has a deep understanding of math and statistics.
The article sold the idea of becoming a data wizard to millions of students, resulting in the meteoric rise of computer science and statistics related degrees. Students were ecstatic to solve business problems using math and code.
In 2012, a total of 1.8 million degrees were awarded in the U.S., where just over 2.5% of the degrees were computer science related majors. Fast forward to 2020, that percentage more than doubled to 5%. Engineering and math related degrees grew much faster than other common majors like education and business. Students bought into the hype cycle — data science and software engineering were in an all-out gold rush.
Trough of disillusionment
Now, more than ten years later, has the hype lived up to the promise? Well, sort of. From a demand perspective, data science-related job postings on Indeed have declined proportionally more than overall job postings. Software development jobs (not data science, per se, but tech-adjacent) in particular were hit hard, declining nearly three-fold since their peak in early 2022. Of course, this contributed to the broader softening in the economy from higher interest rates, but it has had an especially pronounced effect in data science.
The decline in postings for job seekers has driven up competition, increasing the already high barrier to entry for the field. Our internal data on data science postings indicates a strong increase in apply rates, the percentage of job seekers that successfully complete a job application. The near-high of 8% percent last month is nearly double the median across Appcast. Students and career professionals are competing over a dwindling supply of postings – and it has turned an already tough job market into a ruthless one.
One potential explanation for the rush of job seekers (ten years ago and today) is how lucrative the career can be. Traditional consumer goods companies found a significant spike in the demand for the products during the peak of the COVID-pandemic and needed data professionals to clean, analyze, and predict this unexpected shift. Median advertised salaries for data science related postings accordingly increased nearly $10,000 dollars due to employer demand.
That is just a single data scientist’s salary, not a whole team. A successful crew of data professionals often requires software engineers, data analysts, and data engineers. The unfortunate truth is that it is expensive to invest in data science.
The return-on-investment of data science has been a mixed bag, with several companies initiating high-profile layoffs in the past twelve months. On the flip side, the advent of generative AI has skyrocketed. Open AI’s valuation has exploded to $90 billion dollars!
Plateau of productivity
Despite the challenges companies face with growing a successful data science team, several organizations have tied their identity to the power of machine learning. Stitchfix (an online clothing retailer) has a website dedicated to all the different algorithms they use to optimize their business.
As the dust settles from the consumer spending frenzy during the height of the COVID era, the labor market for technical roles has slowly ebbed. Demand for tech workers peaked in mid-2021 and now monthly employment growth is barely positive. For companies that have found their niche (like Stitchfix) the demand for highly qualified statisticians will always persist, but for smaller firms it remains more of a gamble.
What does this mean for recruiters and job seekers?
Laying out all these facts, is data science truly an “unsexy” career? The truth is somewhere in the middle.
For recruiters, often companies spend lavishly on hiring highly specialized computer scientists and statisticians, but are more frugal when they need to hire data analysts and engineers to build reports and maintain data sources. This does not mean the investment doesn’t pay off; it often does in miraculous ways. Understanding the true data needs of a company will go a long way to prevent incorrect hiring decisions.
For job seekers, it’s a mix of the good, bad, and ugly. The good is data science can be lucrative in terms of potential earnings. The bad is the high barrier to entry – postings sometimes require a master’s or PhD with several years of experience, but always necessitate skills in R, Python, SQL, and other tools. The downright ugly is the huge amount of competition exists for openings – postings often get hundreds of applicants in the first 24 hours. On the other side of a competitive job market exists a highly rewarding career – but not without sacrifice.
