A Wakefield Research survey reveals a disturbing state of affairs among data engineering professionals. The study of 600 data engineers suggests an overwhelming majority are burned out and calling for relief.
- 97% report experiencing burnout in their day-to-day jobs.
- 70% say they are likely to leave their current company for another data engineering job in the next 12 months.
- 79% have considered leaving the industry entirely.
Among the highest compensated data professionals, a data engineer prepares data for data analysts, scientists, and people that use self-service BI tools to perform the analytics that guide data-driven decisions. Turnover in this critical role is especially disruptive for companies given the significant influence of data engineers on data productivity and overall business agility.
The data engineers identified significant sources of burnout
- Spending too much time finding and fixing errors.
- Manual, repetitive processes related to data prep and pipelines.
- Relentless pace of requests from colleagues.
Other challenges cited by data engineers include:
- 91% report frequently receiving requests for analytics with unrealistic or unreasonable expectations.
- 87% say they are blamed when things go wrong.
- 69% say their company’s data governance policies make their day-to-day job more difficult.
- 89% report frequent disruptions to work-life balance due to unplanned work.
These factors force data engineers to work long, irregular schedules that take a toll on their well-being. In fact, 78% of survey respondents wish their job came with a therapist to help manage work-related stress.
“Companies are hiring data professionals to work on complex problems with little attention to designing the workflows that enable efficient team collaboration,” said Christopher Bergh, CEO, Head Chef at DataKitchen.
“It’s basically setting an entire, highly skilled class of workers up for failure and then blaming those same workers when projects fail. We hope to see a shift in the importance of process and workflow design in managing teams of data professionals.”
DataOps: Addressing systemic inefficiencies
The problems in data organizations may seem deeply entrenched, but data engineers identified new methodologies like DataOps to address systemic inefficiencies. For example, 78% of engineers and 91% of managers stated that DataOps is very important or essential to incorporate to their data practices.
DataOps, and its counterpart Agile Data Governance, automate end-to-end data lifecycle workflows, including error mitigation, development, deployment, data discovery, observability, and governance. By automating these tasks, data organizations can address the causes of data engineer burnout while improving the quality, agility, veracity, and usability of analytics.
“Our survey serves as a wake-up call to data organizations,” said Bryon Jacob, CTO at data.world. “Data engineers are calling for change. That means instituting reliable, efficient and repeatable workflow processes that will improve analytics collaboration and productivity while restoring sanity to the lives of data engineers.”
