As data centers get smarter, the workforces running them are struggling to keep up.

While the industrywide ideal is for so-called dark data centers — meaning, centers without any on-site staff — this brave new reality, for the moment, is still aspirational and years away.

In fact, industry leaders say the flood of information coming from smart equipment and sensors needed for artificial intelligence and automation created a widespread need for employees capable of actually understanding it.

Those workers are proving very hard to come by.

“The more automation and monitoring you deploy, the more dependent you are on staff in order to interpret and analyze,” said Clae Anderson, director of IT support at Kaiser Permanente, speaking at a Bisnow webinar on data center operations last week.

“Our systems will eventually get to a point where machine learning is able to predict outcomes and deliver solutions to our engineers, but we’re not there yet and it’s going to take a long time to get there.”

Data Center operators increasingly use technologies like sophisticated remote monitoring, AI and automation to run their facilities. Networks of sensors and smart devices — from complex power management systems to simple power strips — provide detailed real-time information about performance and environmental conditions to both automated systems and human analysts, helping to predict and prevent outages, use power more efficiently and ultimately hold down costs.

Anderson points to a network of thermal sensors that allows his team at Kaiser Permanente to model how installing new server equipment will impact the flow of heat throughout the data center.

“We have to understand the impact that these high-density cabinets are going to have in our environment before we install them because we can’t fix it later on,” he told the panel.

“Problem-solving with spreadsheets, experience and tribal knowledge isn’t enough anymore — we need analytics and science behind the decisions.”

Across all sectors of the data center industry, the hope is that the availability of this information will allow AI and machine learning to take over much of the day-to-day decision-making, and data centers will effectively run themselves. But as of now, the potential benefits of this flood of data are hitting a barrier, according to Tamara Budec, vice president of implementation services at Digital Realty Trust.

“We have a dire need to hire folks who can actually read and interpret the data,” she said.

The data center industry as a whole is facing a severe labor shortage. Unprecedented growth and heightened performance standards mean that experienced operations professionals are a prized commodity. And as Bisnow reported previously, much of the data center workforce is nearing retirement age.

When it comes to finding employees qualified to analyze performance data, the situation is worse. Traditionally, working in data center operations required hands-on skills: the ability to repair a generator or quickly modify an HVAC system. While these are highly technical and specialized jobs, the skill set is completely different from the ability to use data center management software and problem-solving based on a digital dashboard.

“Five or 10 years ago, we’d look for staff with screwdriver-and-hammer kind of experience,” Budec said. “Today, you have to supplement that staffing with folks with a software background, who can look at your graphs and monitoring systems and interpret the data and know-how to read it and see divergences or patterns emerging.”

Many large data center operators are trying to address this problem by pouring money into training and workforce development programs through universities and local governments. Near Northern Virginia’s Data Center Alley, both Marist College and Northern Virginia Community College have established data center programs funded largely by the industry. In Loudoun County, the government collaborates with major providers on workforce development for data center operations at the high school level.

Others in the industry are looking to retrain their existing workforce, although these efforts can encounter resistance from old-guard operations staff, according to Donna Jacobs, senior director for technology services at the University of Pennsylvania.

“I think part of that labor transition is bringing them either to be able to do those new types of skills or at least understand the value that it’s bringing, versus seeing it as a threat to their job,” Jacobs said. “It’s about breaking through that cultural barrier to get people to appreciate the information they’re getting and the data we’re analyzing and learn from it.”

Alleviating the labor pressure somewhat is the fact that systems monitoring and analysis does not need to happen on-site, allowing remote teams to analyze input from multiple data centers simultaneously. Data Center operations service provider BCS, for instance, runs what it calls a Tactical Control Center, where analysts and experts specializing in monitoring specific systems from batteries to control systems track data from each facility the company manages.

Panelists at Thursday’s Bisnow webinar agreed that ultimately, the labor squeeze will push industry leaders towards further investment in AI, machine learning and automation. We may be a long way from dark data centers, but, as UPenn’s Jacobs emphasized, necessity is already forcing understaffed companies to leave as much decision-making as possible to machines.

“Artificial intelligence and being able to augment staff with automation is going to get us over this hump,” she said.