machine learning oil and gas

The age for leveraging big data in Oil and Gas has finally arrived.


Finding a way to be profitable (or even just survive) at recent oil prices has Oil and Gas companies turning to new and innovative solutions to cut expenditures. According to GE, the ability to reduce capital expenditures by as little as 1% would save over $90 Billion of accumulated costs over a 15 year period.

But what are these new and innovative technologies really about? What is the Internet of Things, and why are companies all of a sudden professing their love for IoT, big data, and advanced analytics? Quite a lot has changed, notably on 3 fronts:

  1. How we get data
  2. How we use data
  3. How we amplify user intelligence

In order to truly understand IoT, Big Data, Machine Learning, and any other buzzword frequently associated with the IoT/Big Data/Artificial Intelligence movement, it’s necessary to start at the source—the data. Without any data, none of this is possible. Luckily, data is being created faster than ever before because of two unique advancements.

More Endpoints are Coming Online

The Internet of Things is not some magical place we have no exposure to—it’s simply the internet with more connected endpoints.  Historically, the only endpoints on the internet were desktops, servers, laptops, tablets, and even cellular devices.  Now, all sorts of control systems are being added, causing the internet to get much bigger quickly. In fact, Gartner projects that IoT endpoint shipments will approach nearly 6 Billion per year by 2019…more than doubling the expected shipping rate of traditional endpoints!

Oil and Gas IoT connected device

In Oil & Gas, this influx of new connected endpoints means that more and more data is becoming available to better make decisions. For example, connected endpoint devices are beginning to replace route-based monitoring in asset maintenance. Not only do these connected endpoints reduce the likelihood of human error, they are also able to provide more up to date data.

Increased Sensorization

Outside of controlling machines, connected endpoints can also collect data—lots of data—using physical sensors monitoring all sorts of different physical phenomena like temperature, pressure, vibration, and humidity. The reason we can more efficiently sensorize connected endpoints now versus a decade ago is because the price of each sensor has dropped dramatically. According to Goldman Sachs, the price of the average sensor will have dropped from $1.30 in 2004 to nearly $0.50 in 2016. This significant price drop enables each and every new endpoint to have more sensors, creating exponentially more data for operators and business teams to use to drive better decisions.

sensor cost forecast IoT

How We Use Data – Better Analytics

The industry-standard approach for analyzing all of this new data would be to use statistical analysis tools like SPSS and Microsoft Excel. However, tools like these require large amounts of user input and are not well equipped to handle this massive data inflow without manual supervision.  Because of this, IoT and “Big Data” are creating a massive talent gap. According to the McKinsey Global Institute, the United States may face a 50 to 60 percent gap between the supply and the demand for deep analytic talent in the coming future.

To address this data gap, a new approach to how we analyze data must be taken.  The only way to actually ensure lossless analysis when dealing with this much data is with smarter, learning algorithms capable of handling data independently without as much user guidance. This concept has bred a field known as Machine Learning, or in some circles, Artificial Intelligence.

job trends oil and gas

As can be evidenced by’s visualized growth of Data Scientist openings shown above, Machine Learning (a major component of Data Science) is a rapidly growing field. In Oil & Gas, ML algorithms have already been used to build predictions that physics-based models have only dreamed about.  For example, they are able to dynamically predict the probability of an event such as a stuck pipe in real time. Even more impressively, these algorithms can be used to modify numerous control systems (brought about by the IoT) to minimize the likelihood of these types of events in the first place!  Moreover, because each particular well being drilled is unique, the ability for algorithms to dynamically understand operations on a per-well or basin basis provides a much more granular and fine-tuned prediction of events.

Outside of drilling, machine learning has also been used to revolutionize the operation of large power generation assets. Even assets that very infrequently experience catastrophic failures, like combustion turbines, can be better understood using machine learning techniques. Using a technique known as anomaly detection, which falls under the unsupervised branch of machine learning, it is possible to identify when each unique combustion turbine within a fleet is behaving abnormally, or in a fashion that correlates with patterns of degradation. Without getting into the techniques that make this happen, this unsupervised solution contains one pivotal component worth noting—it is universally scalable and does not need to understand anything about the asset in order to detect abnormal behavior.

How We Amplify User Intelligence

The final area of Machine Learning or AI where advances are helping the Oil & Gas industry to get the most out of the big data revolution is the area of NLP, or Natural Language Processing.

Making Use of Unstructured Data

How often do people at your company write things down or take notes? Every email, every update to SalesForce, every maintenance log, every injury report: all of these things are written by employees but very rarely used. Why is that?

The written word falls into a category of data called “unstructured data”, or data without format.  If I asked you to perform calculations on data in a relational database (like an Excel spreadsheet), it would be easy. But if I asked you to quantify something like Wikipedia, it would be much more difficult.  You can read it and derive meaning, but breaking different articles down into Boolean operands is a much more challenging task.  The problem is, most of the data being generated is unstructured. According to Oracle, about 80% of generated data is unstructured:

IoT Oil and Gas

To handle this growth in unstructured data, natural language processing algorithms are now being used to supplement human analysis.  Most people don’t have the time to read through thousands of emails, documents, or other text entries to figure out what is going on at a company-wide level. A computer does, however.  There are recent advances in NLP algorithms, like Latent Semantic Analysis, that can look at numerous text sources and capture meaning in a way similar to a human.  For example, an algorithm can search through a log of work-related incidents and identify which incidents are knee-related injuries. The word “knee” may not appear in all of them, but based on the context and expression of the writing (i.e. somebody writing “I stepped at a funny angle and tore my ACL”), NLP machine learning algorithms can identify that this is similar to other “knee” related incidents. Then, it can be combined with more traditional machine learning algorithms mentioned above to make a safer workplace.

Pulling it all together

So what is the age of IoT really about for Oil & Gas? Three things, really.

  1. Collect more data.
  2. Get more insights.
  3. Act more responsibly.

With that as the vision, we will build a better tomorrow for Oil & Gas.


Contact us to see how we’re applying Machine Learning in the Oil and Gas industry.