ENERGY INTELLIGENCE: detecting new revenue opportunities

  • 508

    ENERGY INTELLIGENCE: detecting new revenue opportunities

    As energy merchants face tighter profitability, they need to look beyond reduced cost to efficiency and automation and explore innovative ways to identify new revenue opportunities around their current portfolio. In this article, Rashed Haq, Abhishek Bhattacharya and Charles Ford discuss approaches for harnessing emerging information services and technology capabilities to more systematically detect new revenue opportunities.


    Over the last decade, it has become harder for many energy merchant organizations to continue growing their bottom line. The economic collapse of 2008, followed by the energy renaissance in the US that caused tectonic shifts in supply, demand and transportation, has made it more difficult for firms to discover new arbitrage opportunities using legacy methods. The problem is compounded by the fact that arbitrage signals tend to be shorter lived, with more constraints to consider in terms of markets and logistics, and potentially have smaller margins.

    As such, to maintain a competitive edge, energy companies that are engaging in trading, whether they are producers, consumers, integrated or merchants, will need to look at more information at a deeper granularity and with faster turnaround to capture the value from potential opportunities. We have labeled this new approach to growing the bottom line: Energy Intelligence.


    The core concept behind Energy Intelligence is to rapidly detect market events that can lead to new trading opportunities and evaluate their effects on near-term market operations by developing a better qualitative and quantitative understanding of the impacts. This enables the next incremental revenue and margin gain without significantly growing the organizational or asset footprint.

    An example of how Energy Intelligence helps a global petroleum merchant (GPM Co.) is depicted in Figure 1. GPM Co. has a title for a vessel of gasoline in the US Gulf Coast and a commitment to deliver it at New York Harbor (solid blue line in Figure 1), ignoring complexities of some physical constraints and specification differences across regions. The company hears about an event—a refinery on the West Coast has shut down due to a fire. GPM believes that this will create a temporary price hike in California and would like to take advantage of that by diverting its vessel to the West Coast (dotted blue line in Figure 1). But before it can make this call, it wants to be sure that there will likely be a price hike in California, and that it would still be able to cover its obligation at New York Harbor without losing all the gains from the West Coast. To rapidly evaluate these considerations, GPM looks at its intelligence map which shows who has ships where, what’s on board these ships and when they will reach their destination. Specifically, GPM would like to see:

    • If there are other vessels destined for California, or could be diverted to California, within a three-day window with a similar grade of gasoline (dotted red line on left side of Figure 1)—if not, then there will more likely be a temporary price increase.
    • If there are vessels that are available to acquire and direct towards New York Harbor (dotted red line on right side of Figure 1) within reasonable delivery ranges, pricing ranges and potential demurrage impacts.

    This information can be used to very quickly build a model for the specific date and price ranges that would work to monetize the situation. It can also be used to short-list the companies or groups to contact externally with the relevant requests. This information allows GPM to gain a better qualitative and quantitative understanding of the near-term markets and how it should respond. Without the rapid and relevant compiling of information, GPM would not have been able to capitalize on the event because it was short lived and required immediate action.

    Figure 1: Intelligence Map—A Graphical View of Relevant Information for the Illustrative Scenario.

    The assumption has long been that most events are highly visible, such as in this example. But as companies are pulling together their Energy Intelligence information, they are discovering that events, both large and small, are happening daily. The main concern is whether firms can see all of them, identify the impacts and decide whether to take advantage of them. Only companies with a robust Energy Intelligence capability have this visibility and insight.

    Another scenario where Energy Intelligence has been important is in the growing significance of ethanol in the North American energy market. As an example, consider a company that has entered into a long-term, fixed-price sale of ethanol that it would like to hedge. Without a consistently reliable and correlated hedge for ethanol available, the trader wanted to use several different commodity instruments to manage the risk—including corn futures, sugar futures, natural gas futures and gasoline futures. To manage the ethanol price risk, the trader needed to buy and sell the various instruments over time, either in combination or individually, based on when and how well they correlated to ethanol prices. To understand the instruments’ effectiveness as a hedge, the trader needed access to a variety of other data inputs and the ability to quickly analyze their impact on the relationships between ethanol prices and the available hedge instruments. The other data inputs needed included current and weather forecasts, gasoline and ethanol inventories, spot ethanol prices and any potential information gleaned from news and social media feeds. Today, a trader might hedge the long-term, fixed-price ethanol contract with a single, poorly correlated instrument or perhaps go through periods where the trade is not hedged at all. With an increased set of hedge instruments available, the trader has the ability to better understand the changing relationships between ethanol, the hedge instruments and market factors and can potentially make better hedge decisions over time.

    Energy Intelligence helps the trader to organize and analyze large amounts of structured and unstructured data to make consistently effective hedging decisions. With it, the trader will be better positioned to quickly recognize important data patterns and correlations—and be a first mover when reacting to market events. This is not only valuable in managing increasingly riskier trades, but also when analyzing trade structures prior to entering a complex trade. Today, a trader’s desktop tools might equip the trader to monitor several market factors consistently but not adequately enough to recognize data patterns across large amounts of structured and unstructured data—or to recognize the importance of market events. In addition, today’s trader desktop tools do not offer a way to visualize all the data but, rather, force the trader to interact with a number of screens to assess a relatively small data set and identify traditional patterns limited to that data set. By viewing a larger set of related data, traders will have the ability to recognize and understand patterns and correlations between ethanol prices with other commodity instruments—and ultimately better manage the risk of a long-term, fixed-price ethanol trade. This capability is becoming increasingly important as complex, structured trades are becoming more commonplace in today’s commodity trading landscape.

    While the examples above are specific and simplified for illustrative purposes, the concept of Energy Intelligence can be useful for any energy company involved in bulk commerce, whether they are producers, consumers or merchants.

    Figure 2: Energy Intelligence Factors to Consider.


    Typically, when a company is looking at external markets (i.e., commercial information outside of the company’s proprietary assets and contracts), it can do long-term fundamental analysis in terms of both supply and demand and also forward views of prices. A number of factors may be included in fundamental analysis models. These factors (some of which are shown in Figure 2) tend to be very long-range projections and forecasts, and the outcome of the fundamental analysis is driven by the accuracy of these input factors. The accuracy of these forecasts drops dramatically the further out in time they are.

    To build an Energy Intelligence capability, firms must bring the fundamental analysis to the immediate and near-future using specific intelligence factors and associated data and models. The shorter time range increases the confidence level on accuracy sufficiently for the company to act on the results. The overall business process for this approach includes four key steps, as shown in Figure 3.

    Figure 3: Business Process Stages for Energy Intelligence.

    The first step is to gather all the data that is of interest and that relates to the relevant intelligence factors. Gathering includes finding the data sources, acquiring data through data service agreements, performing any validations and formatting necessary and storing the data. Eventually, this data will be used for analytics, predictive model building and back-testing these models.

    The next step is to collate all the information that was gathered and prepare it for analysis. This includes data cleansing, correlation, mapping and aggregation. Correlation is critical to data analysis and is done to ensure that event times are comparable—and products and locations match between different data sources. Data collation and aggregations have to address the key problems of volume, velocity and variety—the 3 Vs of BigData.

    Once the information is ready, quants and analysts are able to commercially analyze the data. Generally, this will start with some research and formulating hypotheses of where opportunities may lie—or simply mining the information gathered to explore opportunities. Then, a relevant model may be built that will show the “signal” from any large or small event. This model should be back-tested with historical data to validate that the signal can be discovered and to determine whether it will be profitable, and within what thresholds.

    The final step is for users to monitor for signals based on the previous analysis. This may be as rudimentary as preparing tabular reports or more sophisticated by using visualizations, such as a graph with data overlays allowing users to more quickly spot an opportunity. Adding complex event processing, as used in trade surveillance and other areas, could further enhance usability and speed. Over time, not only will more companies have visualization and complex event processing, but Energy Intelligence will evolve towards Artificial Intelligence. In fact, some companies are already exploring approaches such as these as a possible trigger for better signal detection which analysts then validate.


    The functional architecture required to support business processes for Energy Intelligence closely mirrors the different stages of the business process. A tiered architecture separates concerns, enabling the concurrent development, deployment and maintenance of the architecture. This is critical because new models and new data sources are frequently being added. The four tiers of the functional architecture for Energy Intelligence include: the Management Layer, Enrichment Layer, Insight Layer and Monitoring Layer—each of which consists of several components and functions.

    Figure 4: Functional Architecture for Energy Intelligence.

    The management layer provides a complete set of capabilities for data management across relational, non-relational and streaming data throughout the full data life cycle with the ability to:

    • Ingest several varieties of data, massive volumes of data and real-time data (or events)
    • Seamlessly move data from one type to another
    • Cleanse and refine data in a business context including eliminating redundancies, removing obsolete data, correcting inaccurate data and enriching missing data
    • Correlate datasets through master data
    • Address security and availability concerns

    The enrichment layer provides capabilities for aggregating and summarizing available datasets within the organization. The primary goal of the enrichment layer is to make the data ready for the insight layer, which includes preparing relevant analytical cubes. This layer must also address scale and performance concerns.

    The insight layer provides capabilities for advanced modeling and analytics with the ability to tag for actual and what-if datasets. The platform addresses fast processing, correlation and analytics, and real-time event-based processing.

    The monitoring layer provides capabilities for decision-making and collaboration capabilities to all end users through a variety of devices. Different types of users need different analysis and decision-making capabilities depending on the depth of analysis they need to understand the output from the insight layer—and on the speed with which they need to act on their information.


    A number of recent changes are encouraging energy companies to consider Energy Intelligence:

    • In general, people are becoming more analytics savvy. Analytics is becoming part of many people’s daily lives through the use of consumer electronics and associated applications, such as the iPhone with Nike+
    • Data gathering used to be a major hurdle, but an increasing number of data vendors are providing different aspects of the kind of information that is relevant for Energy Intelligence (e.g., BLOM, Spacetime Insights, Marwood Group)
    • Energy companies are looking at the significant successes that have materialized in other industries (e.g., customer intelligence in retail, automated trading in the equities and financial sector, using missile-guidance technology to move from 95% to 96% accuracy in sports, etc.)
    • Advances in technology are making it easier to build this kind of information capability. High-performance computing, for example, is becoming more available (e.g., parallel computing toolboxes on multicore CPUs or GPUs or GPU clusters and emerging BigData and analytics tools)

    To successfully build an Energy Intelligence capability, firms must:

    • Realize that not all events will be readily visible if they don’t actively look for them
    • Foster the collaborative analytics capability between traders, analysts, quants, schedulers and IT to detect and rapidly validate the signals
    • Build broad partnerships for information gathering and explore where signals may be found

    Companies currently leveraging Energy Intelligence capabilities are already gaining an advantage over their competition in terms of being able to make better near-term operational and commercial decisions in response to market changes.

    The Authors
    Rashed Haq

    Rashed Haq
    is Vice President and Lead for Analytics & Optimization for Commodities at Sapient Global Markets. Based in Houston, Rashed specializes in trading, supply logistics and risk management. He advises oil, gas and power companies to address their most complex challenges in business operations through innovative capabilities, processes and solutions.

    Abhishek Bhattacharya

    Abhishek Bhattacharya
    is a Director of Technology based in India and leads the Technology Capability at Sapient Global Markets. He helps trading organizations leverage innovative technology and analytics to discover, understand and capture new trading opportunities.

    Charles Ford

    Charles Ford
    is a Director at Sapient Global Markets based in Houston. Charles has 18 years of expertise identifying and implementing technology to drive innovations in the commodity trading space. Charles began his career as a refinery engineer and moved into consulting where he was instrumental in creating one of the first commercially available supply and trading platforms.

    Leave a Comment