Insight

Chevron’s Supply Chain Cloud Transformation

Chevron Corporation’s (CVX), global refining system manufactures fuels and other products sold by the company’s marketing, lubricants, and supply and trading organizations under the Chevron®, Texaco® and Caltex® brands. These refining systems are capable of processing heavy crude oils and producing a variety of high-value products, including transportation fuels, chemicals, and lubricants. This value chain supports 2.4 million barrels/day worth of annual products sold by seven refineries across a network of 8,000 retail stations.

The Commodity Supply Chain Management (CSCM) Data and Insights team is responsible for managing more than 200 data pipelines to ingest data from various internal and external sources and standardize it in the Supply Chain Data Foundation (SCDF). SCDF is central to driving data integration and related initiatives across the many functions responsible for managing the flow of crude oil and refined products, including:

• replenishment planning and scheduling
  
• inventory management
  
• price and demand forecasting
  
• contract planning
  
• product quality and blending
  
• margin analysis
  
• common master data

The Chevron team envisioned that the migration to Azure could go beyond the Company’s existing data capabilities to enable aspirational and advanced capabilities, such as

• improved operational efficiency
• improved, agile business decision making
• end-to-end profitability

The Publicis Sapient Product & Data Engineering team partnered with Chevron CSCM to build a new Platform as a Service (PaaS) solution in Azure to deliver on this vision.

The Why: The Benefits of Data in the Cloud

Migrating the data foundation to Azure has substantially:

• Minimized support and disruption costs with a streamlined, fit-for-purpose architecture
• Improved capability to enhance and scale the data platform much more rapidly than the legacy on-premise system
• Enabled future advanced capabilities in Azure including artificial intelligence (AI), streaming/device (IoT) data, and management of unstructured data (PDFs, emails, legal)
• Improved ability to develop, test, and deploy changes quickly via modern DevOps practices

It gave more than 400 manufacturing and value chain optimization users access to:

• All supply chain data integrated into a central, Chevron-wide instance of a data lake to enable cross-department collaboration
• Self-service BI to allow citizen users seamless data exploration and analysis, improving business insight generation
• Supply Chain apps consume data more efficiently from the platform leading to fewer disruptions and an enhanced user experience

More than 200 data integration jobs were converted from the previous solution to Azure Data Factory (ADF). The sequencing of the delivery of these had to be carefully managed due to the internal dependencies of the data platform.
However, to enable delivery and a consistent pattern, multiple frameworks were developed early in the project to facilitate:

• Massive parallel processing using Synapse dedicated pools
  
• Parameterized pipelines to enable re-use for common data patterns
  
• Change data capture framework ensuring management of data change
  
• High performance data warehouse load using polybase connector
  
• Python unit case (Pytest) for scalable, automated test cases saving effort and improving testing quality
  

While the requirements of the timeliness of data depend on the part of the supply chain being supported, critical 24/7 operations like refinery, blending, and scheduling have more strict data needs. In addition, proper design and management of resources in the cloud is crucial to ensure excess Azure costs are not incurred. Major components include:

• Databricks pool cluster is provided to critical business functions on-demand to eliminate typical cloud spin-up time
  
• Databricks multithreading and Spark integration to handle massive tables efficiently
  
• Polybase load to data warehouse (Synapse) to optimize writing to the Synapse data warehouse
  
• Table distribution and results set caching to optimize large read operations on the data warehouse
  
• Workload management to classify the importance and prioritization of certain jobs
  

In addition to storing the data, the data furthermore needed to be transformed and served to the business function needing the data. Streamlined access and proper preparation of the data are crucial to keep the data consumers from experiencing disruption as the system migrates from on-premise to the cloud. Major components include:

• Automated DB database refreshes enabled by DevOps to reset the data to a neutral state to enable agile unit testing and structured QA script execution
• Data quality engine enabled by rules defined in tables, executed in ADF, and reported in Power BI
• Azure Web APIs developed to enable consumers to consume the data efficiently and on their own terms

In less than one year, the Publicis Sapient delivery and Chevron CSCM Platform teams were tasked to migrate the following components to Azure:

• 200 data integration pipelines to Azure Data Factory/Databricks/ LogicApps
• 400 tables modelled and migrated to Azure Synapse
• 450 stored procedures and queries to Synapse
• Conversion of these statements to end-state architecture was facilitated by Microsoft Tradurre, a Microsoft product not yet publically available.
• A data quality engine with 400 rules to Synapse/ADF
• 20 sets of reports migrated to Synapse integrated Power BI Reports

With traditional project management techniques, the SCDF project would have taken years.

However, the Publicis Sapient team, partnering with the larger Chevron platform team leveraging Microsoft workflow and integration tools like Azure DevOps (ADO), combined the following practices to enable a successful project completion, all in less than one year.

Given the size and scope of the project, the coordination required between the Publicis Sapient delivery team and the larger Chevron CSCM platform team was a significant endeavor. Azure DevOps was used to facilitate both internal iteration planning, in addition to managing cross-team dependencies.

More than 20 supply chain application teams were needed to test, plan, and migrate their integrations to the new platform simultaneous with the decommissioning of the legacy on-premise platform. The delivery team embraced Chevron’s newly deployed Scaled Agile process within the CSCM platform in ADO to plan out quarterly PI increments, map dependencies, and monitor execution via ADO dashboards and progress reports to ensure delivery adhered to the schedule.

Agile Program Management and modern DevOps practices go hand-in-hand; one is essential to enable the other and neither are successful independently. While many believe DevOps is a technology or a skillset, the Publicis Sapient delivery team embraced DevOps as a mindset to enable the rapid development, integration, testing, and deployment cycle necessary to meet tight timelines. Without this DevOps mindset, enabled by ADO, Azure Repos, and Azure Automated Deployment Pipelines, the delivery of the platform in less than a year would have been unrealistic.

The new platform was deployed in production using Microsoft Project to coordinate the cutover,” a carefully sequenced set of steps to ensure that the project is deployed simultaneously with all data publishers and consumers reconnecting to the new platform.

Remarkably, the team was able to deliver the platform on time at high quality, despite a surge in the Delta variant of COVID-19 during 2021 which, at times, reduced total team capacity by 30 percent as team members took off for illness and to care for their families .