Method and apparatus for cross domain and cross-layer event correlation

Description

BACKGROUND

As is known in the art, cloud computing systems, even in pre-architected and pre-qualified environments, contain a relatively large number of hardware devices and components and software applications, modules, and components. In the presence of a fault, alert, or other condition needing attention, it can be difficult to identify the source of the fault or alert since there are many complex components that may be provided by multiple vendors which may make it difficult to correlate information in an efficient manner.

For example, in a cloud computing environment, alerts and events from various event sources in platforms normally contain limited information that may not be meaningful and may seem unrelated to the environment from which they originate. It is challenging for IT personnel to extract executable data from the alerts and take appropriate action.

With large volumes of alerts/events constantly coming from various sources, it is time consuming to troubleshoot all of them all any prioritization. It is challenging to prioritize the alerts/events and take appropriate actions without correlating them and knowing which of the alerts or events are root causes and which are just symptoms. In addition, many of the IT resources are managed in silos by IT personnel specialized in certain technology domains. For example, when a blade in the Cisco Unified Computing System (UCS) fails or has performance issues its impact propagates to the ESX server deployed on the blade, to the virtual machines deployed on the ESX server, to the applications or critical services running on those virtual machines, to the critical business that relies on those services. It may take hours or even days to sort through those alerts or events, which may result in significant detrimental impact on an enterprise.

SUMMARY

In one aspect of the invention, a method comprises: in a cloud computing system having system domains including a storage domain, a network domain, a compute domain, and a virtual layer: receiving a first alert in a first domain of the system domains and a second alert in a second domain of the system domains, the first and second alerts caused by an event, determining a topology of the system including physical and logical connections for components in the storage, network, compute domains, and the virtual layer, performing, using a computer processor, connectivity matching to identify connections between a port in the first domain and a port in the second domain, and performing identify matching, using unique identifiers for domain components, to determine that a first component in the first domain associated with the first alert is the same component as a second component associated with the second alert in the second domain for cross domain event correlation.

The method can further including one or more of the following features: the first component comprises a storage volume in the storage domain, the first component comprises a port in the network domain for a switch component, the first component comprises a cable connected to a fabric in the compute domain, the first comprises a server in the virtual layer, querying the first alert, the unique identifiers include a WWPN or a UUID, and/or the first alert includes one of a server (blade or ESX) down or connection lost, a storage volume error, heartbeat timeout between a server and a datastore, a zoneset being deactivated, a port failure, a link down, a host adaptor down, a virtual circuit down, a SAN port down, a Fibre Channel link down, and/or a virtual interface down.

In another aspect of the invention, an article comprises: a computer readable medium containing non-transitory stored instructions that enable a machine to perform, in a cloud computing system having system domains including a storage domain, a network domain, a compute domain, and a virtual layer, receiving a first alert in a first domain of the system domains and a second alert in a second domain of the system domains, the first and second alerts caused by an event, determining a topology of the system including physical and logical connections for components in the storage, network, compute domains, and the virtual layer, performing, using a computer processor, connectivity matching to identify connections between a port in the first domain and a port in the second domain, and performing identify matching, using unique identifiers for domain components, to determine that a first component in the first domain associated with the first alert is the same component as a second component associated with the second alert in the second domain for cross domain event correlation.

The article can further include one or more of the following features: the first component comprises a storage volume in the storage domain, the first component comprises a port in the network domain for a switch component, the first component comprises a cable connected to a fabric in the compute domain, the first component comprises a server in the virtual layer, querying the first alert, the unique identifiers include a WWPN or a UUID, and/or the first alert includes one of a server (blade or ESX) down or connection lost, a storage volume error, heartbeat timeout between a server and a datastore, a zoneset being deactivated, a port failure, a link down, a host adaptor down, a virtual circuit down, a SAN port down, a Fibre Channel link down, and/or a virtual interface down.

In a further aspect of the invention, a cloud computing system having system domains including a storage domain, a network domain, a compute domain, and a virtual layer, the system comprising: a processor, and a memory containing stored instructions to enable the processor to: receive a first alert in a first domain of the system domains and a second alert in a second domain of the system domains, the first and second alerts caused by an event, determine a topology of the system including physical and logical connections for components in the storage, network, compute domains, and the virtual layer, perform connectivity matching to identify connections between a port in the first domain and a port in the second domain, and perform identify matching, using unique identifiers for domain components, to determine that a first component in the first domain associated with the first alert is the same component as a second component associated with the second alert in the second domain for cross domain event correlation.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of this invention, as well as the invention itself, may be more fully understood from the following description of the drawings in which:

FIG. 1 is a high level schematic representation of a cloud computing system having cross domain event correlation in accordance with exemplary embodiments of the invention;

FIG. 2 is a schematic representation showing further detail of a pre-architected and pre-qualified cloud computing system of FIG. 1 including interconnections;

FIG. 3 is a schematic representation showing further detail of a pre-architected and pre-qualified cloud computing system of FIG. 2 including system components;

FIG. 4 is a schematic representation showing further detail of a storage domain of the cloud computing system of FIG. 3 using NAS for the storage domain;

FIG. 5 is a schematic representation of a unified infrastructure manager (UIM) module showing component layering;

FIG. 6 is a tabular representation of alerts caused by a single failure in a pre-architected and pre-qualified cloud computing system;

FIG. 7 is a schematic representation of cross-domain event correlation;

FIG. 8 is a schematic representation of an exemplary system having domains and layers;

FIG. 9 is a schematic representation of components in the event correlation engine in the UIM/Operations product;

FIG. 10 is a schematic representation of a topology stitcher receiving information from components in the system;

FIG. 11 is a schematic representation of output information from the topology stitcher;

FIG. 11A is a schematic representation of the flow of event correlation processing implemented in the UIM/Operations product;

FIG. 12 is an exemplary display of cross-domain alert information;

FIG. 13 is an exemplary display of alert causality and root cause;

FIG. 14 is a schematic representation of an exemplary computer that can perform at least some of the processing described herein.

DETAILED DESCRIPTION

FIG. 1 shows an exemplary cloud computing environment 100 having cross domain event correlation in accordance with exemplary embodiments of the invention. The environment includes an infrastructure layer 110 and a virtual layer 108. The infrastructure layer 110 is further divided into compute domain 102, a network domain 104, and a storage domain 106. The environment may be referred to as a platform. It is understood that any practical number of platforms can be combined into a cloud computing environment.

The compute domain 102 comprises components, such as blade servers, chassis and fabric interconnects that provide the computing power for the platform. The storage domain 106 comprises the storage components for the platform. The network domain 104 comprises the components that provide switching and routing between the compute and storage domains 102, 106 within and between platforms, and to the client or customer network.

FIG. 2 shows further detail for the environment 100 of FIG. 1. The storage domain 106 can include storage components 150, such as CLARIION storage components from EMC Corporation of Hopkinton Mass. The network domain 104 can include a pair of switches 152, such as MDS 9000 Series Multilayer SAN Switches from Cisco of San Jose, Calif., coupled to the storage components and to a LAN. The compute domain 102 can include a pair of fabric interconnects 154, such as CISCO 6100 series devices. The compute domain can further include a number of blade servers 156, such as CISCO 5100 blade chassis.

FIG. 3 shows further detail of an exemplary cloud environment having a compute domain 302, a network domain 304 and a storage domain 306. The network domain 302 is coupled to a customer network 308 in a manner known in the art. The network domain 302 includes switches 310 coupled to the customer network 308. The network domain 302 also includes multilayer fabric switches 312 coupled to fabric interconnects 314 in the compute domain 302 and to storage processors 316 in the storage domain 306. The fabric interconnects 314 are coupled to blade server chassis 318 containing blades. Data movers 320 in the storage domain 306 are coupled between the storage processors 316 and the switches 310 in the network domain. Disk array enclosures 322 are coupled to the storage processors 316. FIG. 4 shows interconnections for a system similar to that shown in FIG. 3 with physical disks 326. In the illustrated embodiment, the storage domain includes 8 to 16 front end fibre channel ports and 2-4 GB iSCSI front end ports. It is understood that a variety of other configurations having different interconnections and storage configuration can be provided to meet the needs of a particular application. The virtual layer can include a number of applications to perform various functions for overall control, configuration, etc of the various platform components. For example, virtual layer applications can include a virtualization function, such as VSPHERE/VCENTER, by VMware of Palto Alto, Calif.

Each of the infrastructure domains and the virtual layer is managed by a separate element management system. One of such element management systems is the Unified Computing System (UCS) Manager by Cisco. It is understood that the blade chassis and fabric interconnection can be considered part of the UCS. Another management application can include a management interface, such as EMC UNISPHERE, to provide a flexible, integrated experience for managing existing storage systems, such as CLARIION and CELERRA storage devices from EMC. A further management application includes a platform element manager, such as Unified Infrastructure Manager (UIM) by EMC, for managing the configuration, provisioning, and compliance of the platform.

FIG. 5 shows an exemplary unified infrastructure manager 500. In one embodiment, the unified infrastructure manager 500 includes a configuration center module 502, a provisioning center module 504, and an operations center module 506 having cross domain event correlation in accordance with exemplary embodiments of the invention. Below these modules are a platform infrastructure service catalog 506 and a cross domain context and visibility module 508.

The unified infrastructure manager 500 further includes a change and configuration management module 510, a policy-based compliance and analysis module 512, a unified infrastructure provisioning module 514, a consolidation topology and event service module 516, and an operational awareness module 518. The various modules interact with platform elements, such as devices in compute, network and storage domains, and other management applications.

The unified infrastructure manager 500 performs platform deployment by abstracting the overall provisioning aspect of the platform(s) and offering granular access to platform components for trouble shooting and fault management.

In one aspect of the invention, a uniform infrastructure management module includes a cross domain event correlation module to process alerts from physical infrastructure, logical services, virtual applications, and tenant/organizations. It is understood that cloud environments can provide storage for multiple tenants/organizations.

For example, a Vblock leverages VMware, Cisco, and EMC packages with integrated computing, network, storage, and management capabilities. Each of the domains has its own management system. When a component, for example, fails, multiple domain management systems may report an alert indication representing the same failure. For example, when a UCS port fails, a fault is triggered in the UCS. At the same time, vCenter also reports an alarm on the corresponding VMNIC. These two events indicate the same failure. To correlate duplicate events across different domains, the affected objects are matched using unique identifiers. In one embodiment, the following is used to perform matching and event correlation: UCS/vHBA and vSphere/vmhba using WWPN; match UCS/FCPort and MDS/FCPort using WWPN; match UCS/Blade and vCenter/ESX using UUID; match Storage/Volume and vCenter/Datastore using UUID; and matching StoragePort and vCenter/ScsiTarget using WWPN.

It is understood that various vendor specific terminology, product name, jargon, etc., may be used herein. It is further understood that such vendor specific information is used to facilitate an understanding of embodiments of the invention and should not limit the invention in any way. Any specific vendor information should be construed mean a generic product, function, or module.

Some exemplary items are set forth below. It is understood that one of ordinary skill in the art is familiar with the generic architecture and functionality of a vendor specific terms.

UIM/Provisioning or UIM/P: EMC Unified Infrastructure Management/Provisioning that provides simplified management for VCE VBLOCK by managing the components of VBLOCK platforms as a single entity and easily define and create infrastructure service profiles to match business requirements.

Cisco UCS: Cisco Unified Computing System.

VMWARE VSPHERE: A virtualization platform for building cloud infrastructures

ESX/ESXi: An enterprise-level computer virtualization product offered by VMware.

VM: Virtual Machine

VBLOCK: A pre-architected and pre-qualified environment for virtualization at scale: storage, fabric, compute, hypervisor, management and security.

WWPN: World Wide Port Name

UUID: Universally Unique Identifier

HBA: Host Bus Adapter

NIC: Network Interface Card

FC: Fiber Channel

The cloud computing environment is broken up into layers: infrastructure (or physical) layer and the virtual layer. In exemplary embodiments, the infrastructure layer is divided into three domains (silos): compute, network, storage, because traditionally they are managed separately with respective element management systems.

FIG. 6 shows an exemplary representation 600 of a port failure in a SAN component listing the domain 602, raw alert 604, alert count 606, and correlation 608, which are described below in detail. In the SAN switch domain 610, there is a link down 612, which is the root cause 614 of the alert as shown in the correlation column. The alert count 616 is one.

The UCS domain 620 lists a series of raw alerts 622 generated by the SAN port failure in SAN switch domain. As can be seen, the UCS alerts 622 include a FC link down, a host HBA down, a virtual interface down, and a virtual circuit down for a count of 13 in the UCS domain. The vCenter domain 624 lists two raw alerts 626 with a total count of 23 alerts (10+13). A total of 37 alerts are generated in three domains due the root cause SAN port failure.

FIG. 7 shows an exemplary representation of cross domain event correlation for root cause events generated in a network domain NL, compute domain CL, storage domain SL, and virtual layer VL. The root causes have respective symptoms. For example, a host HBA down in the compute domain CL has as a symptom a virtual circuit down. While a domain management system may correlate events within that domain, cross-domain event correlation correlates events across the various domains, as described more fully below.

FIG. 8 shows cross domain matching with connectivity and identity of components. The exemplary system 800 includes a storage domain 802, a network domain 804, a compute domain 806 and a virtual 808. The system is shown having a virtual layer 810, a logical layer 812, and a physical infrastructure layer 814.

Each of the storage 802, network 804, compute 806 domains and the virtual layer is managed by respective element management systems. In an exemplary embodiment, cross-domain correlation establishes relationships between objects in different domains or layers, as shown in the dash-lined boxes.

The storage domain 802 comprises a LUN 810 and a storage system 812 having first and second controllers 814a,b and disk arrays 816. The network domain 804 comprises first and second switches 818, 820 coupled between the storage domain 802 and the compute domain 806, which includes first and second fabric switches 822a,b and a UCS blade 824. The virtual layer 808 comprises an ESX server 826, a virtualization application 828, such as VM, and a datastore application 830. As can be seen, the components, domains and layers are connected by physical, logical and transaction connections. For example, physical connections between ports are shown between the first UCS fabric switch B 822a and the first network switch 818. A cross-domain logical connection is shown between the ESX server 826 and the storage LUN 810. A virtual connection is shown between VM 828 and the datastore 830.

In one particular embodiment, first and second categories of alert matching include connectivity matching and identity matching. For connectivity matching, a port 840 in the UCS fabric 822a is connected to a port 842 in the MDS switch via a cable. If the MDS reports that port 842 is down, then it is very likely the UCSM 806 will also report the connected UCS port 840 down. The system stores port-to-port connectivity for the components in the domains.

For identity matching, the same storage volume is represented in both the storage system and the vCenter/vSphere environment, for example. If there is a failure on the storage volume 810, the management systems for the storage domain 802 and the virtual layer 808 will report an alert. The two alerts in the different domains are reconciled to determine the root cause, as described more fully below.

FIG. 9 shows an exemplary implementation of the cross-domain event correlation in accordance with exemplary embodiments of the invention. An event correlation engine/module 900 includes a topology repository module 902 to store the infrastructural topology objects of all domains and their connections, the virtual layer objects, and their relationships with the infrastructural objects. The schema of the topology objects are driven by a predefined model 904. A codebook module 906 takes the repository data (topology, relationship, status) as the input and calculates the causality relationships between the events or alerts.

A topology importer module 910 obtains topology information for the system, such as the UCS, storage, network domains and the virtual layer, for transmission to the topology repository module 902. The topology importer creates and maintains topology objects and intra-domain relationships by periodically querying data sources and instantiating objects. A topology stitcher module 912 creates cross-domain relationships, as described more fully below. In general, the topology stitcher 912 builds relationships among cross-domain objects, which enables the correlation engine 900 to correlate alerts from different domains.

State updates are performed based upon the received alerts by mapping an alert to a state of a topology object. A so-called “event” is triggered by a certain condition of the state, and the correlation engine 900 determines the relationships among the events based on the model 904 and the topology relationships instantiated by the topology importer 910 and stitcher 912.

The causality output 920 from the topology repository 902 enables the system to fetch events, and then for each event, to query the related events. For example, if event A causes event B, then B will be returned in the list of getExplainedBy(A).

FIG. 10 shows further detail for an exemplary topology stitcher 1000. The topology stitcher 1000 examines objects and their unique identifications from different domains. The input from vCenter 1002 to the topology stitcher, for example, is the UUID (Universally Unique IDentifier) of ESX and Datastore, as well as the WWPN (World Wide Port Name) of ScsiTarget and vmhba. The UUID and the WWPN are used as unique identifiers for the topology stitcher to match related objects from other domains.

Similarly, input from the UCS 1004 includes the WWPN for the FC ports, the WWPN for the vHBAs, and UUID for the blade. Input from the UCS 1006 includes the peer WWPN for FC ports and the WWPN for the zones. Input from the storage domain 1008 includes the UUID for the storage volumes and the WWPN for the storage ports.

In one embodiment, the input and the matching criteria are defined in an XML file, so that newly identified matching criteria can be easily added without code changes for the topology stitcher to perform its function. Below is an example:

<ScsiTarget>

. . .

<matching targetClass=“StoragePort” relationship=“LayeredOver”

isRelationshipset=“true”>

<property selfProperty=“WWPN” propertyInTarget=“WWPN”/>

</matching>

</ScsiTarget>

This section of data instructs the topology stitcher to do the following: For objects in class ScsiTarget, find objects in StoragePort that have the same WWPN, and create a relationship called “LayeredOver” between them. In other words, a matching criteria is characterized by the following elements:

• • classA
  • classB
  • A list of matching property pairs {propertyA1, propertyB1; propertyA2, propertyB2; . . . }
  • relationshipName
  • isRelationshipSet—indicates whether the relationship is a one-to-one or one-to-many relationship

FIG. 11 shows exemplary topology stitcher outputs. In one embodiment, the topology stitcher 1100 performs 7 matches, resulting in 5 identity groups of classes shown in the diagram. The 7 pairs of matches, counter-clockwise starting from the left, are as follows:

• • Matching UCS/Blade and vCenter/ESX using UUID 1102
  • Matching UCS/FCPort and MDS/FCPort using Peer WWPN 1104
  • Matching MDS/FCPort and Storage/Port using Peer WWPN 1106
  • Matching Storage/Port and vCenter/ScsiTarget using WWPN 1108
  • Matching Storage/Volume and vCenter/Datastore using UUID 1110
  • Matching UCS/vHBA and vCenter/vmhba using WWPN 1112
  • Matching MDS/Zone and UCS/vHBA using WWPN 1114

Exemplary pseudo code for generating matches is set forth below:

for each matching criteria {

• • for each object objB in classB {
    • generate a string key that concatenate the values of all matching properties: key=objB.propertyB1.value+delim+objB.propertyB2.value+ . . . ;
    • save the object in a hash map, using the generated key: bMap.put(key, objB);
  • }
  • for each object objA in classA {
    • generate a string key that concatenate the values of all matching properties: key=objA.propertyA1.value+delim+objA.propertyA2.value+ . . . ;
    • lookup bMap using the key to see if matches are found: matchedObjBSet=bMap.get(key);
    • for each objB in matchedObjBSet {
      • if (isRelationshipSet) {objA.relationshipName.insert(objB);}
      • else {objA.relationshipName=objB;}
  • }

}

FIG. 11A shows an exemplary sequence of steps for providing cross-domain event correlation. In step 1150, topology data is retrieved from various infrastructure domains and the virtual layer. In step 1152, connectivity data is examined. For example, the port-to-port connectivity for the various components in various domains is examined. In step 1154, connectivity matching of corresponding components from various domains is performed.

In parallel, in step 1156, the objects are examined, regardless of whether there is alert, along with the unique identifiers from the domains. In step 1158, identity matching is performed based on the unique identifiers. In 1160, corresponding relationships are created based on the matches for cross-domain event correlation, as shown in FIG. 11.

In step 1170, an event in the system, such as a link failure, generates alerts in multiple domains that are received by the management domain. In step 1172, the alerts are mapped to object states in the topology store. In step 1180, relationships of alerts are made consumable by users.

FIG. 12 shows an exemplary screen display 1200 of an event correlation result between a StorageVolume alert 1202 and a Datastore alert: 1204. The Storage Volume alert 1202, called StorageVolume-Error, indicates the Storage Volume operational state is Error. The Datastore alert 1204, called EsxProblemVmfsHeartbeatTimeout, indicates the datastore is no longer reachable by the ESX. It is understood that the storage volume alert 1202 corresponds to the root cause of the alert.

FIG. 13 shows an exemplary screen display 1250 of a further event correlation result including the first level of the causality tree under the link down alert of an MDS/FCPort. This event correlates the following alerts:

• • The link down alert 1252 of two UCS/vHBA interfaces since they share the same zone with the MDS/FCPort, shown by the first two lines under the Causes node;
  • The link down alert 1254 of the matched UCS/FCPort, shown by the third line under the Causes node.

The following table illustrates the cross domain identity matching between the storage volume or LUN on storage arrays and the datastore in VMware vSphere

FIG. 14 shows an exemplary computer that can perform at least a part of the processing described herein. A computer includes a processor 1302, a volatile memory 1304, an output device 1305, a non-volatile memory 1306 (e.g., hard disk), and a graphical user interface (GUI) 1308 (e.g., a mouse, a keyboard, a display, for example). The non-volatile memory 1306 stores computer instructions 1312, an operating system 1316 and data 1318, for example. In one example, the computer instructions 1312 are executed by the processor 1302 out of volatile memory 1304 to perform all or part of the processing described above. An article 1319 can comprise a machine-readable medium that stores executable instructions causing a machine to perform any portion of the processing described herein.

Processing is not limited to use with the hardware and software described herein and may find applicability in any computing or processing environment and with any type of machine or set of machines that is capable of running a computer program. Processing may be implemented in hardware, software, or a combination of the two. Processing may be implemented in computer programs executed on programmable computers/machines that each includes a processor, a storage medium or other article of manufacture that is readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and one or more output devices. Programs may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the programs may be implemented in assembly or machine language. The language may be a compiled or an interpreted language and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. A computer program may be stored on a storage medium or device (e.g., CD-ROM, hard disk, or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform processing.

One skilled in the art will appreciate further features and advantages of the invention based on the above-described embodiments. Accordingly, the invention is not to be limited by what has been particularly shown and described, except as indicated by the appended claims. All publications and references cited herein are expressly incorporated herein by reference in their entirety.