Exchange message protocol message transmission between two devices

Description

CROSS-REFERENCE(S) TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. Provisional Application 61/980,599, filed 17 Apr. 2014. This U.S. Provisional Application 61/980,599 is hereby fully incorporated herein by reference.

This application claims the benefit of and priority to U.S. Provisional Application 61/980,602, filed 17 Apr. 2014. This U.S. Provisional Application 61/980,602 is hereby fully incorporated herein by reference.

This application claims the benefit of and priority to U.S. Provisional Application 61/980,594, filed 17 Apr. 2014. This U.S. Provisional Application 61/980,594 is hereby fully incorporated herein by reference.

This application claims the benefit of and priority to U.S. Provisional Application 61/980,579, filed 17 Apr. 2014. This U.S. Provisional Application 61/980,579 is hereby fully incorporated herein by reference.

This application claims the benefit of and priority to U.S. Provisional Application 61/981,188, filed 17 Apr. 2014. This U.S. Provisional Application 61/981,188 is hereby fully incorporated herein by reference.

This application claims the benefit of and priority to U.S. Provisional Application 61/981,195, filed 17 Apr. 2014. This U.S. Provisional Application 61/981,195 is hereby fully incorporated herein by reference.

FIELD

Embodiments of the invention relate generally to data storage systems. Embodiments of the invention also relate to transmissions of frames of an SSD across a transmission path.

DESCRIPTION OF RELATED ART

The background description provided herein is for the purpose of generally presenting the context of the disclosure of the invention. Work of the presently named inventors, to the extent the work is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against this present disclosure of the invention.

In a computer system, a data bus transfers data among and between components of the computer system. For example, data is transferred between a Central Processing Unit (CPU) of the computer and a memory or data storage.

Storage devices such as, for example, Solid State Drives (SSDs), can connect to an input/output (I/O) bus of a computer host. One example of such an I/O bus is a Peripheral Component Interconnect Express (PCIe) computer expansion bus or a similar type of computer bus. However, there is a continuing need to overcome the constraints or disadvantages of current conventional systems.

SUMMARY

Embodiments of the invention relate generally to data storage systems. Embodiments of the invention also relate to transmissions of frames of an SSD (solid state drive) across a transmission path.

Embodiments of the invention provide an apparatus and method on how to send SCSI-compliant (Small Computer System Interface compliant) frames of an SSD across a PCIe (Peripheral Component Interconnect Express) bus, when there is no standard protocol yet in PCIe buses for SSD communications.

An embodiment of the invention also provides an exchange message protocol (EMP) that is used to transfer data frames between a host-side and a memory device side (e.g., non-volatile memory device(s)).

In an embodiment of the invention, a method comprises: transmitting, by a host side, an exchange message protocol (EMP) command frame to a memory device side; informing, by the host side, the memory device side to process the command frame; executing, by the memory device side, the command frame; and transmitting, by the memory device side, an EMP response frame to the host side, in response to the command frame.

In another embodiment of the invention, an apparatus comprises: a host side configured to transmit an exchange message protocol (EMP) command frame to a memory device side; wherein the host side is configured to inform the memory device side to process the command frame; wherein the memory device side is configured to execute the command frame; and wherein the memory device side is configured to transmit an EMP response frame to the host side, in response to the command frame.

In yet another embodiment of the invention, an article of manufacture, comprises: a non-transient computer-readable medium having stored thereon instructions that permit a method comprising: transmitting, by a host side, an exchange message protocol (EMP) command frame to a memory device side; informing, by the host side, the memory device side to process the command frame; executing, by the memory device side, the command frame; and transmitting, by the memory device side, an EMP response frame to the host side, in response to the command frame.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one (several) embodiment(s) of the invention and together with the description, serve to explain the principles of the invention.

BRIEF DESCRIPTION OF DRAWINGS

Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the present invention may admit to other equally effective embodiments.

FIG. 1 is a block diagram of an example data storage system (or data storage apparatus) that can include an embodiment of the invention.

FIG. 2 is block diagram of an Exchange Message Protocol (EMP) command frame format, in accordance with an embodiment of the invention.

FIG. 3 is block diagram of an Exchange Message Protocol (EMP) response frame format, in accordance with an embodiment of the invention.

FIG. 4 is a block diagram that illustrates the usage of base address registers by a host and an endpoint PCI device such as, for example, an embedded disk card (EDC), in accordance with an embodiment of the invention.

FIG. 5 is a block diagram of command ring and a response ring, in accordance with an embodiment of the invention, wherein both rings are currently empty.

FIG. 6 is a block diagram of command ring and a response ring, in accordance with an embodiment of the invention, wherein both rings are currently full.

FIG. 7 is block diagram of a system that shows the interaction between an EMP Host and an EMP Port for EMP frame passing using rings, in accordance with an embodiment of the invention.

FIG. 8 is a flow diagram of a method in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

In the following detailed description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various embodiments of the present invention. Those of ordinary skill in the art will realize that these various embodiments of the present invention are illustrative only and are not intended to be limiting in any way. Other embodiments of the present invention will readily suggest themselves to such skilled persons having the benefit of this disclosure.

In addition, for clarity purposes, not all of the routine features of the embodiments described herein are shown or described. One of ordinary skill in the art would readily appreciate that in the development of any such actual implementation, numerous implementation-specific decisions may be required to achieve specific design objectives. These design objectives will vary from one implementation to another and from one developer to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine engineering undertaking for those of ordinary skill in the art having the benefit of this disclosure. The various embodiments disclosed herein are not intended to limit the scope and spirit of the herein disclosure.

Exemplary embodiments for carrying out the principles of the present invention are described herein with reference to the drawings. However, the present invention is not limited to the specifically described and illustrated embodiments. A person skilled in the art will appreciate that many other embodiments are possible without deviating from the basic concept of the invention. Therefore, the principles of the present invention extend to any work that falls within the scope of the appended claims.

As used herein, the terms “a” and “an” herein do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced items.

In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” (or “coupled”) is intended to mean either an indirect or direct electrical connection (or an indirect or direct optical connection). Accordingly, if one device is coupled to another device, then that connection may be through a direct electrical (or optical) connection, or through an indirect electrical (or optical) connection via other devices and/or other connections.

FIG. 1 is a block diagram of an example data storage system 100 (or data storage apparatus 100) that can include an embodiment of the invention. Those skilled in the art with the benefit of this disclosure will realize that an embodiment of the invention can be included in other suitable types of computing systems or data storage systems.

When the system 100 has initialized and is under normal operation, a software/program 101 (run by the processor requests for the SSD access), for example, will do a read transaction to read data from one or more non-volatile memory devices 102 in the flash storage module 103 or do a write transaction to write data to one or more non-volatile memory devices 102 in the flash storage module 103. Typically, the one or more memory devices 102 form a memory device array 104 in the flash module 103. The memory device array 104 is coupled via a flash interface 105 to a flash memory controller 106.

The flash storage module 103 is coupled via a flash bus 107 (or memory bus 107) to a Direct Memory Access (DMA) controller 108. The DMA controller 108 is coupled via a DMA bus interface 114 to a system bus 109.

A processor 110, system memory 111, and a software/program 101 (run by processor) are all coupled to the system bus 109. The system 100 can include more than one a software/program 101, more than one processor 110, and/or more than one system memory 111. Additionally or alternatively, the system 100 can include more than one DMA controller 108 and more than one flash storage module 103. In an embodiment of the invention that includes a plurality of flash storage modules 103 and a plurality of DMA controllers 108, wherein each flash storage module 103 is coupled via a respective flash bus 107 to a respective DMA controller 108, the plurality of flash storage modules 103 will form an array (not shown) of flash storage modules 103.

System bus 109 is a conduit or data path for transferring data between DMA controller 108, processor 110, system memory 111, and software/program 101. Processor 110, DMA controller 108, and software/program 101 may access system memory 111 via system bus 109 as needed. System memory 111 may be implemented using any form of memory, such as, for example, various types of DRAM (dynamic random access memory), non-volatile memory, or other types of memory devices.

A request 115 for a memory transaction (e.g., read or write transaction) from software/program 101, typically in the form of an input-output descriptor command, is destined for the processor 110. Descriptor commands are detailed instructions to be executed by an engine or a module. The processor 110 interprets that the input-output descriptor command intends to read from memory devices 102 in the flash storage module 103 or intends to write to memory devices 102 in the flash storage module 103. The processor 110 is in-charge of issuing all the needed descriptors to one or more Direct Memory Access (DMA) controllers 108 to execute a read memory transaction or write memory transaction in response to the request 115. Therefore, the DMA controller 108, flash memory controller 106, and processor 110 allow at least one device, such as a software/program 101, to communicate with memory devices 102 within the data storage apparatus 100. Operating under a program control (such as a control by software or firmware), the processor 110 analyzes and responds to a memory transaction request 115 by generating DMA instructions that will cause the DMA controller 108 to read data from or write data to the flash devices 102 in a flash storage module 103 through the flash memory controller 106. If this data is available, the flash memory controller 106 retrieves this data, which is transferred to system memory 111 by the DMA controller 108. Data obtained during this memory read transaction request is hereinafter named “read data”. Similarly, write data provided by software/program 101 will cause the DMA controller 108 to write data to the flash devices 102 through the flash memory controller 106.

A non-volatile memory device 102 in the flash storage module 103 may be, for example, a flash device. This flash device may be implemented by using a flash memory device that complies with the Open NAND Flash Interface Specification, commonly referred to as ONFI Specification. The term “ONFI Specification” is a known device interface standard created by a consortium of technology companies known as the “ONFI Workgroup”. The ONFI Workgroup develops open standards for NAND Flash memory devices and for devices that communicate with these NAND flash memory devices. The ONFI Workgroup is headquartered in Hillsboro, Oreg. Using a flash device that complies with the ONFI Specification is not intended to limit the embodiment(s) disclosed herein. One of ordinary skill in the art having the benefit of this disclosure would readily recognize that other types of flash devices employing different device interface protocols may be used, such as protocols that are compatible with the standards created through the Non-Volatile Memory Host Controller Interface (NVMHCI) working group. Members of the NVMHCI working group include Intel Corporation of Santa Clara, Calif., Dell Inc. of Round Rock, Tex., and Microsoft Corporation of Redmond, Wash.

Those skilled in the art with the benefit of this disclosure will realize that there can be multiple components in the system 100 such as, for example, multiple processors, multiple memory arrays, multiple DMA controllers, and/or multiple flash controllers.

FIG. 2 is block diagram of an Exchange Message Protocol (EMP) command frame format, in accordance with an embodiment of the invention. Exchange Message Protocol is a communication protocol used to transmit messages between two PCI devices. The EMP Host acts as the Root Complex PCI device while the EMP Port functions as the Endpoint PCI device. An EMP frame is defined exclusively to convey the message from the EMP Host to the EMP Port or vice versa. The message contains information of the SCSI command issued by the EMP Host and the command status posted by the EMP Port. The EMP frame (command frame and response frame) encapsulates all the necessary command information into one entity. For example, the EMP command frame 205 includes address information 210 of the EMP host which is the root complex PCI device that transmits an EMP command frame to the EMP port. The EMP command frame 205 also includes an SCSI command information 215 issued by the EMP host. This SCSI command information 215 includes the operation code which is executed by the EMP port to permit the EMP port to perform an operation according to the operation code.

As known to those skilled in the art, in a PCIe system, a root complex PCI device connects the processor and memory subsystem to the PCI Express switch fabric that comprises one or more switch devices, and similar to a host bridge in a PCI system, the root complex generates transaction requests on behalf of the processor which is interconnected through a local bus. Root complex functionality may be implemented as a discrete device, or may be integrated with the processor. A root complex may contain more than one PCI Express port and multiple switch devices can be connected to ports on the root complex or cascaded.

FIG. 3 is block diagram of an Exchange Message Protocol (EMP) response frame format, in accordance with an embodiment of the invention. The EMP response frame 305 includes the command status 310 posted by the EMP port which is the endpoint PCI device. An endpoint PCI device (e.g., device 410 in FIG. 4) transmits the response frame 305 in response to receiving a command frame 205 from a host (e.g., host 405 in FIG. 4).

FIG. 4 is a block diagram of a system 400 that illustrates the usage of base address registers by a host 405 and an endpoint PCI device 410 such as, for example, an embedded disk card (EDC), in accordance with an embodiment of the invention. Therefore, the host 405 can also be defined as a host side 405 in the system 400. As an example, a host side 405 can be defined as (or can include) any suitable device. As another example, the host 405 can be, for example, a host computer such as a host personal computer. Another example of the host 405 can be, e.g., the I/O device 101 in FIG. 1.

The endpoint PCI device 410 can be an EMP port 410 such as, for example, an embedded disk card (EDC). An EDC can include the flash storage module 103 in FIG. 1. Therefore, the EMP port 410 can also be defined as a memory device side 410 in the system 400.

As known to those skilled in the art, PCIe BAR mapping involves mapping a PCIe device into the memory-mapped address space (or other I/O address space) of the computer system. This mapping enables the computer system to address PCIe devices. As also known to those skilled in the art, drivers or an operating system can program the Base Address Registers (BARs). During system startup, PCIe devices are usually identified, numbered, and then addressed. The Peripheral Component Interconnect Special Interest Group (PCI-SIG) provides specifications for PCIe and other computer bus technologies.

In FIG. 4, BAR 0 415 is used as a memory window for the EMP Host 405 to access a first memory region 420 in the EMP Port 410, wherein the first memory region 420 contains the configuration settings 425 of the EMP port 410. Typically, the EMP Host 405 accesses the mailbox memory 450 through BAR 0 415.

The EMP Port 410 exposes the sets of registers 430. The EMP Port 410 includes a second memory region 435 that contains the sets of registers 430. The EMP Host 405 can control the behavior and can determine the current status of EMP Port 420 by accessing the memory-mapped view of the EMP registers 430 as viewed by Bar 1 440.

BAR 0 415 AND BAR 1 440 are mapping of the of the EMP port 410 in the memory-mapped address space (or other I/O address space) of the EMP host 405.

The mailbox 450 is a region of memory that the EMP Host 405 uses to pass synchronous commands 455 to the EMP Port 420.

The mailbox 450 is a mailbox memory that resides in the memory of the EDC 420. The memory area used for the mailbox 450 is typically only big enough to hold a single command 455. As discussed above, typically, the EMP Host 405 accesses the mailbox memory 450 through BAR 0 415.

The EMP Host 405 places a mailbox command 455 in the mailbox 450 and this placement of the mailbox command 455 signals (and informs) the EMP Port 420 when there is a command frame 205 (FIG. 2) to process by the EMP Port 410. When the EMP Port 410 has completed processing of the command frame 205, the EMP Port 410 places the mailbox response 465 in the mailbox 450 and this placement of the mailbox response 465 signals (and informs) the EMP Host 405 that there is a response frame 305 (FIG. 3) to process by the EMP Host 405.

This method disclosed in FIG. 4 imposes a serial communication nature to the commands. The EMP Port 410 will only process one mailbox command 455 at a time and will process a command frame 205 until the EMP Port 410 completes processing of the command frame 205.

Mailbox commands 455 are defined to aid in the communication between the EMP Host 405 and EMP Port 410 during initialization and configuration.

Reference is now made to FIG. 5 and FIG. 6.

FIG. 5 is a block diagram of command ring and a response ring, in accordance with an embodiment of the invention, wherein both rings are currently empty. FIG. 6 is a block diagram of command ring and a response ring, in accordance with an embodiment of the invention, wherein both rings are currently full.

In an embodiment of the invention, a ring is a feature that implements asynchronous communication between the EMP Host 405 and the EMP Port 410. The ring allows multiple outstanding commands and responses to be processed.

The actual structure of each ring is composed of two memory arrays, a Command Ring 505 and Response Ring 510. The Command Ring 505 has a set comprising a Command Get pointer 515 and a Command Put pointer 520. Similarly, the Response Ring 510 has a set comprising a Response Get pointer 525 and a Response Put pointer 530. A Command Ring 505 has a plurality of memory areas 540 for receiving ring values (entries) and a Response Ring 510 has a plurality of memory areas 540 for receiving ring values (entries).

The rings 505 and 510 are allocated in a memory area of the EMP Port 410 (EDC or memory device side 410) in FIG. 4. The memory arrays that form the rings 505 and 510 are treated as circular rings so that when the last element (last ring entry) in the memory array is reached, the next element (next ring entry) to be used (accessed and executed) will be the first element (first ring entry) in the memory array.

The get and put pointers allow the insertion and removal of ring entries. When the get and put pointers contain the same value, the ring is considered empty. In FIG. 5, the Command Ring 505 is considered as empty since the Command get pointer 515 and Command put pointer 520 contain the same value in a given memory area 540 in the Command Ring 505. Similarly, the Response Ring 510 is considered as empty since the Response get pointer 525 and Response put pointer 530 contain the same value in a given memory area 545 in the Response Ring 510.

When the put pointer contains a value immediately before the get pointer, the ring is full. In FIG. 6, since the Command put pointer 520 contains a value in the given memory area 540a of Command Ring 505 and the Command get pointer 515 contains a value in the other given memory area 540b of Command Ring 505, and since the memory area 540a is immediately before the memory area 540b, the Command Ring 505 is full.

Similarly, in FIG. 6, since the Response put pointer 530 contains a value in the given memory area 545a of Response Ring 510 and the Response get pointer 525 contains a value in the other given memory area 545b of Response Ring 510, and since the memory area 545a is immediately before the memory area 545b, the Response Ring 510 is full.

One or more commands 550 are placed in the Command Ring 505 by EMP Host 405 and the EMP Port 410 is signaled that work is available for the EMP Port 410 to process at least one of the commands 550. The EMP Port 410 will process each command 550 and as the EMP Port 410 completes the processing of each command 550, the EMP Port 410 will put a respective response 555 (corresponding to the processed command 550) in the Response Ring 510 and notify the EMP Host 405 that one or more responses 555 are available. A format of a frame 205 of a command 550 is shown in FIG. 2, while a format of a frame 305 of a response 555 is shown in FIG. 3.

FIG. 7 is block diagram of a system 700 that shows the interaction between an EMP Host 405 and an EMP Port 410 for EMP frame passing using rings, in accordance with an embodiment of the invention. The host 405 includes a host memory 705 while the EMP Port 410 includes an EDC memory 710. The host 405 is coupled via PCIe bus 715 to the EMP Port 410. As noted above, the host 405 can also be defined as a host side 405 and the EMP Port 410 can also be defined as a memory device side 410.

The host 405 posts EMP command frames 715(0) through 715(N), where N is an integer, to the Command Ring 505, where the host 405 will posts the command frames starting from the Command Ring Base Address 720(0) through Command Ring Address 720(N). Therefore, the EMP host 405 posts the command frames 715(0), 715(1), 715(2), 715(3), up to 715(N) to the Command Ring Addresses 720(0), 720(1), 720(2), 720(3), up to 720(N), respectively. The posted command frames are shown as command ring entries in the command ring addresses. For example, the posted command frames are Command Ring Entry0, Command Ring Entry1, Command Ring Entry2, Command Ring Entry3, and up to Command Ring EntryN in command ring addresses 720(0), 720(1), 720(2), 720(3), up to 720(N), respectively.

The host 405 tracks EMP response frames 730(0) through 730(N), where N is an integer, from the Response Ring 510, where the EMP Port 410 will posts the response frames starting from the Response Ring Base Address 735(0) through Response Ring Address 735(N). Therefore, the EMP Port 410 posts the response frames 730(0), 730(1), 730(2), 73o(3), up to 730(N) to the Respond Ring Addresses 735(0), 735(1), 735(2), 730(3), up to 730(N), respectively, prior to the EMP Port 410 transmitting the response frames 730(0) to 730(N) to the host 405. The posted response frames are shown as response ring entries in the response ring addresses. For example, the posted response frames are Response Ring Entry0, Response Ring Entry1, Response Ring Entry2, Response Ring Entry3, and up to Response Ring EntryN in response ring addresses 735(0), 735(1), 735(2), 735(3), up to 735(N), respectively.

The EMP host 405 includes a Command Send Module(s) 740 that sends the commands 550 to the EMP Port 410 via PCIe bus 715 and the EMP Port 410 posts these commands 550 in the command ring 505. In an embodiment of the invention, the module(s) 740 can also increment the command put pointer 520 and response get pointer 525. As discussed above with reference to FIG. 3, the EMP host 405 can control the behavior and can determine the current status of the EMP Port 410 by accessing the sets of EMP registers 430. Therefore, the modules(s) 740 in the EMP host 405 can increment the pointers 520 and 525. In another embodiment of the invention, the EMP Port 410 will instead increment the command put pointer 520 and response get pointer 525.

The EMP host 405 includes a Response Receive Module(s) 745 that reads the Command Get pointer 515 and Response Put pointer 530 which have been updated by the EMP Port 410 prior to the EMP Port 410 posting and releasing a ring entry (response 555) via PCIe bus 715. As similarly discussed above, the EMP Port 410 posts the responses 555 in the response ring 510. In an embodiment of the invention, the EMP Port 410 will increment the command get pointer 515 and response put pointer 530 prior to posting and releasing an entry (response) 555 in the response ring 510. In an embodiment of the invention, the EMP Port 410 will post and release a response 555.

FIG. 8 is a flow diagram of a method 800 in accordance with an embodiment of the invention. At 805, a host side 405 transmits an exchange message protocol (EMP) command frame 205 to a memory device side 410.

At 810, the host side 405 informs the memory device side 410 to process the command frame 205. For example, the host side 405 transmit a mailbox message 455 to the memory device side 410 to inform the memory device side 410 to process the command frame 205. In an embodiment of the invention the operations at 805 occurs prior to the operations at 810. In another embodiment of the invention, the operations at 805 occurs after the operations at 810. In another embodiment of the invention, the operations at 805 and 810 can occur concurrently or can occur in a substantially concurrent manner.

At 815, the memory device side 410 executes the command frame 205.

At 820, the memory device side 410 transmits an EMP response frame 305 to the host side 405, in response to the command frame 205.

Foregoing described embodiments of the invention are provided as illustrations and descriptions. They are not intended to limit the invention to precise form described. In particular, it is contemplated that functional implementation of invention described herein may be implemented equivalently in hardware, software, firmware, and/or other available functional components or building blocks, and that networks may be wired, wireless, or a combination of wired and wireless.

It is also within the scope of the present invention to implement a program or code that can be stored in a non-transient machine-readable (or non-transient computer-readable medium) having stored thereon instructions that permit a method (or that permit a computer) to perform any of the inventive techniques described above, or a program or code that can be stored in an article of manufacture that includes a non-transient computer readable medium on which computer-readable instructions for carrying out embodiments of the inventive techniques are stored. Other variations and modifications of the above-described embodiments and methods are possible in light of the teaching discussed herein.

The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.

These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification and the claims. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.