Collection

zero Useful+1

zero

Virtual storage system

Computer Academic Language

With the continuous increase of various multimedia processing services around digitalization and networking, the storage system network platform has become a core platform. At the same time, various applications have increasingly high requirements for the platform, not only in storage capacity It also includes data access performance, data transmission performance data management Ability, storage expansion ability, etc. It can be said that the comprehensive performance of the storage network platform will directly affect the normal operation of the entire system. In order to meet these requirements, an emerging technology is attracting more and more attention, namely Virtual Storage Technology 。

Chinese name: Virtual storage system
Processing business: Digitization and networking

Include: Data transmission performance data management ability
technology: Virtual Storage Technology

catalog

Technical Introduction

Announce

edit

actually Virtualization technology It is not a very new technology. Its development should be said to follow the development of computer technology, starting in the 1970s. Because of the storage capacity , especially Memory capacity The cost is very high, and the capacity is very small, so it is very limited for large applications or multi program applications. In order to overcome such restrictions, people have adopted Virtual Storage The most typical application of virtual memory Technology. With the development of computer technology and related information processing technology With the continuous development of, people's demand for storage is growing. In this way Demand stimulus The emergence of various new technologies, such as better and better disk performance and larger and larger capacity. But in a large number of large and medium-sized information processing system In this case, a single disk cannot meet the needs Storage virtualization Technology has developed. In this development process, there are several stages and several applications. The first is the disk stripe set (RAID, fault tolerant) technology, which combines multiple Physical disk Gather them through certain logical relationships to become a high-capacity Virtual Disk 。 With the continuous increase of data volume and the continuous improvement of data availability requirements, another new storage technology emerges as the times require: Storage Area Network (SAN) technology. The wide area of SAN aims to realize the storage device as a kind of public facility, so that any person and any host can obtain their desired data at any time and anywhere. At present, there are many discussions, including iSCSI , FC Over IP and other technologies, as some relevant standards have not been finalized, but the public use of storage devices and storage network wide area is an irreversible trend.

Virtual Storage Concepts

Announce

edit

so-called Virtual Storage It means that multiple storage media modules (such as hard disk and RAID) are managed together through certain means. All storage modules are in one Storage pool (Storage Pool) host and workstation From the perspective of, we can see that it is not multiple hard disks, but a partition or volume, just like a hard disk with a large capacity (such as 1T or more). This kind of storage system can manage multiple storage devices and provide users with large capacity and high data transmission performance. It is called virtual storage.

Virtual Storage Classification

Announce

edit

at present Virtual Storage There is no unified standard for the development of topological structure There are mainly two ways: symmetrical and asymmetrical. Symmetrical form Virtual Storage Technology Virtual storage control device and storage software system . The switching equipment is integrated into a whole, embedded in Network data transmission In the path; Asymmetric virtual storage technology means that the virtual storage control device is independent of the data transmission path. There are also two ways to realize virtual storage; I.e data block Virtual vs Virtual File System 。 The details are as follows:

Symmetric virtual storage

Figure 1 Schematic diagram of symmetric virtual storage solution

Symmetrical formula shown in Figure 1 Virtual Storage In the structure diagram, the storage control device High Speed Traffic Directors (HSTD) is integrated with the storage pool subsystem Storage Pool to form a SAN Appliance. It can be seen that the storage control device HSTD in this scheme host And Storage pool It plays a central role in the process of data exchange. Virtualization of the scheme stored procedure It is like this: the physical hard disks in the storage pool are virtualized as logic by the storage management system embedded in the HSTD Storage unit (LUN), and proceed Port Mapping (Specify which ports can see a LUN). The host side maps each visible storage unit to a drive letter recognized by the operating system. When the host writes data to the SAN Appliance, the user only needs to specify the data write location as his mapped drive letter (LUN). The data passes through the HSTD high-speed Parallel port , write to the cache first, and the storage management system in the HSTD automatically converts the target location from LUN to physical hard disk. In this process, users only see the virtual Logical unit , regardless of the specific physical organization structure of each LUN. The scheme has the following main features:

(1) Large capacity cache is adopted to significantly improve data transmission speed.

cache It is widely used as an intermediate medium in the I/O path between the host and the storage device in the storage system. When the host reads data from the storage device, it will data storage The data connected to the location is read into the cache, and the data called many times is kept in the cache; When the host reads data, it is very likely to find the required data from the cache. Read directly from the cache. The speed of reading data from the cache is only affected by the propagation speed of the electrical signal (equal to the speed of light), so it is much faster than the mechanical rotation speed of the disk when reading data from the hard disk. When the host writes data to the storage device, it first writes the data to the cache, and then writes the data from the cache to the hard disk after the host end stops writing, which is also higher than the speed of direct writing to the hard disk

(2) Multiport Parallel technology , eliminated I/O Bottleneck.

In traditional FC storage devices Control port It has a fixed relationship with the logical disk. Access to a hard disk can only be through the controller port that controls it. In symmetric virtual storage devices, the relationship between SAN Appliance storage ports and LUNs is virtual, that is, multiple hosts can access the same LUN simultaneously through multiple storage ports (up to 8); stay Fibre Channel Under the premise of 100MB/bandwidth Number of ports The more, the higher the data bandwidth.

(3) Logic Storage unit Provides high disk access speed.

In the video application environment, the application reads and writes data in a fixed size data block Is the unit (from 512byte to 1MB). In order to ensure the bandwidth requirements of applications, the storage system is often designed to transmit data blocks of more than 512byte in order to reach its optimal size I/O Performance. In the traditional SAN structure, when the capacity demand increases, the only solution is to bind multiple disks (physical or logical) as a stripe set to achieve large capacity LUNs. In a symmetric virtual storage system host It provides really large capacity, high-performance LUNs, rather than poor performance Logical volume 。 Compared with stripe set, Power LUN has many advantages. For example, large I/O blocks will be really accepted by the storage system, effectively improving the data transmission speed; And since there is no processing with extent sets, the host CPU can relieve a lot of burden and improve the performance of the host.

(4) Paired HSTD system Fault tolerance performance 。

In a symmetric virtual storage system, HSTD is the only place for data I/O, Storage pool It is the data storage place. Since the data in the storage pool has a fault-tolerant mechanism to ensure security, users will naturally wonder whether the HSTD has fault-tolerant protection. Like many large storage systems, in a mature symmetric virtual storage system, HSTDs are configured in pairs. Each pair of HSTDs is consistent in cache data and communicates with each other through the network management services embedded in SAN Appliance.

(5) On the SAN Appliance, switching devices can be easily connected to realize a SAN with a very large fabric structure.

Because the system maintains a standard SAN structure and provides technical support for the expansion and interconnection of the system, it can easily connect switching devices on the SAN Appliance to realize a SAN with a super large fabric structure.

Asymmetric virtual storage system

In asymmetric virtual storage System structure diagram Every host and virtual machine in the network Storage management Devices are connected to disk array , where the host's data path reaches the disk array through the FC switching device; The virtual storage device virtualizes the disk array connected to the network Storage array The LUNs in are virtual logical strips, and each host on the network is assigned access rights to each strip (writeable, readable, and inaccessible). When the host wants to access a strip, it first needs to access the virtual storage device, read the strip information and access permissions, and then access the data in the actual strip through the exchange device. In this process, the host will only recognize the logical Strip, not the physical hard disk. This scheme has the following characteristics:

(1) Set the capacity Carry out logical combination to realize virtual stripe set Array controller Port binding improves the available bandwidth of the system to a certain extent.

(2) On Switch Number of ports If enough, two virtual storage devices can be installed in one network to realize the redundancy 。

However, the scheme has the following shortcomings:

(1) This scheme is essentially zone concentration—— disk array Structure. Once a disk array controller in the stripe set is damaged, or the copper cable and GBIC on the path from the array to the switch are damaged, a virtual LUN will be offline. The stripe set itself has no fault tolerance capability. A damaged LUN means the loss of data in the entire Strip.

(2) Because the bandwidth improvement of this scheme is achieved through array port binding, the common Fibre Channel Array controller Of Effective bandwidth It is only about 40MB/S, so to reach a bandwidth of several hundred megabytes means to call more than ten arrays, which will occupy dozens of switch ports. This is not possible in small and medium-sized networks with only one or two switches.

(3) Because the performance of disk arrays of different brands and models is not the same, if arrays of different brands and models are bound for virtualization purposes, there will be a problem: data writes or reads are concurrent data stream The speed of is different, which means that the original packet order is disrupted after transmission, and the system needs time and resources to re sort the packets, which will seriously affect the system performance.

Block virtualization and virtual file system

From the perspective of topology, the symmetric and asymmetric types are analyzed Virtual Storage In terms of the implementation principle of virtualized storage, there are two ways to achieve the similarities and differences of the schemes; I.e data block Virtual vs Virtual File System 。

The data block virtual storage scheme focuses on solving the conflict and delay problems in the data transmission process. In a SAN with a large fabric structure composed of multiple switches, since multiple hosts access storage devices through multiple switch ports, latency and data block conflicts are very serious. Block virtual storage scheme utilizes virtual multi ports Parallel technology , providing extremely high bandwidth for multiple clients, minimizing latency and conflicts. In practical applications, the data block virtual storage scheme is symmetric topological structure Is the form of expression.

Virtual file system storage solution focuses on solving large-scale network problems File sharing The security mechanism of. By assigning different access rights to different sites, the security of network files can be ensured. In practical application, Virtual File System The storage scheme takes the form of asymmetric topology.

Implementation mode

Announce

edit

Currently implemented Virtual Storage It is mainly divided into the following categories:

Virtual storage on the server side

The server manufacturer will implement the Virtual Storage 。 Similarly, software manufacturers will also implement virtual storage on the server platform. These virtual storage implementations map images to Peripheral storage device In addition to allocating data, there is no control over peripheral storage devices. The server side usually uses logic Volume Management To implement Virtual Storage Technology 。 Logical volume management provides a virtual layer for mapping from physical storage to logical volumes. The server only needs to process Logical volume Instead of managing the physical parameters of the storage device.

This kind of virtual storage system is a performance bottleneck on the server side, so it is rarely used in the field of multimedia processing.

Virtual storage on the storage subsystem side

Another place to implement virtualization is the storage device itself. such Virtual Storage It is generally implemented by the storage manufacturer, but it is likely to use the manufacturer's exclusive storage products. To avoid this no compatibility , manufacturers may cooperate with server, software or network manufacturers. When virtual storage is implemented on the device side, the logical (virtual) environment and Physical devices In the same control range, the benefits of doing this are: Virtual Disk Highly efficient use of disks capacity , virtual tape uses tape media highly efficiently.

The virtual storage device on the storage subsystem side mainly uses large-scale RAID subsystems and multiple I/O channel Connect to the server, and the intelligent controller provides LUNs access control , cache and other management functions such as data replication. The advantage of this method is that the storage device administrator has complete control over the device, and Server system Separate storage management from multiple Server operating system Isolated and easily adjustable Hardware parameters 。

Implement virtual storage on network device side

The network manufacturer will implement the Virtual Storage , mapping logical images to Peripheral storage device , has no control over peripheral storage devices except for data allocation. It is reasonable to implement virtual storage on the network side, because its implementation is neither on the server side nor on the storage device side, but between the two environments. It may be the most "open" virtual implementation environment, which is most likely to support any server, operating system, application and storage device. Technically, there are two types of structures for implementing virtual storage on the network side: symmetric and asymmetric virtual storage.

From the current Virtual Storage Technology And the actual situation of the product, based on host And storage based methods are most attractive to early adopters because they do not need any additional hardware, but for heterogeneous storage systems and operating systems, the operating effect of the system is not very good. The method based on interconnected devices is in between. It avoids some security problems, Storage virtualization It can reduce the load of a single host and achieve good scalability.

No matter what kind of virtual storage technology is used, its purpose is to provide a high-performance, safe, stable, reliable and scalable storage network platform to meet the stringent requirements of the program production network system. According to the comprehensive Price performance ratio Generally speaking, when based on host And storage device based Virtual Storage Technology When the data processing capability of the system can be guaranteed, priority is given to the two virtual storage technologies, because they are convenient in architecture, simple in management, easy in maintenance, relatively mature in products, and high in performance price ratio. In the case that the pure virtual storage technology based on storage devices cannot guarantee the performance requirements of the storage system, we can consider using the virtual storage technology based on interconnection devices.

Virtual Storage Features

Announce

edit

Virtual Storage It has the following characteristics:

(1) Virtual storage provides a means of centralized management of large capacity storage system, which is managed by a link in the network (such as the server) in a unified way, avoiding the management trouble caused by storage device expansion. For example, with a general storage system, when a new storage device is added, the entire system (including many user devices in the network) needs to be re configured to enable this "new member" to join the storage system. Instead, use Virtual Storage Technology , when adding new storage devices, only network administrator Simple system configuration Change. The client does not need any operation. It feels that the storage system capacity has increased.

(2) The most valuable feature of virtual storage for video network systems is that it can greatly improve the overall access bandwidth of the storage system. The storage system is composed of multiple storage modules, and the virtual storage system can load balance The bandwidth required for each data access is reasonably allocated to each storage module, so that the overall access bandwidth of the system is increased. For example, there are four enclosures in a storage system, and the access of each enclosure bandwidth If it is 50MBps, the total access bandwidth of this storage system can be close to the sum of the bandwidth of each storage module, that is, 200MBps.

（3） Virtual Storage Technology It provides better flexibility for storage resource management, can centrally manage and use different types of storage devices, and ensures the investment of storage devices previously purchased by users.

(4) Virtual storage technology can provide some other useful functions for the network system through management software, such as remote mirror and data snapshot without server（ Snapshot ）Etc.

Virtual storage application

Announce

edit

because Virtual Storage With the above characteristics, Virtual Storage Technology Is gradually becoming shared Storage management The main technologies of

data mirroring

Data mirroring is to establish data replicas between different storage devices through two-way synchronization or one-way synchronization mode. A reasonable solution should be able to provide Storage array And the method of making images between different storage arrays.

Data replication

adopt IP address Distance achieved data migration (Usually Asynchronous transmission ）It is extremely important for enterprises of different scales Data disaster recovery Tools. Good solutions should not rely on special network equipment Support. At the same time, you should not rely on the host to save the enterprise's Administrative expenses 。

Tape Backup Enhancement Device

In the past few years, there have been few new developments in tape backup technology. However, a network storage device platform should also be able to build a bridge between tape and disk to complete the backup work in a high-speed, stable and safe manner.

Real time replica

For testing, expanding and summarizing or some other reasons, enterprises often need to make data copies.

Real time data recovery

Using tape to restore data is the main means of data recovery, but it is often difficult to succeed. data management One of the important new development directions is to transfer the recent backup data (which can be historical data several weeks ago) to disk media instead of tape media. Recovering data with disk is as fast as lightning (all files can be recovered within 60 seconds), and is far safer and more reliable than recovering data with tape. At the same time, the entire volume data can be recovered.

Application integration

Storage management Another new direction of development is to bring services closer to applications. No manager in the field of information technology will buy storage devices simply out of interest. Storage devices are used to serve applications, such as databases, communication systems, etc. By combining storage devices with key Enterprise applications The integration of behaviors can obtain greater value, and at the same time, greatly reduce the problems encountered in the operation process.

Application of virtual storage in digital video network

Now let me focus on Virtual Storage stay digital video Network applications.

Digital video network pair Radio and television industry It is no longer a strange concept. Because it plays an important role in the process of broadcasting and television technology digitization, domestic television stations at all levels pay great attention to it, and begin to construct and apply such systems. In the concept of digital video network, it completely breaks the traditional structure of a video recorder, an editing system, and a broadcast system, Instead, upload workstation, editing and production workstation, broadcast workstation and program storage workstation are used to facilitate operation and management. Program upload, program editing, and program broadcast are completed on workstations with different functions, which can double the work efficiency. At the same time, due to the adoption of Nonlinear editing system , except for the compression loss during acquisition. The signal will not have any loss in the process of production and broadcasting, and the technical quality of the program will be greatly improved.

In the existing video network system, although the computer Dominant frequency The transmission rate of the network and the performance of the switching device can meet the requirements of most applications, but the access bandwidth of the storage device has become a major performance bottleneck of the system. Video editing The production has the characteristics of large data storage, high code stream, strong real-time, and important security. This requires that the storage technology and products used in the video field must have sufficient bandwidth and good stability.

In stand-alone applications, to ensure that one editing site has sufficient data bandwidth, SCSI Technology, local independent disk redundancy RAID (Redundant Array of Independent Disks) technology (including software and hardware) is widely used SCSI HardDisk In addition, the controller forms a large capacity, quick response The storage subsystem with high reliability can be used as a logical disk or Virtual disk , thus greatly improving Data transmission rate and storage capacity At the same time, error correction technology is used to improve the reliability of storage and meet the bandwidth requirements.

With the development of program production demand, 2-3 stations are required to share editing data. At this time, SCSI network technology can be used to achieve this requirement. Several editing sites are configured with high-performance SCSI adapters to connect to shared SCSI disk array , which not only enables several sites to share data, but also ensures the working bandwidth of each single machine.

Fibre Channel technology The mature application of Shared Storage The connection distance between devices is limited from a few meters or ten meters to hundreds or thousands of meters, and the network scale is expanded several times or ten times with Fibre Channel switching equipment. FC (Fibre Channel) disk array RAID Fault tolerance technology , relative SCSI With high bandwidth and large capacity, it has become the core storage device in the video network.

With the development of TV station scale, the application of large-scale video network at the whole station level is proposed. Under this demand, more advanced storage technologies and products must be introduced into the video field. Storage Area Network The development of (SAN) is currently in a period of full speed rise, with various concepts emerging in endlessly. What is of epoch-making significance Virtual Storage The proposal of the concept. Compared with the traditional switch plus RAID array, the host directly accesses the SAN structure of the hard disk in the array through the hardware layer. The positioning of virtual storage is to data storage Functions from the actual, physical data access Abstracted from the process, ordinary users do not have to care about the configuration parameters, physical location and capacity of specific storage devices when accessing data, thus simplifying the work difficulty of users and system managers.

When designing a video network system, the following factors should be considered when selecting the storage system: (1) overall bandwidth performance; (2) Manageability; (3) Safety; （4） Scalability ； (5) System costs.

Of course, these factors sometimes restrict each other, especially the relationship between system cost and performance and security. How to seek reasonable, practical and economic cooperation among these factors is a subject that needs to be solved. Virtual Storage Technology The emergence of provides us with a feasible solution with high performance price ratio when building a video network system.

from topological structure The symmetric scheme has higher bandwidth performance and better Safety features Therefore, it is more suitable for large-scale video network applications. The asymmetric scheme is more suitable for the application of ordinary LAN (such as office network) due to the virtual file principle.

Novice on the road

Growth task Getting Started with Editing Edit Rule Edit by myself

I have questions

Content query Online Service Official post bar Feedback

Complaints and suggestions

Report bad information Failed to appeal through entries Complaint of infringement information Blocking query and unblocking

Jinggong Network Anbei No. 1100000200001