Main viewpoint-based panoramic video mapping method

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national stage filing under 35 U.S.C. § 371 of PCT/CN2018/088806, filed on May 29, 2018 which claims priority to CN Application No. 201710882161.1 filed on Sep. 26, 2017. The applications are incorporated herein by reference in their entirety.

FIELD

The present disclosure generally relates to a field of virtual reality (VR) video technology, specifically relates to a novel main viewpoint-based panoramic video mapping method and a corresponding inverse mapping method, the mapping technology is used for panoramic video, and in the premise of ensuring that the quality of the main viewpoint region of the panoramic video is unchanged, the file size and the coding bit rate of the panoramic video can be greatly reduced.

BACKGROUND

With the development of virtual reality video, the demand for virtual reality video is increasing. Compared with a traditional video with a view angle of about 100 degrees or more, a virtual reality video need to provide a view angle of 360 degrees, therefore, a virtual reality video requires a greater resolution. In addition, virtual reality videos usually need to be watched on head-mounted devices, and in order to avoid causing dizziness to the user, the suggested frame rate needs to be above 60 frames per second, or even 120 frames per second. Due to the increase in resolution and frame rate, the bit rates required for a virtual reality video coding is also greatly increased compared to a traditional video.

Although the virtual reality video provides a view angle of 360 degrees, the user will only watch the image in a vicinity of one viewpoint for a short period of time, and this viewing direction is called the main viewpoint of the user. In view of this feature, asymmetric mapping technology and stream switching technology are proposed to achieve the purpose of saving bit rates. The asymmetric mapping technique uses a higher sampling density to sample a main viewpoint region, while uses a lower sampling density to sample other regions. The stream switching technology encodes and stores multiple bit streams of asymmetrically mapped video with different main viewpoints on a server end, and the client chooses to receive the corresponding bit stream according to the position of the user's viewpoint.

Although the asymmetric mapping technology and the stream switching technology may increase the storage and coding cost of the server side, network transport cost and decoding complexity of the client can be effectively reduced, which is effective in one-to-many application scenarios. However, the coding efficiency of the prior asymmetric mapping technology is low, and it needs to be improved.

SUMMARY

To overcome the drawbacks in the existing technologies, to further improve coding efficiency of the asymmetric mapping technology, the present disclosure provides a panoramic video asymmetrical mapping method and a corresponding inverse mapping method. While ensuring that the video quality of the region of the main viewpoint remains unchanged, the present panoramic video asymmetrical mapping technique greatly reduces the resolution of the remaining regions in the video, effectively reducing the bit rate required for encoding a virtual reality video. The panoramic video asymmetrical inverse mapping technique provides a method for mapping from a planar surface to a spherical surface, and by means of said method, a planar surface video may be mapped back to a spherical surface for rendering and viewing.

The technical solution provided by the present invention is:

A novel main viewpoint-based panoramic video asymmetrical mapping method, mapping a spherical surface corresponding to a panoramic image or video A onto a two-dimensional image or video B; projecting the spherical surface onto an isosceles quadrangular pyramid with a square bottom plane, and further projecting the isosceles quadrangular pyramid onto a planar surface; using isometric projection on a main viewpoint region in the projection and using a relatively high sampling density to ensure that the video quality of the region of the main viewpoint is high, while using a relatively low sample density for non-main viewpoint regions so as to reduce bit rate. Specifically, a right-handed coordinate system is established based on the sphere center O, wherein the Z axis points to the direction at longitude and latitude of (0°, 0°) of the spherical surface, the Y axis points to the direction of the north pole of the spherical surface, and the X axis points to the direction at longitude and latitude of (90°, 0°) of the spherical surface. Then, a quadrangular pyramid W is established, wherein, the center of the bottom plane of the quadrangular pyramid W is on the positive Z axis, the apex is on the negative Z axis, and the base edges are respectively parallel to the X axis and parallel to the Y axis; the connecting line between the center D of the bottom plane of the quadrangular pyramid and the sphere center O is denoted as l₁, and the connecting line between the midpoint of a base edge of the quadrangular pyramid and the sphere center O is denoted as l₂, and the angle formed by l₁ and l₂ is θ; the bottom plane of the quadrangular pyramid is denoted as plane I″, the side surface intersecting with the positive X axis is denoted as plane II″, the side surface intersecting with the positive Y axis is denoted as plane III″, the side surface intersecting with the negative X axis is denoted as plane IV″, and the side surface intersecting with the negative Y axis is denoted as plane V″. When the pitch angle, the yaw angle and the roll angle of the main viewpoint are respectively β₁, β₂ and β₃, a corresponding quadrangular pyramid Q′ is obtained by rotating the quadrangular pyramid Q about the sphere center O by (β₁, β₂, β₃). The sphere center O and the four base edges of the quadrangular pyramid Q′ form four sector planes, and the four sector planes divide the spherical surface into two regions. Wherein, one region including the main viewpoint is referred to as the main viewpoint region, and the main viewpoint region corresponds to the bottom plane of the pyramid and is denoted as region I, and the other regions are referred to as the non-main viewpoint regions; the sphere center O and four side edges of the quadrangular pyramid Q′ form four sector planes, which further divide one non-main viewpoint region into four sub-regions, and the four sub-regions respectively correspond to the four side surfaces II″, III″, IV″ and V″, and are respectively referred to as regions II, III, IV and V. The mapping method of the present invention maps the described region I onto a rectangular plane I′ with a resolution of W_I′×H_I′, and maps the regions II, III, IV and V onto four isosceles triangular planes II′, III′, IV′, and V′, wherein the base edge and the height of the four isosceles triangular planes are respectively

W II ′ × H II ′ 2 , H II ′ × W II ′ 2 , W II ′ × H II ′ 2 , H II ′ × W II ′ 2 ,

and then the four isosceles triangular planes are spliced into a rectangular plane VI′ with a resolution of W_II′×H_II′ The parameters θ, W_I′, H_I′, W_II′, H_II′, β₁, β₂, and β₃ can be set as required. The rectangular plane I′ and rectangular plane VI′ are the two-dimensional image or video B obtained by mapping.

The mapping formats of panoramic image A comprises a longitude and latitude image, a cube-mapped image, or a panoramic video captured by a multi-channel camera, but is not limited thereto. The two-dimensional image or video B includes the rectangular plane I′ and the rectangular plane VI′.

For the panoramic image mapping method, specifically, the mapping a spherical surface corresponding to a panoramic image or video A onto a two-dimensional image or video B including the following steps:

First step: for each pixel in the rectangular plane I′, according to its coordinate (X_I′, Y_I′) in the plane I′, calculating its corresponding coordinate “Coordinate′” in the bottom plane of the quadrangular pyramid Q′, then further calculating its corresponding coordinate “Coordinate” on the spherical surface according to the perspective projection method, and finally according to the spherical coordinate “Coordinate”, taking the pixel value of the corresponding position (or the corresponding pixel value obtained based on neighbor pixels by interpolation) on the spherical surface, as the pixel value of the pixel (X_I, Y_I) in the plane I′. The specific steps to calculate the spherical coordinate “Coordinate” according to the plane coordinate (X_I′, Y_I′) are as follows:

(1.1) the point, corresponding to the point in the plane I′ with the coordinate (X_I′, Y_I′), in the bottom plane of the quadrangular pyramid Q′ is denoted as A_I′, the connecting lines between the midpoints of the opposite base edges of the quadrangular pyramid Q′ are denoted as m₁ and m₂, the projections of the point A_I′ on m₁ and m₂ are respectively denoted as B_I′ and C_I′, and the center of the bottom plane of the quadrangular pyramid Q′ is denoted as D, and the position of A_I′ can be determined based on the angle B_I′OD and the angle C_I′OD; in the present invention, the magnitudes of the angle B_I′OD and the angle C_I′OD can be calculated based on the value of (X_I′, Y_I′), the magnitude of the angle B_I′OD and the value of X_I′ are in a relationship of a linear function, and the magnitude of the angle C_I′OD and the value of Y_I′ are in a relationship of a linear function;

(1.2) The coordinate of A_I′, “Coordinate′”, is determined according to the magnitude of the angle B_I′OD, the magnitude of the angle C_I′OD and the values of β₁, β₂ and β₃;

(1.3) According to the coordinate of the point A_I′, the coordinate of the intersection of the ray {right arrow over (OA_I′)} and the spherical surface, “Coordinate”, is obtained.

Second step: for each pixel in the four isosceles triangular planes II′, III′, IV′ and V′, according to its coordinate (X_II′, Y_II′) (the X and Y coordinate axes are respectively perpendicular to and parallel to the base of the isosceles triangle), calculating its corresponding coordinate “Coordinate′” on the side surface of the quadrangular pyramid Q′, then further calculating its corresponding coordinate “Coordinate” on the spherical surface according to the perspective projection method, and finally according to the spherical coordinate “Coordinate”, taking the pixel value of the corresponding position (or the corresponding pixel value obtained based on neighbor pixels by interpolation) on the spherical surface, as the pixel value of the pixel (X_II′, Y_II′). The specific steps to calculate the spherical coordinate “Coordinate” according to the plane coordinate (X_II′, Y_II′) are as follows:

(2.1) The point, corresponding to the pixel (X_II′, Y_II′), on the side surface of the quadrangular pyramid Q′ is denoted as A_II′, the connecting line between the midpoint of the base edge of the side surface and the apex of the quadrangular pyramid Q′ is denoted as n₁, and the connecting line between the center of the bottom plane and the apex of the quadrangular pyramid Q′ is denoted as n₂, and the projections of point A_II′ on n₁ and n₂ are denoted as B_II′ and C_II′, respectively, and the center of the bottom plane of the quadrangular pyramid Q′ is denoted as D, and the position of A_II′ can be determined based on the angle B_II′OD and the angle C_II′OD; in the present invention, the magnitudes of the angle B_II′OD and the angle A_II′C_II′B_II′ can be calculated based on the value of (X_II′, Y_II′), and the magnitude of the angle A_II′C_II′B_II′ and the value of Yip are in a relationship of a linear function, the relationship between the magnitude of the angle B_II′OD and the value of X_II′ is ∠B_II′OD=f(X_II′), the f function can be any function that satisfies the following conditions:

θ=f(0)

180°=f(H)

Wherein, H is the height of the isosceles triangle, and the heights of the four isosceles triangular planes II′, III′, IV′ and V′ are respectively

H II ′ 2 , W II ′ 2 , H II ′ 2 , W II ′ 2 ;

(2.2) Calculating the coordinate of A_II′, “Coordinate′”, based on the magnitudes of the angle B_II′OD and the angle A_II′C_II′B_II′ and the values of β₁, β₂ and β₃;

(2.3) Calculating the coordinate of the intersection of the ray OA_II′ and the spherical surface, “Coordinate”, based on the coordinate of the point A_II′.

Third step: splicing the four isosceles triangular planes II′, III′, IV′ and V′ obtained in the second step into a rectangular plane VI′ with a resolution of W_II′×H_II′.

The rectangular plane I′ and rectangular plane VI′ are the two-dimensional image or video B obtained by mapping.

For the above-mentioned panoramic video mapping process, further, in (2.1), f(X_II′) may be:

f  ( X II ′ ) = θ + ( 180 ° - θ ) × tan  ( X II ′ ÷ H × 180 ° × C ) tan  ( 180 ° × C )

(Wherein, C is a constant that is greater than 0 and less than 0.5).

On the other hand, in the process of the panoramic image inverse mapping method, mapping a two-dimensional image or video B back to a spherical surface based on a main viewpoint. The two-dimensional image or video B includes a rectangular plane I′ with a resolution of W_I′×H_I′ and a rectangular plane VI′ with a resolution of W_II′×H_II′, wherein, the rectangular plane VI′ may be further divided into four isosceles triangular planes II′, III′, IV′ and V′, and the base edge and the height of the four isosceles triangles are respectively

W II ′ × H II ′ 2 , H II ′ × W II ′ 2 , W II ′ × H II ′ 2 , H II ′ × W II ′ 2 .

The inverse mapping method firstly projects the above-mentioned plane onto a quadrangular pyramid by means of such as isometric projection, and then maps the quadrangular pyramid onto a spherical surface. Specifically, a right-handed coordinate system is established based on the sphere center O, wherein the Z axis points to the direction at longitude and latitude of (0°, 0°) of the spherical surface, the Y axis points to the direction of the north pole of the spherical surface, and the X axis points to the direction at longitude and latitude of (90°, 0°) of the spherical surface. Then, a quadrangular pyramid W is established, the center of the bottom plane of the quadrangular pyramid W is on the positive Z axis, the apex is on the negative Z axis, and the base edges are respectively parallel to the X axis and parallel to the Y axis; the connecting line between the center D of the bottom plane of the quadrangular pyramid and the sphere center O is denoted as l₁, and the connecting line between the midpoint of a base edge of the quadrangular pyramid and the sphere center O is denoted as l₂, and the angle formed by l₁ and l₂ is θ, and θ represents the size of the main viewpoint region; the bottom plane of the quadrangular pyramid is denoted as plane I″, the side surface intersecting with the positive X axis is denoted as plane II″, the side surface intersecting with the positive Y axis is denoted as plane III″, the side surface intersecting with the negative X axis is denoted as plane IV″, and the side surface intersecting with the negative Y axis is denoted as plane V″. When the pitch angle, the yaw angle and the roll angle of the main viewpoint are respectively β₁, β₂ and β₃, a corresponding quadrangular pyramid Q′ is obtained by rotating the quadrangular pyramid Q about the sphere center O by (β₁, β₂, β₃). The sphere center O and the four base edges of the quadrangular pyramid Q′ form four sector planes, and the four sector planes divide the spherical surface into two regions. Wherein, one region including the main viewpoint is referred to as the main viewpoint region, and the main viewpoint region corresponds to the bottom plane of the pyramid and is denoted as region I; the other regions are referred to as the non-main viewpoint regions, and the center O and four side edges of the quadrangular pyramid Q′ form four sector planes, which further divide one non-main viewpoint region into four sub-regions, and the four sub-regions respectively correspond to the four side surfaces II′, III′, IV′ and V′, and are respectively referred to as regions II, III, IV and V. The inverse mapping method of the present invention maps the rectangular plane I′ with the resolution of WI′×H_I′ contained in the two-dimensional image or video B onto the main viewpoint I of the spherical surface, and maps the four isosceles triangular planes II′, III′, IV′ and V′ contained in the two-dimensional image or video B onto the non-main viewpoints II, III, IV and V of the spherical surface. The values of the parameters θ, W_I′, H_I′, W_II′, H_II′, β₁, β₂ and β₃ can be obtained from a bit stream, but is not limited thereto.

For the above-mentioned panoramic image inverse mapping method, further, the method of projecting the planar image or video B back to the spherical surface is to perform the following operations on all points on the spherical surface:

First step: according to the coordinate of the point on the spherical surface, “Coordinate”, and the values of β₁, β₂, and β₃, determining which one of the regions I, II, III, IV and V the point is located in; if the point is in region I, go to the second step; if the point is in region II, III, IV, or V, go to the fifth step;

Second step: the point whose spherical coordinate is “Coordinate” is denoted as A_I′, the connecting lines between the midpoints of the opposite base edges of the quadrangular pyramid Q′ are denoted as m₁ and m₂, and the projections of the point A_I′ on m₁ and m₂ are respectively denoted as B_I′ and C_I′, and the center of the bottom plane of the quadrangular pyramid Q′ is denoted as D; and calculating the magnitudes of the angle B_I′OD and the angle C_I′OD according to the coordinate of A_I′, “Coordinate”, and the values of β₁, β₂ and β₃;

Third step: calculating the value of the plane coordinate (X_I′, Y_I′) according to the magnitudes of the angle B_I′OD and the angle C_I′OD, wherein, the value of X_I′ and the magnitude of the angle B_I′OD are in a relationship of a linear function, and the value of Y_I′ and the magnitude of the angle C_I′OD are in a relationship of a linear function;

Fourth step: using the pixel value (or the pixel value obtained by interpolation based on neighboring pixels) at (X_I′, Y_I′) on the rectangular plane I′ as the pixel value of the point with the coordinate “Coordinate” on the spherical surface; omit the subsequent steps;

Fifth step: the point whose spherical coordinate is “Coordinate” is denoted as A_II′, and, the connecting line between the midpoint of the base edge of the side surface and the apex of the quadrangular pyramid Q′ is denoted as n₁, and the plane passing n₁ and the sphere center O is denoted as α₃, and the connecting line between the center of the bottom plane and the apex of the quadrangular pyramid Q′ is denoted as n₂, and the projections of point A_II′ on α₃ and n₂ are denoted as B_II′ and C_II′, respectively, and the center of the bottom plane of the quadrangular pyramid Q′ is denoted as D; calculating the magnitudes of the angle B_II′OD and the angle A_II′C_II′B_II′ according to the coordinate of A_II′, “Coordinate”, and the values of β₁, β₂ and β₃;

Sixth step: calculating the value of the plane coordinate (X_II′, Y_II′) according to the magnitudes of the angle B_II′OD and the angle A_II′C_II′B_II′ (the X and Y coordinate axes are respectively perpendicular to and parallel to the base of the isosceles triangle), the value of Y_II′ and the magnitude of the angle A_II′C_II′B_II′ are in a relationship of a linear function, and the relationship between the magnitude of the angle B_II′OD and the value of X_II′ is ∠B_II′OD=f(X_II′), the f function can be any function that satisfies the following conditions:

θ=f(0)

180°=f(H)

Wherein, H is the height of the isosceles triangle, and the heights of the four isosceles triangular planes II′, III′, IV′ and V′ are respectively

H II ′ 2 , W II ′ 2 , H II ′ 2 , W II ′ 2 ;

Seventh step: Using the pixel value (or the pixel value obtained by interpolation based on neighboring pixels) at (X_II′, Y_II′) on the triangular plane as the pixel value of the point with the coordinate “Coordinate” on the spherical surface; omit the subsequent steps;

Perform the processes from the first step to the seventh step on all points on the spherical surface, thereby obtaining a panoramic image of the spherical surface.

For the above-mentioned panoramic image inverse mapping method, further, in the sixth step, f(X_II′) may be:

f  ( X II ′ ) = θ + ( 180 ° - θ ) × tan  ( X II ′ ÷ H × 180 ° × C ) tan  ( 180 ° × C )

(Wherein, C is a constant that is greater than 0 and less than 0.5).

Compared with the prior art, the beneficial effects of the present invention are as follows:

Disclosed are a panoramic video asymmetrical mapping method and a corresponding inverse mapping method, mapping a spherical surface corresponding to a panoramic image or video A onto a two-dimensional image or video B; projecting the spherical surface onto an isosceles quadrangular pyramid with a square bottom plane, and further projecting the isosceles quadrangular pyramid onto a planar surface; using isometric projection on a main viewpoint region in the projection and using a relatively high sampling density to ensure that the video quality of the region of the main viewpoint is high, while using a relatively low sample density for non-main viewpoint regions so as to reduce bit rate. While ensuring that the video quality of the region of the main viewpoint remains unchanged, the present panoramic video asymmetrical mapping technique greatly reduces the resolution of the remaining regions in the video, effectively reducing the bit rate required for encoding a virtual reality video. The panoramic video asymmetrical inverse mapping technique provides a method for mapping from a planar surface to a spherical surface, and by means of said method, a planar surface video may be mapped back to a spherical surface for rendering and viewing.

The present invention overcomes the shortcomings of the prior art, further improves coding efficiency of the asymmetric mapping technology, and has the following advantages:

(1) In the present invention, in the main viewpoint region, the angles on the spherical surface and the planar coordinates are in a relationship of a linear function, that is, in the main viewpoint region, equidistant sampling is performed according to the angles, so as to ensure that the sampling of the main viewpoint region is uniform;

(2) In the present invention, the parameter θ and the function f of the panoramic image mapping method are both adjustable, that is, the range of the main viewpoint region and the speed variation in the sampling density are both adjustable;

(3) When the parameter θ and the function f(X_II) are reasonably set, the quality of the main viewpoint region can be guaranteed, and the bit rate can be greatly saved. For example, when θ=45°,

∠   DOB = f  ( X II ′ ) = θ + ( 180 ° - θ ) × tan  ( X II ′ ÷ H × 180 ° × 0.35 ) tan  ( 180 ° × 0.35 ) ,

a video with a resolution of 4096×2048 will be mapped into a video with a resolution of 1024×1024 in both plane I′ and plane VI′, and then plane I and plane II are spliced into a video with a resolution of 2048×1024. Compared with the prior asymmetric mapping technology, the coding efficiency can be improved by about 10-30%.

(4) Through the inverse mapping method, a planar image B in the present invention can be mapped back to a spherical surface for rendering and viewing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a mapping relationship between a spherical surface and a plane in a mapping process of the present invention;

Wherein, a) is a schematic diagram of a corresponding quadrangular pyramid and a spherical surface when the pitch angle, the yaw angle and the roll angle of the main viewpoint are (0°, 0°, 0°), respectively; (b) is a schematic diagram of a corresponding quadrangular pyramid and a spherical surface when the pitch angle, the yaw angle and the roll angle of the main viewpoint are (β₁, β₂, β₃), respectively; c) is a schematic diagram of a rectangular plane I′ and a rectangular plane VI′ obtained by mapping a spherical surface to a plane by the method of the present invention;

In (a): O is the sphere center; the Z axis points to the direction at longitude and latitude of (0°, 0°) of the spherical surface, the Y axis points to the direction of the north pole of the spherical surface, and the X axis points to the direction at longitude and latitude of (90°, 0°) of the spherical surface; in the quadrangular pyramid, a quadrangular pyramid W is established, wherein, the center of the bottom plane of the quadrangular pyramid W is on the positive Z axis, the apex is on the negative Z axis, and the base edges are respectively parallel to the X axis and parallel to the Y axis; the angle formed by the connecting line between the center D of the bottom plane of the quadrangular pyramid and the sphere center O and the connecting line between the midpoint of a base edge of the quadrangular pyramid and the sphere center O is θ, and θ represents the size of the main viewpoint region;

In (b): the quadrangular pyramid is obtained by rotating the quadrangular pyramid in (a) about the sphere center O by the pitch angle, the yaw angle and the roll angle;

In (c): I′ is a rectangular plane; II′, III′, IV′, V′ are four isosceles triangular planes, VI′ is a rectangular plane VI′ stitched by II′, III′, IV′ and V′;

FIG. 2 is a schematic diagram of a mapping relationship between a point with coordinate (X_I′, Y_I′) in a rectangular plane I′ and a point A_I′ on a quadrangular pyramid in the present invention;

Wherein, (a) is a schematic diagram of a point with coordinate (X_I′, Y_I′) in a rectangular plane I′; (b) is a schematic diagram of the point A_I′ on a quadrangular pyramid; wherein, O is the sphere center, Dis the center of the main viewpoint, m₁ and m₂ are connecting lines between the midpoints of the opposite base edges of the quadrangular pyramid, respectively, point A_I′ is the corresponding point on the quadrangular pyramid mapped from the point with coordinate (X_I′, Y_I′) in the rectangular plane I′, B_I′ and C_I′ are projections of point A_I′ on m₁ and m₂, respectively.

FIG. 3 is a schematic diagram of a mapping relationship between a point with coordinate (X_I′, Y_I′) in a triangular plane and a point A_II′ on a quadrangular pyramid in the present invention;

Wherein, (a) is a schematic diagram of a point with coordinate (X_II′, Y_II′) in a triangular plane; (b) is a schematic diagram of the point A_II′ on a quadrangular pyramid; wherein, O is the sphere center, Dis the center of the main viewpoint, n₁ is the connecting line between the midpoint of the base edge of the side surface and the apex of the quadrangular pyramidal, n₂ is the connecting line between the center D of the bottom plane and the apex of the quadrangular pyramidal, point A_II′ is the corresponding point on the quadrangular pyramid mapped from the point with coordinate (X_II′, Y_II′) in the triangular plane, B_II′ and C_II′ are projections of point A_II′ on n₁ and n₂, respectively.

FIG. 4 is an effect diagram of an embodiment of the panoramic image mapping method according to the present invention.

FIG. 5 is a schematic diagram of calculating the angle BOD and the angle COD in the panoramic image inverse mapping method of the present invention;

Wherein, points B and C are projections of point A on the ZOX and ZOY planes, respectively; point D is a point on the positive Z axis, and E is a point on the negative Z axis.

FIG. 6 is a schematic diagram of a mapping relationship between the point A_I′ located on the spherical surface in region land a point with coordinate (X_I′, Y_I′) in the rectangular plane I′ in the panoramic image inverse mapping method according to the present invention;

Wherein, (a) is a schematic diagram of the point A_I′ located in the region Ion a spherical surface, O is the sphere center, Dis the center of the main viewpoint, m₁ and m₂ are connecting lines between the midpoints of the opposite base edges of the quadrangular pyramid, respectively, B_I′ and C_I′ are projections of point A_I′ on m₁ and m₂, respectively; (b) is a schematic diagram of a point with coordinate (X_I′, Y_I′) in the rectangular plane I′.

FIG. 7 is a schematic diagram of a mapping relationship between a point A_II′ in region II, III, IV or V of a spherical surface and a point (X_II′, Y_II′) in a triangular plane in the panoramic image inverse mapping method of the present invention;

Wherein, (a) is a schematic diagram of the point A_II′ in region II, III, IV or V on the spherical surface, O is the sphere center, Dis the center of the main viewpoint, n₁ is the connecting line between the midpoint of the base edge of the side surface and the apex of the quadrangular pyramidal, n₂ is the connecting line between the center D of the bottom plane and the apex of the quadrangular pyramidal, B_II′ and C_II′ are projections of point A_II′ on n₁ and n₂, respectively; (b) is a schematic diagram of a point (X_II′, Y_II′) with coordinate (X_II′, Y_II′) in a triangular plane.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, the present disclosure is further described through the embodiments, but the scope of the present disclosure is not limited in any manner.

Embodiments of the present invention provide a panoramic image mapping method based on a main viewpoint, including a panoramic image mapping method and a corresponding inverse mapping method. Embodiments of the mapping method and the inverse mapping method are described as follows.

In an embodiment of the panoramic image mapping method, a spherical surface corresponding to a panoramic image A of a certain mapping format (such as a latitude and longitude map, a cube map image, etc.) is mapped to a planar image B corresponding to the panoramic mapping designed in the present invention. The planar image B includes a rectangular plane I′ with a resolution of W_I′×H_I′ and a rectangular plane VI′ with a resolution of W_II′×H_II′. The panoramic image mapping method firstly project the spherical surface onto an isosceles quadrangular pyramid with a square bottom plane, and then project the quadrangular pyramid onto a planar surface; in the projection, use isometric projection on a main viewpoint region and use a relatively high sampling density to ensure that the video quality of the region of the main viewpoint is high, while use a relatively low sample density for non-main viewpoint regions so as to reduce bit rate. As shown in FIG. 1, a right-handed coordinate system is established based on the sphere center O, wherein the Z axis points to the direction at longitude and latitude of (0°, 0°) of the spherical surface, the Y axis points to the direction of the north pole of the spherical surface, and the X axis points to the direction at longitude and latitude of (90°, 0°) of the spherical surface. Then, a quadrangular pyramid W is established, wherein, the center of the bottom plane of the quadrangular pyramid W is on the positive Z axis, the apex is on the negative Z axis, and the base edges are respectively parallel to the X axis and parallel to the Y axis; the connecting line between the center D of the bottom plane of the quadrangular pyramid and the sphere center O is denoted as l₁, and the connecting line between the midpoint of a base edge of the quadrangular pyramid and the sphere center O is denoted as l₂, and the angle formed by l₁ and l₂ is θ, and θ represents the size of the main viewpoint region; the bottom plane of the quadrangular pyramid is denoted as plane I″, the side surface intersecting with the positive X axis is denoted as plane II″, the side surface intersecting with the positive Y axis is denoted as plane III″, the side surface intersecting with the negative X axis is denoted as plane IV″, and the side surface intersecting with the negative Y axis is denoted as plane V″. When the pitch angle, the yaw angle and the roll angle of the main viewpoint are respectively β₁, β₂ and β₃, a corresponding quadrangular pyramid Q′ is obtained by rotating the quadrangular pyramid Q about the sphere center O by (β₁, β₂, β₃). The sphere center O and the four base edges of the quadrangular pyramid Q′ form four sector planes, and the four sector planes divide the spherical surface into two regions. Wherein, one region including the main viewpoint is referred to as the main viewpoint region, and the main viewpoint region corresponds to the bottom plane of the pyramid and is denoted as region I, and the other regions are referred to as the non-main viewpoint regions; the sphere center O and four side edges of the quadrangular pyramid Q′ form four sector planes, which further divide one non-main viewpoint region into four sub-regions, and the four sub-regions respectively correspond to the four side surfaces II′, III′, IV′ and V′, and are respectively referred to as regions II, III, IV and V. The mapping method of the present invention maps the described region I onto a rectangular plane I′ with a resolution of W_I′×H_I′, and maps the regions II, III, IV and V onto four isosceles triangular planes II′, III′, IV′, and V′, wherein the base edge and the height of the four isosceles triangular planes are respectively

W II ′ × H II ′ 2 , H II ′ × W II ′ 2 , W II ′ × H II ′ 2 , H II ′ × W II ′ 2 ,

and then the four isosceles triangular planes are spliced into a rectangular plane VI′ with a resolution of W_II′×H_II′. The parameters θ, W_I′, H_I′, W_II′, β₁, β₂, and β₃ can be set as required. The specific processes of the panoramic image mapping method are as follows:

First step: for each pixel in the rectangular plane I′, according to its coordinate (X_I′, Y_I′) in the plane I′, calculating its corresponding coordinate “Coordinate′” in the bottom plane of the quadrangular pyramid Q′, then further calculating its corresponding coordinate “Coordinate” on the spherical surface according to the perspective projection method, and finally according to the spherical coordinate “Coordinate”, taking the pixel value of the corresponding position (or the corresponding pixel value obtained based on neighbor pixels by interpolation) on the spherical surface, as the pixel value of the pixel (X_I, Y_I) in the plane I′. The specific steps to calculate the spherical coordinate “Coordinate” according to the plane coordinate (X_I′, Y_I′) are as follows:

(1.1) As shown in FIG. 2, the point (X_I′, Y_I′) in the plane I′ corresponds to the point A_I′ in the bottom plane of the quadrangular pyramid Q′, and the connecting lines between the midpoints of the opposite base edges of the quadrangular pyramid Q′ are denoted as m₁ and m₂, and the projections of the point A_I′ on m₁ and m₂ are respectively Br and Cr. The angles B_I′OD and C_I′OD are calculated according to the value of (X_I′, Y_I′), and the calculation formulas are as follows:

∠   B I ′  OD = 2  θ × Y I ′ W I ′ - θ

∠   C I ′  OD = - 2  θ × Y X ′ H I ′ + θ

(1.2) The coordinate of A_I′, “Coordinate′”, which is expressed by Cartesian coordinates, is determined according to the magnitude of the angle B_I′OD, the magnitude of the angle C_I′OD and the values of β₁, β₂ and β₃, and the calculation formula thereof is:

( X A I ′ , Y A I ′ , Z A I ′ ) =  ( k × tan  ( ∠   B I ′  OD ) · k × tan  ( ∠   C I ′  OD ) , k ) × T =  k × ( tan  ( ∠   B I ′  OD ) · tan  ( ∠   C I ′  OD ) , 1 ) × T

Wherein, T is a 3×3 dimensional rotation matrix generated according to the pitch angle β₁, the yaw angle β₂ and the roll angle β₃; k is the length of the line segment OD;

(1.3) According to the coordinate of the point A_I′, the coordinate of the intersection of the ray {right arrow over (OA_I′)} and the spherical surface, “Coordinate”, is obtained, and the coordinate “Coordinate” is represented by the Cartesian coordinates, and the calculation formula thereof is:

( X sphere , Y sphere , Z sphere ) =  R × ( X A I ′ , Y A I ′ , Z A I ′ )  X A I ′ , Y A I ′ , Z A I ′  =  R × k × ( tan  ( ∠   B I ′  OD ) , tan  ( ∠   C I ′  OD ) , 1 ) × T k ×  ( tan  ( ∠   B I ′  OD ) , tan  ( ∠   C I ′  OD ) , 1 ) × T  =  R × ( tan  ( ∠   B I ′  OD ) , tan  ( ∠   C I ′  OD ) , 1 ) × T  ( tan  ( ∠   B I ′  OD ) , tan  ( ∠   C I ′  OD ) , 1 ) × T 

Wherein, R is the radius of the sphere;

 X A 1 ′ , Y A 1 ′ , Z A 1 ′ 

represents the modulus of the line segment OA_I′, that is, the length of the line segment OA_I′, and dividing

( X A 1 ′ , Y A 1 ′ , Z A 1 ′ )

by

 X A 1 ′ , Y A 1 ′ , Z A 1 ′ 

is a normalization operation on

( X A 1 ′ , Y A 1 ′ , Z A 1 ′ ) .

It should be noted that the value of k in the above step (1.2) is unknown, but it does not affect the subsequent calculation, because k will be eliminated when the normalization operation is performed on

( X A 1 ′ , Y A 1 ′ , Z A 1 ′ )

in step (1.3).

Second step: for each pixel in the four isosceles triangular planes II′, III′, IV′ and V′, according to its coordinate (X_II′, Y_II′) (the X and Y coordinate axes are respectively perpendicular to and parallel to the base of the isosceles triangle), calculating its corresponding coordinate “Coordinate′” on the side surface of the quadrangular pyramid Q′, then further calculating its corresponding coordinate “Coordinate” on the spherical surface according to the perspective projection method, and finally according to the spherical coordinate “Coordinate”, taking the pixel value of the corresponding position (or the corresponding pixel value obtained based on neighbor pixels by interpolation) on the spherical surface, as the pixel value of the pixel (X_II′, Y_II′). The specific steps to calculate the spherical coordinate “Coordinate” according to the plane coordinate(X_II′, Y_II′) are as follows:

(2.1) As shown in FIG. 3, the pixel (X_II′, Y_II′) corresponds to the point A_II′ on the side surface of the quadrangular pyramid Q′, the connecting line between the midpoint of the base edge of the side surface and the apex of the quadrangular pyramid Q′ is denoted as n₁, and the connecting line between the center of the bottom plane and the apex of the quadrangular pyramid Q′ is denoted as n₂, and the projections of point A_II′ on n₁ and n₂ are denoted as B_II′ and C_II′, respectively. The magnitude of the angle B_II′OD and the angle A_II′C_II′B_II′ are calculated according to the value of (X_II′, Y_II′), and the calculation formulas are as follows:

∠  B II ′  OD = f  ( X II ′ ) ∠   A II ′  C II ′  B II ′ = ( Y II ′ - L 2 ) ÷ ( H - X II ′ H × L ) × 90 °

Wherein, the f function can be any function that satisfies the following conditions:

θ=f(0)

180°=f(H)

Wherein, H is the height of the isosceles triangle, and the heights of the four isosceles triangular planes II′, III′, IV′ and V′ are respectively

H II ′ 2 , W II ′ 2 , H II ′ 2 , W II ′ 2 ;

L is the base of an isosceles triangle, and the heights of the four isosceles triangular planes II′, III′, IV′ and V′ are respectively W_II′, W_II′ and H_II′;

(2.2) The coordinate of A_II′, “Coordinate′”, which is expressed by Cartesian coordinates, is determined according to the magnitude of the angle B_II′OD, the magnitude of the angle A_II′C_II′B_II′ and the values of β₁, β₂ and β₃, and the calculation formula thereof is:

  { ( X A II ′ , Y A II ′ , Z A II ′ ) = k × ( 1 , tan  ( ∠   A II ′  C II ′  B II ′ ) , cot  ( ∠  B II ′  OD ) ) × T Isosceles   triangle   II ′ ( X A II ′ , Y A II ′ , Z A II ′ ) = k × ( tan  ( ∠   A II ′  C II ′  B II ′ ) , 1 , cot  ( ∠  B II ′  OD ) ) × T Isosceles   triangle   III ′ ( X A II ′ , Y A II ′ , Z A II ′ ) = k × ( - 1 , - tan  ( ∠   A II ′  C II ′  B II ′ ) , cot  ( ∠  B II ′  OD ) ) × T Isosceles   triangle   IV ′ ( X A II ′ , Y A II ′ , Z A II ′ ) = k × ( - tan  ( ∠   A II ′  C II ′  B II ′ ) , - 1 , cot  ( ∠  B II ′  OD ) ) × T Isosceles   triangle   V ′

Wherein, T is a 3×3 dimensional rotation matrix generated according to the pitch angle β₁, the yaw angle β₂ and the roll angle β₃; k is the length of the line segment C_II′B_II′;

(2.3) According to the coordinate of the point A_II′, the coordinate of the intersection of the ray {right arrow over (OA_II′)} and the spherical surface, “Coordinate”, is obtained, and the coordinate “Coordinate” is represented by the Cartesian coordinates, and the calculation formula thereof is:

( X sphere , Y sphere , Z sphere ) = R × ( X A I ′ , Y A I ′ , Z A I ′ )  X A I ′ , Y A I ′ , Z A I ′ 

Wherein, R is the radius of the sphere;

 X A I ′ , Y A I ′ , Z A I ′ 

represents the modulus of the line segment OA_I′, that is, the length of the line segment OA_I′, and dividing

( X A I ′ , Y A I ′ , Z A I ′ )

by

 X A I ′ , Y A I ′ , Z A I ′ 

is a normalization operation on

( X A I ′ , Y A I ′ , Z A I ′ ) .

It should be noted that the value of k in the above step (2.2) is unknown, but it does not affect the subsequent calculation, because k will be eliminated when the normalization operation is performed on

( X A I ′ , Y A I ′ , Z A I ′ )

in step (2.3).

Third step: splicing the four isosceles triangular planes II′, III′, IV′ and V′ obtained in the second step into a rectangular plane VI′ with a resolution of W_II′×H_II′;

So far, all the steps of the embodiment of the panoramic image mapping method are completed, and the effect of the embodiment is as shown in FIG. 4.

On the other hand, in an embodiment of the panoramic image inverse mapping method, mapping a two-dimensional image or video B back to a spherical surface based on a main viewpoint. The two-dimensional image or video B includes a rectangular plane I′ with a resolution of W_I′×H_I′ and a rectangular plane VI′ with a resolution of W_II′×B_II′, wherein, the rectangular plane VI′ may be further divided into four isosceles triangular planes II′, III′, IV′ and V′, and the base edge and the height of the four isosceles triangles are respectively

W II ′ × H II ′ 2 , H II ′ × W II ′ 2 , W II ′ × H II ′ 2 , H II ′ × W II ′ 2 .

The inverse mapping method firstly projects the above-mentioned plane onto a quadrangular pyramid by means of such as isometric projection, and then maps the quadrangular pyramid onto a spherical surface. Specifically, a right-handed coordinate system is established based on the sphere center O, wherein the Z axis points to the direction at longitude and latitude of (0°, 0°) of the spherical surface, the Y axis points to the direction of the north pole of the spherical surface, and the X axis points to the direction at longitude and latitude of (90°, 0°) of the spherical surface. Then, a quadrangular pyramid W is established, the center of the bottom plane of the quadrangular pyramid W is on the positive Z axis, the apex is on the negative Z axis, and the base edges are respectively parallel to the X axis and parallel to the Y axis; the connecting line between the center D of the bottom plane of the quadrangular pyramid and the sphere center O is denoted as l₁, and the connecting line between the midpoint of a base edge of the quadrangular pyramid and the sphere center O is denoted as l2, and the angle formed by l₁ and l₂ is θ, and θ represents the size of the main viewpoint region; the bottom plane of the quadrangular pyramid is denoted as plane I″, the side surface intersecting with the positive X axis is denoted as plane II″, the side surface intersecting with the positive Y axis is denoted as plane III″, the side surface intersecting with the negative X axis is denoted as plane IV″, and the side surface intersecting with the negative Y axis is denoted as plane V″. When the pitch angle, the yaw angle and the roll angle of the main viewpoint are respectively β₁, β₂ and β₃, a corresponding quadrangular pyramid Q′ is obtained by rotating the quadrangular pyramid Q about the sphere center O by (β₁, β₂, β₃). The sphere center O and the four base edges of the quadrangular pyramid Q′ form four sector planes, and the four sector planes divide the spherical surface into two regions. Wherein, one region including the main viewpoint is referred to as the main viewpoint region, and the main viewpoint region corresponds to the bottom plane of the pyramid and is denoted as region I; the other regions are referred to as the non-main viewpoint regions, and the center O and four side edges of the quadrangular pyramid Q′ form four sector planes, which further divide one non-main viewpoint region into four sub-regions, and the four sub-regions respectively correspond to the four side surfaces II′, III′, IV′ and V′, and are respectively referred to as regions II, III, IV and V. The inverse mapping method of the present invention maps the rectangular plane I′ with the resolution of W_I′×H_I′ contained in the two-dimensional image or video B onto the main viewpoint I of the spherical surface, and maps the four isosceles triangular planes II′, III′, IV′ and V′ contained in the two-dimensional image or video B onto the non-main viewpoints II, III, IV and V of the spherical surface. The values of the parameters θ, W_I′, H_I′, W_II′, H_II′, β₁, β₂ and β₃ can be obtained from a bit stream, but is not limited thereto. The specific processes of the panoramic image inverse mapping method are as follows:

First step: according to the coordinate of the point on the spherical surface, “Coordinate”, and the values of β₁, β₂, and β₃, determining which one of the regions I, II, III, IV and V the point is located in; the specific determination method is:

Firstly, the coordinate of the point on the sphere, “Coordinate”, is rotated according to β₁, β₂ and β₃, wherein the coordinate “Coordinate” is expressed by Cartesian coordinates as (X_sphere, Y_sphere, Z_sphere), and the point after rotation is referred to as A, and the coordinate of A is:

(X_A,Y_A,Z_A)=(X_sphere,Y_sphere,Z_sphere)×T′

Wherein, T′ is a 3×3 dimensional rotation matrix generated according to a pitch angle −β₁, a yaw angle −β₂ and a roll angle −β₃;

As shown in FIG. 5, the point A is projected onto the ZOX plane and the ZOY plane, the projection points are point B and point C respectively, and the magnitudes of the angle BOD and the angle COD are calculated:

∠COD=arctan(Y_A,Z_A)

∠BOD=arctan(X_A,Z_A)

Determine the region where the point is located according to the angle BOD and the angle COD:

  { if - θ < ∠   COD < θ && - θ < ∠   BOD < θ belongs   to   region   I if - θ < ∠   COD < θ && ∠   BOD = θ belongs   to   region   I   or   II if - θ < ∠   COD < θ && ∠   BOD = - θ belongs   to   region   I   or   VI if ∠   COD = θ && - θ < ∠   BOD < θ belongs   to   region   I   or   III if ∠   COD = - θ && - θ < ∠   BOD < θ belongs   to   region   I   or   V

If not, the point is in region II, III, IV or V, and it need to be further determined according to the values of X_A, and Y_A:

  { X A > Y A > - X A belongs   to   region   II Y A > X A > - Y A belongs   to   region   III - X A > Y A > - X A belongs   to   region   IV - Y A > X A > Y A belongs   to   region   V X A = Y A && X A > - Y A belongs   to   region   II   or   III X A = - Y A && X A < Y A belongs   to   region   III   or   IV X A = Y A && X A < - Y A belongs   to   region   IV   or   V X A = - Y A && X A > Y A belongs   to   region   II   or   V

If the point is in region I, go to the second step; if the point is in region II, III, IV, or V, go to the fifth step;

Second step: as shown in FIG. 6, the point whose spherical coordinate is “Coordinate” is denoted as A_I′, and the coordinate “Coordinate” is denoted as (X_sphere, Y_sphere, Z_sphere) by the Cartesian coordinates. The connecting lines between the midpoints of the opposite base edges of the quadrangular pyramid Q′ are denoted as m₁ and m₂, and the projections of the point A_I′ on m₁ and m₂ are respectively denoted as B_I′ and C_I′, and the center of the bottom plane of the quadrangular pyramid Q′ is denoted as D. Calculate the magnitudes of the angle B_I′OD and the angle C_I′OD according to the coordinate of A_I′, “Coordinate”, and the values of β₁, β₂ and β₃:

( X A I ′ , Y A I ′ , Z A I ′ ) = ( X sphere , Y sphere , Z sphere ) × T ′ ∠   C I ′  OD = arctan  ( Y A I ′ , Z A I ′ ) ∠   B I ′  OD = arctan  ( X A I ′ , Z A I ′ )

Wherein, T′ is a 3×3 dimensional rotation matrix generated according to the pitch angle −β₁, the yaw angle −β₂ and the roll angle β₃;

The calculation of the angle B_I′OD and the angle C_I′OD is the same as the calculation method of the angle BOD and the angle COD in the first step, therefore, the values of the angle BOD and the angle COD in the first step can also be directly used as the values of the angle B_I′OD and the angle C_I′OD;

Third step: calculate the value of the plane coordinate (X_I′, Y_I′) according to the magnitudes of the angle B_I′OD and the angle C_I′OD. The calculation formulas are as follows:

X I ′ = ∠   C I ′  OD - θ - 2   θ × H I ′ Y I ′ = ∠   B I ′  OD + θ 2   θ × W I ′

Fourth step: Use the pixel value at (X_I′, Y_I′) on the rectangular plane I′ (or the pixel value obtained by interpolation based on neighboring pixels) as the pixel value whose coordinate is “Coordinate” on the spherical surface; omit the subsequent steps;

Fifth step: as shown in FIG. 7, the point, whose spherical coordinate is “Coordinate”, is denoted as A_II′, and the coordinate “Coordinate” is denoted as (X_sphere, Y_sphere, Z_sphere) by the Cartesian coordinates. The connecting line between the midpoint of the base edge of the side surface and the apex of the quadrangular pyramid Q′ is denoted as n₁, and the plane passing n₁ and the sphere center O is denoted as α₃, and the connecting line between the center of the bottom plane and the apex of the quadrangular pyramid Q′ is denoted as n₂, and the projections of point A_II′ on α₃ and n₂ are denoted as B_II′ and C_II′, respectively, and the center of the bottom plane of the quadrangular pyramid Q′ is denoted as D. Calculate the magnitudes of the angle B_II′OD and the angle A_II′C_II′B_II′ according to the coordinate of A_II′, “Coordinate”, and the values of β₁, β₂ and β₃;

(X′_AII′,Y′_AII′,Z′_AII′)=(X_sphere,Y_sphere,Z_sphere)×T′

Wherein, T′ is a 3×3 dimensional rotation matrix generated according to the pitch angle −β₁, the yaw angle −β₂ and the roll angle −β₃;

{ ∠   A II ′  C II ′  B II ′ = arctan  ( Y A II ′ , X A II ′ ) , ∠   B II ′  OD = arctan  ( X A II ′ , Z A II ′ ) belongs   to   region   I ∠   A II ′  C II ′  B II ′ = arctan  ( X A II ′ , Y A II ′ ) , ∠   B II ′  OD = arctan  ( Y A II ′ , Z A II ′ ) belongs   to   region   III ∠   A II ′  C II ′  B II ′ = arctan  ( Y A II ′ , X A II ′ ) , ∠   B II ′  OD = arctan  ( - X A II ′ , Z A II ′ ) belongs   to   region   IV ∠   A II ′  C II ′  B II ′ = arctan  ( X A II ′ , Y A II ′ ) , ∠   B II ′  OD = arctan  ( - Y A II ′ , Z A II ′ ) belongs   to   region   V  

Sixth step: calculate the value of the plane coordinate (X_II′, Y_II′) according to the magnitudes of the angle B_II′OD and the angle A_II′C_II′B_II′ (the X and Y coordinate axes are respectively perpendicular to and parallel to the base of the isosceles triangle), The relationship between the value of Yip and the magnitude of the angle A_II′C_II′B_II′ is:

Y II ′ = ∠   A II ′  C II ′  B II ′ ÷ 90  ° × ( H - X II ′ H × L ) + L 2

The relationship between the magnitude of the angle B_II′OD and the value of X_II′ is ∠B_II′OD=f(X_II′), and the f function can be any function that satisfies the following conditions:

θ=f(0)

180°=f(H)

Wherein, H is the height of the isosceles triangle, and the heights of the four isosceles triangular planes II′, III′, IV′ and V′ are respectively

H II ′ 2 , W II ′ 2 , H II ′ 2 , W II ′ 2 ;

Seventh step: Use the pixel value at (X_II′, Y_II′) on the triangular plane (or the pixel value obtained by interpolation based on neighboring pixels) as the pixel value of the point with the coordinate “Coordinate” on the spherical surface; omit the subsequent steps;

So far, all the steps of the embodiment of the panoramic image inverse mapping method are completed.

It needs to be noted that the embodiments as disclosed are intended to facilitating further understanding of the present disclosure; however, those skilled in the art may understand that various substitutions and modifications are possible without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure should not be limited to the contents disclosed in the embodiments, but should be governed by the appended claims.