Real time data warehouse Hologres_Interactive analysis MaxCompute AliCloud

Hologres (original interactive analysis)

Play video

Hologres is a one-stop real-time data warehouse engine, which supports real-time writing, real-time updating and real-time analysis of massive data, supports standard SQL (compatible with PostgreSQL protocol), supports petabyte level data multidimensional analysis (OLAP) and ad hoc analysis (Ad Hoc), supports high concurrency and low latency online data services (Serving), and is deeply integrated with MaxCompute, Flink and DataWorks, Provide off line integrated whole stack digital warehouse solution.

The original price of the package is 5313 yuan per 15000 CU, and only 59 yuan is needed for new purchase within a limited time

Developer community Product documentation 20 lectures on real-time data warehouse 10% discount for read-only instances (monthly package)

Latest information Best Practices The Road to Hologres Serverless - Uncover the Elastic Computing Group Preferential activities No need to export, no query, no charge, and speed up MaxCompute query with a 50% discount Technical disclosure Extreme analysis performance of JSON semi-structured data Best Practices Open download! Real Time Data Warehouse Hologres Best Practice 2.0 getting started Open download! Real Time Data Warehouse Hologres Technology Practice 2.0

Customer Stories Game industry 37 Mobile games are based on Flink+Hologres cloud practice E-commerce industry Construction and application of Flink+Hologres real-time data warehouse in Lazada Retail industry Taocai: the way to upgrade the real-time digital warehouse architecture based on Flink and Hologres Education industry Replace Kudu, and TAL will reduce costs and increase efficiency based on Hologres Advertising industry Alimama Dolphin intelligent computing engine is based on Flink+Hologres practice

Product advantages

One stop real-time data warehouse

Focus on real-time scenarios

Data is written and updated in real time, which is visible when written. It is integrated with Flink natively, and supports real-time data warehouse development with high throughput, low latency, and models to meet the real-time requirements of business insight.

Sub second interactive analysis

It supports sub second interactive analysis of massive data without pre calculation, and supports multidimensional analysis, ad hoc analysis, exploratory analysis, and MaxCompute accelerated analysis to meet the WYSIWYG analysis experience.

Unified data service export

It supports multiple scenarios such as multi-dimensional analysis, high-performance spot check, and data retrieval, supports load isolation, simplifies data architecture, unifies data access interfaces, and practices integration of analysis services (HSAP).

Open ecology

Standard SQL protocol, seamless connection between mainstream BI and SQL development framework, without application rewriting. It supports data lake scenarios, semi-structured data such as JSON, and simple warehousing of OSS and DLF.

Product Functions

Building enterprise level real-time data warehouse and real-time data middle platform

Multi scenario query analysis Support row storage, column storage and other storage modes and multiple kinds of indexes, and meet the diversified analysis and query needs of simple query, complex query, Ad Hoc query and so on. Using large-scale parallel processing architecture, distributed SQL processing, high resource utilization, rapid analysis of massive data, and best practice of Hybrid Serving/Analytical Processing (HSAP).

Sub second interactive analysis (OLAP) The scalable MPP architecture is used for full parallel computing, which gives full play to the maximum computing power of the CPU to the quantization operator. ORC format lists optimized indexes, SSD storage optimizes IO, and supports sub second interactive analysis experience of PB level data.

High performance primary key service The primary key index based on the row storage table and the short path optimization of the query engine support hundreds of thousands of QPS high-performance service type spot checks per second, support high throughput updates, and improve the performance of the open source system by more than 10 times.

Federated query, appearance acceleration Seamless connection to MaxCompute, no data movement required, supports transparent appearance to accelerate BI access, supports correlation analysis of cold and hot data, supports high-speed synchronization of data at millions of levels per second, supports OSS appearance reading and writing, and simplifies data warehousing into the lake.

Cloud native real-time data warehouse In view of the frequent updating and agile processing of real-time data warehouse data, it analyzes the characteristics of flexible self-help, supports highly concurrent real-time writing and updating, supports transaction isolation and atomicity, and data can be checked after being written.

High throughput real-time write and update It is natively integrated with Flink, Spark and other computing frameworks. Through the built-in connector, it supports real-time writing and updating of high-throughput data, multiple scenarios of source tables, result tables, dimension tables, and complex operations such as multi stream consolidation.

WYSIWYG development Data can be queried immediately after being written in real time. It supports DB, Schema and Table three-level systems, view view, update/delete natively, rich expression capabilities such as association, nesting and window, and semi-structured JSON data.

Full link event driven It supports the Binlog transmission capability of table update events, and uses Flink to consume Hologres Binlog to realize real-time development of the full link between data warehouse levels. On the premise of meeting hierarchical governance, it can shorten the end-to-end delay of data processing.

Enterprise level operation and maintenance Storage and computing separation architecture, computing and storage independent elastic scaling, support computing load, access rights and other fine-grained control requirements, provide rich monitoring and alarm, support system hot upgrade, and meet enterprise level safe and reliable operation and maintenance requirements.

data security It supports fine-grained access control policies, BYOK data storage encryption and data desensitization, data protection umbrella, IP white list, RAM, STS, independent account and other authentication systems, and passes PCI-DSS security authentication.

Load isolation Support load isolation based on resource groups, isolate different business requirements, different query types, write and read and other resource competition scenarios, and ensure the continuous stability of the system.

High reliability design Multiple computing instances form a highly reliable deployment mode. Instances share storage, support fault isolation and high availability of online services, and support rapid and automatic recovery of failed nodes. The storage and calculation are separated, and the resources are expanded independently, without the need for local disks. Pangu three copies are highly reliable and redundant storage.

Ecology and scalability It is compatible with PostgreSQL ecology and seamlessly connects with big data computing engine and big data intelligent research and development platform DataWorks. No additional learning is required, and you can start to develop immediately.

Compatible with PostgreSQL ecosystem It is compatible with PostgreSQL ecosystem, provides JDBC/ODBC interface, and easily interfaces with third-party ETL and BI tools, including QuickBI, DataV, Tableau, Fansoft, etc. Support GIS spatial data analysis.

DataWorks development integration It is deeply integrated with DataWorks, provides graphical, intelligent, one-stop digital warehouse building and interactive analysis service tools, and supports enterprise level capabilities such as data assets, data kinship, data real-time synchronization, and data services.

Dharma Hall Proxima vector retrieval Closely combined with the machine learning platform PAI, it has a built-in Proxima vector retrieval plug-in of the Dharma Institute, which supports online real-time feature storage, real-time recall, and vector retrieval.

Application scenarios

Help business take off quickly

E-commerce: real-time recommendation and precision marketing

Social networking on the Internet: real-time multidimensional analysis

Logistics: real-time order analysis and real-time monitoring

Internet service: data middle platform and real-time large screen

Real time precision marketing based on traffic data

With the gradual disappearance of traffic dividends and the rapid rise of innovation costs, the business development of a cross-border e-commerce company needs to gradually transform from the original barbaric growth. The original traditional Lambda architecture is complex, unable to support the multidimensional indicator fine-grained analysis, and unable to achieve the goal of refined operation. Through real-time digital warehouse upgrade, it has steadily supported the Double 11 over the years, saved nearly 50% of costs, improved efficiency by 300%, and achieved real-time precision marketing.

Scenario advantages

Real time service recommendation

Replace HBase with Hologres row storage table, Flink links Hologres dimension table in real time, generates user tags in real time, helps business real-time operation decision-making, and real-time users reach and recommend.

Flexible multidimensional analysis

The standard SQL syntax fully supports the complex multidimensional analysis demands of refined operation. The data processing delay is accelerated from 3 hours to real-time. The multi table association join returns in seconds, and the traffic matching efficiency is improved by 300%.

50% cost savings

The Flink+Hologres architecture supports multiple business scenarios, with flexible capacity expansion, simpler operation and maintenance, and effective resource savings of nearly 50%.

Recommended combination

Real time multidimensional analysis based on user log data

An Internet social networking company used to conduct real-time multi-dimensional analysis by building its own ClickHouse, but with business growth, recommendation business needs to adjust strategies and update models in a more real-time manner in typical OLAP analysis scenarios to achieve the purpose of recall and refined operation. By replacing the self built ClickHouse with Hologres, the whole link refined operation is built to meet the complex exploratory analysis under the ABTest scenario.

Scenario advantages

High real-time update efficiency

Instead of the self built ClickHouse, Hologres has a primary key, which can accurately remove duplicates, and is deeply integrated with Flink. It supports high throughput real-time write and update.

Easily query large amounts of data

Compared with the self built ClickHouse, it can easily conduct real-time multi-dimensional analysis on 7 or even 15 days' data and 100 billion level data to meet different business query requirements.

Operation and maintenance free

Hologres storage and computing are separated, and storage and computing resources are expanded independently. It can easily store 15 days of data, dynamically expand and shrink capacity, and avoid operation and maintenance. It only needs to focus on business development.

Recommended combination

Hologres

Flink

MaxCompute

Real time order analysis and real-time monitoring based on logistics data

The big data department of a freight logistics company has been exploring the construction of a new generation of digital warehouse, but has not made a great breakthrough, unable to make data play a greater value. The new generation of real-time data warehouse established by Hologres replaces the original ES, HBase and other architectures, and solves the problems of slow real-time analysis of tens of millions of order data and difficult real-time logistics scheduling of millions of freight drivers.

Scenario advantages

Flexible analysis of order data

The design reduces dimension degradation, supports real-time multidimensional analysis of ten million order data, and improves business query flexibility and development efficiency.

The delivery efficiency has been improved from a few days to a few minutes

Instead of HBase, Hologres provides online services to monitor logistics and warehouse abnormalities in real time and improve the stability of monitoring services and logistics delivery efficiency.

Unified data service outlet reduces architecture redundancy

Real time and offline are integrated into a set of architecture, sharing a single data, unifying data service outlets, improving data processing timeliness, and reducing architecture redundancy.

Recommended combination

Hologres

MaxCompute

Flink

Real time large screen and data middle platform based on business log

An Internet service company used to build real-time data warehouse through Greenplum and EMR offline architecture, but the timeliness of data update was poor, and it was unable to grasp business dynamics in real time. In order to meet the user growth demand of the whole scenario, Flink+Hologres new generation real-time data warehouse is used to build a real-time large screen and data middle platform system based on business log data to accelerate knowledge data exploration and promote rapid business development.

Scenario advantages

Second level response to report query

Perfect support for real-time report query of revenue, order volume and other indicators, meet the increasing requirements for data timeliness in enterprise operations, and respond in seconds.

Real time monitoring and real-time decision-making

Through the high concurrent read/write capability provided by Hologres and the association of the device status table, the status can be updated in real time to meet the real-time query and monitoring of the CRM system on the device (power bank) and help the business make real-time decisions.

Reduce operation and maintenance expenses

Replace open source Hive, Impala, etc., simplify business architecture, avoid data islands, consistency, security and other problems, and reduce development, operation and maintenance costs.

Recommended combination

Product function comparison

Why Hologres?

product positioning

Scene capability

system architecture

Engine capability

SQL support

Data update

Query engine

Storage Engine

Operation and maintenance capability

Resource elasticity

Cost of ownership

Enterprise level support

Cloud native real-time database Hologres

One stop real-time data warehouse, supporting OLAP, Ad Hoc and high QPS spot check

Cloud native computing storage separation architecture, multi node parallel writing, parallel computing

Support standard SQL, compatible with PostgreSQL 11 protocol

Real time write, real-time update, write visible, support primary key constraint

Distributed, vectorized and asynchronous execution engine with stable performance

There are two kinds of storage: row storage and column storage, which are based on Pangu system and optimized for SSD

Computing and storage expand elastically and independently on demand, without data relocation

Support billing by use, support data cold and hot storage, lower overall cost of ownership

Load isolation, high availability, fine-grained permissions, data encryption, etc

Traditional OLAP engine

Multi dimensional analysis (OLAP) oriented, suitable for static report presentation

Non cloud native architecture, high expansion cost, limited cluster size

Support limited SQL, and support for complex join, sub query, etc. is inefficient

Batch import, unable to update flexibly, low data timeliness

Unstable and jittery in high concurrency and complex query scenarios

Mainly column storage, and the storage is designed as HDD sequential scanning

Resource architecture coupling, resources cannot be expanded independently

The cost of purchasing software licenses is high, and the cost of talent recruitment is high

Mainly open source software, with less ability in governance and security

product positioning

Scene capability

data model

Data modeling

Query interface

Analytical capability

Data development

Development mode

Cost of ownership

Cloud native real-time database Hologres

One stop real-time data warehouse, supporting flexible analysis and high-performance spot check

Strong Schema, supporting rich data types

Perfect support for SQL, highly optimized for Scan scenarios, and high flexibility

Massive data sub second level query, support Join, high QPS PK click query

For data warehouse theme development, support table nesting, association and other complex scenarios

One set of architecture, multiple scenarios, reduce data fragmentation, and lower overall cost of ownership

Traditional Key/Value database

Support high-performance spot check based on Key/Value interface

Weak Schema, weak data type, difficult to troubleshoot data quality problems

Proprietary API, requiring third-party extension to support limited SQL

It supports spot check and prefix scanning, does not support complex multidimensional computing, and supports high QPS spot check

Complex development, system efficiency depends on primary key design, and there are many intermediate tables

Multiple systems, multiple data import and export, complex operation and maintenance

Customer Stories

Sun Tian, General Manager of Noah Fortune Data Intelligence Center:

Noah provides complex asset allocation services for high net worth customers. The business attribute of high-end financial services is naturally characterized by "fewer rows and more columns". The demand is extremely complex, and it is a deep-water area of data services. If it is not for the determination and excellent technology to change the industry with data, it is difficult to serve customers in the financial industry well. Hologres' partners not only provided us with high-level training in person, but also took pains to answer our various questions. Even if we asked questions late at night, they would respond quickly and solve them actively. What moved me more was that Noah, with the support of Hologres, did a lot of calculation optimization to improve the data calculation speed and reduce the cost by more than half. Hologres team is also a teacher and a friend, accompanying us to grow along the way, and we hope to build the best practice of financial big data together in the future!

Rutin, the person in charge of 2D Fire Big Data:

2D Fire is a company that focuses on the research, development and application of cloud computing catering software systems. It has more than 5000 Alibaba Cloud servers and serves 50w merchants. The big data team uses Alibaba Cloud big data service MaxCompute to conduct H+1/T+1 statistics on the purchase, sales, and inventory data of restaurants or retail stores, and uses interactive analysis Hologres to meet the high concurrency, second level analysis and processing of 10 billion level offline data, providing nearly 80 different types of reports, including business, order, food, membership and other reports, Provide multi-dimensional analysis and business exploration for store development.

58 Tao Wangfei, the person in charge of real-time calculation of fast dog big data:

Based on the real-time unified data service provided by Hologres, we provide subsecond simple ad hoc query return efficiency in most cases, which greatly promotes the real-time transformation of the business and solves a series of problems brought by the underlying multiple storage engines. In addition, for some unconventional real-time data requirements, Hologres' high performance is used to directly base on detailed data, reducing targeted development, while enhancing flexibility and improving development efficiency.

Li Yun, Xiaomai Senior Digital Warehouse Development Engineer:

Xiaomai Technology is a digital advertising company focusing on high-quality APP, user growth and commercialization. It has built 10+application systems through big data technology to enable digital business operations. However, with the exponential growth of the number of users, the business team's requirements for real-time and refined data have increased, and the big data system has begun to face challenges. Xiaomai Technology, through the introduction of the real-time data warehouse built by Hologres, has supported the tens of billions of business data complex multidimensional analysis second level response, improved the timeliness of data query and writing, and enhanced the system's high availability through the deployment of Hologres read-write separation instance, helping the rapid development of business.

Product specification

There is always a business suitable for you

Hologres calculation deduction package

Calculation Package Type

Mainland China - general computing package

Calculate packet capacity

15000CU * H (degrees)

term of validity

Please use up within one year, and expire

Click Buy Now to get the latest price

Hologres shared cluster

region

East China 2 (Shanghai)

Commodity characteristics

50% off time limit

Commodity advantages

Charge by scanning volume

Click Buy Now to get the latest price

Product Dynamics

One hand dynamic push

View details

2019-08-30 New Product

Interactive Analytics public beta release

View details

New products on March 20, 2020

Alibaba Cloud Intelligent Interactive Analysis (Hologres) Commercialized Release

View details

2020-07-31 New Region/New Availability Zone

Interactive analysis Hologres was officially launched in Hong Kong, Singapore, the United States (Silicon Valley) and Malaysia (Kuala Lumpur)

View details

2020-07-31 New Features/Specifications

Interactive analysis Hologres - one-stop data development and management platform HoloWeb officially launched

View details

2020-08-26 Function optimization

Alibaba Cloud interactive analysis Hologres and MaxCompute, DataWorks product portfolio sales function release

View details

2020-08-28 New Features/Specifications

Interactive analysis Hologres supports pay as you go subcontracting

View details

2020-10-27 New Features/Specifications

MaxCompute Interactive Analysis (Hologres) V0.8 was released, and the engine capability was enhanced

View details

2020-11-20 New Features/Specifications

Interactive analysis Hologres shared cluster (MaxCompute BI accelerated version) public beta release

View details

2021-01-25 New Features/Specifications

MaxCompute Interactive Analysis (Hologres) V0.9 was released to improve throughput

View details

2021-03-16 new region/new zone

Interactive analysis Hologres officially opened in North China 3 (Zhangjiakou)

View details

2021-04-01 Price adjustment

The user storage unit price of monthly subscription version and pay as you go version in domestic stations will be reduced by 50%

View details

2021-05-17 New Features/Specifications

Hologres is deeply integrated with MaxCompute, and the external speed is increased by 30%. Built in MC DDL functions are executed

View details

2021-05-20 New Features/Specifications

Enhanced operation and maintenance capability, slow query

View details

2021-05-24 New Features/Specifications

Support automatic table sampling to generate better execution plans

View details

2021-06-01 New functions/specifications

Add RoaringBitmap extension to optimize precise de duplication, label filtering and other scenarios under ultra-high base

View details

2021-06-08 New Features/Specifications

Read OSS data on the surface, integrate DLF, and simplify data lake data analysis

View details

2021-06-15 New Features/Specifications

Enhance traffic analysis scenarios, support fine-grained retention analysis, and optimize crowd selection, attribution and other scenarios

View details

2021-07-01 New functions/specifications

The optimized column storage adopts AliORC compression format by default, and the storage compression ratio increases by 30%~50%

View details

2021-08-01 New Features/Specifications

Use a new external table query engine to optimize the query performance of MaxCompute external tables

View details

2021-09-01 New functions/specifications

HoloWeb supports graphical viewing of execution plans to comprehensively improve query and analysis efficiency

View details

2021-10-25 New Region/New Availability Zone

Interactive analysis of Hologres shared cluster (MaxCompute BI accelerated version) was released in Singapore

View details

2021-11-19 New Region/New Availability Zone

Real time Hologres was commercially released in Germany (Frankfurt)

View details

2021-11-29 New Features/Specifications

Publish JSONB index function to speed up JSONB data query

View details

2022-01-06 Function optimization

Real time data warehouse Hologres upgraded new control console

View details

2022-01-10 Function optimization

Upgrade the shared cluster (MaxCompute BI accelerated version) to version 1.1

View details

2022-02-18 Function optimization

Shared cluster (MaxCompute BI accelerated version) starts bill pre push

View details

2022-03-22 New Features/Specifications

HoloWeb supports slow query Gantt chart analysis function

View details

2022-03-27 New Features/Specifications

The Hologres management console is upgraded, and the network configuration supports binding the specified VPC

View details

2022-05-05 New Features/Specifications

Hologres Read only Publish from Instance Self service Configuration

View details

2022-05-05 Function optimization

Query and monitoring interactive experience hall online

View details

2022-05-26 Function optimization

Publish Performance Tuning Guide

View details

2022-05-26 Function optimization

Build product life cycle management system

View details

2022-06-26 New Region/New Availability Zone

Shared cluster (MaxCompute BI accelerated version) opened in Singapore

View details

2022-07-26 Function optimization

Passed the ecological compatibility certification of the Haiguang CPU

View details

2022-07-26 New Features/Specifications

Worker level monitoring indicators are revealed to improve self diagnosis ability

View details

2022-08-17 New Region/New Availability Zone

Open the service at the US (Virginia) site

View details

2022-08-30 New Features/Specifications

Support OSS data lake acceleration: read Hudi/Delta and other surfaces

View details

2022-08-30 New Features/Specifications

Open beta of backup recovery function

View details

2022-09-25 New Features/Specifications

Support transmission encryption

View details

2022-09-27 New Features/Specifications

Support real-time materialized views

View details

2022-10-26 Function optimization

Pass the compatibility test of Feiteng processor

View details

2022-10-27 New Features/Specifications

JSON storage optimization

View details

2022-10-27 New Features/Specifications

Support dynamic partition management

View details

2022-11-22 New Features/Specifications

Data lake acceleration support OSS HDFS

View details

2022-11-22 New Features/Specifications

Hologres Self service Upgrade Function Release

View details

2022-11-23 Function optimization

Write or update the tuning guide

View details

2022-12-26 New Features/Specifications

Shared cluster supports data lake acceleration

View details

2022-12-26 Function optimization

Add UNIQ precise de duplication function

View details

2022-12-28 New Features/Specifications

Hologres docking Flink new function release

View details

2023-02-26 New Region/New Availability Zone

Sharing cluster publishing Shenzhen region

View details

2023-02-26 New Features/Specifications

Hologres realizes the integration of lake and warehouse based on Delta Lake

View details

2023-02-26 New Features/Specifications

Best Practices for Data Writing, Updating, and Spot Checking

View details

2023-03-23 New Features/Specifications

The shared cluster (MaxCompute BI acceleration version) is upgraded to the shared cluster (Hucang acceleration version)

View details

2023-03-23 New Features/Specifications

Supports hot and cold data tiered storage

View details

2023-04-19 New Features/Specifications

Developer experience version release

View details

2023-05-22 New Features/Specifications

Hologres publishes elastic computing group, OLAP analyzes fine-grained resource isolation

View details

2023-06-01 New Features/Specifications

Support Kafka real-time synchronization Hologres and ETL

View details

2023-06-15 New Features/Specifications

Support single instance shard level multiple replicas, improve instance throughput, and improve availability.

View details

2023-06-28 New Features/Specifications

Support instance load balancing and provide instance failover capability

View details

2023-07-01 New functions/specifications

Support hg_stat_activity to enrich SQL runtime diagnostic information

View details

2023-07-01 New functions/specifications

Support Runtime Filter to improve the performance of multi table association

View details

2023-07-01 New functions/specifications

Enrich Explain and Explain Analyze, and simplify SQL optimization methods

View details

2023-08-01 New Features/Specifications

Support OpenAPI to improve instance management capability

View details

2023-08-09 New Region/New Availability Zone

Add Zone J in East China 1 (Hangzhou)

View details

2023-08-09 New Region/New Availability Zone

Newly opened zone F in South China 1 (Shenzhen)

View details

2023-08-31 New Features/Specifications

ClickHouse Whole Database Migration Hologres

View details

2023-08-31 New Features/Specifications

Postgres real-time synchronization Hologres

View details

2023-10-08 Function optimization

Cloud monitoring supports monitoring of Hologres instance category

View details

2023-10-08 New Features/Specifications

Hologres monitoring indicators support memory classification

View details

2023-11-01 New Features/Specifications

Support interval funnel function and analyze traffic conversion in groups

View details

2023-11-01 Function optimization

COUNT DISTINCT is automatically optimized to improve query efficiency

View details

2023-11-01 New Features/Specifications

Add the hg_relation_size function to view the detailed storage

View details

2023-11-17 New Features/Specifications

Runtime Filter supports multiple fields Join

View details

2023-12-01 New Features/Specifications

Support Hologres instance management through Terraform

View details

2024-01-19 Function optimization

Holoweb supports the visualization of the Explain operator

View details

2024-01-25 New Features/Specifications

Holoweb supports Query insight and quickly associates table metadata

View details

2024-03-25 New Features/Specifications

Hologres supports instance level SQL diagnostics

View details

View all logs

Documentation and Tools

Help Documents

Hologres Product Help Document

Developer community

Hologres technology sharing and practice

Video Center

Customer case, best practice sharing

20 lectures on real-time data warehouse

Hologres from beginner to proficient

Start using Hologres

How to create instances and databases

developer's guide

Develop specifications, data types, and compatible functions

CREATE TABLE

How to create a table in Hologres

Supported Data Types

Data types and array types supported by Hologres

Publications and materials

Collection of Hologres e-books, open classes and case collections