Release 0.1.0

Key Features

FeatureDescription
Adaptive Dynamic FilterThe dynamic feature is enhanced so that in addition to bloom filter, hashsets can be used to store the build side values to filter out the probe side. The filters are stored in a distributed memory store so that they can be reused by subsequent queries without having to be rebuilt.
Dynamically Add CatalogAdd catalog REST api allows an administrator to add new catalogs of any connector during run time. Catalog property files are written to a shared storage where they are discovered by openLooKeng cluster nodes and are registered and loaded. This does not require to restart server.
Cross Region Dynamic FilterIn a cross data center scenario when the probe side is in the remote data center and the build side is in the local data center, the adaptive filter will create a bloom filter. The data center connector is enhanced to send the bloom filter over the network to the remote data center, where it can be used to filter out the probe side. This reduces the amount of data that needs to be transferred over the network from the remote data center to the local data center.
Horizontal Scaling SupportIntroduced ISOLATING and ISOLATED node states so that a node can be quiesced during scale-in. ISOLATED nodes do not accept any new tasks.
VDMVirtual Data Mart, allows an administrator to create virtual catalogs, and virtual schemas. Within a virtual schema, the administrator can create views over tables spanning multiple data sources. Virtual data marts simplifies the access to tables across data sources, and across regions.
IUD support for ORCSupport Insert, Update, Delete on Hive transactional ORC tables. Only Hive 3.x is supported
Compaction for ORCSupport compaction of hive transactional ORC tables so that the number of files to be read is reduced there by increasing the data fetch per reader and hence helps in improving the query performance and also improves concurrency
CarbonData connector with IUD supportSupport Insert, Update, Delete Operations on CarbonData tables
Insert OverwriteSupport for Insert overwrite syntax.This is easy method for Truncating and loading into the existing table
ODBC connectorODBC Driver and gateway for the 3rd party BI tools like PowerBI, Tableau, YongHong Desktop to connect to openLooKeng
Dynamic Hive UDFDynamically load custom hive UDFs into openLooKeng
HBase ConnectorHBase Connector supports read data from HBase data source
Enhance Create Table statementAllow users to create Hive transactional ORC table; Allow users to specify the external location of managed hive tables when running CREATE TABLE AS command in openLooKeng
Metadata CacheA generic metadata cache SPI is introduced to provide a transparent caching layer that can be leveraged by any connector. The metadata cache delegates to the connector specific metadata if the cache does not exist. Currently being used by JDBC connectors, and DC Connector
Cross DC ConnectorA new connector is introduced to support responsive queries across a WAN allowing a client to query a data source that is presenting in another data center.
High Availability (Active-Active)Supports HA AA mode by storing runtime state information into a distributed cache like Hazelcast. Hazelcast cluster formation is done using a seed file. Discovery Service, OOM, & CPU Usage uses a distributed lock to ensure only one coordinator starts these services.
Sql Migration ToolSupplementary tool to assist in migrating the Hive SQL to openLooKeng Compatible SQLs
Code EnhancementRawSlice Optimization: This optimization reduces the memory footprint of openLooKeng by reusing the RawSlice object instead of constructing new Slice objects; Implicit Conversion: This feature supports data type implicit conversion. For example: If the query type does not match the table type in the Insert statement, it can convert the query type to the table type implicitly.

Obtaining the Document

For details, see https://gitee.com/openlookeng/hetu-core/tree/010/hetu-docs/en