How to access a new Trino Connector plugin

How to access a new Trino Connector plugin

Background

Starting from version 4.0, Doris supports docking with the Trino Connector plugin. Through the rich Trino Connector plugin and Doris' Trino-Connector Catalog function, Doris can query more data sources.

The purpose of the Trino Connector compatibility framework is to help Doris quickly connect to more data sources to meet user needs.

For data sources such as Hive, Iceberg, Hudi, Paimon, and JDBC, we still recommend using the built-in catalog of Doris for connection to achieve better performance, stability, and compatibility.

This article mainly introduces how to adapt a Trino Connector plugin in Doris.

The following takes Trino's kafka Connector plugin as an example to introduce in detail how to adapt Trino's kafka Connector plugin in Doris, and then access the kafka data source through the Trino-Connector catalog function of Doris.

Note: Trino is an Apache License 2.0 protocol open source software provided by Trino Software Foundation. For details, please visit Trino official website.

Step 1: Compile Kakfa connector plugin

Trino does not provide officially compiled connector plugins, so we need to compile the required connector plugins ourselves.

Note: Since Doris currently uses the 435 version of the trino-main package, it is best to compile the 435 version of the connector plugin. There may be compatibility issues with non-435 versions of the connector plugin. If you encounter any problems, please provide feedback to the Apache Doris community.

1. Clone Trino source code $ git clone https://github.com/trinodb/trino.git
2. Switch Trino source code to version 435 $ git checkout 435
3. Enter the Kafka plugin source code directory $ cd trino/plugin/trino-kafka
4. Compile the Kafka plugin $ mvn clean install -DskipTest
5. After the compilation is completed, the target/trino-kafka-435 directory will be generated in the trino/plugin/trino-kafka/ directory.

Note: Each connector plugin is a subdirectory, not a jar package.

Step 2: Set up Doris's fe.conf / be.conf

After preparing the Kafka connector plug-in, you need to configure Doris's fe.conf and be.conf so that Doris can load the plug-in.

If we store the trino-kafka-435 directory prepared above in the /path/to/connectors directory, and then we should configure:

1. fe.conf

   Configure trino_connector_plugin_dir=/path/to/connectors in the fe.conf file (if the trino_connector_plugin_dir attribute is not configured in fe.conf, the ${Doris_HOME}/fe/connectors directory will be used by default).

2. be.conf

   Configure trino_connector_plugin_dir=/path/to/connectors in the be.conf file (if the trino_connector_plugin_dir attribute is not configured in be.conf, the ${Doris_HOME}/be/connectors directory will be used by default).

Note: Doris uses a lazy loading method to load the Trino Connector plug-in, which means that if it is the first time to use the Trino-Connector Catalog function in Doris, there is no need to restart the FE / BE node, Doris will automatically load the plug-in. However, the plug-in will only be loaded once, so if the plug-in in the /path/to/connectors/ directory changes, you need to restart the FE / BE node before the changed plug-in can be loaded.

Step 3: Using the Trino-Connector catalog feature

After completing the previous two steps, we can use the Trino-Connector Catalog function in Doris.

1. First let's create a Trino-Connector Catalog in Doris:

   create catalog kafka_tpch properties (
   "type"="trino-connector",
   -- The following four properties are derived from trino and are consistent with the properties in etc/catalog/kakfa.properties of trino
   "connector.name"="kafka",
   "kafka.table-names"="tpch.customer,tpch.orders,tpch.lineitem,tpch.part,tpch.partsupp,tpch.supplier,tpch.nation,tpch.region",
   "kafka.nodes"="localhost:9092",
   "kafka.table-description-dir" = "/mnt/datadisk1/fangtiewei"
   );

   explain:

   • type ：The type of catalog, here we must set it to trino-connector.
   • connector.name、kafka.table-names、kafka.nodes、kafka.table-description-dir The following four properties are derived from trino, refer to: Kafka connector

   Different Connector plug-ins should set different properties. You can refer to the official trino documentation: Connectors

2. Use catalog

   After we create the Trino-Connector catalog, there is no difference in use from other catalogs. Switch to the catalog through the switch kafka_tpch statement, and then you can query the data of the Kafka data source.

The following are the Doris Trino-Connector catalog configuration of several commonly used Connector plug-ins.

1. Hive

   create catalog emr_hive properties (
       "type"="trino-connector",
   
       "connector.name"="hive",
       "hive.metastore.uri"="thrift://ip:port",
       "hive.config.resources"="/path/to/core-site.xml,/path/to/hdfs-site.xml"
   );

   Note:

   • You should add Hadoop's user name in the JVM parameters: -DHADOOP_USER_NAME=ftw, which can be configured at the end of the JAVA_OPTS_FOR_JDK_17 parameter in the fe.conf / be.conf file, such as JAVA_OPTS_FOR_JDK_17="...-DHADOOP_USER_NAME=ftw"

2. Mysql

   create catalog trino_mysql properties (
       "type"="trino-connector",
       
       "connector.name"="mysql",
       "connection-url" = "jdbc:mysql://ip:port",
       "connection-user" = "user",
       "connection-password" = "password"
   );

   Note:

   • When encountering the error: Unknown or incorrect time zone: 'Asia/Shanghai', you need to add: -Duser.timezone=Etc/GMT-8 to the JVM startup parameters, which can be configured at the end of the JAVA_OPTS_FOR_JDK_17 parameter in the fe.conf / be.conf file.

3. Kafka

   create catalog kafka properties (
       "type"="trino-connector",
       
       "connector.name"="kafka",
       "kafka.nodes"="localhost:9092",
       "kafka.table-description-supplier"="CONFLUENT",
       "kafka.confluent-schema-registry-url"="http://localhost:8081",
       "kafka.hide-internal-columns" = "false"
   );

4. BigQuery

   create catalog bigquery_catalog properties (
       "type"="trino-connector",
   
       "connector.name"="bigquery",
       "bigquery.project-id"="steam-circlet-388406",
       "bigquery.credentials-file"="/path/to/application_default_credentials.json"
   );