From 1e28ee21d514640f41c2ddc8605d01fef412ec15 Mon Sep 17 00:00:00 2001 From: fengyubiao Date: Wed, 31 May 2023 14:57:18 +0800 Subject: [PATCH 1/5] pip-163 --- pip/pip-263.md | 125 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 pip/pip-263.md diff --git a/pip/pip-263.md b/pip/pip-263.md new file mode 100644 index 0000000000000..fb05cc662cd77 --- /dev/null +++ b/pip/pip-263.md @@ -0,0 +1,125 @@ +# Background knowledge & Motivation + +Discussion: https://lists.apache.org/thread/tp25lom2ggztljlo76krsldo270f293j + +#### Just auto-create no-partitioned DLQ/Retry Topic + +If enabled the config `allowAutoTopicCreation,` Pulsar will auto-create a topic when the client loads it; +After setting config `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, Pulsar will auto-create a +partitioned topic(which have two partitions) when the client loads it. + +After the above, if using the feature [Retry Topic](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic) +and [DLQ](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic) enable topic auto-creation, +we will get a partitioned DLQ and a partitioned Retry Topic like this: +- `{primary_topic_name}-{sub_name}-DLQ` + -`{primary_topic_name}-{sub_name}-DLQ-partition-0` + -`{primary_topic_name}-{sub_name}-DLQ-partition-1` +- `{primary_topic_name}-{sub_name}-RETRY` + -`{primary_topic_name}-{sub_name}-RETRY-partition-0` + -`{primary_topic_name}-{sub_name}-RETRY-partition-1` + +---- + +I feel that almost all users will not use the multi-partitioned DLQ or multi-partitioned Retry topic because there is +a bug that causes the above behavior to be incorrect, but we have yet to receive any issues about it. This bug causes +the above behavior to look like this: When the partitioned DLQ is auto-created for the topic `tp1-partition-0`, Pulsar +will create a partitioned topic meta which has two partitioned but only create a topic +named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named `{primary_topic_name}-{sub_name}-DLQ-partition-x.` +Please look at this [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug description. + +So I want to change the behavior to Just auto-create no-partitioned DLQ/Retry Topic. + + +---- + +#### Prevent auto-create the DLQ for a DLQ +If we use regex-topic(not filter out the Retry topics) consumer and enable retry, and after several times restart the +client. it is possible to create such a topic `persistent://public/default/tp1-sub1-RETRY-sub2-RETRY-sub3-RETRY....`. +Please look at this [Discussion](https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for the detail. + +# Goals + +- Just auto-create no-partitioned DLQ/Retry Topic(with the other words: prevent auto-create partitioned DLQ) +- DLQ/Retry topic should not create for a DLQ/Retry Topic + - roles: + - DLQ will not auto-create for a DLQ + - Retry Topic will not auto-create for a Retry Topic + - DLQ will not auto-create for a Retry Topic + - Retry Topic will not auto-create for a DLQ + - client changes: Clients will not create a DLQ for a DLQ + - broker changes: rejected the request which wants to auto-create a DLQ for a DLQ + + +# High Level Design + + + +# Detailed Design + +## Design & Implementation Details + + + +## Public-facing Changes +nothing change. + +### Public API +nothing change. + +### Binary protocol + +#### CommandSubscribe.java +```java +/** + * This is an enumeration value with tree options: "standard", "dead letter", "retry letter". + */ +private String topicPurpose; +``` + +#### Properties of Topic +```properties +"purposeOfAutoCreatedTopic": value with tree options: "standard", "dead letter", "retry letter" +``` + +Note: Why not use two properties: `isAutoCreated` and `topicPurpose`? +Because there is a scenario like this: auto-create a topic, use it as a DLQ after a few days, and not use it as a DLQ +after a few days, this Topic will be allowed to have DLQ/Retry Topic. We only mark the topics created for DLQ/Retry +purposes. + +# Backward & Forward Compatability + +## Revert + +Nothing todo. + +## Upgrade + +This change and PR https://github.com/apache/pulsar/pull/19841 will be released in the same version. The bug fixed by +https://github.com/apache/pulsar/pull/19841 maybe cause data loss of Retry topic and DLQ. + +The identification of the bug +- `allowAutoTopicCreation = true` +- `allowAutoTopicCreationType = partitioned` +- there has any ZK node(partitioned topic) named `*-partition-*` under the path `/admin/partitioned-topics`, but there has no related partition under the path `/managed-ledgers/{tenant}/{namespace}/persistent/` + +### Workaround +delete the ZK node whose name matches `*-partition-*` under the path `/admin/partitioned-topics` + + + From 89d47d80a8c7f8a746680dac9735902dc0f4c015 Mon Sep 17 00:00:00 2001 From: fengyubiao Date: Wed, 31 May 2023 15:01:29 +0800 Subject: [PATCH 2/5] pip-163 --- pip/pip-263.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/pip/pip-263.md b/pip/pip-263.md index fb05cc662cd77..13df5870c9c3a 100644 --- a/pip/pip-263.md +++ b/pip/pip-263.md @@ -119,7 +119,4 @@ The identification of the bug - there has any ZK node(partitioned topic) named `*-partition-*` under the path `/admin/partitioned-topics`, but there has no related partition under the path `/managed-ledgers/{tenant}/{namespace}/persistent/` ### Workaround -delete the ZK node whose name matches `*-partition-*` under the path `/admin/partitioned-topics` - - - +delete the ZK node whose name matches `*-partition-*` under the path `/admin/partitioned-topics` \ No newline at end of file From 27b845fb039cba5d0d6344978f02ed15292cf85d Mon Sep 17 00:00:00 2001 From: fengyubiao Date: Wed, 31 May 2023 15:11:45 +0800 Subject: [PATCH 3/5] add the link of vote --- pip/pip-263.md | 1 + 1 file changed, 1 insertion(+) diff --git a/pip/pip-263.md b/pip/pip-263.md index 13df5870c9c3a..d0212660c9e9c 100644 --- a/pip/pip-263.md +++ b/pip/pip-263.md @@ -1,6 +1,7 @@ # Background knowledge & Motivation Discussion: https://lists.apache.org/thread/tp25lom2ggztljlo76krsldo270f293j +Vote: https://lists.apache.org/thread/10931v20ryr9m08qrl1yn8g6zlqfb4nc #### Just auto-create no-partitioned DLQ/Retry Topic From adff48640dd713a2db34b31dd552e341e04693b0 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Fri, 2 Jun 2023 10:38:25 +0800 Subject: [PATCH 4/5] Update pip-263.md --- pip/pip-263.md | 59 ++++++++++---------------------------------------- 1 file changed, 11 insertions(+), 48 deletions(-) diff --git a/pip/pip-263.md b/pip/pip-263.md index d0212660c9e9c..258e6ee350391 100644 --- a/pip/pip-263.md +++ b/pip/pip-263.md @@ -1,17 +1,13 @@ # Background knowledge & Motivation -Discussion: https://lists.apache.org/thread/tp25lom2ggztljlo76krsldo270f293j -Vote: https://lists.apache.org/thread/10931v20ryr9m08qrl1yn8g6zlqfb4nc +- Discussion: https://lists.apache.org/thread/tp25lom2ggztljlo76krsldo270f293j +- Vote: https://lists.apache.org/thread/10931v20ryr9m08qrl1yn8g6zlqfb4nc #### Just auto-create no-partitioned DLQ/Retry Topic -If enabled the config `allowAutoTopicCreation,` Pulsar will auto-create a topic when the client loads it; -After setting config `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, Pulsar will auto-create a -partitioned topic(which have two partitions) when the client loads it. +If enabled the config `allowAutoTopicCreation,` Pulsar will auto-create a topic when the client loads it; After setting config `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, Pulsar will auto-create a partitioned topic(which have two partitions) when the client loads it. -After the above, if using the feature [Retry Topic](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic) -and [DLQ](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic) enable topic auto-creation, -we will get a partitioned DLQ and a partitioned Retry Topic like this: +After the above, if using the feature [Retry Topic](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic) and [DLQ](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic) enable topic auto-creation, we will get a partitioned DLQ and a partitioned Retry Topic like this: - `{primary_topic_name}-{sub_name}-DLQ` -`{primary_topic_name}-{sub_name}-DLQ-partition-0` -`{primary_topic_name}-{sub_name}-DLQ-partition-1` @@ -21,12 +17,7 @@ we will get a partitioned DLQ and a partitioned Retry Topic like this: ---- -I feel that almost all users will not use the multi-partitioned DLQ or multi-partitioned Retry topic because there is -a bug that causes the above behavior to be incorrect, but we have yet to receive any issues about it. This bug causes -the above behavior to look like this: When the partitioned DLQ is auto-created for the topic `tp1-partition-0`, Pulsar -will create a partitioned topic meta which has two partitioned but only create a topic -named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named `{primary_topic_name}-{sub_name}-DLQ-partition-x.` -Please look at this [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug description. +I feel that almost all users will not use the multi-partitioned DLQ or multi-partitioned Retry topic because there is a bug that causes the above behavior to be incorrect, but we have yet to receive any issues about it. This bug causes the above behavior to look like this: When the partitioned DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will create a partitioned topic meta which has two partitioned but only create a topic named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at this [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug description. So I want to change the behavior to Just auto-create no-partitioned DLQ/Retry Topic. @@ -34,9 +25,7 @@ So I want to change the behavior to Just auto-create no-partitioned DLQ/Retry To ---- #### Prevent auto-create the DLQ for a DLQ -If we use regex-topic(not filter out the Retry topics) consumer and enable retry, and after several times restart the -client. it is possible to create such a topic `persistent://public/default/tp1-sub1-RETRY-sub2-RETRY-sub3-RETRY....`. -Please look at this [Discussion](https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for the detail. +If we use regex-topic(not filter out the Retry topics) consumer and enable retry, and after several times restart the client. it is possible to create such a topic `persistent://public/default/tp1-sub1-RETRY-sub2-RETRY-sub3-RETRY....`. Please look at this [Discussion](https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for the detail. # Goals @@ -69,39 +58,13 @@ DON'T ## Design & Implementation Details - - -## Public-facing Changes -nothing change. - -### Public API -nothing change. - -### Binary protocol - -#### CommandSubscribe.java -```java -/** - * This is an enumeration value with tree options: "standard", "dead letter", "retry letter". - */ -private String topicPurpose; -``` - -#### Properties of Topic +### Properties of Topic ```properties -"purposeOfAutoCreatedTopic": value with tree options: "standard", "dead letter", "retry letter" +"wasAutoCreated": boolean ``` -Note: Why not use two properties: `isAutoCreated` and `topicPurpose`? -Because there is a scenario like this: auto-create a topic, use it as a DLQ after a few days, and not use it as a DLQ -after a few days, this Topic will be allowed to have DLQ/Retry Topic. We only mark the topics created for DLQ/Retry -purposes. +- If the topic name ends with "-RETRY" or "-DLQ", only non-partitioned topics will be automatically created +- If the property `wasAutoCreated` of the topic is true and the topic name ends with "-RETRY" or "-DLQ", retry topics and dead letter queues will no longer be created for this topic. # Backward & Forward Compatability @@ -120,4 +83,4 @@ The identification of the bug - there has any ZK node(partitioned topic) named `*-partition-*` under the path `/admin/partitioned-topics`, but there has no related partition under the path `/managed-ledgers/{tenant}/{namespace}/persistent/` ### Workaround -delete the ZK node whose name matches `*-partition-*` under the path `/admin/partitioned-topics` \ No newline at end of file +delete the ZK node whose name matches `*-partition-*` under the path `/admin/partitioned-topics` From 968997f304006363ffd61d54b3427ac4738bdb98 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Fri, 2 Jun 2023 10:47:04 +0800 Subject: [PATCH 5/5] Update pip-263.md --- pip/pip-263.md | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-) diff --git a/pip/pip-263.md b/pip/pip-263.md index 258e6ee350391..fad2c64ff1991 100644 --- a/pip/pip-263.md +++ b/pip/pip-263.md @@ -39,26 +39,9 @@ If we use regex-topic(not filter out the Retry topics) consumer and enable retry - client changes: Clients will not create a DLQ for a DLQ - broker changes: rejected the request which wants to auto-create a DLQ for a DLQ - -# High Level Design - - - # Detailed Design -## Design & Implementation Details - -### Properties of Topic +**Properties of Topic** ```properties "wasAutoCreated": boolean ```