{"id":15410,"date":"2020-12-14T09:21:43","date_gmt":"2020-12-14T08:21:43","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/"},"modified":"2020-12-14T09:21:43","modified_gmt":"2020-12-14T08:21:43","slug":"oracle-write-consistency-bug-and-multi-thread-de-queuing","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/","title":{"rendered":"Oracle write consistency bug and multi-thread de-queuing"},"content":{"rendered":"<h2>By Franck Pachot<\/h2>\n<p>.<br \/>\n<i>This was initially posted on CERN Database blog where it seems to be lost. Here is a copy thanks to <a href=\"https:\/\/web.archive.org\/web\/20191229171006\/https:\/\/db-blog.web.cern.ch\/blog\/franck-pachot\/2018-09-oracle-write-consistency-bug-and-multi-thread-de-queuing\" target=\"_blank\" rel=\"noopener noreferrer\">web.archive.org<\/a><br \/>\n<strong>Additional notes:<\/strong><br \/>\n&#8211; I&#8217;ve tested and got the same behaviour in Oracle 21c<br \/>\n&#8211; you will probably enjoy reading Hatem Mahmoud going further on <a href=\"https:\/\/mahmoudhatem.wordpress.com\/2018\/10\/05\/write-consistency-and-dml-restart\/\" target=\"_blank\" rel=\"noopener noreferrer\">Write consistency and DML restart<\/a> <\/i><\/p>\n<p>Posted by <a class=\"username\" title=\"View user profile.\" href=\"https:\/\/twitter.com\/FranckPachot\">Franck Pachot<\/a> on Thursday, 27 September 2018<\/p>\n<p>Here is a quick test I did after encountering an abnormal behavior in write consistency and before finding some references to a bug on <a href=\"https:\/\/stackoverflow.com\/questions\/7577836\/write-consistency-with-nested-subquery-in-oracle\" target=\"_blank\" rel=\"noopener noreferrer\">StackOverflow<\/a> (yes, write consistency questions on StackOverflow!) and <a href=\"https:\/\/asktom.oracle.com\/pls\/asktom\/asktom.search?tag=write-consistency#3954570100346946285\" target=\"_blank\" rel=\"noopener noreferrer\">AskTOM<\/a>. And a bug opened by Tom Kyte in 2011, that is still there in 18c.<\/p>\n<p>The original issue was with a task management system to run jobs. Here is the simple table where all rows have a &#8216;NEW&#8217; status and the goal is to have several threads processing them by updating them to the &#8216;HOLDING&#8217; status&#8217; and adding the process name.<\/p>\n<pre><code>\nset echo on\ndrop table DEMO;\ncreate table DEMO (ID primary key,STATUS,NAME,CREATED)\n as select rownum,cast('NEW' as varchar2(10)),cast(null as varchar2(10)),sysdate+rownum\/24\/60 from xmltable('1 to 10')\n\/\n<\/code><\/pre>\n<p>Now here is the query that selects the 5 oldest rows in status &#8216;NEW&#8217; and updates them to the &#8216;HOLDING&#8217; status:<\/p>\n<pre><code>\nUPDATE DEMO SET NAME = 'NUMBER1', STATUS = 'HOLDING' \nWHERE ID IN (\n SELECT ID FROM (\n  SELECT ID, rownum as counter \n  FROM DEMO \n  WHERE STATUS = 'NEW' \n  ORDER BY CREATED\n ) \nWHERE counter &lt;= 5) \n;\n<\/code><\/pre>\n<p>Note that the update also sets the name of the session which has processed the rows, here &#8216;NUMBER1&#8217;.<\/p>\n<p>Once the query started, and before the commit, I&#8217;ve run the same query from another session, but with &#8216;NUMBER2&#8217;.<\/p>\n<pre><code>\nUPDATE DEMO SET NAME = 'NUMBER2', STATUS = 'HOLDING' \nWHERE ID IN (\n SELECT ID FROM (\n  SELECT ID, rownum as counter \n  FROM DEMO \n  WHERE STATUS = 'NEW' \n  ORDER BY CREATED\n ) \nWHERE counter &lt;= 5) \n;<\/code><\/pre>\n<p>Of course, this waits on row lock from the first session as it has selected the same rows. Then I commit the first session, and check, from the first session what has been updated:<\/p>\n<pre><code>\ncommit;\nset pagesize 1000\nselect versions_operation,versions_xid,DEMO.* from DEMO versions between scn minvalue and maxvalue order by ID,2;\n\nV VERSIONS_XID             ID STATUS     NAME       CREATED        \n- ---------------- ---------- ---------- ---------- ---------------\nU 0500110041040000          1 HOLDING    NUMBER1    27-SEP-18 16:48\n                            1 NEW                   27-SEP-18 16:48\nU 0500110041040000          2 HOLDING    NUMBER1    27-SEP-18 16:49\n                            2 NEW                   27-SEP-18 16:49\nU 0500110041040000          3 HOLDING    NUMBER1    27-SEP-18 16:50\n                            3 NEW                   27-SEP-18 16:50\nU 0500110041040000          4 HOLDING    NUMBER1    27-SEP-18 16:51\n                            4 NEW                   27-SEP-18 16:51\nU 0500110041040000          5 HOLDING    NUMBER1    27-SEP-18 16:52\n                            5 NEW                   27-SEP-18 16:52\n                            6 NEW                   27-SEP-18 16:53\n                            7 NEW                   27-SEP-18 16:54\n                            8 NEW                   27-SEP-18 16:55\n                            9 NEW                   27-SEP-18 16:56\n                           10 NEW                   27-SEP-18 16:57<\/code><\/pre>\n<p>I have used flashback query to see all versions of the rows. All 10 have been created and the the first 5 of them have been updated by NUMBER1.<\/p>\n<p>Now, my second session continues, updating to NUMBER2. I commit and look at the row versions again:<\/p>\n<pre><code>\ncommit;\nset pagesize 1000\nselect versions_operation,versions_xid,DEMO.* from DEMO versions between scn minvalue and maxvalue order by ID,2;\n<\/code><\/pre>\n<pre><code>\nV VERSIONS_XID             ID STATUS     NAME       CREATED        \n- ---------------- ---------- ---------- ---------- ---------------\nU 04001B0057030000          1 HOLDING    NUMBER2    27-SEP-18 16:48\nU 0500110041040000          1 HOLDING    NUMBER1    27-SEP-18 16:48\n                            1 NEW                   27-SEP-18 16:48\nU 04001B0057030000          2 HOLDING    NUMBER2    27-SEP-18 16:49\nU 0500110041040000          2 HOLDING    NUMBER1    27-SEP-18 16:49\n                            2 NEW                   27-SEP-18 16:49\nU 04001B0057030000          3 HOLDING    NUMBER2    27-SEP-18 16:50\nU 0500110041040000          3 HOLDING    NUMBER1    27-SEP-18 16:50\n                            3 NEW                   27-SEP-18 16:50\nU 04001B0057030000          4 HOLDING    NUMBER2    27-SEP-18 16:51\nU 0500110041040000          4 HOLDING    NUMBER1    27-SEP-18 16:51\n                            4 NEW                   27-SEP-18 16:51\nU 04001B0057030000          5 HOLDING    NUMBER2    27-SEP-18 16:52\nU 0500110041040000          5 HOLDING    NUMBER1    27-SEP-18 16:52\n                            5 NEW                   27-SEP-18 16:52\n                            6 NEW                   27-SEP-18 16:53\n                            7 NEW                   27-SEP-18 16:54\n                            8 NEW                   27-SEP-18 16:55\n                            9 NEW                   27-SEP-18 16:56\n                           10 NEW                   27-SEP-18 16:57\n<\/code><\/pre>\n<p>This is not what I expected. I wanted my second session to process the other rows, but here it seems that it has processed the same rows as the first one. What has been done by the NUMBER1 has been lost and overwritten by NUMBER2. This is inconsistent, violates ACID properties, and should not happen. An SQL statement must ensure write consistency: either by locking all the rows as soon as they are read (for non-MVCC databases where reads block writes), or re-starting the update when a mutating row is encountered. Oracle default behaviour is in the second case, NUMBER2 query reads the rows 1 to 5, because the changes by NUMBER1, not committed yet, are invisible from NUMBER2. But the execution should keep track of the columns referenced in the where clause. When attempting to update a row, now that the concurrent change is visible, the update is possible only if the WHERE clause used to select the rows still selects this row. If not, the database should raise an error (this is what happens in serializable isolation level) or re-start the update when in the default statement-level consistency.<\/p>\n<p>Here, probably because of the nested subquery, the write consistency is not guaranteed and this is a bug.<\/p>\n<p>One workaround is not to use subqueries. However, as we need to ORDER BY the rows in order to process the oldest first, we cannot avoid the subquery. The workaround for this is to add STATUS = &#8216;NEW&#8217; in the WHERE clause of the update, so that the update restart works correctly.<\/p>\n<p>However, the goal of multithreading those processes is to be scalable, and multiple update restarts may finally serialize all those updates.<\/p>\n<p>The preferred solution for this is to ensure that the updates do not attempt to touch the same rows. This can be achieved by a SELECT &#8230; FOR UPDATE SKIP LOCKED. As this cannot be added directly to the update statement, we need a cursor. Something like this can do the job:<\/p>\n<pre><code>\ndeclare counter number:=5;\nbegin\n for c in (select \/*+ first_rows(5) *\/ ID FROM DEMO \n           where STATUS = 'NEW' \n           order by CREATED\n           for update skip locked)\n loop\n  counter:=counter-1;\n  update DEMO set NAME = 'NUMBER1', STATUS = 'HOLDING'  where ID = c.ID and STATUS = 'NEW';\n  exit when counter=0;\n end loop;\nend;\n\/\ncommit;\n<\/code><\/pre>\n<p>This can be optimized further but just gives an idea of what is needed for a scalable solution. Waiting for locks is not scalable.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>By Franck Pachot . This was initially posted on CERN Database blog where it seems to be lost. Here is a copy thanks to web.archive.org Additional notes: &#8211; I&#8217;ve tested and got the same behaviour in Oracle 21c &#8211; you will probably enjoy reading Hatem Mahmoud going further on Write consistency and DML restart Posted [&hellip;]<\/p>\n","protected":false},"author":27,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[59],"tags":[2216,96,2217,2218],"type_dbi":[],"class_list":["post-15410","post","type-post","status-publish","format-standard","hentry","category-oracle","tag-cern","tag-oracle","tag-restart","tag-write-consistency"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Oracle write consistency bug and multi-thread de-queuing - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Oracle write consistency bug and multi-thread de-queuing\" \/>\n<meta property=\"og:description\" content=\"By Franck Pachot . This was initially posted on CERN Database blog where it seems to be lost. Here is a copy thanks to web.archive.org Additional notes: &#8211; I&#8217;ve tested and got the same behaviour in Oracle 21c &#8211; you will probably enjoy reading Hatem Mahmoud going further on Write consistency and DML restart Posted [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2020-12-14T08:21:43+00:00\" \/>\n<meta name=\"author\" content=\"Oracle Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Oracle Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/\"},\"author\":{\"name\":\"Oracle Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"headline\":\"Oracle write consistency bug and multi-thread de-queuing\",\"datePublished\":\"2020-12-14T08:21:43+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/\"},\"wordCount\":652,\"commentCount\":0,\"keywords\":[\"CERN\",\"Oracle\",\"restart\",\"write consistency\"],\"articleSection\":[\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/\",\"name\":\"Oracle write consistency bug and multi-thread de-queuing - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2020-12-14T08:21:43+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Oracle write consistency bug and multi-thread de-queuing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\",\"name\":\"Oracle Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"caption\":\"Oracle Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Oracle write consistency bug and multi-thread de-queuing - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/","og_locale":"en_US","og_type":"article","og_title":"Oracle write consistency bug and multi-thread de-queuing","og_description":"By Franck Pachot . This was initially posted on CERN Database blog where it seems to be lost. Here is a copy thanks to web.archive.org Additional notes: &#8211; I&#8217;ve tested and got the same behaviour in Oracle 21c &#8211; you will probably enjoy reading Hatem Mahmoud going further on Write consistency and DML restart Posted [&hellip;]","og_url":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/","og_site_name":"dbi Blog","article_published_time":"2020-12-14T08:21:43+00:00","author":"Oracle Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Oracle Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/"},"author":{"name":"Oracle Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"headline":"Oracle write consistency bug and multi-thread de-queuing","datePublished":"2020-12-14T08:21:43+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/"},"wordCount":652,"commentCount":0,"keywords":["CERN","Oracle","restart","write consistency"],"articleSection":["Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/","url":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/","name":"Oracle write consistency bug and multi-thread de-queuing - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2020-12-14T08:21:43+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/oracle-write-consistency-bug-and-multi-thread-de-queuing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Oracle write consistency bug and multi-thread de-queuing"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee","name":"Oracle Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","caption":"Oracle Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/15410","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/27"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=15410"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/15410\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=15410"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=15410"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=15410"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=15410"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}