{"id":15320,"date":"2020-11-30T10:22:36","date_gmt":"2020-11-30T09:22:36","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/"},"modified":"2020-11-30T10:22:36","modified_gmt":"2020-11-30T09:22:36","slug":"dynamodb-scan-pagination","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/","title":{"rendered":"DynamoDB Scan (and why 128.5 RCU?)"},"content":{"rendered":"

By Franck Pachot<\/h2>\n

.
\nIn the previous post<\/a> I described the PartiSQL SELECT for DynamoDB and mentioned that a SELECT without a WHERE clause on the partition key may result in a Scan, but the result is automatically paginated. This pagination, and the cost of a Scan, is something that may not be very clear from the documentation and I’ll show it here on the regular DynamoDB API. By not very clear, I think this is why many people in the AWS community fear that, with this new PartiQL API, there is a risk to full scan tables, consuming expensive RCUs. I was also misled, when I started to look at DynamoDB, by the AWS CLI “–no-paginate” option, as well as its “Consumed Capacity” always showing 128.5 even for very large scans. So those examples should, hopefully, clear out some doubts.<\/p>\n

I have created a HASH\/RANGE partitioned table and filled it with a few thousands of items:<\/p>\n

\naws dynamodb create-table --attribute-definitions \\\n  AttributeName=MyKeyPart,AttributeType=N \\\n  AttributeName=MyKeySort,AttributeType=N \\\n --key-schema \\\n  AttributeName=MyKeyPart,KeyType=HASH \\\n  AttributeName=MyKeySort,KeyType=RANGE \\\n --billing-mode PROVISIONED \\\n --provisioned-throughput ReadCapacityUnits=25,WriteCapacityUnits=25 \\\n --table-name Demo\n\nfor i in {1..5000} ; do aws dynamodb put-item --table-name Demo --item '{\"MyKeyPart\":{\"N\":\"'$(( $RANDOM \/1000 ))'\"},\"MyKeySort\":{\"N\":\"'$SECONDS'\"},\"MyUnstructuredData\":{\"S\":\"'$(printf %-1000s | tr ' ' x)'\"}}' ; done\n<\/code><\/pre>\n
Here is how those items look like:
\n\"\"<\/a><\/p>\n
I have created large items with a 1000 bytes “MyUnstructuredData” attribute. According to https:\/\/zaccharles.github.io\/dynamodb-calculator\/<\/a> an item size is around 1042 bytes. And that’s exactly the size I see here (5209783\/5000=1041.96) from the console “Items summary” (I waited a few hours to get it updated in the screenshot above). This means that around 1000 items can fit on a 1MB page. We will see why I’m mentioning 1MB here: the title says 128.5 RCU and that’s the consumed capacity when reading 1MB with eventual consistency (0.5 RCU per 4KB read is 128 RCU per 1MB). Basically, this post will try to explain why we see a 128.5 consumed capacity at maximum when scanning any large table:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL\\\n --no-consistent-read --output table\n\n----------------------------------\n|              Scan              |\n+----------+---------------------+\n|   Count  |    ScannedCount     |\n+----------+---------------------+\n|  5000    |  5000               |\n+----------+---------------------+\n||       ConsumedCapacity       ||\n|+----------------+-------------+|\n||  CapacityUnits |  TableName  ||\n|+----------------+-------------+|\n||  128.5         |  Demo       ||\n|+----------------+-------------+|\n[opc@a aws]$\n<\/code><\/pre>\n
TL;DR: this number is wrong \ud83d\ude09
\nYou cannot scan 5000 items of 1000 bytes with 128.5 RCU as this is nearly 5MB scanned and you need 0.5 RCU per 4KB reads.<\/p>\n
Output text<\/h3>\n
I’ve run this with “–output table” for a pretty print of it. Let’s have a look at the other formats (json and text):<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output json\n{\n    \"Count\": 5000,\n    \"ScannedCount\": 5000,\n    \"ConsumedCapacity\": {\n        \"TableName\": \"Demo\",\n        \"CapacityUnits\": 128.5\n    }\n}\n\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL\\\n --no-consistent-read --output text\n\n1007    None    1007\nCONSUMEDCAPACITY        128.5   Demo\n1007    None    1007\n1007    None    1007\n1007    None    1007\n972     None    972\n<\/code><\/pre>\n
The JSON format is similar to the TABLE one, but the TEXT output gives more information about this 128.5 consumed capacity as it appears after a count of 1007 items. Yes, this makes sense, 1007 is approximately the number of my items I expect in 1MB and, as I mentioned earlier, reading 1MB in eventual consistency consumes 128 RCU (0.5 RCU per 4KB). What actually happens here is pagination. A scan call is always limited to read 1MB at maximum (you can compare that to a fetch size in a SQL database except that it is about the amount read rather than returned) and what happens here is that the AWS CLI fetches the next pages in order to get the whole COUNT. Unfortunately, the “–return-consumed-capacity TOTAL” shows the value from the first fetch only. And only the TEXT format shows this count for each call. The TABLE and JSON formats do the sum of count for you (which is nice) but hide the fact that the Consumed Capacity is the one from the first call only.<\/p>\n
Debug<\/h3>\n
The partial display of the consumed capacity is a problem with AWS CLI but each call actually returns the right value, which we can see with “–debug”:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n --no-consistent-read --output table --debug 2>&1 | grep '\"ConsumedCapacity\"'\n\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":128.5,\"TableName\":\"Demo\"},\"Count\":1007,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"2\"},\"MyKeySort\":{\"N\":\"96744\"}},\"ScannedCount\":1007}'\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":128.5,\"TableName\":\"Demo\"},\"Count\":1007,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"27\"},\"MyKeySort\":{\"N\":\"91951\"}},\"ScannedCount\":1007}'\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":128.5,\"TableName\":\"Demo\"},\"Count\":1007,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"20\"},\"MyKeySort\":{\"N\":\"85531\"}},\"ScannedCount\":1007}'\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":128.5,\"TableName\":\"Demo\"},\"Count\":1007,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"29\"},\"MyKeySort\":{\"N\":\"90844\"}},\"ScannedCount\":1007}'\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":124.0,\"TableName\":\"Demo\"},\"Count\":972,\"ScannedCount\":972}'\n<\/code><\/pre>\n
Here the total RCU is 128.5+128.5+128.5+128.5+125=639 which is what we can expect here to scan a 5MB table (0.5*5209783\/4096=636).<\/p>\n
Pagination<\/h3>\n
To make the confusion bigger, there are several meanings in “pagination”. One is about the fact that a scan call reads at maximum 1MB of DynamoDB storage (and then at maximum 128.5 RCU – or 257 for strong consistency). And the other is about the fact that, from the AWS CLI, we can automatically fetch the next pages.<\/p>\n
I can explicitly ask to read only one page with the “–no-paginate” option (misleading name, isn’t it? It actually means “do not automatically read the next pages”):<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output json --no-paginate\n\n{\n    \"Count\": 1007,\n    \"ScannedCount\": 1007,\n    \"LastEvaluatedKey\": {\n        \"MyKeyPart\": {\n            \"N\": \"2\"\n        },\n        \"MyKeySort\": {\n            \"N\": \"96744\"\n        }\n    },\n    \"ConsumedCapacity\": {\n        \"TableName\": \"Demo\",\n        \"CapacityUnits\": 128.5\n    }\n}\n\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output text --no-paginate\n\n1007    1007\nCONSUMEDCAPACITY        128.5   Demo\nMYKEYPART       2\nMYKEYSORT       96744\n\n<\/code><\/pre>\n
Here, in all output formats, things are clear as the consumed capacity matches the number of items. The first 1MB page has 1007 items, which consumes 128.5 RCU, and, in order to know the total number, we need to read the next pages.<\/p>\n
Different than the auto pagination (automatically call the next page until the end), the read pagination always happens for scans: you will never read more than 1MB from the DynamoDB storage in one call. This is how the API avoids any surprise in the response time: the fetch size depends on the cost of data access rather than the result. If I add a filter to my scan so that no rows are returned, the same pagination happens:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output table \\\n--filter-expression \"MyUnstructuredData=:v\" --expression-attribute-values '{\":v\":{\"S\":\"franck\"}}'\n\n----------------------------------\n|              Scan              |\n+----------+---------------------+\n|   Count  |    ScannedCount     |\n+----------+---------------------+\n|  0       |  5000               |\n+----------+---------------------+\n||       ConsumedCapacity       ||\n|+----------------+-------------+|\n||  CapacityUnits |  TableName  ||\n|+----------------+-------------+|\n||  128.5         |  Demo       ||\n|+----------------+-------------+|\n\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output text \\\n--filter-expression \"MyUnstructuredData=:v\" --expression-attribute-values '{\":v\":{\"S\":\"franck\"}}'\n\n0       None    1007\nCONSUMEDCAPACITY        128.5   Demo\n0       None    1007\n0       None    1007\n0       None    1007\n0       None    972\n<\/code><\/pre>\n
Here no items verify my filter (Count=0) but all items had to be scanned (ScanndCount=5000). And then, as displayed with the text output, pagination happened, returning empty pages. This is a very important point to understand DynamoDB scans: as there is no access filter (no key value) it does a Full Table Scan, with the cost of it, and the filtering is done afterwards. This means that empty pages can be returned and you may need multiple roundtrips even for no rows. And this is what I had here with the COUNT: 5 calls to get the answer “Count: 0”.<\/p>\n
For my Oracle Database readers, you can think of DynamoDB scan operation like a “TABLE ACCESS FULL” in an execution plan (but not like a “TABLE ACCESS STORAGE FULL” which offloads the predicates to the storage) where you pay per throttled reads per second. The cost of the operation depends on the volume read (the size of the table) but not on the result. Yes, the message in DynamoDB is “avoid scan as much as possible” like we had “avoid full table scans as much as possible” in SQL databases, when used for OLTP, except for small tables. And guess what? The advantage of DynamoDB scan operation comes when you need to read a large part of the table because it can read many items with one call, with 1MB read I\/O size on the storage… Yes, 1MB, the same as what the db_file_multiblock_read_count default value has always set for maximum I\/O size behind in Oracle for full table scans. APIs change but many concepts are the same.<\/p>\n
Page size<\/h3>\n
We can control the page size, if we want more smaller pages (but you need very good reasons to do so), by specifying the number of items. In order to show that it is about the number of items scanned but not returned after filtering, I keep my filter that removes all items from the result:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output text --filter-expression \"MyUnstructuredData=:v\" --expression-attribute-values '{\":v\":{\"S\":\"franck\"}}' \\\n--page-size 500\n\n0       None    500\nCONSUMEDCAPACITY        64.0    Demo\n0       None    500\n0       None    500\n0       None    500\n0       None    500\n0       None    500\n0       None    500\n0       None    500\n0       None    500\n0       None    500\n0       None    0\n<\/code><\/pre>\n
Here each page is smaller than before, limited to 500 items. Then the RCU consumed by each call is smaller, as well as the response time. However, it is clear that the total number of RCU consumed is still the same, even higher:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output text --filter-expression \"MyUnstructuredData=:v\" --expression-attribute-values '{\":v\":{\"S\":\"franck\"}}' \\\n--page-size 10 \\\n--debug  2>&1 | awk -F, '\/b.{\"ConsumedCapacity\":{\"CapacityUnits\":\/{sub(\/[^0-9.]*\/,\"\");cu=cu+$1}END{print cu}'\n\n750.5\n<\/code><\/pre>\n
and the response time is higher because of additional roundtrips.<\/p>\n
We can define smaller pages, but never larger than 1MB:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output text --filter-expression \"MyUnstructuredData=:v\" --expression-attribute-values '{\":v\":{\"S\":\"franck\"}}' \\\n--page-size 1500\n\n0       None    1067\nCONSUMEDCAPACITY        128.5   Demo\n0       None    1074\n0       None    1057\n0       None    1054\n0       None    748\n<\/code><\/pre>\n
Here, even if I asked for 1500 items per page, I get the same as before because, given the item size, the 1MB limit is reached first.<\/p>\n
A few additional tests:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output table --page-size 1500 --no-paginate                                                                                             \n\nCannot specify --no-paginate along with pagination arguments: --page-size\n\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output table --page-size 1 --debug 2>&1 | grep '\"ConsumedCapacity\"' | head -2\n\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":0.5,\"TableName\":\"Demo\"},\"Count\":1,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"7\"},\"MyKeySort\":{\"N\":\"84545\"}},\"ScannedCount\":1}'\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":0.5,\"TableName\":\"Demo\"},\"Count\":1,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"7\"},\"MyKeySort\":{\"N\":\"85034\"}},\"ScannedCount\":1}'\n\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output table --page-size 10 --debug 2>&1 | grep '\"ConsumedCapacity\"' | head -2\n\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":1.5,\"TableName\":\"Demo\"},\"Count\":10,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"7\"},\"MyKeySort\":{\"N\":\"85795\"}},\"ScannedCount\":10}'\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":1.5,\"TableName\":\"Demo\"},\"Count\":10,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"7\"},\"MyKeySort\":{\"N\":\"86666\"}},\"ScannedCount\":10}'\n\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output table --page-size 100 --debug 2>&1 | grep '\"ConsumedCapacity\"' | head -2\n\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":13.0,\"TableName\":\"Demo\"},\"Count\":100,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"7\"},\"MyKeySort\":{\"N\":\"94607\"}},\"ScannedCount\":100}'\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":13.0,\"TableName\":\"Demo\"},\"Count\":100,\"LastEvaluatedKey\":{\"MyKeyPart\":{\"N\":\"8\"},\"MyKeySort\":{\"N\":\"89008\"}},\"ScannedCount\":100}'\n\n<\/code><\/pre>\n
First, I cannot disable auto pagination when defining a page size. This is why I used the debug mode to get the RCU consumed. Reading only one item (about 1KB) consumed 0.5 RCU because this is the minimum: 0.5 to read up to 4KB.  Then I called for 10 and 100 items per page. This helps to estimate the size of items. 13 RCU for 100 items means that the average item size is 13*4096\/0.5\/100=1065 bytes.<\/p>\n
You probably don’t want to reduce the page size under 1MB, except maybe if your RCU are throttled and you experience timeout. It is a response time vs. throughput decision. And in any case, a scan page should return many items. If I scan all my 5000 items with –page-size 1 will require 2500 RCU because each call is 0.5 at minimum:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL --no-consistent-read --output text --filter-expression \"MyUnstructuredData=:v\" --expression-attribute-values '{\":v\":{\"S\":\"franck\"}}' --page-size 1 --debug  2>&1 | awk -F, '\/b.{\"ConsumedCapacity\":{\"CapacityUnits\":\/{sub(\/[^0-9.]*\/,\"\");cu=cu+$1}END{print cu}'\n\n2500.5\n<\/code><\/pre>\n
This cost with the smallest page size is the same as reading each item with a getItem operation. So you see, except with this extremely small page example, that scan is not always evil. When you need to read many items, scan can get them with less RCU. How much is “many”? The maths is easy here. With 0.5 RCU you can read a whole 1MB page with scan, or just one item with getItem. Then, as long as, on average, you read more than one item per page, you get a benefit from scan. You can estimate the number of items you retreive. And you can divide the table size by 1MB. But keep in mind that if the table grows, the cost of scan increases. And sometimes, you prefer scalable and predictable response time over fast response time.<\/p>\n
Max items<\/h3>\n
In addition to the page size (number of items scanned) we can also paginate the result. This doesn’t work for count:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \\\n--no-consistent-read --output text --max-items 1\n\n1007    None    1007\nCONSUMEDCAPACITY        128.5   Demo\n1007    None    1007\n1007    None    1007\n1007    None    1007\n972     None    972\n<\/code><\/pre>\n
Here, despite the “–max-items 1” the full count has been returned.<\/p>\n
I’m now selecting (projection) two attributes, with a “–max-items 5”:<\/p>\n
[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort \\\n--return-consumed-capacity TOTAL --no-consistent-read --output text --max-items 5\n\n1007    1007\nCONSUMEDCAPACITY        128.5   Demo\nMYKEYPART       7\nMYKEYSORT       84545\nMYKEYPART       7\nMYKEYSORT       85034\nMYKEYPART       7\nMYKEYSORT       85182\nMYKEYPART       7\nMYKEYSORT       85209\nMYKEYPART       7\nMYKEYSORT       85359\nNEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9\n<\/code><\/pre>\n
This, like pagination, gives a “next token” to get the remaining items. <\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort \\\n--return-consumed-capacity TOTAL --no-consistent-read --output text --max-items 5 \\\n--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9\n\n0       0\nCONSUMEDCAPACITY        128.5   Demo\nMYKEYPART       7\nMYKEYSORT       85380\nMYKEYPART       7\nMYKEYSORT       85516\nMYKEYPART       7\nMYKEYSORT       85747\nMYKEYPART       7\nMYKEYSORT       85769\nMYKEYPART       7\nMYKEYSORT       85795\nNEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDEwfQ==\n<\/code><\/pre>\n
The displayed cost here is the same as a full 1MB scan: 128.5 RCU and, if you look at the calls with “–debug” you will see that a thousand of items were returned. However, the ScannedCount is zero:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort \\\n--return-consumed-capacity TOTAL --no-consistent-read --output table --max-items 5 \\\n--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9\n\n-----------------------------------------------------------------------------------------------------------\n|                                                  Scan                                                   |\n+-------+---------------------------------------------------------------------------------+---------------+\n| Count |                                    NextToken                                    | ScannedCount  |\n+-------+---------------------------------------------------------------------------------+---------------+\n|  0    |  eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDEwfQ==   |  0            |\n+-------+---------------------------------------------------------------------------------+---------------+\n||                                           ConsumedCapacity                                            ||\n|+---------------------------------------------------------+---------------------------------------------+|\n||                      CapacityUnits                      |                  TableName                  ||\n|+---------------------------------------------------------+---------------------------------------------+|\n||  128.5                                                  |  Demo                                       ||\n|+---------------------------------------------------------+---------------------------------------------+|\n||                                                 Items                                                 ||\n<\/code><\/pre>\n
Does it make sense? No items scanned but thousand if items retrieved, with the RCU of a 1MB read?<\/p>\n
Let’s try to answer this. I’ll query this again and update one item:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=ALL_ATTRIBUTES --return-consumed-capacity TOTAL <\n--no-consistent-read --output text --max-items 5 \\\n--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9 \\\n| cut -c1-80\n\n0       0\nCONSUMEDCAPACITY        128.5   Demo\nMYKEYPART       7\nMYKEYSORT       85380\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nMYKEYPART       7\nMYKEYSORT       85516\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nMYKEYPART       7\nMYKEYSORT       85747\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nMYKEYPART       7\nMYKEYSORT       85769\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nMYKEYPART       7\nMYKEYSORT       85795\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nNEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6ID\n\n[opc@a aws]$ aws dynamodb execute-statement \\\n--statement "update Demo set MyunstructuredData='Hello' where MyKeyPart=7 and MyKeySort =85516"\n\n------------------\n|ExecuteStatement|\n+----------------+\n<\/code><\/pre>\n
I’ve used the PartiQL SQL-Like API juszt because I find it really convenient for this.<\/p>\n
Now scanning from the same next token:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=ALL_ATTRIBUTES --return-consumed-capacity TOTAL <\n--no-consistent-read --output text --max-items 5 \\\n--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9 \\\n| cut -c1-80\n\n0       0\nCONSUMEDCAPACITY        128.5   Demo\nMYKEYPART       7\nMYKEYSORT       85380\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nMYKEYPART       7\nMYKEYSORT       85516\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nMYUNSTRUCTUREDDATA      Hello\nMYKEYPART       7\nMYKEYSORT       85747\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nMYKEYPART       7\nMYKEYSORT       85769\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nMYKEYPART       7\nMYKEYSORT       85795\nMYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nNEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6ID\n<\/code><\/pre>\n
Besides the fact that my update has added a new attribute MyunstructuredData rather than replacing MyUnstructuredData because I made a typo (not easy to spot with the text output as it uppercases all attributes names) the important point is that I’ve read the new value. Despite the “ScannedCount=0”, I have obviously read the items again. Nothing stays in cache and there are no stateful cursors in a NoSQL database.<\/p>\n
So just be careful with “–max-items”. It limits the result, but not the work done in one page read. RCU is always calculated from the number of 4KB that are read to get the page from the storage, far before any filtering. Where “–max-items” can limit the cost is when using auto pagination to avoid reading more pages than necessary:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort --return-consumed-capacity TOTAL \\\n--no-consistent-read --output text \\\n--max-items 2500 --debug 2>&1 | grep '\"ConsumedCapacity\"' | cut -c1-80\n\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":128.5,\"TableName\":\"Demo\"},\"Count\":1007,\"I\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":128.5,\"TableName\":\"Demo\"},\"Count\":1007,\"I\nb'{\"ConsumedCapacity\":{\"CapacityUnits\":128.5,\"TableName\":\"Demo\"},\"Count\":1007,\"I\n<\/code><\/pre>\n
Here limiting the displayed result to 2500 items only 3 pages (of 1007 items which are not filtered to the result) have been read.<\/p>\n
Consistency<\/h3>\n
I’ve run all those scans with “–no-consistent-read”, which is the default, just to make it implicit that we accept to miss the latest changes. With consistent reads, we are sure to read the latest, but requires more reads (from a quorum of mirrors) and doubles the RCU consumption:<\/p>\n
\n[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort \\\n--return-consumed-capacity TOTAL --consistent-read --output text --max-items 5 \\\n--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9\n\n0       0\nCONSUMEDCAPACITY        257.0   Demo\nMYKEYPART       7\nMYKEYSORT       85380\nMYKEYPART       7\nMYKEYSORT       85516\nMYKEYPART       7\nMYKEYSORT       85747\nMYKEYPART       7\nMYKEYSORT       85769\nMYKEYPART       7\nMYKEYSORT       85795\nNEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6ID==\n<\/code><\/pre>\n
The cost here is 257 RCU as 4KB consistent reads cost 1 RCU instead of 0.5 without caring about consistency.<\/p>\n
Query<\/h3>\n
I focused on the scan operation here, but the same multi-item read applies to the query operation. Except that you do not read the whole table, even with pagination, as you define a specific partition by providing a value for the partition key which will be hashed to one partition.<\/p>\n
Those tests are fully reproducible on the Free Tier. You don’t need billions of items to understand how it works. And once you understand how it works, simple math will tell you how it scales to huge tables.<\/p>\n","protected":false},"excerpt":{"rendered":"
By Franck Pachot . In the previous post I described the PartiSQL SELECT for DynamoDB and mentioned that a SELECT without a WHERE clause on the partition key may result in a Scan, but the result is automatically paginated. This pagination, and the cost of a Scan, is something that may not be very clear […]<\/p>\n","protected":false},"author":89,"featured_media":15322,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1865],"tags":[133,1866,2047,98],"type_dbi":[],"class_list":["post-15320","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aws","tag-aws","tag-dynamodb","tag-scan","tag-sql"],"acf":[],"yoast_head":"\nDynamoDB Scan (and why 128.5 RCU?) - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"DynamoDB Scan (and why 128.5 RCU?)\" \/>\n<meta property=\"og:description\" content=\"By Franck Pachot . In the previous post I described the PartiSQL SELECT for DynamoDB and mentioned that a SELECT without a WHERE clause on the partition key may result in a Scan, but the result is automatically paginated. This pagination, and the cost of a Scan, is something that may not be very clear […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2020-11-30T09:22:36+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2020-11-29-131556.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1978\" \/>\n\t<meta property=\"og:image:height\" content=\"1030\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Cloud Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Cloud Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"15 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/\"},\"author\":{\"name\":\"Cloud Team\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/92adc1c969d57f2b2f51e970b15b2f70\"},\"headline\":\"DynamoDB Scan (and why 128.5 RCU?)\",\"datePublished\":\"2020-11-30T09:22:36+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/\"},\"wordCount\":1905,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2022\\\/04\\\/Screenshot-2020-11-29-131556.jpg\",\"keywords\":[\"AWS\",\"DynamoDB\",\"scan\",\"SQL\"],\"articleSection\":[\"AWS\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/\",\"name\":\"DynamoDB Scan (and why 128.5 RCU?) - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2022\\\/04\\\/Screenshot-2020-11-29-131556.jpg\",\"datePublished\":\"2020-11-30T09:22:36+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/92adc1c969d57f2b2f51e970b15b2f70\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2022\\\/04\\\/Screenshot-2020-11-29-131556.jpg\",\"contentUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2022\\\/04\\\/Screenshot-2020-11-29-131556.jpg\",\"width\":1978,\"height\":1030},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/dynamodb-scan-pagination\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"DynamoDB Scan (and why 128.5 RCU?)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/92adc1c969d57f2b2f51e970b15b2f70\",\"name\":\"Cloud Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1fd4abaa3d2f79eb3c430f8ababb33014273b4168e2652ca915d59c6ef5e8cec?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1fd4abaa3d2f79eb3c430f8ababb33014273b4168e2652ca915d59c6ef5e8cec?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1fd4abaa3d2f79eb3c430f8ababb33014273b4168e2652ca915d59c6ef5e8cec?s=96&d=mm&r=g\",\"caption\":\"Cloud Team\"},\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/author\\\/cloud\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"DynamoDB Scan (and why 128.5 RCU?) - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/","og_locale":"en_US","og_type":"article","og_title":"DynamoDB Scan (and why 128.5 RCU?)","og_description":"By Franck Pachot . In the previous post I described the PartiSQL SELECT for DynamoDB and mentioned that a SELECT without a WHERE clause on the partition key may result in a Scan, but the result is automatically paginated. This pagination, and the cost of a Scan, is something that may not be very clear […]","og_url":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/","og_site_name":"dbi Blog","article_published_time":"2020-11-30T09:22:36+00:00","og_image":[{"width":1978,"height":1030,"url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2020-11-29-131556.jpg","type":"image\/jpeg"}],"author":"Cloud Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Cloud Team","Est. reading time":"15 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/"},"author":{"name":"Cloud Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/92adc1c969d57f2b2f51e970b15b2f70"},"headline":"DynamoDB Scan (and why 128.5 RCU?)","datePublished":"2020-11-30T09:22:36+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/"},"wordCount":1905,"commentCount":0,"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2020-11-29-131556.jpg","keywords":["AWS","DynamoDB","scan","SQL"],"articleSection":["AWS"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/","url":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/","name":"DynamoDB Scan (and why 128.5 RCU?) - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/#primaryimage"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2020-11-29-131556.jpg","datePublished":"2020-11-30T09:22:36+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/92adc1c969d57f2b2f51e970b15b2f70"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/#primaryimage","url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2020-11-29-131556.jpg","contentUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screenshot-2020-11-29-131556.jpg","width":1978,"height":1030},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/dynamodb-scan-pagination\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"DynamoDB Scan (and why 128.5 RCU?)"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/92adc1c969d57f2b2f51e970b15b2f70","name":"Cloud Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1fd4abaa3d2f79eb3c430f8ababb33014273b4168e2652ca915d59c6ef5e8cec?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1fd4abaa3d2f79eb3c430f8ababb33014273b4168e2652ca915d59c6ef5e8cec?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1fd4abaa3d2f79eb3c430f8ababb33014273b4168e2652ca915d59c6ef5e8cec?s=96&d=mm&r=g","caption":"Cloud Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/cloud\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/15320","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/89"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=15320"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/15320\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media\/15322"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=15320"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=15320"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=15320"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=15320"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}