One problem with cloud-native solution is that you need to access the service during the development of your application. This is not a major cost issue because DynamoDB is available on the Free Tier (with limited throughput, but that’s sufficient for development). But users may want to develop offline, on their laptop, without a reliable internet connection. And this is possible because Amazon provides a downloadable version of this database: DynamoDB Local.<\/p>\n
Difference<\/h3>\n
The most important thing is that the API is the same as with the cloud version. For sure, all the partitioning stuff is missing in the local version. And I have no idea if the underlying data format is similar or not: I am doing this installation on OEL 7.7 which is similar to RHEL 7.7 or CentOS 7.7<\/p>\n I have a JRE installed<\/p>\n I’m installing everything in a local temporary directory.<\/p>\n All is documented in: https:\/\/docs.aws.amazon.com\/amazondynamodb\/latest\/developerguide\/DynamoDBLocal.DownloadingAndRunning.html#download-locally<\/a><\/p>\n This simply downloads and extract the DynamoDB local distribution<\/p>\n This will use a persistent file (you can run it in memory only with -inMemory instead of it) in the directory mentioned by -dbPath and -sharedDb will use the following file name:<\/p>\n I’ll tell you more about this file later.<\/p>\n so, when started it displays on which port it listens:<\/p>\n Another port can be defined with -port<\/p>\n I use the AWS commandline interface, here is how to install it:<\/p>\n For the configuration, as you are local, you can put anything you want for the access key and region:<\/p>\n Because this information is not used, I’ll need to define the endpoint “–endpoint-url http:\/\/localhost:8000” with each call.<\/p>\n I mentioned some provisioned capacity ready for my test on the Free Tier but they are actually ignored by DynamoDB local.<\/p>\n Another difference with the cloud version is that this command returns immediately (no “CREATING” status).<\/p>\n I’ll put some items with Python, thus installing it.<\/p>\n boto3 is the AWS SDK for Python<\/p>\n Here is my demo.py program:<\/p>\n I just fill each collection with an increasing number of items.<\/p>\n The nice thing here is that you can see the ConsumedCapacity which gives you an idea about how it scales. Here, I read 45 items that have a size of 81 bytes and this is lower than 4K. Then the cost of it is 0.5 RCU for eventually consistent queries.<\/p>\n You know how I’m curious. If you want to build a local NoSQL database, which storage engine would you use?<\/p>\n Yes, this NoSQL database is actually stored in a SQL database!<\/p>\n They use SQLite for this DynamoDB Local engine, embedded in Java.<\/p>\n I have SQLite installed here and then can look at what is inside with my preferred data API: SQL.<\/p>\n Here is my Demo table accompanied with some internal tables. That looks like the version of the database (Config Table)<\/p>\n Here are the (DynamoDB Metadata) about my table, the DynamoDB ones, like “AttributeName”,”AttributeType” and their mapping to the SQLite “SQLiteColumnName”,”SQLiteDataType”,…<\/p>\n The tables sm, ss, us, and tr are empty and are related with Streams Metadata, Shard Metadata, Streams and Transactions and I may have a look at them for a next post.<\/p>\n Now the most interesting one: my Demo table. For this one, I’ve opened it in DBeaver<\/a>:<\/p>\n Actually, there’s a big advantage to have this NoSQL backed by a SQL database. During the development phase, you don’t only need a database to run your code. You have to verify the integrity of data, even after some race conditions. For example, I’ve inserted more items by increasing the ‘k’ loop in my demo.py and letting it run for 6 hours:<\/p>\n
\nhttps:\/\/twitter.com\/FranckPachot\/status\/1291313397171138561?s=20<\/a>However, this is more for curiosity. The local version just needs a compatible API. You will not measure the performance there. And from what I’ve discovered below, I’m sure the storage is completely different in the cloud version.<\/p>\nInstall<\/h3>\n
\n[oracle@cloud DynamoDBLocal]$ cat \/etc\/oracle-release\nOracle Linux Server release 7.7\n[oracle@cloud DynamoDBLocal]$ cat \/etc\/redhat-release\nRed Hat Enterprise Linux Server release 7.7 (Maipo)\n<\/code><\/pre>\n\n[oracle@cloud DynamoDBLocal]$ java -version\njava version \"1.8.0_231\"\nJava(TM) SE Runtime Environment (build 1.8.0_231-b11)\nJava HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)\n<\/code><\/pre>\n\nmkdir -p \/var\/tmp\/DynamoDBLocal && cd $_\n<\/code><\/pre>\n\ncurl https:\/\/s3.eu-central-1.amazonaws.com\/dynamodb-local-frankfurt\/dynamodb_local_latest.tar.gz | tar -xvzf -\n<\/code><\/pre>\nRun<\/h3>\n
\njava -Djava.library.path=\/var\/tmp\/DynamoDBLocal\/DynamoDBLocal_lib -jar \/var\/tmp\/DynamoDBLocal\/DynamoDBLocal.jar -sharedDb -dbPath \/var\/tmp\/DynamoDBLocal &\n<\/code><\/pre>\n\n[oracle@cloud ~]$ ls -l \/var\/tmp\/DynamoDBLocal\/shared-local-instance.db\n-rw-r--r-- 1 oracle oinstall 12346368 Aug 6 12:20 \/var\/tmp\/DynamoDBLocal\/shared-local-instance.db\n<\/code><\/pre>\n\n\n[oracle@cloud ~]$ pkill -f -- '-jar DynamoDBLocal.jar -sharedDb'\n\n[oracle@cloud ~]$ java -Djava.library.path=\/var\/tmp\/DynamoDBLocal\/DynamoDBLocal_lib -jar \/var\/tmp\/DynamoDBLocal\/DynamoDBLocal.jar -sharedDb -dbPath \/var\/tmp\/DynamoDBLocal &\n[1] 33294\n[oracle@cloud ~]$ Initializing DynamoDB Local with the following configuration:\nPort: 8000\nInMemory: false\nDbPath: \/var\/tmp\/DynamoDBLocal\nSharedDb: true\nshouldDelayTransientStatuses: false\nCorsParams: *\n\n<\/code><\/pre>\nAWS CLI<\/h3>\n
\nwget --continue https:\/\/awscli.amazonaws.com\/awscli-exe-linux-x86_64.zip\nunzip -oq awscli-exe-linux-x86_64.zip\nsudo .\/aws\/install\naws configure\n<\/code><\/pre>\n\n[oracle@cloud ~]$ aws configure\nAWS Access Key ID [****************chot]: @FranckPachot\nAWS Secret Access Key [****************chot]: @FranckPachot\nDefault region name [Lausanne]: Lausanne\nDefault output format [table]:\n<\/code><\/pre>\nCreate table<\/h3>\n
\naws dynamodb --endpoint-url http:\/\/localhost:8000 create-table \\\n --attribute-definitions \\\n AttributeName=MyKeyPart,AttributeType=S \\\n AttributeName=MyKeySort,AttributeType=S \\\n --key-schema \\\n AttributeName=MyKeyPart,KeyType=HASH \\\n AttributeName=MyKeySort,KeyType=RANGE \\\n --billing-mode PROVISIONED \\\n --provisioned-throughput ReadCapacityUnits=25,WriteCapacityUnits=25 \\\n --table-name Demo\n<\/code><\/pre>\n\n[oracle@cloud ~]$ aws dynamodb --endpoint-url http:\/\/localhost:8000 create-table \\\n> --attribute-definitions \\\n> AttributeName=MyKeyPart,AttributeType=S \\\n> AttributeName=MyKeySort,AttributeType=S \\\n> --key-schema \\\n> AttributeName=MyKeyPart,KeyType=HASH \\\n> AttributeName=MyKeySort,KeyType=RANGE \\\n> --billing-mode PROVISIONED \\\n> --provisioned-throughput ReadCapacityUnits=25,WriteCapacityUnits=25 \\\n> --table-name Demo\n--------------------------------------------------------------------------------------------------------------------------------------------------------\n| CreateTable |\n+------------------------------------------------------------------------------------------------------------------------------------------------------+\n|| TableDescription ||\n|+----------------------------------+------------+-----------------------------------------------------+------------+-----------------+---------------+|\n|| CreationDateTime | ItemCount | TableArn | TableName | TableSizeBytes | TableStatus ||\n|+----------------------------------+------------+-----------------------------------------------------+------------+-----------------+---------------+|\n|| 2020-08-06T12:42:23.669000+00:00| 0 | arn:aws:dynamodb:ddblocal:000000000000:table\/Demo | Demo | 0 | ACTIVE ||\n|+----------------------------------+------------+-----------------------------------------------------+------------+-----------------+---------------+|\n||| AttributeDefinitions |||\n||+------------------------------------------------------------------------+-------------------------------------------------------------------------+||\n||| AttributeName | AttributeType |||\n||+------------------------------------------------------------------------+-------------------------------------------------------------------------+||\n||| MyKeyPart | S |||\n||| MyKeySort | S |||\n||+------------------------------------------------------------------------+-------------------------------------------------------------------------+||\n||| KeySchema |||\n||+----------------------------------------------------------------------------------------+---------------------------------------------------------+||\n||| AttributeName | KeyType |||\n||+----------------------------------------------------------------------------------------+---------------------------------------------------------+||\n||| MyKeyPart | HASH |||\n||| MyKeySort | RANGE |||\n||+----------------------------------------------------------------------------------------+---------------------------------------------------------+||\n||| ProvisionedThroughput |||\n||+--------------------------------+---------------------------------+-----------------------------+-----------------------+-------------------------+||\n||| LastDecreaseDateTime | LastIncreaseDateTime | NumberOfDecreasesToday | ReadCapacityUnits | WriteCapacityUnits |||\n||+--------------------------------+---------------------------------+-----------------------------+-----------------------+-------------------------+||\n||| 1970-01-01T00:00:00+00:00 | 1970-01-01T00:00:00+00:00 | 0 | 25 | 25 |||\n||+--------------------------------+---------------------------------+-----------------------------+-----------------------+-------------------------+||\n<\/code><\/pre>\nPython<\/h3>\n
\nyum install -y python3\npip3 install boto3\n<\/code><\/pre>\nInsert some items<\/h3>\n
\nimport boto3, time, datetime\nfrom botocore.config import Config\ndynamodb = boto3.resource('dynamodb',config=Config(retries={'mode':'adaptive','total_max_attempts': 10}),endpoint_url='http:\/\/localhost:8000')\nn=0 ; t1=time.time()\ntry:\n for k in range(0,10):\n for s in range(1,k+1):\n r=dynamodb.Table('Demo').put_item(Item={'MyKeyPart':f\"K-{k:08}\",'MyKeySort':f\"S-{s:08}\",'seconds':int(time.time()-t1),'timestamp':datetime.datetime.now().isoformat()})\n time.sleep(0.05);\n n=n+1\nexcept Exception as e:\n print(str(e))\nt2=time.time()\nprint(f\"Last: %s\\n\\n===> Total: %d seconds, %d keys %d items\/second\\n\"%(r,(t2-t1),k,n\/(t2-t1)))\n<\/code><\/pre>\n\n[oracle@cloud DynamoDBLocal]$ python3 demo.py\nLast: {'ResponseMetadata': {'RequestId': '6b23dcd2-dbb0-404e-bf5d-57e7a9426c9b', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'application\/x-amz-json-1.0', 'x-amz-crc32': '2745614147', 'x-amzn-requestid': '6b23dcd2-dbb0-404e-bf5d-57e7a9426c9b', 'content-length': '2', 'server': 'Jetty(8.1.12.v20130726)'}, 'RetryAttempts': 0}}\n\n===> Total: 3 seconds, 9 keys 14 items\/second\n\n[oracle@cloud DynamoDBLocal]$\n\n<\/code><\/pre>\ncount items<\/h3>\n
\n[oracle@cloud DynamoDBLocal]$ aws dynamodb --endpoint-url http:\/\/localhost:8000 scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL\n----------------------------------\n| Scan |\n+----------+---------------------+\n| Count | ScannedCount |\n+----------+---------------------+\n| 45 | 45 |\n+----------+---------------------+\n|| ConsumedCapacity ||\n|+----------------+-------------+|\n|| CapacityUnits | TableName ||\n|+----------------+-------------+|\n|| 0.5 | Demo ||\n|+----------------+-------------+|\n<\/code><\/pre>\nshared-local-instance.db<\/h3>\n
\n[oracle@cloud DynamoDBLocal]$ cd \/var\/tmp\/DynamoDBLocal\n[oracle@cloud DynamoDBLocal]$ file shared-local-instance.db\nshared-local-instance.db: SQLite 3.x database\n<\/code><\/pre>\n\n[oracle@cloud DynamoDBLocal]$ sudo yum install sqlite\nLoaded plugins: ulninfo, versionlock\nExcluding 247 updates due to versionlock (use \"yum versionlock status\" to show them)\nPackage sqlite-3.7.17-8.el7_7.1.x86_64 already installed and latest version\nNothing to do\n<\/code><\/pre>\n\n[oracle@cloud DynamoDBLocal]$ sqlite3 \/var\/tmp\/DynamoDBLocal\/shared-local-instance.db\nSQLite version 3.7.17 2013-05-20 00:56:22\nEnter \".help\" for instructions\nEnter SQL statements terminated with a \";\"\n\n\nsqlite> .databases\nseq name file\n--- --------------- ----------------------------------------------------------\n0 main \/var\/tmp\/DynamoDBLocal\/shared-local-instance.db\n\nsqlite> .tables\nDemo cf dm sm ss tr us\n<\/code><\/pre>\n
\nLet’s look at the fixed tables there (which I would call the catalog or dictionary if DynamoDB was not a NoSQL database)<\/p>\n\nsqlite> .headers on\nsqlite> .mode column\nsqlite> select * from cf;\nversion\n----------\nv2.4.0\nsqlite>\n<\/code><\/pre>\n\nsqlite> select * from dm;\nTableName CreationDateTime LastDecreaseDate LastIncreaseDate NumberOfDecreasesToday ReadCapacityUnits WriteCapacityUnits TableInfo BillingMode PayPerRequestDateTime\n---------- ---------------- ---------------- ---------------- ---------------------- ----------------- ------------------ --------------------------------------------------------------------------------------------- ----------- ---------------------\nDemo 1596718271246 0 0 0 25 25 {\"Attributes\":[{\"AttributeName\":\"MyKeyPart\",\"AttributeType\":\"S\"},{\"AttributeName\":\"MyKeySort\",\"AttributeType\":\"S\"}],\"GSIList\":[],\"GSIDescList\":[],\"SQLiteIndex\":{\"\":[{\"DynamoDBAttribute\":{\"AttributeName\":\"MyKeyPart\",\"AttributeType\":\"S\"},\"KeyType\":\"HASH\",\"SQLiteColumnName\":\"hashKey\",\"SQLiteDataType\":\"TEXT\"},{\"DynamoDBAttribute\":{\"AttributeName\":\"MyKeySort\",\"AttributeType\":\"S\"},\"KeyType\":\"RANGE\",\"SQLiteColumnName\":\"rangeKey\",\"SQLiteDataType\":\"TEXT\"}]},\"UniqueIndexes\":[{\"DynamoDBAttribute\":{\"AttributeName\":\"MyKeyPart\",\"AttributeType\":\"S\"},\"KeyType\":\"HASH\",\"SQLiteColumnName\":\"hashKey\",\"SQLiteDataType\":\"TEXT\"},{\"DynamoDBAttribute\":{\"AttributeName\":\"MyKeySort\",\"AttributeType\":\"S\"},\"KeyType\":\"RANGE\",\"SQLiteColumnName\":\"rangeKey\",\"SQLiteDataType\":\"TEXT\"}],\"UniqueGSIIndexes\":[]} 0 0\nsqlite>\n<\/code><\/pre>\n![\"\"](\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Annotation-2020-08-06-153013.jpg\")
<\/a>
\nI have one SQLite table per DynamoDB table (global secondary indexes are just indexes on the table), one SQLite row per DynamoDB item, the keys (the HASH for partitioning and the RANGE for sorting within the partition) for which I used a string are stored as TEXT in SQLite but containing their ASCII hexadecimal codes (hashKey and rangeKey). And those are the columns for the SQLite primary key. They are also stored in an even larger binary (hashValue,rangeValue where hashValue is indexed), probably a hash function applied to it. And finally, the full item is stored as JSON in a BLOB. The itemSize is interesting because that’s what counts in Capacity Units (the sum of attribute names and attribute values).<\/p>\nThe power of SQL to verify the NoSQL database<\/h3>\n
\n[oracle@cloud aws]$ time aws dynamodb --endpoint-url http:\/\/localhost:8000 scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL\n----------------------------------\n| Scan |\n+-----------+--------------------+\n| Count | ScannedCount |\n+-----------+--------------------+\n| 338498 | 338498 |\n+-----------+--------------------+\n|| ConsumedCapacity ||\n|+----------------+-------------+|\n|| CapacityUnits | TableName ||\n|+----------------+-------------+|\n|| 128.5 | Demo ||\n|+----------------+-------------+|\n\nreal 0m50.385s\nuser 0m0.743s\nsys 0m0.092s\n<\/code><\/pre>\n