No more extra GSI attributes

Recently, AWS announced support for multi-attribute composite keys on DynamoDB Global Secondary Indexes. It's not the most grounbreaking of the releases, but its definitely one that is welcome when you're building out indexes supporting complex queries across a few attributes. Before, you needed to create extra attributes with concatenated values that would be used as index keys. So for example, a trades table where you want a GSI to look up "all trades for a given wallet, sorted by timestamp" would end up with items shaped like this:

{
  "pk": "TRADE#0xabc123...",
  "sk": "METADATA",
  "walletAddress": "0xdef456...",
  "timestamp": 1730000000,
  "tradeId": "0xabc123...",
  "tokenIn": "USDC",
  "tokenOut": "WETH",
  "amountIn": "1000.00",
  "amountOut": "0.4",
  "gsi1pk": "WALLET#0xdef456...",
  "gsi1sk": "TS#1730000000#0xabc123..."
}

Both the gsi1pk and gsi1sk are created only to support such GSI and they're just made up from walletAddress, timestmp and the primary key of the trade. It requires you to always construct updates in a way that will keep those extra index attributes up-to-date.

With this new feature, the GSI definition itself describes how to compose the key from multiple source attributes. DynamoDB does the composition when it writes the GSI entry, and we no longer need those extra attrs, leaving us with clean:

{
  "pk": "TRADE#0xabc123...",
  "sk": "METADATA",
  "walletAddress": "0xdef456...",
  "timestamp": 1730000000,
  "tradeId": "0xabc123...",
  "tokenIn": "USDC",
  "tokenOut": "WETH",
  "amountIn": "1000.00",
  "amountOut": "0.4"
}

Creating the GSI

Defining a multi-attribute GSI turns out to be pretty simple, we just need to list multiple attributes with the same KeyType in the GSI's KeySchema. There are no separators or composition specs to worry about, DynamoDB takes care of all of that for us under the hood.

await client.send(new CreateTableCommand({
    TableName: 'Trades',
    KeySchema: [
        { AttributeName: 'pk', KeyType: 'HASH' },
        { AttributeName: 'sk', KeyType: 'RANGE' }
    ],
    AttributeDefinitions: [
        { AttributeName: 'pk', AttributeType: 'S' },
        { AttributeName: 'sk', AttributeType: 'S' },
        { AttributeName: 'walletAddress', AttributeType: 'S' },
        { AttributeName: 'timestamp', AttributeType: 'N' },
        { AttributeName: 'tradeId', AttributeType: 'S' }
    ],
    GlobalSecondaryIndexes: [{
        IndexName: 'WalletTradesIndex',
        KeySchema: [
            { AttributeName: 'walletAddress', KeyType: 'HASH' },
            { AttributeName: 'timestamp', KeyType: 'RANGE' },
            { AttributeName: 'tradeId', KeyType: 'RANGE' }
        ],
        Projection: { ProjectionType: 'ALL' }
    }],
    BillingMode: 'PAY_PER_REQUEST'
}));

The interesting bit is the KeySchema for our WalletTradesIndex, where we have one HASH attribute (walletAddress) and two RANGE attributes (timestamp and tradeId). DynamoDB will treat the combination of timestamp and tradeId as the composite sort key, with the ordering driven by the order they're listed in.

Querying the GSI

With our index in place, we can run queries against the natural attributes directly, no exta key construction needed on our side anymore:

// All trades for a given wallet
const q1 = await docClient.send(new QueryCommand({
    TableName: 'Trades',
    IndexName: 'WalletTradesIndex',
    KeyConditionExpression: 'walletAddress = :wallet',
    ExpressionAttributeValues: {
        ':wallet': '0xdef456...'
    }
}));

// Trades for a given wallet after a certain timestamp
const q2 = await docClient.send(new QueryCommand({
    TableName: 'Trades',
    IndexName: 'WalletTradesIndex',
    KeyConditionExpression: 'walletAddress = :wallet AND #ts >= :since',
    ExpressionAttributeNames: { '#ts': 'timestamp' },
    ExpressionAttributeValues: {
        ':wallet': '0xdef456...',
        ':since': 1730000000
    }
}));

// A specific trade by exact timestamp + tradeId
const q3 = await docClient.send(new QueryCommand({
    TableName: 'Trades',
    IndexName: 'WalletTradesIndex',
    KeyConditionExpression: 'walletAddress = :wallet AND #ts = :ts AND tradeId = :tid',
    ExpressionAttributeNames: { '#ts': 'timestamp' },
    ExpressionAttributeValues: {
        ':wallet': '0xdef456...',
        ':ts': 1730000000,
        ':tid': '0xabc123...'
    }
}));

A small thing to keep in mind, when constraining the sort key, the conditions have to go left-to-right, you can't skip an attribute or constrain a later one without constraining the earlier ones first. And if you use an inequality (like >=), it has to be on the last constrained attribute.

A few things worth knowing

• You can have up to 4 attributes in either the partition key or the sort key.
• If you add a new multi-attribute GSI to an existing table, DynamoDB indexes existing items automatically based on their native attributes, so there's no need to backfill anything.

For most single-table designs, this means we can finally get rid of those extra attributes and let DynamoDB take care of the composition for us. Thanks for reading!