Hello!
I need to extract a large volume of data from Grail. It was not possible through the Notebook because I reached the limit of 100000 records,
I need data from the last 30 days, but I can extract it in 24-hour amounts to avoid problems.
The difficulty I am having is that, when executing my query in "/query:execute", because it is large, I receive the status "RUNNING".
Then I run "/query-poll" to receive the results. But two problems arise here. Or I get error 410, saying that the results are expired or the browser crashes (even though my computer has a good configuration), what would you recommend me to do?
My body:
{
"query": "fetch logs\n| filter dt.system.bucket==\"bucketABC\"\n| filter matchesValue(k8s.container.name, \"containerABC\") and matchesPhrase(content, \"content\")\n| parse content, \"\"\"DATA 'for customer ' SPACE? LD:CPF.passo1'\"'\"\"\"\n| fields `timestamp.passo1` = timestamp, `status.passo1` = status, `content.passo1` = content, CPF.passo1\n| lookup [fetch logs\n | filter dt.system.bucket==\"bucketABC\"\n\t | filter ((matchesValue(k8s.container.name, \"containerABC\") and matchesPhrase(content, \"content\") and matchesPhrase(content, \"content\")))\n\t | parse content, \"\"\"DATA 'customerId [' SPACE? LD:CPF.passo2']'\"\"\"\n | fields `timestamp.passo2` = timestamp, `status.passo2` = status, `content.passo2` = content, CPF.passo2], lookupField:CPF.passo2, sourceField:CPF.passo1, prefix:\"-\"\n | lookup [fetch logs\n | filter dt.system.bucket==\"bucketABC\"\n | filter ((matchesValue(k8s.container.name, \"containerABC\") and matchesPhrase(content, \"content\")))\n | parse content, \"\"\"DATA 'customerId [' SPACE? LD:CPF.passo3']'\"\"\"\n | fields `timestamp.passo3` = timestamp, `status.passo3` = status, `content.passo3` = content, CPF.passo3], lookupField:CPF.passo3, sourceField:CPF.passo1, prefix:\"--\"\n | lookup [fetch logs\n | filter dt.system.bucket==\"bucketABC\"\n | filter ((matchesValue(k8s.container.name, \"containerABC\") and (matchesPhrase(content, \"content\"))))\n | parse content, \"\"\"DATA 'customer ' SPACE? LD:CPF.passo4'\"'\"\"\"\n | fields `timestamp.passo4` = timestamp, `status.passo4` = status, `content.passo4` = content, CPF.passo4], lookupField:CPF.passo4, sourceField:CPF.passo1, prefix:\"---\"",
"defaultTimeframeStart": "2024-04-09T00:00:00.123Z",
"defaultTimeframeEnd": "2024-04-09T23:59:59.123Z",
"timezone": "GMT-3",
"locale": "en_US",
"maxResultRecords": 1000000000000,
"maxResultBytes": 1000000,
"fetchTimeoutSeconds": 600,
"requestTimeoutMilliseconds": 10000,
"enablePreview": true,
"defaultScanLimitGbytes": 500
}
}
