Migrating from useDqlQuery to useDql

[andycarlson]

I've migrated a few pages now to useDql without too many difficulties but the most recent one has proved more problematic.

This page uses DQL with some user entered filter parameters to query log data and display the results in a DataTableV2. Because it is querying logs the page only runs the query when the user presses the search button (the query is disabled and refetch() called when the button is pressed)

One issue I found is that the EmptyState text in the table panel no longer worked as expected - in fact often it did not work at all. Having dug through the differences in the DQL result objects I can definitely see that the new one has more complex semantics with new flags like isPending that did not exist for useDqlQuery. My page logic uses isLoading, isSuccess and tests whether the result data is null to control the view but perhaps that is no longer enough.

The second difference I have seen is that when editing the filter field values the data shown in the table will often magically change without the use of the search button and without the usual delay to fetch data from the server. I suspect this happens when the changed value has been 'seen' before and I guess the app engine is fetching it from cache somehow... which might be nice except that it isn't triggering my query completion effect (on the isLoading flag) to update other parts of the UI. 

Is there any more detailed guidance on the differences between useDqlQuery and useDql, particularly around migrating from old to new and integrating with DataTableV2 and whether it is possible to control (or perhaps be notified of) these unexpected table data refreshes somehow?

Thanks.

[imsingh]

Hi Andy

Do you have a code example that I can look at? 

Thanks

[andycarlson]

That is rather tricky without writing a whole new simpler app TBH, but here are some slightly anonymised snippets that hopefully show the relevant bits of the pre-migration page code...

  // mapped to text fields on the page

  const [filterA, setFilterA] = useState('');

  const [filterB, setFilterB] = useState('');

  const [filterC, setFilterC] = useState('');

  // this will be false for the use cases we are talking about

  const [autoFetch, setAutoFetch] = useState<boolean>(false);

  // typing in a text field sets this to true, query completion (effect) resets it

  const [filtersDirty, setFiltersDirty] = useState(false);

...

  const mylogDql = (filterA || filterB || filterC) ? buildQuery(filterA, filterB, filterC, timeframe, limit) : 'data json:"[]"';

  const mylogResult = useDqlQuery(

    { body: { query: mylogDql }, },

    { autoFetch: autoFetch, autoFetchOnUpdate: autoFetch });

  // the user pressed the find button
  function onFindClicked() {
    console.log('find clicked')
    mylogResult.refetch()
  }

...

  {mylogResult.isLoading && <ProgressCircle aria-label='Waiting for mylog log Results' />}

  {mylogResult.data && (

<DataTableV2 data={

filterByMsRange(msRange,

  filterByStatus(showStatus,

filterByLogSource(showLogSource,

  filterByLogType(showLogtype,mylogResult.data.records))))

  } columns={mylogColumns(tsMode)} sortable resizable>

  <DataTableV2.Toolbar>

<DataTableV2.VisibilitySettings />

<DataTableV2.DownloadData />

  </DataTableV2.Toolbar>

  <DataTableV2.Pagination defaultPageSize={10} />

  {((filterA || filterB || filterC) || (

  <DataTableV2.EmptyState>                

Please use at least one filter parameter above.

  </DataTableV2.EmptyState>              

  ))}

  {(filtersDirty && (filterA || filterB || filterC) && (

  <DataTableV2.EmptyState>                

Click the find button when you have finished entering the filter parameters.

  </DataTableV2.EmptyState>              

  ))}

...

[andycarlson]

The effect on query completion - extracts choices for client side filters and then triggers a second query to show a histogram...

 useEffect(() => {

    console.log('mylog Query Effect called mylogResult.isLoading=' + mylogResult.isLoading)

    if (!mylogResult.isLoading && mylogResult.isSuccess) {

        setFiltersDirty(false)

        setAutoFetch(false)

        let logtypes = extractColVals(mylogResult.data?.records, 'logtype');

        let showlt = new Object()

        for (let logtype of logtypes) {

            showlt[logtype] = true

        }

        setShowLogType(showlt)

        // similar for log sources and status (client side) filter choices

// this should trigger a second DQL for a histogram display. The max value is needed to calculate the histogram bucket size

        let maxval = extractMax(mylogResult.data?.records)

        setMaxRt(maxval)

      }

  }, [mylogResult.isLoading]);

I've just tried changing the page to build the query from a second set of filter fields to which the values are copied only when the find button is pressed. That stopped the table values 'magically' changing when typing in the filter fields but still the results seem somehow to be coming from cache when the values have been 'seen' before... so my effect doesn't run and the client side filter choices and histogram don't get updated.

[andycarlson]

I also thought to try forceRefetch()... I think this is re-running the query because I immediately see the data change (from cache I guess) and then a second or two later it changes again (presumably from the server)... but still the effect does not run and the 'loading' circly thing does not appear. My guess is that the result object never goes to isLoading=true - that would explain both of these symptoms.

[andycarlson]

Summarising where I think explanation is lacking...

• User feedback for empty tables/graphs in light of the new states possible with useDQL.
• User feedback during data loading. It seems now that there are cases where the data is being fully loaded from the server (e.g. with new filter parameters), partially loaded from the server (e.g. where the data is requeried because time has passed since the last query) or loaded from cache (which is fast and may not need feedback).
• Notification to the app code when the query results have changed, regardless of whether it came from the server, via a full reload, a partial reload or from cache.

The circly buffering thing is a fine way to do feedback for a full data reload but for a partial load it is probably best to leave the old data on display (which is what happens) but it would be good to have some way to provide feedback to the user to let them know that more data is being fetched to fill in the blank part of the graph (which I think is not happening now).

[imsingh]

Here are couple of things:

- Each time the hook returns new stuff, the whole `mylogResult` gets updated. You can verify that by using a useEffect on the variable. So your last bullet point is already actually happening. This is how react rendering works.

- Instead of using `isLoading`, I'd use `isFetching` to show ProgressCircle or any other loading mechanism. This will work for all the cases when data is being fetched from the server.

- The library is based on tanstack react query, and it doesn't have a boolean like isCached. However, you can create something like the following:
`const isCachedData = mylogResult.isFetched && !mylogResult.isFetching && !mylogResult.isStale;` 

I hope this answers your questions. 

[andycarlson]

Thanks, that is helpful and I will give those a try (and sorry for my slow reply - I was away)

Is there another UI option than ProgressCircle? - for example if only the last few minutes on a graph are being fetched it is a shame to wipe out the whole graph by replacing it with a buffering circly thing, but it is still nice to give the user some feedback to tell them that the missing bars on the right hand end of the axis are being fetched.

As I said earlier, I think it would be good to have some specific 'best practice' guidance somewhere in the developer docs about how to integrate the DQL stuff with the presentation stuff to take care of the multiple cases where data is stale/loading etc.

[andycarlson]

Some updates now that I have tried some of these suggestions...

Firstly, making the effect depend on logResult.data does indeed pick up changes regardless of whether they came from cache or a server side query so that suggestion solves that part of the puzzle thank you.

For feedback during partial updates I have resorted to a ProgressBar below the graph. I'm not sure this fits with the style guidance but it does seem to work. My user feedback code covering both the initial load and partial fetches looks like this...

            {(thruputResult.isLoading) && <ProgressCircle aria-label='Waiting for Query Results' />}

            {(thruputResult.isFetching && thruputResult.data) && <ProgressBar aria-label='Fetching More Results' density='condensed' />}

For queries where I need user control over when they run, simply migrating from useDqlQuery (with autofetch off and calling refetch()) to useDql (using enabled in place of autofetch) does not seem to work for me...

The first problem is that using the filter field state variables directly in the DQL query causes the displayed results to be refreshed (or more often just wiped out) as soon as the user starts any typing in the filter field on screen. The only solution I have found to this is to have a second set of 'shadow' state variables and copy the field values over when the user hits the 'find' button. In my case the update of shadow fields is further complicated by the use of intents and support for the 'back' button but I won't go off on that tangent just now.

I also stopped using 'refetch' for a couple of reasons - firstly in some cases it simply seemed to do absolutely nothing (and changing to forceRefetch did not help). Secondly there were cases where the shadow state variable value updates were pending on the next render. Instead I am now toggling the 'enabled' flag and also toggling the stale time between 60 seconds (when I want the query to run) and 24 hours (when I want the results to stay put).

I think I have something working reliably now but it is a lot more complex than what I had before and testing is complicated by the need to cover cases where the data has become stale or the original relative time window has expired completely.

Thanks again for your help.

[imsingh]

Hi Andy

Does the following code doesn't work?

```

const mylogResult = useDql(mylogDql,{enabled: autoFetch});

```

[andycarlson]

Yes it works in so much as it fetches the data but my issues are the ones that I summarised in my post on 27th Oct. My app needs to provide user feedback and client side filtering and the new API has different/extended semantics in these areas.