Delphi - How to create a generic REST call

Question

Using Delphi Seattle. I have an application that makes various REST calls. Some of these calls can return 10-20 rows via JSON, while others can return 30-40 thousand rows. I set my REST server to return 1000 rows. When the data returns to my client, I use RestDataAdapater, DataSource and the client dataset to expose the data as if it were a local table. This part is working fine. If we are at the end of 1000 lines, I change the URL and request the next batch of 1000 lines.

My task: I would like to abstract this so that one subroutine processes all the scripts (at least for GET calls). Complex parts - how can I process a data set of customer data / data containing 1000 rows? An example might help clarify ... I would like to be able to accomplish something like this ...

...
genericREST_Get(baseURL, resource, suffix);  // This would actually execute the REST call, where the components are in Datamodule DM1.
while not dm1.ds_Generic.DataSet.Eof do
        begin
       ... some kind of processing
       dm1.ds_Generic.DataSet.Next;
        end;

How do I handle the intersection threshold of 1000 lines? When my caller (shown above) goes from line 1000 to 1001, the REST API should request the next set of 1000 lines from the server. Although I know HOW to do this, I do not know WHERE to do this. I want the "get the next 1000 lines" to be in a common routine (e.g. genericREST_Get). I do not want each of the calling procedures to deal with this.

Assume that all routines will ONLY move forward, not backward.

Answer 1

:

1)
30-40 . , , . , , :

  repeat
    PartialData := genericREST_Get(baseURL, resource, suffix);
    // CopyDataSet is actually a FireDac method that I don't see on ClientDataSet
    // Basically just .Append and copy all fields with matching names.
    FullDataMemTable.CopyDataSet(PartialData);
  until PartialData.IsEmpty;

2) , DataSet , (Eof, FieldByName, Next ..). "Next" eof, . , DataModule. , - dm1.ds_Generic.DataSet.Next, dm1.Next.

constructor TDataFetcher.Create(BaseUrl, Resource, Suffix: string);
begin
  FBaseUrl := BaseUrl;
  FResource := Resource;
  FSuffix := Suffix;
end;

procedure TDataFetcher.Open;
begin
  FData := genericREST_Get(FBaseURL, FResource, FSuffix);
end;

procedure TDataFetcher.GetNextData;
begin
  FData := genericREST_Get(FBaseURL, FResource, FSuffix);
end;

function TDataFetcher.Eof: boolean;
begin
  result := FData.Eof;
end;

function TDataFetcher.FieldByName(FieldName: string): TField;
begin
  result := FData.FieldByName(FieldName);
end;

procedure TDataFetcher.Next;
begin
  FData.Next;
  if FData.Eof then
  begin
    GetNextData;
  end;
end;

:
a) TClientDataSet , TClientDataSet MoveBy:

function MoveBy(Distance: Integer): Integer; virtual;  

Inherited MoveBy EOF, . , , , . , , .Last? . , , .

function TMyDataSet.MoveBy(Distance: Integer): Integer; override;  
begin
  inherited MoveBy
  if self.Eof then
  begin
    FetchMoreData;
  end;
end;

b) FetchOnDemand
ClientDataSet FetchOnDemand. , RestDataAdapter. , , , , ClientDataSet .