Geek Answers Handbook

Unable to split pages into EMR cluster using boto

I have about 55 EMR clusters (all of them were completed) and tried to retrieve all 55 EMR clusters using the list_clusters method in boto . I searched for examples on pagination of the number of results from boto, but did not find any examples. Given this statement:

emr_object.list_clusters(cluster_states=["TERMINATED"], marker="what_should_i_use_here").clusters

I kept getting an InvalidRequestException error:

 boto.exception.EmrResponseError: EmrResponseError: 400 Bad Request <ErrorResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31"> <Error> <Type>Sender</Type> <Code>InvalidRequestException</Code> <Message>Marker 'what_should_i_use_here' is not valid.</Message> </Error> <RequestId>555b91bd-c122-11e3-8e31-abc75abdb39d</RequestId> </ErrorResponse>

What should I provide in the parameter marker , so that I can correctly split the request?

Thanks!

+1
source share
2 answers

I tried

 emr_object.describe_jobflows(states=["TERMINATED"])

and it works! This method returns all clusters.

0
source

You can go to None for the first time.

If you get a marker attribute from ClusterListResult, you can pass this later, for example

 m=None while True: try: cluster_list_result=emr_object.describe_jobflows(states=['TERMINATED'], marker=m) .... Do whatever with cluster_list_result.clusters m=cluster_list_result.marker # See if there are more except AttributeError: break
0
source

More articles:


All Articles