bulk data load errors in release 32

[ssteiner]

I've decided to give O2G another go in my provisioning solution. But, it fails right there at the data load (I cache various data so the GUI won't have to fetch it from the PBX every time somebody accesses the GUI). Data caches ist stuff like Entities, Costcenter, Device Templates, etc.

So I extract the list of objects, then I load every object returned in the list of objectIds (I can't use the attributes tag since that was broken from v029 to v032). So, loading my 2000 costcenters results in a majority of successful loads, and a couple dozen of 503s

Code: Select all

HTTP/1.1 503 Service Unavailable
Date: Sun, 12 May 2019 17:18:16 GMT
Server: Apache/2.4.23
Content-Length: 227
Connection: close
Content-Type: application/json

{"httpStatus":"SERVICE_UNAVAILABLE","code":1000,"helpMessage":"The server is not yet available. Please retry later if the initialization is not finished.","type":"NOT_READY","innerMessage":"Resource limitation","canRetry":true}

Subsequent loads work, there doesn't seem to be any regularly, except that in every attempt I make, I have at least a handful of these. When I originally wrote the code, with the pre-release and 1.0 release, things worked fine, so something was broken in subsequent releases.

[yblanchard]

we didn't see anything brought by last version which could explain this behahavior.

However, it will be interesting to know how the requests are thrown: are they in sequential mode or in parallel mode ?

We can already say that OXE CMIS simultaneous client number is limited ( probably maximum 4 parallel requests)

Our lab test (getting all the Subscriber data) show that using 4 parrallel requests leads to the better result, and since 8, some errors appear.

[Plassard]

This issue will be fixed with CRO2G-68 in version 33
Concurrent request number will be limited at O2G level

[ssteiner]

The requests are fired parallel. There's a single thread starting the extraction for every item and these use async (Task Parallel Framework in .NET). Nothing has changed in that code though - once listing items with selected attributes work again, I'll rewrite as much as possible to use the attribute selection approach which means a single request getting large datasets instead of making a large number of requests.

Is there a limit to the number of attributes that can be extracted? Worst case there's perhaps a dozen attributes i need...

When is release 33 due?