[2022] ESI Market History Endpoint

Hey everyone!

I apologize that this post is being made a little later than I would have liked, unfortunately when this issue was originally raised, the length that the endpoint was going to be down for wasn’t as short as we had hoped.

Just to give a bit of a shortened version of events just to let you know what the state of play is concerning it:

• On 2 November the ESI endpoints for the market were essentially getting hammered so hard that it was spiking the CPU Use to 100% between 12:15 and 12:30. Just shy of 100 IP addresses had to get banned to restore it down to normality.

• On 3 November, between 12:15 and 12:30, the same thing happened, except it was a different set of IPs and they were all a part of AWS.

• When it happened again on 4 November, the decision was made that the endpoint was to be taken down due to the constant action it required each day and the subsequent performance issues it was causing.

The endpoint is going to likely have some degree of redesign or auth being added to it before the endpoint will be made available again, the current extent of this work and the teams involved are still being scoped out.

Once I have more details of what’s involved, or if I have any new developments to share with regards to it, I’ll relay that here.

It’s a good thought in the event that anything like this happens in the future, we actually considered it as well as a form of mitigation, but the system was designed to be open to all, then closed to specific IPs as required (i.e a blacklist system).

We also don’t want to punish any (new) developers trying to access the endpoints trying to develop apps that are using dynamic IP ranges that need constant updates too.

There is a small update today though!

It is currently being worked on and we’re exploring different avenues to adjust the impact of the service on our infrastructure going forward. We have generated a small form to help us with learning how everyone used to / intends to use the market history ESI endpoint when we bring it back online. If you can spare any time for some feedback on the form it would be appreciated!

Thank you all

Hey everyone!

• The ESI Market endpoint has been bought back online as of 1300 UTC today!
• The change introduces a rate limit per IP address and a filter for regions that don’t have markets.
• The endpoint is tentatively limited to 300 requests per minute per IP address. The rate limit is subject to review at any time, and we will update according if and when a change is made such that all third-party developers can adjust their apps accordingly.
• When the rate limit is exceeded, users can expect an HTTP 500 error with the original error in the reason text.
• If the rate limit is reached, users must wait 1 minute before trying again. There is also a limiter on the number of errors, so avoid excessive cooldown checks.
• Additionally, requesting stats for a region with no regional market will return a 400 error.
• The above rate limits only affect the Market ESI endpoint. All others can be used as normal for the time being.
• Circumventing the rate limiter by any means may result in a termination of a users developer application

This information pertains to the original topic of this thread which was the history endpoint.

I’ve also been made aware of a defect with regards to pulling information from the Pochven region which results in the error message stating that “This region cannot have a market”. This is not intended and It’s been relayed to relevant teams and I’m hoping that we’ll be able to have that tweaked as soon as we can as well.