Feed version validation using etag and last-modified headers

To improve overall efficiency of our system, we now support version validation of different type of feeds before re-processing them, to fetch newly added content.

We utilise industry standardised etag and last-modified headers, in the response header of the feed resource, to check if a new version of a feed is available for re-processing. This process improves the overall efficiency of the system as feeds which have not updated are not re-processed unnecessarily.

What is a etag header?

An etag or entity tag is part of HTTP, the protocol for the World Wide Web. It is one of several mechanisms that HTTP provides for Web cache validation, which allows a client to make conditional requests. This mechanism allows caches to be more efficient and saves bandwidth, as a Web server does not need to send a full response if the content has not changed. Learn more about etags here.

What is a last-modified header?

A last-modified response HTTP header contains a date and time when the origin server believes the resource was last modified. It is used as a validator to determine if the resource is the same as the previously stored one. Less accurate than an etag header, it is a fallback mechanism. Learn more about Last-Modified here.

Example of etag and last-modified headers:

To check these values for a specific resource, open developer console on Firefox and check the network tab. Now load the resource URL. Inspect the request sent and check for these tags under Response Header.

How we utilise etag and last-modified headers?

Our system re-processes each feed at a set interval of time, to fetch new content from the feeds. Checking etag and last-modified headers ensures that we do not waste resources by re-processing feeds that does not have any new content.

If a feed's response header contains etag and/or last-modified headers, our system will compare their values with the values we stored from the last time we processed that feed.

The flow diagram below describes how we check these headers to re-process a feed

Backward Compatibility

We understand that not all hosting providers/servers support etag and last-modified headers. Therefore, we designed the system in a way, that it re-processes the feed if etag and last-modified headers are not present in the Response Header.

Important Note:

If etag and/or last-modified headers are present in the response header, their value must change when the feed is updated. Failing to do so will result in the feed not re-processing, which in-turn will not update the newly added content in your apps.

FAQs

1. Is adding these headers in the Response Header mandatory?

No. However, it is highly recommended as these will improve system performance and save resources on both your's and our end.

2. Are both headers necessary in the Response Header?

No. etag is the recommended header, whereas last-modified is used as a fall-back mechanism incase etag is not present. Important note: If Etag is present, Last-Modified will not be checked.

3. What if I do not add these headers to my feed's Response Header?

If both headers are not present in the Response Header, the feed will be queued for processing as normal. (This makes it backward compatible as not all servers support etag and last-modified headers).

4. What if these headers are present in the Response Header but their value is not updating when the feed is updated.

The feed won’t be re-processed by our system. These headers are specifically used for version validation. if these tags are not updated when the content/file is updated, then its an issue with the hosting server. In that case we would suggest you to contact your hosting provider.

5. Which feeds does this version validation process affects?