[Doolally]

Hi everyone!

I'm having a bit of difficulty with what I've been working on at TSR recently, it's a way to aggregate the RSS feeds from every forum, and rather than take it straight to the TSR tech team who are up to their eyeballs in work, I thought I'd ask you awesome tech forum users!

The idea is to find a way to aggregate the RSS feeds from all the forums across the site, to create a kind of master feed containing all the posts on TSR. From this, the idea is to filter the results for specific topics/universities, and then feed those results into a tweet deck to create a really cool way for people and advertising clients to see exactly what's being said about something across the whole site

I've been using yahoo pipes to aggregate all the feeds (there's 270) but it seems to be silently truncating the results or capping them at 5000 when there really should be 20,000+. It seems that pipes works fine on a small group of feeds, but dealing with as much info as we have on TSR it gets a bit stuck. Also the filters I put in seem to be a bit buggy but there's no bug forum or much diagnostic info on yahoo pipes, so it's really hard to figure out what's going on. So I've been trying to find a different aggregator, but can't really find anything.

Is there anyone who knows of a programme that can create this master feed? Or anyone that knows yahoo pipes well?

For anyone that wants a look, here's the link to the master pipe - http://pipes.yahoo.com/pipes/pipe.in...61f1f95c23c870

a sample filter one - http://pipes.yahoo.com/pipes/pipe.in...2905ccddd01bad


feel free to have a play around and see what you can come up with, any help would be really appreciated!

[xXedixXx]

So you want ALL of TSR's RSS feeds in one huge feed?

[Chrosson]

(Original post by Doolally)
Hi everyone!

I'm having a bit of difficulty with what I've been working on at TSR recently, it's a way to aggregate the RSS feeds from every forum, and rather than take it straight to the TSR tech team who are up to their eyeballs in work, I thought I'd ask you awesome tech forum users!

The idea is to find a way to aggregate the RSS feeds from all the forums across the site, to create a kind of master feed containing all the posts on TSR. From this, the idea is to filter the results for specific topics/universities, and then feed those results into a tweet deck to create a really cool way for people and advertising clients to see exactly what's being said about something across the whole site

I've been using yahoo pipes to aggregate all the feeds (there's 270) but it seems to be silently truncating the results or capping them at 5000 when there really should be 20,000+. It seems that pipes works fine on a small group of feeds, but dealing with as much info as we have on TSR it gets a bit stuck. Also the filters I put in seem to be a bit buggy but there's no bug forum or much diagnostic info on yahoo pipes, so it's really hard to figure out what's going on. So I've been trying to find a different aggregator, but can't really find anything.

Is there anyone who knows of a programme that can create this master feed? Or anyone that knows yahoo pipes well?

For anyone that wants a look, here's the link to the master pipe - http://pipes.yahoo.com/pipes/pipe.in...61f1f95c23c870

a sample filter one - http://pipes.yahoo.com/pipes/pipe.in...2905ccddd01bad


feel free to have a play around and see what you can come up with, any help would be really appreciated!

Maybe I'm missing something, but why are you using forum specific feeds? For example:
http://www.thestudentroom.co.uk/external.php?forumids=1&type=rss2&lastpost=1&count=150

Note the bit I've bolded (the forum parameter). If we remove this we get
http://www.thestudentroom.co.uk/external.php?type=rss2&lastpost=1&count=150
which seems to be a feed for every publicly viewable post on TSR - i.e. a master feed.

The issue here is that the 'count' parameter is seemingly ignored by TSR, so there is no way to get a larger number of results. I imagine this will be a setting changeable through the admin control panel.
Crank up the count limit to whatever number you want and away you go.

This is arguably a better way to do things anyway - 150 posts in a low activity forum might go back a week compared a day in a higher activity forum, whereas this master RSS feed will always retrieve the most recent 20k (or whatever) posts.

You can then push that feed directly into yahoo pipes and work from there.

[Chrosson]

An addendum to my post above: there are of course issues with the effective derestriction (including crashing TSR if rss feeds are implemented badly...and knowing TSR...), but it's arguably better than the current approach. Any 'real' solution will need dev time.

Also to note that I didn't actually check that feed in detail, but it seemed to plausibly be a master.

[Chrosson]

Ah here we go, the documentation for the RSS feed options: https://www.vbulletin.com/docs/html/...group_external

Be aware that the RSS feeds only seem to get updated every 2 hours (or more I don't know for certain)...the cache value is likely set to 120 or higher at the moment.

[Doolally]

(Original post by xXedixXx)
So you want ALL of TSR's RSS feeds in one huge feed?

Yep, no mean feat!

(Original post by Chrosson)
Maybe I'm missing something, but why are you using forum specific feeds? For example:
http://www.thestudentroom.co.uk/external.php?forumids=1&type=rss2&lastpost=1&count=150

Note the bit I've bolded (the forum parameter). If we remove this we get
http://www.thestudentroom.co.uk/external.php?type=rss2&lastpost=1&count=150
which seems to be a feed for every publicly viewable post on TSR - i.e. a master feed.

The issue here is that the 'count' parameter is seemingly ignored by TSR, so there is no way to get a larger number of results. I imagine this will be a setting changeable through the admin control panel.
Crank up the count limit to whatever number you want and away you go.

This is arguably a better way to do things anyway - 150 posts in a low activity forum might go back a week compared a day in a higher activity forum, whereas this master RSS feed will always retrieve the most recent 20k (or whatever) posts.

You can then push that feed directly into yahoo pipes and work from there.

That's been really useful thank you! We've made a couple changes to the parameters through admin control, just got to wait a while for the caches to time out before seeing real benefit in the pipes. I'm also going to look at creating smaller more specialised feeds, for e.g. higher education, so it won't be as much to grapple with as a master but would still have lots of info in it. I'll keep you updated!

[Chrosson]

(Original post by Doolally)
Yep, no mean feat :s

That's been really useful thank you! We've made a couple changes to the parameters through admin control, just got to wait a while for the caches to time out before seeing real benefit in the pipes. I'm also going to look at creating smaller more specialised feeds, for e.g. higher education, so it won't be as much to grapple with as a master but would still have lots of info in it. I'll keep you updated!

Another hint: you can make TSR do the heavy lifting with multiple forums by using the comma operator in forumids. For example
http://www.thestudentroom.co.uk/external.php?forumids=1,119&type=rss2&lastpost=1&count=150
Will display forum 1 and forum 119 (i.e. current affairs and this forum). The first time you visit a combination of forums I believe it will generate the cache for the first time, after which point it will act as normal. I also don't see a difference with the number of results reported, but maybe that takes a bit longer for some reason...

Something that you might want to look it is the time reported in the RSS feed for posts - it's actually wrong by an hour.

[Chrosson]

Some more docs on the external data provider: https://www.vbulletin.com/docs/html/...r_implementing