Crawling for AC/DC
The case
One early December morning, I received a text message from my friend saying Dude, I’m kind of embarrassed to ask you this, but could you buy two tickets for the AC/DC concert for me and my gf? My payday is soon so… Sure man, I’m getting up soon so no problem. But are there still any tickets available? I replied. Nope, there weren’t (well, not for General Admission or Inner Barrier). It had been December and the concert was in July, so lots of time but no tickets left. It is a very rare event for AC/DC to come to my country. I had bought my ticket a few days earlier. Don’t worry man, there is still a chance. Sometimes you can get one or two tickets at some random time, but you have to watch the site constantly. Well, I wasn’t planning to do that, and neither was my friend. But it is rather easy to automate the whole process. So I did.
The solution
To automate the process, I needed a piece of code. All you have to do is to visit the page, parse the ticket info and check if there are tickets available. If there are, you can buy them (I think, using a credit card, but that is more complex), or you can inform someone (e.g. via email) who will buy them immediately. I was interested in GA or IB tickets. The site I’m using to buy tickets is eventim.pl and it looks like this (I hope the url still works - if not, you can check any other event on eventim - the page template is the same, or you can take a look at the screenshot).
The code
Having a VPS with nodejs installed is a good start, so I sat down (well, there were still two hours before leaving to work, so plenty of time) and started coding. It was pure fun. The working code I came up with after little more than an hour of coding looks like this (not perfect though):
var request = require('request');
var cheerio = require('cheerio');
var nodemailer = require('nodemailer');
var moment = require('moment');
var transporter = nodemailer.createTransport({
service: 'Gmail',
auth: {
user: 'myGmailAddress@gmail.com',
pass: 'myGmailPass'
}
});
var sendMail = function(transporter, mailOptions) {
transporter.sendMail(mailOptions, function(error, info){
if (error) {
console.log(error);
} else {
console.log('Message sent: ' + info.response);
}
});
};
var getTickets = function($) {
if ($('#tableAssortmentList_yTix tr').length === 0) {
throw new Error('No tickets table in request!');
}
var tickets = [];
$('#tableAssortmentList_yTix tr').each(function(i, el) {
if (!$(el).hasClass('disabled') && $(el).find('td.priceCategory').length) {
tickets.push({
description: $(el).find('td.priceCategory').text(),
price: $(el).find('td.price').text()
});
}
});
return tickets;
};
var url = 'http://www.eventim.pl/bilety.html?affiliate=PLE&action=tickets&doc=artistPages/tickets&fun=artist&key=1310956$4670064&language=en';
var prevMsg = '';
var interval = 1000 * 60 * 4;
setInterval(function() {
request(url, function (error, response, html) {
if (!error && response.statusCode == 200) {
var $ = cheerio.load(html);
var allTickets = [];
var intrestedIn = [];
var intrestedInMsg = '';
console.log('200, checking...');
try {
allTickets = getTickets($);
} catch(err) {
var mailOptions = {
from: '<myGmailAddress@gmail.com>',
to: 'krzysztof@krzton.com',
subject: 'Eventim crawlera error...',
text: err.message
};
sendMail(transporter, mailOptions);
}
if (allTickets.length) {
for (var i in allTickets) {
if (allTickets[i].description.toLowerCase() === 'general admission' ||
allTickets[i].description.toLowerCase() === 'inner barrier') {
intrestedIn.push(allTickets[i]);
intrestedInMsg += '\n' + allTickets[i].description + ' for ' + allTickets[i].price;
}
}
var introText = 'we salute you!\n\nBot reports that about ' + moment().format('H:m D.MM.YY')
+ ' it found tickets. If you click on the link there can be any left because they disapear quickly.'
+ ' Don\'t worry bot checks every ' + interval + ' minutes so there is still a huge chance!:D\n';
if (intrestedIn.length && (intrestedInMsg != prevMsg)) {
var mailOptions = {
from: 'EventimBot <krzysztof@krzton.com>',
to: 'krzysztof@krzton.com, myFriendEmail@gmail.com',
subject: 'For those about to rock...',
text: introText + intrestedInMsg + '\nSee here ' + url
};
sendMail(transporter, mailOptions);
}
}
prevMsg = intrestedInMsg;
}
});
}, interval);
Let’s dive deeper
I used 4 modules
var request = require('request'); // http requests
var cheerio = require('cheerio'); // nice lib for manipulating html (in very similar way to jquery)
var nodemailer = require('nodemailer'); // sending emails
var moment = require('moment'); // my favourite lib for time manipulation
Then setup the nodemailer module to send emails via a gmail account
var transporter = nodemailer.createTransport({
service: 'Gmail',
auth: {
user: 'myGmailAddress@gmail.com',
pass: 'myGmailPass'
}
});
Some helper functions for sending emails, with error logging
var sendMail = function(transporter, mailOptions) {
transporter.sendMail(mailOptions, function(error, info){
if (error) {
console.log(error);
} else {
console.log('Message sent: ' + info.response);
}
});
};
Function for parsing tickets (type and price)
var getTickets = function($) {
if ($('#tableAssortmentList_yTix tr').length === 0) {
throw new Error('No tickets table in request!');
}
var tickets = [];
$('#tableAssortmentList_yTix tr').each(function(i, el) {
if (!$(el).hasClass('disabled') && $(el).find('td.priceCategory').length) {
tickets.push({
description: $(el).find('td.priceCategory').text(),
price: $(el).find('td.price').text()
});
}
});
return tickets;
};
Here is the main part of the script, which does the work every x milliseconds (in this case every 4 minutes)
var url = 'http://www.eventim.pl/bilety.html?affiliate=PLE&action=tickets&doc=artistPages/tickets&fun=artist&key=1310956$4670064&language=en';
var prevMsg = '';
var interval = 1000 * 60 * 4;
setInterval(function() {
request(url, function (error, response, html) {
if (!error && response.statusCode == 200) {
var $ = cheerio.load(html);
var allTickets = [];
var intrestedIn = [];
var intrestedInMsg = '';
console.log('200, checking...');
try {
allTickets = getTickets($);
} catch(err) {
var mailOptions = {
from: '<myGmailAddress@gmail.com>',
to: 'krzysztof@krzton.com',
subject: 'Eventim crawlera error...',
text: err.message
};
sendMail(transporter, mailOptions);
}
if (allTickets.length) {
for (var i in allTickets) {
if (allTickets[i].description.toLowerCase() === 'general admission' ||
allTickets[i].description.toLowerCase() === 'inner barrier') {
intrestedIn.push(allTickets[i]);
intrestedInMsg += '\n' + allTickets[i].description + ' za ' + allTickets[i].price;
}
}
var introText = 'we salute you!\n\nBot reports that about ' + moment().format('H:m D.MM.YY')
+ ' it found tickets. If you click on the link there can be any left because they disapear quickly.'
+ ' Don\'t worry bot checks every ' + interval + ' minutes so there is still a huge chance!:D\n';
if (intrestedIn.length && (intrestedInMsg != prevMsg)) {
var mailOptions = {
from: 'EventimBot <krzysztof@krzton.com>',
to: 'krzysztof@krzton.com, myFriendEmail@gmail.com',
subject: 'For those about to rock...',
text: introText + intrestedInMsg + '\nSee here ' + url
};
sendMail(transporter, mailOptions);
}
}
prevMsg = intrestedInMsg;
}
});
}, interval);
First, parse the HTML
var $ = cheerio.load(html);
Then, try to parse ticket data (if there is any error, send me info because it means something may be broken and needs to be fixed)
try {
allTickets = getTickets($);
} catch(err) {
var mailOptions = {
from: '<myGmailAddress@gmail.com>',
to: 'krzysztof@krzton.com',
subject: 'Eventim crawlera error...',
text: err.message
};
sendMail(transporter, mailOptions);
}
Check if there are tickets available, compose the email message and send it! We are counting on the fact that there is my friend on the other end ready to buy tickets!
if (allTickets.length) {
for (var i in allTickets) {
if (allTickets[i].description.toLowerCase() === 'general admission' ||
allTickets[i].description.toLowerCase() === 'inner barrier') {
intrestedIn.push(allTickets[i]);
intrestedInMsg += '\n' + allTickets[i].description + ' for ' + allTickets[i].price;
}
}
var introText = 'we salute you!\n\nBot reports that about ' + moment().format('H:m D.MM.YY')
+ ' it found tickets. If you click on the link there can be any left because they disapear quickly.'
+ ' Don\'t worry bot checks every ' + interval + ' minutes so there is still a huge chance!:D\n';
if (intrestedIn.length && (intrestedInMsg != prevMsg)) {
var mailOptions = {
from: 'EventimBot <krzysztof@krzton.com>',
to: 'krzysztof@krzton.com, myFriendEmail@gmail.com',
subject: 'For those about to rock...',
text: introText + intrestedInMsg + '\nSee here ' + url
};
sendMail(transporter, mailOptions);
}
}
prevMsg = intrestedInMsg;
The message is only sent when the state has changed from the previous check. This deals with the case that tickets are available for about 20 minutes and you got five identical messages when probably one is enough ;).
So as you can see, the flow is simple:
- visit site,
- get HTML,
- parse,
- check for tickets (report possible errors),
- if there are tickets available (and different from previous request) - send info,
- repeat!
I used VPS server to run the script. To keep it going even when it crashes I used the simplest tool - cron and a dead simple bash script.
crontab -e
#eventim-checker - run script every minute
* * * * * /home/apps/private/eventim-checker/run.sh
/home/apps/private/eventim-checker/run.sh
#!/bin/bash
#check if screen with process exists
if ! screen -ls | grep -q "node-eventimchecker"; then
screen -dmS node-eventimchecker nodejs /home/apps/private/eventim-checker/index.js
fi
The result
The result was pretty satisfying. We (me and my friend) received the first mail after only few days - it looked like this.
But it wasn’t easy to buy those tickets: you have like 1-2 minutes and then, bam, disappeared (probably I am not the only one who came up with the idea of automatic checking). But finally, on Sunday 5th of May (so almost after half a year from starting the script!) at exactly 13:04, I received a familiar looking email, checked the link and there were two GA tickets available. Only two, but it was all that I needed. My friend and his gf were very pleased when they received them.
You are the man!
they said. Well… I don’t deny it ;)
I have learnt three things (well, some of them I knew already):
- coding is fun, but when you can experience the results in the real world it is much more fun
- sometimes you need patience to achieve something
- and finally, what I didn’t know before: Sunday is the best day to buy tickets :)
Any thoughts?