Most people on the internet know about PayPal. It's ubiquitous. I'm writing a new PayPal 'backend', intended to manage user accounts. This is not my first time on this particular merry-go-round, so I'm working with a large base of tested and reliable code, but PayPal is special.
Two months ago, PayPal added a new field to it's IPN messages, called "ipn_track_id". It's filled with a big 22 character long unique-looking alphanumeric hash.
Hopefully this is the long-sought-after "idempotency marker", much desired by PayPal module developers in the same way the Arthurian Knights were quite keen on the Holy Grail.
'Idempotency' means 'doing it twice doesn't screw things up'. It has become a very important concept in making mixed internet systems reliable. See, there are 'edge cases' on days when both the network and users are being silly that cause issues that lead directly into the Twlilight Zone...
Consider, if you will, the following user story. It is a day like any other, when Arthur Q Public makes a PayPal transaction for a simple product item on an internet web site. Then, while intending to cancel another order, accidentally reverses the one he wanted to keep. Paypal begins the reversal process, locking Arthur out of further changes. Arthur, upon realizing what he has done, re-purchases the item he really wanted, expecting everything will get sorted somehow. Little does he realize that the IPN message system is passing on his purchases in random order, plus retries, on a day when the PayPal validation callback service is dropping connection from a DDoS attack, or traffic from Random Pop Star's latest album, which from a systems point of view looks like the same thing.
Depending how your system answers the question 'Is this record the same as the other one?' means the purchased will be be correct, (processed, reversed, processed again) punitive (purchased, purchased again but ignored, reversed. leaving none.) doubled (purchased, purchased again, reversal doesn't match latest, leaving two purchases) or worse (multiple interleaved retries and reversals, leaving god knows what.)
Payment records are fairly safe, because their Transaction ID works as the idempotency id. If you get the transaction twice, you can safely ignore the second. The Subscription ID works the same on the signup record... and that's the first problem right there, because suddenly you need to use different ID's depending on the type of the record.
Is there a 'Record Type' field? Sure there is! At least three of them. (Doh.)
Oh, and not all records have an ID that makes them idempotent, especially things like reversals and cancellations
So the sudden appearance of what looks like a consistent idempotency ID is exciting news. It significantly changes the processing logic of an IPN receiver. And remember this new field is being sent by PayPal as part of every single IPN message, it's not like it's an optional request. Everyone who writes or just runs an IPN endpoint (ie: an Internet Business) is expected by PayPal to deal with the existence of this new field in the IPN messages, and deal with the consequences.
You'd expect it to be documented, right?
Right? Suddenly pushing a new field at every single piece of IPN software out there, you'd expect a little Change Management documentation? Here's the highest ranked exchange in PayPal's own developer forums: https://www.x.com/thread/51579
It's short enough that I can include the entire amusing conversation:
1 posts since
Apr 20, 2011
... and that's it. It's been three months.
There's a similar story attached to the "notify_version" field suddenly changing to "3.1" from "3.0", and "internal tracking purposes" was mentioned as part of the reason for that one too.
Remember that the IPN protocol is similar to email; we all run receiver software expecting messages to conform to a particular format. Database tables are created and maintained with columns for each field. (well, mine don't. I know PayPal too well.) Transaction field values are thoroughly checked to prevent fraud and abuse, and deviations from the standard (such as the version changing from "3.0" to "3.1") can literally set off alarm bell sound files. You would expect PayPal to give the developer network a bit of a heads-up before arbitrarily changing the specification, especially because getting these new fields is not optional. There is no way to turn them on or off.
My personal theory is there is a tier of 'paid-up PayPal developers' who are given access to this advance information, with legal non-disclosure requirements that are apparently well enforced, because it's hard to find any information in the usual developer places. This would give them a reason to be hostile towards the Independent Developers, in order to give the paid-up members an advantage for their commercial software.
The alternative is that PayPal simply doesn't care about documenting IPN, and treats all developers like this. Basically, a choice between malevolent or uncaring. Tricky.
I really want to use the 'ipn_track_id' field... If it is what I think it is, it solves a lot of problems. But like everyone else, I apparently have to reverse-engineer it's meaning from the messages we are given.
I have to guess? While writing software expected to process actual money transactions? Really?
Fortunately, I have a large collection of data to look at. Not sandbox transactions, but a year of payments and edge cases like reversals and dispute adjustments. Data that, by definition, I can't share or use as examples. And I can tell you that PayPal's own 'sample transactions' don't even come close to the complexity of what happens during a dispute.
I hope this gives you a tiny insight into why PayPal site integration is so variable, and why so many smaller shops seem to have such difficulty with automatic processing, and go for larger 'checkout' services who take yet another cut on top of PayPal fees, and the original Bank charges, but don't allow for 'deep' user management. Small systems and independent coding projects (like the Drupal module) are fragile in the face of PayPal's arbitrary protocol changes, apparently by design.
But despite their current ubiquitous lock-in, PayPal needs to seriously lift it's game. Google Checkout is coming.
Update:
Nope. A little data trawling has suggested that "ipn_track_id" has an entirely different meaning, and solves a different problem. This value is repeated across two IPN messages that were otherwise hard to match together: the signup and payment transactions that are generated when the client starts a 'subscription' recurring payment.
Yes, it does also match across retries for the same record, but this seems secondary in purpose. It might technically be idempotent when combined with the "txn_type" field, but at that point we're back to magic combinations of fields again.
Also I just want to mention a minor thing - if you happen to be designing an API like PayPal's; while having a field like "reattempt" might sound like a good idea, but it means that each retry data request is subtly different from the original. So otherwise simple and reliable methods like storing message hashes won't work unless you strip those (otherwise quite useless) fields. A shame, more than a problem. It means that we need one set of logic for spurious TCP/HTTP retries, (which can happen, especially if a proxy in involved) and a second set for when PayPal reattempts explicitly, despite it making no difference to us why the glitch happens.
But it fits with PayPal's egocentric worldview. We are apparently supposed to care how many times PayPal tried to give it to us, enough to justify them altering the transaction message. But we don't get a simple field which lets us know which message the retry is for, in case we already got it.
Argh, PayPal strikes again. :(
ReplyDeleteThanks for publishing Jeremy. As you mentioned, the 'ipn_track_id' is both underdocumented and deceptively named. Without good folks like you publishing this info, any developer working with PayPal for more than a simple single-purchase really is helpless.
I've seen the same ipn_track_id across different txn_id transactions. echeck pending payment related? Not to be relied on as a unique identifier for an ipn.
ReplyDelete