I'm talking about an action game with no upper score limit and no way to verify the score on the server by replaying moves etc.
What I really need is the strongest encryption possible in Flash/PHP, and a way to prevent people calling the PHP page other than through my Flash file. I have tried some simple methods in the past of making multiple calls for a single score and completing a checksum / fibonacci sequence etc, and also obfuscating the SWF with Amayeta SWF Encrypt, but they were all hacked eventually.
Thanks to StackOverflow responses I have now found some more info from Adobe - http://www.adobe.com/devnet/flashplayer/articles/secure_swf_apps_12.html and https://github.com/mikechambers/as3corelib - which I think I can use for the encryption. Not sure this will get me around CheatEngine though.
I need to know the best solutions for both AS2 and AS3, if they are different.
The main problems seem to be things like TamperData and LiveHTTP headers, but I understand there are more advanced hacking tools as well - like CheatEngine (thanks Mark Webster)
This is a classic problem with Internet games and contests. Your Flash code works with users to decide a score for a game. But users aren't trusted, and the Flash code runs on the user's computer. You're SOL. There is nothing you can do to prevent an attacker from forging high scores:
Flash is even easier to reverse engineer than you might think it is, since the bytecodes are well documented and describe a high-level language (Actionscript) --- when you publish a Flash game, you're publishing your source code, whether you know it or not.
Attackers control the runtime memory of the Flash interpreter, so that anyone who knows how to use a programmable debugger can alter any variable (including the current score) at any time, or alter the program itself.
The simplest possible attack against your system is to run the HTTP traffic for the game through a proxy, catch the high-score save, and replay it with a higher score.
You can try to block this attack by binding each high score save to a single instance of the game, for instance by sending an encrypted token to the client at game startup, which might look like:
hex-encoding( AES(secret-key-stored-only-on-server, timestamp, user-id, random-number))
(You could also use a session cookie to the same effect).
The game code echoes this token back to the server with the high-score save. But an attacker can still just launch the game again, get a token, and then immediately paste that token into a replayed high-score save.
So next you feed not only a token or session cookie, but also a high-score-encrypting session key. This will be a 128 bit AES key, itself encrypted with a key hardcoded into the Flash game:
hex-encoding( AES(key-hardcoded-in-flash-game, random-128-bit-key))
Now before the game posts the high score, it decrypts the high-score-encrypting-session key, which it can do because you hardcoded the high-score-encrypting-session-key-decrypting-key into the Flash binary. You encrypt the high score with this decrypted key, along with the SHA1 hash of the high score:
hex-encoding( AES(random-128-bit-key-from-above, high-score, SHA1(high-score)))
The PHP code on the server checks the token to make sure the request came from a valid game instance, then decrypts the encrypted high score, checking to make sure the high-score matches the SHA1 of the high-score (if you skip this step, decryption will simply produce random, likely very high, high scores).
So now the attacker decompiles your Flash code and quickly finds the AES code, which sticks out like a sore thumb, although even if it didn't it'd be tracked down in 15 minutes with a memory search and a tracer ("I know my score for this game is 666, so let's find 666 in memory, then catch any operation that touches that value --- oh look, the high score encryption code!"). With the session key, the attacker doesn't even have to run the Flash code; she grabs a game launch token and a session key and can send back an arbitrary high score.
You're now at the point where most developers just give up --- give or take a couple months of messing with attackers by:
Scrambling the AES keys with XOR operations
Replacing key byte arrays with functions that calculate the key
Scattering fake key encryptions and high score postings throughout the binary.
This is all mostly a waste of time. It goes without saying, SSL isn't going to help you either; SSL can't protect you when one of the two SSL endpoints is evil.
Here are some things that can actually reduce high score fraud:
Require a login to play the game, have the login produce a session cookie, and don't allow multiple outstanding game launches on the same session, or multiple concurrent sessions for the same user.
Reject high scores from game sessions that last less than the shortest real games ever played (for a more sophisticated approach, try "quarantining" high scores for game sessions that last less than 2 standard deviations below the mean game duration). Make sure you're tracking game durations serverside.
Reject or quarantine high scores from logins that have only played the game once or twice, so that attackers have to produce a "paper trail" of reasonable looking game play for each login they create.
"Heartbeat" scores during game play, so that your server sees the score growth over the lifetime of one game play. Reject high scores that don't follow reasonable score curves (for instance, jumping from 0 to 999999).
"Snapshot" game state during game play (for instance, amount of ammunition, position in the level, etc), which you can later reconcile against recorded interim scores. You don't even have to have a way to detect anomalies in this data to start with; you just have to collect it, and then you can go back and analyze it if things look fishy.
Disable the account of any user who fails one of your security checks (for instance, by ever submitting an encrypted high score that fails validation).
Remember though that you're only deterring high score fraud here. There's nothing you can do to prevent if. If there's money on the line in your game, someone is going to defeat any system you come up with. The objective isn't to stop this attack; it's to make the attack more expensive than just getting really good at the game and beating it.
You may be asking the wrong question. You seem focused on the methods people are using to game their way up the high score list, but blocking specific methods only goes so far. I have no experience with TamperData, so I can't speak to that.
The question you should be asking is: "How can I verify that submitted scores are valid and authentic?" The specific way to do that is game-dependent. For very simple puzzle games, you might send over the score along with the specific starting state and the sequence of moves that resulted in the end state, and then re-run the game on the server side using the same moves. Confirm that the stated score is the same as the computed score and only accept the score if they match.