Let's make a couple of assumptions. Software is divided into functional components. Licenses are for functional components within that software package. Licenses can be based on time, on version or on a number of uses, i.e you may use the functionality until a set point in time, you may the functionality of the version you purchased or some minor derivative of it or you may use it a number of times. There are two main scenarios you have to solve, where an attacker doesn't have access to a license and where he does.
Attacker with no license
The first scenario is where your attacker does not have access to a valid license to your product. This problem is easy to solve. Simply assign a separate encryption key to each of the functional licenseable parts of your software. Encrypt each functional part with the encryption key designed for that part. Now you can distribute your software without worry of someone being able to decrypt functions they have not licensed since you never send them the key.
Attacker with access to license
The second scenario, which is much harder to solve, is when your attacker has a valid license to your software but he either wants to redistribute the functions he has licensed or to extend his license time wise.
Now you need a reliable time source, this can be solved by:
- embedding a public key into a dongle and having the dongle issue a random challenge which must be forwarded to a time server. The time server responds by signing the current time and the challenge and returning it to the client which then sends it to the key and the key then updates its internal clock and unlocks.
- updating the internal clock based on the time it has been plugged into the computer. The USB port supplies power to your dongle all the time while its plugged in.
- updating the internal clock based on timestamps sent from drivers installed on the machine its attached to. Only allow timestamps forward in time. Only allow movement backwards in time if the time source is a remote trusted time server supplying a signed timestamp.
If your license is based on versions you actually have an attacked who does not have access to a license because your key derivation function for the functional unit takes both the identifier of the functional unit and the version of it as input.
So once you have separate keys for each functional unit your licenses basically becomes a matter of distributing symmetric keys so that they can be sent to the dongle. This is usually done by embedding a secret symmetric key in the dongle, encrypting the license decryption keys with the shared secret key and then signing the encrypted key update files. The signed update files are then passed to the dongle which validates the signature on the update, decrypts the new keys with the shared symmetric key and stores them for later use.
All dongles must have access to secure storage in order to store license decryption keys, expiration timestamps and so on. In general this is not implemented on external flash memory or EEPROM. If it is it must be encrypted with a key internal to the ASIC or FPGA and signed such that it can not be changed.
Plain text hole
Once the user has a license to your functional component, even if he can't extract your secret key, he can use your dongle to decrypt that functional component. This leads to the issue that he may extract all your plain text and replace the decryption call with a direct call to the extracted plain text. Some dongles cover this issue by embedding a processor into the dongle. The functional component is then sent encrypted over to the dongle which decrypts the component and executes it internally. This means that the dongle essentially becomes a black box and the functional components sent to the dongle needs to be probed individually to discover their properties.
A lot of dongles are encryption and decryption oracles which leads to potential issues with Chosen-ciphertext attacks, e.g the recent padding oracle attacks.
Side channel attacks
Besides the oracle issues you also have a lot of concerns with all of the so far well known side channel attacks. You also need to be concerned with any potential but undiscovered side channel.
Be aware that there are a number of companies in the world who specialize in picking apart and auditing secure chips. Some of the most well known companies are probably Chris Tarnovsky of flylogic, now part of IOActive and chipworks. This sort of attack is expensive but may be a real threat depending on the value of your target. It would surprise me if but a few, possibly none of, dongles today are able to withstand this sort of high budget attacker.
Do they work
Given a dongle which is based on strong encryption, isn't time based since you can not expire encryption keys based on time nor is time an absolute, free of any side channel attacks and executes the code on the chip, yes it will make discovering the underlying code equivalent to probing a black box. Most of the breaks that happen with these dongles are based on implementation weaknesses by the licensees of the hardware licensing system due to the implementer being unfamiliar with reverse engineering and computer security in general.
Also, do realize that even software where a majority of the logic is implemented on an internet facing server has been broken simply by probing the black box and inferring server side code based on client code expectations. Always prepare for your application to be broken and develop a plan for how to deal with it when it happens.