As mentioned above by 0xea, @domi007 published 4 blog posts (1,2,3,4) detailing his experience with GSM sniffing and cracking. He also published his recorded presentation about GSM security (slides).
On the same topic, there's also Sniffing GSM with HackRF, Analyzing GSM with Airprobe and Wireshark, three Chaos Computer Club presentations (One discussing Wideband GSM sniffing, another discussing the functioning of GSM networks) and the one mentioned above about GSM Cracking. Last but not least, there's also a Black Hat 2008 Presentation called "Intercepting Mobile Phone/GSM Traffic", by David Hulton and Steve video (.mov).
Observation : There's also this tutorial by Srlabs, which uses their own tool, Kraken, that covers decrypting GSM using Airprobe and the Airprobe's own tutorial on decoding GSM.
When choosing your SDR, I suggest you read this comparison about HackRF, bladeRF and the USRP B210. There is also RTL-SDR . They are all quite nice. I also suggest using Gqrx as it's built on Gnuradio and has a neat interface.
Architecture and theory wise, I'd recommend About GSM Dm Channels, which is a quite complete and detailed beginner paper that explains the GSM Architecture and how it works. I also recommend GSM - Architecture, Protocols and Services and 4G: LTE/LTE-Advanced for Mobile Broadband (if considering LTE), which provide further information and details about the functioning of GSM for those that want to delve into it.
Regarding GSM Encryption and its flaws, I suggest Instant Ciphertext-Only Cryptanalysis of GSM Encrypted Communication, which discusses ciphertext attacks on A5/(1,2,3), Hardware-based Cryptanalysis of the GSM A5/1 Encryption Algorithm - includes a 2 page brief on A5/1 and then goes on to the cryptoanalysis - and A Practical-Time Attack on the A5/3 Cryptosystem Used in Third Generation GSM Telephony - discusses attacks on A5/3. Finally, there is also Real Time Cryptanalysis of A5/1 on a PC, a very nice document called Security of 3G and LTE that discusses the security architecture and the attacks on it's flaws, and Software Hardware Trade-offs - Applications to A5/1 Cryptanalysis - another nice paper on A5/1. Observations : You can find the specifications for A5/3 in the middle of this page. There are also two Blackhat presentions that cover part of those papers in a succinct way Attacking Phone Privacy and Breaking Phone Privacy. If you like animations, there is an A5/1 Cipher Animation on YouTube. @matthew_d_green deserves to be mentioned, considering he wrote a small synthesis of cellular communications crypto flaws in his blog.
Important observation : The GSM specifications are located in 3GPP's website. To find them, you need to determine the numbering of the part you're looking for. Then, you browse into their ftp server and look for the date and release you fancy. (Releases prior to 2012-13 need to be solicited through their contact mail). Supposing you want the latest "3G and Beyond / GSM" Signalling Protocols specifications, you'll need to browse to their "latest" folder, descend to the release you're looking for (i.e. Release 12), and finally download the "Series" that contain the information you need - which in this case would be "Series 24". Therefore, the result of your endeavor would be : ftp://ftp.3gpp.org/specs/latest/Rel-12/24_series/ .
It's not a good user experience, especially because there are several empty directories, but with patience, you can find what you're looking for.
@gat3way deserves a special mention for documenting his experiments in cracking GSM A5/1 in his blog (1,2,3) - includes a brief description of the A5/1 mechanism - and the fact that he has implemented support for cracking A5/1 using Pornin's attack in his password recovery tool, hashkill - git commit.
An existing tool to crack A5/1 is Kraken, by srlabs (available through git://git.srlabs.de/kraken.git), which should be used after recording the data with Gnuradio/Gqrx and parsing it with Airprobe. It needs GSM rainbow tables, available at the jump.
Another tool that can be considered is hashkill, which takes the key to be cracked in a "frame_number:keystream" format - it's author published the code for converting a bin burst into the required input format here. He also published a php script to locate suitable SI5/SI6 encrypted bursts for cracking - you'll need to change the hardcoded values for SI5 frames to fit your location -, which, together with Kraken's xortool, gsmframecoder (that will be used to calculate Timing Advance changes) and Airprobe should be sufficient to crack GSM.