To extend the answer of perror:
Perhaps you should take a look into a recently published whitepaper named Breaking the x86 ISA, by Christopher Domas. It was published on blackhat17 and describes an approach for digging into x86 chips and extracting hidden machine instructions.
Title:
Breaking the x86 ISA
Abstract:
A processor is not a trusted black box for running code; on the contrary, modern x86 chips are packed full of secret instructions and hardware bugs. In this paper, we demonstrate how page fault analysis and some creative processor fuzzing can be used to exhaustively search the x86 instruction set and uncover the secrets buried in a chipset. The approach has revealed critical x86 hardware glitches, previously unknown machine instructions, ubiquitous software bugs, and flaws in enterprise hypervisors.
Author: Christopher Domas
https://www.blackhat.com/docs/us-17/thursday/us-17-Domas-Breaking-The-x86-Instruction-Set-wp.pdf
Related Resources:
Somehow related microcode-internals from academia:
Title:
Reverse Engineering x86 Processor Microcode
Abstract: Microcode is an abstraction layer on top of the physical components of a CPU and present in most generalpurpose CPUs today. In addition to facilitate complex and vast instruction sets, it also provides an update mechanism that allows CPUs to be patched in-place without requiring any special hardware. While it is well-known that CPUs are regularly updated with this mechanism, very little is known about its inner workings given that microcode and the update mechanism are proprietary and have not been throughly analyzed yet. In this paper, we reverse engineer the microcode semantics and inner workings of its update mechanism of conventional COTS CPUs on the example of AMD’s K8 and K10 microarchitectures. Furthermore, we demonstrate how to develop custom microcode updates. We describe the microcode semantics and additionally present a set of microprograms that demonstrate the possibilities offered by this technology. To this end, our microprograms range from CPU-assisted instrumentation to microcoded Trojans that can even be reached from within a web browser and enable remote code execution and cryptographic implementation attacks.
Author: Philipp Koppe, Benjamin Kollenda, Marc Fyrbiak, Christian Kison, Robert Gawlik, Christof Paar, and Thorsten Holz
http://syssec.rub.de/media/emma/veroeffentlichungen/2017/08/16/usenix17-microcode.pdf
Related Work/Reads:
EDIT: I added the reference mentioned by Dan.
Title:
Page Fault Liberation Army or Gained in Translation: a history of creative x86 virtual memory uses
Abstract:
x86 processors contain a surprising amount of built-in memory translation logic, which is driven by various data tables with intricate entry formats, and can produce various kinds of traps and other interesting computational effects. These features are mostly relics of earlier, more civilized times, when Jedi Knights tried to protect the Old Republic OSes with segmentation, supervisor bits, and hardware task support, but were defeated by processor de-optimizations and performance concerns and left unused by both Windows and UNIX systems – and explored only by hackers. For the rest of the world, an x86 PC was a "von Neumann architecture" with most of its strangeness unused.
Author: Julian Bangert and Sergey Bratus
https://media.ccc.de/v/29c3-5265-en-page_fault_liberation_army_h264
EDIT: I added the recently published spectre or meltdown attack, which exploits critical vulnerabilities in modern processors.
Title:
Meltdown and Spectre: Bugs in modern computers leak passwords and sensitive data
TLDR:
Meltdown and Spectre exploit critical vulnerabilities in modern processors. These hardware bugs allow programs to steal data which is currently processed on the computer. While programs are typically not permitted to read data from other programs, a malicious program can exploit Meltdown and Spectre to get hold of secrets stored in the memory of other running programs. This might include your passwords stored in a password manager or browser, your personal photos, emails, instant messages and even business-critical documents.
Paper:
https://meltdownattack.com/meltdown.pdf
https://spectreattack.com/spectre.pdf
Related Resources:
