To extend the answer of perror: 

Perhaps you should take a look into a recently published whitepaper named **Breaking the x86 ISA**, by Christopher Domas. It was published on blackhat17 and describes an approach for digging into x86 chips and extracting hidden machine instructions.

> **Title:**
>
> *Breaking the x86 ISA*
> 
> **Abstract:**
>
> A processor is not a trusted black box for running code; on the
> contrary, modern x86 chips are packed full of secret instructions and
> hardware bugs. In this paper, we demonstrate how page fault analysis
> and some creative processor fuzzing can be used to exhaustively search
> the x86 instruction set and uncover the secrets buried in a chipset.
> The approach has revealed critical x86 hardware glitches, previously
> unknown machine instructions, ubiquitous software bugs, and flaws in
> enterprise hypervisors.
>
> **Author:** Christopher Domas
>
> https://www.blackhat.com/docs/us-17/thursday/us-17-Domas-Breaking-The-x86-Instruction-Set-wp.pdf
>
> **Related Resources:**
> 
> - https://github.com/xoreaxeaxeax/sandsifter

Somehow related microcode-internals from academia:

> **Title:**
>
> *Reverse Engineering x86 Processor Microcode*
> 
> **Abstract:**
> Microcode is an abstraction layer on top of the physical
components of a CPU and present in most generalpurpose
CPUs today. In addition to facilitate complex and
vast instruction sets, it also provides an update mechanism
that allows CPUs to be patched in-place without requiring
any special hardware. While it is well-known that CPUs
are regularly updated with this mechanism, very little is
known about its inner workings given that microcode and
the update mechanism are proprietary and have not been
throughly analyzed yet.
In this paper, we reverse engineer the microcode semantics
and inner workings of its update mechanism of conventional
COTS CPUs on the example of AMD’s K8 and
K10 microarchitectures. Furthermore, we demonstrate
how to develop custom microcode updates. We describe
the microcode semantics and additionally present a set of
microprograms that demonstrate the possibilities offered
by this technology. To this end, our microprograms range
from CPU-assisted instrumentation to microcoded Trojans
that can even be reached from within a web browser
and enable remote code execution and cryptographic implementation
attacks.
>
> **Author:** Philipp Koppe, Benjamin Kollenda, Marc Fyrbiak, Christian Kison, Robert Gawlik, Christof Paar, and Thorsten Holz
>
> http://syssec.rub.de/media/emma/veroeffentlichungen/2017/08/16/usenix17-microcode.pdf
>
> **Related Work/Reads**:

> - http://inertiawar.com/microcode/

> - https://argp.github.io/2017/09/14/re-x86-microcode/
>


EDIT: I added the reference mentioned by Dan.

> **Title:**
>
> Page Fault Liberation Army or Gained in Translation: a history of creative x86 virtual memory uses
>
> **Abstract:**
>
> x86 processors contain a surprising amount of built-in memory translation logic, which is driven by various data tables with intricate entry formats, and can produce various kinds of traps and other interesting computational effects. These features are mostly relics of earlier, more civilized times, when Jedi Knights tried to protect the Old Republic OSes with segmentation, supervisor bits, and hardware task support, but were defeated by processor de-optimizations and performance concerns and left unused by both Windows and UNIX systems – and explored only by hackers. For the rest of the world, an x86 PC was a "von Neumann architecture" with most of its strangeness unused.
>
> **Author:** Julian Bangert and Sergey Bratus
>
> https://media.ccc.de/v/29c3-5265-en-page_fault_liberation_army_h264

EDIT: I added the recently published **spectre** or **meltdown** attack, which exploits critical vulnerabilities in modern processors. 

> **Title:**
> 
> Meltdown and Spectre: Bugs in modern computers leak passwords and sensitive data
> 
> **TLDR:**
>
> Meltdown and Spectre exploit critical vulnerabilities in modern processors. These hardware bugs allow programs to steal data which is currently processed on the computer. While programs are typically not permitted to read data from other programs, a malicious program can exploit Meltdown and Spectre to get hold of secrets stored in the memory of other running programs. This might include your passwords stored in a password manager or browser, your personal photos, emails, instant messages and even business-critical documents.
> 
>
> **Paper:**
>
> https://meltdownattack.com/meltdown.pdf
>
> https://spectreattack.com/spectre.pdf
>
> 
> 
>
> **Related Resources:**
>
> - https://googleprojectzero.blogspot.de/2018/01/reading-privileged-memory-with-side.html?m=1
>
> - http://pythonsweetness.tumblr.com/post/169166980422/the-mysterious-case-of-the-linux-page-table
>
> - https://lkml.org/lkml/2017/12/27/2

> - https://gruss.cc/files/kaiser.pdf

> - https://lwn.net/Articles/741878/

> - https://lwn.net/Articles/738975/