In 2011 Google realized they had a problem. They were getting serious about deep learning networks with computational demands that strained their resources. Google calculated they would have to have twice as many data centers as they already had if people used their deep learning speech recognition models for voice search for just three minutes a day. They needed more powerful and efficient processing chips.
What kind of chip did they need? Central processing units (CPUs) are built to handle a wide variety of computational tasks very efficiently. Their limitation is that they can only handle a relatively small number of tasks at one time. GPUs, on the other hand, are less efficient at carrying out a single task and they can handle a much smaller variety of tasks. Their strength is that they can carry out many tasks at the same time. If you have to multiply three floating point numbers, a CPU will crush a GPU; if you have to multiply a million sets of three floating point numbers, the GPU will blow the CPU out of the room.
GPUs are ideal for deep learning applications because sophisticated deep learning networks perform millions of computations that can be carried out at the same time. Nvidia is the go-to chip company for GPUs that are designed for machine learning. Google uses Nvidia GPUs but they needed something faster. They also needed a chip that is more efficient. A single GPU doesn’t consume a lot of energy but when you have millions of servers running 24/7 as Google does, energy consumption becomes a serious problem. Google decided to build a chip of their own.
GPUs are ideal for deep learning applications because sophisticated deep learning networks perform millions of computations that can be carried out at the same time. Nvidia is the go-to chip company for GPUs that are designed for machine learning. Google uses Nvidia GPUs but they needed something faster. They also needed a chip that is more efficient. A single GPU doesn’t consume a lot of energy but when you have millions of servers running 24/7 as Google does, energy consumption becomes a serious problem. Google decided to build a chip of their own.
Google told the world they had designed their own chip in May of last year. They called it a Tensor processing unit (TPU) because it’s custom designed to work with Tensorflow, Google’s open-source software library for machine learning. A good deal of speculation about what Google had in mind for the TPU accompanied the announcement because Google did not provide very much information about the chip’s architecture or what it could do. That information came in a
paper Google released last week and it is very compelling.
The paper compares performance of the first generation TPU (which has been running in Google’s data centers since 2015) with Intel’s Haswell CPUs and Nvidia’s K80 Kepler dual GPU. A
deep learning network must be trained before it can be used to infer information from data. Google’s first-generation TPU was designed for inference and thus the performance comparison was limited to inference operations.
Google compared the chips in terms of speed and efficiency. Speed was measured as tera (trillion)-operations performed per second as a function of memory bandwidth. The TPU was 15x to 30x faster than the CPUs and GPUs. Efficiency was measured as tera-operations performed per Watt of energy consumed. The TPU was 30x to 80x more efficient than the CPUs and GPUs.
These are extraordinary numbers but several caveats must be noted before jumping to the conclusion that the TPU is the future of deep learning computing. Google’s tests were carried out using chips that were contemporary in early 2015. Google, Nvidia and Intel have all improved their chips since then and it is unknown how today’s chips compare. Still, the TPU’s advantages were so great two years ago that it’s unlikely Intel and Nvidia have completely closed the gap.
A more important consideration is the nature of the chips being compared. Intel’s CPUs are general purpose chips designed for flexibility and speed running a limited number of processes at one time. Nvidia’s GPUs are general purpose chips designed for running many neural net computations at one time. Google’s TPU is an ASIC (application specific integrated circuit) that is custom designed to carry out specific functions in Tensorflow.
The CPU has maximum flexibility. It can run a wide variety of programs including deep learning networks performing both learning and inference using many software libraries. The GPU is not as flexible as the CPU but it is better at deep learning computation, it can carry out both learning and inference and it is also not limited to a single software library. The tested TPU has almost no flexibility. It does one thing, inference in Tensorflow, but it does it brilliantly.
Chip deployment in deep learning computing is not a zero-sum game. Real-world deep learning networks need a GPU in the system to communicate with either GPUs or ASICs like Google’s TPU. GPUs are ideal for work environments where deep learning flexibility is required or the necessary ASICs have yet to be built. ASIC’s are ideal when a full commitment has been made to a software library or platform.
Google has obviously made that commitment to Tensorflow and the TPUs superior performance makes it highly likely that Tensorflow and the TPU will evolve together. The tight connection between the TPU and specific functions within particular builds of Tensorflow makes it unclear whether it's sensible for Google to market their chips outside the company. However, third parties that make use of Google’s cloud services for machine learning solutions can reap the benefits of the TPU’s exceptional performance metrics.
don't be an air head ...