Timezone: »

 
MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point
Tao Yu · Wentao Guo · Canal Li · Tiancheng Yuan · Christopher De Sa

Sat Jul 23 12:00 PM (PDT) @
in Hardware-aware efficient training (HAET) »

In this paper, we introduce MCTensor, a library based on PyTorch for providing general-purpose and high-precision arithmetic for DL training. MCTensor is used in the same way as PyTorch Tensor: we implement multiple basic, matrix-level computation operators and NN modules for MCTensor with identical PyTorch interface. Our algorithms achieve high precision computation and also benefits from heavily-optimized PyTorch floating-point arithmetic. We evaluate MCTensor arithmetic against PyTorch native arithmetic for a series of tasks, where models using MCTensor in float16 would match or outperform the PyTorch model with float32 or float64 precision.

Author Information

Tao Yu (Cornell University)
Wentao Guo (Cornell University)
Wentao Guo

I am a master of engineering student in CS at Cornell University. Previously I also obtained my bachelor degree in CS at Cornell University.

Canal Li (Cornell University)
Tiancheng Yuan (Cornell University)
Christopher De Sa (Cornell)

More from the Same Authors