Timezone: »

Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE
Juntang Zhuang · Nicha Dvornek · Xiaoxiao Li · Sekhar Tatikonda · Xenophon Papademetris · James Duncan

Thu Jul 16 06:00 AM -- 06:45 AM & Thu Jul 16 05:00 PM -- 05:45 PM (PDT) @ Virtual

The empirical performance of neural ordinary differential equations (NODEs) is significantly inferior to discrete-layer models on benchmark tasks (e.g. image classification). We demonstrate an explanation is the inaccuracy of existing gradient estimation methods: the adjoint method has numerical errors in reverse-mode integration; the naive method suffers from a redundantly deep computation graph. We propose the Adaptive Checkpoint Adjoint (ACA) method: ACA applies a trajectory checkpoint strategy which records the forward- mode trajectory as the reverse-mode trajectory to guarantee accuracy; ACA deletes redundant components for shallow computation graphs; and ACA supports adaptive solvers. On image classification tasks, compared with the adjoint and naive method, ACA achieves half the error rate in half the training time; NODE trained with ACA outperforms ResNet in both accuracy and test-retest reliability. On time-series modeling, ACA outperforms competing methods. Furthermore, NODE with ACA can incorporate physical knowledge to achieve better accuracy.

Author Information

Juntang Zhuang (Yale University)
Nicha Dvornek (Yale University)
Xiaoxiao Li (Yale University)
Sekhar Tatikonda (Yale)
Xenophon Papademetris (Yale University)
James Duncan (Yale University)

More from the Same Authors

  • 2021 Workshop: Interpretable Machine Learning in Healthcare »
    Yuyin Zhou · Xiaoxiao Li · Vicky Yao · Pengtao Xie · DOU QI · Nicha Dvornek · Julia Schnabel · Judy Wawira · Yifan Peng · Ronald Summers · Alan Karthikesalingam · Lei Xing · Eric Xing