DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Xuxi Chen, Tianlong Chen, Yu Cheng, Wei Chen, Zhangyang Wang, Ahmed Awadallah
ACL 2023 | May 2023
Reducing AI’s Carbon Footprint
Pursuing more efficient AI
Publications
-
-
-
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
-
What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression
Mengnan Du, Subhabrata (Subho) Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Awadallah
EACL | April 2023
-
-
-
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
-
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
-
Measuring the Carbon Intensity of AI in Cloud Instances
Remi Tachet des Combes, Erika Odmark, Will Buchanan
ACM Conference on Fairness, Accountability, and Transparency | June 2022
-
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
-
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
ICLR 2022 | April 2022
-
-
-
The Elastic Lottery Ticket Hypothesis
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
35th Annual Conference on Neural Information Processing System (NeurIPS 2021) | December 2021
-
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
-
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Atlas Wang, Jingjing Liu
ACL-IJCNLP 2021 | August 2021
-
Hant: Hardware-Aware Network Transformation
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, Arash Vahdat
July 2021
arXiv | July 2021
-
XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
Subhabrata (Subho) Mukherjee, Ahmed Awadallah, Jianfeng Gao
ArXiv | June 2021
arXiv:2106.04563
-
Initialization and Regularization of Factorized Neural Layers
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, Nicolo Fusi
2021 International Conference on Learning Representations | May 2021
-
-
-
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Subhabrata (Subho) Mukherjee, Ahmed Awadallah
Annual Conference of the Association for Computational Linguistics (ACL 2020) | July 2020
-
-
-
Probabilistic Neural Architecture Search
Francesco Paolo Casale, Jonathan Gordon, Nicolo Fusi
February 2019
arXiv | February 2019
-
-
-
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Xuxi Chen, Tianlong Chen, Yu Cheng, Wei Chen, Zhangyang Wang, Ahmed Awadallah
ACL 2023 | May 2023
-
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
-
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
-
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
-
Measuring the Carbon Intensity of AI in Cloud Instances
Remi Tachet des Combes, Erika Odmark, Will Buchanan
ACM Conference on Fairness, Accountability, and Transparency | June 2022
-
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
-
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
ICLR 2022 | April 2022
-
The Elastic Lottery Ticket Hypothesis
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
35th Annual Conference on Neural Information Processing System (NeurIPS 2021) | December 2021
-
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
-
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Atlas Wang, Jingjing Liu
ACL-IJCNLP 2021 | August 2021
-
Hant: Hardware-Aware Network Transformation
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, Arash Vahdat
July 2021
arXiv | July 2021
-
Initialization and Regularization of Factorized Neural Layers
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, Nicolo Fusi
2021 International Conference on Learning Representations | May 2021
-
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Subhabrata (Subho) Mukherjee, Ahmed Awadallah
Annual Conference of the Association for Computational Linguistics (ACL 2020) | July 2020
-
Probabilistic Neural Architecture Search
Francesco Paolo Casale, Jonathan Gordon, Nicolo Fusi
February 2019
arXiv | February 2019
-
-
-
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
-
What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression
Mengnan Du, Subhabrata (Subho) Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Awadallah
EACL | April 2023
-
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
-
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
-
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
-
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
ICLR 2022 | April 2022
-
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Atlas Wang, Jingjing Liu
ACL-IJCNLP 2021 | August 2021
-
XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
Subhabrata (Subho) Mukherjee, Ahmed Awadallah, Jianfeng Gao
ArXiv | June 2021
arXiv:2106.04563
-
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Subhabrata (Subho) Mukherjee, Ahmed Awadallah
Annual Conference of the Association for Computational Linguistics (ACL 2020) | July 2020
-
-
-
Measuring the Carbon Intensity of AI in Cloud Instances
Remi Tachet des Combes, Erika Odmark, Will Buchanan
ACM Conference on Fairness, Accountability, and Transparency | June 2022
-
-
-
The Elastic Lottery Ticket Hypothesis
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
35th Annual Conference on Neural Information Processing System (NeurIPS 2021) | December 2021
-
Hant: Hardware-Aware Network Transformation
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, Arash Vahdat
July 2021
arXiv | July 2021
-
-
-
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
-
-
-
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
-
-
-
Initialization and Regularization of Factorized Neural Layers
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, Nicolo Fusi
2021 International Conference on Learning Representations | May 2021
-
-
-
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Xuxi Chen, Tianlong Chen, Yu Cheng, Wei Chen, Zhangyang Wang, Ahmed Awadallah
ACL 2023 | May 2023
-
AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Ganesh Jawahar, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Awadallah, Sébastien Bubeck, Jianfeng Gao
ACL | May 2023
-
What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression
Mengnan Du, Subhabrata (Subho) Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Awadallah
EACL | April 2023
-
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Awadallah, Jianfeng Gao
NeurIPS 2022 | December 2022
-
Measuring the Carbon Intensity of AI in Cloud Instances
Remi Tachet des Combes, Erika Odmark, Will Buchanan
ACM Conference on Fairness, Accountability, and Transparency | June 2022
-
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
ICLR 2022 | April 2022
-
The Elastic Lottery Ticket Hypothesis
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
35th Annual Conference on Neural Information Processing System (NeurIPS 2021) | December 2021
-
AsyMo: Scalable and Efficient Deep-Learning Inference on Asymmetric Mobile CPUs
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, Fengyuan Xu
The 27th Annual International Conference On Mobile Computing And Networking (MobiCom’21) | October 2021
-
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Atlas Wang, Jingjing Liu
ACL-IJCNLP 2021 | August 2021
-
Initialization and Regularization of Factorized Neural Layers
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, Nicolo Fusi
2021 International Conference on Learning Representations | May 2021
-
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Subhabrata (Subho) Mukherjee, Ahmed Awadallah
Annual Conference of the Association for Computational Linguistics (ACL 2020) | July 2020
-
-
-
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
Yaqing Wang, Sahaj Agarwal, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
EMNLP | December 2022
-
LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners
Yaqing Wang, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao
NAACL 2022 | April 2022
-
XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
Subhabrata (Subho) Mukherjee, Ahmed Awadallah, Jianfeng Gao
ArXiv | June 2021
arXiv:2106.04563
-
-
-
Hant: Hardware-Aware Network Transformation
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, Arash Vahdat
July 2021
arXiv | July 2021
-
Probabilistic Neural Architecture Search
Francesco Paolo Casale, Jonathan Gordon, Nicolo Fusi
February 2019
arXiv | February 2019
-