Skip to content

    microsoft/MPNet

    Repository files navigation

    MPNet

    MPNet: Masked and Permuted Pre-training for Language Understanding, by Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu, is a novel pre-training method for language understanding tasks. It solves the problems of MLM (masked language modeling) in BERT and PLM (permuted language modeling) in XLNet and achieves better accuracy.

    News: We have updated the pre-trained models now.

    Supported Features

    • A unified view and implementation of several pre-training models including BERT, XLNet, MPNet, etc.
    • Code for pre-training and fine-tuning for a variety of language understanding (GLUE, SQuAD, RACE, etc) tasks.

    Installation

    We implement MPNet and this pre-training toolkit based on the codebase of fairseq. The installation is as follow:

    pip install --editable pretraining/
    pip install pytorch_transformers==1.0.0 transformers scipy sklearn
    

    Pre-training MPNet

    Our model is pre-trained with bert dictionary, you first need to pip install transformers to use bert tokenizer. We provide a script encode.py and a dictionary file dict.txt to tokenize your corpus. You can modify encode.py if you want to use other tokenizers (like roberta).

    1) Preprocess data

    We choose WikiText-103 as a demo. The running script is as follow:

    wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip
    unzip wikitext-103-raw-v1.zip
    
    for SPLIT in train valid test; do \
        python MPNet/encode.py \
            --inputs wikitext-103-raw/wiki.${SPLIT}.raw \
            --outputs wikitext-103-raw/wiki.${SPLIT}.bpe \
            --keep-empty \
            --workers 60; \
    done
    

    Then, we need to binarize data. The command of binarizing data is following:

    fairseq-preprocess \
        --only-source \
        --srcdict MPNet/dict.txt \
        --trainpref wikitext-103-raw/wiki.train.bpe \
        --validpref wikitext-103-raw/wiki.valid.bpe \
        --testpref wikitext-103-raw/wiki.test.bpe \
        --destdir data-bin/wikitext-103 \
        --workers 60
    

    2) Pre-train MPNet

    The below command is to train a MPNet model:

    TOTAL_UPDATES=125000    # Total number of training steps
    WARMUP_UPDATES=10000    # Warmup the learning rate over this many updates
    PEAK_LR=0.0005          # Peak learning rate, adjust as needed
    TOKENS_PER_SAMPLE=512   # Max sequence length
    MAX_POSITIONS=512       # Num. positional embeddings (usually same as above)
    MAX_SENTENCES=16        # Number of sequences per batch (batch size)
    UPDATE_FREQ=16          # Increase the batch size 16x
    
    DATA_DIR=data-bin/wikitext-103
    
    fairseq-train --fp16 $DATA_DIR \
        --task masked_permutation_lm --criterion masked_permutation_cross_entropy \
        --arch mpnet_base --sample-break-mode complete --tokens-per-sample $TOKENS_PER_SAMPLE \
        --optimizer adam --adam-betas '(0.9,0.98)' --adam-eps 1e-6 --clip-norm 0.0 \
        --lr-scheduler polynomial_decay --lr $PEAK_LR --warmup-updates $WARMUP_UPDATES --total-num-update $TOTAL_UPDATES \
        --dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 \
        --max-sentences $MAX_SENTENCES --update-freq $UPDATE_FREQ \
        --max-update $TOTAL_UPDATES --log-format simple --log-interval 1 --input-mode 'mpnet'
    

    Notes: You can replace arch with mpnet_rel_base and add command --mask-whole-words --bpe bert to use relative position embedding and whole word mask.

    Notes: You can specify --input-mode as mlm or plm to train masked language model or permutation language model.

    Pre-trained models

    We have updated the final pre-trained MPNet model for fine-tuning.

    You can load the pre-trained MPNet model like this:

    from fairseq.models.masked_permutation_net import MPNet
    mpnet = MPNet.from_pretrained('checkpoints', 'checkpoint_best.pt', 'path/to/data', bpe='bert')
    assert isinstance(mpnet.model, torch.nn.Module)

    Fine-tuning MPNet on down-streaming tasks

    Acknowledgements

    Our code is based on fairseq-0.8.0. Thanks for their contribution to the open-source commuity.

    Reference

    If you find this toolkit useful in your work, you can cite the corresponding papers listed below:

    @article{song2020mpnet,
        title={MPNet: Masked and Permuted Pre-training for Language Understanding},
        author={Song, Kaitao and Tan, Xu and Qin, Tao and Lu, Jianfeng and Liu, Tie-Yan},
        journal={arXiv preprint arXiv:2004.09297},
        year={2020}
    }
    

    Related Works

    About

    MPNet: Masked and Permuted Pre-training for Language Understanding https://arxiv.org/pdf/2004.09297.pdf

    Resources

    License

    Code of conduct

    Security policy

    Stars

    Watchers

    Forks

    Releases

    No releases published

    Packages

    No packages published

    Contributors 3

    •  
    •  
    •  
    主站蜘蛛池模板: 国产精品女同一区二区| 熟女精品视频一区二区三区| 日本v片免费一区二区三区| 国产成人久久精品麻豆一区| 日本一区午夜爱爱| 中文无码精品一区二区三区| av无码人妻一区二区三区牛牛| 国语精品一区二区三区| 亚洲国产成人精品无码一区二区| 亚洲一区二区三区无码国产| 麻豆精品人妻一区二区三区蜜桃| 国产福利一区二区精品秒拍| 国产一区三区二区中文在线| 亚洲视频一区在线播放| 无码人妻aⅴ一区二区三区| 丰满人妻一区二区三区视频53| 国产精品日本一区二区不卡视频 | 亚洲AV无码第一区二区三区| 国产韩国精品一区二区三区久久| 香蕉久久ac一区二区三区| 免费萌白酱国产一区二区三区 | 久久毛片一区二区| 一区二区三区在线视频播放| 色窝窝无码一区二区三区色欲 | 亚洲国产成人久久一区WWW| 中文字幕精品一区二区日本| 日本精品一区二区三区视频| 久久99精品波多结衣一区| 成人丝袜激情一区二区| 亚洲毛片αv无线播放一区| 精品无码国产一区二区三区51安| 中文字幕日本一区| 在线视频国产一区| 日韩免费无码视频一区二区三区| 狠狠爱无码一区二区三区| 麻豆国产一区二区在线观看| 成人精品一区二区三区不卡免费看| 久久精品成人一区二区三区 | 亚洲乱码一区二区三区国产精品 | 久草新视频一区二区三区| 波多野结衣一区在线|